Data Engineer

About the role

Greenlight is dedicated to providing a comprehensive platform that enables families to manage their finances, safety, and daily responsibilities through a suite of innovative tools, including debit cards and GPS trackers. We are actively seeking a talented Data Engineer to join our Data Platform team. In this role, you will be instrumental in designing, developing, and maintaining the infrastructure that supports our product offerings and business operations. Your work will involve solving complex data challenges to ensure our systems are reliable, scalable, and efficient, especially as we serve over 6.5 million family members across various regions. You will collaborate closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to translate business requirements into robust technical solutions. This position offers an exciting opportunity to work on large-scale data systems, contribute to product growth, and help shape the future of our data infrastructure.

Key facts

What you'll do

• Develop, implement, and maintain scalable and reliable data pipelines using Apache Airflow to orchestrate workflows that move and transform data into Snowflake and Databricks environments.
• Design, build, and optimize data models within Snowflake that support reporting, analytics, and machine learning initiatives, ensuring data is structured for efficient querying and analysis.
• Automate cloud infrastructure provisioning and management on AWS using Terraform, applying infrastructure as code principles to streamline resource deployment and updates.
• Implement comprehensive monitoring, alerting, and data quality checks to verify the accuracy, completeness, and timeliness of data flowing through pipelines.
• Continuously analyze and improve workflow performance to optimize costs, reduce latency, and handle increasing data volumes as the platform scales.
• Investigate pipeline failures, troubleshoot technical issues, and implement fixes to ensure minimal downtime and data integrity.
• Collaborate with teams across India and the United States to understand business needs and translate them into technical data solutions, including data ingestion, transformation, and storage strategies.
• Maintain detailed documentation of data infrastructure, pipelines, data models, and processes to facilitate knowledge sharing and onboarding within the team.
• Participate in code reviews, adhere to best practices in software development, and contribute to the evolution of the data engineering standards and practices.
• Support data governance initiatives by ensuring compliance with security, privacy, and regulatory requirements, including ITAR if applicable.
• Assist in the evaluation and integration of new data tools and technologies to enhance the platform's capabilities.
• Contribute to the development of automated testing frameworks for data pipelines to ensure robustness and reliability.
• Engage in continuous learning to stay updated with the latest trends, tools, and best practices in data engineering and cloud infrastructure.

Requirements

• Minimum of 2 years of professional experience in data engineering roles, with a proven track record of building and maintaining large-scale data pipelines.
• Strong proficiency in Python, with experience in scripting, automation, and data processing tasks.
• Hands-on experience with workflow orchestration tools, particularly Apache Airflow, including creating, scheduling, and monitoring DAGs.
• Advanced SQL skills, with extensive experience working with cloud data warehouses such as Snowflake, including data modeling, query optimization, and performance tuning.
• Familiarity with AWS cloud services such as S3, Lambda, EC2, and IAM, and experience integrating these services into data workflows.
• Ability to work effectively with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver solutions.
• Demonstrated ownership of data pipelines, with a focus on reliability, performance, and scalability.
• Strong problem-solving skills, with the ability to troubleshoot complex technical issues and implement effective solutions.
• Experience with infrastructure automation tools like Terraform, including writing and managing infrastructure as code.
• Excellent communication skills, capable of documenting processes, sharing knowledge, and collaborating across teams.
• Knowledge of data security and compliance standards relevant to handling sensitive information, including ITAR if applicable.
• Ability to work in a fast-paced environment, managing multiple priorities and deadlines effectively.
• A proactive mindset with a focus on continuous improvement and learning.

Nice to have

• Experience with dbt (data build tool) for transformation workflows and analytics engineering.
• Knowledge of Databricks platform, Spark optimization techniques, and Delta Lake architecture.
• Practical experience with CI/CD pipelines for data systems, including automated testing and deployment.
• Familiarity with data modeling techniques such as star schemas, snowflake schemas, and dimensional modeling.
• Exposure to real-time data processing frameworks and streaming data architectures.
• Understanding of data governance, privacy, and security best practices.
• Experience working in an environment with strict compliance requirements, such as ITAR.
• Knowledge of containerization technologies like Docker and orchestration tools like Kubernetes.
• Experience with version control systems such as Git.

Skills & tools

• Python, SQL, Airflow, Snowflake, Databricks, AWS (S3, Lambda, EC2, IAM), Terraform.

Practical notes

• Greenlight is committed to fostering an inclusive and diverse workplace and is an equal opportunity employer.
• We welcome applications from candidates of all backgrounds and experiences.
• If you require reasonable accommodations during the hiring process, please contact your recruiter or email accomodations@greenlight.me.
• This role is based on-site in Bengaluru, Karnataka, and involves working at our office location.
• Candidates should be prepared to work in an environment that values collaboration, innovation, and continuous improvement.
• Our team values transparency, integrity, and a proactive approach to problem-solving.
• We look forward to welcoming a motivated and skilled data engineer to help us build the future of family financial management.