Senior Lead Data Engineer

About the role

This position involves developing and expanding Brillio's data and analytics infrastructure. You will focus on creating, building, and managing efficient, dependable, and cost-effective data pipelines and platforms within a cloud environment. The role requires a hands-on approach to designing data systems from scratch.

Key facts

What you'll do

• Design, build, and operate cloud-based data pipelines that are scalable, reliable, and performant.
• Create and maintain large, complex datasets that fulfill both functional and non-functional business needs.
• Implement database management best practices, including logical data modeling, physical database optimization, and data security.
• Develop and improve data quality checks to ensure the accuracy, integrity, and reliability of data pipelines.
• Collaborate with Data & Analytics experts to continuously enhance data platform capabilities and usability.
• Support various teams, systems, and products by providing consistent and optimized data delivery architectures.
• Participate in on-call support rotations and proactively resolve user issues based on severity and business impact.
• Contribute to improving operational processes, monitoring, and the reliability of data platforms.

Requirements

• 5+ years of experience designing, building, and operating data pipelines for large and complex datasets.
• Strong experience with structured and unstructured data, including JSON.
• Extensive hands-on experience with advanced SQL for data transformation and analysis.
• 3+ years of experience with Python and PySpark for large-scale data processing.
• Hands-on experience transforming and processing data using Databricks.
• 3+ years of experience building data solutions on AWS.
• Experience using Git and CI/CD practices for data and platform development.
• Strong collaboration skills to support data engineers, analysts, data scientists, and software engineers.
• Self-directed, ownership-driven mindset with the ability to manage multiple concurrent priorities.

Nice to have

• Experience with Power BI or similar business intelligence and visualization tools.
• Familiarity with modern data lake and data warehouse architectures.
• Experience implementing observability, monitoring, and alerting for data pipelines.
• Exposure to streaming data technologies or near real-time data processing.
• Experience in a fast-growing, platform-focused or SaaS environment.

Skills & tools

• Athena
• SNS
• SQS
• CloudWatch
• Kinesis
• Redshift
• AWS Data Engineering Advanced: Associate Data Engineer specialization