Lead Data Engineer - R01568940

About the role

Brillio is seeking a Lead Data Engineer to join our Data Pipeline team. In this role, you will engage with advanced technologies to construct efficient data pipelines and models that support critical business decisions. You will collaborate with various teams to enhance data-driven practices and contribute to the development of scalable data architectures.

Key facts

What you'll do

• Design and implement scalable ETL processes, ensuring data quality and movement.
• Build and maintain data pipelines using technologies such as Databricks, Azure Synapse Analytics, and Azure Data Explorer.
• Work on large-scale lakehouse and warehouse systems that handle both real-time and batch data processing.
• Collaborate with business, design, test, and development teams to optimize data capture and enhance test-driven methodologies.
• Support the development of a high-quality, modern data architecture that meets the needs of the organization.

Requirements

• Minimum of 5 years of experience working with SQL.
• At least 5 years of experience in designing and implementing scalable ETL processes, including data movement and quality assurance tools.
• A minimum of 3 years of experience in modern Big Data Analytics, utilizing Data Lake, Spark, and formats such as Parquet.
• At least 2 years of experience in building cloud-hosted data systems, with a strong preference for Azure.

Nice to have

• Experience in building data pipelines using Azure Databricks, Fabric, or Spark.
• Familiarity with data in delta lake format and working with Azure Data Explorer (Kusto).
• Knowledge of applying AI/ML techniques to data engineering tasks, including feature engineering and model monitoring.
• Experience in preparing datasets for AI applications, including A/B testing and privacy-aware data access.

Skills & tools

• Proficiency in AKS, Event Hub, Azure DevOps, Cosmos DB, and Azure Functions.

Practical notes

• Candidates must be eligible to work in the United States.
• Brillio offers competitive benefits and a supportive work environment that fosters professional growth.
• Interested applicants should submit their resumes by the application deadline.