Lead Data Engineer

About the role

HackerRank is currently undergoing a significant transformation in its data platform. Following a successful migration from Redshift to StarRocks and Apache Hudi, we have achieved notable reductions in export latencies. With the essential infrastructure now established, our focus is shifting towards the creation of an AI-centric data layer designed to enhance functionalities such as natural language querying for our HackerRank for Work clients. In the position of Lead Data Engineer, you will play a crucial role as an individual contributor, making vital platform decisions and working closely with various teams to implement data-driven features that will drive revenue growth.

Key facts

What you'll do

• Supervise and optimize the data platform, leveraging technologies such as StarRocks (OLAP), Apache Hudi (Data Lake), Trino, Spark, and Apache Ranger to guarantee peak performance, reliability, and security.
• Design and implement an advanced AI-optimized data layer that delivers clean and structured datasets, facilitating natural language querying and AI functionalities for HackerRank for Work.
• Oversee in-product data features, which include exports, insights dashboards, interview analytics, and a self-service Custom Reports interface.
• Develop self-service data pipelines for internal teams, reducing the need for ad-hoc requests and enhancing data accessibility throughout the organization.
• Establish robust data security protocols, including access controls and policies through Apache Ranger, along with implementing confidence-scoring for AI-generated outputs.
• Lead technical design discussions and set engineering standards for the data team to follow.
• Collaborate with product managers and business stakeholders to identify and articulate AI-driven data use cases.
• Mentor junior data engineers and promote best practices within the team to foster a culture of continuous improvement.
• Analyze performance metrics and provide insights to optimize data workflows and processes.
• Participate in code reviews and contribute to the development of documentation for data engineering processes and standards.
• Engage with external partners and vendors to explore new technologies and tools that can enhance our data capabilities.
• Stay updated on industry trends and advancements in data engineering and AI to ensure our platform remains competitive.

Requirements

• A minimum of 6 years of experience in data engineering, with at least 2 years in a senior or leadership capacity.
• Extensive hands-on experience with OLAP databases such as StarRocks, ClickHouse, or Druid.
• Strong expertise in data lake technologies, including Apache Hudi, Iceberg, or Delta Lake.
• Proficient in distributed query engines like Trino or Presto, as well as batch and stream processing using Apache Spark.
• A solid understanding of data security practices, including role-based access control and tools like Apache Ranger.
• Experience working in a hybrid environment that integrates AWS with open-source self-managed solutions.
• Exceptional communication skills, with the ability to explain technical concepts to non-technical stakeholders and manage cross-functional projects independently.
• Proven track record of delivering data-driven solutions that contribute to business objectives.
• Ability to work collaboratively in a fast-paced environment and adapt to changing priorities.

Nice to have

• Experience with AI-related data tasks, such as confidence scoring, agentic pipelines, or vector stores.
• Familiarity with operationalizing AI concepts in a production setting.
• Background in scaling data infrastructure for SaaS or B2B products.
• Knowledge of natural language querying interfaces or the development of data products aimed at end-users.
• Exposure to data governance frameworks and best practices.
• Understanding of machine learning concepts and their application in data engineering.

Skills & tools

• StarRocks, Apache Hudi, Trino, Apache Spark, Apache Ranger, data lakes, OLAP databases, data security, AWS, open-source technologies, machine learning concepts, natural language processing.

Practical notes

HackerRank is committed to being an equal opportunity employer. We strive to create a diverse and inclusive workplace where all individuals are treated with respect and fairness. All applicants will be evaluated without regard to race, religion, national origin, gender identity or expression, sexual orientation, age, marital status, veteran status, or disability. Your application will be handled confidentially in accordance with EEO guidelines.

If you are interested in this opportunity, please submit your application through our official website.