Senior AI Data Operations Manager

About the role

HackerRank is on the lookout for a Senior AI Data Operations Manager to lead the development of our extensive data creation pipeline for large language models (LLMs). This role is pivotal in managing relationships with vendors, working closely with engineering teams to evaluate model performance, and ensuring that the data utilized for training adheres to high-quality standards. You will oversee the AI data pipeline that supports more than ten AI features within our platform, ensuring seamless operations and data integrity.

In this capacity, you will play a crucial role in connecting various teams, including machine learning engineers, internal annotators, and external vendors, translating technical specifications into effective data workflows. Your focus will also be on automating processes to boost efficiency and streamline overall operations, making a significant impact on our AI initiatives.

Key facts

What you'll do

• Oversee the complete lifecycle of AI feature development, managing everything from data sourcing to quality assurance and vendor relationship management.
• Design and implement automated workflows for critical processes, including vendor briefings, verification of annotations, and tracking of data.
• Evaluate model performance metrics such as precision, recall, and edge case identification to guide focused data creation strategies.
• Collaborate with Engineering Managers, Machine Learning Engineers, and Product Managers to forecast model data needs and incorporate insights into pipeline improvements.
• Maintain comprehensive documentation for data batches, annotation protocols, and standard operating procedures for vendors.
• Facilitate communication between technical and non-technical stakeholders to ensure alignment on project goals and data quality expectations.
• Identify and implement opportunities for process automation to enhance operational efficiency within the AI data pipeline.
• Monitor and report on the performance of AI features, providing insights to inform future data strategies and operational adjustments.

Requirements

• A minimum of 5 years of experience in technical program or operations management, with at least 3 years dedicated to AI/ML data operations and vendor management in a tech-focused environment.
• Strong proficiency in AI tools, with the ability to discuss automation possibilities and model behavior with both technical and non-technical audiences.
• Hands-on experience in developing or utilizing AI agents, complemented by exceptional organizational skills to juggle multiple vendors, deadlines, and data pipelines simultaneously.
• Solid understanding of LLM evaluation principles and the capability to engage in data quality discussions with engineering teams.
• Excellent communication skills, both written and verbal, to effectively convey complex concepts to diverse audiences.
• A proactive approach to problem-solving, with the ability to adapt to changing priorities and work in a fast-paced environment.

Nice to have

• Experience with natural language processing (NLP) or AI training data pipelines, enhancing your ability to contribute to our projects.
• Familiarity with annotation tools such as CVAT, Label Studio, Scale AI, Surge AI, or similar platforms, which can aid in streamlining our processes.
• A background in a startup or rapidly evolving environment, providing insights into agile methodologies and process adaptation.

Skills & tools

• Strong analytical capabilities, project management skills, and a solid understanding of AI and machine learning principles.
• Familiarity with data quality metrics and best practices in data management, ensuring that our data meets the highest standards.
• Proficiency in using project management software and data analysis tools to track progress and optimize workflows.

Practical notes

HackerRank is proud to be an equal opportunity employer. We are dedicated to fostering a workplace that is free from discrimination based on race, religion, national origin, gender identity or expression, sexual orientation, age, marital status, veteran status, or disability. All applications will be handled confidentially in accordance with EEO guidelines. We encourage individuals from diverse backgrounds to apply and join our team in shaping the future of AI data operations.