Computational Linguist, AI Evaluation

About the role

TwelveLabs is at the forefront of creating an intelligence layer for video data on a global scale, enabling machines to understand visual, auditory, and motion-related information. We are looking for a dedicated professional to join our Machine Learning Data Operations team, focusing on the management of video-language data and the evaluation of model performance. This role is integral to ensuring that our models are effectively trained and evaluated, contributing to the advancement of our AI capabilities.

Key facts

What you'll do

• Design and execute evaluation frameworks for video-language models, translating research objectives into standardized benchmarks that can be consistently applied.
• Create data processing pipelines that transform raw usage statistics into meaningful insights, helping to pinpoint areas where model performance can be enhanced.
• Oversee extensive data collection and labeling initiatives, focusing on automating repetitive tasks to boost overall operational efficiency.
• Guide external vendor partners to ensure the delivery of high-quality results by providing clear instructions and establishing effective feedback mechanisms.
• Work in close collaboration with engineering and research teams to prioritize data needs and communicate findings through comprehensive dashboards.
• Conduct thorough analyses of complex datasets to generate clear documentation and develop annotation guidelines that facilitate the evaluation process.
• Implement strategies to streamline the lifecycle of video-language data, ensuring that all aspects of data handling are optimized for performance.
• Monitor and report on model performance metrics, identifying trends and areas for improvement to inform future development efforts.

Requirements

• A minimum of 5 years of experience in a data operations role focused on artificial intelligence.
• Demonstrated expertise in designing and managing large-scale data initiatives, encompassing collection, labeling, and subsequent processing.
• Hands-on experience in constructing model evaluation pipelines, including frameworks for human evaluation and automated scoring systems.
• Strong analytical skills with the ability to interpret complex datasets and produce clear, actionable documentation.
• Proficiency in Python and familiarity with automation tools such as agentic coding.
• Excellent project management abilities, capable of juggling multiple projects simultaneously while maintaining high standards of quality.
• A foundational understanding of large language models (LLMs), vision-language models (VLMs), and multimodal AI technologies.

Nice to have

• Experience in data collection and labeling specifically tailored for multimodal language models.
• Previous collaboration with research scientists and engineers in a technical environment.
• A track record of planning and implementing new data tools or labeling systems to enhance operational workflows.
• Knowledge or interest in video-related fields such as advertising, sports, or content creation, which can provide valuable context for the role.

Skills & tools

• Technical: Proficient in Python, FFMPEG, and AWS services (including S3 and EC2)
• Project Management: Familiarity with tools such as Linear, Notion, Google Suite, and Encord for effective project tracking and collaboration

Practical notes

• Employees enjoy comprehensive benefits, including full health, dental, and vision insurance, along with flexible paid time off and parental leave.
• The company supports continuous learning and development through an annual stipend, as well as a monthly wellness allowance to promote employee well-being.
• For those based in San Francisco, a transportation stipend is provided, along with complimentary lunch and dinner options.
• The office observes a closure during the week of Christmas and New Year's, allowing employees to enjoy the holiday season with their families.

Join us at TwelveLabs and be part of a dynamic team that is shaping the future of AI and video data interpretation. Your expertise in computational linguistics and data operations will play a crucial role in driving our mission forward.

About the company

At Twelve Labs, we are pioneering the development of multimodal foundation models that have the ability to comprehend videos just like humans do. Our models have redefined the standards in video-language modeling, empowering us with more intuitive and far-reaching capabilities, and fundamentally transforming the way we interact with and analyze various forms of media.