Member of Technical Staff, Model Evaluation

About the role

Inception is building the Mercury model, a diffusion-based LLM that achieves five times the speed and efficiency of standard autoregressive models. We are looking for engineers and scientists to build the evaluation infrastructure that defines how we measure and improve our frontier AI performance.

Key facts

What you'll do

• Create and maintain benchmarking systems to assess model performance across various domains.
• Develop quantitative metrics to track model quality, safety, reliability, and regressions.
• Build automated pipelines for evaluation that integrate with our training and deployment processes.
• Perform statistical analysis on model outputs to detect biases, failure modes, and performance gaps.
• Collaborate with product teams to turn real-world use cases into specific evaluation criteria.

Requirements

• BS, MS, or PhD in Computer Science, Statistics, Machine Learning, or a related field.
• Minimum of 2 years of professional experience in applied ML research or model evaluation.
• Proficiency in Python and PyTorch.
• Experience building evaluation metrics and benchmarks for generative AI.
• Understanding of LLM concepts including instruction tuning, RLHF, autoregressive generation, decoding strategies, and in-context learning.
• Background in experimental design, hypothesis testing, and statistics.
• Experience with Docker and Git.
• Ability to communicate complex data findings to stakeholders.

Nice to have

• Experience with human-in-the-loop systems such as red-teaming, pairwise ranking, or Likert-scale annotation.
• Familiarity with alignment and safety evaluation, including hallucination detection and toxicity.
• Knowledge of benchmark suites like BIG-Bench, HELM, MMLU, and HumanEval.
• Experience with cloud-based evaluation infrastructure on AWS, GCP, or Azure.
• Background in MLOps, CI/CD, data engineering, or synthetic data generation.

Skills & tools

• Python, PyTorch, Git, Docker, Cloud platforms (AWS/GCP/Azure).

Practical notes

• Annual base salary range: $200,000 to $350,000 USD, plus equity and benefits.
• Benefits include health, dental, and vision insurance, 401k match, flexible PTO, catered meals, and commuter subsidies.
• Inception is an equal opportunity employer.