Member of Technical Staff - Research Engineering, Evaluation

About the role

This role is centered on the design and implementation of a comprehensive evaluation system for our research organization. You will be responsible for developing the foundational tools and methodologies that enable precise measurement and comparison of our large physics foundation model. Your contributions will establish the standard framework that allows every research team to assess model advancements with confidence and consistency. The position requires a deep commitment to building robust, scalable, and interpretable evaluation processes from the ground up. You will work at the intersection of software engineering and scientific measurement, ensuring that data tells a clear and reliable story. The success of this role is defined by the accuracy, usability, and adoption of the evaluation systems you create. Ultimately, you will empower the entire research organization to validate improvements and drive innovation effectively.

Key facts

What you'll do

• Architect a central, reusable evaluation framework that serves as the foundational layer for all research teams across the organization.
• Design and implement end-to-end evaluation pipelines that ingest model outputs, apply transformations, and produce structured, quantitative assessments.
• Curate and maintain benchmark suites and baseline results that provide a consistent reference point for measuring progress over time.
• Engineer advanced visualization and interactive dashboard tools that translate complex evaluation metrics into clear, actionable insights for researchers.
• Integrate robust statistical methods and testing protocols to ensure observed performance gains are genuine and not attributable to randomness or noise.
• Partner closely with research scientists and domain experts to identify, define, and automate domain-specific performance metrics for diverse project areas.
• Optimize data workflows for scalability and reliability, ensuring that evaluation processes can handle large volumes of experimental data efficiently.
• Document evaluation methodologies, assumptions, and results to maintain transparency and enable reproducibility across all research initiatives.
• Lead the implementation of monitoring systems that track model behavior and evaluation health in near real-time, highlighting anomalies or regressions.
• Contribute to the open source ecosystem where appropriate, releasing tools and frameworks that can benefit the broader scientific community.
• Conduct code reviews and provide technical mentorship to junior engineers, fostering a culture of quality and best practices in evaluation engineering.
• Collaborate with product and research stakeholders to translate high-level objectives into concrete evaluation criteria and success metrics.
• Iterate on existing evaluation tools based on user feedback, improving usability, performance, and the overall researcher experience.
• Stay current with advancements in evaluation theory, measurement techniques, and visualization strategies to continuously enhance the system.

Requirements

• Demonstrated strong software engineering skills with a proven track record of constructing large-scale data and evaluation pipelines that are maintainable and efficient.
• Showcased ability to convert complex research outputs, model predictions, and experimental data into reliable metrics, benchmarks, and visual representations.
• Solid understanding of probability theory and statistical inference, with the capacity to design evaluation methods that accurately distinguish signal from noise.
• Full-stack development expertise encompassing both robust backend data pipelines and intuitive frontend visualization components.
• Ownership of project deliverables, driving tasks from initial requirements gathering through architecture design, implementation, testing, and independent execution.
• Experience working with high-dimensional data typical in physics-based models, including the ability to derive meaningful summaries and insights.
• Proficiency in writing clean, modular, and well-tested code that can be integrated into a larger, collaborative software ecosystem.
• Comfort working in a fast-paced research environment where evaluation criteria evolve alongside scientific discovery and experimentation.

Nice to have

There are no preferred items specified for this role beyond the core requirements and technical skills outlined in the job description.

Practical notes

This is a full-time position based in San Francisco. The role requires immediate availability and a long-term commitment to building foundational evaluation infrastructure. No specific visa sponsorship details or application deadlines are provided in the current job description. The engagement is full-time, and the hours are standard business hours aligned with the San Francisco office schedule.