Staff Engineer (ML Engineer)

About the role

You will ensure the reliability of the software stack that powers Graphcore hardware. This position focuses on benchmarking and validating complex AI systems to guarantee performance before they reach the end user. You own the creation and execution of rigorous validation suites that confirm the integrity of machine learning workflows. This role requires you to act as a quality gatekeeper for numerical correctness and performance across the entire stack. You will investigate subtle discrepancies between theoretical models and their actual hardware execution. Your work will directly influence the confidence of product teams in releasing new features. You are responsible for automating the detection of regressions before they impact external customers. This position demands a meticulous approach to testing and a commitment to uncompromising standards of reliability.

Key facts

What you'll do

• Execute benchmarks using open source models to establish baseline performance metrics and automate these workflows for continuous validation.
• Engineer specialized test suites designed to probe low-level behaviors such as quantization, numerical precision, attention mechanisms, and distributed execution strategies.
• Analyze system logs and performance telemetry to identify regressions, correctness errors, and performance bottlenecks across a wide array of frameworks and deployment environments.
• Interface with hardware engineers to correlate software anomalies with specific architectural features or silicon characteristics.
• Work alongside software developers to refine testing methodologies and improve the robustness of existing validation tools.
• Partner with infrastructure teams to provision and manage the large-scale compute resources required for exhaustive model validation.
• Decompose complex system failures into reproducible test cases that isolate root causes efficiently.
• Document testing procedures and results to create a clear audit trail for compliance and historical reference.
• Evaluate new ML frameworks and libraries to determine their compatibility with Graphcore hardware through proactive experimentation.
• Lead the development of automation scripts that reduce manual effort and increase the coverage of quality assurance activities.
• Synthesize test outcomes into actionable insights that guide product and engineering decisions.
• Ensure all testing activities align with the broader goals of product stability and user satisfaction.
• Contribute to the evolution of best practices for testing machine learning systems at scale.
• Mentor junior engineers on effective strategies for debugging intricate AI workloads.

Requirements

• Proven background in Machine Learning or related software engineering demonstrated through professional experience or significant projects.
• Deep understanding of neural network training, inference, and performance optimization principles in real-world scenarios.
• Proficiency in Python for the purpose of benchmarking, automation, and sophisticated data analysis.
• Experience managing and analyzing ML model experiments to track variables, metrics, and outcomes effectively.
• Strong Linux debugging skills essential for diagnosing issues at the system and application level.
• Practical knowledge of major ML frameworks such as PyTorch, JAX, TensorFlow, or Triton for building reliable test pipelines.
• Ability to read and interpret complex technical documentation to quickly adapt to new technologies.
• Commitment to writing clean, maintainable, and well-documented code for all testing artifacts.

Skills & tools

• Python
• PyTorch
• TensorFlow
• JAX
• Triton
• Linux

Practical notes

• Benefits include unlimited annual leave, up to 5% matched pension, and phantom equity.
• Additional perks include a health cash plan, life assurance, income protection, and optional private medical or dental insurance.
• The office provides free food and an on-site barista.
• Graphcore is part of the SoftBank Group.
• The company supports interview adjustments for candidates who require them.

This role is based in London, United Kingdom, and is offered as a full-time engagement within the ML QA team. The successful candidate will be expected to work primarily from the Graphcore office, collaborating closely with cross-functional teams on a daily basis. The position requires a high degree of self-motivation and the ability to manage complex testing workflows with minimal supervision. You will interact frequently with hardware designers, compiler engineers, and application developers to ensure comprehensive test coverage. The role involves a significant amount of hands-on work with command-line interfaces, scripting, and analysis of large data sets. You must be comfortable navigating fast-paced environments where priorities can shift based on project needs. Strong communication skills are essential for documenting issues and presenting findings to technical audiences. The successful applicant will contribute to the continuous improvement of testing infrastructure and methodologies. This is an opportunity to shape the quality standards for emerging AI hardware and software ecosystems. The company provides a supportive environment where engineers are encouraged to take ownership of their work and drive initiatives forward. Professional development is supported, and the role offers exposure to cutting-edge technology within the SoftBank Group portfolio. Candidates must be legally authorized to work in the United Kingdom without sponsorship, as the position does not offer visa sponsorship. The interview process may include technical assessments and practical coding challenges related to machine learning and system-level debugging. Please ensure you meet the outlined requirements before applying.