Senior Engineer (ML Engineer)

About the role

You will ensure the reliability and performance of the Graphcore software stack by validating how it interacts with our AI accelerator hardware. This role is centered on the rigorous testing and benchmarking of complex systems to uncover regressions and performance bottlenecks before they reach production deployment. You will own the validation processes that bridge the gap between theoretical model performance and real-world execution on Graphcore processors. The position demands a meticulous approach to system-level testing where software meets specialized silicon. You will work at the intersection of machine learning frameworks and low-level hardware interaction. Your work will directly influence the stability and efficiency of the software tools used by researchers and developers. This is a critical role in maintaining the integrity of the Graphcore software ecosystem. You will be responsible for creating the experiments that prove our hardware performs as expected under diverse and demanding conditions.

Within the ML QA team, you act as the primary gatekeeper for quality assurance, implementing robust methodologies that ensure Graphcore's hardware delivers on its promise. This involves a deep dive into the entire software stack, from high-level model execution down to the compiler and runtime behavior. You will validate that the hardware correctly implements the intended computations defined by the software. The role requires a proactive mindset focused on preventing issues rather than just identifying them. By establishing rigorous testing pipelines, you provide the confidence needed for engineers to ship features rapidly and safely. Your findings will guide critical architectural and software decisions.

Key facts

What you'll do

• Execute open source models and develop automated pipelines for benchmarking to establish consistent performance metrics across diverse workloads.
• Create specific tests for low-level ML operations including quantization, numerical precision, and distributed execution to ensure correctness and compliance with specifications.
• Diagnose correctness issues and performance limits across various frameworks and model subgraphs through systematic investigation and root cause analysis.
• Collaborate with hardware, software, and infrastructure teams to improve testing methodologies and drive higher quality standards across the organization.
• Design and implement experiments that isolate hardware-specific behaviors and validate software optimizations under varying conditions.
• Analyze large datasets of performance metrics to identify trends, anomalies, and potential areas for optimization, translating data into actionable insights.
• Develop and maintain testing frameworks that are robust, scalable, and provide clear diagnostic output for debugging complex interactions.
• Work closely with software engineers to reproduce issues and verify fixes in a timely and efficient manner, minimizing downtime and ensuring stability.
• Contribute to the documentation of test procedures and results to support the wider engineering organization and create a knowledge base for future efforts.
• Drive the creation of automated test suites that integrate seamlessly into the existing CI/CD pipeline, ensuring continuous validation.
• Investigate the interaction between the Graphcore compiler, runtime, and system libraries to uncover hidden inefficiencies and potential improvements.
• Provide clear and actionable reports to stakeholders regarding test outcomes, risks, and recommendations for deployment, aiding in informed decision-making.

Requirements

• Proven background in Machine Learning or related software engineering, demonstrating a history of delivering complex software solutions.
• Proficiency in Python for benchmarking, automation, and data analysis, leveraging libraries and frameworks effectively.
• Understanding of neural network training, inference, and performance trade-offs, including the impact of precision and architecture.
• Experience with ML frameworks such as JAX, PyTorch, TensorFlow, or Triton, showing the ability to work with diverse toolchains.
• Ability to debug complex issues within Linux environments, utilizing command-line tools and scripting for problem-solving.
• Experience designing and running ML model experiments, from hypothesis to conclusion, with a focus on empirical validation.
• Strong grasp of software development principles and best practices, ensuring maintainable and efficient test code.
• Excellent problem-solving skills and a methodical approach to complex system failures, capable of navigating ambiguity.

Skills & tools

• Python
• PyTorch
• TensorFlow
• JAX
• Triton
• Linux

Practical notes

Benefits include unlimited annual leave, up to 5% matched pension, and phantom equity. The package also covers a health cash plan, income protection, life assurance, and optional private medical or dental insurance. The office provides free food and an on-site barista. Graphcore is part of the SoftBank Group and supports flexible working arrangements.