Applied AI Researcher, Agent Systems & Evaluation

About the role You own the evaluation pipeline for autonomous agent systems, ensuring real workflow integrity from intake to shipped output at Nuro. Decisions rest on evidence, not intuition, as you close the loop between production behavior and model improvement. You amplify engineering impact by making unattended autonomous work reliably measurable and safe. About the team Frontier models are fungible. Any team can rent the same intelligence we can, and the model we build on today will be replaced within a month. What is not fungible is the infrastructure that decides whether an autonomous system's output can be trusted - evaluation, verification, and the discipline to gate on evidence instead of impressions. Nuro has spent a decade building exactly that discipline for a robot that drives on public roads, and this team turns it inward: we build the platform that lets AI agents operate autonomously inside Nuro's own engineering organization, under the same standard of proof we apply to the vehicle. Our mandate is to amplify the output of every engineer and researcher at Nuro by 100x. Not a better IDE, not a faster build - a change in what a single person can attempt. That number is a target, not a claim, and reaching it depends on one thing above all: autonomous work has to be trustworthy enough to run unattended. So our central ambition is to build the most rigorous closed-loop evaluation system for AI work anywhere. use follows from trust, and trust follows from measurement. We operate as a startup inside a company that has already shipped a hard thing. Small team, no established playbook, direct access to compute and to the systems we are automating. You will work directly with engineering leadership and the CEO, and the decisions you make will be yours to make rather than yours to implement. About the role Most teams building agents make design decisions by intuition and anecdote. Someone tries a new memory scheme, it feels better, it ships. We think that is the central failure of the field right now, and we are building this team to work the other way: every decision about how our agent systems are constructed should be settled by evidence. You would not be starting from zero. We already operate a substantial agent system in production, a fleet of agents with an extensive library of skills and plugins, integrated into the tools our engineers use daily, serving real users with real work. So every hypothesis you form can be tested against genuine production traffic from your first month. And the system is now complex enough that intuition has stopped being sufficient to improve it, which is precisely why this role exists. Everything this team builds is centered on frontier-lab transformer models. We are not inventing architectures. We are extracting the maximum from the best models that exist, and adapting them ourselves in the narrow places where our data gives us an advantage nobody else has. Your charter has two halves. Make the system perform against real-world data, not public benchmarks or tasks we invented to look good, but our codebase, our infrastructure, and our engineers' actual requests, with all the ambiguity that implies. The gap between benchmark performance and real-task performance is where most agent systems quietly fail. And turn any task into a closed loop: for any workflow an agent takes on, you should be able to say what success looks like, where the evaluation data comes from, how signal is collected, and how results feed the next iteration. Own the evaluation pipeline end to end. Three stages, and the value is in owning all three. Eval data collection. Where ground truth comes from. Mining production traces for labeled outcomes, capturing human accept or edit signal as it happens, building task sets that reflect the real distribution of work rather than the tasks that are easy to score, and knowing when a model-based judge is trustworthy and when it is laundering an assumption. Eval loop construction. Turning a fuzzy objective into a measurement that runs on every change. Noise floors, statistical standards for acceptance, task suites that resist gaming, and experiments designed for production setting. Evaluation architecture. Instrumenting the full agent run, storing traces, defining metrics, and maintaining a registry of tasks with stable failure modes. What you'll do Setting eval data strategy that captures ground truth from production traces and human signals. Designing measurement standards that define success for agent workflows from intake to ship. Guarding quality by filtering noise floors and accepting only statistically valid task suites. Building closed-loop evaluation so every agent workflow turns into a testable, improvable cycle. Partnering with infrastructure teams to align data collection with real engineering tool usage. Shipping evaluation modules that integrate into developer workflows without slowing momentum. Adapting frontier transformer models to extract maximum performance from existing Nuro capabilities. Verifying that real-task outcomes close the gap left by public benchmarks and proxy tests. Requirements You hold a PhD or equivalent experience in related fields with 3-5 years of hands-on work. You have shipped agent systems or evaluation platforms that prove decisions through evidence. You understand transformer architectures and how to adapt them to narrow, high-value domains. You can trace production traffic into structured evaluation data that informs iteration. You communicate findings clearly to both engineers and leadership under tight uncertainty. Nice to have Experience with