Senior LLMOps Engineer

About the role

You will architect and operate the LLMOps infrastructure that underpins every Heidi product decision. This role owns the critical layer that transforms raw model output into clinically reliable intelligence. You will ensure that every deployment is observable, every incident is traceable, and every model earns its place in production. You will build the feedback flywheel that turns real-world clinician interactions into model improvements. This position requires a practitioner who thrives on ambiguity and delivers production-grade systems under pressure. You will partner directly with researchers to embed observability into the core of the model stack. Your work will directly determine whether Heidi can scale its impact without sacrificing safety or reliability.

Key facts

What you'll do

• Construct a deployment health dashboard that provides live visibility into every model in production, including health metrics, monitoring, and proactive incident alerting that flags issues before they impact clinicians.
• Engineer a complete session-to-model lineage system that enables incidents to be traced from degraded sessions to affected user profiles, specific model IDs, and root causes within minutes.
• Architect an improvement flywheel that consumes Intercom tickets and qualitative CSAT feedback, using AI agents to pinpoint the exact sessions that inform product and model decisions.
• Retrieve and synthesize the complete execution trace for every flagged session, delivering summaries and context that allow non-engineering stakeholders to act on complex model behavior.
• Implement a closed-loop process that filters high-value feedback into training data, triggers model improvements, and monitors post-deployment performance to ensure every release is measurably better.
• Establish per-model profit and loss accountability by measuring revenue against inference cost, and use those unit economics to guide model selection and deployment strategy.
• Elevate the standard of LLMOps at Heidi by importing and adapting best practices from the most mature AI organizations, including tracing, evaluation, and model incident response protocols.
• Collaborate across the model team with researchers and engineers responsible for ASR, note generation, Evidence, and Dictate models to ensure observability is designed in rather than patched on.

Requirements

• You have spent the last 2-3 years hands-on in an LLMOps role, building observability, tracing, evaluation, and feedback systems around production LLMs and owning them through real incidents.
• That experience originates from an AI company operating at or ahead of Heidi's maturity, most likely in the US or China, where LLMOps practice is deeply established and sophisticated.
• You have a proven track record of shipping the core systems this role owns, including monitoring and alerting platforms such as Datadog, distributed tracing across multi-step LLM pipelines, and session and event data models that remain robust at scale.
• You have direct experience building with LLMs, not merely operating them, demonstrated by your ability to leverage agents for triaging feedback and automatically matching tickets to sessions.
• You are comfortable integrating cost and revenue data to construct per-model unit economics that provide leadership with actionable insights.
• You operate at a senior level, taking ambiguous mandates, designing robust systems, and driving them through to stable production operation without needing a PhD to validate impact.
• You care about evidence, trust the patient, and understand that the stakes of your work are measured in human outcomes, not just technical metrics.
• You are prepared to commit to the demands of this role, including the required hours and the expectation to lead complex initiatives that protect and enhance the clinician-patient relationship.

Nice to have

• A broader engineering foundation that spans backend, data platform, or ML infrastructure, providing depth beyond LLMOps specialization.
• Experience wiring product feedback tools like Intercom into engineering systems to automate responses and insights.
• Prior time working in healthcare or other regulated, safety-critical domains where reliability and compliance are non-negotiable.

Practical notes

This role is full-time based in Melbourne.