DevOps Engineer - AI Runtime Services

About the role

The AI Runtime Services team manages the infrastructure supporting all artificial intelligence features across the Demandbase platform. You will build and maintain the core services that allow product teams to deploy LLM-based tools, focusing on gateway management, cost tracking, and operational reliability.

Key facts

What you'll do

• Manage the LLM gateway, which serves as the primary access point for all model providers, including routing, rate limiting, and failover.
• Own the reliability of the runtime environment by managing on-call rotations, incident response, and post-mortem analysis.
• Implement cost control measures to track, report, and reduce spending on model usage per team and per model.
• Develop observability systems to monitor traces, spans, and billing data for all AI-driven features.
• Build infrastructure for experimentation and evaluation, providing teams with the tools to test and score model performance.
• Configure caching layers to reduce latency and deploy guardrails for PII handling and policy enforcement.

Requirements

• Strong background in production infrastructure using Kubernetes, AWS, and Terraform.
• Experience with GitOps workflows.
• Proficiency in Python for developing and maintaining services and internal tooling.
• Proven experience in on-call rotations and incident management, including the ability to analyze and improve system stability after failures.
• Deep understanding of observability, including instrumenting systems, managing metrics and logs, and configuring effective alerting.
• Practical experience building LLM-based systems, such as RAG pipelines or agents, with a focus on tokens, context windows, and evaluation frameworks.

Nice to have

• Production experience with LLM proxies like LiteLLM.
• Familiarity with LLM observability tools such as LangSmith, Braintrust, Arize, or Datadog LLM Observability.
• Experience implementing evaluation frameworks or LLM-as-judge scoring in a production environment.
• Background in FinOps, specifically regarding cost attribution and quota management.
• Knowledge of content filtering, PII detection, or guardrail systems.
• Experience working with GCP in addition to AWS.

Skills & tools

• Infrastructure: AWS, GCP, EKS, Terraform, Flux, GitOps
• Languages: Python
• Observability: Datadog, Prometheus, Loki, ClickHouse
• AI/ML: LiteLLM, GitLab CI

Practical notes

Benefits include group medical, personal accident, and term life insurance. Coverage extends to dental, vision, and outpatient care, alongside mental health resources, fitness benefits, and a car lease policy. Gratuity is provided for long-term financial support. Demandbase does not accept unsolicited resumes from third-party recruiting agencies.