Staff Software Engineer

About the role

You will define and drive the technical vision for core Tempo components including ingestion, storage, query, and metrics generation across a globally distributed deployment. You will lead multi-quarter technical initiatives such as trace aggregation APIs, Limitless Tempo, autoscaling cells, customer limits, and query engine improvements from problem framing through execution and rollout. You will set the bar for architecture reviews, design decisions, and operational excellence, ensuring that trade-offs between performance, cost, and complexity are clearly documented and justified. You will design APIs for both humans and agents, shaping deterministic, structured, and discoverable interfaces that enable Act 3 products, LLM-driven assistants, and external integrations to build reliably on Tempo. You will own the end-to-end delivery of high-impact features, balancing product needs with platform scalability, reliability, and long-term maintainability. You will mentor engineers across the Tempo and broader Grafana Databases organization, elevating code quality, testing practices, and operational runbooks through clear communication and ownership. You will collaborate closely with product managers, SRE, and other platform teams to align on roadmap priorities, resolve cross-cutting dependencies, and ensure consistent developer experience. You will contribute to the open source project, guiding community engagement and ensuring that internal needs and upstream contributions are aligned.

Key facts

What you'll do

Define and drive the technical vision for core Tempo components including ingestion, storage, query, and metrics generation.

Lead multi-quarter technical initiatives from problem framing through execution and rollout, such as trace aggregation APIs and query engine improvements.

Own the architecture of core Tempo components and drive design reviews that make sharp trade-offs on performance, cost, and complexity.

Design APIs for humans and agents, shaping deterministic, structured, and discoverable interfaces for Act 3 products and external integrations.

Elevate operational excellence at scale as Tempo grows from close to 50 cells this year into triple digits, focusing on autoscaling, parameterized rollouts, and toil reduction.

Evolve Tempo into a platform enabler with higher-density APIs, trace aggregation, TraceQL metrics math, and machine/LLM-friendly interfaces.

Push performance further to achieve faster query latency at hundreds of MB/s ingestion and performant 30-day query ranges to match competitors.

Prepare Tempo for an agent-driven world by supporting larger, burstier, higher-cardinality workloads and AI-powered workflows like assistant-driven triage.

Contribute to the open source project, guiding community engagement and aligning internal needs with upstream contributions.

Collaborate with product managers, SRE, and platform teams to resolve cross-cutting dependencies and ensure consistent developer experience.

Define and implement runbooks and operational practices that improve reliability, debuggability, and maintainability of large-scale deployments.

Partner with the distributed tracing product team to ensure Tempo remains a best-in-class backend for Grafana Cloud Traces and GET.

Lead post-incident reviews and blameless postmortems to drive continuous improvement of system design and operational processes.

Champion testing strategies, including unit, integration, and load testing, to ensure robust and predictable platform behavior.

Requirements

You have a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

You have 8+ years of professional software engineering experience, with a strong track record of delivering complex distributed systems.

You have 3+ years of hands-on experience building and operating backend platforms or observability infrastructure at scale.

You are proficient in Go, with a deep understanding of concurrency patterns, performance profiling, and production debugging.

You have experience designing and operating high-throughput ingestion systems that handle millions of events per second.

You understand distributed systems concepts well, including consensus, replication, partitioning, and fault tolerance in large-scale deployments.

You have experience with storage systems and query engines, and you can evaluate trade-offs between performance, cost, and usability.

You are comfortable working in a fast-paced, open-source-first environment where priorities evolve quickly and context switches are common.

Nice to have

Experience contributing to open source observability projects such as Tempo, Prometheus, Loki, or similar platforms.

Familiarity with AI-assisted development workflows and agent-computer interaction patterns.

Experience building platform products used by internal and external consumers.

Practical notes

This is a remote opportunity located in Republic of Ireland.

Employment is full-time.

Grafana Labs is an equal opportunity employer and welcomes applicants from all backgrounds.