Software Engineer- GPU Fabric Observability

About the role

Baseten is developing proprietary GPU infrastructure to support high-scale AI inference. We are seeking a lead engineer to architect a diagnostic system capable of identifying complex, intermittent failures within our GPU fabric, such as network congestion and hardware-level bottlenecks.

Key facts

What you'll do

• Architect and maintain the infrastructure for GPU fabric observability and root-cause analysis.
• Develop telemetry pipelines that aggregate data from switches, NICs, hosts, GPUs, Kubernetes, and inference services.
• Map network topology, service ownership, and failure domains to improve incident response.
• Distinguish between fabric-level faults and issues originating in the driver, kernel, scheduler, or serving layer.
• Design operator workflows that provide clear, actionable insights during active incidents.
• Build real-time systems to ingest and correlate high-cardinality telemetry data.
• Implement diagnostic probes to detect latency, packet drops, and congestion across the network.
• Link network performance metrics to specific RDMA operations, GPUDirect transfers, and KV cache movement.

Requirements

• Staff or Senior Staff level experience in production infrastructure software engineering.
• Background in distributed systems, specifically regarding control-plane software, streaming systems, or telemetry pipelines.
• Experience managing and processing high-volume, noisy operational data.
• Proficiency in networking fundamentals and high-performance network environments.
• Ability to translate low-level infrastructure signals into functional anomaly detection and root-cause analysis tools.

Skills & tools

• Distributed systems architecture
• Telemetry and observability pipelines
• Networking (InfiniBand, RoCE)
• RDMA and GPUDirect
• Kubernetes
• GPU and host-level diagnostics

Practical notes

• Benefits include 100% medical, dental, and vision coverage for employees and dependents.
• Includes a flexible PTO policy with a mandatory company-wide winter break.
• Offers paid parental leave and a fertility/family-building stipend via Carrot.
• Includes a company-facilitated 401(k) plan.
• Baseten is an equal opportunity employer and complies with the San Francisco Fair Chance Ordinance.