AI Systems engineer

About the role

Tilted AI is seeking an AI Systems Engineer to join our founding team. This position offers the chance to work on a unique operating system designed for an agent economy. You will play a crucial role in establishing our in-house pretraining program, which will expand significantly over the coming years.

Key facts

What you'll do

• Manage the training infrastructure, ensuring effective distributed training across various dimensions including data, tensor, pipeline, and expert parallelism.
• Oversee Model FLOPs utilization, a critical metric that impacts financial outcomes in later phases.
• Implement fault tolerance measures, including checkpointing, automatic recovery, and silent data corruption detection.
• Address the MoE communication challenge, which is essential for optimizing throughput in sparse models.
• Develop a scalable data pipeline for efficient tokenization, sharding, and streaming loaders to keep thousands of GPUs operational.
• Integrate the entire pipeline into TiltedOS's praxis mesh, enabling self-scheduling, retries, and audits.
• Build observability tools, including dashboards and alerts, to monitor training runs proactively.

First 90 days

• Collaborate with the pretraining lead to establish the training monorepo within TiltedOS.
• Set up the initial reserved cluster and execute a complete multi-node training run.
• Implement a checkpoint/restart system that can recover from node failures autonomously.
• Launch the ablation grid as scheduled mesh jobs, ensuring results are processed automatically.

Requirements

• Proven experience in running distributed training on multi-node GPU clusters, including troubleshooting and debugging.
• In-depth knowledge of PyTorch internals, including autograd, memory management, and performance profiling.
• Familiarity with NCCL and collective communication, with the ability to differentiate between network, kernel, and data-loader issues.
• Strong understanding of systems fundamentals, including Linux, networking, storage, and containers.
• Willingness to handle on-call responsibilities during long training runs, with a proactive approach to problem-solving.

Nice to have

• Experience with CUDA/Triton kernel development or custom kernel integration.
• Familiarity with frameworks such as FSDP, DeepSpeed, Megatron-LM, or TorchTitan at large GPU scales.
• Knowledge of MoE systems, including expert parallelism and optimization strategies.
• Experience with mixed-precision training (FP8 / FP4) in production environments.
• Background in high-performance storage and checkpointing at a petabyte scale.
• Understanding of NATS, event-sourced systems, or job meshes.

How you work

• You prioritize instrumentation and data analysis over assumptions.
• You view a 5% drop in throughput as a significant issue that needs resolution.
• You prefer to automate recovery processes rather than manually intervene.

Why this role is different

This position is not just another cog in the machine. You will work closely with one ML lead and another engineer, engaging with all layers of the stack from the kernel to the scheduler. Your contributions will directly impact throughput metrics.

You will be working on innovative infrastructure, not simply attaching a training pipeline to standard cloud services. You will be developing within an OS that already has structured job management, making this a unique systems challenge.

We have a clear roadmap for scaling, starting with hundreds of GPUs and aiming for much more as we progress.

Compensation

Base salary range: competitive

Equity: meaningful founding equity

Benefits: comprehensive benefits package

Additional: hardware and conference budget

Apply

To apply, please send your application to Apollo@tilted.xyz. We encourage you to share something substantial, such as a profiling report, a challenging distributed-systems issue you resolved, a relevant repository, or a brief note on how you would investigate a sudden drop in MFU from 42% to 31%. A cover letter is not required.

Tilted AI is an equal opportunity employer.