Senior Software Engineer, Applied AI

About the role

At Astronomer, we are defining what it means to build intelligence directly into the data stack. As the company behind Apache Airflow - the open-source orchestration engine used by the world's largest enterprises - we have a unique vantage point into how data moves, transforms, and creates value across organizations. This role is centered on Architecting the semantic mapping layer that translates pipeline metadata into structured, queryable context for retrieval and reasoning. You will design systems that deeply understand data lineage, execution patterns, and metadata to drive automation, search, code generation, and root cause analysis. You will operate at the intersection of applied research and production engineering, transforming early AI prototypes into reliable, scalable features that serve data practitioners globally. If you thrive in fast-moving, research-driven environments and care deeply about how data professionals work, this role is for you.

What you'll do

• Architect and implement the core intelligence layer that powers global context for data across the entire observability and AI stack.
• Design scalable ingestion and indexing pipelines that process lineage, metrics, configuration, and runtime data from heterogeneous sources into a unified semantic model.
• Optimize vector search and metadata filtering strategies to support low-latency queries across massive, multi-tenant data ecosystems.
• Prototype and integrate LLM-driven agents that automate root cause analysis, impact assessment, and decision suggestions for data practitioners.
• Design and expose context-aware APIs and developer tools that integrate seamlessly with IDEs, orchestrators, and internal platforms.
• Partner closely with machine learning researchers and product teams to translate experimental models into production services that meet strict reliability, performance, and correctness requirements.
• Instrument end-to-end data workflows to ensure observability, performance, and correctness of AI-assisted features, including retrieval quality and generation accuracy.
• Lead design reviews and contribute to open-source projects that underpin the Astronomer platform and the broader Apache Airflow ecosystem.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a closely related technical field.
• Demonstrated experience building backend systems that handle large-scale data processing and high-concurrency workloads with strict SLAs.
• Strong proficiency in Python, including asynchronous frameworks, concurrency models, and integration patterns for distributed services.
• Solid understanding of database systems, indexing strategies (including vector indices), and query optimization techniques across structured and unstructured data.
• Hands-on experience with containerization and orchestration technologies, especially Kubernetes and Docker, including production-grade deployment patterns.
• Deep familiarity with distributed systems concepts such as idempotency, retries, backpressure, backoff strategies, and eventual consistency in large-scale pipelines.
• Comfort working with REST and GraphQL APIs to integrate disparate services into cohesive, platform-level products.
• Excellent written and verbal communication skills to collaborate effectively with engineers, researchers, product managers, and non-technical stakeholders.
• Empathy for data professionals and a demonstrated interest in improving their workflows, tooling, and day-to-day experience.

Nice to have

• Passion for AI systems tailored to data, developer tools, and machine learning infrastructure.
• Familiarity with Apache Airflow or other orchestration engines and metadata standards in the data ecosystem.
• Demonstrated contributions to open-source projects, especially those related to data platforms, observability, or AI tooling.
• Experience in information retrieval, semantic search, and large-scale data infrastructure that spans warehouses, lakes, and pipelines.
• Exposure to early-stage startups or R&D organizations where ambiguity is the norm and rapid iteration is essential.
• Experience building agentic systems on top of frontier models, including prompt engineering, tool use, and guardrails for production workloads.

Practical notes

-

Engagement: Full-time.

-

Location: USA

• Work arrangement: Hybrid.

-

Compensation: The estimated total compensation for this role ranges from $168,000 - $230,000 based on leveling and geography, along with an equity component and a comprehensive benefits package. Actual compensation may vary based on skills, experience, and qualifications.

• At Astronomer, we are an equal opportunity employer and value diversity. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, or other legally protected status.