Senior Software Engineer, Airflow Infrastructure

About the role

Astronomer is seeking a Senior Software Engineer to join the Airflow Infrastructure team. This role involves developing the core layer connecting open-source Airflow to a scalable cloud platform. Your contributions will directly impact how global organizations manage data pipelines, enhancing their speed, dependability, and ease of use. You will play a critical part in translating complex infrastructure challenges into robust, scalable backend solutions. The position requires a deep understanding of both software engineering principles and data orchestration concepts. You will be instrumental in shaping the developer experience for a global user base. Your work will ensure that the platform remains at the forefront of the data engineering ecosystem. This is an opportunity to build systems that power real-world data workflows across industries.

Key facts

What you'll do

• Architect and develop backend services using high-quality, maintainable, and well-tested code in Python or Golang.
• Partner with engineering peers, product managers, customer reliability support, and leadership to define system evolution and achieve business objectives.
• Conduct thorough code reviews and provide constructive, actionable feedback to elevate the quality of the entire codebase.
• Drive improvements in the performance, reliability, and scalability of existing backend services to meet growing demands.
• Investigate, prototype, and propose innovative solutions that significantly enhance the user experience for platform operators.
• Author and maintain clear technical documentation for systems and processes to ensure long-term maintainability and knowledge sharing.
• Participate in an on-call rotation to troubleshoot production issues and resolve incidents promptly and effectively.
• Design and implement integrations that connect open-source Airflow with the underlying cloud platform infrastructure.
• Optimize database queries and system interactions to ensure efficient data flow and low-latency responses.
• Collaborate with the reliability team to implement best practices for monitoring, alerting, and incident response.
• Evaluate new technologies and tools to determine their applicability for improving the infrastructure stack.
• Guide junior engineers through mentorship and technical leadership, fostering a culture of quality and excellence.
• Translate ambiguous requirements into concrete technical specifications and implementation plans.
• Ensure all deliverables adhere to security, compliance, and operational standards.

Requirements

• Bring 5 or more years of experience building and delivering Software as a Service products in a professional setting.
• Demonstrate strong proficiency in Python or Golang, with a deep understanding of language-specific nuances and ecosystem.
• Possess practical experience with Kubernetes, including cluster management, deployments, and networking concepts.
• Show a solid understanding of integrating RESTful APIs and designing distributed systems that are resilient and scalable.
• Have familiarity with testing frameworks like pytest and a commitment to writing comprehensive test suites.
• Exhibit strong written and verbal communication skills, including experience creating detailed technical specifications.
• Show a strong commitment to reliability and operational excellence in all aspects of software development.
• Prove the ability to define work scope and coordinate across multiple teams to manage risks and ensure successful delivery.
• Have hands-on experience with software development best practices, including code reviews, testing, CI/CD, version control, automation, and debugging.
• Demonstrate adaptability to change and thrive in a fast-paced development environment.
• Take a proactive approach to identifying and resolving issues, with a focus on ownership and accountability for outcomes.
• Have a history of delivering high-impact software solutions that improve user productivity and system performance.
• Understand the principles of infrastructure as code and configuration management.

Nice to have

• Bring experience with Apache Airflow to the role, including DAG development and scheduler optimization.
• Possess knowledge of cloud provider services such as AWS, GCP, or Azure.
• Have contributed to open-source projects or demonstrated a strong understanding of open-source collaboration.
• Familiarity with data pipeline patterns and ETL processes.
• Experience with containerization tools beyond Kubernetes, such as Docker.
• Understanding of database systems, including both SQL and NoSQL solutions.
• Exposure to data security and governance frameworks.

Practical notes

This is a hybrid role requiring at least 3 days per week in the New York City office. Astronomer is an equal opportunity employer. All employment decisions are made without regard to race, color, religion, sex, national origin, age, disability, veteran status, or other protected status. The position requires the ability to work in a dynamic team environment and adapt to shifting priorities. Effective collaboration is essential, as you will be working closely with cross-functional partners. The successful candidate will be comfortable managing multiple tasks and maintaining a high degree of organization. The role demands a proactive mindset and the willingness to engage in continuous learning. You will be expected to mentor less experienced engineers and contribute to technical roadmaps. The ability to communicate complex technical concepts to non-technical stakeholders is highly valued. This role is based in New York City and requires consistent collaboration with the local team. The hybrid model allows for flexibility while ensuring regular in-person presence.