Staff Software Engineer

About the role

This role focuses on advancing Spark Structured Streaming within Apache Spark and the Databricks Data Intelligence Platform. You will help define the technical direction for stream processing, working on both open-source components and Databricks-specific features. The goal is to enhance the engine's capabilities and architecture for improved performance and cost efficiency.

Key facts

What you'll do

• Establish and guide the technical roadmap for Spark Structured Streaming, covering both open-source and Databricks platform elements.
• Design and implement core engine features, including state management, new operators, and architectural improvements for latency and throughput.
• Improve engineering quality and operability, ensuring software is robust and easy to deploy in production.
• Contribute to company-wide impact by promoting stream processing adoption across Databricks products.
• Advise on long-term architectural decisions and technical debt, balancing these with product development goals.
• Provide technical guidance to team members and collaborate with the Engineering Manager on talent acquisition and development.

Requirements

• Bachelor's degree or higher in Computer Science or a related technical field, or equivalent practical work experience.
• At least 8 years of experience building related systems, such as big-data ecosystems, Apache Spark, or database internals.
• Strong interest in database systems, storage systems, distributed systems, language design, or performance optimization.
• Ability to work towards a multi-year vision through incremental deliverables.
• Proven history of delivering features while maintaining high operational excellence and engineering quality.
• Comfortable collaborating with product management and directly with customers, with a deep understanding of product and user needs.

Skills & tools

• Apache Spark
• Distributed systems
• Database internals
• Stream processing

Practical notes

Compensation includes eligibility for an annual performance bonus, equity, and benefits. Specific benefit details are available by region. Databricks is committed to diversity and inclusion. Employment decisions are made without regard to protected characteristics. Access to export-controlled technology or source code may require a U.S. government license, which Databricks may or may not pursue.