Staff Software Engineer, Data Governance & Foundations

About the role

You will translate Instacart's data strategy into a multi-year architecture roadmap for monetization, federated access, and real-time capabilities. You will own the open lakehouse foundation, defining unified table formats, storage governance, and access patterns for the entire organization. You will drive the real-time and streaming infrastructure for critical domains such as Ads, Fraud, and ML, establishing deployment patterns and operational standards. You will pioneer the application of LLM and AI tools across the data infrastructure lifecycle to accelerate development and optimize costs. You will lead architecture reviews and mentor senior and staff engineers to elevate the technical excellence of the team. You will communicate complex trade-offs clearly to both technical and executive stakeholders to ensure alignment across the business. You will collaborate closely with engineering leadership and stakeholders across Data Science, ML Platform, Ads Infrastructure, Finance Engineering, Product Engineering, and Security.

Key facts

What you'll do

Translate Instacart's data strategy (e.g., monetization, federated access, real-time) into an actionable multi-year architecture roadmap; align with leadership while evolving the platform for scale, maturity, and cost efficiency.

Own the open lakehouse foundation: define and deliver unified table formats, storage governance, and a multi-engine compute portfolio (interactive, batch, streaming) that enables portability and prevents lock-in.

Drive real-time and streaming infrastructure for critical use cases (Ads, Fraud, ML): set deployment patterns, SLAs, and operational practices that balance performance, availability, and spend.

Pioneer AI-native data infrastructure engineering by applying LLM/AI tools to the platform lifecycle - accelerating development, automation, observability, and cost optimization - and partnering to embed AI-powered capabilities into the platform.

Elevate engineering excellence: lead architecture reviews, mentor senior/staff engineers, influence hiring, and clearly communicate complex trade-offs to both technical and executive audiences to ensure cross-org alignment.

Collaborate closely with engineering leadership and stakeholders across Data Science, ML Platform, Ads Infrastructure, Finance Engineering, Product Engineering, and Security.

Operate with a high degree of ownership in a fast-paced environment where architectural decisions have real technical and financial consequences.

Contribute to the design and delivery of a modern open data lakehouse on Apache Iceberg, a multi-engine compute platform, and self-serve tooling that helps Product, Data Science, ML, Ads, Finance, and engineering teams move fast with data.

Requirements

10+ years of software engineering experience building and operating data infrastructure or distributed systems at production scale.

Hands-on expertise with modern data lakehouse architectures including open table formats and storage systems such as Apache Iceberg.

Strong experience with stream processing and messaging systems such as Apache Kafka and Apache Flink.

Deep proficiency in SQL and query engines, with hands-on experience running and optimizing interactive workloads using engines such as Trino, ClickHouse, or similar.

Experience with data processing frameworks such as Apache Spark and/or dbt, and with orchestration tools such as Airflow.

Solid understanding of cloud infrastructure and object storage, including performance and cost trade-offs on AWS.

Experience with containerization and orchestration technologies such as Docker and Kubernetes, and familiarity with CI/CD practices for data platforms.

Strong foundational knowledge of data security and governance concepts, including authentication, authorization, and auditing.

Nice to have

Experience with Databricks and Snowflake.

Experience with Confluent and Delta Lake.

Experience building and operating platforms that serve data consumers across Finance, Data Science, ML, and Product Engineering.

Practical notes

This role operates within a Flex First framework where employees have flexibility in where they do their best work, whether from home, an office, or a preferred coffee shop, while staying connected through regular in-person events. The position is based in the United States and is fully remote. The team values pragmatism, clarity, and impact, operating as a focused group of engineers dedicated to delivering the foundational systems that power Instacart's data ecosystem.