Staff Software Engineer, Datalake Platform

About the role

This role focuses on the evolution of the data infrastructure that serves all business units at Stripe. You will lead the migration to modern open source standards while managing the storage abstraction layer for petabytes of data.

Key facts

What you'll do

• Design a unified metastore service to act as the primary source of truth for Iceberg table management across Spark, Trino, Flink, and PyIceberg.
• Manage the migration strategy from Hive Metastore, including cross-team coordination and maintaining operational stability during transitions.
• Build the object storage abstraction layer, covering bucket provisioning, security policies, and client libraries for internal developers.
• Integrate compliance and security requirements directly into the platform, including audit logging and data segregation.
• Optimize storage costs and efficiency by implementing automated tools for data lifecycle and snapshot management.
• Oversee technical design reviews and mentor senior staff to ensure high standards for reliability and security.

Requirements

• 10+ years of professional software engineering experience.
• Experience designing and operating large-scale distributed storage or data infrastructure.
• Proficiency with object storage systems like S3 or Azure Blob, including IAM and lifecycle management at petabyte scale.
• Ability to lead multi-quarter infrastructure projects involving complex cross-team dependencies.
• Background in designing authorization and access control models for distributed systems.

Nice to have

• Expertise in Apache Iceberg internals, specifically the REST Catalog specification, compaction, and compute engine integration.
• Experience with compliance-sensitive infrastructure such as SOX or ICFR and translating these into technical controls.
• History of executing large-scale data migrations with a focus on data integrity and rollback strategies.
• Experience building ergonomic developer abstractions that reduce operational toil.

Skills & tools

• Apache Iceberg
• Spark
• Trino
• Flink
• PyIceberg
• S3 / Azure Blob
• Hive Metastore
• IAM / Access Control
• Data Compliance (SOX/ICFR)