Staff Engineer - Experimentation Team

About the role

Become a vital part of LaunchDarkly's Experimentation team, where you will contribute to the development of a robust platform that empowers data-informed product choices. In this position, you will create systems that facilitate A/B testing, assess the impact of features, and enhance user experiences, all while integrating with a feature management system that processes trillions of evaluations daily. This role uniquely combines elements of data science and platform engineering, with a focus on statistical engines, data warehouse analysis pipelines, and innovative experimentation methodologies.

Key facts

What you'll do

• Develop and refine the statistical engine that supports experimentation, focusing on key methodologies such as hypothesis testing, sequential analysis, variance reduction techniques like CUPED, and power analysis to ensure statistical integrity across various experiment types.
• Architect and implement data warehouse-native experimentation functionalities that allow for analysis directly within customer data warehouses, including Snowflake, Databricks, Redshift, and BigQuery, creating flexible, warehouse-agnostic computation layers.
• Spearhead the design of adaptive experimentation systems, leveraging machine learning models such as contextual bandits and Bayesian optimization to enhance allocation strategies beyond traditional A/B testing.
• Collaborate closely with product management, design, and data science teams to influence the product and technical roadmap for the experimentation platform.
• Work in partnership with other engineering teams, including Warehouse Integrations, SDK, Platform, and Data Science, to ensure seamless product development and integration.
• Mentor and guide fellow engineers, raising the team's standards for statistical accuracy and system architecture.
• Take responsibility for operational aspects of the systems, including monitoring, observability, incident response, and on-call duties, to maintain high levels of system reliability.
• Engage in continuous improvement of experimentation methodologies and tools, ensuring that the platform remains at the forefront of industry standards.

Requirements

• At least 10 years of experience in developing large-scale experimentation platforms, statistical analysis systems, or data-intensive backend services.
• Strong understanding of applied statistics, including hypothesis testing, sequential analysis, variance reduction techniques (e.g., CUPED), power analysis, and experiment design, with a grasp of both frequentist and Bayesian methodologies.
• Proven experience with machine learning techniques applicable to adaptive experimentation, such as contextual bandits, Thompson sampling, Bayesian optimization, or reinforcement learning-based strategies for allocation.
• Demonstrated ability to design systems that operate effectively across various data warehouses, including Snowflake, Databricks, Redshift, and BigQuery.
• Proficiency in backend programming languages like Go or Python, suitable for service development and statistical computations.
• Experience with event-driven architectures, data pipelines, and handling large data volumes efficiently.
• Familiarity with cloud platforms such as AWS and GCP, along with infrastructure-as-code methodologies.
• A track record of technical leadership, including establishing technical direction, breaking down complex problems, and influencing cross-functional teams.
• Excellent communication skills, capable of articulating complex statistical concepts to both product and engineering stakeholders.

Nice to have

• Experience with contemporary data warehousing concepts and distributed computing frameworks.
• Familiarity with advanced data visualization tools and techniques.
• Background in software development best practices and agile methodologies.

Skills & tools

Go, Python, Snowflake, Databricks, Redshift, BigQuery, AWS, GCP, Infrastructure as Code, Event-driven architectures, Data Pipelines, Machine Learning, Applied Statistics.

Practical notes

• Zone 1 (San Francisco/Bay Area, NYC Metro, Boston, Seattle): $214,800 - $295,350
• Zone 2 (Irvine, LA, Monterey, Santa Barbara, Santa Rosa, Austin, Portland, Philadelphia, Chicago): $193,400 - $265,870
• Zone 3 (All other US locations): $182,600 - $251,020

Exact salary will be determined based on your skills, experience, and location.

Benefits include Restricted Stock Units (RSUs), comprehensive health, vision, and dental insurance, as well as mental health support. LaunchDarkly is committed to being an equal opportunity employer.

Please note that recruiters will only reach out to candidates using @launchdarkly.com email addresses or verified LinkedIn profiles. Be cautious of unsolicited requests for financial information or payments.