Senior Data Platform Engineer

About the role

Adyen is at the forefront of financial and payment technology, serving a diverse range of global clients such as Meta, Uber, Microsoft, and H&M. In this position, you will become part of a dedicated team that oversees a vast on-premise big data infrastructure, which is crucial for managing thousands of daily jobs and supporting hundreds of developers. Your expertise will help enhance the robustness and efficiency of our data systems, ensuring that we continue to deliver exceptional service to our clients.

Key facts

What you'll do

• Construct and sustain a sophisticated infrastructure that encompasses over 500 nodes, more than 25,000 CPU cores, upwards of 240 terabytes of RAM, and around 23 petabytes of storage capacity.
• Create and refine tools tailored for the OLAP ecosystem, with a strong emphasis on technologies such as Druid, FastAPI, Java, and Kafka.
• Architect data engineering frameworks that include data contracts, validation mechanisms, and efficient Kafka ingestion pipelines.
• Develop comprehensive systems for metadata collection, ensuring data lineage is traceable, and implementing platform-wide quality monitoring solutions.
• Enhance continuous integration and continuous deployment (CI/CD) processes within our private cloud infrastructure to streamline operations.
• Build and optimize stream processing applications that facilitate quicker insights for data scientists and analysts, thereby improving decision-making processes.
• Collaborate closely with cross-functional teams to align infrastructure, security, and data science initiatives, ensuring seamless integration and performance.
• Troubleshoot and resolve issues within the data platform, ensuring minimal disruption and maintaining high availability.
• Stay updated with industry trends and emerging technologies, integrating relevant advancements into our existing systems to maintain a competitive edge.
• Mentor junior engineers and contribute to the overall growth of the team by sharing knowledge and best practices.

Requirements

• A minimum of 6 years of professional experience working with Java or Python in a data engineering context.
• Demonstrated expertise in designing and implementing distributed compute and data systems, with proficiency in technologies such as Spark, Trino, Druid, or ClickHouse.
• Solid experience in constructing reliable real-time and batch data pipelines utilizing Kafka and Spark Streaming.
• Strong capability to serve as a liaison between infrastructure, security, and data science teams, ensuring cohesive collaboration.
• Familiarity with data governance principles and practices, particularly in the context of large-scale data environments.
• Excellent problem-solving skills and the ability to work under pressure while managing multiple priorities.

Nice to have

• Experience with cloud platforms such as AWS, Google Cloud, or Azure, particularly in relation to data services.
• Knowledge of containerization technologies like Docker and orchestration tools such as Kubernetes.
• Familiarity with data visualization tools and techniques, enhancing the presentation of data insights.
• Previous involvement in open-source projects or contributions to the data engineering community.

Skills & tools

• Proficient in Python and Java programming languages.
• Experienced with data processing frameworks including Spark, Trino, Druid, and ClickHouse.
• Knowledgeable in messaging systems like Kafka and stream processing with Spark Streaming.
• Familiar with FastAPI for building APIs and microservices.
• Competent in CI/CD practices and managing private cloud infrastructure.

Practical notes

• Adyen operates with an office-first approach and does not accommodate remote-only work arrangements.
• We strive to respond to all applications within a timeframe of 5 business days.
• The standard interview process typically spans around 4 weeks, allowing for thorough evaluation and consideration.
• We actively encourage applications from individuals belonging to underrepresented groups, even if they do not meet every single qualification listed.

Join us at Adyen and be part of a dynamic team that is redefining the landscape of financial technology. Your contributions will directly impact our mission to provide seamless payment solutions to clients worldwide.

About the company

Adyen is a financial technology platform providing end-to-end payments infrastructure, data analytics, and financial products. Founded by Pieter van der Does and Arnout Schuijff in 2006, Adyen went public on Euronext Amsterdam in 2018. Its clients include Meta, Uber, Spotify, and Microsoft.