Lead Data Engineer

About the role

At PitchBook, we understand that our Data Operations team plays a role in setting us apart in the competitive landscape. We are dedicated to providing our clients with top-notch data while nurturing a collaborative and responsive work culture. Our team consists of motivated and detail-oriented professionals who are enthusiastic about evolving in a fast-paced industry.

Key facts

What you'll do

• Lead the design and implementation of data pipelines and workflows tailored for Data Operations.
• Develop and uphold best practices related to data modeling, pipeline orchestration, testing, and monitoring to ensure data integrity.
• Improve the data architecture with a focus on warehousing, transformation layers, and user access patterns.
• Identify opportunities for tool modernization to boost performance and reduce operational expenses.
• Provide mentorship and guidance to a diverse group of data engineers with varying levels of experience.
• Enhance the team's capabilities in SQL, Python, data modeling, and system architecture.
• Establish coding standards and review protocols while promoting best practices across the team.
• Foster collaboration between operational team members and software engineering disciplines to streamline processes.
• Act as the primary technical point of contact for Data Operations leadership and key stakeholders.
• Work closely with Product, Engineering, and Data Platform teams to ensure alignment on architecture and shared capabilities.
• Convert business and operational needs into actionable technical specifications that drive project execution.
• Influence the prioritization and strategic roadmap for Data Operations initiatives.
• Design and maintain data solutions that effectively combine manual workflows with automated processes.
• Create monitoring and reporting tools to guarantee data quality, pipeline reliability, and operational transparency.
• Develop a deep understanding of Data Operations processes to pinpoint automation and optimization opportunities.
• Balance the demands for speed and quality in a dynamic, often ambiguous work environment.
• Exemplify and promote the company's vision and values through your actions and interactions.
• Participate in various company initiatives and projects as required.

Requirements

• A Bachelor's degree in Computer Science, Engineering, Statistics, Information Systems, or a related discipline.
• At least 5 years of experience in data engineering or related fields.
• Strong proficiency in SQL, with extensive experience managing large and complex datasets.
• Advanced skills in Python for data processing, automation, and pipeline development.
• Demonstrated experience in constructing and maintaining ETL/ELT pipelines and data workflows.
• Familiarity with data warehousing principles and contemporary data stacks, such as Snowflake or comparable technologies.
• Ability to architect scalable and maintainable data models and systems.
• Exceptional communication skills, enabling effective engagement with both technical and non-technical stakeholders.
• Experience in mentoring or coaching fellow engineers, whether in formal or informal capacities.
• Background in managing large-scale proprietary data environments.
• Knowledge of orchestration tools like Airflow and familiarity with modern data technologies.
• Experience with business intelligence platforms such as Tableau or Power BI.
• Exposure to cloud service providers including AWS, Azure, or GCP.

Nice to have

• Experience with machine learning frameworks and data science methodologies.
• Familiarity with data governance and compliance standards.
• Understanding of data security principles and practices.
• Experience with real-time data processing technologies like Kafka or similar.
• Knowledge of containerization and orchestration tools such as Docker and Kubernetes.

Skills & tools

• Proficiency in SQL and Python programming languages.
• Experience with data warehousing solutions, particularly Snowflake.
• Familiarity with orchestration tools like Apache Airflow.
• Knowledge of cloud platforms such as AWS, Azure, or Google Cloud.
• Experience with business intelligence tools like Tableau or Power BI.
• Strong understanding of data modeling and ETL/ELT processes.

Practical notes

This role is open to candidates requiring visa sponsorship. The company provides a comprehensive benefits package, which includes health insurance, retirement plans, and opportunities for professional growth. Applications will be accepted until the position is filled, and we encourage interested candidates to apply promptly.