Member of Technical Staff - Data Ingestion & Quality

About the role

In this pivotal position, you will oversee the management of data throughout its entire lifecycle, from the initial stages of acquisition to ensuring its readiness for training purposes. Your primary responsibility will be to develop and maintain the infrastructure that supports our extensive physics foundation model. This role is essential for advancing artificial intelligence that can predict and influence various physical systems.

Key facts

What you'll do

• Identify and source diverse types of physical data, including sensor readings, 3D point clouds, and radar data, by collaborating with partners, vendors, or utilizing public datasets.
• Develop robust data pipelines capable of processing petabytes of information using technologies like Apache Spark, ensuring that the data is stored in a standardized format that is ready for both batch and real-time analysis.
• Establish methodologies for evaluating data quality, focusing on aspects such as coverage, accuracy, and consistency, while also identifying subtle issues like sensor bias or processing errors that could impact model performance.
• Design and implement automated quality assurance protocols that continuously monitor data quality and manage the results of these evaluations.
• Create comprehensive documentation of technical requirements and provide constructive feedback to external data providers and collaborators to enhance data quality.
• Collaborate with researchers to validate that newly acquired or improved datasets contribute to enhanced model performance, ensuring that the data meets the necessary standards.
• Engage in troubleshooting and resolving data-related issues, ensuring that any discrepancies are addressed promptly to maintain the integrity of the data.
• Stay updated with the latest advancements in data ingestion technologies and methodologies to continuously improve the data management processes within the organization.

Requirements

• Demonstrated experience in building large-scale data pipelines, quality assurance systems, or evaluation frameworks, utilizing technologies such as Spark, Ray, or Beam.
• A keen eye for detail, with the ability to identify minor inconsistencies and issues in data that could potentially affect quality and model performance.
• A proactive approach to thoroughly reviewing new source materials, including format specifications, sensor documentation, and vendor guidelines, to ensure accurate and efficient data ingestion.
• Previous experience working collaboratively with external data vendors and partners, from conducting technical assessments to providing ongoing feedback for improvement.
• Strong project management skills, with the ability to oversee projects from inception to completion, including gathering requirements and executing tasks independently.
• Excellent communication skills, both written and verbal, to effectively convey technical information to non-technical stakeholders and collaborate with team members.

Nice to have

• Familiarity with machine learning concepts and how data quality impacts model training and performance.
• Experience in working with cloud-based data storage solutions and data processing frameworks.
• Knowledge of data governance and compliance standards relevant to data ingestion and management.
• Understanding of physical systems and their data requirements, particularly in the context of AI applications.

Skills & tools

• Proficient in Apache Spark for large-scale data processing.
• Experience with Ray for distributed computing tasks.
• Familiarity with Beam for stream and batch data processing.
• Knowledge of data quality assessment tools and methodologies.
• Understanding of data visualization tools to present findings effectively.

Practical notes

• This position is based in San Francisco, a hub for technology and innovation, providing a dynamic work environment.
• The role requires a full-time commitment, and while compensation details are not specified, it is expected to be competitive within the industry.
• Candidates should be prepared to work collaboratively within a team-oriented culture, contributing to the overall success of the infrastructure team and the organization as a whole.
• A willingness to adapt and learn new technologies and methodologies is essential, as the field of data ingestion and quality assurance is constantly evolving.

META

Company: Causal

Title: Member of Technical Staff - Data Ingestion & Quality

Listed

location: San Francisco

Job type: Full-Time