Data Engineer
Job description
About the role
Vulcan Elements is dedicated to the production of American rare-earth permanent magnets, which play a crucial role in enhancing national security and fostering economic stability. We are on the lookout for a skilled Data Engineer who will be instrumental in developing and implementing robust data infrastructure aimed at optimizing both operational and business data. Initially, this position will support a pilot project, with plans to scale operations to a facility capable of producing 10,000 tons annually.
Key facts
What you'll do
- Design and oversee the architecture of Vulcan's data systems, encompassing everything from operational data storage to advanced analytics and AI layers.
- Evaluate and select appropriate platforms for the data Lakehouse, ETL tools, and operational databases, prioritizing scalability, compliance, and cost-effectiveness.
- Develop and update data architecture documentation to ensure compliance with technical standards and regulations such as CUI and ITAR.
- Document design decisions clearly to enhance understanding and facilitate onboarding for future team members.
- Ensure that the data architecture is designed for scalability, allowing for a seamless transition from pilot to full-scale production without significant redesigns.
- Implement engineering best practices, including version control, testing protocols, and comprehensive documentation, as the data team grows.
- Create and manage ETL pipelines that facilitate the transfer of data from operational systems into the data Lakehouse, ensuring that data is contextually enriched for analytics and AI applications.
- Establish reliable data ingestion pathways for diverse data types sourced from manufacturing and laboratory systems.
- Work collaboratively with engineering, operations, and IT teams to identify data flows and integration requirements, translating these into architectural frameworks.
- Automate manual data workflows, replacing them with monitored data pipelines to enhance efficiency.
- Identify and resolve data quality issues, incorporating monitoring mechanisms into pipelines for early detection and resolution.
- Develop data models that support operational queries and analytical tasks, ensuring readiness for future AI and machine learning applications.
- Set standards for data contextualization to ensure that data points are meaningful and well-documented for users.
- Contribute to the design of schemas for operational data stores to maintain consistency across the organization.
- Assist in creating tools for reporting and visibility, providing insights into process and quality data.
- Produce clear technical documentation covering architecture, data models, pipeline designs, and operational procedures.
Responsibilities and tasks outlined are not exhaustive and may evolve based on the business's needs.
Requirements
- A minimum of 8 years of experience in data engineering or a closely related technical field, with a solid track record of managing and delivering production systems.
- Proven experience in designing and constructing data lakes, Lakehouses, or analytical data stores, with the ability to assess and justify platform selections.
- Strong expertise in developing ETL/ELT pipelines that enhance data quality and context.
- Proficient in data modeling and quality assurance methodologies to ensure data integrity and usability.
- Familiarity with compliance standards, particularly CUI and ITAR, and their impact on data architecture.
Nice to have
- Experience with cloud service providers such as AWS, Azure, or Google Cloud.
- Understanding of machine learning applications and their associated data requirements.
- Familiarity with data visualization tools and reporting frameworks that aid in data interpretation.
Skills & tools
- Proficiency in programming languages such as Python or Scala.
- Experience with both SQL and NoSQL database systems.
- Familiarity with data pipeline orchestration tools, such as Apache Airflow or similar technologies.
- Understanding of data warehousing principles and tools.
Practical notes
This role is based in Research Triangle Park, NC. Candidates must possess authorization to work in the United States. The position offers a range of benefits, including health insurance, retirement plans, and opportunities for professional growth. Applications will be accepted until the position is filled.