Forward Deployed Data Engineer (Manassas, VA

About the role

We are seeking a talented Forward Deployed Data Engineer to join our innovative team at Spear Ai, where you will play a pivotal role in shaping the future of data management and artificial intelligence solutions focused on maritime domain awareness. In this position, you will engage directly with clients on-site, integrating advanced systems and tackling intricate technical challenges. Your contributions will be vital to our mission's success, as you blend your engineering expertise with advisory responsibilities to deliver impactful solutions. You will act as a critical bridge between our technical product capabilities and the specific needs of our maritime clients. This role requires a unique individual who thrives in translating complex data infrastructure concepts into tangible business value. You will be responsible for ensuring that the data pipelines and analytical systems we build are not only robust but also provide clear insight and efficiency to the end user. Expect to dive deep into the intricacies of the client's environment to diagnose issues and optimize performance. Your work will directly influence the strategic direction of our data products and their application in the field. This is a hands-on position where your technical acumen will be on the front lines of every deployment.

Key facts

What you'll do

You will design and implement scalable data warehouses utilizing Postgres, Apache Iceberg, Parquet, and S3 to handle the demanding data loads of maritime tracking. You will architect and optimize data models that ensure high-performance query execution for real-time domain awareness. You will validate, normalize, and integrate diverse data sources from various sensors and systems to maintain strict data integrity and usability. You will refine local development workflows and enhance CI/CD processes by leveraging modern tools and GitHub Actions to accelerate delivery. You will work in tandem with cross-functional teams to guarantee the seamless integration of data solutions into existing operational systems. You will provide expert technical guidance and support to clients, ensuring their specific data needs are not only met but exceeded. You will actively participate in code reviews to uphold code quality and contribute to the continuous improvement of engineering best practices. You will monitor industry trends and emerging technologies to identify opportunities for enhancing our data engineering capabilities and competitive edge. You will create comprehensive documentation for processes and workflows to facilitate knowledge sharing within the team and with stakeholders. You will troubleshoot complex issues in production environments, applying advanced debugging techniques to restore system integrity swiftly.

Requirements

You must possess a current or active U.S. Secret clearance. You must demonstrate a strong background in time-series data processing and analysis, including techniques such as windowing, resampling, and interpolation. You must exhibit proficiency in the programming languages Python and Rust, specifically applying them to data engineering tasks and challenges. You must have proven experience in parsing binary message formats effectively to handle diverse data ingestion scenarios. You must show familiarity with both row-oriented and column-oriented data storage formats and their appropriate use cases. You must have knowledge and experience with OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) databases to select the right tool for the job. You must understand distributed systems, streaming architectures, and batch processing methodologies to build resilient data solutions. You must have practical experience with batch orchestration tools like Dagster or Airflow to manage complex workflow dependencies. You must possess hands-on experience with streaming platforms, particularly Redpanda or Kafka, to handle real-time data ingestion. You must be proficient in working with binary message formats, such as Protobuf, to ensure efficient and reliable data communication.

Nice to have

You may have experience with Internet of Things (IoT) devices and sensor data integration, which is highly relevant to our maritime focus. You may have a background in digital signal processing techniques that can provide an edge in handling noisy maritime sensor data. You may be familiar with geospatial analysis and Geographic Information Systems (GIS) to visualize and analyze vessel movements and patterns. You may have experience working within monorepos to manage codebases efficiently and maintain consistency across the project.

Practical notes

This role is based in Manassas, Virginia and requires on-site presence. The position is offered as a full-time engagement.