Data Scientist

About the role

We are seeking a Data Scientist to join our team focused on optimizing locally-hosted language models for enterprise IT operations. You will move beyond standard predictive modeling to evaluate and improve the reliability, reasoning, and agentic behavior of non-deterministic LLM systems.

Key facts

What you'll do

• Create and maintain evaluation frameworks, including golden sets and rubric-based scoring for LLM and agent outputs.
• Build LLM-as-judge pipelines and validate them against human labels to ensure accuracy.
• Perform red-teaming and stress testing to identify failure modes, prompt injections, and jailbreak vulnerabilities.
• Analyze retrieval quality and multi-step agent trajectories to improve tool-use and intent classification.
• Develop and deploy production models for anomaly detection, trend forecasting, and early-warning signals using operational telemetry.
• Monitor production systems for behavioral drift and implement fixes through prompt engineering, retrieval adjustments, or model selection.
• Quantify system economics and link performance metrics to business outcomes like MTTR.

Requirements

• Bachelor or Master degree in Data Science, Computer Science, Statistics, Mathematics, or equivalent experience.
• 3+ years of professional experience in data science, ML, or quantitative analysis.
• Proficiency in Python and strong SQL skills for querying large datasets.
• Applied statistics expertise for designing experiments on noisy, non-deterministic outputs.
• Experience deploying and maintaining predictive or time-series models in production.
• Demonstrated background in evaluating or improving NLP and LLM systems.
• Hands-on experience with LLM evaluation frameworks, serving libraries, and prompting techniques.

Nice to have

• Experience optimizing small or self-hosted models under memory and latency constraints.
• Background in retrieval-augmented generation (RAG) and retrieval evaluation.
• Knowledge of agentic frameworks and replayable-state analysis.
• Familiarity with IT operations, AIOps, observability, or log anomaly detection.
• Experience with large-scale data stores, cloud ML workloads, and enterprise security compliance.

Skills & tools

• Python, SQL, LLM evaluation frameworks, time-series forecasting, anomaly detection, AIOps, retrieval-augmented generation (RAG), and agentic orchestration.

Practical notes

• Benefits include medical, dental, vision, 401(k) with match, flexible time off, volunteer time off, paid parental leave, and a 5-year sabbatical program.
• The company maintains a headquarters in Reston Town Center.
• We encourage applications from all qualified candidates, even if you do not meet every listed qualification.