AI Evaluation Engineer

About the role

This position focuses on maintaining quality standards for agentic AI systems. You will work alongside the evaluation lead to manage benchmarks, refine metrics, and ensure product readiness for voice and chat solutions.

Key facts

What you'll do

• Create and carry out validation plans for NLP, speech, and agentic workflows across staging and release environments.
• Manage regression testing, red teaming, and A/B comparisons to verify model performance.
• Develop and calibrate LLM-judge metrics and refine associated prompts.
• Oversee data annotation tasks to ensure consistent evaluation datasets.
• Build and maintain QA infrastructure, including notebooks and pipeline tools for scalable testing.
• Analyze system errors to determine if they require engineering fixes, updated test sets, or further investigation.
• Coordinate with product, engineering, and applied science teams.

Requirements

• Bachelor or Master degree in Software Engineering, Computer Science, Computational Linguistics, or a related field.
• Minimum 3 years of experience in model evaluation, test engineering, or QA for AI products.
• Proven ability to design structured testing strategies for both manual and automated processes.
• Familiarity with complex AI architectures, including LLMs, NLP, and agentic systems.
• Experience creating benchmark datasets, gold sets, or adversarial test sets.
• Strong analytical capabilities for identifying failure patterns and communicating technical findings.
• Ability to work effectively within cross-functional teams.

Skills & tools

• LLM and NLP evaluation frameworks
• Data annotation management
• QA tooling and pipeline development
• Notebook-based analysis (e.g., Jupyter)
• Technical documentation and reporting

Practical notes

The provided salary range applies to Ontario-based roles and excludes bonuses, equity, and benefits. Dialpad is an equal-opportunity employer. Candidates who do not meet every listed requirement but possess the necessary drive and foundational skills are encouraged to apply.