Sr. SDET

About the role

You will lead test automation and quality strategy for our AI Voice Agent services within the Agentic QA team. This position involves building frameworks to validate end-to-end product experiences, including frontend interfaces, backend services, APIs, and audio-text interactions. You will work closely with developers to ensure our AI platform remains stable, performant, and reliable as we scale.

Key facts

What you'll do

• Manage end-to-end quality for agentic workflows, covering strategy, execution, and release.
• Create automation tooling for AI and LLM systems, including prompt flows and agent orchestration.
• Build evaluation frameworks to track accuracy, response quality, and hallucination rates.
• Maintain 80 percent or higher automation coverage for critical AI workflows using deterministic and probabilistic methods.
• Integrate AI quality checks into CI/CD pipelines to ensure PR feedback in under 15 minutes.
• Develop observability and debugging tools for LLMs, such as response analysis and prompt tracing.
• Collaborate with Applied AI teams on model selection, prompt engineering, and evaluation.
• Conduct performance and load testing to measure latency, throughput, and cost efficiency.
• Define and monitor AI quality KPIs like precision, recall, and task success rates.
• Participate in architectural reviews to ensure system testability and resilience.
• Mentor team members and improve quality engineering standards.

Requirements

• 5 or more years of experience in SDET or software engineering roles.
• Proficiency in Python, Java, or JavaScript.
• Experience testing cloud-native SaaS systems and APIs.
• Demonstrated ability to use AI agents to improve code quality and development speed.
• Hands-on work with LLMs or AI/ML systems such as Gemini, Claude, or OpenAI.
• Understanding of probabilistic testing and non-deterministic system behavior.
• Experience building scalable test frameworks.
• Familiarity with AI evaluation techniques like golden datasets, benchmarking, and human-in-the-loop validation.
• Experience with CI/CD tools such as GitHub Actions or Jenkins.
• Bachelor degree in Computer Science or equivalent practical experience.

Skills & tools

• Backend: Python, Go, Google Cloud Platform, Cloud Run, App Engine, Kubernetes, Datastore, Redis, ElasticSearch.
• Frontend: Vue3, React.
• AI Stack: LLM APIs, LiveKit, prompt orchestration frameworks, evaluation tooling.

Practical notes

This role is based in our Canadian offices and reports to a QA Engineering Manager located in the United States. Compensation listed is base salary only and excludes bonuses, equity, and benefits.