Sr. Software Development Engineer in Test

About the role

You will lead test automation and quality frameworks for the AI Voice Agent services at Dialpad. This position involves building infrastructure to validate end-to-end product experiences, including frontend, backend, API, and audio interactions. You will report to a QA Engineering Manager based in the United States.

Key facts

What you'll do

• Manage end-to-end quality strategy, development, and release qualification for agentic features.
• Create automation tooling for AI and LLM systems, covering prompt flows and orchestration.
• Build evaluation frameworks to track response accuracy, hallucination rates, and quality.
• Aim for 80% or higher automation coverage for critical AI workflows using deterministic and probabilistic methods.
• Integrate quality checks into CI/CD pipelines to ensure PR validation in under 15 minutes.
• Develop observability and debugging tools for LLMs, such as prompt tracing.
• Collaborate with Applied AI teams on model selection and prompt engineering.
• Conduct performance and load testing for AI services regarding latency, cost, and throughput.
• Define and monitor AI quality KPIs like precision, recall, and task success rates.
• Participate in architecture reviews to improve system testability and resilience.
• Mentor team members on quality engineering practices.

Requirements

• 5+ years of experience in software engineering or SDET roles.
• Proficiency in Python, Java, or JavaScript.
• Experience testing cloud-native, distributed SaaS systems and APIs.
• Practical knowledge of LLMs or AI/ML systems such as OpenAI, Claude, or Gemini.
• Understanding of non-deterministic systems and probabilistic testing.
• Experience building scalable test frameworks.
• Familiarity with AI evaluation techniques like benchmarking and golden datasets.
• Experience with CI/CD tools like GitHub Actions or Jenkins.
• Bachelor degree in Computer Science or equivalent experience.

Skills & tools

• Backend: Python, Go, Google Cloud Platform, Cloud Run, App Engine, Kubernetes, Datastore, Redis, ElasticSearch.
• Frontend: Vue3, React.
• AI Stack: LLM APIs, LiveKit, prompt orchestration frameworks, evaluation tooling.

Practical notes

• This role is based in a Canadian office.
• Compensation listed is base salary only and excludes equity, bonuses, and benefits.