Senior Software Engineer in Test

About the role

We are seeking a Senior Software Engineer in Test to concentrate on the demanding field of AI Agentic Systems within our growing technology organization. In this capacity, you will be responsible for architecting and building robust testing frameworks and strategies that ensure the unwavering reliability of our automated health intelligence tools. The role requires a deep partnership with cross-functional engineering teams to embed quality checks directly into the development lifecycle from the earliest stages. You will focus on validating complex behaviors within agentic workflows to maintain the highest standards for system stability and data accuracy. Because we handle sensitive health-related applications, your work will be instrumental in safeguarding the integrity of the systems we deliver. This position is centered in our AI Agentic Systems team, where you will tackle some of the most challenging testing problems in modern software. Your contributions will directly influence the trustworthiness of the products that power our mission in the healthcare sector.

Key facts

What you'll do

• Architect and implement automated testing solutions specifically designed for intricate AI agent workflows and multi-step decision processes.
• Partner closely with software engineers to integrate quality gates and validation checks throughout the entire development lifecycle of our products.
• Investigate and isolate performance bottlenecks and functional gaps that emerge within our sophisticated agentic architecture and data pipelines.
• Define and uphold stringent standards for system stability, reliability, and data accuracy across all health-related applications and services.
• Develop and maintain comprehensive test suites that cover the dynamic and non-deterministic nature of machine learning and agent-based systems.
• Collaborate with product managers and engineers to translate complex requirements into actionable test strategies and executable test cases.
• Analyze system logs and test results to identify root causes of failures, ensuring issues are resolved before they impact end users.
• Contribute to the continuous improvement of testing methodologies, tools, and best practices as they evolve with new AI and software engineering technologies.
• Mentor junior engineers and testers on effective testing practices for complex backend systems and AI-driven applications.
• Act as a quality advocate for the engineering organization, promoting a culture of testing excellence and proactive defect prevention.

Requirements

• Bring professional experience in software testing or quality engineering focused on complex backend systems and distributed architectures.
• Demonstrate a proven ability to write, maintain, and scale automated test suites for intricate software applications.
• Show a strong understanding of software development lifecycles, quality assurance methodologies, and testing best practices.
• Exhibit the capability to work on-site in either our Lehi, Utah or Plano, Texas offices consistently and reliably.
• Possess a meticulous attention to detail and a strong commitment to identifying edge cases and potential system failures.
• Display effective communication skills to collaborate with technical and non-technical stakeholders regarding testing status and quality metrics.
• Have a foundational understanding of the principles that govern backend software engineering and system architecture.
• Demonstrate the ability to adapt to rapidly changing priorities and technical environments in a fast-paced startup setting.

Skills & tools

• Expertise with automated testing frameworks and tools for building scalable test solutions.
• Solid experience in validating AI and machine learning systems, including model performance and data integrity checks.
• Profound knowledge of backend software engineering principles, APIs, and data integration patterns.
• Familiarity with testing practices specific to agentic systems and workflows that involve autonomous decision-making.
• Understanding of health data concepts and the requirements for maintaining data accuracy in healthcare applications.

Nice to have

No additional preferred items were specified for this role beyond the core requirements and skills.

Practical notes

This is a full-time position that requires on-site work in either the Lehi, UT or Plano, TX offices. Collective Health provides comprehensive benefits packages for full-time employees. Please submit your application through our careers portal to be considered for this position.