Staff Product Manager

About the role

At Workato, we are on a mission to create a robust infrastructure for the agentic era, delivering a platform that integrates data, applications, and artificial intelligence for a significant portion of the Fortune 500. We are in search of a Staff Product Manager who will spearhead the creation of evaluation frameworks specifically designed for AI agents. This role will involve developing tools that empower both our internal teams and our clients to assess and enhance the performance of these agents effectively.

Key facts

What you'll do

• Lead the development and management of the internal evaluation framework tailored for AI agent functionalities, beginning with Agent Studio.
• Craft the user-facing experience that allows developers to test, monitor, and optimize the agents they create.
• Strike the right balance between technical precision and user-friendliness in the evaluation tools.
• Collaborate closely with the Build Experience Product Manager to integrate evaluation processes seamlessly into the user workflow.
• Work in partnership with machine learning engineers and platform teams to ensure that the frameworks are both technically robust and practical for real-world applications.
• Establish success metrics to gauge the quality of internal agents and the adoption rates from customers.
• Conduct thorough user research to gain insights into customer challenges and their understanding of agent performance.
• Analyze feedback and data to iterate on evaluation tools and frameworks, ensuring they meet evolving user needs.
• Facilitate training sessions and workshops to educate internal teams and clients on the effective use of evaluation tools.
• Monitor industry trends and competitor offerings to continuously refine our evaluation strategies.
• Advocate for the user experience in all product decisions, ensuring that our tools provide maximum value.

Requirements

• A minimum of 7 years of experience in Product Management, with a strong track record of success.
• Direct, hands-on experience in crafting evaluations for AI, machine learning systems, large language models, or agents.
• Proven ability to deliver technical products that cater to both internal stakeholders and external customers.
• Demonstrated experience in promoting the adoption of new frameworks or practices within engineering teams.
• A bachelor's degree or equivalent professional experience in a relevant field.
• Capability to distill complex evaluation concepts into user-friendly, high-value product features.
• Experience navigating ambiguous environments, including defining initial product versions and iterating based on user feedback and data analysis.
• Strong understanding of B2B product dynamics, with a focus on simplifying intricate workflows for users.

Nice to have

• A practical background in machine learning engineering, solutions architecture, or technical program management.
• Experience in developing tools for developers or products at the platform level.
• Familiarity with agent architectures, retrieval-augmented generation systems, or the development of applications using large language models.
• Knowledge of specific evaluation methodologies, including human evaluation pipelines, automated benchmarks, or red-teaming practices.

Skills & tools

• Expertise in product roadmapping and leading cross-functional teams.
• Proficiency in AI and machine learning evaluation methodologies.
• Ability to write detailed technical specifications.
• Skills in user research and customer discovery processes.

Practical notes

• REQ ID: 2538.
• Workato fosters a culture centered on trust and flexibility, prioritizing a balance between productivity and employee well-being. We believe that a supportive work environment enhances creativity and innovation, allowing our team members to thrive both personally and professionally. Our commitment to employee satisfaction is reflected in our policies and practices, ensuring that everyone can contribute to their fullest potential while maintaining a healthy work-life balance.

Join us at Workato to be part of a forward-thinking team that is shaping the future of AI and data integration. If you are passionate about product management and eager to make a significant impact in the tech industry, we would love to hear from you!