Machine Learning Eval Engineer

About the role

You will design and own the evaluation frameworks that define how we measure model quality across our document understanding stack. You will work at the intersection of product, research, and infrastructure to turn messy enterprise behavior into actionable benchmarks. This role requires you to take full ownership from metric design to deployment of evaluation tooling. You will surface hard failure cases before they reach customers and help prioritize model improvements. Your work will directly influence which capabilities get built and how we scale quality assurance. You will act as the internal expert on evaluation best practices for ML and GTM teams. You will ensure that our benchmarks remain rigorous, reproducible, and aligned with real customer workflows.

Key facts

What you'll do

• Design and maintain evaluation benchmarks that expose strengths and weaknesses in model performance across document types.
• Develop metrics, heuristics, and automated workflows to detect new failure modes in large, noisy enterprise datasets.
• Partner with ML engineers to translate evaluation insights into concrete model improvements and training priorities.
• Work directly with unstructured enterprise documents such as PDFs and spreadsheets to find edge cases and hard examples.
• Build lightweight internal and user-facing tools using Python frameworks like Flask to help teams inspect results and analyze model behavior.
• Collaborate with customers and internal stakeholders to understand real-world data needs and create benchmarks that highlight Reducto's strengths.
• Own the end to end lifecycle of evaluation experiments, from hypothesis to measurement and reporting.
• Drive standardization of evaluation practices so that model comparisons are consistent and meaningful over time.
• Surface insights about data quality, model drift, and user expectations to product and engineering teams.
• Contribute to the broader ML research community by publishing internal findings and patterns where appropriate.

Requirements

• Hold a high bar for quality and precision in all evaluation work and deliverables.
• Enjoy solving complex problems and building solutions from first principles.
• Have strong Python skills and the ability to independently build clean, reliable technical solutions.
• Be comfortable working with data infrastructure such as AWS S3 and OLAP or analytics systems like Tinybird.
• Love working with unstructured data and chasing down difficult failure cases in document workflows.
• Operate effectively in fast changing, high growth environments with shifting priorities.
• Collaborate smoothly across technical and non technical teams including product, engineering, and customer facing groups.
• Take full ownership from strategy through execution without needing constant direction.

Nice to have

• Have experience at an early stage or high growth startup.
• Have some background in product thinking and the ability to build simple, polished user facing interfaces.
• Be comfortable working directly with customers to understand their workflows and data needs.
• Have experience in AI or ML, data infrastructure, enterprise software, or document understanding systems.
• Care deeply about combining technical excellence with measurable business impact.

Practical notes

This is an in person role at our office in SF. We are an early stage company which means that the role requires working hard and moving quickly. Please only apply if that excites you.

Benefits at Reducto

At Reducto, we are invested in the well being and growth of our team. Here is what we currently offer.

• Unlimited PTO: We believe great work requires recharging.
• Lunch: Receive a free lunch to eat with your teammates daily at the office.
• Reimbursed Transportation: Provide us with your receipts and we will take care of the costs.
• Insurance: Generous health insurance covering medical, dental, and vision.
• Health and Wellness Budget: We provide up to $150 per month reimbursement for health and wellness spending, such as gym memberships, fitness classes, or similar.
• Parental Leave: Work with us to build a leave schedule that works for you and your family.

Reducto is an Equal Opportunity Employer committed to diversity and inclusion in the workplace. All qualified applicants will receive consideration for employment without regard to sex, race, color, age, national origin, religion, physical and mental disability, genetic information, marital status, sexual orientation, gender identity or assignment, citizenship, pregnancy or maternity, protected veteran status, or any other status prohibited by applicable national, federal, state or local law.