Research Scientist

About the role

This position involves conducting novel research to improve the understanding, assessment, and selection of large language models. You will work with an extensive collection of AI-generated text from various models and providers to drive advancements in the field. Your primary contribution will be through the quality and influence of your research findings.

Key facts

What you'll do

• Lead and execute a research plan focused on evaluating LLM quality, optimizing model selection, and understanding usage trends, generating new insights for the AI community.
• Create innovative methods for assessing model performance that go beyond standard metrics, utilizing real-world data to understand how models function in diverse situations.
• Perform extensive studies on LLM behavior, comparing models across different platforms, tracking performance evolution, and identifying model strengths and weaknesses through usage data.
• Establish the statistical and mathematical underpinnings for our model selection systems, developing algorithms that guide intelligent choices between providers and models.
• Discover ways to apply research outcomes to enhance OpenRouter's products and platform.
• Engage with external researchers, model developers, and the open-source community to foster a collective understanding of LLM capabilities.
• Work with product and engineering teams to integrate research discoveries into the OpenRouter platform, without being bound by strict release schedules.

Requirements

• Hold a Master's or PhD degree in a quantitative discipline such as machine learning, statistics, computer science, mathematics, or computational linguistics.
• Possess a proven history of original research, evidenced by publications where you are the primary author, substantial contributions to open-source projects, or equivalent industry research achievements.
• Demonstrate deep knowledge in statistics, experimental design, and causal inference, enabling the creation of sound studies and careful analysis of validity and bias.
• Exhibit strong programming abilities in Python, including experience building data pipelines, conducting large-scale experiments, and prototyping models efficiently.
• Be proficient in SQL for querying large analytical databases like ClickHouse or BigQuery.
• Have practical experience with current machine learning and natural language processing techniques, including LLM evaluation, fine-tuning, embeddings, classification, or reinforcement learning from human feedback.
• Be familiar with the current LLM landscape, including model architectures, provider ecosystems, benchmark suites, and the capabilities and limitations of prominent models.

Nice to have

• A deeply curious and self-motivated approach to identifying and pursuing important research questions.
• A balance of scientific rigor and pragmatic execution suitable for a fast-paced startup environment.
• Extensive personal use of LLMs, coding assistants, and modern AI tools in your research process, with clear preferences for effective tools.
• Excellent communication skills, capable of explaining complex findings to both technical and non-technical audiences through various formats.
• A collaborative spirit, working effectively with product and engineering teams to translate research insights into practical recommendations.

Skills & tools

• Python
• SQL
• LLM Evaluation
• Machine Learning
• Statistics
• Experimental Design
• Causal Inference
• ClickHouse
• BigQuery

Practical notes

• OpenRouter sponsors visas.
• Travel is not required for this role.
• Benefits include equity.
• If you are interested in the role but do not meet every listed qualification, please still apply. We value enthusiasm and potential.