AI - Sr. Application Engineer

About the role

In this position, you will contribute to the intelligence layer for various applications, ensuring the quality of models, retrieval-augmented generation (RAG) accuracy, prompt engineering, and AI safety. Your work will involve developing a Socratic tutor persona, an adaptive learning recommendation engine, and a multi-modal AI system that utilizes both text and voice.

Key facts

What you'll do

• Oversee the quality and performance of AI models across multiple applications.
• Develop and implement RAG evaluation frameworks and feedback loops to enhance retrieval processes.
• Orchestrate multi-step LLM call chains, ensuring compatibility and efficiency.
• Ensure that all production-grade AI features meet accuracy and safety standards before release.

Requirements

• A minimum of 10 years of experience in IT.
• At least 4 to 7 years in software engineering, with a focus on LLM application development for production environments, not research or internal tools.
• Proven experience in delivering an LLM-powered product that is relied upon by real users, with responsibility for quality metrics.
• Experience in implementing AI safety measures for customer-facing products, including designing and testing safety layers.
• Proficiency in building RAG evaluation pipelines and making release decisions based on accuracy thresholds.
• Experience in optimizing multi-step LLM call chains for performance and latency.

LLM Application Development

• Expertise in LLM prompt engineering, including system prompts and instruction following.
• Proficient in orchestrating multi-step LLM chains using tools like LangChain or LlamaIndex.
• Advanced skills in managing multi-turn conversations, including context window management and session memory.
• Experience with streaming LLM response handling, including token-by-token streaming.
• Ability to select and benchmark models based on task requirements, balancing latency, cost, and accuracy.

RAG Pipeline Design & Quality

• Expertise in designing RAG pipelines, including chunking strategies and retrieval configurations.
• Advanced skills in tuning vector similarity search parameters and retrieval depth.
• Proficiency in reranking and relevance scoring methodologies.
• Experience with RAG evaluation frameworks and automated evaluation pipelines.
• Familiarity with hybrid search techniques that combine dense vector retrieval with traditional keyword search is a plus.

AI Safety & Guardrails

• Advanced knowledge of prompt injection detection and mitigation strategies.
• Experience in conducting jailbreak testing and red-teaming for LLM systems.
• Integration of content safety classifiers and hallucination detection strategies.
• Ability to enforce topical control on LLM responses.

Evaluation & Production Quality

• Skills in designing automated evaluation pipelines, including test set curation and regression detection.
• Familiarity with A/B testing methodologies for model and prompt changes.
• Experience in latency profiling for identifying bottlenecks in LLM call chains.
• Ability to design feedback loops for user signal collection and integration.
• Experience in monitoring production models for accuracy drift and quality degradation.

Development

• Expertise in Python for ML/AI application development, including asynchronous programming.
• Advanced skills in designing APIs for AI services, including error handling and timeout management.
• Proficiency in operations related to embedding models, including batch embedding and index updates.

Nice to have

• Experience with adaptive learning systems or personalization engines.
• Familiarity with knowledge graph integration in RAG contexts.
• Understanding of multi-agent orchestration patterns.
• Experience integrating with ServiceNow APIs.
• Prior experience in developing AI products on NVIDIA infrastructure.

Practical notes

This role is open to candidates with valid work authorization. Benefits and travel details will be discussed during the interview process.