Note: The job is a remote job and is open to candidates in USA. OVA.Work is seeking an AI Evaluation Engineer to design, implement, and maintain evaluation frameworks for AI and machine learning systems, focusing on Large Language Models and generative AI applications. The role involves developing evaluation methodologies, assessing model performance, and collaborating with various teams to enhance AI system reliability.
Responsibilities
- Design and implement evaluation frameworks for AI, machine learning, and generative AI systems
- Develop automated and manual evaluation pipelines to assess model quality and performance
- Define evaluation metrics for accuracy, relevance, factuality, consistency, completeness, latency, and user satisfaction
- Create benchmark datasets, test suites, and evaluation scenarios for AI models
- Evaluate LLMs and AI applications for hallucinations, bias, toxicity, fairness, robustness, and safety
- Measure Retrieval-Augmented Generation (RAG) performance, including retrieval quality and response grounding
- Conduct A/B testing and comparative evaluations of models, prompts, and AI workflows
- Analyze evaluation results and provide actionable recommendations for model improvement
- Collaborate with AI engineers, data scientists, product managers, and QA teams throughout the AI development lifecycle
- Monitor production AI systems and identify performance degradation, data drift, and model drift
- Document evaluation methodologies, findings, and best practices
- Ensure compliance with organizational AI governance, privacy, security, and responsible AI policies
Skills
- Bachelor's degree in Computer Science, Artificial Intelligence, Data Science, Statistics, Mathematics, Software Engineering, or a related field
- 3–5+ years of experience in AI, machine learning, software engineering, data science, or model evaluation
- Strong understanding of machine learning concepts and evaluation methodologies
- Experience evaluating AI or generative AI applications
- Proficiency in Python and SQL
- Experience with REST APIs and cloud-based applications
- Master's degree in AI, Machine Learning, Data Science, Computer Science, or a related field
- Experience evaluating Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems
- Knowledge of Responsible AI, AI safety, and AI governance principles
- Experience with MLOps and model lifecycle management
- AI or cloud certifications (AWS, Azure, Google Cloud)
- Generative AI applications
- Conversational AI and chatbots
- Enterprise AI platforms
- Retrieval-Augmented Generation (RAG)
- Machine learning model validation
- AI-powered SaaS applications
- Regulated industries such as healthcare, finance, or insurance
Company Overview