Note: The job is a remote job and is open to candidates in USA. EPAM Systems is a leading company in the tech industry, and they are seeking a skilled Senior AI QA Engineer with strong experience in both manual and automated testing. The ideal candidate will test a variety of AI-based applications, ensuring reliability and accuracy while contributing to the development of automation capabilities.
Responsibilities
- Research and evolve automation frameworks in line with Gen AI tooling and best practices
- Design and automate the evaluation of Gen AI features — grounding, answer accuracy, determinism/reproducibility, precision, recall, and criteria recall
- Build automated LLM test harnesses that scale evaluation beyond human-in-the-loop
- Select and apply Gen AI evaluation frameworks, measuring answer quality and pipeline efficiency
- Perform manual testing as needed to validate new features, integrations, and user stories
- Build and maintain test cases from requirements and user stories
- Test applications that may include AI agents, APIs, databases, and other integrations
- Collaborate with product, engineering, and operations teams to understand requirements and deployment environments
- Track and report test results, defects, and quality metrics
- Assist with troubleshooting production issues; escalate risks as needed
- Guide and support team members, including onshore and offshore consultants
Skills
- 3+ years of experience in software QA, with at least 1 year focused on testing AI agents, agentic solutions, or LLM-based systems
- Hands-on experience with both manual and automated testing of AI agents, including prompt/instruction testing and evaluation of agentic workflows
- Strong programming skills in Python for test automation — pytest or equivalent, scripting, and AI/ML library integration
- Experience with AI agent frameworks, prompt engineering, and evaluation metrics for LLM-based systems
- Demonstrated experience in testing and evaluating Gen AI / LLM applications — grounding, answer accuracy, and hallucination/determinism checks
- Applied knowledge of Gen AI / LLM evaluation frameworks and metrics — precision, recall, criteria recall, and efficiency
- Experience with issue and test management tools (e.g., Jira, QMetry, TestRail)
- Experience with version control systems and integrating tests into CI/CD pipelines
- Experience using AI-powered tools for QA (e.g., GitHub Copilot, LLM-based test generation)
- Understanding of cloud environments, particularly AWS
- Excellent communication, collaboration, and leadership skills
- Strong English communication skills (B2 level or higher)
- Experience with agentic AI platforms (e.g., LangChain, OpenAI Function Calling, or similar)
- Skills in AI safety, bias, and reliability testing
- Background in test data generation for AI/ML systems
Benefits
- International projects with top brands
- Work with global teams of highly skilled, diverse peers
- Healthcare benefits
- Employee financial programs
- Paid time off and sick leave
- Upskilling, reskilling and certification courses
- Unlimited access to the LinkedIn Learning library and 22,000+ courses
- Global career opportunities
- Volunteer and community involvement opportunities
- EPAM Employee Groups
- Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn
Company Overview
EPAM leverages its core engineering expertise as a leading global product development and digital platform engineering services company. It was founded in 1993, and is headquartered in Newtown, Pennsylvania, USA, with a workforce of 10001+ employees. Its website is https://www.epam.com.Company H1B Sponsorship
EPAM Systems has a track record of offering H1B sponsorships, with 11 in 2026, 120 in 2025, 172 in 2024, 232 in 2023, 373 in 2022, 359 in 2021, 502 in 2020. Please note that this does not guarantee sponsorship for this specific role.