Senior QA Automation

About:

This role sits at the intersection of advanced AI development and rigorous quality assurance

within a leading technology organization focused on delivering secure, reliable solutions for

mission-critical sectors. The Senior QA Automation Engineer will play a key part in ensuring that

innovative AI and supply chain systems meet the highest standards of reliability and

compliance, supporting the organization’s commitment to operational excellence and

technological advancement.

Responsibilities:

• Lead the design and implementation of automated testing frameworks for AI and LLM-based

systems

• Develop and maintain automated test suites to validate AI outputs for accuracy, consistency,

and alignment with ground truth datasets

• Build and execute integration tests to ensure seamless data flow between AI agents and

enterprise systems such as ERP platforms

• Create and run performance tests to benchmark AI model latency, throughput, and reliability in

real-time environments

• Automate API testing to verify secure data exchange, RBAC enforcement, and PII redaction

before data reaches AI models

• Map automated test cases to system requirements and user needs to support formal

verification and validation processes

• Prepare and deliver test readiness documentation and evidence for stage gate and production

release reviews

• Manage defect identification, tracking, and resolution, ensuring traceability to specific model

versions or datasets

• Collaborate with cross-functional teams, including data scientists and systems engineers, to

align testing with acceptance criteria

Skills:

• Expert proficiency in Python for test automation
• Strong experience with Pytest, Selenium, and Playwright
• Advanced skills in performance testing tools (Locust, JMeter, K6)
• Hands-on experience evaluating LLMs and AI models (DeepEval, TruLens, custom evaluators)
• Proficiency in SQL and data validation tools (Great Expectations)
• Experience integrating automated tests into CI/CD pipelines (GitLab)
• Familiarity with requirements management and traceability tools (Jira, Jama, Polarion)
• Strong knowledge of test management and reporting tools (TestRail, Allure)
• Solid understanding of version control systems (Git, GitLab)
• Deep knowledge of modern quality engineering practices (Shift Left, Test Pyramid, mono-repo

architecture)