QE for AI Engineer

About the Role

In this role as a Quality Engineer for AI Engineer, you will be responsible for evaluating, hardening, and governing GenAI and agentic systems before and after they go live. This involves ensuring that what we build works consistently, reliably, and safely at scale. You will partner closely with AI engineers, architects, and clients to define what correct means in an AI world and prove it.

This role sits at the intersection of engineering, QA, and AI safety, making it a unique opportunity to combine your technical skills with your passion for AI.

Key Responsibilities

• Evaluate and design evaluation frameworks for Large Language Models (LLMs), Retrieval Augmentation Generation (RAG) pipelines, and multi-agent systems.
• Develop test strategies for non-deterministic systems, including semantic correctness, hallucination detection, and consistency checks.
• Build testing pipelines for LLM evaluation, including offline and online evaluations.
• Establish ground truth datasets, benchmarks, and scoring metrics for AI systems.
• Validate multi-agent orchestration flows and test failure modes, including hallucination and incomplete reasoning.
• Ensure robustness across multi-step workflows and chained reasoning tasks.
• Automate prompt testing, regression testing, and response comparison, and integrate AI validation into CI/CD pipelines.

Skills & Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Experience with AI and machine learning, particularly in LLMs and RAG pipelines.
• Strong understanding of software development life cycles and testing methodologies.
• Proficiency in programming languages such as Python, Java, or C++.
• Familiarity with testing frameworks and tools, such as JUnit or Pytest.
• Experience with CI/CD pipelines and automation tools, such as Jenkins or GitLab CI/CD.
• Strong analytical and problem-solving skills, with the ability to debug complex AI systems.
• Excellent communication and collaboration skills, with the ability to work with cross-functional teams.

What You'll Learn

In this role, you will have the opportunity to learn about the latest advancements in AI and machine learning, particularly in the areas of LLMs and RAG pipelines. You will also gain experience with testing and validation methodologies, as well as automation tools and CI/CD pipelines.

Additionally, you will have the chance to work with a talented team of engineers, architects, and clients to define and implement AI solutions that meet the needs of our clients.

Resume Tip

When applying for this role, be sure to highlight your experience with AI and machine learning, particularly in LLMs and RAG pipelines. Emphasize your understanding of software development life cycles and testing methodologies, as well as your proficiency in programming languages and testing frameworks.

Also, be sure to mention any experience you have with CI/CD pipelines and automation tools, as well as your analytical and problem-solving skills.