Most reviews like this come from course marketers, but I personally tested each resource as a Senior ML Eval Engineer who successfully transitioned into testing ML systems from Software Quality Assurance (QA).
My name is Lilia Urmazova, and I spent 20 years as a QA engineer (Manual, Automation, Lead, and QA Manager). A few years ago, when I transitioned into ML, a lot of my QA skills turned out to be directly transferable. As a result, my salary increased significantly, and my job security improved sharply.
Of course, with a career change like this, there’s a lot to learn, and that’s where things get tricky. Finding quality learning resources for AI app testing is challenging. While studying, I kept running into superficial, AI-generated books with 4.5+ ratings on Amazon. LLMs can help with the fundamentals, but their knowledge may not reflect the latest releases in this rapidly evolving industry. So it’s better not to look for a single “best” course, but rather to piece together the most up-to-date information bit by bit.
Intro Level: Traditional QA Using AI Tools
First, make sure you clearly distinguish between Traditional QA using AI, an AI QA Engineer, and an ML Evaluation Engineer:
The first uses AI tools to test traditional applications.
The second tests AI features built into standard applications.
The third tests applications where AI is the core functionality.
0. Generative AI in Software Testing Training (Course) – For Traditional QA using AI tools, this course is more than sufficient. Coursera may offer eligible learners a 7-day trial, which can be enough time to complete it.
Level I: AI QA Engineer
An AI QA Engineer tests the AI functionality built into traditional apps – the bulk of today’s market. This often involves black-box testing.
Working with AI applications without knowing how to code is difficult. Java is fine, but Python is the lingua franca of ML/AI – its ecosystem for models, stats, and transformers is unmatched. If you’re in Java QA, it’s worth learning Python. You can skip a formal Python course if you learn best by building projects. I opted for a tough pet project paired with Claude Code (pro tip: set Output style → Learning mode in the settings). Don’t devolve into lazy “vibe coding” – write code yourself and dissect the AI’s logic. I found the paid subscription valuable, though it’s not essential for everyone.
1. LLM Evaluation for Beginners (Free Guide) – A brief guide to getting started with AI application testing.
2. How To Test AI Applications (Free Course) – You may find yourself, just as I once did, tasked with testing an AI application for the first time as early as tomorrow. Finding very few resources back then, I later ended up building this free introductory course to cover the basics.
3. Claude 101, Claude Code in Action, Introduction to Subagents, AI Capabilities and Limitations (Free Courses) – Who can explain AI better than its creators? Claude Code is a useful tool for IT professionals. That’s exactly why you should take these introductory courses at Anthropic Academy.
4. Building with the Claude API (Free Course) – To test AI applications, you’ll need to constantly use AI via APIs – essentially, writing your own small AI applications for testing. This course from Anthropic is much longer than the previous ones, but it will give you a strong foundation (which can also be applied when working with OpenAI models).
Don’t just read through these materials – code your way through all of them alongside a coding assistant such as Claude Code.
Level II: ML Evaluation Engineer
Simply put, an ML Evaluation Engineer is an AI Quality Assurance Engineer – after all, AI model evaluation involves many of the same principles as QA. At this stage, you can choose between two specializations: LLM Model Evaluation (assessing the performance, accuracy, and behavior of individual models) and LLM System Evaluation (evaluating entire AI applications against business requirements). In both cases, QA may increasingly involve gray-box testing.
5. Chip Huyen – AI Engineering: Building Applications with Foundation Models (Book) – Highly recommended as the bible for AI application testing. Don’t let the title fool you – evaluation is a major focus throughout the book. It costs around $50, but the ROI is huge.
6. Evaluating AI Agents (Free Course) – How do you test AI agents and trace their steps and tool invocations? This concise hands-on course provides the blueprint using Phoenix (an AI observability tool) as the primary example. Other tools, including Langfuse, follow similar principles.
7. Building and Evaluating Advanced RAG (Free Course) – A slightly more challenging hands-on course introducing the RAG testing process an ML Eval Engineer faces daily. You’ll learn the basics using TruLens (an LLM evaluation framework similar to Ragas or DeepEval).
Level III: ML Engineer
An ML Evaluation Engineer builds metrics, benchmarks, and eval infrastructure, while an ML Engineer designs, trains, and deploys models.
Today, that ML Engineer path splits into two distinct tracks: Classic ML Engineer (focused on tabular data, statistical modeling, and feature pipelines) and AI/GenAI Engineer (focused on LLMs, RAG, and fine-tuning foundation models).
Because ML Eval Engineers already write code, work with LLM stacks, and evaluate model behavior, the jump to an AI/GenAI ML Engineer role – with the potential for a salary increase – is far smaller than a traditional QA-to-Dev pivot.
8. Machine Learning Crash Course (Free Course) – A course from Google on machine learning concepts and algorithms. The material is very well organized, so it’s worth reviewing when preparing for interviews.
9. Building LLMs From Scratch (Free Course) – You can’t evaluate what you don’t understand. I had to learn how models are built from scratch to understand how to measure them. Special thanks to Dr. Raj Abhijit Dandekar for a fantastic course that covers LLM fundamentals from the ground up, with an in-depth look at Transformer architecture along the way.
10. Chip Huyen – Designing Machine Learning Systems (Book) – Yes, this is the same author as the book above – Chip brings a rare combination of deep, Stanford-level academic expertise and practical experience at Nvidia, Netflix, and other companies. And this is a book I highly recommend reading at this level. To get the most out of this book, don’t just read it – work through the code examples in each chapter. Check GitHub for code snippets.
I’m always exploring new AI testing/ML Eval resources, so I’ll keep updating this list here on my page.
