AI Evals Explained: The Skill Every AI Engineer Needs

Your agent works. Does it work correctly?

Connect a model, give it some tools, write a few prompts, and suddenly you have an agent that answers questions, searches databases and takes actions for you. It is genuinely easy to get to that point today.

Now take an AI customer support agent as an example. You test it yourself, ask a few questions, and everything looks great. Then you release it to real customers, someone asks for a refund, and instead of following your company's refund policy the agent confidently invents a policy that does not exist. This is very common when there are no guardrails. Worse, the system does not tell you it made a mistake, because it does not know it made one.

That is why I think evaluations, or evals for short, are one of the most important skills for anyone becoming an AI engineer. Building the agent is going to be the easy half of the job. The other half is knowing whether you built it reliably and whether you can trust it. If you are earlier on that path, start with my roadmap on how I would become an AI engineer from zero.

Why normal software tests are not enough

Think about a hotel booking site. You pick your dates, choose a hotel, select a room and pay. The flow is designed by developers and is fairly predictable. If you book three nights at $200 a night, the system should show $600 before fees, and a developer can write an automated test to check that maths. That is how software has been tested for the past decade.

Now compare that with an AI chatbot. Instead of a calendar and a few buttons, you give users an empty text box. One person asks, "Can I get a refund?" Another says your app charged them twice. Someone else tells it to ignore its previous instructions and listen to them instead. The AI has to understand what each person wants, look up the right information and decide what to do. And unlike normal code, the same question can produce different answers.

When traditional software fails, you usually get an error. When AI fails, you might get an answer that is well written, polite and completely wrong. You cannot anticipate every question, because the space is close to infinite, and passing a test once does not guarantee the system passes it every time.

What an eval actually is

An AI eval is simply a way of measuring whether your AI system is doing what it is supposed to do. If you have done software engineering before, it is basically testing. The difference is that you are often evaluating something without one fixed correct answer.

Ask a support agent "Where is my order?" and "Your package arrives tomorrow" is a good answer. So is "I have checked your tracking and delivery is scheduled for Wednesday." Different words, both correct. So you cannot write a unit test that matches one exact sentence. Instead you define what success looks like, like a checklist: did it retrieve the right order, was the information accurate, did it follow company policy, and if the situation needed a human, did it escalate? Those become your criteria, and now you can measure performance instead of trusting answers that merely look good.

Model evals versus product evals

A question I see a lot on Reddit: if OpenAI, Google, Anthropic and xAI already evaluate their models, why do we need our own evals? Those leaderboard scores come from model evals. SWE-bench measures how well coding agents solve real problems from GitHub repositories, and MMLU-Pro tests knowledge and reasoning across subjects. They are useful for knowing what a model can do in general.

But a high benchmark score does not mean the model will perform well inside your product. Say you build an admissions assistant for a school. It needs to understand your courses, fees, payment plans, refund policy, entry requirements and what happens when a student wants to move to the next cohort. It might need to read your CRM and know when a student should speak to a human counsellor. No benchmark tells you whether it can do that, because none was designed around your business.

Model evals tell you how capable a model is on standardised tasks. Product evals tell you how well your complete system performs in your environment. When you build real AI products, product evals are what tell you whether you are ready for users.

Three ways to evaluate an AI system

You do not need AI for all of these. Each has a job, and each catches mistakes the others miss.

Building your first eval in five steps

Take an AI support agent for an online store. Customers message it about orders, payments, refunds and account issues, and we want to know when it should solve something itself and when it should hand over to a human.

Step one is a dataset. Do not let the word scare you. It is a spreadsheet with the customer's message, what the AI replied and what the correct behaviour should have been. A customer says they were charged twice, and the AI tells them to request a refund from the orders page, when policy says duplicate charges go to the billing team. Another says someone changed the email on their account, and the AI tells them to update it in settings, when the account may be compromised and should go to the security team.

Step two is spotting the pattern. Reviewing these, you notice the AI handles normal questions well but struggles to recognise when a human is needed. So instead of evaluating everything at once, you focus on that one problem and build an escalation eval. Step three is deciding exactly what you are measuring: did the AI make the right call on escalation? You can use tools like LangSmith or the open source Arize Phoenix, but a spreadsheet and a normal LLM API are enough to start.

Step four is the rubric, which is just the checklist an experienced support manager would use. Suspected fraud, account security issues or duplicate charges must be escalated. A customer who explicitly asks for a human after repeated failed attempts should be escalated. Otherwise, resolve normal questions without unnecessary escalation. If it should have escalated and did not, that is a fail. Step five is running it: the judge fills its own column with pass or fail and a short reason, so you can see what failed and why, and compare results before and after you change the agent. The tool matters less than having a repeatable way to measure whether your AI does the job. For how this fits into the wider system around the model, see my breakdown of harnesses, loops, evals and RAG.

Who judges the judge? A lesson from auditing

An AI judge is still AI. It can misunderstand instructions, misread context or give wrong scores. Anthropic shared an example where one of their models initially scored 42% on a research benchmark, and after researchers fixed problems in how the benchmark was designed and improved the agent setup, the score rose to 95%. That shows how misleading an evaluation can be when the evaluation itself is flawed.

This reminds me of my first job. I studied accounting and finance and started my career in financial audit in 2019. An auditor checks whether a company's reported cash, sales and expenses are real, but a company can have millions of transactions, so you never check them one by one. You take a random sample, and you also deliberately check the material items, the ones that would have a significant impact if they were wrong. If those check out, you have reasonable confidence the records are true and fair. Not perfect, but supported by enough evidence.

Apply the same thinking to evals. If your judge has scored 100,000 conversations, have a human check a random sample, say 100, and see whether they agree with the judge. On top of that, review the material cases: refunds, cancellations, any action that affects a customer financially. If you want to estimate the judge's overall accuracy, that needs a properly selected, large enough random sample. When the judge gets things wrong, improve its instructions and criteria. That is the loop.

Evaluate what it does, not just what it says

Especially with agents, do not only evaluate the words. If your agent promises to escalate a ticket, check that the ticket was escalated. AI always gives you a convincing response, so verify the action itself. This is why combining code-based checks, AI judges and human review is so powerful.

And evals are not something you build once before launch and forget. Real customers will ask in a different language, give incomplete information, or you will update the model and something that used to work starts failing. Good AI engineering is a continuous loop: build a dataset of test cases, run your AI against them, find failures, improve the system, release, monitor real conversations, and feed new failures back into the dataset. Each time you change something, you re-run the evals to check you improved things and did not break something that used to work. That is regression testing, and it will feel familiar if you have written normal software tests.

Your homework for the next agent you build

Do not stop at the impressive demo. Write 10 to 20 realistic questions people would actually ask, run them through your agent, review the responses, find one thing it consistently gets wrong, and build an eval to measure that problem. It does not need to be complicated or need a fancy platform.

Building an AI system is about making it work. Engineering an AI system is about making it work reliably. That distinction will matter more and more as AI moves from demos into real businesses, and it rests on solid software fundamentals: you cannot write a code-based eval if you cannot read the code. That is why our AI-Native Software Development Programme teaches fundamentals first and then the AI layer, evals included. If you want to see whether this work suits you first, try the free trial.