Product Update

Agent Evals: how we test AI agents with simulated customers and human-aligned judges

Ish Jindal7 minutes read

Agent Evals: simulated customers, human-aligned judges

✓ Last Updated: September 19, 2026

Our AI agent evaluation works by making our AI agents hold full conversations with simulated customers, which we call Simulation Users, before a real customer ever arrives. We start by writing down what the agent is supposed to do, and we check that against what customers really ask. Then a person reads the transcripts and marks the exact words where the agent went wrong. Only after that does anything automatic get to score a conversation.

Most companies selling AI agents say theirs is reliable. It is much harder to find one that shows how it checks. So here is how we check ours.

The first time I ran our evaluation setup against one of my own agents, every simulation came back failed. My honest reaction sat somewhere between "good, it works" and "so what do I do with this now?" Answering that second question is where most of the work has gone since.

We call this Agent Evals. It is how we test the AI agents we deploy.

The agent evaluation loop: a written brief, topics checked against the brief, customer profiles, objectives, a plan of test cases, scenarios, simulations, human review, named failure modes and approved evaluators — The loop, in the order we work through it. Orange steps are done by a person.

Why AI agent evaluation is not like testing software

Normal software testing works because the same input gives you the same output. A conversation does not behave that way. The customer gives you half the information. They change their mind in the third message. So you cannot write a fixed test case for one. You need a test customer who talks back.

That is why AI agent evaluation judges complete conversations and not single answers. Did the agent give correct information? Did it use the right tools and follow the process it was supposed to follow? Did it hand off to a person when it should have?

We start by writing down what the agent is for

Before anything is generated, we fill in an Evaluation Brief for the agent. It is a plain form that asks for the agent's goals, the policies and guardrails it has to respect, and who its customers are.

We added it because of a gap we kept hitting. Past conversations tell you what has happened. They do not tell you what the business needs the agent to get right. A refund rule that no customer has tested yet is still a rule the agent has to follow.

The brief also means we can start before an agent has had a single conversation.

The Evaluate section of an agent, open on the Evaluation Brief tab before a brief has been written. The tabs read Evaluation Brief, Topics, Scenarios, Simulations, Analysis and Evaluators — Where the loop starts. The brief can be written before the agent has had a single conversation.

Then we check the brief against what customers really ask

The brief is only our idea of the customer, so we check it against what customers really ask.

Topics are the actual reasons customers write in. Refunds, delivery delays, plan upgrades, whatever it is for that agent. They are pulled from past conversations and then compared with the brief.

Each topic gets a label that says where it came from. Observed means it showed up in conversations. Intended means it came from the brief. Hybrid means both. Topics that came from conversations show how many sit behind them, and we approve or archive topics until the list matches reality.

Customer Profiles are built the same way. A profile says who the customer is, how they behave, and how they write, with the conversations that back up each part.

New profiles wait for review. Nothing is approved without a person, and each topic and profile says which source it came from.

We pick the test cases before any scenario gets written

With the brief, the topics and the customer profiles in place, we set Objectives. Each objective states what an evaluation has to prove. We write some by hand. Others get drafted for us from those three inputs, and each one waits until a person approves it.

Objectives keep the test cases pointed at what the business cares about. The approved topics, customer profiles and objectives produce a list of proposed cases. Each case is one situation in one sentence, marked required, recommended or optional.

We pick the ones worth writing, and only those become scenarios.

An earlier version tried to fill every combination of topic and customer profile. Most of those combinations were not useful.

A Simulation User never sees the answer

A scenario is what a Simulation User works from. It says what the situation is, how the customer behaves and follows up, and when the customer stops. It also says what the agent must do and must not do.

The Simulation User never sees what the agent is expected to do, so a test cannot lead the agent to the right answer. If a scenario's wording sits too close to the expected answer, it gets flagged for a person to check.

Each scenario waits for a person to approve it before it is used.

We first called these fake users. That name did not survive its first review, because "fake" lowers trust in the thing you are measuring with. They are Simulation Users. We act on what these tests say, so the name matters.

A simulation run gives us transcripts to read

The chosen scenarios run against the agent as a suite. Each run is recorded with its transcript, shown the way the customer saw it, with the chat bubbles, buttons and images in place.

An earlier version stamped every run pass or fail. We took that out, because those results could give a misleading view of how good a simulation really was. A run now shows that it finished and keeps the transcript. It carries no verdict until a person gives it one.

The part I did not expect to care about is running the same scenario several times. An agent that handles a question well once and badly the next two times is a flaky agent, and that is very hard to catch by hand-testing.

A person reads the transcripts before any machine scores them

When we read a run and something is wrong, we select the exact words that went wrong and attach a short note. That marks the run as one that went wrong and files it in a review queue. A run only gets there because a person read it and marked the words.

In the queue, the reviewer adds the reasons. Each conversation gets one verdict, and the highlights are the evidence for it.

This is the slow part. The reviewed transcripts become the reference for the automated checks that follow.

Naming a failure gives us something we can actually fix

As we review, similar failures get grouped into named problems. Gives the wrong refund policy. Does not hand off to a human. Each suggested name is grounded in the exact lines we highlighted.

We confirm, rename and merge those names until they describe reality. What comes out is a known-issues list backed by transcripts.

"The agent is unreliable" is not something anyone can work on. "The agent gives the wrong refund policy when the order is over thirty days old" is.

A grader has to agree with us before its score counts

For each failure mode we can build an automatic checker, called an evaluator. It is either a simple rule or an AI judge that reads a conversation and calls it.

Before an evaluator is allowed to count for anything, it is run against the examples we already reviewed by hand. We see how often it agrees with us, and we see where it does not. Only evaluators we approve start running.

This is the step I would keep if I had to throw away everything else. A grader nobody has checked still gives you a score. You just have no idea what the score is worth until you know how often it agrees with a person who read the same transcript.

The hard part turned out to be the input

I assumed the difficult bit of this would be the testing. It turned out to be the quality of what goes in.

If the brief is vague and the topics are sloppy, the scenarios built on them are sloppy, and the result you get at the end is confident nonsense. Garbage in, garbage out is the oldest line in software, and it is most of the story of agent evaluation.

So the steps we protect are the human ones: writing the brief, approving the topics and scenarios, and reading the transcripts ourselves. The automation is there to repeat a judgement we already made.

What this means if you are deploying an AI agent

Whatever platform you are on, three questions are worth asking.

Where do the test cases come from? Real conversations show what customers ask. They do not cover a policy nobody has tested yet. A written brief does, but on its own it is only your idea of your customers. Ask for both.

Has a person read the transcripts themselves, and not just a summary of them?

Does your automated grader agree with that person, and do you know by how much? If nobody has ever measured that, the quality score on your dashboard is decoration.

Whether the agent found the right information in the first place is a separate measurement, and we wrote about it in Retrieval Evals. That post is about whether the agent found the right passage. This one is about whether the customer got what they came for.

In healthcare, banking, or anywhere else a wrong answer has consequences, I would rather find the failures myself than have a customer find them for me.

Like what you've read? Why not share it with a friend!

Build innovative AI Agents that deliver results

Get started for free

Ish Jindal

Ish is the co-founder at Tars. His day-to-day activities primarily involve making sure that the Tars tech team doesn’t burn the office to the ground. In the process, Ish has become the world champion at using a fire extinguisher and intends to participate in the World Fire Extinguisher championship next year.