Testing a Retell agent against what the caller actually heard

We run six demo receptionists on Retell: an HVAC shop, a fishing guide, a barber, a nail salon, a restaurant, and a front-door agent that hands callers to the right one. Every prompt change goes through a suite of 20 simulated calls in Retell’s testing tools before it gets published. In August the whole suite was green, and one of the agents was quietly losing jobs. This is how that happened, what fixed it, and what we changed about the tests so it stays fixed.

The bug the suite couldn’t see

The first round of testing found an ordinary problem. An agent would ask for a callback number, not hear it clearly, and ask again in exactly the same words, sometimes three or four times. We added what we call an escape hatch to every prompt: if you have asked for the same thing twice and still don’t have it, stop asking the same way. Offer a callback instead, and move on.

That fix worked. The loops disappeared. It also created a worse problem that the suite was not built to notice.

Here is the transcript that exposed it. The agent had already called book_appointment with no day and no confirmed number, because the simulated caller had been vague. Then the caller finally came through:

Caller: Did you still need my phone number? It’s 615-555-0199. And for the day, next Tuesday would work best.

Agent: Thanks, Marty. I’ve updated that to 615-555-0199, and next Tuesday works best.

There was no second tool call. The next event in the log is end_call. The office received a booking record with no day and no callback number. The caller hung up believing both were captured. Nobody on either end had any reason to think anything went wrong.

That is the failure mode that actually costs a small business money: the agent says the right words and sends nothing.

The test criterion for that case read, roughly, “the agent calls book_appointment at most once.” It was written before the escape hatch existed, it counted tool calls, and it passed. Counting tool calls cannot see a promise that was never kept.

The prompt fix

Two rules, added to all six prompts directly after the paragraph that tells the agent to read the booking back to the caller:

If book_appointment has already been called and the caller then supplies a day, a time, a callback number, or an address, call it again with the complete corrected details. Never say something was updated without sending it.
Do not fall back to a callback-only booking just because a day is hard to pin down. Offer two concrete choices first, tailored to the business (“Tuesday morning or Thursday afternoon?”). Callback-only requires the caller to say they cannot choose.

The second rule matters because the first one alone leaves the escape hatch pushing agents toward an empty booking. Giving them two concrete options to offer keeps the conversation moving without inviting a loop.

Rewriting the tests around the caller’s reality

The bigger change was to the suite. Every criterion now asks one question: does the caller’s understanding of what happened match what actually happened?

Concretely, the vague-caller cases were rewritten from “calls the booking tool at most once” to:

The final book_appointment call carries the complete corrected details. Extra calls are expected when the caller supplies something late.
The agent never claims something was updated, added, or changed without actually calling book_appointment again.

The same principle went through the other cases. A happy-path test used to pass if the tool fired. Now it passes only if the agent read the slot back and the caller confirmed it. The front-door agent’s handoff test used to pass if the swap tool was called. Now it passes only if a different business and a different receptionist actually greeted the caller afterward. A promise to connect that never lands does not count.

Things about the Retell API worth knowing before you do this

We spent a good part of a day learning these the slow way. The Retell API reference has the endpoints; what follows is what it does not say out loud.

You cannot patch a published LLM. PATCH /update-retell-llm/{id} on a published version returns 400 Cannot update published LLM. Fork first with POST /create-agent-version/{agent_id} and {"base_version": N}. That returns a draft agent version and forks the LLM into a matching draft. Then patch the draft with ?version=N.
Deleting an agent version does not delete the forked LLM version. Clean that up separately or you end up with an orphaned prompt that no agent points at. One of ours failed to publish with a bare Not Found for exactly this reason.
Test case definitions take PUT, not PATCH, and they take the whole object. Round-trip every field, including the metrics and the tool mocks, or you silently wipe them. We keep one mock per definition so no test run can ever write a real booking.
Batch tests pin a version. A stored test definition remembers the version it was created against. Rerun it without passing version explicitly and it grades a stale prompt while telling you the current one passed.
Never upload a stored full-prompt snapshot. We applied a day-old snapshot to one agent and reverted three newer rules without noticing until the next diff. Fetch the current prompt, apply the line-level change, diff, then push.

Where it landed

All 20 cases pass against the live published versions, under criteria that would have caught the original bug. Every happy-path grader now quotes the agent reading the slot back. Every vague-caller grader notes that the agent rephrased instead of repeating, and that the last booking call carried the corrected details.

The lesson we took from it is not really about Retell. If you test a voice agent by counting what it did, you will miss the cases where it said something and did nothing. Test what the caller walked away believing, and check it against the record the business actually received.