Teaching a voice agent to wait: holds, interruptions, and per-call playbooks

Kate is the outbound assistant I use for my own errands. She runs on Retell, and I can start a call from Claude on my phone. She places food orders, calls offices, and sits through phone trees so I don’t have to. The receptionists we build for clients answer the phone. Kate makes the call, which means she’s the one on the receiving end of “can you hold one second?” and “what else?” Those two moments were where she kept going wrong.

This post covers the fix in one afternoon: two short prompt sections, three speech settings, and a way to give her a different playbook for each kind of call. Most of it applies to any Retell agent that has to share a phone line politely.

Some of the problem was in the settings, not the prompt

The first step was to read Kate’s agent config before changing anything. Two of the settings I wanted to change weren’t in the response at all.

Retell’s get-agent leaves out any field that was never set, and the agent then runs on the default. Here are three defaults that matter on a call that includes a hold:

Setting	Kate’s value	What it means
`reminder_trigger_ms`	unset (10,000)	After 10 seconds of silence, say something
`reminder_max_count`	unset (1)	Do that once
`responsiveness`	unset (1.0)	Reply as fast as possible

Put those together and an agent that has just been told “hold on” waits 10 seconds, then speaks into the hold. No instruction in the prompt overrides that, because the nudge comes from the agent settings, not from the model deciding to talk.

We changed three values:

reminder_max_count → 0. No silence nudges at all. Retell still hangs up after 10 minutes of total silence, which is the safety net.
responsiveness → 0.55. She waits a beat longer before answering, which costs a little snappiness and avoids stepping on someone who is still talking.
interruption_sensitivity → 0.85, up from 0.8. When the other person starts talking, she stops sooner.

If you are debugging an agent that “won’t stay quiet,” check these before rewriting the prompt.

The prompt half: let the other side lead

The prompt additions were short and concrete. The hold rule:

If anyone asks you to hold, wait, or “one second,” reply only “Sure, no problem” and stay silent until a person speaks to you first. Silence, hold music, or background noise is not a cue to talk.

The ordering rules came from watching how a counter person actually takes an order. They lead, they ask “what else?”, and they want one item at a time. Kate now answers whatever was just asked before raising anything new. She gives one complete item per turn, says up front how many items there are, and only asks about the total and pickup time after the store has the full order and a name. A final, required read-back of every item, the name, the ready time, and the total comes before goodbye.

Appending rules can create a contradiction

The instruction was to append these sections without removing anything. Before applying, we diffed the result and found a line further up that the new rule contradicted:

If the user says “hold on”, reply exactly: “NO_RESPONSE_NEEDED”.

With both in the prompt, the model has two different instructions for the same moment, and one of them is a token that sounds terrible if it ever gets spoken aloud. We replaced that single line with a pointer to the new section. If you only ever append, read the whole prompt for rules about the same situation first. A long prompt collects these.

A draft version means changes go live with no undo

Retell versions agents: you edit a draft, publish it, and published versions are read-only. Kate’s calls were pinned to version 1, and version 1 had never been published. So every edit went live on her next call, and there was no older published version to fall back to.

That’s fine for a personal assistant. It’s the setup you want for fast iteration. But it means the backup has to come from you. We saved the full agent and LLM JSON before each change, committed them to a private repo, and wrote a README listing which file came from which change.

Re-read the prompt before every write

Between our first update and our second, about an hour apart, I edited Kate’s prompt directly. I fixed a house number and reworked the ordering read-back. If we had built the second change from the text we’d saved locally, we would have silently reverted all of that.

We already learned this one the hard way testing our receptionists, and it held up again: fetch the live prompt, apply the change to that, diff, then write. The apply step now also checks that the prompt’s last-modified time hasn’t changed since the proposal was reviewed.

Playbooks: different instructions per kind of call

A food order, a pharmacy refill, and a billing dispute each need different instructions. Packing all three into one prompt makes it longer and full of conflicts. On a billing call, Kate should present as the account contact. On a food order, she’s plainly “Steve’s assistant.” Those two rules can’t both sit in the general prompt without clashing.

So the general prompt now ends with one section:

## SCENARIO PLAYBOOK
Instructions for this specific kind of call. Where they conflict with the general
guidance above, follow the playbook. If nothing appears below this line, use the
general guidance.

{{playbook}}

The playbooks themselves live in a Cloudflare KV namespace, one key per scenario. When Claude places a call, it passes a scenario name. The Worker looks up the text and sends it to Retell as the playbook dynamic variable. Editing a playbook is a write to KV, with no prompt change and no redeploy.

Three details made this work.

An unsent variable stays in the prompt as literal text. Retell leaves a variable that has no value exactly as written, curly braces included. Our older call path, a shell script, doesn’t send playbook at all. Retell LLMs have a default_dynamic_variables field, and an empty string counts as a real value, so setting playbook: "" there fixes the old path without touching it.

Reject unknown scenarios instead of falling back. The first design fell back to the general playbook when a scenario name didn’t exist. We changed that: an unknown name now places no call, and the error lists the scenarios that do exist. A typo like pharmacey should fail loudly, not quietly run a refill call with no instructions.

Give the model a way to undo. The tools that write a playbook, or change the prompt itself, return the previous text in their response. If an edit made from the phone goes wrong, the rollback is already in the conversation.

Per-call limits for long holds

Some calls are mostly waiting: insurance companies, pharmacies, anything with “your call is important to us.” Kate’s normal limits are 10 minutes of silence and an hour total. Raising those for every call would mean a stuck call could run for hours.

Retell’s create-phone-call accepts an agent_override with an agent object that changes agent settings for one call only. A long_hold: true flag on our call tool raises end_call_after_silence_ms and max_call_duration_ms to two hours for that call. Two hours is the documented ceiling for call duration. Retell documents no ceiling for the silence timeout, so we matched it to the call length, since the silence can’t outlast the call anyway.

Retell API changes worth knowing about this fall

If you have scripts against Retell’s agent version endpoints, two changed recently:

GET /get-agent-versions/{agent_id} was removed on September 15, 2026. Use GET /list-agent-versions/{agent_id}, which is paginated.
POST /publish-agent/{agent_id} is deprecated. Use POST /publish-agent-version/{agent_id} with {"version": N}, which publishes a draft in place and keeps its number.

Where it stands

The settings and prompt changes are live, and the playbook tools are deployed. The playbook store is still empty, which is deliberate: every call runs exactly as before until we write the first playbook. Billing goes first, since it’s the scenario where the general rules genuinely conflict.

The lesson I’d keep is the one from the top of this post. When a voice agent misbehaves, rewriting the prompt feels like the obvious fix. But “talks when it should be quiet” is a timing problem, and the timing lives in the agent settings. Read the settings first, including the ones that aren’t in the response.