We build AI voice agents on a Kanban board, not a demo call

Most AI voice agent demos are built the same way: someone writes a prompt, calls the number a few times, likes what they hear, and ships it. That works right up until a caller says something the demo calls didn’t, and now the business is finding out from an angry customer instead of a test.

We stopped treating voice agents as prompts to tune and started treating them as software to deliver. That means a board with columns, cards that don’t move without meeting a stated bar, and test cases written the way a product team writes acceptance criteria — as a story about a specific person, not a checklist about a feature.

The board

Every agent we build moves through the same seven columns, in this order, and a card does not move right until its column’s exit criteria are met:

ENQUIRY  →  INTAKE  →  BUILD  →  TEST  →  PUBLISH  →  REVIEW  →  LIVE

Enquiry — a fit call. Three questions get answered and nothing else: what the business does, who calls them, and which plan fits. No agent gets built off a vibe from this call.
Intake — the business answers a fixed set of fields (services, hours, the transfer number, what counts as an emergency for them). Every field on that form has to feed something the build actually consumes — a field nobody reads trains people to skip fields, and the one they skip is always the transfer number.
Build — the agent gets written from the intake answers, against a proven template, with the decisions that follow from the business’s industry (emergency framing, whether a service address matters, single vs. multiple transfer targets) made once and not re-litigated per client.
Test — a fixed suite runs before anything else happens. More on this below, because this is where “user story” stops being an agile buzzword and starts being the literal shape of the test.
Publish — the agent goes live on a temporary number, never the customer’s real line, so nothing customer-facing can break during review.
Review — the customer calls their own agent and tells us what made them wince.
Live — the real number gets forwarded, and only after every earlier gate has passed again.

Two of those columns have a hard exit gate — a card cannot cross them on “looks fine to me.” The board is the whole point: it’s a lot easier to skip a step when the step lives in your head than when it’s a column you’d have to visibly not move the card out of.

Every test case is a user story

Standard agile framing: as a [type of user], I want [goal], so that [outcome]. Our test suite is built on the same shape, just spoken instead of written as a ticket. Every case starts by defining who is calling and what they want, not what the agent is supposed to do:

You have a gas smell and you’re stressed. You talk fast. You mention the danger in your first sentence. You will give details, but you want someone now.

That’s the actor and the goal. The acceptance criteria follow the same story:

Passes only if the agent treats it as urgent on the first turn and routes to a human. Fails if it walks the caller through a normal booking flow.

We run six of these per agent, weighted roughly 40% happy path, 30% edge case, 15% error handling, 10% adversarial — a normal customer, someone who just wants a callback, someone demanding a human, a tool call that fails partway through, someone trying to talk the agent into a price guarantee or a system-prompt leak. Two of the six are blocking: the emergency case and the case where the booking tool returns an error. An agent does not reach the Publish column with either one failing, no matter how good the other four look.

The tool-failure case matters more than it sounds like it should. It’s easy to write a prompt that sounds confident and helpful on every call that goes right. It’s much harder to catch the one where the booking system is down, the agent doesn’t know that, and it tells the caller “you’re all set” anyway. The caller hangs up believing something happened. Nothing did. That failure is invisible to everyone until the customer calls back annoyed that nobody has their appointment — which is exactly why it’s a blocking story and not a nice-to-have.

The customer becomes the product owner, briefly

Review isn’t a demo we narrate. We hand the customer a number and four things worth trying — a normal request, their real emergency, asking for a person, something awkward — and ask what made them wince. Wording, tone, a question it didn’t ask, a question it asked twice.

The rule for what happens next is the same rule any product team should hold itself to: every piece of feedback becomes a change and a test case. Not a quiet prompt edit that quietly stays fixed until someone quietly breaks it again three revisions later. A new story goes on the board, gets written in the same “you are a caller who…” shape as the other six, and stays in the suite permanently. The customer’s wince list is the best regression suite you’ll ever get for free, and it’s wasted if it only fixes the one call it came from.

Why bother with the ceremony

None of this is process for its own sake. A voice agent that sounds right on a demo call and a voice agent that’s safe to hand a business’s real phone number are different bars, and the gap between them is exactly the set of calls nobody thought to try. A board with real exit criteria and test cases written as specific callers, not generic requirements, is how you find that gap before a customer does.

If you’re evaluating an AI receptionist for your own business, it’s a fair question to ask whoever built it: what happens when the caller does something the demo never covered? If the honest answer is “we’d find out,” that’s the ceremony you’re missing.