Your AI voice agent talks over callers because its barge-in handling is mistuned, not because the model is bad. When a caller starts speaking, the agent has to stop the audio already streaming down the phone line and dump the text-to-speech buffer queued behind it. Most stacks only stop generating new speech, so the two or three seconds already buffered keep playing straight over the caller. Swapping to a faster or smarter model does nothing for this. It's a pipeline setting, and it's usually fixable in an afternoon.
We run inbound and outbound voice agents for clients on Retell, and interruption handling is the single thing we tune most after launch. Nobody ever complains that the agent's script is bad. They complain that it "doesn't listen."
What barge-in actually is
Barge-in is the agent's ability to shut up the instant a human starts talking. Sounds trivial. It isn't, because four separate things have to cooperate:
- Voice activity detection — hearing that there is human speech on the line at all, and not a dog, a TV, or line noise.
- End-of-turn detection — deciding whether the caller is taking the turn or just backchanneling. "Mm-hm" is not an interruption. "Wait, no, I said Tuesday" is.
- Audio flush — killing playback and clearing the queued speech fast enough that the caller doesn't hear the collision. The 2026 production bar for this flush is under 60ms.
- Context recovery — resuming intelligently. If the agent already confirmed the caller's email and then gets interrupted, it should not go back and ask for the email again.
Get one of those wrong and the call feels off in a way clients describe as "robotic" even though the voice itself sounds great.
The three causes we find on almost every broken agent
The buffer never gets flushed
This is the big one. Text-to-speech streams ahead of playback — that's what makes it sound smooth. But it means at any given moment there's a chunk of speech already generated and sitting in a queue. If your interrupt handler stops generation without clearing that queue, the caller gets talked over for as long as the buffer is deep. On a chatty agent with long responses, that's a genuinely rude two seconds.
Test it in thirty seconds: call your agent, wait for it to start a long sentence, then say something. If it stops mid-word, your flush is working. If it finishes the sentence first, it isn't.
Interruption sensitivity is set for a quiet room
Every platform ships a sensitivity dial. Retell exposes it directly as interruption sensitivity in the agent's basic settings, and the default assumes a reasonably quiet caller.
Real calls are not quiet. Someone calls from a car with the radio on. A kid yells in the background. A speakerphone echoes the agent's own voice back into the mic and the agent interrupts itself — which sounds completely unhinged on a recording, and we've heard it more than once.
High sensitivity buys responsiveness and costs you false interruptions. Low sensitivity buys resilience and costs you the ability to cut in. There is no universally right number, and that's the point: this should be set per campaign, not once globally. Our outbound agents calling mobile numbers run noticeably less sensitive than our inbound line for a professional services client.
Turn detection is guessing from silence alone
If the only signal for "the caller is done" is a silence timer, you get both failure modes at once. Set it short and the agent jumps in while someone is still thinking mid-sentence. Set it long and every exchange gains an awkward beat.
The fix is to stop treating silence as the only evidence. Most speech-to-text providers expose the silence threshold directly — Deepgram calls it endpointing — but the threshold alone is a blunt instrument. Use the partial transcript. Require a minimum number of real transcribed words before you hand over the turn, so filler stays filler. Semantic turn detection — actually looking at whether the sentence sounds finished — is what separates the agents that feel human from the ones that feel like an IVR with a better voice.
The latency budget we hold ourselves to
Once interruptions behave, latency is the next thing callers feel. Humans hand off turns in roughly 200ms. Published 2026 benchmarks put production voice agents around 680ms at the median and 1,180ms at p95, and the practical rule matches what we hear on recordings: under 800ms feels like a conversation, over 1,500ms feels broken.
We budget the round trip like this:
- Network and carrier hop — 30–80ms. Mostly geography and your SIP provider. Cheap to fix by moving regions, impossible to fix by tuning.
- Turn detection — 150–300ms. The biggest single variable, and the one most teams never touch.
- Speech-to-text — 50–100ms after end of speech, assuming you're streaming. If you're not streaming, this number is a lot worse and you should fix that before anything else.
- Model time to first token — 150–400ms. Driven by model choice and prompt size. A bloated system prompt is a latency bill you pay on every single turn.
- Text-to-speech time to first audio — 100–200ms.
- Function calls — 100–500ms. Any CRM lookup or calendar check lands here, and this is where a slow client API quietly wrecks an otherwise fast agent.
That's the honest reason we push clients to trim system prompts and cache lookups: it's not tidiness, it's 200ms a turn. The same discipline shows up in the cost per minute math — shorter, tighter turns are both faster and cheaper.
What we actually change first
In order, because order matters:
- Flush the buffer properly. Nothing else matters until an interruption truly stops the audio.
- Drop interruption sensitivity for noisy campaigns. Mobile-heavy outbound lists almost always need this.
- Gate interruptions on transcribed words, so backchannels don't hijack the turn.
- Cut the system prompt. We've taken 400ms off a turn by deleting instructions the agent was never using.
- Move function calls off the critical path where possible — pre-fetch the CRM record when the call connects instead of mid-sentence.
- Only then shop for a faster model or voice.
Most teams do this list backwards, starting with model shopping, and wonder why a benchmark-winning stack still gets complaints.
Why outbound is harder than inbound
Inbound callers are ready to talk. They dialed you, they're usually somewhere they can hear, and they expect to have a conversation. Outbound is the opposite on every count, and the same agent config will fall apart on it.
Someone picking up an outbound call is doing something else. They're driving, they're in a store, they've got a TV on. They answer with "hello?" and then say nothing while they figure out who this is — which a silence-based turn detector reads as the end of their turn, so the agent charges ahead into its opener while the caller is still saying "…hello?"
Then there's the echo problem. Speakerphone in a car is the worst case we deal with: the agent's own voice comes back through the microphone loud enough that a sensitive detector treats it as the caller barging in, and the agent cuts itself off mid-sentence for no reason a human listener can identify. If you've heard your agent stutter and restart on a recording and couldn't explain it, that's what happened.
We keep separate agent configs for inbound and outbound rather than one tuned to the average of both, because the average is wrong for both.
The two numbers worth tracking
Forget the dashboard vanity metrics. We watch two:
- Collision rate — the share of calls with at least one moment where both parties spoke for more than about 300ms. This is the number clients actually feel. The 2026 production target for false barge-ins is under 2%.
- p95 response latency, not p50. Your median can look great while one turn in twenty takes 1.8 seconds — and that one turn is the one where the caller decides the thing is broken.
If you're picking a platform rather than fixing one, we broke down how the two main options compare in Vapi vs Retell for agencies, and our full deployment process is in how we deploy Retell agents for client lead follow-up. The payoff for getting this right is real — clean turn-taking is a big part of why voice agents cut no-shows instead of annoying people.
One last thing about testing
Test calls from your own desk are useless for this. Your office is quiet, your headset is good, and you know what the agent is about to say so you never interrupt it naturally. Every barge-in bug we've found came from listening to recordings of real leads on real phones in real cars.
Pull ten of those, mark every collision with a timestamp, tune, then replay the same ten. If the collision count doesn't drop, you changed the wrong setting.
Get a free automation audit
If your voice agent is live and callers keep saying it doesn't listen, we'll listen to the recordings with you and tell you which of the three causes above you have. No pitch deck.
Get a free automation audit — or see how we build and tune voice AI agents for clients.