How Much Latency Can an AI Voice Agent Have Before Callers Hang Up?

Target under 800ms voice-to-voice; callers start noticing around 500ms. Human conversation turns over in about 200ms, so anything past half a second reads as hesitation. Peer-reviewed turn-taking research and ITU telephony standards both support the low end. The widely-quoted "callers hang up 40% more past one second" figure does not have a traceable study behind it.

What latency do humans actually expect?

About 200 milliseconds, and this is the one number in this article with genuinely solid science behind it. Human conversational turn-taking clusters tightly around a fifth of a second, across every language studied, which is why sub-second AI response times still feel slow rather than fast.

The reference is Stivers et al. (2009) in PNAS, which measured question-answer gaps across ten languages on five continents. The findings are precise: all distributions were unimodal with the highest number of transitions occurring between 0 and 200 ms, an overall mode of 0 ms, a cross-linguistic median of +100 ms, and a full-dataset mean response offset of +208 ms. Language-specific medians ran from 0 ms (English, Japanese, Tzeltal, Yélî-Dnye) to +300 ms (Danish, Lao, ‡Ākhoe Hai‖om). Danish was slowest at +469 ms mean; Japanese fastest at +7 ms.

The useful implication isn't "hit 200ms" — no production phone agent does. It's that the entire cross-cultural range of normal human response fits inside half a second. When your agent takes 900ms, it isn't slightly slow. It's outside the range of anything a caller has experienced from a human being, which is why the delay registers as something being wrong rather than as someone thinking.

What do telephony standards say about delay?

That 150ms one-way is effectively invisible and 400ms is the planning ceiling — and critically, this budget is consumed by the phone network before your agent has processed anything. ITU-T Recommendation G.114 is the standard here, and it's been the basis of voice network design for decades.

The verbatim language from G.114: although a few applications may be slightly affected by end-to-end "mouth-to-ear" delays of less than 150 ms, "if delays can be kept below this figure, most applications, both speech and non-speech, will experience essentially transparent interactivity." And: "While delays above 400 ms are unacceptable for general network planning purposes, it is recognized that in some exceptional cases this limit will be exceeded."

Two things follow that most latency articles miss. First, G.114 governs transmission delay — the network carrying audio — not processing time. Your agent's thinking time sits on top of it. Second, if your carrier leg is eating 70–160ms round trip, the standard's own "transparent" budget is largely spent before speech recognition starts. That's why the honest target for a phone agent is under 800ms rather than under 200ms: you're working with what the network leaves you.

What are the real thresholds callers react to?

There are four bands, and the boundaries are approximate rather than precise. Under 500ms feels like conversation. 500–800ms feels like a slightly slow person. 800ms–1.2s produces the "hello?" behaviour. Past 1.2s callers assume the line dropped and either talk over the agent or hang up.

Broken down further: under 300ms the caller does nothing at all — it's indistinguishable from a person. Between 300 and 500ms it still reads as natural. From 500–800ms they perceive a pause but stay comfortable, and at 800ms–1.2s they say "hello?", repeat themselves, or start to double-talk.

Past that, behaviour degrades quickly. Over 1.2 seconds the caller assumes the call dropped and either abandons or presses 0. Anything over 2 seconds reads as outright system failure.

Be clear about where these bands come from. The boundaries are consistent across many industry sources and match the human baseline above, and I'd act on them. They are not the output of a controlled study with published methodology. The "hello?" behaviour at 800ms is a real, widely-reported observation from people running these systems, and it is not the same category of evidence as the PNAS data.

The 800ms target is the defensible one because it's where two independent things converge: it's roughly 4x the human mean, and it's where practitioners consistently report the conversational contract breaking. Advantage Labs describes its voice agents as generating responses in milliseconds using low-latency speech recognition tuned for natural conversation — the number to ask for in a demo is voice-to-voice p95, not average, for reasons the next-but-one section covers.

Where does the "40% more hang-ups" number come from?

Nowhere I could verify — and it appears in dozens of articles as though it were established. If you're building a business case on it, don't.

The claim circulates in several forms: "customers hang up 40% more frequently past one second," "call abandonment increases 25% when latency exceeds 1 second," "each 100ms beyond 800ms reduces task completion by 4–6%." I tried to trace all three. Every instance leads to another vendor blog, a "contact-centre studies show" construction with no named study, or a page that has since 403'd. No named dataset, no methodology, no sample size, no publication.

That doesn't make the figures wrong. Directionally they're almost certainly right — latency does drive abandonment, and the mechanism is obvious. But an unsourced percentage repeated across an industry is a rumour with a decimal point, and this specific topic is thick with them. If you need to justify latency work internally, use two things instead: the PNAS baseline, which is real, and your own call recordings, which are free. Ten minutes of listening for callers saying "hello?" is more persuasive than a borrowed statistic your CFO can't source either.

What latency do production voice agents actually hit?

Roughly 600ms to 2s depending on architecture, and vendor homepage numbers are systematically better than third-party measurements of the same platforms. The gap between them is the most useful thing in the benchmark data.

Published vendor claims cluster optimistically: sub-second voice-to-voice, p50 under 500ms, "as low as" 600ms. Third-party testing tells a different story. One comparative benchmark collects measurements across platforms — a 500-production-call-per-platform study reporting medians around 680–850ms and p95s from 920ms to 1,180ms, and a separate fixed-stack test putting p50s between 1.73s and 2.34s when every platform runs the same models. Carrier-leg testing puts the network hop alone at 71–89ms p50.

Three lessons in that spread:

• "As low as" means best case: A vendor quoting 400ms and a tester measuring 850ms median can both be honest. Ask which percentile, measured how, over how many calls.

• Configuration dominates the platform: The fixed-stack test showing 1.7 - 2.3s across the board tells you model choice and prompt size move latency more than which vendor's logo is on the dashboard.

• p95 is the number that matters: A 600ms median with a 1,200ms p95 means one call in twenty hits the abandonment band. Averages hide exactly the calls you're losing.

Where is the delay actually coming from?

The dominant cost is turn-taking detection and LLM time-to-first-token, not speech recognition. This surprises people, because transcription feels like a heavy step — but it runs while the caller is still talking, so it adds almost nothing after they stop.

A typical voice-to-voice budget breaks down roughly like this. Network round trip takes 30–80ms and is largely fixed by geography and carrier. Turn-taking, or the VAD decision, runs 150–300ms and is the biggest single variable — it's how long silence must last before the agent decides you're done talking.

The remaining stages are smaller. Speech-to-text finalisation adds 50–100ms, running in parallel with speech for a small additive cost. LLM time-to-first-token runs 150–400ms depending on model size and prompt length, and TTS time-to-first-audio adds 100–200ms, where streaming matters more than voice quality.

Add the middles and you're near 700ms — which is why sub-500ms on a real phone call is hard rather than a matter of buying better software. Two consequences are worth acting on.

Your system prompt is a latency setting. A 4,000-token prompt costs measurable time-to-first-token on every turn. Teams add instructions for months without ever re-measuring, and the agent gets slower one paragraph at a time. This is the most common self-inflicted latency problem and the cheapest to fix.

Turn-taking tuning is a trade-off, not an optimisation. Shortening the silence threshold makes the agent faster and makes it interrupt people mid-sentence. Lengthening it makes it patient and slow. There's no setting that wins both, and the right answer differs by call type — a scheduling agent can be snappier than one taking an address.

How do you measure and fix your own latency?

Measure voice-to-voice from your own phone, at p95, on the network your callers actually use — then fix in order of cost. Vendor dashboards commonly report component latency or server-side processing, which excludes the carrier leg and the turn-taking delay where much of the problem lives.

1. Record ten real calls and measure the gap from the caller's last word to the agent's first sound. A phone recording and any audio editor is enough; you don't need instrumentation to find a 1.5-second gap.

2. Take the worst of the ten, not the average. You're hunting the p95.

3. Cut the system prompt. Move static knowledge to a retrieval step and delete accumulated instructions. Free, and often the largest single win.

4. Try a smaller or faster model for the routine turns. Most scheduling conversations don't need frontier reasoning.

5. Tune the turn-taking threshold in 100ms steps, listening for interruptions after each change.

6. Check streaming is actually on end to end. A TTS engine that buffers a full sentence before speaking adds its entire generation time.

7. Add a filler only if you've done 3 through 6. "Let me check that for you" masks a wait; it doesn't shorten one. Used to paper over an untuned stack, it becomes its own irritation.

8. Re-measure after every prompt change. Latency regresses silently.

Steps 3 and 6 are where most teams find their problem, and both are configuration rather than procurement. If you've done all eight and you're still past a second, the constraint is architectural — stitched multi-vendor stacks carry a network hop between each stage, and that's a rebuild conversation rather than a tuning one.

Frequently asked questions

What is a good response time benchmark for an AI voice agent?

Under 800ms voice-to-voice at p95, with under 500ms as the target for a genuinely natural feel. Measure end of caller speech to start of agent audio, including the carrier leg. Component-level numbers from a vendor dashboard will look considerably better than what your caller experiences.

Is 1 second of latency too slow?

It's past the comfortable band. At around one second callers begin repeating themselves or saying "hello?" into the gap, and some assume the line dropped. It's usable for informational calls and noticeably degraded for anything transactional. Treat one second as a ceiling you've already exceeded, not a target.

Does a filler phrase fix high latency?

It masks it, which is worth something and isn't a fix. A brief acknowledgement before a genuine lookup reads as natural; the same phrase on every turn to cover an untuned pipeline becomes its own annoyance. Fix the prompt size, model choice, and streaming first, then add fillers only where a real external lookup happens.

Why does my agent interrupt callers?

Your turn-taking threshold is too short — the agent decides the caller has finished during a natural mid-sentence pause. This is the direct trade-off against latency: every 100ms you shave off the silence threshold buys speed and costs interruptions. Tune by listening, and expect to land somewhere imperfect.

Do callers care more about latency or accuracy?

Latency, up to a point, then accuracy sharply. A fast agent that mishears an address loses the booking; a slow one that gets it right merely annoys. Callers forgive a 900ms pause far more readily than a wrong appointment time, so don't trade correctness for speed.

Does latency matter less for inbound than outbound?

It matters more. An inbound caller has a problem and chose to call, so a laggy agent stands between them and something they want. Someone answering an outbound call has lower expectations from the outset. Inbound is where latency costs you bookings you'd otherwise have won.

Can AI voice agents ever match human response time?

Not on a standard phone call today. The carrier leg alone consumes a meaningful share of the human 200ms mean before processing starts. Speech-to-speech architectures that skip separate transcription and generation steps are closing the gap; on ordinary telephony, sub-500ms remains the practical frontier.

Conclusion

Latency is a configuration problem far more often than a procurement one. Measure voice-to-voice from your own phone at p95, take the worst of ten calls rather than the average, then work the cheap fixes first: trim the system prompt, confirm streaming is on end to end, and tune the turn-taking threshold in small steps while listening for interruptions. Hold 800ms as the ceiling and 500ms as the goal, and re-measure after every prompt change, because latency regresses quietly. Advantage Labs tunes these stacks for natural conversation and can tell you whether your gap is a prompt, a model, or an architecture problem. Schedule a consultation with Advantage Labs once you've recorded your ten calls and found your worst number.