This post is for engineers and technical leads building or fixing a voice agent that feels slow. It gives the latency budget behind our production deployments, where teams lose whole seconds, and the six architectural techniques that bring a 2.5-second baseline under one second.
A voice agent feels human at sub-1-second p95 latency. It feels janky at 1.5 seconds. It feels broken at 2.5 seconds. The threshold is psychological: the moment a user perceives a hang, they stop listening for the answer and start wondering whether the agent is still there. Below a second the conversation flows. Above it, every turn is a small interruption.
Our production voice deployments hold p95 latency between 700ms and 950ms across English, Spanish, German, and Hindi conversations. Here is how the budget breaks down and how to buy the time back.
The latency budget
End-to-end voice latency is STT finalization plus LLM generation plus TTS first audio. Real numbers from our LiveKit, Deepgram, Claude, and ElevenLabs stack:
- STT finalization (Deepgram streaming): 80-150ms after the user stops speaking
- LLM first token (Claude Sonnet, prompt cached): 200-400ms
- LLM full sentence (to first natural break): 300-600ms more
- TTS first audio chunk (ElevenLabs streaming): 250-400ms after first LLM token
- Network and telephony overhead (Twilio or LiveKit): 80-200ms
Stream the STT; do not wait for finalization
The biggest single win is consuming the streaming transcript before the user has finished speaking. Most voice agents wait for the "final" transcript event from Deepgram or Whisper, then send it to the LLM. That waits on endpoint-of-speech detection, which adds 200-400ms on top of the actual speech.
Instead, consume the interim transcript stream, run a short "is the user done speaking" classifier on each delta, and start the LLM call as soon as the classifier signals end of turn. The LLM is already generating by the time STT finalizes.
Speculative LLM responses on partial transcripts
Take it further: kick off two or three speculative LLM calls on partial transcripts, then cancel the wrong ones once the user finishes. If the partial transcript at 600ms reads "can I change my appointment to...", start one LLM call assuming a date is coming and another assuming a cancellation. When the final transcript arrives at 900ms you have already saved 200-300ms, because one of the speculative calls is mid-generation.
This works because LLM API cost is negligible next to the user-experience cost of latency. We typically run two speculative calls per turn with a 70-90% hit rate on the right one. The wasted tokens cost cents per thousand turns; the latency win is felt on every turn.
TTS chunking and pre-warming
Most TTS providers (ElevenLabs, Cartesia, OpenAI) support streaming synthesis: they start generating audio while text is still arriving. Pipe LLM tokens into the TTS endpoint as they generate, not after the full response completes. The user hears the first words while the LLM is still finishing the sentence.
- Use sentence-boundary chunking: send each completed sentence to TTS as it arrives from the LLM
- Pre-warm the TTS connection at conversation start so the first chunk has no cold-start penalty
- Cache and replay common phrases (greetings, "one moment please", hold messages) instead of re-synthesizing them
- For multilingual agents, pre-load the voice model for the detected language during STT, not after LLM generation
Cut LLM time with prompt caching and right-sized models
Voice does not need the same model size as a deep RAG query. For agent dialogue with structured tool calls, Claude Sonnet or a small open-weight model such as an 8B Llama is usually enough, and 2-4x faster than the top-tier model. Reserve the larger model for the small fraction of turns that need deep reasoning, and route to it dynamically with a classifier on the partial transcript.
Prompt caching cuts another 30-60% off first-token latency for the system prompt and tool definitions, which on a voice agent repeat every turn. Anthropic and OpenAI both support it. The latency win is immediate, and the cached portion bills at up to 90% less than base input price, which compounds at scale.
Tool calls without blocking the user
If the agent needs to call a CRM, a calendar, or a billing system, those tool calls happen mid-conversation. The naive pattern is: user speaks, STT, LLM decides to call the tool, wait for the tool, continue. Latency death.
Better: have the agent emit a verbal acknowledgement ("let me check that for you") while the tool call runs in the background. The acknowledgement is a 1.5-second audio buffer that hides the entire tool round trip, and unlike silence it does not feel like a hang. The conversation stays warm while the system does real work behind it. This is the same bounded tool-calling pattern we use in AI agent development generally; voice just makes the timing visible.
Putting it together
Streaming STT consumption, speculative LLM calls, streaming TTS with sentence chunking, prompt caching, right-sized models, and verbal acknowledgements for tool calls. Each buys 100-400ms; together they take a 2.5-second baseline into the 700-900ms range where voice feels human.
The hard part is instrumentation. You need per-step latency telemetry on every turn in production and an alert on p95 regression. Without it, latency creeps back up as the prompt grows, the corpus grows, or the model provider changes something, and user-facing quality degrades until churn shows up in the dashboard.
If your voice agent sits above a second and you want the budget above applied to your stack, our voice agent development team runs exactly this instrumentation-first process. Contact us for a 30-minute latency review.
Frequently asked questions
What latency does a voice agent need to feel natural?
Under about 1 second p95 per turn. Around 1.5 seconds it feels sluggish, and by 2.5 seconds users perceive the agent as broken.
Why do sequential latency numbers overstate real voice latency?
Because a production pipeline is streaming and speculative. STT, the LLM, and TTS overlap, so perceived latency is closer to STT finalization plus LLM first token plus TTS first chunk, not the sum of every stage.
Do speculative LLM calls cost too much?
No. Two speculative calls per turn with a 70-90% hit rate waste tokens that cost cents per thousand turns, while the 200-300ms saved is felt on every turn.