People forgive a slow chatbot. They will not forgive a slow phone call. In text, a two-second pause is nothing. On a call, a two-second pause is the other person wondering if you are still there. That is the whole problem with voice agents: you have roughly one second to answer, every turn, or the conversation starts to feel broken.
Human conversation runs on tight timing. The natural gap between turns is about 200 to 500 milliseconds. The targets voice teams design to:
- under ~500ms voice-to-voice feels snappy,
- around ~800ms is where callers start noticing the pause,
- past ~1.5s it feels broken.
That budget has to cover three stages and the network in between. Here is where it goes, and how you claw it back.
The pipeline
A standard voice agent is three models in a row. Speech-to-text turns the caller's audio into words. A language model reads those words and writes a reply. Text-to-speech turns the reply back into audio.
The trick that makes this feel live is streaming. Nothing waits for the previous stage to finish. Speech-to-text emits partial transcripts as the caller talks. The language model streams tokens as it generates. Text-to-speech starts synthesizing on the first sentence while the model is still writing the second. Do it sequentially and you stack 2 to 4 seconds of delay. Overlap it and you can get under a second.
The consequence: what matters is not how long each stage takes to finish, but how long it takes to produce its first output. First partial transcript, first token, first chunk of audio. The whole call lives or dies on those.
Where the milliseconds go
A rough breakdown for a tuned stack. These are vendor and blog figures, so read them as best cases, not your production median:
- audio transport over WebRTC: under ~50ms,
- speech-to-text first partial: ~100 to 200ms,
- language model first token: ~200 to 400ms,
- text-to-speech first audio: ~100 to 300ms.
Add it up and a good stack lands under a second. Add turn-detection delay and a slow model, and plenty of real deployments sit at 800ms to 2s.
The single largest and most variable piece is the language model's time to first token. It is roughly 40% of a sub-second budget, and a frontier reasoning model can blow past a second on its own. It is also the piece you control least: it depends on the model size, the length of your prompt and retrieved context, provider load, and whether the model has to make a tool call, which adds a whole extra round trip. This is why voice teams reach for smaller, faster models, a GPT-4o-mini class model or an open model on a fast host, and keep prompts short. In our own voice work, the model's first token was the thing standing between us and a sub-800ms loop, not the speech models.
Turn-taking is the hard part
Getting the models fast is the easy half. The hard half is knowing when the caller has actually finished talking.
The blunt tool is voice activity detection: is there speech energy right now or not. Silero is the open-source model most people use. It knows sound from silence and nothing else. So you set a silence timeout, say 800ms, and when the caller goes quiet for that long you assume they are done. That timeout is pure added latency on every turn, and it forces a bad trade: shorten it and you cut people off mid-thought, lengthen it and the agent feels slow.
The better tool is semantic turn detection: a model that reads the words and the prosody and predicts whether the sentence is actually complete, often before the silence. Pipecat ships an open-source turn classifier for this; LiveKit has a model-based detector alongside its VAD. It is what lets an agent answer quickly without talking over someone who just paused to think.
Then there is barge-in, the caller interrupting the agent. That means running turn detection while the agent is speaking, and the moment the caller starts, canceling the in-flight model and text-to-speech and flushing the audio already queued. Get the cancellation wrong and the classic bug appears: the agent keeps playing a stale sentence over the person who interrupted it. You also have to stop the agent from hearing itself, which is an echo-cancellation problem. None of this is helped by phone audio being narrow, noisy, and full of "uh-huh" backchannels that must not be mistaken for the end of a turn.
Cascaded pipeline, or one speech-to-speech model
The three-stage pipeline is one approach. The other is a single model that takes audio in and puts audio out with no text handoffs: OpenAI's Realtime API (gpt-realtime, generally available since August 2025) and Google's Gemini Live.
Speech-to-speech is lower latency, because there are no hops between services, and it keeps the tone and emotion that speech-to-text throws away. Small tests put both around 380 to 410ms voice-to-voice. The costs: you are locked to that provider's model for reasoning, the synthesized voice still trails dedicated text-to-speech on naturalness, and you lose the clean text boundary where you would otherwise inject retrieval, run guardrails, and log a transcript. Audio-token pricing also makes long calls expensive.
The cascaded pipeline is more work and a little more latency, but you pick the best model at each stage, swap them as the field moves, and keep a text checkpoint to ground answers, enforce policy, and record what happened. For anything that touches money or compliance, that checkpoint is worth the milliseconds.
Getting onto the phone
A browser demo is not a phone call. Real calls come over the PSTN, and the audio is narrowband: 8kHz mu-law, mono, in 20ms frames. Twilio's Media Streams opens a WebSocket and forwards the caller's audio in exactly that format, and you have to send audio back in the same format or it breaks. That 8kHz ceiling is a real quality hit versus the 16kHz-plus audio a browser gives you, and it lands hardest on names, numbers, and accents, which is exactly the content that matters on a sales or support call. SIP is increasingly first class here: LiveKit has a SIP bridge, and OpenAI's Realtime API added phone calling at launch.
The ways it goes wrong
The failure modes are specific, and most demos never hit them:
- Hallucinated confirmations. The agent says "done, your refund is processed" before the backend confirmed anything, or invents a policy that does not exist. On a recorded call in a regulated business, that is not a bad experience, it is a liability. The fix is grounding answers in retrieval, strict tool use, and never claiming an action succeeded until the system says it did.
- Barge-in leftovers. Stale audio that plays on after the caller has already interrupted.
- Hangup handling. Detecting that the caller hung up so you flush the recording, transcript, and summary and stop billing cleanly, rather than leaving a session running.
- Cross-call leaks. In a multi-tenant system, audio frames that are not bound to a session id can leak between concurrent calls. That is a real production bug, not a hypothetical.
- Background noise. Demos are quiet. Real calls are not, and 8kHz phone audio plus noise spikes transcription errors.
The India angle
For a market like India, the differentiator is not latency, it is language. Hindi, Hinglish with constant code-switching mid-sentence, Tamil, Telugu, and a long list of regional languages and accents. The stacks that work here route transcription per language rather than defaulting to one provider. Indian model companies like Sarvam build speech and language models for exactly this, and India-first calling platforms price per minute in rupees because cost sensitivity is acute: a few cents a minute, against roughly twenty-plus cents for a US-style stack. Lead qualification, appointment reminders, order and payment verification, and collections are the workloads people run first, because they are high volume and the script is narrow enough to keep the agent grounded.
What it comes down to
A voice agent is a latency budget with a personality. Under a second, every turn, or it feels broken. Most of that second is the language model's first token, so that is where you optimize first: a fast model, a short prompt, and a text checkpoint you can ground and log at. Get the turn-taking right so you are not adding a silence timeout to every reply. And test on a real phone line, in a real noisy room, before you trust any number from a quiet demo.
