In conversation, people expect a reply within a few hundred milliseconds of finishing a sentence. Pauses longer than about a second feel awkward; two seconds feel broken. Our first voice assistant prototype, built to answer delivery status questions over the phone, typically answered after two to three seconds. Callers repeated themselves, talked over the agent, or hung up.

The obvious suspect was the language model. It turned out to be responsible for less than a third of the delay. This post describes how we measured the pipeline, set a budget per stage, and brought typical response time to about 800 milliseconds.

The pipeline

A typical cascaded voice agent:

caller audio -> transport -> voice activity / end-of-turn detection -> speech-to-text (STT)
            -> LLM (with tools) -> text-to-speech (TTS) -> transport -> caller hears reply

Each stage adds latency. Some of it is unavoidable, much of it comes from waiting for a stage to finish before starting the next.

Measure first

We instrumented every stage with timestamps relative to the moment the caller stopped speaking, which we defined from the audio itself (last frame above the voice activity threshold). The first measurements, median over 200 test calls:

Stage Median added latency
End-of-turn detection (silence timeout) 800 ms
STT final transcript after end of speech 350 ms
LLM time to first token 450 ms
LLM until full response 900 ms (waited for completely)
TTS time to first audio 400 ms (started after full LLM text)
Transport and buffering 150 ms
Total to first audio ~2,600 ms

The biggest single item was not a model. It was the 800 ms of silence we waited for before deciding the caller had finished.

Stage 1: turn detection

Deciding when someone has finished speaking is the hardest part of a voice agent. Too eager, and the agent interrupts people who pause mid-sentence. Too cautious, and every reply starts with dead air.

The prototype used voice activity detection (VAD) with a fixed 800 ms silence threshold. Improvements:

  • Shorter silence threshold with semantic checks. We reduced the threshold to about 300 ms, but only ended the turn if the partial transcript looked complete. A sentence ending in "my order number is" is not complete; "where is my order" is. A small classifier on the partial transcript (and turn-detection models that some speech providers now offer) makes that call.
  • Context awareness. When the agent has just asked for a number, longer pauses are expected while the caller reads it; the threshold is extended for that turn.

Median end-of-turn delay dropped from 800 ms to about 350 ms, with fewer interruptions than before.

Stage 2: streaming speech-to-text

Batch STT waits for the whole utterance, then transcribes. Streaming STT transcribes while the caller speaks, producing partial results and a final result shortly after speech ends. Switching to a streaming STT provider brought the final transcript to about 150 ms after end of speech, and the partial transcripts fed the turn detection above.

Stage 3: the language model

Three changes:

  • Stream tokens instead of waiting for the full response.
  • Shorter prompts and a smaller model for the conversational layer. The system prompt had grown to 3,000 tokens; trimming it and moving rarely needed instructions into tool descriptions reduced time to first token. A smaller, faster model handled the conversation; complex lookups ran through tools.
  • Keep replies short. Voice is not chat. Instructions asked for one or two sentences, which also helps everything downstream.

For questions that need a tool call (looking up a delivery), the agent says a short acknowledgement first ("Let me check that for you") while the tool runs, which keeps the conversation alive.

Median time to first token: about 250 ms.

Stage 4: streaming text-to-speech

The prototype waited for the full LLM response before synthesising speech. Now LLM tokens are buffered into short phrases (split at punctuation) and sent to a streaming TTS service as they arrive. The first audio plays while the model is still generating the rest of the answer. Median TTS time to first audio after the first phrase: about 150 ms.

Stage 5: transport

For phone calls, audio comes through the telephony provider's media stream; for the web app, through WebRTC. We removed an unnecessary jitter buffer in our own media server and kept the services in the same region as the telephony provider's media endpoints, saving about 80 ms.

The result

Stage Before After
End-of-turn 800 ms 350 ms
STT final 350 ms 150 ms
LLM first token 450 ms (plus full wait) 250 ms
TTS first audio 400 ms (after full text) 150 ms
Transport 150 ms 70 ms
First audio ~2,600 ms ~800 ms

The stages now overlap: STT runs during speech, the LLM starts on the final transcript, TTS starts on the first phrase.

Barge-in

Faster replies made another problem visible: callers interrupting the agent. If the caller speaks while the agent is talking, the agent must stop speaking quickly, discard the rest of its planned answer, and listen. Our implementation:

  • detect caller speech during playback with VAD, ignoring echo of the agent's own audio;
  • stop TTS playback and cancel the in-flight LLM and TTS streams within about 200 ms;
  • record what the caller actually heard, so the conversation history reflects the truncated reply.

That last point is easy to miss. If the history contains the full intended reply, the model believes the caller heard information they never did.

Cost and quality trade-offs

Faster stages sometimes cost quality. A shorter end-of-turn threshold occasionally cut callers off; we tuned it with recordings of real calls (with consent) and accepted a slightly longer threshold while callers read out order numbers. A smaller conversational model handled small talk and routing well but was weaker at explaining unusual delivery situations, so those questions go through a tool that drafts the answer with a larger model while the agent says a short acknowledgement. Measuring latency and task success together kept us from optimising one at the expense of the other.

Measuring in production

We track, per call, the time from end of caller speech to first agent audio, the number of interruptions by the agent (agent started speaking while the caller was still talking), and caller barge-ins. The p90 of response time matters more than the median: a few slow turns ruin a call.

Common mistakes

Mistake Effect Do this instead
Blaming the LLM first Optimising the smaller part Measure every stage from end of speech
Fixed long silence timeout Dead air on every turn Short threshold plus semantic end-of-turn
Waiting for each stage to finish Latencies add up Stream STT, LLM tokens and TTS
Long, chatty replies Slow and tiring to listen to One or two sentences per turn
Ignoring barge-in Agent talks over callers Cancel playback and generation quickly

Key takeaways

  • Measure from the moment the caller stops speaking to the first audio of the reply.
  • Turn detection is often the largest single delay; combine VAD with transcript-based completeness.
  • Stream every stage and overlap them.
  • Keep the conversational model fast and replies short; use acknowledgements during tool calls.
  • Handle barge-in by cancelling quickly and recording what was actually heard.

References