Latency in Conversational Voice Agents: Sources, Perception Gaps, and a Dynamic Approach to Dead Air
End-to-end latency in AI voice agents is typically framed as a performance optimization problem. We argue it is better understood as two separate problems: actual latency — the measurable pipeline delay — and perceived latency — the psychological weight of dead air on the user. This post breaks down where time is spent in a production STT→LLM→TTS pipeline, why the LLM accounts for up to 80% of total delay, and how a parallel sub-prompt architecture for context-aware dynamic filler generation systematically reduces perceived latency independent of model speed.
Abstract
End-to-end latency in AI voice agents is commonly framed as a performance optimization problem. We argue it is better understood as two distinct problems: actual latency — the measurable delay between user turn-end and agent response onset — and perceived latency — the psychological weight of that delay on the user.
Optimizing for actual latency alone is necessary but insufficient. In this post, we break down the latency stack of a production voice agent pipeline, identify the LLM as the dominant bottleneck (60–80% of total delay on complex prompts), review established mitigation practices, and describe a proprietary technique we developed at callin.io: context-aware dynamic filler generation via a parallel sub-prompt, which systematically reduces perceived latency independent of underlying model speed.
1. Introduction
There is a precise moment at which a telephone conversation with an AI agent stops feeling intelligent and starts feeling broken. It is not when the agent produces an incorrect response. It happens much earlier: during that one-and-a-half seconds of complete silence after the user finishes speaking.
The line goes quiet. The user waits. Then comes the inevitable question: "Hello? Is anyone there?"
At that point, the conversation is already compromised. Even if the subsequent response were perfect — accurate, relevant, well-phrased — the damage is done. The user has stepped outside the conversational flow. They have stopped talking with the agent and started talking at it. The distinction is subtle but consequential: it is the difference between a call that converts and one that ends prematurely.
This degradation is caused by latency. And it is, in practical terms, the primary failure mode of deployed voice agents — more common and more damaging than incorrect responses, because it occurs on every single call regardless of model quality.
2. Decomposing the Latency Stack
To reason about solutions, we first need a precise model of where latency originates. A voice agent pipeline is not a single operation — it is a sequential chain of processes, each contributing its own delay. Figure 1 presents a realistic latency budget for a production deployment, measured from the moment the user stops speaking to the moment they hear the agent's first audio output.
Figure 1. Latency budget for a production STT→LLM→TTS voice agent pipeline. Ranges reflect typical production deployments; long-context or reasoning-intensive prompts push LLM contribution toward the upper bound.
The perceptual thresholds are well-established in the conversational AI literature and broadly consistent with telephony UX research: delays under 500ms feel natural; 500ms to 1 second is acceptable; beyond 1 second, users begin to perceive system malfunction. Beyond 2 seconds, the perception of failure is near-universal.
Twilio's engineering analysis of voice agent architecture corroborates the sequential nature of the pipeline: each component waits for the previous one to complete. Delays accumulate — they do not compensate each other.
The distribution of that accumulated delay is the key observation. STT and TTS together rarely exceed 25–30% of total pipeline latency. The dominant contributor, consistently and by a wide margin, is the LLM inference step.
3. Why the LLM Dominates Latency
A language model does not produce output instantaneously upon receiving a prompt. The inference process involves three sequential stages:
Tokenization. The input text — user utterance, system prompt, conversation history, any retrieved context — is converted into a token sequence. This is computationally cheap but not free.
Prefill (context processing). The model processes the entire input context to build its key-value (KV) cache, from which autoregressive generation begins. This is where prompt length has its most significant latency impact. As a conversation progresses and the context window grows, prefill time increases accordingly. A 10-turn conversation carries a context roughly 5–8× larger than a 2-turn exchange. The model must reprocess this full context on every new turn.
Autoregressive decoding. The model generates output tokens one at a time, each requiring a full forward pass through the network. This phase is bounded by hardware throughput (tokens/second) rather than prompt length.
The key metric for voice agent optimization is TTFT — Time to First Token: the interval between prompt submission and receipt of the first output token. For a voice agent, the practical TTFT budget is approximately 400ms — the amount of time the LLM can consume before the overall pipeline exceeds user tolerance thresholds, assuming optimized STT (~200ms) and TTS (~150ms with streaming).
The challenge is that many capable models — those that handle tool calling reliably, maintain multi-turn coherence, and perform well on reasoning tasks — regularly exceed 600ms TTFT, particularly on longer or more complex prompts. This gap between the TTFT budget and actual model behavior is the central latency problem in production voice agents today.
4. Benchmarking LLM Speed: Reference Resources
Selecting a model for voice-first deployment requires access to reliable, continuously updated latency data. The following resources are among the most useful for this purpose:
Artificial Analysis — The most cited independent benchmark in the field. Tracks TTFT, tokens per second, end-to-end latency, and output quality across all major providers and models. Updated continuously. Essential for anyone building production voice systems.
BenchLM.ai — LLM Speed Leaderboard — Ranks models by generation speed (tokens/second) and TTFT. Covers recent model releases with current provider-level data.
AIMultiple LLM Latency Benchmark — Comparative analysis across use case categories (Q&A, summarization, reasoning). Useful for understanding how latency varies as a function of task type, not just model architecture.
Hamming AI — Voice AI Latency Guide — A practitioner-focused reference specifically for voice agent deployments, with model selection recommendations indexed by use case and latency requirements.
Among models consistently performing well for voice-first applications, current benchmarks indicate:
Gemini 2.5 Flash — Strong balance of low TTFT, high output quality, and reliable tool-calling behavior. Widely deployed in production conversational agents.
GPT-4o mini — Low TTFT and competitive cost; suitable for simple conversational phases where reasoning depth is not required.
Grok 4.1 Fast — High tokens/second after generation onset; favorable for longer responses where generation throughput matters more than initial latency.
Qwen (various versions) — Open-weight models with competitive latency profiles; well-suited for self-hosted deployments where infrastructure control reduces network overhead.
Mistral Large — Among top-tier models, one of the most consistent performers for sub-second TTFT across prompt lengths.
Knowing which models are fastest, however, addresses only half of the problem.
5. Standard Mitigation Practices
Before describing our approach, it is useful to summarize the engineering practices that represent the current state of the art for actual latency reduction. These are well-understood and broadly applicable.
Token streaming end-to-end. The LLM should begin forwarding output tokens to the TTS synthesis layer as they are generated, rather than waiting for complete response generation. With streaming, TTS begins synthesis on the first few tokens while the LLM is still generating the remainder. This parallelization meaningfully reduces perceived latency. As Twilio's engineering documentation states, the absence of streaming support in an LLM API is an immediate disqualifier for voice applications.
Context window discipline. Every token in the prompt carries a prefill cost. Verbose system prompts, unpruned conversation histories, and inefficiently structured retrieved context all increase TTFT. Production voice prompts should be compact and structurally clean, without sacrificing the information the model needs to maintain conversational coherence.
Precise end-of-turn detection. Every additional millisecond the VAD layer requires to confirm that the user has stopped speaking contributes to perceived latency from the user's perspective. A well-calibrated VAD reduces initial dead time without prematurely truncating user utterances — a balance that requires empirical tuning per deployment.
Model tiering by conversational phase. Not every phase of a conversation requires the most capable available model. Greetings, confirmations, and simple clarification requests impose minimal reasoning demands. Deploying a lightweight, fast model for these phases and reserving higher-capability models for complex reasoning or tool-calling tasks reduces mean TTFT across the full call. This is the principle underlying our LLM Switcher architecture, discussed in a prior post.
Geographic infrastructure proximity. Network round-trip time between the user's device and the inference host accumulates across every conversational turn. Region-local deployments reduce this overhead structurally.
These practices collectively address actual latency — measurable milliseconds removed from the pipeline. They do not, however, address perceived latency when actual latency cannot be reduced below a certain threshold. This is where the most significant opportunity lies.
6. The Actual vs. Perceived Latency Distinction
The distinction between actual and perceived latency is underappreciated in the voice agent literature and largely absent from standard engineering guidance.
Actual latency is the measurable interval between user turn-end and agent response onset. Perceived latency is the psychological weight of that interval — how much it degrades the user's experience of the conversation.
These two quantities are related but not equivalent. And the relationship is not linear.
A person calling a human call center does not expect an instantaneous response. They understand that the agent must think, look something up, reason through the situation. What they do not tolerate is silence. Silence is absence. Silence is the sensation that the system has failed.
Human agents manage this through what linguistics terms fillers — conversational elements that maintain the interactional thread while cognition proceeds: "Let me just pull that up...", "Right, so...", "One moment, I'm looking at your account now..." These phrases carry no propositional content. Their function is entirely pragmatic: they signal that processing is occurring, that the conversational partner is present, that a response is coming.
A naive AI voice agent has two options when LLM inference takes longer than the perceptual threshold: respond immediately (not possible if the model is slow) or remain silent (which triggers the failure cascade described in the introduction). Neither is satisfactory.
The most common market solution is static filler audio: pre-recorded clips played automatically after a configurable silence timeout. A fixed "One moment please..." audio segment, identical across every invocation, independent of context.
This approach provides marginal improvement over silence but introduces its own problems. It is immediately recognizable as mechanical — users perceive the repetition. More importantly, it is contextually uncoupled: a brisk "Sure, one moment!" played in response to an emotionally charged complaint produces dissonance that compounds rather than mitigates user frustration.
7. Dynamic Contextual Fillers: Our Approach
At callin.io, we have developed a technique that addresses perceived latency through a different mechanism: context-aware dynamic filler generation via a dedicated parallel sub-prompt.
The architecture is as follows. When the main inference pipeline receives an input whose estimated TTFT exceeds a threshold (empirically set around 600ms in our current deployment), a second, independent prompt is immediately dispatched to a lightweight, fast model. This sub-prompt has a single, narrow objective: generate a bridging phrase that is (a) semantically coherent with the user's most recent utterance, (b) tonally appropriate to the emotional register of the exchange, and (c) structured to connect naturally to the forthcoming full response.
The sub-prompt does not reason about the full response. It does not consult external systems. It answers one question: "Given the last user utterance and the current conversational context, what would a skilled human agent say while taking time to think?"
The full response pipeline and the filler generation pipeline run in parallel. The filler arrives first — typically within 150–250ms — is played to the user, and the main response follows directly, linguistically attached.
The result differs fundamentally from a static filler.
7.1 Example: Product Availability Query
User: "I'd like to know if you still have the Pro 4K model available for delivery by this week — I need it for an event on Saturday."
Situation: Agent must query product database via MCP, verify stock and express shipping availability. Estimated main LLM TTFT: 900ms.
Static filler response:
"One moment please..." [800ms silence] "Yes, the Pro 4K is available and..."
Dynamic filler (callin.io):
"Got it — with the event on Saturday, timing is what matters here. Let me check availability and fast-delivery options right now..." [main response continues] "...the model is in stock, and we can guarantee Friday delivery if you order before 3pm today."
The bridging phrase encodes the specific constraint (Saturday event), contextualizes the processing delay as purposeful action, and transitions seamlessly into the full response. The user's experience is of an agent that understood the urgency and acted on it.
7.2 Example: Emotionally Charged Complaint
User: "This is the third time I'm calling about the same issue. Every time you tell me it's resolved and then it starts again. I'm really tired of this."
Situation: Agent must retrieve full customer history and previous ticket chain. Estimated TTFT: 800ms.
Static filler response:
"I'm sorry for the inconvenience, one moment..." [700ms silence] "I have your profile here..."
Dynamic filler (callin.io):
"Three calls for the same issue — that shouldn't happen, and I completely understand the frustration. I'm pulling up your full case history right now..." [main response continues] "...I can see all three contacts. You're right — the issue was never properly closed on our end."
The filler phrase matches the emotional register of the complaint, acknowledges the specific failure (three contacts), and transforms the processing interval into an active retrieval moment rather than a system pause.
7.3 Example: Complex Technical Query
User: "If I integrate your system with my Salesforce CRM via API, can I trigger automated outbound calls based on lead scoring events?"
Situation: Multi-layered technical question requiring structured reasoning. Estimated TTFT (Claude): 1,200ms.
Static filler response:
"Great question, just a moment..." [1,100ms silence] "Yes, that's possible..."
Dynamic filler (callin.io):
"Yes, Salesforce integration with lead scoring triggers is actually one of the more interesting use cases — let me give you the full picture on how that works..." [main response continues] "...the cleanest approach is using Salesforce Flow to fire a webhook to our inbound endpoint when a lead crosses your scoring threshold..."
The user receives immediate confirmation that the question was understood and that a detailed answer is being constructed — not that the system is processing or loading.
8. The Cascade Effect: Why Dead Air Compounds
There is a dynamic familiar to anyone who has managed contact center operations at scale: conversations have a rhythm, and when that rhythm breaks, recovery is disproportionately difficult.
Dead air beyond the perceptual threshold does not simply produce momentary discomfort. It triggers a cascade that degrades every subsequent exchange:
The user questions system function → trust is eroded
The user breaks the natural conversational flow → begins asking system-state questions ("Can you hear me?")
The agent responds to the meta-question rather than the original query → loses conversational thread
The user, already skeptical, applies heightened scrutiny to subsequent responses
The call terminates prematurely, before intended outcomes are reached
A contextually appropriate dynamic filler interrupts this cascade at step 1. Not by deceiving the user — they know processing is occurring — but by transforming the waiting interval from a system pause into a recognized conversational beat. The difference between "Hello? Is anyone there?" and "I see — give me just a second..." is entirely a function of what fills the gap.
9. A Unified Framework: Actual and Perceived Latency Together
Production voice systems that perform well under real-world conditions address both dimensions simultaneously.
Tier 1 — Actual latency reduction:
Model selection indexed to TTFT benchmarks (Artificial Analysis, BenchLM) for each conversational phase
Native LLM-to-TTS token streaming with no intermediate buffering
Prompt engineering discipline: lean context, structured conversation history, efficient RAG/MCP retrieval injection
VAD calibration for accurate, low-latency end-of-turn detection
Region-local infrastructure deployment
Phase-appropriate model tiering via LLM Switcher
Tier 2 — Perceived latency reduction:
Context-aware dynamic filler generation via parallel sub-prompt
Tonal calibration of filler to emotional and informational register of input
Seamless linguistic attachment between filler and main response
Elimination of silence events beyond perceptual threshold, independent of main model TTFT
The combined effect is a system that, even when the primary model requires 1–1.5 seconds to generate a complete response, never stops projecting presence, attentiveness, and control of the conversational thread.
10. Why This Matters Beyond Engineering
Latency is not, ultimately, a technical problem. It is a conversion problem.
A voice agent sustaining perceived latency below 500ms closes more calls, generates higher user trust, and produces lower mid-call abandonment rates. An agent suffering from dead air produces the inverse — even when its responses are substantively correct.
At callin.io, we instrument both dimensions: component-level actual latency across the full pipeline, and silence rate — the proportion of conversational time during which the user perceives silence beyond the perceptual threshold. Silence rate, in our production data, is the metric that most directly correlates with call completion rates and conversion outcomes.
Reducing silence rate is not a UX nicety. It is a fundamental design commitment: that the system will respect the conversational contract of a phone call. People do not call to interact with a pipeline. They call to speak with something that behaves as though it understands that, on a phone call, silence carries more meaning than words.
References and Further Reading
Twilio Engineering: Core Latency in AI Voice Agents
Hamming AI: Voice AI Latency — What's Fast, What's Slow, and How to Fix It
Softcery: Choosing an LLM for Voice Agents: Speed, Accuracy, Cost
Artificial Analysis: LLM Speed & Latency Leaderboard
BenchLM: LLM Speed Comparison
AIMultiple: LLM Latency Benchmark by Use Case
NVIDIA Technical Blog: LLM Inference Benchmarking: Fundamental Concepts
callin.io is an AI voice agent platform built for companies that can't afford to get it wrong. If you'd like to benchmark your current agent's latency profile or explore how dynamic filler generation affects your call completion rates, contact us.
