⬅ Back to all blogs

Architecting Sub-700ms Real-Time Voice AI Agents with Pipecat and Twilio

Architecting Sub-700ms Real-Time Voice AI Agents with Pipecat and Twilio

Building conversational voice AI that feels human is one of the hardest engineering problems in generative AI today. Humans process conversational turn-taking within 200 to 400 milliseconds. When an automated telephone agent takes 1.5 to 2.5 seconds to reply, the caller immediately recognizes the delay, begins talking over the bot, and experiences severe cognitive friction.

To deliver true conversational fluidity over standard telephone lines (PSTN) and WebRTC, the total round-trip latency must stay consistently under 700 milliseconds.

In this architectural breakdown, I will walk through the pipeline design, voice activity detection (VAD), audio frame transport, and stateful tool integration required for enterprise-ready voice agents using Pipecat, Twilio WebRTC, and FastAPI.

The Latency Anatomy: Where Every Millisecond Goes

A real-time voice pipeline consists of four discrete stages:

  1. Audio Ingestion & Voice Activity Detection (VAD): Detecting when the speaker stops talking. Silero VAD running locally on raw 16kHz PCM audio typically takes 150ms to 250ms of trailing silence to confidently determine sentence completion.
  2. Speech-to-Text (STT): Streaming transcription via Deepgram Nova-2 with WebSocket streaming achieves initial transcript emission in 100ms to 180ms.
  3. LLM Inference & First Token Latency (TTFT): Utilizing low-latency models like Groq Llama 3.3 70B, Claude 3.5 Haiku, or OpenAI GPT-4o mini via streaming tokens. Time to first token (TTFT) takes between 150ms and 220ms.
  4. Text-to-Speech (TTS) & Frame Buffering: Streaming audio generation with Cartesia Sonic, ElevenLabs Turbo v2.5, or Deepgram Aura delivers the first audio chunk in 80ms to 150ms.

Summing these critical paths:

200ms (VAD) + 120ms (STT) + 180ms (LLM TTFT) + 110ms (TTS First Chunk) = 610ms Total Voice-to-Voice Latency

This 610ms baseline falls squarely inside the sub-700ms threshold for natural conversation.

Transport Architecture: WebSockets and Twilio Media Streams

Traditional HTTP REST endpoints cannot support real-time audio. Instead, we establish a bidirectional WebSocket connection directly to Twilio Media Streams:

PSTN Caller <---> Twilio Gateway <--- μ-law 8kHz Audio Stream ---> FastAPI WebSocket Worker (Pipecat)

Twilio streams audio encoded in 8kHz μ-law. Inside the Pipecat pipeline, an audio resampler converts the stream to 16kHz linear PCM before feeding it to Silero VAD and Deepgram STT. When the TTS engine streams back audio, the pipeline downsamples and converts it to μ-law before broadcasting it back to the Twilio WebSocket.

Handling Interruptions: The Barge-In Problem

The true test of a voice agent is handling user interruptions gracefully (barge-in).

If a caller interrupts the agent while the agent is speaking:

  1. Instant Cut-Off: The local VAD detects user voice energy while the assistant output stream is active.
  2. Buffer Cancellation: The pipeline immediately sends an audio clear frame (clear event) to Twilio to stop playing buffered audio packets in the caller’s receiver.
  3. Task Cancellation: The active LLM stream task and TTS synthesis worker are cancelled instantly to prevent token waste and race conditions.
  4. State Transition: The agent state machine transitions from SPEAKING to LISTENING within 50 milliseconds.

Stateful Execution and Tool Calling

Voice agents must do more than talk: they must look up records in a CRM, check calendar availability, and book appointments during live conversation.

Running slow database queries or external API calls inside the primary speech loop will stall audio synthesis and create awkward silence. To solve this, we implement asynchronous speculative tool execution:

  • Audio Filler Dispatch: If a database query exceeds 400ms, the agent dispatches natural audio fillers (“Let me pull that up for you right now…”) to mask backend processing time.
  • Structured JSON Validation: Function calls are parsed and validated with Pydantic models before executing against PostgreSQL or CRM webhooks.
  • Human Transfer Fallbacks: If sentiment drops or confidence scores fall below acceptable thresholds, the system triggers a SIP transfer to route the call to a human specialist.

Conclusion

Designing real-time voice AI requires moving beyond standard LLM wrapper patterns into distributed systems, low-latency streaming pipelines, and robust audio processing.

By pairing Pipecat with optimized streaming providers, asynchronous tool loops, and resilient interruption logic, teams can deploy voice agents that handle customer conversations with exceptional speed and natural intelligence.