Building Real-Time Voice Agents: TEN, Pipecat, and LiveKit

Real-time voice is the one LLM application category where infrastructure beats model quality. A stack that answers in 1.4 seconds with a frontier model loses to one that answers in 600ms with something smaller, because people talk over silence and then hang up. This is a build guide to the three open-source frameworks that dominate the category as of July 2026, the VAD, turn detection, ASR and TTS pieces you bolt onto them, and the transport and cost decisions that follow.

The latency budget dictates the architecture

Every design choice in a voice agent is a withdrawal from one account: the gap between the user finishing a sentence and the first syllable of the reply. Human conversation has a modal turn-taking gap around 200ms. No cascaded LLM pipeline hits that, and in practice the pause starts reading as unnatural somewhere past the half-second mark and as a dropped call past a second or so.

A cascaded pipeline spends that budget in five places:

Stage	Typical 2026 budget	What eats it
Network round trip	30-80ms	Client to SFU to agent worker, region distance
Endpointing / turn detection	100-300ms	Silence timers, semantic classifier inference
ASR finalization	150-300ms	Chunk size, decoder settling, punctuation pass
LLM time to first token	150-700ms	Prompt length, model size, cold routing
TTS time to first audio	100-200ms	Vocoder warmup, chunk granularity

Add the middle of each range and a well-configured cascade lands around 500 to 800ms voice to voice, while an untuned one sits past a second. That spread is almost entirely turn detection and TTS chunking, not model choice.

Two structural facts matter more than any individual number. First, the stages overlap: streaming ASR emits partial hypotheses while the user is still talking, and streaming TTS vocodes from the first sentence boundary rather than the full response. Serialize them and you pay the sum instead of the max. Second, endpointing latency is charged twice when you get it wrong, because recovering from a false endpoint costs far more than the 200ms it saved.

Three frameworks, three theories of control

The open-source field consolidated around three projects, and they disagree about where the abstraction boundary belongs.

Pipecat is a Python pipeline framework built around frames and processors. Audio, text, and control events flow as typed frames through a chain of processors, and you compose behavior by inserting or reordering them. It is BSD-2-Clause, requires Python 3.11 or newer, and shipped 1.6.0 on July 21, 2026. Recent versions built out multi-worker orchestration: a WorkerRunner supervises PipelineWorker instances exchanging messages over a WorkerBus, and the Flows examples include a multi_worker_handoff pattern where a router hands control to a specialist worker and takes it back. Transport is pluggable across Daily WebRTC, LiveKit, Vonage, a FastAPI WebSocket integration, a bare WebSocket server, and local devices.

uv tool install "pipecat-ai[cli]"
pipecat init
uv add "pipecat-ai[deepgram,cartesia,openai]"

LiveKit Agents starts from the transport and works up. It is the agent SDK layered on LiveKit's WebRTC SFU, Apache-2.0 licensed, with 1.6.7 released July 25, 2026 and support for Python 3.10 through 3.14. Its distinguishing features are operational rather than architectural: job scheduling and dispatch APIs for assigning agent workers to rooms, first-class SIP telephony, MCP tool support, and a test framework with LLM-as-judge assertions for conversation flows. If you plan to run hundreds of concurrent sessions and want the worker pool to be somebody else's problem, this is the one that treats that as a product requirement.

pip install "livekit-agents[openai,deepgram,cartesia,turn-detector,silero]"

TEN Framework is the outlier and the most interesting one if you are not a Python shop. It is a graph runtime written in C++ with extension bindings for Python, Go, TypeScript, and C++, where each ASR, LLM, TTS, or avatar integration is an extension node in a declaratively defined graph. Releases move fast: 0.11.69 landed July 27, 2026, and 0.11.68 before it added a Mistral Voxtral TTS extension, Gradium TTS support, sentence termination handling for Soniox ASR, and Opus on by default for the Spatius avatar. The license is Apache 2.0 with additional restrictions, worth reading before you commit commercially. It is backed by Agora, and the quickstart assumes Agora RTC plus Docker Compose.

git clone https://github.com/TEN-framework/ten-framework.git
cd ten-framework/ai_agents && cp .env.example .env
docker compose up -d
docker exec -it ten_agent_dev bash -c "cd agents/examples/voice-assistant && task install && task run"

	Pipecat	LiveKit Agents	TEN Framework
License	BSD-2-Clause	Apache-2.0	Apache-2.0 with additional restrictions
Core language	Python	Python, Node.js	C++ core, Python/Go/TS/C++ extensions
Version checked	1.6.0 (Jul 21, 2026)	1.6.7 (Jul 25, 2026)	0.11.69 (Jul 27, 2026)
Abstraction	Frame pipeline	Session + agent workers	Extension graph
Default transport	Daily WebRTC	LiveKit WebRTC SFU	Agora RTC
Transport portability	High (5+ backends)	Tied to LiveKit	RTC or WebSocket
Bundled turn detector	Smart Turn v3.2 (local ONNX)	LiveKit audio turn detector v1 / v1-mini	TEN Turn Detection (text, Qwen2.5-7B)
Telephony	Via transport provider	Native SIP	Via RTC provider
Best fit	Maximum component control	Scaling managed sessions	Polyglot teams, embedded targets

Turn detection is the actual hard problem

Voice activity detection tells you whether there is speech in a 10 to 30ms frame. That is solved, and it is not what you need. You need to know whether a person who just went quiet is finished talking or merely thinking, and no amount of energy thresholding answers that.

Silero VAD remains the default first stage almost everywhere. It is MIT licensed, the JIT model is roughly two megabytes, it handles 8kHz and 16kHz input in chunks of 30ms and up, and it runs in under a millisecond per chunk on a single CPU thread. It was trained on corpora spanning over 6000 languages, which is why it generalizes better than the old WebRTC VAD. TEN VAD is the alternative: 306KB on Linux against Silero's 2.16MB JIT and 2.22MB ONNX builds, 16kHz audio with 160 or 256 sample hops (10ms or 16ms), and precompiled builds for Linux, Windows, macOS, WASM, Android, and iOS. TEN's published precision-recall curves put it ahead of both WebRTC VAD and Silero, and it specifically claims Silero suffers a delay of several hundred milliseconds on speech-to-non-speech transitions. Treat vendor benchmarks with skepticism, but the size and platform coverage advantage is real if you are targeting a browser or a phone.

The semantic layer is where the frameworks differ most sharply.

Smart Turn (Pipecat)

Pipecat's Smart Turn is a native-audio end-of-turn classifier: Whisper Tiny as a base with a linear classifier layer, roughly 8M parameters, ONNX, taking 16kHz mono PCM up to 8 seconds. The quantized CPU build is 8MB and the unquantized GPU build is 32MB, and the project reports inference in as little as 10ms on some CPUs, under 100ms on most cloud instances, and around 65ms on a Pipecat Cloud standard 1x instance. Current release is v3.2 across 23 languages. Per-language accuracy claims move between releases, so benchmark on your own audio, but the licensing is the real differentiator: BSD-2-Clause with weights, training data, and training code all published, which makes fine-tuning on your own call recordings tractable.

from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3

turn_analyzer = LocalSmartTurnAnalyzerV3(cpu_count=2)

LiveKit audio turn detector

LiveKit replaced its earlier text-based English and multilingual models with a unified audio turn detector that reasons over speech directly, combining lexical content with intonation and rhythm rather than waiting for a transcript. It ships in two flavors: v1, served through LiveKit Inference and free for agents on LiveKit Cloud, and v1-mini, which runs locally on CPU in under 500MB of RAM at no cost anywhere. It covers 14 languages with per-language confidence thresholds and ships inside livekit-agents 1.6.1 and later, no extra install. One caveat from the docs: v1-mini executes in a shared CPU process and wants compute-optimized instances such as AWS c6i or c7i, not burstable ones, or CPU credit limits will produce inference timeouts.

TEN Turn Detection

TEN takes the opposite approach: a Qwen2.5-7B-based text classifier that sorts an ASR transcript into finished, unfinished, or wait states. It handles English and Chinese and understands genuinely ambiguous cases that acoustics alone cannot resolve. The cost is structural. It requires a transcript, so it inherits ASR finalization latency, and a 7B model is a real GPU line item rather than a rounding error on a vCPU. Use it where semantic precision justifies the spend, not as the default.

A fourth option worth benchmarking is STT-native endpointing, where the recognizer itself makes the end-of-turn call. Deepgram Flux has turn detection built in, and AssemblyAI is the plugin LiveKit's docs name as recommended for STT-based endpointing. If you delegate, turn the framework's own min_delay down so you are not stacking a silence timer on top of a model that already decided.

Interruption handling: barge-in without whiplash

Detecting the interruption is the easy half. Deciding which detections deserve a response is the hard half. Backchannels ("mhm", "right"), coughs, a dog barking, and the agent's own audio leaking through a bad echo canceller all look like speech onset.

LiveKit exposes this as an explicit policy surface whose knobs map directly to real failure modes:

session = AgentSession(
    turn_detection=TurnDetector(version="v1-mini"),
    interruption=Interruption(
        mode="adaptive",                 # or "vad"
        min_duration=0.4,                # ignore blips shorter than this
        min_words=2,                     # require content, not just noise
        false_interruption_timeout=2.0,  # silence after a stop = probably false
        resume_false_interruption=True,  # default on: pick the sentence back up
    ),
)

min_duration and min_words filter noise before it reaches the state machine. false_interruption_timeout plus resume_false_interruption handle the case that ruins demos: the agent stops mid-sentence for a cough, then sits in silence waiting for a turn that never comes. Adaptive mode uses acoustic cues to separate real interruptions from backchannels, and it is the default on LiveKit Cloud with most STT providers rather than an opt-in. One more knob worth knowing is discard_audio_if_uninterruptible, which drops buffered audio while the agent is speaking and cannot be interrupted.

Whatever framework you pick, the non-negotiable engineering requirement is that interruption must cancel work in flight, not just mute the speaker. That means aborting the in-progress LLM stream, flushing queued TTS audio, and truncating context to what the user actually heard. Append the full generated response to history when only three words were spoken and the agent will reference things it never said, a bug invisible in transcripts and obvious to users.

Streaming ASR: the 30-second window is the enemy

Whisper's architecture is the single biggest source of avoidable latency in naive voice stacks. The encoder-decoder was trained on fixed 30-second windows and pads shorter clips with zeros, so the encoder does the same amount of work for a one-second utterance as for a thirty-second one, and the cost is charged on every chunk. Every real-time Whisper deployment is working around that constraint.

faster-whisper is the standard workaround: a CTranslate2 reimplementation the project describes as up to 4 times faster than openai/whisper at the same accuracy while using less memory, with INT8 on CPU and INT8/float16 on GPU. Its benchmark puts large-v2 at INT8 on GPU near 2.9GB of VRAM against 4.7GB for the reference package at fp16. It is excellent for batch and passable for streaming with careful chunking, but do not assume CPU is viable at the top end: the published CPU benchmarks cover the small model, and the absent large-v3 CPU row is itself the answer. distil-whisper large-v3 trims to 756M parameters at a claimed 6.3x the speed of large-v3, within 1% WER on sequential long-form audio, at the cost of being English-only.

The architecturally correct answer is a model built to stream. Moonshine accepts any length of audio and spends compute only on that input, with no zero padding. It claims 5x faster or more than Whisper in live speech, but the response-latency table is the more useful artifact: Tiny Streaming at 34M parameters returns in 34ms on a MacBook Pro, 69ms on Linux x86, and 237ms on a Raspberry Pi 5, while Medium Streaming at 245M returns in 107ms on the MacBook against 11,286ms for Whisper Large v3. The smallest variant, Tiny, is 26M parameters. Moonshine v2 adds streaming caches and flexible input windows through an ergodic streaming encoder built for latency-critical use.

On the accuracy end, NVIDIA's NeMo family is hard to beat on throughput per point of WER. Canary Qwen 2.5B posts a 5.63 mean WER at 418 RTFx under CC-BY-4.0, and Parakeet TDT 1.1B, also CC-BY-4.0, trades accuracy for throughput at 2,390 RTFx: 1.39 WER on LibriSpeech test-clean, but 9.55 on GigaSpeech and 15.90 on AMI meeting audio. For anything that has to run on constrained hardware or ship inside a client, sherpa-onnx gives you streaming transducer models with C, Go, Swift, Kotlin, and WASM bindings, and SenseVoice is a strong choice for Chinese-dominant deployments.

Whichever recognizer you choose, run it through ONNX Runtime or TensorRT rather than raw PyTorch in production. The graph-level fusion and quantization are usually worth 2x, and the memory footprint reduction is what lets you colocate the VAD, turn detector, and ASR on the same worker.

TTS: time to first audio is the only number that matters

Total synthesis speed is a throughput metric. The only thing on a voice agent's critical path is how long until the first audio chunk exists, because everything after that streams behind the playback clock.

Kokoro is the current default for latency-sensitive open deployments: 82M parameters under Apache 2.0, a StyleTTS 2 architecture with an ISTFTNet vocoder and no diffusion in the loop. The project publishes no real-time factor, so measure your own, but at that size it shares a GPU with your ASR instead of demanding a second card, and that is why it wins. Quality is good rather than remarkable, and cloning is not the point. Piper goes further down the curve: a fast local neural TTS engine embedding espeak-ng for phonemization, aimed at constrained hardware, which makes it right for offline and embedded agents. Check the license first, because it changed: the maintained piper1-gpl repository is GPL-3.0, while the older rhasspy/piper tree was MIT.

At the higher-fidelity end, Orpheus runs on a Llama-3b backbone under Apache-2.0 and reports roughly 200ms streaming latency, reducible to about 100ms with input streaming, plus zero-shot cloning and emotion tags such as <laugh> inlined in the prompt. Kyutai TTS is 1.8B parameters despite the 1.6b in its name, a 1B backbone plus a 600M depth transformer, pretrained on 2.5 million hours of public audio with weights under CC-BY 4.0; its delayed streams modeling starts speaking before the full text arrives. The one published density figure, 64 simultaneous connections on a single L40S at 3x real-time, is for the STT half of that stack, not TTS, so benchmark the TTS side yourself. Chatterbox from Resemble AI is MIT licensed and now a family: Turbo at 350M for English with its decoder cut from ten steps to one, Nano at 110M running 3x faster than realtime on 8 CPU cores, and Multilingual V3 at 500M across 23 or more languages. Resemble evaluated Turbo against ElevenLabs Turbo v2.5 and Cartesia Sonic 3 on Podonos and published the reports, so read those. For CPU-only or mobile targets, sherpa-onnx TTS covers the same ground as its ASR counterpart.

The practical tuning lever regardless of model is chunking. Send the first clause to TTS the moment the LLM emits a sentence boundary rather than waiting for the full response. That single change routinely removes 300 to 600ms from perceived latency and costs nothing but a sentence splitter.

Transport: WebRTC, WebSockets, and the phone network

The transport decision is really a decision about what happens when the network degrades, and it has one defensible default.

WebRTC runs over UDP and accepts packet loss in exchange for consistent timing. WebSockets run over TCP, which means a single lost packet stalls everything behind it until retransmission completes. A missing 20ms audio frame is nearly inaudible; a 200ms head-of-line stall is a stutter users notice. WebRTC also gives you an adaptive jitter buffer, native Opus encoding in the browser, and RTCP clock synchronization for free, none of which you want to reimplement in JavaScript. Opus packet loss concealment absorbs modest loss with little audible damage, and above that you can set useinbandfec=1 in the SDP offer to trade bitrate for forward error correction.

Use WebSockets for server-to-server legs, telephony bridges where the carrier already hands you a reliable stream, and prototypes. Use WebRTC for anything with a real human on a real network at the other end. Pipecat's FastAPIWebsocketTransport is a fine starting point precisely because it defers the WebRTC decision until you have a working agent.

Telephony is a third category. SIP and PSTN mean 8kHz narrowband audio, G.711 or G.722 rather than Opus, and DTMF handling. LiveKit Agents treats SIP as a first-class transport with native trunk configuration; Pipecat and TEN reach the phone network through their transport providers. If a meaningful share of your traffic is inbound calls, that difference outweighs any latency benchmark, because the alternative is operating a media gateway.

Self-hosting versus hosted transport, and what a minute costs

Pricing across the ecosystem converged on a per-minute agent fee plus metered model usage, and the numbers are close enough that this is not really a cost decision.

LiveKit Cloud charges $0.0100 per agent session minute past the included allotment, and US local inbound telephony is also $0.0100 per minute. The free Build tier includes 1,000 agent session minutes, 5 concurrent sessions, and one agent deployment; Ship starts at $50 per month for 5,000 minutes, 20 concurrent sessions, and two deployments; Scale starts at $500 per month for 50,000 minutes, up to 600 concurrent sessions, and four deployments. Bundled model inference is metered separately: the pricing calculator's phone-call example lists LLM at $0.0014 per minute, STT at $0.0058, and TTS at $0.0300, each varying with the model you select. Data transfer includes 250GB on Ship before $0.12 per GB, and 3TB on Scale before $0.10 per GB.

Pipecat Cloud prices its agent-1x instance at $0.01 per active minute and $0.0005 per minute reserved, with agent-2x and agent-3x at two and three times those rates. WebRTC transport is free for 1:1 voice sessions on Pipecat Cloud; the $0.004 per participant minute only starts once you add video.

Two observations. First, for pure voice, bandwidth is not the cost driver, and anyone telling you to self-host to save on egress has not done the arithmetic. Opus at 32 kbps is 0.24MB per minute per direction, so a gigabyte covers roughly 4,000 agent minutes. At $0.12 per GB that is about $0.00003 per minute, three orders of magnitude under the session fee. Self-hosting the SFU somewhere with single-cent-per-GB egress saves a rounding error unless you are also pushing video.

Second, what you actually save by self-hosting is the $0.01 per minute agent fee, and you spend it on engineering. At 100,000 minutes per month that is $1,000, which does not cover one engineer babysitting TURN servers, region failover, and worker autoscaling. At 10 million minutes it is $100,000 and the calculus inverts. Both LiveKit's server and Pipecat's runtime are open source, so this decision can be deferred, and deferring it is usually correct.

Where self-hosting does pay early is the model layer. Serving your own LLM with vLLM and your own TTS with Kokoro or Piper on one rented GPU can undercut per-minute API pricing at surprisingly low volumes, because the metered STT and TTS lines above are frequently larger than the transport line. Route through LiteLLM so swapping a hosted frontier model for your own endpoint is a config change, and instrument with Langfuse so you see per-stage latency distributions instead of guessing which component regressed.

One architectural fork worth naming: speech-to-speech models that skip the text layer. They win on paralinguistic nuance and stack simplicity, but depending on vendor they are either cheaper than a cascade or multiples more expensive, and the spread across realtime audio APIs runs well over two orders of magnitude per minute. Worse for planning, cost grows with conversation length as audio context accumulates. A cascade stays in a band you can compute in advance from the STT, LLM, and TTS lines above. Cascade still dominates production in 2026 for the boring reasons: you can log the transcript, swap the LLM, and pass a compliance audit.

How to choose

If you are writing Python and want to control every stage, start with Pipecat. The frame pipeline is the most legible mental model of the three, Smart Turn v3.2 is fully open including training data, and transport portability means you are not locked to one provider's pricing.

If you expect to run hundreds of concurrent sessions, need inbound phone calls, and would rather not build a worker scheduler, start with LiveKit Agents. The dispatch APIs, native SIP, and built-in evaluation harness are the parts that are tedious to rebuild, and v1-mini turn detection runs locally so you are not forced onto Cloud to get good endpointing.

If your team is not Python-first, or you are targeting embedded and browser runtimes where a C++ core matters, evaluate TEN Framework. Read the license addendum before you commit, and budget for the fact that the extension graph is a less common pattern with a correspondingly smaller pool of examples.

The component defaults that hold up: Silero VAD or TEN VAD for frame-level detection, a native-audio turn detector rather than a text classifier unless you need semantic disambiguation, Moonshine or a NeMo transducer for streaming ASR rather than Whisper behind a chunking wrapper, and Kokoro for TTS until quality becomes the binding constraint. Then instrument time-to-first-audio at every stage boundary, because measuring is the only way to find the 400ms you are wasting.