Ultravox

Skipping the transcription step is the core idea of Ultravox: a speech encoder projects audio directly into the embedding space of a backbone LLM, so the model consumes spoken input natively instead of reading ASR output, cutting latency and preserving cues that transcripts drop. Training freezes both the encoder and the language model and fits only the projector adapter, a run the team pegged at 2 to 3 hours on 8 H100 GPUs for 14,000 steps in v0.4. Successive releases moved the backbone from Llama 3.1 8B up to Llama 3.3 70B with Gemma and Mistral variants, and the v0.7 checkpoints from December 2025 are published on Hugging Face under the fixie-ai organization. Output today is streaming text, with direct speech token generation planned, so a separate TTS stage completes a full voice agent. The training and inference code is MIT licensed and the weights are open, while Ultravox Realtime offers the same models as a paid hosted API for production telephony and web agents. Running the open checkpoints locally needs a CUDA GPU sized for the chosen backbone, and it ranks among the most adopted open speech LLMs for real-time voice work.