Beyond Whisper: Parakeet, SenseVoice and ASR in 2026

Whisper is still the reflexive answer to "how do I transcribe audio," and in 2026 that reflex is wrong more often than it is right. This article covers the open speech recognition models that have passed it on accuracy, on throughput, or on both: NVIDIA Parakeet, SenseVoice, Kimi-Audio, Ultravox and Moshi, along with the voice activity detection and diarization pieces that no single ASR model hands you. It is aimed at developers picking an ASR stack, with word error rates, real-time factors, licenses and hardware floors attached.

The leaderboard split into two races

The Hugging Face Open ASR Leaderboard is the closest thing the field has to a shared scoreboard. The paper behind it benchmarks 86 open and proprietary systems across 12 datasets and three tracks: English short-form, English long-form, and multilingual short-form. The extra tracks exist because short-form English averages had stopped discriminating.

The interesting result is not who won, but that the leaderboard measures two things pulling in opposite directions. The paper's own conclusion is that Conformer encoders paired with large decoders produce the best word error rates, while CTC and token-and-duration transducer (TDT) decoders produce the best inverse real-time factors and are better suited to long-form and batched processing. NVIDIA's Canary-Qwen-2.5B topped the short-form English table in the paper at 5.63 percent average WER.

The gap is not subtle. On the long-form track, Whisper large-v3 posts 6.43 WER at an RTFx of 68.56. Parakeet CTC 1.1B posts 6.68 WER at an RTFx of 2793.75. Roughly the same transcript quality, roughly forty times the audio processed per unit of wall clock.

The top of the accuracy column kept moving through 2026. Cohere Transcribe, a 2B Apache 2.0 model released on 26 March 2026, took first place at 5.42 percent average WER, and IBM shipped Granite Speech 4.1 2B at 5.33 percent about five weeks later, on 30 April. Whisper large-v3, at 1.55B parameters and 7.44 percent on the short-form English average, is no longer in that conversation. It is still the most widely deployed speech model on the planet, which is a different claim entirely.

Parakeet: throughput first, accuracy close behind

Parakeet TDT is NVIDIA's FastConformer plus token-and-duration transducer family, and the 0.6B v3 checkpoint is the one worth defaulting to. Six hundred million parameters, CC-BY-4.0, 25 European languages with automatic language detection, and no prompting: hand it audio, get punctuated and capitalized text.

The model card reports 6.34 percent average WER on the Open ASR Leaderboard at an RTFx of 3332.74, which makes it the throughput leader among multilingual open models by a wide margin. Training used roughly 670,000 hours: 10,000 hours of human-transcribed NeMo ASR Set 3.0 plus 660,000 hours of pseudo-labeled Granary data. The sibling Canary 1B v2, 978M parameters despite the name, posts 7.15 percent mean English WER and is described by NVIDIA as matching the quality of models 3x larger while running up to 10x faster.

Two practical details matter more than the WER. First, Parakeet v3 emits word-level, segment-level and character-level timestamps natively, which removes an entire alignment stage from most pipelines. Second, it handles long audio: up to 24 minutes with full attention on an A100 80GB, and up to roughly 3 hours with local attention. Whisper's 30-second window and the chunking machinery built around it stop being a design constraint.

Getting it running through NeMo:

pip install -U "nemo_toolkit[asr]"

import nemo.collections.asr as nemo_asr

asr = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v3")
out = asr.transcribe(["meeting.wav"], timestamps=True)
print(out[0].text)

The catch is the NVIDIA-shaped hole in the deployment story. NeMo targets Ampere, Blackwell, Hopper and Volta, with a stated floor of 2 GB of memory. If that is not your hardware, you want the ONNX export path covered further down.

SenseVoice and the non-autoregressive argument

SenseVoice went the other direction from the LLM-decoder crowd. SenseVoice-Small is a compact encoder-only model, non-autoregressive, with a single CTC head and four learned task tokens prepended to the input. There is no decoder loop, so inference cost is close to flat with respect to transcript length.

The numbers that follow from that design, per the model card: about 70 milliseconds to process 10 seconds of audio, more than 5 times faster than Whisper-Small and 15 times faster than Whisper-Large. The card's charts also put it ahead of Whisper on Chinese recognition, which is the more important claim. If your audio is Mandarin, Cantonese, Japanese or Korean, the Whisper family has never been the right answer and SenseVoice is a direct upgrade rather than a tradeoff.

The same four task tokens give you spoken language identification, speech emotion recognition and audio event detection out of a single forward pass. For call analytics and content moderation that is real value at zero marginal cost.

The caveats are real. Repository code is MIT, but the model weights sit under the FunASR Model Open Source License Agreement rather than a standard OSI license, so read it before shipping. There is no native streaming mode, with third-party truncated-attention projects standing in. And the "50+ languages" line belongs to SenseVoice-Large: the FunAudioLLM paper is explicit that the Small variant delivers low-latency ASR for five languages, namely Mandarin, Cantonese, English, Japanese and Korean.

SenseVoice ships inside FunASR, which is the part most people underrate. FunASR is a full pipeline: VAD, punctuation restoration, speaker verification and diarization, all composable with ASR in one AutoModel call, plus the Paraformer family for streaming. The streaming Paraformer example runs a 600ms chunk configuration, and FunASR's own benchmark table puts Paraformer-Large at 10.18 percent CER on Chinese at 120x realtime on GPU and 15x realtime on CPU. Toolkit source is MIT; weights are licensed per model card.

pip install -U funasr

from funasr import AutoModel

model = AutoModel(model="FunAudioLLM/SenseVoiceSmall", vad_model="fsmn-vad", device="cuda:0")
res = model.generate(input="call.wav", language="auto", use_itn=True)

For Mandarin specifically, also evaluate FireRedASR. The 8.3B LLM variant reports 3.05 percent average CER across four public Mandarin benchmarks; the 1.1B AED variant reports 3.18 percent. On KeSpeech, the multi-dialect Mandarin corpus, the same two report 3.56 and 4.48 percent, which is the harder and more interesting number. FireRedASR2S landed on 12 February 2026 and bundles FireRedASR2-AED with matching VAD, language ID and punctuation models.

Audio LLMs: when the transcript is a side effect

Kimi-Audio is a different category of thing. The 7B Instruct checkpoint is an audio foundation model initialized from Qwen 2.5 7B, pretrained on more than 13 million hours of audio and text, handling ASR, audio question answering, captioning, emotion recognition and end-to-end speech conversation from one set of weights. Its ASR numbers are excellent: 1.28 WER on LibriSpeech test-clean, 2.42 on test-other, 0.60 CER on AISHELL-1 and 2.56 on the AISHELL-2 iOS subset. Code derived from Qwen2.5-7B is Apache 2.0; the rest is MIT.

The 7B in the name undersells the footprint. The published checkpoint totals about 10B parameters once the audio encoder and detokenizer are counted, roughly 20 GB of bf16 weights, which puts a single 24 GB card at the exact edge of viable. If all you need is a transcript, you are paying a large multiple of a compact CTC model's compute for a WER improvement you may not be able to measure. Kimi-Audio earns its keep when the same model has to answer questions about the recording, not just write it down.

Ultravox narrows the idea further. It is a speech encoder, whisper-large-v3-turbo, projected through a multimodal adapter into a text LLM, and v0.7, released in December 2025 under MIT, uses GLM 4.6 as the backbone. Its card reports 97.00 percent on Big Bench Audio and a 90.75 VoiceBench overall score with reasoning enabled, falling to 91.80 and 87.05 without it, plus 2.28 percent WER on LibriSpeech. The v0.6 line pairs the same recipe with Llama 3.3 70B, Gemma 3 27B and Qwen 3 32B if you need a different backbone.

The thing to internalize is that Ultravox does not transcribe. It consumes an <|audio|> pseudo-token and emits text responses, skipping the intermediate transcript entirely. That is the point: no ASR error propagating into an LLM prompt, no extra round trip, and prosody survives into the model's context instead of being flattened into characters. It is also exactly why it is the wrong tool if what you need is subtitles. Serve it with vLLM if you want production concurrency.

Voxtral sits between the two poles: Apache 2.0, an enhancement of Ministral 3B, doing both straight transcription and audio understanding. Voxtral Mini 3B does automatic language detection across eight widely used languages and carries a 32k context that takes 30 minutes of audio for transcription or 40 for understanding, plus a transcription-only mode. Qwen3-ASR and the Omni-series models cover adjacent ground with audio output attached.

Moshi and the full-duplex problem

Every stack described so far is turn-based, even the streaming ones. Moshi is not. Kyutai's model treats a conversation as two parallel audio streams, one for the model and one for the user, generated simultaneously, so barge-in and backchannel noises fall out of the architecture instead of being bolted on with a VAD and a cancel button.

Architecturally it is a 7B temporal transformer paired with a small depth transformer that models inter-codebook dependencies, running on top of Mimi, a streaming neural codec that compresses 24 kHz audio to a 12.5 Hz representation at 1.1 kbps with 80ms of frame latency. Theoretical end-to-end latency is 160ms, 80ms of frame plus 80ms of acoustic delay, with a practical 200ms measured on an L4.

Licensing is split: Python code MIT, Rust backend Apache, weights CC-BY-4.0. Unquantized PyTorch inference wants a 24 GB GPU, but an experimental int8 path exists in PyTorch, the Rust and Candle backend does int8 and bf16, and the MLX backend adds int4 on Apple Silicon.

pip install -U moshi        # PyTorch, CUDA
pip install -U moshi_mlx    # Apple Silicon

Moshi is closer to a research artifact than a product, and its recognition accuracy is not the reason to reach for it. Use it when the interaction model matters more than the transcript. For everything else, Pipecat, LiveKit Agents or RealtimeSTT will get you a cascaded voice pipeline with less risk and more knobs. Kyutai also ships separate streaming STT models under CC-BY-4.0: a 2.6B English model with a 2.5 second delay that reports 6.4 mean WER on the Open ASR Leaderboard set, and a 1B English and French model with a 0.5 second delay. Either is the saner choice if you want low-latency transcription rather than a conversation partner.

The numbers side by side

Model	Params	License	Languages	Reported accuracy	Speed or latency
Whisper large-v3	1.55B	MIT	99	7.44% WER short-form EN, 6.43 long-form	RTFx 145.51 short, 68.56 long
Distil-Whisper large-v3	756M	MIT	English	within 1% WER of large-v3 long-form	6.3x faster than large-v3
Parakeet TDT 0.6B v3	600M	CC-BY-4.0	25 European	6.34% average WER	RTFx 3332.74
Canary-1B-v2	978M	CC-BY-4.0	25 European	7.15% mean English WER	up to 10x faster than 3x larger models
Cohere Transcribe	2B	Apache 2.0	14	5.42% average WER	up to 3x the offline throughput of same-size peers
Granite Speech 4.1 2B	2B	Apache 2.0	6 for ASR	5.33% mean WER	NAR variant reported near RTFx 1820 on one H100
SenseVoice-Small	compact, encoder-only	FunASR model license	5 low-latency	ahead of Whisper on Chinese benchmarks	70ms per 10s of audio
Kimi-Audio-7B-Instruct	~10B on disk	MIT plus Apache 2.0	multilingual	1.28 / 2.42 WER LibriSpeech	~20 GB of bf16 weights
Ultravox v0.7	GLM 4.6 backbone	MIT	multilingual	97.00% Big Bench Audio, 90.75 VoiceBench	no transcript stage, serve on vLLM
Moshi	7B	CC-BY-4.0 weights	English	not an ASR benchmark target	200ms practical on an L4
Moonshine Medium Streaming	245M	MIT	8	6.65% WER against 7.44% for large-v3	107ms against 11,286ms on a MacBook Pro
Voxtral Mini	3B	Apache 2.0	8	transcription plus audio understanding	30 min of audio in a 32k context

That accuracy column mixes benchmarks on purpose, because that is how these models are published: short-form averages, long-form averages, Mandarin CER and LibriSpeech WER are not interchangeable. Treat the table as a shortlist generator, not a ranking, and run your own eval on your own audio before committing.

Punctuation, timestamps, and why alignment is still a step

Whisper's timestamps have always been approximate. The model emits segment boundaries as text tokens and they drift, sometimes badly, on long or noisy audio. That is why WhisperX exists: it runs batched faster-whisper for the transcript, then a phoneme-level wav2vec2 model for forced alignment, producing word-level timestamps that are materially tighter. Reported throughput is around 70x realtime with large-v2 in under 8 GB of VRAM at beam_size=5. whisper-timestamped takes a lighter cross-attention approach if you would rather not load a second model.

The newer encoders make most of that unnecessary. Parakeet v3 gives you word, segment and character timestamps plus punctuation and capitalization directly from the transducer. Canary-1B-v2 produces word-level and segment-level timestamps through an auxiliary CTC model, and segment-level timestamps survive into translated output. SenseVoice supports CTC-alignment timestamps and inverse text normalization through the use_itn flag.

The rule of thumb: if your model is a CTC or transducer encoder, timestamps are cheap and accurate because the alignment already exists inside the model. If your model is autoregressive, which covers Whisper and every audio LLM on this list, timestamps are an inference and you should align separately.

VAD and diarization are still bolt-ons

No model above will segment speakers for you. This part of the stack has not been absorbed and probably will not be soon.

Silero VAD remains the right default for voice activity detection: MIT licensed, about 2 MB as a JIT model, 8 kHz and 16 kHz sample rates, chunks of 30 ms and up, and under 1 millisecond per chunk on a single CPU thread. It runs under ONNX Runtime, ExecuTorch and OpenVINO, with bindings for C++, Rust, Go, Java and C#, so it drops into non-Python services.

pip install silero-vad

Putting a VAD in front of an offline ASR model is not an optimization, it is a correctness fix. Whisper in particular hallucinates confident text on silence and music, and gating on speech probability is the cheapest mitigation available.

For diarization, pyannote.audio 4.x with the community-1 pipeline is the open baseline. Published DERs against the legacy 3.1 pipeline: AMI IHM 17.0 versus 18.8, AMI SDM 19.9 versus 22.7, DIHARD 3 full 20.2 versus 21.4, AliMeeting channel 1 20.3 versus 24.5, CALLHOME part 2 26.7 versus 28.5, MSDWild 22.8 versus 25.4. VoxConverse v0.3 is flat at 11.2 and REPERE phase 2 actually regresses from 7.9 to 8.9, so this is a strong upgrade rather than a universal one. Weights are CC-BY-4.0 behind a gated Hugging Face token, and the pipeline runs on CPU by default.

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    token="hf_...",
)
diarization = pipeline("meeting.wav")

Community-1 adds an exclusive diarization mode that guarantees one active speaker per segment. It exists to simplify reconciling diarization timestamps with transcription timestamps, so if you are joining the two streams yourself, turn it on. SpeechBrain is the alternative when you need to train your own embedding or segmentation models rather than consume a finished pipeline.

Running on modest hardware

Two questions decide this: do you have a GPU, and is it NVIDIA.

Yes to both, and faster-whisper on CTranslate2 is still the pragmatic Whisper deployment, reported at up to 4 times faster than openai/whisper for the same accuracy while using less memory, with INT8 quantization on CPU and GPU to cut it further. But if you are on NVIDIA anyway, Parakeet through NeMo is faster than that by an order of magnitude, and the reason to stay on faster-whisper is ecosystem inertia rather than performance.

No GPU, and whisper.cpp is the general answer. MIT licensed, with ggml integer quantization, ARM NEON and Accelerate on Apple Silicon, Metal, and Core ML execution of the encoder on the Apple Neural Engine, which the project reports at more than 3x faster than CPU-only. It also builds against CUDA, ROCm/HIP, Vulkan and OpenVINO, so one codebase covers the Intel Arc and AMD cards that CUDA-only stacks ignore. WhisperKit is the Swift-native option if you are shipping into iOS or macOS.

To run Parakeet or SenseVoice off NVIDIA, go through ONNX. sherpa-onnx ships an int8 conversion of parakeet-tdt-0.6b-v2 and a SenseVoice build among its pretrained models, exposes APIs in twelve languages plus WebAssembly, and targets Raspberry Pi, riscv64 boards and Jetson class hardware. SenseVoice also ships its own ONNX and libtorch exports.

At the bottom of the size range, Moonshine is the genuinely interesting one. MIT licensed, with a 26M Tiny, a 34M streaming Tiny, a 58M Base, and 123M and 245M streaming variants, covering English, Spanish, Mandarin, Japanese, Korean, Vietnamese, Ukrainian and Arabic. The original paper claimed a 5x reduction in compute for a 10-second segment with no increase in WER against Whisper tiny.en. The current repo claims more: Medium Streaming at 245M posts 6.65 percent WER against Whisper large-v3's 7.44 percent, and on a MacBook Pro the published latencies are 34ms for Tiny Streaming against Whisper Tiny's 277ms, and 107ms for Medium Streaming against Whisper large-v3's 11,286ms, with Raspberry Pi 5 figures published alongside.

pip install moonshine-voice
moonshine-voice mic

Distil-Whisper large-v3, at 756M, English only, MIT, runs 6.3x faster than large-v3 and stays within 1 percent WER of it on long-form audio. The distil-large-v3.5 checkpoint posts 7.21 percent on the leaderboard's short-form English track at an RTFx of 202.03, against 7.44 percent and 145.51 for large-v3 itself. Vosk, the older Kaldi-derived option, is still the smallest thing that reliably works offline in embedded contexts, but treat it as a fallback rather than a first pick in 2026.

Multilingual coverage and what the language counts hide

Whisper's 99 languages is the number everyone quotes and the least useful number in any of these tables. Coverage is wildly uneven: strong in the top ten, thin below that, and close to unusable in the tail.

The 2026 models made a different trade. Parakeet v3 and Canary-1B-v2 cover 25 European languages and cover them properly. SenseVoice-Small covers five. Voxtral Mini claims eight, Cohere Transcribe fourteen. Meta's Omnilingual ASR went the opposite direction: more than 1,600 languages, 500 of them never transcribed by a model before, with character error rates below 10 for 78 percent of them, released Apache 2.0 in sizes from 300M to 7B. That is the only serious attempt at the long tail so far.

The practical consequence: language count is a filter, not a ranking. Work out your language mix first, then choose. A 25-language model that is a point and a half better on the five languages you care about beats a 99-language model every time, and costs less compute to get there.

If you need translation alongside transcription, Canary-1B-v2 does speech-to-text translation from 24 languages into English and from English into 24 languages in the same weights, which removes an entire model from the pipeline.

How to choose

• Bulk English or European transcription on NVIDIA hardware: Parakeet TDT 0.6B v3. Nothing is close on throughput per point of accuracy, and timestamps and punctuation are free.
• Chinese, Japanese, Korean or Cantonese: SenseVoice-Small inside FunASR, or FireRedASR if you need the last point of Mandarin CER and can afford the parameters. Check the weight license first.
• Lowest achievable WER, cost no object: Cohere Transcribe or Granite Speech 4.1 from the top of the leaderboard. Budget for far less throughput than Parakeet unless you take a non-autoregressive variant.
• CPU, edge or battery: Moonshine at the small end, whisper.cpp for portability across every backend, sherpa-onnx if you want Parakeet-class accuracy without CUDA.
• Voice agents where the transcript is not the deliverable: Ultravox for text-out reasoning over speech, Moshi if genuine full duplex is the requirement rather than fast turn-taking.
• Anything with more than one speaker: add Silero VAD and pyannote community-1 regardless of which ASR model you picked.

What to watch next is whether the accuracy leaders close the throughput gap. Granite Speech 4.1's non-autoregressive editing variant, reported near an RTFx of 1820 on a single H100, is the first credible attempt at both from one vendor. For now "most accurate" and "fastest" remain two different products built on two different decoder architectures, and every pipeline in production is a compromise between them. Given how much of this article was rewritten by releases in the last twelve months, betting against that gap closing seems unwise.