Hugging Face Tokenizers

Tokenizers converts raw text into the integer IDs a language model consumes, and back again, using a Rust engine with Python and Node.js bindings. It implements BPE, WordPiece, Unigram, and word-level models inside a configurable chain of normalizers, pre-tokenizers, post-processors, and decoders, and loads any tokenizer.json published on the Hugging Face Hub; Transformers depends on it for its fast tokenizer classes. The stable 0.23 line on PyPI also trains new vocabularies, returns character offsets for alignment, and pads or truncates batches. The 1.0 release candidates, at rc.2 as of September 2026, split the core into separate crates for encoding, training, serialization, and legacy config conversion, add a SIMD bitstream pre-tokenizer that exactly reproduces the splitting rules of GPT-2, cl100k, o200k, DeepSeek, Kimi K2, and Mistral's Tekken, and temporarily leave out Python features such as training and offsets until the final release. It runs on CPU, installs with pip or cargo, and is Apache-2.0. Reach for it when tokenization speed matters in data preparation or model serving.