SentencePiece

Nearly every modern multilingual language model owes something to SentencePiece, Google's unsupervised subword tokenizer and detokenizer. It trains BPE and unigram language model tokenizers directly from raw text, treating input as a sequence of Unicode characters with no pre-tokenization step, which makes it genuinely language independent and equally at home with Japanese, Chinese, or English. Whitespace is preserved through the tokenization round trip, so detokenization is lossless, and subword regularization plus BPE-dropout provide sampling-based augmentation that makes downstream models more robust. The core is C++ for speed, with the README citing segmentation at around 50,000 sentences per second and a memory footprint of about 6 MB, and Python bindings install with pip install sentencepiece; C++ and command-line interfaces ship as well. Model families including T5 and Gemma tokenize with it, and it remains the default choice for training a tokenizer from scratch on raw corpora. The project is Apache-2.0 licensed, has roughly 12,000 stars, and stays actively maintained.