Hugging Face Tokenizers
Ultra-fast text tokenization library in Rust with Python bindings.
About
Hugging Face Tokenizers is a tokenization library with a Rust core and bindings for Python and other languages, built for speed and versatility. It implements Byte-Pair Encoding, WordPiece, and Unigram models, trains custom tokenizers in a couple of lines, and handles configurable pre- and post-processing, tokenizing about a gigabyte of text in under twenty seconds on a server CPU. Released under the Apache 2.0 license.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- AI Frameworks & Libraries
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Easy (2/5)
- License
- Apache-2.0
- Added
- Apr 3, 2026
Related Tools
JAX neural network library with PyTorch-like module system.
High-level deep learning API supporting JAX, TensorFlow, and PyTorch backends.
Neural network library for JAX by Google providing Linen module system.
Deep learning platform by Baidu with industrial-strength deployment tools.
Hugging Face library for state-of-the-art NLP models
End-to-end open-source ML platform by Google for training and deployment.