DataTrove

FineWeb, Hugging Face's flagship web pretraining dataset, was built with DataTrove, and the library exists to make that class of processing repeatable: platform-agnostic pipeline blocks read WARC, JSONL, CSV, and Parquet, extract text from HTML with Trafilatura, filter by language and quality heuristics, deduplicate with MinHash, exact substring, or sentence level methods, and write results to disk, S3, or the Hugging Face Hub. The same pipeline definition runs on a multi-core local executor, a Slurm executor for HPC clusters, or an experimental Ray executor, with per-task logging, completion tracking, and resumption so multi-day jobs survive node failures. The repository ships the full FineWeb reproduction config, which doubles as the best tutorial for serious use. Everything is Python, Apache 2.0 licensed, installed with optional extras for S3, Ray, deduplication, and multilingual support, and runs on CPU. At 3,200 GitHub stars it is the reference open pipeline for teams building LLM pretraining corpora from web crawl scale sources without writing their own distributed plumbing.