DataTrove

Library for processing, filtering, and deduplicating LLM training text at scale, locally or on clusters.

Open SourceSelf HostedOffline Capable

0.0 (0)

About

FineWeb, Hugging Face's flagship web pretraining dataset, was built with DataTrove, and the library exists to make that class of processing repeatable: platform-agnostic pipeline blocks read WARC, JSONL, CSV, and Parquet, extract text from HTML with Trafilatura, filter by language and quality heuristics, deduplicate with MinHash, exact substring, or sentence level methods, and write results to disk, S3, or the Hugging Face Hub. The same pipeline definition runs on a multi-core local executor, a Slurm executor for HPC clusters, or an experimental Ray executor, with per-task logging, completion tracking, and resumption so multi-day jobs survive node failures. The repository ships the full FineWeb reproduction config, which doubles as the best tutorial for serious use. Everything is Python, Apache 2.0 licensed, installed with optional extras for S3, Ray, deduplication, and multilingual support, and runs on CPU. At 3,200 GitHub stars it is the reference open pipeline for teams building LLM pretraining corpora from web crawl scale sources without writing their own distributed plumbing.

Should you use DataTrove?

Pick it when

Use it when you build an LLM pretraining corpus from Common Crawl WARC files or large JSONL or Parquet dumps on CPU machines or a Slurm cluster, and want resumable jobs plus the shipped FineWeb config as a proven starting pipeline.

Look elsewhere when

It is text only, so multimodal corpora belong in Data-Juicer or NeMo Curator, and its Ray executor is experimental, which matters if your cluster runs Ray rather than Slurm. For a fine-tuning set of a few thousand rows, Hugging Face Datasets is enough.

Alternatives to DataTrove

Data-Juicer
Multimodal and Ray native with 200+ operators; DataTrove is narrower but runs on Slurm and ships a full FineWeb reproduction to copy.
NeMo Curator
GPU-accelerated fuzzy and semantic dedup on RAPIDS, faster on large GPU clusters, but its GPU path needs Linux, CUDA 12, and 16 GB GPUs where DataTrove runs on CPU.
Hugging Face Datasets
Loads, maps, and streams Hub datasets, fine for modest data, but you would write the dedup and resumable cluster job tracking DataTrove provides.

Reviews (0)

Leave a Review

No reviews yet. Be the first to review!