NeMo Curator

NVIDIA's answer to data curation bottlenecks moves the whole pipeline onto GPUs: NeMo Curator uses RAPIDS libraries such as cuDF, cuML, and cuGraph, orchestrated over Ray based executors, to filter, classify, deduplicate, and transform training data across text, image, video, and audio modalities. The team reports fuzzy deduplication of a RedPajama v2 subset dropping from 10.7 hours to 0.65 hours, roughly a 16x speedup, on three H100 nodes compared with CPU alternatives. Text tooling covers language identification, quality classifiers, and exact, fuzzy, and semantic dedup; images get aesthetic and NSFW filtering plus embedding generation; video supports scene detection and clip extraction; audio adds transcription and quality filtering. The same stack powers the data pipelines behind NVIDIA's Nemotron model family. The GPU path expects Linux x86_64, CUDA 12, and about 16 GB of GPU memory, with installation through uv packages or Docker containers. The code is Apache 2.0 licensed with about 1,700 GitHub stars, aimed squarely at teams curating pretraining scale corpora on GPU clusters.