Data-Juicer

Data-Juicer treats training data preparation as composable infrastructure: more than 200 operators for cleaning, filtering, deduplicating, synthesizing, and analyzing data chain together in declarative recipe files, covering text, image, audio, and video, including dedicated processing for embodied AI and vision language action models. The system scales from a laptop to clusters through native Ray integration, and the team reports processing 70 billion samples in about two hours on 6,400 cores across 50 Ray nodes. A sandbox module supports data and model co-development experiments, and a hub collects more than 50 community recipes for reproducing known dataset pipelines. Alibaba Cloud has integrated the project into its PAI machine learning platform, which anchors it in production use. Installation is pip install py-data-juicer or Docker, most operators run on CPU with optional GPU acceleration for the model-based ones, and the codebase is Apache 2.0 with about 6,800 GitHub stars and a v1.5.4 release in July 2026. It suits teams curating multimodal corpora who want reproducible configs instead of one-off scripts.