RedPajama
Open-source reproduction of LLaMA training dataset and models.
About
An open reproduction effort from Together AI, RedPajama set out to recreate the training data behind LLaMA-class models and publish every step. RedPajama-V1 released a roughly 1 trillion token corpus following the original LLaMA recipe, along with the RedPajama-INCITE models trained on it through a fully open pipeline. RedPajama-V2 scales far beyond that: over 100 billion raw documents from 84 CommonCrawl snapshots processed with the CCNet pipeline, yielding about 30 trillion tokens across English, German, French, Spanish, and Italian, with 30 billion documents carrying quality signals covering natural language metrics, repetitiveness, toxicity, and deduplication. Rather than shipping one filtered dataset, V2 exposes these annotations so teams can apply their own filtering recipes. The code is Apache 2.0 licensed while the data follows Common Crawl terms of use, and the corpora have served as pretraining material for hundreds of open models.
Should you use RedPajama?
Pick it when
Pick this when you are pretraining your own model and want a huge web corpus with per-document quality signals, so you can apply your own filtering recipe across about 30T tokens in English, German, French, Spanish, and Italian.
Look elsewhere when
Skip it if you want a model to run: RedPajama is mainly a dataset, and the INCITE models are dated. Skip V2 too if you need Asian or other non-European languages, since it covers only five Western European languages.
Alternatives to RedPajama
- OLMo
Ships trained models alongside its data recipes and checkpoints, if you want an open corpus and a reproducible model together.
- DataTrove
A library for extracting, filtering, and deduplicating your own web text at scale, rather than a prebuilt corpus with fixed signals.
- NeMo Curator
GPU-accelerated curation toolkit for building training data from your own sources, where RedPajama hands you finished Common Crawl output.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- Large Language Models (LLMs)
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Intermediate (3/5)
- License
- Apache-2.0
- Minimum VRAM
- 8 GB
- Added
- Apr 3, 2026