torchtitan

PyTorch's own answer to large-scale pretraining infrastructure, torchtitan demonstrates how the framework's native distributed features compose: FSDP2 with per-parameter sharding, tensor parallel including an async variant, pipeline parallel, and context parallel for sequences up to one million tokens, all stackable into 4D parallelism with torch.compile support. Float8 and MXFP8 training paths target recent NVIDIA hardware including Blackwell GPUs, while distributed checkpointing, activation checkpointing, and Flight Recorder debugging cover the production concerns. Reference configurations train Llama 3.1 at 8B, 70B, and 405B, supervised fine-tuning on chat-formatted datasets is supported, and the accompanying paper was accepted to ICLR 2025. Installation comes from stable pip or conda builds, though the newest features track PyTorch nightly. The BSD-3-Clause license permits commercial use. With around 5,600 GitHub stars and an AMD-optimized fork released in late 2025, it functions less as a wrapper over someone else's stack and more as the place where PyTorch itself develops pretraining best practice.