ColossalAI

Colossal-AI from HPC-AI Tech tackles the gap between single-GPU training code and the distributed infrastructure needed for models with billions of parameters. The framework lets developers keep writing largely sequential PyTorch code while it applies data, pipeline, and sequence parallelism, tensor parallelism in 1D, 2D, 2.5D, and 3D variants, ZeRO redundancy elimination, and automatic parallelization. Heterogeneous memory management spills model states between GPU and CPU memory, which the project reports can cut GPU memory consumption for GPT-2 training by 11x and allow far larger models on the same hardware. Ready-made recipes cover LLaMA pretraining, Stable Diffusion, and other large models, and the framework requires PyTorch 2.2 or newer on Linux with CUDA. Released under the Apache 2.0 license, Colossal-AI is used by research groups and companies that need cost-efficient multi-GPU training and inference without writing low-level distributed systems code.