Megatron-LM

NVIDIA's large-scale training stack comes as two layers in one repository. Megatron Core is the pip-installable library (package name megatron-core) holding GPU-tuned transformer blocks, optimizers, dataset loaders and the parallelism machinery: tensor, pipeline, data, expert and context splitting, with numeric precision down to FP8 and FP4 alongside the usual BF16 and FP16. Megatron-LM wraps that library with ready training scripts for GPT, Mamba, hybrid and vision-language models, plus post-training code for quantization, distillation, pruning and reinforcement learning. A companion project, Megatron Bridge, converts checkpoints to and from Hugging Face format. Expect NVIDIA GPUs, Python 3.12 or newer and recent PyTorch; NGC containers are the smoothest install, and a source build can exhaust RAM unless parallel compile jobs are capped. NVIDIA's own code ships under a BSD 3-clause license that allows commercial use, while bundled third-party files keep their original notices. This is a tool for teams pretraining or continuing training of models across many GPUs, not for single-card hobby fine-tuning.