GPT-NeoX

EleutherAI's GPT-NeoX is the training codebase behind GPT-NeoX-20B, the Pythia suite spanning 70M to 12B parameters, and Polyglot-Ko, which makes it one of the foundational open pretraining frameworks. Built on Megatron and DeepSpeed, it implements tensor, pipeline, and ZeRO parallelism for training autoregressive transformers across GPU clusters, with rotary and ALiBi positional embeddings, flash attention, and mixture-of-experts support layered on top. It runs on NVIDIA systems and AMD MI250X GPUs and launches through Slurm, MPI, or IBM Job Step Manager, reflecting heavy use on academic and national lab supercomputers. Configuration is file-driven through documented neox_arguments, installation is pip install -r requirements/requirements.txt from the repository, and experiment tracking integrates with Weights and Biases, TensorBoard, and Comet ML. Apache-2.0 licensed with about 7.4k GitHub stars, the project remains maintained, most recently adding Transformer Engine integration. Research groups choose it when they need proven large-scale pretraining recipes rather than an inference stack.