llm.c

Training GPT-2 in roughly 1,000 lines of clean C is the whole point of llm.c, Andrej Karpathy's systems-level reimplementation of the LLM training loop with no PyTorch or Python dependency. The CPU reference in train_gpt2.c is readable end to end, while the CUDA path adds mixed precision, multi-GPU training over NCCL, and multi-node runs over MPI, with the maintainers measuring it about 7 percent faster than PyTorch Nightly on their benchmarks. The repository reproduces the 124M-parameter GPT-2, and repo discussions document scaled-up runs through the larger GPT-2 sizes on H100 nodes. Getting started means downloading a starter pack of tokenized data and pretrained weights, then compiling with make; meaningful training requires an NVIDIA GPU, though the CPU build runs on a laptop for study. The code is MIT licensed. Around 30,700 GitHub stars and a long list of ports to other languages reflect its role as the reference for understanding what deep learning frameworks actually do during LLM training, one matrix multiplication at a time.