nanoGPT

Two files of about 300 lines each, model.py and train.py, are the entire core of nanoGPT, Andrej Karpathy's minimal codebase for training and fine-tuning medium-sized GPTs. It reproduces the 124M-parameter GPT-2 on OpenWebText in about four days on an 8xA100 40GB node, loads OpenAI's released GPT-2 checkpoints for fine-tuning, and scales down to a character-level Shakespeare model that trains in minutes on a single GPU or even a CPU. Dependencies are just PyTorch, NumPy, transformers, datasets, tiktoken, wandb, and tqdm, with PyTorch 2.0 compilation on by default, and multi-node training works over DDP. The MIT-licensed repository has around 61,700 stars and spawned an active speedrun community that competes on wall-clock time to GPT-2 quality. A README notice added in late 2025 steers newcomers to the full-stack successor nanochat, but nanoGPT remains the standard teaching codebase for LLM pretraining: readable in an afternoon, hackable by design, and the starting point for countless research forks, ports, and educational videos.