Grok-1

Weighing in at 314B parameters, Grok-1 was one of the largest open-weight releases of 2024 when xAI published it under Apache-2.0 in March of that year. It is a mixture-of-experts transformer with 8 experts of which 2 activate per token, 64 layers, 48 query heads with 8 key-value heads, 6144-dimensional embeddings, an 8192-token context, rotary position embeddings, and a 131k-token SentencePiece tokenizer, with support for activation sharding and 8-bit quantization. The release is the raw pretraining checkpoint rather than a chat-tuned model, so it needs alignment work before conversational use. What ships is deliberately minimal: JAX example code for loading and sampling, with weights distributed via torrent and Hugging Face; the README notes that the MoE implementation favors correctness over efficiency and that running the model takes a machine with substantial GPU memory. At roughly 52k GitHub stars it drew enormous attention, and although the repository has not been touched since 2024 it remains a reference artifact for studying large MoE checkpoints, with its size keeping practical self-hosting out of reach for most users.