RWKV

Where transformer LLMs keep a growing cache of past tokens, RWKV models carry a fixed-size recurrent state forward, so generation speed and memory stay flat no matter how long the context gets, while training still runs in parallel across a sequence. The seventh generation, nicknamed Goose, drops attention entirely and adjusts its state at every token, which the author frames as learning from context on the fly. The BlinkDL/RWKV-LM repository holds the reference training code, built on PyTorch, DeepSpeed, and custom CUDA kernels, whose default training config needs one GPU with about 7 GB of VRAM. Pretrained G1 weights from about 0.1B to 13.3B parameters, trained on multilingual web, code, synthetic, and reasoning data, are posted on Hugging Face and refreshed every few weeks. For inference there is the rwkv pip package, RWKV-Runner as a desktop GUI, the Ai00 server, community GGUF conversions, and mobile runtimes. Code and weights are Apache 2.0, so commercial use is fine, and the project belongs to LF AI and Data. Best for researchers probing attention alternatives and teams needing cheap long-context or batched inference.