verl

ByteDance's Volcano Engine team built verl as the open-source implementation of the HybridFlow paper, whose hybrid-controller programming model lets complex post-training dataflows like PPO be expressed in a few lines of code while still executing efficiently across large clusters. The library pairs training backends, FSDP, FSDP2, and Megatron-LM, with the vLLM and SGLang rollout engines, scales to models with hundreds of billions of parameters including expert parallelism for MoE architectures such as DeepSeek and Qwen, and implements PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME, DAPO, and DrGRPO. Throughput features include sequence packing, dynamic batching, and resharding that eliminates memory redundancy when switching between training and generation phases. Installation is pip from GitHub, multi-GPU NVIDIA hardware is assumed, and documentation lives at verl.readthedocs.io. Apache-2.0 licensed with about 22,700 GitHub stars, it became the dominant open framework for reasoning-model RL, used widely across academia and industry to reproduce and extend DeepSeek-R1-style training.