OpenRLHF

Built on Ray for distributed scheduling, vLLM for fast rollout generation, and DeepSpeed ZeRO-3 for memory-efficient training, OpenRLHF is a high-performance framework for reinforcement learning from human feedback that scales past 70 billion parameters. Algorithm coverage includes PPO, GRPO and Dr. GRPO, GSPO, REINFORCE++ and its baseline variant, RLOO, DAPO with dynamic filtering, and preference methods such as DPO and IPO, with implementation details documented in the maintainers' technical reports. Recent releases added vision-language model support, asynchronous agentic training, and multi-turn agent rollouts, pushing it beyond classic RLHF into agent post-training. Models load directly from Hugging Face, installation is pip install openrlhf with optional vLLM extras or prebuilt Docker images, and serious use assumes a multi-GPU cluster. The project is Apache-2.0 licensed with documentation at openrlhf.readthedocs.io. Close to 10,000 GitHub stars and citations across academic and industry post-training work have made it one of the reference implementations for RLHF at scale.