HELM

Benchmark sprawl is what HELM was built to tame: Stanford CRFM's Holistic Evaluation of Language Models wraps datasets like MMLU-Pro, GPQA, IFEval, and WildBench into standardized scenarios, runs them against any model behind a unified adapter layer, and scores results on metrics beyond accuracy, including calibration, robustness, bias, toxicity, and efficiency. The same pipeline powers the public HELM Capabilities and HELM Safety leaderboards plus specialized suites such as VHELM for vision-language models and MedHELM for clinical tasks, and every prompt and completion behind a leaderboard number can be inspected in the web UI, which is the project's transparency argument. The framework installs as the crfm-helm package from PyPI under Apache 2.0, evaluates hosted APIs and local Hugging Face models alike, and writes results as JSON its visualizer renders. The project moved to maintenance mode in June 2026 and still accepts fixes; it remains the citation standard for holistic LLM evaluation, with the original 2022 paper among the most cited in the evaluation literature.