LightEval

Whether a model sits in memory, on a GPU cluster, or behind an API, LightEval evaluates it the same way: the Hugging Face toolkit abstracts benchmark execution across backends including Transformers with Accelerate, vLLM, SGLang, Nanotron for distributed checkpoints, Text Generation Inference, Hugging Face Inference Endpoints, LiteLLM for hosted APIs, and an Inspect-AI integration. More than 1,000 tasks come built in, spanning MMLU, GSM8K, AIME, coding, multilingual, and chat-model suites, and adding a custom task or metric is a documented Python extension point rather than a fork. Sample-by-sample result inspection is a core design goal, letting failures be debugged example by example instead of through a single aggregate score, and results can push to the Hub for sharing. Installation is pip install lighteval, with a CLI dispatching to whichever backend the flags specify. Hugging Face built it for internal use alongside nanotron and datatrove during model training, and the MIT-licensed project, at about 2,500 stars, is the main open alternative to EleutherAI's lm-evaluation-harness.