LM Evaluation Harness

EleutherAI's evaluation harness has become the standard way to benchmark language models, implementing more than 60 academic benchmarks with hundreds of subtasks and variants behind a single command line interface. Tasks are defined in YAML configuration files, which keeps prompts reproducible and makes new benchmarks straightforward to add. Models load through HuggingFace transformers, including PEFT and LoRA adapters, or through vLLM, SGLang, GPT-NeoX, Megatron-DeepSpeed, and OpenAI-compatible APIs, so the same task definitions apply to local checkpoints and commercial endpoints alike, with prototype multimodal support via the hf-multimodal and vllm-vlm model types. Installation is a pip install of the lm_eval package with optional extras per backend, and hardware needs depend entirely on the model under test. The code is MIT licensed with no paid tier. It serves as the backend of the Hugging Face Open LLM Leaderboard, is used internally at NVIDIA, Cohere, BigScience, and MosaicML, and has appeared in hundreds of papers, making its prompt formats a common reference point for comparable numbers.