RouteLLM

The LMSYS team behind Chatbot Arena built RouteLLM on a simple observation: most queries do not need the most expensive model. The framework serves and evaluates learned routers that send each prompt to either a strong or a weak model, and its paper reports cost reductions up to 85 percent while preserving 95 percent of GPT-4 performance on MT-Bench, matching commercial routing products while being over 40 percent cheaper. Four pretrained routers ship out of the box: matrix factorization (the recommended default), Elo-based similarity-weighted ranking, a BERT classifier, and an LLM-based causal classifier, all trained on preference data released alongside the router models on Hugging Face. It installs with pip install routellm[serve,eval], acts as a drop-in replacement for the OpenAI client or runs as an OpenAI-compatible server, and includes an evaluation harness for MMLU, GSM8K, and MT-Bench. The code is Apache-2.0 licensed with about 5.3k GitHub stars, and the accompanying paper is on arXiv (2406.18665) for teams weighing routing against fine-tuning for cost control.