The LLM Evaluation Stack: Ragas, LightEval, OpenCompass

An LLM evaluation stack is three separate things that routinely get conflated: benchmarks that score a model, assertions that score your application, and telemetry that scores what actually shipped. This piece covers the open-source tooling for each layer as of July 2026, which projects are actually being maintained, and where LLM-as-judge scoring quietly misleads you. It assumes you already have a RAG pipeline or an agent in production and no trustworthy signal about whether last week's prompt change helped.

Three layers, three different tools

Model benchmarks are static, reference-based, and reusable across teams. They answer "is Qwen3 better than Llama 4 at math." That is a procurement question, not an engineering one, and the tools built for it (LightEval, OpenCompass, lm-evaluation-harness, Inspect AI) are optimized for throughput across thousands of fixed items.

Application evals are your data, your retrieval corpus, your prompts. They answer "did commit abc123 make citation quality worse." Ragas, DeepEval, and promptfoo live here. The set is small, hand-curated, and goes stale within a quarter.

Online eval is scoring on live traffic that nobody labeled. It answers "what are users actually asking, and how often are we failing." This is where Phoenix, LangWatch, Laminar, OpenLIT, Langfuse, and Opik operate.

The mistake is treating these as substitutes. A curated offline set cannot contain the input that breaks you next Tuesday. Production traffic has no ground truth. You need both, and the connective tissue is a trace store you can mine for new test cases.

Ragas: the best RAG metric taxonomy, and a maintenance problem

Ragas is Apache-2.0, requires Python 3.9 or later, and its latest release is 0.4.3 from 13 January 2026. The repository has moved to the vibrantlabsai org, where it carries roughly 15,000 stars, 1,590 forks, and 543 open issues.

It is also quiet. The default branch's last push was 24 February 2026, five months before this was written. The project is not archived and the metric definitions are still the clearest published taxonomy for RAG evaluation, but if you build on it, pin the version and expect to read the source when something misbehaves.

The taxonomy is worth internalizing even if you implement the metrics yourself. Ragas keeps its RAG metrics in one group, and that group splits cleanly into retrieval-side (Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity) and generation-side (Faithfulness, Response Relevancy), with a separate Nvidia-contributed group (Answer Accuracy, Context Relevance, Response Groundedness), an agent group (Topic Adherence, Tool Call Accuracy, Tool Call F1, Agent Goal Accuracy), and a set of non-LLM comparisons (BLEU, ROUGE, CHRF, Exact Match, String Presence).

That split is the diagnostic. Low faithfulness with high context recall means the generator is inventing things and a prompt or model change might fix it. Low context recall means the answer was never retrievable and no amount of prompt work will help; go fix chunking, embeddings, or the reranker.

Version 0.4 broke a lot. Metrics moved from ragas.metrics to ragas.metrics.collections, the evaluate() entry point gave way to an @experiment() decorator, metrics now return a MetricResult with .value and .reason rather than a bare float, ground_truths became a single reference string, and LLM setup collapsed into one llm_factory(). AspectCritic and SimpleCriteria were removed in favor of a @discrete_metric decorator, and AnswerSimilarity became SemanticSimilarity.

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
faithfulness = Faithfulness(llm=llm)

result = await faithfulness.ascore(
    user_input="What is the refund window?",
    response="You have 30 days to request a refund.",
    retrieved_contexts=["Refunds are accepted within 30 days of purchase."],
)
print(result.value, result.reason)

One trap: the stable documentation's metric index still lists Aspect Critic and Simple Criteria Scoring, while the v0.3-to-v0.4 migration guide tells you they are gone. Trust the migration guide and your installed version, not the docs index.

Ragas also ships a synthetic test set generator that builds a starting golden dataset from your corpus. Treat its output as a draft. Send it through Argilla or an annotation queue and have a human delete the third of it that is nonsense before it becomes your regression baseline.

Model benchmarks: LightEval, OpenCompass, and the harness

LightEval is MIT, Python 3.10 or later, and comes out of Hugging Face's leaderboard and evals team. The last tagged release is 0.13.0 from 24 November 2025, but the default branch was still receiving fixes through 29 June 2026, so read the release cadence as slow rather than dead.

The interesting change is architectural. Recent versions added Inspect AI as a backend, and the README now flags it as the preferred one. The others remain: Accelerate, Nanotron, vLLM, SGLang, Text Generation Inference, Hugging Face inference endpoints and inference providers, LiteLLM, and custom models. Version 0.13.0 shipped a breaking reorganization to one file per task definition, so older local task definitions will need moving.

pip install lighteval

lighteval eval "hf-inference-providers/openai/gpt-oss-20b" gpqa:diamond

OpenCompass is the opposite profile: Apache-2.0, about 7,200 stars, and merging pull requests daily. Release 0.5.3 landed 29 June 2026 and added AIME2026 and HMMT February 2026 datasets, a RawPromptTemplate, and support for the OpenAI Responses API, the Gemini SDK, and Claude SDK thinking content. The catalogue covers 70-plus datasets, roughly 400,000 questions across five capability dimensions, with 20-plus preconfigured HuggingFace and API model wrappers.

pip install -U opencompass

opencompass --models hf_internlm2_5_1_8b_chat \
    --datasets demo_gsm8k_chat_gen \
    -a lmdeploy

lm-evaluation-harness remains the reference implementation for the task variants everyone cites in papers. Version 0.4.12 (11 May 2026) added TensorRT-LLM, Megatron-LM, Intel Gaudi, and LiteLLM backends plus native tensor parallelism on the HuggingFace backend, and new tasks including InfiniteBench for contexts over 100K tokens, CRUXEval, and JFinQA. It also renamed SteeredHF to SteeredModel and raised the vLLM floor to 0.18.

Inspect AI itself is at 0.3.251, MIT, maintained by the UK AI Security Institute, with a companion inspect_evals benchmark suite built alongside Arcadia Impact and the Vector Institute. Given that LightEval now defers to it, learning Inspect's task and solver model is the higher-leverage investment.

For retrieval specifically, MTEB (Apache-2.0, still pushed daily) is where you settle embedding model arguments before anyone touches a prompt.

None of these tell you anything about your application. Benchmark contamination is real, and your document corpus is not in MMLU. Use them for model selection, then stop.

DeepEval: evals shaped like tests

DeepEval is Apache-2.0, around 17,300 stars, and the most actively released project in this article. Version python-v4.1.4 shipped 29 July 2026.

The 4.0 release repositioned it as an evaluation harness for coding agents: deepeval generate synthesizes a dataset by inferring your use case from the codebase, deepeval test run executes the suite, deepeval inspect opens a local trace viewer, and results land in local file storage so an agent can read them and iterate. It ships a large built-in metric library, component-level scoring through tracing, and single-line tracing integrations for LangChain, Pydantic AI, OpenAI Agents, Anthropic, CrewAI, and LlamaIndex.

from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams


def test_correctness():
    metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[
            SingleTurnParams.ACTUAL_OUTPUT,
            SingleTurnParams.EXPECTED_OUTPUT,
        ],
        threshold=0.5,
    )
    case = LLMTestCase(
        input="I have a persistent cough and fever. Should I be worried?",
        actual_output=run_pipeline("I have a persistent cough and fever."),
        expected_output="A persistent cough and fever can indicate anything from a mild viral infection to pneumonia.",
    )
    assert_test(test_case=case, metrics=[metric])

deepeval test run test_example.py

The most honest thing in the 4.1.4 changelog is a new flaky flag for metrics and test cases. A judge-scored assertion is nondeterministic and will sometimes fail for no reason connected to your code. A framework that gives you a first-class way to say so is more useful than one that pretends otherwise.

If your team would rather write configuration than Python, promptfoo (MIT, roughly 23,700 stars) does declarative YAML test suites with a CLI built for CI, and doubles as a red-teaming scanner. For dedicated adversarial work, garak and Giskard go further than either.

Tool comparison

Tool	License	Layer	Latest	Judge model needed	Deploy
Ragas	Apache-2.0	RAG metrics	0.4.3, Jan 2026	Yes, for most metrics	Library
DeepEval	Apache-2.0	App assertions	4.1.4, Jul 2026	Yes, for GEval	Library
promptfoo	MIT	Prompts, red team	Active	Optional	CLI
LightEval	MIT	Model benchmarks	0.13.0, Nov 2025	No	Library
OpenCompass	Apache-2.0	Model benchmarks	0.5.3, Jun 2026	Only subjective evals	Library
lm-evaluation-harness	MIT	Model benchmarks	0.4.12, May 2026	No	Library
Inspect AI	MIT	Model and agent evals	0.3.251	Optional	Library
Phoenix	Elastic 2.0	Tracing, online eval	19.10.0, Jul 2026	Optional	Docker
LangWatch	Apache-2.0, ee carve-out	Tracing, online eval	3.7.0, Jul 2026	Optional	Docker, Helm
Laminar	Apache-2.0	Tracing, online eval	0.2.1, Jul 2026	Optional	Docker
OpenLIT	Apache-2.0	Tracing, online eval	1.24.2, Jul 2026	Optional	Docker Compose

LLM as judge: the failure modes that actually bite

Every metric above that reads "judge model needed" inherits a body of measured, reproducible bias. This is not a theoretical caveat.

Position bias is the best studied. "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge" examined 15 judge models across MTBench and DevBench, covering 22 tasks and roughly 40 answer-generating models for over 150,000 evaluation instances. It introduced repetition stability, position consistency, and preference fairness as measurements, and concluded that position bias is not random noise: it varies systematically by judge and by task, and is driven more by the quality gap between the two candidates than by prompt length.

Style bias is larger and less discussed. "Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines" (April 2026) compared nine debiasing strategies across five judge models from four provider families, over MT-Bench, LLMBar, and a 375-pair controlled set. It measured preferences of 0.10 to 0.76 for markdown-formatted content over plain prose, against position bias of 0.04 or less in the same setup. If your candidate outputs differ in formatting, you are measuring formatting.

Verbosity bias is not uniform across model families, which breaks the common assumption that you can correct for it globally. In the same study, Gemini Pro, Gemini Flash, and Llama favored longer responses by +0.24 to +0.44, Claude preferred concise answers at -0.12, and GPT-4o sat near neutral at -0.04. On truncation controls every model correctly picked the complete response, at 0.88 to 1.00 accuracy, so this is a taste for length rather than a failure to notice cut-off text.

Self-preference has its own measurement problem: a judge rating its own output highly might simply be right. "Beyond the Surface: Measuring Self-Preference in LLM Judgments" proposes a DBG score that compares judge scores against gold judgments, which separates genuine bias from legitimate quality differences. The broader CALM framework catalogues 12 distinct bias types.

The practical numbers are sobering. In that mitigation study the best configuration, Gemini 2.5 Flash running a combined strategy of chain-of-thought plus a five-point rubric across accuracy, relevance, completeness, clarity, and reasoning depth, executed twice with positions reversed, reached 71.0 percent agreement with human labels at kappa 0.549 and roughly $0.001 per evaluation. The best frontier setup, Claude Sonnet 4, managed 69.5 percent at roughly $0.015, about 15 times the cost. The same combined strategy improved Claude by 11.5 percentage points and Gemini Flash by 7.5. Position swap on its own hurt every model on the adversarial LLMBar set by 4 to 13 points, because averaging over both orders discards correct verdicts when one response is unambiguously better.

So: normalize formatting and length before judging, use a rubric with explicit criteria rather than "rate this 1 to 10", swap positions on natural comparison data but not blindly on adversarial sets, and calibrate against a few hundred human labels before you trust any threshold. Seventy percent agreement is the foundation you are building on, not a rounding error.

Tracing and online eval

Every online eval strategy starts with traces, so pick that layer first.

Phoenix is at arize-phoenix 19.10.0 (28 July 2026), built on OpenTelemetry through OpenInference instrumentation, with versioned datasets and experiments for tracking changes to prompts, models, and retrieval. Note the license: Elastic License 2.0, not OSI-approved, which forbids offering it as a hosted service. Fine for internal use, a problem if you are a platform vendor.

docker run -p 6006:6006 arizephoenix/phoenix:latest

LangWatch is Apache-2.0 apart from its enterprise modules, which need a commercial license for production use, with a Helm chart at 3.7.0 (26 July 2026). Beyond tracing and evaluations it runs multi-turn agent simulations against your real stack, tools, state, and user simulator included, does prompt management with a GitHub integration, and ships an OpenAI and Anthropic compatible gateway with virtual keys, hierarchical budgets, and inline guardrails. The simulation feature is the differentiator if you ship conversational agents.

Laminar is Apache-2.0, OpenTelemetry-native, ClickHouse-backed, at v0.2.1 (9 July 2026). Two features stand out. Signals let you describe an outcome or a failure in plain language and extract structured events from traces for alerting, which is the most usable online eval primitive in this group. The debugger records runs and replays them with cached results, so you can iterate on a prompt without re-executing the whole agent. Offline evals use an evaluate() call with datapoints, an executor, and evaluator functions, run either directly or with lmnr eval / npx lmnr eval over an evals/ directory.

OpenLIT is Apache-2.0, SDK 1.44.0 with the platform at 1.24.2 (28 July 2026). It has OpenTelemetry-native SDKs for Python, TypeScript, and Go covering 50-plus providers, frameworks, and vector databases, 11 built-in evaluation types (hallucination, bias, toxicity, safety, instruction following, completeness, conciseness, sensitivity, relevance, coherence, and faithfulness), real-time guardrails, prompt management, a secrets vault, and OpenTelemetry-native GPU monitoring. Self-hosting is docker compose with ClickHouse behind an OpenTelemetry collector.

import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

Langfuse (MIT core, with ee/ directories under a separate enterprise license) and Opik (Apache-2.0) round out the realistic shortlist. Helicone and OpenLLMetry cover the gateway and instrumentation-only ends respectively, and AgentOps is narrower but agent-specific.

The OpenTelemetry caveat

"OpenTelemetry-native" is doing less work than vendors imply. Every gen_ai.* attribute, metric, event, and span was deprecated out of open-telemetry/semantic-conventions in v1.42.0 (12 June 2026) and moved to a dedicated semantic-conventions-genai repository. By v1.43.0 (3 July 2026) all that survives under model/gen-ai in the old repo is a deprecated folder. The new repository has no tags and no releases as of late July 2026, and its definitions are still marked development, so there is no stable schema URL to pin against.

Attribute names have already churned: gen_ai.system was renamed to gen_ai.provider.name in v1.37.0. The gen_ai.evaluation.result event, the one part of this defined specifically for eval tooling, now lives in the new repository and is still at development stability. What OTel-native buys you today is transport portability. Budget for an attribute mapping layer if you ever migrate backends.

Wiring evals into CI without a runaway bill

Split the suite by cost, not by topic. Deterministic checks (JSON schema validity, tool call names, citation presence, refusal detection, p95 latency, token budget) are free and belong on every pull request. Judge-scored metrics belong on a smaller curated set, run nightly or behind a label.

Pin the judge. A judge model version upgrade is a breaking change to your metric, and if you leave it floating you will spend a day debugging a "regression" that is your evaluator drifting. Record the judge model string in the results.

Measure your own noise before setting thresholds. Run the identical suite three times against the same commit and look at the spread. Absolute pass/fail thresholds on judge scores are brittle; comparing against the previous commit's score on the same set and failing on regression beyond that measured spread is not.

Cache aggressively. Laminar's cached replay and DeepEval's local result storage exist precisely so the second run of an unchanged component costs nothing.

On budget: at roughly $0.001 per judged evaluation with a cheap judge, 500 cases times four metrics is about $2 a run, which is nothing. The same suite on a frontier judge is around 15 times that, and if you fire it on every push across a team of ten it stops being nothing. Decide which metrics justify the frontier model and run the rest cheap.

Keep the golden dataset in version control next to the code that it tests. A dataset that lives only in a SaaS dashboard will drift out of sync with the prompts, and nobody will notice until a regression slips through.

What to actually measure

Retrieval. Recall at k against known-relevant chunks, and hit rate. If you have labels, plain recall@k is cheaper and more trustworthy than any LLM-graded equivalent; Ragas Context Recall is the fallback when you do not. Measure before and after reranking so you can attribute wins to rerankers rather than to the vector store.

Grounding. Faithfulness or groundedness, plus citation coverage: the fraction of factual claims in the answer that have a supporting span in the retrieved context. This is the number users notice, because unsupported claims are what they call hallucinations.

Task success. A binary per use case, defined by someone who is not an engineer. For agents, Tool Call F1 and Agent Goal Accuracy get closer to this than any text-similarity metric. Everything else on this list is diagnostic; this one is the outcome.

Operations. p95 latency, tokens per request, and cost per resolved task. Cost per call is a vanity metric; cost per resolved task exposes the retry loops.

Safety, in both directions. Refusal rate on questions you should answer is as much a defect as completion rate on questions you should not. Guardrails AI and NeMo Guardrails handle enforcement; garak handles probing.

What to skip: BLEU and ROUGE on free-form answers, any aggregate "quality score" without a denominator, and any single number that can move for six unrelated reasons. If a metric goes down and nobody can say which component caused it, it is not a metric, it is a mood.

How to choose

Start with the trace store, because everything else reads from it. If a permissive license and self-hosting are hard requirements, that is Laminar, OpenLIT, or Opik. LangWatch and Langfuse are permissive at the core with enterprise modules carved out, LangWatch under Apache-2.0 and Langfuse under MIT. Phoenix under Elastic 2.0 is fine unless you resell it.

For model selection, OpenCompass has the breadth and the fastest-moving maintainers. lm-evaluation-harness has the exact task implementations papers cite. LightEval makes sense if you are already in the Hugging Face stack or want Inspect AI underneath without writing Inspect tasks yourself.

For application assertions, DeepEval if your team writes pytest, promptfoo if it would rather write YAML. Both integrate with the tracing backends above, so this is a taste question, not an architecture one.

For RAG metrics, Ragas still has the best vocabulary, but check the commit log before you depend on it and pin the version either way. If it stays quiet through the autumn, the metric definitions are portable and reimplementing the four you actually use is a day of work.

Three things worth watching: whether the GenAI semantic conventions repository cuts a tagged release with a pinnable schema URL, whether Ragas resumes development under the new org, and whether judge-human agreement climbs meaningfully above the 71 percent ceiling. That last number is the binding constraint on this entire category, and no amount of tooling around it changes the arithmetic.