RAG evaluation: scoring what a system retrieves and what it does with it.

What is RAG evaluation?

RAG evaluation is the practice of measuring how well a retrieval-augmented generation system performs, by scoring its two halves separately: retrieval (did it fetch the right context) and generation (did the model use that context to produce a correct, grounded, relevant answer). It combines classic information-retrieval metrics with LLM-based judges of the generated text.

RAG evaluation builds on the base metrics and judge techniques of language-model evaluation, so the mechanics of scoring text, LLM-as-a-judge, and its biases carry over. For those foundations see our guide to LLM evaluation; this page focuses on what is specific to retrieval-augmented systems.

Why evaluate retrieval and generation separately?

You evaluate retrieval and generation separately because they fail in different ways and a single end-to-end score hides which half broke. A perfect generator on top of weak retrieval still fails the user, and a strong retriever feeding a careless generator still hallucinates. Scoring each half localizes the failure so you know what to fix.

This is the organizing principle of RAG evaluation, and it maps onto the pipeline: the retriever finds context, the generator writes an answer from it. Retrieval metrics ask whether the right evidence was found; generation metrics ask whether the answer used it correctly.

What are the retrieval metrics for RAG?

The retrieval metrics for RAG include context precision (of the retrieved chunks, how many are relevant and well-ranked), context recall (of the evidence that exists, how much was retrieved), and the classic information-retrieval ranking metrics, hit rate and recall at k, mean reciprocal rank (MRR), and normalized discounted cumulative gain (nDCG). Recall-style metrics need ground truth; precision-style ones often do not.

What metrics measure RAG generation quality?

The core generation metrics are faithfulness (also called groundedness), answer relevancy, and answer correctness. Faithfulness checks that every claim is supported by the retrieved context; answer relevancy checks that the answer addresses the question; answer correctness checks the answer against a known-good reference. Hallucination rate and citation coverage round out the set.

The distinction that trips people up is that faithfulness and correctness are not the same. A faithful answer only says what the context supports, but the context could be wrong or incomplete; a correct answer matches the truth, which needs a reference. In production, where references are scarce, faithfulness plus answer relevancy is the workhorse pair because neither needs a gold answer.

What is the RAG triad?

The RAG triad, introduced by TruLens, is a set of three reference-free checks that together cover a RAG pipeline: context relevance (is the retrieved context relevant to the question), groundedness (is the answer supported by that context), and answer relevance (does the answer address the question). All three are scored by an LLM judge, so they can run without a gold answer, including on live traffic.

The triad is popular because it is reference-free and covers the whole pipeline with three cheap judge calls, which is why it is used for continuous production monitoring, not just offline testing. A note on terminology: TruLens's "groundedness" is the same idea as RAGAS's "faithfulness," and TruLens's single "context relevance" is a query-time judgment, distinct from RAGAS's context precision and recall, which score the whole retrieved set. Naming differs across frameworks; the underlying checks do not.

What is the difference between reference-free and reference-based RAG evaluation?

Reference-free RAG evaluation scores an answer using only the question, the retrieved context, and the output, so it needs no gold answer and can run on live traffic. Reference-based evaluation compares against a known-good answer or labeled relevant documents, which is stricter but requires ground truth. Context recall and answer correctness are reference-based; the RAG triad is reference-free.

The general mechanics of reference-based versus reference-free scoring, and of LLM-as-a-judge, are covered in our LLM evaluation guide; the point here is that reference-free metrics are what make continuous, on-your-own-data RAG monitoring practical.

How do you build a golden dataset for RAG evaluation?

You build a golden dataset for RAG by assembling question, context, and answer triplets: real questions, the passages that should answer them, and a known-good answer. You can generate candidates synthetically (frameworks like RAGAS produce test questions across types, from simple to multi-hop) but the contexts and answers should be verified by people who know the domain. Realism beats volume.

The general method, for any task type, is in how to build a golden dataset; this is the same groundwork that makes any owned evaluation possible, and where data readiness for AI pays off. Without a golden set drawn from your own documents, you are evaluating retrieval against someone else's notion of relevance.

How do chunking, embeddings, and retrieval choices affect evaluation?

Chunking, embedding, and retrieval choices largely determine your retrieval scores, because retrieval failure is the most common RAG failure: if the right chunk is never fetched, the generator cannot recover. Poor chunk boundaries, weak embeddings, or pure vector search with no reranking all show up as low context recall, which is why those metrics are where tuning pays off.

Common fixes are hybrid search (combining dense vector search with keyword search like BM25) and a reranking step. Anthropic's Contextual Retrieval, a technique that prepends model-generated context to each chunk before embedding, was reported to cut retrieval errors materially, and features like Claude Citations help tie each claim back to its source for attribution scoring. Evaluation is how you tell whether any of these actually helped on your data.

What is the difference between component and end-to-end RAG evaluation?

Component evaluation scores each stage of the pipeline (retrieval quality, then generation quality) to localize failures; end-to-end evaluation scores the final answer the user sees, on correctness, latency, cost, and safety. You need both: component scores tell you what to fix, and the end-to-end score tells you whether the system is actually good enough to ship.

How do you monitor a RAG system in production?

You monitor a RAG system in production by running the reference-free metrics, the RAG triad, on live traffic, sampling rather than scoring every request to control cost, and alerting when any leg drifts: context relevance dropping signals a retrieval or index problem, groundedness dropping signals hallucination, answer relevance dropping signals the model losing the thread. Latency and cost are tracked alongside.

Production monitoring is where reference-free evaluation earns its keep, because you cannot label a gold answer for every live query. The offline golden set catches regressions before you ship; the online triad catches drift after you do.

What tools and frameworks evaluate RAG?

The common frameworks include RAGAS (the de-facto open-source RAG metrics), TruLens (which originated the RAG triad and is now part of Snowflake), DeepEval, Arize Phoenix, LlamaIndex's built-in evaluators, and LangSmith. They differ on open-source versus commercial, whether they score retrieval and generation separately, and whether they can run self-hosted inside your perimeter. We compare the categories in AI evaluation tools, compared.

How do you evaluate a RAG system on your own data?

You evaluate a RAG system on your own data by building a golden set from your own documents and real questions, scoring retrieval and generation separately against what your business considers a good answer, running the whole thing inside your perimeter, and keeping the documents and criteria invisible to the model vendors you are judging. Public RAG benchmarks cannot do this; only an owned evaluation can.

In practice that is four steps, run as a continuous loop:

1. Build a golden set from real questions: question, context, and answer triplets from your own documents.
2. Score retrieval and generation separately: context precision and recall, then faithfulness and answer relevancy.
3. Run it inside your perimeter: keep the documents, criteria, and results on infrastructure you own.
4. Monitor and re-score on change: run the reference-free triad on live traffic, and re-run the golden set on every model, prompt, or index change.

Your relevance judgments and gold answers encode what a good answer means in your domain, which makes them intellectual property, and a metric your vendor can see is a metric your vendor will optimize for. That is why the RAG evaluation an enterprise trusts is private and owned. We make the full argument in what private AI evaluation is. We do not evaluate your RAG system for you; we give you the private machinery to evaluate it yourself.

Common questions about RAG evaluation