A retrieval-augmented generation (RAG) system can look impressive in a demo and still fail quietly in production. It might answer confidently while missing a key document, or sound accurate while fabricating a detail the retrieved context never mentioned. Because a RAG pipeline has two independent stages, retrieval and generation, and each can fail in ways the other stage does not catch, evaluating one without the other gives a false sense of security.
This guide explains how to evaluate a RAG system properly: the metrics that matter at each stage, the frameworks used to compute them, and how to build a testing process that catches regressions before users do.
Why RAG Evaluation Is Different From Standard LLM Evaluation
A standalone language model is evaluated on how well it generates text. A RAG system depends on two things working together: whether the retriever finds the right information, and whether the generator uses that information correctly. A high-quality generator fed the wrong context will still produce a wrong answer, and a perfect retriever paired with a generator that ignores context will still hallucinate.
This is why a real-world regression can hide from an incomplete evaluation setup. Consider a legal research assistant that scores 0.91 on a faithfulness metric, which looks strong. Weeks later, users report that the system regularly misses a second relevant statute on multi-part questions. The faithfulness score stays at 0.91 the entire time, because the generator is still being faithful to whatever context it received. The real problem is a context recall score sitting at 0.62: the retriever is failing to surface all the necessary information in the first place. A dashboard tracking only generation-stage metrics never flags this. This is the core argument for evaluating both stages, not just one.
The Two-Stage Evaluation Framework
A complete RAG evaluation program tracks at least one metric for retrieval quality and at least one metric for generation quality, since each stage fails independently.
Stage 1: Retrieval Metrics
Retrieval evaluation asks a simple question: did the system find the right information for the query? It has two classic failure modes: missing relevant chunks (low recall) and surfacing irrelevant chunks (low precision).
Precision@k measures what proportion of the top k retrieved chunks are actually relevant to the query. Low precision usually points to noisy embeddings, poor chunking, or an under-tuned similarity threshold.
Recall@k measures what proportion of all relevant chunks in the corpus were retrieved within the top k results. Low recall means the retriever is leaving important information behind, regardless of how well the generator performs afterward.
Mean Reciprocal Rank (MRR) measures how high up the first relevant result appears, on average, across queries. This matters because generators often weigh earlier context more heavily, so a relevant chunk buried at position ten is less useful than one at position one.
Normalized Discounted Cumulative Gain (NDCG) extends this idea by accounting for the graded relevance and ranking position of every retrieved result, not just the first relevant one. It rewards systems that both find relevant content and rank it near the top.
Hit Rate simply measures whether at least one relevant chunk appears anywhere in the retrieved set. It is a coarser but useful sanity check, especially early in development.
Stage 2: Generation Metrics
Generation evaluation asks whether the model used the retrieved context correctly to produce a good answer.
Faithfulness, sometimes called groundedness, measures whether the generated answer contains only claims that are supported by the retrieved context. It is typically computed by breaking the answer into individual claims and checking each one against the retrieved chunks. Low faithfulness is the clearest signal of hallucination, since it means the model is stating things the context never said.
Answer Relevancy measures whether the generated answer actually addresses the question that was asked. An answer can be perfectly faithful to the retrieved context and still miss the point of the question, so faithfulness and relevancy need to be tracked separately.
Answer Correctness compares the generated answer against a known ground-truth answer, when one is available. This is a stricter and more expensive metric than faithfulness, since faithfulness only checks consistency with the retrieved context, not correctness against real-world fact.
It is worth being precise about the difference between faithfulness and correctness here. An answer can be faithful to an incorrect or incomplete retrieved context while still being wrong overall. Faithfulness tells you the generator did its job given what it was handed. It does not tell you the retriever handed it the right thing.
The Hybrid Metrics: Context Precision and Context Recall
Two additional metrics sit between the two stages and are widely used in popular evaluation libraries such as RAGAS.
Context Precision evaluates whether the most relevant retrieved chunks are ranked near the top of the result set, which matters because generators are sensitive to where useful information sits within a long context window.
Context Recall evaluates whether the retrieved context contains everything needed to answer the question completely, catching the exact failure mode described in the legal research example earlier.
| Metric | Stage | What it catches |
|---|---|---|
| Precision@k, Recall@k, MRR, NDCG, Hit Rate | Retrieval | Missed or irrelevant chunks, poor ranking |
| Context Precision | Retrieval-Generation boundary | Relevant content buried too low to be used well |
| Context Recall | Retrieval-Generation boundary | Missing information the retriever failed to surface |
| Faithfulness (Groundedness) | Generation | Fabricated claims not supported by retrieved context |
| Answer Relevancy | Generation | Answers that are grounded but off-topic |
| Answer Correctness | Generation | Wrong answers, even if internally consistent |
An Emerging Fifth Dimension: Context Trustworthiness
Standard RAG metrics generally assume that the underlying knowledge index itself is trustworthy. That assumption does not always hold. A system can score very well on faithfulness and still return the wrong business answer if the indexed content is outdated, unowned, or drifted from the canonical source of truth. This has led some practitioners to argue for tracking context trustworthiness as a distinct evaluation dimension, covering the freshness, ownership, and lineage of the documents feeding the retriever, rather than only the retrieval and generation behavior built on top of them. For any RAG system built on a knowledge base that changes over time, this is worth treating as a real evaluation category rather than an afterthought.
Tools and Frameworks
Several open-source and commercial frameworks have become standard for computing these metrics without building everything from scratch.
RAGAS is the most widely adopted open-source library for RAG evaluation. It computes faithfulness, answer relevancy, context precision, and context recall directly from a set of questions, retrieved contexts, and generated answers, and can also generate synthetic test questions from a provided corpus.
ARES automates RAG evaluation using a technique called prediction-powered inference, which combines a small set of human-annotated examples, typically around 150, with a larger volume of automatically scored examples to produce statistically calibrated scores.
TruLens and DeepEval are commonly used for tracing RAG pipelines and testing whether system instructions and grounding requirements are actually being followed in practice, often integrated into CI/CD pipelines as automated quality gates.
Benchmark datasets such as BEIR, RGB, and CRAG provide standardized test sets for measuring retrieval robustness, multi-document integration, and multi-hop reasoning, which is useful for comparing approaches beyond a single company’s private data.
Most of these tools rely heavily on an “LLM-as-judge” approach, where a separate language model scores faithfulness or relevancy by comparing the generated answer against the retrieved context. This is efficient and scales well, but it inherits the judge model’s own blind spots, which is why spot-checking with human review remains important, especially early in a project.
Building a Practical Testing Framework
A workable RAG evaluation process generally follows a consistent structure.
Start with a golden test set. Assemble a set of representative questions covering easy cases, edge cases, multi-hop questions that require combining information from multiple chunks, and known historically difficult queries. Include expected answers or expected supporting chunks where possible.
Evaluate both stages, not just one. Track at least one retrieval metric and at least one generation metric for every test run. A generation-only dashboard, as shown in the legal research example, can hide a retrieval regression for weeks.
Automate evaluation in CI/CD. Run the test set automatically whenever the retriever, embedding model, chunking strategy, or prompt changes, and treat significant metric drops as a blocking issue rather than something to review later.
Test robustness deliberately. Include adversarial or difficult cases: ambiguous questions, questions with no good answer in the corpus, and questions requiring synthesis across multiple documents. A system that performs well only on straightforward single-document questions will struggle in production.
Monitor in production, not just offline. Offline evaluation happens against a fixed test set, but real user queries drift over time. Continuous monitoring, paired with logging user corrections and complaints as a live signal, closes the loop that a one-time offline benchmark cannot.
Separate pre-deployment testing from live monitoring, but connect them. Using the same evaluation configuration and metric definitions for both stages makes it possible to compare production behavior directly against what was measured before launch, rather than working with two disconnected sets of numbers.
Common Mistakes
- Tracking only generation-stage metrics like faithfulness while ignoring retrieval quality, which hides exactly the kind of regression described earlier in this guide.
- Treating a single overall RAGAS-style average score as sufficient, when the individual component metrics often reveal where the actual problem sits.
- Relying entirely on LLM-as-judge scoring without any human spot-checking, especially for a new or high-stakes domain.
- Evaluating only with easy, well-formed questions instead of the messy, ambiguous, and multi-hop queries real users actually submit.
- Treating an offline evaluation score as a permanent guarantee, rather than recognizing that document freshness, corpus changes, and shifting user behavior require ongoing monitoring.
- Ignoring the trustworthiness of the underlying knowledge index itself, which no faithfulness or relevancy score can catch on its own.
Frequently Asked Questions
What is the single most important RAG metric? There is no single most important metric, because retrieval and generation fail independently. Faithfulness is often highlighted as the clearest signal of hallucination, but tracking it alone will miss retrieval-stage failures like the one described earlier in this guide.
How is faithfulness different from answer correctness? Faithfulness measures whether an answer is consistent with the retrieved context, regardless of whether that context is actually correct. Answer correctness compares the answer against real-world ground truth. A faithful answer can still be wrong if the retrieved context itself was incomplete or outdated.
Do I need human evaluation if I am already using RAGAS or a similar tool? Automated, LLM-as-judge scoring scales well but carries the judge model’s own blind spots. Periodic human review, especially on a sample of production traffic, remains valuable for catching issues automated scoring misses.
How often should a RAG system be re-evaluated? At minimum, whenever the retriever, embedding model, chunking strategy, prompt, or underlying knowledge base changes. Beyond that, continuous production monitoring is necessary because real user queries and the underlying document corpus both drift over time.
Key Takeaways
A RAG system has two independent stages, retrieval and generation, and a complete evaluation program measures both rather than relying on generation-stage metrics alone. Precision@k, Recall@k, MRR, and NDCG cover retrieval quality, while faithfulness, answer relevancy, and answer correctness cover generation quality, with context precision and context recall bridging the two. Frameworks like RAGAS, ARES, TruLens, and DeepEval make these metrics practical to compute at scale, but LLM-as-judge scoring still benefits from human spot-checking. Beyond the standard metrics, the trustworthiness and freshness of the underlying knowledge index deserves its own attention, since no faithfulness score can fix a retriever pulling from a stale or unowned source.
Conclusion
Evaluating a RAG system well means resisting the temptation to reduce it to a single score. The strength of a two-stage evaluation framework is that it tells you not just whether something is wrong, but where. A team that tracks both retrieval and generation metrics, tests against realistic and adversarial queries, and keeps evaluation running continuously in production will catch the kind of quiet regression that a one-time faithfulness score of 0.91 can mask for weeks. As RAG systems move deeper into legal, medical, financial, and other high-stakes domains, that level of rigor is quickly becoming the baseline expectation rather than a nice-to-have.
AiVoogle – AI Tutorials & AI Tools
AiVoogle is an AI-focused platform sharing practical AI tutorials, AI tools, guides, reviews, and the latest trends in artificial intelligence. Our goal is to make AI simple, useful, and accessible for everyone—from beginners and creators to marketers, developers, and businesses.
The AiVoogle team researches and covers the latest AI tools and technologies to help readers discover the right tools and learn how to use AI effectively.
Focus: AI Tutorials | AI Tools | AI Guides | AI News | AI Reviews
