RAG Evaluation: Ragas, BEIR, Faithfulness and Groundedness

"A RAG pipeline that retrieves perfectly and writes beautifully can still hallucinate. Evaluation is the only place we find out which of the three things failed."

Eval, Vibe-Averse AI Agent

RAG eval looks similar to ordinary LLM eval until you try to debug a regression. A drop in answer accuracy might mean the retriever missed the right chunk, the retriever found the chunk and the generator ignored it, or both worked and the question had no answer in the corpus. The three-layer eval cake separates these failure modes so you know which component to fix. The rest of this section makes that separation operational.

43.1.1 The Three-Layer Eval Cake

A production RAG pipeline has at minimum three failure surfaces, and a useful eval suite reports each separately. The first layer, retrieval, asks whether the relevant chunks are in the top-k. The second layer, generation, asks whether the model uses those chunks faithfully and answers the question. The third layer, end-to-end, asks whether the system gives the user a correct, helpful answer overall, including the case where it correctly refuses to answer.

Retrieval Metrics

Retrieval evaluation borrows directly from classical information retrieval. For a query with a known set of relevant documents, you report:

• Recall@k: fraction of relevant documents found in the top-k results. The single most important RAG retrieval metric. Recall@5 = 0.92 means 92 percent of relevant chunks were retrieved within the first 5 results.
• MRR (Mean Reciprocal Rank): the average of 1/rank of the first relevant result. Sensitive to whether the right answer appears at position 1, 2, or further down. Critical when the generator only reads the top-1 chunk.
• nDCG@k (Normalized Discounted Cumulative Gain): rewards relevant results that appear earlier in the ranking, with logarithmic discounting. Captures both relevance and order; standard in BEIR.
• Precision@k: fraction of retrieved chunks that are actually relevant. Important when the LLM context budget is tight and irrelevant chunks crowd out useful ones.

Formally, for a ranked list of retrieved items with binary relevance labels:

where $\operatorname{rank}_q^{*}$ is the rank of the first relevant document for query q, and $\operatorname{DCG}@k(q) = \sum_{i=1}^{k} \operatorname{rel}_i / \log_2(i+1)$.

Generation Metrics

Generation metrics assume retrieval succeeded and ask: given the retrieved context, did the LLM produce a useful answer? Three metrics dominate the 2026 toolkit:

• Faithfulness: is the answer derivable from the retrieved context? Computed by decomposing the answer into atomic claims and checking each claim against the context. A faithfulness score of 0.8 means 80 percent of the answer's claims are entailed by the retrieved chunks.
• Answer Relevance: does the answer actually address the user's question? Computed by reverse-generating questions from the answer and measuring similarity to the original query. Catches the "long, technically correct, but off-topic" failure mode.
• Context Precision and Context Recall: were the right chunks retrieved (precision) and were enough relevant chunks retrieved (recall)? These bridge retrieval and generation by using the ground-truth answer to label which chunks should have been used.

End-to-End Metrics

End-to-end metrics ignore the internals and judge the final answer:

• Answer Correctness: a weighted combination of factual accuracy (semantic similarity to ground-truth answer) and completeness (no key facts missing). Ragas computes this as a blend of F1 over claim entailment plus an embedding similarity term.
• Refusal Calibration: when the corpus does not contain the answer, does the system correctly refuse rather than confabulating? Measured by separating golden-set examples into "answerable" and "unanswerable" buckets and reporting accuracy on each.
• Semantic Similarity / Answer Similarity: embedding-based similarity to the gold answer; quick and cheap, but a weaker signal than correctness when paraphrase matters.

Figure 43.1.1: The three-layer RAG evaluation cake. Each layer answers a different diagnostic question and points to a different repair. (Diagram to commission for final styling.)

43.1.2 Faithfulness vs Groundedness (Commonly Confused)

These two terms get used interchangeably in blog posts, vendor decks, and even academic papers. They are not the same, and the difference matters in production.

Operationally, faithfulness is computed by an LLM judge that scores claim-by-claim entailment against the union of retrieved chunks. Groundedness uses the same atomic-claim decomposition but requires each claim to be linked to a specific chunk identifier; the metric is the fraction of claims with a valid citation that holds up under inspection.

Ragas as of late 2025 ships both metrics. The Faithfulness metric uses the chunk union; the ContextEntityRecall and the newer CitationFaithfulness (in the experimental branch) probe groundedness. DeepEval ships GEval and a separate CitationAccuracy metric. TruLens emphasizes its RAG triad (context relevance, groundedness, answer relevance) as the canonical generation-side instrumentation.

43.1.3 Ragas 2024-2026: From 4 Metrics to 12+

The original Ragas paper (Es et al., 2023) shipped four metrics: faithfulness, answer relevance, context precision, and context recall. The library through 2024-2026 has expanded into a 12+ metric toolkit that covers RAG, summarization, and agentic pipelines under a single API. The headline additions:

The expansion reflects how the RAG eval problem grew. The 2023 toolkit assumed a clean QA pipeline; the 2026 toolkit handles agentic loops, multi-document summaries, and policy boundaries. The downside is that metric selection became a small product-management problem: pick the wrong subset and your eval gate fires on regressions you do not care about while ignoring ones you do.

43.1.4 BEIR for Retrieval-Only Evaluation

BEIR (Benchmarking IR; Thakur et al., 2021) is a heterogeneous suite of 18 retrieval tasks spanning question answering, fact verification, biomedical search, news, and Stack Overflow-style technical Q&A. Unlike Ragas, BEIR scores only the retrieval layer: given a query and a corpus, how good is the ranked list of returned documents? Standard metrics are nDCG@10 and recall@100.

BEIR is now integrated into MTEB (Massive Text Embedding Benchmark; Muennighoff et al., 2022 with 2024-2026 expansions), the standard suite for embedding-model evaluation. MTEB v1.x covers seven task categories: retrieval (drawn from BEIR), reranking, clustering, classification, pair classification, STS, and summarization. As of 2026, MTEB has grown to encompass over 60 languages and 100+ datasets, with leaderboard submissions from every major embedding provider.

43.1.5 Two Case Studies: Where Faithfulness Suffices and Where Groundedness Is Mandatory

The Ragas/BEIR/MTEB toolkit looks complete on paper, but only the choice between faithfulness and groundedness determines whether your evaluation strategy will survive a real audit. The two case studies that follow contrast a regulated-finance RAG, where every sentence needs a cited source, with a customer-support chatbot, where over-citation would degrade the experience. The right metric depends entirely on which mistake costs more.

Case A , Regulated Finance RAG: Groundedness Is Mandatory

A mid-sized asset manager deploys a RAG assistant for portfolio managers to query the firm's internal research and regulatory filings. The corpus includes 10-K reports, internal credit-risk memos, and SEC-published guidance. The deployment is governed by SR 11-7 (the US Federal Reserve's supervisory guidance on model risk management), which requires every model-generated recommendation entering an investment decision to have a documented evidence chain.

The team initially shipped a Ragas pipeline tuned for faithfulness: claims had to be entailed by retrieved chunks. An audit two months in flagged the problem. A claim like "our credit team has held a negative view on issuer X since Q2" was scored as faithful because the entailment was supported by the union of retrieved memos. But the auditor could not point at which memo supported the claim, and the same claim with a different supporting chunk would have scored identically. Audit failed.

The fix was a redesign around groundedness. Every answer was forced to emit per-sentence citations via a structured-output prompt; a separate verification pass cross-checked that each cited chunk actually entailed the cited sentence. Ragas was supplemented with TruLens's groundedness metric, and a custom CitationCoverage metric (fraction of sentences with a valid cited chunk) became the headline ship gate. The Ragas faithfulness score barely budged: it had been 0.86 before, it was 0.85 after. But the citation coverage went from "indeterminate" (the old pipeline produced inline references but did not verify them) to a measured 0.94. Audit passed.

Lesson: in regulated settings, faithfulness is necessary but not sufficient. The evidence chain must be measurable, not implicit. Choose groundedness, not faithfulness, as your ship gate.

Case B , Customer Support Chatbot: Faithfulness Suffices

A SaaS company deploys a RAG-backed customer support assistant. The corpus is the company's documentation portal plus 2,000 resolved support tickets. Users ask questions like "How do I configure SSO for Okta?" or "Why is my export taking 30 minutes?" The product surface is a chat widget on the docs page. There is no regulatory audit.

The team picked faithfulness as the headline metric. Their reasoning: users care about getting the right answer, not about source citation. When the chatbot does cite, citations help, but inline citation is a UX cost (longer, denser responses) and the team chose to leave them off. Faithfulness scores trended at 0.91 after a quarter of golden-set tuning. Answer relevance was 0.94. End-to-end correctness, measured against a 500-ticket gold standard, was 0.82.

A regression alarm fired three months in: faithfulness dropped to 0.79. Investigation found that the team had added a "helpful follow-up" prompt that asked the LLM to suggest next steps. The next-step suggestions were not grounded in retrieved chunks; the LLM was riffing. The fix was a faithfulness-gated follow-up: suggestions were generated, then filtered by a second-pass faithfulness check, and only the surviving suggestions appeared in the response. Faithfulness recovered to 0.89, and the new prompt design shipped.

Lesson: in non-regulated, helpfulness-first deployments, faithfulness is enough. The ROI on full groundedness instrumentation does not pay off when no one is auditing the evidence trail. Choose faithfulness, instrument the metric, and gate prompt changes against it.

43.1.6 Code: A Ragas Evaluation Pipeline End-to-End

The code below wires Ragas against a small RAG pipeline (10 question-answer pairs with retrieved context). It loads a dataset, runs four core metrics, and applies a threshold gate suitable for a CI hook. The structure is the canonical Ragas 0.2+ pattern: build an evaluation dataset, pick metrics, evaluate, gate. Figure 43.1.2 shows how the four headline Ragas metrics partition the question, retrieved context, generated answer, and ground truth into a 2x2 of failure modes.

Figure 43.1.2a: The four headline Ragas metrics partitioned by which inputs they compare. Faithfulness checks whether each claim in the answer is entailed by the retrieved context (hallucination detector). Answer relevance checks whether the answer addresses the question (off-topic detector). Context precision checks whether the retrieved chunks are ranked usefully against the question (reranker quality). Context recall checks whether the retrieved chunks cover the ground-truth claims (retriever coverage). Faithfulness + answer relevance can be computed without ground truth; context recall requires a labelled gold answer.

# Ragas evaluation pipeline wired to a small RAG (10-15 lines of business logic)
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    answer_correctness,
)

# 1. Build the eval dataset: questions, retrieved contexts, generated answers, ground truths
samples = {
    "question": questions,            # list[str]: user queries
    "contexts": retrieved_chunks,    # list[list[str]]: top-k chunks per query
    "answer": generated_answers,     # list[str]: what your pipeline returned
    "ground_truth": gold_answers,     # list[str]: human-curated reference answers
}
eval_ds = Dataset.from_dict(samples)

# 2. Pick a 4-metric default; faithfulness + answer_relevancy + context_recall + correctness
result = evaluate(
    eval_ds,
    metrics=[faithfulness, answer_relevancy, context_recall, answer_correctness],
)

# 3. Threshold gate: block CI merge if any metric falls below floor
floors = {"faithfulness": 0.80, "answer_relevancy": 0.85,
          "context_recall": 0.80, "answer_correctness": 0.75}
scores = result.to_pandas().mean(numeric_only=True).to_dict()
regressions = {k: (scores[k], floors[k]) for k in floors if scores[k] < floors[k]}
if regressions:
    raise SystemExit(f"RAG eval regression: {regressions}")
print("RAG eval gate: PASS", scores)

Note three production-grade additions that the snippet omits for brevity. First, Ragas calls a judge LLM under the hood; for stability, pin the judge model and version (e.g., gpt-4o-2024-08-06 rather than gpt-4o) and average across N=3 runs. Second, the floors should be tuned per-domain; the values shown are reasonable defaults for general QA but harsh for entity-heavy domains. Third, the gate should compare against a moving baseline of the last 7 evaluations, not a fixed floor, so seasonal drift in queries does not trigger false alarms.

43.1.7 Common Failure Modes in RAG Evaluation

Three additional failure modes recur across teams:

• Positional bias in judge models. When an LLM judge scores faithfulness by reading "[claim] [context]", the order of the inputs can shift scores. The fix is the same swap-and-average pattern as in §34.1's LLM-as-judge discussion: run each judgement twice, once with claim-first and once with context-first, and only trust consistent verdicts.
• Refusal-vs-answer confusion in answer correctness. A pipeline that correctly refuses an unanswerable question scores 0 on naive answer correctness (because there is no overlap with the "gold answer", which is itself a refusal). The fix is a two-bucket evaluation: separate answerable from unanswerable golden-set entries, and report per-bucket accuracy, weighted by traffic mix.
• Single-judge model collapse. If you use the same LLM family for generation and for the Ragas judge, the judge tends to reward outputs that look like its own style. Mitigation: use a different judge family, or use a panel of three judges from different families and report agreement.