RAG Indexing, Evaluation & Long-Context Tradeoff

The best students are not the ones who memorize the most. They are the ones who know exactly which book to open and which page to turn to.

RAG, Well-Read AI Agent

This continuation of Section 32.1 moves from "what RAG is" to "how to make RAG work in production." It covers the indexing choices you must make for large corpora, how to measure whether your RAG pipeline is actually retrieving the right things, and the practical comparison between RAG and the increasingly capable long-context windows of frontier models.

32.2.1 Indexing Strategies for Large Corpora

When your knowledge base contains millions of documents, naive flat indexing becomes impractical. Several indexing strategies help maintain retrieval quality at scale.

32.2.1.1 Hierarchical Indexing

Hierarchical indexing creates multiple levels of abstraction. At the top level, document summaries are indexed. When a query matches a summary, the system then searches within that document's chunks for specific passages. This two-stage approach dramatically reduces the search space while maintaining recall. Figure 32.2.3 illustrates how hierarchical indexing narrows the search space.

Figure 32.2.3a: Hierarchical indexing narrows search from document summaries to section-level chunks, reducing search space while maintaining precision.

32.2.1.2 Metadata Filtering

Adding metadata to chunks enables pre-retrieval filtering that narrows the search space before vector similarity is computed. Common metadata fields include document type, creation date, author, department, language, and topic tags. This filtering can be combined with vector search for efficient hybrid retrieval.

32.2.1.3 RAPTOR: Recursive Cluster-and-Summarize Trees

Naive hierarchical indexing assumes the corpus already exposes a clean document outline (book chapter, then section, then paragraph). Many real corpora do not. A pile of support tickets, a dump of research papers, or a podcast transcript archive has no built-in hierarchy at all. RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval, Sarthi et al., ICLR 2024) builds the hierarchy automatically by recursively clustering and summarizing the chunks themselves.

The construction algorithm is a tight three-step loop. First, embed every leaf chunk with a sentence-embedding model. Second, cluster those embeddings (RAPTOR uses soft Gaussian mixture clustering over UMAP-reduced embeddings, so a chunk can belong to multiple parent topics). Third, for each cluster, call an LLM to write a short summary of the chunks inside, then treat that summary as a new node and re-embed it. Repeat: cluster the summary embeddings, summarize again, embed again, until the top of the tree contains only a handful of nodes that cover the entire corpus.

At retrieval time, RAPTOR offers two traversal modes. Tree traversal starts at the root, picks the top branches whose summary is most similar to the query, and descends layer by layer until reaching leaf chunks. Collapsed-tree retrieval ignores the hierarchy at query time and treats every node (leaf and summary) as a flat candidate pool, retrieving the top-$k$ by cosine similarity. The collapsed variant turns out to win in most benchmarks because it lets the retriever mix abstract summary nodes with concrete leaf chunks in a single context, which is exactly what multi-hop questions need. RAPTOR reports a 20% absolute improvement on the QuALITY long-document QA benchmark over the strongest flat-chunk baseline.

Figure 32.2.4: The RAPTOR tree built bottom-up from raw chunks. Each upper-layer node is an LLM-written summary of the cluster below it. Tree traversal at query time picks a small subtree; collapsed-tree retrieval treats all nodes as a single flat candidate pool.

Algorithm: BUILD-RAPTOR-TREE
Input:  leaf chunks C = {c_1, ..., c_n}, embedding model E,
        summarizer LLM S, max tree height H
Output: tree T with leaves C and summary nodes above

  layer_0 := [(c_i, E(c_i)) for c_i in C]
  for h = 1 .. H:
      clusters := SOFT-CLUSTER(layer_{h-1}, method = GMM-over-UMAP)
      layer_h  := []
      for cluster K in clusters:
          text_K   := concatenate text of nodes in K
          summary  := S("Summarize:", text_K)
          layer_h.append((summary, E(summary)))
      if |layer_h| <= 2: break    # stop at near-root
  Return T (all nodes across all layers)

Algorithm: SEARCH-RAPTOR-COLLAPSED
Input:  query q, tree T, top-k count k
Output: top-k context chunks (mix of leaves and summaries)

  pool   := all nodes in T (leaves and every summary level)
  scores := [cos(E(q), node.vec) for node in pool]
  Return TOP-K(pool, scores, k)

Algorithm: SEARCH-RAPTOR-TREE-TRAVERSAL
Input:  query q, tree T, branch width b, levels to descend L
Output: context nodes along the best root-to-leaf path

  frontier := root.children
  for l = 1 .. L:
      keep     := TOP-K(frontier, cos(E(q), .), b)
      frontier := union of (n.children for n in keep)
  Return keep ∪ (descended leaves)

32.2.2 Evaluation and Common Failure Modes

Evaluating a RAG system requires measuring both retrieval quality and generation quality independently. The RAG triad framework assesses three dimensions: context relevance (did we retrieve the right documents?), groundedness (does the answer stick to the retrieved context?), and answer relevance (does the answer address the original question?).

32.2.2.1 Common Failure Modes

• Retrieval failure: The correct document exists in the knowledge base but is not retrieved, often because the query and document use different terminology.
• Context poisoning: Irrelevant or contradictory documents are retrieved, causing the LLM to generate incorrect answers grounded in bad context.
• Lost in the middle: The relevant document is retrieved but placed in the middle of the context where the LLM pays less attention to it.
• Abstention failure: The model generates a confident answer instead of admitting the context is insufficient to answer the question.
• Context overflow: Too many retrieved chunks exceed the token budget, causing truncation or performance degradation.

The RAG triad above is the intrinsic evaluation track: every metric scores the system against the retrieved context rather than against a gold answer, which means no labeled test set is required. The complementary extrinsic track scores generated answers against human-written reference answers using the classical NLG metrics covered in depth in Section 42.1. BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) compare n-gram overlap between candidate and reference and were originally designed for machine translation and summarization respectively; BERTScore (Zhang et al., 2020) replaces the n-gram match with cosine similarity over contextual embeddings and correlates better with human judgments for free-form answers. For RAG specifically, BLEU and ROUGE are too rigid (the same correct answer can be phrased many ways), so practitioners default to either BERTScore for surface-form similarity or LLM-as-judge correctness scores from RAGAS or ARES (Section 32.4.8.2) for semantic correctness. The intrinsic-vs-extrinsic split matters because they fail in different ways: an unfaithful answer scores high on BLEU if it happens to share n-grams with the reference, and a faithful answer scores low if it phrases the truth differently.

32.2.3 RAG vs. Long Context Windows

The rapid expansion of context windows raises a fundamental question: if a model can accept 1 million tokens or more in a single prompt, is RAG still necessary? Models like Gemini 1.5 Pro (1M tokens), Claude 3 (200K tokens), and GPT-4o (128K tokens) can process entire codebases, full books, or large document collections in a single request. Some practitioners have concluded that "just stuff everything in the context" eliminates the need for retrieval infrastructure. This conclusion, while tempting, overlooks several critical factors that keep RAG relevant even in an era of enormous context windows.

32.2.3.1 Why RAG Still Matters

Five distinct advantages keep RAG relevant even when the context window is large enough to hold your entire corpus. We walk through each in turn.

Cost and economics. Long-context inference is expensive. Processing 1M tokens through a frontier model costs orders of magnitude more than embedding a query and retrieving 5 relevant chunks. For a production system handling thousands of queries per hour against a 10M-token knowledge base, the cost difference between "retrieve then generate" and "load everything into context" can be 100x or greater. RAG allows you to pay embedding costs once at indexing time, then pay only for the retrieved subset at query time. For most enterprise workloads, this economic advantage is decisive.

Latency. Prefill time scales with context length. Processing 1M tokens takes seconds to tens of seconds even on high-end hardware, whereas a RAG system can retrieve relevant chunks and generate a response with a context of a few thousand tokens in under a second. For interactive applications where users expect sub-second responses, RAG provides a latency profile that long-context approaches cannot match. The time-to-first-token for a 1M token prompt is fundamentally constrained by the compute required to process the full attention matrix.

Freshness and dynamic knowledge. RAG systems can incorporate new information the moment it is indexed. When a document is updated, the RAG pipeline re-chunks and re-embeds it, making the new content immediately available to queries. Long-context approaches require re-loading the entire corpus for every request, and they cannot incorporate changes that occur between requests without rebuilding the full prompt. For knowledge bases that change hourly or daily (support tickets, news feeds, regulatory updates), RAG provides a natural mechanism for staying current.

Privacy and access control. RAG pipelines can enforce document-level access controls during retrieval. A user with clearance for Department A's documents sees only Department A's chunks in their context, even though the full knowledge base spans all departments. With a long-context approach, access control requires either maintaining separate prompts per user role or implementing complex filtering logic before context assembly. RAG's retrieval stage is a natural enforcement point for per-document permissions.

Accuracy at scale. Research on the "lost in the middle" problem (discussed in Section 4) shows that models struggle to attend equally to all parts of very long contexts. Information placed in the middle of a 100K+ token context is less likely to be used accurately than information placed at the beginning or end. RAG sidesteps this problem by placing only the most relevant chunks in context, ensuring they are in a position where the model attends to them reliably. Empirical studies have shown that for needle-in-a-haystack retrieval tasks, RAG with good retrieval outperforms naive long-context approaches once the corpus exceeds roughly 50K tokens.

32.2.3.2 When Long Context Wins

Long context does offer genuine advantages in specific scenarios. For tasks requiring holistic understanding of a single large document (such as summarizing an entire book, analyzing patterns across a full codebase, or answering questions that require synthesizing information scattered across many pages), long-context models can outperform RAG because they have access to the complete document and its structure. RAG's chunking process inevitably loses cross-chunk relationships and document-level structure. Additionally, for small, static knowledge bases (under 50K tokens), simply including the full content in context avoids the complexity of building and maintaining a retrieval pipeline.

32.2.3.3 Hybrid Approaches

The most effective production systems increasingly combine both approaches. A retrieve-then-read hybrid uses RAG to narrow a large corpus to the most relevant documents, then passes those complete documents (not just chunks) into a long-context model. This combines RAG's precision in finding relevant material with the long-context model's ability to reason across the full document structure. Another pattern uses RAG as a first pass, then applies a long-context model to re-rank and synthesize across the top retrieved results.

Google's Gemini team has documented a context caching approach that offers a middle ground. Frequently accessed documents are cached in the model's context at a reduced per-token rate, allowing repeated queries against the same corpus without re-processing the full context each time. This reduces the cost disadvantage of long-context approaches while preserving their holistic understanding advantage. However, caching is currently limited to a single session and does not solve the freshness or access control challenges.

32.2.3.4 Cache-Augmented Generation (CAG)

The academic name for the same middle-ground pattern is Cache-Augmented Generation (CAG). The recipe is: take the documents you would have retrieved, run them through the model once at indexing time, and persist the resulting KV cache to disk. At query time, the LLM loads that pre-computed cache, attends to it, and generates an answer without ever rerunning the (expensive) prefill over those documents. Conceptually, the documents have been baked into "soft tokens" that live in the attention key-value store rather than in the prompt.

CAG is a sensible production sweet spot for corpora that are small enough to fit in the context window of the chosen model (typically a few hundred thousand tokens), change rarely, and serve high query volume. Examples include the public API reference for a SaaS product, the engineering wiki for a single team, or a regulatory rulebook. The retrieval step disappears entirely (so no chunker, no embedder, no vector index to operate), latency drops because prefill is skipped, and you keep the LLM's holistic reasoning over the full corpus. The cost is that any document update invalidates the cache and forces a full re-prefill, so CAG is the wrong pattern for fast-moving knowledge bases (where RAG wins) or for multi-tenant access control (where per-user prompt slicing is harder than per-user retrieval filtering).

Gemini context caching, Anthropic prompt caching, and OpenAI's response-cache hints are the three production realizations of CAG you will meet most often. Each charges a discounted per-token rate (typically 10% to 25% of the input price) for cached tokens, which is exactly the economic mechanism that makes CAG viable. Think of CAG as the third corner of a triangle with RAG and naive long-context: RAG retrieves and prefills a small context per query, long-context prefills a large context per query, CAG prefills a large context once and reuses it for many queries.

The per-query cost difference between naive long-context and CAG is easy to write down. Let $C_{\text{prefill}}$ be the cost of running the corpus through the model and $C_{\text{generate}}$ the cost of a single long-context call (which includes corpus prefill plus question prefill plus decode). Then a CAG call avoids the corpus prefill on every cached hit:

$$ C_{\text{CAG}} \;=\; C_{\text{generate}} - C_{\text{prefill}} \;+\; \alpha \cdot C_{\text{prefill}}, $$

where $\alpha \in [0.10, 0.25]$ is the cached-token discount charged by the provider. For $N$ queries against the same corpus the amortized cost is $C_{\text{prefill}} + N \cdot (C_{\text{generate}} - C_{\text{prefill}} + \alpha C_{\text{prefill}})$, which converges to $\alpha C_{\text{prefill}} + (C_{\text{generate}} - C_{\text{prefill}})$ per query as $N$ grows large, an order of magnitude cheaper than the naive $N \cdot C_{\text{generate}}$ baseline.

Figure 32.2.5: CAG separates the expensive corpus prefill (run once at indexing time) from the cheap per-query work (load the cached keys and values, prefill only the new question, decode the answer). The "soft tokens" framing reflects that the cached K and V tensors are exactly what the model would have computed by reading the documents word by word.

In practice the cache is exposed as a server-side handle that the client passes back on every request. The following snippet shows the Anthropic API form, where a single cache_control marker on the corpus message turns one prompt into an indexing call and every subsequent prompt into a cached-prefix lookup:

import anthropic

client = anthropic.Anthropic()
CORPUS = open("api_reference.md").read()  # 120K tokens, changes weekly

def ask(question: str) -> str:
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=512,
        system=[
            {
                "type": "text",
                "text": "You answer questions strictly from the provided reference.",
            },
            {
                "type": "text",
                "text": CORPUS,
                # First call: pay full input price, server stores KV cache for ~5 min.
                # Later calls within the TTL: cached read at ~10% of input price.
                "cache_control": {"type": "ephemeral"},
            },
        ],
        messages=[{"role": "user", "content": question}],
    )
    usage = response.usage
    print(f"cache_creation={usage.cache_creation_input_tokens}, "
          f"cache_read={usage.cache_read_input_tokens}")
    return response.content[0].text

print(ask("What is the rate limit for POST /v1/messages?"))  # warm path
print(ask("How do I list models?"))                          # cached path

knowledge_base = [
    {"id": "doc1", "title": "Vacation Policy",
    "text": "All full-time employees receive 20 days of paid vacation per year. Vacation days accrue monthly at 1.67 days per month. Unused days carry over up to 10 days maximum. Requests for 3+ consecutive days need 2 weeks advance notice."},
    {"id": "doc2", "title": "Remote Work Policy",
    "text": "Employees may work remotely up to 3 days per week with manager approval. Core hours are 10 AM to 3 PM local time. A stable internet connection and dedicated workspace are required. International remote work requires HR and Legal approval."},
    {"id": "doc3", "title": "Health Benefits",
    "text": "Three health plans available: Basic (80% coverage, $1500 deductible), Standard (90%, $750), Premium (95%, $250). Dental and vision included in Standard and Premium. Open enrollment occurs annually in November."},
    {"id": "doc4", "title": "Expense Reimbursement",
    "text": "Submit expenses within 30 days. Receipts required over $25. Meals reimbursed up to $75/day during travel. Domestic flights: economy class. International flights over 6 hours: business class. Hotels up to $200/night."},
    {"id": "doc5", "title": "Performance Reviews",
    "text": "Reviews conducted in June and December. Self-assessment required before each review. Manager rating on 1 to 5 scale. Rating 4+ qualifies for bonus. Two consecutive ratings below 2 triggers a performance improvement plan."},
    ]
print(f"Knowledge base: {len(knowledge_base)} documents")
for doc in knowledge_base:
    print(f" [{doc['id']}] {doc['title']} ({len(doc['text'])} chars)")

from sentence_transformers import SentenceTransformer
import numpy as np
embed_model = SentenceTransformer("all-MiniLM-L6-v2")
doc_texts = [doc['text'] for doc in knowledge_base]
doc_embeddings = embed_model.encode(doc_texts)
doc_norms = np.linalg.norm(doc_embeddings, axis=1)
print(f"Index built: {doc_embeddings.shape}")

import numpy as np
def retrieve(query, top_k=2):
    """Retrieve top-k documents by cosine similarity."""
    q_emb = embed_model.encode(query)
    scores = np.dot(doc_embeddings, q_emb) / (doc_norms * np.linalg.norm(q_emb))
    top_idx = np.argsort(scores)[::-1][:top_k]
    return [(knowledge_base[i], scores[i]) for i in top_idx]
# Test against four representative queries from the lab's policy KB.
for query in ["How many vacation days do I get?", "Can I work from home?",
    "What health insurance options exist?", "How do I submit expenses?"]:
    results = retrieve(query, top_k=2)
    print(f"\nQ: {query}")
    for doc, score in results:
        print(f" [{score:.3f}] {doc['title']}")

from openai import OpenAI
client = OpenAI()
def rag_answer(query, top_k=2):
    """Retrieve context and generate a grounded answer."""
    retrieved = retrieve(query, top_k)
    # Build context with source labels
    context = "\n\n".join(
        f"[Source {i+1}: {doc['title']}]\n{doc['text']}"
        for i, (doc, _) in enumerate(retrieved))
    system = ("You are a helpful HR assistant. Answer ONLY from the provided "
        "context. Cite sources with [Source N]. If the context does not "
        "cover the question, say so clearly.")
    user_msg = f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:"
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": system},
        {"role": "user", "content": user_msg}],
        temperature=0.3, max_tokens=300)
    answer = response.choices[0].message.content
    sources = [doc['title'] for doc, _ in retrieved]
    return answer, sources
# Test the full pipeline
for q in ["How many vacation days do I get?", "Can I work from home?",
    "What health plans are available?", "How do I expense meals?"]:
    answer, sources = rag_answer(q)
    print(f"\nQ: {q}\nA: {answer}\nSources: {sources}\n{'='*50}")

out_of_scope = [
    "What is the company's stock price?",
    "Who is the CEO?",
    "What programming languages does engineering use?",
    ]
print("=== Out-of-Scope Query Test ===")
for q in out_of_scope:
    answer, sources = rag_answer(q)
    print(f"\nQ: {q}\nA: {answer}")

from sentence_transformers import SentenceTransformer
from openai import OpenAI
import numpy as np
client = OpenAI()
embed_model = SentenceTransformer("all-MiniLM-L6-v2")
kb = [
    {"id":"doc1","title":"Vacation Policy","text":"20 days PTO per year. Accrue 1.67/month. Carry over max 10 days. 2 weeks notice for 3+ days."},
    {"id":"doc2","title":"Remote Work","text":"Up to 3 days/week remote with manager approval. Core hours 10AM-3PM. International needs HR/Legal."},
    {"id":"doc3","title":"Health Benefits","text":"Basic (80%/$1500), Standard (90%/$750), Premium (95%/$250). Dental/vision in Standard+. November enrollment."},
    {"id":"doc4","title":"Expenses","text":"Submit within 30 days. Receipts over $25. Meals $75/day. Domestic economy, international biz class 6hr+. Hotels $200/night."},
    {"id":"doc5","title":"Reviews","text":"June and December. Self-assessment required. 1-5 scale. 4+ for bonus. Two sub-2 ratings triggers PIP."},
    ]
texts = [d['text'] for d in kb]
embs = embed_model.encode(texts)
norms = np.linalg.norm(embs, axis=1)
def retrieve(q, k=2):
    qe = embed_model.encode(q)
    scores = np.dot(embs, qe) / (norms * np.linalg.norm(qe))
    idx = np.argsort(scores)[::-1][:k]
    return [(kb[i], scores[i]) for i in idx]
def rag(q, k=2):
    ret = retrieve(q, k)
    ctx = "\n\n".join(f"[Source {i+1}: {d['title']}]\n{d['text']}" for i,(d,_) in enumerate(ret))
    r = client.chat.completions.create(model="gpt-4o-mini",
        messages=[{"role":"system","content":"Answer ONLY from context. Cite [Source N]. Say if info insufficient."},
        {"role":"user","content":f"Context:\n{ctx}\n\nQ: {q}\nA:"}],
        temperature=0.3, max_tokens=300)
    return r.choices[0].message.content, [d['title'] for d,_ in ret]
for q in ["How many vacation days?","Can I work remotely?","Health plan options?","Who is the CEO?"]:
    a, s = rag(q)
    print(f"Q: {q}\nA: {a}\nSources: {s}\n{'='*50}")