RAG Foundations: Pipeline & Why It Beats Fine-Tuning

The best students are not the ones who memorize the most. They are the ones who know exactly which book to open and which page to turn to.

RAG, Well-Read AI Agent

32.1.0 The Knowledge Storage Spectrum

Every LLM system stores knowledge somewhere. The choice is not binary; it is a 2-D spectrum with two axes:

• Access latency: how fast can the model use the knowledge during inference? Parametric weights are instantaneous (no extra round trip). Long-context windows pay a quadratic attention cost. RAG pays a vector-search cost. Agent memory pays a multi-hop reasoning cost.
• Knowledge currency: how easily can the knowledge be updated? Parametric weights require retraining. Long-context windows expire when the conversation ends. RAG and agent memory can be updated continuously without touching the model.

Three practical implications follow from this spectrum:

1. The right answer to "RAG or fine-tuning?" is usually neither. Most knowledge that practitioners want to add to a model is best stored as RAG (current, verifiable) or in the prompt (one-shot, freshest). Fine-tuning is for changing behavior, not for adding facts. See Section 16.1's adaptation decision tree.
2. Long-context windows are not a replacement for RAG. Even at 1M tokens, models exhibit the "lost in the middle" effect (Liu et al. 2023): performance is best for information at the start or end of context, with up to 40% accuracy drop in the middle. RAG with smart chunking can outperform a stuffed long context on multi-hop questions.
3. Agent memory and RAG are not the same thing. RAG retrieves documents by similarity. Agent memory retrieves experiences (past conversations, prior tool calls, learned user preferences) by relevance to the current task. They use similar infrastructure (vector stores) but answer different questions. See Section 26.6.

32.1.1 Why Retrieval-Augmented Generation?

Large language models store knowledge implicitly in their parameters during pretraining data. This parametric knowledge has three fundamental limitations. First, it has a knowledge cutoff: the model knows nothing about events after its training data was collected. Second, it is incomplete: no model can memorize every fact from its training corpus, especially rare or domain-specific information. Third, it is unverifiable: when a model generates a claim, there is no way to trace that claim back to a specific source document.

Retrieval-Augmented Generation, introduced by Lewis et al. (2020), addresses all three limitations by adding an explicit retrieval step before generation. The model receives both the user's query and a set of retrieved documents, then generates a response grounded in the retrieved evidence. This approach combines the generative fluency of LLMs with the factual precision of information retrieval systems.

Think of it like the difference between a closed-book exam and an open-book exam. A standard LLM takes a closed-book exam: it can only answer from what it memorized during training. RAG gives the model an open book. It can look up relevant passages before answering, cite its sources, and say "I could not find that information" when the book does not cover the topic. The result is answers that are more accurate, more current, and more trustworthy.

32.1.1.1 The Core RAG Loop

Every RAG system follows the same fundamental loop. Algorithm 1 formalizes the naive RAG pipeline, which serves as the foundation for the more advanced retrieval patterns covered later in this chapter.

Input: user query q, knowledge base KB, embedding model E, LLM G, top-k parameter k
Output: grounded response with citations
1. Encode: q_vec = E(q) // embed the query
2. Retrieve: docs = top_k_similar(q_vec, KB, k) // vector similarity search
3. Augment: prompt = format(q, docs) // insert docs into prompt template
e.g., "Given the following context: {docs}\n\nAnswer: {q}"
4. Generate: response = G(prompt) // LLM generates grounded answer
5. return response (with source citations from docs)

The simplicity of this pipeline is both its strength and its limitation. Each stage introduces potential failure points: the query embedding may not capture intent (step 1), retrieval may return irrelevant documents (step 2), the prompt may exceed the context window (step 3), or the LLM may ignore the retrieved context (step 4). The rest of this chapter addresses these failure modes systematically. Figure 32.1.1a illustrates the four stages of a naive RAG pipeline.

Figure 32.1.1a: An accelerated RAG pipeline showing the ingest flow (document processing, embedding, vector storage) and the query flow (retrieval, context augmentation, LLM generation). Source: NVIDIA, 2023. RAG 101: Demystifying Retrieval-Augmented Generation Pipelines.

32.1.1.2 Reference Implementation: HuggingFace RAG (DPR + BART)

Before stitching together a custom retriever and generator, it helps to look at the canonical reference implementation that the Hugging Face transformers library exposes as a single model. The RagTokenForGeneration class wraps DPR (Dense Passage Retrieval, Karpukhin et al., 2020) as the retriever and BART (Lewis et al., 2019) as the generator, trained end-to-end on the original Lewis 2020 RAG paper recipe. It is the cleanest example of the bi-encoder + generator split described in Section 31.1.

DPR is a dual encoder: two BERT towers, one for questions and one for passages, trained with an InfoNCE contrastive loss against in-batch negatives so that a question vector ends up close to the vector of any passage that answers it. At indexing time the passage encoder runs once per corpus chunk and the vectors go into FAISS; at query time only the question encoder runs, exactly the asymmetry that makes retrieval cheap. BART then takes the top-$k$ retrieved passages, concatenates each with the question, and marginalizes the encoder-decoder output across the passages to produce the answer. The four-line HF version is:

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

tok = RagTokenizer.from_pretrained("facebook/rag-token-nq")
retriever = RagRetriever.from_pretrained(
    "facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True,
)
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
ids = tok("Who founded the linux kernel?", return_tensors="pt").input_ids
print(tok.batch_decode(model.generate(ids), skip_special_tokens=True))

Production RAG stacks rarely ship the exact facebook/rag-token-nq checkpoint, but the architecture (frozen passage index, lightweight query encoder, generator that fuses the top-$k$) is the template every modern stack inherits. The DPR question encoder lives on as a strong bi-encoder baseline that is still competitive with newer sentence-transformer models on the Natural Questions benchmark.

Figure 32.1.2a: The DPR bi-encoder: two independent BERT towers map questions and passages to a shared 768-d space. At indexing time only the passage tower runs (once per chunk, results cached in FAISS); at query time only the question tower runs (once per query). The training objective pulls the matching (question, passage) pair closer than every other passage in the same mini-batch.

The DPR training objective is the InfoNCE contrastive loss with in-batch negatives. For a mini-batch of $B$ (question, positive-passage) pairs, each question $q_i$ uses the other $B-1$ passages in the batch as cheap negatives:

Here $\mathrm{sim}(q, p) = q \cdot p$ is the dot product, $p_i^{+}$ is the gold passage for $q_i$, and $\tau$ is a temperature (DPR sets $\tau = 1$ and relies on the unnormalized similarity scale). Larger batch sizes mean more (and harder) negatives at no extra annotation cost, which is why the original DPR paper trained with $B = 128$ on 8 GPUs. The same recipe powers every modern sentence-transformer (SBERT, E5, BGE), with refinements like hard-negative mining and longer context windows.

32.1.2 The Ingestion Pipeline

Before retrieval can happen, documents must be processed and indexed. The ingestion pipeline transforms raw documents (PDFs, web pages, databases, Markdown files) into searchable chunks stored in a vector database. The quality of this pipeline directly determines the quality of retrieved results, making it one of the most important components of any RAG system.

32.1.2.1 Document Loading and Preprocessing

The first step is loading documents from their source format and extracting clean text. This involves handling diverse formats (PDF, HTML, DOCX, CSV), removing boilerplate content (headers, footers, navigation), preserving meaningful structure (headings, tables, lists), and extracting metadata (title, author, date, source URL) for later filtering.

32.1.2.2 Chunking Strategies

Raw documents are typically too long to fit in a single retrieval result. Chunking splits documents into smaller segments that can be independently embedded and retrieved. The choice of chunking strategy profoundly affects retrieval quality: chunks that are too small lose context, while chunks that are too large dilute relevance and waste context window space.

Common Chunking Approaches

This snippet compares fixed-size, sentence-based, and recursive chunking strategies for splitting documents.

# Two chunking primitives: fixed-window-with-overlap and heading-aware.
# tiktoken gives us exact token counts so chunks fit a known embedding budget.
import tiktoken

def chunk_by_tokens(text: str, max_tokens: int = 512, overlap: int = 50) -> list[dict]:
    """Sliding-window chunking with token-level overlap to avoid splitting facts."""
    encoder = tiktoken.encoding_for_model("gpt-4o")
    tokens = encoder.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = start + max_tokens
        window = tokens[start:end]
        chunks.append({
            "text": encoder.decode(window),
            "token_count": len(window),
            "start_token": start,
        })
        start = end - overlap  # slide window, keep `overlap` tokens of context
    return chunks

def chunk_by_structure(markdown_text: str) -> list[dict]:
    """Split a markdown doc on heading boundaries; each section becomes one chunk."""
    sections, current = [], {"heading": "", "content": []}
    for line in markdown_text.split("\n"):
        if line.startswith("#"):
            # A new heading closes the previous section.
            if current["content"]:
                sections.append(current)
            current = {"heading": line.lstrip("# "), "content": []}
        else:
            current["content"].append(line)
    if current["content"]:  # flush the trailing section
        sections.append(current)
    return [
        {"text": "\n".join(s["content"]), "metadata": {"heading": s["heading"]}}
        for s in sections
    ]

32.1.2.3 Embedding and Indexing

After chunking, each chunk is converted into a dense vector using an embedding model and stored in a vector database. The embedding model's quality is critical: it determines whether semantically similar queries and documents will have similar vector representations. Popular embedding models include OpenAI's text-embedding-3-small, Cohere's embed-v3, and open-source options like BAAI/bge-large-en-v1.5.

# Ingest a list of text chunks: embed via the OpenAI Embeddings API,
# then persist them in a local ChromaDB collection configured for cosine similarity.
from openai import OpenAI
import chromadb
client = OpenAI()
chroma = chromadb.PersistentClient(path="./chroma_db")
collection = chroma.get_or_create_collection(
    name="documents",
    metadata={"hnsw:space": "cosine"}
)
def ingest_chunks(chunks, source_doc):
    """Embed and store chunks in ChromaDB."""
    texts = [c["text"] for c in chunks]
    # Batch embed (max 2048 texts per API call)
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    embeddings = [item.embedding for item in response.data]
    # Store with metadata for filtering
    collection.add(
        ids=[f"{source_doc}_chunk_{i}" for i in range(len(chunks))],
        embeddings=embeddings,
        documents=texts,
        metadatas=[{
        "source": source_doc,
        "chunk_index": i,
        "heading": chunks[i].get("metadata", {}).get("heading", "")
        } for i in range(len(chunks))]
    )
    return len(chunks)

For local, cost-free embeddings, sentence-transformers plus FAISS replaces the API call entirely:

# Library shortcut: local embeddings + FAISS (pip install sentence-transformers faiss-cpu)
from sentence_transformers import SentenceTransformer
import faiss, numpy as np
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
texts = ["chunk one text...", "chunk two text...", "chunk three text..."]
embeddings = model.encode(texts, normalize_embeddings=True)
index = faiss.IndexFlatIP(embeddings.shape[1])
index.add(embeddings.astype(np.float32))
# Query
q_vec = model.encode(["What is the vacation policy?"], normalize_embeddings=True)
dists, ids = index.search(q_vec.astype(np.float32), k=3)
print(f"Top chunks: {ids[0]}, scores: {dists[0]}")

Figure 32.1.3a: The ingestion pipeline transforms raw documents into indexed vector embeddings through parsing, chunking, and embedding stages.

32.1.3 Naive RAG: The Retrieve-and-Generate Pattern

The simplest RAG implementation follows a straightforward pattern: embed the user query, retrieve the top-k most similar chunks from the vector database, concatenate them into a prompt, and pass the augmented prompt to the LLM. Despite its simplicity, this "naive RAG" approach delivers substantial improvements over ungrounded generation for many use cases. The function below implements this retrieve-and-generate pattern end to end.

# Naive RAG in three steps: vector-search the chunks, stitch them into a prompt,
# and call gpt-4o to generate a grounded answer with explicit source citations.
def naive_rag(query: str, k: int = 5) -> dict:
    """Simple retrieve-and-generate RAG pipeline."""
    # Step 1: Retrieve relevant chunks
    results = collection.query(
        query_texts=[query],
        n_results=k
    )
    retrieved_docs = results["documents"][0]
    sources = results["metadatas"][0]
    # Step 2: Build augmented prompt
    context = "\n\n---\n\n".join(
        [f"[Source: {s['source']}]\n{doc}"
        for doc, s in zip(retrieved_docs, sources)]
    )
    prompt = f"""Answer the question based on the provided context.
    If the context does not contain enough information, say so clearly.
    Cite the source documents used in your answer.
    Context:
    {context}
    Question: {query}
    Answer:"""
    # Step 3: Generate grounded response
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1
    )
    return {
        "answer": response.choices[0].message.content,
        "sources": sources,
        "num_chunks_used": len(retrieved_docs)
    }

pip install llama-index llama-index-llms-openai llama-index-embeddings-openai
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

docs = SimpleDirectoryReader("./policies").load_data()
index = VectorStoreIndex.from_documents(docs)
engine = index.as_query_engine(similarity_top_k=5, response_mode="compact")
answer = engine.query("What is our vacation policy?")

The same retrieve-and-generate pipeline can be assembled in six lines using LangChain's built-in chain:

# Library shortcut: RAG with LangChain (pip install langchain langchain-openai langchain-chroma)
from langchain_chroma import Chroma
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain.chains import RetrievalQA
vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=OpenAIEmbeddings())
qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o", temperature=0.1),
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)
result = qa.invoke("What is our vacation policy?")
print(result["result"])

32.1.4 Context Window Management

Modern LLMs have large context windows (128K for GPT-4o, 1M for GPT-4.1, 200K for Claude, 1M-2M for Gemini 1.5/2.5, and 10M for Llama 4 Scout), but stuffing the entire context window with retrieved documents is rarely optimal. Research has revealed important patterns in how LLMs process long contexts that directly affect RAG system design.

32.1.4.1 The Lost-in-the-Middle Problem

Liu et al. (2024) demonstrated that LLMs attend more strongly to information at the beginning and end of their context, with reduced attention to content in the middle. This "U-shaped" attention pattern means that documents placed in the middle of a long context are more likely to be ignored. For RAG systems, this implies that simply concatenating many retrieved documents can actually hurt performance if the most relevant document ends up in the middle of the context.

32.1.4.2 Optimal Context Sizing

This snippet experiments with different context window sizes to find the optimal balance between recall and relevance.

# Greedy context packer: keep adding relevance-sorted chunks until the budget runs out.
# Highest-relevance chunks first so the primacy effect lands the best evidence at the top.
def build_context_with_budget(retrieved_chunks: list[dict], token_budget: int = 4000) -> tuple[str, int]:
    """Pack chunks into one context string while respecting a hard token budget."""
    encoder = tiktoken.encoding_for_model("gpt-4o")
    context_parts, total_tokens = [], 0
    for chunk in retrieved_chunks:  # Assumed pre-sorted by descending relevance
        chunk_tokens = len(encoder.encode(chunk["text"]))
        if total_tokens + chunk_tokens > token_budget:
            break  # Adding this chunk would exceed the budget; stop packing.
        context_parts.append(chunk["text"])
        total_tokens += chunk_tokens
    return "\n\n---\n\n".join(context_parts), total_tokens

32.1.5 When RAG Beats Fine-Tuning

RAG and fine-tuning are complementary approaches to adapting LLMs, not competing ones. However, understanding when each approach is more appropriate helps practitioners avoid costly mistakes. The decision framework depends on several factors including knowledge volatility, the nature of the task, and available resources.