RAG Architecture & Fundamentals

The best students are not the ones who memorize the most. They are the ones who know exactly which book to open and which page to turn to.

RAG, Well-Read AI Agent

1. Why Retrieval-Augmented Generation?

Large language models store knowledge implicitly in their parameters during pretraining. This parametric knowledge has three fundamental limitations. First, it has a knowledge cutoff: the model knows nothing about events after its training data was collected. Second, it is incomplete: no model can memorize every fact from its training corpus, especially rare or domain-specific information. Third, it is unverifiable: when a model generates a claim, there is no way to trace that claim back to a specific source document.

Retrieval-Augmented Generation, introduced by Lewis et al. (2020), addresses all three limitations by adding an explicit retrieval step before generation. The model receives both the user's query and a set of retrieved documents, then generates a response grounded in the retrieved evidence. This approach combines the generative fluency of LLMs with the factual precision of information retrieval systems.

Think of it like the difference between a closed-book exam and an open-book exam. A standard LLM takes a closed-book exam: it can only answer from what it memorized during training. RAG gives the model an open book. It can look up relevant passages before answering, cite its sources, and say "I could not find that information" when the book does not cover the topic. The result is answers that are more accurate, more current, and more trustworthy.

1.1 The Core RAG Loop

Every RAG system follows the same fundamental loop. Algorithm 1 formalizes the naive RAG pipeline, which serves as the foundation for the more advanced retrieval patterns covered later in this chapter.

Input: user query q, knowledge base KB, embedding model E, LLM G, top-k parameter k
Output: grounded response with citations

1. Encode: q_vec = E(q)                        // embed the query
2. Retrieve: docs = top_k_similar(q_vec, KB, k) // vector similarity search
3. Augment: prompt = format(q, docs)             // insert docs into prompt template
     e.g., "Given the following context: {docs}\n\nAnswer: {q}"
4. Generate: response = G(prompt)                // LLM generates grounded answer
5. return response (with source citations from docs)

The simplicity of this pipeline is both its strength and its limitation. Each stage introduces potential failure points: the query embedding may not capture intent (step 1), retrieval may return irrelevant documents (step 2), the prompt may exceed the context window (step 3), or the LLM may ignore the retrieved context (step 4). The rest of this chapter addresses these failure modes systematically. Figure 20.1.3 illustrates the four stages of a naive RAG pipeline.

Figure 20.1.3: An accelerated RAG pipeline showing the ingest flow (document processing, embedding, vector storage) and the query flow (retrieval, context augmentation, LLM generation). Source: NVIDIA, 2023. RAG 101: Demystifying Retrieval-Augmented Generation Pipelines.

2. The Ingestion Pipeline

Before retrieval can happen, documents must be processed and indexed. The ingestion pipeline transforms raw documents (PDFs, web pages, databases, Markdown files) into searchable chunks stored in a vector database. The quality of this pipeline directly determines the quality of retrieved results, making it one of the most important components of any RAG system.

2.1 Document Loading and Preprocessing

The first step is loading documents from their source format and extracting clean text. This involves handling diverse formats (PDF, HTML, DOCX, CSV), removing boilerplate content (headers, footers, navigation), preserving meaningful structure (headings, tables, lists), and extracting metadata (title, author, date, source URL) for later filtering.

2.2 Chunking Strategies

Raw documents are typically too long to fit in a single retrieval result. Chunking splits documents into smaller segments that can be independently embedded and retrieved. The choice of chunking strategy profoundly affects retrieval quality: chunks that are too small lose context, while chunks that are too large dilute relevance and waste context window space. Code Fragment 20.1.3 below puts this into practice.

Common Chunking Approaches

This snippet compares fixed-size, sentence-based, and recursive chunking strategies for splitting documents.

# implement chunk_by_tokens, chunk_by_structure
# Key operations: chunking strategy

import tiktoken

def chunk_by_tokens(text, max_tokens=512, overlap=50):
 """Split text into chunks with token-level control."""
 encoder = tiktoken.encoding_for_model("gpt-4")
 tokens = encoder.encode(text)
 chunks = []

 start = 0
 while start < len(tokens):
 end = start + max_tokens
 chunk_tokens = tokens[start:end]
 chunk_text = encoder.decode(chunk_tokens)
 chunks.append({
 "text": chunk_text,
 "token_count": len(chunk_tokens),
 "start_token": start
 })
 start = end - overlap # Slide window with overlap

 return chunks

def chunk_by_structure(markdown_text):
 """Split markdown by headings, preserving document structure."""
 sections = []
 current_section = {"heading": "", "content": []}

 for line in markdown_text.split("\n"):
 if line.startswith("#"):
 if current_section["content"]:
 sections.append(current_section)
 current_section = {
 "heading": line.strip("# "),
 "content": []
 }
 else:
 current_section["content"].append(line)

 if current_section["content"]:
 sections.append(current_section)

 return [{
 "text": "\n".join(s["content"]),
 "metadata": {"heading": s["heading"]}
 } for s in sections]

2.3 Embedding and Indexing

After chunking, each chunk is converted into a dense vector using an embedding model and stored in a vector database. The embedding model's quality is critical: it determines whether semantically similar queries and documents will have similar vector representations. Popular embedding models include OpenAI's text-embedding-3-small, Cohere's embed-v3, and open-source options like BAAI/bge-large-en-v1.5. Code Fragment 20.1.3 below puts this into practice.

# implement ingest_chunks
# Key operations: embedding lookup, chunking strategy, API interaction

from openai import OpenAI
import chromadb

client = OpenAI()
chroma = chromadb.PersistentClient(path="./chroma_db")
collection = chroma.get_or_create_collection(
 name="documents",
 metadata={"hnsw:space": "cosine"}
)

def ingest_chunks(chunks, source_doc):
 """Embed and store chunks in ChromaDB."""
 texts = [c["text"] for c in chunks]

 # Batch embed (max 2048 texts per API call)
 response = client.embeddings.create(
 model="text-embedding-3-small",
 input=texts
 )
 embeddings = [item.embedding for item in response.data]

 # Store with metadata for filtering
 collection.add(
 ids=[f"{source_doc}_chunk_{i}" for i in range(len(chunks))],
 embeddings=embeddings,
 documents=texts,
 metadatas=[{
 "source": source_doc,
 "chunk_index": i,
 "heading": chunks[i].get("metadata", {}).get("heading", "")
 } for i in range(len(chunks))]
 )
 return len(chunks)

For local, cost-free embeddings, sentence-transformers plus FAISS replaces the API call entirely:

# Library shortcut: local embeddings + FAISS (pip install sentence-transformers faiss-cpu)
from sentence_transformers import SentenceTransformer
import faiss, numpy as np

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
texts = ["chunk one text...", "chunk two text...", "chunk three text..."]
embeddings = model.encode(texts, normalize_embeddings=True)

index = faiss.IndexFlatIP(embeddings.shape[1])
index.add(embeddings.astype(np.float32))
# Query
q_vec = model.encode(["What is the vacation policy?"], normalize_embeddings=True)
dists, ids = index.search(q_vec.astype(np.float32), k=3)
print(f"Top chunks: {ids[0]}, scores: {dists[0]}")

3. Naive RAG: The Retrieve-and-Generate Pattern

The simplest RAG implementation follows a straightforward pattern: embed the user query, retrieve the top-k most similar chunks from the vector database, concatenate them into a prompt, and pass the augmented prompt to the LLM. Despite its simplicity, this "naive RAG" approach delivers substantial improvements over ungrounded generation for many use cases. Figure 20.1.4 shows the complete ingestion pipeline.

# implement naive_rag
# Key operations: retrieval pipeline, chunking strategy, RAG pipeline

def naive_rag(query, k=5):
 """Simple retrieve-and-generate RAG pipeline."""

 # Step 1: Retrieve relevant chunks
 results = collection.query(
 query_texts=[query],
 n_results=k
 )
 retrieved_docs = results["documents"][0]
 sources = results["metadatas"][0]

 # Step 2: Build augmented prompt
 context = "\n\n---\n\n".join(
 [f"[Source: {s['source']}]\n{doc}"
 for doc, s in zip(retrieved_docs, sources)]
 )

 prompt = f"""Answer the question based on the provided context.
If the context does not contain enough information, say so clearly.
Cite the source documents used in your answer.

Context:
{context}

Question: {query}

Answer:"""

 # Step 3: Generate grounded response
 response = client.chat.completions.create(
 model="gpt-4o",
 messages=[{"role": "user", "content": prompt}],
 temperature=0.1
 )

 return {
 "answer": response.choices[0].message.content,
 "sources": sources,
 "num_chunks_used": len(retrieved_docs)
 }

The same retrieve-and-generate pipeline can be assembled in six lines using LangChain's built-in chain:

# Library shortcut: RAG with LangChain (pip install langchain langchain-openai langchain-chroma)
from langchain_chroma import Chroma
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain.chains import RetrievalQA

vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=OpenAIEmbeddings())
qa = RetrievalQA.from_chain_type(
 llm=ChatOpenAI(model="gpt-4o", temperature=0.1),
 retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)
result = qa.invoke("What is our vacation policy?")
print(result["result"])

4. Context Window Management

Modern LLMs have large context windows (128K tokens for GPT-4o, 200K for Claude), but stuffing the entire context window with retrieved documents is rarely optimal. Research has revealed important patterns in how LLMs process long contexts that directly affect RAG system design.

4.1 The Lost-in-the-Middle Problem

Liu et al. (2024) demonstrated that LLMs attend more strongly to information at the beginning and end of their context, with reduced attention to content in the middle. This "U-shaped" attention pattern means that documents placed in the middle of a long context are more likely to be ignored. For RAG systems, this implies that simply concatenating many retrieved documents can actually hurt performance if the most relevant document ends up in the middle of the context.

4.2 Optimal Context Sizing

This snippet experiments with different context window sizes to find the optimal balance between recall and relevance.

# implement build_context_with_budget
# Key operations: retrieval pipeline, chunking strategy

def build_context_with_budget(retrieved_chunks, token_budget=4000):
 """Pack chunks into context respecting a token budget.
 Places highest-relevance chunks first (primacy effect)."""
 encoder = tiktoken.encoding_for_model("gpt-4o")
 context_parts = []
 total_tokens = 0

 for chunk in retrieved_chunks: # Already sorted by relevance
 chunk_tokens = len(encoder.encode(chunk["text"]))
 if total_tokens + chunk_tokens > token_budget:
 break
 context_parts.append(chunk["text"])
 total_tokens += chunk_tokens

 return "\n\n---\n\n".join(context_parts), total_tokens

5. When RAG Beats Fine-Tuning

RAG and fine-tuning are complementary approaches to adapting LLMs, not competing ones. However, understanding when each approach is more appropriate helps practitioners avoid costly mistakes. The decision framework depends on several factors including knowledge volatility, the nature of the task, and available resources.

6. Indexing Strategies for Large Corpora

When your knowledge base contains millions of documents, naive flat indexing becomes impractical. Several indexing strategies help maintain retrieval quality at scale.

6.1 Hierarchical Indexing

Hierarchical indexing creates multiple levels of abstraction. At the top level, document summaries are indexed. When a query matches a summary, the system then searches within that document's chunks for specific passages. This two-stage approach dramatically reduces the search space while maintaining recall. Figure 20.1.5 illustrates how hierarchical indexing narrows the search space.

6.2 Metadata Filtering

Adding metadata to chunks enables pre-retrieval filtering that narrows the search space before vector similarity is computed. Common metadata fields include document type, creation date, author, department, language, and topic tags. This filtering can be combined with vector search for efficient hybrid retrieval.

7. Evaluation and Common Failure Modes

Evaluating a RAG system requires measuring both retrieval quality and generation quality independently. The RAG triad framework assesses three dimensions: context relevance (did we retrieve the right documents?), groundedness (does the answer stick to the retrieved context?), and answer relevance (does the answer address the original question?).

7.1 Common Failure Modes

• Retrieval failure: The correct document exists in the knowledge base but is not retrieved, often because the query and document use different terminology.
• Context poisoning: Irrelevant or contradictory documents are retrieved, causing the LLM to generate incorrect answers grounded in bad context.
• Lost in the middle: The relevant document is retrieved but placed in the middle of the context where the LLM pays less attention to it.
• Abstention failure: The model generates a confident answer instead of admitting the context is insufficient to answer the question.
• Context overflow: Too many retrieved chunks exceed the token budget, causing truncation or performance degradation.

8. RAG vs. Long Context Windows

The rapid expansion of context windows raises a fundamental question: if a model can accept 1 million tokens or more in a single prompt, is RAG still necessary? Models like Gemini 1.5 Pro (1M tokens), Claude 3 (200K tokens), and GPT-4o (128K tokens) can process entire codebases, full books, or large document collections in a single request. Some practitioners have concluded that "just stuff everything in the context" eliminates the need for retrieval infrastructure. This conclusion, while tempting, overlooks several critical factors that keep RAG relevant even in an era of enormous context windows.

8.1 Why RAG Still Matters

Cost and economics. Long-context inference is expensive. Processing 1M tokens through a frontier model costs orders of magnitude more than embedding a query and retrieving 5 relevant chunks. For a production system handling thousands of queries per hour against a 10M-token knowledge base, the cost difference between "retrieve then generate" and "load everything into context" can be 100x or greater. RAG allows you to pay embedding costs once at indexing time, then pay only for the retrieved subset at query time. For most enterprise workloads, this economic advantage is decisive.

Latency. Prefill time scales with context length. Processing 1M tokens takes seconds to tens of seconds even on high-end hardware, whereas a RAG system can retrieve relevant chunks and generate a response with a context of a few thousand tokens in under a second. For interactive applications where users expect sub-second responses, RAG provides a latency profile that long-context approaches cannot match. The time-to-first-token for a 1M token prompt is fundamentally constrained by the compute required to process the full attention matrix.

Freshness and dynamic knowledge. RAG systems can incorporate new information the moment it is indexed. When a document is updated, the RAG pipeline re-chunks and re-embeds it, making the new content immediately available to queries. Long-context approaches require re-loading the entire corpus for every request, and they cannot incorporate changes that occur between requests without rebuilding the full prompt. For knowledge bases that change hourly or daily (support tickets, news feeds, regulatory updates), RAG provides a natural mechanism for staying current.

Privacy and access control. RAG pipelines can enforce document-level access controls during retrieval. A user with clearance for Department A's documents sees only Department A's chunks in their context, even though the full knowledge base spans all departments. With a long-context approach, access control requires either maintaining separate prompts per user role or implementing complex filtering logic before context assembly. RAG's retrieval stage is a natural enforcement point for per-document permissions.

Accuracy at scale. Research on the "lost in the middle" problem (discussed in Section 4) shows that models struggle to attend equally to all parts of very long contexts. Information placed in the middle of a 100K+ token context is less likely to be used accurately than information placed at the beginning or end. RAG sidesteps this problem by placing only the most relevant chunks in context, ensuring they are in a position where the model attends to them reliably. Empirical studies have shown that for needle-in-a-haystack retrieval tasks, RAG with good retrieval outperforms naive long-context approaches once the corpus exceeds roughly 50K tokens.

8.2 When Long Context Wins

Long context does offer genuine advantages in specific scenarios. For tasks requiring holistic understanding of a single large document (such as summarizing an entire book, analyzing patterns across a full codebase, or answering questions that require synthesizing information scattered across many pages), long-context models can outperform RAG because they have access to the complete document and its structure. RAG's chunking process inevitably loses cross-chunk relationships and document-level structure. Additionally, for small, static knowledge bases (under 50K tokens), simply including the full content in context avoids the complexity of building and maintaining a retrieval pipeline.

8.3 Hybrid Approaches

The most effective production systems increasingly combine both approaches. A retrieve-then-read hybrid uses RAG to narrow a large corpus to the most relevant documents, then passes those complete documents (not just chunks) into a long-context model. This combines RAG's precision in finding relevant material with the long-context model's ability to reason across the full document structure. Another pattern uses RAG as a first pass, then applies a long-context model to re-rank and synthesize across the top retrieved results.

Google's Gemini team has documented a context caching approach that offers a middle ground. Frequently accessed documents are cached in the model's context at a reduced per-token rate, allowing repeated queries against the same corpus without re-processing the full context each time. This reduces the cost disadvantage of long-context approaches while preserving their holistic understanding advantage. However, caching is currently limited to a single session and does not solve the freshness or access control challenges.

pip install openai sentence-transformers numpy

knowledge_base = [
 {"id": "doc1", "title": "Vacation Policy",
 "text": "All full-time employees receive 20 days of paid vacation per year. Vacation days accrue monthly at 1.67 days per month. Unused days carry over up to 10 days maximum. Requests for 3+ consecutive days need 2 weeks advance notice."},
 {"id": "doc2", "title": "Remote Work Policy",
 "text": "Employees may work remotely up to 3 days per week with manager approval. Core hours are 10 AM to 3 PM local time. A stable internet connection and dedicated workspace are required. International remote work requires HR and Legal approval."},
 {"id": "doc3", "title": "Health Benefits",
 "text": "Three health plans available: Basic (80% coverage, $1500 deductible), Standard (90%, $750), Premium (95%, $250). Dental and vision included in Standard and Premium. Open enrollment occurs annually in November."},
 {"id": "doc4", "title": "Expense Reimbursement",
 "text": "Submit expenses within 30 days. Receipts required over $25. Meals reimbursed up to $75/day during travel. Domestic flights: economy class. International flights over 6 hours: business class. Hotels up to $200/night."},
 {"id": "doc5", "title": "Performance Reviews",
 "text": "Reviews conducted in June and December. Self-assessment required before each review. Manager rating on 1 to 5 scale. Rating 4+ qualifies for bonus. Two consecutive ratings below 2 triggers a performance improvement plan."},
]

print(f"Knowledge base: {len(knowledge_base)} documents")
for doc in knowledge_base:
 print(f" [{doc['id']}] {doc['title']} ({len(doc['text'])} chars)")

from sentence_transformers import SentenceTransformer
import numpy as np

embed_model = SentenceTransformer("all-MiniLM-L6-v2")

doc_texts = [doc['text'] for doc in knowledge_base]
doc_embeddings = embed_model.encode(doc_texts)
doc_norms = np.linalg.norm(doc_embeddings, axis=1)

print(f"Index built: {doc_embeddings.shape}")

def retrieve(query, top_k=2):
 """Retrieve top-k documents by cosine similarity."""
 q_emb = embed_model.encode(query)
 scores = np.dot(doc_embeddings, q_emb) / (doc_norms * np.linalg.norm(q_emb))
 top_idx = np.argsort(scores)[::-1][:top_k]
 return [(knowledge_base[i], scores[i]) for i in top_idx]

# Test
for q in ["How many vacation days do I get?", "Can I work from home?",
 "What health insurance options exist?", "How do I submit expenses?"]:
 results = retrieve(q, top_k=2)
 print(f"\nQ: {q}")
 for doc, score in results:
 print(f" [{score:.3f}] {doc['title']}")

from openai import OpenAI

client = OpenAI()

def rag_answer(query, top_k=2):
 """Retrieve context and generate a grounded answer."""
 retrieved = retrieve(query, top_k)

 # Build context with source labels
 context = "\n\n".join(
 f"[Source {i+1}: {doc['title']}]\n{doc['text']}"
 for i, (doc, _) in enumerate(retrieved))

 system = ("You are a helpful HR assistant. Answer ONLY from the provided "
 "context. Cite sources with [Source N]. If the context does not "
 "cover the question, say so clearly.")

 user_msg = f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:"

 response = client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[{"role": "system", "content": system},
 {"role": "user", "content": user_msg}],
 temperature=0.3, max_tokens=300)

 answer = response.choices[0].message.content
 sources = [doc['title'] for doc, _ in retrieved]
 return answer, sources

# Test the full pipeline
for q in ["How many vacation days do I get?", "Can I work from home?",
 "What health plans are available?", "How do I expense meals?"]:
 answer, sources = rag_answer(q)
 print(f"\nQ: {q}\nA: {answer}\nSources: {sources}\n{'='*50}")

out_of_scope = [
 "What is the company's stock price?",
 "Who is the CEO?",
 "What programming languages does engineering use?",
]

print("=== Out-of-Scope Query Test ===")
for q in out_of_scope:
 answer, sources = rag_answer(q)
 print(f"\nQ: {q}\nA: {answer}")

from sentence_transformers import SentenceTransformer
from openai import OpenAI
import numpy as np

client = OpenAI()
embed_model = SentenceTransformer("all-MiniLM-L6-v2")

kb = [
 {"id":"doc1","title":"Vacation Policy","text":"20 days PTO per year. Accrue 1.67/month. Carry over max 10 days. 2 weeks notice for 3+ days."},
 {"id":"doc2","title":"Remote Work","text":"Up to 3 days/week remote with manager approval. Core hours 10AM-3PM. International needs HR/Legal."},
 {"id":"doc3","title":"Health Benefits","text":"Basic (80%/$1500), Standard (90%/$750), Premium (95%/$250). Dental/vision in Standard+. November enrollment."},
 {"id":"doc4","title":"Expenses","text":"Submit within 30 days. Receipts over $25. Meals $75/day. Domestic economy, international biz class 6hr+. Hotels $200/night."},
 {"id":"doc5","title":"Reviews","text":"June and December. Self-assessment required. 1-5 scale. 4+ for bonus. Two sub-2 ratings triggers PIP."},
]

texts = [d['text'] for d in kb]
embs = embed_model.encode(texts)
norms = np.linalg.norm(embs, axis=1)

def retrieve(q, k=2):
 qe = embed_model.encode(q)
 scores = np.dot(embs, qe) / (norms * np.linalg.norm(qe))
 idx = np.argsort(scores)[::-1][:k]
 return [(kb[i], scores[i]) for i in idx]

def rag(q, k=2):
 ret = retrieve(q, k)
 ctx = "\n\n".join(f"[Source {i+1}: {d['title']}]\n{d['text']}" for i,(d,_) in enumerate(ret))
 r = client.chat.completions.create(model="gpt-4o-mini",
 messages=[{"role":"system","content":"Answer ONLY from context. Cite [Source N]. Say if info insufficient."},
 {"role":"user","content":f"Context:\n{ctx}\n\nQ: {q}\nA:"}],
 temperature=0.3, max_tokens=300)
 return r.choices[0].message.content, [d['title'] for d,_ in ret]

for q in ["How many vacation days?","Can I work remotely?","Health plan options?","Who is the CEO?"]:
 a, s = rag(q)
 print(f"Q: {q}\nA: {a}\nSources: {s}\n{'='*50}")