Agentic RAG: Retrieval-Augmented Agents

"Retrieval is not enough. Sometimes you need to search, reason, search again, and then admit you still need more evidence."

RAG, Iteratively Augmented AI Agent

1. From Static RAG to Agentic RAG

Traditional RAG follows a fixed pipeline: receive query, retrieve documents, generate response. The retrieval step happens once, with no ability to reformulate the query if results are poor, no mechanism to follow up on partial answers, and no way to combine information from multiple retrieval strategies. Agentic RAG transforms retrieval into an agent capability: the agent decides when to retrieve, what query to use, how to evaluate retrieved results, and whether to retrieve again with a refined query.

This shift is significant because real information needs rarely map to a single retrieval query. A user asking "How does our pricing compare to competitors for enterprise customers?" might require: (1) retrieving the company's pricing page, (2) searching for competitor pricing, (3) finding the enterprise tier definitions, and (4) retrieving recent sales data. An agentic RAG system treats each of these as separate retrieval actions orchestrated by the agent's planning loop, rather than hoping a single query captures all the needed context.

Three key research threads have advanced agentic RAG. Corrective RAG (CRAG) evaluates retrieved documents for relevance and triggers web search as a fallback when the knowledge base lacks the answer. Adaptive-RAG classifies queries by complexity and routes simple queries to direct retrieval while routing complex queries through multi-step agentic reasoning. Self-RAG teaches the model to generate special tokens that trigger retrieval only when the model determines it needs external information, rather than retrieving for every query.

Corrective RAG Pattern

This snippet implements a corrective RAG loop that evaluates retrieval quality and re-queries when the initial results are insufficient.

from langgraph.graph import StateGraph, END
from typing import TypedDict, List

class CRAGState(TypedDict):
 query: str
 documents: List[dict]
 relevance_scores: List[float]
 needs_web_search: bool
 final_answer: str

def retrieve(state: CRAGState) -> dict:
 """Retrieve documents from the knowledge base."""
 docs = vector_store.similarity_search(state["query"], k=5)
 return {"documents": docs}

def evaluate_relevance(state: CRAGState) -> dict:
 """Score each document for relevance to the query."""
 scores = []
 for doc in state["documents"]:
 score = llm.invoke(
 f"Rate the relevance of this document to the query "
 f"on a scale of 0 to 1.\n"
 f"Query: {state['query']}\n"
 f"Document: {doc['content'][:500]}\n"
 f"Score (just the number):"
 )
 scores.append(float(score.content.strip()))
 needs_web = all(s < 0.5 for s in scores)
 return {"relevance_scores": scores, "needs_web_search": needs_web}

def web_search_fallback(state: CRAGState) -> dict:
 """Search the web when knowledge base results are insufficient."""
 web_results = web_search_tool(state["query"])
 return {"documents": state["documents"] + web_results}

def route_after_eval(state: CRAGState) -> str:
 if state["needs_web_search"]:
 return "web_search"
 return "generate"

graph = StateGraph(CRAGState)
graph.add_node("retrieve", retrieve)
graph.add_node("evaluate", evaluate_relevance)
graph.add_node("web_search", web_search_fallback)
graph.add_node("generate", generate_answer)
graph.set_entry_point("retrieve")
graph.add_edge("retrieve", "evaluate")
graph.add_conditional_edges("evaluate", route_after_eval)
graph.add_edge("web_search", "generate")
graph.add_edge("generate", END)

2. Multi-Source Retrieval Agents

Production knowledge systems span multiple data sources: vector databases, SQL databases, APIs, document stores, and web search. An agentic RAG system treats each source as a tool the agent can invoke. The agent decides which sources to query based on the question type: factual questions about internal data go to the SQL database, conceptual questions go to the vector store, current events go to web search, and complex questions may require querying multiple sources and synthesizing results.

The agent's ability to route queries to the right source is a form of learned metadata. By including source descriptions in the tool definitions ("This database contains customer transaction records from 2020 to present" vs. "This vector store contains product documentation and user guides"), the model can make intelligent routing decisions. This source-aware routing often outperforms naive approaches that embed everything into a single vector store, because it preserves the structure and query capabilities of each data source.

3. Knowledge-Grounded Agents

Knowledge-grounded agents go beyond retrieval to maintain a persistent knowledge state. Instead of retrieving fresh context for every query, they build and maintain a knowledge graph or structured memory of facts extracted from documents. When a new document is added or a conversation reveals new information, the agent updates its knowledge state. When answering questions, it reasons over this structured knowledge rather than raw retrieved text.

This approach is especially effective for domains with complex, interconnected information: medical knowledge bases, legal case law, technical documentation. The knowledge graph captures relationships (drug A interacts with drug B, regulation X applies to industry Y) that are difficult to retrieve through vector similarity alone. The agent can traverse these relationships to answer complex queries that require multi-hop reasoning across documents.

Exercises