Retrieval as a Tool Call

"To my agent, the vector database is just another endpoint with a schema. The interesting question is when it decides to call me."

RAG, Tool-Surface-Aware AI Agent

27.5.1 Retrieval From the Agent Side

The RAG literature usually frames retrieval as the spine of an architecture: documents come in, queries route through a pipeline, the LLM rides on top. A tool-using agent inverts that framing. From inside the agent loop, retrieval is just another callable function alongside send_email, run_sql, or read_file. The agent does not "do RAG"; it makes a tool call whose name happens to be search_docs, receives a payload, decides whether to call it again, and continues planning.

This shift matters because the tool-use machinery from Section 27.1 already gives the agent everything it needs to drive retrieval: a schema describing how to query, a way to inspect results, a way to retry, and a way to combine results with calls to other tools. Most of the patterns described as "agentic RAG" in the literature are simply instances of good tool design applied to a retriever. The agent-side question is therefore not "what does an agentic RAG architecture look like?" (answered in Section 32.3) but "what does a well-shaped retrieval tool look like, and when should the agent call it?".

27.5.2 Shaping a Retrieval Tool

A retrieval function exposed to an agent needs a tool schema that the model can reliably target. The schema design follows the rules from Section 27.4, with three retrieval-specific obligations: a query parameter the agent can populate from natural language, a way to scope the search (a corpus name, a filter, or a date range), and a stable result shape with provenance so the agent can cite or re-rank what it gets back.

# A retrieval tool exposed to an agent via the function-calling interface.
# The agent picks `query`, `corpus`, and optional filters; the wrapper
# handles vector search, formatting, and error surfaces.
from typing import Literal, Optional
from pydantic import BaseModel, Field

class SearchDocsArgs(BaseModel):
    query: str = Field(..., description="A natural-language search query.")
    corpus: Literal["product_docs", "runbooks", "slack_archive"] = Field(
        description="Which knowledge base to search."
    )
    since: Optional[str] = Field(None, description="ISO date filter (YYYY-MM-DD).")
    k: int = Field(5, ge=1, le=20, description="Number of passages to return.")

def search_docs(args: SearchDocsArgs) -> dict:
    """Vector-search a named corpus and return structured passages.

    Returns a stable JSON shape: a list of passages with source_id,
    score, and excerpt, plus a `status` the agent can branch on.
    """
    hits = retriever.search(
        corpus=args.corpus,
        query=args.query,
        k=args.k,
        filter={"date_gte": args.since} if args.since else None,
    )
    if not hits:
        return {"status": "no_results", "passages": [],
                "hint": "Try a different corpus or broader query."}
    return {
        "status": "ok",
        "passages": [{"source_id": h.id,
                      "score": round(h.score, 3),
                      "excerpt": h.text[:800]}
                     for h in hits],
    }

Three schema choices in this example matter more than the implementation. First, the corpus is an enum, not a free-form string. Agents will hallucinate plausible-sounding collection names ("technical_docs", "internal_kb") if given the option; a closed set forces them to pick a real one or fail loudly. Second, every passage carries a source_id; this is the hook the agent uses later to attribute claims in its final answer. Third, the empty case returns status: no_results with a hint, not an exception. As described in Section 27.4, errors that the model can read and act on outperform errors that bubble up as Python tracebacks.

27.5.3 When the Agent Decides to Retrieve

A naive RAG system retrieves on every turn; an agent only retrieves when it judges it needs to. The decision lives in the model's chain of reasoning, not in pipeline plumbing, and the prompt shapes it the same way a prompt shapes any other tool selection. Three failure modes are worth budgeting for.

• Over-retrieval. Without a clear instruction, models often call search_docs for trivia they already know (capitals, definitions, public-domain facts) because retrieval feels safe. A one-line system prompt ("Only call search_docs for facts that depend on internal knowledge or recent events") suppresses most of this.
• Under-retrieval. The opposite failure mode: the model confidently answers from parametric memory when it should look something up. The mitigation is to make the cost of not retrieving visible, by requiring source citations in the final answer.
• Premature retrieval. The agent fires a query before the question is clear, retrieves noise, and lets that noise contaminate downstream reasoning. Tools like clarify_question or an explicit planning step before any tool call reduce this; see also the Adaptive-RAG pattern in Section 32.3.

Inside the loop, an agent can issue several retrieval calls in sequence, refining the query each time. The mechanism is identical to chaining any other tool: the model reads the previous result, generates a new tool_call, and the runtime executes it. The retrieval-architecture concerns of CRAG (grade results then fall back), Adaptive-RAG (route by query complexity), and Self-RAG (in-model retrieval gating) are covered in Section 32.3; at the tool-use level, each of those is one more loop iteration with a different prompt.

27.5.4 Error Handling for Retrieval Tool Calls

Retrieval tools fail in characteristic ways that the agent should be able to recover from. The four canonical failure modes:

• Empty result. The corpus had nothing relevant. Return status: no_results with a hint pointing at other corpora or query reformulation.
• Low-confidence result. Passages came back but the top score is below a threshold. Return them with a low_confidence flag rather than silently passing them through; this lets the agent decide whether to trust them, requery, or escalate to a human.
• Backend timeout. The vector index or web search is slow or down. Surface status: timeout so the agent can fall back to a cached corpus, a different tool, or an apology, rather than hanging the entire loop.
• Quota or rate-limit error. Some retrieval tools (paid search APIs in particular) have per-minute caps. The tool wrapper should return a structured error with a recommended wait time, which the agent can use to pace its calls; see the broader tool-economy treatment in Section 27.6.

Exercises