Planning & Agentic Reasoning

"A plan is just a list of things that will not happen in order."

Agent X, Pragmatically Planning AI Agent

26.2.1 From Simple Loops to Strategic Planning

A basic ReAct agent operates one step at a time: observe, think, act, repeat. This works well for simple tasks but breaks down when the problem requires coordinating multiple steps, managing dependencies between actions, or recovering from dead ends. Planning gives agents the ability to think ahead, decompose complex tasks into manageable subtasks, and reason about the order in which those subtasks should be executed.

The simplest planning approach is plan-and-execute: the agent first generates a complete plan (a numbered list of steps), then executes each step sequentially, checking results against expectations after each step. If a step fails or produces unexpected output, the agent can revise its plan before continuing. This separation of planning from execution makes the agent's reasoning transparent and debuggable.

More sophisticated approaches treat planning as a search problem. Tree of Thoughts (ToT) explores multiple reasoning paths in parallel, evaluating each path's promise before committing resources. Language Agent Tree Search (LATS) combines Monte Carlo tree search with LLM-based evaluation, treating each potential action sequence as a node in a search tree and using the model to estimate the value of unexplored branches.

ReWoo: a single-pass plan with cross-task variables

Plan-and-execute as described above interleaves planning with execution: the model plans, runs a step, observes, possibly replans, runs the next step. ReWoo (Reasoning WithOut Observation, Xu et al., 2023) takes the opposite design choice: it commits to the entire plan in one LLM call and only afterwards calls the tools, with no observation-driven replanning during execution. The key trick that makes this practical is variable substitution. The planner emits a numbered list whose tool-call arguments may reference the results of earlier steps by name ($E1, $E2, etc.), so the plan reads like a small dataflow program:

Plan:
  E1 = Search["population of Japan 2024"]
  E2 = Search["population of South Korea 2024"]
  E3 = Calculator["$E1 - $E2"]
Solver(question, $E1, $E2, $E3): <final answer>

A separate executor walks the plan in order, dispatches each tool call (substituting any Ei placeholder with the result of step i), and stores the result back in a results dictionary. After all calls return, a separate solver LLM call receives the original question together with the populated E1, E2, ... values and writes the final natural-language answer. Because the planner runs only once and the solver only once, ReWoo uses far fewer LLM calls than plan-and-execute for tasks with k tool calls (two LLM calls plus k tool calls, rather than 1 + k replan checks). The trade-off is that ReWoo cannot adapt when an intermediate result invalidates a later step: the plan is committed up front. This makes ReWoo the right choice for predictable compound queries (multi-hop fact lookups, financial calculations from API data, ETL-style aggregations) and a poor choice for exploratory tasks where every observation might rewrite the plan. In LangGraph, ReWoo is the cleanest three-node graph in this section: planner → executor → solver, no conditional edges needed.

import re

def rewoo(question, llm, tools):
    """Planner -> worker (executor) -> solver, with $Ei variable substitution."""
    plan = llm(f"Write a numbered plan for: {question}\n"
               f"Each line: Ei = tool_name[args]. Tools: {list(tools)}")
    results = {}
    for line in plan.splitlines():
        m = re.match(r"\s*(E\d+)\s*=\s*(\w+)\[(.+)\]", line)
        if not m:
            continue
        var, tool, args = m.groups()
        for k, v in results.items():
            args = args.replace(f"${k}", str(v))   # variable substitution
        results[var] = tools[tool](args)
    return llm(f"Question: {question}\nEvidence: {results}\nAnswer:")

from collections import deque

def baby_agi(objective, llm, max_steps=10):
    """Three-role autonomous loop: create -> prioritize -> execute."""
    queue = deque([f"Develop a first plan for: {objective}"])
    history = []
    for _ in range(max_steps):
        if not queue:
            break
        task = queue.popleft()                              # execute
        result = llm(f"Objective: {objective}\nTask: {task}\nResult:")
        history.append((task, result))
        new_tasks = llm(f"Given result '{result}', list new subtasks "
                        f"toward objective. One per line.").splitlines()
        queue.extend(t.strip("-* ") for t in new_tasks if t.strip())
        ranked = llm(f"Reorder tasks by importance for '{objective}': "
                     f"{list(queue)}").splitlines()
        queue = deque(t.strip("-* 1234567890.") for t in ranked if t.strip())
    return history

Plan-and-Execute Architecture

Input: task T, tool set Tools, LLM M, max replans R
Output: final answer
1. plan = M("Decompose T into numbered steps") // planning phase
2. results = []
3. for step_idx = 0 to len(plan):
a. result = execute_step(plan[step_idx], Tools, results)
b. results.append(result)
c. // reflection: check if plan still valid
d. verdict = M("Given results so far, does remaining plan make sense?")
e. if verdict == "replan" and replans_remaining > 0:
plan = M("Revise plan given results: " + results)
replans_remaining -= 1
// continue with updated plan
4. answer = M("Synthesize final answer from all results")
return answer

from langgraph.graph import StateGraph, END
from typing import TypedDict, List, Optional

class PlanExecuteState(TypedDict):
    task: str
    plan: List[str]
    current_step: int
    results: List[str]
    final_answer: Optional[str]

def create_plan(state: PlanExecuteState) -> dict:
    """Generate a multi-step plan for the task."""
    response = llm.invoke(
        f"Create a step-by-step plan to accomplish this task:\n"
        f"{state['task']}\n\n"
        f"Return a numbered list of concrete, actionable steps."
    )
    steps = parse_numbered_list(response.content)
    return {"plan": steps, "current_step": 0, "results": []}

def execute_step(state: PlanExecuteState) -> dict:
    """Execute the current step of the plan."""
    step = state["plan"][state["current_step"]]
    previous = "\n".join(
        f"Step {i+1}: {r}" for i, r in enumerate(state["results"])
    )
    result = agent_executor.invoke(
        f"Execute this step: {step}\n\nPrevious results:\n{previous}"
    )
    return {
        "results": state["results"] + [result],
        "current_step": state["current_step"] + 1,
    }

def should_replan(state: PlanExecuteState) -> str:
    """Check if the plan needs revision after the latest step."""
    if state["current_step"] >= len(state["plan"]):
        return "synthesize"
    # Ask the LLM if the plan still makes sense
    check = llm.invoke(
        f"Given the results so far, does the remaining plan still make sense?\n"
        f"Results: {state['results']}\n"
        f"Remaining steps: {state['plan'][state['current_step']:]}"
    )
    if "replan" in check.content.lower():
        return "replan"
    return "execute"

# Build the graph
graph = StateGraph(PlanExecuteState)
graph.add_node("plan", create_plan)
graph.add_node("execute", execute_step)
graph.add_node("replan", create_plan)
graph.add_node("synthesize", synthesize_answer)
graph.set_entry_point("plan")
graph.add_edge("plan", "execute")
graph.add_conditional_edges("execute", should_replan)
graph.add_edge("replan", "execute")
graph.add_edge("synthesize", END)

from pydantic_ai import Agent
agent = Agent(
    "openai:gpt-4o",
    system_prompt=(
    "You are a research assistant. Break the user's request "
    "into steps, execute each, and synthesize a final answer."
    ),
)
result = agent.run_sync("Plan a migration from session-based to JWT auth")
print(result.data)

26.2.2 Tree of Thoughts and LATS

Tree of Thoughts (ToT) extends chain-of-thought prompting by exploring multiple reasoning paths simultaneously. Instead of committing to a single chain of reasoning, the agent generates several candidate next steps, evaluates each one's promise using the LLM as a heuristic, and selects the most promising branch to continue exploring. This is particularly effective for problems where the first approach attempted is unlikely to be optimal, such as creative writing, mathematical proofs, or complex code refactoring.

The ToT loop is short enough to write in a dozen lines of Python pseudocode: generate $k$ candidate next thoughts, score each with the LLM, expand the best, repeat. The snippet below uses a simple beam (keep top $b$ at every depth) and stops when a candidate self-reports a final answer.

def tree_of_thoughts(problem, llm, depth=3, k=3, beam=2):
    """Generate -> evaluate -> expand -> select best across `depth` levels."""
    frontier = [(problem, 0.0)]  # (partial_path, cumulative_score)
    for _ in range(depth):
        scored = []
        for path, prior in frontier:
            children = llm.generate(f"{path}\nPropose {k} next steps:", n=k)
            for child in children:
                score = float(llm.score(f"Rate this step 1-10:\n{child}"))
                scored.append((path + "\n" + child, prior + score))
        frontier = sorted(scored, key=lambda x: -x[1])[:beam]  # keep top-b
        if any("FINAL:" in p for p, _ in frontier):
            break
    return max(frontier, key=lambda x: x[1])[0]

LATS (Zhou et al., 2024) takes this further by applying Monte Carlo tree search (MCTS) principles. Each node in the tree represents a state (the agent's observations and actions so far). MCTS runs four operations per iteration: selection (descend from root using a UCB-style score that balances exploit and explore), expansion (generate child action candidates with the LLM), evaluation (simulate or use the LLM as a value model on the new state), and backpropagation (update the value estimates of ancestors). LATS also augments each rollout with verbal self-reflection so subsequent iterations avoid known failure modes. LATS has shown strong results on challenging benchmarks like HumanEval and WebShop, where the search-based approach discovers solutions that greedy single-path agents miss.

Figure 26.2.2 walks the four MCTS stages around the tree: selection descends by UCB1, expansion adds a child by prompting the LLM, simulation scores it with an LLM value head, and backpropagation pushes that value back up the visited path.

The MAP (Multi-Agent Planning) approach distributes planning across multiple agents. A decomposition agent breaks the problem into independent subproblems. Specialist agents solve each subproblem in parallel. A synthesis agent combines the results. This is useful when subproblems have minimal dependencies, enabling parallelism that reduces wall-clock time compared to sequential planning approaches.

26.2.3 Reflection and Self-Correction

Reflection is the agent's ability to evaluate its own outputs and improve them. After completing a task or receiving feedback (from tools, tests, or humans), a reflective agent asks itself: "Did this work? What went wrong? How can I do better?" This self-evaluation step is what separates agents that improve over a task from those that simply execute a fixed strategy regardless of results.

Reflexion (Shinn et al., 2023) formalized this into a three-component architecture: an Actor that takes actions, an Evaluator that assesses the outcome, and a Self-Reflection module that generates verbal feedback. The reflection output is stored in memory and included in the prompt for subsequent attempts, enabling the agent to learn from its mistakes within a single task episode. This approach has shown significant improvements on coding benchmarks, where the agent can learn from failed test cases.

To build intuition for why reflection works: consider a student who writes an essay, then re-reads it with fresh eyes. The re-reading step catches errors that were invisible during writing because the student's mental state has "reset" enough to see the text as a reader would. LLM reflection works similarly. The critic prompt shifts the model's attention from generating to evaluating, engaging different patterns in the model's weights. This is why separate generator and critic roles outperform a single "generate then check" instruction: the role switch creates a meaningful shift in the model's behavior.

RefleXion: a Responder + Revisor pair with structured JSON and tool-augmented citations

The Actor / Evaluator / Self-Reflection split of Reflexion is general; a useful concrete instantiation that has shown up across LangGraph notebooks, the AutoGen recipe gallery, and Anthropic's research assistant prompts is the Responder + Revisor pair. Two LLM roles drive the loop. The Responder reads the user question and produces an initial structured answer; its output schema (a Pydantic BaseModel in practice) carries three fields: answer, reflection (a short, self-critical note about what is missing or shaky), and search_queries (a list of follow-up questions the model would ask to fix the weak parts). The Revisor takes the previous answer, the reflection, and the results of executing the search queries against a real retrieval tool, and emits the same schema but additionally populates a references field with citation IDs drawn from the search results. The pair alternates for a small fixed number of rounds (typically 2 to 3) or until the reflection field reports "no further weaknesses".

Three engineering details make this pattern reliable. First, the structured-output schema is enforced via tool-call parsing: the LLM "calls" a dummy function named Responder or Revisor whose argument schema is the Pydantic model, which means any malformed output triggers the function-calling validator and the model is retried until parse succeeds. Second, the search queries are not free text; the Revisor runs them through the agent's real retrieval tool (a vector store, a web search, an internal docs MCP server) before its next turn, so the loop integrates new external evidence on each round rather than spinning on the model's parametric memory. Third, the references field forces the model to cite specific retrieved passages by source_id, which lets a downstream evaluator check that the cited passages actually support the claims, the same idea as faithfulness scoring in agentic-RAG (Section 27.5).

from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI

class AnswerWithReflection(BaseModel):
    answer: str = Field(description="Best current answer to the question.")
    reflection: str = Field(description="Self-critique: what is missing or weak.")
    search_queries: list[str] = Field(
        description="Up to 3 follow-up queries to address the reflection.")
    references: list[str] = Field(
        default_factory=list,
        description="source_id list for citations; empty on the first round.")

llm = ChatOpenAI(model="gpt-4o", temperature=0)

# The Responder writes the first draft with no citations yet.
responder = llm.with_structured_output(AnswerWithReflection)

# The Revisor is the same schema but consumes the previous draft + tool results.
def revise(draft: AnswerWithReflection, tool_results: list[dict]) -> AnswerWithReflection:
    return responder.invoke(
        f"Previous draft:\n{draft.model_dump_json()}\n\n"
        f"New evidence from tool calls:\n{tool_results}\n\n"
        "Revise the answer to address the reflection. Cite every claim using the "
        "source_id values in tool_results and put them in `references`.")

# One round of Responder + Revisor with a real retrieval call in between.
draft = responder.invoke("Why did the Concorde program end in 2003?")
hits = retrieval_tool.batch(draft.search_queries)   # real search / MCP call
final = revise(draft, hits)
print(final.answer, final.references)

pip install letta
from letta_client import Letta
client = Letta(base_url="http://localhost:8283")
agent = client.agents.create(name="me",
    memory_blocks=[{"label": "human", "value": "User loves transformers."},
                   {"label": "persona", "value": "Helpful research assistant."}],
    model="openai/gpt-4o-mini",
    embedding="openai/text-embedding-3-small")
client.agents.messages.create(agent_id=agent.id,
    messages=[{"role": "user", "content": "What did I tell you I love?"}])

Exercises