Memory Architecture for Agents

"A chatbot needs to remember the conversation. An agent needs to remember what it did and what it still has to do."

KV, Carefully Curated AI Agent

26.6.1 What Is Agent-Specific About Agent Memory

A pure chatbot has a single memory problem: keep the conversation coherent while staying inside the context window. That problem is handled at length in Section 37.3, which covers sliding windows, summarization buffers, semantic / vector-store memory, cross-session user profiles, and self-managing systems like MemGPT and Letta. Everything in that section applies to agents too; an agent that talks to a user inherits the entire conversational stack.

What agents add is state that is not part of the dialogue. An agent executing a multi-step plan needs to know which steps it has finished, which tools it has called and what they returned, which sub-goal it is currently pursuing, and what intermediate values it has computed and may need to refer back to. None of that lives in the user-facing transcript, and most of it is too verbose to keep in the context window at full fidelity. Four memory shapes are characteristic of agents and rarely show up in pure chatbots:

• Plan / scratchpad memory: the current task tree, which steps are complete, which are in flight, and which are blocked. This is what lets a ReAct loop know it has already tried option A and should try option B.
• Tool-call history: a structured log of every tool the agent invoked, the arguments it passed, and the result that came back. This is the actions side of the agent's experience, distinct from anything the user said.
• Episodic memory of completed tasks: compact summaries of past tasks ("on 2026-04-12 the agent was asked to triage the outage, ran list_incidents three times, escalated to on-call after two retries"). Lets the agent recognize "I have done this before" across sessions.
• Agent-state checkpoints: serialized snapshots of the agent's internal state at well-defined moments, so a long-running task can resume after a crash, a sandbox restart, or a human handoff.

26.6.2 Working Memory for Multi-Step Plans

An agent's working memory is not the same thing as the LLM's context window, even though both share the name. The context window is the prompt the model sees on a single call; the agent's working memory is the structured plan state that the orchestrator maintains across calls and renders into the prompt selectively. A plan with twenty steps does not belong in the prompt verbatim on every turn; what belongs in the prompt is "you are on step 7 of 20, here is what you have completed, here is what comes next".

A workable plan-memory record holds, for each step: the step description, the status (pending / running / done / failed), any tool calls or sub-results attached to it, and a dependency edge to its parent step. Rendering the plan into the prompt becomes a projection problem: keep finished steps as one-line summaries, keep the current step in full, and elide the not-yet-relevant future. The same idea drives the LangGraph state graphs and the OpenAI Assistants thread-state APIs.

# Plan / scratchpad memory: structured progress state that the agent
# orchestrator maintains across LLM calls and renders into the prompt
# selectively.
from dataclasses import dataclass, field
from enum import Enum
from typing import Any

class StepStatus(Enum):
    PENDING = "pending"
    RUNNING = "running"
    DONE    = "done"
    FAILED  = "failed"

@dataclass
class PlanStep:
    step_id: str
    description: str
    status: StepStatus = StepStatus.PENDING
    depends_on: list[str] = field(default_factory=list)
    summary: str = ""          # short result blurb after completion
    tool_calls: list[dict] = field(default_factory=list)

class PlanMemory:
    """Working memory for a multi-step agent plan."""
    def __init__(self):
        self.steps: list[PlanStep] = []

    def render_for_prompt(self, current: str) -> str:
        """Project the plan into a compact prompt slice.

        Finished steps become one-liners ("done: looked up customer record").
        Current step is verbatim.
        Future steps are name-only.
        """
        lines = []
        for s in self.steps:
            if s.status == StepStatus.DONE:
                lines.append(f"[done] {s.description} -> {s.summary}")
            elif s.step_id == current:
                lines.append(f"[current] {s.description}")
            else:
                lines.append(f"[{s.status.value}] {s.description}")
        return "\n".join(lines)

    def mark_done(self, step_id: str, summary: str) -> None:
        for s in self.steps:
            if s.step_id == step_id:
                s.status, s.summary = StepStatus.DONE, summary

26.6.3 Tool-Call History

Every tool invocation an agent makes is a piece of memory the agent will want to look back on. The naive design (just leave the raw tool messages in the conversation) blows up fast: a single 10K-character API response on turn 3 will sit in the prompt for the rest of the task, distorting every subsequent reasoning step. The structured alternative is to keep the tool log in a side channel, render only the parts that matter into the prompt, and put everything else behind a deterministic ID the agent can reference.

A useful tool-log schema has four fields per call: a stable call_id, the tool name, the arguments (JSON), and a result summary rather than the raw output. The raw output goes to a separate store keyed by call_id so the agent can ask for it explicitly via a recall_tool_result(call_id) tool when it actually needs to re-read the bytes. This pattern, which the OpenAI Assistants API formalizes as "thread runs with tool outputs", keeps the per-turn prompt budget bounded by the number of currently relevant tool calls, not by the cumulative history.

26.6.4 Episodic Memory of Completed Tasks

When a task ends (success, failure, or human handoff), the agent should write an episode summarizing what it tried, what worked, and what failed. Episodes are typed records with an outcome label, a 3 to 5-sentence narrative, the tools that were called, and metadata like elapsed time and cost. They are stored in the same vector index as conversational long-term memory (covered in Section 37.3), but with a memory_type: "agent_episode" tag that distinguishes them from raw chat history.

Episodes are how agents learn from themselves without fine-tuning. When a new task arrives, the agent's loader can retrieve the three most similar past episodes ("you have triaged this kind of outage before; here is what worked the last time and what wasted time") and inject them into the system prompt as one-shot guidance. The Reflexion line of work (Shinn et al., 2023) and the AutoGen "previous-attempt memory" patterns are both formalizations of this idea: turn each task into a structured episode, then retrieve relevant episodes at the start of the next one.

26.6.5 Agent-State Checkpointing

Long-running agents fail. The sandbox dies, the model rate-limits, the user closes the tab, a tool call times out. An agent that has been working for forty minutes and loses everything on a crash is a worse product than a chatbot that loses its last reply. Checkpointing fixes this by serializing the agent's state at well-defined moments (end of each plan step, before any expensive tool call, on graceful shutdown) and restoring it on the next run.

A minimal checkpoint includes: the plan-memory state, the tool-call log, the running summary of any conversation context, the current step pointer, and a monotonically increasing version counter so multiple writers do not stomp on each other. Storage is usually a key-value store or a single JSON blob in object storage keyed by session_id. Frameworks like LangGraph and Temporal expose this as a first-class concept ("checkpointers", "workflows with replay"); even hand-rolled agents benefit from the discipline, because it forces the orchestrator to keep agent state separable from in-memory Python objects.

from dataclasses import asdict
import json, pathlib

class AgentCheckpoint:
    """Serialize and restore agent state across crashes and restarts."""
    def __init__(self, root: str="./checkpoints"):
        self.root = pathlib.Path(root); self.root.mkdir(exist_ok=True)

    def save(self, session_id: str, plan: PlanMemory,
             tool_log: list[dict], current_step: str,
             version: int) -> None:
        blob = {
            "version": version,
            "current_step": current_step,
            "plan": [asdict(s) for s in plan.steps],
            "tool_log": tool_log,
        }
        (self.root / f"{session_id}.json").write_text(json.dumps(blob))

    def load(self, session_id: str) -> dict | None:
        p = self.root / f"{session_id}.json"
        return json.loads(p.read_text()) if p.exists() else None

26.6.6 Privacy and PII for Agent Memory

Agent memory inherits all the privacy obligations of conversational memory (PII redaction, user-scoped queries, right-to-be-forgotten) and adds a few of its own. Tool-call logs are dense in sensitive data: SQL queries return PII, web search results return user IDs, debugging tool calls return secrets. The same PII filter that Section 37.3 uses for conversational memory should run over tool results before they are stored or summarized, not just over user-facing text.

For right-to-be-forgotten compliance, every memory record (plan steps, tool calls, episodes, checkpoints) must carry the originating user_id as a mandatory metadata field. A purge then becomes a single user-scoped delete across every memory store, including the side-channel raw-output store, vector indexes, and any checkpoint blobs. The regulation-and-compliance treatment in Chapter 53 covers the broader compliance landscape; from the agent's perspective the design rule is simply tag everything, scope everything, and write the purge path before you write the storage path.

26.6.7 Putting It Together

A production agent's memory subsystem is a small composition: a conversational layer (provided by the Section 37.3 stack), a plan-memory object, a tool-call log with side-channel raw storage, an episode writer, and a checkpointer. The agent-system orchestrator from Section 26.5 calls all five at well-defined points: load conversational context + relevant episodes + plan snapshot at the start of each turn, append to the tool log on every tool call, write an episode and clear ephemeral state on task completion, and checkpoint after each plan step.

Exercises