Agent Memory Systems

"I remember everything you told me. The hard part is knowing which memories matter right now."

Agent X, Nostalgic but Context-Limited AI Agent

1. Why Agents Need Memory

A stateless agent forgets everything between turns. It cannot recall what tools it used five minutes ago, what the user preferred last week, or what strategies failed yesterday. Memory transforms an agent from a reactive system into one that learns, adapts, and improves over time. Without memory, every interaction starts from scratch, forcing the agent to re-derive context that should already be available.

Agent memory systems fall into three broad categories borrowed from cognitive science. Episodic memory stores specific past experiences: what happened, when, and in what context. Semantic memory captures general knowledge and facts extracted from those experiences. Procedural memory encodes learned skills and action sequences that the agent can reuse. Each type serves a different purpose in the agent loop, and effective agents typically combine all three.

The fundamental challenge is fitting useful memory into a finite context window. A model with a 128K token context cannot store every past interaction verbatim. Memory systems must therefore implement strategies for compression, retrieval, and forgetting. This is analogous to how human memory works: we do not remember every detail of every conversation, but we can recall relevant information when cued by context.

MemGPT and Virtual Context Management

MemGPT (now called Letta) introduced a breakthrough concept: treating the LLM's context window like an operating system treats RAM. Just as an OS pages data between RAM and disk, MemGPT pages information between the model's active context and an external store. The agent can explicitly decide to save important information to long-term storage, retrieve relevant memories when needed, and discard outdated context to make room for new information. This gives the agent an effectively unlimited memory capacity within a fixed context window.

The architecture has three tiers. The main context is the LLM's current prompt, containing the system message, recent conversation turns, and currently relevant memories. The recall storage holds the full conversation history, searchable by recency or content. The archival storage is a vector database holding long-term facts, preferences, and learned procedures. The agent issues memory management function calls (save, search, delete) alongside its regular tool calls.

# Simplified MemGPT-style memory management
from letta import create_client

# Create a Letta client with persistent memory
client = create_client()

# Create an agent with memory tiers
agent = client.create_agent(
 name="research_assistant",
 memory_blocks=[
 {"label": "user_preferences", "value": "User prefers concise answers with code examples."},
 {"label": "project_context", "value": "Working on a Python ML pipeline for text classification."},
 ],
 tools=["web_search", "code_execution", "memory_save", "memory_search"],
)

# The agent can now manage its own memory
response = agent.send_message(
 "Remember that I prefer PyTorch over TensorFlow for new projects."
)
# Agent internally calls memory_save to persist this preference

# Later, when asked about frameworks, it retrieves the preference
response = agent.send_message(
 "What framework should I use for my next model?"
)
# Agent calls memory_search, finds the PyTorch preference, responds accordingly

Mem0 and A-MEM: Zettelkasten for Agents

Mem0 provides a simpler, API-first approach to agent memory. Rather than requiring the agent to manage its own memory explicitly, Mem0 automatically extracts and stores key facts from conversations. When the agent needs context, Mem0 retrieves relevant memories based on the current query. This reduces the cognitive burden on the agent itself, letting the memory layer operate as a transparent middleware.

A-MEM takes a different approach inspired by the Zettelkasten method, a note-taking system where ideas are stored as atomic, interconnected notes. Each memory is a self-contained unit with explicit links to related memories, forming a knowledge graph. When the agent retrieves a memory, it also gets the connected context, enabling richer reasoning about relationships between past experiences. This is particularly useful for research agents that need to synthesize information across many sources.

2. Episodic, Semantic, and Procedural Memory

Episodic memory stores timestamped records of specific interactions. When did the user last ask about deployment? What error message appeared during the last debugging session? These memories are retrieved by similarity to the current situation, enabling the agent to say "we encountered a similar issue last Tuesday" and apply the same resolution strategy. Implementation typically uses a vector store with metadata (timestamps, session IDs, outcome labels).

Semantic memory captures distilled facts and relationships. Rather than storing the full conversation where the user mentioned their database is PostgreSQL 15, semantic memory stores the fact itself: "user's database = PostgreSQL 15." This is more token-efficient and avoids the noise of raw conversation history. Knowledge graphs are a natural fit for semantic memory, with entities as nodes and relationships as edges.

Procedural memory records successful action sequences. If the agent discovered that deploying to staging requires running migrations first, then updating environment variables, then restarting the service, that procedure is stored and can be replayed in future deployments. This is the agent equivalent of muscle memory: learned skills that can be executed without re-deriving the steps each time.

import mem0

# Initialize Mem0 with different memory types
memory = mem0.Memory.from_config({
 "vector_store": {"provider": "qdrant", "config": {"url": "localhost:6333"}},
 "llm": {"provider": "openai", "config": {"model": "gpt-4o-mini"}},
})

# Store episodic memory (what happened)
memory.add(
 "User reported a 502 error on the /api/payments endpoint. "
 "Root cause was a connection pool exhaustion in the database layer. "
 "Fixed by increasing max_connections from 20 to 50.",
 user_id="user_123",
 metadata={"type": "episodic", "category": "incident", "resolved": True},
)

# Store semantic memory (what we know)
memory.add(
 "User's production stack: Python 3.11, FastAPI, PostgreSQL 15, Redis 7, "
 "deployed on AWS ECS with Fargate.",
 user_id="user_123",
 metadata={"type": "semantic", "category": "infrastructure"},
)

# Retrieve relevant memories for a new query
results = memory.search(
 "The payments API is returning errors again",
 user_id="user_123",
 limit=5,
)
# Returns the previous incident memory, enabling the agent to check
# connection pool settings as a first diagnostic step

3. Context Window Management Strategies

Even with external memory stores, the agent must decide what to put in its context window for each turn. The context window budget must be allocated across the system prompt (fixed), retrieved memories (variable), tool definitions (fixed per tool set), conversation history (growing), and space reserved for the model's response. A well-designed agent treats context allocation as an optimization problem: maximize the relevance of included information while staying within token limits.

Common strategies include sliding window (keep the N most recent messages), summarization (periodically compress older messages into summaries), retrieval-based selection (embed the current query and retrieve the most relevant past messages), and hybrid approaches that combine all three. The optimal strategy depends on the task: customer support benefits from recent context, while research tasks benefit from relevance-based retrieval across a longer history.

Exercises