Memory Architectures That Improve Execution

"Intelligence without memory is reaction. Intelligence with memory is adaptation."

Sage, Memory Cherishing AI Agent

1. A Taxonomy of Agent Memory

Cognitive science distinguishes several types of human memory, and this taxonomy maps usefully onto agent systems. Each memory type serves a different purpose in agent execution and requires different storage, indexing, and retrieval strategies.

1.1 Episodic Memory: What Happened

Episodic memory stores records of specific past experiences: "On March 15, the user asked me to refactor the authentication module. I used the code search tool to find all auth-related files, then applied changes to three files. The user approved two changes but rejected the third because it broke backwards compatibility." Each episode contains the context, the actions taken, the outcomes, and any feedback received.

In production, episodic memory enables the agent to avoid repeating mistakes. If the agent encountered a specific error pattern before and learned that a particular tool invocation sequence resolved it, episodic retrieval surfaces that past experience when a similar situation arises. The connection to RAG (covered in Section 20.1) is direct: episodic memories are embedded, indexed, and retrieved using the same vector similarity techniques used for document retrieval.

1.2 Semantic Memory: What Is Known

Semantic memory stores general knowledge that has been distilled from experience. While episodic memory records "what happened on March 15," semantic memory records the extracted lesson: "The authentication module uses a plugin architecture; changes to the base class require updating all registered plugins." Semantic memories are more compact and more broadly applicable than episodic memories.

Building semantic memory from episodic memory requires a distillation process. After a set of related episodes accumulates, the agent (or a background process) synthesizes them into general principles. This is analogous to how human learners extract rules from examples. In practice, this distillation is often performed by a separate LLM call that takes a batch of related episodes and produces a concise summary of the learned principle.

1.3 Procedural Memory: How To Do Things

Procedural memory stores learned procedures and workflows: "To deploy to staging, first run the test suite, then build the Docker image, push to the registry, and update the Kubernetes manifest." Procedural memories are sequences of actions that have been validated through experience. They differ from semantic memories in that they are action-oriented rather than fact-oriented.

In agent systems, procedural memory can be implemented as stored tool call sequences (playbooks) that the agent can invoke for recurring tasks. When the agent encounters a task that matches a stored procedure, it can execute the procedure directly rather than reasoning from first principles each time. This dramatically improves both speed and reliability for common operations.

2. Production Memory Systems

Several open-source and commercial systems have emerged to provide memory capabilities for LLM agents. Each takes a different approach to the storage, retrieval, and management of agent memories.

2.1 MemGPT and Virtual Context Management

MemGPT (Packer et al., 2023) introduced the concept of virtual context management for LLMs, inspired by operating system virtual memory. The core insight is that an LLM's context window is analogous to RAM: limited in size but fast to access. MemGPT adds a "disk" layer (a persistent database) and gives the model explicit tools to move information between the context window and long-term storage. The model learns to page information in and out of its context as needed, maintaining a working set of relevant memories while keeping the full history available on demand.

# Persistent agent memory store: records observations with timestamps,
# importance scores, and access counts. Supports semantic search and
# time-weighted retrieval for long-running conversational agents.
from dataclasses import dataclass, field
from datetime import datetime
import json

@dataclass
class MemoryEntry:
 """A single memory entry with metadata for retrieval."""
 content: str
 memory_type: str # "episodic", "semantic", "procedural"
 created_at: datetime = field(default_factory=datetime.now)
 last_accessed: datetime = field(default_factory=datetime.now)
 access_count: int = 0
 importance_score: float = 0.5
 tags: list[str] = field(default_factory=list)
 source_episode_ids: list[str] = field(default_factory=list)

class AgentMemoryManager:
 """Manages episodic, semantic, and procedural memory stores.

 Provides write, retrieve, and garbage collection operations.
 Uses a vector store for similarity search and a relational
 store for metadata queries.
 """

 def __init__(self, vector_store, max_context_memories: int = 5):
 self.vector_store = vector_store
 self.max_context_memories = max_context_memories
 self.working_memory: list[MemoryEntry] = []

 def remember(self, content: str, memory_type: str, **kwargs):
 """Store a new memory."""
 entry = MemoryEntry(
 content=content,
 memory_type=memory_type,
 **kwargs,
 )
 # Compute importance based on content characteristics
 entry.importance_score = self._score_importance(entry)

 # Store in vector database for similarity retrieval
 self.vector_store.upsert(
 id=f"{memory_type}_{entry.created_at.isoformat()}",
 text=content,
 metadata={
 "type": memory_type,
 "importance": entry.importance_score,
 "tags": json.dumps(entry.tags),
 },
 )
 return entry

 def recall(self, query: str, memory_type: str = None,
 top_k: int = 5) -> list[MemoryEntry]:
 """Retrieve relevant memories for a given query."""
 filters = {}
 if memory_type:
 filters["type"] = memory_type

 results = self.vector_store.query(
 text=query,
 top_k=top_k,
 filters=filters,
 )

 entries = []
 for result in results:
 entry = MemoryEntry(
 content=result.text,
 memory_type=result.metadata["type"],
 importance_score=result.metadata["importance"],
 )
 entry.last_accessed = datetime.now()
 entry.access_count += 1
 entries.append(entry)
 return entries

 def load_into_context(self, query: str) -> str:
 """Load the most relevant memories into a context string.

 This is the 'page in' operation from MemGPT's
 virtual context management pattern.
 """
 memories = self.recall(query, top_k=self.max_context_memories)
 self.working_memory = memories

 if not memories:
 return "No relevant memories found."

 context_parts = []
 for i, mem in enumerate(memories, 1):
 context_parts.append(
 f"[Memory {i} ({mem.memory_type})]: {mem.content}"
 )
 return "\n".join(context_parts)

 def _score_importance(self, entry: MemoryEntry) -> float:
 """Score memory importance for retention priority."""
 base = 0.5
 if entry.memory_type == "procedural":
 base = 0.7 # procedures are high value
 if entry.tags:
 base += 0.1 # tagged memories are curated
 return min(base, 1.0)

2.2 Zep: Continuous Memory for Conversational Agents

Zep provides a managed memory layer specifically designed for conversational AI. Its distinctive feature is automatic memory extraction: as conversations proceed, Zep continuously extracts facts, preferences, and relationships from the dialogue and stores them as structured memory entries. This removes the burden of explicit memory management from the agent and ensures that important information is captured even when the agent does not explicitly choose to "remember" something.

Zep maintains both a message store (raw conversation history) and a memory store (extracted facts and summaries). When the agent needs context for a new conversation, Zep retrieves relevant memories from both stores and injects them into the prompt. The temporal relevance scoring ensures that recent memories are weighted more heavily than old ones, while high-importance memories (e.g., the user's name, their key preferences) are always included regardless of recency.

2.3 Mem0: User-Centric Memory Graphs

Mem0 takes a user-centric approach, organizing memories around entities (users, projects, organizations) rather than conversations. Each entity has a memory graph that accumulates knowledge over time. When the agent interacts with a user, Mem0 retrieves the user's memory graph and provides it as context. This enables personalization that persists across sessions and even across different agent applications that share the same Mem0 backend.

The graph structure is particularly useful for capturing relationships between entities: "User A is the manager of User B" or "Project X depends on Service Y." These relational memories enable the agent to reason about organizational context in ways that flat key-value stores cannot support.

3. Memory Indexing, Retrieval, and Garbage Collection

As an agent accumulates memories over weeks or months of operation, the memory store grows large enough to create retrieval quality and performance challenges. Effective memory management requires strategies for indexing, retrieval ranking, and garbage collection.

3.1 Hybrid Retrieval for Memories

Pure vector similarity search (as used in basic RAG, see Section 20.2) is insufficient for memory retrieval because relevance depends on more than semantic similarity. A memory about a specific tool failure from last week is more relevant than a semantically similar memory from six months ago. Hybrid retrieval combines vector similarity with metadata filters (recency, memory type, importance score, access frequency) to produce rankings that reflect true relevance.

3.2 Memory Consolidation

Over time, multiple episodic memories about the same topic should be consolidated into fewer, richer semantic memories. This process mirrors human memory consolidation during sleep. A background job periodically scans the episodic memory store, identifies clusters of related episodes, and generates consolidated semantic memories that capture the essential patterns. The original episodes can then be archived (moved to cold storage) or deleted, keeping the active memory store compact.

3.3 Garbage Collection Strategies

Not all memories are worth keeping. A memory about a transient API error that was resolved has no long-term value. Garbage collection for agent memories uses a scoring function that considers recency (when was the memory last accessed?), frequency (how often has it been retrieved?), importance (was it explicitly marked as important?), and redundancy (is the same information captured in a more recent or more general memory?). Memories that score below a threshold are candidates for deletion.

4. Measuring the Impact of Memory on Task Completion

Memory systems add complexity and cost. To justify their inclusion, you must measure their impact on agent performance. The most direct measurement is an A/B test: run the agent with and without memory on the same task distribution and compare task completion rates, step counts, error rates, and user satisfaction scores.

Common findings from memory impact studies include: 15% to 30% reduction in average step count for recurring task types (the agent does not need to re-discover solutions it has already found); 10% to 20% improvement in task completion rate for personalization-sensitive tasks (the agent remembers user preferences and context); and significant reduction in user frustration for long-running relationships (users do not need to repeat information across sessions).

However, memory can also hurt performance if poorly implemented. Stale or incorrect memories can lead the agent down wrong paths. Excessive memory retrieval can bloat the context and increase latency. And the privacy implications of persistent memory (see Section 32.2) require careful data governance. Always measure net impact, not just the benefits.

5. Connections to RAG and Vector Database Architecture

Agent memory and RAG share infrastructure but serve different purposes. RAG retrieves from a static or slowly-changing knowledge base to ground the model's responses in factual information. Agent memory retrieves from a dynamic, continuously-updated store of experiences to inform the model's behavior. The key differences are in the data lifecycle (RAG corpora are curated; memories are auto-generated), the update frequency (RAG corpora change on a schedule; memories change with every interaction), and the relevance criteria (RAG optimizes for factual accuracy; memory optimizes for behavioral improvement).

In practice, many production systems combine both: a RAG pipeline provides factual grounding from a knowledge base (as described in Chapter 20), while a memory system provides experiential context from past interactions. The agent's prompt includes both retrieved documents and retrieved memories, giving it access to both "what is true" and "what has worked before." The vector database infrastructure (covered in Chapter 19) can serve both systems, though the indexing strategies and update patterns differ.

Exercises