Sim's Brain Stew
AI Agent Memory

Article

AI Agent Memory

Published on
Authors

Ask most people what “memory” means for an AI agent, and they’ll describe one thing: the agent remembers what you said earlier in the conversation. That’s true, but it’s only one of at least seven distinct memory systems that show up in real agent architectures — and conflating them is one of the most common mistakes in agent design.

The truth is, “memory” in an AI system isn’t a single feature. It’s a set of different mechanisms, each solving a different problem, each with different trade-offs around cost, latency, and durability. A well-designed agent doesn’t just have “memory” — it deliberately chooses which of these seven (or more) systems to use, and for what.

This post walks through all seven, following the same system design lens each time: what it is, why it exists, where it shows up in production, a concrete example, and a rough sketch of how data flows through it. At the end, we’ll cover a few additional memory patterns that didn’t make the original list but are increasingly showing up in real agent architectures.

mindmap
  root((AI Agent Memory))
    In-Context / Working
      System prompt
      Message thread
      Tool outputs
    Semantic
      Facts
      Preferences
      Domain knowledge
    Episodic
      Past events
      Successes / Failures
    Procedural
      Skills
      Workflows
      Tool usage patterns
    External / Retrieval
      Vector DB
      RAG
    Parametric
      Model weights
    Prospective
      Task queue
      Goals
      Reminders

1. In-Context / Working Memory

What it is: The active context window — everything the model can “see” right now: the system prompt, the current message thread, tool outputs, and intermediate reasoning steps.

Why it’s used: It’s the agent’s scratchpad, keeping a conversation or task coherent turn to turn. Without it, every message would be answered in a vacuum, with no memory of what was just said.

Where it’s used: Every multi-turn agent or chatbot relies on this, typically managed through a “checkpointer” that persists the current thread’s state between calls.

Example: You ask an agent “what did I just say?” — it looks back into its current context window and answers correctly, because that message is still sitting right there in the active context.

Interview signal: “How do you manage context across turns?” — a strong answer touches on context window management, checkpointing strategy, and what happens when the conversation grows too long to fit (truncation, summarization, or windowing strategies).

Workflow sketch:

flowchart LR
    C["Clients (Web / Mobile)"] --> AR["Agent Runtime"]
    subgraph AR2["In-Context Memory"]
        M1[Messages]
        M2[System Prompt]
        M3[Tool Output]
        M4[Reasoning]
    end
    AR --> AR2
    AR2 <--> CP["Checkpointer (Thread ID)"]
    AR2 --> TS["Tools & Systems<br/>LLM · Tools/Functions · Vector DB · APIs"]

The critical thing to understand about working memory is that it’s the only memory type that’s genuinely free to access — it’s just sitting in the prompt already. Everything else on this list exists precisely because working memory has a hard limit: the context window. Once a conversation or task outgrows that window, you need somewhere else to put things, which is where the next six memory types come in.

2. Semantic Memory

What it is: A persistent store of facts, preferences, and domain knowledge about a user or topic — knowledge that outlives any single conversation.

Why it’s used: It enables personalized, consistent, and accurate responses across sessions, instead of the agent starting from scratch with every new conversation.

Where it’s used: Personalized assistants, customer support bots, and enterprise knowledge agents — anywhere the system needs to “know” things about a specific user or domain long-term.

Example: An agent remembers that a user prefers metric units, and quietly applies that preference the next time they ask a question involving measurements, days or weeks later.

Interview signal: “How do you design long-term memory for an AI agent?” — good answers cover the data model (what gets stored and how it’s structured), the storage choice (usually a vector database or profile store), the retrieval strategy, and privacy considerations around storing personal facts long-term.

Workflow sketch:

flowchart TD
    C["Clients / Agents<br/>Chat, App, Agents"] --> SML["Semantic Memory Layer"]
    SML <--> VS["Vector DB / Profile Store<br/>(embeddings, structured data)"]
    SML --> CO["Consumers<br/>LLM/Inference · Tools · Personalization · RAG"]

Semantic memory is what most people actually mean when they casually say “the AI remembers me.” It’s decoupled from when a fact was learned — it just persists as a durable fact about the world or the user, ready to be retrieved whenever it’s relevant.

3. Episodic Memory

What it is: A record of past events, conversations, and task outcomes — specifically what happened, when, and whether it succeeded or failed.

Why it’s used: It lets agents learn from experience and avoid repeating the same mistakes, rather than making the identical error in every fresh session.

Where it’s used: Coding agents, decision-making agents, and support systems that need audit trails of what was tried and what happened.

Example: The Reflexion technique (Shinn et al., NeurIPS 2023) has an agent write a self-reflection after each failed attempt at a task, store that reflection, and reference it on the next attempt — a technique that pushed accuracy from GPT-4’s 80% up to 91% on a coding benchmark, purely by remembering and learning from its own past failures.

Interview signal: “How would you design memory so agents can learn from past failures?” — strong answers mention event logging, retrieval by similarity (finding past episodes relevant to the current situation), summarization of long histories, and retention policies for how long to keep old episodes around.

Workflow sketch:

flowchart TD
    A["Agent / Client<br/>conversation, task run, event"] --> EMS["Episodic Memory Store"]
    EMS <--> EL["Event Log DB<br/>(structured logs, metadata + results)"]
    EMS --> AR["Agent Runtime:<br/>Recall past events → Learn & adapt → Avoid repeated mistakes"]

The distinction between episodic and semantic memory trips a lot of people up. Semantic memory is distilled knowledge (“the user prefers metric units”). Episodic memory is the raw record of what happened (“on March 3rd, the agent tried approach A, it failed with error X, and it then tried approach B successfully”). Often, semantic memory is actually built from episodic memory over time, by summarizing and generalizing across many individual episodes.

4. Procedural Memory

What it is: The agent’s knowledge of how to do things — skills, workflows, tool usage patterns, and behavioral rules.

Why it’s used: It lets agents follow known, proven procedures instead of re-deriving the steps to a task from scratch every single time.

Where it’s used: Workflow automation and coding agents, typically stored as prompt templates, tool registries, and skill libraries.

Example: Voyager, an autonomous Minecraft-playing agent, builds up a growing library of executable skills as it plays. For new tasks, instead of figuring everything out from first principles again, it reuses and composes skills it already learned and stored earlier.

Interview signal: “How do you design an agent that can learn and reuse procedures?” — good answers cover skill libraries, versioning (what happens when a skill needs to be updated), retrieval (finding the right skill for a new but similar task), and safe execution boundaries.

Workflow sketch:

flowchart TD
    A["Agent / User<br/>task/goal, tool request"] --> PML["Procedural Memory Layer"]
    PML <--> SS["Skills / Procedures Store<br/>(workflows, tool schemas, templates, scripts)"]
    PML --> TS["Tools & Systems<br/>APIs · Code Executors · Databases · External Services"]

If semantic memory is “knowing that,” procedural memory is “knowing how” — the classic distinction from cognitive science, borrowed directly into agent design. A coding agent’s procedural memory might be a library of tested code patterns; a customer service agent’s might be a set of proven de-escalation scripts.

5. External / Retrieval Memory

What it is: Outside knowledge stored in a vector database and pulled into context at inference time, based on similarity search — this is the mechanism underneath most RAG (Retrieval-Augmented Generation) systems.

Why it’s used: A model’s context window is finite, but the amount of knowledge a system might need access to is effectively unlimited. Retrieval memory lets an agent access vast, current information without needing to fit it all into the prompt at once.

Where it’s used: Knowledge-base Q&A, document analysis, and support agents that need to look things up on demand rather than having everything pre-loaded.

Example: A support agent embeds a company’s documentation into a vector database ahead of time, then, when a user asks a question, retrieves just the most relevant chunks of that documentation and injects them into the prompt before generating an answer.

Interview signal: “How would you design retrieval for your support agent?” — strong answers address embedding strategy, chunking (how documents get split up), indexing, similarity search method, and how you keep the retrieved information fresh as source documents change.

Workflow sketch:

flowchart LR
    U["User / Agent<br/>question / request"] --> EQ[Embed Query]
    EQ --> SS[Similarity Search]
    SS --> TK[Top-K Chunks]
    TK --> IC[Inject into Context]
    SS <--> VD[("Vector Database<br/>embeddings, metadata, sources")]
    IC --> LI["LLM / Inference"]
    LI --> AN["Answer to User"]

Retrieval memory overlaps conceptually with semantic memory — both often live in a vector database — but the distinction is about scope and freshness. Semantic memory tends to be durable facts specifically about a user or ongoing relationship. Retrieval memory is typically a much larger, more general knowledge base (like an entire documentation set) that gets searched fresh on every relevant query.

6. Parametric Memory

What it is: Knowledge baked directly into the model’s weights during training — language patterns, reasoning ability, and general world knowledge, with no retrieval step required.

Why it’s used: It’s the fastest and cheapest form of memory there is, since there’s no lookup involved — the knowledge is simply already “inside” the model, instantly accessible.

Where it’s used: Everywhere, implicitly, in every single model response. Its major limitation is that it’s frozen at whatever point training ended, and it’s genuinely hard to update or audit what a model “knows” this way.

Example: A model knows what a REST API is without ever being told, because that knowledge was baked into its weights during training — but it won’t know anything about a library that was released last month, because that information didn’t exist yet when training happened.

Interview signal: “How do you decide what to store in parametric memory versus external memory?” — the best answers recognize this isn’t really a design choice you make per-fact; it’s a trade-off you navigate: parametric memory is fast, cheap, and always available, but frozen and unauditable, while external memory (retrieval, semantic) is fresh, updatable, and traceable, but costs an extra lookup step.

Workflow sketch:

flowchart TD
    U["User / Agent<br/>question / prompt"] --> LLM["LLM (Model)"]
    subgraph PM["Parametric Memory (stored in weights)"]
        L1[Language Knowledge]
        L2[Reasoning Patterns]
        L3[World Knowledge]
        L4[Facts & Concepts]
    end
    LLM --> PM
    PM --> R["Response<br/>Answer, Reasoning, Generated Content"]

Parametric memory is the one type on this list that isn’t really “designed” by the application developer at all — it’s a property of the underlying model itself, set during training by the model provider. But understanding it matters enormously for system design, because it defines the baseline your other six memory systems are compensating for: anything the model doesn’t already “just know,” some other memory system needs to supply.

7. Prospective Memory

What it is: A record of future intentions, scheduled goals, and follow-up actions the agent has committed to but not yet executed — remembering what you plan to do next, not just what already happened.

Why it’s used: It offloads “what do I need to do next” to a persistent store, so long-horizon agents don’t lose the thread when context resets or time passes between steps.

Where it’s used: Planning agents, autonomous task runners, and multi-agent pipelines — typically stored as task queues, goal stacks, or reminder logs, often alongside episodic memory.

Example: An agent completes step 2 of a 5-step task, then gets interrupted. It logs “resume at step 3 with these inputs.” When it restarts — potentially much later, or in an entirely new session — it reads back that intent and continues exactly where it left off, instead of starting over or losing track of the plan.

Interview signal: “How do you ensure your agent doesn’t forget what it planned to do next?” — the expected answer is essentially the definition of this pattern: persist intents explicitly, set concrete triggers (time-based, event-based, or agent-based) for when to act on them, and resume execution at the right moment.

Workflow sketch:

flowchart TD
    A["Agent<br/>Current Task in Progress"] --> PMS["Prospective Memory Store"]
    subgraph PMS2[" "]
        PT["Pending Tasks<br/>(Task Queue)"]
        GP["Goals/Plans<br/>(Goal Stack)"]
        RM["Reminders<br/>(Time/Triggers)"]
    end
    PMS --> PMS2
    PMS2 --> TR["Trigger / Time / Event"]
    TR --> RE["Retrieve intent & resume execution"]
    RE --> A

Prospective memory is the newest and least widely implemented of the seven — it matters most for agents that operate over long horizons (hours, days, or weeks) rather than a single conversational session. Without it, an agent that gets interrupted mid-task effectively has amnesia about its own plans, which is a serious problem for anything doing genuinely autonomous, multi-step work.

A Few More Worth Knowing

The seven above cover the core taxonomy well, but a handful of related patterns show up often enough in real systems that they’re worth knowing too.

Entity Memory — a narrower, more structured cousin of semantic memory that tracks specific facts about specific named entities mentioned in a conversation (a person, a company, a product) rather than general facts and preferences. It’s common in customer-facing agents that need to track “who are we talking about right now” across a long conversation, updating a structured profile as new details emerge.

Buffer / Summary Memory — a practical technique for extending working memory’s effective reach without a separate storage system: instead of keeping the full raw conversation history in context, the system periodically summarizes older turns into a condensed form, keeping recent messages verbatim but compressing everything older. This is less a distinct memory type and more a coping strategy for the hard limits of working memory.

Shared / Multi-Agent Memory — as multi-agent systems become more common, a growing pattern is a memory store that multiple agents read from and write to collectively, rather than each agent maintaining fully isolated memory. This raises its own design challenges around consistency (what happens when two agents write conflicting information) and access control (which agents should see which memories).

Graph / Relational Memory — an increasingly popular evolution of semantic memory that stores facts not just as flat embeddings, but as an explicit graph of entities and their relationships (similar in spirit to a knowledge graph). This makes certain kinds of multi-hop reasoning — “who reports to the person who approved this project?” — far more reliable than pure vector similarity search alone, since the relationships are stored explicitly rather than needing to be inferred from proximity in embedding space.

The Practical Takeaway

You don’t need all seven-plus memory systems for every agent, and building them all from day one is usually over-engineering. A sensible progression looks like this:

  • Start with working memory. Every multi-turn agent needs this at minimum — it’s not optional.
  • Add semantic memory once users expect the agent to remember them specifically across separate sessions, not just within one conversation.
  • Add retrieval memory once the agent needs access to a knowledge base too large to fit in context — most RAG systems, in practice.
  • Layer in episodic, procedural, and prospective memory only once your agent needs to genuinely plan ahead, learn from its own past failures, and adapt its behavior over time — this is where things get architecturally complex, and it’s rarely worth the overhead for a simple Q&A bot.
  • Parametric memory isn’t something you build — it’s the baseline capability of whatever model you choose, and it quietly defines how much of the other six systems you actually need.

Knowing the difference between these systems isn’t just academic. It’s the difference between an agent that feels genuinely intelligent over time — remembering you, learning from its mistakes, picking up exactly where it left off — and one that resets to a blank slate every single time you open a new chat.

Cheers,

Sim