07 Sep 2024 AI Engineering 5 min read

Agent Memory Beyond the Context Window

Context windows are working memory, not memory. This article separates the two concepts and reviews the three memory mechanisms in production use for LLM agents in 2024 — conversation summaries, vector memories and structured state — with their failure modes, why deliberate forgetting improves correctness, and which boundaries the GDPR sets for persistent agent memory.

A Context Window Is Not Memory

An LLM is stateless. Every API call starts from zero. What looks like memory in a chat interface is replay: the client resends the previous turns with each request. The context window — 128,000 tokens for GPT-4o, 200,000 for Claude 3.5 Sonnet, up to 2 million for Gemini 1.5 Pro since 27 June 2024 — is working memory at best. After the call it is empty.

For agents this distinction is structural. An agent that operates for weeks, across sessions and tasks, accumulates more history than any window holds — and it needs specific facts back at unpredictable moments. Memory is therefore an engineering problem outside the model: what to store, how to retrieve it, when to delete it. This article covers the three mechanisms in production use in 2024.

A task arrives — the agent plans its next step. 1/4

Why Larger Windows Do Not Solve It

Three arguments against "just use a bigger window". First, cost: input tokens are billed on every call, so a history replayed in full makes every turn more expensive than the last. Second, latency: prefill time grows with prompt length. Third, recall quality: Liu et al. showed in July 2023 ("Lost in the Middle", arXiv 2307.03172) that models use information from the middle of long contexts significantly worse than information at the beginning or end.

Even a perfect window would not persist anything. The window is an argument to a function call, not a store. When the session ends, its content is gone. Persistence needs a write path, a retrieval path and a deletion path — and those are system design questions, not model questions.

Conversation Summaries

The oldest mechanism is rolling summarization. When the transcript approaches the token budget, the model compresses older turns into a summary; the summary replaces those turns in the prompt. LangChain ships this as ConversationSummaryMemory; most agent frameworks have an equivalent. Cost stays bounded. The prompt stays short.

Summarization is lossy by design, and the loss is uncontrolled. The model decides at compression time what survives, without knowing what will be asked next week. Precise values — an order number, a version constraint mentioned once — are the first casualties. Summaries preserve gist and tone. They do not preserve facts reliably, and they are not a system of record.

Vector Memories

Vector memory applies retrieval-augmented generation to the agent's own history. Turns or extracted statements are embedded and written to a vector index; at query time the top-k nearest entries are injected into the prompt. Storage scales to years of history. Retrieval cost stays constant regardless of how much has accumulated.

The limits are the limits of embedding similarity. Semantic nearness is not relevance: "customer cancelled the contract" and "customer extended the contract" embed close together. Time is not modeled: a preference from January and its revision from June both match the same query, and the index does not know which one is current. Vector memory retrieves. It does not reason.

Structured State

The third mechanism stores facts as explicit, typed state: a profile, a task list, key-value pairs the agent reads and writes through tool calls. MemGPT (Packer et al., October 2023) formalized this with an operating-system analogy: a small core memory pinned in the context, larger stores paged in on demand, the LLM issuing the memory operations itself via function calls.

Products followed. OpenAI announced memory for ChatGPT on 13 February 2024 and extended it to Free, Plus, Team and Enterprise users on 5 September 2024 — as visible, editable, deletable entries. That visibility is the point: structured state can be audited. The price is schema design up front and extraction logic that decides what becomes a fact.

Mechanism	Write path	Read path	Weakest when
Conversation summary	Model compresses older turns	Summary prepended to every prompt	Specific details are needed later
Vector memory	Embed and upsert statements	Top-k similarity search at query time	Recency or negation decides relevance
Structured state	Extraction into a defined schema	Direct lookup or pinned in context	Facts do not fit the schema

Forgetting as a Feature

A memory that only grows degrades. Stale facts crowd out current ones, retrieval precision drops, contradictions accumulate. Park et al. showed the alternative in April 2023: the "Generative Agents" memory stream scores every entry by recency, importance and relevance, with recency decaying exponentially. Forgetting was load-bearing in that design, not an afterthought.

In practice this means: time-to-live on episodic entries, decay scores that demote what is never retrieved, and supersede-on-write rules so a new fact replaces its predecessor instead of coexisting with it. An agent that remembers a cancelled plan keeps acting on it. Deletion is not data loss here. It is state hygiene.

Privacy Boundaries

Persistent memory turns an agent into a personal data store, and in the EU that has defined consequences. Stored user facts are personal data under the GDPR: Article 15 grants access, Article 17 erasure. "Delete my data" must reach summaries, vector entries and structured state alike — including derived facts the user never wrote verbatim.

Architecture follows from this. Memory must be partitioned per user and per purpose, never shared across tenants, and erasable by key. A memory the user cannot inspect should not exist. We treat inspectable and erasable memory as a hard requirement in enterprise agents, not as a feature. OpenAI's memory controls and temporary chats point the same way.

Outlook From September 2024

We expect the three mechanisms to converge into managed memory layers: hierarchical stores in which summaries, vectors and structured facts are tiers of one system, with MemGPT-style paging logic moving into frameworks and platform APIs. Model vendors will offer memory as an API primitive, not only as a product feature in their chat frontends.

Two predictions for the next twelve months. First, memory quality becomes a benchmark category next to reasoning; today it is barely measured. Second, forgetting becomes a compliance feature that enterprise buyers ask for by name. Context windows will keep growing, and none of this becomes obsolete. A window is where an agent thinks. Memory is what it keeps.