An LLM call by itself has no memory. Send a message, get a response, send another message — each call starts fresh. For a basic chatbot, you solve this by appending the conversation history to each call. For a production agent that needs to handle complex, multi-session workflows, that's nowhere near enough.
Production agents use four distinct types of memory, and the architecture decisions for each are genuinely different. Most implementations get two of them right and miss the other two entirely.
Why stateless LLM calls don't make agents
The fundamental constraint: an LLM only knows what's in its context window during a specific call. It doesn't remember yesterday's conversation. It doesn't know what happened in a previous session. It can't learn from repeated interactions unless you design a system to feed that learning back in.
This is not a limitation of the technology — it's the expected behavior of a stateless function. The memory architecture around the LLM is what transforms stateless inference into a coherent, context-aware agent.
Working memory: the context window
Working memory is everything in the current context window: the system prompt, conversation history, recent tool results, and any retrieved content.
What fits
Modern context windows are large — 128K to 200K tokens is common, with some models reaching 1M. A 128K-token context holds roughly 90,000 words, or about 300 pages of text. In practice, you don't want to fill the context window completely:
- Quality can degrade on some models when context is very full
- You're paying for every token, every call
- The model doesn't weight all context equally — information in the middle of a very long context is often less reliably used than information at the beginning or end
What to keep and what to evict
Working memory should hold: the system prompt, the last 10–15 turns of conversation, and the most recent tool results. For most agent tasks, this is sufficient context.
As conversations grow longer, implement a summarization step: take the oldest 5–10 turns, compress them into a 100–200 token summary, replace the raw turns with the summary. The model retains the essential information at a fraction of the token cost.
The "lost in the middle" problem
Research consistently shows that LLMs are less reliable at using information positioned in the middle of long contexts compared to information at the beginning or end. For agents, this means: put critical instructions in the system prompt (beginning) and the most recent tool result directly before the current turn (end). Don't bury the most important context in the middle of a 200-turn history.
Semantic memory: the retrieval layer
Semantic memory is your knowledge base — product documentation, policy documents, support articles, internal wikis. It's too large to fit in the context window, so you retrieve the relevant chunks at query time.
This is the RAG pattern. The agent has a search_knowledge_base tool that it calls when it needs information from the knowledge base. The retrieval step returns the relevant chunks, which get injected into working memory for the current call.
Implementation patterns
Flat retrieval: embed all chunks with a single embedding model, store in a vector database, search by cosine similarity. Works well for homogeneous content (all FAQs, all support articles).
Hybrid retrieval: combine dense (embedding) search with sparse (BM25 keyword) search and rerank the combined results. Better for heterogeneous content where some queries are better served by keyword match.
Hierarchical retrieval: search at the sentence level, return the parent section. More precise matching, more complete context. See our RAG chunking guide for implementation details.
The staleness problem
Semantic memory reflects the knowledge base at index time. When your documentation changes, the embeddings need to be updated. Build a re-indexing pipeline from the start — not as an afterthought. For most business use cases, nightly re-indexing of changed documents is sufficient.
Episodic memory: conversation history across sessions
Episodic memory is what the agent remembers about past interactions with a specific user. A customer service agent that doesn't know a user called last week and resolved a billing issue will ask redundant questions and frustrate users. A sales agent that doesn't remember a prospect's specific objections will repeat them.
The storage pattern
At the end of each conversation, run a summarization step:
summary_prompt = f"""
Summarize this conversation in 3-5 sentences. Focus on:
- The user's name and any stated preferences
- The main issues raised or questions asked
- Resolutions reached or open items
- Any follow-up commitments made
Conversation: {conversation_transcript}
"""
summary = llm.complete(summary_prompt)
db.write(user_id=user_id, timestamp=now(), summary=summary)
At the start of the next session, retrieve the N most recent (or most relevant) summaries and inject them into the system prompt:
[PREVIOUS INTERACTIONS]
2026-02-20: User called about a billing discrepancy on invoice #4421. Issue was resolved — duplicate charge was reversed. User was satisfied with resolution.
2026-01-15: User asked about upgrading their plan. Was interested in the Professional tier but wanted to see the annual pricing before deciding.
How much history to retrieve
More is not always better. A system prompt loaded with 20 sessions of history becomes noisy. Start with the 3–5 most recent interactions. For agents where recency matters less than relevance (a long-term account manager agent), use embedding similarity to retrieve the most relevant past interactions for the current topic.
Procedural memory: cached reasoning patterns
Procedural memory is the most underused type. It's the set of task-specific reasoning patterns, few-shot examples, and decision trees that the agent uses to solve known problem types.
Unlike the other three memory types, procedural memory is static — it lives in the system prompt and doesn't change between calls. But it's powerful because it encodes institutional knowledge that would otherwise require the model to derive from scratch every time.
When procedural memory is worth building
- Narrow, high-volume tasks where you want consistent, predictable behavior: classification tasks, extraction templates, specific reasoning chains
- Tasks where few-shot examples significantly outperform zero-shot: most structured output tasks benefit from 2–5 concrete examples in the prompt
- Agents with specific domain expertise requirements: a medical coding agent needs domain-specific reasoning patterns; a general-purpose agent does not
The maintenance burden
Procedural memory requires maintenance. When the right reasoning pattern changes — your product changes, regulations change, your process changes — the few-shot examples and decision trees need to be updated. Version your system prompts like you version code.
Which combination to use
The right memory architecture depends on your agent type:
| Agent type | Working | Semantic | Episodic | Procedural | |---|---|---|---|---| | Simple Q and A chatbot | Yes | If knowledge base exists | No | Optional | | Customer support agent | Yes | Yes | Yes | Yes | | Internal workflow agent | Yes | Yes | No | Yes | | Sales development agent | Yes | Yes | Yes | Yes | | Document processing agent | Yes | No | No | Yes |
Start with working memory — you always need it. Add semantic memory if there's a knowledge base. Add episodic memory if users return for multiple sessions and context continuity matters. Add procedural memory when you identify task-specific patterns worth encoding.
The AI agent development projects that struggle most aren't missing a better LLM — they're missing a coherent memory architecture. An agent that can't remember context from last week isn't a production agent; it's a prototype that feels like it should work but doesn't in practice.