Your agent forgets things it should remember.
It remembers things it should have forgotten. It mixes context from different users. It starts every session cold even when someone has used your product for months.
Most teams reach for a bigger context window.
The actual fix is to design memory on purpose across four layers that do different jobs.
Why "just use a bigger context window" fails
A larger context window does not solve the memory problem. It delays it.
Stuffing everything into the context window works until the window fills up, until retrieval gets noisy because there is too much to attend to, until costs spike at scale, until the model loses track of something that appeared three thousand tokens ago.
It also lumps together four problems that need separate answers.
The four layers
Layer 1: Working memory
This is your context window: what the agent can reason over right now. Fast, flexible, bounded. Every agent has this layer. Most treat it as the only one.
The design question is not how big it should be. It is what belongs here and what should be evicted.
Working memory should hold what the agent needs for the current task: the immediate goal, the last few exchanges, the current tool results. Everything older, larger, or less relevant belongs somewhere else.
A working memory that holds everything eventually holds nothing useful. The agent loses the thread. Responses get less coherent. Retrieval gets noisy. You add more tokens to compensate. The problem compounds.
Layer 2: Episodic memory
What happened in this session and in recent sessions. Not raw conversation history. Structured, summarised state.
What did the user ask for? What did the agent try? What was the outcome? What does the user seem to prefer? What should the agent remember next time?
This is what makes an agent feel like it knows you after the first interaction. Without it, every session is a first meeting. Users repeat context they already gave. The experience degrades. They stop trusting the system.
Episodic memory lives outside the context window, in a database, structured and queryable. It gets retrieved selectively at session start, not dumped wholesale into the prompt.
The design questions: at what granularity do you store episodes? When does an episode expire, get archived, or get compressed into a longer-term user profile?
Layer 3: Semantic memory
Your retrieved knowledge. Documents, product information, policies, anything the agent needs to answer questions accurately. This is where RAG lives.
Most teams build this layer. Fewer connect it properly to the others.
The planner needs to know when to query semantic memory, what to query it with, and how much retrieved context to pull into working memory for this specific task. That routing decision, which most teams leave implicit or hardcode, is where most retrieval quality problems actually start.
Semantic memory is not a static store. It should update as the world changes and as your product evolves. A system that retrieves from a knowledge base three months stale is not reliable.
Layer 4: Procedural memory
How to do things: successful tool call patterns, learned strategies for similar tasks, user preferences for how things get done.
This is the rarest layer and the one that makes an agentic system feel smarter over time. Instead of rediscovering the right approach to a recurring task, the system retrieves what worked before and starts there.
Most teams do not build this layer at all. It is harder. You have to decide what counts as a successful strategy, how to store it in a retrievable form, and when to apply past strategies versus try something new.
The teams that build it have agents that improve with use instead of staying flat.
The connection between layers
The layers are not independent. They work together through a memory manager whose job is deciding what goes where and what comes back when.
At task start: working memory gets populated from episodic context (what the user has done before), relevant semantic memory (what the agent needs to know), and procedural context (how similar tasks were handled).
During a task: working memory is maintained actively. Old context is evicted. New results are added. The current goal stays prominent.
At task end: significant events are written to episodic memory. Successful strategies are evaluated for procedural storage.
This cycle, populate, maintain, persist, is what makes memory feel coherent instead of accidental.
How to build this incrementally
You do not need all four layers on day one.
Start with working memory and episodic memory. Together they solve most "my agent feels stateless" complaints. Users feel remembered. Sessions feel connected.
Add semantic memory when you have a knowledge domain worth retrieving from. This is when RAG enters the system, designed explicitly as a memory layer, not bolted on as a feature.
Add procedural memory when you have enough usage data to spot recurring task patterns. That is a later optimisation, not an early requirement.
At every stage: each layer should have a single owner in your codebase. One component reads from and writes to episodic memory. One manages working memory. One handles retrieval from semantic storage.
Mixing these responsibilities, like mixing planning and execution in orchestration, gives you a memory system that is hard to debug and hard to improve without breaking something unintended.
The question worth asking now
Look at your current agentic system and ask: if a user comes back tomorrow and picks up where they left off, what does the agent actually know?
The answer tells you which layer you are missing.