Stateless by Design: What "Give the Model Memory" Actually Means

By Adrian Vance · What "give the model memory" actually resolves to, and why fine-tuning is the wrong tool.

A request that lands on every applied AI team eventually: "can it remember what we talked about last time?"

The request is reasonable. The mental model behind it usually is not, and the gap causes teams to reach for expensive tools that do not solve the problem. Fine-tuning gets proposed. Sometimes it gets funded.

Worth being precise about what is happening, because the precision points directly at the cheap solution.

The model does not remember anything, ever

A language model is a pure function. Tokens in, tokens out. It holds no state between calls. The weights are frozen at inference time and identical for every request the service handles.

What feels like memory in a chat interface is the client resending the entire prior conversation on every turn. Turn twelve is not the model recalling turn three. It is turn three being pasted back in, along with everything else, and the whole thing re-read from scratch. Take that transcript away and the model is exactly as ignorant as it was on the first request.

This is not a limitation waiting to be fixed. It is the architecture, and it is why the same service can handle millions of unrelated conversations concurrently without them contaminating each other.

Once that is clear, "give it memory" resolves into its real form: something outside the model must decide what to put in the context window on this call. Every memory feature that has ever shipped is a retrieval and assembly problem wearing a cognitive costume.

Why fine-tuning is the wrong tool for this

Fine-tuning updates weights. It is the right choice for teaching a model a format, a style, or a narrow task shape it handles poorly, and you need a training corpus in the thousands of examples to move the needle.

It is the wrong choice for facts, and especially wrong for facts that change.

Facts encoded in weights cannot be updated without another training run, cannot be deleted on request, cannot be cited, and cannot be audited. If a customer's account tier is baked into the weights and the customer upgrades, you now have a model that confidently states something false, with no way to correct it short of retraining. Every governance property you want, correction, provenance, deletion, access control, is a property of a database and not a property of a weight matrix.

The rule that holds up in practice: weights for behavior, context for knowledge. If it is something the model should be able to do, fine-tuning is on the table. If it is something the model should know, it belongs in a store you can query, version, and revoke.

The three tiers that actually make up "memory"

Working systems almost always decompose into the same three layers, with very different lifecycles.

Durable instructions. The stable definition of the task: role, output format, domain vocabulary, rules of engagement. This changes rarely and should be identical across every invocation of a given task type. It is code, and it deserves the treatment code gets: a file in version control, a review before it changes, a release note when it does.

The failure mode here is the instruction block living in individual users' heads and getting retyped from memory at the start of each session, slightly differently every time. That is not a memory problem, that is an unmanaged artifact. One document per recurring task type, owned by the team that owns the task, fixes it, and costs nothing.

Structured state. The facts: who the user is, what the entities are, what happened, what the current status is. This belongs in a real store, relational or otherwise, queried at request time and injected as context. The key design decision is retrieval, not storage. You cannot inject everything, so something must select what is relevant to this request.

Teams reach for vector search by default here. It is a good fit for genuinely unstructured recall over documents. It is a poor fit when the relevant facts are structured and the relationships are known, where a query against the actual data model is more precise, cheaper, and far easier to debug. If your system already has a data model with entities and relationships, retrieve against it directly. Embedding a row and hoping cosine similarity finds it again is a downgrade from a WHERE clause.

Episodic summary. The compressed record of prior interactions. Full transcripts grow without bound and quickly cost more than they contribute, because most of what was said does not matter later. The workable pattern is to summarize at the end of a session, extract the durable facts into the structured store, and keep a short rolling narrative for continuity.

The design question worth being deliberate about is what gets promoted from episodic to structured. Something inferred once in conversation, then written to the fact store and read back forever, will look like ground truth to every future call. Mark provenance on write: was this stated by the user, or inferred by the system? Without that flag, inference hardens into fact silently, and nobody can tell later which is which.

Assembly is where the engineering is

With three tiers defined, each request assembles a context window: durable instructions, plus the retrieved facts, plus the recent narrative, plus the current input. This assembly step is the actual system. It has a token budget, an ordering, a truncation policy, and failure modes, and it deserves tests.

Two things reliably go wrong.

Budget contention. All three tiers want space and the window is fixed. Without an explicit budget per tier, retrieval bloat quietly pushes out the instructions, and the model starts ignoring rules that are technically still present but buried behind four thousand tokens of retrieved noise. Allocate explicitly, and log what actually got assembled when something goes wrong.

Untested retrieval. If the retrieval step returns the wrong facts, the model will use them, confidently. Retrieval quality is measurable independently of generation quality, and it should be measured that way. A set of representative requests with known correct retrievals will find failures that end-to-end output review never surfaces, because a fluent answer built on the wrong record looks fine until someone checks it.

Where this leaves the original request

"Can it remember what we talked about last time" is a request for a retrieval system with a storage schema, a promotion policy, and a context budget. None of that requires touching the model.

That is good news, because it means the work is ordinary engineering with ordinary properties. You can version it, test it, correct it, and delete from it. You can explain to an auditor where a fact came from. You can fix a bad memory in one row rather than one training run.

The systems that feel like they remember are not the ones with the best models. They are the ones that made deliberate choices about what to write down, what to retrieve, and what to leave out.

Adrian Vance writes about applied AI, evaluation, and decision systems for technical audiences.