Chatbot memory for long conversations: summaries, facts and retrieval

A conversation with an AI character starts well. Three weeks and six hundred turns later, the user mentions the name they gave their dog on the first evening, and the character has no idea what they mean. Or worse, it remembers a different name. The first version of most chat products sends the whole history with every request, and that works until it does not. We built the summarisation for the EroPlay.ai MVP as technology partner, by hand, so that a character stayed consistent long past what a model can read at once. This is how we would build that memory now.

Why the full history stops fitting

The context limit is the obvious reason, and it is rarely the first one to bite. Cost comes first. Every turn resends everything before it, so the total input over a conversation grows with the square of its length. Say, as an example, that one exchange of user message and reply is 200 tokens. Turn 500 sends about 100,000 tokens of history, and the whole conversation has sent about 25 million input tokens. Cap the history at 8,000 tokens and the same 500 turns send under 4 million.

Latency follows the same curve. The model reads every input token before it writes the first output token, so the wait before a reply starts grows as the history does.

The third reason is that a longer context is not a better one. The Lost in the Middle paper found that models use information at the start and end of their input best and do markedly worse when it sits in the middle, including models built for long inputs. Chroma’s context rot report ran the LongMemEval conversational memory benchmark two ways: with a focused prompt of about 300 tokens holding only the relevant part, and with the full history of about 113,000 tokens. Every model it tested scored significantly higher on the focused version. A bigger window lets you send more. It does not make the model read it well.

What goes into each request

What replaces the full history is a request assembled from parts, each with its own rules: the character or assistant definition, a rolling summary, pinned facts, the recent turns verbatim, passages retrieved from older turns, then the new message.

The order matters for two reasons. Models attend best to the ends of their input, so the persona sits at the top and the recent turns at the bottom, next to the new message. And prompt caching, covered below, rewards putting the parts that change least at the front.

Keep the recent turns verbatim

The last stretch of a conversation is the part a summary handles worst. It carries the tone, the exact words the user chose, the joke that is still running and the question that has not been answered yet. Summarise it and the character starts to sound like someone who was briefed about the conversation rather than someone who was in it.

We size this window in tokens, not messages, because messages vary from one word to a page. We cut only at exchange boundaries, so a user message never arrives without the reply it got, and a tool call never arrives without its result. The window has a floor as well as a ceiling: even if one enormous message fills the budget, the previous exchange stays.

A rolling summary of the chat history

When the recent window overflows, the oldest turns are evicted and folded into the summary. One model call takes the current summary and the evicted turns and writes the new summary. That is the whole mechanism, and most of the quality comes from what the summary is asked to contain.

Part	What it holds	How it changes
Character definition	the persona and the standing instructions	written once, sent whole, never summarised
Recent turns, verbatim	the tone, the user’s exact words, the running joke, the unanswered question	the oldest are folded into the summary
Rolling summary	what has happened, in order and briefly; open threads and anything promised; the current state of the scene or task; how the user likes to be spoken to	rewritten at each compaction
Facts store	names, preferences, boundaries, dates, and what the character invents about itself	extracted after each exchange; the newest value wins
Retrieved passages	older exchanges that bear on this message, with turn numbers and dates	searched when a message refers back to something

A summary written as a story reads well and loses the details people come back for. We ask for the fixed sections in the table instead. Each item carries the turn numbers it came from, so a later step can fetch the source when the summary is too vague to answer.

The failure mode is drift. A summary of a summary of a summary loses something each time, and it can amplify a mistake the first pass made. The summary has a length budget, and when it reaches it, it is compacted from its own sections with the instruction to drop the least recent narrative first and never the open threads. Durable facts are not left to the summary at all. In the LoCoMo study of very long conversations, giving the model session summaries did not significantly improve its answers, which the authors put down to information lost in turning dialogue into summaries. Short stored observations, much like the facts store below, did better.

It is tempting to give the summariser the cheapest model, since nobody reads its output directly. That is where a cheap model costs the most. A dropped fact shows up thirty turns later as a wrong answer, and nothing connects the two. We give summarisation a mid-tier model and an evaluation set of its own, and route the cheap model to work whose mistakes are visible at once. The routing decision itself is the subject of cheap and frontier models side by side.

Long-term memory as a store of facts

Some things should never depend on a summariser remembering them: a name, a preference, a boundary the user set, a date that matters to them. These go into a facts store, extracted after each exchange by a model call that returns structured records: a key, a value, the turn it came from and when.

Facts get updated. A user who moved city in month two should not be asked about their old flat in month four. The extractor is given the existing facts for the same keys and asked whether the new message confirms, changes or contradicts them. The newest value wins in the prompt, and the old one stays in the store with its date, because “you used to live in Leeds” is sometimes the right thing to know. Extraction is a structured output problem with all the usual failure modes, and the schema discipline in reliable structured output from LLMs applies here unchanged.

While the facts number in the dozens, all of them go into every request. Once they number in the hundreds, they are retrieved by relevance like anything else. In a consumer product, the user should be able to see this store and delete from it. What the product remembers about a person is their data.

Retrieving older turns

The summary says the user and the character named a ship together around turn 140. The facts store may not have caught it. The user asks what the ship was called. This is where retrieval over the stored history earns its place: embed every exchange as it is stored, search on the new message, and return the best matches verbatim, with their turn numbers and dates, marked as earlier conversation.

Retrieval brings its own errors. It finds a passage that was later contradicted, or one that is about a different ship. Dates and turn numbers let the model weigh it, and when retrieval and the facts store disagree, the facts store wins. Not every message needs a search either. A cheap check on whether the message refers back to something, or a search whose results are dropped below a similarity score, keeps old passages from crowding out the recent ones. The wider choice between retrieval and training a model on your data is covered in RAG versus fine-tuning for company data.

Keeping a character consistent

A persona stays consistent when the parts of the prompt that define it never pass through a summariser. The character definition is written once and sent with every request, whole. Everything the character invents about itself during the conversation, a sister, a home town, a scar with a story, is a fact about the character, and goes into the facts store next to the facts about the user. Left to the summary, those inventions are among the first details a compaction drops, and the character later contradicts itself.

The EroPlay.ai MVP also orchestrated several models within one conversation. When more than one model can write a turn, memory is what lets them speak as one character: each receives the same definition, the same summary and the same facts, assembled the same way.

When to summarise

Summarising on every turn is wasteful and, as the next section shows, defeats caching. We use two thresholds. When the recent window reaches a high-water mark, the oldest turns are folded into the summary until the window is back at a low-water mark. As an example, compact at 6,000 tokens down to 3,000, and with exchanges of 200 tokens the summariser runs about once every fifteen exchanges.

The compaction runs after the reply has been sent, in a background job, so the user never waits for it. The next request uses the new summary if it is ready and the old summary with a slightly longer window if it is not. Memory has a version number, and a job that finishes against a state that has moved on, because the user sent two messages quickly, is discarded and rerun rather than applied. If the window gets close to the model’s hard limit while a job is still running, the request waits for the summary or truncates the oldest turns. Both are rare enough to log and look at.

Prompt caching and the order of the parts

Providers cache a processed prompt prefix, so repeating it exactly is cheaper and faster. On the Claude API, the cache covers tools, then system, then messages, and at the time of writing (October 2026) a cache read costs a tenth of the normal input price or less, depending on the model, a write to the default five-minute cache costs 1.25 times, and an entry lives for five minutes from its last use unless you pay more for an hour. OpenAI’s prompt caching is automatic, requires the entire rendered prefix to match, and keeps entries for 30 minutes after their last use on recent models.

Ordering the parts by how often they change, persona first, then the summary, then the facts, then the recent turns, means most requests reuse almost everything. Between compactions, the recent turns only grow at the end, so the prefix before them stays intact. A compaction changes the summary once and invalidates what follows it once, and the high-water and low-water marks make that happen every fifteen exchanges rather than every turn. Facts are extracted more often than the summary is rewritten, so new facts are merged into the facts block at the next compaction; until then, the turn that stated them is still in the recent window. Retrieved passages change on every request, so they go last, next to the new message. Put them, or the current time, near the top and the cache misses on every turn.

The other interaction is time. A user who takes ten minutes to reply has outlived a five-minute cache. Whether the hour-long cache is worth its higher write price is a number to read off real traffic, not guess.

Testing that summaries do not drop facts

A memory system fails quietly. It answers fluently and wrongly, many turns after the cause. The only defence is a test that knows the answers.

We build long test conversations with facts planted at known turns: a name at turn 12, a preference at turn 90, a change to that preference at turn 300. The test runs the real pipeline to the end, with the real summariser, extractor and thresholds, then asks probe questions and checks the answers. It also checks the intermediate state directly: after each compaction, is every planted fact still in the summary or the store? A fact that vanishes at compaction four points at the compaction prompt. A fact answered with its old value points at the extractor. Every change to a memory prompt or model runs through these conversations before it ships, like any other evaluation, described in building an evaluation set from past cases.

In production, the cheapest signal is the user correcting the product. Messages like “I already told you” or “no, her name is” can be caught with a small classifier and counted per thousand conversations. When that number moves after a release, the memory changed, whatever else the release notes say.

If you are building a chat product where conversations run for weeks and a character has to stay itself, the custom AI apps page describes how we take on that work, and you can write to us with what your conversations look like.