The Current LLMs Thread and Memory Problems

Two months into a single chat thread with an AI assistant, I asked for a small edit. A few words. It consumed a quarter of my session allowance.

Nothing was broken. I had simply run into the way the tool works, and it is worth understanding before you decide how to organize your practice around it.

Why a small request can cost a fortune

A chat thread has no memory in the human sense. On every turn, the assistant rereads the whole conversation from the top, then answers. The cost of a message is therefore not the size of your request. It is the size of everything that came before it, plus your request.

My thread had grown for two months. Every question I asked (for example, “fix this comma”) cost a whole bunch of tokens of rereading before the assistant produced a word. The request was small. The context was not, and the request inherited it.

I had heard the folklore: start a new thread from time to time because a chat will degrade and cost more and more tokens as it grows. I had been doing it monthly, asking the assistant for a handoff document and opening a fresh conversation. I decided to test whether it was actually necessary by skipping it once. This confirmed it was necessary.

And then comes the automated proposed solution: move up a plan or buy more tokens.

The human factors reading

Set aside the pricing question for a moment and look at this as an interface.

You have a resource that depletes. It depletes nonlinearly, because the cost of each action grows with accumulated state. And that state is invisible. The chat interface shows no gauge. The command-line version shows a counter climbing, without a ceiling, so you know you are spending but not how much you have. At one point my request was consuming 160,000 tokens, but I had no idea if it was a lot or not.

In most places where I work, that combination would fail reviews on the first pass. No state visibility. No feedback on consumption. No way for the operator to build an accurate mental model of the system. It reminds me of a similar situation where, in Canada, home internet plans used to have download caps (not only cellphones). I think almost everybody has faced at least once an absurdly high bill because there was no way to know how much was used. With everything being billed at the end of the month, it was a bitter surprise.

The predictable outcome is that folklore begins to build up. Ask for a handoff and start a new thread every month. Use such and such skills. Build prompts in such and such a way. Basically, you try to learn what consumes and what doesn’t. You read forum threads about which features are expensive. Each of these can be real, useful advice, and each of them is the product’s job shifted onto the person using it.

I’m not saying this is a dark pattern, designed to trap anyone. Context is an unsolved engineering constraint, and the people building these tools know it better than I do. But you don’t need intent for an incentive to be misaligned. In this case, opacity favours the seller, and a automated proposed solution of “upgrade” when the actual fix is “start a new thread” tells you the product has not yet taken ownership of the problem.

Workarounds bring their own problems

The folklore then created a second problem. When you rotate threads, each thread accumulates subjects. My marketing thread held inbound content, outbound prospecting, a client authorization question and a tooling discussion, all in one scroll. I tried broader threads to reduce rotation, and they became worse: harder to search, harder to hand off, and the handoff document grew until it had the same problem the thread did.

That is when it became clear the issue was not thread length. It was that I had been using a chat thread as memory, and a chat thread is a terrible memory. It is linear, it is expensive to reread, it mixes subjects, and it belongs to the vendor.

What I built instead

I stopped treating threads as memory. State now lives outside the chat.

Concretely: a vault of plain-text markdown notes, one subject per note, stored in Obsidian on a home server I already run. A small MCP server exposes the vault to the assistant through a tunnel, so any AI surface can read and write it. A short rule file in the vault tells the assistant how to behave: read the master index and the relevant status note at the start of a session, write back facts and decisions at the end, record every edit in a changelog, never write chat content, one subject per note, and put anything uncertain in an inbox folder with a proposed location.

Threads became sessions. I open one for a piece of work, the assistant reads a few thousand tokens of vaults instead of a hundred and sixty thousand tokens of history, we do the work, it writes back what changed, and the thread can die. The next session starts from the vault, not from a handoff.

The side benefits turned out to matter as much as the token savings. The vault is versioned, so I can see what changed and when. It is backed up. It is portable, so I can switch tools or point two tools at the same work. And it is mine, in a format that will still open in twenty years.

What it costs

It really isn’t a perfect solution, especially given how the tools are currently designed.

This is not a consumer solution. It took a home server, a Cloudflare tunnel, an afternoon with a coding agent to wire the server, and a willingness to maintain infrastructure. Most people should not do this, and the fact that they would have to do this is the point of the first half of this article.

It runs on discipline. A vault that is not written back to decays into a stale mirror of the world, which is worse than no vault, because you trust it. The rules file helps. The assistant is good at following it. But the write-back step is a habit, and habits fail.

Threads began to multiply much faster, and I’m essentially not using the chat tool as it was intended. The context each thread builds duplicates the vault and the many short threads created to sidestep the problem leaves behind a plethora of unusable chats. The chat history becomes an unused tool that is now just a distraction.

What I would want from the product

First, predictability. It doesn’t need to be perfect, but we need an idea of what will be consumed. This shouldn’t be hard to build as AI tools already know when to compact, and they know how it affects token consumption. We don’t need a usage dashboard buried in settings. We need a visible indicator of what this thread costs to continue, right where the message is typed. That single element would have replaced the folklore for me.

Second, a native separation between session and state. The pieces exist, projects and memory features and so on, but they are still vendor-owned and still opaque. The pattern I built by hand is where these tools are heading. I would just like it to arrive in a form that doesn’t require a server in my basement.

Until then, the vault holds. It put the gauge back where I can see it, and it stopped my tools from forgetting what matters.