I injected memory 1,112 times and I don't know whether it helped

There's an idea that surfaces whenever someone builds a system with agents: give it memory. Let it learn from what already happened, stop repeating the same mistakes, accumulate experience across sessions.

I built it in May. By June it was being injected 1,112 times in 30 days. And when I went to evaluate it, I found I had no way to answer the most basic question: does it help?

What it actually does

The system stores two kinds of thing. Patterns that worked — ways of solving something that produced good results — and known failures, so they aren't repeated. At the time of measuring there were 127 patterns and 180 failures on record.

When an agent starts a task, the system searches that store for whatever resembles the work about to be done and injects it into the context. The search is by trigram similarity, with a threshold of 0.30 and a cap of three results per query.

It's simple, it's cheap and it runs without errors. That last point turned out to be the problem.

The component that never complains

Everything else in the system makes it onto my review list because it makes noise. The quality gate fails and stops a task. The watchdog kills a process. An agent runs out of turns. The queue jams. Each of those events produces a signal that forces me to look at it.

Memory doesn't. It gets injected, the agent works, the task closes. No error, no alert, nothing to review. It became invisible through good behaviour.

And so a good month went by with over a thousand accumulated injections, without anyone — me — asking whether those injections were improving anything or simply taking up room in every agent's context.

The textbook technical answer, and why I didn't take it

When I finally sat down to look at it, my first reaction was the obvious one: trigram search is primitive, this should move to embeddings.

It's true that it's primitive. It compares character sequences, not meaning. It's the kind of retrieval anyone would replace today with a vector search, and there are libraries to do it in an afternoon.

I didn't, and I think that was the right call. Improving retrieval without an impact metric is betting blind, and paying for the privilege: embeddings carry a per-query cost, so I'd be spending more to improve something I don't even know contributes anything.

The right question wasn't "how do I make retrieval better?" but "is retrieval making any difference?". And I had never asked that second one.

The infrastructure to measure it already existed

Here's what surprised me most: I could measure it, and I didn't know.

Every memory injection records the identifier of the session it happened in. That, which I'd added without thinking about it much, enables a natural experiment: there are sessions with memory injected and sessions without, on the same system, over the same period, doing comparable work.

It's an A/B that's been running for months. All that was missing was looking at it.

The cross-reference to make is against two things also on record: each session's incidents and the quality gate's outcome. The questions are concrete. Do sessions with memory injected repeat fewer reads of the same file? Do they run fewer searches that find nothing? Do they pass on the first attempt more often?

If the answer is yes, memory helps, and only then does investing in better retrieval make sense — with the cost justified by a measured effect.

The question almost nobody asks

But there's a third possible outcome, and it's the one I find most interesting: that memory doesn't move the needle at all.

If that were the case, the correct conclusion isn't to improve it. It's the opposite: raise the similarity threshold, lower the result cap, inject less. Because every injection carries a cost that is certain — it consumes context, and context is finite — against a benefit that would be zero.

That's the uncomfortable part of measuring seriously: one of the possible outcomes is having to dismantle something that took work to build. I suspect the resistance to measuring comes largely from there, not from technical difficulty.

The criterion I took away

A component that never fails isn't the same as a component that helps. They're two different things and they're easy to confuse, because day to day you operate by exception: you look at whatever complains.

Now, before adding any new piece to the system, I try to answer up front: what would the world look like if this piece didn't exist? If I can't answer that, it isn't that the piece is wrong — it's that I'll be stuck with it, unable to ever justify it.

With memory I arrived at that question late. I'm 1,112 injections in and still don't have the answer.

What came next

By July the system was running fast and verifying well. But each task still cost close to an hour of checks before it could close, and that made the backlog crawl.

The problem was no longer what to check. It was a more uncomfortable one: what should block the work, and what shouldn't. That's what the next entry is about.