What Do Memory Benchmarks Actually Measure? (Hint: Not Storage)
If you're building agents, there's a decent chance your memory system is a markdown file the agent appends to. No judgment — ours started there too, and for one agent with a small working set it's honestly fine. This post is about what happens when you try to measure whether a memory system is any good, because the answer turned out to be more interesting than we expected, and it changed how we think about the whole problem.
The short version: the best long-horizon memory benchmarks we have barely test storage at all. Almost every hard question in them tests a decision — what to keep, what supersedes what, what to say when you shouldn't trust your own notes. The benchmarks quietly agree with a thesis we've been arguing across this series: memory is a job, not a dump. Below, we map what the benchmarks actually probe, share our numbers and how to reproduce them, and end with a gap we think this community is exactly the right place to close.
The pattern, and why it can't be measured into health
The append-only file (or its cousin, the append-only store) has one behavior: it grows. Nothing merges the four near-duplicate notes about the same quirk. Nothing expires the fact that changed in March. And the agent managing the file is the same agent doing the work — spending task tokens on "is this worth keeping?" and grading its own homework about what matters.
You can feel that this degrades something. The question is what, and by how much, and whether a benchmark can catch it.
Mapping the benchmarks to the actual job
Two long-horizon benchmarks have become the de facto scoreboard for agent memory: LongMemEval (arXiv:2604.22085) and LoCoMo (arXiv:2601.11557). If you haven't read the papers, here's the part that matters: both are built from long, multi-session interaction histories, and both organize their questions by ability — and the abilities are revealing.
Take LongMemEval's categories. Information extraction is the only one that looks like a storage test — can you find a fact you saw once. The rest probe something else entirely:
| Benchmark probes | The question it really asks | The job function it exercises |
|---|---|---|
| Knowledge updates | A fact changed across sessions — do you answer with the current version, or the fossil? | Reconcile (supersession) |
| Temporal reasoning | What was true as of session N? What happened before what? | Lifecycle / provenance |
| Multi-session reasoning | Synthesize across interactions that never mention each other | Consolidate |
| Abstention | The history doesn't support an answer — do you say so, or hallucinate from stale notes? | Curate (noise vs. signal, trust calibration) |
| Retrieval under long horizons | Hundreds of sessions in — can you surface the relevant slice, not the whole pile? | Brief |
LoCoMo's very-long-term conversational setup pushes the same abilities across even longer horizons — with temporal and multi-hop questions doing the heavy lifting.
Look at the right-hand column. Curate, reconcile, consolidate, brief — that's four of the five verbs we'd argue constitute the actual job of memory (the fifth, provision — bootstrapping a new agent with what the fleet already knows — is the one benchmarks don't touch yet; more on that below). The benchmarks were designed empirically, from where systems actually fail. And where systems actually fail is not "the fact wasn't stored." It's "the fact was stored, along with its contradiction, its four duplicates, and its expired predecessor — and nothing in the stack owned sorting that out."
An append-only dump fails knowledge-update questions by construction: both versions of the fact are in there, equally weighted, and the newer-looking one isn't always the truer one. Abstention is arguably crueler — a system that trusts its own uncurated notes will confidently answer questions it should decline. The benchmark is testing whether anyone is doing the librarian's job. Usually, no one is.
Our numbers, and how to check them
Since we're making claims, here's our scoreboard: Memanto — our open-source (MIT) Memory Agent, a dedicated agent whose only function is managing other agents' memories (all five verbs, running as its own loop, on your own model) — currently scores 89.8% on LongMemEval and 87.1% on LoCoMo.
We'd genuinely rather you reproduce than believe: the code is at github.com/moorcheh-ai, both benchmarks are public, and if you get different numbers we want the issue filed. We hold ourselves to the norm this community runs on — a claim without a reproducible eval is a vibe — and we'd encourage you to apply the same standard to every memory system you evaluate, very much including in-house ones. Run your memory.md pattern through LongMemEval's knowledge-update split sometime. It's clarifying.
Equally important is what these scores don't capture, and we want to be straight about it:
- Both benchmarks are essentially single-user, single-timeline. They test one memory serving one interaction history.
- Nothing evaluates fleet-level memory. Multiple agents, shared knowledge, cross-agent contradictions — the regime where two agents' notes disagree for eleven weeks and no component is responsible for noticing. In production, that's where the worst incidents live. In evals, it's a blank space.
- Provisioning is unmeasured. How well does a memory system onboard agent N+1 with what agents 1..N already learned? No benchmark asks.
The open format part (or: what GGUF taught everyone)
One more thing this community understands better than anyone: ecososystems form around open formats, not products. Local AI didn't take off because one runtime was great; it took off because the artifacts were portable and everyone could build against them.
We think agent memory is at the same fork. A memory estate — the accumulated, curated knowledge of a fleet — is going to be one of the most valuable artifacts a team owns, and it should never be trapped in whoever's store it happens to sit in. That's why Memanto's estate lives in the Open Knowledge Format (OKF), with a shipped memanto migrate CLI as the working exit. Judge every memory system, ours included, by whether its state exports in a format something else can read. A component that can't do that isn't a component — it's a dependency.
We've started sketching task designs and would much rather build this in the open than ship it as homework: multi-agent histories with seeded contradictions, supersession chains with provenance, provisioning tasks scored on day-one competence. If you've built evals, run agent fleets, or have production horror stories that should become test cases (anonymized specimens gratefully accepted — the worst ones are the best ones), come argue in the repo: github.com/moorcheh-ai/memanto. Datasets and a leaderboard Space are the obvious shape for it, and this is the natural home.
Memory is a job, not a dump. The benchmarks already know it. Let's build the one that proves it at fleet scale.