Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents
Abstract
WikiTrace links each distilled lesson to its source trajectory and rationale, enabling LLM agents to verify and correct memories before acting, improving performance and reducing memory-induced degradation across agentic benchmarks.
External memory enables LLM agents to improve from experience by distilling trajectories of past tasks into concise lessons for new tasks. However, memory can also lead to memory-induced degradation, where an agent fails on tasks it would solve without memory, often because it follows a lesson that is incorrect, or misapplies or misreads a valid one. This is because lessons are inferred from trajectories labeled only with task success or failure, yet are typically provided without the trajectories they came from, leaving the agent no way to check them. To address this, we introduce WikiTrace, an evidence-linked memory system designed to let agents check each lesson against its original trajectory. Specifically, motivated by the finding that a trajectory often still holds the evidence needed to catch an erroneous lesson, WikiTrace stores each lesson as a concise insight linked to a summary of why its trajectory succeeded or failed and to the raw trajectory itself. Then, on a new task, the agent starts from all insights and, whenever one lacks details needed to act, conflicts with another, or comes from a failed task, descends through its links with terminal tools to fill in missing details or correct the insight before acting. We validate WikiTrace on five agentic benchmarks, showing that it consistently outperforms existing memory approaches and a baseline given the same insights alone, while also reducing memory-induced degradation. Our code is available at https://github.com/yukyeongleee/wikitrace
Community
Don’t just trust your memory.
A bad lesson can make an agent fail a task it could solve without memory. We introduce WikiTrace to reduce this memory-induced degradation. WikiTrace links each lesson to its original run and a diagnosis of the outcome, so agents can check the evidence before acting. Across five benchmarks, WikiTrace scored 3.2 points higher on average than using the same lessons alone.
Separating Gain from Regression is a really useful framing, and the fact that simply rerunning the agent without memory already flips 9.7% of solved tasks puts a lot of memory results in perspective. I also liked that descents after a blocked attempt lead Insights Only by 17.3 points, while routine checks add only 3.4, so the links matter most exactly when something goes wrong. Since every insight currently goes into the first message, I'm curious how you see this scaling: could the compilation layer itself act as the retrieval index once the memory no longer fits in context?
Thanks for the thoughtful comment! I’m glad you found it useful to look at Gain and Regression separately.
My take on the trigger analysis is that a specific question helps the agent know where to look in a long run. Why was my attempt blocked? Can I trust this lesson? Either question gives the agent something concrete to check.
I agree that we’ll need to retrieve insights as memory grows. I’d try two similarity-based retrieval methods: (1) comparing the current task with the task in each insight’s source run and (2) comparing the current task with each insight’s title. In addition, all three layers already have an index with a one-line summary of each document. The agent can scan the insight index and use Bash to open any insight it wants to read.
In an early version, we gave the agent only the indexes and let it decide what to open. It often started acting after reading just the summaries, or skipped memory altogether. It also missed general lessons that could help it understand how the environment works.
So I’d put a few selected insights including both (1) lessons relevant to the current task and (2) general lessons that apply across tasks in the same environment directly in the first message.
Lost a day to this last quarter: a lesson distilled from one successful run got retrieved into an unrelated task and executed like a rule, because nothing in the retrieved text said where it came from. Linking the evidence fixes the wrong lesson. It doesn't fix the right lesson applied to the wrong task — that trajectory looks compatible enough to cite, and the agent cites it while confidently doing the wrong thing.
The baseline I want it measured against is cheaper: store the lesson with its preconditions and make the agent pass an explicit check before applying it. If a citation only helps by making the model more hesitant, that's a prompt effect wearing a memory architecture's clothes.
My test would be boring — replay tasks the agent already solved under memory off, memory on, memory on with evidence, memory on with precondition gating, same seeds, and report regressions as a fraction of available headroom instead of raw accuracy. Until then the stores stay append-only and retrieval hands back the raw trace rather than the summary, since the summary is where the provenance got dropped in the first place.
Get this paper in your agent:
hf papers read 2611.14320 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper