--- title: Lethe Agent Memory emoji: 🧠 colorFrom: indigo colorTo: gray sdk: docker app_port: 7860 pinned: true license: mit short_description: Agent memory server that learns what to recall tags: - agent-memory - llm-agents - retrieval - go - reranking - embeddings - feedback-learning models: - sentence-transformers/all-MiniLM-L6-v2 --- [![CI](https://github.com/shaikn6/lethe/actions/workflows/ci.yml/badge.svg)](https://github.com/shaikn6/lethe/actions/workflows/ci.yml)   **Source:** [github.com/shaikn6/lethe](https://github.com/shaikn6/lethe) (Go + Python, MIT) # lethe: live demo **Memory server for AI agents that learns which memories help. On a hand-written held-out test of 36 questions the right memory ranks first 86% of the time and in the top 3 every time (small, paraphrase-heavy test; see Limits).** Store a few memories, ask a question, and mark which recalled memories helped. This runs the real [lethe](https://github.com/shaikn6/lethe) Go server inside the Space. Turn embeddings off in the Recall tab to see the difference. Memories are embedded with `sentence-transformers/all-MiniLM-L6-v2` (a small CPU model). With embeddings on, "Who should avoid nuts?" finds the peanut-allergy memory. With them off, the same question returns nothing, because it shares no words with it. How ranking was checked: 72 hand-written questions, each a paraphrase of one stored memory, spread over 12 separate stores. Each store holds 24 memories (the other 23 are distractors, many sharing a word with the question) plus the 7 sample memories. Settings were chosen on 36 questions, then frozen and scored once on the other 36, through this Space's API with exploration off. On those 36 held-out questions, with embeddings on, the right memory ranked first in 31 (86%, 95% bootstrap interval 75-97%) and in the top 3 in all 36; mean reciprocal rank 0.93 (0.86-0.98). The previous settings ranked it first in 6 (17%, 6-31%), top 3 in 31, reciprocal rank 0.52 (0.44-0.60). With embeddings off (keyword ranking only) nothing changed: first in 2 of 36, top 3 in 11. Twelve short exact-term lookups (names, codes such as "HX-55120") stayed first in 12 of 12. What changed: the starting weights are keyword 0.1 and embedding 4 instead of 2 and 2, so one shared word no longer outranks a memory whose meaning answers the question, and the embedding model is all-MiniLM-L6-v2 instead of bge-small-en-v1.5. Almost all of the gain comes from the weights: the old model with the new weights reached 0.92 on the same held-out questions (scored offline, without the sample memories). On the 36 tuning questions the bge query prefix helped bge-base and bge-large but not bge-small, and the larger models (bge-base, bge-large, mxbai-embed-large) were at best level with all-MiniLM-L6-v2 (0.977 vs 0.972) at 2 to 13 times its download size. With exploration on, the right memory moves around within the top 5 from run to run, because exploring deliberately tries different rankings so the server can learn. Exploration is therefore off by default. ## How it works, benchmark and limitations Each recall scores every candidate as a weighted sum of nine features (keyword and semantic match, recency, utility, importance, a time-aware recency, agreement between the two searches, and the keyword and semantic scores of neighbouring memories). Embedding recall scores each memory on its own, so a distractor that shares the question's wording can edge out the real answer. Two parts of the demo address that, and both are measurable in the app rather than asserted: - **Benchmark tab.** The cross-encoder reranker reads each `(question, memory)` pair together and reorders the pool the server returned. On a bundled set of eight weak cases (each answer paired with a higher-overlap distractor, including the quarterly-review case above), run live against a throwaway server, raw embedding recall ranks the answer first in **7 of 8** (MRR 0.938) and the reranker in **8 of 8** (MRR 1.000), with **0** cases made worse. Fixed cases, fixed seed, so the numbers reproduce. - **Learning loop tab.** Starting from the server's default even weights (keyword 2, semantic 2), a batch of simulated feedback that marks the right memory useful (exploration on) shifts the learned weights, most visibly semantic 2 to about 4.2 and utility 0 to about 2.8. Scored on all eight weak cases afterwards, recall@1 rises from **7 of 8 to 8 of 8** (MRR 0.938 to 1.000). The loop runs on its own throwaway server and never touches the shared policy the live demo uses. These two run the real pipeline in-Space. They are separate from the LoCoMo and LongMemEval numbers in the "Measured results" tab, which come from the repository README with simulated feedback and are **not** reproduced here. Limits of this demo: - Each visitor gets a private namespace, but the server keeps data in memory and wipes it on restart. Do not store anything sensitive. It holds at most 2,000 memories. - The check above uses short, hand-written questions that lean toward paraphrase, which favours embeddings. A known weak case: "Who is presenting at the quarterly review?" ranks a memory that only mentions the review just above "Marcus will present the fraud dashboard at the quarterly review." - A few feedback clicks in the Recall tab will not visibly retrain the ranking; the Learning loop tab uses dozens of simulated cycles to make the shift legible. The LoCoMo and LongMemEval numbers in the "Measured results" tab come from the repository README, not from this demo. - The in-Space benchmark above uses eight short weak cases, not a standard dataset; it shows the reranker and the learning loop working on hard examples, not a general accuracy figure. - The server was developed against the LoCoMo and LongMemEval benchmarks; the gains reported there were larger on LoCoMo than on LongMemEval, as the repository README explains. ## Architecture ```mermaid flowchart LR A[add_memory / recall] --> B[Gradio UI] B --> C[lethe_client] C --> D[Go lethed daemon
HTTP :8787] D --> E[Embedding recall
all-MiniLM-L6-v2] E --> F[Cross-encoder rerank
ms-marco-MiniLM-L-6-v2] F --> G[Learned 9-feature ranking policy] G --> H[Ranked memories] H -.feedback.-> G ``` ## Limitations / failure modes - Demo memory pool is small (seeded samples), so default weights already rank most answers first; the learning tab shows the weight shift and recall@1 lift rather than a single dramatic flip. - The live demo's ranking policy is global (not per-namespace); benchmarks run against a throwaway server so they don't perturb it. - Reranker is an off-the-shelf cross-encoder; the product is the server + learned ranking, not the reranker.