Spaces:
Running
Running
|
Download README.md from 9mark9/lethe-agent-memory: direct link, hf CLI and curl.
- Browser
- Download file 6.79 kB
-
https://huggingface.co/spaces/9mark9/lethe-agent-memory/resolve/main/README.md
- Command line
-
hf download hf://spaces/9mark9/lethe-agent-memory/README.md
-
curl -L -o README.md https://huggingface.co/spaces/9mark9/lethe-agent-memory/resolve/main/README.md
6.79 kB
| title: Lethe Agent Memory | |
| emoji: 🧠 | |
| colorFrom: indigo | |
| colorTo: gray | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: true | |
| license: mit | |
| short_description: Agent memory server that learns what to recall | |
| tags: | |
| - agent-memory | |
| - llm-agents | |
| - retrieval | |
| - go | |
| - reranking | |
| - embeddings | |
| - feedback-learning | |
| models: | |
| - sentence-transformers/all-MiniLM-L6-v2 | |
| [](https://github.com/shaikn6/lethe/actions/workflows/ci.yml) **Source:** [github.com/shaikn6/lethe](https://github.com/shaikn6/lethe) (Go + Python, MIT) | |
| # lethe: live demo | |
| **Memory server for AI agents that learns which memories help. On a hand-written held-out test of 36 questions the right memory ranks first 86% of the time and in the top 3 every time (small, paraphrase-heavy test; see Limits).** | |
| Store a few memories, ask a question, and mark which recalled memories helped. | |
| This runs the real [lethe](https://github.com/shaikn6/lethe) Go server inside the Space. | |
| Turn embeddings off in the Recall tab to see the difference. Memories are embedded with | |
| `sentence-transformers/all-MiniLM-L6-v2` (a small CPU model). With embeddings on, "Who should avoid nuts?" finds the | |
| peanut-allergy memory. With them off, the same question returns nothing, because it shares no words with it. | |
| How ranking was checked: 72 hand-written questions, each a paraphrase of one stored memory, spread over 12 | |
| separate stores. Each store holds 24 memories (the other 23 are distractors, many sharing a word with the question) | |
| plus the 7 sample memories. Settings were chosen on 36 questions, then frozen and scored once on the other 36, | |
| through this Space's API with exploration off. On those 36 held-out questions, with embeddings on, the right memory | |
| ranked first in 31 (86%, 95% bootstrap interval 75-97%) and in the top 3 in all 36; mean reciprocal rank 0.93 | |
| (0.86-0.98). The previous settings ranked it first in 6 (17%, 6-31%), top 3 in 31, reciprocal rank 0.52 | |
| (0.44-0.60). With embeddings off (keyword ranking only) nothing changed: first in 2 of 36, top 3 in 11. | |
| Twelve short exact-term lookups (names, codes such as "HX-55120") stayed first in 12 of 12. | |
| What changed: the starting weights are keyword 0.1 and embedding 4 instead of 2 and 2, so one shared word no longer | |
| outranks a memory whose meaning answers the question, and the embedding model is all-MiniLM-L6-v2 instead of | |
| bge-small-en-v1.5. Almost all of the gain comes from the weights: the old model with the new weights reached 0.92 | |
| on the same held-out questions (scored offline, without the sample memories). On the 36 tuning questions the bge | |
| query prefix helped bge-base and bge-large but not bge-small, and the larger models (bge-base, bge-large, | |
| mxbai-embed-large) were at best level with all-MiniLM-L6-v2 (0.977 vs 0.972) at 2 to 13 times its download size. | |
| With exploration on, the right memory moves around within the top 5 from run to run, because exploring | |
| deliberately tries different rankings so the server can learn. Exploration is therefore off by default. | |
| ## How it works, benchmark and limitations | |
| Each recall scores every candidate as a weighted sum of nine features (keyword and semantic match, recency, | |
| utility, importance, a time-aware recency, agreement between the two searches, and the keyword and semantic | |
| scores of neighbouring memories). Embedding recall scores each memory on its own, so a distractor that shares the | |
| question's wording can edge out the real answer. Two parts of the demo address that, and both are measurable in | |
| the app rather than asserted: | |
| - **Benchmark tab.** The cross-encoder reranker reads each `(question, memory)` pair together and reorders the | |
| pool the server returned. On a bundled set of eight weak cases (each answer paired with a higher-overlap | |
| distractor, including the quarterly-review case above), run live against a throwaway server, raw embedding | |
| recall ranks the answer first in **7 of 8** (MRR 0.938) and the reranker in **8 of 8** (MRR 1.000), with **0** | |
| cases made worse. Fixed cases, fixed seed, so the numbers reproduce. | |
| - **Learning loop tab.** Starting from the server's default even weights (keyword 2, semantic 2), a batch of | |
| simulated feedback that marks the right memory useful (exploration on) shifts the learned weights, most visibly | |
| semantic 2 to about 4.2 and utility 0 to about 2.8. Scored on all eight weak cases afterwards, recall@1 rises | |
| from **7 of 8 to 8 of 8** (MRR 0.938 to 1.000). The loop runs on its own throwaway server and never touches the | |
| shared policy the live demo uses. | |
| These two run the real pipeline in-Space. They are separate from the LoCoMo and LongMemEval numbers in the | |
| "Measured results" tab, which come from the repository README with simulated feedback and are **not** reproduced | |
| here. | |
| Limits of this demo: | |
| - Each visitor gets a private namespace, but the server keeps data in memory and wipes it on restart. | |
| Do not store anything sensitive. It holds at most 2,000 memories. | |
| - The check above uses short, hand-written questions that lean toward paraphrase, which favours embeddings. A known | |
| weak case: "Who is presenting at the quarterly review?" ranks a memory that only mentions the review just above | |
| "Marcus will present the fraud dashboard at the quarterly review." | |
| - A few feedback clicks in the Recall tab will not visibly retrain the ranking; the Learning loop tab uses dozens | |
| of simulated cycles to make the shift legible. The LoCoMo and LongMemEval numbers in the "Measured results" tab | |
| come from the repository README, not from this demo. | |
| - The in-Space benchmark above uses eight short weak cases, not a standard dataset; it shows the reranker and the | |
| learning loop working on hard examples, not a general accuracy figure. | |
| - The server was developed against the LoCoMo and LongMemEval benchmarks; the gains reported there were | |
| larger on LoCoMo than on LongMemEval, as the repository README explains. | |
| ## Architecture | |
| ```mermaid | |
| flowchart LR | |
| A[add_memory / recall] --> B[Gradio UI] | |
| B --> C[lethe_client] | |
| C --> D[Go lethed daemon<br/>HTTP :8787] | |
| D --> E[Embedding recall<br/>all-MiniLM-L6-v2] | |
| E --> F[Cross-encoder rerank<br/>ms-marco-MiniLM-L-6-v2] | |
| F --> G[Learned 9-feature ranking policy] | |
| G --> H[Ranked memories] | |
| H -.feedback.-> G | |
| ``` | |
| ## Limitations / failure modes | |
| - Demo memory pool is small (seeded samples), so default weights already rank most answers first; the learning tab shows the weight shift and recall@1 lift rather than a single dramatic flip. | |
| - The live demo's ranking policy is global (not per-namespace); benchmarks run against a throwaway server so they don't perturb it. | |
| - Reranker is an off-the-shelf cross-encoder; the product is the server + learned ranking, not the reranker. | |