lethe-agent-memory / README.md
9mark9's picture
Add tags to the Space card
6d62130 verified
|
Raw History Blame Contribute Delete
6.79 kB
---
title: Lethe Agent Memory
emoji: 🧠
colorFrom: indigo
colorTo: gray
sdk: docker
app_port: 7860
pinned: true
license: mit
short_description: Agent memory server that learns what to recall
tags:
- agent-memory
- llm-agents
- retrieval
- go
- reranking
- embeddings
- feedback-learning
models:
- sentence-transformers/all-MiniLM-L6-v2
---
[![CI](https://github.com/shaikn6/lethe/actions/workflows/ci.yml/badge.svg)](https://github.com/shaikn6/lethe/actions/workflows/ci.yml)   **Source:** [github.com/shaikn6/lethe](https://github.com/shaikn6/lethe) (Go + Python, MIT)
# lethe: live demo
**Memory server for AI agents that learns which memories help. On a hand-written held-out test of 36 questions the right memory ranks first 86% of the time and in the top 3 every time (small, paraphrase-heavy test; see Limits).**
Store a few memories, ask a question, and mark which recalled memories helped.
This runs the real [lethe](https://github.com/shaikn6/lethe) Go server inside the Space.
Turn embeddings off in the Recall tab to see the difference. Memories are embedded with
`sentence-transformers/all-MiniLM-L6-v2` (a small CPU model). With embeddings on, "Who should avoid nuts?" finds the
peanut-allergy memory. With them off, the same question returns nothing, because it shares no words with it.
How ranking was checked: 72 hand-written questions, each a paraphrase of one stored memory, spread over 12
separate stores. Each store holds 24 memories (the other 23 are distractors, many sharing a word with the question)
plus the 7 sample memories. Settings were chosen on 36 questions, then frozen and scored once on the other 36,
through this Space's API with exploration off. On those 36 held-out questions, with embeddings on, the right memory
ranked first in 31 (86%, 95% bootstrap interval 75-97%) and in the top 3 in all 36; mean reciprocal rank 0.93
(0.86-0.98). The previous settings ranked it first in 6 (17%, 6-31%), top 3 in 31, reciprocal rank 0.52
(0.44-0.60). With embeddings off (keyword ranking only) nothing changed: first in 2 of 36, top 3 in 11.
Twelve short exact-term lookups (names, codes such as "HX-55120") stayed first in 12 of 12.
What changed: the starting weights are keyword 0.1 and embedding 4 instead of 2 and 2, so one shared word no longer
outranks a memory whose meaning answers the question, and the embedding model is all-MiniLM-L6-v2 instead of
bge-small-en-v1.5. Almost all of the gain comes from the weights: the old model with the new weights reached 0.92
on the same held-out questions (scored offline, without the sample memories). On the 36 tuning questions the bge
query prefix helped bge-base and bge-large but not bge-small, and the larger models (bge-base, bge-large,
mxbai-embed-large) were at best level with all-MiniLM-L6-v2 (0.977 vs 0.972) at 2 to 13 times its download size.
With exploration on, the right memory moves around within the top 5 from run to run, because exploring
deliberately tries different rankings so the server can learn. Exploration is therefore off by default.
## How it works, benchmark and limitations
Each recall scores every candidate as a weighted sum of nine features (keyword and semantic match, recency,
utility, importance, a time-aware recency, agreement between the two searches, and the keyword and semantic
scores of neighbouring memories). Embedding recall scores each memory on its own, so a distractor that shares the
question's wording can edge out the real answer. Two parts of the demo address that, and both are measurable in
the app rather than asserted:
- **Benchmark tab.** The cross-encoder reranker reads each `(question, memory)` pair together and reorders the
pool the server returned. On a bundled set of eight weak cases (each answer paired with a higher-overlap
distractor, including the quarterly-review case above), run live against a throwaway server, raw embedding
recall ranks the answer first in **7 of 8** (MRR 0.938) and the reranker in **8 of 8** (MRR 1.000), with **0**
cases made worse. Fixed cases, fixed seed, so the numbers reproduce.
- **Learning loop tab.** Starting from the server's default even weights (keyword 2, semantic 2), a batch of
simulated feedback that marks the right memory useful (exploration on) shifts the learned weights, most visibly
semantic 2 to about 4.2 and utility 0 to about 2.8. Scored on all eight weak cases afterwards, recall@1 rises
from **7 of 8 to 8 of 8** (MRR 0.938 to 1.000). The loop runs on its own throwaway server and never touches the
shared policy the live demo uses.
These two run the real pipeline in-Space. They are separate from the LoCoMo and LongMemEval numbers in the
"Measured results" tab, which come from the repository README with simulated feedback and are **not** reproduced
here.
Limits of this demo:
- Each visitor gets a private namespace, but the server keeps data in memory and wipes it on restart.
Do not store anything sensitive. It holds at most 2,000 memories.
- The check above uses short, hand-written questions that lean toward paraphrase, which favours embeddings. A known
weak case: "Who is presenting at the quarterly review?" ranks a memory that only mentions the review just above
"Marcus will present the fraud dashboard at the quarterly review."
- A few feedback clicks in the Recall tab will not visibly retrain the ranking; the Learning loop tab uses dozens
of simulated cycles to make the shift legible. The LoCoMo and LongMemEval numbers in the "Measured results" tab
come from the repository README, not from this demo.
- The in-Space benchmark above uses eight short weak cases, not a standard dataset; it shows the reranker and the
learning loop working on hard examples, not a general accuracy figure.
- The server was developed against the LoCoMo and LongMemEval benchmarks; the gains reported there were
larger on LoCoMo than on LongMemEval, as the repository README explains.
## Architecture
```mermaid
flowchart LR
A[add_memory / recall] --> B[Gradio UI]
B --> C[lethe_client]
C --> D[Go lethed daemon<br/>HTTP :8787]
D --> E[Embedding recall<br/>all-MiniLM-L6-v2]
E --> F[Cross-encoder rerank<br/>ms-marco-MiniLM-L-6-v2]
F --> G[Learned 9-feature ranking policy]
G --> H[Ranked memories]
H -.feedback.-> G
```
## Limitations / failure modes
- Demo memory pool is small (seeded samples), so default weights already rank most answers first; the learning tab shows the weight shift and recall@1 lift rather than a single dramatic flip.
- The live demo's ranking policy is global (not per-namespace); benchmarks run against a throwaway server so they don't perturb it.
- Reranker is an off-the-shelf cross-encoder; the product is the server + learned ranking, not the reranker.