README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Base
library_name: peft
tags: [lora, parametric-memory, multi-hop-qa, musique, generative-retrieval]
---

# MemAnchor - parametric memory adapters

LoRA adapters for `Qwen/Qwen3-4B-Base`. Each one is a **memory**: a corpus of 759 Wikipedia
paragraphs written into the weights, so that asking a question makes the model
generate the relevant paragraph. There is no index and no retrieved text at question
time - the paragraph is regenerated.

Every adapter here stores the *same* 759 paragraphs with the *same* trainer
(r=64, alpha=128, 30 epochs, lr 2e-4, max_len 384; the reader is r=16/alpha=32).
The only thing that differs between the `memory/` folders is **what (question ->
paragraph) training pairs were written**, which is what the study varies.

## Loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from huggingface_hub import snapshot_download

path = snapshot_download("0010SS/MemAnchor", allow_patterns="memory/questions-9subject-9mention/*")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Base")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Base", dtype="bfloat16")
mem = PeftModel.from_pretrained(base, f"{path}/memory/questions-9subject-9mention")

ids = tok("Question: Who distributed the film UHF?\nAnswer:", return_tensors="pt")
print(tok.decode(mem.generate(**ids, max_new_tokens=200)[0]))   # recites the paragraph
```

Disable the adapter (`with mem.disable_adapter():`) to get the frozen base back - that
is how the planner and the untrained reader run in our pipeline.

## Contents

| folder | what was written per paragraph |
|---|---|
| `memory/questions-9subject-9mention` | Generated questions, 9 subject-anchored + 9 mention-anchored per paragraph (12,762 pairs).
| `memory/questions-5subject-5mention` | Same generator and prompts at 5 of each (9,733 pairs) - the matched-budget comparator.
| `memory/keys-subject` | Extracted 'subject ; relation ; object' keys as addresses, 10 per paragraph (7,307 pairs).
| `memory/keys-mention` | Same keys with half anchored on mentioned entities instead of the subject (6,971 pairs).
| `memory/graph-titleset` | Entity -> paragraph-title-set graph written into the weights (23,812 pairs), frozen-test filtered.
| `memory/graph-titleset-unfiltered` | Same graph with node addresses exempt from the frozen-test filter (23,985 pairs).
| `memory/questions-plus-templated-keys` | Row 1's questions plus every extracted key rendered into two templates (25,709 pairs).
| `memory/memo-synthesis` | MeMo's four-stage question synthesis, our storage target (5,984 pairs, 624/759 covered).
| `reader/v2-hard-distractors` | Span-extraction reader, 2,000 train-split items, hard distractors, 4-doc contexts (6,406 pairs).
| `reader/v1-random-distractors` | Earlier reader revision, 2 randomly sampled distractors - the closed-book end-to-end numbers use this.

## Evaluation

Two tests, both over the same 759-passage corpus:

- **Retrieval (168 already-decomposed sub-questions)**: does the generated paragraph
  match the correct one, by best word-overlap over all 759. 84 *direct* questions name
  the paragraph's own subject; 84 *bridge* questions name an intermediate entity.
- **End to end (42 original two-hop questions)**: the model decomposes, queries its own
  memory twice, and answers. Exact match and `contains`.

## Known caveats

Read these before quoting numbers.

1. The `graph-titleset` adapters were scored by a **different evaluator** than the other
   memories (title-space argmax with set membership over up to 6 fetched passages, versus
   passage-text top-1 over 759). Their retrieval figures are not directly comparable to
   the other folders'.
2. `questions-9subject-9mention` was selected as the best of a ten-point sweep evaluated
   on the same 42-item test set. Treat its end-to-end number as an optimistic point
   estimate, not an unbiased one; the full sweep is in the repo.
3. Run-to-run variance on the 42-item test is roughly +/-0.05 (about two items).
   Differences smaller than that are not differences.
4. Neither `questions-*` nor `memo-synthesis` had its training questions filtered against
   the test questions; the `keys-*` and `graph-*` arms did. An audit found 85 of 12,475
   generated questions would be flagged, concentrated on 36 paragraphs.

## Not included

The supervised upper bound and the frozen-test-filtered arm were trained on a
collaborator's cluster and are not in this snapshot. The JANUS memory matrices, SCoL
consolidation checkpoints, librarian writer-RL arms, legacy meta-RL per-episode
adapters, and LoCoMo per-conversation memories are excluded as out of scope.

Code, training data and all result files: <https://github.com/0010SS/MemAnchor>

## Attribution

Built from MuSiQue (CC-BY-4.0) over Wikipedia text (CC-BY-SA). Training questions were
generated with Qwen2.5-7B-Instruct; adapters are LoRA weights over `Qwen/Qwen3-4B-Base`.