CLERK paper draft v0.1: method, benchmark spec, protocol; results tables pending observed runs

paper/paper.md

@@ -0,0 +1,112 @@
+# CLERK: Learned Write-Time Consolidation for Fixed-Budget Memory in Evolving Multi-Session Dialogue
+
+**Justin Wolcott**
+*fallnai-research.org*
+
+**Draft v0.1 — September 2026.** Method, benchmark, and experimental protocol are final. All quantitative results in §7 come from runs of the released pipeline (`run_all.sh`); this draft carries the tables with values *pending compute*. No result in this paper is estimated, extrapolated, or fabricated.
+
+---
+
+## Abstract
+
+Long, evolving interactions — assistants that remember a user across weeks of sessions — currently force a choice between full-transcript context (quadratic cost growth; stale facts competing with fresh ones) and external memory updated by prompt heuristics (whose errors compound; LLM-consolidated memories have been shown to corrupt over repeated updates). We present **CLERK** (Consolidated Ledger with Eviction and Rewrite Keys), a memory architecture in which a *trained* consolidation policy performs write-time memory management: at each session boundary, a small model reads the current memory — a fixed-budget JSON ledger of atomic fact slots — together with the session transcript and emits a structured edit program (ADD / UPDATE / TOMBSTONE / EVICT) that a deterministic reducer applies. Supervision is exact by construction: evolving multi-session dialogues are generated programmatically, so gold ledger transitions — including gold evictions under budget pressure — are known without any LLM judge. Reading costs O(budget) tokens regardless of interaction length, supersession is explicit, and the write policy is *learned*, not prompted. We release a seeded, invariant-checked synthetic evolving-session benchmark, a LoRA-trained 0.5B policy on Qwen2.5-0.5B-Instruct, and a five-system evaluation protocol (trained CLERK, prompted-ledger, full-context, sliding window, RAG) on the held-out benchmark plus zero-shot on LoCoMo-MC10.
+
+## 1. Introduction
+
+Three failure modes dominate current approaches to long-term conversational memory:
+
+1. **Cost blow-up.** Full-transcript context grows linearly with history; attention grows quadratically. A multi-week interaction cannot ride in context forever.
+2. **Stale facts.** When facts change ("I was a barista" → "I'm a chef now"), transcripts retain both statements. Retrieval is similarity-based and cannot adjudicate currency; the model must resolve temporal conflicts implicitly, per question, forever.
+3. **Untrained write policies.** Agent-style systems (MemGPT, Mem0, Zep) consolidate memory with prompted LLM calls. Recent work shows prompted consolidation corrupts memory over successive updates, and a 2026 implicit-conflict benchmark reports the best frontier model at 55.2%. The write path is the weak link — and nobody trains it.
+
+**Contributions.**
+
+- **C1 (Architecture).** A ledger memory with typed slots and a five-op edit language executed by a deterministic reducer; write-time consolidation at session boundaries; an O(budget) read-cost amortization guarantee (§4.4).
+- **C2 (Exact supervision).** A seeded synthetic-benchmark generator whose gold ledger transitions are known by construction, making consolidation trainable without any LLM judge, with machine-checked dataset invariants.
+- **C3 (Evaluation).** A five-system comparison on a held-out evolving-session benchmark with a supersession-error metric, plus zero-shot transfer on LoCoMo-MC10 (1,986 items, five ability types).
+- **C4 (Release).** Fully reproducible: seeded generation, invariant-gated data, pinned dependencies, one-command pipeline.
+
+## 2. Related Work
+
+**Architectural memory.** Transformer-XL (Dai et al. 2019), Compressive Transformer (Rae et al. 2020), Memorizing Transformers (Wu et al. 2022), Block-Recurrent Transformer (Hutchins et al. 2022), Recurrent Memory Transformer (Bulatov et al. 2022; 1M-token scaling 2023), Infini-attention (Munkhdalai et al. 2024). Memory3 (Wang et al. 2024) and Memory Layers at Scale (Bergsma et al. 2024) retrieve static key-value facts; ATLAS (2025) and Memory Decoder (2025) pretrain dedicated memory modules. These memories are opaque, unversioned, and lack update semantics; all require from-scratch or pretraining-scale training. CLERK is interpretable, reducer-checked, and LoRA-scale.
+
+**Agent-style memory.** MemGPT (Packer et al. 2023), Mem0 (2025), Zep/Graphiti (2025), A-Mem (2025), MemoryBank (2023), MemoryOS (2025), MemOS (2025), HippoRAG 1/2 (2024/2025). Consolidation in all of these is prompt-heuristic. Two published findings motivate training it instead: STALE/CUPMem (2026) shows frontier models reach only 55.2% on implicit-conflict memory scenarios, and arXiv:2605.12978 (2026) shows LLM-consolidated memories degrade across repeated updates relative to raw episodic logs.
+
+**Learned memory policies.** Memory-T1 (2025) trains a read-time temporal evidence-selection policy with RL at 7B scale; MemAgent (2025) learns a fixed-size overwrite policy. Both act when *reading*. CLERK trains the *write* path, with structured semantics and a hard budget, at 0.5B scale.
+
+**Benchmarks.** RULER (2024), LongMemEval (2024, includes knowledge-update questions), LoCoMo (2024) and its MC10 derivative (1,986 items). Our synthetic benchmark is complementary: only a generated world can provide exact gold memory states — including which facts were evicted and why — at supervision grade.
+
+## 3. Problem Setting
+
+A user interacts with an assistant across sessions $X_1, X_2, \dots$ The assistant may be asked questions about the user at any time. A memory system must support: **retention** (old facts), **supersession** (answer with current value), **invalidation** (facts no longer true), **temporal recall** (what a fact used to be), **acknowledged ignorance** (never-stated facts), all at per-question cost independent of interaction length.
+
+## 4. Method
+
+### 4.1 Ledger state
+
+Memory is a ledger $M$ of $B$ slots ($B = 12$). Slot $i$: $m_i = (\mathrm{id}, \mathrm{subject}, \mathrm{predicate}, \mathrm{object}, t_{\mathrm{last}}, v)$, serialized as compact JSON with rows `{i, s, p, o, t, v}`. $v{=}0$ marks a tombstoned slot (fact no longer true; value retained for temporal recall). Tombstoned slots remain visible until EVICT.
+
+### 4.2 Edit language and deterministic reducer
+
+The policy emits an edit program $\Pi$; the reducer applies it deterministically:
+
+| Op | Semantics |
+|---|---|
+| ADD | write fact into lowest free slot; dropped + counted if none |
+| UPDATE | overwrite slot $i$'s fact, bump $t_{\mathrm{last}}$; slot identity preserved ⇒ explicit supersession |
+| TOMBSTONE | set $v{=}0$; value retained |
+| EVICT | free slot $i$ |
+| NOOP | nothing |
+
+The reducer is the single source of truth: the generator emits gold programs, training consumes gold programs, and every evaluation — including simulated model rollouts — applies programs through the same reducer. Malformed model output is surfaced as a metric (JSON validity rate), never silently repaired.
+
+### 4.3 Training
+
+LoRA SFT on Qwen/Qwen2.5-0.5B-Instruct ($r{=}32$, all-linear, lr $2{\times}10^{-4}$, 2 epochs, seq len 2048, effective batch 16, seed 137), via TRL's SFTTrainer. One mixture of two tasks sharing the model: **CONSOLIDATE** $(M_t, X_{t+1}) \mapsto \Pi_{t+1}$ (gold programs, in transcript order, with budget-aware EVICTs) and **ANSWER** $(M_t, q) \mapsto y$ (gold answers machine-checked against $M_t$). Prompt strings are defined once in `clerk/prompts.py` and imported by both trainer and evaluators — train/test formats cannot drift.
+
+### 4.4 Amortization
+
+Full-context reading at session $k$ costs $kL$ tokens; over a stream of $N$ sessions, $\Theta(N^2 L)$. CLERK pays one consolidation call $O(L + B)$ per session plus $O(B)$ per read: $\Theta(N(L + B))$ total, with per-question context cost *flat in history length*.
+
+### 4.5 Eviction supervision
+
+When facts exceed budget, the generator's documented salience oracle $s(m) = 4\,n_{\mathrm{uses}} + 3\,v + \max(0, 4 - \mathrm{age})$ selects eviction victims; gold programs EVICT *before* ADD when the ledger is full, so every supervised program is executable by construction. The oracle is a labeling device, not a deployment component; learned eviction without oracle supervision is future work.
+
+## 5. Benchmark
+
+`clerk.generator` produces seeded personas over 8 sessions, with 18 predicates (job, city, pet, partner, car, hobby, boss, project, deadline, …), dedicated update/invalidation phrasings, chatter distractors, and a Q&A block per session. Question types: **stale**, **superseded** (current value must win over the old value still present in transcript history), **negated** (tombstoned), **temporal** (the tombstoned slot's retained value), **unknown** (refuse). Splits: 800/100/100 personas (seeds 137/991/4242; disjoint name pools). With $B{=}12$ and up to ~18 facts per persona, eviction pressure is guaranteed.
+
+Machine-checked invariants (`clerk.tests_smoke`): gold programs are executable and reproduce stored ledger states exactly; budgets are never exceeded; serializers round-trip; QA supervision matches the memory state answered against.
+
+## 6. Systems
+
+1. **CLERK** (trained adapter; ledger-only reads) 2. **Prompted-ledger** (same base model, same prompts, no training — isolates the effect of training) 3. **Full-context** (entire transcript) 4. **Window-2** 5. **RAG** (top-5 user statements by embedding) 6. on LoCoMo: **summary-RAG** (top-3 shipped session summaries).
+
+Metrics: per-type accuracy; supersession error rate (answers with the stale value); tokens/question incl. amortized write cost; JSON validity; valid-fact precision/recall/F1 vs gold; LoCoMo accuracy overall and per ability type.
+
+## 7. Experiments
+
+**Status: results pending observed runs.** The following tables are produced by `bash run_all.sh` and filled from its JSON outputs (`results/results_synthetic.json`, `results/results_locomo.json`). This v0.1 draft ships with them empty rather than with estimated numbers.
+
+- Table 1: held-out accuracy by question type, per system — *pending*
+- Table 2: supersession error rate on superseded questions — *pending*
+- Table 3: tokens per question (incl. amortized writes) vs history length — *pending*
+- Table 4: consolidation fidelity (JSON validity, fact P/R/F1 vs gold) — *pending*
+- Table 5: LoCoMo-MC10 zero-shot, overall + five ability types — *pending*
+- Table 6: ablations (budget ∈ {8,12,24}; trained vs prompted eviction) — *pending*
+
+**Pre-registered hypotheses** (stated before any run): H1 CLERK > prompted-ledger on superseded/negated accuracy; H2 CLERK's tokens/question flat in history length; H3 full-context and RAG show the highest supersession error; H4 trained eviction degrades gracefully under tighter budgets. Deviations are reported as findings, not tuned away.
+
+**Transfer caveat.** The policy trains on single-user fact-statement dialogues; LoCoMo conversations are peer dialogues with event-rich structure. The LoCoMo section measures transfer of the learned policy, and a weak result is a finding about generality, not something to tune away post hoc.
+
+## 8. Limitations
+
+Template-bounded fact schemas (18 predicates); open-predicate behavior measured only via MC accuracy; oracle-supervised eviction; 0.5B scale only; English only.
+
+## 9. Reproducibility
+
+Code + paper: https://huggingface.co/Falln87/clerk-memory · benchmark card: https://huggingface.co/datasets/Falln87/clerk-synthetic-sessions · adapter: https://huggingface.co/Falln87/clerk-qwen2.5-0.5b-lora. One command regenerates data byte-identically and reruns everything; smoke tests gate GPU jobs.
+
+## References (abridged)
+
+Bergsma et al. 2024 (arXiv:2412.09764) · Bulatov et al. 2022 (2207.06881; 2304.11062) · Dai et al. 2019 (1904.09737) · Hutchins et al. 2022 (2203.07852) · Hsieh et al. 2024 (2404.06654) · Maharana et al. 2024 (2402.17753) · Munkhdalai et al. 2024 (2404.07143) · Packer et al. 2023 (2310.08560) · Rae et al. 2020 (1911.05507) · Wang et al. 2024 (2407.01178) · Wu et al. 2022 (2203.08913) · Wu et al. 2024 (2410.10813) · Zhang et al. 2025 (2505.00675) · Memory-T1 (2512.20092) · MemAgent (2507.02259) · STALE/CUPMem (2605.06527) · 2605.12978 · ATLAS (2505.23735) · Memory Decoder (2508.09874) · LM2 (2502.06049) · HippoRAG (2405.14831; 2502.14802) · A-Mem (2502.12110) · MemoryBank (2305.10250) · MemoryOS (2506.06326) · MemOS (2505.22101) · Mem0 (2504.19413) · Zep (2501.13956).