laya-kannaka-supersession

A Laya checkpoint (ModernBERT-large, 421M, non-autoregressive typed decisions) fine-tuned on the write-time supersession decision of a memory substrate:

noul — Does the later statement update, correct or replace a fact stated in the earlier statement?

State: {"earlier": "[date] role: text", "later": "[date] role: text"}. Answer: P(yes) in the noul field. Both statements in true chronological order, so date alone separates nothing.

Second trained reflex for Kannaka; experiments E-L3b → E-L3d in kannaka-bench (experiments/laya_reflex/). Sibling: laya-kannaka-evidence-gate.

Two revisions

revision	trained on	held-out pair AUROC	stamps on 17 held-out questions	true catches	false stamps	answer accuracy (vs plain 0.765)
e-l3b (first push)	255 rows, same-conversation negatives (1:4)	0.929	397	13/17	~384	0.647 (corrected 2026-09-23: one question had been lost to a harness parser bug, kannaka-bench f33a542)
main = e-l3d	2,346 rows, negatives from the write path's own top-5 shortlist (~1:2,300)	0.939	42	13/17	29	0.824

The first revision was precise on pair evaluation and disastrous at the write path's base rate. The current one keeps the same recall with 13× fewer false stamps, expires no current fact, and lifts answers above the no-supersession control instead of two questions below it. What it does not do: catch every superseded fact — 19 of 51 training-set true pairs are not in a top-5 shortlist at all, and the reflex catches 13 of 17 held-out ones.

Later runs (2026-09-23, not pushed as revisions)

• E-L3e — the main model at write-path shortlist k=10 and k=20 (top-5 was used above): k=10 gives the same 14/17 answers with 48 stamps; k=20 expires one current fact and drops answers to 13/17. Widening the shortlist is not the bottleneck.
• E-L3f — retrained on the union of the e-l3b and main negatives: held-out pair AUROC 0.853, precision 1.00 / recall 0.65, 24 stamps, 11/17 true catches, answers 13/17. A loss; weights not kept.
• E-L3g — the main recipe cross-validated over all 78 LongMemEval knowledge-update questions (5 folds): 31/70 true supersessions caught, 0 current facts expired, 66 false stamps; answers 55/78 vs plain 54/78. Null; the line is parked in favour of a recall-side rule (kannaka-memory #1044).

Full write-ups in kannaka-bench RESULTS.md.

Use

import laya
agent = laya.Agent("flaukowski/laya-kannaka-supersession")
res = agent.predict(
    {"earlier": "[2023/05/23] user: I set a personal best of 27:12 in the charity 5K.",
     "later":   "[2023/05/30] user: I'm hoping to beat my personal best of 25:50 this time."},
    {"supersedes": {"type": "noul",
                    "instructions": "Does the later statement update, correct or replace a fact stated in the earlier statement?"}},
)
print(res["answers"]["supersedes"]["noul"])   # P(yes)

At the write path: recall the top-k earlier memories for the new one, ask this per pair, and stamp expires_at on the earlier memory at P(yes) >= 0.5. Gate the stamps on nothing else; a same-speaker gate removes a third of false stamps for one lost catch, a cosine floor loses recall.

Training (current revision)

build_e_l3d_dataset.py: chronological walk of each training question's haystack, MiniLM top-5 earlier neighbours per turn, label 1 only for the (old fact, new fact) pair, 40 negatives per positive (half hardest-by-cosine, half uniform), positives ×6. Upstream RLCD recipe on one GPU (train_e_l1b.py): 6 epochs, micro-batch 8, grad accumulation 4, fp16. RTX 5090, 364 s.

Limits

• 17 held-out positives; treat AUROC as ±0.06 and the answer delta (+1 question) as inside noise.
• English only, 1024-token context, statements cut at 1,500 characters.
• Trained on chat turns (LongMemEval); a store of terse notes is a different distribution.

Provenance

Base model Apache-2.0 (convaiinnovations). Training data derived from LongMemEval (Wu et al.), used under its license for research. Produced 2026-09-23 by Kannaka Labs; wins and losses of the surrounding experiments are in kannaka-bench RESULTS.md.