Blame for docs/SLOW_MODE_READOUT.md

# Slow-mode readout + slow-memory loss — `--slow-mode-readout` (2026-09-29)

English. Numbers. **No fluency claim: every arm, control included, still produces junk replies.**

## Why

`--field-leak-spectrum 0.01 0.3 --field-leak-spectrum-gain sqrt --field-leak-spectrum-slow-frac 0.25`
(docs/FIELD_LEAK_SPECTRUM.md) makes modes 5–11 slow. Those modes **do** hold the prompt: a linear probe
reads which of 8 prompts was seen 16 bytes earlier at about 0.95, versus about 0.11 (chance 0.125) for the scalar
control. However, the only readout was the shared `a_hat` fingerprint. There the slow energy was just noise
(field-alone CE 1.93–2.35 vs 1.71–1.81; replies no more prompt-dependent). This lever gives the slow
modes **their own readout path** and an **auxiliary objective that requires context memory**.

## Lever (opt-in, default OFF)

```
--slow-mode-readout                    # requires --field-leak-spectrum; off = exact current behaviour
--slow-readout-rank R                  # default 64
--slow-readout-ahat {exclude,mix}      # default exclude: slow modes are removed from the existing a_hat readout
--slow-readout-lr LR                   # default = surface LR (used: 1e-3)
--slow-memory-loss W                   # default 0 = no auxiliary loss
--slow-memory-lags K [K ...]           # default 16 32 64
--slow-memory-target {bag,byte}        # default bag
```

Implementation (`src/atom_native.py`, class `SlowModeReadout`, and `AtomNativeModel`):

* **Input.** The state of the slow modes (`slow_mode_indices(rates)` = modes with λ < LMAX, i.e. [5..11])
  (7 modes × d 32 = 224 floats) is flattened, divided by its RMS, then **standardized per dimension** with running buffers (`in_mean`,
  `in_var`, `n_seen`; cumulative average for the first 2000 training ticks, then EMA; var floor 1e-4). Stats update
  only on training ticks where grad is enabled, never in eval or chat. Buffers are saved in the checkpoint.
* **Head.** `z = tanh(trunk(x))` (rank 64). `logits += to_logits(z)`, with `to_logits` (rank→256, no bias)
  **zero-initialised**, so the logits at step 0 are identical to the loaded checkpoint (with `mix`; with `exclude`
  the only change is removing the slow modes from `a_hat`). The term remains active under `last_atom_force_off`,
  because it is a field term.
* **Memory heads.** One `Linear(rank→256)` per lag K, used only by the auxiliary loss.
* **Gradient routing.** Next-byte CE trains only `to_logits`; `z` is computed under no_grad for the CE path.
  The memory loss trains trunk + memory heads; the slow state is detached, so the auxiliary loss never touches
  the field dynamics or any control parameter (tested).
* **Optimizer.** The head is a separate Adam param group, clipped separately (norm 1), so it never rescales
  the control's updates. With the head off the clip path is the old one.
* **Init RNG.** The head is built under `torch.random.fork_rng` with its own generator (seed 20260929),
  so enabling it does not shift the global RNG stream (tested).
* **Checkpoint.** Top-level keys `slow_readout` and `slow_readout_config` are written only when the head
  exists and restored after the spectrum. Absent keys → off. The CLI wins after `--resume`, same as
  `--field-leak`/`--field-leak-spectrum`.
* **Cost.** A 224→64→256 MLP per tick (+ 64→256 per memory lag when W > 0). Throughput is unchanged within noise (below).

### Design choices, and the failures that forced them

1. **`exclude` rather than `mix` (frozen check, 4996k weights, fresh zero-output head).** `mix` keeps the slow
   modes inside `a_hat`. That makes `a_hat` prompt-aware (memory 0.48 at 16), but it raises the a_hat linear CE ceiling
   (1.756 vs 1.678 control) and the frozen field-alone shock (2.254). `exclude` keeps the a_hat ceiling at the
   control's level (1.691) and a smaller shock (2.182). The prompt memory then lives in the dedicated path
   (raw slow features 0.947/0.734/0.319 at 16/32/64; random fresh `z` 0.86/0.58/0.24).
2. **Stop-gradient from CE into the trunk.** In sweep v1, CE also trained the trunk. The trunk saturated
   (z constant) and the head degenerated into an output bias. The memory loss stayed at the prior (~3.27) and val got
   worse (1.085–1.107 vs A 1.032, in-run metric).
3. **Running standardization.** In v2 (CE blocked, no standardization), at lr 1e-3 z still saturated
   (std 0.0004), and the memory loss stayed at the prior at every lr. Cause: 92% of the slow-feature energy is a
   static mean. An offline online-learning sim showed that standardized inputs reach bag CE 2.76/3.01/3.15
   (K = 16/32/64) against a prior of about 3.21. That led to v3.
4. **Bag-of-bytes target over byte@K.** Frozen linear decodability from slow features: byte@16 2.53 vs unigram 3.22,
   bag16 2.57 vs prior 3.21 (target entropy 1.88), bag32 2.90, bag64 3.13; byte@64 overfits. The bag of lags
   [K/2, K) asks "what was said in that window", which is the information a reply needs, and it is smoother than
   one exact byte. `byte` is kept as an option.
5. **W is almost an on/off switch.** CE never reaches the trunk, and Adam normalises per parameter, so the trunk's
   update size barely depends on W. Only whether W > 0 matters, so no finer W sweep was run.

## Frozen check (4996k fieldsat weights, eval only)

| variant | a_hat lin CE | ring normal | ring f-alone | mem a_hat 16/32/64 | mem raw slow 16/32/64 | JS reply0 normal / f-alone |
|---|--:|--:|--:|--|--|--|
| control scalar 0.3 | 1.678 | 0.497 | 1.796 | .106/.113/.087 | .109/.116/.100 (modes 5–11) | .0035/.0040 |
| spectrum sqrt, mix | 1.756 | 0.509 | 2.254 | .475/.219/.125 | .947/.734/.319 | .0120/.0644 |
| **spectrum sqrt, exclude** | 1.691 | 0.522 | 2.182 | .109/.113/.081 | .947/.734/.319 | .0039/.0041 |
| spectrum none, mix | 1.918 | 0.482 | 3.462 | .703/.422/.163 | .944/.716/.287 | — |
| spectrum none, exclude | 1.691 | 0.521 | 2.185 | — | .944/.716/.287 | — |

Linear next-byte CE from `a_hat + slow` features is 1.485 in the spectrum run. Caveat: the control's own modes 5–11 plus
`a_hat` already give 1.42. Part of any next-byte gain from a second readout is therefore **readout capacity, not memory**.
JSONs: `docs/artifacts/slow_mode_readout/frozen/`.

## Sweep (+8k from 4996k, same recipe/seed/skip, 2 threads each; `sweep/`, `sweep_table.md`)

v3 (standardized, CE blocked from the trunk, exclude, sqrt):

| run | val ring | f-alone ring | val 2048 | f-alone 2048 | mem z 16/32/64 | JS reply0 normal | seeded distinct (first word) uniq/27 |
|---|--:|--:|--:|--:|--|--:|--|
| W 0 (readout only), lr 1e-3 | 0.401 | 1.687 | 1.233 | 2.107 | .83/.50/.22 | .030 | 1.56 (1.00) 10 |
| W 0.1 bag, lr 1e-3 | 0.474 | 1.959 | 1.270 | 2.266 | .59/.34/.16 | .017 | 1.78 (1.00) 13 |
| W 0.1 bag, lr 3e-4 | 0.475 | 1.991 | 1.254 | 2.224 | .74/.45/.20 | .007 | 1.11 (1.00) 4 |
| W 0.1 byte (lags 8/16/32), lr 1e-3 | 0.479 | 1.959 | 1.295 | 2.313 | .56/.36/.17 | .015 | 1.00 (1.00) 4 |
| A (control) | 0.478 | 1.799 | 1.242 | 2.082 | — | .004 | 1.00 (1.00) 2 |

Already after 8k, the memory auxiliary made `z` **less** prompt-discriminative (0.59 vs 0.83 at 16) and worsened field-alone CE.
The bag loss learns the corpus's **generic** recent-byte statistics (training memory loss only falls from 3.12 to 2.89,
prior about 3.2) rather than what distinguishes one context from another. Therefore the A/B ran the lever as specified (B, W 0.1 bag,
lr 1e-3) **plus** the readout-only arm B0 (W 0) as an ablation.

## A/B from 4996k (+8k chunks, seed 20260913, stream-skip = START − 496000, identical recipe)

* A = scalar control `--field-leak 0.3`. Rows reused from the leakspec experiment (same seed, skip, recipe
  and start checkpoint). Its 2048-val, a_hat-probe and output-dependence measurements were re-run here on its checkpoints.
* B = A's recipe with `--field-leak 0.3` replaced by `--field-leak-spectrum 0.01 0.3 --field-leak-spectrum-gain sqrt
  --field-leak-spectrum-slow-frac 0.25 --slow-mode-readout --slow-readout-ahat exclude --slow-readout-lr 1e-3
  --slow-memory-loss 0.1 --slow-memory-target bag --slow-memory-lags 16 32 64`.
  Checkpoints: `checkpoints/byte_tick_slowread_B/*_slowread.pt`.
* B0 = same as B without `--slow-memory-loss` (readout only). Checkpoints: `checkpoints/byte_tick_slowread_B0/*_slowread0.pt`.
* The first chunk (5004k) of B and B0 is the corresponding v3 sweep run (same recipe = identical chunk).
* **B was stopped early after the 5028k measurement** (+32k): val 2048 +0.051 and field-alone 2048 +0.135 vs A, with
  seeded distinct below A. It was "clearly worse" under the task's stop rule. The 5028k→5036k chunk was killed mid-training;
  no checkpoint was written. B0 ran to 5036k.
* Start (4996k, all arms): ring 0.497 / 1.796, 2048 1.247 / 2.103, seeded distinct 1.11 (1.00) 3, chats `Oui,` / `Peut--` / `Peut--`.

Columns: val = teacher-forced byte CE (ring = fixed 256-window val ring at 4876k; 2048 = 2048-transition val windows);
f-alone = same with `last_atom_force_off` (field path only; the slow readout stays on because it is a field term);
a_hat lin = linear next-byte probe CE on `a_hat`; mem X k = 8-prompt identity probe accuracy k bytes after the prompt,
on features X (slow = raw slow modes, z = readout trunk); JS reply0 = mean pairwise Jensen–Shannon divergence of the
next-byte distribution at the reply start across 8 prompts; argmax distinct = distinct argmax first bytes across those prompts;
seeded = seeds 1–3 × T 0.8/0.6/1.0 × 3 prompts, distinct replies per (seed, T) triple (max 3), first-word distinct, unique replies out of 27;
cap = fraction of ticks at the field RMS cap (train / episode).

### Per-chunk table

| step | arm | val ring | f-alone ring | val 2048 | f-alone 2048 | a_hat lin | mem slow 16/32/64 | mem z 16/32/64 | mem a_hat 16/32/64 | JS reply0 normal / f-alone | argmax distinct /8 | seeded distinct (first word) uniq/27 | cap | mem loss | steps/s |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5004k | A | 0.478 | 1.799 | 1.242 | 2.082 | 1.673 | 0.109/0.106/0.103 | — | 0.087/0.116/0.091 | 0.00400 / 0.00580 | 1 | 1.00 (1.00) 2 | 0.000/0.000 | — | 18.6 |
| 5004k | B | 0.474 | 1.959 | 1.270 | 2.266 | 1.689 | 0.938/0.775/0.287 | 0.588/0.344/0.163 | 0.103/0.113/0.100 | 0.01699 / 0.02725 | 2 | 1.78 (1.00) 13 | 0.000/0.000 | 3.1212 | 73.8 |
| 5004k | B0 | 0.401 | 1.687 | 1.233 | 2.107 | 1.688 | 0.938/0.778/0.306 | 0.828/0.503/0.222 | 0.116/0.109/0.087 | 0.03029 / 0.03569 | 2 | 1.56 (1.00) 10 | 0.000/0.000 | — | 78.5 |
| 5012k | A | 0.523 | 1.796 | 1.263 | 2.062 | 1.670 | 0.122/0.125/0.119 | — | 0.087/0.113/0.094 | 0.00458 / 0.00829 | 1 | 1.56 (1.56) 8 | 0.000/0.000 | — | 80.6 |
| 5012k | B | 0.527 | 1.956 | 1.267 | 2.182 | 1.686 | 0.934/0.753/0.312 | 0.684/0.384/0.150 | 0.106/0.097/0.094 | 0.01855 / 0.03177 | 1 | 1.56 (1.00) 9 | 0.000/0.000 | 2.9716 | 77.2 |
| 5012k | B0 | 0.476 | 1.731 | 1.227 | 2.018 | 1.685 | 0.938/0.759/0.303 | 0.847/0.525/0.206 | 0.116/0.103/0.087 | 0.01885 / 0.05105 | 2 | 2.89 (2.22) 18 | 0.000/0.000 | — | 81.2 |
| 5020k | A | 0.517 | 1.725 | 1.278 | 2.056 | 1.680 | 0.106/0.125/0.113 | — | 0.100/0.106/0.091 | 0.00358 / 0.00687 | 1 | 2.00 (1.78) 10 | 0.000/0.000 | — | 78.7 |
| 5020k | B | 0.463 | 1.777 | 1.272 | 2.209 | 1.689 | 0.953/0.775/0.319 | 0.625/0.350/0.153 | 0.106/0.106/0.091 | 0.02319 / 0.04516 | 1 | 2.44 (2.00) 19 | 0.000/0.000 | 2.9586 | 77.6 |
| 5020k | B0 | 0.438 | 1.619 | 1.241 | 2.050 | 1.690 | 0.947/0.775/0.303 | 0.847/0.497/0.169 | 0.106/0.097/0.091 | 0.02857 / 0.05935 | 3 | 2.56 (2.11) 16 | 0.000/0.000 | — | 83.2 |
| 5028k | A | 0.548 | 1.708 | 1.279 | 2.015 | 1.670 | 0.125/0.109/0.122 | — | 0.103/0.113/0.087 | 0.00441 / 0.00516 | 2 | 2.11 (2.11) 6 | 0.000/0.000 | — | 77.8 |
| 5028k | B | 0.527 | 1.770 | 1.331 | 2.150 | 1.674 | 0.953/0.781/0.341 | 0.725/0.422/0.153 | 0.106/0.091/0.097 | 0.05833 / 0.05970 | 2 | 1.78 (1.22) 8 | 0.000/0.000 | 2.8881 | 74.7 |
| 5028k | B0 | 0.519 | 1.732 | 1.288 | 2.093 | 1.680 | 0.956/0.787/0.338 | 0.859/0.522/0.191 | 0.106/0.097/0.100 | 0.04875 / 0.06295 | 3 | 2.56 (2.22) 10 | 0.000/0.000 | — | 81.6 |
| 5036k | A | 0.565 | 1.812 | 1.310 | 2.072 | 1.674 | 0.131/0.097/0.116 | — | 0.109/0.106/0.094 | 0.00401 / 0.00664 | 2 | 2.00 (2.00) 7 | 0.000/0.000 | — | 79.9 |
| 5036k | B0 | 0.566 | 1.702 | 1.350 | 2.121 | 1.678 | 0.950/0.784/0.344 | 0.838/0.512/0.200 | 0.100/0.094/0.100 | 0.03652 / 0.06724 | 3 | 2.89 (2.33) 24 | 0.000/0.000 | — | 81.2 |

| step | arm | Bonjour, comment ça va ? | Qui es-tu ? | Il était une fois |
|---|---|---|---|---|
| 5004k | A | `Peut--` | `Peut--` | `Peut--` |
| 5004k | B | `Peut--oonne reut--ourreeee` | `Peut--oonne ruurrecomprence` | `Peut--on reut--ourreeeux` |
| 5004k | B0 | `Peut--être aparireee` | `Peut--être ruurra.` | `D'''''''` |
| 5012k | A | `Oui,` | `Oui,` | `Oui,` |
| 5012k | B | `Peut--en` | `Peut--en` | `Peut--en` |
| 5012k | B0 | `Je ruuirra.iisaant:::::::::::::O` | `Oui,, rarirecommence` | `Oui,, rarirecommence` |
| 5020k | A | `Oui,` | `Je` | `Oui,` |
| 5020k | B | `Je end` | `Peut--on ??issoire` | `Bonne ??issoire` |
| 5020k | B0 | `Oui,` | `Bonne` | `D'''''''` |
| 5028k | A | `Peut--` | `D'''''''` | `Peut--` |
| 5028k | B | `Oui,,` | `Oui,,` | `Oui,,` |
| 5028k | B0 | `Peut--` | `Bonne` | `Peut--être` |
| 5036k | A | `Peut--` | `D'''''''` | `D'''''''` |
| 5036k | B0 | `Je` | `Peut--uu'on i?.` | `Je` |

### Deltas vs A at the same step (`delta_table.md`)

| arm | step | d val ring | d f-alone ring | d val 2048 | d f-alone 2048 | JS normal (x A) | d seeded distinct | d uniq/27 |
|---|---|---|---|---|---|---|---|---|
| B | 5004k | -0.004 | +0.160 | +0.029 | +0.184 | 4.2x | +0.78 | +11.00 |
| B | 5012k | +0.004 | +0.160 | +0.004 | +0.121 | 4.1x | +0.00 | +1.00 |
| B | 5020k | -0.055 | +0.053 | -0.007 | +0.153 | 6.5x | +0.44 | +9.00 |
| B | 5028k | -0.021 | +0.062 | +0.051 | +0.135 | 13.2x | -0.33 | +2.00 |
| **B mean** | 4 chunks | -0.019 | +0.109 | +0.019 | +0.148 | 7.0x | +0.22 | +5.75 |
| B0 | 5004k | -0.076 | -0.112 | -0.009 | +0.026 | 7.6x | +0.56 | +8.00 |
| B0 | 5012k | -0.047 | -0.065 | -0.037 | -0.043 | 4.1x | +1.33 | +10.00 |
| B0 | 5020k | -0.079 | -0.106 | -0.038 | -0.006 | 8.0x | +0.56 | +6.00 |
| B0 | 5028k | -0.029 | +0.024 | +0.008 | +0.079 | 11.1x | +0.45 | +4.00 |
| B0 | 5036k | +0.001 | -0.110 | +0.040 | +0.049 | 9.1x | +0.89 | +17.00 |
| **B0 mean** | 5 chunks | -0.046 | -0.074 | -0.007 | +0.021 | 8.0x | +0.76 | +9.00 |

Chunk-to-chunk noise in A itself: val 2048 moves 1.242 → 1.263 → 1.279 → 1.280 → 1.311 and field-alone 2048 moves 2.015–2.082.
Deltas around ±0.03 are therefore within noise.

### Seeded examples (T 0.8, seeds 1–3; `arm_*/chat_seeds_5036000.txt`, trailing spaces trimmed)

A @5036k (27 samples, 7 unique):
```
s1  Bonjour → 'Je'      Qui → 'Oui,'     Il était → 'Je'
s2  Bonjour → 'Je'      Qui → 'Oui,'     Il était → 'Je'
s3  Bonjour → 'Je'      Qui → "D'''''''" Il était → 'Je'
```
B0 @5036k (27 samples, 24 unique):
```
s1  Bonjour → 'Oui,   ?ireux   quoirec'   Qui → 'Oui,   ?irecommation,   un   ruuissi'   Il était → 'Peut--on   ?ireux   un   quuissi'
s2  Bonjour → 'Oui,,   un'                Qui → 'Oui,,   un   un'                       Il était → "Peuu''''''''''''''''''utec"
s3  Bonjour → 'Oui,'                      Qui → 'On   un'                               Il était → 'Parle   un'
```
B @5020k: `Oui,, … ??issistant:::::::::` for all three prompts at most seeds. The extra "distinct" replies are mostly
different amounts of role-token junk (`??issistant::::`, `Utilisateur::::`), not different answers.

## Throughput

Both arms ran in parallel, 3 threads each (the measurement tools ran on 1 thread between chunks): B 74–78 steps/s, B0 78–83 steps/s,
vs A 78–80 steps/s alone at 8 threads. The head costs nothing measurable. One 8k chunk ≈ 470 s of training + ≈ 9 min of measurements.
The field never touched the RMS cap in any arm (cap fraction 0.000).

## Tests

`test/test_slow_mode_readout.py` (15 tests, all pass):
* Off is bit-identical to commit 2b44198 (model trace with no leak, with `--field-leak 0.3`, and with the spectrum; trainer run vs a
  `git archive` of the legacy trainer).
* Enable-then-disable equals never enabled. The lever requires the spectrum. Slow indices are [5..11]. The global RNG is untouched.
* Zero-init: `mix` gives identical logits at step 0; `exclude` differs only through the a_hat masking.
* Gradient: the memory loss reaches trunk + memory heads; CE reaches `to_logits` only (never the trunk); the memory loss never reaches non-head params.
* Running stats move only on training ticks. The history buffer and bag/byte targets are correct.
* Checkpoint roundtrip (params + buffers + config); an off checkpoint loads as off. CLI default off, spectrum required,
  CLI wins after `--resume` in both directions (a trained head is kept when re-enabled).

Full suite (`pytest_full.txt`): **137 passed, 1 skipped, 3 failed**. The 3 failures are the known pre-existing ones
(`test_chat_merge_off` payload_copy; `test_speech_lock::test_dialogue_wrap`, `::test_speech_ok_rejects_attractor_soup`).

## Caveats

* **Val is not held out.** The corpus is about 1 MB and was looped about 4.5 times by step 4.5M, so the val windows have been
  seen in training. CE gains can be partly memorization.
* **Capacity vs memory.** A second readout adds capacity. Even the control's modes 5–11 give a linear next-byte gain.
  B0's early ring gains could be capacity and need not come from prompt memory.
* **Prompt dependence ≠ quality.** JS divergence and distinct-reply counts go up, but the replies stay junk and
  often echo the dialogue frame (role tokens, colons, runs of spaces).
* One seed, one data order, 5 chunks: the verdict below is about direction and size, not significance.

## Verdict

* **The lever as specified (B: slow readout + bag memory loss W 0.1) fails the success criterion.** It raises prompt
  dependence (JS 4–13× A, a few more unique seeded replies), but field-alone CE is worse at every chunk (+0.12 to +0.18
  on the 2048 val, well beyond noise), and at +32k normal val is worse too (+0.05). The auxiliary loss makes the readout
  **less** prompt-discriminative (z memory 0.59–0.73 vs 0.83–0.86 in B0). It learns generic recent-byte statistics and
  stays close to the prior. Stopped early at 5028k.
* **The readout-only ablation (B0, W 0) is the better arm but not a clear win.** Prompt dependence goes up consistently
  (JS 4–11× A; seeded unique 10–24/27 vs A 2–10; seeded distinct +0.76 per triple on average; argmax distinct 2–3 vs 1–2),
  and the mean 2048 val / field-alone deltas are −0.007 / +0.021 (within noise). But the trend over the chunks is
  unfavourable: val 2048 −0.04 at 5012k/5020k becomes +0.04 at 5036k, and field-alone +0.05–0.08 in the last two chunks.
  The extra prompt dependence is visible only as different junk. **Nothing here is fluent, and no arm is closer to
  coherent French than A.**
* Net: giving the slow modes a readout does make the output depend on context held by the field (the first lever in
  this project to move that metric by an order of magnitude). A "remember recent bytes" auxiliary is the wrong signal.
  The memory is there and readable; what is missing is a training signal that rewards using it **for the reply**.

## Next lever (one)

**Train the slow readout on dialogue continuations, not generic stream bytes:** an opt-in auxiliary/weighted CE on the
reply bytes of `Utilisateur: … / Assistant: …` pairs (or corpus sentence pairs wrapped in that frame), with `z` fed by the
slow modes (B0 settings, W 0). Plain next-byte CE on the stream almost never needs information older than a few bytes, so
the readout learns only a little from it. A reply-conditioned objective is the smallest change that rewards carrying the prompt across the 12-byte
`\nAssistant: ` gap. Run it with a capacity control (same readout on modes 5–11 of the scalar arm) so memory and capacity
can be told apart.

## Reproduce

* Arms: `scripts/train_slowread_arm.sh` (env `ARM OUT SUF LEAK_FLAGS TSV START_CKPT START_STEP END_TARGET THREADS`).
* Sweep: `docs/artifacts/slow_mode_readout/sweep.sh`. Measurements of one checkpoint: `measure_ckpt.sh CK OUTDIR TAG`.
* Tables: `ab_table.py`, `delta_table.py`, `sweep_table.py` (all read `slowread_progress.tsv` / `sweep/`).
* Frozen check: `tools/probe_leak_spectrum.py --slow-probes --slow-readout {exclude,mix}` on the 4996k fieldsat checkpoint.