ATOM (atom-ai): code snapshot (100fae0 + batched-train), README with measured numbers, curated checkpoints
9f1076a verified | # Decode-label fix + truncated BPTT through the field | |
| Follow-up to `docs/ROOT_CAUSE.md` (commit f8a8918). Code: commit fbea452 (on top of f8a8918). Scripts, raw JSON and | |
| logs: `docs/artifacts/decfix_bptt/`. All checkpoints are in **new** directories; nothing was overwritten or deleted. | |
| Metric: next-byte CE in nats/byte on the fixed 2048-transition val (`eval_windows.py`: 4 × 512, reset per window); | |
| "field-alone" = same with the `last_atom` bigram readout zeroed. | |
| ## TL;DR | |
| 1. **Decode fix (step 1) done.** In byte-tick mode a generated byte is now committed through the *training* | |
| encode path, so its label and features are bit-identical to training. Teacher-forced and generation logits now | |
| match exactly (0.0). Doubled punctuation in 24 probe chats went from 7 to **0**. GRU-judge NLL of the replies | |
| went from 9.69 to **1.37** (4996k), and to **1.27** after a 16k-step fine-tune with efference off (`_decfix`). | |
| Val CE barely moved (1.247 → 1.287): the decode fix changes decoding, not what the model knows. | |
| 2. **BPTT (step 2) passes the decisive test.** With `--field-bptt 64/128` plus a learnable per-mode leak, a fresh | |
| ATOM memorises the 2 KB excerpt: **938 bytes** regenerated verbatim past the 32-byte prefix (K=128) and | |
| **648** (K=64) in 60 min. The target was > 500. Before: ATOM ~10–13, the K=0 control 10, GRU 1,453. TF argmax | |
| accuracy on the text is 99.7 % (5 errors in 1,939 bytes); the control reaches 89.9 %. No NaN, the field cap is | |
| never hit (0 %), and gradient now reaches lags 1–64. Before it was exactly 0 at every lag ≥ 1. | |
| 3. **Full corpus (step 3) is running and improving.** Val 2048 went from 1.247 to **0.48–0.53** (fresh init + BPTT | |
| K=64, 400–512k steps, ~3 h on 1 core; best row 0.478 at 496k) and to **0.65–0.73** (from `_decfix` + BPTT, 400k steps, | |
| plateauing; stopped). The old recipe sat at 1.25 for millions of steps. The fresh run is now at the byte 4-gram | |
| level (0.48), well under a trigram (0.96), and still far from the GRU (0.10). Field-alone went 2.10 → **0.56–0.64**: | |
| the field now carries most of the prediction. GRU-judge NLL of the replies went 1.37 → **0.88**. | |
| 4. **Honest verdict: it does not speak fluent French yet.** Replies contain more whole corpus lines (`Avec plaisir, | |
| continuons.`, `Oui, tout à fait.`, `Bonne question, je vais réfléchir.`), but still with word salad in between | |
| and still largely independent of the prompt. Measured context grew only moderately: the perturbation KL is | |
| non-negligible out to ~12–16 bytes, versus 3–4 before. The learnable leak drifts *up* (0.05 → mean 0.13, e-fold | |
| ~7 bytes), so the optimiser prefers short memory on this data. | |
| ## 1. What changed (all opt-in / escape-flagged; defaults proven bit-identical) | |
| | file | change | flag / default | | |
| |---|---|---| | |
| | `src/io/atomizer.py` | `packet_from_payload(payload, "generated")`: when `max_span_bytes == 1` and the payload is 1 byte, commit via `encode_bytes(payload, reset=False, flush=False)`, i.e. the exact training code. The result is `max_span` (LF/CR `newline`) in linguistic mode, and in eager mode whatever eager training gives. New attribute `Atomizer.legacy_gen_labels` (runtime only, not saved). | **automatic in byte-tick**; `--legacy-gen-labels` (trainer, `chat_atom_native.py`, `chat_seeds_probe.py`) restores the old `_infer_commit_boundary` labels exactly. Non-byte-tick is unchanged. | | |
| | `src/atom_native.py` | `field_bptt` (`set_field_bptt`, `detach_field_graph`, `_bptt_prev_alpha`). In train mode with grad on, `_advance` builds α_t = leak(α_{t-1}) + projection(atom_t) **out of place with autograd** (same ops and order as the legacy in-place path) and keeps α_t's graph for the next tick. Eval and `no_grad` ticks never keep a graph. A stale graph (after reset, load or eval ticks) falls back to a detached copy. | `--field-bptt K` (0 = off = legacy per-tick detach + `no_grad` injection) | | |
| | `src/atom_native.py` | Learnable per-mode leak `field_leak_logit` [n_modes, 1], λ_m = sigmoid, initialised at `--field-leak`. It only gets gradient under BPTT and is exclusive with `--field-leak-spectrum`. | `--field-leak-learnable` (off = no parameter) | | |
| | `src/atom_native.py` | Checkpoint: keys `field_bptt` / `field_leak_logit` are written **only when on** (off = layout unchanged). Load restores them; absent = off. | | | |
| | `tools/run_atom_native.py` | With K > 0: accumulate K tick losses (mean), one backward, clip 1.0, one AdamW step, `stabilize_dynamics_parameters`, cut the graph. A window also closes at the end of the run and at each `--episode-reset` boundary. CLI wins after `--resume` in both directions. Config/metrics keys are added only when on. | | | |
| | `scripts/train_decfix.sh`, `scripts/train_bptt.sh` | Step-1 fine-tune driver; resumable step-3 driver (8k chunks, progress TSV every 2 chunks). | | | |
| Efference with BPTT: `efference_loss` predicts under `no_grad`, which cuts the graph at that tick (documented; | |
| efference is off in every BPTT run here). Batching several sequences through a vectorised field was **not** | |
| implemented: the model is single-sequence throughout (atom list, atomizer, last_atom). One process runs about | |
| 120 ticks/s on one core; parallel arms use the other cores instead. | |
| ### Tests (`test/test_decode_fix_bptt.py`, 15 tests, all pass) | |
| * Byte-tick generated commits are identical to `encode_bytes` of the same stream (payload, start, label, level, | |
| features), in both pack modes. The linguistic labels ⊂ {max_span, newline}. | |
| * `legacy_gen_labels=True` reproduces the f8a8918 atomizer exactly (labels + features), and `generate_packets` | |
| output is byte-identical to the f8a8918 model for 6 sampled (prompt, seed) pairs. Non-byte-tick is unchanged | |
| with either flag. | |
| * TF path vs generation path: max |Δlogit| **== 0.0**. With legacy labels it is > 0 (sanity check). | |
| * `--field-bptt 0` is **bit-identical to f8a8918** on a 40-step train + 20-step eval trace (no leak, leak 0.3, leak | |
| spectrum). The **trainer** checkpoint (all tensors, config, loss trajectory) is bit-identical with defaults + | |
| efference off, and with efference on + `--legacy-gen-labels`. | |
| * BPTT on: forward values (loss, logits, α) are bit-identical to off at equal weights. d loss_T / d atom_{T−L} > 0 at | |
| L = 1, 2, 3, 5, 8 when on, and **== 0** when off. A window cut stops the gradient exactly. | |
| * Learnable leak: initialisation equals the scalar leak; it receives gradient under BPTT. | |
| * Checkpoint roundtrip. An old or off checkpoint loads as off. Trainer CLI: default off, on, CLI wins after resume. | |
| Full suite: 152 passed, 1 skipped, 3 failed (log: artifacts/decfix_bptt/pytest_full.txt). The 3 failures (`test_chat_merge_off::payload_copy_sees_distinct_atoms`, | |
| `test_speech_lock::dialogue_wrap`, `test_speech_lock::speech_ok_rejects_attractor_soup`) **also fail on f8a8918** | |
| (checked in a clean worktree). Two existing tests were updated on purpose: | |
| `test_last_atom_readout::test_commit_boundary_never_generated` now asserts the fixed labels plus the legacy flag. | |
| `test_slow_mode_readout::test_trainer_off_bit_identical_to_legacy` uses `--efference-every 2`, so it now passes | |
| `--legacy-gen-labels` to the new trainer. The fix changes efference commits by design; with the flag the output is | |
| bit-identical to the old code. | |
| ## 2. Step 1: decode fix + `_decfix` fine-tune | |
| Fine-tune: `scripts/train_decfix.sh`, from `checkpoints/byte_tick_fieldsat/…4996000…fieldsat.pt`, 2 × 8k steps. The | |
| recipe is exactly the fieldsat one (LR 1.5e-5 / SLR 3.6e-5, leak 0.3, last-atom dropout 0.5, field loss 0.05, episode 8000, | |
| stream skip = START−496000) except **`--efference-every 0`**. Output: `checkpoints/byte_tick_decfix/atom_native_step_{5004000,5012000}_d32_ms1M_lrd_b4828_fieldsat_decfix.pt`. | |
| Measurements (`measure_decfix.py`; chats use the probe settings: no role prime, 96 packets / 100 bytes, T 0.8, top-k 12; | |
| "24 samples" = 3 prompts × seeds 0–7; judge = GRU 2×512 at val 0.10, first reply line, NLL/byte): | |
| | ckpt | decode | val 2048 | field-alone | doubled punct (24 chats) | space runs ≥3 | judge NLL | same reply for all 3 prompts | | |
| |---|---|---|---|---|---|---|---| | |
| | fieldsat 4996k | legacy (as shipped) | 1.247 | 2.103 | 7 / 20 punct | 29 | 9.69 | 3/8 seeds | | |
| | fieldsat 4996k | **fixed** | 1.247 | 2.103 | **0 / 83** | 0 | **1.37** | 3/8 | | |
| | decfix 5012k | legacy | 1.287 | 2.061 | 10 / 31 | 53 | 9.16 | 0/8 | | |
| | decfix 5012k | **fixed** | 1.287 | 2.061 | **0 / 152** | 0 | **1.27** | 0/8 | | |
| * Removing efference for 16k steps does **not** remove the learned copy rule for OOD labels (legacy decode on 5012k | |
| is still junk). The fix is in the decode path, as intended. | |
| * Val got slightly worse (1.247 → 1.287), within the chunk-to-chunk noise of this branch; field-alone improved | |
| slightly. The fine-tune is not a quality gain; its purpose is to stop training the copy shortcut. | |
| Exact chats, decfix 5012k, fixed decode (unseeded draw, then seeds 1/2/3; `probe_chats` in `measure_decfix_5012k.json`): | |
| * Bonjour, comment ça va ? → `Peut-être ce prends d'accord ?\nAssistant: Bon, je ais d'accord avec toi.` | |
| * Qui es-tu ? → `Peut-être ce tu proposes ?\nAssistant: Je suis d'accord ?\nAssistant: Oui, je suis d'accord ?` | |
| * Il était une fois → `Je suis ci.` | |
| * seed 1: `Peut-être ce d'accord ?…` / `Oui, on verra.` / `Je suis d'accord ?\nAssistant: Je suis d'accord avec toi.` | |
| * seed 2: `Peux qu'on proposes ?…` / (same) / `Je vais laisir, coute.` | |
| * seed 3: `Peut-être ce nuis d'accord ?…` / `Oui, on verra.` / `Je suis d'accord ?\nAssistant: Peut-être un parle marci.` | |
| Legacy decode on the same checkpoint: `Je` + ~30 spaces + `,,` + spaces `iissis` …, `Peut--..\nUtilisateur::` + spaces. | |
| ## 3. Step 2: truncated BPTT, the decisive 2 KB memorisation test | |
| `memorize_bptt.py` uses the same per-tick logic as the trainer (AdamW wd .01, clip 1, stabilize after each step, | |
| atom flush every 64, field loss .05, last-atom dropout .5, reset per pass, **efference off**) plus the trainer's window | |
| rule. Text: `root_cause/memorize_2k.txt` (1,940 B). 60 min, 1 thread each, run in parallel. Regeneration: greedy from the | |
| first 32 bytes through the (fixed) chat commit path. The count is the number of bytes identical to the text past the prefix. | |
| | run | init | K | leak | lr | passes | updates | TF CE | **regen past prefix** | TF argmax acc | cap | NaN | learned λ mean (min–max) | | |
| |---|---|---|---|---|---|---|---|---|---|---|---|---| | |
| | f_k0_l05_lr3e4 (**control, no BPTT**) | fresh | 0 | .05 | 3e-4 | 256 | 496k | 0.601 | **10** | 89.9 % | 0 | no | fixed | | |
| | f_k64_l05_lr1e3 | fresh | 64 | .05 | 1e-3 | 305 | 9.5k | 0.027 | 13 | 99.0 % | 0 | no | fixed | | |
| | f_k64_l05_lr3e3 | fresh | 64 | .05 | 3e-3 | 301 | 9.3k | 0.030 | 249 | 99.0 % | 0 | no | fixed | | |
| | f_k64_l30_lr2e3 | fresh | 64 | .30 | 2e-3 | 305 | 9.5k | 0.018 | 137 | 99.2 % | 0 | no | fixed | | |
| | f_k128_l05_lr2e3 | fresh | 128 | .05 | 2e-3 | 303 | 4.8k | 0.025 | 33 | 99.2 % | 0 | no | fixed | | |
| | **f_k64_learn05_lr2e3** | fresh | 64 | learn (init .05) | 2e-3 | 305 | 9.5k | 0.012 | **648** | 99.7 % | 0 | no | .124 (.051–.326) | | |
| | **f_k128_learn05_lr2e3** | fresh | 128 | learn (init .05) | 2e-3 | 304 | 4.9k | **0.006** | **938** | 99.7 % | 0 | no | .123 (.070–.221) | | |
| | c_decfix_k64_learn05_lr2e3 | decfix 5012k | 64 | learn (init .05) | 2e-3 | 295 | 9.1k | 0.059 | 10 | 98.6 % | 0 | no | .133 (.054–.243) | | |
| | *(root cause) ATOM, old recipe* | fresh / 4996k | 0 | .3 | 3e-4 | 117 / 106 | | 0.31 / 0.255 | 13 / 10 | | | | | | |
| | *(root cause) GRU 124k* | | | | | 5,500 | | 0.005 | 1,453 | | | | | | |
| * **Pass: > 500 bytes (938 and 648).** Without BPTT, the same budget and leak stay at TF CE 0.60 and 10 bytes. | |
| * Regeneration from 20 later prefixes (`diag_mem_*.json`): mean verbatim run is 320 bytes (K128-learn), 263 | |
| (K64-learn), 96–99 (fixed-leak BPTT runs), 10 (control), 3.9 (decfix 5012k). The best models fail at only 5–6 | |
| positions of the text. Those are genuinely ambiguous template continuations: the same reply prefix appears after | |
| different questions, more than ~30 bytes back. | |
| * A learnable per-mode leak clearly beats a fixed one. Starting from the trained decfix checkpoint memorises | |
| *worse* than fresh (TF 0.059, 10 bytes). The trained readout/last-atom solution seems to resist being repurposed | |
| in 60 min. | |
| * Numerical stability: no NaN or Inf in any run. The field RMS cap (3.0) was never hit (0 % of ticks). energy_decay, | |
| coupling and phase_sync stay inside their clamps. Gradient norms are finite, with clipping at 1.0. | |
| * Gradient through time (`grad_by_lag`, loss at tick 200 of val window 1): ‖∂L/∂atom_{T−L}‖ for L = 0/1/2/4/8/16/32/64. | |
| decfix 5012k with K=64 gives 1.53 / 0.73 / 0.59 / 0.25 / 0.049 / 0.0032 / 8e-6 / 1e-10. The legacy path gives | |
| 1.44 / **0 / 0 / 0 …**. For the corpus-trained BPTT model (5252k) with K=64 it is 2.44 / 3.17 / 1.17 / 1.54 / 0.83 / | |
| 0.32 / 0.029 / 0.0013. Gradient reaches 16–32 ticks back and dies by 64. That is the combined effect of the leak | |
| and the field dynamics; no explosion. | |
| ### Perturbation sensitivity (KL of the next-byte distribution L bytes after flipping one letter; val windows, 24 points) | |
| Corpus-trained checkpoints (the meaningful comparison; memorisation models are off-distribution on these windows): | |
| | ckpt | L=1 | 2 | 3 | 4 | 6 | 8 | 12 | 16 | 24 | 32 | 64 | | |
| |---|---|---|---|---|---|---|---|---|---|---|---| | |
| | fieldsat 4996k (ROOT_CAUSE) normal | 7.56 | 4.33 | 0.52 | 0.29 | 0.14 | 0.081 | 0.057 | 0.008 | 0.0004 | 0.0001 | 0 | | |
| | decfix 5012k normal | 8.43 | 4.62 | 0.52 | 0.43 | 0.21 | 0.054 | 0.066 | 0.005 | 0.0009 | 0.0001 | 0 | | |
| | BPTT from decfix 5252k normal | 4.92 | 2.72 | 0.37 | 0.19 | 0.11 | 0.106 | 0.112 | 0.023 | 0.0023 | 0.0002 | 0 | | |
| | BPTT fresh 256k normal | 7.07 | 3.49 | 0.72 | 0.36 | 0.34 | 0.062 | 0.032 | 0.007 | 0.0016 | 0.0001 | 0 | | |
| | fieldsat 4996k field-alone | 0.83 | 0.31 | 0.27 | 0.21 | 0.056 | 0.040 | 0.010 | 0.0015 | 0.0004 | 0 | 0 | | |
| | BPTT from decfix 5252k field-alone | 2.92 | 0.99 | 0.82 | 1.56 | 0.26 | 0.25 | 0.12 | 0.033 | 0.0056 | 0.0007 | 0 | | |
| | BPTT fresh 256k field-alone | 7.41 | 2.31 | 0.37 | 1.42 | 0.29 | 0.12 | 0.029 | 0.008 | 0.0019 | 0.0001 | 0 | | |
| The field now carries the last ~4–8 bytes strongly (field-alone KL at lag 4 is 1.4–1.6 vs 0.21). The tail beyond 16 | |
| bytes is still tiny, as a mean learned leak of 0.12–0.13 implies. BPTT makes the field **usable** as a memory. It does | |
| not by itself give the long-range (prompt-length) memory: on this templated corpus the optimiser does not ask for it. | |
| On the 2 KB text, where it is necessary, it does learn it (938 bytes verbatim needs > 30-byte dependencies). | |
| ## 4. Step 3: long full-corpus BPTT runs (running) | |
| Driver: `scripts/train_bptt.sh` (chunks of 8k steps, one new checkpoint per chunk, progress row every 2 chunks: | |
| val 2048 normal / field-alone, train stats, 3 probe chats unseeded + seed 1). Recipe = fieldsat recipe + `--efference-every 0 | |
| --field-bptt K --field-leak 0.05 --field-leak-learnable`, LR = SLR. Stream position: `(CUR−496000) mod 1,000,452` | |
| transitions. This is the same text position as the legacy skip without replaying 4.5M packets (≈5 min per chunk). | |
| The 256-period phase feature differs by a constant offset, which does not matter for new branches. About 120 ticks/s, | |
| about 130 s per 8k chunk. | |
| Arms (all started at the same time and run in parallel, 1 core each; box times PDT, i.e. ET−3): | |
| | arm | init | K | LR=SLR | dir | status at the 3 h mark (05:42 PDT) | | |
| |---|---|---|---|---|---| | |
| | **fresh (best)** | fresh init (`checkpoints/byte_tick_bptt_fresh/atom_native_step_0_fresh_init.pt`, trainer constructor, seed 20260913) | 64 | 2e-3 | `checkpoints/byte_tick_bptt_fresh/` suffix `_d32_bptt_fresh` | **running**, 512k steps | | |
| | fresh K128 | same fresh init | 128 | 2e-3 | `checkpoints/byte_tick_bptt_fresh_k128/` | **running**, 336k steps | | |
| | main (from decfix) | pilotA 5044k = decfix 5012k + 32k BPTT steps (`checkpoints/byte_tick_bptt_pilotA/`) | 64 | 1e-3 | `checkpoints/byte_tick_bptt/` suffix `_d32_ms1M_fieldsat_decfix_bptt` | stopped at 5444k (plateau ~0.66–0.73; dominated by fresh) | | |
| | from decfix, LR 2e-3 | same pilotA 5044k | 64 | 2e-3 | `checkpoints/byte_tick_bptt_lr2e3/` | stopped at 5444k (worse than main) | | |
| Val 2048 normal / field-alone (from `bptt*_progress.tsv`; one row every 16k steps; rows fluctuate ±0.05): | |
| | steps of BPTT | fresh K64 | fresh K128 | from decfix, LR 1e-3 | from decfix, LR 2e-3 | | |
| |---|---|---|---|---| | |
| | 0 | (random) | (random) | 1.287 / 2.061 (decfix 5012k) | same | | |
| | 16k | 1.496 / 2.435 | 1.606 / 2.762 | 1.017 / 2.584 (5060k, incl. pilot) | 1.128 / 2.456 | | |
| | 96k | 0.941 / 1.253 | 1.021 / 1.454 | 0.782 / 1.674 (5188k) | 0.812 / 1.540 | | |
| | 192k | 0.754 / 0.915 | 0.766 / 1.053 | 0.742 / 1.446 (5252k) | 0.781 / 1.295 | | |
| | 320k | 0.562 / 0.676 | 0.624 / 0.831 | 0.658 / 1.186 (5364k) | 0.675 / 1.054 | | |
| | 400–512k | **0.478–0.531 / 0.56–0.64** (496k: 0.478 / 0.562) | – | 0.66–0.73 / 1.14–1.25 (5444k) | 0.76–0.80 / 1.09–1.14 | | |
| Learned leak mean (all modes): fresh 0.05 → 0.155, main 0.05 → 0.098. Cap fraction 0.000, no NaN/Inf, all chunks finite. | |
| Wall time per 8k chunk ≈ 130 s (≈ 120 ticks/s). | |
| Final measurements (`measure_bptt_final.json`, `diag_bptt_final.json`): | |
| | ckpt | val 2048 | field-alone | judge NLL (fixed decode) | doubled punct (24 chats) | KL lag 8 / 12 / 16 / 24 | grad lag 8 / 16 / 32 | | |
| |---|---|---|---|---|---|---| | |
| | fieldsat 4996k (reference) | 1.247 | 2.103 | 1.37 | 0 | 0.081 / 0.057 / 0.008 / 0.0004 | 0 / 0 / 0 | | |
| | main 5412k (from decfix) | 0.654 | 1.185 | 1.24 | 0 | 0.115 / 0.145 / 0.028 / 0.003 | 0.042 / 0.016 / 0.001 | | |
| | **fresh 448k** | **0.570** | **0.683** | **0.88** | 0 | 0.049 / 0.035 / 0.010 / 0.0007 | 1.73 / 0.94 / 0.053 | | |
| | GRU 124k (ROOT_CAUSE) | 0.099 | | 0.41 (samples) | | | | | |
| The fresh run never saw efference, so it never learned the copy rule: even the *legacy* decode gives no junk on it | |
| (judge 0.80, 0 doubled punctuation). | |
| Exact probe chats, fresh 448k (fixed decode; unseeded draw, then seeds 1/2/3; first line shown in full, rest cut at 100 bytes): | |
| * Bonjour, comment ça va ? → `Avec plaisir, continuons.` | |
| * Qui es-tu ? → `On mairdi da de tu fait.\nUtilisateur: Tu ais demps a.je te t'éccor ?\nAssistant: Oui, je t'écco` | |
| * Il était une fois → `Bonne question, je vais réfléchir.\nUtilisateur: Tu veux qu'on recommence ?\nAssistant: Je suis ` | |
| * seed 1: `Oui, tout à fait.\nUtilisateur: Tu vis as lusisav ex unf ?…` / `Oui, tout à fait.\nUtilisateur: Tu vis as cconter ça ?…` / `Bonne question, je vais réfléchir.\nUtilisateur: Tu vis d'accord ?…` | |
| * seed 2: `Oui, tout à fait.\nUtilisateur: Tu es d'accord ?\nAssistant: Je suis d'accord avec toi.` (identical for all 3 prompts) | |
| * seed 3: `On paraisin de mat ça ?` / `On m'appelle his quelle une sist ron ?\nAssistant: Je suis d'accord avec toi.…` / `D'accord, je t'éccoute.\nUtilisateur: Qu'est-ce que tu proposes ?\nAssistant: Bonne question, je …` | |
| Main 5412k (from decfix): `Peut-être ce tu fait.` / `Bonne quest-tu de j'approposes ?\nAssistant: Je suis d'accord avec toi.…` / | |
| `Mit à fait.\nUtilisateur: Tu es d'accord ?\nAssistant: Monti, tout à fait.…`; seed 3 gives the same | |
| `J'ai précoute.\nUtilisateur: Et toi, qu'en penses-tu ?…` for all 3 prompts. | |
| Reading: many first lines are now exact corpus replies, and the model continues the dialogue format (it writes the next | |
| `Utilisateur:` turn itself). Replies do not depend on the prompt (same seed → often the same reply for all 3 prompts), | |
| and there is still word salad inside turns. It is not fluent. | |
| Monitoring / restart (runs are resumable: re-running resumes from the newest checkpoint in `OUT`): | |
| ``` | |
| # fresh K64 (driver pid 3132056; log logs/bptt_fresh/master.log, chunk logs logs/bptt_fresh/train_*_bptt.log) | |
| nohup env INIT=checkpoints/byte_tick_bptt_fresh/atom_native_step_0_fresh_init.pt START=0 BASE=0 K=64 LEAK=0.05 LEARN=1 LR=2e-3 \ | |
| OUT=checkpoints/byte_tick_bptt_fresh SUF=d32_bptt_fresh TSV=docs/artifacts/decfix_bptt/bptt_fresh_progress.tsv \ | |
| LOGD=logs/bptt_fresh bash scripts/train_bptt.sh >> logs/bptt_fresh/master.log 2>&1 < /dev/null & | |
| # fresh K128 (driver pid 3149305; logs/bptt_fresh_k128/): same with K=128 OUT=checkpoints/byte_tick_bptt_fresh_k128 | |
| # SUF=d32_bptt_fresh_k128 TSV=docs/artifacts/decfix_bptt/bptt_fresh_k128_progress.tsv LOGD=logs/bptt_fresh_k128 | |
| # main from decfix (stopped; resume): INIT=checkpoints/byte_tick_bptt_pilotA/atom_native_step_5044000_d32_ms1M_fieldsat_decfix_bptt_pilotA.pt | |
| # START=5044000 K=64 LEAK=0.05 LEARN=1 LR=1e-3 bash scripts/train_bptt.sh >> logs/bptt/master.log | |
| # stop: kill <driver pid> $(pgrep -P <driver pid>) | |
| ``` | |
| END_TARGET defaults to START + 2,000,000 steps (≈ 9 h per arm), so the running arms will keep going unless stopped. | |
| ## 5. Verdict | |
| * **The decode bug is fixed properly** (automatic in byte-tick, exact escape flag, bit-identical tests). ATOM's | |
| output went from punctuation and space floods to French-looking words and corpus phrases. That is a real, free improvement. | |
| * **The field is now trained through time and it works as a memory.** Decisive test passed (938 / 648 bytes vs 10), | |
| gradient reaches past atoms, no instability. On the full corpus, val CE fell from the 1.25 plateau to ~0.5 | |
| (fresh) in 3 h and was still falling slowly. Field-alone went 2.10 → ~0.6. Fresh init beats continuing from | |
| the decfix checkpoint. | |
| * **Not fluent, not prompt-aware.** CE ~0.5 is at byte 4-gram level, versus 0.10 for a 124k GRU. Chats | |
| are corpus fragments stitched with word salad. The same seed often gives the same reply for different prompts. | |
| The measured context is ~8–16 bytes, not the ≥ 30 bytes needed to condition on the question. | |
| * Next levers (not done): (1) let the learnable leak reach smaller values, e.g. a per-mode init ladder from | |
| 0.005 to 0.3, or a penalty/cap; the optimiser drifts λ up; (2) longer K for the corpus run (K=128 arm is running); | |
| (3) a batched field (B sequences) for 10–30× throughput; (4) more diverse data (26 KB of unique text caps what any model can say). | |
| ## Artifact index (`docs/artifacts/decfix_bptt/`) | |
| | file | content | | |
| |---|---| | |
| | `dfb_common.py` | shared helpers: val 2048, perturbation KL, gradient-by-lag, chat samples, doubled-punct rate, GRU judge (reuses `root_cause/`) | | |
| | `measure_decfix.py`, `measure_4996k.json`, `measure_decfix_5012k.json`, `measure_bptt_interim.json`, `measure_bptt_final.json` | step-1 / step-3 measurements, fixed vs legacy decode, all chat samples | | |
| | `memorize_bptt.py`, `mem_*.json` | 2 KB memorisation runs (log, regen head, args) | | |
| | `diag_mem.py`, `diag_mem_{a,b,c,d}.json`, `diag_decfix_5012k.json`, `diag_bptt_*.json` | per-position TF errors, regeneration from 20 prefixes, perturbation KL, gradient-by-lag | | |
| | `tsv_bptt.py`, `bptt*_progress.tsv`, `pilotA_progress.tsv` | step-3 progress rows | | |
| | `logs/` | copies of the run logs | | |
| ### Addendum: latest progress rows at commit time (05:52 PDT / 08:52 ET) | |
| - fresh K64: 512k 0.494/0.578, 528k 0.504/0.585 (best is still 0.478/0.562 at 496k, so it is plateauing). Still running; driver PID 3132056. | |
| - fresh K128: 352k 0.640/0.816, 368k 0.580/0.780 (still improving). Still running; driver PID 3149305. | |
| - Chats: still prompt-independent formulaic replies with word salad. **Not fluent.** | |