ATOM (atom-ai): code snapshot (100fae0 + batched-train), README with measured numbers, curated checkpoints
9f1076a verified | # last_atom dropout — 2026-09-28 | |
| English. Numbers. **No fluency claim.** | |
| ## Problem | |
| Under `field_obligatory_hard` + `--last-atom-readout` the byte-tick hard logits are | |
| ``` | |
| logits = obl·α_MLP(a_hat) + 0.3·frozen_JL(a_hat) + payload + S·last_atom(last_byte, prev_byte, φ) | |
| ``` | |
| Audit: teacher-forced CE/val is carried by the `last_atom` term (bigram + phase | |
| shortcut); the field (α) path contributes little. Measured here on the b4828 start tip | |
| (4868k): val CE **0.401** normal vs **6.12** with `last_atom` zeroed at eval — worse than | |
| uniform (ln 256 = 5.55). The field path alone is not even a unigram model. In free-run | |
| chat the model falls into attractors (`Bonne`, `Oui,,`, `Peut--`). | |
| Already falsified: fixed/adaptive `last_atom_scale` S (letter soup), `field_hard_couple` | |
| S=0.25/1.0 (W barely trains), free-run-aux (brief diversity, then collapse). | |
| ## Lever (opt-in, default OFF) | |
| ``` | |
| --last-atom-dropout P # default 0.0 = exact current behavior (no RNG draw) | |
| ``` | |
| Training only (`module.training` **and** grad enabled). Each surface forward is one | |
| byte-tick position. With probability P that position's `la_byte` term is set to 0 | |
| (per-position Bernoulli; no inverted-dropout rescale). On those positions CE has to be | |
| carried by α-MLP + frozen JL. The flag is ignored in eval, generation, validation, and | |
| no-grad predictions. So efference's own-byte pick still uses the full model, the same way | |
| generation does. | |
| * `src/atom_native.py`: `AtomSurfaceHead.last_atom_dropout` (saved in checkpoint config, restored on | |
| load as provenance). Also adds runtime-only `last_atom_force_off` (not saved), a diagnostic | |
| that zeroes `la_byte` in any mode. | |
| * `tools/run_atom_native.py`: `--last-atom-dropout`. CLI wins after `--resume`, using the same | |
| pattern as `field_loss_weight` and `last_atom_scale`. It is recorded in `atomization_stats` / `run_metrics`. | |
| * `test/test_last_atom_dropout.py`: | |
| - P=0 gives bit-identical logits and draws no global RNG. | |
| - P=1 in train mode gives exactly the no-last_atom logits. | |
| - Eval and no-grad are unaffected. | |
| - P=0.5 is a per-position Bernoulli (~50%). | |
| - `force_off` works. | |
| - The value survives a checkpoint roundtrip. | |
| * Diagnostics: `tools/eval_last_atom_ablation.py` rebuilds the exact stream-ring val of | |
| `run_atom_native` and reports CE normal vs `last_atom`-zeroed. `tools/chat_seeds_probe.py` | |
| runs the coherence-probe prompts/params with fixed torch seeds. | |
| ## Probe (+8k, P=0.5) | |
| Resume: `atom_native_step_4868000_d32_ms1M_lrd_b4828.pt`. This was the newest b4828 tip at probe start, | |
| 2026-09-28 13:45 ET. The recipe was the exact live one from `scripts/train_until_coherent.sh`: | |
| `stream-skip 4372000`, d32, `max-span-bytes 1`, `field-obligatory-hard`, `no-enable-merge`, | |
| `no-payload-copy`, `last-atom-readout`, `efference-every 10`, `atom-flush-every 64`, | |
| `episode-length 8000 --episode-reset`, `field-loss-weight 0.05`, contrast 0, LR 1.5e-5 / | |
| SLR 3.6e-5, seed 20260913, `--allow-parallel-train`. The only addition was **`--last-atom-dropout 0.5`**. | |
| Output: `checkpoints/byte_tick_ladrop/atom_native_step_4876000_d32_ms1M_lrd_b4828_ladrop.pt` | |
| (separate dir, so the live `run_metrics.json` was never touched). Baseline: the live | |
| `atom_native_step_4876000_d32_ms1M_lrd_b4828.pt` (same start, same data, P=0). | |
| ### Val (stream ring, 256 transitions — identical window for all three rows) | |
| | ckpt | val loss (normal) | byte CE normal | byte CE **last_atom zeroed** | | |
| |------|------------------:|---------------:|-----------------------------:| | |
| | start 4868k (b4828) | 0.4010 | 0.3990 | **6.120** | | |
| | live 4876k (P=0) | 0.4024 | 0.4003 | **6.195** | | |
| | ladrop 4876k (P=0.5) | 0.5363 | 0.5309 | **3.399** | | |
| Reference on the same window, from corpus counts: unigram CE ≈ 3.24, bigram CE ≈ 1.77, uniform 5.55 | |
| (`val_window_baselines.json`). The eval script reproduces `run_metrics` val exactly (0.4010 / | |
| 0.4024 / 0.5363). | |
| Read it plainly: | |
| * The field path alone went from worse-than-uniform (6.12) to about unigram level (3.40) in 8k | |
| steps. It learned the byte marginal. It did not learn context: 3.40 is still far above the | |
| bigram's 1.77. | |
| * Normal val got worse, 0.401 → 0.536 (+0.13), because `last_atom` now only has half the | |
| positions to fit. | |
| * The live P=0 run moved field-alone CE the other way (6.12 → 6.20). Without pressure, the field path decays. | |
| ### Chat | |
| The coherence probe CLI takes only the **first line** of the response (`sed … | head -1`), with | |
| unseeded sampling, T=0.8, top-k 12, `--no-role-prime`, 96 packets, max-length 100: | |
| | prompt | live 4876k | ladrop 4876k | | |
| |--------|-----------|--------------| | |
| | Bonjour, comment ça va ? | `Bonne` | `Bonne` | | |
| | Qui es-tu ? | `Oui,,` | `Bonne` | | |
| | Il était une fois | `Oui,,` | `Bonne` | | |
| Seeded probe (seeds 1,2,3 × T 0.6/0.8/1.0 × 3 prompts = 27 full replies per ckpt, | |
| `chat_seeds_*.txt`): | |
| | ckpt | unique replies /27 | mean distinct replies across the 3 prompts (same T, seed) | first words | | |
| |------|------------------:|-----------------------:|-------------| | |
| | start 4868k | 23 | 2.56 | Oui,, 9 / Je 9 / Peut-- … | | |
| | live 4876k | 21 | 2.56 | Je 16 / Peut-- … | | |
| | ladrop 4876k | 15 | **1.67** | **Je 27/27** | | |
| Example, T=0.8 seed=1, same reply for all three prompts on ladrop: | |
| `'Je …\nsoirends …\nAsssistant:::::::::::::::::::O \nuurrraais'`. | |
| Chat got **worse**. Replies depend less on the prompt and always open with `Je`, which fits a field | |
| path that learned only the unconditional marginal and now biases every opening. | |
| ### Other observations | |
| * Train-log `field=96.0 rms=3.0 scale≈0.999` on both runs: α sits pinned at the | |
| `field_max_rms=3.0` clamp for most of each episode. A saturated field can carry little | |
| position-specific information. This is a plausible reason the field path tops out at unigram. | |
| * Median train loss: live 0.039, ladrop 2.98 (half the positions have no shortcut). | |
| ## Verdict | |
| P=0.5 for 8k steps is a **technical PASS, not a fluency lever yet**. It proves the field path | |
| can be pushed to carry probability mass (field-alone CE 6.12 → 3.40), but what it learned | |
| is the marginal, not context. Chat is less prompt-conditioned. Normal val is +0.13 worse. | |
| ## Next (not done here) | |
| 1. Longer P=0.5 run (≥40k) tracking **field-alone CE**. If it stays near unigram (~3.2), the | |
| α state holds no usable context, and the bottleneck is upstream (saturated field at the RMS | |
| clamp, α features), not the readout. | |
| 2. P=1.0 for 8k steps is a clean "field-only" ceiling test. It matches P=0.5 on field-alone CE if the field | |
| is capacity-limited. Do not chat-evaluate it with `last_atom` on (train/eval mismatch). | |
| 3. Only if field-alone CE clearly beats bigram (≲1.8): consider a stacked recipe (P anneal | |
| 0.5→0.2). | |
| ## Artifacts | |
| `docs/artifacts/last_atom_dropout/`: `val_ablation.{json,txt}`, `val_window_baselines.json`, | |
| `chat_seeds_{start_4868000,live_4876000,ladrop_4876000}.{json,txt}`, | |
| `gen_coherent_4876000_{b4828,ladrop}.txt`, `run_metrics_*`, `train_ladrop.log`, | |
| `yield_to_live.py` + `measure.sh` (see CPU note). | |
| ## CPU note (parallel probes) | |
| The live trainer runs torch with 8 OMP threads that busy-wait. Any other busy process on | |
| one core cut live throughput from about 77 to about 2 steps/s (measured with SIGSTOP on and off). The probe | |
| and evals were therefore run single-threaded (`OMP_NUM_THREADS=1`) under | |
| `yield_to_live.py`, which SIGSTOPs only the probe's own process group while the live chunk is past its | |
| single-threaded stream-skip phase. The live loop was never signalled. | |