docs/LAST_ATOM_DROPOUT.md

last_atom dropout — 2026-09-28

English. Numbers. No fluency claim.

Problem

Under field_obligatory_hard + --last-atom-readout the byte-tick hard logits are

logits = obl·α_MLP(a_hat) + 0.3·frozen_JL(a_hat) + payload + S·last_atom(last_byte, prev_byte, φ)

Audit: teacher-forced CE/val is carried by the last_atom term (bigram + phase shortcut); the field (α) path contributes little. Measured here on the b4828 start tip (4868k): val CE 0.401 normal vs 6.12 with last_atom zeroed at eval — worse than uniform (ln 256 = 5.55). The field path alone is not even a unigram model. In free-run chat the model falls into attractors (Bonne, Oui,,, Peut--).

Already falsified: fixed/adaptive last_atom_scale S (letter soup), field_hard_couple S=0.25/1.0 (W barely trains), free-run-aux (brief diversity, then collapse).

Lever (opt-in, default OFF)

--last-atom-dropout P      # default 0.0 = exact current behavior (no RNG draw)

Training only (module.training and grad enabled). Each surface forward is one byte-tick position. With probability P that position's la_byte term is set to 0 (per-position Bernoulli; no inverted-dropout rescale). On those positions CE has to be carried by α-MLP + frozen JL. The flag is ignored in eval, generation, validation, and no-grad predictions. So efference's own-byte pick still uses the full model, the same way generation does.

• src/atom_native.py: AtomSurfaceHead.last_atom_dropout (saved in checkpoint config, restored on load as provenance). Also adds runtime-only last_atom_force_off (not saved), a diagnostic that zeroes la_byte in any mode.
• tools/run_atom_native.py: --last-atom-dropout. CLI wins after --resume, using the same pattern as field_loss_weight and last_atom_scale. It is recorded in atomization_stats / run_metrics.
• test/test_last_atom_dropout.py:
  • P=0 gives bit-identical logits and draws no global RNG.
  • P=1 in train mode gives exactly the no-last_atom logits.
  • Eval and no-grad are unaffected.
  • P=0.5 is a per-position Bernoulli (~50%).
  • force_off works.
  • The value survives a checkpoint roundtrip.
• Diagnostics: tools/eval_last_atom_ablation.py rebuilds the exact stream-ring val of run_atom_native and reports CE normal vs last_atom-zeroed. tools/chat_seeds_probe.py runs the coherence-probe prompts/params with fixed torch seeds.

Probe (+8k, P=0.5)

Resume: atom_native_step_4868000_d32_ms1M_lrd_b4828.pt. This was the newest b4828 tip at probe start, 2026-09-28 13:45 ET. The recipe was the exact live one from scripts/train_until_coherent.sh: stream-skip 4372000, d32, max-span-bytes 1, field-obligatory-hard, no-enable-merge, no-payload-copy, last-atom-readout, efference-every 10, atom-flush-every 64, episode-length 8000 --episode-reset, field-loss-weight 0.05, contrast 0, LR 1.5e-5 / SLR 3.6e-5, seed 20260913, --allow-parallel-train. The only addition was --last-atom-dropout 0.5. Output: checkpoints/byte_tick_ladrop/atom_native_step_4876000_d32_ms1M_lrd_b4828_ladrop.pt (separate dir, so the live run_metrics.json was never touched). Baseline: the live atom_native_step_4876000_d32_ms1M_lrd_b4828.pt (same start, same data, P=0).

Val (stream ring, 256 transitions — identical window for all three rows)

ckpt	val loss (normal)	byte CE normal	byte CE last_atom zeroed
start 4868k (b4828)	0.4010	0.3990	6.120
live 4876k (P=0)	0.4024	0.4003	6.195
ladrop 4876k (P=0.5)	0.5363	0.5309	3.399

Reference on the same window, from corpus counts: unigram CE ≈ 3.24, bigram CE ≈ 1.77, uniform 5.55 (val_window_baselines.json). The eval script reproduces run_metrics val exactly (0.4010 / 0.4024 / 0.5363).

Read it plainly:

• The field path alone went from worse-than-uniform (6.12) to about unigram level (3.40) in 8k steps. It learned the byte marginal. It did not learn context: 3.40 is still far above the bigram's 1.77.
• Normal val got worse, 0.401 → 0.536 (+0.13), because last_atom now only has half the positions to fit.
• The live P=0 run moved field-alone CE the other way (6.12 → 6.20). Without pressure, the field path decays.

Chat

The coherence probe CLI takes only the first line of the response (sed … | head -1), with unseeded sampling, T=0.8, top-k 12, --no-role-prime, 96 packets, max-length 100:

prompt	live 4876k	ladrop 4876k
Bonjour, comment ça va ?	Bonne	Bonne
Qui es-tu ?	Oui,,	Bonne
Il était une fois	Oui,,	Bonne

Seeded probe (seeds 1,2,3 × T 0.6/0.8/1.0 × 3 prompts = 27 full replies per ckpt, chat_seeds_*.txt):

ckpt	unique replies /27	mean distinct replies across the 3 prompts (same T, seed)	first words
start 4868k	23	2.56	Oui,, 9 / Je 9 / Peut-- …
live 4876k	21	2.56	Je 16 / Peut-- …
ladrop 4876k	15	1.67	Je 27/27

Example, T=0.8 seed=1, same reply for all three prompts on ladrop: 'Je …\nsoirends …\nAsssistant:::::::::::::::::::O \nuurrraais'. Chat got worse. Replies depend less on the prompt and always open with Je, which fits a field path that learned only the unconditional marginal and now biases every opening.

Other observations

• Train-log field=96.0 rms=3.0 scale≈0.999 on both runs: α sits pinned at the field_max_rms=3.0 clamp for most of each episode. A saturated field can carry little position-specific information. This is a plausible reason the field path tops out at unigram.
• Median train loss: live 0.039, ladrop 2.98 (half the positions have no shortcut).

Verdict

P=0.5 for 8k steps is a technical PASS, not a fluency lever yet. It proves the field path can be pushed to carry probability mass (field-alone CE 6.12 → 3.40), but what it learned is the marginal, not context. Chat is less prompt-conditioned. Normal val is +0.13 worse.

Next (not done here)

1. Longer P=0.5 run (≥40k) tracking field-alone CE. If it stays near unigram (~3.2), the α state holds no usable context, and the bottleneck is upstream (saturated field at the RMS clamp, α features), not the readout.
2. P=1.0 for 8k steps is a clean "field-only" ceiling test. It matches P=0.5 on field-alone CE if the field is capacity-limited. Do not chat-evaluate it with last_atom on (train/eval mismatch).
3. Only if field-alone CE clearly beats bigram (≲1.8): consider a stacked recipe (P anneal 0.5→0.2).

Artifacts

docs/artifacts/last_atom_dropout/: val_ablation.{json,txt}, val_window_baselines.json, chat_seeds_{start_4868000,live_4876000,ladrop_4876000}.{json,txt}, gen_coherent_4876000_{b4828,ladrop}.txt, run_metrics_*, train_ladrop.log, yield_to_live.py + measure.sh (see CPU note).

CPU note (parallel probes)

The live trainer runs torch with 8 OMP threads that busy-wait. Any other busy process on one core cut live throughput from about 77 to about 2 steps/s (measured with SIGSTOP on and off). The probe and evals were therefore run single-threaded (OMP_NUM_THREADS=1) under yield_to_live.py, which SIGSTOPs only the probe's own process group while the live chunk is past its single-threaded stream-skip phase. The live loop was never signalled.