ATOM d64 leak-frozen 3.9M + honest README (not fluent; operable field model)

README.md

@@ -1,194 +1,104 @@
----
-language:
-- fr
-- en
-library_name: pytorch
-pipeline_tag: text-generation
-tags:
-- byte-level
-- no-tokenizer
-- recurrent
-- toroidal-field
-- cpu-inference
-- research
-- experimental
-- french
-- english
-- bilingual
+# ATOM (atom-ai)
+
+Experimental **byte-level toroidal-field** language model. Not a Transformer.
+
+**Status (2026-10-01):** research code. ATOM does **not** produce coherent answers yet. Weights are alive; generation is still salad with French function words. Read this as a lab notebook, not a product card.
+
+Repo on the Hub: [thefinalboss/atom-ai](https://huggingface.co/thefinalboss/atom-ai)  
+Code lineage also on GitHub: [AFKmoney/atom-ai](https://github.com/AFKmoney/atom-ai)
+
 ---
 
-# ATOM (atom-ai): a bilingual French + English byte-level toroidal-field language model (research, not yet fluent)
-
-> **Status: experimental research code. ATOM does NOT produce coherent text yet, in either language.**
-> ATOM is designed as a **bilingual French + English** model. The checkpoints published here were trained on **French data only**
-> and generate French-looking text: real words, corpus phrases and names mixed with pseudo-words.
-> English is being added in the ongoing GPU run; no English results are available yet.
-> Replies are mostly independent of the prompt. Read the numbers below as a lab notebook, not a product claim.
-
-ATOM is an independent attempt to build a small language model **without a Transformer**. It has no attention,
-no BPE, no Hugging Face tokenizer and no GPT-2 components. Text goes in and out as raw UTF-8 **bytes**, which a
-custom byte-level front end called the *Atomizer* turns into packets. A persistent **toroidal field** carries the
-state from one byte to the next. The target is coherent **French and English** with **CPU inference**; training can run on CPU or GPU.
-
-## Bilingual by design (French + English)
-
-* **No tokenizer change is needed for two languages.** The Atomizer works on raw UTF-8 bytes, so French (including accents such as
-  `é`/`ç`, which are 2-byte UTF-8 sequences) and English go through exactly the same input and output path (256 byte values). There is no
-  vocabulary to extend and no per-language component.
-* **Main GPU training corpus: bilingual FR + EN.** The target is about 50 % French and 50 % English, with about 20 % dialogue within each language:
-  * Wikipedia (FR and EN),
-  * Project Gutenberg books (FR and EN),
-  * OpenSubtitles-style dialogue, formatted as `Utilisateur: … / Assistant: …` in French and `User: … / Assistant: …` in English.
-* **Builders.** `scripts/fr_big/download.sh` + `scripts/fr_big/build_fr_big.py` build the French part. The English build is being added:
-  `scripts/fr_big/download_en.sh` fetches the English sources, and `build_fr_big.py --lang en` applies the same cleaning with English
-  rules and the `User:` tag. The combined bilingual corpus is not included in this repo (multi-GB), and no English corpus sizes are reported yet.
-* **What is published today is French-only.** Every checkpoint in `checkpoints/` was trained on French data only (the template dialogue corpus
-  and the 7.4 MB French mix described below). All numbers and samples in this README are French. English results will be added once the GPU run produces them.
-
-This repository contains the full code, the documentation and measurement artifacts, a small French-only training corpus
-(7.4 MB) with held-out validation sets, and a curated set of checkpoints (see [`checkpoints/README.md`](checkpoints/README.md)).
-
-## Architecture in plain words
-
-* **Atomizer (tokenizer replacement).** `src/io/atomizer.py`. Every UTF-8 byte is one "tick". The Atomizer turns the byte
-  plus a small rolling context into a 320-dim packet feature: byte identity, boundary label (`max_span` / `newline` in
-  byte-tick mode), and a context sketch. There is no vocabulary to learn and no merges. The output alphabet is the 256 byte values.
-* **Atoms and the field.** `src/atom_native.py`, `src/toroidal/`. Each packet is compiled into an *atom* (an embedding).
-  Atoms are injected into a complex field α over 32 modes arranged on a torus. At every tick the field is leaked
-  (per-mode, learnable leak), receives the new atom, and is advanced by an RK4 step with phase-sync coupling between
-  neighbouring modes. An RMS controller keeps it bounded. The field is persistent across the whole stream (reset every 8,000 ticks).
-* **Readout.** The next byte is predicted from the field α (MLP + frozen JL projection) plus a small `last_atom`
-  readout (bigram-like). "Field-alone" numbers below zero that `last_atom` path, so they measure what the field itself carries.
-* **Training.** Next-byte cross-entropy (nats/byte) with **truncated BPTT through the field** (`--field-bptt K`, K = 64 or 128).
-  `tools/run_atom_native.py` trains one stream on CPU. `tools/train_batched.py` trains B streams in lockstep on CPU or CUDA.
-* **Size.** The current d=32 checkpoints have **479,548 parameters** (266,828 trainable). Inference is a plain Python/PyTorch
-  loop on CPU (about 2 s for one short reply on this machine, one thread).
-
-## What was found (root cause) and fixed
-
-The full reports are in `docs/ROOT_CAUSE.md`, `docs/DECODE_FIX_AND_BPTT.md`, `docs/MIX_DATA_LONG_MEMORY.md` and `docs/BATCHED_TRAINING.md`.
-
-1. **Decode bug.** At generation time, generated spaces and punctuation received a different Atomizer boundary label than in
-   training. An off-by-one "efference" loss had also taught the model to *copy* such packets. Result: output floods
-   such as `Oui,,`, `Peut--`, `Je` + 94 spaces. With the fix, the teacher-forced and generation logits match exactly (max |Δlogit| = 0.0).
-   Doubled punctuation in 24 probe chats went from 7 to 0, and a GRU-judge NLL of replies went from 9.69 to 1.37. (Fixed in commit fbea452, automatic in byte-tick mode.)
-2. **The field was never trained as a memory.** The field was detached every tick and injected under `no_grad`, so the gradient to any
-   past byte was exactly 0. Together with a fixed leak of 0.3, that left an effective context of **2–3 bytes**: validation CE plateaued
-   at **1.247 nats/byte**, between a byte bigram (1.950) and a byte trigram (0.959). Data and parameter count were ruled out:
-   on the same data a 123,904-parameter **GRU reaches 0.099**.
-3. **Fix: truncated BPTT through the field + learnable leak.** On a 2 KB memorisation test (greedy regeneration past a 32-byte prefix),
-   ATOM went from **10 bytes** (no-BPTT control; 10–13 with the old recipe) to **938 bytes** verbatim (K=128) and 648 (K=64). The GRU reference does 1,453.
-4. **Why memory is still short.** An exact per-tick Jacobian shows two causes: the learned leak drifts up (mean 0.18, e-fold about 13 ticks),
-   and phase-sync coupling pinned at its clamp drains slow modes (e-fold 13.4 → 8.0 ticks). An opt-in "slow block" lever was tested
-   and not adopted (`docs/MIX_DATA_LONG_MEMORY.md` §4).
-5. **Batched training for GPU.** The model gained a lockstep batched runtime (B streams). It is bit-identical to the single-stream path at B=1 (tested) and
-   produces checkpoints the CPU chat tool loads unchanged.
-
-## Measured numbers (all from files in this repo)
-
-Metric: next-byte cross-entropy in **nats/byte** on fixed validation windows (512-transition windows, state reset per window).
-Lower is better.
-
-### Baselines vs ATOM, template corpus (`docs/ROOT_CAUSE.md`, `docs/artifacts/root_cause/`)
-
-| model | val CE (2048-transition val) |
-|---|---|
-| ATOM old recipe @4996k (no BPTT) | 1.247 |
-| byte 2-gram / 3-gram / 5-gram | 1.950 / 0.959 / 0.293 |
-| GRU 1×128 (123,904 params) | **0.099** |
-| ATOM, fresh init + BPTT K=64, best row (1232k) | **0.3560** (field-alone 0.3945) |
-| ATOM, fresh init + BPTT K=128, best row (1104k) | **0.3954** (field-alone 0.4653) |
+## What it is
 
-Fresh BPTT rows fluctuate between about 0.36 and 0.46 late in training (`docs/artifacts/decfix_bptt/bptt_fresh_progress.tsv`,
-`bptt_fresh_k128_progress.tsv`). Caveat: this corpus has only 588 unique template lines (26 KB of unique text), so these
-numbers mostly measure template memorisation and are not a sign of general French.
+Text in and out is raw UTF-8 **bytes**. There is no BPE, no Hugging Face tokenizer, no GPT-2 stack, no attention.
 
-### Mixed French corpus, true held-out sets (`docs/artifacts/mix_long_memory/bptt_k128_mix_progress.tsv`)
+1. **Atomizer** — each UTF-8 byte is one tick. Byte + a small rolling context → a packet (feature dim 320 in the current code). Alphabet = 256 byte values. French accents (`é`, `ç`) are just 2-byte UTF-8 sequences on the same path as English.
+2. **Field** — packets become atoms injected into a complex field `α` on a torus (`n_modes` modes). Each tick: leak, inject, RK4 step with phase-sync between neighbours, RMS cap. Field persists across the stream (episode reset every 8,000 ticks on the current GPU recipe).
+3. **Readout** — next-byte CE from the field (MLP + frozen JL) plus a small `last_atom` bigram-like term. Training can drop `last_atom` (`--last-atom-dropout 0.5`) so the field has to carry CE. Eval/generation use the full readout.
+4. **Train** — truncated BPTT through the field (`--field-bptt K`). Batched lockstep streams on CPU or CUDA (`tools/train_batched.py`). Inference is a Python/PyTorch loop; CPU works.
 
-The K128 run (from 1104k) continued on a 7.39 MB mix: 75 % public-domain Project Gutenberg French books and 25 % dialogue in
-`Utilisateur:/Assistant:` format. It is evaluated on held-out dialogue and held-out book text that never appear in the training stream.
+This is meant to be **operable**: keep the `.pt`, change a lever (leak, BPTT, decode), resume. Not “wipe and become a Transformer.”
 
-| step | held-out dialogue (normal / field-alone) | held-out varied books (normal / field-alone) |
-|---|---|---|
-| 1104k (before the mix) | 1.286 / 1.291 | 6.772 / 6.591 |
-| 1520k | 1.0818 / 1.1577 | 2.1145 / 2.1183 |
-| 1600k | 1.0212 / 1.0337 | **2.0581** / 2.0701 |
-| **1680k (latest measured row)** | **1.0053** / 1.0317 | 2.0600 / 2.0768 |
+---
+
+## What it is not
 
-About 2.06 nats/byte on unseen book text is weak, roughly byte-trigram level. The run was still training when this was uploaded
-(latest checkpoint 1720k, not yet evaluated).
+- Not fluent French or English.
+- Not a chat model. Teacher-force CE going down ≠ the free-run following the prompt.
+- Not the slop fork that adds `nn.MultiheadAttention` + a GPT-2 tokenizer. That is a different repo. ATOM’s main path does not import `transformers`.
 
-### Batched training throughput (`docs/BATCHED_TRAINING.md`, `docs/artifacts/batched_training/`)
+---
+
+## Live GPU run (this card’s current tip)
+
+Machine: 1× RTX 5090. Recipe name: **`bi_d64_leak05`**.
 
-| setting | throughput |
+| | |
 |---|---|
-| CPU, legacy single stream (1 thread) | 160 bytes per CPU-second (older estimate in ROOT_CAUSE: ~80 bytes/s wall) |
-| CPU batched B=32 / B=128 (1 thread) | 3,108 / 6,540 bytes per CPU-second = **×19 / ×41** |
-| GPU RTX 5090, batched K=128, B=1024 | ~85,900 bytes/s |
-| GPU RTX 5090, B=1536 / B=1792 | **~125,200 / ~144,800 bytes/s** (CUDA peak 21.5 / 25.1 GB) |
+| Size | `d_model=64`, `n_modes=64` |
+| Data | bilingual shards `data/[ef][nr]_big/train_*.txt` (~13.8B bytes on the box) |
+| Batch | `B=1920`, BPTT `K=256`, episode 8000, atom flush 64 |
+| Leak | **frozen at 0.05** (`--no-field-leak-learnable`) |
+| Other | last-atom dropout 0.5 train-only, merge off, payload copy off, field-loss 0.05 |
 
-GPU smoke run (`checkpoints/gpu_smoke_b512/progress.tsv`): B=512, initialised from the CPU checkpoint 1616k,
-4,000 ticks = 2.05 M bytes. Held-out val CE went **1.2957 → 1.2307** (ticks 1000 → 4000). Rows were logged every ~20 s, so the run took about 80 s
-(~38–40k training bytes/s; evaluation and samples are included in the wall time).
+### Why the leak was frozen
 
-## Real samples (verbatim, CPU)
+A previous continuation (`bi_d64_cont`) started with `--field-leak 0.05 --field-leak-learnable`. The learned leak drifted to **~0.35**. At that rate the field forgets the prompt in a handful of ticks. Generation collapsed to the same openings (`That th`, `Tout le…`) on every question. Train CE could still fall. Same class of bug as an earlier recipe that **detached** the field every tick (effective context 2–3 bytes).
 
-Produced for this upload with `tools/chat_atom_native.py --device cpu`. The chat tool wraps the prompt as `Utilisateur: … / Assistant:`.
+Cut at `atom_native_step_3200000_bi_d64_cont.pt`. Resume in a new dir with leak **fixed** at 0.05. Optimizer groups that held `field_leak_logit` were not restored (the parameter is gone). Model weights and batched stream state were.
 
-Latest mixed-corpus checkpoint (1720k), default sampling (T=0.7, top-k 8):
+After the cut, train loss jumped **1.71 → ~2.45** (expected: the field now keeps more past) and came back down.
 
-```
-Bonjour, comment ça va ?  →  Bonjusi. 
-                             Utilisateur: Tu vais compordu de la mois que peter l'avoist apes compre fait à préc
-Qui es-tu ?               →  Oui, ton parin, estau le plus lusers que t'étous chre du de l'avoit, mett de chalon qu'estion,
-```
+### Numbers on disk (UTC 2026-10-01)
 
-Same checkpoint, `--deterministic` (greedy): it falls into a loop for all three test prompts:
+| step | train CE (approx) | note |
+|---|---|---|
+| 3,200,000 | ~1.71 | last `bi_d64_cont` row before the cut; leak_mean ~0.35 |
+| 3,300,000 | ~2.19 | first `bi_d64_leak05` ckpt |
+| 3,900,000 | ~1.95 | latest uploaded tip when this README was written |
+| live | ~1.95 at ~3.97M / 7.2M target | still training |
 
-```
-Je suis d'accord ?
-Assistant: Je suis d'accord ?
-Assistant: Je suis d'accord ?
-```
+Held-out `val_ce` is **not** logged on this pass (`nan` in the TSV). Do not treat train CE as a fluency score.
 
-Fresh K64 checkpoint (1232k, template corpus), `--deterministic`:
+Throughput on the 5090 is on the order of **400k bytes/s** for this recipe.
 
-```
-Bonjour, comment ça va ?  →  Peut-être ce matin, on verra.
-Il était une fois         →  Pas trop, je préfère lire.
-                             Utilisateur: Tu veux qu'on recommence ?
-```
+---
 
-(These are exact corpus lines; the model ignores the prompt.)
+## Generation (honest)
 
-GPU smoke checkpoint (4000 ticks at B=512), default sampling:
+Decode lock used for probes and for the TSV sample column:
 
-```
-Qui es-tu ?  →  Peut-être à ce soilluise de luisir, le son préssi de du prerais l'ampe même précés pronti
-```
+- chat defaults: temperature **0.3**, top-k **2**, optional greedy
+- `speech_ok` is a **unique-prefix** gate (salad with four real letters no longer counts as speech)
+
+**3.2M (`bi_d64_cont`, leak still high), greedy lock:** all three probe prompts → `That th` (gate cuts at 8 bytes).
 
-Before the fixes (old recipe @4996k, from `docs/ROOT_CAUSE.md`): `Je` + 94 spaces / `Peut--ommends` / `Oui,`.
+**3.3M (`bi_d64_leak05`):**  
+`Je se conte que se avais serve ave aition de la prour la pour…`  
+Still not an answer. Regime changed off the single English stub.
 
-## Honest status
+**3.9M TSV samples (greedy lock):**  
+- Bonjour → `Je vous sur la premier de la provise de la projettes…`  
+- Qui es-tu → `Je vous seil et proput our le cour la mais de la mais…`  
+- Il était une fois → `Il n'est proposse de la proter de la provise…`
+
+French function words showed up. The prompt is still not what is being answered. Loops on `de la mais` / `de la provise`. That is the current truth.
+
+Batched training is **teacher-forced** next-byte CE. No scheduled sampling / free-run aux on this GPU path. That gap is still open.
+
+---
 
-* It is **not coherent**, in French or English. The published checkpoints have seen **no English training data**. On the template corpus it recites whole corpus lines but ignores the prompt. On the mixed corpus it produces
-  French-sounding word salad with some real words.
-* The measured context is still short: perturbation sensitivity is non-negligible out to about 12–16 bytes, and gradients die by about 64 ticks.
-* The model is tiny (d=32, ~0.48 M parameters) and has been trained on at most a few MB of distinct text.
+## Checkpoints
 
-## Roadmap
+Add-only sync. Old runs stay on the Hub.
 
-1. **Large bilingual FR + EN corpus on GPU** (ongoing): about 50/50 French/English, about 20 % dialogue in each language, trained with
-   `tools/train_batched.py` at large B. French part: `scripts/fr_big/` (French Wikipedia ≤ 4.2 GB and French Project Gutenberg ≤ 1.4 GB by default,
-   plus OpenSubtitles dialogue at a ~20 % share, i.e. roughly 7 GB). English part: being added (`download_en.sh`, `build_fr_big.py --lang en`).
-   English results are not in yet.
-2. **Bigger model**: d=64/128 (the code supports Net2Net-style growth, `tools/grow_checkpoint.py`).
-3. **Parallel-scan field**: the field recurrence is linear in α (`docs/MIX_DATA_LONG_MEMORY.md` §3), which should allow a parallel scan
-   over time instead of a Python loop per tick.
-4. **Longer memory**: keep slow modes out of the phase-sync drain and cap the leak; use a recall curriculum.
-5. **A fair benchmark against a small Transformer** (and the GRU baseline) at equal parameters, data and compute.
+- Current tip: `checkpoints/bi_d64_leak05/atom_native_step_3900000_bi_d64_leak05.pt`  
+  (+ `checkpoints/bi_d64_leak05/progress.tsv`)
+- Previous continuation: `checkpoints/gpu_bi_d64_cont/` (sync copies under the `gpu_` prefix)
+- Older French-only / d32 work: see `checkpoints/` and `docs/`
 
-## Quick start (CPU inference)
+Load with `tools/chat_atom_native.py`. Config in the `.pt` must match (`d_model`, `n_modes`, byte-tick).
 
 ```bash
 git lfs install
@@ -198,33 +108,24 @@ python3 -m venv .venv
 .venv/bin/pip install --index-url https://download.pytorch.org/whl/cpu torch
 .venv/bin/pip install -r requirements.txt
 PYTHONPATH=. .venv/bin/python tools/chat_atom_native.py \
-  --checkpoint checkpoints/byte_tick_bptt_k128_mix/atom_native_step_1720000_d32_bptt_k128_mix.pt \
-  --prompt "Bonjour, comment ça va ?" --device cpu
+  --checkpoint checkpoints/bi_d64_leak05/atom_native_step_3900000_bi_d64_leak05.pt \
+  --prompt "Bonjour, comment ça va ?" --device cpu --deterministic
 ```
 
-Use `requirements.txt` (torch + numpy) rather than `pip install -e .`. The legacy `pyproject.toml` lists `transformers`/`datasets`,
-which ATOM does not import anywhere. More details, including how to resume training on CPU or GPU, are in
-[`checkpoints/README.md`](checkpoints/README.md).
+Do not `pip install -e .` if that pulls `transformers` from the leftover `pyproject.toml`. ATOM does not use that stack on the main path.
 
-## Repository layout
+---
 
-| path | content |
-|---|---|
-| `src/atom_native.py`, `src/toroidal/`, `src/io/atomizer.py` | model, field dynamics, Atomizer |
-| `src/atom_batched.py`, `src/io/batched_stream.py` | batched (B streams) runtime and memmapped loader |
-| `tools/run_atom_native.py` | single-stream trainer (CPU) |
-| `tools/train_batched.py`, `tools/bench_batched.py` | batched trainer (CPU/CUDA) and benchmark |
-| `tools/chat_atom_native.py` | chat / generation (CPU) |
-| `tools/build_corpus_fr_mix.py`, `data/` | corpus builder, 7.4 MB mixed corpus shards, held-out sets, template corpus |
-| `scripts/` | resumable drivers (`train_bptt.sh`, `train_batched.sh`, `setup_gpu_box.sh`, `fr_big/`) |
-| `docs/` | reports (start with ROOT_CAUSE → DECODE_FIX_AND_BPTT → MIX_DATA_LONG_MEMORY → BATCHED_TRAINING) and raw artifacts |
-| `test/` | test suite (`pytest test/`) |
-| `docs/README_legacy.md` | the previous project README (historical; some of its claims predate the root-cause analysis) |
-
-Code snapshot: branch `ms1M-continue-d32` at commit 100fae0 plus the batched-training commits (91c7477..5fc07bc on branch `batched-train`)
-and the GPU runbook scripts. Run logs are not included.
-
-## Data and licence
-
-No licence file exists in the source repository, so none is declared here. Corpus provenance: the `data/corpus_fr_mix_*` book text comes from Project Gutenberg
-(public domain in the USA, boilerplate stripped); the dialogue is synthetic and generated locally (`data/corpus_fr_mix_manifest.json`, `docs/MIX_DATA_LONG_MEMORY.md` §1).
+## Earlier findings (still true)
+
+Documented in `docs/ROOT_CAUSE.md`, `docs/DECODE_FIX_AND_BPTT.md`, `docs/LAST_ATOM_DROPOUT.md`:
+
+1. **Decode mismatch** (fixed): generated space/punct used a different Atomizer boundary label than train → floods (`Oui,,`, 94 spaces). After the fix, teacher-forced and generate logits match (`max |Δlogit| = 0`).
+2. **Field never trained as memory** (fixed): field detached every tick + leak 0.3 → context 2–3 bytes. Truncated BPTT + learnable leak was the first repair. Learnable leak **drifting back up** is the 2026-10-01 repair (this README).
+3. Small d32 French-only numbers on a 7.4 MB mix (~2.06 nats/byte held-out books) are in the older README / `docs/artifacts/`. They are not this bilingual d64 run.
+
+---
+
+## License / intent
+
+Experimental research. Independent architecture. Operable checkpoints. No fluency claim.

checkpoints/bi_d64_leak05/atom_native_step_3900000_bi_d64_leak05.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac1b5589365bdcfefb5da6896f9f100f6468c867569dfc69342f61f414f7d928
+size 73950027