atom-ai / ARCHITECTURE.md

ATOM (atom-ai): code snapshot (100fae0 + batched-train), README with measured numbers, curated checkpoints

9f1076a verified about 1 month ago

8.61 kB

# ATOM architecture

A tick-based toroidal field. **Not a Transformer.**
No softmax attention, no BPE, no KV-cache, no HF tokenizer on the live path.
**New race: infinite, growable, CPU-scalable, live-modifiable.**

## One sentence

A UTF-8 byte becomes an atom; atoms live in a field α that RK4 advances;
the surface reads **the last atom + α** and emits the next byte.
The same `.pt` can be grown (d16→d32→d64…) and taught forever — stream, no forgetting.

## Why this is a new race (v2 proof: 0→1080k ms1M, d32, no patches)

**Growable to infinity:** `tools/grow_checkpoint.py` Net2Net-preserving:
- doubled output rows tiled+noise (symmetry breakers)
- doubled input cols zero-padded (old outputs exact)
- embeddings tiled+noise
- JL interleaved + frozen grown (α-branch exact on tiled α)
Result: d16 120k (val 1.352 / teacher 63.2%) → d32 grown init → 8k later 56% teacher, 16k 61.3% = d16 best. No restart. You can chain d32→d64→d128 same lineage. Field and readout are co-adapted — exact function ≠ same trajectory because d32 saturates at ~3000 steps vs ~850 d16 — but training re-adapts in 1 chunk.

**Infinitely learnable, not frozen:**
- Stream forever: `--stream --data-glob 'data/corpus_fr_medium*.txt' --loop-shards --chunk-bytes 65536 --stream-skip-packets k*steps` → fresh 64kB chunks each run.
- No replay (quarantined — drove CE 2.99→0.008 by reciting ring), no MERGE, no payload-copy (both off at train AND chat).
- `episode-reset --episode-length 8000` (was 6000 until 48k, 8000 sweet spot since 56k, tested 12000 worse): ~37.5% transient / 62.5% saturated for d32, exposes readout to both regimes. Episodes are far longer than saturation transient, so field persists across dialogue scales — not a wipe.
- `atom-flush-every 64` mandatory everywhere (train + primer probes) or atoms grow unbounded → 20× slowdown (same for probing).
- Same checkpoint lineage 0→120k d16 → 0→496k s21 → 504k→1M ms1M (s21 500k + s7 500k =1M) → 1080k 2nd epoch: val 1.765→0.551, teacher 56%→75.2%, no catastrophic forgetting. Val spikes at shard boundaries (1.645@632k, 1.49@976k) then descends — knowledge intact.

**CPU-scalable, GPU-fast:**
- No attention = O(n) not O(n²). Surface = `α-MLP(α) + 0.3*frozen_JL(α) + last_atom(2-gram + dentate top-25% + φ)`.
- CPU: d16 50-72 tick/s, d32 37-50 tps. 8k steps = 3-4 min. 1M = 125 chunks = ~8h CPU. GPU would be 10-100× — 10s GB in hours.
- `field-contrast-weight 0` — dead hinge (grads zero, 3 forwards/step wasted), cut for free speedup. `efference-every 10` = 1 tick/10 predict own byte no-grad → commit → CE on gold next — ~0% tps cost, 800/8000 eff ticks verified, not guilty of collapse.

**Live-modifiable without breaking + live survey:**
- `.pt` is infinitely trainable: `PYTHONPATH=. .venv/bin/python tools/run_atom_native.py --resume your.pt --data your_corpus.txt ...` on CPU, even after GPU pre-train on 10s GB.
- Read-only probes never stop training: `tools/probe_field_persistence.py` (field_rms cold ~0.04 / primed 0.26@560k→0.43@768k→0.62@872k→0.87@1M→1.06@1080k towards 3.0), `tools/probe_field_regimes.py --primer-packets 3500` (cold vs primed 3500), `tools/diagnose_teacher.py --max-span-bytes 1` (must, default span-4 gives ~22% artefact). Probe shows field differs across prompts but surface logits nearly identical when field not reliably read — Transformer-regime risk. Our scaling improves it.
- You can watch the model live while it trains — `field_rms`, `n_atoms`, generation `Peut-être`, `Peux qu'`, `Parler,`, `Je il`, clean stop `\n\n`.

## One sentence (original)

A UTF-8 byte becomes an atom; atoms live in a field α that RK4 advances;
the surface reads **the last atom + α** and emits the next byte.

## Tick graph

```
UTF-8 stream
    |
    v
Atomizer          1 byte → AtomPacket {payload, features, boundary, phase}
    |
    v
Compiler          packet features → ToroidalAtom {r, φ, ω, E, payload}
    |
    v
Field core        α ∈ R^{n_modes × d}   RK4 + interaction + optional MERGE
    |
    v
Surface           logits[256] = α-MLP(α) + 0.3·JL(α) + last_atom(byte, prev, φ)
    |
    v
decode            argmax / sample one byte → commit as next atom
```

Train and generate **share this graph**. That is the speech contract
(`docs/BYTE_TICK.md`). Linguistic spans (5–32 bytes) broke it: train
saw a whole span, generate decoded slots independently.

## Pieces

### Atomizer (`src/io/atomizer.py`)

Cuts the byte stream into packets. Speech line: `max_span_bytes=1`.
Features are a fixed local observation (histogram, first/last nibble,
whitespace/alpha rates) — not a learned vocab.

`packet_from_payload` (generate commit) infers a train-like boundary
(`newline|punct|whitespace|max_span`). Tag `"generated"` is OOD and
must not be written.

### Toroidal atom (`src/toroidal/atom.py`)

One living unit:

| field | meaning |
|-------|---------|
| `r` | content vector |
| `φ` | phase (oscillatory binding) |
| `ω` | frequency |
| `E` | energy |
| `payload` | exact bytes this atom carries |

The collection is a list + stacked tensors. MERGE can fuse two atoms
when phase coherence is high. Speech line keeps MERGE **off**.

### Field α (`src/toroidal/state.py`, `dynamics.py`)

α is the persistent field, shape `(n_modes, d_model)`.
Dynamics integrate with RK4. Energy decay is bounded.
`field_max_rms` clips amplitude (sandbox: 3.0).

α is **not** a residual stream of a Transformer block. It is the
state that survives ticks, flushes, and (when you allow it) sessions.

### Interaction / aggregation / abstraction / consolidation

Still in the core. They are the structural story (atoms → aggregates
→ abstractions → persistent buffer). They are **not** the speech
readout. Speech is the surface.

### Surface / hard readout (`AtomSurfaceHead` in `src/atom_native.py`)
Under `--field-obligatory-hard`:
```
a_hat        = JL flatten(α)
train_logits = α-MLP(a_hat)          # trainable
frozen       = 0.3 · frozen_JL(a_hat) # buffer, not trained
last         = last_atom_readout(payloads, φ)
logits       = train + frozen + last
```
No RMS cap on the trainable branch (the cap re-collapsed CE).
**Last-atom readout** (corpus bridge #20 + #1):
1. embed last committed byte
2. embed previous byte (2-gram)
3. dentate: expand concat → ReLU → keep top 25% → project back
4. add `cos φ, sin φ` of the last atom
5. linear to 256 logits
This is **not** payload-copy (global hist / winner copy). Copy is off.
### Generate (`generate_packets`)
1. wrap prompt as `Utilisateur: …\nAssistant: ` (`src/speech_lock.py`)
2. ingest wrap packets into the live field
3. loop: predict one byte, commit `packet_from_payload`, append
4. n-gram anti-repeat: if the last 2–6 bytes already cycled, forbid
   that byte and draw once more
5. stop on empty, hard cycle, or `max_packets`
`speech_ok` is a phrase gate, not a language model. Leave
`speech_gate=False` while measuring.
### Train (`tools/run_atom_native.py`)
For each step: take packets `(current, target)`, `transition_loss` =
CE(next byte) + optional tiny aux, clip grad 1.0, step.
Optional **efference** (bridge #9): every 10 steps, commit the model's
own predicted byte, then CE on the gold target from that state.
Optional **replay** exists and is **off**. It drove CE to 0.008 by
reciting a ring.
Stream mode (`--stream`) reads shards online. Use it above ~200 kB
when `max_span_bytes=1` (otherwise encode holds one packet per byte).
## What ATOM is not
| banned | why |
|--------|-----|
| softmax attention / Flash / GQA | different machine |
| HF / GPT-2 tokenizer | Atomizer is the tokenizer |
| payload-copy attractor | surface copies instead of predicting |
| `"generated"` boundary | train/generate mismatch |
| field wipe to drop CE | destroys the thesis |
| stacking 4 new levers in one run | unreadable science |
## Neuro map (own corpus, not papers)
`docs/NEURO_BRIDGES_ATOM.md`. Wired: φ multiplex, dentate 2-gram,
efference. Replay implemented and gated off.
## Files that matter
```
src/atom_native.py          field + surface + generate
src/io/atomizer.py          packets
src/io/stream_corpus.py     online shards
src/speech_lock.py          wrap + gate
src/toroidal/*              atoms, RK4, MERGE, persist
tools/run_atom_native.py    trainer
tools/chat_atom_native.py   interactive
TRAIN.md                    how to run
docs/MEASURE_LOG.md         numbers
docs/STATUS.md              what is live
```
Legacy GPT-era: `src/main.py`, `src/training/trainer.py`, `legacy/`.
Do not start those for speech.

	# ATOM architecture

	A tick-based toroidal field. Not a Transformer.
	No softmax attention, no BPE, no KV-cache, no HF tokenizer on the live path.
	New race: infinite, growable, CPU-scalable, live-modifiable.

	## One sentence

	A UTF-8 byte becomes an atom; atoms live in a field α that RK4 advances;
	the surface reads the last atom + α and emits the next byte.
	The same `.pt` can be grown (d16→d32→d64…) and taught forever — stream, no forgetting.

	## Why this is a new race (v2 proof: 0→1080k ms1M, d32, no patches)

	Growable to infinity: `tools/grow_checkpoint.py` Net2Net-preserving:
	- doubled output rows tiled+noise (symmetry breakers)
	- doubled input cols zero-padded (old outputs exact)
	- embeddings tiled+noise
	- JL interleaved + frozen grown (α-branch exact on tiled α)
	Result: d16 120k (val 1.352 / teacher 63.2%) → d32 grown init → 8k later 56% teacher, 16k 61.3% = d16 best. No restart. You can chain d32→d64→d128 same lineage. Field and readout are co-adapted — exact function ≠ same trajectory because d32 saturates at ~3000 steps vs ~850 d16 — but training re-adapts in 1 chunk.

	Infinitely learnable, not frozen:
	- Stream forever: `--stream --data-glob 'data/corpus_fr_medium.txt' --loop-shards --chunk-bytes 65536 --stream-skip-packets ksteps` → fresh 64kB chunks each run.
	- No replay (quarantined — drove CE 2.99→0.008 by reciting ring), no MERGE, no payload-copy (both off at train AND chat).
	- `episode-reset --episode-length 8000` (was 6000 until 48k, 8000 sweet spot since 56k, tested 12000 worse): ~37.5% transient / 62.5% saturated for d32, exposes readout to both regimes. Episodes are far longer than saturation transient, so field persists across dialogue scales — not a wipe.
	- `atom-flush-every 64` mandatory everywhere (train + primer probes) or atoms grow unbounded → 20× slowdown (same for probing).
	- Same checkpoint lineage 0→120k d16 → 0→496k s21 → 504k→1M ms1M (s21 500k + s7 500k =1M) → 1080k 2nd epoch: val 1.765→0.551, teacher 56%→75.2%, no catastrophic forgetting. Val spikes at shard boundaries (1.645@632k, 1.49@976k) then descends — knowledge intact.

	CPU-scalable, GPU-fast:
	- No attention = O(n) not O(n²). Surface = `α-MLP(α) + 0.3*frozen_JL(α) + last_atom(2-gram + dentate top-25% + φ)`.
	- CPU: d16 50-72 tick/s, d32 37-50 tps. 8k steps = 3-4 min. 1M = 125 chunks = ~8h CPU. GPU would be 10-100× — 10s GB in hours.
	- `field-contrast-weight 0` — dead hinge (grads zero, 3 forwards/step wasted), cut for free speedup. `efference-every 10` = 1 tick/10 predict own byte no-grad → commit → CE on gold next — ~0% tps cost, 800/8000 eff ticks verified, not guilty of collapse.

	Live-modifiable without breaking + live survey:
	- `.pt` is infinitely trainable: `PYTHONPATH=. .venv/bin/python tools/run_atom_native.py --resume your.pt --data your_corpus.txt ...` on CPU, even after GPU pre-train on 10s GB.
	- Read-only probes never stop training: `tools/probe_field_persistence.py` (field_rms cold ~0.04 / primed 0.26@560k→0.43@768k→0.62@872k→0.87@1M→1.06@1080k towards 3.0), `tools/probe_field_regimes.py --primer-packets 3500` (cold vs primed 3500), `tools/diagnose_teacher.py --max-span-bytes 1` (must, default span-4 gives ~22% artefact). Probe shows field differs across prompts but surface logits nearly identical when field not reliably read — Transformer-regime risk. Our scaling improves it.
	- You can watch the model live while it trains — `field_rms`, `n_atoms`, generation `Peut-être`, `Peux qu'`, `Parler,`, `Je il`, clean stop `\n\n`.

	## One sentence (original)

	A UTF-8 byte becomes an atom; atoms live in a field α that RK4 advances;
	the surface reads the last atom + α and emits the next byte.

	## Tick graph

	```
	UTF-8 stream
	\|
	v
	Atomizer 1 byte → AtomPacket {payload, features, boundary, phase}
	\|
	v
	Compiler packet features → ToroidalAtom {r, φ, ω, E, payload}
	\|
	v
	Field core α ∈ R^{n_modes × d} RK4 + interaction + optional MERGE
	\|
	v
	Surface logits[256] = α-MLP(α) + 0.3·JL(α) + last_atom(byte, prev, φ)
	\|
	v
	decode argmax / sample one byte → commit as next atom
	```

	Train and generate share this graph. That is the speech contract
	(`docs/BYTE_TICK.md`). Linguistic spans (5–32 bytes) broke it: train
	saw a whole span, generate decoded slots independently.

	## Pieces

	### Atomizer (`src/io/atomizer.py`)

	Cuts the byte stream into packets. Speech line: `max_span_bytes=1`.
	Features are a fixed local observation (histogram, first/last nibble,
	whitespace/alpha rates) — not a learned vocab.

	`packet_from_payload` (generate commit) infers a train-like boundary
	(`newline\|punct\|whitespace\|max_span`). Tag `"generated"` is OOD and
	must not be written.

	### Toroidal atom (`src/toroidal/atom.py`)

	One living unit:

	\| field \| meaning \|
	\|-------\|---------\|
	\| `r` \| content vector \|
	\| `φ` \| phase (oscillatory binding) \|
	\| `ω` \| frequency \|
	\| `E` \| energy \|
	\| `payload` \| exact bytes this atom carries \|

	The collection is a list + stacked tensors. MERGE can fuse two atoms
	when phase coherence is high. Speech line keeps MERGE off.

	### Field α (`src/toroidal/state.py`, `dynamics.py`)

	α is the persistent field, shape `(n_modes, d_model)`.
	Dynamics integrate with RK4. Energy decay is bounded.
	`field_max_rms` clips amplitude (sandbox: 3.0).

	α is not a residual stream of a Transformer block. It is the
	state that survives ticks, flushes, and (when you allow it) sessions.

	### Interaction / aggregation / abstraction / consolidation

	Still in the core. They are the structural story (atoms → aggregates
	→ abstractions → persistent buffer). They are not the speech
	readout. Speech is the surface.

	### Surface / hard readout (`AtomSurfaceHead` in `src/atom_native.py`)

	Under `--field-obligatory-hard`:

	```
	a_hat = JL flatten(α)
	train_logits = α-MLP(a_hat) # trainable
	frozen = 0.3 · frozen_JL(a_hat) # buffer, not trained
	last = last_atom_readout(payloads, φ)
	logits = train + frozen + last
	```

	No RMS cap on the trainable branch (the cap re-collapsed CE).

	Last-atom readout (corpus bridge #20 + #1):

	1. embed last committed byte
	2. embed previous byte (2-gram)
	3. dentate: expand concat → ReLU → keep top 25% → project back
	4. add `cos φ, sin φ` of the last atom
	5. linear to 256 logits

	This is not payload-copy (global hist / winner copy). Copy is off.

	### Generate (`generate_packets`)

	1. wrap prompt as `Utilisateur: …\nAssistant: ` (`src/speech_lock.py`)
	2. ingest wrap packets into the live field
	3. loop: predict one byte, commit `packet_from_payload`, append
	4. n-gram anti-repeat: if the last 2–6 bytes already cycled, forbid
	that byte and draw once more
	5. stop on empty, hard cycle, or `max_packets`

	`speech_ok` is a phrase gate, not a language model. Leave
	`speech_gate=False` while measuring.

	### Train (`tools/run_atom_native.py`)

	For each step: take packets `(current, target)`, `transition_loss` =
	CE(next byte) + optional tiny aux, clip grad 1.0, step.

	Optional efference (bridge #9): every 10 steps, commit the model's
	own predicted byte, then CE on the gold target from that state.

	Optional replay exists and is off. It drove CE to 0.008 by
	reciting a ring.

	Stream mode (`--stream`) reads shards online. Use it above ~200 kB
	when `max_span_bytes=1` (otherwise encode holds one packet per byte).

	## What ATOM is not

	\| banned \| why \|
	\|--------\|-----\|
	\| softmax attention / Flash / GQA \| different machine \|
	\| HF / GPT-2 tokenizer \| Atomizer is the tokenizer \|
	\| payload-copy attractor \| surface copies instead of predicting \|
	\| `"generated"` boundary \| train/generate mismatch \|
	\| field wipe to drop CE \| destroys the thesis \|
	\| stacking 4 new levers in one run \| unreadable science \|

	## Neuro map (own corpus, not papers)

	`docs/NEURO_BRIDGES_ATOM.md`. Wired: φ multiplex, dentate 2-gram,
	efference. Replay implemented and gated off.

	## Files that matter

	```
	src/atom_native.py field + surface + generate
	src/io/atomizer.py packets
	src/io/stream_corpus.py online shards
	src/speech_lock.py wrap + gate
	src/toroidal/* atoms, RK4, MERGE, persist
	tools/run_atom_native.py trainer
	tools/chat_atom_native.py interactive
	TRAIN.md how to run
	docs/MEASURE_LOG.md numbers
	docs/STATUS.md what is live
	```

	Legacy GPT-era: `src/main.py`, `src/training/trainer.py`, `legacy/`.
	Do not start those for speech.