docs/technical/04_the_head.md: paraphrase the program lead's directives (no inline quotations)
1996c80 verified |
Download docs/technical/04_the_head.md from AbstractPhil/alephllm-mini-beatrix-training: direct link, hf CLI and curl.
- Browser
- Download file 13.1 kB
-
https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training/resolve/main/docs/technical/04_the_head.md
- Command line
-
hf download hf://AbstractPhil/alephllm-mini-beatrix-training/docs/technical/04_the_head.md
-
curl -L -o 04_the_head.md https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training/resolve/main/docs/technical/04_the_head.md
13.1 kB
| <!-- Technical ledger extracted 2026-09-04 from the verified deep rewrite behind the beatrix-ft2 article (every number checked against its ledger and sources). This is the technical record; the human-readable article summarizes it. --> | |
| ## 4. The head | |
| **(2026-08-27 β 09-01.)** The aleph *head* is a second output read beside the craft's linear base head w_h (a 1024β256 linear): a projection, a 256-anchor book at address dimension 256, and an output W_s born weight-zero under the born-null law. Through 24,000 steps (6.3 billion tokens; 6.5 by the halt at 24,860) its head-off toggle sat between β2.3e-6 and β3.4e-7 at every boundary (warmup 1,145 at 0.30B, validation 2.3626 bpb; cap 8,320; fineweb 20,219 at 1.1172; s0 22,890 at 1.2522 β one craft, fineweb-edu holdout) while the hubs went +1.50 β +2.54 and the banks +0.04 β +2.70 (warmup to s0, hub JSON). TensorBoard shows what the toggle could not: W_s norm growing 19.1 (step 2,000) β 40.6 (24,000) on a read carrying nothing, consumed rank falling 53.9 β 9.2 (16,000) before climbing back to 40.2. **Magnitude without function, again** β the mother's arc (0.027 β 24.845 with head-off 0.0000 throughout) at a new scale. One rider fell at once: reading the warmup null as "v1 head-election lore" was **retracted the same day** β the lore itself was retracted (below). | |
| **(2026-08-27, 12:11: the dead head named.)** An offline diagnostic on the step-12,000 checkpoint (3.1B tokens; local 4090; fp32 head math; one checkpoint) found the read dead by *learned avoidance*: signed-read norm ~6e-4 per position; proj(h)'s cosines against all 256 anchors ~1e-3 against chance 1/β256 = 0.0625 β 60Γ *below* chance; W_s at norm 31 multiplying nothing; head gradients five orders below w_h's; input rank intact (334 of 1,024 post-norm). A ridge solve on the dead reads recovered 0.0006 bpb: **least-squares ignition alone was refuted**. The mechanism: early aleph noise is cheapest suppressed by rotating the projection into codebook-orthogonality, silencing the address before W_s can earn. An eight-arm trial on the same checkpoint (committed 12:48; ~65k held-out positions, paired per-block bars) priced an offline write: as-trained +0.000001 at 0.014Γ chance; ridge on the buried reads β0.0014 Β±0.0005; the full revival (whitening projection, data-seeded governed book, fp64 ridge) β0.0007 Β±0.0003 at 3.3Γ chance; a fresh random book instead β0.0023 Β±0.0008; the op without the 45Β° projection β0.0008 Β±0.0003, a tie with the governed arm; frozen random β0.0023 Β±0.0007, a tie with the fresh book; gradient ratio ||g_Ws||/||g_wh|| 1.4e-5 β 6.9e-3 (500Γ). **Offline extraction is worth one to two millibits in every live arm** β the linear base already takes everything linearly available β so function can only be earned nonlinearly in training; k-means seeding bought nothing and the **governor was exonerated**. A retro sweep found burial complete in the first 2,000 steps β 0.032Γ of chance at the first stored checkpoint (its step not stated), deepening to 0.014Γ at 12,000 β so prevention must be at birth; the pre-revival read at 24,860 later gave 0.0103Γ. No JSON of the trial shipped. | |
| **(2026-08-27, 12:48: the lineage correction.)** The same hour re-read the mother. "Never elected" holds on every record β head-off exactly 0.0 at 51,882, 58,664 and 88,508 (one craft; fineweb gauge). The autopsy's weight claim does not: it recorded "||W_s|| = 0.0 exactly, untouched zero-init after 8.4B tokens," but the shipped mini-beatrix-1 AutoModel is the step-58,664, 17.301B-byte post-fold core, and a direct safetensors read of its head on 09-04 gives W_s norm 0.0270 (the 0.0015 gate folded into a 17.9-norm weight on 08-15), gamma 1.0, K=512 at D=32, all 131,072 entries nonzero. **The function claim survives; the zero-weight line is retracted here** as checkpoint-specific and numerically off; the graduate at 88,508 carries 24.845 with head-off β0.0000. The "patience precedent" (v1 elected late, so wait) fell with the lore; born-null burial stood at 2/2 crafts; the mother's model card still ships the head-election asset. | |
| **(2026-08-29, 07:55: a third category of intervention.)** A closed-form write carrying task signal fit neither the governor statute nor the no-balance-machinery law, so **boundary-write** was minted (S-1003) as a third category β data-dependent, so outside the governor; never in the task gradient, so outside balance machinery β permitted only at run boundaries (phase edges, halted-session resumes), every application logging provenance. alephllm 0.8.2 shipped the op β projection β whitening-PCA of the post-norm states; book β fresh Gaussian seed plus 45Β° minimum-separation; W_s β fp64 ridge on the base head's cross-entropy residual, alpha swept 0.25β32 held-out β with a frozen-address flag, a liveness gauge in every census (chance from 512 random unit inputs through the same book; buried below 0.1Γ). Applied to the halted craft at step 24,860 (in s1, ~6.5B tokens; one craft, book seed 13), promoted at 08:08 on Phil's go: zero separation pushes, alpha 8.0 interior, W_s installed at norm 481.5, liveness 0.0103Γ β 0.9964Γ of chance, held-out loss 1.2768 β 1.2339 bpb (β0.0429) on 65,536 positions. That delta, twenty times the trial's millibits, is **flagged as partly stage-drift recovery** β the base head had drifted under the s0 diet, and a ridge fit absorbs what is linearly recoverable. The freeze is verified in the log: codebook rank 197.2 at 24,000 β 155.1 at 25,000, constant through 61,000; consumed rank 40.2 β 144.9. | |
| **(2026-08-29 β 08-31: the head lives, then wobbles.)** Seven hundred steps later the s1 boundary (25,561; 6.70B) read **+0.025805 bpb** (head-off 1.280245 vs 1.254440) β the first nonzero aleph-head toggle in the lineage β with W_s trimmed 481.5 β 478.1 by the gradient, not dismantled. The install's β0.043 became +0.026 on the trainer's gauge as the base head reabsorbed the drift-recovery share (two gauges). The trajectory by boundary, one craft on the fineweb-edu holdout, paired per-block bars ~Β±0.0008 recorded in prose only: | |
| | boundary | step | tokens | toggle | val bpb | | |
| |---|---|---|---|---| | |
| | s1 | 25,561 | 6.70B | +0.0258 | 1.2544 | | |
| | s2 | 29,376 | 7.70B | +0.0140 | 1.2177 | | |
| | s3 | 33,954 | 8.90B | +0.0072 | 1.2318 | | |
| | s4 | 38,532 | 10.10B | +0.0130 | 1.2356 | | |
| | s5 | 42,347 | 11.10B | +0.0278 | 1.2504 | | |
| | cap (session) | 43,130 | 11.31B | +0.0140 | 1.2614 | | |
| | s6 | 45,399 | 11.90B | +0.0126 | 1.2683 | | |
| | s7 | 50,740 | 13.30B | β0.0010 | 1.1883 | | |
| | s8 (fixed gauge) | 53,792 | 14.10B | +0.0149 | 1.1911 | | |
| | prose anneal | 57,607 | 15.10B | +0.0101 | 1.1153 | | |
| | chat anneal, final | 61,422 | 16.10B | +0.0123 | 1.1097 | | |
| The s8 row is a **retraction stated where it fell**: the curriculum_s8 report ran on the trainer's synthetic fallback (1.0737 bpb, head +0.0090, "do not cite"); the fixed-gauge re-issue is the +0.0149 above, yet +0.0090 still travels in the canon toggle chains, the 09-04 audit's head row and this section's previous draft. The narrative between boundaries fell three times β "the decay reversed" at s2 (oscillation, by 30,000), a "durable floor near +0.012" (broken at 32,000: +0.0024 on the s3 rule-chain diet), then "slow reabsorption" β before the settled read at s3, whose own "+0.002 to +0.015, never zero" widened when s7 touched zero exactly as the base head dropped 1.268 β 1.188. The settled read: **a small, diet-sensitive, volatile niche, band β0.001 to +0.028, mean about +0.010, never a floor** β one craft, a candidate, and the source of a reporting rule: head verdicts at phase boundaries or on band-exit only. Its two exits above the band arrived on the richest diets (s5 causal +0.0278; eval 48,000 on s7's mix +0.0253) β a demand-diversity growth hypothesis the anneal did not honor (+0.0101, +0.0123). Beneath the toggle, W_s grew monotonically 475.1 (28,000) β 515.9 shipped while function did not; consumed rank drifted 147 β 136; the frozen address, logged only from 54,000, sits at 0.81β0.84Γ of chance against the 0.9964Γ installed β a drift the record never remarks. | |
| **(2026-08-31 β 09-04: the third craft.)** The softmax twin of section 5 trained its born-null head with a *free* address for its whole life β a near-miss fixed in 0.8.7 before launch, since it would otherwise have inherited the treatment's frozen address and been buried by construction. Its head-off toggle never left Β±1.1e-5 at any boundary report through 16.1B tokens (+5.9e-6 at 1,145 to β1.06e-5 at 61,422; fourteen rows, one craft). Liveness read 0.23Γ at step 500, 0.0295Γ by 3,000 (buried flag set from 3,000 through 61,000), 0.0098Γ at 38,000, 0.028Γ at 60,500; W_s climbed 13 β 127 on nothing; consumed rank reached 66β103 mid-run β structure without function. **Born-null self-burial is a statute at 3/3 crafts (S-1002)**: regardless of governor, address freezing, attention family or scale, a weight-zero aleph head racing a same-target linear base buries within ~2,000 steps. The 08-30 selection-mechanism review named the precondition structural: abstention and burial are the *same* geometry in the signed read; hubs never buried because attention *is* their read β no bypass β while the head races a parallel path. | |
| **(2026-08-31: the cultivation screen.)** Phil's question: can the head be cultivated from baseline within a thousand steps or so with the new head state, or does it need an architectural change. Four fresh 237M crafts on the mission card (wikitext-103, ctx 4,096, micro 8, seed 1337, 1,200 steps, gauges every 300 on a light 32,768-position val slice; one seed): **A**, born null β buried at the first gauge, liveness 0.089Γ β 0.045Γ, toggle within 1.1e-5 of zero, W_s 9 β 18; **B**, the revival op at step 0 (birth ridge β2.95 against an untrained base, alpha 16, W_s 6,345) β toggle +3.27 β +3.50, liveness 0.975Γ β 1.125Γ; **C**, the finished craft's 61,422 head transplanted (projection, book, W_s; not w_h) onto a newborn β +0.0375 β +0.0498 β +0.0723 β +0.0922 (monotone, accelerating), liveness 1.56Γ β 2.16Γ, W_s 516.5 β 520.5; **D**, B's address with W_s zeroed β toggle between β1.2e-4 and β4.6e-5, the address alive at 1.03β1.05Γ. D is decisive: **weight-zero cannot bootstrap against the linear base even without burial; the solve is mandatory.** B and C carried and grew: architecture acquitted, bottleneck rung deprioritized. Two caveats: the op fits on the validation batches the gauges later use (surviving inflation unmeasured), and B's toggle measures relocation β at birth the solved head is the primary readout. | |
| **(2026-08-31, 13:49 β 23:42: born with function, and its price.)** Phil designated the v3 doctrine mid-screen β heads born live, address frozen so burial is structurally impossible β and D tightened it to born *with function*: address construction plus the birth ridge, or a matured warm-start. The owed supplement ran that night on the local 4090 (B2; fineweb-edu validation over 32,768 positions, micro 2, seed 1337, 1,200 steps): born-null A 3.1576 β 3.0750 bpb with head toggle within 3e-5 of zero; born-live B 3.1740 β 3.0968 with the head carrying **+2.2343 bpb** of the model's function. B led at 900 steps (3.1340 vs 3.1369) and finished **+0.0218 worse** β one seed, a candidate: **born-live relocates labor at a small early tax**; the doctrine keeps its structural justification (guaranteed participation, transplantability, conditioning-facing roles) and loses the free lunch. Whether a trunk co-organized around a live head pays later is a full-run question with no data; arm C has no full-gauge number. Two grading inconsistencies stand: the law-family file carries S-1002 as candidate rows beside the statute, and grades B2 a law where both dockets say candidate. | |
| **(2026-09-01.)** What the head is still for: B3 (01:48, local 4090) trained only W_s of the finished craft on ~24k plain-English captions (ctx 1,024, pure Adam 3e-4, 1,500 steps; one seed, one domain): caption validation 1.6914 β 1.6728 (β0.0186, not plateaued), arm-off restoring the original W_s bit-exactly, fineweb interference +0.0003 β **the whole specialization lives in a swappable W_s**. Then the Fable 5.1 review (13:02) called the lane a sunk cost β burial 3/3, a revived head worth ~12 millibits, born-live paying early β and deprioritized head cultivation below the layer-14 attach of section 7; Phil amended rather than agreed: v3 must carry a **guaranteed global-frame organ** if the head cannot encompass the behavior, its mechanism decided before the v3 spec (S-1004). The born-state statutes now chain: gate-zero deadlocks (S-1001), weight-zero self-buries (S-1002), function is written at a boundary or born with (S-1003, S-1004), the read modulates and never sole-carries (S-1006). The refinement rungs β abstain channel, ΟΒ·βD ladder, w_h bottleneck, ΞΈ ladder β remain unarmed; every number in this section is one seed. | |