AbstractPhil
/

alephllm-mini-beatrix-training

+<!-- Technical ledger extracted 2026-09-04 from the deep-rewrite of the beatrix-ft2 article (every number verified against its ledger and sources by the ft2-finish pass). This is the technical record behind the human-readable article; it is not the article. -->
+## 6. The battery night
+**(2026-08-31 23:07 → 09-01 01:48 for the four cells, the local 4090 under the 18 GB rule, one seed per cell; then 02:09 → 08:08 for the wide arm and to 09:35 for its probes.)** The journal dates the night 09-01 by convention though its first verdict landed at 23:42 on 08-31, and several of the evening's headers say 09-01 too; commit timestamps are primary here. Three of Phil's directives framed it. The caption-expert docket (08-30 21:22): "Will be wise to train an arm for the behavior, so we can get proper conditioning … we'll finetune using a large array of assorted shorter captions to enhance the caption capacity for booru and plain english capacity" — the arm teaches the register, a later alignment the target space. The priority steer (08-31 22:59) that parked the rank chase behind the diffusion line: "The structure at small scale doesn't matter as much as a large scale diffusion potential." And the application-driven doctrine (23:04) that opened the transplant line: "it's potentially viable to transplant hubs into pretrained models and train them as distillation mechanisms … The AlephLM has unique components that simply cannot be understood without testing their applications." The plan (23:07) put four cells in one script, sequential and fault-isolated, under the v1 arm laws — frozen trunk; structure born weight-zero, never gate-zero; arm-off versus arm-on *is* the gauge; a near-zero delta reads as *vacuous*; fit and evaluation surfaces disjoint; weights ship on completion; a VRAM assert every step. Three cells were owed by the evening's antecedents (the rank trio in section 5, the C7 screen in section 4); the fourth opened the caption line.
+| cell | bed | what trained | headline (one seed, one validation draw) | grade |
+|---|---|---|---|---|
+| B2, the C7 supplement | fresh 237M 2s crafts, wikitext-103, 1,200 steps; fineweb validation, 4 batches × 2 × 4,096 | the whole model; B's head revived at step 0 | A 3.0750 vs B 3.0968 bpb (+0.0218); head toggle B +2.2343, A −4e-6 | candidate |
+| B3, the caption head-arm | frozen 2s final (step 61,422); COCO captions, 1,500 steps | `head.w_s` only, 65,536 parameters | caption 1.6914 → 1.6728; toggle +0.0186 = the whole delta; fineweb +0.0003 | candidate |
+| B1, the hub transplant | frozen 40M sdpa host (d 512, 12 layers), wikitext-103, 1,500 steps | a 4-book hub at layers 1–3, output projection born zero | 1.9660 → 1.8810 bpb; toggle +0.0851; early rank 34/22/32 → 35/24/36 | candidate |
+| B4, the update-rank gauge | matched 12-layer splat/sdpa pair, one 1,024-position probe sequence | nothing — a gauge | layers 0–2: sdpa 4.3/1.9/4.6 vs splat 5.3/7.8/9.7; mid-stack inverted | candidate |
+**B2: relocated, not better.** The cultivation screen's arm B toggle (3.27 → 3.50) had been read on a light fixed-slice gauge, so whether a born-live head makes a *better model* or a *relocated* one was unmeasured. B2 rebuilt A (born null, the mission default) and B (the revival op at step 0: whitened projection, fresh governed book, fp64 ridge for the output weights, projection and codebook then frozen) as matched 237M crafts, same seed, 1,200 steps, judged every 300 on four fineweb batches. A: 3.1576, 3.1458, 3.1369, 3.0750 bpb, head toggle never past −3e-5 — the born-null head contributing nothing, as in every craft before it. B: 3.1740, 3.1578, 3.1340, 3.0968, toggle +2.1257, +2.1756, +2.2619, +2.2343. **The born-live head carries 2.23 bpb of the function and the model is +0.0218 bpb worse at 1,200 steps.** Two honesty lines the journal does not carry: the tax is one seed, one draw, no noise bar, and its sign was not steady — at step 900 B led, 3.1340 against 3.1369; and the script solved B's revival ridge on `val[:2]`, the same two fineweb batches that score the toggle and make up half the full gauge, so the fit/eval-disjoint law was not honored for that install — the toggle may be inflated by an unmeasured amount, the tax's direction is unaffected. Downstream the v3 born-live doctrine kept its structural case — guaranteed participation, transplantability — and lost the free-lunch claim; the head lane fell below L14-attach in the Fable 5.1 review. The journal grades the cell a candidate in one place and a law in two (`law_families` :192 and :201) — a tension stated here, not resolved upward.
+**B3: the entire specialization lives in the swappable matrix.** The caption line's opener trained one tensor of the finished craft — the head's 256 × 256 output weights — with Adam 3e-4, no warmup, no clip, for 1,500 steps of byte-packed COCO captions (26,000 kept at 15–250 characters; 2,048 held out, 23,952 fit; context 1,024, micro 8); the held-out gauge is eight packed batches, 65,536 bytes per evaluation. Caption validation 1.6914 �� 1.6728 (−0.0186, near-linear, not plateaued); restoring the original weights returns 1.6913971745 to the last printed digit, so the toggle +0.0186 is the whole delta; fineweb moved +0.000255 on four batches; ‖w_s‖ 517. **Feasibility confirmed** — specialist in, base back, no host damage — at one seed, one domain (plain-English COCO, not the docket's booru register), one mechanism, and no saved trajectory (the every-300-step prints are not in the ledger). The cell took four launches and an operating-system restart in a driver-crash cascade (registry rebuilt 01:43, verdict 01:48); the lessons became a standing memory, not a result.
+**B1: function grafts, geometry does not.** The transplant opener rebuilt the rank screen's sdpa arm (d 512, 12 layers, 8 heads, no hubs, seed 1337, 1,500 wikitext-103 steps under Muon 2e-2 and Adam 3e-4), froze it, grafted a four-book `CausalSplatHUB` (K 32, D 64, τ 0.1, seed 99) additively beside attention at layers 1–3 — the collapse zone — with its output projection born zero, and trained the hubs alone 1,500 steps on a third seed; the gauge is wikitext-103 validation on two batches, not fineweb, and the rank readout one probe sequence. Host 1.9660 → 1.8810 bpb; toggle +0.0628 at 300, +0.0809, +0.0827, +0.0813, **+0.0851 at 1,500**; hidden effective rank after layers 1–3 33.6/21.8/32.3 → 35.1/24.1/36.4 (layer 0, the embedding, unchanged at 74.8). Real function, none of the collapse repaired: a supplement path lowers loss without buying the splat's rank sustenance, which by the causal rung's reading requires owning the mixer. The host's base ranks differ from the rank screen's A2 readout (39/28/44) — different probe batch and validation, unreconciled. The line was declared viable for distillation into hub-free hosts; its docketed mission-scale host, the control craft, ended destabilized at 2.8846 bpb (section 5), so host fitness is unmeasured. The journal carries a [LAW] copy (:2196) of this one-seed cell.
+**B4: the parked gauge, run anyway.** The steer had parked the update-rank discriminator; it ran at 00:41 beside B1 on the rebuilt screen pair (the plan's one-book arm absent). One probe sequence per craft; the effective rank of each layer's attention update — 1,024 positions × 512 channels, before the residual add. Layers 0–2: sdpa 4.28/1.88/4.60 against splat 5.31/7.79/9.74; layers 3–9: sdpa 24–38 against splat 5–15; the full vectors are in section 5. Naive H-content — update diversity at any depth — is **inverted mid-stack** while sdpa's hidden stream stays collapsed, and holds only at the collapse site; that minted H-CONTENT-EARLY as a refinement, not a confirmation. Its falsifier — swap only the first two or three mixers, splat-early/sdpa-late against the inverse — is unrun; the journal holds it as a candidate at :2999 and a law at :1745.
+**The prototype behind the wide arm** was amoe2's ConstellationRelay, committed at 23:18 on a worktree branch: a multi-book governed adapter fusing the v2-era hub mechanisms into the settled arm laws, each answering a recorded lesson: the supply law asserted at construction (K ≤ 2D, the ROUND 5e crowding result); per-book reads composed by concatenation, no cross-book softmax (the comparative-selector failure class); birth minimum-separation at 45°, governed outside the task gradient; a consume path with weight *and* bias zeroed (v1 zeroed the weight only, a documented ~+0.5 ppl leak); a liveness tripwire that compares real read norms against the same books' self-calibrated chance and flags burial below 0.1× (buried crafts had sat at 0.003–0.03×); and an optional attach-time ridge ignition returning provenance (C7's mandatory solve as a boundary-write). Six of six CPU invariants passed, with no stored transcript. It is a prototype — not imported by the library, no GPU certification before the run — and the branch is **not on the GitHub remote as of 09-04**; the public arm library is unchanged since 08-17.
+**The wide arm.** Phil's order at 02:09: "Make a wider MOE with enough params to encompass the behavior of a 6 hour run and then test it when it's done for diffusion-capable utility." The plan sized it from B3's head-only backward throughput (120–140k tokens/s, discounted to 90–110k for the relay's ~1.6× per-token FLOPs): 2.0–2.4B tokens, 45–60 per parameter — as 64 slots × 4 books, 64 anchors at dimension 32 (K = 2D, the supply boundary), hidden 4,096: **46,158,849 trainable parameters** around the frozen craft's final norm, zero-initialized with no ignition so that all function is attributable to the caption gradient. A pre-launch adversarial panel caught a gauge hole — liveness on random slot inputs reads ~1.0 by construction — so the shipped gauge reads real activations. Pure Adam 3e-4, 1,000-step warmup, clip 1.0, governance every 500 steps, micro 16 × 1,024 bytes of cc12m-berts captions, the first 4,096 of shard 0 held out as 24 packed batches (393,600 bytes — the script's "≈ full held set" is unverified against the ~897k bytes those captions hold), 5.5 training hours in a six-hour wall; inertness at birth passed at |Δ| < 1e-7. The run: **57,003 steps, 934,849,200 tokens, 5.503 hours at 47.2k tokens/s** — roughly half the planned rate, less than half the planned tokens, ~20 per parameter — through 4,331,396 captions from 9 of 67 shards, about an eighth of the 33.5M set. Held-out caption bpb 1.3029 → 1.0927 at 2,500 steps, 0.9782 at 10k, 0.9430 at 20k, 0.9068 at 40k, 0.8966 at 50k, **0.8918 final** (−0.4110; −0.0048 over the last 7,003 steps — flattening, not flat); the post-training bypass reproduces the base, 1.3028627515, to every printed digit (no tensor-level check logged), so the toggle +0.4110 *is* the gain. Fineweb 1.1239 → 1.2464 with the arm on (+0.1225; the band ran +0.0762 at 22.5k to +0.1225 at the end) — **a modal specialist, not a free one**. Liveness 0.489 at 2,500, a minimum 0.411 at 7,500, 0.730 final (the journal's "0.43 → 0.73" starts from the second row); book drift 0.067 → 0.285; the tripwire never fired; no spikes; one seed, one held-out draw, one data order. The journal prints 700× B3's parameters for 22× the caption gain at 400× the interference; by the ledgers 704×, 22× and ~480×, on an unmatched pair — different diets, held-out sets, steps, evaluation slices, mechanisms — a characterization, not a scaling law.
+**What the probes made of it.** PROBE-3 (08:13) scored the arm against CLIP-B/32 on 2,048 COCO captions, one draw: relay reads +.2715 Spearman, the hidden they read from +.2052 arm-off and +.1989 arm-on, the consume path's hidden layer +.1859, layer 14 +.3954. Its reading — caption mastery is not conditioning geometry — was **retracted at 09:04** under T5-v1.1-XXL: reads +.3709 against L14's +.4117, the head surface *improved* by the arm (+.3044 on, +.2816 off), the two rulers at −.0016 global rank correlation with each other. On five rulers (09:06) the reads beat the untouched surface on every one (+.371, +.297, +.272, +.556, +.524 against +.282, +.221, +.205, +.381, +.355); arm-on wins the three language-model rulers by +.019 to +.023 and loses the two CLIP rulers by −.004 and −.006 — a language-model-cluster geometry. At the causal tier (09:35) the journal says only that the reads "do not survive"; the hub ledger has the numbers: contrast accuracy .503 (chance .50), group .10 (chance .25), families .60 attribute / .71 count / .17 relation (relation confounded by anchor order), beside L14's .560, .277 and .71/.81/.13. The numbers stand; the convergent lesson with B1 and B4 — a head-side attachment cannot reach mid-stack structure — is carried as a law at :216 on candidate inputs, and its falsifier is a constellation attached *at* layer 14. Section 7 carries the rulers.
+**The ceiling nobody has measured.** The Fable 5.1 review reopened a pre-window law: dense signed dispatch shares a conserved amplitude budget (w_k/z = sinh(u_k)/Σ cosh(u_j)), so member shares shrink as the roster grows — implied totals .850 → .809 → .785 → .757 from two to six arms, and the one measurement, three arms at .868 against .809 predicted, already disagrees. The relay's books compose by concatenation, not dispatch, so the question belongs to an AMOE spine's roster, not this arm; the 4/8/16/32/64 screen at matched capacity is queued, unscheduled. Every cell of the night is one seed and stands as a candidate; the retractions are PROBE-3's reading, the C7 light-gauge toggle as a headline, and the antecedent chain's three mechanisms; the debts are three unrun falsifiers — the early-mixer chimera, the L14-attach constellation, the arm-count screen — and an unpublished branch.