AbstractPhil
/

alephllm-mini-beatrix-training

+<!-- Technical ledger extracted 2026-09-04 from the deep-rewrite of the beatrix-ft2 article (every number verified against its ledger and sources by the ft2-finish pass). This is the technical record behind the human-readable article; it is not the article. -->
+## 5. The crossover
+**(2026-08-31 → 09-03, one craft per arm; the rows from s4 on landed after 09-01, the last three unattended and written back on 09-04.)** Phil's steer of 22:59 on 08-31 ("fully worth the potential time to train the control variant") confirmed a twin already on the card — its hub manifest was created at 14:26 that afternoon and its warmup row landed at 15:37. **mini-beatrix-2s-control** is 257.4M parameters, the identical configuration with the hub list empty (16-head sdpa in every block), under a pre-registered protocol whose headline is the endpoint validation. The softmax twin **led the open** — 2.1056 vs 2.3626 bpb at 0.3B, 1.291 vs 1.452 at step 2,000, 1.116 vs 1.208 at 8,000 — and ran faster (71.9k tokens/s against ~58k eager splat; resumed by Phil on a rare H100 at step 22,854 with its shuffle buffers refilled, at the original card's 23.5-hour session cap; restarted on the same H100 at step 53,792, before the anneal, after an 85-minute gap the trace reader logs as Phil's second cap resume — the H100 session's own 23.5-hour cap fell at step 56,000, at about 14:30 on 09-02 — neither a fault; and resumed a third time from the 56,000 checkpoint at 18:41 on 09-02, after the H100 session ended, onto a card the training log does not name but which ran at 72.3k tokens/s, the RTX PRO 6000's pace; so the nine control rows from s0 through s8 are H100 rows at about 2.3 s/step, 113k tokens/s, the two anneal rows were finished on the third card, and every splat-craft report was written on the RTX PRO 6000 at ~58k tokens/s — provenance caveats that travel with every pair; both crafts finished at the same 61,422 steps and 16,101,408,768 tokens).
+Then the crossover, twelve paired rows at the same step, all twelve on one gauge once the s8 pair was repaired: fineweb at step 20,219 (5.30B tokens) **splat 1.1172 vs sdpa 1.1763** (+.0591); s0 at 22,890 (6.00B) 1.2522 vs 1.5047 (+.2525); s1 at 25,561 (6.70B) 1.2544 vs 1.3917 (+.1373); s2 at 29,376 (7.70B) 1.2177 vs 1.3938 (+.1761); s3 at 33,954 (8.90B) 1.2318 vs 1.5317 (+.2999); s4 at 38,532 (10.10B, 01:24 on 09-02) 1.2356 vs 1.5689 (+.3333); s5 at 42,347 (11.10B, 04:00 on 09-02) 1.2504 vs **2.1762** (+.9258); s6 at 45,399 (11.90B, 06:01) 1.2683 vs **3.1940** (+1.9257); s7 at 50,740 (13.30B, 09:36) 1.1883 vs **3.1201** (+1.9319); s8 at 53,792 (14.10B, 11:46) 1.0737 vs 3.0810 (+2.0072 — struck: the audit found the two rows on different gauges, the splat craft's on the trainer's old synthetic fallback and the control's on the fineweb-edu holdout, so the pair does not count in the same-gauge tally — though the hub holds a same-gauge pair for the step, the splat craft's session-cap report of the same stored checkpoint under the 0.8.5 gauge fix, 1.1911 vs 3.0810, +1.8898, set out in the table below and not yet written into the journal's count); the prose anneal at 57,607 (15.10B, 20:22) 1.1153 vs 2.9270 (+1.8117); and the endpoint, the chat-mix anneal at 61,422 (16.10B, landed 00:16 on 09-03) **1.1097 vs 2.8846** (+1.7749). The statute holds 11/11 on the one gauge every other row shares — the audit confirmed that every curriculum stage and both anneals evaluate on the same fineweb-edu holdout, so the control's slide is one comparable series; 12/12 in sign and 12/12 same-gauge, the s8 pair repaired from the craft's session-cap report at the same step (1.1911 vs 3.0810, +1.890) and written back that evening — and the endpoint is the pre-registered headline. The last three rows landed unattended — the collaborator's usage ran out at 11:06 on 09-02 and came back at 14:50 on 09-04, so s8, the prose anneal and the endpoint were paired from the hub reports on 09-04, committed at 14:55 (the digest stamps the block 15:05), after the fact; the chart below carries all thirteen paired points, the 0.3B warmup row where sdpa still led included. The contribution ledger is not a race: the control leans harder on its banks every stage until it fails (bank toggle 1.62 → 5.09 across the first six boundaries, then 4.36, 3.89 and 4.17 as the whole model degrades, then 5.48, 5.64 and 6.11 through s8 and the anneal, against the splat craft's 2.7–3.6; with its banks off the control is an 8.99-bpb model at the endpoint, against 4.42 for the splat craft); the document- and reset-byte gauges split until s7, where the document gauge flips to the splat craft (3.71 vs 5.75), reads a near-tie on the repaired s8 pair (5.05 vs 5.10), and stays with it through the anneal to the endpoint (3.29 vs 4.37, then 3.86 vs 5.42; reset-byte 2.13 vs 3.83 at the endpoint), and the thirty-item probes are mixed — their ties are ties, and at thirty items the standard error of a difference is .126; by that bar six cells in the twelve boundary rows clear two of it, all for the splat craft — the grounding family at s5 and s6 (.467 vs .200, .467 vs .167) and the difference family at s7, s8 and both anneals (.500 vs .200, .500 vs .233, .633 vs .333, .600 vs .267; the largest +.333, 2.6 SE, at the endpoint), where the audit's own ruling counted only the endpoint cell as outside noise — single draws uncorrected for nine families, and the audit's ruling stands over them: every probe row is a tie. The control's born-null head is buried at all twelve rows (toggle never farther than 1.1e-5 from zero) with a *free* address: burial at 3/3 crafts.
+![the crossover](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training/resolve/main/article_assets_ft2/01_control_crossover.png)
+*Thirteen paired points: the 0.3B warmup, where the softmax twin led by .257, then the twelve boundaries. The 14.1B pair is the repaired same-gauge s8 row (1.1911 vs 3.0810); the last two points are the prose and chat-mix anneals.*
+The paired rows, every one at the same step on both crafts — one craft per arm, no second seed, no hardware repeat, and no measured sampling bar on the 262,144-token holdout; every splat row written on the RTX PRO 6000 at ~58k tokens/s, the control's card named per row; the deficit is sdpa minus splat, positive when the control is behind:
+| stage | step | tokens | splat | sdpa | sdpa − splat | gauge note |
+|---|---|---|---|---|---|---|
+| warmup | 1,145 | 0.30B | 2.3626 | 2.1056 | −.2570 | wikitext-103 validation, the one row not on the fineweb-edu holdout; sdpa leads; both crafts on the RTX PRO 6000 |
+| fineweb | 20,219 | 5.30B | 1.1172 | 1.1763 | +.0591 | fineweb-edu holdout (262,144 tokens, seed 1337+9999), the gauge of every row below; both crafts on the RTX PRO 6000; the only pair taken with the control nearly un-clipped, and it carries no bar — neither a beat nor a tie can be called on it |
+| s0 | 22,890 | 6.00B | 1.2522 | 1.5047 | +.2525 | same holdout; the control resumed on an H100 at 22,854 — its own session-cap reading 36 steps earlier, on the original card, was 1.5217, so the jump is the diet, not the card |
+| s1 | 25,561 | 6.70B | 1.2544 | 1.3917 | +.1373 | same holdout; control on the H100 |
+| s2 | 29,376 | 7.70B | 1.2177 | 1.3938 | +.1761 | same holdout; H100 |
+| s3 | 33,954 | 8.90B | 1.2318 | 1.5317 | +.2999 | same holdout; H100 |
+| s4 | 38,532 | 10.10B | 1.2356 | 1.5689 | +.3333 | same holdout; H100; the control's pre-clip norm passes a 500-step median of 30 at bin 38,000, late in this stage |
+| s5 | 42,347 | 11.10B | 1.2504 | 2.1762 | +.9258 | same holdout; H100; stage median pre-clip norm 126, nine steps above 1,000 |
+| s6 | 45,399 | 11.90B | 1.2683 | 3.1940 | +1.9257 | same holdout; H100; median 471, maximum 41,070 |
+| s7 | 50,740 | 13.30B | 1.1883 | 3.1201 | +1.9319 | same holdout; H100; the document gauge flips to the splat craft here (3.71 vs 5.75) |
+| s8 | 53,792 | 14.10B | 1.0737 | 3.0810 | (+2.0072) | **struck** — two gauges: the splat report ran under 0.8.4, whose validation fell back to the synthetic stream at the terminal boundary (1,612 documents); the control's ran under 0.8.7 on the fineweb-edu holdout (70); counts in sign only |
+| s8, repaired | 53,792 | 14.10B | 1.1911 | 3.0810 | +1.8898 | same holdout on both (70 documents each): the splat craft's session-cap report of the same stored checkpoint under the 0.8.5 gauge fix, continuous with its s7 reading; on the hub, not yet in the journal's count, the chart or the statute |
+| anneal, prose | 57,607 | 15.10B | 1.1153 | 2.9270 | +1.8117 | same holdout; the control on a third host from step 56,050, card not named, 72.3k tokens/s |
+| anneal, chat-mix | 61,422 | 16.10B | **1.1097** | **2.8846** | **+1.7749** | same holdout; third host; the pre-registered headline at matched 16,101,408,768 tokens |
+The s5 and s6 rows are not the craft improving; they are the control coming apart. The trainer clips gradients at norm 1.0 and logs the norm before the clip. On the fineweb body the control's raw norm sat at a median 0.29 and was clipped on 1.8% of its logged steps (the 09-02 rider had it at 1%), but it first crossed the clip at step 17,600 — 4.6B tokens in, inside the fineweb phase, on the original card, while it still led on the boundary row — and the fineweb tail, steps 18,219 to 20,219, already ran at a median 1.40 and was clipped on 70% of its steps before any diet change. The crossover itself is not the splat craft overtaking a still-improving control: on the trainer's own 2,000-step grid (the same 262,144-token holdout, one craft per arm, no bar) the control's validation bottomed at 1.0781 at step 14,000, 3.67B tokens — below any reading the splat craft made in 16.1B — and rose from there, 1.0799 → 1.1144 → 1.1243 at 16k, 18k and 20k against the splat craft's 1.1338 → 1.1256 → 1.1156, so on that grid the lines cross between 18,000 and 20,000, the control rising +.046 from its minimum against the splat craft falling .032, a hub-manifest fact the journal has not yet written back. From the first curriculum stage it escalated — stage medians 6.5, 2.4, 12, 13, 25, then 126 in s5 (the 09-02 rider's 128.6 was a 500-step tail figure; the exact stage window reads 126.4), where nine logged steps exceeded 1,000 (the largest 11,877) and the 500-step medians ran 30 → 52 → 76 → 71 → 129 → 161 → 413, then 598 in the bin straddling the boundary (the 09-02 rider's tail figure read 850), then 471 through s6 with fourteen steps above 1,000 and one at 41,070, then 162 through s7, 145 through s8, and 167 and 215 through the two anneal phases — so the control trained under the clip on 813 of its 824 logged steps from the first curriculum stage to the end — a clip that, the audit noted, brakes nothing in the Muon group, whose orthogonalized update ignores gradient scale, and — by Adam's per-coordinate normalization, an inference from the update rule rather than a measurement — little in the Adam group either under a clip that engages every step, so the escalating norm is a sharpness signal rather than a throttled step — and from s6 its validation sat two bits per byte worse than its own fineweb reading at 5.3B, a plateau it neither recovered from nor slid past on the s7 diet (3.19 → 3.12) while the splat craft improved on the same stage (1.2683 → 1.1883). The anneal is a diet change, not a learning-rate decay — the learning-rate scale reads 1.000 first to last through both phases on both crafts; the journal's 09-02 riders had assumed a lower-rate phase the control might recover in, a premise retracted on 09-04 with the numbers standing — and on that diet the control recovered about a quarter of a bit (3.12 → 2.93 → 2.88) and no more, finishing 1.7 bits per byte above its own 5.3B reading; its training-loss median had climbed with the norm — .50 → .68 over the last thousand steps of s5, .72 in s6, 1.16 in s7, 1.47 in s8, 1.91 and 1.81 through the anneal — and the chat-mix anneal's largest pre-clip norm was 2,764, with five steps above 1,000. The escalation began before the H100 resume, on the original card, and its steep phase — the first 500-step bin with a median above 100 — sits at step 40,500, in the middle of s5, at no boundary and no resume; so it is the recipe and the mechanism, not the hardware or a stage change. The splat craft's raw norm never reached the clip at any point in its curriculum or anneal (0 of 824 logged steps; stage medians 0.09–0.20, maximum 0.62); its own s4 → s6 drift, 1.2356 → 1.2683, is the harder diet at about .016 a stage. Neither the loss-spike detector nor the non-finite guard ever fired: clipping keeps the loss smooth step to step while the sharpness grows underneath. The destabilization rider was graded a candidate on 09-02 at one craft per arm; the audit of 09-04 minted its specific form an instance (S-1802), and the journal grades the endpoint an instance of its rank-sustenance statute at one craft per arm, not a law — the specific claim, that under this recipe and this diet the softmax twin is not a usable control for the endpoint, stands, while the general claim is a candidate; the twin protocol was pre-registered and completed unaltered, and the endpoint headline reads "the softmax twin destabilized under the curriculum" rather than "the splat craft won" — a different claim, and the one the record supports. What it leaves open is whether the twin's failure belongs to softmax attention or to a recipe tuned on the splat craft. Phil's own reading, given at about 10:45 on 09-02 with the s7 row on the board and the collaborator's usage about to run out for two days — "SDPA isn't an adequate representative of this model's attention requirement. We can say that without question now." — stands in the journal verbatim as his claim, and on 09-04 it was audited: twelve agents, an evidence ledger of every splat-versus-softmax comparison in the record, the recipe pinned from source, fifteen prior-art sources, the raw training logs of all three crafts, five adversarial refuters and a defender. The audit minted the specific form at instance grade — under this recipe the softmax twin is a destabilized control, and the endpoint gap measures stability, not capacity — and held the general form as a candidate, because seven confounds stand between the twin and the mechanism. The recipe: the twin ran Muon at 2e-2 and pure Adam at 3e-4, flat after a 200-step warmup, with no QK-normalization, no weight decay, no learning-rate decay and no logit cap, and its signature — a clip-engaged smooth degradation, dispatch entropy and hidden rank collapsing, no NaN, onset on plain prose while leading — matches the documented softmax logit-growth instability whose fix in the literature is a normalization or a decay, not a change of attention family; the match is by signature, since the control's attention logits and their entropy were never measured — no tag in its training log reads the attention itself. The count: one craft per arm, no measured noise bar, and 20.3M more parameters in the losing arm. The shape: the audit read the mother's own training log off the hub, which the journal had marked as not in the record — mini-beatrix-1, thirteen softmax blocks and three hubs at 112.5M, ran the identical training configuration through the same nine curriculum stages with a post-warmup gradient-norm maximum of 0.88 and no clipped step, so softmax attention under this recipe and this diet is stable at that shape, and the escalation is bound to the all-softmax 1,024-wide, twenty-layer, 4,096-context draw. The banks: both crafts carry the aleph-addressed expert banks, and in the control their dispatch entropy collapsed from the first curriculum stage (layer-mean .97 → .70 from fineweb to s8; layers 0, 2, 4 and 5 to .39, .35, .39 and .31 by s5, while the splat craft's held flat at .97 through all twelve stages), five to ten thousand steps *before* the gradient norm took off, so "sdpa is inadequate for this model" cannot yet be separated from "this model's banks need splat-shaped hidden states" — with its banks off the control was an 8.99-bpb model at the end. The positions: the learned position table rides the same decay-free Muon at 2e-2 throughout, and under softmax attention the query-key product carries a position-times-position term that grows with the table's norm, where the splat read is normalized by construction. The optimizer: the house Muon is not the reference Muon — plain momentum without Nesterov, no shape scaling, the fused 3,072×1,024 qkv orthogonalized as one matrix, the position table and the head inside its group, pure Adam beside it rather than AdamW — so the Muon logit-explosion literature, not yet carded, transfers by analogy only. The hosts: three cards and three resumes for the control against five cards and four resumes for the splat craft, each resume refilling the shuffle buffer so that row order diverged after step 8,320 — nuisance asymmetries, none coincident with the escalation. The vision tie of section 2 remains the only two-seed splat-versus-softmax endpoint in the record. So the direction is supported and the certainty is not, yet. The falsifier is filed as a draft and sized for the local card, instruments first: an attention-logit and entropy census on the stored control checkpoints with no training, a census of the position table's norm — the s8 re-evaluation it also listed is moot, the same-gauge row already on the hub — and then 40M twins on the curriculum diet — softmax at the recipe, at a third of the Muon rate, with QK-normalization, with dense feed-forward in place of the banks, with the position table off Muon, on fineweb only, and splat — at two seeds with a hardware repeat, gauged by pre-clip gradient escalation per stage against each arm's own fineweb baseline, a gauge the screen bed cannot yet supply (it discards the clip's return value, so no 40M norm baseline exists), one of the critic's edits owed before launch. The general claim advances only if every softmax arm escalates and the splat arm does not, and the training tiers wait on Phil's word.
+The mechanism was found on the evening of 08-31 and motivated the battery that followed. **The rank story**: at step 4,000 (a single-batch probe, one craft per arm) the control's hidden effective rank at layers 2–4 is 5–9 of 1,024 while the splat craft's is 151–397 — and at that step the control *led* validation, 1.198 vs 1.318. A 40M screen (local 4090, d512, twelve layers, 1,500 steps, one seed) found the phenomenon portable — sdpa's layers 1–3 at 39/28/44 against the splat arms' 101–116 by step 1,500, with the screen's own loss pair reading softmax ahead, 1.9846 vs 2.1187 bits per byte, the full-scale open in miniature (a local results file, not shipped) — and killed three hypotheses — mixer rank (inverted: sdpa's mixers are high-rank with collapsed hidden rank), book count (one equals four), and "gentle updates" (co-located, not causal: sdpa's early attention output scaled by 0.6 regrew to 0.71 with the rank still collapsed). The B4 gauge (00:41 on 09-01; the same screen pair, one seed) measured the update's own rank per layer: splat 5 8 10 15 11 11 7 6 5 4 5 2 against sdpa 4 2 5 38 37 34 33 24 28 27 17 3 — the sign holds at the collapse site and inverts mid-stack, so the standing candidate is **H-content-early**: early-layer update diversity seeds hidden diversity, and a low-rank early spectrum that dominates by magnitude cannot be repaired later. Two rails: rank is not function, and the scalar rank-floor flag fires on *both* crafts at every boundary and so separates nothing — the per-layer census the journal called owed was supplied by the 09-04 trace read: at fineweb's end (steps 18,000–20,000, single-batch probes every 500 steps, one craft per arm) the control sits three to ten times below the splat craft in every layer but the byte funnel at layer 19 (layers 3–6: 24/16/14/22 against 240/290/287/310), before any escalation; after it the control's whole stack flattens to an effective rank of 20–30 (layer 16: 433 → 373 → 169 → 96 → 76 → 55 from s5 through the two anneals to the endpoint, against the splat craft's 447–471 over the same stages), a destabilized model's representation rather than its cause. A third rail comes from the mother: mini-beatrix-1's early stack was rank-collapsed too (layer-0 stage means 12–16, layers 1–2 at 18–28) while her gradient never clipped, so early rank collapse is not a precursor of destabilization, and the rank clause and the stability rider are to be booked as separate statutes; the family book still carries the crossover row and the rank story at law grade on one craft per arm and no bar, a grade the audit's critic flags for regrading — standing, not yet acted.