README: the softmax twin (endpoint, the claim audit)
Browse files
README.md
CHANGED
|
@@ -96,12 +96,48 @@ under `mini-beatrix-2s/`.
|
|
| 96 |
Final validation: **1.1097 bpb** on the fineweb-edu holdout β the run's best
|
| 97 |
reading, set during the chat phase.
|
| 98 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
## Lineage
|
| 100 |
|
| 101 |
Code: [AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)
|
| 102 |
(this repo vendors the model files verbatim; alephllm 0.8.6).
|
| 103 |
Sibling: [mini-beatrix-1](https://huggingface.co/AbstractPhil/mini-beatrix-1)
|
| 104 |
-
(112M, 3-hub hybrid β the previous rung).
|
|
|
|
|
|
|
| 105 |
design decision is maintained privately; laws referenced in the code
|
| 106 |
docstrings (supply β€ 2D, min-sep governance, budget composition, born-null
|
| 107 |
weight-zero) are measured, not aesthetic.
|
|
|
|
| 96 |
Final validation: **1.1097 bpb** on the fineweb-edu holdout β the run's best
|
| 97 |
reading, set during the chat phase.
|
| 98 |
|
| 99 |
+
## The softmax twin
|
| 100 |
+
|
| 101 |
+
A pre-registered control ran the identical recipe and diet beside this craft:
|
| 102 |
+
**mini-beatrix-2s-control** β the same preset with the hub list empty, so 16-head
|
| 103 |
+
causal softmax attention (SDPA) in every one of the 20 blocks (257.4M parameters
|
| 104 |
+
against this craft's 237M), 16.101B tokens in 61,422 steps, one craft per arm. At
|
| 105 |
+
the matched endpoint (anneal_mix, step 61,422, same evaluation) the twin read
|
| 106 |
+
**2.8846 bpb against this craft's 1.1097**, and this craft led at 11 of 11
|
| 107 |
+
same-gauge paired boundaries after the twin's early lead (0.3B: twin 2.106 vs
|
| 108 |
+
this craft 2.363; the first paired row at 5.3B: this craft 1.1172 vs twin
|
| 109 |
+
1.1763); the s8 pair is struck because its two rows were measured on different
|
| 110 |
+
gauges. The twin did not merely trail β it destabilized: its validation, on the
|
| 111 |
+
one fineweb-edu holdout every row shares, went 1.18 (fineweb) β 1.57 (s4) β 2.18
|
| 112 |
+
(s5) β 3.19 (s6) β 3.12 (s7), and the two anneal phases recovered it only to
|
| 113 |
+
2.88; its pre-clip gradient norm first crossed the 1.0 clip at step 17,600 inside
|
| 114 |
+
the fineweb phase while it still led, then escalated stage by stage (medians 0.29
|
| 115 |
+
β 6.5 β 2.4 β 12 β 13 β 24.5 β 126 β 471 β 162 β 145 β 167 β 215, spikes to
|
| 116 |
+
41,070, clipped on 813 of 824 logged curriculum and anneal steps) while this
|
| 117 |
+
craft's never reached the clip (0 of 824, max 0.62). The anneal was a diet change
|
| 118 |
+
(1B of prose, then 1B with the chat frame), not a learning-rate decay: lr_scale
|
| 119 |
+
read 1.000 throughout on both crafts. This is one craft per arm, and the twin
|
| 120 |
+
was given none of softmax attention's usual stabilizers (no QK-normalization, no
|
| 121 |
+
weight decay, no learning-rate decay, the same 1.0 clip). The claim that SDPA is
|
| 122 |
+
not an adequate representative of this model's attention requirement is
|
| 123 |
+
AbstractPhil's (2026-09-02); its audit (2026-09-04) grades the specific form an
|
| 124 |
+
instance β under this recipe the twin is a destabilized control, and the gap
|
| 125 |
+
measures stability rather than capacity β and the general form a candidate
|
| 126 |
+
behind the recipe, the single seed, the shape (mini-beatrix-1's thirteen softmax
|
| 127 |
+
blocks trained clean under the identical configuration), the aleph-addressed
|
| 128 |
+
expert banks both crafts carry, and the learned position table on decay-free
|
| 129 |
+
Muon. Paired reports: `mini-beatrix-2s-control/reports/v2/` beside this craft's
|
| 130 |
+
under [alephllm-mini-beatrix-training](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training);
|
| 131 |
+
the paired chart is `article_assets_ft2/01_control_crossover.png` there.
|
| 132 |
+
|
| 133 |
## Lineage
|
| 134 |
|
| 135 |
Code: [AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)
|
| 136 |
(this repo vendors the model files verbatim; alephllm 0.8.6).
|
| 137 |
Sibling: [mini-beatrix-1](https://huggingface.co/AbstractPhil/mini-beatrix-1)
|
| 138 |
+
(112M, 3-hub hybrid β the previous rung). Control twin: mini-beatrix-2s-control
|
| 139 |
+
(softmax attention in every block; weights, resume states and 15 reports under
|
| 140 |
+
`alephllm-mini-beatrix-training/mini-beatrix-2s-control/`). The research memory behind every
|
| 141 |
design decision is maintained privately; laws referenced in the code
|
| 142 |
docstrings (supply β€ 2D, min-sep governance, budget composition, born-null
|
| 143 |
weight-zero) are measured, not aesthetic.
|