AbstractPhil commited on
Commit
06bbac1
Β·
verified Β·
1 Parent(s): 90a06b7

README: the softmax twin (endpoint, the claim audit)

Browse files
Files changed (1) hide show
  1. README.md +37 -1
README.md CHANGED
@@ -96,12 +96,48 @@ under `mini-beatrix-2s/`.
96
  Final validation: **1.1097 bpb** on the fineweb-edu holdout β€” the run's best
97
  reading, set during the chat phase.
98
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
99
  ## Lineage
100
 
101
  Code: [AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)
102
  (this repo vendors the model files verbatim; alephllm 0.8.6).
103
  Sibling: [mini-beatrix-1](https://huggingface.co/AbstractPhil/mini-beatrix-1)
104
- (112M, 3-hub hybrid β€” the previous rung). The research memory behind every
 
 
105
  design decision is maintained privately; laws referenced in the code
106
  docstrings (supply ≀ 2D, min-sep governance, budget composition, born-null
107
  weight-zero) are measured, not aesthetic.
 
96
  Final validation: **1.1097 bpb** on the fineweb-edu holdout β€” the run's best
97
  reading, set during the chat phase.
98
 
99
+ ## The softmax twin
100
+
101
+ A pre-registered control ran the identical recipe and diet beside this craft:
102
+ **mini-beatrix-2s-control** β€” the same preset with the hub list empty, so 16-head
103
+ causal softmax attention (SDPA) in every one of the 20 blocks (257.4M parameters
104
+ against this craft's 237M), 16.101B tokens in 61,422 steps, one craft per arm. At
105
+ the matched endpoint (anneal_mix, step 61,422, same evaluation) the twin read
106
+ **2.8846 bpb against this craft's 1.1097**, and this craft led at 11 of 11
107
+ same-gauge paired boundaries after the twin's early lead (0.3B: twin 2.106 vs
108
+ this craft 2.363; the first paired row at 5.3B: this craft 1.1172 vs twin
109
+ 1.1763); the s8 pair is struck because its two rows were measured on different
110
+ gauges. The twin did not merely trail β€” it destabilized: its validation, on the
111
+ one fineweb-edu holdout every row shares, went 1.18 (fineweb) β†’ 1.57 (s4) β†’ 2.18
112
+ (s5) β†’ 3.19 (s6) β†’ 3.12 (s7), and the two anneal phases recovered it only to
113
+ 2.88; its pre-clip gradient norm first crossed the 1.0 clip at step 17,600 inside
114
+ the fineweb phase while it still led, then escalated stage by stage (medians 0.29
115
+ β†’ 6.5 β†’ 2.4 β†’ 12 β†’ 13 β†’ 24.5 β†’ 126 β†’ 471 β†’ 162 β†’ 145 β†’ 167 β†’ 215, spikes to
116
+ 41,070, clipped on 813 of 824 logged curriculum and anneal steps) while this
117
+ craft's never reached the clip (0 of 824, max 0.62). The anneal was a diet change
118
+ (1B of prose, then 1B with the chat frame), not a learning-rate decay: lr_scale
119
+ read 1.000 throughout on both crafts. This is one craft per arm, and the twin
120
+ was given none of softmax attention's usual stabilizers (no QK-normalization, no
121
+ weight decay, no learning-rate decay, the same 1.0 clip). The claim that SDPA is
122
+ not an adequate representative of this model's attention requirement is
123
+ AbstractPhil's (2026-09-02); its audit (2026-09-04) grades the specific form an
124
+ instance β€” under this recipe the twin is a destabilized control, and the gap
125
+ measures stability rather than capacity β€” and the general form a candidate
126
+ behind the recipe, the single seed, the shape (mini-beatrix-1's thirteen softmax
127
+ blocks trained clean under the identical configuration), the aleph-addressed
128
+ expert banks both crafts carry, and the learned position table on decay-free
129
+ Muon. Paired reports: `mini-beatrix-2s-control/reports/v2/` beside this craft's
130
+ under [alephllm-mini-beatrix-training](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training);
131
+ the paired chart is `article_assets_ft2/01_control_crossover.png` there.
132
+
133
  ## Lineage
134
 
135
  Code: [AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)
136
  (this repo vendors the model files verbatim; alephllm 0.8.6).
137
  Sibling: [mini-beatrix-1](https://huggingface.co/AbstractPhil/mini-beatrix-1)
138
+ (112M, 3-hub hybrid β€” the previous rung). Control twin: mini-beatrix-2s-control
139
+ (softmax attention in every block; weights, resume states and 15 reports under
140
+ `alephllm-mini-beatrix-training/mini-beatrix-2s-control/`). The research memory behind every
141
  design decision is maintained privately; laws referenced in the code
142
  docstrings (supply ≀ 2D, min-sep governance, budget composition, born-null
143
  weight-zero) are measured, not aesthetic.