docs/technical/03_designed_life_2s.md: paraphrase the program lead's directives (no inline quotations)
Browse files
docs/technical/03_designed_life_2s.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
| 6 |
|
| 7 |
### The craft, the recipe, and why this craft
|
| 8 |
|
| 9 |
-
The preset is the lawful rung above the mother: d 1024, 20 layers, context 4,096, a byte-trigram embedding over vocab 256, a CausalSplatHUB in every one of the twenty blocks (4 books × 64 anchors at address dimension 128, τ 0.1, supply 0.5× of the K ≤ 2D bound), a 3 × ff-1024 anchored expert bank in every layer, a dual head (linear plus a 256-anchor aleph read at D 256), the governor at 45° every 8 steps, and hub_ckpt 0 because the retained scan fits 96GB at this size; 237.1M parameters measured at build. The recipe: Muon at 2e-2 on the transport weights, pure Adam at 3e-4 with weight decay 0 everywhere else, 200 warmup steps then flat — the logged lr_scale reads 0.25 at step 50 and 1.000 first to last after, both anneal phases included — gradient clip 1.0, bf16 autocast over fp32 masters, 262,144 tokens a step. No QK-normalization, no weight decay, no learning-rate decay: **the anneal is a diet change, not a decay** — the 09-02 riders' premise that "the anneal's lower learning rate" might rescue the control was retracted on 09-04 because no lower-rate phase exists — and that recipe, tuned on this craft, is the open confound section 5 carries. Why this craft: with the 849M at 24–26 days on one card, Phil ruled on 08-26
|
| 10 |
|
| 11 |
### The diet
|
| 12 |
|
|
|
|
| 6 |
|
| 7 |
### The craft, the recipe, and why this craft
|
| 8 |
|
| 9 |
+
The preset is the lawful rung above the mother: d 1024, 20 layers, context 4,096, a byte-trigram embedding over vocab 256, a CausalSplatHUB in every one of the twenty blocks (4 books × 64 anchors at address dimension 128, τ 0.1, supply 0.5× of the K ≤ 2D bound), a 3 × ff-1024 anchored expert bank in every layer, a dual head (linear plus a 256-anchor aleph read at D 256), the governor at 45° every 8 steps, and hub_ckpt 0 because the retained scan fits 96GB at this size; 237.1M parameters measured at build. The recipe: Muon at 2e-2 on the transport weights, pure Adam at 3e-4 with weight decay 0 everywhere else, 200 warmup steps then flat — the logged lr_scale reads 0.25 at step 50 and 1.000 first to last after, both anneal phases included — gradient clip 1.0, bf16 autocast over fp32 masters, 262,144 tokens a step. No QK-normalization, no weight decay, no learning-rate decay: **the anneal is a diet change, not a decay** — the 09-02 riders' premise that "the anneal's lower learning rate" might rescue the control was retracted on 09-04 because no lower-rate phase exists — and that recipe, tuned on this craft, is the open confound section 5 carries. Why this craft: with the 849M at 24–26 days on one card, Phil ruled on 08-26 that the next stage up from Beatrix v1 would be trained, the large one deferred until a series of logistics problems was worked out. The 849M is shelved, not retired; 2s became the bed every v2-era mechanism screens on first.
|
| 10 |
|
| 11 |
### The diet
|
| 12 |
|