AbstractPhil commited on
Commit
b1ef2a6
·
verified ·
1 Parent(s): 9ae6daa

docs/technical/03_designed_life_2s.md: paraphrase the program lead's directives (no inline quotations)

Browse files
docs/technical/03_designed_life_2s.md CHANGED
@@ -6,7 +6,7 @@
6
 
7
  ### The craft, the recipe, and why this craft
8
 
9
- The preset is the lawful rung above the mother: d 1024, 20 layers, context 4,096, a byte-trigram embedding over vocab 256, a CausalSplatHUB in every one of the twenty blocks (4 books × 64 anchors at address dimension 128, τ 0.1, supply 0.5× of the K ≤ 2D bound), a 3 × ff-1024 anchored expert bank in every layer, a dual head (linear plus a 256-anchor aleph read at D 256), the governor at 45° every 8 steps, and hub_ckpt 0 because the retained scan fits 96GB at this size; 237.1M parameters measured at build. The recipe: Muon at 2e-2 on the transport weights, pure Adam at 3e-4 with weight decay 0 everywhere else, 200 warmup steps then flat — the logged lr_scale reads 0.25 at step 50 and 1.000 first to last after, both anneal phases included — gradient clip 1.0, bf16 autocast over fp32 masters, 262,144 tokens a step. No QK-normalization, no weight decay, no learning-rate decay: **the anneal is a diet change, not a decay** — the 09-02 riders' premise that "the anneal's lower learning rate" might rescue the control was retracted on 09-04 because no lower-rate phase exists — and that recipe, tuned on this craft, is the open confound section 5 carries. Why this craft: with the 849M at 24–26 days on one card, Phil ruled on 08-26 — "Lets train the next stage up from the beatrix v1, we can't train the large one currently. Not until we work out a series of logistics problems." The 849M is shelved, not retired; 2s became the bed every v2-era mechanism screens on first.
10
 
11
  ### The diet
12
 
 
6
 
7
  ### The craft, the recipe, and why this craft
8
 
9
+ The preset is the lawful rung above the mother: d 1024, 20 layers, context 4,096, a byte-trigram embedding over vocab 256, a CausalSplatHUB in every one of the twenty blocks (4 books × 64 anchors at address dimension 128, τ 0.1, supply 0.5× of the K ≤ 2D bound), a 3 × ff-1024 anchored expert bank in every layer, a dual head (linear plus a 256-anchor aleph read at D 256), the governor at 45° every 8 steps, and hub_ckpt 0 because the retained scan fits 96GB at this size; 237.1M parameters measured at build. The recipe: Muon at 2e-2 on the transport weights, pure Adam at 3e-4 with weight decay 0 everywhere else, 200 warmup steps then flat — the logged lr_scale reads 0.25 at step 50 and 1.000 first to last after, both anneal phases included — gradient clip 1.0, bf16 autocast over fp32 masters, 262,144 tokens a step. No QK-normalization, no weight decay, no learning-rate decay: **the anneal is a diet change, not a decay** — the 09-02 riders' premise that "the anneal's lower learning rate" might rescue the control was retracted on 09-04 because no lower-rate phase exists — and that recipe, tuned on this craft, is the open confound section 5 carries. Why this craft: with the 849M at 24–26 days on one card, Phil ruled on 08-26 that the next stage up from Beatrix v1 would be trained, the large one deferred until a series of logistics problems was worked out. The 849M is shelved, not retired; 2s became the bed every v2-era mechanism screens on first.
10
 
11
  ### The diet
12