| # b4 full epoch at 32k -- 2026-09-10 |
|
|
| Continuation of the 16k->32k YaRN extension through the whole of b4. |
|
|
| ## Setup |
|
|
| | | | |
| |---|---| |
| | model | 1.06B (36 layers, d_model 1536, 12q/4kv, hd 128, SwiGLU 4096) | |
| | window | 32,768 (YaRN SCALE=2 ORIG_MAXPOS=16384, product == 32768 exactly) | |
| | parent | ext32k step 750 (itself extended from l2, the best 16k arm) | |
| | data | b4_32768_65536 -- min doc 32,768 == the window, so EVERY sample is one contiguous document | |
| | tokens | steps 750 -> 2227 = 1.168 B of b4's 1.171 B train split; b4 seen EXACTLY ONCE, no wrap | |
| | batch | B=1 x grad_accum 16 = 524,288 tokens/step, identical to every 16k arm | |
| | LR | peak 1.8e-4, floor 6e-5 (see the note below), WSD: re-warm 100 from step 750, flat to 2027, 200-step decay tail | |
| | hardware | 2x H200, cp=2, peak 107.4 GB of 143 GB | |
| | eval | niah_single, N=100, greedy, substring match, one shared haystack (b2_8192_16384 val) | |
|
|
| ## Results |
|
|
| niah_single accuracy %, +-95% Wilson half-width |
| |
| | step | b4 seen | 8,192 | 16,360 | 32,744 | decayed | |
| |---|---|---|---|---|---| |
| | 750 | 34% | 60.0 +-9.4 | 45.0 +-9.6 | 27.0 +-8.6 | yes | |
| | 1100 | 49% | 55.0 +-9.6 | 40.0 +-9.4 | 24.0 +-8.3 | no | |
| | 1650 | 74% | 64.0 +-9.2 | 55.0 +-9.6 | 33.0 +-9.1 | no | |
| | **2227** | **100%** | **70.0 +-8.8** | **58.0 +-9.5** | **43.0 +-9.5** | **yes** | |
| |
| The 1100 and 1650 checkpoints sit mid-plateau at constant LR and are UNDECAYED, so |
| they understate. Only 750 and 2227 are decayed and mutually comparable -- which is |
| why 1100 reads BELOW 750 and is not a regression. |
| |
| ### The comparison: step 750 vs 2227, both decayed |
| |
| | length | 750 | 2227 | delta | combined 95% bar | resolvable | |
| |---|---|---|---|---|---| |
| | 8,192 | 60.0 | 70.0 | +10.0 | +-12.9 | no | |
| | 16,360 | 45.0 | 58.0 | +13.0 | +-13.5 | no | |
| | 32,744 | 27.0 | 43.0 | **+16.0** | +-12.8 | **YES** | |
| |
| The gains GROW with length (+10, +13, +16). That is the slope flattening, not a |
| uniform lift. Slope on the decayed points: 8k->16k -15.0 -> -12.0 pp/doubling; |
| 16k->32k -18.0 -> -15.0. |
| |
| ### 16k retrieval across the project, same harness, matched cache |
| |
| | model | 16,360 | |
| |---|---| |
| | armA_c (flux, 400 steps) | 33.0 | |
| | l2 (b2, 400 steps) | 35.0 | |
| | ext32k (b4, 750 steps) | 45.0 | |
| | **b4 full epoch (2227 steps)** | **58.0** | |
|
|
| ## Conclusion, and what it is not |
|
|
| Training steps AT the target length is the lever that moved the slope. Document |
| length shifted the intercept only (l3 vs l2 at 16k was +0.0pp); the YaRN cache fix |
| was a one-off +27pp at the window boundary; model scale remains untested. The b3 |
| long arm at 16k found the same thing independently (-29 -> -19 -> -11 pp/doubling |
| across 400/1000/2000 steps). |
|
|
| Only the 32,744 cell is individually resolvable at N=100. The slope figures are |
| derived from two points each and inherit those error bars. |
|
|
| ## KNOWN ISSUE -- CP_MAX_LR and CP_MIN_LR do not exist |
|
|
| bluscriptCP.cpp hardcodes max_lr=6e-4 (line 131) and min_lr=6e-5 (line 132), and |
| the env-parsing block never reads CP_MAX_LR or CP_MIN_LR -- there is not even a |
| float env helper. Every script in this project passed them as a no-op. So the real |
| schedule has always been 6e-4 * CP_REWARMUP_PEAK(0.3) = 1.8e-4 peak, 6e-5 floor, |
| verified from ext32k's log (1.8000e-04 at steps 99..649, 6.0030e-05 at 749). |
|
|
| This is benign for comparability: EVERY run in this project used the same values, |
| so l1/l2/l3, ext32k, the b3 long arm and this run are all mutually comparable. Only |
| prose descriptions of the LR were wrong. The LR was deliberately NOT changed here, |
| because doing so would have broken continuity with ext32k's first 750 steps. |
|
|
| ## The resume that required the sidecar |
|
|
| A first attempt at this continuation was aborted at step 763. The .zero_g1 |
| optimizer sidecar had not been uploaded with the step-750 checkpoint, and per |
| bluscriptCP.cpp:2032 a bf16 run then "resumes with group 1's Adam m/v at zero". |
| Observed effect: loss 2.95 -> 5.52 at the first optimizer step, gradient norm |
| 0.15 -> 9.55. After recovering the sidecar from the original instance the same |
| resume gave loss 2.95 -> 3.16, norm 0.15 -> 0.18, and the run completed with |
| median norm 0.20. ALWAYS upload the .zero_g1 sidecar alongside the .ckpt. |
|
|
| ## Raw eval output |
|
|
| ### step 1100, PROMPT_LEN=8192 |
| ``` |
| task,b4c1100
|
| niah_single,55.0
|
| ``` |
| |
| ### step 1100, PROMPT_LEN=16360 |
| ``` |
| task,b4c1100
|
| niah_single,40.0
|
| ``` |
| |
| ### step 1100, PROMPT_LEN=32744 |
| ``` |
| task,b4c1100
|
| niah_single,24.0
|
| ``` |
|
|
| ### step 1650, PROMPT_LEN=8192 |
| ``` |
| task,b4c1650
|
| niah_single,64.0
|
| ``` |
| |
| ### step 1650, PROMPT_LEN=16360 |
| ``` |
| task,b4c1650
|
| niah_single,55.0
|
| ``` |
| |
| ### step 1650, PROMPT_LEN=32744 |
| ``` |
| task,b4c1650
|
| niah_single,33.0
|
| ``` |
|
|
| ### step 2227, PROMPT_LEN=8192 |
| ``` |
| task,b4c2227
|
| niah_single,70.0
|
| ``` |
| |
| ### step 2227, PROMPT_LEN=16360 |
| ``` |
| task,b4c2227
|
| niah_single,58.0
|
| ``` |
| |
| ### step 2227, PROMPT_LEN=32744 |
| ``` |
| task,b4c2227
|
| niah_single,43.0
|
| ``` |
|
|
|
|
