b4 full epoch at 32k -- 2026-09-10
Continuation of the 16k->32k YaRN extension through the whole of b4.
Setup
| model | 1.06B (36 layers, d_model 1536, 12q/4kv, hd 128, SwiGLU 4096) |
| window | 32,768 (YaRN SCALE=2 ORIG_MAXPOS=16384, product == 32768 exactly) |
| parent | ext32k step 750 (itself extended from l2, the best 16k arm) |
| data | b4_32768_65536 -- min doc 32,768 == the window, so EVERY sample is one contiguous document |
| tokens | steps 750 -> 2227 = 1.168 B of b4's 1.171 B train split; b4 seen EXACTLY ONCE, no wrap |
| batch | B=1 x grad_accum 16 = 524,288 tokens/step, identical to every 16k arm |
| LR | peak 1.8e-4, floor 6e-5 (see the note below), WSD: re-warm 100 from step 750, flat to 2027, 200-step decay tail |
| hardware | 2x H200, cp=2, peak 107.4 GB of 143 GB |
| eval | niah_single, N=100, greedy, substring match, one shared haystack (b2_8192_16384 val) |
Results
niah_single accuracy %, +-95% Wilson half-width
| step | b4 seen | 8,192 | 16,360 | 32,744 | decayed |
|---|---|---|---|---|---|
| 750 | 34% | 60.0 +-9.4 | 45.0 +-9.6 | 27.0 +-8.6 | yes |
| 1100 | 49% | 55.0 +-9.6 | 40.0 +-9.4 | 24.0 +-8.3 | no |
| 1650 | 74% | 64.0 +-9.2 | 55.0 +-9.6 | 33.0 +-9.1 | no |
| 2227 | 100% | 70.0 +-8.8 | 58.0 +-9.5 | 43.0 +-9.5 | yes |
The 1100 and 1650 checkpoints sit mid-plateau at constant LR and are UNDECAYED, so they understate. Only 750 and 2227 are decayed and mutually comparable -- which is why 1100 reads BELOW 750 and is not a regression.
The comparison: step 750 vs 2227, both decayed
| length | 750 | 2227 | delta | combined 95% bar | resolvable |
|---|---|---|---|---|---|
| 8,192 | 60.0 | 70.0 | +10.0 | +-12.9 | no |
| 16,360 | 45.0 | 58.0 | +13.0 | +-13.5 | no |
| 32,744 | 27.0 | 43.0 | +16.0 | +-12.8 | YES |
The gains GROW with length (+10, +13, +16). That is the slope flattening, not a uniform lift. Slope on the decayed points: 8k->16k -15.0 -> -12.0 pp/doubling; 16k->32k -18.0 -> -15.0.
16k retrieval across the project, same harness, matched cache
| model | 16,360 |
|---|---|
| armA_c (flux, 400 steps) | 33.0 |
| l2 (b2, 400 steps) | 35.0 |
| ext32k (b4, 750 steps) | 45.0 |
| b4 full epoch (2227 steps) | 58.0 |
Conclusion, and what it is not
Training steps AT the target length is the lever that moved the slope. Document length shifted the intercept only (l3 vs l2 at 16k was +0.0pp); the YaRN cache fix was a one-off +27pp at the window boundary; model scale remains untested. The b3 long arm at 16k found the same thing independently (-29 -> -19 -> -11 pp/doubling across 400/1000/2000 steps).
Only the 32,744 cell is individually resolvable at N=100. The slope figures are derived from two points each and inherit those error bars.
KNOWN ISSUE -- CP_MAX_LR and CP_MIN_LR do not exist
bluscriptCP.cpp hardcodes max_lr=6e-4 (line 131) and min_lr=6e-5 (line 132), and the env-parsing block never reads CP_MAX_LR or CP_MIN_LR -- there is not even a float env helper. Every script in this project passed them as a no-op. So the real schedule has always been 6e-4 * CP_REWARMUP_PEAK(0.3) = 1.8e-4 peak, 6e-5 floor, verified from ext32k's log (1.8000e-04 at steps 99..649, 6.0030e-05 at 749).
This is benign for comparability: EVERY run in this project used the same values, so l1/l2/l3, ext32k, the b3 long arm and this run are all mutually comparable. Only prose descriptions of the LR were wrong. The LR was deliberately NOT changed here, because doing so would have broken continuity with ext32k's first 750 steps.
The resume that required the sidecar
A first attempt at this continuation was aborted at step 763. The .zero_g1 optimizer sidecar had not been uploaded with the step-750 checkpoint, and per bluscriptCP.cpp:2032 a bf16 run then "resumes with group 1's Adam m/v at zero". Observed effect: loss 2.95 -> 5.52 at the first optimizer step, gradient norm 0.15 -> 9.55. After recovering the sidecar from the original instance the same resume gave loss 2.95 -> 3.16, norm 0.15 -> 0.18, and the run completed with median norm 0.20. ALWAYS upload the .zero_g1 sidecar alongside the .ckpt.
Raw eval output
step 1100, PROMPT_LEN=8192
task,b4c1100
niah_single,55.0
step 1100, PROMPT_LEN=16360
task,b4c1100
niah_single,40.0
step 1100, PROMPT_LEN=32744
task,b4c1100
niah_single,24.0
step 1650, PROMPT_LEN=8192
task,b4c1650
niah_single,64.0
step 1650, PROMPT_LEN=16360
task,b4c1650
niah_single,55.0
step 1650, PROMPT_LEN=32744
task,b4c1650
niah_single,33.0
step 2227, PROMPT_LEN=8192
task,b4c2227
niah_single,70.0
step 2227, PROMPT_LEN=16360
task,b4c2227
niah_single,58.0
step 2227, PROMPT_LEN=32744
task,b4c2227
niah_single,43.0