Upload docs/B4_FULL_EPOCH_32K_RESULTS_2026-09-10.md with huggingface_hub

docs/B4_FULL_EPOCH_32K_RESULTS_2026-09-10.md

@@ -0,0 +1,144 @@
+# b4 full epoch at 32k -- 2026-09-10
+
+Continuation of the 16k->32k YaRN extension through the whole of b4.
+
+## Setup
+
+| | |
+|---|---|
+| model | 1.06B (36 layers, d_model 1536, 12q/4kv, hd 128, SwiGLU 4096) |
+| window | 32,768 (YaRN SCALE=2 ORIG_MAXPOS=16384, product == 32768 exactly) |
+| parent | ext32k step 750 (itself extended from l2, the best 16k arm) |
+| data | b4_32768_65536 -- min doc 32,768 == the window, so EVERY sample is one contiguous document |
+| tokens | steps 750 -> 2227 = 1.168 B of b4's 1.171 B train split; b4 seen EXACTLY ONCE, no wrap |
+| batch | B=1 x grad_accum 16 = 524,288 tokens/step, identical to every 16k arm |
+| LR | peak 1.8e-4, floor 6e-5 (see the note below), WSD: re-warm 100 from step 750, flat to 2027, 200-step decay tail |
+| hardware | 2x H200, cp=2, peak 107.4 GB of 143 GB |
+| eval | niah_single, N=100, greedy, substring match, one shared haystack (b2_8192_16384 val) |
+
+## Results
+
+niah_single accuracy %, +-95% Wilson half-width
+
+| step | b4 seen | 8,192 | 16,360 | 32,744 | decayed |
+|---|---|---|---|---|---|
+| 750 | 34% | 60.0 +-9.4 | 45.0 +-9.6 | 27.0 +-8.6 | yes |
+| 1100 | 49% | 55.0 +-9.6 | 40.0 +-9.4 | 24.0 +-8.3 | no |
+| 1650 | 74% | 64.0 +-9.2 | 55.0 +-9.6 | 33.0 +-9.1 | no |
+| **2227** | **100%** | **70.0 +-8.8** | **58.0 +-9.5** | **43.0 +-9.5** | **yes** |
+
+The 1100 and 1650 checkpoints sit mid-plateau at constant LR and are UNDECAYED, so
+they understate. Only 750 and 2227 are decayed and mutually comparable -- which is
+why 1100 reads BELOW 750 and is not a regression.
+
+### The comparison: step 750 vs 2227, both decayed
+
+| length | 750 | 2227 | delta | combined 95% bar | resolvable |
+|---|---|---|---|---|---|
+| 8,192 | 60.0 | 70.0 | +10.0 | +-12.9 | no |
+| 16,360 | 45.0 | 58.0 | +13.0 | +-13.5 | no |
+| 32,744 | 27.0 | 43.0 | **+16.0** | +-12.8 | **YES** |
+
+The gains GROW with length (+10, +13, +16). That is the slope flattening, not a
+uniform lift. Slope on the decayed points: 8k->16k -15.0 -> -12.0 pp/doubling;
+16k->32k -18.0 -> -15.0.
+
+### 16k retrieval across the project, same harness, matched cache
+
+| model | 16,360 |
+|---|---|
+| armA_c (flux, 400 steps) | 33.0 |
+| l2 (b2, 400 steps) | 35.0 |
+| ext32k (b4, 750 steps) | 45.0 |
+| **b4 full epoch (2227 steps)** | **58.0** |
+
+## Conclusion, and what it is not
+
+Training steps AT the target length is the lever that moved the slope. Document
+length shifted the intercept only (l3 vs l2 at 16k was +0.0pp); the YaRN cache fix
+was a one-off +27pp at the window boundary; model scale remains untested. The b3
+long arm at 16k found the same thing independently (-29 -> -19 -> -11 pp/doubling
+across 400/1000/2000 steps).
+
+Only the 32,744 cell is individually resolvable at N=100. The slope figures are
+derived from two points each and inherit those error bars.
+
+## KNOWN ISSUE -- CP_MAX_LR and CP_MIN_LR do not exist
+
+bluscriptCP.cpp hardcodes max_lr=6e-4 (line 131) and min_lr=6e-5 (line 132), and
+the env-parsing block never reads CP_MAX_LR or CP_MIN_LR -- there is not even a
+float env helper. Every script in this project passed them as a no-op. So the real
+schedule has always been 6e-4 * CP_REWARMUP_PEAK(0.3) = 1.8e-4 peak, 6e-5 floor,
+verified from ext32k's log (1.8000e-04 at steps 99..649, 6.0030e-05 at 749).
+
+This is benign for comparability: EVERY run in this project used the same values,
+so l1/l2/l3, ext32k, the b3 long arm and this run are all mutually comparable. Only
+prose descriptions of the LR were wrong. The LR was deliberately NOT changed here,
+because doing so would have broken continuity with ext32k's first 750 steps.
+
+## The resume that required the sidecar
+
+A first attempt at this continuation was aborted at step 763. The .zero_g1
+optimizer sidecar had not been uploaded with the step-750 checkpoint, and per
+bluscriptCP.cpp:2032 a bf16 run then "resumes with group 1's Adam m/v at zero".
+Observed effect: loss 2.95 -> 5.52 at the first optimizer step, gradient norm
+0.15 -> 9.55. After recovering the sidecar from the original instance the same
+resume gave loss 2.95 -> 3.16, norm 0.15 -> 0.18, and the run completed with
+median norm 0.20. ALWAYS upload the .zero_g1 sidecar alongside the .ckpt.
+
+## Raw eval output
+
+### step 1100, PROMPT_LEN=8192
+```
+task,b4c1100
+niah_single,55.0
+```
+
+### step 1100, PROMPT_LEN=16360
+```
+task,b4c1100
+niah_single,40.0
+```
+
+### step 1100, PROMPT_LEN=32744
+```
+task,b4c1100
+niah_single,24.0
+```
+
+### step 1650, PROMPT_LEN=8192
+```
+task,b4c1650
+niah_single,64.0
+```
+
+### step 1650, PROMPT_LEN=16360
+```
+task,b4c1650
+niah_single,55.0
+```
+
+### step 1650, PROMPT_LEN=32744
+```
+task,b4c1650
+niah_single,33.0
+```
+
+### step 2227, PROMPT_LEN=8192
+```
+task,b4c2227
+niah_single,70.0
+```
+
+### step 2227, PROMPT_LEN=16360
+```
+task,b4c2227
+niah_single,58.0
+```
+
+### step 2227, PROMPT_LEN=32744
+```
+task,b4c2227
+niah_single,43.0
+```
+