| # 32k -> 64k YaRN extension on b5 -- 2026-09-10 |
|
|
| Parent: b4 full-epoch step 2227 (best 32k model: 70.0/58.0/43.0 at 8k/16k/32k). |
| Data: b5_65536_131072 -- min doc 65,536 == the window, so every sample is |
| ONE contiguous document. |
| YaRN: SCALE=2 ORIG_MAXPOS=32768 -> 65536 exactly. |
| Train: 695 steps x 524288 = .364 B tokens |
| = 42% of b5's 0.866 B train split (no wrap). |
| WSD warmup 100 / stable 593 / decay tail 2 -- final ckpt DECAYED. |
| LR peak 1.8e-4, floor 6e-5 (CP_MAX_LR is a no-op; see B4 results doc). |
| 4x H200, cp=4, B=1 x grad_accum 8. |
| Eval: niah_single, N=100, greedy, shared haystack b2_8192_16384 val. |
| |
| ## Results |
|
|
| | context | niah_single | |
| |---|---| |
| | 8192 | 67.0
| |
| | 16360 | 65.0
| |
| | 32744 | 52.0
| |
| | 65512 | 29.0
| |
| |
| For comparison, the 32k parent at the same lengths: 70.0 / 58.0 / 43.0 / n/a |
| (it cannot run at 64k -- its RoPE cache is sized 32,768). |
| |
| NOTE: this is a PARTIAL b5 epoch, sized to a wall-clock deadline. It is a |
| first 64k data point, not a converged extension. |
| |
