File size: 1,033 Bytes
cb0d955 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # 32k -> 64k YaRN extension on b5 -- 2026-09-10
Parent: b4 full-epoch step 2227 (best 32k model: 70.0/58.0/43.0 at 8k/16k/32k).
Data: b5_65536_131072 -- min doc 65,536 == the window, so every sample is
ONE contiguous document.
YaRN: SCALE=2 ORIG_MAXPOS=32768 -> 65536 exactly.
Train: 695 steps x 524288 = .364 B tokens
= 42% of b5's 0.866 B train split (no wrap).
WSD warmup 100 / stable 593 / decay tail 2 -- final ckpt DECAYED.
LR peak 1.8e-4, floor 6e-5 (CP_MAX_LR is a no-op; see B4 results doc).
4x H200, cp=4, B=1 x grad_accum 8.
Eval: niah_single, N=100, greedy, shared haystack b2_8192_16384 val.
## Results
| context | niah_single |
|---|---|
| 8192 | 67.0
|
| 16360 | 65.0
|
| 32744 | 52.0
|
| 65512 | 29.0
|
For comparison, the 32k parent at the same lengths: 70.0 / 58.0 / 43.0 / n/a
(it cannot run at 64k -- its RoPE cache is sized 32,768).
NOTE: this is a PARTIAL b5 epoch, sized to a wall-clock deadline. It is a
first 64k data point, not a converged extension.
|