32k -> 64k YaRN extension on b5 -- 2026-09-10

Parent: b4 full-epoch step 2227 (best 32k model: 70.0/58.0/43.0 at 8k/16k/32k). Data: b5_65536_131072 -- min doc 65,536 == the window, so every sample is ONE contiguous document. YaRN: SCALE=2 ORIG_MAXPOS=32768 -> 65536 exactly. Train: 695 steps x 524288 = .364 B tokens = 42% of b5's 0.866 B train split (no wrap). WSD warmup 100 / stable 593 / decay tail 2 -- final ckpt DECAYED. LR peak 1.8e-4, floor 6e-5 (CP_MAX_LR is a no-op; see B4 results doc). 4x H200, cp=4, B=1 x grad_accum 8. Eval: niah_single, N=100, greedy, shared haystack b2_8192_16384 val.

Results

context	niah_single
8192	67.0

16360	65.0

32744	52.0

65512	29.0

For comparison, the 32k parent at the same lengths: 70.0 / 58.0 / 43.0 / n/a (it cannot run at 64k -- its RoPE cache is sized 32,768).

NOTE: this is a PARTIAL b5 epoch, sized to a wall-clock deadline. It is a first 64k data point, not a converged extension.