logs/cont64k/RESULTS.md

64k b5 continuation to a full epoch -- 2026-09-11

Parent: ext64k run3 step 695 (67.0/65.0/52.0/29.0 at 8k/16k/32k/64k), loaded WEIGHTS-ONLY: ZeRO sidecars cannot cross world_size 4 -> 8. Data: b5_65536_131072, 9 shards = .865 B tokens. Train: 1651 steps x 524288 = .865 B tokens = a full b5 epoch. The loader restarts at token 0, so b5's first 42% is seen a second time and the rest for the first time. WSD warmup 100 / stable 1451 / decay tail 100. 8x H100, cp=8, B=1 x grad_accum 8. Eval: niah_single, N=100, greedy, shared haystack b2_8192_16384 val.

Results

context	niah_single
8192	45.0
16360	17.0
32744	2.0
65512	1.0

Parent for comparison: 67.0 / 65.0 / 52.0 / 29.0.