Upload docs/CONT64K_B5_RESULTS_2026-09-11.md with huggingface_hub
Browse files
docs/CONT64K_B5_RESULTS_2026-09-11.md
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 0 |
|
|
|
|
|
| 1 |
|
|
|
|
|
| 2 |
|
|
|
|
|
| 3 |
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 64k b5 continuation to a full epoch -- 2026-09-11
|
| 2 |
+
|
| 3 |
+
Parent: ext64k run3 step 695 (67.0/65.0/52.0/29.0 at 8k/16k/32k/64k),
|
| 4 |
+
loaded WEIGHTS-ONLY: ZeRO sidecars cannot cross world_size 4 -> 8.
|
| 5 |
+
Data: b5_65536_131072, 9 shards = .865 B tokens.
|
| 6 |
+
Train: 1651 steps x 524288 = .865 B tokens = a full b5 epoch.
|
| 7 |
+
The loader restarts at token 0, so b5's first 42% is seen a
|
| 8 |
+
second time and the rest for the first time.
|
| 9 |
+
WSD warmup 100 / stable 1451 / decay tail 100.
|
| 10 |
+
8x H100, cp=8, B=1 x grad_accum 8.
|
| 11 |
+
Eval: niah_single, N=100, greedy, shared haystack b2_8192_16384 val.
|
| 12 |
+
|
| 13 |
+
## Results
|
| 14 |
+
|
| 15 |
+
| context | niah_single |
|
| 16 |
+
|---|---|
|
| 17 |
+
| 8192 | 45.0
|
| 18 |
|
|
| 19 |
+
| 16360 | 17.0
|
| 20 |
|
|
| 21 |
+
| 32744 | 2.0
|
| 22 |
|
|
| 23 |
+
| 65512 | 1.0
|
| 24 |
|
|
| 25 |
+
|
| 26 |
+
Parent for comparison: 67.0 / 65.0 / 52.0 / 29.0.
|