| [14:27:33] === CONTINUE 64k b5 to a FULL EPOCH (8x H100, cp=8) === |
| [14:27:33] parent: ext64k run3 step 695 (64k model), weights-only via CP_INIT_FROM |
| [14:27:33] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536 OK |
| [14:27:33] b5: 9 train shards = .865 B tokens on disk |
| [14:27:33] disk: 123 GB free |
| [14:27:33] PHASE 1: 3-step probe (8,192 tok/rank; extrapolates to ~67 GB of 81.5 GB) |
| [14:28:49] PROBE OK: peak |
| [14:28:50] probe rate (UPPER BOUND, inflated by fixed costs x grad_accum): 0.0 s/step |
| [14:28:50] PHASE 2: 1651 steps x 524288 = .865 B tokens = a full b5 epoch |
| [14:28:50] WSD: warmup 100, stable 1451, decay tail 100 -> final checkpoint IS decayed |
| [14:30:03] FATAL: run 2 wrote NO checkpoint |
| [14:37:39] === CONTINUE 64k b5 to a FULL EPOCH (8x H100, cp=8) === |
| [14:37:39] parent: ext64k run3 step 695 (64k model), weights-only via CP_INIT_FROM |
| [14:37:39] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536 OK |
| [14:37:39] b5: 9 train shards = .865 B tokens on disk |
| [14:37:39] disk: 123 GB free |
| [14:37:39] PHASE 1: 3-step probe (8,192 tok/rank; extrapolates to ~67 GB of 81.5 GB) |
| [14:40:04] PROBE OK: 3 steps, peak mem: 71843MB of 81559 MiB/card |
| [14:40:04] probe rate (UPPER BOUND, inflated by fixed costs x grad_accum): 84.100 s/step |
| [14:40:04] PHASE 2: 1651 steps x 524288 = .865 B tokens = a full b5 epoch |
| [14:40:04] WSD: warmup 100, stable 1451, decay tail 100 -> final checkpoint IS decayed |
| [18:22:46] cont64k -> run 4, blumodelcp_run4_step_1651.ckpt, loss: 1.143309 |
| [18:22:46] peak mem mem: 72099MB |
| [18:22:46] final lr lr 6.0030e-05 (want ~6e-05 = decayed) |
| [18:22:46] PHASE 3: eval cont64k |
| [19:07:25] len 8192 -> niah_single,45.0
|
| [19:07:25] len 16360 -> niah_single,17.0
|
| [19:07:25] len 32744 -> niah_single,2.0
|
| [19:07:25] len 65512 -> niah_single,1.0
|
| [19:07:25] PHASE 4: upload |
| [19:07:26] sent docs/CONT64K_B5_RESULTS_2026-09-11.md |
| [19:07:29] sent logs/cont64k_len_8192/ |
| [19:07:32] sent logs/cont64k_len_16360/ |
| [19:07:35] sent logs/cont64k_len_32744/ |
| [19:07:39] sent logs/cont64k_len_65512/ |
| [19:07:42] sent logs/cont64k/ |
| [19:07:43] sent scripts/cont64k_b5_h100.sh |
| [19:07:43] uploading the model itself (the part that was lost yesterday) |
| [19:07:50] sent checkpoints/cont64k_b5_rank0_step_1651.ckpt |
| VERIFIED checkpoints/cont64k_b5_rank0_step_1651.ckpt local=3,750,897,182 remote=3750897182 |
| [19:07:53] sent checkpoints/cont64k_b5_rank0_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank0_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:07:56] sent checkpoints/cont64k_b5_rank1_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank1_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:07:59] sent checkpoints/cont64k_b5_rank2_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank2_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:08:02] sent checkpoints/cont64k_b5_rank3_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank3_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:08:05] sent checkpoints/cont64k_b5_rank4_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank4_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:08:07] sent checkpoints/cont64k_b5_rank5_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank5_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:08:10] sent checkpoints/cont64k_b5_rank6_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank6_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:08:13] sent checkpoints/cont64k_b5_rank7_step_1651.zero_g1 |
| VERIFIED checkpoints/cont64k_b5_rank7_step_1651.zero_g1 local=116,082,596 remote=116082596 |
| [19:08:13] === CONT64K DONE |
|
|
