File size: 3,697 Bytes
345518d 7b11354 c497868 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | [14:27:33] === CONTINUE 64k b5 to a FULL EPOCH (8x H100, cp=8) ===
[14:27:33] parent: ext64k run3 step 695 (64k model), weights-only via CP_INIT_FROM
[14:27:33] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536 OK
[14:27:33] b5: 9 train shards = .865 B tokens on disk
[14:27:33] disk: 123 GB free
[14:27:33] PHASE 1: 3-step probe (8,192 tok/rank; extrapolates to ~67 GB of 81.5 GB)
[14:28:49] PROBE OK: peak
[14:28:50] probe rate (UPPER BOUND, inflated by fixed costs x grad_accum): 0.0 s/step
[14:28:50] PHASE 2: 1651 steps x 524288 = .865 B tokens = a full b5 epoch
[14:28:50] WSD: warmup 100, stable 1451, decay tail 100 -> final checkpoint IS decayed
[14:30:03] FATAL: run 2 wrote NO checkpoint
[14:37:39] === CONTINUE 64k b5 to a FULL EPOCH (8x H100, cp=8) ===
[14:37:39] parent: ext64k run3 step 695 (64k model), weights-only via CP_INIT_FROM
[14:37:39] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536 OK
[14:37:39] b5: 9 train shards = .865 B tokens on disk
[14:37:39] disk: 123 GB free
[14:37:39] PHASE 1: 3-step probe (8,192 tok/rank; extrapolates to ~67 GB of 81.5 GB)
[14:40:04] PROBE OK: 3 steps, peak mem: 71843MB of 81559 MiB/card
[14:40:04] probe rate (UPPER BOUND, inflated by fixed costs x grad_accum): 84.100 s/step
[14:40:04] PHASE 2: 1651 steps x 524288 = .865 B tokens = a full b5 epoch
[14:40:04] WSD: warmup 100, stable 1451, decay tail 100 -> final checkpoint IS decayed
[18:22:46] cont64k -> run 4, blumodelcp_run4_step_1651.ckpt, loss: 1.143309
[18:22:46] peak mem mem: 72099MB
[18:22:46] final lr lr 6.0030e-05 (want ~6e-05 = decayed)
[18:22:46] PHASE 3: eval cont64k -- 4 lengths, one GPU each
[19:07:25] len 8192 -> niah_single,45.0
[19:07:25] len 16360 -> niah_single,17.0
[19:07:25] len 32744 -> niah_single,2.0
[19:07:25] len 65512 -> niah_single,1.0
[19:07:25] PHASE 4: upload
[19:07:26] sent docs/CONT64K_B5_RESULTS_2026-09-11.md
[19:07:29] sent logs/cont64k_len_8192/
[19:07:32] sent logs/cont64k_len_16360/
[19:07:35] sent logs/cont64k_len_32744/
[19:07:39] sent logs/cont64k_len_65512/
[19:07:42] sent logs/cont64k/
[19:07:43] sent scripts/cont64k_b5_h100.sh
[19:07:43] uploading the model itself (the part that was lost yesterday)
[19:07:50] sent checkpoints/cont64k_b5_rank0_step_1651.ckpt
VERIFIED checkpoints/cont64k_b5_rank0_step_1651.ckpt local=3,750,897,182 remote=3750897182
[19:07:53] sent checkpoints/cont64k_b5_rank0_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank0_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:07:56] sent checkpoints/cont64k_b5_rank1_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank1_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:07:59] sent checkpoints/cont64k_b5_rank2_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank2_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:02] sent checkpoints/cont64k_b5_rank3_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank3_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:05] sent checkpoints/cont64k_b5_rank4_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank4_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:07] sent checkpoints/cont64k_b5_rank5_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank5_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:10] sent checkpoints/cont64k_b5_rank6_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank6_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:13] sent checkpoints/cont64k_b5_rank7_step_1651.zero_g1
VERIFIED checkpoints/cont64k_b5_rank7_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:13] === CONT64K DONE -- all uploads verified, safe to tear down ===
|