[14:27:33] === CONTINUE 64k b5 to a FULL EPOCH (8x H100, cp=8) ===
[14:27:33] parent: ext64k run3 step 695 (64k model), weights-only via CP_INIT_FROM
[14:27:33] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536  OK
[14:27:33] b5: 9 train shards = .865 B tokens on disk
[14:27:33] disk: 123 GB free
[14:27:33] PHASE 1: 3-step probe (8,192 tok/rank; extrapolates to ~67 GB of 81.5 GB)
[14:28:49] PROBE OK: peak 
[14:28:50] probe rate (UPPER BOUND, inflated by fixed costs x grad_accum): 0.0 s/step
[14:28:50] PHASE 2: 1651 steps x 524288 = .865 B tokens = a full b5 epoch
[14:28:50]   WSD: warmup 100, stable 1451, decay tail 100 -> final checkpoint IS decayed
[14:30:03] FATAL: run 2 wrote NO checkpoint
[14:37:39] === CONTINUE 64k b5 to a FULL EPOCH (8x H100, cp=8) ===
[14:37:39] parent: ext64k run3 step 695 (64k model), weights-only via CP_INIT_FROM
[14:37:39] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536  OK
[14:37:39] b5: 9 train shards = .865 B tokens on disk
[14:37:39] disk: 123 GB free
[14:37:39] PHASE 1: 3-step probe (8,192 tok/rank; extrapolates to ~67 GB of 81.5 GB)
[14:40:04] PROBE OK: 3 steps, peak mem: 71843MB of 81559 MiB/card
[14:40:04] probe rate (UPPER BOUND, inflated by fixed costs x grad_accum): 84.100 s/step
[14:40:04] PHASE 2: 1651 steps x 524288 = .865 B tokens = a full b5 epoch
[14:40:04]   WSD: warmup 100, stable 1451, decay tail 100 -> final checkpoint IS decayed
[18:22:46] cont64k -> run 4, blumodelcp_run4_step_1651.ckpt, loss: 1.143309
[18:22:46]   peak mem mem: 72099MB
[18:22:46]   final lr lr 6.0030e-05  (want ~6e-05 = decayed)
[18:22:46] PHASE 3: eval cont64k -- 4 lengths, one GPU each
[19:07:25]   len 8192 -> niah_single,45.0
[19:07:25]   len 16360 -> niah_single,17.0
[19:07:25]   len 32744 -> niah_single,2.0
[19:07:25]   len 65512 -> niah_single,1.0
[19:07:25] PHASE 4: upload
[19:07:26]   sent docs/CONT64K_B5_RESULTS_2026-09-11.md
[19:07:29]   sent logs/cont64k_len_8192/
[19:07:32]   sent logs/cont64k_len_16360/
[19:07:35]   sent logs/cont64k_len_32744/
[19:07:39]   sent logs/cont64k_len_65512/
[19:07:42]   sent logs/cont64k/
[19:07:43]   sent scripts/cont64k_b5_h100.sh
[19:07:43] uploading the model itself (the part that was lost yesterday)
[19:07:50]   sent checkpoints/cont64k_b5_rank0_step_1651.ckpt
  VERIFIED checkpoints/cont64k_b5_rank0_step_1651.ckpt local=3,750,897,182 remote=3750897182
[19:07:53]   sent checkpoints/cont64k_b5_rank0_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank0_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:07:56]   sent checkpoints/cont64k_b5_rank1_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank1_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:07:59]   sent checkpoints/cont64k_b5_rank2_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank2_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:02]   sent checkpoints/cont64k_b5_rank3_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank3_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:05]   sent checkpoints/cont64k_b5_rank4_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank4_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:07]   sent checkpoints/cont64k_b5_rank5_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank5_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:10]   sent checkpoints/cont64k_b5_rank6_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank6_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:13]   sent checkpoints/cont64k_b5_rank7_step_1651.zero_g1
  VERIFIED checkpoints/cont64k_b5_rank7_step_1651.zero_g1 local=116,082,596 remote=116082596
[19:08:13] === CONT64K DONE -- all uploads verified, safe to tear down ===