Upload logs/mix64k/master.log with huggingface_hub
Browse files- logs/mix64k/master.log +36 -0
logs/mix64k/master.log
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[09:25:01] manifest: usable_steps=1716, running 1714
|
| 2 |
+
[09:25:01] === 32k -> 64k on the MIXED corpus (8x H200, cp=4 dp=2) ===
|
| 3 |
+
[09:25:01] parent: b4 epoch step 2227 (32k model), weights-only via CP_INIT_FROM
|
| 4 |
+
[09:25:01] data: Data_Loader/Data/mix50 b5_65536_131072=0.5000 b3_16384_32768=0.3000 b1_4096_8192=0.2000
|
| 5 |
+
[09:25:01] YaRN: SCALE=2 ORIG=32768 -> 65536 == CP_T=65536 OK
|
| 6 |
+
[09:25:01] WSD: warmup 100, stable 1612, tail 2 -> plateau at 1.8e-4 throughout
|
| 7 |
+
[09:25:01] evals: 695 (A/B vs ext64k 67/65/52/29), 1200 (collapse check)
|
| 8 |
+
[09:25:01] disk: 138 GB free
|
| 9 |
+
[09:25:01] PHASE 1: 3-step probe (16,384 tok/rank; measured 108,493 MB of 146,000 on H200)
|
| 10 |
+
[09:25:17] PROBE FAILED (rc=1 steps=0 peak='')
|
| 11 |
+
[09:34:40] manifest: usable_steps=1716, running 1300
|
| 12 |
+
[09:34:40] === 32k -> 64k on the MIXED corpus (8x H200, cp=4 dp=2) ===
|
| 13 |
+
[09:34:40] parent: ext64k step 695 (64k model, 67/65/52/29) -- weights-only: its 4-rank ZeRO sidecars cannot load at world_size 8
|
| 14 |
+
[09:34:40] data: Data_Loader/Data/mix50 b5_65536_131072=0.5000 b3_16384_32768=0.3000 b1_4096_8192=0.2000
|
| 15 |
+
[09:34:40] YaRN: SCALE=2 ORIG=32768 -> 65536 == CP_T=65536 OK
|
| 16 |
+
[09:34:40] WSD: warmup 100, stable 1198, tail 2 -> plateau at 1.8e-4 throughout
|
| 17 |
+
[09:34:40] evals: 695 (mid-run health), 1200 (the failed run cliffed at its step 1199)
|
| 18 |
+
[09:34:40] disk: 133 GB free
|
| 19 |
+
[09:34:40] PHASE 1: 3-step probe (16,384 tok/rank; measured 108,493 MB of 146,000 on H200)
|
| 20 |
+
[09:34:55] PROBE FAILED (rc=1 steps=0 peak='')
|
| 21 |
+
[09:36:49] manifest: usable_steps=1716, running 1300
|
| 22 |
+
[09:36:49] === 32k -> 64k on the MIXED corpus (8x H200, cp=4 dp=2) ===
|
| 23 |
+
[09:36:49] parent: ext64k step 695 (64k model, 67/65/52/29) -- weights-only: its 4-rank ZeRO sidecars cannot load at world_size 8
|
| 24 |
+
[09:36:49] data: Data_Loader/Data/mix50 b5_65536_131072=0.5000 b3_16384_32768=0.3000 b1_4096_8192=0.2000
|
| 25 |
+
[09:36:49] YaRN: SCALE=2 ORIG=32768 -> 65536 == CP_T=65536 OK
|
| 26 |
+
[09:36:49] WSD: warmup 100, stable 1198, tail 2 -> plateau at 1.8e-4 throughout
|
| 27 |
+
[09:36:49] evals: 695 (mid-run health), 1200 (the failed run cliffed at its step 1199)
|
| 28 |
+
[09:36:49] disk: 133 GB free
|
| 29 |
+
[09:36:49] PHASE 1: 3-step probe (16384 tok/rank, global_batch 131072)
|
| 30 |
+
[09:38:54] PROBE OK: 3 steps, peak mem: 105747MB
|
| 31 |
+
[09:38:54] starting fresh from the 32k parent, target step 695
|
| 32 |
+
VERIFIED checkpoints/mix64k_step_300.ckpt local=3,750,897,182 remote=3750897182
|
| 33 |
+
VERIFIED checkpoints/mix64k_step_600.ckpt local=3,750,897,182 remote=3750897182
|
| 34 |
+
VERIFIED checkpoints/mix64k_step_900.ckpt local=3,750,897,182 remote=3750897182
|
| 35 |
+
VERIFIED checkpoints/mix64k_step_1200.ckpt local=3,750,897,182 remote=3750897182
|
| 36 |
+
[12:02:20] FATAL: step 695 checkpoint missing (0/8 ranks)
|