Upload logs/mix64k/finish.log with huggingface_hub
Browse files- logs/mix64k/finish.log +49 -0
logs/mix64k/finish.log
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[11:57:40] === corrected finisher armed ===
|
| 2 |
+
[11:57:40] waiting for the trainer to exit (it runs to step 1300, then frees the GPUs)
|
| 3 |
+
[12:02:42] trainer exited; last step: step 1299
|
| 4 |
+
[12:02:42] preserved checkpoints: 12
|
| 5 |
+
[12:02:42] === evaluating the FINAL checkpoint, step 1200 ===
|
| 6 |
+
[12:02:42] EVAL 1200 from keep_step_1200.ckpt -- 4 lengths, one GPU each
|
| 7 |
+
[12:15:42] 1200 len 8192 -> niah_single,83.0
|
| 8 |
+
[12:15:42] 1200 len 16360 -> niah_single,70.0
|
| 9 |
+
[12:15:42] 1200 len 32744 -> niah_single,50.0
|
| 10 |
+
parent {8192: 67.0, 16360: 65.0, 32744: 52.0, 65512: 29.0}
|
| 11 |
+
pure-b5 {8192: 45.0, 16360: 17.0, 32744: 2.0, 65512: 1.0}
|
| 12 |
+
mix@1200 {8192: 83.0, 16360: 70.0, 32744: 50.0}
|
| 13 |
+
wrote mix64k_results_1200.png (397 KB)
|
| 14 |
+
wrote mix64k_results_1200.html (531 KB)
|
| 15 |
+
[12:15:43] plotted mix64k_results_1200.{png,html}
|
| 16 |
+
VERIFIED plots/mix64k_results_1200.png local=407,387 remote=407387
|
| 17 |
+
VERIFIED plots/mix64k_results_1200.html local=544,619 remote=544619
|
| 18 |
+
[12:15:51] EARLY PLOT for 1200 is on HF (8k/16k/32k)
|
| 19 |
+
[12:31:34] FINAL 1200 len 8192 -> niah_single,83.0
|
| 20 |
+
[12:31:34] FINAL 1200 len 16360 -> niah_single,70.0
|
| 21 |
+
[12:31:34] FINAL 1200 len 32744 -> niah_single,50.0
|
| 22 |
+
[12:31:34] FINAL 1200 len 65512 -> niah_single,31.0
|
| 23 |
+
parent {8192: 67.0, 16360: 65.0, 32744: 52.0, 65512: 29.0}
|
| 24 |
+
pure-b5 {8192: 45.0, 16360: 17.0, 32744: 2.0, 65512: 1.0}
|
| 25 |
+
mix@1200 {8192: 83.0, 16360: 70.0, 32744: 50.0, 65512: 31.0}
|
| 26 |
+
wrote mix64k_results_1200.png (410 KB)
|
| 27 |
+
wrote mix64k_results_1200.html (548 KB)
|
| 28 |
+
[12:31:35] plotted mix64k_results_1200.{png,html}
|
| 29 |
+
VERIFIED plots/mix64k_results_1200.png local=420,109 remote=420109
|
| 30 |
+
VERIFIED plots/mix64k_results_1200.html local=561,590 remote=561590
|
| 31 |
+
[12:31:41] --- verdict for step 1200 against the parent (67/65/52/29) ---
|
| 32 |
+
length parent mix delta bar verdict
|
| 33 |
+
8192 67.0 83.0 +16.0 11.7 ok
|
| 34 |
+
16360 65.0 70.0 +5.0 12.8 ok
|
| 35 |
+
32744 52.0 50.0 -2.0 13.6 ok
|
| 36 |
+
65512 29.0 31.0 +2.0 12.5 ok
|
| 37 |
+
VERDICT GOOD no resolvable degradation at 16k/32k/64k
|
| 38 |
+
[12:31:41] verdict: GOOD
|
| 39 |
+
[12:31:41] === final is GOOD -- skipping the 1200 eval (saves ~38 min of GPU) ===
|
| 40 |
+
[12:31:41] cliff detector on the MIX run: CLEAN (onset=0 baseline=2.778 current=2.711)
|
| 41 |
+
[12:31:41] for contrast, 100% b5 from the SAME parent: CLIFF at step 1199, ended 45/17/2/1
|
| 42 |
+
[12:31:41] === uploading everything ===
|
| 43 |
+
[12:31:44] sent (dir) logs/mix64k_1200_len_16360/
|
| 44 |
+
[12:31:47] sent (dir) logs/mix64k_1200_len_32744/
|
| 45 |
+
[12:31:51] sent (dir) logs/mix64k_1200_len_65512/
|
| 46 |
+
[12:31:54] sent (dir) logs/mix64k_1200_len_8192/
|
| 47 |
+
VERIFIED logs/mix64k/master.log local=2,673 remote=2673
|
| 48 |
+
VERIFIED logs/mix64k/train.log local=317,801 remote=317801
|
| 49 |
+
VERIFIED logs/mix64k/probe.log local=4,361 remote=4361
|