Upload folder using huggingface_hub

logs/l3long_run17/janitor.log

@@ -0,0 +1,30 @@
+[15:29:53] freed 928 MB, 45G avail
+[15:44:54] freed 928 MB, 54G avail
+[15:49:54] freed 928 MB, 57G avail
+[15:54:54] freed 928 MB, 57G avail
+[15:59:54] freed 928 MB, 57G avail
+[16:04:54] freed 928 MB, 57G avail
+[16:09:55] freed 928 MB, 57G avail
+[16:14:55] freed 928 MB, 57G avail
+[16:19:55] freed 928 MB, 57G avail
+[16:24:55] freed 928 MB, 57G avail
+[16:29:55] freed 928 MB, 57G avail
+[16:34:55] freed 928 MB, 57G avail
+[16:39:56] freed 928 MB, 57G avail
+[16:44:56] freed 928 MB, 41G avail
+[16:54:56] freed 928 MB, 41G avail
+[17:09:56] freed 928 MB, 33G avail
+[17:14:57] freed 928 MB, 36G avail
+[17:19:57] freed 928 MB, 38G avail
+[17:24:57] freed 928 MB, 41G avail
+[17:29:57] freed 928 MB, 41G avail
+[17:34:57] freed 928 MB, 41G avail
+[17:39:57] freed 928 MB, 41G avail
+[17:44:58] freed 928 MB, 41G avail
+[17:49:58] freed 928 MB, 41G avail
+[17:54:58] freed 928 MB, 41G avail
+[17:59:58] freed 928 MB, 41G avail
+[18:04:58] freed 928 MB, 41G avail
+[18:09:59] freed 928 MB, 41G avail
+[18:14:59] freed 928 MB, 41G avail
+[18:19:59] freed 928 MB, 40G avail

logs/l3long_run17/master.log

@@ -0,0 +1,26 @@
+[06:54:49] waiting for the running b3 experiment to release the GPUs
+[07:06:18] waiting for the running b3 experiment to release the GPUs
+[09:43:20] === long b3 arm: START ===
+[09:43:20] parent run7' (same as l3@400, so l3@400 vs this isolates STEP COUNT)
+[09:43:20] steps=4000  stable=4000 -> decay_start=4100 (> 4000, so NO decay: flat 3.0e-4)
+[09:43:20] ckpt every 25 steps, keep 2 (= 34.56 GB), free 83G
+[09:43:20] tokens 2097152000 = 70.0% of b3's 2.93 B
+[09:43:22] trainer exited rc=1
+[09:43:22] run number: unknown
+[09:43:22] LR trace:  at step 100
+[09:43:22]            at step 2000  (must equal step 100)
+[09:43:22]            at the last logged step
+[09:43:22] to continue:  ./b3_long_run.sh resume <run>
+[09:46:00] === long b3 arm: START ===
+[09:46:00] parent run7' (same as l3@400, so l3@400 vs this isolates STEP COUNT)
+[09:46:00] steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail
+[09:46:01] ckpt every 25 steps, keep 2 (= 34.56 GB), free 83G
+[09:46:01] tokens 2097152000 = 70.0% of b3's 2.93 B
+[13:17:43] RESUMING run 17 (anchor pinned to 0 so warmup does not restart)
+[13:17:43] steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail
+[13:17:43] ckpt every 25 steps, keep 2 (= 34.56 GB), free 2G
+[13:17:43] tokens 2097152000 = 70.0% of b3's 2.93 B
+[16:49:01] RESUMING run 17 (anchor pinned to 0 so warmup does not restart)
+[16:49:01] steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail
+[16:49:01] ckpt every 25 steps, keep 2 (= 34.56 GB), free 41G
+[16:49:01] tokens 2097152000 = 70.0% of b3's 2.93 B

logs/l3long_run17/runnum

@@ -0,0 +1 @@
+17