[06:54:49] waiting for the running b3 experiment to release the GPUs
[07:06:18] waiting for the running b3 experiment to release the GPUs
[09:43:20] === long b3 arm: START ===
[09:43:20] parent run7' (same as l3@400, so l3@400 vs this isolates STEP COUNT)
[09:43:20] steps=4000  stable=4000 -> decay_start=4100 (> 4000, so NO decay: flat 3.0e-4)
[09:43:20] ckpt every 25 steps, keep 2 (= 34.56 GB), free 83G
[09:43:20] tokens 2097152000 = 70.0% of b3's 2.93 B
[09:43:22] trainer exited rc=1
[09:43:22] run number: unknown
[09:43:22] LR trace:  at step 100
[09:43:22]            at step 2000  (must equal step 100)
[09:43:22]            at the last logged step
[09:43:22] to continue:  ./b3_long_run.sh resume <run>
[09:46:00] === long b3 arm: START ===
[09:46:00] parent run7' (same as l3@400, so l3@400 vs this isolates STEP COUNT)
[09:46:00] steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail
[09:46:01] ckpt every 25 steps, keep 2 (= 34.56 GB), free 83G
[09:46:01] tokens 2097152000 = 70.0% of b3's 2.93 B
[13:17:43] RESUMING run 17 (anchor pinned to 0 so warmup does not restart)
[13:17:43] steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail
[13:17:43] ckpt every 25 steps, keep 2 (= 34.56 GB), free 2G
[13:17:43] tokens 2097152000 = 70.0% of b3's 2.93 B
[16:49:01] RESUMING run 17 (anchor pinned to 0 so warmup does not restart)
[16:49:01] steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail
[16:49:01] ckpt every 25 steps, keep 2 (= 34.56 GB), free 41G
[16:49:01] tokens 2097152000 = 70.0% of b3's 2.93 B