| waiting for the running b3 experiment to release the GPUs |
| waiting for the running b3 experiment to release the GPUs |
| === long b3 arm: START === |
| parent run7' (same as l3@400, so l3@400 vs this isolates STEP COUNT) |
| steps=4000 stable=4000 -> decay_start=4100 (> 4000, so NO decay: flat 3.0e-4) |
| ckpt every 25 steps, keep 2 (= 34.56 GB), free 83G |
| tokens 2097152000 = 70.0% of b3's 2.93 B |
| trainer exited rc=1 |
| run number: unknown |
| LR trace: at step 100 |
| at step 2000 (must equal step 100) |
| at the last logged step |
| to continue: ./b3_long_run.sh resume <run> |
| === long b3 arm: START === |
| parent run7' (same as l3@400, so l3@400 vs this isolates STEP COUNT) |
| steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail |
| ckpt every 25 steps, keep 2 (= 34.56 GB), free 83G |
| tokens 2097152000 = 70.0% of b3's 2.93 B |
| RESUMING run 17 (anchor pinned to 0 so warmup does not restart) |
| steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail |
| ckpt every 25 steps, keep 2 (= 34.56 GB), free 2G |
| tokens 2097152000 = 70.0% of b3's 2.93 B |
| RESUMING run 17 (anchor pinned to 0 so warmup does not restart) |
| steps=4000 stable=3800 -> decay_start=3900; flat 3.0e-4 for steps 100..3899, then 100-step tail |
| ckpt every 25 steps, keep 2 (= 34.56 GB), free 41G |
| tokens 2097152000 = 70.0% of b3's 2.93 B |
|
|
