[11:12:40] watching for blumodelcp_run17_step_1000.ckpt on both ranks (run 17, target step 1000) [11:13:44] watching for blumodelcp_run17_step_1000.ckpt on both ranks (run 17, target step 1000) [13:08:45] step 1000 present on 2 ranks -- stopping trainer NOW to halt pruning [13:08:57] trainer stopped at step 1001 [13:09:07] preserved as l3long1000_blumodelcp_run17_step_1000.ckpt (2 ranks, 16G) [13:09:07] eval models: armA_c l1 l2 l3 l3long (only l3long's cells are new) [13:09:07] PROMPT_LEN=4096 [13:10:40] -> task,armA_c,l1,l2,l3,l3long / niah_single,73.0,75.0,80.0,74.0,65.0 [13:10:40] PROMPT_LEN=8192 [13:12:56] -> task,armA_c,l1,l2,l3,l3long / niah_single,53.0,48.0,62.0,64.0,55.0 [13:12:56] PROMPT_LEN=16360 [13:17:43] -> task,armA_c,l1,l2,l3,l3long / niah_single,33.0,30.0,35.0,35.0,36.0 [13:17:43] === EVAL@1000 DONE === [13:17:43] resuming run 17 to step 4000 [13:19:13] resumed: step 1004 logged, 3 trainer proc(s) [13:19:13] LR after resume: lr 1.8000e-04 (must still be ~3.0e-4, NOT ramping)