File size: 975 Bytes
a2d6f5f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | [15:38:16] watching for blumodelcp_run17_step_2000.ckpt on both ranks (run 17, target step 2000)
[16:40:47] step 2000 present on 2 ranks -- stopping trainer NOW to halt pruning
[16:40:59] trainer stopped at step 1662
[16:41:10] preserved as l3long2000_blumodelcp_run17_step_2000.ckpt (2 ranks, 16G)
[16:41:10] eval models: armA_c l1 l2 l3 l3long l3long2k (only l3long's cells are new)
[16:41:10] PROMPT_LEN=4096
[16:42:35] -> task,armA_c,l1,l2,l3,l3long,l3long2k
/ niah_single,73.0,75.0,80.0,74.0,65.0,68.0
[16:42:35] PROMPT_LEN=8192
[16:44:43] -> task,armA_c,l1,l2,l3,l3long,l3long2k
/ niah_single,53.0,48.0,62.0,64.0,55.0,54.0
[16:44:43] PROMPT_LEN=16360
[16:49:01] -> task,armA_c,l1,l2,l3,l3long,l3long2k
/ niah_single,33.0,30.0,35.0,35.0,36.0,43.0
[16:49:01] === EVAL@1000 DONE ===
[16:49:01] resuming run 17 to step 4000
[16:50:31] resumed: step 1665 logged, 3 trainer proc(s)
[16:50:31] LR after resume: lr 1.8000e-04 (must still be ~3.0e-4, NOT ramping)
|