Upload folder using huggingface_hub

a2d6f5f verified 2 months ago

975 Bytes

[15:38:16] watching for blumodelcp_run17_step_2000.ckpt on both ranks (run 17, target step 2000)
[16:40:47] step 2000 present on 2 ranks -- stopping trainer NOW to halt pruning
[16:40:59] trainer stopped at step 1662
[16:41:10] preserved as l3long2000_blumodelcp_run17_step_2000.ckpt (2 ranks, 16G)
[16:41:10] eval models: armA_c l1 l2 l3 l3long l3long2k  (only l3long's cells are new)
[16:41:10] PROMPT_LEN=4096
[16:42:35]   -> task,armA_c,l1,l2,l3,l3long,l3long2k
 / niah_single,73.0,75.0,80.0,74.0,65.0,68.0
[16:42:35] PROMPT_LEN=8192
[16:44:43]   -> task,armA_c,l1,l2,l3,l3long,l3long2k
 / niah_single,53.0,48.0,62.0,64.0,55.0,54.0
[16:44:43] PROMPT_LEN=16360
[16:49:01]   -> task,armA_c,l1,l2,l3,l3long,l3long2k
 / niah_single,33.0,30.0,35.0,35.0,36.0,43.0
[16:49:01] === EVAL@1000 DONE ===
[16:49:01] resuming run 17 to step 4000
[16:50:31] resumed: step 1665 logged, 3 trainer proc(s)
[16:50:31] LR after resume: lr 1.8000e-04  (must still be ~3.0e-4, NOT ramping)

	[15:38:16] watching for blumodelcp_run17_step_2000.ckpt on both ranks (run 17, target step 2000)
	[16:40:47] step 2000 present on 2 ranks -- stopping trainer NOW to halt pruning
	[16:40:59] trainer stopped at step 1662
	[16:41:10] preserved as l3long2000_blumodelcp_run17_step_2000.ckpt (2 ranks, 16G)
	[16:41:10] eval models: armA_c l1 l2 l3 l3long l3long2k (only l3long's cells are new)
	[16:41:10] PROMPT_LEN=4096
	[16:42:35] -> task,armA_c,l1,l2,l3,l3long,l3long2k / niah_single,73.0,75.0,80.0,74.0,65.0,68.0
	[16:42:35] PROMPT_LEN=8192
	[16:44:43] -> task,armA_c,l1,l2,l3,l3long,l3long2k / niah_single,53.0,48.0,62.0,64.0,55.0,54.0
	[16:44:43] PROMPT_LEN=16360
	[16:49:01] -> task,armA_c,l1,l2,l3,l3long,l3long2k / niah_single,33.0,30.0,35.0,35.0,36.0,43.0
	[16:49:01] === EVAL@1000 DONE ===
	[16:49:01] resuming run 17 to step 4000
	[16:50:31] resumed: step 1665 logged, 3 trainer proc(s)
	[16:50:31] LR after resume: lr 1.8000e-04 (must still be ~3.0e-4, NOT ramping)