Upload folder using huggingface_hub

d8065aa verified 2 months ago

1.02 kB

[11:12:40] watching for blumodelcp_run17_step_1000.ckpt on both ranks (run 17, target step 1000)
[11:13:44] watching for blumodelcp_run17_step_1000.ckpt on both ranks (run 17, target step 1000)
[13:08:45] step 1000 present on 2 ranks -- stopping trainer NOW to halt pruning
[13:08:57] trainer stopped at step 1001
[13:09:07] preserved as l3long1000_blumodelcp_run17_step_1000.ckpt (2 ranks, 16G)
[13:09:07] eval models: armA_c l1 l2 l3 l3long  (only l3long's cells are new)
[13:09:07] PROMPT_LEN=4096
[13:10:40]   -> task,armA_c,l1,l2,l3,l3long
 / niah_single,73.0,75.0,80.0,74.0,65.0
[13:10:40] PROMPT_LEN=8192
[13:12:56]   -> task,armA_c,l1,l2,l3,l3long
 / niah_single,53.0,48.0,62.0,64.0,55.0
[13:12:56] PROMPT_LEN=16360
[13:17:43]   -> task,armA_c,l1,l2,l3,l3long
 / niah_single,33.0,30.0,35.0,35.0,36.0
[13:17:43] === EVAL@1000 DONE ===
[13:17:43] resuming run 17 to step 4000
[13:19:13] resumed: step 1004 logged, 3 trainer proc(s)
[13:19:13] LR after resume: lr 1.8000e-04  (must still be ~3.0e-4, NOT ramping)

	[11:12:40] watching for blumodelcp_run17_step_1000.ckpt on both ranks (run 17, target step 1000)
	[11:13:44] watching for blumodelcp_run17_step_1000.ckpt on both ranks (run 17, target step 1000)
	[13:08:45] step 1000 present on 2 ranks -- stopping trainer NOW to halt pruning
	[13:08:57] trainer stopped at step 1001
	[13:09:07] preserved as l3long1000_blumodelcp_run17_step_1000.ckpt (2 ranks, 16G)
	[13:09:07] eval models: armA_c l1 l2 l3 l3long (only l3long's cells are new)
	[13:09:07] PROMPT_LEN=4096
	[13:10:40] -> task,armA_c,l1,l2,l3,l3long / niah_single,73.0,75.0,80.0,74.0,65.0
	[13:10:40] PROMPT_LEN=8192
	[13:12:56] -> task,armA_c,l1,l2,l3,l3long / niah_single,53.0,48.0,62.0,64.0,55.0
	[13:12:56] PROMPT_LEN=16360
	[13:17:43] -> task,armA_c,l1,l2,l3,l3long / niah_single,33.0,30.0,35.0,35.0,36.0
	[13:17:43] === EVAL@1000 DONE ===
	[13:17:43] resuming run 17 to step 4000
	[13:19:13] resumed: step 1004 logged, 3 trainer proc(s)
	[13:19:13] LR after resume: lr 1.8000e-04 (must still be ~3.0e-4, NOT ramping)