BluScriptCP / logs /eval_run7c /eval_master.log

Upload folder using huggingface_hub

68c9404 verified 2 months ago

1.32 kB

[07:53:36] arm B still training -- waiting for it to finish (not killing it: its
[07:53:36]   step-200 shard on rank_0 was lost to a run-number collision, so a
[07:53:36]   resume is impossible and a kill would cost 41 min, not 12)
[08:01:42] GPUs free
[08:01:42] registered base_c = blumodelcp_run6_step_750.ckpt  (run7', corrected cache)
[08:01:42] registered armB_c = blumodelcp_run7_step_400.ckpt  (all 4 shards present)
[08:01:42] models: base_c armB_c 
[08:01:42] eval PROMPT_LEN=1024  N=100  tasks: niah_single val_backward
[08:05:15]   -> 3 rows in newck_len_1024/summary.csv
[08:05:15] eval PROMPT_LEN=2048  N=100  tasks: niah_single val_backward
[08:09:11]   -> 3 rows in newck_len_2048/summary.csv
[08:09:11] eval PROMPT_LEN=4096  N=100  tasks: niah_single val_backward
[08:14:19]   -> 3 rows in newck_len_4096/summary.csv
[08:14:19] eval PROMPT_LEN=8192  N=100  tasks: niah_single val_backward
[08:22:44]   -> 3 rows in newck_len_8192/summary.csv
[08:22:44] eval PROMPT_LEN=16360  N=100  tasks: niah_single val_backward
[08:35:06]   -> 3 rows in newck_len_16360/summary.csv
[08:35:06] EVAL DONE
[08:35:06]   newck_len_1024/summary.csv OK
[08:35:06]   newck_len_2048/summary.csv OK
[08:35:06]   newck_len_4096/summary.csv OK
[08:35:06]   newck_len_8192/summary.csv OK
[08:35:06]   newck_len_16360/summary.csv OK

	[07:53:36] arm B still training -- waiting for it to finish (not killing it: its
	[07:53:36] step-200 shard on rank_0 was lost to a run-number collision, so a
	[07:53:36] resume is impossible and a kill would cost 41 min, not 12)
	[08:01:42] GPUs free
	[08:01:42] registered base_c = blumodelcp_run6_step_750.ckpt (run7', corrected cache)
	[08:01:42] registered armB_c = blumodelcp_run7_step_400.ckpt (all 4 shards present)
	[08:01:42] models: base_c armB_c
	[08:01:42] eval PROMPT_LEN=1024 N=100 tasks: niah_single val_backward
	[08:05:15] -> 3 rows in newck_len_1024/summary.csv
	[08:05:15] eval PROMPT_LEN=2048 N=100 tasks: niah_single val_backward
	[08:09:11] -> 3 rows in newck_len_2048/summary.csv
	[08:09:11] eval PROMPT_LEN=4096 N=100 tasks: niah_single val_backward
	[08:14:19] -> 3 rows in newck_len_4096/summary.csv
	[08:14:19] eval PROMPT_LEN=8192 N=100 tasks: niah_single val_backward
	[08:22:44] -> 3 rows in newck_len_8192/summary.csv
	[08:22:44] eval PROMPT_LEN=16360 N=100 tasks: niah_single val_backward
	[08:35:06] -> 3 rows in newck_len_16360/summary.csv
	[08:35:06] EVAL DONE
	[08:35:06] newck_len_1024/summary.csv OK
	[08:35:06] newck_len_2048/summary.csv OK
	[08:35:06] newck_len_4096/summary.csv OK
	[08:35:06] newck_len_8192/summary.csv OK
	[08:35:06] newck_len_16360/summary.csv OK