File size: 1,323 Bytes
68c9404 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | [07:53:36] arm B still training -- waiting for it to finish (not killing it: its
[07:53:36] step-200 shard on rank_0 was lost to a run-number collision, so a
[07:53:36] resume is impossible and a kill would cost 41 min, not 12)
[08:01:42] GPUs free
[08:01:42] registered base_c = blumodelcp_run6_step_750.ckpt (run7', corrected cache)
[08:01:42] registered armB_c = blumodelcp_run7_step_400.ckpt (all 4 shards present)
[08:01:42] models: base_c armB_c
[08:01:42] eval PROMPT_LEN=1024 N=100 tasks: niah_single val_backward
[08:05:15] -> 3 rows in newck_len_1024/summary.csv
[08:05:15] eval PROMPT_LEN=2048 N=100 tasks: niah_single val_backward
[08:09:11] -> 3 rows in newck_len_2048/summary.csv
[08:09:11] eval PROMPT_LEN=4096 N=100 tasks: niah_single val_backward
[08:14:19] -> 3 rows in newck_len_4096/summary.csv
[08:14:19] eval PROMPT_LEN=8192 N=100 tasks: niah_single val_backward
[08:22:44] -> 3 rows in newck_len_8192/summary.csv
[08:22:44] eval PROMPT_LEN=16360 N=100 tasks: niah_single val_backward
[08:35:06] -> 3 rows in newck_len_16360/summary.csv
[08:35:06] EVAL DONE
[08:35:06] newck_len_1024/summary.csv OK
[08:35:06] newck_len_2048/summary.csv OK
[08:35:06] newck_len_4096/summary.csv OK
[08:35:06] newck_len_8192/summary.csv OK
[08:35:06] newck_len_16360/summary.csv OK
|