| [06:41:49] === length-bucket experiment === |
| [06:41:49] free disk 113G |
| [06:41:49] l1: 4 train shards, 1 val shards in lendata/b1_4096_8192 |
| [06:41:49] l1: attempting CP_B=2 (grad_accum=16, global 524288) |
| [07:29:38] l1 -> run 1, blumodelcp_run1_step_400.ckpt, B=2, final loss loss: 2.458802 |
| [07:29:38] l1 peak mem mem: 68769MB |
| [07:29:38] l2: 4 train shards, 1 val shards in lendata/b2_8192_16384 |
| [07:29:38] l2: attempting CP_B=2 (grad_accum=16, global 524288) |
| [08:17:23] l2 -> run 2, blumodelcp_run2_step_400.ckpt, B=2, final loss loss: 2.794149 |
| [08:17:23] l2 peak mem mem: 68769MB |
| [08:17:23] eval models: armA_c l1 l2 |
| [08:17:23] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin (195370858 B) |
| [08:17:23] eval PROMPT_LEN=1024 N=100 |
| [08:20:56] eval PROMPT_LEN=2048 N=100 |
| [08:24:24] eval PROMPT_LEN=4096 N=100 |
| [08:27:53] eval PROMPT_LEN=8192 N=100 |
| [08:31:22] eval PROMPT_LEN=16360 N=100 |
| [08:34:50] eval done |
| [08:34:50] wrote lenbucket_logs/RESULTS.md |
| [08:34:50] === LENBUCKET DONE === |
| [08:56:22] === length-bucket experiment === |
| [08:56:22] free disk 70G |
| [08:56:22] l1 already trained -> checkpoints_bluscriptcp/rank_0/blumodelcp_run1_step_400.ckpt |
| [08:56:22] l2 already trained -> checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_400.ckpt |
| [08:56:22] eval models: armA_c l1 l2 |
| [08:56:22] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin (195370858 B) |
| [08:56:22] eval PROMPT_LEN=1024 N=100 |
| [09:01:59] -> niah_single,94.0,95.0,96.0
|
| [09:01:59] eval PROMPT_LEN=2048 N=100 |
| [09:08:18] -> niah_single,94.0,96.0,94.0
|
| [09:08:18] eval PROMPT_LEN=4096 N=100 |
| [09:16:08] -> niah_single,84.0,88.0,92.0
|
| [09:16:08] eval PROMPT_LEN=8192 N=100 |
| [09:27:23] -> niah_single,62.0,67.0,78.0
|
| [09:27:23] eval PROMPT_LEN=16360 N=100 |
| [09:45:19] -> niah_single,33.0,39.0,48.0
|
| [09:45:19] eval done |
| [09:45:19] wrote lenbucket_logs/RESULTS.md |
| [09:45:19] === LENBUCKET DONE === |
|
|
