[06:41:49] === length-bucket experiment ===
[06:41:49] free disk 113G
[06:41:49] l1: 4 train shards, 1 val shards in lendata/b1_4096_8192
[06:41:49] l1: attempting CP_B=2 (grad_accum=16, global 524288)
[07:29:38] l1 -> run 1, blumodelcp_run1_step_400.ckpt, B=2, final loss loss: 2.458802
[07:29:38] l1  peak mem mem: 68769MB
[07:29:38] l2: 4 train shards, 1 val shards in lendata/b2_8192_16384
[07:29:38] l2: attempting CP_B=2 (grad_accum=16, global 524288)
[08:17:23] l2 -> run 2, blumodelcp_run2_step_400.ckpt, B=2, final loss loss: 2.794149
[08:17:23] l2  peak mem mem: 68769MB
[08:17:23] eval models: armA_c l1 l2 
[08:17:23] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin (195370858 B)
[08:17:23] eval PROMPT_LEN=1024 N=100
[08:20:56] eval PROMPT_LEN=2048 N=100
[08:24:24] eval PROMPT_LEN=4096 N=100
[08:27:53] eval PROMPT_LEN=8192 N=100
[08:31:22] eval PROMPT_LEN=16360 N=100
[08:34:50] eval done
[08:34:50] wrote lenbucket_logs/RESULTS.md
[08:34:50] === LENBUCKET DONE ===
[08:56:22] === length-bucket experiment ===
[08:56:22] free disk 70G
[08:56:22] l1 already trained -> checkpoints_bluscriptcp/rank_0/blumodelcp_run1_step_400.ckpt
[08:56:22] l2 already trained -> checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_400.ckpt
[08:56:22] eval models: armA_c l1 l2 
[08:56:22] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b3val_haystack.bin (195370858 B)
[08:56:22] eval PROMPT_LEN=1024 N=100
[09:01:59]   -> niah_single,94.0,95.0,96.0
[09:01:59] eval PROMPT_LEN=2048 N=100
[09:08:18]   -> niah_single,94.0,96.0,94.0
[09:08:18] eval PROMPT_LEN=4096 N=100
[09:16:08]   -> niah_single,84.0,88.0,92.0
[09:16:08] eval PROMPT_LEN=8192 N=100
[09:27:23]   -> niah_single,62.0,67.0,78.0
[09:27:23] eval PROMPT_LEN=16360 N=100
[09:45:19]   -> niah_single,33.0,39.0,48.0
[09:45:19] eval done
[09:45:19] wrote lenbucket_logs/RESULTS.md
[09:45:19] === LENBUCKET DONE ===