| [13:18:27] === 16k -> 32k YaRN extension on b4 === |
| [13:18:27] 6 train shards = .586 B tokens; need .393 B for 750 steps |
| [13:18:27] YaRN: SCALE=2 ORIG_MAXPOS=16384 -> 32768 (must equal CP_T=32768) |
| [13:18:27] PHASE 1: 3-step memory probe (projected 130.8 GB of 143 GB = 91 |
| [13:19:33] PROBE OK: peak mem: 108619MB |
| [13:19:33] PHASE 2: 750 steps, stable=550, ckpt every 50 keep 1 |
| [16:26:10] ext32k -> run 2, blumodelcp_run2_step_750.ckpt, loss loss: 2.570224 |
| [16:26:10] peak mem mem: 108811MB |
| [16:26:10] PHASE 3: eval models ext32k l2 |
| [16:26:10] eval PROMPT_LEN=4096 |
| [16:28:26] -> task,ext32k,l2
/ niah_single,75.0,76.0
|
| [16:28:26] eval PROMPT_LEN=8192 |
| [16:32:02] -> task,ext32k,l2
/ niah_single,60.0,36.0
|
| [16:32:02] eval PROMPT_LEN=16360 |
| [16:37:39] -> task,ext32k,l2
/ niah_single,45.0,20.0
|
| [16:37:39] eval PROMPT_LEN=32744 |
| [16:48:53] -> task,ext32k,l2
/ niah_single,27.0,11.0
|
| [16:48:53] === EXT32K DONE === |
|
|
