File size: 924 Bytes
cbacfbc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | [13:18:27] === 16k -> 32k YaRN extension on b4 ===
[13:18:27] 6 train shards = .586 B tokens; need .393 B for 750 steps
[13:18:27] YaRN: SCALE=2 ORIG_MAXPOS=16384 -> 32768 (must equal CP_T=32768)
[13:18:27] PHASE 1: 3-step memory probe (projected 130.8 GB of 143 GB = 91%)
[13:19:33] PROBE OK: peak mem: 108619MB
[13:19:33] PHASE 2: 750 steps, stable=550, ckpt every 50 keep 1
[16:26:10] ext32k -> run 2, blumodelcp_run2_step_750.ckpt, loss loss: 2.570224
[16:26:10] peak mem mem: 108811MB
[16:26:10] PHASE 3: eval models ext32k l2
[16:26:10] eval PROMPT_LEN=4096
[16:28:26] -> task,ext32k,l2
/ niah_single,75.0,76.0
[16:28:26] eval PROMPT_LEN=8192
[16:32:02] -> task,ext32k,l2
/ niah_single,60.0,36.0
[16:32:02] eval PROMPT_LEN=16360
[16:37:39] -> task,ext32k,l2
/ niah_single,45.0,20.0
[16:37:39] eval PROMPT_LEN=32744
[16:48:53] -> task,ext32k,l2
/ niah_single,27.0,11.0
[16:48:53] === EXT32K DONE ===
|