ai-2 archive: reader9 before the box is cleared

ai2-archive/reader9/TRAIN.log

@@ -0,0 +1,100 @@
+[23:37:28 +  1.5m] student streamed: 4.9 GB
+[23:37:31 +  1.6m] LAYER_SPLIT [15, 30]: cuda:0: layers 0-14 | cuda:1: layers 15-29 | cuda:2: layers 30-39
+[23:37:31 +  1.6m] LoRA r=8 on 84 attention/GDN projections + routed experts of layers [0, 1, 2]..25
+[23:37:33 +  1.6m] reply-ending tokens: [248044, 248046] (hash reset on 248044)
+[23:37:33 +  1.6m] PLE table: 8 heads x 976,640 rows x 256 = 2.00B params, injected before layer 2
+[23:37:39 +  1.7m] resumed expert LoRA
+[23:37:39 +  1.7m] resumed linear LoRA on 78 projections
+[23:37:39 +  1.7m] resumed HC/PLE/gates/routers from /home/support/whittle-train/reader7/keep_final.pt
+[23:37:40 +  1.7m] lm_head + embed_tokens on cuda:2 (LMHEAD_DEV)
+[23:37:40 +  1.7m] FREEZE_HC: HC, head mixer and shared gates frozen; PLE projections + LoRA train
+[23:37:40 +  1.7m] LoRA/AdamW: 215.5M adapter params at lr 1.0e-04
+[23:37:40 +  1.7m] trainable: Muon 21.0M | AdamW 0.06M | table 2.00B (sparse SGD)
+[23:37:40 +  1.7m] corpus source: /home/support/whittle-train/corpus_code_all.npy (264,425,588 tokens)
+[23:37:42 +  1.8m] held-out extra: /home/support/whittle-train/slice_cs/held_code.npy (66,156 tokens, 16 windows)
+[23:37:42 +  1.8m] held-out extra2: /home/support/whittle-train/slice_cs/held_sci.npy (27,167 tokens, 16 windows)
+[23:37:45 +  1.8m] teacher cache: 2,438 complete replies from /home/support/whittle-train/tcache_mix (K=128); TCACHE_FRAC=0.9 of chat rows
+[23:45:03 +  9.1m] step 0 (HC identity, PLE no-op == the checkpoint): held-out CE SFT 1.5335 | OOD 2.2147
+[23:50:48 + 14.9m]   PLE gain @ 0: SFT +0.0062 | OOD -0.0012  (CE with memory OFF minus ON on 8+8 held rows; positive = the table helps)
+[transformers] Calling `crop` with a positive value is deprecated and will be removed in version 5.18. Please use a negative integer to remove that number of tokens from the cache instead.
+[23:51:11 + 15.3m]   prefix-KD row: prefix 896 tok in 1 chunks, suffix 512 tok, cache layers 40
+/home/support/whittle-train/train_next36.py:936: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
+Consider using tensor.detach() first. (Triggered internally at /pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:836.)
+  if step <= 6 or step % 200 == 0 or not math.isfinite(float(gn)): log(f"  prefix step {step}: loss {float(loss):.4f} kd {kd_v/max(n,1):.4f} grad-norm {float(gn):.3e}")
+[23:51:25 + 15.5m]   prefix step 1: loss 1.3075 kd 0.4780 grad-norm 1.904e+00
+[23:51:25 + 15.5m] step1: grad-norm 1.904e+00 | HC probe max|delta| 0.000e+00 | table rows touched 0
+[23:51:25 + 15.5m] peak GPU cuda:0 9.2 GB | cuda:1 8.5 GB | cuda:2 8.7 GB
+[23:54:47 + 18.9m] OOM at step 9 (T=829, stage fwd): Tried to allocate 648.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 508.81 MiB is free., skipped
+[23:55:02 + 19.1m] OOM at step 10 (T=581, stage fwd): Tried to allocate 596.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 528.81 MiB is free., skipped
+[23:55:17 + 19.3m] OOM at step 11 (T=666, stage fwd): Tried to allocate 560.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 528.81 MiB is free., skipped
+[23:55:41 + 19.7m] OOM at step 13 (T=512, stage fwd): Tried to allocate 528.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 530.81 MiB is free., skipped
+[23:55:43 + 19.8m] OOM at step 14 (T=512, stage fwd): Tried to allocate 616.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 530.81 MiB is free., skipped
+[23:55:45 + 19.8m] OOM at step 15 (T=1799, stage fwd): Tried to allocate 584.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 504.81 MiB is free., skipped
+[23:55:47 + 19.8m] OOM at step 16 (T=512, stage fwd): Tried to allocate 632.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 530.81 MiB is free., skipped
+[23:55:52 + 19.9m] OOM at step 17 (T=317, stage fwd): Tried to allocate 620.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 590.81 MiB is free., skipped
+[23:55:54 + 20.0m] OOM at step 18 (T=512, stage fwd): Tried to allocate 588.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 550.81 MiB is free., skipped
+[23:55:59 + 20.0m] OOM at step 19 (T=994, stage fwd): Tried to allocate 612.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 528.81 MiB is free., skipped
+[23:56:01 + 20.1m] OOM at step 20 (T=337, stage fwd): Tried to allocate 596.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 590.81 MiB is free., skipped
+[23:56:24 + 20.5m] OOM at step 22 (T=512, stage fwd): Tried to allocate 584.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 470.81 MiB is free., skipped
+[23:56:26 + 20.5m] OOM at step 23 (T=512, stage fwd): Tried to allocate 544.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 490.81 MiB is free., skipped
+[23:56:30 + 20.6m] OOM at step 24 (T=512, stage fwd): Tried to allocate 548.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 530.81 MiB is free., skipped
+[23:56:43 + 20.8m] OOM at step 25 (T=635, stage fwd): Tried to allocate 636.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 510.81 MiB is free., skipped
+[23:56:47 + 20.9m] OOM at step 26 (T=315, stage fwd): Tried to allocate 624.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 590.81 MiB is free., skipped
+[23:56:50 + 20.9m] OOM at step 27 (T=1119, stage fwd): Tried to allocate 612.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 528.81 MiB is free., skipped
+[23:56:53 + 21.0m] OOM at step 28 (T=376, stage fwd): Tried to allocate 584.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 570.81 MiB is free., skipped
+[23:56:55 + 21.0m] OOM at step 29 (T=512, stage fwd): Tried to allocate 600.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 550.81 MiB is free., skipped
+[23:56:57 + 21.0m] OOM at step 30 (T=512, stage fwd): Tried to allocate 528.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 488.81 MiB is free., skipped
+[23:57:02 + 21.1m] OOM at step 31 (T=361, stage fwd): Tried to allocate 608.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 570.81 MiB is free., skipped
+[23:57:04 + 21.1m] OOM at step 32 (T=512, stage fwd): Tried to allocate 516.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 508.81 MiB is free., skipped
+[23:57:07 + 21.2m] OOM at step 33 (T=512, stage fwd): Tried to allocate 548.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 530.81 MiB is free., skipped
+[23:57:19 + 21.4m] OOM at step 34 (T=562, stage fwd): Tried to allocate 640.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 550.81 MiB is free., skipped
+[23:57:21 + 21.4m] OOM at step 35 (T=512, stage fwd): Tried to allocate 576.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 570.81 MiB is free., skipped
+[23:57:22 + 21.4m] OOM at step 36 (T=512, stage fwd): Tried to allocate 572.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 550.81 MiB is free., skipped
+[23:57:47 + 21.9m] OOM at step 37 (T=582, stage fwd): Tried to allocate 500.00 MiB. GPU 2 has a total capacity of 11.63 GiB of which 482.81 MiB is free., skipped
+[23:57:49 + 21.9m] OOM at step 38 (T=512, stage fwd): Tried to allocate 572.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[23:57:54 + 22.0m] OOM at step 39 (T=512, stage fwd): Tried to allocate 548.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 486.81 MiB is free., skipped
+[23:57:55 + 22.0m] OOM at step 40 (T=512, stage fwd): Tried to allocate 568.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[23:57:57 + 22.0m] OOM at step 41 (T=512, stage fwd): Tried to allocate 576.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 566.81 MiB is free., skipped
+[23:58:18 + 22.4m] OOM at step 42 (T=172, stage fwd): Tried to allocate 456.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 412.81 MiB is free., skipped
+[23:58:31 + 22.6m] OOM at step 43 (T=512, stage fwd): Tried to allocate 508.00 MiB. GPU 2 has a total capacity of 11.63 GiB of which 436.81 MiB is free., skipped
+[23:58:35 + 22.7m] OOM at step 44 (T=512, stage fwd): Tried to allocate 564.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 526.81 MiB is free., skipped
+[23:58:37 + 22.7m] OOM at step 45 (T=512, stage fwd): Tried to allocate 596.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 466.81 MiB is free., skipped
+[23:58:40 + 22.7m] OOM at step 46 (T=512, stage fwd): Tried to allocate 588.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[23:58:42 + 22.8m] OOM at step 47 (T=512, stage fwd): Tried to allocate 560.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 466.81 MiB is free., skipped
+[23:58:43 + 22.8m] OOM at step 48 (T=512, stage fwd): Tried to allocate 572.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[23:58:48 + 22.9m] OOM at step 49 (T=512, stage fwd): Tried to allocate 556.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[23:59:06 + 23.2m] OOM at step 50 (T=267, stage fwd): Tried to allocate 532.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 532.81 MiB is free., skipped
+[23:59:21 + 23.4m] OOM at step 51 (T=783, stage fwd): Tried to allocate 580.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 538.81 MiB is free., skipped
+[23:59:23 + 23.4m] OOM at step 52 (T=390, stage fwd): Tried to allocate 592.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[23:59:26 + 23.5m] OOM at step 53 (T=512, stage fwd): Tried to allocate 576.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 446.81 MiB is free., skipped
+[23:59:28 + 23.5m] OOM at step 54 (T=512, stage fwd): Tried to allocate 580.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[23:59:33 + 23.6m] OOM at step 55 (T=465, stage fwd): Tried to allocate 576.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 566.81 MiB is free., skipped
+[23:59:36 + 23.7m] OOM at step 56 (T=512, stage fwd): Tried to allocate 536.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 526.81 MiB is free., skipped
+[23:59:37 + 23.7m] OOM at step 57 (T=446, stage fwd): Tried to allocate 580.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[23:59:39 + 23.7m] OOM at step 58 (T=512, stage fwd): Tried to allocate 560.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[00:00:00 + 24.1m] OOM at step 59 (T=173, stage fwd): Tried to allocate 488.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 410.81 MiB is free., skipped
+[00:00:02 + 24.1m] OOM at step 60 (T=512, stage fwd): Tried to allocate 580.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[00:00:04 + 24.1m] OOM at step 61 (T=512, stage fwd): Tried to allocate 620.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 466.81 MiB is free., skipped
+[00:00:06 + 24.2m] OOM at step 62 (T=512, stage fwd): Tried to allocate 592.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 566.81 MiB is free., skipped
+[00:00:08 + 24.2m] OOM at step 63 (T=512, stage fwd): Tried to allocate 528.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 486.81 MiB is free., skipped
+[00:00:22 + 24.4m] OOM at step 64 (T=518, stage fwd): Tried to allocate 600.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 502.81 MiB is free., skipped
+[00:00:26 + 24.5m] OOM at step 65 (T=512, stage fwd): Tried to allocate 528.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[00:00:30 + 24.6m] OOM at step 66 (T=449, stage fwd): Tried to allocate 572.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[00:00:32 + 24.6m] OOM at step 67 (T=512, stage fwd): Tried to allocate 592.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[00:00:54 + 25.0m] OOM at step 68 (T=573, stage fwd): Tried to allocate 484.00 MiB. GPU 2 has a total capacity of 11.63 GiB of which 338.81 MiB is free., skipped
+[00:00:56 + 25.0m] OOM at step 69 (T=379, stage fwd): Tried to allocate 596.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 566.81 MiB is free., skipped
+[00:00:59 + 25.0m] OOM at step 70 (T=512, stage fwd): Tried to allocate 600.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 586.81 MiB is free., skipped
+[00:01:00 + 25.1m] OOM at step 71 (T=512, stage fwd): Tried to allocate 540.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[00:01:02 + 25.1m] OOM at step 72 (T=1403, stage fwd): Tried to allocate 604.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 582.81 MiB is free., skipped
+[00:01:22 + 25.4m] OOM at step 73 (T=244, stage fwd): Tried to allocate 488.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 454.81 MiB is free., skipped
+[00:01:25 + 25.5m] OOM at step 74 (T=496, stage fwd): Tried to allocate 528.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[00:01:27 + 25.5m] OOM at step 75 (T=512, stage fwd): Tried to allocate 596.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 566.81 MiB is free., skipped
+[00:01:30 + 25.6m] OOM at step 76 (T=512, stage fwd): Tried to allocate 556.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[00:01:32 + 25.6m] OOM at step 77 (T=512, stage fwd): Tried to allocate 568.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 566.81 MiB is free., skipped
+[00:01:34 + 25.6m] OOM at step 78 (T=512, stage fwd): Tried to allocate 588.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[00:01:36 + 25.7m] OOM at step 79 (T=377, stage fwd): Tried to allocate 556.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 546.81 MiB is free., skipped
+[00:01:38 + 25.7m] OOM at step 80 (T=512, stage fwd): Tried to allocate 540.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 506.81 MiB is free., skipped
+[00:01:39 + 25.7m] OOM at step 81 (T=512, stage fwd): Tried to allocate 604.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 526.81 MiB is free., skipped
+[00:01:41 + 25.8m] OOM at step 82 (T=512, stage fwd): Tried to allocate 556.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 526.81 MiB is free., skipped
+[00:01:43 + 25.8m] OOM at step 83 (T=512, stage fwd): Tried to allocate 544.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 486.81 MiB is free., skipped
+[00:01:55 + 26.0m] OOM at step 84 (T=599, stage fwd): Tried to allocate 540.00 MiB. GPU 1 has a total capacity of 11.63 GiB of which 522.81 MiB is free., skipped

ai2-archive/reader9/ngram_table.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:beb55b5ca14166b3f0d941ccb3383d579ac44bf148cec1cdf32e86982466a1a6
+size 4000317568

ai2-archive/reader9/ple_hash.json

@@ -0,0 +1,12 @@
+{
+ "multipliers": [
+  32473892291597,
+  22010589867355,
+  6403636407469
+ ],
+ "rows_per_head": 976640,
+ "heads": 8,
+ "ngram": 3,
+ "eos": 248044,
+ "head_dim": 256
+}

ai2-archive/reader9/train.pid

@@ -0,0 +1 @@
+1815029