ai-2 archive: reader7 before the box is cleared

ai2-archive/reader7/TRAIN.log

@@ -0,0 +1,313 @@
+[23:32:04 +  1.3m] student streamed: 4.9 GB
+[23:32:04 +  1.3m] LoRA r=8 on 78 attention/GDN projections + routed experts of layers [2, 3, 4]..25
+[23:32:05 +  1.3m] reply-ending tokens: [248044, 248046] (hash reset on 248044)
+[23:32:05 +  1.3m] PLE table: 8 heads x 976,640 rows x 256 = 2.00B params, injected before layer 2
+[23:32:09 +  1.3m] resumed expert LoRA
+[23:32:09 +  1.3m] resumed linear LoRA on 78 projections
+[23:32:09 +  1.3m] resumed HC/PLE/gates/routers from /home/support/whittle-train/reader6/keep_final.pt
+[23:32:11 +  1.4m] lm_head + embed_tokens on cuda:1 (LMHEAD_DEV)
+[23:32:11 +  1.4m] FREEZE_HC: HC, head mixer and shared gates frozen; PLE projections + LoRA train
+[23:32:11 +  1.4m] LoRA freeze: expert adapters frozen on layers [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]; linear adapters frozen; trainable adapter layers [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]
+[23:32:11 +  1.4m] LoRA/AdamW: 113.5M adapter params at lr 1.0e-04
+[23:32:11 +  1.4m] trainable: Muon 21.0M | AdamW 0.06M | table 2.00B (sparse SGD)
+[23:32:11 +  1.4m] corpus source: /home/support/whittle-train/corpus_code_all.npy (264,425,588 tokens)
+[23:32:13 +  1.4m] held-out extra: /home/support/whittle-train/slice_cs/held_code.npy (66,156 tokens, 16 windows)
+[23:32:13 +  1.4m] held-out extra2: /home/support/whittle-train/slice_cs/held_sci.npy (27,167 tokens, 16 windows)
+[23:32:15 +  1.4m] teacher cache: 1,518 complete replies from /home/support/whittle-train/tcache_all (K=128); TCACHE_FRAC=0.9 of chat rows
+[23:33:44 +  2.9m] step 0 (HC identity, PLE no-op == the checkpoint): held-out CE SFT 1.7533 | OOD 2.4101
+[23:34:49 +  4.0m]   PLE gain @ 0: SFT +0.0083 | OOD +0.0030  (CE with memory OFF minus ON on 8+8 held rows; positive = the table helps)
+[23:34:55 +  4.1m] step1: grad-norm 1.992e+00 | HC probe max|delta| 0.000e+00 | table rows touched 1888
+[23:34:55 +  4.1m] peak GPU 10.4 GB
+[23:36:46 +  6.0m] step 20 loss 0.8173 (kd 0.2309 ce 0.5863) | 5.87s/step | table rows seen 31,676
+[23:38:46 +  8.0m] step 40 loss 0.7233 (kd 0.1442 ce 0.5791) | 5.94s/step | table rows seen 62,148
+[23:40:42 +  9.9m] step 60 loss 0.6124 (kd 0.1211 ce 0.4913) | 5.89s/step | table rows seen 90,104
+[23:42:43 + 11.9m] step 80 loss 0.6577 (kd 0.1584 ce 0.4993) | 5.93s/step | table rows seen 121,800
+[23:44:45 + 13.9m] step 100 loss 0.8605 (kd 0.1850 ce 0.6755) | 5.96s/step | table rows seen 150,936
+[23:46:41 + 15.9m] step 120 loss 0.6397 (kd 0.1661 ce 0.4736) | 5.94s/step | table rows seen 176,640
+[23:48:41 + 17.9m] step 140 loss 0.5811 (kd 0.0649 ce 0.5161) | 5.94s/step | table rows seen 203,320
+[23:50:38 + 19.8m] step 160 loss 0.6030 (kd 0.1980 ce 0.4050) | 5.93s/step | table rows seen 225,632
+[23:52:38 + 21.8m] step 180 loss 0.6684 (kd 0.2328 ce 0.4357) | 5.94s/step | table rows seen 248,548
+[23:54:37 + 23.8m] step 200 loss 0.5646 (kd 0.1958 ce 0.3688) | 5.94s/step | table rows seen 274,188
+[23:55:57 + 25.1m]   held-out CE @ 200: SFT 1.7894 (start 1.7533) | OOD 2.4616 (start 2.4101)
+[23:55:57 + 25.1m]   table decay @ 200: visited 274,188 rows, visits 366,868 | |row| median 0.0373 p90 0.1046 | below snap thr 0.01: 25.2%
+[23:57:05 + 26.3m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 570.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.35 GiB is allocated by PyTorch, and 527.83 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[23:57:08 + 26.3m]   HC stats @ 200: gate|dev| median 0.0256 (L0 0.0246, L39 0.0262) | inject w|dev| median 0.2708 | stream spread median 0.4355 (L39 0.4830)
+[23:59:04 + 28.3m] step 220 loss 0.5716 (kd 0.1358 ce 0.4357) | 6.61s/step | table rows seen 293,284
+[00:01:02 + 30.2m] step 240 loss 0.6974 (kd 0.2142 ce 0.4832) | 6.56s/step | table rows seen 316,940
+[00:02:59 + 32.2m] step 260 loss 0.4998 (kd 0.1290 ce 0.3708) | 6.50s/step | table rows seen 336,444
+[00:04:57 + 34.1m] step 280 loss 0.4580 (kd 0.1824 ce 0.2755) | 6.46s/step | table rows seen 356,008
+[00:06:51 + 36.0m] step 300 loss 0.4392 (kd 0.1336 ce 0.3056) | 6.41s/step | table rows seen 370,296
+[00:08:50 + 38.0m] step 320 loss 0.5607 (kd 0.1915 ce 0.3692) | 6.38s/step | table rows seen 394,760
+[00:10:41 + 39.9m] step 340 loss 0.6100 (kd 0.0870 ce 0.5230) | 6.33s/step | table rows seen 414,352
+[00:12:39 + 41.8m] step 360 loss 0.5422 (kd 0.0925 ce 0.4496) | 6.31s/step | table rows seen 439,752
+[00:14:36 + 43.8m] step 380 loss 0.5957 (kd 0.1862 ce 0.4095) | 6.28s/step | table rows seen 463,136
+[00:16:39 + 45.8m] step 400 loss 0.7040 (kd 0.1903 ce 0.5136) | 6.28s/step | table rows seen 487,676
+[00:17:59 + 47.2m]   held-out CE @ 400: SFT 1.8010 (start 1.7533) | OOD 2.5117 (start 2.4101)
+[00:17:59 + 47.2m]   table decay @ 400: visited 487,676 rows, visits 731,052 | |row| median 0.0364 p90 0.0992 | below snap thr 0.01: 25.3%
+[00:19:06 + 48.3m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 574.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 540.81 MiB is free. Including non-PyTorch memory, this process has 11.03 GiB memory in use. Of the allocated memory 10.36 GiB is allocated by PyTorch, and 532.08 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[00:19:08 + 48.3m]   HC stats @ 400: gate|dev| median 0.0256 (L0 0.0246, L39 0.0263) | inject w|dev| median 0.2706 | stream spread median 0.4352 (L39 0.4828)
+[00:21:10 + 50.4m] step 420 loss 0.7128 (kd 0.1778 ce 0.5349) | 6.62s/step | table rows seen 508,536
+[00:23:07 + 52.3m] step 440 loss 0.4724 (kd 0.1734 ce 0.2990) | 6.59s/step | table rows seen 525,688
+[00:25:07 + 54.3m] step 460 loss 0.7365 (kd 0.1295 ce 0.6071) | 6.56s/step | table rows seen 546,912
+[00:27:03 + 56.2m] step 480 loss 0.4643 (kd 0.1558 ce 0.3085) | 6.53s/step | table rows seen 563,224
+[00:29:03 + 58.2m] step 500 loss 0.7099 (kd 0.1797 ce 0.5303) | 6.51s/step | table rows seen 587,164
+[00:31:03 + 60.2m] step 520 loss 0.5536 (kd 0.2010 ce 0.3526) | 6.49s/step | table rows seen 601,560
+[00:33:02 + 62.2m] step 540 loss 0.4727 (kd 0.0673 ce 0.4054) | 6.47s/step | table rows seen 619,972
+[00:34:58 + 64.2m] step 560 loss 0.5178 (kd 0.1603 ce 0.3575) | 6.44s/step | table rows seen 634,512
+[00:37:00 + 66.2m] step 580 loss 0.6476 (kd 0.1903 ce 0.4574) | 6.43s/step | table rows seen 654,752
+[00:39:01 + 68.2m] step 600 loss 0.6592 (kd 0.1987 ce 0.4606) | 6.42s/step | table rows seen 673,864
+[00:40:22 + 69.6m]   held-out CE @ 600: SFT 1.8103 (start 1.7533) | OOD 2.5606 (start 2.4101)
+[00:40:22 + 69.6m]   table decay @ 600: visited 673,864 rows, visits 1,097,024 | |row| median 0.0358 p90 0.0951 | below snap thr 0.01: 25.2%
+[00:41:28 + 70.7m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 576.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 500.81 MiB is free. Including non-PyTorch memory, this process has 11.07 GiB memory in use. Of the allocated memory 10.37 GiB is allocated by PyTorch, and 567.48 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[00:41:31 + 70.7m]   HC stats @ 600: gate|dev| median 0.0255 (L0 0.0246, L39 0.0262) | inject w|dev| median 0.2686 | stream spread median 0.4353 (L39 0.4796)
+[00:43:30 + 72.7m] step 620 loss 0.5741 (kd 0.1941 ce 0.3800) | 6.65s/step | table rows seen 690,032
+[00:45:33 + 74.7m] step 640 loss 0.5463 (kd 0.0891 ce 0.4572) | 6.63s/step | table rows seen 713,580
+[00:47:33 + 76.7m] step 660 loss 0.7170 (kd 0.1168 ce 0.6001) | 6.61s/step | table rows seen 731,808
+[00:49:35 + 78.8m] step 680 loss 0.6353 (kd 0.1939 ce 0.4414) | 6.60s/step | table rows seen 756,140
+[00:51:32 + 80.7m] step 700 loss 0.7104 (kd 0.2445 ce 0.4659) | 6.58s/step | table rows seen 773,340
+[00:53:30 + 82.7m] step 720 loss 0.5729 (kd 0.2193 ce 0.3536) | 6.56s/step | table rows seen 792,552
+[00:55:31 + 84.7m] step 740 loss 0.6594 (kd 0.1420 ce 0.5174) | 6.54s/step | table rows seen 813,312
+[00:57:29 + 86.7m] step 760 loss 0.4991 (kd 0.1197 ce 0.3794) | 6.53s/step | table rows seen 829,188
+[00:59:32 + 88.7m] step 780 loss 0.6209 (kd 0.1805 ce 0.4404) | 6.52s/step | table rows seen 848,176
+[01:01:31 + 90.7m] step 800 loss 0.7362 (kd 0.1646 ce 0.5716) | 6.50s/step | table rows seen 868,428
+[01:02:52 + 92.1m]   held-out CE @ 800: SFT 1.8988 (start 1.7533) | OOD 2.6286 (start 2.4101)
+[01:02:52 + 92.1m]   table decay @ 800: visited 868,428 rows, visits 1,468,240 | |row| median 0.0353 p90 0.0927 | below snap thr 0.01: 25.3%
+[01:03:58 + 93.2m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 586.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 460.81 MiB is free. Including non-PyTorch memory, this process has 11.11 GiB memory in use. Of the allocated memory 10.41 GiB is allocated by PyTorch, and 567.50 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[01:04:01 + 93.2m]   HC stats @ 800: gate|dev| median 0.0254 (L0 0.0246, L39 0.0262) | inject w|dev| median 0.2649 | stream spread median 0.4348 (L39 0.4768)
+[01:05:59 + 95.2m] step 820 loss 0.4140 (kd 0.1478 ce 0.2662) | 6.67s/step | table rows seen 880,628
+[01:07:54 + 97.1m] step 840 loss 0.7382 (kd 0.2617 ce 0.4765) | 6.65s/step | table rows seen 893,836
+[01:09:52 + 99.1m] step 860 loss 0.6383 (kd 0.1952 ce 0.4431) | 6.63s/step | table rows seen 911,360
+[01:11:49 +101.0m] step 880 loss 0.4455 (kd 0.1067 ce 0.3388) | 6.61s/step | table rows seen 927,184
+[01:13:49 +103.0m] step 900 loss 0.5921 (kd 0.2316 ce 0.3605) | 6.60s/step | table rows seen 942,652
+[01:15:49 +105.0m] step 920 loss 0.6040 (kd 0.1010 ce 0.5031) | 6.59s/step | table rows seen 960,020
+[01:17:48 +107.0m] step 940 loss 0.6458 (kd 0.2003 ce 0.4454) | 6.57s/step | table rows seen 974,824
+[01:19:43 +108.9m] step 960 loss 0.6295 (kd 0.1525 ce 0.4770) | 6.56s/step | table rows seen 986,808
+[01:21:43 +110.9m] step 980 loss 0.5536 (kd 0.1471 ce 0.4065) | 6.55s/step | table rows seen 999,932
+[01:23:40 +112.9m] step 1000 loss 0.6798 (kd 0.1501 ce 0.5297) | 6.53s/step | table rows seen 1,017,816
+[01:25:01 +114.2m]   held-out CE @ 1000: SFT 1.8808 (start 1.7533) | OOD 2.5993 (start 2.4101)
+[01:25:01 +114.2m]   table decay @ 1000: visited 1,017,816 rows, visits 1,831,580 | |row| median 0.0353 p90 0.0930 | below snap thr 0.01: 25.3%
+[01:26:07 +115.3m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 586.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 500.81 MiB is free. Including non-PyTorch memory, this process has 11.07 GiB memory in use. Of the allocated memory 10.40 GiB is allocated by PyTorch, and 530.21 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[01:26:10 +115.4m]   HC stats @ 1000: gate|dev| median 0.0255 (L0 0.0246, L39 0.0262) | inject w|dev| median 0.2680 | stream spread median 0.4346 (L39 0.4785)
+[01:28:08 +117.3m] step 1020 loss 0.5522 (kd 0.1748 ce 0.3774) | 6.67s/step | table rows seen 1,033,036
+[01:30:04 +119.3m] step 1040 loss 0.4276 (kd 0.1595 ce 0.2681) | 6.65s/step | table rows seen 1,046,192
+[01:32:02 +121.2m] step 1060 loss 0.5667 (kd 0.1997 ce 0.3669) | 6.64s/step | table rows seen 1,062,992
+[01:34:04 +123.3m] step 1080 loss 0.5873 (kd 0.1170 ce 0.4703) | 6.63s/step | table rows seen 1,079,652
+[01:36:02 +125.2m] step 1100 loss 0.5698 (kd 0.1460 ce 0.4238) | 6.61s/step | table rows seen 1,093,336
+[01:38:04 +127.3m] step 1120 loss 0.7454 (kd 0.2307 ce 0.5147) | 6.60s/step | table rows seen 1,106,688
+[01:40:01 +129.2m] step 1140 loss 0.6547 (kd 0.1825 ce 0.4721) | 6.59s/step | table rows seen 1,119,736
+[01:41:59 +131.2m] step 1160 loss 0.5519 (kd 0.1676 ce 0.3843) | 6.58s/step | table rows seen 1,138,960
+[01:43:56 +133.1m] step 1180 loss 0.4903 (kd 0.2014 ce 0.2889) | 6.57s/step | table rows seen 1,152,256
+[01:45:58 +135.2m] step 1200 loss 0.7020 (kd 0.1304 ce 0.5715) | 6.56s/step | table rows seen 1,170,912
+[01:47:19 +136.5m]   held-out CE @ 1200: SFT 1.8812 (start 1.7533) | OOD 2.5828 (start 2.4101)
+[01:47:19 +136.5m]   table decay @ 1200: visited 1,170,912 rows, visits 2,189,432 | |row| median 0.0349 p90 0.0902 | below snap thr 0.01: 25.3%
+[01:48:26 +137.6m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 582.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 480.81 MiB is free. Including non-PyTorch memory, this process has 11.09 GiB memory in use. Of the allocated memory 10.39 GiB is allocated by PyTorch, and 561.23 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[01:48:28 +137.7m]   HC stats @ 1200: gate|dev| median 0.0254 (L0 0.0246, L39 0.0261) | inject w|dev| median 0.2683 | stream spread median 0.4339 (L39 0.4750)
+[01:50:24 +139.6m] step 1220 loss 0.4216 (kd 0.1654 ce 0.2561) | 6.67s/step | table rows seen 1,182,328
+[01:52:23 +141.6m] step 1240 loss 0.5264 (kd 0.2018 ce 0.3246) | 6.66s/step | table rows seen 1,189,564
+[01:54:24 +143.6m] step 1260 loss 0.6620 (kd 0.1766 ce 0.4854) | 6.65s/step | table rows seen 1,204,692
+[01:56:22 +145.6m] step 1280 loss 0.4430 (kd 0.1172 ce 0.3258) | 6.64s/step | table rows seen 1,217,112
+[01:58:22 +147.6m] step 1300 loss 0.6894 (kd 0.1808 ce 0.5086) | 6.63s/step | table rows seen 1,229,832
+[02:00:16 +149.5m] step 1320 loss 0.5314 (kd 0.1032 ce 0.4281) | 6.61s/step | table rows seen 1,244,392
+[02:02:18 +151.5m] step 1340 loss 0.5778 (kd 0.1175 ce 0.4603) | 6.60s/step | table rows seen 1,259,804
+[02:04:19 +153.5m] step 1360 loss 0.6926 (kd 0.2782 ce 0.4143) | 6.60s/step | table rows seen 1,272,800
+[02:06:16 +155.5m] step 1380 loss 0.5384 (kd 0.1575 ce 0.3808) | 6.58s/step | table rows seen 1,284,608
+[02:08:13 +157.4m] step 1400 loss 0.4500 (kd 0.1317 ce 0.3183) | 6.57s/step | table rows seen 1,294,740
+[02:09:33 +158.7m]   held-out CE @ 1400: SFT 1.8905 (start 1.7533) | OOD 2.5782 (start 2.4101)
+[02:09:33 +158.7m]   table decay @ 1400: visited 1,294,740 rows, visits 2,543,908 | |row| median 0.0352 p90 0.0898 | below snap thr 0.01: 25.2%
+[02:10:40 +159.9m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 580.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 520.81 MiB is free. Including non-PyTorch memory, this process has 11.05 GiB memory in use. Of the allocated memory 10.38 GiB is allocated by PyTorch, and 532.62 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[02:10:42 +159.9m]   HC stats @ 1400: gate|dev| median 0.0255 (L0 0.0246, L39 0.0262) | inject w|dev| median 0.2674 | stream spread median 0.4345 (L39 0.4786)
+[02:12:42 +161.9m] step 1420 loss 0.5831 (kd 0.1807 ce 0.4024) | 6.67s/step | table rows seen 1,306,888
+[02:14:42 +163.9m] step 1440 loss 0.6087 (kd 0.2052 ce 0.4035) | 6.66s/step | table rows seen 1,319,984
+[02:16:39 +165.8m] step 1460 loss 0.5474 (kd 0.1793 ce 0.3681) | 6.65s/step | table rows seen 1,330,696
+[02:18:36 +167.8m] step 1480 loss 0.5017 (kd 0.0952 ce 0.4064) | 6.64s/step | table rows seen 1,341,184
+[02:20:38 +169.8m] step 1500 loss 0.6211 (kd 0.1625 ce 0.4586) | 6.63s/step | table rows seen 1,357,352
+[02:22:38 +171.8m] step 1520 loss 0.5527 (kd 0.1964 ce 0.3563) | 6.62s/step | table rows seen 1,367,104
+[02:24:37 +173.8m] step 1540 loss 0.5491 (kd 0.1068 ce 0.4423) | 6.62s/step | table rows seen 1,379,184
+[02:26:38 +175.8m] step 1560 loss 0.6363 (kd 0.2277 ce 0.4086) | 6.61s/step | table rows seen 1,390,020
+[02:28:41 +177.9m] step 1580 loss 0.6884 (kd 0.1538 ce 0.5347) | 6.60s/step | table rows seen 1,402,564
+[02:30:47 +180.0m] step 1600 loss 0.6548 (kd 0.0970 ce 0.5578) | 6.60s/step | table rows seen 1,420,160
+[02:32:07 +181.3m]   held-out CE @ 1600: SFT 1.9101 (start 1.7533) | OOD 2.6105 (start 2.4101)
+[02:32:07 +181.3m]   table decay @ 1600: visited 1,420,160 rows, visits 2,917,344 | |row| median 0.0348 p90 0.0887 | below snap thr 0.01: 25.3%
+[02:33:14 +182.4m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 570.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 540.81 MiB is free. Including non-PyTorch memory, this process has 11.03 GiB memory in use. Of the allocated memory 10.34 GiB is allocated by PyTorch, and 553.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[02:33:16 +182.5m]   HC stats @ 1600: gate|dev| median 0.0254 (L0 0.0246, L39 0.0261) | inject w|dev| median 0.2658 | stream spread median 0.4345 (L39 0.4765)
+[02:35:15 +184.4m] step 1620 loss 0.5087 (kd 0.1287 ce 0.3800) | 6.68s/step | table rows seen 1,430,192
+[02:37:12 +186.4m] step 1640 loss 0.5823 (kd 0.1191 ce 0.4632) | 6.67s/step | table rows seen 1,442,500
+[02:39:09 +188.3m] step 1660 loss 0.6742 (kd 0.2191 ce 0.4551) | 6.66s/step | table rows seen 1,451,800
+[02:41:07 +190.3m] step 1680 loss 0.6573 (kd 0.2034 ce 0.4539) | 6.65s/step | table rows seen 1,461,652
+[02:43:07 +192.3m] step 1700 loss 0.6714 (kd 0.1723 ce 0.4991) | 6.65s/step | table rows seen 1,471,060
+[02:45:08 +194.3m] step 1720 loss 0.6793 (kd 0.2334 ce 0.4459) | 6.64s/step | table rows seen 1,480,972
+[02:47:07 +196.3m] step 1740 loss 0.6492 (kd 0.0935 ce 0.5557) | 6.63s/step | table rows seen 1,494,928
+[02:49:03 +198.2m] step 1760 loss 0.7562 (kd 0.1289 ce 0.6273) | 6.62s/step | table rows seen 1,508,040
+[02:51:00 +200.2m] step 1780 loss 0.6347 (kd 0.0749 ce 0.5598) | 6.61s/step | table rows seen 1,519,420
+[02:52:58 +202.2m] step 1800 loss 0.5984 (kd 0.1291 ce 0.4693) | 6.61s/step | table rows seen 1,530,556
+[02:54:18 +203.5m]   held-out CE @ 1800: SFT 1.8747 (start 1.7533) | OOD 2.5599 (start 2.4101)
+[02:54:18 +203.5m]   table decay @ 1800: visited 1,530,556 rows, visits 3,273,372 | |row| median 0.0346 p90 0.0870 | below snap thr 0.01: 25.4%
+[02:55:25 +204.6m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 564.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.33 GiB is allocated by PyTorch, and 549.85 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[02:55:28 +204.7m]   HC stats @ 1800: gate|dev| median 0.0255 (L0 0.0246, L39 0.0261) | inject w|dev| median 0.2682 | stream spread median 0.4347 (L39 0.4763)
+[02:57:32 +206.7m] step 1820 loss 0.6834 (kd 0.1623 ce 0.5211) | 6.68s/step | table rows seen 1,540,700
+[02:59:31 +208.7m] step 1840 loss 0.7122 (kd 0.1139 ce 0.5983) | 6.68s/step | table rows seen 1,551,912
+[03:01:34 +210.8m] step 1860 loss 0.7852 (kd 0.1268 ce 0.6584) | 6.67s/step | table rows seen 1,565,164
+[03:03:34 +212.8m] step 1880 loss 0.6612 (kd 0.1837 ce 0.4775) | 6.66s/step | table rows seen 1,575,748
+[03:05:32 +214.7m] step 1900 loss 0.7345 (kd 0.1529 ce 0.5816) | 6.65s/step | table rows seen 1,591,900
+[03:07:31 +216.7m] step 1920 loss 0.5936 (kd 0.1874 ce 0.4062) | 6.65s/step | table rows seen 1,601,708
+[03:09:29 +218.7m] step 1940 loss 0.7406 (kd 0.1496 ce 0.5910) | 6.64s/step | table rows seen 1,613,856
+[03:11:25 +220.6m] step 1960 loss 0.8465 (kd 0.1207 ce 0.7257) | 6.63s/step | table rows seen 1,627,492
+[03:13:24 +222.6m] step 1980 loss 0.6249 (kd 0.1784 ce 0.4464) | 6.62s/step | table rows seen 1,637,908
+[03:15:21 +224.5m] step 2000 loss 0.6492 (kd 0.2176 ce 0.4316) | 6.62s/step | table rows seen 1,646,932
+[03:16:42 +225.9m]   held-out CE @ 2000: SFT 1.8647 (start 1.7533) | OOD 2.5415 (start 2.4101)
+[03:16:42 +225.9m]   table decay @ 2000: visited 1,646,932 rows, visits 3,642,040 | |row| median 0.0346 p90 0.0871 | below snap thr 0.01: 25.3%
+[03:17:48 +227.0m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 560.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.31 GiB is allocated by PyTorch, and 567.18 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[03:17:51 +227.0m]   HC stats @ 2000: gate|dev| median 0.0255 (L0 0.0246, L39 0.0261) | inject w|dev| median 0.2668 | stream spread median 0.4342 (L39 0.4772)
+[03:19:48 +229.0m] step 2020 loss 0.7244 (kd 0.1782 ce 0.5462) | 6.68s/step | table rows seen 1,658,016
+[03:21:48 +231.0m] step 2040 loss 0.5568 (kd 0.1432 ce 0.4135) | 6.68s/step | table rows seen 1,667,080
+[03:23:47 +233.0m] step 2060 loss 0.8077 (kd 0.1375 ce 0.6703) | 6.67s/step | table rows seen 1,680,216
+[03:25:48 +235.0m] step 2080 loss 0.8207 (kd 0.1372 ce 0.6836) | 6.66s/step | table rows seen 1,696,292
+[03:27:46 +237.0m] step 2100 loss 0.7976 (kd 0.1959 ce 0.6017) | 6.66s/step | table rows seen 1,707,348
+[03:29:48 +239.0m] step 2120 loss 0.5224 (kd 0.2019 ce 0.3205) | 6.65s/step | table rows seen 1,716,704
+[03:31:48 +241.0m] step 2140 loss 0.4393 (kd 0.1352 ce 0.3041) | 6.64s/step | table rows seen 1,721,868
+[03:33:46 +243.0m] step 2160 loss 0.6873 (kd 0.1317 ce 0.5555) | 6.64s/step | table rows seen 1,733,460
+[03:35:47 +245.0m] step 2180 loss 0.8518 (kd 0.2838 ce 0.5680) | 6.63s/step | table rows seen 1,748,216
+[03:37:47 +247.0m] step 2200 loss 0.7762 (kd 0.2200 ce 0.5562) | 6.63s/step | table rows seen 1,758,736
+[03:39:07 +248.3m]   held-out CE @ 2200: SFT 1.9216 (start 1.7533) | OOD 2.5608 (start 2.4101)
+[03:39:07 +248.3m]   table decay @ 2200: visited 1,758,736 rows, visits 4,006,364 | |row| median 0.0346 p90 0.0857 | below snap thr 0.01: 25.3%
+[03:40:14 +249.4m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 568.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.34 GiB is allocated by PyTorch, and 534.34 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[03:40:17 +249.5m]   HC stats @ 2200: gate|dev| median 0.0255 (L0 0.0246, L39 0.0262) | inject w|dev| median 0.2661 | stream spread median 0.4341 (L39 0.4787)
+[03:42:19 +251.5m] step 2220 loss 0.5416 (kd 0.1468 ce 0.3947) | 6.69s/step | table rows seen 1,766,656
+[03:44:17 +253.5m] step 2240 loss 0.5638 (kd 0.0914 ce 0.4724) | 6.68s/step | table rows seen 1,774,740
+[03:46:16 +255.5m] step 2260 loss 0.6221 (kd 0.1401 ce 0.4819) | 6.68s/step | table rows seen 1,785,172
+[03:48:16 +257.5m] step 2280 loss 0.6286 (kd 0.1823 ce 0.4463) | 6.67s/step | table rows seen 1,796,448
+[03:50:17 +259.5m] step 2300 loss 0.5680 (kd 0.1532 ce 0.4149) | 6.66s/step | table rows seen 1,804,780
+[03:52:18 +261.5m] step 2320 loss 0.8297 (kd 0.2387 ce 0.5909) | 6.66s/step | table rows seen 1,814,932
+[03:54:15 +263.4m] step 2340 loss 0.8192 (kd 0.1763 ce 0.6429) | 6.65s/step | table rows seen 1,827,736
+[03:56:10 +265.4m] step 2360 loss 0.4520 (kd 0.1861 ce 0.2659) | 6.64s/step | table rows seen 1,831,656
+[03:58:07 +267.3m] step 2380 loss 0.6634 (kd 0.1427 ce 0.5208) | 6.64s/step | table rows seen 1,840,828
+[04:00:08 +269.3m] step 2400 loss 0.7993 (kd 0.2432 ce 0.5562) | 6.63s/step | table rows seen 1,850,636
+[04:01:29 +270.7m]   held-out CE @ 2400: SFT 1.8942 (start 1.7533) | OOD 2.5737 (start 2.4101)
+[04:01:29 +270.7m]   table decay @ 2400: visited 1,850,636 rows, visits 4,366,876 | |row| median 0.0345 p90 0.0842 | below snap thr 0.01: 25.3%
+[04:02:36 +271.8m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 564.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.32 GiB is allocated by PyTorch, and 552.83 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[04:02:38 +271.8m]   HC stats @ 2400: gate|dev| median 0.0254 (L0 0.0246, L39 0.0262) | inject w|dev| median 0.2663 | stream spread median 0.4343 (L39 0.4759)
+[04:04:36 +273.8m] step 2420 loss 0.8667 (kd 0.1604 ce 0.7063) | 6.69s/step | table rows seen 1,861,140
+[04:06:36 +275.8m] step 2440 loss 0.8831 (kd 0.2375 ce 0.6457) | 6.68s/step | table rows seen 1,869,844
+[04:08:33 +277.7m] step 2460 loss 0.9722 (kd 0.1964 ce 0.7758) | 6.68s/step | table rows seen 1,879,044
+[04:10:35 +279.8m] step 2480 loss 1.0044 (kd 0.0932 ce 0.9112) | 6.67s/step | table rows seen 1,893,988
+[04:12:35 +281.8m] step 2500 loss 1.0264 (kd 0.2801 ce 0.7462) | 6.67s/step | table rows seen 1,905,204
+[04:14:38 +283.8m] step 2520 loss 0.9850 (kd 0.1625 ce 0.8225) | 6.66s/step | table rows seen 1,915,944
+[04:16:36 +285.8m] step 2540 loss 0.7974 (kd 0.2119 ce 0.5855) | 6.66s/step | table rows seen 1,922,184
+[04:18:34 +287.8m] step 2560 loss 0.6704 (kd 0.1285 ce 0.5419) | 6.65s/step | table rows seen 1,930,252
+[04:20:37 +289.8m] step 2580 loss 1.0782 (kd 0.3930 ce 0.6852) | 6.65s/step | table rows seen 1,938,924
+[04:22:40 +291.9m] step 2600 loss 0.8636 (kd 0.2005 ce 0.6632) | 6.64s/step | table rows seen 1,947,824
+[04:24:00 +293.2m]   held-out CE @ 2600: SFT 1.7960 (start 1.7533) | OOD 2.4333 (start 2.4101)
+[04:24:00 +293.2m]   table decay @ 2600: visited 1,947,824 rows, visits 4,732,064 | |row| median 0.0344 p90 0.0843 | below snap thr 0.01: 25.4%
+[04:25:07 +294.3m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 568.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 520.81 MiB is free. Including non-PyTorch memory, this process has 11.05 GiB memory in use. Of the allocated memory 10.34 GiB is allocated by PyTorch, and 575.58 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[04:25:09 +294.3m]   HC stats @ 2600: gate|dev| median 0.0256 (L0 0.0246, L39 0.0263) | inject w|dev| median 0.2693 | stream spread median 0.4348 (L39 0.4819)
+[04:27:04 +296.3m] step 2620 loss 0.8109 (kd 0.2169 ce 0.5940) | 6.69s/step | table rows seen 1,954,768
+[04:29:03 +298.2m] step 2640 loss 0.6391 (kd 0.1967 ce 0.4424) | 6.69s/step | table rows seen 1,962,372
+[04:31:01 +300.2m] step 2660 loss 0.9793 (kd 0.1643 ce 0.8150) | 6.68s/step | table rows seen 1,972,104
+[04:32:57 +302.2m] step 2680 loss 0.9577 (kd 0.2097 ce 0.7481) | 6.67s/step | table rows seen 1,981,504
+[04:34:53 +304.1m] step 2700 loss 0.8327 (kd 0.1880 ce 0.6446) | 6.67s/step | table rows seen 1,987,708
+[04:36:50 +306.0m] step 2720 loss 0.7980 (kd 0.0937 ce 0.7043) | 6.66s/step | table rows seen 1,995,456
+[04:38:47 +308.0m] step 2740 loss 0.9654 (kd 0.1788 ce 0.7866) | 6.66s/step | table rows seen 2,006,556
+[04:40:44 +309.9m] step 2760 loss 0.8031 (kd 0.1966 ce 0.6065) | 6.65s/step | table rows seen 2,012,144
+[04:42:45 +311.9m] step 2780 loss 0.9755 (kd 0.0811 ce 0.8945) | 6.65s/step | table rows seen 2,024,092
+[04:44:46 +314.0m] step 2800 loss 0.7922 (kd 0.1204 ce 0.6718) | 6.64s/step | table rows seen 2,032,732
+[04:46:06 +315.3m]   held-out CE @ 2800: SFT 1.7833 (start 1.7533) | OOD 2.4167 (start 2.4101)
+[04:46:07 +315.3m]   table decay @ 2800: visited 2,032,732 rows, visits 5,087,896 | |row| median 0.0343 p90 0.0847 | below snap thr 0.01: 25.3%
+[04:47:13 +316.4m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 556.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 540.81 MiB is free. Including non-PyTorch memory, this process has 11.03 GiB memory in use. Of the allocated memory 10.30 GiB is allocated by PyTorch, and 599.39 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[04:47:16 +316.5m]   HC stats @ 2800: gate|dev| median 0.0256 (L0 0.0246, L39 0.0263) | inject w|dev| median 0.2719 | stream spread median 0.4347 (L39 0.4818)
+[04:49:16 +318.5m] step 2820 loss 0.9896 (kd 0.1834 ce 0.8062) | 6.69s/step | table rows seen 2,040,548
+[04:51:08 +320.3m] step 2840 loss 0.7147 (kd 0.0851 ce 0.6296) | 6.68s/step | table rows seen 2,048,964
+[04:53:09 +322.3m] step 2860 loss 0.8436 (kd 0.1426 ce 0.7010) | 6.68s/step | table rows seen 2,057,964
+[04:55:07 +324.3m] step 2880 loss 0.7397 (kd 0.2528 ce 0.4869) | 6.67s/step | table rows seen 2,064,296
+[04:57:07 +326.3m] step 2900 loss 0.7150 (kd 0.2374 ce 0.4776) | 6.67s/step | table rows seen 2,069,048
+[04:58:58 +328.2m] step 2920 loss 0.7821 (kd 0.1213 ce 0.6608) | 6.66s/step | table rows seen 2,076,808
+[05:00:58 +330.2m] step 2940 loss 0.8063 (kd 0.1945 ce 0.6118) | 6.66s/step | table rows seen 2,085,408
+[05:02:57 +332.1m] step 2960 loss 0.6143 (kd 0.1003 ce 0.5140) | 6.65s/step | table rows seen 2,091,416
+[05:04:53 +334.1m] step 2980 loss 0.9879 (kd 0.3081 ce 0.6797) | 6.65s/step | table rows seen 2,095,112
+[05:06:55 +336.1m] step 3000 loss 1.0458 (kd 0.3033 ce 0.7425) | 6.64s/step | table rows seen 2,105,156
+[05:08:16 +337.5m]   held-out CE @ 3000: SFT 1.7724 (start 1.7533) | OOD 2.4289 (start 2.4101)
+[05:08:16 +337.5m]   table decay @ 3000: visited 2,105,156 rows, visits 5,450,236 | |row| median 0.0339 p90 0.0830 | below snap thr 0.01: 25.3%
+[05:09:22 +338.6m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 568.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.34 GiB is allocated by PyTorch, and 534.14 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[05:09:25 +338.6m]   HC stats @ 3000: gate|dev| median 0.0256 (L0 0.0246, L39 0.0263) | inject w|dev| median 0.2708 | stream spread median 0.4348 (L39 0.4823)
+[05:11:22 +340.6m] step 3020 loss 0.8186 (kd 0.1296 ce 0.6890) | 6.69s/step | table rows seen 2,112,776
+[05:13:19 +342.5m] step 3040 loss 1.1066 (kd 0.1611 ce 0.9454) | 6.68s/step | table rows seen 2,124,824
+[05:15:12 +344.4m] step 3060 loss 0.7212 (kd 0.1858 ce 0.5354) | 6.67s/step | table rows seen 2,130,284
+[05:17:14 +346.4m] step 3080 loss 1.0642 (kd 0.1097 ce 0.9545) | 6.67s/step | table rows seen 2,142,812
+[05:19:13 +348.4m] step 3100 loss 0.7522 (kd 0.1856 ce 0.5666) | 6.67s/step | table rows seen 2,148,100
+[05:21:12 +350.4m] step 3120 loss 0.6941 (kd 0.2071 ce 0.4870) | 6.66s/step | table rows seen 2,155,036
+[05:23:08 +352.3m] step 3140 loss 0.6744 (kd 0.1297 ce 0.5447) | 6.66s/step | table rows seen 2,163,020
+[05:25:10 +354.4m] step 3160 loss 1.1384 (kd 0.1295 ce 1.0089) | 6.65s/step | table rows seen 2,172,540
+[05:27:07 +356.3m] step 3180 loss 0.9803 (kd 0.1181 ce 0.8622) | 6.65s/step | table rows seen 2,182,772
+[05:29:09 +358.3m] step 3200 loss 1.0212 (kd 0.2356 ce 0.7856) | 6.64s/step | table rows seen 2,190,496
+[05:30:29 +359.7m]   held-out CE @ 3200: SFT 1.7614 (start 1.7533) | OOD 2.4030 (start 2.4101)
+[05:30:29 +359.7m]   table decay @ 3200: visited 2,190,496 rows, visits 5,811,712 | |row| median 0.0340 p90 0.0835 | below snap thr 0.01: 25.4%
+[05:31:36 +360.8m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 558.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.31 GiB is allocated by PyTorch, and 569.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[05:31:38 +360.8m]   HC stats @ 3200: gate|dev| median 0.0256 (L0 0.0246, L39 0.0263) | inject w|dev| median 0.2716 | stream spread median 0.4345 (L39 0.4825)
+[05:31:39 +360.8m]   new best by OOD CE: step 3200
+[05:33:42 +362.9m] step 3220 loss 1.2304 (kd 0.0784 ce 1.1520) | 6.69s/step | table rows seen 2,202,804
+[05:35:42 +364.9m] step 3240 loss 0.9116 (kd 0.1923 ce 0.7193) | 6.68s/step | table rows seen 2,212,264
+[05:37:41 +366.9m] step 3260 loss 1.1084 (kd 0.1817 ce 0.9268) | 6.68s/step | table rows seen 2,222,832
+[05:39:41 +368.9m] step 3280 loss 0.7796 (kd 0.1743 ce 0.6052) | 6.67s/step | table rows seen 2,227,880
+[05:41:39 +370.8m] step 3300 loss 0.8441 (kd 0.1637 ce 0.6804) | 6.67s/step | table rows seen 2,234,240
+[05:43:41 +372.9m] step 3320 loss 1.0024 (kd 0.1197 ce 0.8827) | 6.67s/step | table rows seen 2,245,272
+[05:45:38 +374.8m] step 3340 loss 1.0320 (kd 0.2108 ce 0.8212) | 6.66s/step | table rows seen 2,252,544
+[05:47:39 +376.8m] step 3360 loss 0.7988 (kd 0.1225 ce 0.6762) | 6.66s/step | table rows seen 2,260,016
+[05:49:33 +378.7m] step 3380 loss 0.9051 (kd 0.1145 ce 0.7906) | 6.65s/step | table rows seen 2,268,396
+[05:51:32 +380.7m] step 3400 loss 0.6971 (kd 0.1240 ce 0.5731) | 6.65s/step | table rows seen 2,275,920
+[05:52:52 +382.1m]   held-out CE @ 3400: SFT 1.7461 (start 1.7533) | OOD 2.3833 (start 2.4101)
+[05:52:52 +382.1m]   table decay @ 3400: visited 2,275,920 rows, visits 6,176,988 | |row| median 0.0339 p90 0.0823 | below snap thr 0.01: 25.4%
+[05:54:15 +383.4m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 614.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 360.81 MiB is free. Including non-PyTorch memory, this process has 11.20 GiB memory in use. Of the allocated memory 10.51 GiB is allocated by PyTorch, and 560.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[05:54:17 +383.5m]   HC stats @ 3400: gate|dev| median 0.0256 (L0 0.0246, L39 0.0263) | inject w|dev| median 0.2746 | stream spread median 0.4347 (L39 0.4863)
+[05:54:18 +383.5m]   new best by OOD CE: step 3400
+[05:56:16 +385.5m] step 3420 loss 1.2096 (kd 0.1976 ce 1.0120) | 6.69s/step | table rows seen 2,286,816
+[05:58:15 +387.4m] step 3440 loss 1.0763 (kd 0.1675 ce 0.9088) | 6.69s/step | table rows seen 2,298,660
+[06:00:15 +389.4m] step 3460 loss 1.1326 (kd 0.1790 ce 0.9536) | 6.68s/step | table rows seen 2,308,852
+[06:02:13 +391.4m] step 3480 loss 0.9283 (kd 0.2384 ce 0.6899) | 6.68s/step | table rows seen 2,314,864
+[06:04:12 +393.4m] step 3500 loss 0.8880 (kd 0.1580 ce 0.7300) | 6.68s/step | table rows seen 2,325,612
+[06:06:03 +395.2m] step 3520 loss 0.7127 (kd 0.1334 ce 0.5793) | 6.67s/step | table rows seen 2,332,536
+[06:08:03 +397.2m] step 3540 loss 0.8496 (kd 0.1710 ce 0.6786) | 6.67s/step | table rows seen 2,338,736
+[06:10:00 +399.2m] step 3560 loss 0.8284 (kd 0.0945 ce 0.7338) | 6.66s/step | table rows seen 2,347,004
+[06:11:57 +401.1m] step 3580 loss 0.7573 (kd 0.1165 ce 0.6408) | 6.66s/step | table rows seen 2,354,836
+[06:13:52 +403.1m] step 3600 loss 0.9234 (kd 0.1274 ce 0.7960) | 6.65s/step | table rows seen 2,361,168
+[06:15:12 +404.4m]   held-out CE @ 3600: SFT 1.7412 (start 1.7533) | OOD 2.3884 (start 2.4101)
+[06:15:12 +404.4m]   table decay @ 3600: visited 2,361,168 rows, visits 6,539,612 | |row| median 0.0338 p90 0.0826 | below snap thr 0.01: 25.4%
+[06:16:32 +405.7m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 564.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 560.81 MiB is free. Including non-PyTorch memory, this process has 11.01 GiB memory in use. Of the allocated memory 10.33 GiB is allocated by PyTorch, and 542.18 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[06:16:35 +405.8m]   HC stats @ 3600: gate|dev| median 0.0256 (L0 0.0246, L39 0.0263) | inject w|dev| median 0.2748 | stream spread median 0.4346 (L39 0.4831)
+[06:18:37 +407.8m] step 3620 loss 1.1276 (kd 0.1029 ce 1.0247) | 6.69s/step | table rows seen 2,370,428
+[06:20:35 +409.8m] step 3640 loss 1.1933 (kd 0.2100 ce 0.9833) | 6.69s/step | table rows seen 2,377,240
+[06:22:34 +411.8m] step 3660 loss 1.1197 (kd 0.2041 ce 0.9156) | 6.68s/step | table rows seen 2,383,760
+[06:24:33 +413.7m] step 3680 loss 0.8583 (kd 0.2508 ce 0.6075) | 6.68s/step | table rows seen 2,387,428
+[06:26:31 +415.7m] step 3700 loss 1.3863 (kd 0.2641 ce 1.1222) | 6.68s/step | table rows seen 2,397,680
+[06:28:31 +417.7m] step 3720 loss 0.9425 (kd 0.1212 ce 0.8212) | 6.67s/step | table rows seen 2,403,884
+[06:30:29 +419.7m] step 3740 loss 0.9291 (kd 0.3084 ce 0.6207) | 6.67s/step | table rows seen 2,407,980
+[06:32:28 +421.7m] step 3760 loss 0.9854 (kd 0.0934 ce 0.8920) | 6.66s/step | table rows seen 2,414,944
+[06:34:25 +423.6m] step 3780 loss 1.1863 (kd 0.1145 ce 1.0718) | 6.66s/step | table rows seen 2,423,356
+[06:36:23 +425.6m] step 3800 loss 1.0825 (kd 0.1895 ce 0.8930) | 6.66s/step | table rows seen 2,429,420
+[06:37:43 +426.9m]   held-out CE @ 3800: SFT 1.6932 (start 1.7533) | OOD 2.3166 (start 2.4101)
+[06:37:43 +426.9m]   table decay @ 3800: visited 2,429,420 rows, visits 6,903,576 | |row| median 0.0339 p90 0.0817 | below snap thr 0.01: 25.3%
+[06:38:50 +428.0m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 568.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 540.81 MiB is free. Including non-PyTorch memory, this process has 11.03 GiB memory in use. Of the allocated memory 10.33 GiB is allocated by PyTorch, and 562.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[06:38:52 +428.1m]   HC stats @ 3800: gate|dev| median 0.0257 (L0 0.0246, L39 0.0264) | inject w|dev| median 0.2850 | stream spread median 0.4347 (L39 0.4927)
+[06:38:54 +428.1m]   new best by OOD CE: step 3800
+[06:40:52 +430.1m] step 3820 loss 0.7778 (kd 0.0991 ce 0.6787) | 6.69s/step | table rows seen 2,434,700
+[06:42:53 +432.1m] step 3840 loss 1.2140 (kd 0.2644 ce 0.9495) | 6.69s/step | table rows seen 2,442,100
+[06:44:48 +434.0m] step 3860 loss 0.9394 (kd 0.2484 ce 0.6910) | 6.68s/step | table rows seen 2,446,052
+[06:46:51 +436.0m] step 3880 loss 1.2116 (kd 0.2976 ce 0.9140) | 6.68s/step | table rows seen 2,453,588
+[06:48:52 +438.1m] step 3900 loss 0.9262 (kd 0.1174 ce 0.8088) | 6.68s/step | table rows seen 2,460,416
+[06:50:52 +440.1m] step 3920 loss 1.4123 (kd 0.1614 ce 1.2509) | 6.67s/step | table rows seen 2,470,800
+[06:52:50 +442.0m] step 3940 loss 1.0984 (kd 0.1208 ce 0.9776) | 6.67s/step | table rows seen 2,479,500
+[06:54:49 +444.0m] step 3960 loss 1.4449 (kd 0.1535 ce 1.2915) | 6.67s/step | table rows seen 2,490,056
+[06:56:52 +446.1m] step 3980 loss 0.8194 (kd 0.1847 ce 0.6348) | 6.66s/step | table rows seen 2,495,020
+[06:58:48 +448.0m] step 4000 loss 0.6888 (kd 0.1744 ce 0.5144) | 6.66s/step | table rows seen 2,498,388
+[07:00:08 +449.3m]   held-out CE @ 4000: SFT 1.6626 (start 1.7533) | OOD 2.3022 (start 2.4101)
+[07:00:08 +449.3m]   table decay @ 4000: visited 2,498,388 rows, visits 7,266,108 | |row| median 0.0337 p90 0.0811 | below snap thr 0.01: 25.3%
+[07:01:15 +450.4m]   PLE gain failed: OutOfMemoryError('CUDA out of memory. Tried to allocate 572.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 520.81 MiB is free. Including non-PyTorch memory, this process has 11.05 GiB memory in use. Of the allocated memory 10.36 GiB is allocated by PyTorch, and 558.08 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)')
+[07:01:17 +450.5m]   HC stats @ 4000: gate|dev| median 0.0257 (L0 0.0246, L39 0.0264) | inject w|dev| median 0.2905 | stream spread median 0.4351 (L39 0.4960)
+[07:01:18 +450.5m]   new best by OOD CE: step 4000
+[07:03:14 +452.4m] step 4020 loss 1.4851 (kd 0.2164 ce 1.2687) | 6.69s/step | table rows seen 2,508,132
+[07:05:19 +454.5m] step 4040 loss 1.2833 (kd 0.1434 ce 1.1398) | 6.69s/step | table rows seen 2,517,448
+[07:07:16 +456.5m] step 4060 loss 0.8777 (kd 0.1172 ce 0.7605) | 6.69s/step | table rows seen 2,524,092
+[07:09:14 +458.4m] step 4080 loss 1.1179 (kd 0.1861 ce 0.9317) | 6.68s/step | table rows seen 2,532,048
+[07:11:13 +460.4m] step 4100 loss 1.1634 (kd 0.0749 ce 1.0885) | 6.68s/step | table rows seen 2,541,952
+[07:13:12 +462.4m] step 4120 loss 1.1729 (kd 0.0990 ce 1.0739) | 6.68s/step | table rows seen 2,550,712
+[07:14:54 +464.1m] TIME BUDGET: stopping at step 4138
+[07:14:56 +464.1m]   saved the final state as next_step4137 (4 Sep: a run must never end without a checkpoint)
+[07:14:56 +464.1m] DONE best step 4000 OOD 2.3022 (start 2.4101) | checkpoints in /home/support/whittle-train/reader7/ckpts | table /home/support/whittle-train/reader7/ngram_table.npy

ai2-archive/reader7/keep_final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82ff45b62b979693e9dedd2e05312c1929a1f4e7a50771c78878b544771b97ae
+size 2651263951

ai2-archive/reader7/ngram_table.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:beb55b5ca14166b3f0d941ccb3383d579ac44bf148cec1cdf32e86982466a1a6
+size 4000317568

ai2-archive/reader7/ple_hash.json

@@ -0,0 +1,12 @@
+{
+ "multipliers": [
+  32473892291597,
+  22010589867355,
+  6403636407469
+ ],
+ "rows_per_head": 976640,
+ "heads": 8,
+ "ngram": 3,
+ "eos": 248044,
+ "head_dim": 256
+}

ai2-archive/reader7/ple_hash.json.keep

@@ -0,0 +1,12 @@
+{
+ "multipliers": [
+  32473892291597,
+  22010589867355,
+  6403636407469
+ ],
+ "rows_per_head": 976640,
+ "heads": 8,
+ "ngram": 3,
+ "eos": 248044,
+ "head_dim": 256
+}

ai2-archive/reader7/train.pid

@@ -0,0 +1 @@
+3103454