19:43:45 pkm2 (query BN, keys lr 1e-04) test from global round 537 step 140250: 1000 steps x 64 rows, arms ['mem', 'ctrl'], 2097152 slots
19:57:06 data ready in 801s
19:57:20 pkm arm: 2097152 slots (537M value params), 3.9M query/out params, read before blocks [3, 9, 14], top 16 of 8^3
20:00:13   mem step 50 loss 3.3036 (173s) |wo| 1.955,1.914,1.888  slots used/batch 194944,286423,285886
20:02:58   mem step 100 loss 3.7698 (338s) |wo| 3.134,3.135,3.059  slots used/batch 145583,255082,267721
20:05:42   mem step 150 loss 3.1698 (502s) |wo| 3.965,3.990,3.883  slots used/batch 96545,209817,253652
20:08:27   mem step 200 loss 3.3056 (667s) |wo| 4.639,4.692,4.556  slots used/batch 75128,181667,200741
20:11:11   mem step 250 loss 2.9771 (831s) |wo| 5.227,5.300,5.142  slots used/batch 89016,186179,212295
20:11:19 EVAL mem step 250 | facts mc 0.5543 nll 6.0017 em 0.0800
20:14:04   mem step 300 loss 2.9544 (1004s) |wo| 5.753,5.812,5.681  slots used/batch 71559,149147,193290
20:16:48   mem step 350 loss 3.3512 (1168s) |wo| 6.230,6.311,6.164  slots used/batch 77739,164676,202873
20:19:33   mem step 400 loss 3.2505 (1333s) |wo| 6.660,6.761,6.596  slots used/batch 80859,150350,184488
20:22:17   mem step 450 loss 3.2095 (1497s) |wo| 7.057,7.181,6.997  slots used/batch 78922,139819,186551
20:25:02   mem step 500 loss 2.9035 (1662s) |wo| 7.432,7.579,7.378  slots used/batch 73173,137114,184838
20:25:03 EVAL mem step 500 | facts mc 0.5476 nll 6.1074 em 0.0762
20:27:48   mem step 550 loss 3.9067 (1828s) |wo| 7.795,7.955,7.747  slots used/batch 67600,121021,165555
20:30:33   mem step 600 loss 3.0004 (1993s) |wo| 8.142,8.298,8.090  slots used/batch 78596,138816,184164
20:33:18   mem step 650 loss 3.0662 (2158s) |wo| 8.474,8.626,8.425  slots used/batch 71319,125133,178872
20:36:02   mem step 700 loss 2.9204 (2322s) |wo| 8.785,8.941,8.740  slots used/batch 68821,112064,161948
20:38:47   mem step 750 loss 3.5282 (2487s) |wo| 9.092,9.253,9.030  slots used/batch 67685,107322,150310
20:38:48 EVAL mem step 750 | facts mc 0.5438 nll 6.1373 em 0.0629
20:41:34   mem step 800 loss 4.0744 (2654s) |wo| 9.376,9.549,9.312  slots used/batch 72349,109279,148882
20:44:18   mem step 850 loss 3.5503 (2818s) |wo| 9.662,9.831,9.592  slots used/batch 80672,118638,161610
20:47:03   mem step 900 loss 2.9436 (2983s) |wo| 9.917,10.105,9.865  slots used/batch 83581,125061,167604
20:49:48   mem step 950 loss 3.4850 (3148s) |wo| 10.190,10.381,10.130  slots used/batch 76186,124659,149468
20:52:32   mem step 1000 loss 2.9999 (3312s) |wo| 10.454,10.644,10.391  slots used/batch 75856,123367,151795
20:52:35 EVAL mem step 1000 | facts mc 0.5410 nll 6.1012 em 0.0752  long 2.9045  short 4.0013
20:55:21   ctrl step 50 loss 3.3033 (156s)
20:57:57   ctrl step 100 loss 3.7699 (311s)
21:00:32   ctrl step 150 loss 3.1699 (467s)
21:03:08   ctrl step 200 loss 3.3056 (623s)
21:05:44   ctrl step 250 loss 2.9774 (778s)
21:05:45 EVAL ctrl step 250 | facts mc 0.5543 nll 6.0053 em 0.0781
21:08:21   ctrl step 300 loss 2.9543 (936s)
21:10:57   ctrl step 350 loss 3.3510 (1091s)
21:13:32   ctrl step 400 loss 3.2504 (1247s)
21:16:08   ctrl step 450 loss 3.2093 (1403s)
21:18:43   ctrl step 500 loss 2.9035 (1558s)
21:18:45 EVAL ctrl step 500 | facts mc 0.5457 nll 6.1106 em 0.0743
21:21:21   ctrl step 550 loss 3.9069 (1716s)
21:23:57   ctrl step 600 loss 3.0005 (1872s)
21:26:33   ctrl step 650 loss 3.0659 (2027s)
21:29:08   ctrl step 700 loss 2.9203 (2183s)
21:31:44   ctrl step 750 loss 3.5284 (2339s)
21:31:45 EVAL ctrl step 750 | facts mc 0.5457 nll 6.1399 em 0.0629
21:34:21   ctrl step 800 loss 4.0746 (2496s)
21:36:57   ctrl step 850 loss 3.5510 (2652s)
21:39:33   ctrl step 900 loss 2.9436 (2808s)
21:42:09   ctrl step 950 loss 3.4849 (2963s)
21:44:44   ctrl step 1000 loss 3.0001 (3119s)
21:44:46 EVAL ctrl step 1000 | facts mc 0.5381 nll 6.3159 em 0.0590  long 2.9046  short 4.0030
21:44:47 PKM DONE