File size: 3,891 Bytes
a02b918 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | 19:43:45 pkm2 (query BN, keys lr 1e-04) test from global round 537 step 140250: 1000 steps x 64 rows, arms ['mem', 'ctrl'], 2097152 slots
19:57:06 data ready in 801s
19:57:20 pkm arm: 2097152 slots (537M value params), 3.9M query/out params, read before blocks [3, 9, 14], top 16 of 8^3
20:00:13 mem step 50 loss 3.3036 (173s) |wo| 1.955,1.914,1.888 slots used/batch 194944,286423,285886
20:02:58 mem step 100 loss 3.7698 (338s) |wo| 3.134,3.135,3.059 slots used/batch 145583,255082,267721
20:05:42 mem step 150 loss 3.1698 (502s) |wo| 3.965,3.990,3.883 slots used/batch 96545,209817,253652
20:08:27 mem step 200 loss 3.3056 (667s) |wo| 4.639,4.692,4.556 slots used/batch 75128,181667,200741
20:11:11 mem step 250 loss 2.9771 (831s) |wo| 5.227,5.300,5.142 slots used/batch 89016,186179,212295
20:11:19 EVAL mem step 250 | facts mc 0.5543 nll 6.0017 em 0.0800
20:14:04 mem step 300 loss 2.9544 (1004s) |wo| 5.753,5.812,5.681 slots used/batch 71559,149147,193290
20:16:48 mem step 350 loss 3.3512 (1168s) |wo| 6.230,6.311,6.164 slots used/batch 77739,164676,202873
20:19:33 mem step 400 loss 3.2505 (1333s) |wo| 6.660,6.761,6.596 slots used/batch 80859,150350,184488
20:22:17 mem step 450 loss 3.2095 (1497s) |wo| 7.057,7.181,6.997 slots used/batch 78922,139819,186551
20:25:02 mem step 500 loss 2.9035 (1662s) |wo| 7.432,7.579,7.378 slots used/batch 73173,137114,184838
20:25:03 EVAL mem step 500 | facts mc 0.5476 nll 6.1074 em 0.0762
20:27:48 mem step 550 loss 3.9067 (1828s) |wo| 7.795,7.955,7.747 slots used/batch 67600,121021,165555
20:30:33 mem step 600 loss 3.0004 (1993s) |wo| 8.142,8.298,8.090 slots used/batch 78596,138816,184164
20:33:18 mem step 650 loss 3.0662 (2158s) |wo| 8.474,8.626,8.425 slots used/batch 71319,125133,178872
20:36:02 mem step 700 loss 2.9204 (2322s) |wo| 8.785,8.941,8.740 slots used/batch 68821,112064,161948
20:38:47 mem step 750 loss 3.5282 (2487s) |wo| 9.092,9.253,9.030 slots used/batch 67685,107322,150310
20:38:48 EVAL mem step 750 | facts mc 0.5438 nll 6.1373 em 0.0629
20:41:34 mem step 800 loss 4.0744 (2654s) |wo| 9.376,9.549,9.312 slots used/batch 72349,109279,148882
20:44:18 mem step 850 loss 3.5503 (2818s) |wo| 9.662,9.831,9.592 slots used/batch 80672,118638,161610
20:47:03 mem step 900 loss 2.9436 (2983s) |wo| 9.917,10.105,9.865 slots used/batch 83581,125061,167604
20:49:48 mem step 950 loss 3.4850 (3148s) |wo| 10.190,10.381,10.130 slots used/batch 76186,124659,149468
20:52:32 mem step 1000 loss 2.9999 (3312s) |wo| 10.454,10.644,10.391 slots used/batch 75856,123367,151795
20:52:35 EVAL mem step 1000 | facts mc 0.5410 nll 6.1012 em 0.0752 long 2.9045 short 4.0013
20:55:21 ctrl step 50 loss 3.3033 (156s)
20:57:57 ctrl step 100 loss 3.7699 (311s)
21:00:32 ctrl step 150 loss 3.1699 (467s)
21:03:08 ctrl step 200 loss 3.3056 (623s)
21:05:44 ctrl step 250 loss 2.9774 (778s)
21:05:45 EVAL ctrl step 250 | facts mc 0.5543 nll 6.0053 em 0.0781
21:08:21 ctrl step 300 loss 2.9543 (936s)
21:10:57 ctrl step 350 loss 3.3510 (1091s)
21:13:32 ctrl step 400 loss 3.2504 (1247s)
21:16:08 ctrl step 450 loss 3.2093 (1403s)
21:18:43 ctrl step 500 loss 2.9035 (1558s)
21:18:45 EVAL ctrl step 500 | facts mc 0.5457 nll 6.1106 em 0.0743
21:21:21 ctrl step 550 loss 3.9069 (1716s)
21:23:57 ctrl step 600 loss 3.0005 (1872s)
21:26:33 ctrl step 650 loss 3.0659 (2027s)
21:29:08 ctrl step 700 loss 2.9203 (2183s)
21:31:44 ctrl step 750 loss 3.5284 (2339s)
21:31:45 EVAL ctrl step 750 | facts mc 0.5457 nll 6.1399 em 0.0629
21:34:21 ctrl step 800 loss 4.0746 (2496s)
21:36:57 ctrl step 850 loss 3.5510 (2652s)
21:39:33 ctrl step 900 loss 2.9436 (2808s)
21:42:09 ctrl step 950 loss 3.4849 (2963s)
21:44:44 ctrl step 1000 loss 3.0001 (3119s)
21:44:46 EVAL ctrl step 1000 | facts mc 0.5381 nll 6.3159 em 0.0590 long 2.9046 short 4.0030
21:44:47 PKM DONE
|