Hugging Face's logo

SlayerLab
/
dowody400m-checkpoints

PyTorch
polish
checkpoints
training-history
Model card Files
xet
 Community
dowody400m-checkpoints / K22 /run /train.log
Maggio33's picture
Maggio33
Add run records (args, metrics, logs, val, environment) for all 400M runs, fala-1 finals K22/R1-R3, resume test, code
c28a890 verified
Raw History Blame Contribute Delete
21.9 kB
[13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
[13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
[13:59:58] model GPT-ref: 180.9M param (L22 d768 h12)
[13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
[13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
[13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[14:00:00] torch.compile enabled
[14:00:00] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
[14:04:48] step 20/4768 loss 10.3787 lr 2.40e-05 gnorm 2.38 36,394 tok/s peakVRAM 13.7GB
[14:08:15] step 40/4768 loss 9.6271 lr 4.80e-05 gnorm 1.40 50,498 tok/s peakVRAM 13.7GB
[14:11:43] step 60/4768 loss 8.8371 lr 7.20e-05 gnorm 1.11 50,510 tok/s peakVRAM 13.7GB
[14:15:11] step 80/4768 loss 8.1619 lr 9.60e-05 gnorm 1.04 50,497 tok/s peakVRAM 13.7GB
[14:18:38] step 100/4768 loss 7.6838 lr 1.20e-04 gnorm 0.69 50,466 tok/s peakVRAM 13.7GB
[14:22:06] step 120/4768 loss 7.2235 lr 1.44e-04 gnorm 0.83 50,441 tok/s peakVRAM 13.7GB
[14:25:34] step 140/4768 loss 6.8162 lr 1.68e-04 gnorm 0.88 50,431 tok/s peakVRAM 13.7GB
[14:29:02] step 160/4768 loss 6.4518 lr 1.92e-04 gnorm 0.96 50,418 tok/s peakVRAM 13.7GB
[14:32:30] step 180/4768 loss 6.1645 lr 2.16e-04 gnorm 0.75 50,412 tok/s peakVRAM 13.7GB
[14:35:58] step 200/4768 loss 5.9285 lr 2.40e-04 gnorm 0.78 50,411 tok/s peakVRAM 13.7GB
[14:39:26] step 220/4768 loss 5.7298 lr 2.64e-04 gnorm 0.55 50,409 tok/s peakVRAM 13.7GB
[14:42:54] step 240/4768 loss 5.5502 lr 2.88e-04 gnorm 0.49 50,408 tok/s peakVRAM 13.7GB
[14:46:22] step 260/4768 loss 5.3819 lr 3.12e-04 gnorm 0.59 50,407 tok/s peakVRAM 13.7GB
[14:49:50] step 280/4768 loss 5.2249 lr 3.36e-04 gnorm 0.58 50,410 tok/s peakVRAM 13.7GB
[14:53:18] step 300/4768 loss 5.0776 lr 3.60e-04 gnorm 0.52 50,409 tok/s peakVRAM 13.7GB
[14:56:46] step 320/4768 loss 4.9310 lr 3.84e-04 gnorm 0.63 50,410 tok/s peakVRAM 13.7GB
[15:00:14] step 340/4768 loss 4.7736 lr 4.08e-04 gnorm 0.59 50,408 tok/s peakVRAM 13.7GB
[15:03:42] step 360/4768 loss 4.6485 lr 4.32e-04 gnorm 0.58 50,401 tok/s peakVRAM 13.7GB
[15:07:10] step 380/4768 loss 4.5306 lr 4.56e-04 gnorm 0.55 50,404 tok/s peakVRAM 13.7GB
[15:10:38] step 400/4768 loss 4.4408 lr 4.80e-04 gnorm 0.48 50,402 tok/s peakVRAM 13.7GB
[15:14:06] step 420/4768 loss 4.3643 lr 5.04e-04 gnorm 0.43 50,404 tok/s peakVRAM 13.7GB
[15:17:34] step 440/4768 loss 4.2946 lr 5.28e-04 gnorm 0.50 50,403 tok/s peakVRAM 13.7GB
[15:21:02] step 460/4768 loss 4.2252 lr 5.52e-04 gnorm 0.44 50,392 tok/s peakVRAM 13.7GB
[15:24:31] step 480/4768 loss 4.1773 lr 5.76e-04 gnorm 0.43 50,388 tok/s peakVRAM 13.7GB
[15:27:59] step 500/4768 loss 4.1259 lr 6.00e-04 gnorm 0.40 50,392 tok/s peakVRAM 13.7GB
[15:31:27] step 520/4768 loss 4.0899 lr 6.00e-04 gnorm 0.33 50,395 tok/s peakVRAM 13.7GB
[15:34:55] step 540/4768 loss 4.0379 lr 6.00e-04 gnorm 0.25 50,396 tok/s peakVRAM 13.7GB
[15:38:23] step 560/4768 loss 3.9871 lr 6.00e-04 gnorm 0.33 50,389 tok/s peakVRAM 13.7GB
[15:41:51] step 580/4768 loss 3.9425 lr 6.00e-04 gnorm 0.29 50,387 tok/s peakVRAM 13.7GB
[15:45:19] step 600/4768 loss 3.8993 lr 6.00e-04 gnorm 0.26 50,382 tok/s peakVRAM 13.7GB
[15:48:47] step 620/4768 loss 3.8736 lr 6.00e-04 gnorm 0.25 50,384 tok/s peakVRAM 13.7GB
[15:52:15] step 640/4768 loss 3.8464 lr 6.00e-04 gnorm 0.28 50,384 tok/s peakVRAM 13.7GB
[15:55:44] step 660/4768 loss 3.8137 lr 6.00e-04 gnorm 0.29 50,378 tok/s peakVRAM 13.7GB
[15:59:12] step 680/4768 loss 3.7883 lr 6.00e-04 gnorm 0.22 50,379 tok/s peakVRAM 13.7GB
[16:02:40] step 700/4768 loss 3.7559 lr 6.00e-04 gnorm 0.25 50,379 tok/s peakVRAM 13.7GB
[16:06:08] step 720/4768 loss 3.7238 lr 6.00e-04 gnorm 0.25 50,377 tok/s peakVRAM 13.7GB
[16:09:36] step 740/4768 loss 3.7190 lr 6.00e-04 gnorm 0.28 50,374 tok/s peakVRAM 13.7GB
[16:13:04] step 760/4768 loss 3.6942 lr 6.00e-04 gnorm 0.22 50,379 tok/s peakVRAM 13.7GB
[16:16:32] step 780/4768 loss 3.6677 lr 6.00e-04 gnorm 0.22 50,381 tok/s peakVRAM 13.7GB
[16:20:01] step 800/4768 loss 3.6530 lr 6.00e-04 gnorm 0.20 50,378 tok/s peakVRAM 13.7GB
[16:23:29] step 820/4768 loss 3.6361 lr 6.00e-04 gnorm 0.21 50,380 tok/s peakVRAM 13.7GB
[16:26:57] step 840/4768 loss 3.6213 lr 6.00e-04 gnorm 0.21 50,379 tok/s peakVRAM 13.7GB
[16:30:25] step 860/4768 loss 3.5994 lr 6.00e-04 gnorm 0.18 50,383 tok/s peakVRAM 13.7GB
[16:33:53] step 880/4768 loss 3.5940 lr 6.00e-04 gnorm 0.21 50,370 tok/s peakVRAM 13.7GB
[16:37:21] step 900/4768 loss 3.5738 lr 6.00e-04 gnorm 0.18 50,372 tok/s peakVRAM 13.7GB
[16:40:49] step 920/4768 loss 3.5661 lr 6.00e-04 gnorm 0.19 50,372 tok/s peakVRAM 13.7GB
[16:44:18] step 940/4768 loss 3.5609 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
[16:47:46] step 960/4768 loss 3.5345 lr 6.00e-04 gnorm 0.17 50,374 tok/s peakVRAM 13.7GB
[16:51:14] step 980/4768 loss 3.5308 lr 6.00e-04 gnorm 0.17 50,370 tok/s peakVRAM 13.7GB
[16:54:42] step 1000/4768 loss 3.5204 lr 6.00e-04 gnorm 0.17 50,369 tok/s peakVRAM 13.7GB
[16:58:10] step 1020/4768 loss 3.5146 lr 6.00e-04 gnorm 0.19 50,360 tok/s peakVRAM 13.7GB
[17:01:39] step 1040/4768 loss 3.4869 lr 6.00e-04 gnorm 0.18 50,364 tok/s peakVRAM 13.7GB
[17:05:07] step 1060/4768 loss 3.4986 lr 6.00e-04 gnorm 0.18 50,368 tok/s peakVRAM 13.7GB
[17:08:35] step 1080/4768 loss 3.4884 lr 6.00e-04 gnorm 0.16 50,372 tok/s peakVRAM 13.7GB
[17:12:03] step 1100/4768 loss 3.4719 lr 6.00e-04 gnorm 0.21 50,367 tok/s peakVRAM 13.7GB
[17:15:31] step 1120/4768 loss 3.4548 lr 6.00e-04 gnorm 0.18 50,363 tok/s peakVRAM 13.7GB
[17:18:59] step 1140/4768 loss 3.4481 lr 6.00e-04 gnorm 0.16 50,367 tok/s peakVRAM 13.7GB
[17:22:28] step 1160/4768 loss 3.4420 lr 6.00e-04 gnorm 0.17 50,367 tok/s peakVRAM 13.7GB
[17:25:56] step 1180/4768 loss 3.4437 lr 6.00e-04 gnorm 0.16 50,366 tok/s peakVRAM 13.7GB
[17:29:24] step 1200/4768 loss 3.4253 lr 6.00e-04 gnorm 0.19 50,364 tok/s peakVRAM 13.7GB
[17:32:52] step 1220/4768 loss 3.4227 lr 6.00e-04 gnorm 0.14 50,357 tok/s peakVRAM 13.7GB
[17:36:20] step 1240/4768 loss 3.4135 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
[17:39:49] step 1260/4768 loss 3.3992 lr 6.00e-04 gnorm 0.15 50,366 tok/s peakVRAM 13.7GB
[17:43:17] step 1280/4768 loss 3.4017 lr 6.00e-04 gnorm 0.15 50,361 tok/s peakVRAM 13.7GB
[17:46:45] step 1300/4768 loss 3.3952 lr 6.00e-04 gnorm 0.16 50,359 tok/s peakVRAM 13.7GB
[17:50:13] step 1320/4768 loss 3.3867 lr 6.00e-04 gnorm 0.16 50,362 tok/s peakVRAM 13.7GB
[17:53:42] step 1340/4768 loss 3.3866 lr 6.00e-04 gnorm 0.15 50,360 tok/s peakVRAM 13.7GB
[17:57:10] step 1360/4768 loss 3.3772 lr 6.00e-04 gnorm 0.13 50,360 tok/s peakVRAM 13.7GB
[18:00:38] step 1380/4768 loss 3.3684 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
[18:04:06] step 1400/4768 loss 3.3574 lr 6.00e-04 gnorm 0.14 50,367 tok/s peakVRAM 13.7GB
[18:07:34] step 1420/4768 loss 3.3459 lr 6.00e-04 gnorm 0.16 50,368 tok/s peakVRAM 13.7GB
[18:11:02] step 1440/4768 loss 3.3518 lr 6.00e-04 gnorm 0.16 50,366 tok/s peakVRAM 13.7GB
[18:14:31] step 1460/4768 loss 3.3493 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
[18:17:59] step 1480/4768 loss 3.3528 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
[18:21:27] step 1500/4768 loss 3.3260 lr 6.00e-04 gnorm 0.16 50,361 tok/s peakVRAM 13.7GB
[18:24:55] step 1520/4768 loss 3.3333 lr 6.00e-04 gnorm 0.14 50,362 tok/s peakVRAM 13.7GB
[18:28:23] step 1540/4768 loss 3.3187 lr 6.00e-04 gnorm 0.15 50,362 tok/s peakVRAM 13.7GB
[18:31:52] step 1560/4768 loss 3.3264 lr 6.00e-04 gnorm 0.16 50,363 tok/s peakVRAM 13.7GB
[18:35:20] step 1580/4768 loss 3.3143 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
[18:38:48] step 1600/4768 loss 3.3036 lr 6.00e-04 gnorm 0.15 50,366 tok/s peakVRAM 13.7GB
[18:42:16] step 1620/4768 loss 3.3041 lr 6.00e-04 gnorm 0.14 50,364 tok/s peakVRAM 13.7GB
[18:45:44] step 1640/4768 loss 3.3000 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
[18:49:13] step 1660/4768 loss 3.3013 lr 6.00e-04 gnorm 0.13 50,376 tok/s peakVRAM 13.7GB
[18:52:41] step 1680/4768 loss 3.2893 lr 6.00e-04 gnorm 0.13 50,375 tok/s peakVRAM 13.7GB
[18:56:09] step 1700/4768 loss 3.2815 lr 6.00e-04 gnorm 0.12 50,373 tok/s peakVRAM 13.7GB
[18:59:37] step 1720/4768 loss 3.2840 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
[19:03:05] step 1740/4768 loss 3.2778 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[19:06:34] step 1760/4768 loss 3.2767 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
[19:10:02] step 1780/4768 loss 3.2649 lr 6.00e-04 gnorm 0.14 50,359 tok/s peakVRAM 13.7GB
[19:13:30] step 1800/4768 loss 3.2683 lr 6.00e-04 gnorm 0.13 50,354 tok/s peakVRAM 13.7GB
[19:16:58] step 1820/4768 loss 3.2637 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
[19:20:26] step 1840/4768 loss 3.2567 lr 6.00e-04 gnorm 0.12 50,360 tok/s peakVRAM 13.7GB
[19:23:55] step 1860/4768 loss 3.2561 lr 6.00e-04 gnorm 0.13 50,361 tok/s peakVRAM 13.7GB
[19:27:23] step 1880/4768 loss 3.2509 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
[19:30:51] step 1900/4768 loss 3.2456 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
[19:34:19] step 1920/4768 loss 3.2454 lr 6.00e-04 gnorm 0.11 50,365 tok/s peakVRAM 13.7GB
[19:37:47] step 1940/4768 loss 3.2371 lr 6.00e-04 gnorm 0.13 50,361 tok/s peakVRAM 13.7GB
[19:41:16] step 1960/4768 loss 3.2425 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
[19:44:44] step 1980/4768 loss 3.2365 lr 6.00e-04 gnorm 0.12 50,369 tok/s peakVRAM 13.7GB
[19:48:12] step 2000/4768 loss 3.2297 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
[19:51:40] step 2020/4768 loss 3.2300 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
[19:55:08] step 2040/4768 loss 3.2229 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[19:58:36] step 2060/4768 loss 3.2234 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
[20:02:05] step 2080/4768 loss 3.2119 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
[20:05:33] step 2100/4768 loss 3.2206 lr 6.00e-04 gnorm 0.11 50,369 tok/s peakVRAM 13.7GB
[20:09:01] step 2120/4768 loss 3.2098 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[20:12:29] step 2140/4768 loss 3.2225 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
[20:15:57] step 2160/4768 loss 3.2009 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
[20:19:26] step 2180/4768 loss 3.2029 lr 6.00e-04 gnorm 0.11 50,369 tok/s peakVRAM 13.7GB
[20:22:54] step 2200/4768 loss 3.2040 lr 6.00e-04 gnorm 0.12 50,375 tok/s peakVRAM 13.7GB
[20:26:22] step 2220/4768 loss 3.1966 lr 6.00e-04 gnorm 0.15 50,370 tok/s peakVRAM 13.7GB
[20:29:50] step 2240/4768 loss 3.1929 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
[20:33:18] step 2260/4768 loss 3.1895 lr 6.00e-04 gnorm 0.12 50,364 tok/s peakVRAM 13.7GB
[20:36:46] step 2280/4768 loss 3.1854 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
[20:40:15] step 2300/4768 loss 3.1888 lr 6.00e-04 gnorm 0.12 50,365 tok/s peakVRAM 13.7GB
[20:43:43] step 2320/4768 loss 3.1856 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[20:47:11] step 2340/4768 loss 3.1806 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
[20:50:39] step 2360/4768 loss 3.1701 lr 6.00e-04 gnorm 0.12 50,374 tok/s peakVRAM 13.7GB
[20:54:07] step 2380/4768 loss 3.1692 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
[20:57:36] step 2400/4768 loss 3.1698 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
[21:01:04] step 2420/4768 loss 3.1726 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
[21:04:32] step 2440/4768 loss 3.1677 lr 6.00e-04 gnorm 0.14 50,365 tok/s peakVRAM 13.7GB
[21:08:00] step 2460/4768 loss 3.1653 lr 6.00e-04 gnorm 0.15 50,361 tok/s peakVRAM 13.7GB
[21:11:28] step 2480/4768 loss 3.1631 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
[21:14:57] step 2500/4768 loss 3.1575 lr 6.00e-04 gnorm 0.11 50,371 tok/s peakVRAM 13.7GB
[21:18:25] step 2520/4768 loss 3.1594 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
[21:21:53] step 2540/4768 loss 3.1598 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
[21:25:21] step 2560/4768 loss 3.1564 lr 6.00e-04 gnorm 0.13 50,377 tok/s peakVRAM 13.7GB
[21:28:49] step 2580/4768 loss 3.1529 lr 6.00e-04 gnorm 0.14 50,374 tok/s peakVRAM 13.7GB
[21:32:17] step 2600/4768 loss 3.1433 lr 6.00e-04 gnorm 0.12 50,370 tok/s peakVRAM 13.7GB
[21:35:46] step 2620/4768 loss 3.1518 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
[21:39:14] step 2640/4768 loss 3.1528 lr 6.00e-04 gnorm 0.15 50,373 tok/s peakVRAM 13.7GB
[21:42:42] step 2660/4768 loss 3.1508 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
[21:46:10] step 2680/4768 loss 3.1459 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
[21:49:38] step 2700/4768 loss 3.1484 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
[21:53:06] step 2720/4768 loss 3.1315 lr 6.00e-04 gnorm 0.15 50,371 tok/s peakVRAM 13.7GB
[21:56:35] step 2740/4768 loss 3.1328 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
[22:00:03] step 2760/4768 loss 3.1356 lr 6.00e-04 gnorm 0.14 50,367 tok/s peakVRAM 13.7GB
[22:03:31] step 2780/4768 loss 3.1263 lr 6.00e-04 gnorm 0.14 50,365 tok/s peakVRAM 13.7GB
[22:06:59] step 2800/4768 loss 3.1220 lr 6.00e-04 gnorm 0.15 50,367 tok/s peakVRAM 13.7GB
[22:10:27] step 2820/4768 loss 3.1191 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
[22:13:56] step 2840/4768 loss 3.1272 lr 6.00e-04 gnorm 0.16 50,369 tok/s peakVRAM 13.7GB
[22:17:24] step 2860/4768 loss 3.1184 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
[22:20:52] step 2880/4768 loss 3.1203 lr 6.00e-04 gnorm 0.16 50,371 tok/s peakVRAM 13.7GB
[22:24:20] step 2900/4768 loss 3.1217 lr 6.00e-04 gnorm 0.14 50,369 tok/s peakVRAM 13.7GB
[22:27:48] step 2920/4768 loss 3.1290 lr 6.00e-04 gnorm 0.17 50,368 tok/s peakVRAM 13.7GB
[22:31:16] step 2940/4768 loss 3.1193 lr 6.00e-04 gnorm 0.17 50,375 tok/s peakVRAM 13.7GB
[22:34:45] step 2960/4768 loss 3.1210 lr 6.00e-04 gnorm 0.18 50,374 tok/s peakVRAM 13.7GB
[22:38:13] step 2980/4768 loss 3.1083 lr 6.00e-04 gnorm 0.17 50,368 tok/s peakVRAM 13.7GB
[22:41:41] step 3000/4768 loss 3.1096 lr 6.00e-04 gnorm 0.15 50,369 tok/s peakVRAM 13.7GB
[22:45:09] step 3020/4768 loss 3.1041 lr 6.00e-04 gnorm 0.18 50,373 tok/s peakVRAM 13.7GB
[22:48:37] step 3040/4768 loss 3.1019 lr 6.00e-04 gnorm 0.16 50,369 tok/s peakVRAM 13.7GB
[22:52:05] step 3060/4768 loss 3.1055 lr 6.00e-04 gnorm 0.18 50,371 tok/s peakVRAM 13.7GB
[22:55:34] step 3080/4768 loss 3.0979 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
[22:59:02] step 3100/4768 loss 3.0916 lr 6.00e-04 gnorm 0.19 50,370 tok/s peakVRAM 13.7GB
[23:02:30] step 3120/4768 loss 3.1072 lr 6.00e-04 gnorm 0.20 50,365 tok/s peakVRAM 13.7GB
[23:05:58] step 3140/4768 loss 3.0934 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
[23:09:26] step 3160/4768 loss 3.0890 lr 6.00e-04 gnorm 0.17 50,366 tok/s peakVRAM 13.7GB
[23:12:55] step 3180/4768 loss 3.0953 lr 6.00e-04 gnorm 0.20 50,363 tok/s peakVRAM 13.7GB
[23:16:23] step 3200/4768 loss 3.0965 lr 6.00e-04 gnorm 0.21 50,369 tok/s peakVRAM 13.7GB
[23:19:51] step 3220/4768 loss 3.1058 lr 6.00e-04 gnorm 0.24 50,367 tok/s peakVRAM 13.7GB
[23:23:19] step 3240/4768 loss 3.0879 lr 6.00e-04 gnorm 0.22 50,369 tok/s peakVRAM 13.7GB
[23:26:47] step 3260/4768 loss 3.0842 lr 6.00e-04 gnorm 0.23 50,365 tok/s peakVRAM 13.7GB
[23:30:16] step 3280/4768 loss 3.0803 lr 6.00e-04 gnorm 0.20 50,360 tok/s peakVRAM 13.7GB
[23:33:44] step 3300/4768 loss 3.0907 lr 6.00e-04 gnorm 0.23 50,362 tok/s peakVRAM 13.7GB
[23:37:12] step 3320/4768 loss 3.0850 lr 6.00e-04 gnorm 0.27 50,360 tok/s peakVRAM 13.7GB
[23:40:40] step 3340/4768 loss 3.0855 lr 6.00e-04 gnorm 0.23 50,356 tok/s peakVRAM 13.7GB
[23:44:08] step 3360/4768 loss 3.0760 lr 6.00e-04 gnorm 0.23 50,359 tok/s peakVRAM 13.7GB
[23:47:37] step 3380/4768 loss 3.0862 lr 6.00e-04 gnorm 0.25 50,372 tok/s peakVRAM 13.7GB
[23:51:05] step 3400/4768 loss 3.0713 lr 6.00e-04 gnorm 0.25 50,370 tok/s peakVRAM 13.7GB
[23:54:33] step 3420/4768 loss 3.0788 lr 6.00e-04 gnorm 0.26 50,372 tok/s peakVRAM 13.7GB
[23:58:01] step 3440/4768 loss 3.0709 lr 6.00e-04 gnorm 0.23 50,365 tok/s peakVRAM 13.7GB
[00:01:29] step 3460/4768 loss 3.0802 lr 6.00e-04 gnorm 0.26 50,363 tok/s peakVRAM 13.7GB
[00:04:58] step 3480/4768 loss 3.0792 lr 6.00e-04 gnorm 0.24 50,364 tok/s peakVRAM 13.7GB
[00:08:26] step 3500/4768 loss 3.0655 lr 6.00e-04 gnorm 0.25 50,372 tok/s peakVRAM 13.7GB
[00:11:54] step 3520/4768 loss 3.0694 lr 6.00e-04 gnorm 0.24 50,369 tok/s peakVRAM 13.7GB
[00:15:22] step 3540/4768 loss 3.0713 lr 6.00e-04 gnorm 0.27 50,368 tok/s peakVRAM 13.7GB
[00:18:50] step 3560/4768 loss 3.0752 lr 6.00e-04 gnorm 0.26 50,361 tok/s peakVRAM 13.7GB
[00:22:18] step 3580/4768 loss 3.0689 lr 6.00e-04 gnorm 0.25 50,364 tok/s peakVRAM 13.7GB
[00:25:47] step 3600/4768 loss 3.0714 lr 6.00e-04 gnorm 0.30 50,363 tok/s peakVRAM 13.7GB
[00:29:15] step 3620/4768 loss 3.0629 lr 6.00e-04 gnorm 0.26 50,363 tok/s peakVRAM 13.7GB
[00:32:43] step 3640/4768 loss 3.0709 lr 6.00e-04 gnorm 0.28 50,359 tok/s peakVRAM 13.7GB
[00:36:11] step 3660/4768 loss 3.0597 lr 6.00e-04 gnorm 0.28 50,361 tok/s peakVRAM 13.7GB
[00:39:40] step 3680/4768 loss 3.0664 lr 6.00e-04 gnorm 0.33 50,361 tok/s peakVRAM 13.7GB
[00:43:08] step 3700/4768 loss 3.0636 lr 6.00e-04 gnorm 0.29 50,359 tok/s peakVRAM 13.7GB
[00:46:36] step 3720/4768 loss 3.0570 lr 6.00e-04 gnorm 0.28 50,362 tok/s peakVRAM 13.7GB
[00:50:04] step 3740/4768 loss 3.0528 lr 6.00e-04 gnorm 0.28 50,366 tok/s peakVRAM 13.7GB
[00:53:32] step 3760/4768 loss 3.0513 lr 6.00e-04 gnorm 0.27 50,365 tok/s peakVRAM 13.7GB
[00:57:01] step 3780/4768 loss 3.0523 lr 6.00e-04 gnorm 0.28 50,361 tok/s peakVRAM 13.7GB
[01:00:29] step 3800/4768 loss 3.0538 lr 6.00e-04 gnorm 0.30 50,363 tok/s peakVRAM 13.7GB
[01:03:57] step 3820/4768 loss 3.0467 lr 5.61e-04 gnorm 0.23 50,364 tok/s peakVRAM 13.7GB
[01:07:25] step 3840/4768 loss 3.0445 lr 5.13e-04 gnorm 0.27 50,369 tok/s peakVRAM 13.7GB
[01:10:53] step 3860/4768 loss 3.0375 lr 4.83e-04 gnorm 0.23 50,368 tok/s peakVRAM 13.7GB
[01:14:22] step 3880/4768 loss 3.0346 lr 4.59e-04 gnorm 0.20 50,368 tok/s peakVRAM 13.7GB
[01:17:50] step 3900/4768 loss 3.0270 lr 4.39e-04 gnorm 0.19 50,370 tok/s peakVRAM 13.7GB
[01:21:18] step 3920/4768 loss 3.0222 lr 4.21e-04 gnorm 0.23 50,370 tok/s peakVRAM 13.7GB
[01:24:46] step 3940/4768 loss 3.0140 lr 4.05e-04 gnorm 0.20 50,372 tok/s peakVRAM 13.7GB
[01:28:14] step 3960/4768 loss 3.0142 lr 3.89e-04 gnorm 0.15 50,372 tok/s peakVRAM 13.7GB
[01:31:42] step 3980/4768 loss 3.0063 lr 3.75e-04 gnorm 0.15 50,378 tok/s peakVRAM 13.7GB
[01:35:10] step 4000/4768 loss 3.0050 lr 3.62e-04 gnorm 0.16 50,379 tok/s peakVRAM 13.7GB
[01:38:39] step 4020/4768 loss 3.0023 lr 3.50e-04 gnorm 0.16 50,377 tok/s peakVRAM 13.7GB
[01:42:07] step 4040/4768 loss 2.9931 lr 3.38e-04 gnorm 0.14 50,372 tok/s peakVRAM 13.7GB
[01:45:35] step 4060/4768 loss 2.9928 lr 3.26e-04 gnorm 0.15 50,371 tok/s peakVRAM 13.7GB
[01:49:03] step 4080/4768 loss 2.9841 lr 3.15e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
[01:52:31] step 4100/4768 loss 2.9894 lr 3.05e-04 gnorm 0.15 50,373 tok/s peakVRAM 13.7GB
[01:55:59] step 4120/4768 loss 2.9777 lr 2.95e-04 gnorm 0.14 50,378 tok/s peakVRAM 13.7GB
[01:59:28] step 4140/4768 loss 2.9702 lr 2.85e-04 gnorm 0.14 50,375 tok/s peakVRAM 13.7GB
[02:02:56] step 4160/4768 loss 2.9881 lr 2.75e-04 gnorm 0.13 50,375 tok/s peakVRAM 13.7GB
[02:06:24] step 4180/4768 loss 2.9810 lr 2.66e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
[02:09:52] step 4200/4768 loss 2.9741 lr 2.57e-04 gnorm 0.13 50,372 tok/s peakVRAM 13.7GB
[02:13:20] step 4220/4768 loss 2.9675 lr 2.48e-04 gnorm 0.13 50,372 tok/s peakVRAM 13.7GB
[02:16:48] step 4240/4768 loss 2.9741 lr 2.40e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[02:20:17] step 4260/4768 loss 2.9662 lr 2.31e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
[02:23:45] step 4280/4768 loss 2.9653 lr 2.23e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[02:27:13] step 4300/4768 loss 2.9630 lr 2.15e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[02:30:41] step 4320/4768 loss 2.9580 lr 2.07e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
[02:34:09] step 4340/4768 loss 2.9571 lr 1.99e-04 gnorm 0.13 50,362 tok/s peakVRAM 13.7GB
[02:37:38] step 4360/4768 loss 2.9512 lr 1.92e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
[02:41:06] step 4380/4768 loss 2.9582 lr 1.84e-04 gnorm 0.11 50,367 tok/s peakVRAM 13.7GB
[02:44:34] step 4400/4768 loss 2.9549 lr 1.77e-04 gnorm 0.11 50,368 tok/s peakVRAM 13.7GB
[02:48:02] step 4420/4768 loss 2.9487 lr 1.70e-04 gnorm 0.12 50,481 tok/s peakVRAM 13.7GB
[02:51:29] step 4440/4768 loss 2.9545 lr 1.63e-04 gnorm 0.12 50,542 tok/s peakVRAM 13.7GB
[02:54:57] step 4460/4768 loss 2.9502 lr 1.56e-04 gnorm 0.12 50,557 tok/s peakVRAM 13.7GB
[02:58:24] step 4480/4768 loss 2.9451 lr 1.49e-04 gnorm 0.11 50,559 tok/s peakVRAM 13.7GB
[03:01:51] step 4500/4768 loss 2.9426 lr 1.42e-04 gnorm 0.11 50,565 tok/s peakVRAM 13.7GB
[03:05:19] step 4520/4768 loss 2.9441 lr 1.36e-04 gnorm 0.12 50,564 tok/s peakVRAM 13.7GB
[03:08:46] step 4540/4768 loss 2.9361 lr 1.29e-04 gnorm 0.12 50,562 tok/s peakVRAM 13.7GB
[03:12:14] step 4560/4768 loss 2.9384 lr 1.23e-04 gnorm 0.12 50,565 tok/s peakVRAM 13.7GB
[03:15:41] step 4580/4768 loss 2.9421 lr 1.16e-04 gnorm 0.11 50,562 tok/s peakVRAM 13.7GB
[03:19:08] step 4600/4768 loss 2.9408 lr 1.10e-04 gnorm 0.11 50,556 tok/s peakVRAM 13.7GB
[03:22:36] step 4620/4768 loss 2.9273 lr 1.04e-04 gnorm 0.10 50,487 tok/s peakVRAM 13.7GB
[03:26:03] step 4640/4768 loss 2.9367 lr 9.78e-05 gnorm 0.10 50,539 tok/s peakVRAM 13.7GB
[03:29:31] step 4660/4768 loss 2.9295 lr 9.18e-05 gnorm 0.11 50,554 tok/s peakVRAM 13.7GB
[03:32:58] step 4680/4768 loss 2.9272 lr 8.58e-05 gnorm 0.10 50,523 tok/s peakVRAM 13.7GB
[03:36:26] step 4700/4768 loss 2.9233 lr 7.99e-05 gnorm 0.10 50,502 tok/s peakVRAM 13.7GB
[03:39:54] step 4720/4768 loss 2.9241 lr 7.41e-05 gnorm 0.10 50,527 tok/s peakVRAM 13.7GB
[03:43:21] step 4740/4768 loss 2.9223 lr 6.83e-05 gnorm 0.10 50,522 tok/s peakVRAM 13.7GB
[03:46:49] step 4760/4768 loss 2.9311 lr 6.26e-05 gnorm 0.10 50,546 tok/s peakVRAM 13.7GB
[03:48:13] ckpt @ 4768
[03:48:13] DONE