Hugging Face's logo

SlayerLab
/
dowody400m-checkpoints

PyTorch
polish
checkpoints
training-history
Model card Files
xet
 Community
dowody400m-checkpoints / R1 /run /train.log
Maggio33's picture
Maggio33
Add run records (args, metrics, logs, val, environment) for all 400M runs, fala-1 finals K22/R1-R3, resume test, code
c28a890 verified
Raw History Blame Contribute Delete
21.9 kB
[13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
[13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
[13:59:58] model GPT-ref: 183.6M param (L16 d896 h14)
[13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
[13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
[13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[13:59:59] torch.compile enabled
[13:59:59] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
[14:04:20] step 20/4768 loss 10.3757 lr 2.40e-05 gnorm 2.72 40,193 tok/s peakVRAM 13.7GB
[14:07:32] step 40/4768 loss 9.5564 lr 4.80e-05 gnorm 1.29 54,573 tok/s peakVRAM 13.7GB
[14:10:45] step 60/4768 loss 8.6944 lr 7.20e-05 gnorm 0.71 54,577 tok/s peakVRAM 13.7GB
[14:13:57] step 80/4768 loss 8.0439 lr 9.60e-05 gnorm 1.18 54,575 tok/s peakVRAM 13.7GB
[14:17:09] step 100/4768 loss 7.5900 lr 1.20e-04 gnorm 0.92 54,564 tok/s peakVRAM 13.7GB
[14:20:21] step 120/4768 loss 7.1203 lr 1.44e-04 gnorm 0.82 54,555 tok/s peakVRAM 13.7GB
[14:23:33] step 140/4768 loss 6.7051 lr 1.68e-04 gnorm 0.74 54,546 tok/s peakVRAM 13.7GB
[14:26:46] step 160/4768 loss 6.3498 lr 1.92e-04 gnorm 0.75 54,544 tok/s peakVRAM 13.7GB
[14:29:58] step 180/4768 loss 6.0849 lr 2.16e-04 gnorm 0.78 54,537 tok/s peakVRAM 13.7GB
[14:33:10] step 200/4768 loss 5.8639 lr 2.40e-04 gnorm 0.87 54,535 tok/s peakVRAM 13.7GB
[14:36:22] step 220/4768 loss 5.6756 lr 2.64e-04 gnorm 0.87 54,530 tok/s peakVRAM 13.7GB
[14:39:35] step 240/4768 loss 5.4997 lr 2.88e-04 gnorm 0.77 54,530 tok/s peakVRAM 13.7GB
[14:42:47] step 260/4768 loss 5.3285 lr 3.12e-04 gnorm 0.84 54,528 tok/s peakVRAM 13.7GB
[14:45:59] step 280/4768 loss 5.1732 lr 3.36e-04 gnorm 0.91 54,528 tok/s peakVRAM 13.7GB
[14:49:12] step 300/4768 loss 5.0283 lr 3.60e-04 gnorm 0.78 54,530 tok/s peakVRAM 13.7GB
[14:52:24] step 320/4768 loss 4.8935 lr 3.84e-04 gnorm 0.63 54,530 tok/s peakVRAM 13.7GB
[14:55:36] step 340/4768 loss 4.7505 lr 4.08e-04 gnorm 0.59 54,527 tok/s peakVRAM 13.7GB
[14:58:49] step 360/4768 loss 4.6245 lr 4.32e-04 gnorm 0.72 54,527 tok/s peakVRAM 13.7GB
[15:02:01] step 380/4768 loss 4.5067 lr 4.56e-04 gnorm 0.46 54,527 tok/s peakVRAM 13.7GB
[15:05:13] step 400/4768 loss 4.4020 lr 4.80e-04 gnorm 0.52 54,528 tok/s peakVRAM 13.7GB
[15:08:25] step 420/4768 loss 4.3259 lr 5.04e-04 gnorm 0.47 54,524 tok/s peakVRAM 13.7GB
[15:11:38] step 440/4768 loss 4.2479 lr 5.28e-04 gnorm 0.40 54,526 tok/s peakVRAM 13.7GB
[15:14:50] step 460/4768 loss 4.1796 lr 5.52e-04 gnorm 0.41 54,527 tok/s peakVRAM 13.7GB
[15:18:02] step 480/4768 loss 4.1299 lr 5.76e-04 gnorm 0.42 54,525 tok/s peakVRAM 13.7GB
[15:21:15] step 500/4768 loss 4.0777 lr 6.00e-04 gnorm 0.36 54,523 tok/s peakVRAM 13.7GB
[15:24:27] step 520/4768 loss 4.0391 lr 6.00e-04 gnorm 0.31 54,520 tok/s peakVRAM 13.7GB
[15:27:39] step 540/4768 loss 3.9885 lr 6.00e-04 gnorm 0.26 54,520 tok/s peakVRAM 13.7GB
[15:30:52] step 560/4768 loss 3.9369 lr 6.00e-04 gnorm 0.20 54,521 tok/s peakVRAM 13.7GB
[15:34:04] step 580/4768 loss 3.8963 lr 6.00e-04 gnorm 0.22 54,519 tok/s peakVRAM 13.7GB
[15:37:16] step 600/4768 loss 3.8546 lr 6.00e-04 gnorm 0.23 54,522 tok/s peakVRAM 13.7GB
[15:40:29] step 620/4768 loss 3.8328 lr 6.00e-04 gnorm 0.21 54,520 tok/s peakVRAM 13.7GB
[15:43:41] step 640/4768 loss 3.8063 lr 6.00e-04 gnorm 0.19 54,519 tok/s peakVRAM 13.7GB
[15:46:53] step 660/4768 loss 3.7745 lr 6.00e-04 gnorm 0.20 54,520 tok/s peakVRAM 13.7GB
[15:50:06] step 680/4768 loss 3.7530 lr 6.00e-04 gnorm 0.17 54,519 tok/s peakVRAM 13.7GB
[15:53:18] step 700/4768 loss 3.7221 lr 6.00e-04 gnorm 0.19 54,519 tok/s peakVRAM 13.7GB
[15:56:30] step 720/4768 loss 3.6912 lr 6.00e-04 gnorm 0.16 54,517 tok/s peakVRAM 13.7GB
[15:59:43] step 740/4768 loss 3.6873 lr 6.00e-04 gnorm 0.18 54,513 tok/s peakVRAM 13.7GB
[16:02:55] step 760/4768 loss 3.6627 lr 6.00e-04 gnorm 0.17 54,513 tok/s peakVRAM 13.7GB
[16:06:07] step 780/4768 loss 3.6369 lr 6.00e-04 gnorm 0.15 54,512 tok/s peakVRAM 13.7GB
[16:09:20] step 800/4768 loss 3.6232 lr 6.00e-04 gnorm 0.16 54,508 tok/s peakVRAM 13.7GB
[16:12:32] step 820/4768 loss 3.6076 lr 6.00e-04 gnorm 0.15 54,512 tok/s peakVRAM 13.7GB
[16:15:45] step 840/4768 loss 3.5929 lr 6.00e-04 gnorm 0.15 54,510 tok/s peakVRAM 13.7GB
[16:18:57] step 860/4768 loss 3.5715 lr 6.00e-04 gnorm 0.13 54,510 tok/s peakVRAM 13.7GB
[16:22:09] step 880/4768 loss 3.5661 lr 6.00e-04 gnorm 0.15 54,508 tok/s peakVRAM 13.7GB
[16:25:22] step 900/4768 loss 3.5479 lr 6.00e-04 gnorm 0.14 54,510 tok/s peakVRAM 13.7GB
[16:28:34] step 920/4768 loss 3.5404 lr 6.00e-04 gnorm 0.13 54,511 tok/s peakVRAM 13.7GB
[16:31:46] step 940/4768 loss 3.5349 lr 6.00e-04 gnorm 0.13 54,511 tok/s peakVRAM 13.7GB
[16:34:59] step 960/4768 loss 3.5101 lr 6.00e-04 gnorm 0.14 54,511 tok/s peakVRAM 13.7GB
[16:38:11] step 980/4768 loss 3.5061 lr 6.00e-04 gnorm 0.14 54,509 tok/s peakVRAM 13.7GB
[16:41:23] step 1000/4768 loss 3.4964 lr 6.00e-04 gnorm 0.13 54,509 tok/s peakVRAM 13.7GB
[16:44:36] step 1020/4768 loss 3.4905 lr 6.00e-04 gnorm 0.12 54,506 tok/s peakVRAM 13.7GB
[16:47:48] step 1040/4768 loss 3.4633 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
[16:51:01] step 1060/4768 loss 3.4752 lr 6.00e-04 gnorm 0.14 54,506 tok/s peakVRAM 13.7GB
[16:54:13] step 1080/4768 loss 3.4640 lr 6.00e-04 gnorm 0.13 54,502 tok/s peakVRAM 13.7GB
[16:57:25] step 1100/4768 loss 3.4485 lr 6.00e-04 gnorm 0.14 54,502 tok/s peakVRAM 13.7GB
[17:00:38] step 1120/4768 loss 3.4306 lr 6.00e-04 gnorm 0.11 54,497 tok/s peakVRAM 13.7GB
[17:03:50] step 1140/4768 loss 3.4239 lr 6.00e-04 gnorm 0.13 54,497 tok/s peakVRAM 13.7GB
[17:07:03] step 1160/4768 loss 3.4182 lr 6.00e-04 gnorm 0.12 54,502 tok/s peakVRAM 13.7GB
[17:10:15] step 1180/4768 loss 3.4205 lr 6.00e-04 gnorm 0.13 54,503 tok/s peakVRAM 13.7GB
[17:13:27] step 1200/4768 loss 3.4020 lr 6.00e-04 gnorm 0.12 54,501 tok/s peakVRAM 13.7GB
[17:16:40] step 1220/4768 loss 3.3999 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
[17:19:52] step 1240/4768 loss 3.3910 lr 6.00e-04 gnorm 0.13 54,502 tok/s peakVRAM 13.7GB
[17:23:05] step 1260/4768 loss 3.3769 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
[17:26:17] step 1280/4768 loss 3.3803 lr 6.00e-04 gnorm 0.11 54,504 tok/s peakVRAM 13.7GB
[17:29:29] step 1300/4768 loss 3.3732 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
[17:32:42] step 1320/4768 loss 3.3656 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
[17:35:54] step 1340/4768 loss 3.3657 lr 6.00e-04 gnorm 0.13 54,499 tok/s peakVRAM 13.7GB
[17:39:06] step 1360/4768 loss 3.3566 lr 6.00e-04 gnorm 0.11 54,498 tok/s peakVRAM 13.7GB
[17:42:19] step 1380/4768 loss 3.3479 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
[17:45:31] step 1400/4768 loss 3.3375 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
[17:48:44] step 1420/4768 loss 3.3260 lr 6.00e-04 gnorm 0.14 54,506 tok/s peakVRAM 13.7GB
[17:51:56] step 1440/4768 loss 3.3326 lr 6.00e-04 gnorm 0.10 54,506 tok/s peakVRAM 13.7GB
[17:55:08] step 1460/4768 loss 3.3299 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
[17:58:21] step 1480/4768 loss 3.3338 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
[18:01:33] step 1500/4768 loss 3.3074 lr 6.00e-04 gnorm 0.11 54,507 tok/s peakVRAM 13.7GB
[18:04:46] step 1520/4768 loss 3.3141 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
[18:07:58] step 1540/4768 loss 3.2981 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
[18:11:10] step 1560/4768 loss 3.3069 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
[18:14:23] step 1580/4768 loss 3.2950 lr 6.00e-04 gnorm 0.11 54,497 tok/s peakVRAM 13.7GB
[18:17:35] step 1600/4768 loss 3.2844 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
[18:20:48] step 1620/4768 loss 3.2848 lr 6.00e-04 gnorm 0.10 54,498 tok/s peakVRAM 13.7GB
[18:24:00] step 1640/4768 loss 3.2817 lr 6.00e-04 gnorm 0.10 54,501 tok/s peakVRAM 13.7GB
[18:27:12] step 1660/4768 loss 3.2826 lr 6.00e-04 gnorm 0.11 54,496 tok/s peakVRAM 13.7GB
[18:30:25] step 1680/4768 loss 3.2713 lr 6.00e-04 gnorm 0.12 54,497 tok/s peakVRAM 13.7GB
[18:33:37] step 1700/4768 loss 3.2644 lr 6.00e-04 gnorm 0.11 54,496 tok/s peakVRAM 13.7GB
[18:36:50] step 1720/4768 loss 3.2667 lr 6.00e-04 gnorm 0.12 54,494 tok/s peakVRAM 13.7GB
[18:40:02] step 1740/4768 loss 3.2596 lr 6.00e-04 gnorm 0.12 54,492 tok/s peakVRAM 13.7GB
[18:43:14] step 1760/4768 loss 3.2590 lr 6.00e-04 gnorm 0.10 54,497 tok/s peakVRAM 13.7GB
[18:46:27] step 1780/4768 loss 3.2461 lr 6.00e-04 gnorm 0.11 54,501 tok/s peakVRAM 13.7GB
[18:49:39] step 1800/4768 loss 3.2504 lr 6.00e-04 gnorm 0.11 54,498 tok/s peakVRAM 13.7GB
[18:52:52] step 1820/4768 loss 3.2456 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
[18:56:04] step 1840/4768 loss 3.2390 lr 6.00e-04 gnorm 0.11 54,501 tok/s peakVRAM 13.7GB
[18:59:16] step 1860/4768 loss 3.2379 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
[19:02:29] step 1880/4768 loss 3.2338 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
[19:05:41] step 1900/4768 loss 3.2283 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
[19:08:54] step 1920/4768 loss 3.2283 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
[19:12:06] step 1940/4768 loss 3.2208 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
[19:15:18] step 1960/4768 loss 3.2250 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
[19:18:31] step 1980/4768 loss 3.2196 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
[19:21:43] step 2000/4768 loss 3.2127 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
[19:24:56] step 2020/4768 loss 3.2123 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
[19:28:08] step 2040/4768 loss 3.2057 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
[19:31:20] step 2060/4768 loss 3.2061 lr 6.00e-04 gnorm 0.10 54,508 tok/s peakVRAM 13.7GB
[19:34:33] step 2080/4768 loss 3.1947 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
[19:37:45] step 2100/4768 loss 3.2031 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
[19:40:57] step 2120/4768 loss 3.1923 lr 6.00e-04 gnorm 0.11 54,510 tok/s peakVRAM 13.7GB
[19:44:10] step 2140/4768 loss 3.2055 lr 6.00e-04 gnorm 0.11 54,509 tok/s peakVRAM 13.7GB
[19:47:22] step 2160/4768 loss 3.1841 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
[19:50:35] step 2180/4768 loss 3.1859 lr 6.00e-04 gnorm 0.11 54,503 tok/s peakVRAM 13.7GB
[19:53:47] step 2200/4768 loss 3.1875 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
[19:56:59] step 2220/4768 loss 3.1795 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
[20:00:12] step 2240/4768 loss 3.1762 lr 6.00e-04 gnorm 0.10 54,506 tok/s peakVRAM 13.7GB
[20:03:24] step 2260/4768 loss 3.1724 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
[20:06:36] step 2280/4768 loss 3.1688 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
[20:09:49] step 2300/4768 loss 3.1722 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
[20:13:01] step 2320/4768 loss 3.1690 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
[20:16:14] step 2340/4768 loss 3.1635 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
[20:19:26] step 2360/4768 loss 3.1542 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
[20:22:38] step 2380/4768 loss 3.1533 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
[20:25:51] step 2400/4768 loss 3.1534 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
[20:29:03] step 2420/4768 loss 3.1563 lr 6.00e-04 gnorm 0.09 54,498 tok/s peakVRAM 13.7GB
[20:32:16] step 2440/4768 loss 3.1502 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
[20:35:28] step 2460/4768 loss 3.1489 lr 6.00e-04 gnorm 0.10 54,499 tok/s peakVRAM 13.7GB
[20:38:40] step 2480/4768 loss 3.1461 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
[20:41:53] step 2500/4768 loss 3.1406 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
[20:45:05] step 2520/4768 loss 3.1425 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
[20:48:18] step 2540/4768 loss 3.1422 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
[20:51:30] step 2560/4768 loss 3.1397 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
[20:54:42] step 2580/4768 loss 3.1356 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
[20:57:55] step 2600/4768 loss 3.1255 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
[21:01:07] step 2620/4768 loss 3.1343 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
[21:04:20] step 2640/4768 loss 3.1348 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
[21:07:32] step 2660/4768 loss 3.1334 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
[21:10:44] step 2680/4768 loss 3.1281 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
[21:13:57] step 2700/4768 loss 3.1312 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
[21:17:09] step 2720/4768 loss 3.1138 lr 6.00e-04 gnorm 0.09 54,499 tok/s peakVRAM 13.7GB
[21:20:22] step 2740/4768 loss 3.1155 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
[21:23:34] step 2760/4768 loss 3.1185 lr 6.00e-04 gnorm 0.09 54,500 tok/s peakVRAM 13.7GB
[21:26:46] step 2780/4768 loss 3.1098 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
[21:29:59] step 2800/4768 loss 3.1045 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
[21:33:11] step 2820/4768 loss 3.1017 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
[21:36:23] step 2840/4768 loss 3.1099 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
[21:39:36] step 2860/4768 loss 3.1015 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
[21:42:48] step 2880/4768 loss 3.1037 lr 6.00e-04 gnorm 0.10 54,501 tok/s peakVRAM 13.7GB
[21:46:01] step 2900/4768 loss 3.1048 lr 6.00e-04 gnorm 0.09 54,497 tok/s peakVRAM 13.7GB
[21:49:13] step 2920/4768 loss 3.1120 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
[21:52:25] step 2940/4768 loss 3.1023 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
[21:55:38] step 2960/4768 loss 3.1040 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
[21:58:50] step 2980/4768 loss 3.0913 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
[22:02:03] step 3000/4768 loss 3.0920 lr 6.00e-04 gnorm 0.09 54,502 tok/s peakVRAM 13.7GB
[22:05:15] step 3020/4768 loss 3.0874 lr 6.00e-04 gnorm 0.11 54,504 tok/s peakVRAM 13.7GB
[22:08:27] step 3040/4768 loss 3.0845 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
[22:11:40] step 3060/4768 loss 3.0876 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
[22:14:52] step 3080/4768 loss 3.0810 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
[22:18:05] step 3100/4768 loss 3.0741 lr 6.00e-04 gnorm 0.11 54,507 tok/s peakVRAM 13.7GB
[22:21:17] step 3120/4768 loss 3.0905 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
[22:24:29] step 3140/4768 loss 3.0764 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
[22:27:42] step 3160/4768 loss 3.0722 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
[22:30:54] step 3180/4768 loss 3.0783 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
[22:34:06] step 3200/4768 loss 3.0794 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
[22:37:19] step 3220/4768 loss 3.0889 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
[22:40:31] step 3240/4768 loss 3.0708 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
[22:43:44] step 3260/4768 loss 3.0677 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
[22:46:56] step 3280/4768 loss 3.0627 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
[22:50:08] step 3300/4768 loss 3.0740 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
[22:53:21] step 3320/4768 loss 3.0677 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
[22:56:33] step 3340/4768 loss 3.0681 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
[22:59:46] step 3360/4768 loss 3.0593 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
[23:02:58] step 3380/4768 loss 3.0687 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
[23:06:10] step 3400/4768 loss 3.0538 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
[23:09:23] step 3420/4768 loss 3.0617 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
[23:12:35] step 3440/4768 loss 3.0533 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
[23:15:47] step 3460/4768 loss 3.0629 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
[23:19:00] step 3480/4768 loss 3.0618 lr 6.00e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
[23:22:12] step 3500/4768 loss 3.0479 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
[23:25:25] step 3520/4768 loss 3.0510 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
[23:28:37] step 3540/4768 loss 3.0538 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
[23:31:49] step 3560/4768 loss 3.0581 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
[23:35:02] step 3580/4768 loss 3.0513 lr 6.00e-04 gnorm 0.08 54,502 tok/s peakVRAM 13.7GB
[23:38:14] step 3600/4768 loss 3.0537 lr 6.00e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
[23:41:27] step 3620/4768 loss 3.0451 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
[23:44:39] step 3640/4768 loss 3.0531 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
[23:47:51] step 3660/4768 loss 3.0418 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
[23:51:04] step 3680/4768 loss 3.0485 lr 6.00e-04 gnorm 0.09 54,510 tok/s peakVRAM 13.7GB
[23:54:16] step 3700/4768 loss 3.0457 lr 6.00e-04 gnorm 0.09 54,511 tok/s peakVRAM 13.7GB
[23:57:28] step 3720/4768 loss 3.0391 lr 6.00e-04 gnorm 0.09 54,509 tok/s peakVRAM 13.7GB
[00:00:41] step 3740/4768 loss 3.0351 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
[00:03:53] step 3760/4768 loss 3.0340 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
[00:07:05] step 3780/4768 loss 3.0346 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
[00:10:18] step 3800/4768 loss 3.0353 lr 6.00e-04 gnorm 0.08 54,509 tok/s peakVRAM 13.7GB
[00:13:30] step 3820/4768 loss 3.0289 lr 5.61e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
[00:16:43] step 3840/4768 loss 3.0271 lr 5.13e-04 gnorm 0.08 54,505 tok/s peakVRAM 13.7GB
[00:19:55] step 3860/4768 loss 3.0204 lr 4.83e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
[00:23:07] step 3880/4768 loss 3.0173 lr 4.59e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[00:26:20] step 3900/4768 loss 3.0108 lr 4.39e-04 gnorm 0.08 54,502 tok/s peakVRAM 13.7GB
[00:29:32] step 3920/4768 loss 3.0052 lr 4.21e-04 gnorm 0.08 54,504 tok/s peakVRAM 13.7GB
[00:32:45] step 3940/4768 loss 2.9974 lr 4.05e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[00:35:57] step 3960/4768 loss 2.9981 lr 3.89e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
[00:39:09] step 3980/4768 loss 2.9899 lr 3.75e-04 gnorm 0.08 54,503 tok/s peakVRAM 13.7GB
[00:42:22] step 4000/4768 loss 2.9894 lr 3.62e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[00:45:34] step 4020/4768 loss 2.9862 lr 3.50e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
[00:48:46] step 4040/4768 loss 2.9767 lr 3.38e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[00:51:59] step 4060/4768 loss 2.9768 lr 3.26e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[00:55:11] step 4080/4768 loss 2.9678 lr 3.15e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
[00:58:24] step 4100/4768 loss 2.9733 lr 3.05e-04 gnorm 0.07 54,503 tok/s peakVRAM 13.7GB
[01:01:36] step 4120/4768 loss 2.9621 lr 2.95e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[01:04:48] step 4140/4768 loss 2.9543 lr 2.85e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
[01:08:01] step 4160/4768 loss 2.9723 lr 2.75e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
[01:11:13] step 4180/4768 loss 2.9649 lr 2.66e-04 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
[01:14:25] step 4200/4768 loss 2.9585 lr 2.57e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
[01:17:38] step 4220/4768 loss 2.9518 lr 2.48e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
[01:20:50] step 4240/4768 loss 2.9587 lr 2.40e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[01:24:03] step 4260/4768 loss 2.9504 lr 2.31e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
[01:27:15] step 4280/4768 loss 2.9496 lr 2.23e-04 gnorm 0.07 54,512 tok/s peakVRAM 13.7GB
[01:30:27] step 4300/4768 loss 2.9478 lr 2.15e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
[01:33:40] step 4320/4768 loss 2.9426 lr 2.07e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
[01:36:52] step 4340/4768 loss 2.9412 lr 1.99e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[01:40:04] step 4360/4768 loss 2.9350 lr 1.92e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
[01:43:17] step 4380/4768 loss 2.9421 lr 1.84e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
[01:46:29] step 4400/4768 loss 2.9398 lr 1.77e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
[01:49:42] step 4420/4768 loss 2.9329 lr 1.70e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
[01:52:54] step 4440/4768 loss 2.9391 lr 1.63e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[01:56:06] step 4460/4768 loss 2.9348 lr 1.56e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
[01:59:19] step 4480/4768 loss 2.9299 lr 1.49e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[02:02:31] step 4500/4768 loss 2.9271 lr 1.42e-04 gnorm 0.07 54,502 tok/s peakVRAM 13.7GB
[02:05:43] step 4520/4768 loss 2.9291 lr 1.36e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[02:08:56] step 4540/4768 loss 2.9205 lr 1.29e-04 gnorm 0.07 54,503 tok/s peakVRAM 13.7GB
[02:12:08] step 4560/4768 loss 2.9235 lr 1.23e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[02:15:21] step 4580/4768 loss 2.9269 lr 1.16e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
[02:18:33] step 4600/4768 loss 2.9252 lr 1.10e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
[02:21:45] step 4620/4768 loss 2.9120 lr 1.04e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[02:24:58] step 4640/4768 loss 2.9213 lr 9.78e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
[02:28:10] step 4660/4768 loss 2.9140 lr 9.18e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
[02:31:22] step 4680/4768 loss 2.9119 lr 8.58e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
[02:34:35] step 4700/4768 loss 2.9078 lr 7.99e-05 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[02:37:47] step 4720/4768 loss 2.9089 lr 7.41e-05 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
[02:41:00] step 4740/4768 loss 2.9066 lr 6.83e-05 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
[02:44:12] step 4760/4768 loss 2.9157 lr 6.26e-05 gnorm 0.07 54,512 tok/s peakVRAM 13.7GB
[02:45:31] ckpt @ 4768
[02:45:31] DONE