Hugging Face's logo

SlayerLab
/
dowody400m-checkpoints

PyTorch
polish
checkpoints
training-history
Model card Files
xet
 Community
dowody400m-checkpoints
File size: 11,066 Bytes
c28a890
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
[04:48:32] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
[04:48:32] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
[04:48:34] model GPT-ref: 183.6M param (L16 d896 h14)
[04:48:34] optimizer=muon muon_lr=0.04 (mult=66.7x)
[04:48:34] r11 pamiec: act_ckpt=True loss_chunk=1024
[04:48:34] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[04:48:35] torch.compile enabled
[04:48:35] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
[04:52:32] step 20/4768 loss 10.2553 lr 2.40e-05 gnorm 2.32 44,304 tok/s peakVRAM 13.7GB
[04:55:44] step 40/4768 loss 9.2968 lr 4.80e-05 gnorm 1.00 54,450 tok/s peakVRAM 13.7GB
[04:58:57] step 60/4768 loss 8.4951 lr 7.20e-05 gnorm 1.46 54,449 tok/s peakVRAM 13.7GB
[05:02:10] step 80/4768 loss 7.8249 lr 9.60e-05 gnorm 0.69 54,439 tok/s peakVRAM 13.7GB
[05:05:22] step 100/4768 loss 7.3027 lr 1.20e-04 gnorm 0.65 54,421 tok/s peakVRAM 13.7GB
[05:08:35] step 120/4768 loss 6.7967 lr 1.44e-04 gnorm 0.71 54,420 tok/s peakVRAM 13.7GB
[05:11:48] step 140/4768 loss 6.4050 lr 1.68e-04 gnorm 0.49 54,415 tok/s peakVRAM 13.7GB
[05:15:00] step 160/4768 loss 6.1028 lr 1.92e-04 gnorm 0.54 54,413 tok/s peakVRAM 13.7GB
[05:18:13] step 180/4768 loss 5.8786 lr 2.16e-04 gnorm 0.51 54,414 tok/s peakVRAM 13.7GB
[05:21:26] step 200/4768 loss 5.6752 lr 2.40e-04 gnorm 0.56 54,412 tok/s peakVRAM 13.7GB
[05:24:38] step 220/4768 loss 5.4946 lr 2.64e-04 gnorm 0.62 54,411 tok/s peakVRAM 13.7GB
[05:27:51] step 240/4768 loss 5.3280 lr 2.88e-04 gnorm 0.57 54,408 tok/s peakVRAM 13.7GB
[05:31:04] step 260/4768 loss 5.1680 lr 3.12e-04 gnorm 0.43 54,408 tok/s peakVRAM 13.7GB
[05:34:17] step 280/4768 loss 5.0263 lr 3.36e-04 gnorm 0.37 54,403 tok/s peakVRAM 13.7GB
[05:37:29] step 300/4768 loss 4.8846 lr 3.60e-04 gnorm 0.45 54,404 tok/s peakVRAM 13.7GB
[05:40:42] step 320/4768 loss 4.7472 lr 3.84e-04 gnorm 0.32 54,402 tok/s peakVRAM 13.7GB
[05:43:55] step 340/4768 loss 4.6110 lr 4.08e-04 gnorm 0.33 54,399 tok/s peakVRAM 13.7GB
[05:47:08] step 360/4768 loss 4.5043 lr 4.32e-04 gnorm 0.26 54,397 tok/s peakVRAM 13.7GB
[05:50:20] step 380/4768 loss 4.4053 lr 4.56e-04 gnorm 0.22 54,390 tok/s peakVRAM 13.7GB
[05:53:33] step 400/4768 loss 4.3194 lr 4.80e-04 gnorm 0.25 54,389 tok/s peakVRAM 13.7GB
[05:56:46] step 420/4768 loss 4.2544 lr 5.04e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
[05:59:59] step 440/4768 loss 4.1844 lr 5.28e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
[06:03:12] step 460/4768 loss 4.1207 lr 5.52e-04 gnorm 0.18 54,390 tok/s peakVRAM 13.7GB
[06:06:24] step 480/4768 loss 4.0799 lr 5.76e-04 gnorm 0.22 54,389 tok/s peakVRAM 13.7GB
[06:09:37] step 500/4768 loss 4.0291 lr 6.00e-04 gnorm 0.18 54,386 tok/s peakVRAM 13.7GB
[06:09:39] ckpt @ 500
[06:12:52] step 520/4768 loss 3.9946 lr 6.00e-04 gnorm 0.16 53,918 tok/s peakVRAM 13.7GB
[06:16:04] step 540/4768 loss 3.9497 lr 6.00e-04 gnorm 0.16 54,388 tok/s peakVRAM 13.7GB
[06:19:17] step 560/4768 loss 3.8994 lr 6.00e-04 gnorm 0.20 54,386 tok/s peakVRAM 13.7GB
[06:22:30] step 580/4768 loss 3.8611 lr 6.00e-04 gnorm 0.15 54,384 tok/s peakVRAM 13.7GB
[06:25:43] step 600/4768 loss 3.8206 lr 6.00e-04 gnorm 0.15 54,386 tok/s peakVRAM 13.7GB
[06:28:56] step 620/4768 loss 3.7999 lr 6.00e-04 gnorm 0.14 54,384 tok/s peakVRAM 13.7GB
[06:32:09] step 640/4768 loss 3.7752 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
[06:35:21] step 660/4768 loss 3.7472 lr 6.00e-04 gnorm 0.11 54,379 tok/s peakVRAM 13.7GB
[06:38:34] step 680/4768 loss 3.7249 lr 6.00e-04 gnorm 0.13 54,376 tok/s peakVRAM 13.7GB
[06:41:47] step 700/4768 loss 3.6959 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
[06:45:00] step 720/4768 loss 3.6670 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
[06:48:13] step 740/4768 loss 3.6638 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
[06:51:25] step 760/4768 loss 3.6386 lr 6.00e-04 gnorm 0.12 54,383 tok/s peakVRAM 13.7GB
[06:54:38] step 780/4768 loss 3.6130 lr 6.00e-04 gnorm 0.10 54,386 tok/s peakVRAM 13.7GB
[06:57:51] step 800/4768 loss 3.6005 lr 6.00e-04 gnorm 0.12 54,385 tok/s peakVRAM 13.7GB
[07:01:04] step 820/4768 loss 3.5843 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
[07:04:17] step 840/4768 loss 3.5713 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
[07:07:29] step 860/4768 loss 3.5506 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
[07:10:42] step 880/4768 loss 3.5451 lr 6.00e-04 gnorm 0.11 54,385 tok/s peakVRAM 13.7GB
[07:13:55] step 900/4768 loss 3.5280 lr 6.00e-04 gnorm 0.11 54,387 tok/s peakVRAM 13.7GB
[07:17:08] step 920/4768 loss 3.5207 lr 6.00e-04 gnorm 0.10 54,382 tok/s peakVRAM 13.7GB
[07:20:21] step 940/4768 loss 3.5164 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[07:23:34] step 960/4768 loss 3.4913 lr 6.00e-04 gnorm 0.08 54,381 tok/s peakVRAM 13.7GB
[07:26:46] step 980/4768 loss 3.4890 lr 6.00e-04 gnorm 0.08 54,380 tok/s peakVRAM 13.7GB
[07:29:59] step 1000/4768 loss 3.4788 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
[07:30:02] ckpt @ 1000
[07:33:14] step 1020/4768 loss 3.4739 lr 6.00e-04 gnorm 0.12 53,735 tok/s peakVRAM 13.7GB
[07:36:27] step 1040/4768 loss 3.4472 lr 6.00e-04 gnorm 0.09 54,384 tok/s peakVRAM 13.7GB
[07:39:40] step 1060/4768 loss 3.4589 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[07:42:53] step 1080/4768 loss 3.4485 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
[07:46:06] step 1100/4768 loss 3.4334 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
[07:49:18] step 1120/4768 loss 3.4154 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
[07:52:31] step 1140/4768 loss 3.4083 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[07:55:44] step 1160/4768 loss 3.4036 lr 6.00e-04 gnorm 0.09 54,383 tok/s peakVRAM 13.7GB
[07:58:57] step 1180/4768 loss 3.4058 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
[08:02:10] step 1200/4768 loss 3.3883 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
[08:05:23] step 1220/4768 loss 3.3856 lr 6.00e-04 gnorm 0.10 54,376 tok/s peakVRAM 13.7GB
[08:08:35] step 1240/4768 loss 3.3775 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
[08:11:48] step 1260/4768 loss 3.3644 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
[08:15:01] step 1280/4768 loss 3.3676 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[08:18:14] step 1300/4768 loss 3.3609 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[08:21:27] step 1320/4768 loss 3.3531 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
[08:24:39] step 1340/4768 loss 3.3537 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[08:27:52] step 1360/4768 loss 3.3450 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
[08:31:05] step 1380/4768 loss 3.3367 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
[08:34:18] step 1400/4768 loss 3.3268 lr 6.00e-04 gnorm 0.09 54,382 tok/s peakVRAM 13.7GB
[08:37:31] step 1420/4768 loss 3.3143 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
[08:40:44] step 1440/4768 loss 3.3218 lr 6.00e-04 gnorm 0.08 54,383 tok/s peakVRAM 13.7GB
[08:43:56] step 1460/4768 loss 3.3195 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
[08:47:09] step 1480/4768 loss 3.3233 lr 6.00e-04 gnorm 0.08 54,379 tok/s peakVRAM 13.7GB
[08:50:22] step 1500/4768 loss 3.2963 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
[08:50:24] ckpt @ 1500
[08:53:37] step 1520/4768 loss 3.3041 lr 6.00e-04 gnorm 0.09 53,713 tok/s peakVRAM 13.7GB
[08:56:50] step 1540/4768 loss 3.2883 lr 6.00e-04 gnorm 0.09 54,374 tok/s peakVRAM 13.7GB
[09:00:03] step 1560/4768 loss 3.2969 lr 6.00e-04 gnorm 0.08 54,377 tok/s peakVRAM 13.7GB
[09:03:16] step 1580/4768 loss 3.2851 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
[09:06:29] step 1600/4768 loss 3.2750 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
[09:09:41] step 1620/4768 loss 3.2753 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
[09:13:59] step 1640/4768 loss 3.2726 lr 6.00e-04 gnorm 0.09 40,752 tok/s peakVRAM 13.7GB
[09:18:26] step 1660/4768 loss 3.2733 lr 6.00e-04 gnorm 0.09 39,290 tok/s peakVRAM 13.7GB
[09:22:46] step 1680/4768 loss 3.2623 lr 6.00e-04 gnorm 0.10 40,313 tok/s peakVRAM 13.7GB
[09:25:59] step 1700/4768 loss 3.2555 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[09:29:11] step 1720/4768 loss 3.2578 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[09:32:24] step 1740/4768 loss 3.2513 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
[09:35:37] step 1760/4768 loss 3.2508 lr 6.00e-04 gnorm 0.10 54,389 tok/s peakVRAM 13.7GB
[09:38:50] step 1780/4768 loss 3.2381 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
[09:42:03] step 1800/4768 loss 3.2418 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
[09:45:18] step 1820/4768 loss 3.2374 lr 6.00e-04 gnorm 0.08 53,646 tok/s peakVRAM 13.7GB
[09:51:02] step 1840/4768 loss 3.2313 lr 6.00e-04 gnorm 0.09 30,467 tok/s peakVRAM 13.7GB
[09:54:58] step 1860/4768 loss 3.2303 lr 6.00e-04 gnorm 0.09 44,403 tok/s peakVRAM 13.7GB
[10:00:30] step 1880/4768 loss 3.2265 lr 6.00e-04 gnorm 0.08 31,623 tok/s peakVRAM 13.7GB
[10:06:04] step 1900/4768 loss 3.2209 lr 6.00e-04 gnorm 0.09 31,365 tok/s peakVRAM 13.7GB
[10:11:58] step 1920/4768 loss 3.2205 lr 6.00e-04 gnorm 0.08 29,665 tok/s peakVRAM 13.7GB
[10:17:51] step 1940/4768 loss 3.2141 lr 6.00e-04 gnorm 0.08 29,662 tok/s peakVRAM 13.7GB
[10:23:36] step 1960/4768 loss 3.2180 lr 6.00e-04 gnorm 0.08 30,457 tok/s peakVRAM 13.7GB
[10:29:02] step 1980/4768 loss 3.2126 lr 6.00e-04 gnorm 0.09 32,076 tok/s peakVRAM 13.7GB
[10:34:30] step 2000/4768 loss 3.2058 lr 6.00e-04 gnorm 0.09 32,047 tok/s peakVRAM 13.7GB
[10:34:32] ckpt @ 2000
[10:40:00] step 2020/4768 loss 3.2055 lr 6.00e-04 gnorm 0.09 31,786 tok/s peakVRAM 13.7GB
[10:45:28] step 2040/4768 loss 3.1983 lr 6.00e-04 gnorm 0.09 31,965 tok/s peakVRAM 13.7GB
[10:50:55] step 2060/4768 loss 3.1986 lr 6.00e-04 gnorm 0.08 31,995 tok/s peakVRAM 13.7GB
[10:56:23] step 2080/4768 loss 3.1876 lr 6.00e-04 gnorm 0.08 32,010 tok/s peakVRAM 13.7GB
[11:01:51] step 2100/4768 loss 3.1961 lr 6.00e-04 gnorm 0.10 31,946 tok/s peakVRAM 13.7GB
[11:07:19] step 2120/4768 loss 3.1859 lr 6.00e-04 gnorm 0.09 31,941 tok/s peakVRAM 13.7GB
[11:12:47] step 2140/4768 loss 3.1989 lr 6.00e-04 gnorm 0.08 32,013 tok/s peakVRAM 13.7GB
[11:18:15] step 2160/4768 loss 3.1776 lr 6.00e-04 gnorm 0.08 31,944 tok/s peakVRAM 13.7GB
[11:23:44] step 2180/4768 loss 3.1793 lr 6.00e-04 gnorm 0.09 31,909 tok/s peakVRAM 13.7GB
[11:28:11] step 2200/4768 loss 3.1803 lr 6.00e-04 gnorm 0.08 39,315 tok/s peakVRAM 13.7GB
[11:33:16] step 2220/4768 loss 3.1734 lr 6.00e-04 gnorm 0.09 34,363 tok/s peakVRAM 13.7GB
[11:39:24] step 2240/4768 loss 3.1700 lr 6.00e-04 gnorm 0.08 28,500 tok/s peakVRAM 13.7GB
[11:44:46] step 2260/4768 loss 3.1662 lr 6.00e-04 gnorm 0.08 32,518 tok/s peakVRAM 13.7GB
[11:50:32] step 2280/4768 loss 3.1625 lr 6.00e-04 gnorm 0.08 30,346 tok/s peakVRAM 13.7GB
[11:56:29] step 2300/4768 loss 3.1659 lr 6.00e-04 gnorm 0.08 29,311 tok/s peakVRAM 13.7GB
[12:01:34] step 2320/4768 loss 3.1626 lr 6.00e-04 gnorm 0.08 34,480 tok/s peakVRAM 13.7GB
[12:07:48] step 2340/4768 loss 3.1578 lr 6.00e-04 gnorm 0.09 28,010 tok/s peakVRAM 13.7GB