Hugging Face's logo

SlayerLab
/
dowody400m-checkpoints

PyTorch
polish
checkpoints
training-history
Model card Files
xet
 Community
dowody400m-checkpoints
File size: 13,429 Bytes
c28a890
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
[04:33:11] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
[04:33:11] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
[04:33:13] model GPT-ref: 183.6M param (L16 d896 h14)
[04:33:14] optimizer=muon muon_lr=0.02 (mult=33.3x)
[04:33:14] r11 pamiec: act_ckpt=True loss_chunk=1024
[04:33:14] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[04:33:15] torch.compile enabled
[04:33:15] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
[04:37:15] step 20/4768 loss 10.3757 lr 2.40e-05 gnorm 2.72 43,562 tok/s peakVRAM 13.2GB
[04:40:27] step 40/4768 loss 9.5564 lr 4.80e-05 gnorm 1.29 54,802 tok/s peakVRAM 13.2GB
[04:43:38] step 60/4768 loss 8.6944 lr 7.20e-05 gnorm 0.71 54,811 tok/s peakVRAM 13.2GB
[04:46:49] step 80/4768 loss 8.0439 lr 9.60e-05 gnorm 1.17 54,800 tok/s peakVRAM 13.2GB
[04:50:01] step 100/4768 loss 7.5902 lr 1.20e-04 gnorm 0.90 54,779 tok/s peakVRAM 13.2GB
[04:53:12] step 120/4768 loss 7.1203 lr 1.44e-04 gnorm 1.05 54,753 tok/s peakVRAM 13.2GB
[04:56:24] step 140/4768 loss 6.7067 lr 1.68e-04 gnorm 0.96 54,738 tok/s peakVRAM 13.2GB
[04:59:35] step 160/4768 loss 6.3500 lr 1.92e-04 gnorm 0.76 54,719 tok/s peakVRAM 13.2GB
[05:02:47] step 180/4768 loss 6.0843 lr 2.16e-04 gnorm 0.72 54,707 tok/s peakVRAM 13.2GB
[05:05:59] step 200/4768 loss 5.8631 lr 2.40e-04 gnorm 0.79 54,698 tok/s peakVRAM 13.2GB
[05:09:10] step 220/4768 loss 5.6758 lr 2.64e-04 gnorm 0.73 54,717 tok/s peakVRAM 13.2GB
[05:12:22] step 240/4768 loss 5.5004 lr 2.88e-04 gnorm 0.74 54,726 tok/s peakVRAM 13.2GB
[05:15:34] step 260/4768 loss 5.3312 lr 3.12e-04 gnorm 0.65 54,722 tok/s peakVRAM 13.2GB
[05:18:45] step 280/4768 loss 5.1733 lr 3.36e-04 gnorm 0.82 54,720 tok/s peakVRAM 13.2GB
[05:21:57] step 300/4768 loss 5.0295 lr 3.60e-04 gnorm 0.71 54,721 tok/s peakVRAM 13.2GB
[05:25:09] step 320/4768 loss 4.8940 lr 3.84e-04 gnorm 0.77 54,728 tok/s peakVRAM 13.2GB
[05:28:20] step 340/4768 loss 4.7543 lr 4.08e-04 gnorm 0.60 54,722 tok/s peakVRAM 13.2GB
[05:31:32] step 360/4768 loss 4.6273 lr 4.32e-04 gnorm 0.51 54,717 tok/s peakVRAM 13.2GB
[05:34:43] step 380/4768 loss 4.5068 lr 4.56e-04 gnorm 0.51 54,718 tok/s peakVRAM 13.2GB
[05:37:55] step 400/4768 loss 4.4057 lr 4.80e-04 gnorm 0.47 54,715 tok/s peakVRAM 13.2GB
[05:41:07] step 420/4768 loss 4.3239 lr 5.04e-04 gnorm 0.49 54,712 tok/s peakVRAM 13.2GB
[05:44:18] step 440/4768 loss 4.2493 lr 5.28e-04 gnorm 0.40 54,707 tok/s peakVRAM 13.2GB
[05:47:30] step 460/4768 loss 4.1796 lr 5.52e-04 gnorm 0.39 54,709 tok/s peakVRAM 13.2GB
[05:50:42] step 480/4768 loss 4.1352 lr 5.76e-04 gnorm 0.36 54,715 tok/s peakVRAM 13.2GB
[05:53:53] step 500/4768 loss 4.0760 lr 6.00e-04 gnorm 0.30 54,704 tok/s peakVRAM 13.2GB
[05:53:55] ckpt @ 500
[05:57:07] step 520/4768 loss 4.0400 lr 6.00e-04 gnorm 0.31 54,202 tok/s peakVRAM 13.2GB
[06:00:19] step 540/4768 loss 3.9888 lr 6.00e-04 gnorm 0.27 54,667 tok/s peakVRAM 13.2GB
[06:03:30] step 560/4768 loss 3.9374 lr 6.00e-04 gnorm 0.21 54,666 tok/s peakVRAM 13.2GB
[06:06:42] step 580/4768 loss 3.8953 lr 6.00e-04 gnorm 0.24 54,661 tok/s peakVRAM 13.2GB
[06:09:54] step 600/4768 loss 3.8538 lr 6.00e-04 gnorm 0.24 54,647 tok/s peakVRAM 13.2GB
[06:13:06] step 620/4768 loss 3.8324 lr 6.00e-04 gnorm 0.20 54,649 tok/s peakVRAM 13.2GB
[06:16:18] step 640/4768 loss 3.8060 lr 6.00e-04 gnorm 0.21 54,653 tok/s peakVRAM 13.2GB
[06:19:30] step 660/4768 loss 3.7743 lr 6.00e-04 gnorm 0.16 54,649 tok/s peakVRAM 13.2GB
[06:22:42] step 680/4768 loss 3.7522 lr 6.00e-04 gnorm 0.18 54,642 tok/s peakVRAM 13.2GB
[06:25:54] step 700/4768 loss 3.7206 lr 6.00e-04 gnorm 0.16 54,646 tok/s peakVRAM 13.2GB
[06:29:05] step 720/4768 loss 3.6912 lr 6.00e-04 gnorm 0.16 54,647 tok/s peakVRAM 13.2GB
[06:32:17] step 740/4768 loss 3.6871 lr 6.00e-04 gnorm 0.17 54,646 tok/s peakVRAM 13.2GB
[06:35:29] step 760/4768 loss 3.6618 lr 6.00e-04 gnorm 0.15 54,634 tok/s peakVRAM 13.2GB
[06:38:41] step 780/4768 loss 3.6361 lr 6.00e-04 gnorm 0.16 54,634 tok/s peakVRAM 13.2GB
[06:41:53] step 800/4768 loss 3.6228 lr 6.00e-04 gnorm 0.14 54,641 tok/s peakVRAM 13.2GB
[06:45:05] step 820/4768 loss 3.6064 lr 6.00e-04 gnorm 0.17 54,635 tok/s peakVRAM 13.2GB
[06:48:17] step 840/4768 loss 3.5923 lr 6.00e-04 gnorm 0.15 54,631 tok/s peakVRAM 13.2GB
[06:51:29] step 860/4768 loss 3.5710 lr 6.00e-04 gnorm 0.13 54,635 tok/s peakVRAM 13.2GB
[06:54:41] step 880/4768 loss 3.5654 lr 6.00e-04 gnorm 0.14 54,631 tok/s peakVRAM 13.2GB
[06:57:53] step 900/4768 loss 3.5473 lr 6.00e-04 gnorm 0.15 54,636 tok/s peakVRAM 13.2GB
[07:01:05] step 920/4768 loss 3.5404 lr 6.00e-04 gnorm 0.14 54,640 tok/s peakVRAM 13.2GB
[07:04:17] step 940/4768 loss 3.5347 lr 6.00e-04 gnorm 0.13 54,638 tok/s peakVRAM 13.2GB
[07:07:29] step 960/4768 loss 3.5092 lr 6.00e-04 gnorm 0.13 54,630 tok/s peakVRAM 13.2GB
[07:10:40] step 980/4768 loss 3.5068 lr 6.00e-04 gnorm 0.14 54,640 tok/s peakVRAM 13.2GB
[07:13:52] step 1000/4768 loss 3.4964 lr 6.00e-04 gnorm 0.12 54,636 tok/s peakVRAM 13.2GB
[07:13:55] ckpt @ 1000
[07:17:06] step 1020/4768 loss 3.4906 lr 6.00e-04 gnorm 0.14 54,035 tok/s peakVRAM 13.2GB
[07:20:18] step 1040/4768 loss 3.4636 lr 6.00e-04 gnorm 0.13 54,680 tok/s peakVRAM 13.2GB
[07:23:30] step 1060/4768 loss 3.4751 lr 6.00e-04 gnorm 0.12 54,676 tok/s peakVRAM 13.2GB
[07:26:42] step 1080/4768 loss 3.4651 lr 6.00e-04 gnorm 0.14 54,670 tok/s peakVRAM 13.2GB
[07:29:54] step 1100/4768 loss 3.4490 lr 6.00e-04 gnorm 0.13 54,677 tok/s peakVRAM 13.2GB
[07:33:05] step 1120/4768 loss 3.4309 lr 6.00e-04 gnorm 0.12 54,668 tok/s peakVRAM 13.2GB
[07:36:17] step 1140/4768 loss 3.4241 lr 6.00e-04 gnorm 0.13 54,670 tok/s peakVRAM 13.2GB
[07:39:29] step 1160/4768 loss 3.4195 lr 6.00e-04 gnorm 0.14 54,669 tok/s peakVRAM 13.2GB
[07:42:41] step 1180/4768 loss 3.4214 lr 6.00e-04 gnorm 0.13 54,674 tok/s peakVRAM 13.2GB
[07:45:53] step 1200/4768 loss 3.4033 lr 6.00e-04 gnorm 0.11 54,676 tok/s peakVRAM 13.2GB
[07:49:04] step 1220/4768 loss 3.4008 lr 6.00e-04 gnorm 0.11 54,665 tok/s peakVRAM 13.2GB
[07:52:16] step 1240/4768 loss 3.3927 lr 6.00e-04 gnorm 0.14 54,664 tok/s peakVRAM 13.2GB
[07:55:28] step 1260/4768 loss 3.3785 lr 6.00e-04 gnorm 0.13 54,668 tok/s peakVRAM 13.2GB
[07:58:40] step 1280/4768 loss 3.3814 lr 6.00e-04 gnorm 0.11 54,669 tok/s peakVRAM 13.2GB
[08:01:52] step 1300/4768 loss 3.3748 lr 6.00e-04 gnorm 0.11 54,668 tok/s peakVRAM 13.2GB
[08:05:03] step 1320/4768 loss 3.3672 lr 6.00e-04 gnorm 0.12 54,674 tok/s peakVRAM 13.2GB
[08:08:15] step 1340/4768 loss 3.3675 lr 6.00e-04 gnorm 0.13 54,666 tok/s peakVRAM 13.2GB
[08:11:27] step 1360/4768 loss 3.3585 lr 6.00e-04 gnorm 0.13 54,661 tok/s peakVRAM 13.2GB
[08:14:39] step 1380/4768 loss 3.3499 lr 6.00e-04 gnorm 0.11 54,661 tok/s peakVRAM 13.2GB
[08:17:51] step 1400/4768 loss 3.3391 lr 6.00e-04 gnorm 0.11 54,657 tok/s peakVRAM 13.2GB
[08:21:03] step 1420/4768 loss 3.3271 lr 6.00e-04 gnorm 0.13 54,658 tok/s peakVRAM 13.2GB
[08:24:14] step 1440/4768 loss 3.3344 lr 6.00e-04 gnorm 0.13 54,655 tok/s peakVRAM 13.2GB
[08:27:26] step 1460/4768 loss 3.3319 lr 6.00e-04 gnorm 0.11 54,661 tok/s peakVRAM 13.2GB
[08:30:38] step 1480/4768 loss 3.3359 lr 6.00e-04 gnorm 0.12 54,661 tok/s peakVRAM 13.2GB
[08:33:50] step 1500/4768 loss 3.3086 lr 6.00e-04 gnorm 0.12 54,664 tok/s peakVRAM 13.2GB
[08:33:52] ckpt @ 1500
[08:37:04] step 1520/4768 loss 3.3161 lr 6.00e-04 gnorm 0.12 53,964 tok/s peakVRAM 13.2GB
[08:40:16] step 1540/4768 loss 3.3000 lr 6.00e-04 gnorm 0.11 54,622 tok/s peakVRAM 13.2GB
[08:43:28] step 1560/4768 loss 3.3088 lr 6.00e-04 gnorm 0.11 54,621 tok/s peakVRAM 13.2GB
[08:46:40] step 1580/4768 loss 3.2972 lr 6.00e-04 gnorm 0.11 54,645 tok/s peakVRAM 13.2GB
[08:49:52] step 1600/4768 loss 3.2864 lr 6.00e-04 gnorm 0.12 54,681 tok/s peakVRAM 13.2GB
[08:53:04] step 1620/4768 loss 3.2873 lr 6.00e-04 gnorm 0.13 54,610 tok/s peakVRAM 13.2GB
[08:56:16] step 1640/4768 loss 3.2835 lr 6.00e-04 gnorm 0.11 54,603 tok/s peakVRAM 13.2GB
[08:59:28] step 1660/4768 loss 3.2847 lr 6.00e-04 gnorm 0.11 54,609 tok/s peakVRAM 13.2GB
[09:02:40] step 1680/4768 loss 3.2738 lr 6.00e-04 gnorm 0.10 54,612 tok/s peakVRAM 13.2GB
[09:05:52] step 1700/4768 loss 3.2668 lr 6.00e-04 gnorm 0.11 54,611 tok/s peakVRAM 13.2GB
[09:09:04] step 1720/4768 loss 3.2687 lr 6.00e-04 gnorm 0.11 54,604 tok/s peakVRAM 13.2GB
[09:12:16] step 1740/4768 loss 3.2622 lr 6.00e-04 gnorm 0.11 54,617 tok/s peakVRAM 13.2GB
[09:15:28] step 1760/4768 loss 3.2621 lr 6.00e-04 gnorm 0.12 54,605 tok/s peakVRAM 13.2GB
[09:18:40] step 1780/4768 loss 3.2487 lr 6.00e-04 gnorm 0.11 54,605 tok/s peakVRAM 13.2GB
[09:21:52] step 1800/4768 loss 3.2527 lr 6.00e-04 gnorm 0.11 54,603 tok/s peakVRAM 13.2GB
[09:25:04] step 1820/4768 loss 3.2477 lr 6.00e-04 gnorm 0.11 54,613 tok/s peakVRAM 13.2GB
[09:28:16] step 1840/4768 loss 3.2415 lr 6.00e-04 gnorm 0.10 54,610 tok/s peakVRAM 13.2GB
[09:31:28] step 1860/4768 loss 3.2405 lr 6.00e-04 gnorm 0.11 54,619 tok/s peakVRAM 13.2GB
[09:34:40] step 1880/4768 loss 3.2366 lr 6.00e-04 gnorm 0.10 54,626 tok/s peakVRAM 13.2GB
[09:37:52] step 1900/4768 loss 3.2313 lr 6.00e-04 gnorm 0.11 54,622 tok/s peakVRAM 13.2GB
[09:41:04] step 1920/4768 loss 3.2317 lr 6.00e-04 gnorm 0.11 54,616 tok/s peakVRAM 13.2GB
[09:44:16] step 1940/4768 loss 3.2239 lr 6.00e-04 gnorm 0.11 54,614 tok/s peakVRAM 13.2GB
[09:47:28] step 1960/4768 loss 3.2283 lr 6.00e-04 gnorm 0.10 54,610 tok/s peakVRAM 13.2GB
[09:50:40] step 1980/4768 loss 3.2230 lr 6.00e-04 gnorm 0.11 54,610 tok/s peakVRAM 13.2GB
[09:53:52] step 2000/4768 loss 3.2165 lr 6.00e-04 gnorm 0.12 54,610 tok/s peakVRAM 13.2GB
[09:53:54] ckpt @ 2000
[09:57:06] step 2020/4768 loss 3.2159 lr 6.00e-04 gnorm 0.11 53,972 tok/s peakVRAM 13.2GB
[10:00:18] step 2040/4768 loss 3.2089 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[10:03:30] step 2060/4768 loss 3.2095 lr 6.00e-04 gnorm 0.11 54,629 tok/s peakVRAM 13.2GB
[10:06:42] step 2080/4768 loss 3.1978 lr 6.00e-04 gnorm 0.10 54,624 tok/s peakVRAM 13.2GB
[10:09:54] step 2100/4768 loss 3.2070 lr 6.00e-04 gnorm 0.10 54,619 tok/s peakVRAM 13.2GB
[10:13:06] step 2120/4768 loss 3.1962 lr 6.00e-04 gnorm 0.11 54,623 tok/s peakVRAM 13.2GB
[10:16:18] step 2140/4768 loss 3.2091 lr 6.00e-04 gnorm 0.10 54,626 tok/s peakVRAM 13.2GB
[10:19:30] step 2160/4768 loss 3.1877 lr 6.00e-04 gnorm 0.11 54,621 tok/s peakVRAM 13.2GB
[10:22:42] step 2180/4768 loss 3.1903 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[10:25:54] step 2200/4768 loss 3.1908 lr 6.00e-04 gnorm 0.11 54,631 tok/s peakVRAM 13.2GB
[10:29:06] step 2220/4768 loss 3.1834 lr 6.00e-04 gnorm 0.10 54,628 tok/s peakVRAM 13.2GB
[10:32:18] step 2240/4768 loss 3.1802 lr 6.00e-04 gnorm 0.10 54,631 tok/s peakVRAM 13.2GB
[10:35:30] step 2260/4768 loss 3.1768 lr 6.00e-04 gnorm 0.11 54,632 tok/s peakVRAM 13.2GB
[10:38:42] step 2280/4768 loss 3.1731 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[10:41:54] step 2300/4768 loss 3.1769 lr 6.00e-04 gnorm 0.10 54,627 tok/s peakVRAM 13.2GB
[10:45:06] step 2320/4768 loss 3.1732 lr 6.00e-04 gnorm 0.11 54,626 tok/s peakVRAM 13.2GB
[10:48:17] step 2340/4768 loss 3.1687 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[10:51:29] step 2360/4768 loss 3.1588 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[10:54:41] step 2380/4768 loss 3.1581 lr 6.00e-04 gnorm 0.10 54,632 tok/s peakVRAM 13.2GB
[10:57:53] step 2400/4768 loss 3.1582 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[11:01:05] step 2420/4768 loss 3.1613 lr 6.00e-04 gnorm 0.09 54,632 tok/s peakVRAM 13.2GB
[11:04:17] step 2440/4768 loss 3.1547 lr 6.00e-04 gnorm 0.11 54,634 tok/s peakVRAM 13.2GB
[11:07:29] step 2460/4768 loss 3.1544 lr 6.00e-04 gnorm 0.10 54,628 tok/s peakVRAM 13.2GB
[11:10:41] step 2480/4768 loss 3.1510 lr 6.00e-04 gnorm 0.11 54,633 tok/s peakVRAM 13.2GB
[11:13:53] step 2500/4768 loss 3.1457 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[11:13:55] ckpt @ 2500
[11:17:07] step 2520/4768 loss 3.1475 lr 6.00e-04 gnorm 0.10 53,971 tok/s peakVRAM 13.2GB
[11:20:19] step 2540/4768 loss 3.1478 lr 6.00e-04 gnorm 0.10 54,626 tok/s peakVRAM 13.2GB
[11:23:31] step 2560/4768 loss 3.1449 lr 6.00e-04 gnorm 0.11 54,619 tok/s peakVRAM 13.2GB
[11:26:43] step 2580/4768 loss 3.1403 lr 6.00e-04 gnorm 0.10 54,627 tok/s peakVRAM 13.2GB
[11:29:55] step 2600/4768 loss 3.1308 lr 6.00e-04 gnorm 0.10 54,624 tok/s peakVRAM 13.2GB
[11:33:07] step 2620/4768 loss 3.1397 lr 6.00e-04 gnorm 0.10 54,625 tok/s peakVRAM 13.2GB
[11:36:19] step 2640/4768 loss 3.1406 lr 6.00e-04 gnorm 0.10 54,625 tok/s peakVRAM 13.2GB
[11:39:31] step 2660/4768 loss 3.1389 lr 6.00e-04 gnorm 0.09 54,628 tok/s peakVRAM 13.2GB
[11:42:43] step 2680/4768 loss 3.1341 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[11:45:55] step 2700/4768 loss 3.1367 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[11:49:07] step 2720/4768 loss 3.1201 lr 6.00e-04 gnorm 0.11 54,629 tok/s peakVRAM 13.2GB
[11:52:19] step 2740/4768 loss 3.1216 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[11:55:31] step 2760/4768 loss 3.1249 lr 6.00e-04 gnorm 0.10 54,632 tok/s peakVRAM 13.2GB
[11:58:43] step 2780/4768 loss 3.1154 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[12:01:55] step 2800/4768 loss 3.1102 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[12:05:07] step 2820/4768 loss 3.1080 lr 6.00e-04 gnorm 0.09 54,619 tok/s peakVRAM 13.2GB
[12:08:19] step 2840/4768 loss 3.1164 lr 6.00e-04 gnorm 0.10 54,612 tok/s peakVRAM 13.2GB
[12:11:31] step 2860/4768 loss 3.1075 lr 6.00e-04 gnorm 0.10 54,620 tok/s peakVRAM 13.2GB