Hugging Face's logo

codeShare
/
kromoss-lora-exports

Safetensors
Model card Files
xet
 Community
codeShare commited on
Commit
eb3242b
·
verified ·
1 Parent(s): 5b664b8

Upload Kromoss dual-T4 training log

Browse files
Files changed (1) hide show
  1. training_logs/kromoss_training_log.txt +73 -1356
training_logs/kromoss_training_log.txt CHANGED
@@ -1,1356 +1,73 @@
1
- 2026-11-01 23:20:06 | INFO | ================================================================================
2
- 2026-11-01 23:20:06 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
- 2026-11-01 23:20:06 | INFO | ================================================================================
4
- 2026-11-01 23:20:06 | INFO | PyTorch version: 2.11.0+cu128
5
- 2026-11-01 23:20:06 | INFO | CUDA version: 12.8
6
- 2026-11-01 23:20:06 | INFO | Dataset size: 7
7
- 2026-11-01 23:20:06 | INFO | Presentations per image: 10
8
- 2026-11-01 23:20:06 | INFO | Total optimizer updates: 70
9
- 2026-11-01 23:20:06 | INFO | Checkpoint steps: [14, 35, 70]
10
- 2026-11-01 23:20:06 | INFO | Initial global learning rate: 5.00000000e-05
11
- 2026-11-01 23:20:06 | INFO | Final global learning rate: 2.50000000e-05
12
- 2026-11-01 23:20:06 | INFO | Weight decay: 0.01
13
- 2026-11-01 23:20:06 | INFO | Gradient clipping: 0.5
14
- 2026-11-01 23:20:06 | INFO | Loss scale: 128.0
15
- 2026-11-01 23:20:06 | INFO | LoRA rank: 16
16
- 2026-11-01 23:20:06 | INFO | LoRA alpha: 16
17
- 2026-11-01 23:20:06 | INFO | Trainable parameters: 37,158,912
18
- 2026-11-01 23:20:06 | INFO | GPU0 LoRA parameters: 17,694,720
19
- 2026-11-01 23:20:06 | INFO | GPU1 LoRA parameters: 19,464,192
20
- 2026-11-01 23:20:06 | INFO | Continuation loaded: True
21
- 2026-11-01 23:20:06 | INFO | Continuation loading enabled: True
22
- 2026-11-01 23:20:06 | INFO | HF repository: codeShare/kromoss-lora-exports
23
- 2026-11-01 23:20:06 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
- 2026-11-01 23:20:06 | INFO | GPU0: Tesla T4 | 14.562 GB
25
- 2026-11-01 23:20:06 | INFO | GPU1: Tesla T4 | 14.562 GB
26
- 2026-11-01 23:20:06 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
- 2026-11-01 23:20:06 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
- 2026-11-01 23:20:06 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
- 2026-11-01 23:20:06 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
- 2026-11-01 23:20:06 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
- 2026-11-01 23:20:06 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
- 2026-11-01 23:20:06 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
- 2026-11-01 23:20:06 | INFO | --------------------------------------------------------------------------------
34
- 2026-11-01 23:20:06 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
- 2026-11-01 23:20:06 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
- 2026-11-01 23:20:06 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
- 2026-11-01 23:20:06 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
- 2026-11-01 23:20:06 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
- 2026-11-01 23:20:06 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
- 2026-11-01 23:20:06 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
- 2026-11-01 23:20:06 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
- 2026-11-01 23:20:06 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
- 2026-11-01 23:20:06 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
- 2026-11-01 23:20:06 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
- 2026-11-01 23:20:06 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
- 2026-11-01 23:20:06 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
- 2026-11-01 23:20:06 | INFO | ================================================================================
48
- 2026-11-01 23:20:06 | INFO | STARTING TRAINING
49
- 2026-11-01 23:20:06 | INFO | ================================================================================
50
- 2026-11-01 23:20:06 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
- 2026-11-01 23:20:06 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
- 2026-11-01 23:20:06 | INFO | AdamW optimizer state is retained across all 70 updates.
53
- 2026-11-01 23:20:06 | INFO | Learning rate does NOT reset when the dataset image changes.
54
- 2026-11-01 23:20:06 | INFO | Gradient clipping threshold: 0.5
55
- 2026-11-01 23:20:06 | INFO | Noise is generated in FP32 and cast to FP16.
56
- 2026-11-01 23:20:06 | INFO | Timestep is logged for every optimizer update.
57
- 2026-11-01 23:20:06 | INFO | Continuation loading enabled: True
58
- 2026-11-01 23:20:06 | INFO | Continuation actually loaded: True
59
- 2026-11-01 23:20:21 | INFO | --------------------------------------------------------------------------------
60
- 2026-11-01 23:20:21 | INFO | STEP 1/70 | epoch=1/10 | position=1/7 | dataset_item=1
61
- 2026-11-01 23:20:21 | INFO | DATASET FILE: 00000001.pt
62
- 2026-11-01 23:20:21 | INFO | LOSS: 1.06019557
63
- 2026-11-01 23:20:21 | INFO | LR: 5.00000000e-05
64
- 2026-11-01 23:20:21 | INFO | TIMESTEP: 0.32641813
65
- 2026-11-01 23:20:21 | INFO | GLOBAL GRADIENT NORM: 0.07360940
66
- 2026-11-01 23:20:21 | INFO | MAX ABSOLUTE GRADIENT: 2.46238708e-03
67
- 2026-11-01 23:20:21 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
68
- 2026-11-01 23:20:21 | INFO | MISSING GRADIENT TENSORS: 0
69
- 2026-11-01 23:20:21 | INFO | GPU0 FORWARD: 2.191s
70
- 2026-11-01 23:20:21 | INFO | GPU1 FORWARD: 2.165s
71
- 2026-11-01 23:20:21 | INFO | GPU1 BACKWARD: 5.611s
72
- 2026-11-01 23:20:21 | INFO | GPU0 BACKWARD: 5.219s
73
- 2026-11-01 23:20:21 | INFO | UPDATE WALL TIME: 15.407s
74
- 2026-11-01 23:20:21 | INFO | ELAPSED TRAINING TIME: 15.46s
75
- 2026-11-01 23:20:21 | INFO | STEP 1 | GPU0 | allocated=5.070 GB | reserved=12.859 GB | peak=12.624 GB | free=1.565 GB
76
- 2026-11-01 23:20:21 | INFO | STEP 1 | GPU1 | allocated=4.984 GB | reserved=12.785 GB | peak=12.519 GB | free=1.639 GB
77
- 2026-11-01 23:20:37 | INFO | --------------------------------------------------------------------------------
78
- 2026-11-01 23:20:37 | INFO | STEP 2/70 | epoch=1/10 | position=2/7 | dataset_item=3
79
- 2026-11-01 23:20:37 | INFO | DATASET FILE: 00000003.pt
80
- 2026-11-01 23:20:37 | INFO | LOSS: 1.13788939
81
- 2026-11-01 23:20:37 | INFO | LR: 4.96376812e-05
82
- 2026-11-01 23:20:37 | INFO | TIMESTEP: 0.26634458
83
- 2026-11-01 23:20:37 | INFO | GLOBAL GRADIENT NORM: 0.09846493
84
- 2026-11-01 23:20:37 | INFO | MAX ABSOLUTE GRADIENT: 5.50842285e-03
85
- 2026-11-01 23:20:37 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
86
- 2026-11-01 23:20:37 | INFO | MISSING GRADIENT TENSORS: 0
87
- 2026-11-01 23:20:37 | INFO | GPU0 FORWARD: 2.041s
88
- 2026-11-01 23:20:37 | INFO | GPU1 FORWARD: 2.241s
89
- 2026-11-01 23:20:37 | INFO | GPU1 BACKWARD: 5.726s
90
- 2026-11-01 23:20:37 | INFO | GPU0 BACKWARD: 5.318s
91
- 2026-11-01 23:20:37 | INFO | UPDATE WALL TIME: 15.473s
92
- 2026-11-01 23:20:37 | INFO | ELAPSED TRAINING TIME: 30.94s
93
- 2026-11-01 23:20:37 | INFO | STEP 2 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.898 GB | free=1.429 GB
94
- 2026-11-01 23:20:37 | INFO | STEP 2 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.805 GB | free=1.503 GB
95
- 2026-11-01 23:20:53 | INFO | --------------------------------------------------------------------------------
96
- 2026-11-01 23:20:53 | INFO | STEP 3/70 | epoch=1/10 | position=3/7 | dataset_item=4
97
- 2026-11-01 23:20:53 | INFO | DATASET FILE: 00000004.pt
98
- 2026-11-01 23:20:53 | INFO | LOSS: 1.15446568
99
- 2026-11-01 23:20:53 | INFO | LR: 4.92753623e-05
100
- 2026-11-01 23:20:53 | INFO | TIMESTEP: 0.25900331
101
- 2026-11-01 23:20:53 | INFO | GLOBAL GRADIENT NORM: 0.08567047
102
- 2026-11-01 23:20:53 | INFO | MAX ABSOLUTE GRADIENT: 3.32260132e-03
103
- 2026-11-01 23:20:53 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
104
- 2026-11-01 23:20:53 | INFO | MISSING GRADIENT TENSORS: 0
105
- 2026-11-01 23:20:53 | INFO | GPU0 FORWARD: 2.051s
106
- 2026-11-01 23:20:53 | INFO | GPU1 FORWARD: 2.290s
107
- 2026-11-01 23:20:53 | INFO | GPU1 BACKWARD: 5.770s
108
- 2026-11-01 23:20:53 | INFO | GPU0 BACKWARD: 5.389s
109
- 2026-11-01 23:20:53 | INFO | UPDATE WALL TIME: 15.645s
110
- 2026-11-01 23:20:53 | INFO | ELAPSED TRAINING TIME: 46.60s
111
- 2026-11-01 23:20:53 | INFO | STEP 3 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.862 GB | free=1.429 GB
112
- 2026-11-01 23:20:53 | INFO | STEP 3 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.768 GB | free=1.503 GB
113
- 2026-11-01 23:21:08 | INFO | --------------------------------------------------------------------------------
114
- 2026-11-01 23:21:08 | INFO | STEP 4/70 | epoch=1/10 | position=4/7 | dataset_item=2
115
- 2026-11-01 23:21:08 | INFO | DATASET FILE: 00000002.pt
116
- 2026-11-01 23:21:08 | INFO | LOSS: 1.09719360
117
- 2026-11-01 23:21:08 | INFO | LR: 4.89130435e-05
118
- 2026-11-01 23:21:08 | INFO | TIMESTEP: 0.32806167
119
- 2026-11-01 23:21:08 | INFO | GLOBAL GRADIENT NORM: 0.09014856
120
- 2026-11-01 23:21:08 | INFO | MAX ABSOLUTE GRADIENT: 4.48989868e-03
121
- 2026-11-01 23:21:08 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
122
- 2026-11-01 23:21:08 | INFO | MISSING GRADIENT TENSORS: 0
123
- 2026-11-01 23:21:08 | INFO | GPU0 FORWARD: 2.097s
124
- 2026-11-01 23:21:08 | INFO | GPU1 FORWARD: 2.272s
125
- 2026-11-01 23:21:08 | INFO | GPU1 BACKWARD: 5.790s
126
- 2026-11-01 23:21:08 | INFO | GPU0 BACKWARD: 5.476s
127
- 2026-11-01 23:21:08 | INFO | UPDATE WALL TIME: 15.788s
128
- 2026-11-01 23:21:08 | INFO | ELAPSED TRAINING TIME: 62.40s
129
- 2026-11-01 23:21:08 | INFO | STEP 4 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.835 GB | free=1.429 GB
130
- 2026-11-01 23:21:08 | INFO | STEP 4 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.742 GB | free=1.503 GB
131
- 2026-11-01 23:21:24 | INFO | --------------------------------------------------------------------------------
132
- 2026-11-01 23:21:24 | INFO | STEP 5/70 | epoch=1/10 | position=5/7 | dataset_item=6
133
- 2026-11-01 23:21:24 | INFO | DATASET FILE: 00000006.pt
134
- 2026-11-01 23:21:24 | INFO | LOSS: 1.18125892
135
- 2026-11-01 23:21:24 | INFO | LR: 4.85507246e-05
136
- 2026-11-01 23:21:24 | INFO | TIMESTEP: 0.22171448
137
- 2026-11-01 23:21:24 | INFO | GLOBAL GRADIENT NORM: 0.11910705
138
- 2026-11-01 23:21:24 | INFO | MAX ABSOLUTE GRADIENT: 3.12423706e-03
139
- 2026-11-01 23:21:24 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
140
- 2026-11-01 23:21:24 | INFO | MISSING GRADIENT TENSORS: 0
141
- 2026-11-01 23:21:24 | INFO | GPU0 FORWARD: 2.169s
142
- 2026-11-01 23:21:24 | INFO | GPU1 FORWARD: 2.286s
143
- 2026-11-01 23:21:24 | INFO | GPU1 BACKWARD: 5.805s
144
- 2026-11-01 23:21:24 | INFO | GPU0 BACKWARD: 5.700s
145
- 2026-11-01 23:21:24 | INFO | UPDATE WALL TIME: 16.102s
146
- 2026-11-01 23:21:24 | INFO | ELAPSED TRAINING TIME: 78.51s
147
- 2026-11-01 23:21:24 | INFO | STEP 5 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.882 GB | free=1.429 GB
148
- 2026-11-01 23:21:24 | INFO | STEP 5 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.788 GB | free=1.503 GB
149
- 2026-11-01 23:21:41 | INFO | --------------------------------------------------------------------------------
150
- 2026-11-01 23:21:41 | INFO | STEP 6/70 | epoch=1/10 | position=6/7 | dataset_item=0
151
- 2026-11-01 23:21:41 | INFO | DATASET FILE: 00000000.pt
152
- 2026-11-01 23:21:41 | INFO | LOSS: 1.17807746
153
- 2026-11-01 23:21:41 | INFO | LR: 4.81884058e-05
154
- 2026-11-01 23:21:41 | INFO | TIMESTEP: 0.14347191
155
- 2026-11-01 23:21:41 | INFO | GLOBAL GRADIENT NORM: 0.09497192
156
- 2026-11-01 23:21:41 | INFO | MAX ABSOLUTE GRADIENT: 3.74603271e-03
157
- 2026-11-01 23:21:41 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
158
- 2026-11-01 23:21:41 | INFO | MISSING GRADIENT TENSORS: 0
159
- 2026-11-01 23:21:41 | INFO | GPU0 FORWARD: 2.218s
160
- 2026-11-01 23:21:41 | INFO | GPU1 FORWARD: 2.291s
161
- 2026-11-01 23:21:41 | INFO | GPU1 BACKWARD: 5.917s
162
- 2026-11-01 23:21:41 | INFO | GPU0 BACKWARD: 5.780s
163
- 2026-11-01 23:21:41 | INFO | UPDATE WALL TIME: 16.356s
164
- 2026-11-01 23:21:41 | INFO | ELAPSED TRAINING TIME: 94.88s
165
- 2026-11-01 23:21:41 | INFO | STEP 6 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.821 GB | free=1.429 GB
166
- 2026-11-01 23:21:41 | INFO | STEP 6 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.728 GB | free=1.503 GB
167
- 2026-11-01 23:21:58 | INFO | --------------------------------------------------------------------------------
168
- 2026-11-01 23:21:58 | INFO | STEP 7/70 | epoch=1/10 | position=7/7 | dataset_item=5
169
- 2026-11-01 23:21:58 | INFO | DATASET FILE: 00000005.pt
170
- 2026-11-01 23:21:58 | INFO | LOSS: 0.63784271
171
- 2026-11-01 23:21:58 | INFO | LR: 4.78260870e-05
172
- 2026-11-01 23:21:58 | INFO | TIMESTEP: 0.57760578
173
- 2026-11-01 23:21:58 | INFO | GLOBAL GRADIENT NORM: 0.09459939
174
- 2026-11-01 23:21:58 | INFO | MAX ABSOLUTE GRADIENT: 2.33268738e-03
175
- 2026-11-01 23:21:58 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
176
- 2026-11-01 23:21:58 | INFO | MISSING GRADIENT TENSORS: 0
177
- 2026-11-01 23:21:58 | INFO | GPU0 FORWARD: 2.358s
178
- 2026-11-01 23:21:58 | INFO | GPU1 FORWARD: 2.387s
179
- 2026-11-01 23:21:58 | INFO | GPU1 BACKWARD: 6.309s
180
- 2026-11-01 23:21:58 | INFO | GPU0 BACKWARD: 6.067s
181
- 2026-11-01 23:21:58 | INFO | UPDATE WALL TIME: 17.271s
182
- 2026-11-01 23:21:58 | INFO | ELAPSED TRAINING TIME: 112.16s
183
- 2026-11-01 23:21:58 | INFO | STEP 7 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
184
- 2026-11-01 23:21:58 | INFO | STEP 7 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
185
- 2026-11-01 23:22:16 | INFO | --------------------------------------------------------------------------------
186
- 2026-11-01 23:22:16 | INFO | STEP 8/70 | epoch=2/10 | position=1/7 | dataset_item=3
187
- 2026-11-01 23:22:16 | INFO | DATASET FILE: 00000003.pt
188
- 2026-11-01 23:22:16 | INFO | LOSS: 1.11418712
189
- 2026-11-01 23:22:16 | INFO | LR: 4.74637681e-05
190
- 2026-11-01 23:22:16 | INFO | TIMESTEP: 0.27784425
191
- 2026-11-01 23:22:16 | INFO | GLOBAL GRADIENT NORM: 0.08368929
192
- 2026-11-01 23:22:16 | INFO | MAX ABSOLUTE GRADIENT: 2.32315063e-03
193
- 2026-11-01 23:22:16 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
194
- 2026-11-01 23:22:16 | INFO | MISSING GRADIENT TENSORS: 0
195
- 2026-11-01 23:22:16 | INFO | GPU0 FORWARD: 2.423s
196
- 2026-11-01 23:22:16 | INFO | GPU1 FORWARD: 2.406s
197
- 2026-11-01 23:22:16 | INFO | GPU1 BACKWARD: 6.315s
198
- 2026-11-01 23:22:16 | INFO | GPU0 BACKWARD: 6.153s
199
- 2026-11-01 23:22:16 | INFO | UPDATE WALL TIME: 17.448s
200
- 2026-11-01 23:22:16 | INFO | ELAPSED TRAINING TIME: 129.62s
201
- 2026-11-01 23:22:16 | INFO | STEP 8 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
202
- 2026-11-01 23:22:16 | INFO | STEP 8 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
203
- 2026-11-01 23:22:33 | INFO | --------------------------------------------------------------------------------
204
- 2026-11-01 23:22:33 | INFO | STEP 9/70 | epoch=2/10 | position=2/7 | dataset_item=6
205
- 2026-11-01 23:22:33 | INFO | DATASET FILE: 00000006.pt
206
- 2026-11-01 23:22:33 | INFO | LOSS: 0.57072091
207
- 2026-11-01 23:22:33 | INFO | LR: 4.71014493e-05
208
- 2026-11-01 23:22:33 | INFO | TIMESTEP: 0.73534703
209
- 2026-11-01 23:22:33 | INFO | GLOBAL GRADIENT NORM: 0.08181708
210
- 2026-11-01 23:22:33 | INFO | MAX ABSOLUTE GRADIENT: 2.39753723e-03
211
- 2026-11-01 23:22:33 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
212
- 2026-11-01 23:22:33 | INFO | MISSING GRADIENT TENSORS: 0
213
- 2026-11-01 23:22:33 | INFO | GPU0 FORWARD: 2.459s
214
- 2026-11-01 23:22:33 | INFO | GPU1 FORWARD: 2.446s
215
- 2026-11-01 23:22:33 | INFO | GPU1 BACKWARD: 6.383s
216
- 2026-11-01 23:22:33 | INFO | GPU0 BACKWARD: 6.250s
217
- 2026-11-01 23:22:33 | INFO | UPDATE WALL TIME: 17.682s
218
- 2026-11-01 23:22:33 | INFO | ELAPSED TRAINING TIME: 147.32s
219
- 2026-11-01 23:22:33 | INFO | STEP 9 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
220
- 2026-11-01 23:22:33 | INFO | STEP 9 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
221
- 2026-11-01 23:22:51 | INFO | --------------------------------------------------------------------------------
222
- 2026-11-01 23:22:51 | INFO | STEP 10/70 | epoch=2/10 | position=3/7 | dataset_item=2
223
- 2026-11-01 23:22:51 | INFO | DATASET FILE: 00000002.pt
224
- 2026-11-01 23:22:51 | INFO | LOSS: 0.71565700
225
- 2026-11-01 23:22:51 | INFO | LR: 4.67391304e-05
226
- 2026-11-01 23:22:51 | INFO | TIMESTEP: 0.57242364
227
- 2026-11-01 23:22:51 | INFO | GLOBAL GRADIENT NORM: 0.06862144
228
- 2026-11-01 23:22:51 | INFO | MAX ABSOLUTE GRADIENT: 2.21061707e-03
229
- 2026-11-01 23:22:51 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
230
- 2026-11-01 23:22:51 | INFO | MISSING GRADIENT TENSORS: 0
231
- 2026-11-01 23:22:51 | INFO | GPU0 FORWARD: 2.503s
232
- 2026-11-01 23:22:51 | INFO | GPU1 FORWARD: 2.380s
233
- 2026-11-01 23:22:51 | INFO | GPU1 BACKWARD: 6.149s
234
- 2026-11-01 23:22:51 | INFO | GPU0 BACKWARD: 6.307s
235
- 2026-11-01 23:22:51 | INFO | UPDATE WALL TIME: 17.486s
236
- 2026-11-01 23:22:51 | INFO | ELAPSED TRAINING TIME: 164.81s
237
- 2026-11-01 23:22:51 | INFO | STEP 10 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
238
- 2026-11-01 23:22:51 | INFO | STEP 10 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
239
- 2026-11-01 23:23:08 | INFO | --------------------------------------------------------------------------------
240
- 2026-11-01 23:23:08 | INFO | STEP 11/70 | epoch=2/10 | position=4/7 | dataset_item=4
241
- 2026-11-01 23:23:08 | INFO | DATASET FILE: 00000004.pt
242
- 2026-11-01 23:23:08 | INFO | LOSS: 0.55634642
243
- 2026-11-01 23:23:08 | INFO | LR: 4.63768116e-05
244
- 2026-11-01 23:23:08 | INFO | TIMESTEP: 0.72895420
245
- 2026-11-01 23:23:08 | INFO | GLOBAL GRADIENT NORM: 0.08422560
246
- 2026-11-01 23:23:08 | INFO | MAX ABSOLUTE GRADIENT: 2.57110596e-03
247
- 2026-11-01 23:23:08 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
248
- 2026-11-01 23:23:08 | INFO | MISSING GRADIENT TENSORS: 0
249
- 2026-11-01 23:23:08 | INFO | GPU0 FORWARD: 2.535s
250
- 2026-11-01 23:23:08 | INFO | GPU1 FORWARD: 2.383s
251
- 2026-11-01 23:23:08 | INFO | GPU1 BACKWARD: 6.230s
252
- 2026-11-01 23:23:08 | INFO | GPU0 BACKWARD: 6.204s
253
- 2026-11-01 23:23:08 | INFO | UPDATE WALL TIME: 17.500s
254
- 2026-11-01 23:23:08 | INFO | ELAPSED TRAINING TIME: 182.33s
255
- 2026-11-01 23:23:08 | INFO | STEP 11 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
256
- 2026-11-01 23:23:08 | INFO | STEP 11 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
257
- 2026-11-01 23:23:26 | INFO | --------------------------------------------------------------------------------
258
- 2026-11-01 23:23:26 | INFO | STEP 12/70 | epoch=2/10 | position=5/7 | dataset_item=0
259
- 2026-11-01 23:23:26 | INFO | DATASET FILE: 00000000.pt
260
- 2026-11-01 23:23:26 | INFO | LOSS: 1.15281069
261
- 2026-11-01 23:23:26 | INFO | LR: 4.60144928e-05
262
- 2026-11-01 23:23:26 | INFO | TIMESTEP: 0.10170523
263
- 2026-11-01 23:23:26 | INFO | GLOBAL GRADIENT NORM: 0.09061066
264
- 2026-11-01 23:23:26 | INFO | MAX ABSOLUTE GRADIENT: 4.42504883e-03
265
- 2026-11-01 23:23:26 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
266
- 2026-11-01 23:23:26 | INFO | MISSING GRADIENT TENSORS: 0
267
- 2026-11-01 23:23:26 | INFO | GPU0 FORWARD: 2.471s
268
- 2026-11-01 23:23:26 | INFO | GPU1 FORWARD: 2.365s
269
- 2026-11-01 23:23:26 | INFO | GPU1 BACKWARD: 6.184s
270
- 2026-11-01 23:23:26 | INFO | GPU0 BACKWARD: 6.154s
271
- 2026-11-01 23:23:26 | INFO | UPDATE WALL TIME: 17.321s
272
- 2026-11-01 23:23:26 | INFO | ELAPSED TRAINING TIME: 199.66s
273
- 2026-11-01 23:23:26 | INFO | STEP 12 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
274
- 2026-11-01 23:23:26 | INFO | STEP 12 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
275
- 2026-11-01 23:23:43 | INFO | --------------------------------------------------------------------------------
276
- 2026-11-01 23:23:43 | INFO | STEP 13/70 | epoch=2/10 | position=6/7 | dataset_item=5
277
- 2026-11-01 23:23:43 | INFO | DATASET FILE: 00000005.pt
278
- 2026-11-01 23:23:43 | INFO | LOSS: 0.51763958
279
- 2026-11-01 23:23:43 | INFO | LR: 4.56521739e-05
280
- 2026-11-01 23:23:43 | INFO | TIMESTEP: 0.83249700
281
- 2026-11-01 23:23:43 | INFO | GLOBAL GRADIENT NORM: 0.15706175
282
- 2026-11-01 23:23:43 | INFO | MAX ABSOLUTE GRADIENT: 1.00936890e-02
283
- 2026-11-01 23:23:43 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
284
- 2026-11-01 23:23:43 | INFO | MISSING GRADIENT TENSORS: 0
285
- 2026-11-01 23:23:43 | INFO | GPU0 FORWARD: 2.444s
286
- 2026-11-01 23:23:43 | INFO | GPU1 FORWARD: 2.444s
287
- 2026-11-01 23:23:43 | INFO | GPU1 BACKWARD: 6.431s
288
- 2026-11-01 23:23:43 | INFO | GPU0 BACKWARD: 6.234s
289
- 2026-11-01 23:23:43 | INFO | UPDATE WALL TIME: 17.704s
290
- 2026-11-01 23:23:43 | INFO | ELAPSED TRAINING TIME: 217.37s
291
- 2026-11-01 23:23:43 | INFO | STEP 13 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
292
- 2026-11-01 23:23:43 | INFO | STEP 13 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
293
- 2026-11-01 23:24:01 | INFO | --------------------------------------------------------------------------------
294
- 2026-11-01 23:24:01 | INFO | STEP 14/70 | epoch=2/10 | position=7/7 | dataset_item=1
295
- 2026-11-01 23:24:01 | INFO | DATASET FILE: 00000001.pt
296
- 2026-11-01 23:24:01 | INFO | LOSS: 0.58677816
297
- 2026-11-01 23:24:01 | INFO | LR: 4.52898551e-05
298
- 2026-11-01 23:24:01 | INFO | TIMESTEP: 0.68447441
299
- 2026-11-01 23:24:01 | INFO | GLOBAL GRADIENT NORM: 0.07559282
300
- 2026-11-01 23:24:01 | INFO | MAX ABSOLUTE GRADIENT: 2.25257874e-03
301
- 2026-11-01 23:24:01 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
302
- 2026-11-01 23:24:01 | INFO | MISSING GRADIENT TENSORS: 0
303
- 2026-11-01 23:24:01 | INFO | GPU0 FORWARD: 2.439s
304
- 2026-11-01 23:24:01 | INFO | GPU1 FORWARD: 2.383s
305
- 2026-11-01 23:24:01 | INFO | GPU1 BACKWARD: 6.212s
306
- 2026-11-01 23:24:01 | INFO | GPU0 BACKWARD: 6.142s
307
- 2026-11-01 23:24:01 | INFO | UPDATE WALL TIME: 17.323s
308
- 2026-11-01 23:24:01 | INFO | ELAPSED TRAINING TIME: 234.71s
309
- 2026-11-01 23:24:01 | INFO | STEP 14 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
310
- 2026-11-01 23:24:01 | INFO | STEP 14 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
311
- 2026-11-01 23:24:01 | INFO | ================================================================================
312
- 2026-11-01 23:24:01 | INFO | CHECKPOINT TRIGGERED AT OPTIMIZER STEP 14
313
- 2026-11-01 23:24:05 | INFO | CHECKPOINT SUCCESS: kromoss_lora_step_000014.safetensors
314
- 2026-11-01 23:24:05 | INFO | CHECKPOINT SIZE: 70.93 MB
315
- 2026-11-01 23:24:05 | INFO | ================================================================================
316
- 2026-11-01 23:24:22 | INFO | --------------------------------------------------------------------------------
317
- 2026-11-01 23:24:22 | INFO | STEP 15/70 | epoch=3/10 | position=1/7 | dataset_item=3
318
- 2026-11-01 23:24:22 | INFO | DATASET FILE: 00000003.pt
319
- 2026-11-01 23:24:22 | INFO | LOSS: 1.12059379
320
- 2026-11-01 23:24:22 | INFO | LR: 4.49275362e-05
321
- 2026-11-01 23:24:22 | INFO | TIMESTEP: 0.27099666
322
- 2026-11-01 23:24:22 | INFO | GLOBAL GRADIENT NORM: 0.08447098
323
- 2026-11-01 23:24:22 | INFO | MAX ABSOLUTE GRADIENT: 3.82041931e-03
324
- 2026-11-01 23:24:22 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
325
- 2026-11-01 23:24:22 | INFO | MISSING GRADIENT TENSORS: 0
326
- 2026-11-01 23:24:22 | INFO | GPU0 FORWARD: 2.469s
327
- 2026-11-01 23:24:22 | INFO | GPU1 FORWARD: 2.370s
328
- 2026-11-01 23:24:22 | INFO | GPU1 BACKWARD: 6.226s
329
- 2026-11-01 23:24:22 | INFO | GPU0 BACKWARD: 6.292s
330
- 2026-11-01 23:24:22 | INFO | UPDATE WALL TIME: 17.502s
331
- 2026-11-01 23:24:22 | INFO | ELAPSED TRAINING TIME: 256.37s
332
- 2026-11-01 23:24:22 | INFO | STEP 15 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
333
- 2026-11-01 23:24:22 | INFO | STEP 15 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
334
- 2026-11-01 23:24:40 | INFO | --------------------------------------------------------------------------------
335
- 2026-11-01 23:24:40 | INFO | STEP 16/70 | epoch=3/10 | position=2/7 | dataset_item=2
336
- 2026-11-01 23:24:40 | INFO | DATASET FILE: 00000002.pt
337
- 2026-11-01 23:24:40 | INFO | LOSS: 1.07068217
338
- 2026-11-01 23:24:40 | INFO | LR: 4.45652174e-05
339
- 2026-11-01 23:24:40 | INFO | TIMESTEP: 0.34128180
340
- 2026-11-01 23:24:40 | INFO | GLOBAL GRADIENT NORM: 0.09242336
341
- 2026-11-01 23:24:40 | INFO | MAX ABSOLUTE GRADIENT: 4.89044189e-03
342
- 2026-11-01 23:24:40 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
343
- 2026-11-01 23:24:40 | INFO | MISSING GRADIENT TENSORS: 0
344
- 2026-11-01 23:24:40 | INFO | GPU0 FORWARD: 2.515s
345
- 2026-11-01 23:24:40 | INFO | GPU1 FORWARD: 2.413s
346
- 2026-11-01 23:24:40 | INFO | GPU1 BACKWARD: 6.276s
347
- 2026-11-01 23:24:40 | INFO | GPU0 BACKWARD: 6.269s
348
- 2026-11-01 23:24:40 | INFO | UPDATE WALL TIME: 17.625s
349
- 2026-11-01 23:24:40 | INFO | ELAPSED TRAINING TIME: 274.00s
350
- 2026-11-01 23:24:40 | INFO | STEP 16 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
351
- 2026-11-01 23:24:40 | INFO | STEP 16 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
352
- 2026-11-01 23:24:57 | INFO | --------------------------------------------------------------------------------
353
- 2026-11-01 23:24:57 | INFO | STEP 17/70 | epoch=3/10 | position=3/7 | dataset_item=4
354
- 2026-11-01 23:24:57 | INFO | DATASET FILE: 00000004.pt
355
- 2026-11-01 23:24:57 | INFO | LOSS: 1.04439282
356
- 2026-11-01 23:24:57 | INFO | LR: 4.42028986e-05
357
- 2026-11-01 23:24:57 | INFO | TIMESTEP: 0.00303028
358
- 2026-11-01 23:24:57 | INFO | GLOBAL GRADIENT NORM: 0.28457307
359
- 2026-11-01 23:24:57 | INFO | MAX ABSOLUTE GRADIENT: 1.71508789e-02
360
- 2026-11-01 23:24:57 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
361
- 2026-11-01 23:24:57 | INFO | MISSING GRADIENT TENSORS: 0
362
- 2026-11-01 23:24:57 | INFO | GPU0 FORWARD: 2.454s
363
- 2026-11-01 23:24:57 | INFO | GPU1 FORWARD: 2.417s
364
- 2026-11-01 23:24:57 | INFO | GPU1 BACKWARD: 6.183s
365
- 2026-11-01 23:24:57 | INFO | GPU0 BACKWARD: 6.217s
366
- 2026-11-01 23:24:57 | INFO | UPDATE WALL TIME: 17.418s
367
- 2026-11-01 23:24:57 | INFO | ELAPSED TRAINING TIME: 291.43s
368
- 2026-11-01 23:24:57 | INFO | STEP 17 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
369
- 2026-11-01 23:24:57 | INFO | STEP 17 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
370
- 2026-11-01 23:25:15 | INFO | --------------------------------------------------------------------------------
371
- 2026-11-01 23:25:15 | INFO | STEP 18/70 | epoch=3/10 | position=4/7 | dataset_item=1
372
- 2026-11-01 23:25:15 | INFO | DATASET FILE: 00000001.pt
373
- 2026-11-01 23:25:15 | INFO | LOSS: 0.74975574
374
- 2026-11-01 23:25:15 | INFO | LR: 4.38405797e-05
375
- 2026-11-01 23:25:15 | INFO | TIMESTEP: 0.98469913
376
- 2026-11-01 23:25:15 | INFO | GLOBAL GRADIENT NORM: 0.22894168
377
- 2026-11-01 23:25:15 | INFO | MAX ABSOLUTE GRADIENT: 2.11334229e-02
378
- 2026-11-01 23:25:15 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
379
- 2026-11-01 23:25:15 | INFO | MISSING GRADIENT TENSORS: 0
380
- 2026-11-01 23:25:15 | INFO | GPU0 FORWARD: 2.430s
381
- 2026-11-01 23:25:15 | INFO | GPU1 FORWARD: 2.377s
382
- 2026-11-01 23:25:15 | INFO | GPU1 BACKWARD: 6.254s
383
- 2026-11-01 23:25:15 | INFO | GPU0 BACKWARD: 6.152s
384
- 2026-11-01 23:25:15 | INFO | UPDATE WALL TIME: 17.361s
385
- 2026-11-01 23:25:15 | INFO | ELAPSED TRAINING TIME: 308.80s
386
- 2026-11-01 23:25:15 | INFO | STEP 18 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
387
- 2026-11-01 23:25:15 | INFO | STEP 18 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
388
- 2026-11-01 23:25:32 | INFO | --------------------------------------------------------------------------------
389
- 2026-11-01 23:25:32 | INFO | STEP 19/70 | epoch=3/10 | position=5/7 | dataset_item=5
390
- 2026-11-01 23:25:32 | INFO | DATASET FILE: 00000005.pt
391
- 2026-11-01 23:25:32 | INFO | LOSS: 0.60971481
392
- 2026-11-01 23:25:32 | INFO | LR: 4.34782609e-05
393
- 2026-11-01 23:25:32 | INFO | TIMESTEP: 0.60690606
394
- 2026-11-01 23:25:32 | INFO | GLOBAL GRADIENT NORM: 0.07898372
395
- 2026-11-01 23:25:32 | INFO | MAX ABSOLUTE GRADIENT: 3.92532349e-03
396
- 2026-11-01 23:25:32 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
397
- 2026-11-01 23:25:32 | INFO | MISSING GRADIENT TENSORS: 0
398
- 2026-11-01 23:25:32 | INFO | GPU0 FORWARD: 2.528s
399
- 2026-11-01 23:25:32 | INFO | GPU1 FORWARD: 2.436s
400
- 2026-11-01 23:25:32 | INFO | GPU1 BACKWARD: 6.354s
401
- 2026-11-01 23:25:32 | INFO | GPU0 BACKWARD: 6.257s
402
- 2026-11-01 23:25:32 | INFO | UPDATE WALL TIME: 17.720s
403
- 2026-11-01 23:25:32 | INFO | ELAPSED TRAINING TIME: 326.54s
404
- 2026-11-01 23:25:32 | INFO | STEP 19 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
405
- 2026-11-01 23:25:32 | INFO | STEP 19 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
406
- 2026-11-01 23:25:50 | INFO | --------------------------------------------------------------------------------
407
- 2026-11-01 23:25:50 | INFO | STEP 20/70 | epoch=3/10 | position=6/7 | dataset_item=6
408
- 2026-11-01 23:25:50 | INFO | DATASET FILE: 00000006.pt
409
- 2026-11-01 23:25:50 | INFO | LOSS: 0.68332738
410
- 2026-11-01 23:25:50 | INFO | LR: 4.31159420e-05
411
- 2026-11-01 23:25:50 | INFO | TIMESTEP: 0.59872955
412
- 2026-11-01 23:25:50 | INFO | GLOBAL GRADIENT NORM: 0.08477836
413
- 2026-11-01 23:25:50 | INFO | MAX ABSOLUTE GRADIENT: 2.14004517e-03
414
- 2026-11-01 23:25:50 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
415
- 2026-11-01 23:25:50 | INFO | MISSING GRADIENT TENSORS: 0
416
- 2026-11-01 23:25:50 | INFO | GPU0 FORWARD: 2.487s
417
- 2026-11-01 23:25:50 | INFO | GPU1 FORWARD: 2.406s
418
- 2026-11-01 23:25:50 | INFO | GPU1 BACKWARD: 6.320s
419
- 2026-11-01 23:25:50 | INFO | GPU0 BACKWARD: 6.288s
420
- 2026-11-01 23:25:50 | INFO | UPDATE WALL TIME: 17.649s
421
- 2026-11-01 23:25:50 | INFO | ELAPSED TRAINING TIME: 344.20s
422
- 2026-11-01 23:25:50 | INFO | STEP 20 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
423
- 2026-11-01 23:25:50 | INFO | STEP 20 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
424
- 2026-11-01 23:26:08 | INFO | --------------------------------------------------------------------------------
425
- 2026-11-01 23:26:08 | INFO | STEP 21/70 | epoch=3/10 | position=7/7 | dataset_item=0
426
- 2026-11-01 23:26:08 | INFO | DATASET FILE: 00000000.pt
427
- 2026-11-01 23:26:08 | INFO | LOSS: 1.02696204
428
- 2026-11-01 23:26:08 | INFO | LR: 4.27536232e-05
429
- 2026-11-01 23:26:08 | INFO | TIMESTEP: 0.32843283
430
- 2026-11-01 23:26:08 | INFO | GLOBAL GRADIENT NORM: 0.13469819
431
- 2026-11-01 23:26:08 | INFO | MAX ABSOLUTE GRADIENT: 5.49697876e-03
432
- 2026-11-01 23:26:08 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
433
- 2026-11-01 23:26:08 | INFO | MISSING GRADIENT TENSORS: 0
434
- 2026-11-01 23:26:08 | INFO | GPU0 FORWARD: 2.502s
435
- 2026-11-01 23:26:08 | INFO | GPU1 FORWARD: 2.368s
436
- 2026-11-01 23:26:08 | INFO | GPU1 BACKWARD: 6.205s
437
- 2026-11-01 23:26:08 | INFO | GPU0 BACKWARD: 6.243s
438
- 2026-11-01 23:26:08 | INFO | UPDATE WALL TIME: 17.487s
439
- 2026-11-01 23:26:08 | INFO | ELAPSED TRAINING TIME: 361.70s
440
- 2026-11-01 23:26:08 | INFO | STEP 21 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
441
- 2026-11-01 23:26:08 | INFO | STEP 21 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
442
- 2026-11-01 23:26:25 | INFO | --------------------------------------------------------------------------------
443
- 2026-11-01 23:26:25 | INFO | STEP 22/70 | epoch=4/10 | position=1/7 | dataset_item=2
444
- 2026-11-01 23:26:25 | INFO | DATASET FILE: 00000002.pt
445
- 2026-11-01 23:26:25 | INFO | LOSS: 0.73557556
446
- 2026-11-01 23:26:25 | INFO | LR: 4.23913043e-05
447
- 2026-11-01 23:26:25 | INFO | TIMESTEP: 0.55734289
448
- 2026-11-01 23:26:25 | INFO | GLOBAL GRADIENT NORM: 0.08130765
449
- 2026-11-01 23:26:25 | INFO | MAX ABSOLUTE GRADIENT: 2.48908997e-03
450
- 2026-11-01 23:26:25 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
451
- 2026-11-01 23:26:25 | INFO | MISSING GRADIENT TENSORS: 0
452
- 2026-11-01 23:26:25 | INFO | GPU0 FORWARD: 2.486s
453
- 2026-11-01 23:26:25 | INFO | GPU1 FORWARD: 2.420s
454
- 2026-11-01 23:26:25 | INFO | GPU1 BACKWARD: 6.312s
455
- 2026-11-01 23:26:25 | INFO | GPU0 BACKWARD: 6.133s
456
- 2026-11-01 23:26:25 | INFO | UPDATE WALL TIME: 17.497s
457
- 2026-11-01 23:26:25 | INFO | ELAPSED TRAINING TIME: 379.20s
458
- 2026-11-01 23:26:25 | INFO | STEP 22 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
459
- 2026-11-01 23:26:25 | INFO | STEP 22 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
460
- 2026-11-01 23:26:43 | INFO | --------------------------------------------------------------------------------
461
- 2026-11-01 23:26:43 | INFO | STEP 23/70 | epoch=4/10 | position=2/7 | dataset_item=5
462
- 2026-11-01 23:26:43 | INFO | DATASET FILE: 00000005.pt
463
- 2026-11-01 23:26:43 | INFO | LOSS: 0.55721855
464
- 2026-11-01 23:26:43 | INFO | LR: 4.20289855e-05
465
- 2026-11-01 23:26:43 | INFO | TIMESTEP: 0.67500418
466
- 2026-11-01 23:26:43 | INFO | GLOBAL GRADIENT NORM: 0.08756581
467
- 2026-11-01 23:26:43 | INFO | MAX ABSOLUTE GRADIENT: 1.85298920e-03
468
- 2026-11-01 23:26:43 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
469
- 2026-11-01 23:26:43 | INFO | MISSING GRADIENT TENSORS: 0
470
- 2026-11-01 23:26:43 | INFO | GPU0 FORWARD: 2.458s
471
- 2026-11-01 23:26:43 | INFO | GPU1 FORWARD: 2.421s
472
- 2026-11-01 23:26:43 | INFO | GPU1 BACKWARD: 6.387s
473
- 2026-11-01 23:26:43 | INFO | GPU0 BACKWARD: 6.245s
474
- 2026-11-01 23:26:43 | INFO | UPDATE WALL TIME: 17.661s
475
- 2026-11-01 23:26:43 | INFO | ELAPSED TRAINING TIME: 396.88s
476
- 2026-11-01 23:26:43 | INFO | STEP 23 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
477
- 2026-11-01 23:26:43 | INFO | STEP 23 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
478
- 2026-11-01 23:27:00 | INFO | --------------------------------------------------------------------------------
479
- 2026-11-01 23:27:00 | INFO | STEP 24/70 | epoch=4/10 | position=3/7 | dataset_item=0
480
- 2026-11-01 23:27:00 | INFO | DATASET FILE: 00000000.pt
481
- 2026-11-01 23:27:00 | INFO | LOSS: 0.55830789
482
- 2026-11-01 23:27:00 | INFO | LR: 4.16666667e-05
483
- 2026-11-01 23:27:00 | INFO | TIMESTEP: 0.67845470
484
- 2026-11-01 23:27:00 | INFO | GLOBAL GRADIENT NORM: 0.08147401
485
- 2026-11-01 23:27:00 | INFO | MAX ABSOLUTE GRADIENT: 6.52694702e-03
486
- 2026-11-01 23:27:00 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
487
- 2026-11-01 23:27:00 | INFO | MISSING GRADIENT TENSORS: 0
488
- 2026-11-01 23:27:00 | INFO | GPU0 FORWARD: 2.454s
489
- 2026-11-01 23:27:00 | INFO | GPU1 FORWARD: 2.386s
490
- 2026-11-01 23:27:00 | INFO | GPU1 BACKWARD: 6.216s
491
- 2026-11-01 23:27:00 | INFO | GPU0 BACKWARD: 6.232s
492
- 2026-11-01 23:27:00 | INFO | UPDATE WALL TIME: 17.432s
493
- 2026-11-01 23:27:00 | INFO | ELAPSED TRAINING TIME: 414.32s
494
- 2026-11-01 23:27:00 | INFO | STEP 24 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
495
- 2026-11-01 23:27:00 | INFO | STEP 24 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
496
- 2026-11-01 23:27:18 | INFO | --------------------------------------------------------------------------------
497
- 2026-11-01 23:27:18 | INFO | STEP 25/70 | epoch=4/10 | position=4/7 | dataset_item=3
498
- 2026-11-01 23:27:18 | INFO | DATASET FILE: 00000003.pt
499
- 2026-11-01 23:27:18 | INFO | LOSS: 0.53031009
500
- 2026-11-01 23:27:18 | INFO | LR: 4.13043478e-05
501
- 2026-11-01 23:27:18 | INFO | TIMESTEP: 0.74678826
502
- 2026-11-01 23:27:18 | INFO | GLOBAL GRADIENT NORM: 0.07190437
503
- 2026-11-01 23:27:18 | INFO | MAX ABSOLUTE GRADIENT: 4.97055054e-03
504
- 2026-11-01 23:27:18 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
505
- 2026-11-01 23:27:18 | INFO | MISSING GRADIENT TENSORS: 0
506
- 2026-11-01 23:27:18 | INFO | GPU0 FORWARD: 2.522s
507
- 2026-11-01 23:27:18 | INFO | GPU1 FORWARD: 2.390s
508
- 2026-11-01 23:27:18 | INFO | GPU1 BACKWARD: 6.218s
509
- 2026-11-01 23:27:18 | INFO | GPU0 BACKWARD: 6.223s
510
- 2026-11-01 23:27:18 | INFO | UPDATE WALL TIME: 17.495s
511
- 2026-11-01 23:27:18 | INFO | ELAPSED TRAINING TIME: 431.83s
512
- 2026-11-01 23:27:18 | INFO | STEP 25 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
513
- 2026-11-01 23:27:18 | INFO | STEP 25 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
514
- 2026-11-01 23:27:35 | INFO | --------------------------------------------------------------------------------
515
- 2026-11-01 23:27:35 | INFO | STEP 26/70 | epoch=4/10 | position=5/7 | dataset_item=6
516
- 2026-11-01 23:27:35 | INFO | DATASET FILE: 00000006.pt
517
- 2026-11-01 23:27:35 | INFO | LOSS: 1.11747813
518
- 2026-11-01 23:27:35 | INFO | LR: 4.09420290e-05
519
- 2026-11-01 23:27:35 | INFO | TIMESTEP: 0.05546549
520
- 2026-11-01 23:27:35 | INFO | GLOBAL GRADIENT NORM: 0.25360535
521
- 2026-11-01 23:27:35 | INFO | MAX ABSOLUTE GRADIENT: 1.23672485e-02
522
- 2026-11-01 23:27:35 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
523
- 2026-11-01 23:27:35 | INFO | MISSING GRADIENT TENSORS: 0
524
- 2026-11-01 23:27:35 | INFO | GPU0 FORWARD: 2.518s
525
- 2026-11-01 23:27:35 | INFO | GPU1 FORWARD: 2.429s
526
- 2026-11-01 23:27:35 | INFO | GPU1 BACKWARD: 6.286s
527
- 2026-11-01 23:27:35 | INFO | GPU0 BACKWARD: 6.234s
528
- 2026-11-01 23:27:35 | INFO | UPDATE WALL TIME: 17.618s
529
- 2026-11-01 23:27:35 | INFO | ELAPSED TRAINING TIME: 449.46s
530
- 2026-11-01 23:27:35 | INFO | STEP 26 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
531
- 2026-11-01 23:27:35 | INFO | STEP 26 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
532
- 2026-11-01 23:27:53 | INFO | --------------------------------------------------------------------------------
533
- 2026-11-01 23:27:53 | INFO | STEP 27/70 | epoch=4/10 | position=6/7 | dataset_item=1
534
- 2026-11-01 23:27:53 | INFO | DATASET FILE: 00000001.pt
535
- 2026-11-01 23:27:53 | INFO | LOSS: 0.76441371
536
- 2026-11-01 23:27:53 | INFO | LR: 4.05797101e-05
537
- 2026-11-01 23:27:53 | INFO | TIMESTEP: 0.99145281
538
- 2026-11-01 23:27:53 | INFO | GLOBAL GRADIENT NORM: 0.27154939
539
- 2026-11-01 23:27:53 | INFO | MAX ABSOLUTE GRADIENT: 3.07769775e-02
540
- 2026-11-01 23:27:53 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
541
- 2026-11-01 23:27:53 | INFO | MISSING GRADIENT TENSORS: 0
542
- 2026-11-01 23:27:53 | INFO | GPU0 FORWARD: 2.435s
543
- 2026-11-01 23:27:53 | INFO | GPU1 FORWARD: 2.364s
544
- 2026-11-01 23:27:53 | INFO | GPU1 BACKWARD: 6.230s
545
- 2026-11-01 23:27:53 | INFO | GPU0 BACKWARD: 6.115s
546
- 2026-11-01 23:27:53 | INFO | UPDATE WALL TIME: 17.290s
547
- 2026-11-01 23:27:53 | INFO | ELAPSED TRAINING TIME: 466.76s
548
- 2026-11-01 23:27:53 | INFO | STEP 27 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
549
- 2026-11-01 23:27:53 | INFO | STEP 27 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
550
- 2026-11-01 23:28:10 | INFO | --------------------------------------------------------------------------------
551
- 2026-11-01 23:28:10 | INFO | STEP 28/70 | epoch=4/10 | position=7/7 | dataset_item=4
552
- 2026-11-01 23:28:10 | INFO | DATASET FILE: 00000004.pt
553
- 2026-11-01 23:28:10 | INFO | LOSS: 0.89462411
554
- 2026-11-01 23:28:10 | INFO | LR: 4.02173913e-05
555
- 2026-11-01 23:28:10 | INFO | TIMESTEP: 0.44384053
556
- 2026-11-01 23:28:10 | INFO | GLOBAL GRADIENT NORM: 0.11380419
557
- 2026-11-01 23:28:10 | INFO | MAX ABSOLUTE GRADIENT: 4.21142578e-03
558
- 2026-11-01 23:28:10 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
559
- 2026-11-01 23:28:10 | INFO | MISSING GRADIENT TENSORS: 0
560
- 2026-11-01 23:28:10 | INFO | GPU0 FORWARD: 2.452s
561
- 2026-11-01 23:28:10 | INFO | GPU1 FORWARD: 2.425s
562
- 2026-11-01 23:28:10 | INFO | GPU1 BACKWARD: 6.343s
563
- 2026-11-01 23:28:10 | INFO | GPU0 BACKWARD: 6.241s
564
- 2026-11-01 23:28:10 | INFO | UPDATE WALL TIME: 17.607s
565
- 2026-11-01 23:28:10 | INFO | ELAPSED TRAINING TIME: 484.38s
566
- 2026-11-01 23:28:10 | INFO | STEP 28 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
567
- 2026-11-01 23:28:10 | INFO | STEP 28 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
568
- 2026-11-01 23:28:28 | INFO | --------------------------------------------------------------------------------
569
- 2026-11-01 23:28:28 | INFO | STEP 29/70 | epoch=5/10 | position=1/7 | dataset_item=6
570
- 2026-11-01 23:28:28 | INFO | DATASET FILE: 00000006.pt
571
- 2026-11-01 23:28:28 | INFO | LOSS: 1.18787646
572
- 2026-11-01 23:28:28 | INFO | LR: 3.98550725e-05
573
- 2026-11-01 23:28:28 | INFO | TIMESTEP: 0.19323446
574
- 2026-11-01 23:28:28 | INFO | GLOBAL GRADIENT NORM: 0.13397921
575
- 2026-11-01 23:28:28 | INFO | MAX ABSOLUTE GRADIENT: 5.12695312e-03
576
- 2026-11-01 23:28:28 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
577
- 2026-11-01 23:28:28 | INFO | MISSING GRADIENT TENSORS: 0
578
- 2026-11-01 23:28:28 | INFO | GPU0 FORWARD: 2.449s
579
- 2026-11-01 23:28:28 | INFO | GPU1 FORWARD: 2.399s
580
- 2026-11-01 23:28:28 | INFO | GPU1 BACKWARD: 6.248s
581
- 2026-11-01 23:28:28 | INFO | GPU0 BACKWARD: 6.220s
582
- 2026-11-01 23:28:28 | INFO | UPDATE WALL TIME: 17.461s
583
- 2026-11-01 23:28:28 | INFO | ELAPSED TRAINING TIME: 501.85s
584
- 2026-11-01 23:28:28 | INFO | STEP 29 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
585
- 2026-11-01 23:28:28 | INFO | STEP 29 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
586
- 2026-11-01 23:28:45 | INFO | --------------------------------------------------------------------------------
587
- 2026-11-01 23:28:45 | INFO | STEP 30/70 | epoch=5/10 | position=2/7 | dataset_item=3
588
- 2026-11-01 23:28:45 | INFO | DATASET FILE: 00000003.pt
589
- 2026-11-01 23:28:45 | INFO | LOSS: 0.53480107
590
- 2026-11-01 23:28:45 | INFO | LR: 3.94927536e-05
591
- 2026-11-01 23:28:45 | INFO | TIMESTEP: 0.94193894
592
- 2026-11-01 23:28:45 | INFO | GLOBAL GRADIENT NORM: 0.17610645
593
- 2026-11-01 23:28:45 | INFO | MAX ABSOLUTE GRADIENT: 1.18408203e-02
594
- 2026-11-01 23:28:45 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
595
- 2026-11-01 23:28:45 | INFO | MISSING GRADIENT TENSORS: 0
596
- 2026-11-01 23:28:45 | INFO | GPU0 FORWARD: 2.495s
597
- 2026-11-01 23:28:45 | INFO | GPU1 FORWARD: 2.423s
598
- 2026-11-01 23:28:45 | INFO | GPU1 BACKWARD: 6.340s
599
- 2026-11-01 23:28:45 | INFO | GPU0 BACKWARD: 6.265s
600
- 2026-11-01 23:28:45 | INFO | UPDATE WALL TIME: 17.674s
601
- 2026-11-01 23:28:45 | INFO | ELAPSED TRAINING TIME: 519.54s
602
- 2026-11-01 23:28:45 | INFO | STEP 30 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
603
- 2026-11-01 23:28:45 | INFO | STEP 30 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
604
- 2026-11-01 23:29:03 | INFO | --------------------------------------------------------------------------------
605
- 2026-11-01 23:29:03 | INFO | STEP 31/70 | epoch=5/10 | position=3/7 | dataset_item=5
606
- 2026-11-01 23:29:03 | INFO | DATASET FILE: 00000005.pt
607
- 2026-11-01 23:29:03 | INFO | LOSS: 1.11169529
608
- 2026-11-01 23:29:03 | INFO | LR: 3.91304348e-05
609
- 2026-11-01 23:29:03 | INFO | TIMESTEP: 0.22701375
610
- 2026-11-01 23:29:03 | INFO | GLOBAL GRADIENT NORM: 0.16362807
611
- 2026-11-01 23:29:03 | INFO | MAX ABSOLUTE GRADIENT: 1.30233765e-02
612
- 2026-11-01 23:29:03 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
613
- 2026-11-01 23:29:03 | INFO | MISSING GRADIENT TENSORS: 0
614
- 2026-11-01 23:29:03 | INFO | GPU0 FORWARD: 2.557s
615
- 2026-11-01 23:29:03 | INFO | GPU1 FORWARD: 2.429s
616
- 2026-11-01 23:29:03 | INFO | GPU1 BACKWARD: 6.369s
617
- 2026-11-01 23:29:03 | INFO | GPU0 BACKWARD: 6.334s
618
- 2026-11-01 23:29:03 | INFO | UPDATE WALL TIME: 17.838s
619
- 2026-11-01 23:29:03 | INFO | ELAPSED TRAINING TIME: 537.39s
620
- 2026-11-01 23:29:03 | INFO | STEP 31 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
621
- 2026-11-01 23:29:03 | INFO | STEP 31 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
622
- 2026-11-01 23:29:21 | INFO | --------------------------------------------------------------------------------
623
- 2026-11-01 23:29:21 | INFO | STEP 32/70 | epoch=5/10 | position=4/7 | dataset_item=1
624
- 2026-11-01 23:29:21 | INFO | DATASET FILE: 00000001.pt
625
- 2026-11-01 23:29:21 | INFO | LOSS: 1.12055850
626
- 2026-11-01 23:29:21 | INFO | LR: 3.87681159e-05
627
- 2026-11-01 23:29:21 | INFO | TIMESTEP: 0.05574556
628
- 2026-11-01 23:29:21 | INFO | GLOBAL GRADIENT NORM: 0.19391994
629
- 2026-11-01 23:29:21 | INFO | MAX ABSOLUTE GRADIENT: 1.56402588e-02
630
- 2026-11-01 23:29:21 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
631
- 2026-11-01 23:29:21 | INFO | MISSING GRADIENT TENSORS: 0
632
- 2026-11-01 23:29:21 | INFO | GPU0 FORWARD: 2.429s
633
- 2026-11-01 23:29:21 | INFO | GPU1 FORWARD: 2.391s
634
- 2026-11-01 23:29:21 | INFO | GPU1 BACKWARD: 6.190s
635
- 2026-11-01 23:29:21 | INFO | GPU0 BACKWARD: 6.234s
636
- 2026-11-01 23:29:21 | INFO | UPDATE WALL TIME: 17.395s
637
- 2026-11-01 23:29:21 | INFO | ELAPSED TRAINING TIME: 554.79s
638
- 2026-11-01 23:29:21 | INFO | STEP 32 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
639
- 2026-11-01 23:29:21 | INFO | STEP 32 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
640
- 2026-11-01 23:29:38 | INFO | --------------------------------------------------------------------------------
641
- 2026-11-01 23:29:38 | INFO | STEP 33/70 | epoch=5/10 | position=5/7 | dataset_item=0
642
- 2026-11-01 23:29:38 | INFO | DATASET FILE: 00000000.pt
643
- 2026-11-01 23:29:38 | INFO | LOSS: 1.08607912
644
- 2026-11-01 23:29:38 | INFO | LR: 3.84057971e-05
645
- 2026-11-01 23:29:38 | INFO | TIMESTEP: 0.03580014
646
- 2026-11-01 23:29:38 | INFO | GLOBAL GRADIENT NORM: 0.19619415
647
- 2026-11-01 23:29:38 | INFO | MAX ABSOLUTE GRADIENT: 1.38320923e-02
648
- 2026-11-01 23:29:38 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
649
- 2026-11-01 23:29:38 | INFO | MISSING GRADIENT TENSORS: 0
650
- 2026-11-01 23:29:38 | INFO | GPU0 FORWARD: 2.467s
651
- 2026-11-01 23:29:38 | INFO | GPU1 FORWARD: 2.357s
652
- 2026-11-01 23:29:38 | INFO | GPU1 BACKWARD: 6.107s
653
- 2026-11-01 23:29:38 | INFO | GPU0 BACKWARD: 6.194s
654
- 2026-11-01 23:29:38 | INFO | UPDATE WALL TIME: 17.279s
655
- 2026-11-01 23:29:38 | INFO | ELAPSED TRAINING TIME: 572.08s
656
- 2026-11-01 23:29:38 | INFO | STEP 33 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
657
- 2026-11-01 23:29:38 | INFO | STEP 33 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
658
- 2026-11-01 23:29:55 | INFO | --------------------------------------------------------------------------------
659
- 2026-11-01 23:29:55 | INFO | STEP 34/70 | epoch=5/10 | position=6/7 | dataset_item=2
660
- 2026-11-01 23:29:55 | INFO | DATASET FILE: 00000002.pt
661
- 2026-11-01 23:29:55 | INFO | LOSS: 0.75155067
662
- 2026-11-01 23:29:55 | INFO | LR: 3.80434783e-05
663
- 2026-11-01 23:29:55 | INFO | TIMESTEP: 0.54746217
664
- 2026-11-01 23:29:55 | INFO | GLOBAL GRADIENT NORM: 0.08184964
665
- 2026-11-01 23:29:55 | INFO | MAX ABSOLUTE GRADIENT: 5.22613525e-03
666
- 2026-11-01 23:29:55 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
667
- 2026-11-01 23:29:55 | INFO | MISSING GRADIENT TENSORS: 0
668
- 2026-11-01 23:29:55 | INFO | GPU0 FORWARD: 2.450s
669
- 2026-11-01 23:29:55 | INFO | GPU1 FORWARD: 2.401s
670
- 2026-11-01 23:29:55 | INFO | GPU1 BACKWARD: 6.268s
671
- 2026-11-01 23:29:55 | INFO | GPU0 BACKWARD: 6.187s
672
- 2026-11-01 23:29:55 | INFO | UPDATE WALL TIME: 17.457s
673
- 2026-11-01 23:29:55 | INFO | ELAPSED TRAINING TIME: 589.55s
674
- 2026-11-01 23:29:55 | INFO | STEP 34 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
675
- 2026-11-01 23:29:55 | INFO | STEP 34 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
676
- 2026-11-01 23:30:13 | INFO | --------------------------------------------------------------------------------
677
- 2026-11-01 23:30:13 | INFO | STEP 35/70 | epoch=5/10 | position=7/7 | dataset_item=4
678
- 2026-11-01 23:30:13 | INFO | DATASET FILE: 00000004.pt
679
- 2026-11-01 23:30:13 | INFO | LOSS: 0.72215509
680
- 2026-11-01 23:30:13 | INFO | LR: 3.76811594e-05
681
- 2026-11-01 23:30:13 | INFO | TIMESTEP: 0.56448138
682
- 2026-11-01 23:30:13 | INFO | GLOBAL GRADIENT NORM: 0.07382349
683
- 2026-11-01 23:30:13 | INFO | MAX ABSOLUTE GRADIENT: 5.40924072e-03
684
- 2026-11-01 23:30:13 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
685
- 2026-11-01 23:30:13 | INFO | MISSING GRADIENT TENSORS: 0
686
- 2026-11-01 23:30:13 | INFO | GPU0 FORWARD: 2.469s
687
- 2026-11-01 23:30:13 | INFO | GPU1 FORWARD: 2.401s
688
- 2026-11-01 23:30:13 | INFO | GPU1 BACKWARD: 6.229s
689
- 2026-11-01 23:30:13 | INFO | GPU0 BACKWARD: 6.201s
690
- 2026-11-01 23:30:13 | INFO | UPDATE WALL TIME: 17.448s
691
- 2026-11-01 23:30:13 | INFO | ELAPSED TRAINING TIME: 607.01s
692
- 2026-11-01 23:30:13 | INFO | STEP 35 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
693
- 2026-11-01 23:30:13 | INFO | STEP 35 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
694
- 2026-11-01 23:30:13 | INFO | ================================================================================
695
- 2026-11-01 23:30:13 | INFO | CHECKPOINT TRIGGERED AT OPTIMIZER STEP 35
696
- 2026-11-01 23:30:17 | INFO | CHECKPOINT SUCCESS: kromoss_lora_step_000035.safetensors
697
- 2026-11-01 23:30:17 | INFO | CHECKPOINT SIZE: 70.93 MB
698
- 2026-11-01 23:30:17 | INFO | ================================================================================
699
- 2026-11-01 23:30:34 | INFO | --------------------------------------------------------------------------------
700
- 2026-11-01 23:30:34 | INFO | STEP 36/70 | epoch=6/10 | position=1/7 | dataset_item=5
701
- 2026-11-01 23:30:34 | INFO | DATASET FILE: 00000005.pt
702
- 2026-11-01 23:30:34 | INFO | LOSS: 1.09069192
703
- 2026-11-01 23:30:34 | INFO | LR: 3.73188406e-05
704
- 2026-11-01 23:30:34 | INFO | TIMESTEP: 0.03447602
705
- 2026-11-01 23:30:34 | INFO | GLOBAL GRADIENT NORM: 0.24189210
706
- 2026-11-01 23:30:34 | INFO | MAX ABSOLUTE GRADIENT: 1.19171143e-02
707
- 2026-11-01 23:30:34 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
708
- 2026-11-01 23:30:34 | INFO | MISSING GRADIENT TENSORS: 0
709
- 2026-11-01 23:30:34 | INFO | GPU0 FORWARD: 2.529s
710
- 2026-11-01 23:30:34 | INFO | GPU1 FORWARD: 2.428s
711
- 2026-11-01 23:30:34 | INFO | GPU1 BACKWARD: 6.350s
712
- 2026-11-01 23:30:34 | INFO | GPU0 BACKWARD: 6.344s
713
- 2026-11-01 23:30:34 | INFO | UPDATE WALL TIME: 17.801s
714
- 2026-11-01 23:30:34 | INFO | ELAPSED TRAINING TIME: 628.56s
715
- 2026-11-01 23:30:34 | INFO | STEP 36 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
716
- 2026-11-01 23:30:34 | INFO | STEP 36 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
717
- 2026-11-01 23:30:52 | INFO | --------------------------------------------------------------------------------
718
- 2026-11-01 23:30:52 | INFO | STEP 37/70 | epoch=6/10 | position=2/7 | dataset_item=0
719
- 2026-11-01 23:30:52 | INFO | DATASET FILE: 00000000.pt
720
- 2026-11-01 23:30:52 | INFO | LOSS: 0.48794767
721
- 2026-11-01 23:30:52 | INFO | LR: 3.69565217e-05
722
- 2026-11-01 23:30:52 | INFO | TIMESTEP: 0.84535515
723
- 2026-11-01 23:30:52 | INFO | GLOBAL GRADIENT NORM: 0.11927136
724
- 2026-11-01 23:30:52 | INFO | MAX ABSOLUTE GRADIENT: 7.75909424e-03
725
- 2026-11-01 23:30:52 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
726
- 2026-11-01 23:30:52 | INFO | MISSING GRADIENT TENSORS: 0
727
- 2026-11-01 23:30:52 | INFO | GPU0 FORWARD: 2.465s
728
- 2026-11-01 23:30:52 | INFO | GPU1 FORWARD: 2.395s
729
- 2026-11-01 23:30:52 | INFO | GPU1 BACKWARD: 6.345s
730
- 2026-11-01 23:30:52 | INFO | GPU0 BACKWARD: 6.203s
731
- 2026-11-01 23:30:52 | INFO | UPDATE WALL TIME: 17.553s
732
- 2026-11-01 23:30:52 | INFO | ELAPSED TRAINING TIME: 646.12s
733
- 2026-11-01 23:30:52 | INFO | STEP 37 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
734
- 2026-11-01 23:30:52 | INFO | STEP 37 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
735
- 2026-11-01 23:31:10 | INFO | --------------------------------------------------------------------------------
736
- 2026-11-01 23:31:10 | INFO | STEP 38/70 | epoch=6/10 | position=3/7 | dataset_item=3
737
- 2026-11-01 23:31:10 | INFO | DATASET FILE: 00000003.pt
738
- 2026-11-01 23:31:10 | INFO | LOSS: 0.96387875
739
- 2026-11-01 23:31:10 | INFO | LR: 3.65942029e-05
740
- 2026-11-01 23:31:10 | INFO | TIMESTEP: 0.39176139
741
- 2026-11-01 23:31:10 | INFO | GLOBAL GRADIENT NORM: 0.10431517
742
- 2026-11-01 23:31:10 | INFO | MAX ABSOLUTE GRADIENT: 3.28636169e-03
743
- 2026-11-01 23:31:10 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
744
- 2026-11-01 23:31:10 | INFO | MISSING GRADIENT TENSORS: 0
745
- 2026-11-01 23:31:10 | INFO | GPU0 FORWARD: 2.502s
746
- 2026-11-01 23:31:10 | INFO | GPU1 FORWARD: 2.404s
747
- 2026-11-01 23:31:10 | INFO | GPU1 BACKWARD: 6.275s
748
- 2026-11-01 23:31:10 | INFO | GPU0 BACKWARD: 6.248s
749
- 2026-11-01 23:31:10 | INFO | UPDATE WALL TIME: 17.579s
750
- 2026-11-01 23:31:10 | INFO | ELAPSED TRAINING TIME: 663.72s
751
- 2026-11-01 23:31:10 | INFO | STEP 38 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
752
- 2026-11-01 23:31:10 | INFO | STEP 38 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
753
- 2026-11-01 23:31:27 | INFO | --------------------------------------------------------------------------------
754
- 2026-11-01 23:31:27 | INFO | STEP 39/70 | epoch=6/10 | position=4/7 | dataset_item=4
755
- 2026-11-01 23:31:27 | INFO | DATASET FILE: 00000004.pt
756
- 2026-11-01 23:31:27 | INFO | LOSS: 0.82819331
757
- 2026-11-01 23:31:27 | INFO | LR: 3.62318841e-05
758
- 2026-11-01 23:31:27 | INFO | TIMESTEP: 0.48779073
759
- 2026-11-01 23:31:27 | INFO | GLOBAL GRADIENT NORM: 0.08383896
760
- 2026-11-01 23:31:27 | INFO | MAX ABSOLUTE GRADIENT: 3.06701660e-03
761
- 2026-11-01 23:31:27 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
762
- 2026-11-01 23:31:27 | INFO | MISSING GRADIENT TENSORS: 0
763
- 2026-11-01 23:31:27 | INFO | GPU0 FORWARD: 2.474s
764
- 2026-11-01 23:31:27 | INFO | GPU1 FORWARD: 2.404s
765
- 2026-11-01 23:31:27 | INFO | GPU1 BACKWARD: 6.283s
766
- 2026-11-01 23:31:27 | INFO | GPU0 BACKWARD: 6.236s
767
- 2026-11-01 23:31:27 | INFO | UPDATE WALL TIME: 17.549s
768
- 2026-11-01 23:31:27 | INFO | ELAPSED TRAINING TIME: 681.28s
769
- 2026-11-01 23:31:27 | INFO | STEP 39 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
770
- 2026-11-01 23:31:27 | INFO | STEP 39 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
771
- 2026-11-01 23:31:44 | INFO | --------------------------------------------------------------------------------
772
- 2026-11-01 23:31:44 | INFO | STEP 40/70 | epoch=6/10 | position=5/7 | dataset_item=1
773
- 2026-11-01 23:31:44 | INFO | DATASET FILE: 00000001.pt
774
- 2026-11-01 23:31:44 | INFO | LOSS: 0.56365967
775
- 2026-11-01 23:31:44 | INFO | LR: 3.58695652e-05
776
- 2026-11-01 23:31:44 | INFO | TIMESTEP: 0.72132856
777
- 2026-11-01 23:31:44 | INFO | GLOBAL GRADIENT NORM: 0.08286749
778
- 2026-11-01 23:31:44 | INFO | MAX ABSOLUTE GRADIENT: 4.44030762e-03
779
- 2026-11-01 23:31:44 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
780
- 2026-11-01 23:31:44 | INFO | MISSING GRADIENT TENSORS: 0
781
- 2026-11-01 23:31:44 | INFO | GPU0 FORWARD: 2.438s
782
- 2026-11-01 23:31:44 | INFO | GPU1 FORWARD: 2.358s
783
- 2026-11-01 23:31:44 | INFO | GPU1 BACKWARD: 6.180s
784
- 2026-11-01 23:31:44 | INFO | GPU0 BACKWARD: 6.156s
785
- 2026-11-01 23:31:44 | INFO | UPDATE WALL TIME: 17.279s
786
- 2026-11-01 23:31:44 | INFO | ELAPSED TRAINING TIME: 698.57s
787
- 2026-11-01 23:31:44 | INFO | STEP 40 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
788
- 2026-11-01 23:31:44 | INFO | STEP 40 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
789
- 2026-11-01 23:32:02 | INFO | --------------------------------------------------------------------------------
790
- 2026-11-01 23:32:02 | INFO | STEP 41/70 | epoch=6/10 | position=6/7 | dataset_item=6
791
- 2026-11-01 23:32:02 | INFO | DATASET FILE: 00000006.pt
792
- 2026-11-01 23:32:02 | INFO | LOSS: 1.13808560
793
- 2026-11-01 23:32:02 | INFO | LR: 3.55072464e-05
794
- 2026-11-01 23:32:02 | INFO | TIMESTEP: 0.06130258
795
- 2026-11-01 23:32:02 | INFO | GLOBAL GRADIENT NORM: 0.29548634
796
- 2026-11-01 23:32:02 | INFO | MAX ABSOLUTE GRADIENT: 1.49993896e-02
797
- 2026-11-01 23:32:02 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
798
- 2026-11-01 23:32:02 | INFO | MISSING GRADIENT TENSORS: 0
799
- 2026-11-01 23:32:02 | INFO | GPU0 FORWARD: 2.493s
800
- 2026-11-01 23:32:02 | INFO | GPU1 FORWARD: 2.401s
801
- 2026-11-01 23:32:02 | INFO | GPU1 BACKWARD: 6.259s
802
- 2026-11-01 23:32:02 | INFO | GPU0 BACKWARD: 6.314s
803
- 2026-11-01 23:32:02 | INFO | UPDATE WALL TIME: 17.613s
804
- 2026-11-01 23:32:02 | INFO | ELAPSED TRAINING TIME: 716.19s
805
- 2026-11-01 23:32:02 | INFO | STEP 41 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
806
- 2026-11-01 23:32:02 | INFO | STEP 41 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
807
- 2026-11-01 23:32:20 | INFO | --------------------------------------------------------------------------------
808
- 2026-11-01 23:32:20 | INFO | STEP 42/70 | epoch=6/10 | position=7/7 | dataset_item=2
809
- 2026-11-01 23:32:20 | INFO | DATASET FILE: 00000002.pt
810
- 2026-11-01 23:32:20 | INFO | LOSS: 1.17065358
811
- 2026-11-01 23:32:20 | INFO | LR: 3.51449275e-05
812
- 2026-11-01 23:32:20 | INFO | TIMESTEP: 0.26449913
813
- 2026-11-01 23:32:20 | INFO | GLOBAL GRADIENT NORM: 0.14144331
814
- 2026-11-01 23:32:20 | INFO | MAX ABSOLUTE GRADIENT: 4.80651855e-03
815
- 2026-11-01 23:32:20 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
816
- 2026-11-01 23:32:20 | INFO | MISSING GRADIENT TENSORS: 0
817
- 2026-11-01 23:32:20 | INFO | GPU0 FORWARD: 2.432s
818
- 2026-11-01 23:32:20 | INFO | GPU1 FORWARD: 2.415s
819
- 2026-11-01 23:32:20 | INFO | GPU1 BACKWARD: 6.246s
820
- 2026-11-01 23:32:20 | INFO | GPU0 BACKWARD: 6.173s
821
- 2026-11-01 23:32:20 | INFO | UPDATE WALL TIME: 17.410s
822
- 2026-11-01 23:32:20 | INFO | ELAPSED TRAINING TIME: 733.61s
823
- 2026-11-01 23:32:20 | INFO | STEP 42 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
824
- 2026-11-01 23:32:20 | INFO | STEP 42 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
825
- 2026-11-01 23:32:37 | INFO | --------------------------------------------------------------------------------
826
- 2026-11-01 23:32:37 | INFO | STEP 43/70 | epoch=7/10 | position=1/7 | dataset_item=4
827
- 2026-11-01 23:32:37 | INFO | DATASET FILE: 00000004.pt
828
- 2026-11-01 23:32:37 | INFO | LOSS: 1.11446595
829
- 2026-11-01 23:32:37 | INFO | LR: 3.47826087e-05
830
- 2026-11-01 23:32:37 | INFO | TIMESTEP: 0.29757562
831
- 2026-11-01 23:32:37 | INFO | GLOBAL GRADIENT NORM: 0.10815306
832
- 2026-11-01 23:32:37 | INFO | MAX ABSOLUTE GRADIENT: 5.15747070e-03
833
- 2026-11-01 23:32:37 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
834
- 2026-11-01 23:32:37 | INFO | MISSING GRADIENT TENSORS: 0
835
- 2026-11-01 23:32:37 | INFO | GPU0 FORWARD: 2.473s
836
- 2026-11-01 23:32:37 | INFO | GPU1 FORWARD: 2.380s
837
- 2026-11-01 23:32:37 | INFO | GPU1 BACKWARD: 6.260s
838
- 2026-11-01 23:32:37 | INFO | GPU0 BACKWARD: 6.292s
839
- 2026-11-01 23:32:37 | INFO | UPDATE WALL TIME: 17.552s
840
- 2026-11-01 23:32:37 | INFO | ELAPSED TRAINING TIME: 751.17s
841
- 2026-11-01 23:32:37 | INFO | STEP 43 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
842
- 2026-11-01 23:32:37 | INFO | STEP 43 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
843
- 2026-11-01 23:32:55 | INFO | --------------------------------------------------------------------------------
844
- 2026-11-01 23:32:55 | INFO | STEP 44/70 | epoch=7/10 | position=2/7 | dataset_item=5
845
- 2026-11-01 23:32:55 | INFO | DATASET FILE: 00000005.pt
846
- 2026-11-01 23:32:55 | INFO | LOSS: 0.60464799
847
- 2026-11-01 23:32:55 | INFO | LR: 3.44202899e-05
848
- 2026-11-01 23:32:55 | INFO | TIMESTEP: 0.61395168
849
- 2026-11-01 23:32:55 | INFO | GLOBAL GRADIENT NORM: 0.21718825
850
- 2026-11-01 23:32:55 | INFO | MAX ABSOLUTE GRADIENT: 1.10473633e-02
851
- 2026-11-01 23:32:55 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
852
- 2026-11-01 23:32:55 | INFO | MISSING GRADIENT TENSORS: 0
853
- 2026-11-01 23:32:55 | INFO | GPU0 FORWARD: 2.543s
854
- 2026-11-01 23:32:55 | INFO | GPU1 FORWARD: 2.445s
855
- 2026-11-01 23:32:55 | INFO | GPU1 BACKWARD: 6.391s
856
- 2026-11-01 23:32:55 | INFO | GPU0 BACKWARD: 6.278s
857
- 2026-11-01 23:32:55 | INFO | UPDATE WALL TIME: 17.806s
858
- 2026-11-01 23:32:55 | INFO | ELAPSED TRAINING TIME: 768.99s
859
- 2026-11-01 23:32:55 | INFO | STEP 44 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
860
- 2026-11-01 23:32:55 | INFO | STEP 44 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
861
- 2026-11-01 23:33:12 | INFO | --------------------------------------------------------------------------------
862
- 2026-11-01 23:33:12 | INFO | STEP 45/70 | epoch=7/10 | position=3/7 | dataset_item=1
863
- 2026-11-01 23:33:12 | INFO | DATASET FILE: 00000001.pt
864
- 2026-11-01 23:33:12 | INFO | LOSS: 0.59427971
865
- 2026-11-01 23:33:12 | INFO | LR: 3.40579710e-05
866
- 2026-11-01 23:33:12 | INFO | TIMESTEP: 0.67698693
867
- 2026-11-01 23:33:12 | INFO | GLOBAL GRADIENT NORM: 0.06830193
868
- 2026-11-01 23:33:12 | INFO | MAX ABSOLUTE GRADIENT: 3.22151184e-03
869
- 2026-11-01 23:33:12 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
870
- 2026-11-01 23:33:12 | INFO | MISSING GRADIENT TENSORS: 0
871
- 2026-11-01 23:33:12 | INFO | GPU0 FORWARD: 2.446s
872
- 2026-11-01 23:33:12 | INFO | GPU1 FORWARD: 2.363s
873
- 2026-11-01 23:33:12 | INFO | GPU1 BACKWARD: 6.226s
874
- 2026-11-01 23:33:12 | INFO | GPU0 BACKWARD: 6.185s
875
- 2026-11-01 23:33:12 | INFO | UPDATE WALL TIME: 17.365s
876
- 2026-11-01 23:33:12 | INFO | ELAPSED TRAINING TIME: 786.37s
877
- 2026-11-01 23:33:12 | INFO | STEP 45 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
878
- 2026-11-01 23:33:12 | INFO | STEP 45 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
879
- 2026-11-01 23:33:30 | INFO | --------------------------------------------------------------------------------
880
- 2026-11-01 23:33:30 | INFO | STEP 46/70 | epoch=7/10 | position=4/7 | dataset_item=2
881
- 2026-11-01 23:33:30 | INFO | DATASET FILE: 00000002.pt
882
- 2026-11-01 23:33:30 | INFO | LOSS: 1.10766661
883
- 2026-11-01 23:33:30 | INFO | LR: 3.36956522e-05
884
- 2026-11-01 23:33:30 | INFO | TIMESTEP: 0.03403377
885
- 2026-11-01 23:33:30 | INFO | GLOBAL GRADIENT NORM: 0.26566451
886
- 2026-11-01 23:33:30 | INFO | MAX ABSOLUTE GRADIENT: 2.34985352e-02
887
- 2026-11-01 23:33:30 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
888
- 2026-11-01 23:33:30 | INFO | MISSING GRADIENT TENSORS: 0
889
- 2026-11-01 23:33:30 | INFO | GPU0 FORWARD: 2.515s
890
- 2026-11-01 23:33:30 | INFO | GPU1 FORWARD: 2.375s
891
- 2026-11-01 23:33:30 | INFO | GPU1 BACKWARD: 6.185s
892
- 2026-11-01 23:33:30 | INFO | GPU0 BACKWARD: 6.256s
893
- 2026-11-01 23:33:30 | INFO | UPDATE WALL TIME: 17.481s
894
- 2026-11-01 23:33:30 | INFO | ELAPSED TRAINING TIME: 803.86s
895
- 2026-11-01 23:33:30 | INFO | STEP 46 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
896
- 2026-11-01 23:33:30 | INFO | STEP 46 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
897
- 2026-11-01 23:33:47 | INFO | --------------------------------------------------------------------------------
898
- 2026-11-01 23:33:47 | INFO | STEP 47/70 | epoch=7/10 | position=5/7 | dataset_item=6
899
- 2026-11-01 23:33:47 | INFO | DATASET FILE: 00000006.pt
900
- 2026-11-01 23:33:47 | INFO | LOSS: 0.55366147
901
- 2026-11-01 23:33:47 | INFO | LR: 3.33333333e-05
902
- 2026-11-01 23:33:47 | INFO | TIMESTEP: 0.76543015
903
- 2026-11-01 23:33:47 | INFO | GLOBAL GRADIENT NORM: 0.09578727
904
- 2026-11-01 23:33:47 | INFO | MAX ABSOLUTE GRADIENT: 8.12530518e-03
905
- 2026-11-01 23:33:47 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
906
- 2026-11-01 23:33:47 | INFO | MISSING GRADIENT TENSORS: 0
907
- 2026-11-01 23:33:47 | INFO | GPU0 FORWARD: 2.505s
908
- 2026-11-01 23:33:47 | INFO | GPU1 FORWARD: 2.433s
909
- 2026-11-01 23:33:47 | INFO | GPU1 BACKWARD: 6.334s
910
- 2026-11-01 23:33:47 | INFO | GPU0 BACKWARD: 6.230s
911
- 2026-11-01 23:33:47 | INFO | UPDATE WALL TIME: 17.664s
912
- 2026-11-01 23:33:47 | INFO | ELAPSED TRAINING TIME: 821.54s
913
- 2026-11-01 23:33:47 | INFO | STEP 47 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
914
- 2026-11-01 23:33:47 | INFO | STEP 47 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
915
- 2026-11-01 23:34:05 | INFO | --------------------------------------------------------------------------------
916
- 2026-11-01 23:34:05 | INFO | STEP 48/70 | epoch=7/10 | position=6/7 | dataset_item=3
917
- 2026-11-01 23:34:05 | INFO | DATASET FILE: 00000003.pt
918
- 2026-11-01 23:34:05 | INFO | LOSS: 0.49302477
919
- 2026-11-01 23:34:05 | INFO | LR: 3.29710145e-05
920
- 2026-11-01 23:34:05 | INFO | TIMESTEP: 0.83247983
921
- 2026-11-01 23:34:05 | INFO | GLOBAL GRADIENT NORM: 0.10111229
922
- 2026-11-01 23:34:05 | INFO | MAX ABSOLUTE GRADIENT: 4.60815430e-03
923
- 2026-11-01 23:34:05 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
924
- 2026-11-01 23:34:05 | INFO | MISSING GRADIENT TENSORS: 0
925
- 2026-11-01 23:34:05 | INFO | GPU0 FORWARD: 2.461s
926
- 2026-11-01 23:34:05 | INFO | GPU1 FORWARD: 2.403s
927
- 2026-11-01 23:34:05 | INFO | GPU1 BACKWARD: 6.299s
928
- 2026-11-01 23:34:05 | INFO | GPU0 BACKWARD: 6.173s
929
- 2026-11-01 23:34:05 | INFO | UPDATE WALL TIME: 17.480s
930
- 2026-11-01 23:34:05 | INFO | ELAPSED TRAINING TIME: 839.03s
931
- 2026-11-01 23:34:05 | INFO | STEP 48 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
932
- 2026-11-01 23:34:05 | INFO | STEP 48 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
933
- 2026-11-01 23:34:22 | INFO | --------------------------------------------------------------------------------
934
- 2026-11-01 23:34:22 | INFO | STEP 49/70 | epoch=7/10 | position=7/7 | dataset_item=0
935
- 2026-11-01 23:34:22 | INFO | DATASET FILE: 00000000.pt
936
- 2026-11-01 23:34:22 | INFO | LOSS: 1.19202948
937
- 2026-11-01 23:34:22 | INFO | LR: 3.26086957e-05
938
- 2026-11-01 23:34:22 | INFO | TIMESTEP: 0.11672945
939
- 2026-11-01 23:34:22 | INFO | GLOBAL GRADIENT NORM: 0.22136453
940
- 2026-11-01 23:34:22 | INFO | MAX ABSOLUTE GRADIENT: 1.06353760e-02
941
- 2026-11-01 23:34:22 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
942
- 2026-11-01 23:34:22 | INFO | MISSING GRADIENT TENSORS: 0
943
- 2026-11-01 23:34:22 | INFO | GPU0 FORWARD: 2.447s
944
- 2026-11-01 23:34:22 | INFO | GPU1 FORWARD: 2.405s
945
- 2026-11-01 23:34:22 | INFO | GPU1 BACKWARD: 6.186s
946
- 2026-11-01 23:34:22 | INFO | GPU0 BACKWARD: 6.269s
947
- 2026-11-01 23:34:22 | INFO | UPDATE WALL TIME: 17.453s
948
- 2026-11-01 23:34:22 | INFO | ELAPSED TRAINING TIME: 856.50s
949
- 2026-11-01 23:34:22 | INFO | STEP 49 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
950
- 2026-11-01 23:34:22 | INFO | STEP 49 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
951
- 2026-11-01 23:34:40 | INFO | --------------------------------------------------------------------------------
952
- 2026-11-01 23:34:40 | INFO | STEP 50/70 | epoch=8/10 | position=1/7 | dataset_item=2
953
- 2026-11-01 23:34:40 | INFO | DATASET FILE: 00000002.pt
954
- 2026-11-01 23:34:40 | INFO | LOSS: 1.08855247
955
- 2026-11-01 23:34:40 | INFO | LR: 3.22463768e-05
956
- 2026-11-01 23:34:40 | INFO | TIMESTEP: 0.33134848
957
- 2026-11-01 23:34:40 | INFO | GLOBAL GRADIENT NORM: 0.09441752
958
- 2026-11-01 23:34:40 | INFO | MAX ABSOLUTE GRADIENT: 7.21359253e-03
959
- 2026-11-01 23:34:40 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
960
- 2026-11-01 23:34:40 | INFO | MISSING GRADIENT TENSORS: 0
961
- 2026-11-01 23:34:40 | INFO | GPU0 FORWARD: 2.470s
962
- 2026-11-01 23:34:40 | INFO | GPU1 FORWARD: 2.372s
963
- 2026-11-01 23:34:40 | INFO | GPU1 BACKWARD: 6.200s
964
- 2026-11-01 23:34:40 | INFO | GPU0 BACKWARD: 6.261s
965
- 2026-11-01 23:34:40 | INFO | UPDATE WALL TIME: 17.456s
966
- 2026-11-01 23:34:40 | INFO | ELAPSED TRAINING TIME: 873.96s
967
- 2026-11-01 23:34:40 | INFO | STEP 50 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
968
- 2026-11-01 23:34:40 | INFO | STEP 50 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
969
- 2026-11-01 23:34:57 | INFO | --------------------------------------------------------------------------------
970
- 2026-11-01 23:34:57 | INFO | STEP 51/70 | epoch=8/10 | position=2/7 | dataset_item=4
971
- 2026-11-01 23:34:57 | INFO | DATASET FILE: 00000004.pt
972
- 2026-11-01 23:34:57 | INFO | LOSS: 0.54909748
973
- 2026-11-01 23:34:57 | INFO | LR: 3.18840580e-05
974
- 2026-11-01 23:34:57 | INFO | TIMESTEP: 0.73580515
975
- 2026-11-01 23:34:57 | INFO | GLOBAL GRADIENT NORM: 0.07353024
976
- 2026-11-01 23:34:57 | INFO | MAX ABSOLUTE GRADIENT: 4.26483154e-03
977
- 2026-11-01 23:34:57 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
978
- 2026-11-01 23:34:57 | INFO | MISSING GRADIENT TENSORS: 0
979
- 2026-11-01 23:34:57 | INFO | GPU0 FORWARD: 2.498s
980
- 2026-11-01 23:34:57 | INFO | GPU1 FORWARD: 2.380s
981
- 2026-11-01 23:34:57 | INFO | GPU1 BACKWARD: 6.299s
982
- 2026-11-01 23:34:57 | INFO | GPU0 BACKWARD: 6.186s
983
- 2026-11-01 23:34:57 | INFO | UPDATE WALL TIME: 17.508s
984
- 2026-11-01 23:34:57 | INFO | ELAPSED TRAINING TIME: 891.48s
985
- 2026-11-01 23:34:57 | INFO | STEP 51 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
986
- 2026-11-01 23:34:57 | INFO | STEP 51 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
987
- 2026-11-01 23:35:15 | INFO | --------------------------------------------------------------------------------
988
- 2026-11-01 23:35:15 | INFO | STEP 52/70 | epoch=8/10 | position=3/7 | dataset_item=1
989
- 2026-11-01 23:35:15 | INFO | DATASET FILE: 00000001.pt
990
- 2026-11-01 23:35:15 | INFO | LOSS: 0.60545063
991
- 2026-11-01 23:35:15 | INFO | LR: 3.15217391e-05
992
- 2026-11-01 23:35:15 | INFO | TIMESTEP: 0.65707183
993
- 2026-11-01 23:35:15 | INFO | GLOBAL GRADIENT NORM: 0.07113345
994
- 2026-11-01 23:35:15 | INFO | MAX ABSOLUTE GRADIENT: 2.50625610e-03
995
- 2026-11-01 23:35:15 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
996
- 2026-11-01 23:35:15 | INFO | MISSING GRADIENT TENSORS: 0
997
- 2026-11-01 23:35:15 | INFO | GPU0 FORWARD: 2.433s
998
- 2026-11-01 23:35:15 | INFO | GPU1 FORWARD: 2.370s
999
- 2026-11-01 23:35:15 | INFO | GPU1 BACKWARD: 6.190s
1000
- 2026-11-01 23:35:15 | INFO | GPU0 BACKWARD: 6.235s
1001
- 2026-11-01 23:35:15 | INFO | UPDATE WALL TIME: 17.378s
1002
- 2026-11-01 23:35:15 | INFO | ELAPSED TRAINING TIME: 908.87s
1003
- 2026-11-01 23:35:15 | INFO | STEP 52 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
1004
- 2026-11-01 23:35:15 | INFO | STEP 52 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
1005
- 2026-11-01 23:35:33 | INFO | --------------------------------------------------------------------------------
1006
- 2026-11-01 23:35:33 | INFO | STEP 53/70 | epoch=8/10 | position=4/7 | dataset_item=5
1007
- 2026-11-01 23:35:33 | INFO | DATASET FILE: 00000005.pt
1008
- 2026-11-01 23:35:33 | INFO | LOSS: 1.14804661
1009
- 2026-11-01 23:35:33 | INFO | LR: 3.11594203e-05
1010
- 2026-11-01 23:35:33 | INFO | TIMESTEP: 0.03221190
1011
- 2026-11-01 23:35:33 | INFO | GLOBAL GRADIENT NORM: 0.99011529
1012
- 2026-11-01 23:35:33 | INFO | MAX ABSOLUTE GRADIENT: 7.36083984e-02
1013
- 2026-11-01 23:35:33 | INFO | GRADIENT CLIP COEFFICIENT: 0.50499119
1014
- 2026-11-01 23:35:33 | INFO | MISSING GRADIENT TENSORS: 0
1015
- 2026-11-01 23:35:33 | INFO | GPU0 FORWARD: 2.556s
1016
- 2026-11-01 23:35:33 | INFO | GPU1 FORWARD: 2.444s
1017
- 2026-11-01 23:35:33 | INFO | GPU1 BACKWARD: 6.351s
1018
- 2026-11-01 23:35:33 | INFO | GPU0 BACKWARD: 6.392s
1019
- 2026-11-01 23:35:33 | INFO | UPDATE WALL TIME: 17.899s
1020
- 2026-11-01 23:35:33 | INFO | ELAPSED TRAINING TIME: 926.78s
1021
- 2026-11-01 23:35:33 | INFO | STEP 53 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
1022
- 2026-11-01 23:35:33 | INFO | STEP 53 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
1023
- 2026-11-01 23:35:50 | INFO | --------------------------------------------------------------------------------
1024
- 2026-11-01 23:35:50 | INFO | STEP 54/70 | epoch=8/10 | position=5/7 | dataset_item=3
1025
- 2026-11-01 23:35:50 | INFO | DATASET FILE: 00000003.pt
1026
- 2026-11-01 23:35:50 | INFO | LOSS: 1.17867565
1027
- 2026-11-01 23:35:50 | INFO | LR: 3.07971014e-05
1028
- 2026-11-01 23:35:50 | INFO | TIMESTEP: 0.06505132
1029
- 2026-11-01 23:35:50 | INFO | GLOBAL GRADIENT NORM: 0.63161639
1030
- 2026-11-01 23:35:50 | INFO | MAX ABSOLUTE GRADIENT: 5.27038574e-02
1031
- 2026-11-01 23:35:50 | INFO | GRADIENT CLIP COEFFICIENT: 0.79161848
1032
- 2026-11-01 23:35:50 | INFO | MISSING GRADIENT TENSORS: 0
1033
- 2026-11-01 23:35:50 | INFO | GPU0 FORWARD: 2.469s
1034
- 2026-11-01 23:35:50 | INFO | GPU1 FORWARD: 2.417s
1035
- 2026-11-01 23:35:50 | INFO | GPU1 BACKWARD: 6.321s
1036
- 2026-11-01 23:35:50 | INFO | GPU0 BACKWARD: 6.343s
1037
- 2026-11-01 23:35:50 | INFO | UPDATE WALL TIME: 17.706s
1038
- 2026-11-01 23:35:50 | INFO | ELAPSED TRAINING TIME: 944.50s
1039
- 2026-11-01 23:35:50 | INFO | STEP 54 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
1040
- 2026-11-01 23:35:50 | INFO | STEP 54 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
1041
- 2026-11-01 23:36:08 | INFO | --------------------------------------------------------------------------------
1042
- 2026-11-01 23:36:08 | INFO | STEP 55/70 | epoch=8/10 | position=6/7 | dataset_item=0
1043
- 2026-11-01 23:36:08 | INFO | DATASET FILE: 00000000.pt
1044
- 2026-11-01 23:36:08 | INFO | LOSS: 0.51088017
1045
- 2026-11-01 23:36:08 | INFO | LR: 3.04347826e-05
1046
- 2026-11-01 23:36:08 | INFO | TIMESTEP: 0.75211012
1047
- 2026-11-01 23:36:08 | INFO | GLOBAL GRADIENT NORM: 0.10390844
1048
- 2026-11-01 23:36:08 | INFO | MAX ABSOLUTE GRADIENT: 9.74273682e-03
1049
- 2026-11-01 23:36:08 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1050
- 2026-11-01 23:36:08 | INFO | MISSING GRADIENT TENSORS: 0
1051
- 2026-11-01 23:36:08 | INFO | GPU0 FORWARD: 2.481s
1052
- 2026-11-01 23:36:08 | INFO | GPU1 FORWARD: 2.380s
1053
- 2026-11-01 23:36:08 | INFO | GPU1 BACKWARD: 6.208s
1054
- 2026-11-01 23:36:08 | INFO | GPU0 BACKWARD: 6.127s
1055
- 2026-11-01 23:36:08 | INFO | UPDATE WALL TIME: 17.345s
1056
- 2026-11-01 23:36:08 | INFO | ELAPSED TRAINING TIME: 961.86s
1057
- 2026-11-01 23:36:08 | INFO | STEP 55 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
1058
- 2026-11-01 23:36:08 | INFO | STEP 55 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
1059
- 2026-11-01 23:36:25 | INFO | --------------------------------------------------------------------------------
1060
- 2026-11-01 23:36:25 | INFO | STEP 56/70 | epoch=8/10 | position=7/7 | dataset_item=6
1061
- 2026-11-01 23:36:25 | INFO | DATASET FILE: 00000006.pt
1062
- 2026-11-01 23:36:25 | INFO | LOSS: 0.55729181
1063
- 2026-11-01 23:36:25 | INFO | LR: 3.00724638e-05
1064
- 2026-11-01 23:36:25 | INFO | TIMESTEP: 0.75565380
1065
- 2026-11-01 23:36:25 | INFO | GLOBAL GRADIENT NORM: 0.09028654
1066
- 2026-11-01 23:36:25 | INFO | MAX ABSOLUTE GRADIENT: 6.85501099e-03
1067
- 2026-11-01 23:36:25 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1068
- 2026-11-01 23:36:25 | INFO | MISSING GRADIENT TENSORS: 0
1069
- 2026-11-01 23:36:25 | INFO | GPU0 FORWARD: 2.476s
1070
- 2026-11-01 23:36:25 | INFO | GPU1 FORWARD: 2.403s
1071
- 2026-11-01 23:36:25 | INFO | GPU1 BACKWARD: 6.321s
1072
- 2026-11-01 23:36:25 | INFO | GPU0 BACKWARD: 6.275s
1073
- 2026-11-01 23:36:25 | INFO | UPDATE WALL TIME: 17.623s
1074
- 2026-11-01 23:36:25 | INFO | ELAPSED TRAINING TIME: 979.49s
1075
- 2026-11-01 23:36:25 | INFO | STEP 56 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
1076
- 2026-11-01 23:36:25 | INFO | STEP 56 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
1077
- 2026-11-01 23:36:43 | INFO | --------------------------------------------------------------------------------
1078
- 2026-11-01 23:36:43 | INFO | STEP 57/70 | epoch=9/10 | position=1/7 | dataset_item=1
1079
- 2026-11-01 23:36:43 | INFO | DATASET FILE: 00000001.pt
1080
- 2026-11-01 23:36:43 | INFO | LOSS: 1.22109509
1081
- 2026-11-01 23:36:43 | INFO | LR: 2.97101449e-05
1082
- 2026-11-01 23:36:43 | INFO | TIMESTEP: 0.06928300
1083
- 2026-11-01 23:36:43 | INFO | GLOBAL GRADIENT NORM: 0.95304904
1084
- 2026-11-01 23:36:43 | INFO | MAX ABSOLUTE GRADIENT: 6.16760254e-02
1085
- 2026-11-01 23:36:43 | INFO | GRADIENT CLIP COEFFICIENT: 0.52463143
1086
- 2026-11-01 23:36:43 | INFO | MISSING GRADIENT TENSORS: 0
1087
- 2026-11-01 23:36:43 | INFO | GPU0 FORWARD: 2.480s
1088
- 2026-11-01 23:36:43 | INFO | GPU1 FORWARD: 2.339s
1089
- 2026-11-01 23:36:43 | INFO | GPU1 BACKWARD: 6.129s
1090
- 2026-11-01 23:36:43 | INFO | GPU0 BACKWARD: 6.204s
1091
- 2026-11-01 23:36:43 | INFO | UPDATE WALL TIME: 17.301s
1092
- 2026-11-01 23:36:43 | INFO | ELAPSED TRAINING TIME: 996.80s
1093
- 2026-11-01 23:36:43 | INFO | STEP 57 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
1094
- 2026-11-01 23:36:43 | INFO | STEP 57 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
1095
- 2026-11-01 23:37:00 | INFO | --------------------------------------------------------------------------------
1096
- 2026-11-01 23:37:00 | INFO | STEP 58/70 | epoch=9/10 | position=2/7 | dataset_item=0
1097
- 2026-11-01 23:37:00 | INFO | DATASET FILE: 00000000.pt
1098
- 2026-11-01 23:37:00 | INFO | LOSS: 0.51689565
1099
- 2026-11-01 23:37:00 | INFO | LR: 2.93478261e-05
1100
- 2026-11-01 23:37:00 | INFO | TIMESTEP: 0.74342495
1101
- 2026-11-01 23:37:00 | INFO | GLOBAL GRADIENT NORM: 0.11001887
1102
- 2026-11-01 23:37:00 | INFO | MAX ABSOLUTE GRADIENT: 6.96945190e-03
1103
- 2026-11-01 23:37:00 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1104
- 2026-11-01 23:37:00 | INFO | MISSING GRADIENT TENSORS: 0
1105
- 2026-11-01 23:37:00 | INFO | GPU0 FORWARD: 2.440s
1106
- 2026-11-01 23:37:00 | INFO | GPU1 FORWARD: 2.393s
1107
- 2026-11-01 23:37:00 | INFO | GPU1 BACKWARD: 6.307s
1108
- 2026-11-01 23:37:00 | INFO | GPU0 BACKWARD: 6.185s
1109
- 2026-11-01 23:37:00 | INFO | UPDATE WALL TIME: 17.472s
1110
- 2026-11-01 23:37:00 | INFO | ELAPSED TRAINING TIME: 1014.29s
1111
- 2026-11-01 23:37:00 | INFO | STEP 58 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
1112
- 2026-11-01 23:37:00 | INFO | STEP 58 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
1113
- 2026-11-01 23:37:18 | INFO | --------------------------------------------------------------------------------
1114
- 2026-11-01 23:37:18 | INFO | STEP 59/70 | epoch=9/10 | position=3/7 | dataset_item=3
1115
- 2026-11-01 23:37:18 | INFO | DATASET FILE: 00000003.pt
1116
- 2026-11-01 23:37:18 | INFO | LOSS: 1.21733952
1117
- 2026-11-01 23:37:18 | INFO | LR: 2.89855072e-05
1118
- 2026-11-01 23:37:18 | INFO | TIMESTEP: 0.03509389
1119
- 2026-11-01 23:37:18 | INFO | GLOBAL GRADIENT NORM: 1.21723954
1120
- 2026-11-01 23:37:18 | INFO | MAX ABSOLUTE GRADIENT: 1.07666016e-01
1121
- 2026-11-01 23:37:18 | INFO | GRADIENT CLIP COEFFICIENT: 0.41076515
1122
- 2026-11-01 23:37:18 | INFO | MISSING GRADIENT TENSORS: 0
1123
- 2026-11-01 23:37:18 | INFO | GPU0 FORWARD: 2.491s
1124
- 2026-11-01 23:37:18 | INFO | GPU1 FORWARD: 2.375s
1125
- 2026-11-01 23:37:18 | INFO | GPU1 BACKWARD: 6.226s
1126
- 2026-11-01 23:37:18 | INFO | GPU0 BACKWARD: 6.308s
1127
- 2026-11-01 23:37:18 | INFO | UPDATE WALL TIME: 17.550s
1128
- 2026-11-01 23:37:18 | INFO | ELAPSED TRAINING TIME: 1031.85s
1129
- 2026-11-01 23:37:18 | INFO | STEP 59 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
1130
- 2026-11-01 23:37:18 | INFO | STEP 59 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
1131
- 2026-11-01 23:37:36 | INFO | --------------------------------------------------------------------------------
1132
- 2026-11-01 23:37:36 | INFO | STEP 60/70 | epoch=9/10 | position=4/7 | dataset_item=5
1133
- 2026-11-01 23:37:36 | INFO | DATASET FILE: 00000005.pt
1134
- 2026-11-01 23:37:36 | INFO | LOSS: 0.92453688
1135
- 2026-11-01 23:37:36 | INFO | LR: 2.86231884e-05
1136
- 2026-11-01 23:37:36 | INFO | TIMESTEP: 0.39239028
1137
- 2026-11-01 23:37:36 | INFO | GLOBAL GRADIENT NORM: 0.24530688
1138
- 2026-11-01 23:37:36 | INFO | MAX ABSOLUTE GRADIENT: 8.63647461e-03
1139
- 2026-11-01 23:37:36 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1140
- 2026-11-01 23:37:36 | INFO | MISSING GRADIENT TENSORS: 0
1141
- 2026-11-01 23:37:36 | INFO | GPU0 FORWARD: 2.542s
1142
- 2026-11-01 23:37:36 | INFO | GPU1 FORWARD: 2.431s
1143
- 2026-11-01 23:37:36 | INFO | GPU1 BACKWARD: 6.382s
1144
- 2026-11-01 23:37:36 | INFO | GPU0 BACKWARD: 6.512s
1145
- 2026-11-01 23:37:36 | INFO | UPDATE WALL TIME: 18.008s
1146
- 2026-11-01 23:37:36 | INFO | ELAPSED TRAINING TIME: 1049.87s
1147
- 2026-11-01 23:37:36 | INFO | STEP 60 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
1148
- 2026-11-01 23:37:36 | INFO | STEP 60 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
1149
- 2026-11-01 23:37:53 | INFO | --------------------------------------------------------------------------------
1150
- 2026-11-01 23:37:53 | INFO | STEP 61/70 | epoch=9/10 | position=5/7 | dataset_item=6
1151
- 2026-11-01 23:37:53 | INFO | DATASET FILE: 00000006.pt
1152
- 2026-11-01 23:37:53 | INFO | LOSS: 1.15974355
1153
- 2026-11-01 23:37:53 | INFO | LR: 2.82608696e-05
1154
- 2026-11-01 23:37:53 | INFO | TIMESTEP: 0.31237870
1155
- 2026-11-01 23:37:53 | INFO | GLOBAL GRADIENT NORM: 0.34283411
1156
- 2026-11-01 23:37:53 | INFO | MAX ABSOLUTE GRADIENT: 1.44805908e-02
1157
- 2026-11-01 23:37:53 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1158
- 2026-11-01 23:37:53 | INFO | MISSING GRADIENT TENSORS: 0
1159
- 2026-11-01 23:37:53 | INFO | GPU0 FORWARD: 2.524s
1160
- 2026-11-01 23:37:53 | INFO | GPU1 FORWARD: 2.403s
1161
- 2026-11-01 23:37:53 | INFO | GPU1 BACKWARD: 6.212s
1162
- 2026-11-01 23:37:53 | INFO | GPU0 BACKWARD: 6.232s
1163
- 2026-11-01 23:37:53 | INFO | UPDATE WALL TIME: 17.519s
1164
- 2026-11-01 23:37:53 | INFO | ELAPSED TRAINING TIME: 1067.40s
1165
- 2026-11-01 23:37:53 | INFO | STEP 61 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
1166
- 2026-11-01 23:37:53 | INFO | STEP 61 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
1167
- 2026-11-01 23:38:11 | INFO | --------------------------------------------------------------------------------
1168
- 2026-11-01 23:38:11 | INFO | STEP 62/70 | epoch=9/10 | position=6/7 | dataset_item=2
1169
- 2026-11-01 23:38:11 | INFO | DATASET FILE: 00000002.pt
1170
- 2026-11-01 23:38:11 | INFO | LOSS: 1.24041653
1171
- 2026-11-01 23:38:11 | INFO | LR: 2.78985507e-05
1172
- 2026-11-01 23:38:11 | INFO | TIMESTEP: 0.24778335
1173
- 2026-11-01 23:38:11 | INFO | GLOBAL GRADIENT NORM: 0.31334188
1174
- 2026-11-01 23:38:11 | INFO | MAX ABSOLUTE GRADIENT: 1.16882324e-02
1175
- 2026-11-01 23:38:11 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1176
- 2026-11-01 23:38:11 | INFO | MISSING GRADIENT TENSORS: 0
1177
- 2026-11-01 23:38:11 | INFO | GPU0 FORWARD: 2.425s
1178
- 2026-11-01 23:38:11 | INFO | GPU1 FORWARD: 2.392s
1179
- 2026-11-01 23:38:11 | INFO | GPU1 BACKWARD: 6.243s
1180
- 2026-11-01 23:38:11 | INFO | GPU0 BACKWARD: 6.219s
1181
- 2026-11-01 23:38:11 | INFO | UPDATE WALL TIME: 17.430s
1182
- 2026-11-01 23:38:11 | INFO | ELAPSED TRAINING TIME: 1084.84s
1183
- 2026-11-01 23:38:11 | INFO | STEP 62 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
1184
- 2026-11-01 23:38:11 | INFO | STEP 62 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
1185
- 2026-11-01 23:38:28 | INFO | --------------------------------------------------------------------------------
1186
- 2026-11-01 23:38:28 | INFO | STEP 63/70 | epoch=9/10 | position=7/7 | dataset_item=4
1187
- 2026-11-01 23:38:28 | INFO | DATASET FILE: 00000004.pt
1188
- 2026-11-01 23:38:28 | INFO | LOSS: 0.93932259
1189
- 2026-11-01 23:38:28 | INFO | LR: 2.75362319e-05
1190
- 2026-11-01 23:38:28 | INFO | TIMESTEP: 0.43518484
1191
- 2026-11-01 23:38:28 | INFO | GLOBAL GRADIENT NORM: 0.16493047
1192
- 2026-11-01 23:38:28 | INFO | MAX ABSOLUTE GRADIENT: 1.10321045e-02
1193
- 2026-11-01 23:38:28 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1194
- 2026-11-01 23:38:28 | INFO | MISSING GRADIENT TENSORS: 0
1195
- 2026-11-01 23:38:28 | INFO | GPU0 FORWARD: 2.460s
1196
- 2026-11-01 23:38:28 | INFO | GPU1 FORWARD: 2.407s
1197
- 2026-11-01 23:38:28 | INFO | GPU1 BACKWARD: 6.250s
1198
- 2026-11-01 23:38:28 | INFO | GPU0 BACKWARD: 6.262s
1199
- 2026-11-01 23:38:28 | INFO | UPDATE WALL TIME: 17.526s
1200
- 2026-11-01 23:38:28 | INFO | ELAPSED TRAINING TIME: 1102.38s
1201
- 2026-11-01 23:38:28 | INFO | STEP 63 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
1202
- 2026-11-01 23:38:28 | INFO | STEP 63 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
1203
- 2026-11-01 23:38:46 | INFO | --------------------------------------------------------------------------------
1204
- 2026-11-01 23:38:46 | INFO | STEP 64/70 | epoch=10/10 | position=1/7 | dataset_item=2
1205
- 2026-11-01 23:38:46 | INFO | DATASET FILE: 00000002.pt
1206
- 2026-11-01 23:38:46 | INFO | LOSS: 0.76127219
1207
- 2026-11-01 23:38:46 | INFO | LR: 2.71739130e-05
1208
- 2026-11-01 23:38:46 | INFO | TIMESTEP: 0.54496264
1209
- 2026-11-01 23:38:46 | INFO | GLOBAL GRADIENT NORM: 0.11708542
1210
- 2026-11-01 23:38:46 | INFO | MAX ABSOLUTE GRADIENT: 7.44628906e-03
1211
- 2026-11-01 23:38:46 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1212
- 2026-11-01 23:38:46 | INFO | MISSING GRADIENT TENSORS: 0
1213
- 2026-11-01 23:38:46 | INFO | GPU0 FORWARD: 2.503s
1214
- 2026-11-01 23:38:46 | INFO | GPU1 FORWARD: 2.408s
1215
- 2026-11-01 23:38:46 | INFO | GPU1 BACKWARD: 6.299s
1216
- 2026-11-01 23:38:46 | INFO | GPU0 BACKWARD: 6.249s
1217
- 2026-11-01 23:38:46 | INFO | UPDATE WALL TIME: 17.607s
1218
- 2026-11-01 23:38:46 | INFO | ELAPSED TRAINING TIME: 1120.00s
1219
- 2026-11-01 23:38:46 | INFO | STEP 64 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.835 GB | free=1.389 GB
1220
- 2026-11-01 23:38:46 | INFO | STEP 64 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.742 GB | free=1.464 GB
1221
- 2026-11-01 23:39:03 | INFO | --------------------------------------------------------------------------------
1222
- 2026-11-01 23:39:03 | INFO | STEP 65/70 | epoch=10/10 | position=2/7 | dataset_item=4
1223
- 2026-11-01 23:39:03 | INFO | DATASET FILE: 00000004.pt
1224
- 2026-11-01 23:39:03 | INFO | LOSS: 1.15107083
1225
- 2026-11-01 23:39:03 | INFO | LR: 2.68115942e-05
1226
- 2026-11-01 23:39:03 | INFO | TIMESTEP: 0.33064201
1227
- 2026-11-01 23:39:03 | INFO | GLOBAL GRADIENT NORM: 0.26678461
1228
- 2026-11-01 23:39:03 | INFO | MAX ABSOLUTE GRADIENT: 1.66015625e-02
1229
- 2026-11-01 23:39:03 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1230
- 2026-11-01 23:39:03 | INFO | MISSING GRADIENT TENSORS: 0
1231
- 2026-11-01 23:39:03 | INFO | GPU0 FORWARD: 2.488s
1232
- 2026-11-01 23:39:03 | INFO | GPU1 FORWARD: 2.366s
1233
- 2026-11-01 23:39:03 | INFO | GPU1 BACKWARD: 6.240s
1234
- 2026-11-01 23:39:03 | INFO | GPU0 BACKWARD: 6.281s
1235
- 2026-11-01 23:39:03 | INFO | UPDATE WALL TIME: 17.521s
1236
- 2026-11-01 23:39:03 | INFO | ELAPSED TRAINING TIME: 1137.53s
1237
- 2026-11-01 23:39:03 | INFO | STEP 65 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.862 GB | free=1.389 GB
1238
- 2026-11-01 23:39:03 | INFO | STEP 65 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.768 GB | free=1.464 GB
1239
- 2026-11-01 23:39:21 | INFO | --------------------------------------------------------------------------------
1240
- 2026-11-01 23:39:21 | INFO | STEP 66/70 | epoch=10/10 | position=3/7 | dataset_item=3
1241
- 2026-11-01 23:39:21 | INFO | DATASET FILE: 00000003.pt
1242
- 2026-11-01 23:39:21 | INFO | LOSS: 0.86686647
1243
- 2026-11-01 23:39:21 | INFO | LR: 2.64492754e-05
1244
- 2026-11-01 23:39:21 | INFO | TIMESTEP: 0.46732426
1245
- 2026-11-01 23:39:21 | INFO | GLOBAL GRADIENT NORM: 0.14438007
1246
- 2026-11-01 23:39:21 | INFO | MAX ABSOLUTE GRADIENT: 9.64355469e-03
1247
- 2026-11-01 23:39:21 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1248
- 2026-11-01 23:39:21 | INFO | MISSING GRADIENT TENSORS: 0
1249
- 2026-11-01 23:39:21 | INFO | GPU0 FORWARD: 2.477s
1250
- 2026-11-01 23:39:21 | INFO | GPU1 FORWARD: 2.391s
1251
- 2026-11-01 23:39:21 | INFO | GPU1 BACKWARD: 6.313s
1252
- 2026-11-01 23:39:21 | INFO | GPU0 BACKWARD: 6.230s
1253
- 2026-11-01 23:39:21 | INFO | UPDATE WALL TIME: 17.562s
1254
- 2026-11-01 23:39:21 | INFO | ELAPSED TRAINING TIME: 1155.10s
1255
- 2026-11-01 23:39:21 | INFO | STEP 66 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
1256
- 2026-11-01 23:39:21 | INFO | STEP 66 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
1257
- 2026-11-01 23:39:38 | INFO | --------------------------------------------------------------------------------
1258
- 2026-11-01 23:39:38 | INFO | STEP 67/70 | epoch=10/10 | position=4/7 | dataset_item=1
1259
- 2026-11-01 23:39:38 | INFO | DATASET FILE: 00000001.pt
1260
- 2026-11-01 23:39:38 | INFO | LOSS: 0.68726248
1261
- 2026-11-01 23:39:38 | INFO | LR: 2.60869565e-05
1262
- 2026-11-01 23:39:38 | INFO | TIMESTEP: 0.58846909
1263
- 2026-11-01 23:39:38 | INFO | GLOBAL GRADIENT NORM: 0.10802477
1264
- 2026-11-01 23:39:38 | INFO | MAX ABSOLUTE GRADIENT: 7.20596313e-03
1265
- 2026-11-01 23:39:38 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1266
- 2026-11-01 23:39:38 | INFO | MISSING GRADIENT TENSORS: 0
1267
- 2026-11-01 23:39:38 | INFO | GPU0 FORWARD: 2.420s
1268
- 2026-11-01 23:39:38 | INFO | GPU1 FORWARD: 2.345s
1269
- 2026-11-01 23:39:38 | INFO | GPU1 BACKWARD: 6.145s
1270
- 2026-11-01 23:39:38 | INFO | GPU0 BACKWARD: 6.197s
1271
- 2026-11-01 23:39:38 | INFO | UPDATE WALL TIME: 17.252s
1272
- 2026-11-01 23:39:38 | INFO | ELAPSED TRAINING TIME: 1172.37s
1273
- 2026-11-01 23:39:38 | INFO | STEP 67 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.756 GB | free=1.389 GB
1274
- 2026-11-01 23:39:38 | INFO | STEP 67 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.664 GB | free=1.464 GB
1275
- 2026-11-01 23:39:56 | INFO | --------------------------------------------------------------------------------
1276
- 2026-11-01 23:39:56 | INFO | STEP 68/70 | epoch=10/10 | position=5/7 | dataset_item=6
1277
- 2026-11-01 23:39:56 | INFO | DATASET FILE: 00000006.pt
1278
- 2026-11-01 23:39:56 | INFO | LOSS: 1.29654109
1279
- 2026-11-01 23:39:56 | INFO | LR: 2.57246377e-05
1280
- 2026-11-01 23:39:56 | INFO | TIMESTEP: 0.24980362
1281
- 2026-11-01 23:39:56 | INFO | GLOBAL GRADIENT NORM: 0.76038267
1282
- 2026-11-01 23:39:56 | INFO | MAX ABSOLUTE GRADIENT: 2.78167725e-02
1283
- 2026-11-01 23:39:56 | INFO | GRADIENT CLIP COEFFICIENT: 0.65756278
1284
- 2026-11-01 23:39:56 | INFO | MISSING GRADIENT TENSORS: 0
1285
- 2026-11-01 23:39:56 | INFO | GPU0 FORWARD: 2.514s
1286
- 2026-11-01 23:39:56 | INFO | GPU1 FORWARD: 2.400s
1287
- 2026-11-01 23:39:56 | INFO | GPU1 BACKWARD: 6.262s
1288
- 2026-11-01 23:39:56 | INFO | GPU0 BACKWARD: 6.344s
1289
- 2026-11-01 23:39:56 | INFO | UPDATE WALL TIME: 17.666s
1290
- 2026-11-01 23:39:56 | INFO | ELAPSED TRAINING TIME: 1190.04s
1291
- 2026-11-01 23:39:56 | INFO | STEP 68 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.882 GB | free=1.389 GB
1292
- 2026-11-01 23:39:56 | INFO | STEP 68 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.788 GB | free=1.464 GB
1293
- 2026-11-01 23:40:13 | INFO | --------------------------------------------------------------------------------
1294
- 2026-11-01 23:40:13 | INFO | STEP 69/70 | epoch=10/10 | position=6/7 | dataset_item=0
1295
- 2026-11-01 23:40:13 | INFO | DATASET FILE: 00000000.pt
1296
- 2026-11-01 23:40:13 | INFO | LOSS: 0.49238670
1297
- 2026-11-01 23:40:13 | INFO | LR: 2.53623188e-05
1298
- 2026-11-01 23:40:13 | INFO | TIMESTEP: 0.81050289
1299
- 2026-11-01 23:40:13 | INFO | GLOBAL GRADIENT NORM: 0.12303477
1300
- 2026-11-01 23:40:13 | INFO | MAX ABSOLUTE GRADIENT: 3.88908386e-03
1301
- 2026-11-01 23:40:13 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1302
- 2026-11-01 23:40:13 | INFO | MISSING GRADIENT TENSORS: 0
1303
- 2026-11-01 23:40:13 | INFO | GPU0 FORWARD: 2.473s
1304
- 2026-11-01 23:40:13 | INFO | GPU1 FORWARD: 2.399s
1305
- 2026-11-01 23:40:13 | INFO | GPU1 BACKWARD: 6.264s
1306
- 2026-11-01 23:40:13 | INFO | GPU0 BACKWARD: 6.199s
1307
- 2026-11-01 23:40:13 | INFO | UPDATE WALL TIME: 17.481s
1308
- 2026-11-01 23:40:13 | INFO | ELAPSED TRAINING TIME: 1207.54s
1309
- 2026-11-01 23:40:13 | INFO | STEP 69 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.821 GB | free=1.389 GB
1310
- 2026-11-01 23:40:13 | INFO | STEP 69 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.728 GB | free=1.464 GB
1311
- 2026-11-01 23:40:31 | INFO | --------------------------------------------------------------------------------
1312
- 2026-11-01 23:40:31 | INFO | STEP 70/70 | epoch=10/10 | position=7/7 | dataset_item=5
1313
- 2026-11-01 23:40:31 | INFO | DATASET FILE: 00000005.pt
1314
- 2026-11-01 23:40:31 | INFO | LOSS: 0.66494811
1315
- 2026-11-01 23:40:31 | INFO | LR: 2.50000000e-05
1316
- 2026-11-01 23:40:31 | INFO | TIMESTEP: 0.57378495
1317
- 2026-11-01 23:40:31 | INFO | GLOBAL GRADIENT NORM: 0.16218607
1318
- 2026-11-01 23:40:31 | INFO | MAX ABSOLUTE GRADIENT: 6.96182251e-03
1319
- 2026-11-01 23:40:31 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
1320
- 2026-11-01 23:40:31 | INFO | MISSING GRADIENT TENSORS: 0
1321
- 2026-11-01 23:40:31 | INFO | GPU0 FORWARD: 2.531s
1322
- 2026-11-01 23:40:31 | INFO | GPU1 FORWARD: 2.469s
1323
- 2026-11-01 23:40:31 | INFO | GPU1 BACKWARD: 6.403s
1324
- 2026-11-01 23:40:31 | INFO | GPU0 BACKWARD: 6.281s
1325
- 2026-11-01 23:40:31 | INFO | UPDATE WALL TIME: 17.826s
1326
- 2026-11-01 23:40:31 | INFO | ELAPSED TRAINING TIME: 1225.37s
1327
- 2026-11-01 23:40:31 | INFO | STEP 70 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
1328
- 2026-11-01 23:40:31 | INFO | STEP 70 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
1329
- 2026-11-01 23:40:31 | INFO | ================================================================================
1330
- 2026-11-01 23:40:31 | INFO | CHECKPOINT TRIGGERED AT OPTIMIZER STEP 70
1331
- 2026-11-01 23:40:35 | INFO | CHECKPOINT SUCCESS: kromoss_lora_step_000070.safetensors
1332
- 2026-11-01 23:40:35 | INFO | CHECKPOINT SIZE: 70.93 MB
1333
- 2026-11-01 23:40:35 | INFO | ================================================================================
1334
- 2026-11-01 23:40:35 | INFO | ================================================================================
1335
- 2026-11-01 23:40:35 | INFO | ALL 70 OPTIMIZER UPDATES COMPLETED SUCCESSFULLY
1336
- 2026-11-01 23:40:35 | INFO | ================================================================================
1337
- 2026-11-01 23:40:35 | INFO | UPDATING CONTINUATION LORA
1338
- 2026-11-01 23:40:36 | INFO | CONTINUATION LORA UPDATED: codeShare/kromoss-lora-exports/continue.safetensors
1339
- 2026-11-01 23:40:36 | INFO | CONTINUATION URL: https://huggingface.co/codeShare/kromoss-lora-exports/commit/6a73ffbe4417a5fcdeafd972bf800b5225ba38cf
1340
- 2026-11-01 23:40:36 | INFO | ================================================================================
1341
- 2026-11-01 23:40:36 | INFO | TRAINING FINISHED SUCCESSFULLY
1342
- 2026-11-01 23:40:36 | INFO | ================================================================================
1343
- 2026-11-01 23:40:36 | INFO | Optimizer updates: 70
1344
- 2026-11-01 23:40:36 | INFO | Dataset items: 7
1345
- 2026-11-01 23:40:36 | INFO | Presentations per item: 10
1346
- 2026-11-01 23:40:36 | INFO | Checkpoint steps: [14, 35, 70]
1347
- 2026-11-01 23:40:36 | INFO | Initial global learning rate: 5.00000000e-05
1348
- 2026-11-01 23:40:36 | INFO | Final global learning rate: 2.50000000e-05
1349
- 2026-11-01 23:40:36 | INFO | AdamW state persisted across all seven dataset items.
1350
- 2026-11-01 23:40:36 | INFO | Learning rate was smoothly scheduled across all 70 steps.
1351
- 2026-11-01 23:40:36 | INFO | Dataset presentation order was deterministically shuffled.
1352
- 2026-11-01 23:40:36 | INFO | Final export: /kaggle/working/kromoss_lora_out/comfyui/kromoss_lora_step_000070.safetensors
1353
- 2026-11-01 23:40:36 | INFO | Continuation file: continue.safetensors
1354
- 2026-11-01 23:40:36 | INFO | HF repository: codeShare/kromoss-lora-exports
1355
- 2026-11-01 23:40:36 | INFO | Total elapsed time: 1230.10s
1356
- 2026-11-01 23:40:36 | INFO | ================================================================================
 
1
+ 2026-11-01 23:41:22 | INFO | ================================================================================
2
+ 2026-11-01 23:41:22 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
+ 2026-11-01 23:41:22 | INFO | ================================================================================
4
+ 2026-11-01 23:41:22 | INFO | PyTorch version: 2.11.0+cu128
5
+ 2026-11-01 23:41:22 | INFO | CUDA version: 12.8
6
+ 2026-11-01 23:41:22 | INFO | Dataset size: 7
7
+ 2026-11-01 23:41:22 | INFO | Presentations per image: 10
8
+ 2026-11-01 23:41:22 | INFO | Total optimizer updates: 70
9
+ 2026-11-01 23:41:22 | INFO | Checkpoint steps: [14, 35, 70]
10
+ 2026-11-01 23:41:22 | INFO | Initial global learning rate: 5.00000000e-05
11
+ 2026-11-01 23:41:22 | INFO | Final global learning rate: 2.50000000e-05
12
+ 2026-11-01 23:41:22 | INFO | Weight decay: 0.01
13
+ 2026-11-01 23:41:22 | INFO | Gradient clipping: 0.5
14
+ 2026-11-01 23:41:22 | INFO | Loss scale: 128.0
15
+ 2026-11-01 23:41:22 | INFO | LoRA rank: 16
16
+ 2026-11-01 23:41:22 | INFO | LoRA alpha: 16
17
+ 2026-11-01 23:41:22 | INFO | Trainable parameters: 37,158,912
18
+ 2026-11-01 23:41:22 | INFO | GPU0 LoRA parameters: 17,694,720
19
+ 2026-11-01 23:41:22 | INFO | GPU1 LoRA parameters: 19,464,192
20
+ 2026-11-01 23:41:22 | INFO | Continuation loaded: True
21
+ 2026-11-01 23:41:22 | INFO | Continuation loading enabled: True
22
+ 2026-11-01 23:41:22 | INFO | HF repository: codeShare/kromoss-lora-exports
23
+ 2026-11-01 23:41:22 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
+ 2026-11-01 23:41:22 | INFO | GPU0: Tesla T4 | 14.562 GB
25
+ 2026-11-01 23:41:22 | INFO | GPU1: Tesla T4 | 14.562 GB
26
+ 2026-11-01 23:41:22 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
+ 2026-11-01 23:41:22 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
+ 2026-11-01 23:41:22 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
+ 2026-11-01 23:41:22 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
+ 2026-11-01 23:41:22 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
+ 2026-11-01 23:41:22 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
+ 2026-11-01 23:41:22 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
+ 2026-11-01 23:41:22 | INFO | --------------------------------------------------------------------------------
34
+ 2026-11-01 23:41:22 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
+ 2026-11-01 23:41:22 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
+ 2026-11-01 23:41:22 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
+ 2026-11-01 23:41:22 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
+ 2026-11-01 23:41:22 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
+ 2026-11-01 23:41:22 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
+ 2026-11-01 23:41:22 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
+ 2026-11-01 23:41:22 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
+ 2026-11-01 23:41:22 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
+ 2026-11-01 23:41:22 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
+ 2026-11-01 23:41:22 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
+ 2026-11-01 23:41:22 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
+ 2026-11-01 23:41:22 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
+ 2026-11-01 23:41:22 | INFO | ================================================================================
48
+ 2026-11-01 23:41:22 | INFO | STARTING TRAINING
49
+ 2026-11-01 23:41:22 | INFO | ================================================================================
50
+ 2026-11-01 23:41:22 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
+ 2026-11-01 23:41:22 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
+ 2026-11-01 23:41:22 | INFO | AdamW optimizer state is retained across all 70 updates.
53
+ 2026-11-01 23:41:22 | INFO | Learning rate does NOT reset when the dataset image changes.
54
+ 2026-11-01 23:41:22 | INFO | Gradient clipping threshold: 0.5
55
+ 2026-11-01 23:41:22 | INFO | Noise is generated in FP32 and cast to FP16.
56
+ 2026-11-01 23:41:22 | INFO | Timestep is logged for every optimizer update.
57
+ 2026-11-01 23:41:22 | INFO | Continuation loading enabled: True
58
+ 2026-11-01 23:41:22 | INFO | Continuation actually loaded: True
59
+ 2026-11-01 23:41:22 | ERROR | ================================================================================
60
+ 2026-11-01 23:41:22 | ERROR | TRAINING FAILED
61
+ 2026-11-01 23:41:22 | ERROR | Traceback (most recent call last):
62
+ File "/tmp/ipykernel_841/4050970492.py", line 1153, in <cell line: 0>
63
+ result = train_one_image(
64
+ dataset_index=dataset_index,
65
+ step=step
66
+ )
67
+ File "/tmp/ipykernel_841/4050970492.py", line 809, in train_one_image
68
+ for group in optimizer.param_groups:
69
+ ^^^^^^^^^
70
+ NameError: name 'optimizer' is not defined. Did you mean: 'optimizer0'?
71
+
72
+ 2026-11-01 23:41:22 | INFO | ERROR STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.856 GB | free=9.587 GB
73
+ 2026-11-01 23:41:22 | INFO | ERROR STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.749 GB | free=9.696 GB