Hugging Face's logo

codeShare
/
kromoss-lora-exports

Safetensors
Model card Files
xet
 Community
codeShare commited on
Commit
041ec2c
·
verified ·
1 Parent(s): e45c287

Upload Kromoss dual-T4 training log

Browse files
Files changed (1) hide show
  1. training_logs/kromoss_training_log.txt +148 -202
training_logs/kromoss_training_log.txt CHANGED
@@ -1,202 +1,148 @@
1
- 2026-11-01 23:10:18 | INFO | ================================================================================
2
- 2026-11-01 23:10:18 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
- 2026-11-01 23:10:18 | INFO | ================================================================================
4
- 2026-11-01 23:10:18 | INFO | PyTorch version: 2.11.0+cu128
5
- 2026-11-01 23:10:18 | INFO | CUDA version: 12.8
6
- 2026-11-01 23:10:18 | INFO | Dataset size: 7
7
- 2026-11-01 23:10:18 | INFO | Presentations per image: 10
8
- 2026-11-01 23:10:18 | INFO | Total optimizer updates: 70
9
- 2026-11-01 23:10:18 | INFO | Checkpoint steps: [14, 35, 70]
10
- 2026-11-01 23:10:18 | INFO | Initial global learning rate: 5.00000000e-05
11
- 2026-11-01 23:10:18 | INFO | Final global learning rate: 2.50000000e-05
12
- 2026-11-01 23:10:18 | INFO | Weight decay: 0.01
13
- 2026-11-01 23:10:18 | INFO | Gradient clipping: 0.5
14
- 2026-11-01 23:10:18 | INFO | Loss scale: 128.0
15
- 2026-11-01 23:10:18 | INFO | LoRA rank: 16
16
- 2026-11-01 23:10:18 | INFO | LoRA alpha: 16
17
- 2026-11-01 23:10:18 | INFO | Trainable parameters: 37,158,912
18
- 2026-11-01 23:10:18 | INFO | GPU0 LoRA parameters: 17,694,720
19
- 2026-11-01 23:10:18 | INFO | GPU1 LoRA parameters: 19,464,192
20
- 2026-11-01 23:10:18 | INFO | Continuation loaded: True
21
- 2026-11-01 23:10:18 | INFO | Continuation loading enabled: True
22
- 2026-11-01 23:10:18 | INFO | HF repository: codeShare/kromoss-lora-exports
23
- 2026-11-01 23:10:18 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
- 2026-11-01 23:10:18 | INFO | GPU0: Tesla T4 | 14.562 GB
25
- 2026-11-01 23:10:18 | INFO | GPU1: Tesla T4 | 14.562 GB
26
- 2026-11-01 23:10:18 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
- 2026-11-01 23:10:18 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
- 2026-11-01 23:10:18 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
- 2026-11-01 23:10:18 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
- 2026-11-01 23:10:18 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
- 2026-11-01 23:10:18 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
- 2026-11-01 23:10:18 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
- 2026-11-01 23:10:18 | INFO | --------------------------------------------------------------------------------
34
- 2026-11-01 23:10:18 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
- 2026-11-01 23:10:18 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
- 2026-11-01 23:10:18 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
- 2026-11-01 23:10:18 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
- 2026-11-01 23:10:18 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
- 2026-11-01 23:10:18 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
- 2026-11-01 23:10:18 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
- 2026-11-01 23:10:18 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
- 2026-11-01 23:10:18 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
- 2026-11-01 23:10:18 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
- 2026-11-01 23:10:18 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
- 2026-11-01 23:10:18 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
- 2026-11-01 23:10:18 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
- 2026-11-01 23:10:18 | INFO | ================================================================================
48
- 2026-11-01 23:10:18 | INFO | STARTING TRAINING
49
- 2026-11-01 23:10:18 | INFO | ================================================================================
50
- 2026-11-01 23:10:18 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
- 2026-11-01 23:10:18 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
- 2026-11-01 23:10:18 | INFO | AdamW optimizer state is retained across all 70 updates.
53
- 2026-11-01 23:10:18 | INFO | Learning rate does NOT reset when the dataset image changes.
54
- 2026-11-01 23:10:18 | INFO | Gradient clipping threshold: 0.5
55
- 2026-11-01 23:10:18 | INFO | Noise is generated in FP32 and cast to FP16.
56
- 2026-11-01 23:10:18 | INFO | Timestep is logged for every optimizer update.
57
- 2026-11-01 23:10:18 | INFO | Continuation loading enabled: True
58
- 2026-11-01 23:10:18 | INFO | Continuation actually loaded: True
59
- 2026-11-01 23:10:35 | INFO | --------------------------------------------------------------------------------
60
- 2026-11-01 23:10:35 | INFO | STEP 1/70 | epoch=1/10 | position=1/7 | dataset_item=1
61
- 2026-11-01 23:10:35 | INFO | DATASET FILE: 00000001.pt
62
- 2026-11-01 23:10:35 | INFO | LOSS: 1.06019557
63
- 2026-11-01 23:10:35 | INFO | LR: 5.00000000e-05
64
- 2026-11-01 23:10:35 | INFO | TIMESTEP: 0.32641813
65
- 2026-11-01 23:10:35 | INFO | GLOBAL GRADIENT NORM: 0.07359142
66
- 2026-11-01 23:10:35 | INFO | MAX ABSOLUTE GRADIENT: 2.46238708e-03
67
- 2026-11-01 23:10:35 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
68
- 2026-11-01 23:10:35 | INFO | MISSING GRADIENT TENSORS: 0
69
- 2026-11-01 23:10:35 | INFO | GPU0 FORWARD: 2.516s
70
- 2026-11-01 23:10:35 | INFO | GPU1 FORWARD: 2.314s
71
- 2026-11-01 23:10:35 | INFO | GPU1 BACKWARD: 6.042s
72
- 2026-11-01 23:10:35 | INFO | GPU0 BACKWARD: 6.113s
73
- 2026-11-01 23:10:35 | INFO | UPDATE WALL TIME: 17.465s
74
- 2026-11-01 23:10:35 | INFO | ELAPSED TRAINING TIME: 17.52s
75
- 2026-11-01 23:10:35 | INFO | STEP 1 | GPU0 | allocated=5.070 GB | reserved=12.859 GB | peak=12.624 GB | free=1.565 GB
76
- 2026-11-01 23:10:35 | INFO | STEP 1 | GPU1 | allocated=4.984 GB | reserved=12.785 GB | peak=12.519 GB | free=1.639 GB
77
- 2026-11-01 23:10:53 | INFO | --------------------------------------------------------------------------------
78
- 2026-11-01 23:10:53 | INFO | STEP 2/70 | epoch=1/10 | position=2/7 | dataset_item=3
79
- 2026-11-01 23:10:53 | INFO | DATASET FILE: 00000003.pt
80
- 2026-11-01 23:10:53 | INFO | LOSS: 1.13790405
81
- 2026-11-01 23:10:53 | INFO | LR: 4.96376812e-05
82
- 2026-11-01 23:10:53 | INFO | TIMESTEP: 0.26634458
83
- 2026-11-01 23:10:53 | INFO | GLOBAL GRADIENT NORM: 0.09819196
84
- 2026-11-01 23:10:53 | INFO | MAX ABSOLUTE GRADIENT: 5.51986694e-03
85
- 2026-11-01 23:10:53 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
86
- 2026-11-01 23:10:53 | INFO | MISSING GRADIENT TENSORS: 0
87
- 2026-11-01 23:10:53 | INFO | GPU0 FORWARD: 2.487s
88
- 2026-11-01 23:10:53 | INFO | GPU1 FORWARD: 2.395s
89
- 2026-11-01 23:10:53 | INFO | GPU1 BACKWARD: 6.313s
90
- 2026-11-01 23:10:53 | INFO | GPU0 BACKWARD: 6.420s
91
- 2026-11-01 23:10:53 | INFO | UPDATE WALL TIME: 17.812s
92
- 2026-11-01 23:10:53 | INFO | ELAPSED TRAINING TIME: 35.66s
93
- 2026-11-01 23:10:53 | INFO | STEP 2 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.898 GB | free=1.429 GB
94
- 2026-11-01 23:10:53 | INFO | STEP 2 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.805 GB | free=1.503 GB
95
- 2026-11-01 23:11:12 | INFO | --------------------------------------------------------------------------------
96
- 2026-11-01 23:11:12 | INFO | STEP 3/70 | epoch=1/10 | position=3/7 | dataset_item=4
97
- 2026-11-01 23:11:12 | INFO | DATASET FILE: 00000004.pt
98
- 2026-11-01 23:11:12 | INFO | LOSS: 1.15447319
99
- 2026-11-01 23:11:12 | INFO | LR: 4.92753623e-05
100
- 2026-11-01 23:11:12 | INFO | TIMESTEP: 0.25900331
101
- 2026-11-01 23:11:12 | INFO | GLOBAL GRADIENT NORM: 0.08647851
102
- 2026-11-01 23:11:12 | INFO | MAX ABSOLUTE GRADIENT: 3.39889526e-03
103
- 2026-11-01 23:11:12 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
104
- 2026-11-01 23:11:12 | INFO | MISSING GRADIENT TENSORS: 0
105
- 2026-11-01 23:11:12 | INFO | GPU0 FORWARD: 2.605s
106
- 2026-11-01 23:11:12 | INFO | GPU1 FORWARD: 2.442s
107
- 2026-11-01 23:11:12 | INFO | GPU1 BACKWARD: 6.381s
108
- 2026-11-01 23:11:12 | INFO | GPU0 BACKWARD: 6.471s
109
- 2026-11-01 23:11:12 | INFO | UPDATE WALL TIME: 18.096s
110
- 2026-11-01 23:11:12 | INFO | ELAPSED TRAINING TIME: 54.08s
111
- 2026-11-01 23:11:12 | INFO | STEP 3 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.862 GB | free=1.429 GB
112
- 2026-11-01 23:11:12 | INFO | STEP 3 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.768 GB | free=1.503 GB
113
- 2026-11-01 23:11:30 | INFO | --------------------------------------------------------------------------------
114
- 2026-11-01 23:11:30 | INFO | STEP 4/70 | epoch=1/10 | position=4/7 | dataset_item=2
115
- 2026-11-01 23:11:30 | INFO | DATASET FILE: 00000002.pt
116
- 2026-11-01 23:11:30 | INFO | LOSS: 1.09720421
117
- 2026-11-01 23:11:30 | INFO | LR: 4.89130435e-05
118
- 2026-11-01 23:11:30 | INFO | TIMESTEP: 0.32806167
119
- 2026-11-01 23:11:30 | INFO | GLOBAL GRADIENT NORM: 0.09105441
120
- 2026-11-01 23:11:30 | INFO | MAX ABSOLUTE GRADIENT: 4.51660156e-03
121
- 2026-11-01 23:11:30 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
122
- 2026-11-01 23:11:30 | INFO | MISSING GRADIENT TENSORS: 0
123
- 2026-11-01 23:11:30 | INFO | GPU0 FORWARD: 2.499s
124
- 2026-11-01 23:11:30 | INFO | GPU1 FORWARD: 2.367s
125
- 2026-11-01 23:11:30 | INFO | GPU1 BACKWARD: 6.205s
126
- 2026-11-01 23:11:30 | INFO | GPU0 BACKWARD: 6.134s
127
- 2026-11-01 23:11:30 | INFO | UPDATE WALL TIME: 17.396s
128
- 2026-11-01 23:11:30 | INFO | ELAPSED TRAINING TIME: 71.80s
129
- 2026-11-01 23:11:30 | INFO | STEP 4 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.835 GB | free=1.429 GB
130
- 2026-11-01 23:11:30 | INFO | STEP 4 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.742 GB | free=1.503 GB
131
- 2026-11-01 23:11:47 | INFO | --------------------------------------------------------------------------------
132
- 2026-11-01 23:11:47 | INFO | STEP 5/70 | epoch=1/10 | position=5/7 | dataset_item=6
133
- 2026-11-01 23:11:47 | INFO | DATASET FILE: 00000006.pt
134
- 2026-11-01 23:11:47 | INFO | LOSS: 1.18126702
135
- 2026-11-01 23:11:47 | INFO | LR: 4.85507246e-05
136
- 2026-11-01 23:11:47 | INFO | TIMESTEP: 0.22171448
137
- 2026-11-01 23:11:47 | INFO | GLOBAL GRADIENT NORM: 0.12018501
138
- 2026-11-01 23:11:47 | INFO | MAX ABSOLUTE GRADIENT: 3.15666199e-03
139
- 2026-11-01 23:11:47 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
140
- 2026-11-01 23:11:47 | INFO | MISSING GRADIENT TENSORS: 0
141
- 2026-11-01 23:11:47 | INFO | GPU0 FORWARD: 2.448s
142
- 2026-11-01 23:11:47 | INFO | GPU1 FORWARD: 2.381s
143
- 2026-11-01 23:11:47 | INFO | GPU1 BACKWARD: 6.240s
144
- 2026-11-01 23:11:47 | INFO | GPU0 BACKWARD: 6.148s
145
- 2026-11-01 23:11:47 | INFO | UPDATE WALL TIME: 17.415s
146
- 2026-11-01 23:11:47 | INFO | ELAPSED TRAINING TIME: 89.55s
147
- 2026-11-01 23:11:47 | INFO | STEP 5 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.882 GB | free=1.429 GB
148
- 2026-11-01 23:11:47 | INFO | STEP 5 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.788 GB | free=1.503 GB
149
- 2026-11-01 23:12:05 | INFO | --------------------------------------------------------------------------------
150
- 2026-11-01 23:12:05 | INFO | STEP 6/70 | epoch=1/10 | position=6/7 | dataset_item=0
151
- 2026-11-01 23:12:05 | INFO | DATASET FILE: 00000000.pt
152
- 2026-11-01 23:12:05 | INFO | LOSS: 1.17809653
153
- 2026-11-01 23:12:05 | INFO | LR: 4.81884058e-05
154
- 2026-11-01 23:12:05 | INFO | TIMESTEP: 0.14347191
155
- 2026-11-01 23:12:05 | INFO | GLOBAL GRADIENT NORM: 0.09921148
156
- 2026-11-01 23:12:05 | INFO | MAX ABSOLUTE GRADIENT: 3.68690491e-03
157
- 2026-11-01 23:12:05 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
158
- 2026-11-01 23:12:05 | INFO | MISSING GRADIENT TENSORS: 0
159
- 2026-11-01 23:12:05 | INFO | GPU0 FORWARD: 2.442s
160
- 2026-11-01 23:12:05 | INFO | GPU1 FORWARD: 2.346s
161
- 2026-11-01 23:12:05 | INFO | GPU1 BACKWARD: 6.138s
162
- 2026-11-01 23:12:05 | INFO | GPU0 BACKWARD: 6.269s
163
- 2026-11-01 23:12:05 | INFO | UPDATE WALL TIME: 17.383s
164
- 2026-11-01 23:12:05 | INFO | ELAPSED TRAINING TIME: 107.29s
165
- 2026-11-01 23:12:05 | INFO | STEP 6 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.821 GB | free=1.429 GB
166
- 2026-11-01 23:12:05 | INFO | STEP 6 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.728 GB | free=1.503 GB
167
- 2026-11-01 23:12:23 | INFO | --------------------------------------------------------------------------------
168
- 2026-11-01 23:12:23 | INFO | STEP 7/70 | epoch=1/10 | position=7/7 | dataset_item=5
169
- 2026-11-01 23:12:23 | INFO | DATASET FILE: 00000005.pt
170
- 2026-11-01 23:12:23 | INFO | LOSS: 0.63783616
171
- 2026-11-01 23:12:23 | INFO | LR: 4.78260870e-05
172
- 2026-11-01 23:12:23 | INFO | TIMESTEP: 0.57760578
173
- 2026-11-01 23:12:23 | INFO | GLOBAL GRADIENT NORM: 0.08457435
174
- 2026-11-01 23:12:23 | INFO | MAX ABSOLUTE GRADIENT: 2.31361389e-03
175
- 2026-11-01 23:12:23 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
176
- 2026-11-01 23:12:23 | INFO | MISSING GRADIENT TENSORS: 0
177
- 2026-11-01 23:12:23 | INFO | GPU0 FORWARD: 2.627s
178
- 2026-11-01 23:12:23 | INFO | GPU1 FORWARD: 2.445s
179
- 2026-11-01 23:12:23 | INFO | GPU1 BACKWARD: 6.427s
180
- 2026-11-01 23:12:23 | INFO | GPU0 BACKWARD: 6.464s
181
- 2026-11-01 23:12:23 | INFO | UPDATE WALL TIME: 18.150s
182
- 2026-11-01 23:12:23 | INFO | ELAPSED TRAINING TIME: 125.75s
183
- 2026-11-01 23:12:23 | INFO | STEP 7 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
184
- 2026-11-01 23:12:23 | INFO | STEP 7 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
185
- 2026-11-01 23:12:41 | INFO | --------------------------------------------------------------------------------
186
- 2026-11-01 23:12:41 | INFO | STEP 8/70 | epoch=2/10 | position=1/7 | dataset_item=3
187
- 2026-11-01 23:12:41 | INFO | DATASET FILE: 00000003.pt
188
- 2026-11-01 23:12:41 | INFO | LOSS: 1.11423683
189
- 2026-11-01 23:12:41 | INFO | LR: 4.74637681e-05
190
- 2026-11-01 23:12:41 | INFO | TIMESTEP: 0.27784425
191
- 2026-11-01 23:12:41 | INFO | GLOBAL GRADIENT NORM: 0.08444677
192
- 2026-11-01 23:12:41 | INFO | MAX ABSOLUTE GRADIENT: 2.31170654e-03
193
- 2026-11-01 23:12:41 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
194
- 2026-11-01 23:12:41 | INFO | MISSING GRADIENT TENSORS: 0
195
- 2026-11-01 23:12:41 | INFO | GPU0 FORWARD: 2.535s
196
- 2026-11-01 23:12:41 | INFO | GPU1 FORWARD: 2.413s
197
- 2026-11-01 23:12:41 | INFO | GPU1 BACKWARD: 6.291s
198
- 2026-11-01 23:12:41 | INFO | GPU0 BACKWARD: 6.266s
199
- 2026-11-01 23:12:41 | INFO | UPDATE WALL TIME: 17.697s
200
- 2026-11-01 23:12:41 | INFO | ELAPSED TRAINING TIME: 143.76s
201
- 2026-11-01 23:12:41 | INFO | STEP 8 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
202
- 2026-11-01 23:12:41 | INFO | STEP 8 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB
 
1
+ 2026-11-01 23:13:54 | INFO | ================================================================================
2
+ 2026-11-01 23:13:54 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
+ 2026-11-01 23:13:54 | INFO | ================================================================================
4
+ 2026-11-01 23:13:54 | INFO | PyTorch version: 2.11.0+cu128
5
+ 2026-11-01 23:13:54 | INFO | CUDA version: 12.8
6
+ 2026-11-01 23:13:54 | INFO | Dataset size: 7
7
+ 2026-11-01 23:13:54 | INFO | Presentations per image: 10
8
+ 2026-11-01 23:13:54 | INFO | Total optimizer updates: 70
9
+ 2026-11-01 23:13:54 | INFO | Checkpoint steps: [14, 35, 70]
10
+ 2026-11-01 23:13:54 | INFO | Initial global learning rate: 5.00000000e-05
11
+ 2026-11-01 23:13:54 | INFO | Final global learning rate: 2.50000000e-05
12
+ 2026-11-01 23:13:54 | INFO | Weight decay: 0.01
13
+ 2026-11-01 23:13:54 | INFO | Gradient clipping: 0.5
14
+ 2026-11-01 23:13:54 | INFO | Loss scale: 128.0
15
+ 2026-11-01 23:13:54 | INFO | LoRA rank: 16
16
+ 2026-11-01 23:13:54 | INFO | LoRA alpha: 16
17
+ 2026-11-01 23:13:54 | INFO | Trainable parameters: 37,158,912
18
+ 2026-11-01 23:13:54 | INFO | GPU0 LoRA parameters: 17,694,720
19
+ 2026-11-01 23:13:54 | INFO | GPU1 LoRA parameters: 19,464,192
20
+ 2026-11-01 23:13:54 | INFO | Continuation loaded: True
21
+ 2026-11-01 23:13:54 | INFO | Continuation loading enabled: True
22
+ 2026-11-01 23:13:54 | INFO | HF repository: codeShare/kromoss-lora-exports
23
+ 2026-11-01 23:13:54 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
+ 2026-11-01 23:13:54 | INFO | GPU0: Tesla T4 | 14.562 GB
25
+ 2026-11-01 23:13:54 | INFO | GPU1: Tesla T4 | 14.562 GB
26
+ 2026-11-01 23:13:54 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
+ 2026-11-01 23:13:54 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
+ 2026-11-01 23:13:54 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
+ 2026-11-01 23:13:54 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
+ 2026-11-01 23:13:54 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
+ 2026-11-01 23:13:54 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
+ 2026-11-01 23:13:54 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
+ 2026-11-01 23:13:54 | INFO | --------------------------------------------------------------------------------
34
+ 2026-11-01 23:13:54 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
+ 2026-11-01 23:13:54 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
+ 2026-11-01 23:13:54 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
+ 2026-11-01 23:13:54 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
+ 2026-11-01 23:13:54 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
+ 2026-11-01 23:13:54 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
+ 2026-11-01 23:13:54 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
+ 2026-11-01 23:13:54 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
+ 2026-11-01 23:13:54 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
+ 2026-11-01 23:13:54 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
+ 2026-11-01 23:13:54 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
+ 2026-11-01 23:13:54 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
+ 2026-11-01 23:13:54 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
+ 2026-11-01 23:13:54 | INFO | ================================================================================
48
+ 2026-11-01 23:13:54 | INFO | STARTING TRAINING
49
+ 2026-11-01 23:13:54 | INFO | ================================================================================
50
+ 2026-11-01 23:13:54 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
+ 2026-11-01 23:13:54 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
+ 2026-11-01 23:13:54 | INFO | AdamW optimizer state is retained across all 70 updates.
53
+ 2026-11-01 23:13:54 | INFO | Learning rate does NOT reset when the dataset image changes.
54
+ 2026-11-01 23:13:54 | INFO | Gradient clipping threshold: 0.5
55
+ 2026-11-01 23:13:54 | INFO | Noise is generated in FP32 and cast to FP16.
56
+ 2026-11-01 23:13:54 | INFO | Timestep is logged for every optimizer update.
57
+ 2026-11-01 23:13:54 | INFO | Continuation loading enabled: True
58
+ 2026-11-01 23:13:54 | INFO | Continuation actually loaded: True
59
+ 2026-11-01 23:14:10 | INFO | --------------------------------------------------------------------------------
60
+ 2026-11-01 23:14:10 | INFO | STEP 1/70 | epoch=1/10 | position=1/7 | dataset_item=1
61
+ 2026-11-01 23:14:10 | INFO | DATASET FILE: 00000001.pt
62
+ 2026-11-01 23:14:10 | INFO | LOSS: 1.06019557
63
+ 2026-11-01 23:14:10 | INFO | LR: 5.00000000e-05
64
+ 2026-11-01 23:14:10 | INFO | TIMESTEP: 0.32641813
65
+ 2026-11-01 23:14:10 | INFO | GLOBAL GRADIENT NORM: 0.07359263
66
+ 2026-11-01 23:14:10 | INFO | MAX ABSOLUTE GRADIENT: 2.46238708e-03
67
+ 2026-11-01 23:14:10 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
68
+ 2026-11-01 23:14:10 | INFO | MISSING GRADIENT TENSORS: 0
69
+ 2026-11-01 23:14:10 | INFO | GPU0 FORWARD: 2.407s
70
+ 2026-11-01 23:14:10 | INFO | GPU1 FORWARD: 2.312s
71
+ 2026-11-01 23:14:10 | INFO | GPU1 BACKWARD: 5.825s
72
+ 2026-11-01 23:14:10 | INFO | GPU0 BACKWARD: 5.906s
73
+ 2026-11-01 23:14:10 | INFO | UPDATE WALL TIME: 16.712s
74
+ 2026-11-01 23:14:10 | INFO | ELAPSED TRAINING TIME: 16.76s
75
+ 2026-11-01 23:14:10 | INFO | STEP 1 | GPU0 | allocated=5.070 GB | reserved=12.859 GB | peak=12.624 GB | free=1.565 GB
76
+ 2026-11-01 23:14:10 | INFO | STEP 1 | GPU1 | allocated=4.984 GB | reserved=12.785 GB | peak=12.519 GB | free=1.639 GB
77
+ 2026-11-01 23:14:28 | INFO | --------------------------------------------------------------------------------
78
+ 2026-11-01 23:14:28 | INFO | STEP 2/70 | epoch=1/10 | position=2/7 | dataset_item=3
79
+ 2026-11-01 23:14:28 | INFO | DATASET FILE: 00000003.pt
80
+ 2026-11-01 23:14:28 | INFO | LOSS: 1.13790202
81
+ 2026-11-01 23:14:28 | INFO | LR: 4.96376812e-05
82
+ 2026-11-01 23:14:28 | INFO | TIMESTEP: 0.26634458
83
+ 2026-11-01 23:14:28 | INFO | GLOBAL GRADIENT NORM: 0.09680419
84
+ 2026-11-01 23:14:28 | INFO | MAX ABSOLUTE GRADIENT: 5.51986694e-03
85
+ 2026-11-01 23:14:28 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
86
+ 2026-11-01 23:14:28 | INFO | MISSING GRADIENT TENSORS: 0
87
+ 2026-11-01 23:14:28 | INFO | GPU0 FORWARD: 2.423s
88
+ 2026-11-01 23:14:28 | INFO | GPU1 FORWARD: 2.320s
89
+ 2026-11-01 23:14:28 | INFO | GPU1 BACKWARD: 6.151s
90
+ 2026-11-01 23:14:28 | INFO | GPU0 BACKWARD: 6.167s
91
+ 2026-11-01 23:14:28 | INFO | UPDATE WALL TIME: 17.282s
92
+ 2026-11-01 23:14:28 | INFO | ELAPSED TRAINING TIME: 34.06s
93
+ 2026-11-01 23:14:28 | INFO | STEP 2 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.898 GB | free=1.429 GB
94
+ 2026-11-01 23:14:28 | INFO | STEP 2 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.805 GB | free=1.503 GB
95
+ 2026-11-01 23:14:45 | INFO | --------------------------------------------------------------------------------
96
+ 2026-11-01 23:14:45 | INFO | STEP 3/70 | epoch=1/10 | position=3/7 | dataset_item=4
97
+ 2026-11-01 23:14:45 | INFO | DATASET FILE: 00000004.pt
98
+ 2026-11-01 23:14:45 | INFO | LOSS: 1.15447021
99
+ 2026-11-01 23:14:45 | INFO | LR: 4.92753623e-05
100
+ 2026-11-01 23:14:45 | INFO | TIMESTEP: 0.25900331
101
+ 2026-11-01 23:14:45 | INFO | GLOBAL GRADIENT NORM: 0.08597146
102
+ 2026-11-01 23:14:45 | INFO | MAX ABSOLUTE GRADIENT: 3.32641602e-03
103
+ 2026-11-01 23:14:45 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
104
+ 2026-11-01 23:14:45 | INFO | MISSING GRADIENT TENSORS: 0
105
+ 2026-11-01 23:14:45 | INFO | GPU0 FORWARD: 2.473s
106
+ 2026-11-01 23:14:45 | INFO | GPU1 FORWARD: 2.404s
107
+ 2026-11-01 23:14:45 | INFO | GPU1 BACKWARD: 6.269s
108
+ 2026-11-01 23:14:45 | INFO | GPU0 BACKWARD: 6.358s
109
+ 2026-11-01 23:14:45 | INFO | UPDATE WALL TIME: 17.689s
110
+ 2026-11-01 23:14:45 | INFO | ELAPSED TRAINING TIME: 51.76s
111
+ 2026-11-01 23:14:45 | INFO | STEP 3 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.862 GB | free=1.429 GB
112
+ 2026-11-01 23:14:45 | INFO | STEP 3 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.768 GB | free=1.503 GB
113
+ 2026-11-01 23:15:03 | INFO | --------------------------------------------------------------------------------
114
+ 2026-11-01 23:15:03 | INFO | STEP 4/70 | epoch=1/10 | position=4/7 | dataset_item=2
115
+ 2026-11-01 23:15:03 | INFO | DATASET FILE: 00000002.pt
116
+ 2026-11-01 23:15:03 | INFO | LOSS: 1.09718907
117
+ 2026-11-01 23:15:03 | INFO | LR: 4.89130435e-05
118
+ 2026-11-01 23:15:03 | INFO | TIMESTEP: 0.32806167
119
+ 2026-11-01 23:15:03 | INFO | GLOBAL GRADIENT NORM: 0.09288023
120
+ 2026-11-01 23:15:03 | INFO | MAX ABSOLUTE GRADIENT: 4.47082520e-03
121
+ 2026-11-01 23:15:03 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
122
+ 2026-11-01 23:15:03 | INFO | MISSING GRADIENT TENSORS: 0
123
+ 2026-11-01 23:15:03 | INFO | GPU0 FORWARD: 2.571s
124
+ 2026-11-01 23:15:03 | INFO | GPU1 FORWARD: 2.407s
125
+ 2026-11-01 23:15:03 | INFO | GPU1 BACKWARD: 6.305s
126
+ 2026-11-01 23:15:03 | INFO | GPU0 BACKWARD: 6.346s
127
+ 2026-11-01 23:15:03 | INFO | UPDATE WALL TIME: 17.814s
128
+ 2026-11-01 23:15:03 | INFO | ELAPSED TRAINING TIME: 69.58s
129
+ 2026-11-01 23:15:03 | INFO | STEP 4 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.835 GB | free=1.429 GB
130
+ 2026-11-01 23:15:03 | INFO | STEP 4 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.742 GB | free=1.503 GB
131
+ 2026-11-01 23:15:21 | INFO | --------------------------------------------------------------------------------
132
+ 2026-11-01 23:15:21 | INFO | STEP 5/70 | epoch=1/10 | position=5/7 | dataset_item=6
133
+ 2026-11-01 23:15:21 | INFO | DATASET FILE: 00000006.pt
134
+ 2026-11-01 23:15:21 | INFO | LOSS: 1.18127048
135
+ 2026-11-01 23:15:21 | INFO | LR: 4.85507246e-05
136
+ 2026-11-01 23:15:21 | INFO | TIMESTEP: 0.22171448
137
+ 2026-11-01 23:15:21 | INFO | GLOBAL GRADIENT NORM: 0.11971906
138
+ 2026-11-01 23:15:21 | INFO | MAX ABSOLUTE GRADIENT: 3.08036804e-03
139
+ 2026-11-01 23:15:21 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
140
+ 2026-11-01 23:15:21 | INFO | MISSING GRADIENT TENSORS: 0
141
+ 2026-11-01 23:15:21 | INFO | GPU0 FORWARD: 2.513s
142
+ 2026-11-01 23:15:21 | INFO | GPU1 FORWARD: 2.394s
143
+ 2026-11-01 23:15:21 | INFO | GPU1 BACKWARD: 6.270s
144
+ 2026-11-01 23:15:21 | INFO | GPU0 BACKWARD: 6.145s
145
+ 2026-11-01 23:15:21 | INFO | UPDATE WALL TIME: 17.513s
146
+ 2026-11-01 23:15:21 | INFO | ELAPSED TRAINING TIME: 87.11s
147
+ 2026-11-01 23:15:21 | INFO | STEP 5 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.882 GB | free=1.429 GB
148
+ 2026-11-01 23:15:21 | INFO | STEP 5 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.788 GB | free=1.503 GB