Hugging Face's logo

codeShare
/
kromoss-lora-exports

Safetensors
Model card Files
xet
 Community
codeShare commited on
Commit
e45c287
·
verified ·
1 Parent(s): 66c5602

Upload Kromoss dual-T4 training log

Browse files
Files changed (1) hide show
  1. training_logs/kromoss_training_log.txt +202 -108
training_logs/kromoss_training_log.txt CHANGED
@@ -1,108 +1,202 @@
1
- 2026-11-01 23:05:09 | INFO | ================================================================================
2
- 2026-11-01 23:05:09 | INFO | KROMOSS DUAL-T4 SPEED-OPTIMIZED TRAINING
3
- 2026-11-01 23:05:09 | INFO | ================================================================================
4
- 2026-11-01 23:05:09 | INFO | PyTorch version: 2.11.0+cu128
5
- 2026-11-01 23:05:09 | INFO | CUDA version: 12.8
6
- 2026-11-01 23:05:09 | INFO | Dataset size: 7
7
- 2026-11-01 23:05:09 | INFO | Repeats per image: 10
8
- 2026-11-01 23:05:09 | INFO | Total optimizer updates: 70
9
- 2026-11-01 23:05:09 | INFO | Checkpoint steps: [14, 35, 70]
10
- 2026-11-01 23:05:09 | INFO | Initial learning rate: 5.00000000e-05
11
- 2026-11-01 23:05:09 | INFO | Final global learning rate: 2.50000000e-05
12
- 2026-11-01 23:05:09 | INFO | Weight decay: 0.01
13
- 2026-11-01 23:05:09 | INFO | Gradient clipping: 0.5
14
- 2026-11-01 23:05:09 | INFO | Loss scale: 128.0
15
- 2026-11-01 23:05:09 | INFO | LoRA rank: 16
16
- 2026-11-01 23:05:09 | INFO | LoRA alpha: 16
17
- 2026-11-01 23:05:09 | INFO | Trainable parameters: 37,158,912
18
- 2026-11-01 23:05:09 | INFO | GPU0 LoRA parameters: 17,694,720
19
- 2026-11-01 23:05:09 | INFO | GPU1 LoRA parameters: 19,464,192
20
- 2026-11-01 23:05:09 | INFO | Continuation loaded: True
21
- 2026-11-01 23:05:09 | INFO | HF repository: codeShare/kromoss-lora-exports
22
- 2026-11-01 23:05:09 | INFO | Checkpoint group size: 8
23
- 2026-11-01 23:05:09 | INFO | GPU0: Tesla T4 | 14.562 GB
24
- 2026-11-01 23:05:09 | INFO | GPU1: Tesla T4 | 14.562 GB
25
- 2026-11-01 23:05:09 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
26
- 2026-11-01 23:05:09 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
27
- 2026-11-01 23:05:09 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
28
- 2026-11-01 23:05:09 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
29
- 2026-11-01 23:05:09 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
30
- 2026-11-01 23:05:09 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
31
- 2026-11-01 23:05:09 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
32
- 2026-11-01 23:05:09 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
33
- 2026-11-01 23:05:09 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
34
- 2026-11-01 23:05:09 | INFO | ================================================================================
35
- 2026-11-01 23:05:09 | INFO | STARTING SPEED-OPTIMIZED TRAINING
36
- 2026-11-01 23:05:09 | INFO | ================================================================================
37
- 2026-11-01 23:05:09 | INFO | Using shuffled SAMPLE_PLAN from CELL 2.
38
- 2026-11-01 23:05:09 | INFO | Global learning-rate decay across all 70 optimizer updates.
39
- 2026-11-01 23:05:09 | INFO | Activation checkpoint group size: 8
40
- 2026-11-01 23:05:09 | INFO | Per-step CUDA memory reporting disabled.
41
- 2026-11-01 23:05:09 | INFO | Per-step Python garbage collection disabled.
42
- 2026-11-01 23:05:15 | ERROR | ================================================================================
43
- 2026-11-01 23:05:15 | ERROR | TRAINING FAILED
44
- 2026-11-01 23:05:15 | ERROR | Traceback (most recent call last):
45
- File "/tmp/ipykernel_303/3626853694.py", line 1069, in <cell line: 0>
46
- result = train_one_image(
47
- dataset_index=dataset_index,
48
- step=step
49
- )
50
- File "/tmp/ipykernel_303/3626853694.py", line 875, in train_one_image
51
- grad_x1 = stage1_backward(result)
52
- File "/tmp/ipykernel_303/3626853694.py", line 549, in stage1_backward
53
- gradients = torch.autograd.grad(
54
- scaled_loss,
55
- [result["x1"]] + stage1_params,
56
- allow_unused=True
57
- )
58
- File "/usr/local/lib/python3.13/dist-packages/torch/autograd/__init__.py", line 532, in grad
59
- result = _engine_run_backward(
60
- outputs,
61
- ...<5 lines>...
62
- accumulate_grad=False,
63
- )
64
- File "/usr/local/lib/python3.13/dist-packages/torch/autograd/graph.py", line 869, in _engine_run_backward
65
- return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
66
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
67
- t_outputs, *args, **kwargs
68
- ^^^^^^^^^^^^^^^^^^^^^^^^^^
69
- ) # Calls into the C++ engine to run the backward pass
70
- ^
71
- File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1173, in unpack_hook
72
- _run_fn_with_dynamo_disabled(frame.recompute_fn, *args)
73
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
74
- File "/usr/local/lib/python3.13/dist-packages/torch/_compile.py", line 54, in inner
75
- return disable_fn(*args, **kwargs)
76
- File "/usr/local/lib/python3.13/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
77
- return fn(*args, **kwargs)
78
- File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1139, in _run_fn_with_dynamo_disabled
79
- return fn(*args, **kwargs)
80
- File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1607, in recompute_fn
81
- fn(*args, **kwargs)
82
- ~~^^^^^^^^^^^^^^^^^
83
- File "/tmp/ipykernel_303/3626853694.py", line 319, in run_group
84
- x = base.layers[layer_index](
85
- x,
86
- rotary,
87
- mod
88
- )
89
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
90
- return self._call_impl(*args, **kwargs)
91
- ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
92
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
93
- return forward_call(*args, **kwargs)
94
- File "/kaggle/working/pipeline_moss_image_patched.py", line 147, in forward
95
- hidden, gate = self.proj['ffn_up_gate'](y.to(dtype)).chunk(2, -1)
96
- ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
97
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
98
- return self._call_impl(*args, **kwargs)
99
- ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
100
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
101
- return forward_call(*args, **kwargs)
102
- File "/usr/local/lib/python3.13/dist-packages/peft/tuners/lora/layer.py", line 1102, in forward
103
- result = result + lora_B(lora_A(dropout(x))) * scaling
104
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~
105
- torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 146.00 MiB. GPU 1 has a total capacity of 14.56 GiB of which 82.81 MiB is free. Including non-PyTorch memory, this process has 14.48 GiB memory in use. Of the allocated memory 14.24 GiB is allocated by PyTorch, and 104.85 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
106
-
107
- 2026-11-01 23:05:15 | INFO | ERROR STATE | GPU0 | allocated=4.980 GB | reserved=5.717 GB | peak=5.628 GB | free=8.716 GB
108
- 2026-11-01 23:05:15 | INFO | ERROR STATE | GPU1 | allocated=14.027 GB | reserved=14.344 GB | peak=14.241 GB | free=0.081 GB
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-11-01 23:10:18 | INFO | ================================================================================
2
+ 2026-11-01 23:10:18 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
+ 2026-11-01 23:10:18 | INFO | ================================================================================
4
+ 2026-11-01 23:10:18 | INFO | PyTorch version: 2.11.0+cu128
5
+ 2026-11-01 23:10:18 | INFO | CUDA version: 12.8
6
+ 2026-11-01 23:10:18 | INFO | Dataset size: 7
7
+ 2026-11-01 23:10:18 | INFO | Presentations per image: 10
8
+ 2026-11-01 23:10:18 | INFO | Total optimizer updates: 70
9
+ 2026-11-01 23:10:18 | INFO | Checkpoint steps: [14, 35, 70]
10
+ 2026-11-01 23:10:18 | INFO | Initial global learning rate: 5.00000000e-05
11
+ 2026-11-01 23:10:18 | INFO | Final global learning rate: 2.50000000e-05
12
+ 2026-11-01 23:10:18 | INFO | Weight decay: 0.01
13
+ 2026-11-01 23:10:18 | INFO | Gradient clipping: 0.5
14
+ 2026-11-01 23:10:18 | INFO | Loss scale: 128.0
15
+ 2026-11-01 23:10:18 | INFO | LoRA rank: 16
16
+ 2026-11-01 23:10:18 | INFO | LoRA alpha: 16
17
+ 2026-11-01 23:10:18 | INFO | Trainable parameters: 37,158,912
18
+ 2026-11-01 23:10:18 | INFO | GPU0 LoRA parameters: 17,694,720
19
+ 2026-11-01 23:10:18 | INFO | GPU1 LoRA parameters: 19,464,192
20
+ 2026-11-01 23:10:18 | INFO | Continuation loaded: True
21
+ 2026-11-01 23:10:18 | INFO | Continuation loading enabled: True
22
+ 2026-11-01 23:10:18 | INFO | HF repository: codeShare/kromoss-lora-exports
23
+ 2026-11-01 23:10:18 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
+ 2026-11-01 23:10:18 | INFO | GPU0: Tesla T4 | 14.562 GB
25
+ 2026-11-01 23:10:18 | INFO | GPU1: Tesla T4 | 14.562 GB
26
+ 2026-11-01 23:10:18 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
+ 2026-11-01 23:10:18 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
+ 2026-11-01 23:10:18 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
+ 2026-11-01 23:10:18 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
+ 2026-11-01 23:10:18 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
+ 2026-11-01 23:10:18 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
+ 2026-11-01 23:10:18 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
+ 2026-11-01 23:10:18 | INFO | --------------------------------------------------------------------------------
34
+ 2026-11-01 23:10:18 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
+ 2026-11-01 23:10:18 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
+ 2026-11-01 23:10:18 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
+ 2026-11-01 23:10:18 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
+ 2026-11-01 23:10:18 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
+ 2026-11-01 23:10:18 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
+ 2026-11-01 23:10:18 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
+ 2026-11-01 23:10:18 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
+ 2026-11-01 23:10:18 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
+ 2026-11-01 23:10:18 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
+ 2026-11-01 23:10:18 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
+ 2026-11-01 23:10:18 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
+ 2026-11-01 23:10:18 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
+ 2026-11-01 23:10:18 | INFO | ================================================================================
48
+ 2026-11-01 23:10:18 | INFO | STARTING TRAINING
49
+ 2026-11-01 23:10:18 | INFO | ================================================================================
50
+ 2026-11-01 23:10:18 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
+ 2026-11-01 23:10:18 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
+ 2026-11-01 23:10:18 | INFO | AdamW optimizer state is retained across all 70 updates.
53
+ 2026-11-01 23:10:18 | INFO | Learning rate does NOT reset when the dataset image changes.
54
+ 2026-11-01 23:10:18 | INFO | Gradient clipping threshold: 0.5
55
+ 2026-11-01 23:10:18 | INFO | Noise is generated in FP32 and cast to FP16.
56
+ 2026-11-01 23:10:18 | INFO | Timestep is logged for every optimizer update.
57
+ 2026-11-01 23:10:18 | INFO | Continuation loading enabled: True
58
+ 2026-11-01 23:10:18 | INFO | Continuation actually loaded: True
59
+ 2026-11-01 23:10:35 | INFO | --------------------------------------------------------------------------------
60
+ 2026-11-01 23:10:35 | INFO | STEP 1/70 | epoch=1/10 | position=1/7 | dataset_item=1
61
+ 2026-11-01 23:10:35 | INFO | DATASET FILE: 00000001.pt
62
+ 2026-11-01 23:10:35 | INFO | LOSS: 1.06019557
63
+ 2026-11-01 23:10:35 | INFO | LR: 5.00000000e-05
64
+ 2026-11-01 23:10:35 | INFO | TIMESTEP: 0.32641813
65
+ 2026-11-01 23:10:35 | INFO | GLOBAL GRADIENT NORM: 0.07359142
66
+ 2026-11-01 23:10:35 | INFO | MAX ABSOLUTE GRADIENT: 2.46238708e-03
67
+ 2026-11-01 23:10:35 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
68
+ 2026-11-01 23:10:35 | INFO | MISSING GRADIENT TENSORS: 0
69
+ 2026-11-01 23:10:35 | INFO | GPU0 FORWARD: 2.516s
70
+ 2026-11-01 23:10:35 | INFO | GPU1 FORWARD: 2.314s
71
+ 2026-11-01 23:10:35 | INFO | GPU1 BACKWARD: 6.042s
72
+ 2026-11-01 23:10:35 | INFO | GPU0 BACKWARD: 6.113s
73
+ 2026-11-01 23:10:35 | INFO | UPDATE WALL TIME: 17.465s
74
+ 2026-11-01 23:10:35 | INFO | ELAPSED TRAINING TIME: 17.52s
75
+ 2026-11-01 23:10:35 | INFO | STEP 1 | GPU0 | allocated=5.070 GB | reserved=12.859 GB | peak=12.624 GB | free=1.565 GB
76
+ 2026-11-01 23:10:35 | INFO | STEP 1 | GPU1 | allocated=4.984 GB | reserved=12.785 GB | peak=12.519 GB | free=1.639 GB
77
+ 2026-11-01 23:10:53 | INFO | --------------------------------------------------------------------------------
78
+ 2026-11-01 23:10:53 | INFO | STEP 2/70 | epoch=1/10 | position=2/7 | dataset_item=3
79
+ 2026-11-01 23:10:53 | INFO | DATASET FILE: 00000003.pt
80
+ 2026-11-01 23:10:53 | INFO | LOSS: 1.13790405
81
+ 2026-11-01 23:10:53 | INFO | LR: 4.96376812e-05
82
+ 2026-11-01 23:10:53 | INFO | TIMESTEP: 0.26634458
83
+ 2026-11-01 23:10:53 | INFO | GLOBAL GRADIENT NORM: 0.09819196
84
+ 2026-11-01 23:10:53 | INFO | MAX ABSOLUTE GRADIENT: 5.51986694e-03
85
+ 2026-11-01 23:10:53 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
86
+ 2026-11-01 23:10:53 | INFO | MISSING GRADIENT TENSORS: 0
87
+ 2026-11-01 23:10:53 | INFO | GPU0 FORWARD: 2.487s
88
+ 2026-11-01 23:10:53 | INFO | GPU1 FORWARD: 2.395s
89
+ 2026-11-01 23:10:53 | INFO | GPU1 BACKWARD: 6.313s
90
+ 2026-11-01 23:10:53 | INFO | GPU0 BACKWARD: 6.420s
91
+ 2026-11-01 23:10:53 | INFO | UPDATE WALL TIME: 17.812s
92
+ 2026-11-01 23:10:53 | INFO | ELAPSED TRAINING TIME: 35.66s
93
+ 2026-11-01 23:10:53 | INFO | STEP 2 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.898 GB | free=1.429 GB
94
+ 2026-11-01 23:10:53 | INFO | STEP 2 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.805 GB | free=1.503 GB
95
+ 2026-11-01 23:11:12 | INFO | --------------------------------------------------------------------------------
96
+ 2026-11-01 23:11:12 | INFO | STEP 3/70 | epoch=1/10 | position=3/7 | dataset_item=4
97
+ 2026-11-01 23:11:12 | INFO | DATASET FILE: 00000004.pt
98
+ 2026-11-01 23:11:12 | INFO | LOSS: 1.15447319
99
+ 2026-11-01 23:11:12 | INFO | LR: 4.92753623e-05
100
+ 2026-11-01 23:11:12 | INFO | TIMESTEP: 0.25900331
101
+ 2026-11-01 23:11:12 | INFO | GLOBAL GRADIENT NORM: 0.08647851
102
+ 2026-11-01 23:11:12 | INFO | MAX ABSOLUTE GRADIENT: 3.39889526e-03
103
+ 2026-11-01 23:11:12 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
104
+ 2026-11-01 23:11:12 | INFO | MISSING GRADIENT TENSORS: 0
105
+ 2026-11-01 23:11:12 | INFO | GPU0 FORWARD: 2.605s
106
+ 2026-11-01 23:11:12 | INFO | GPU1 FORWARD: 2.442s
107
+ 2026-11-01 23:11:12 | INFO | GPU1 BACKWARD: 6.381s
108
+ 2026-11-01 23:11:12 | INFO | GPU0 BACKWARD: 6.471s
109
+ 2026-11-01 23:11:12 | INFO | UPDATE WALL TIME: 18.096s
110
+ 2026-11-01 23:11:12 | INFO | ELAPSED TRAINING TIME: 54.08s
111
+ 2026-11-01 23:11:12 | INFO | STEP 3 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.862 GB | free=1.429 GB
112
+ 2026-11-01 23:11:12 | INFO | STEP 3 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.768 GB | free=1.503 GB
113
+ 2026-11-01 23:11:30 | INFO | --------------------------------------------------------------------------------
114
+ 2026-11-01 23:11:30 | INFO | STEP 4/70 | epoch=1/10 | position=4/7 | dataset_item=2
115
+ 2026-11-01 23:11:30 | INFO | DATASET FILE: 00000002.pt
116
+ 2026-11-01 23:11:30 | INFO | LOSS: 1.09720421
117
+ 2026-11-01 23:11:30 | INFO | LR: 4.89130435e-05
118
+ 2026-11-01 23:11:30 | INFO | TIMESTEP: 0.32806167
119
+ 2026-11-01 23:11:30 | INFO | GLOBAL GRADIENT NORM: 0.09105441
120
+ 2026-11-01 23:11:30 | INFO | MAX ABSOLUTE GRADIENT: 4.51660156e-03
121
+ 2026-11-01 23:11:30 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
122
+ 2026-11-01 23:11:30 | INFO | MISSING GRADIENT TENSORS: 0
123
+ 2026-11-01 23:11:30 | INFO | GPU0 FORWARD: 2.499s
124
+ 2026-11-01 23:11:30 | INFO | GPU1 FORWARD: 2.367s
125
+ 2026-11-01 23:11:30 | INFO | GPU1 BACKWARD: 6.205s
126
+ 2026-11-01 23:11:30 | INFO | GPU0 BACKWARD: 6.134s
127
+ 2026-11-01 23:11:30 | INFO | UPDATE WALL TIME: 17.396s
128
+ 2026-11-01 23:11:30 | INFO | ELAPSED TRAINING TIME: 71.80s
129
+ 2026-11-01 23:11:30 | INFO | STEP 4 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.835 GB | free=1.429 GB
130
+ 2026-11-01 23:11:30 | INFO | STEP 4 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.742 GB | free=1.503 GB
131
+ 2026-11-01 23:11:47 | INFO | --------------------------------------------------------------------------------
132
+ 2026-11-01 23:11:47 | INFO | STEP 5/70 | epoch=1/10 | position=5/7 | dataset_item=6
133
+ 2026-11-01 23:11:47 | INFO | DATASET FILE: 00000006.pt
134
+ 2026-11-01 23:11:47 | INFO | LOSS: 1.18126702
135
+ 2026-11-01 23:11:47 | INFO | LR: 4.85507246e-05
136
+ 2026-11-01 23:11:47 | INFO | TIMESTEP: 0.22171448
137
+ 2026-11-01 23:11:47 | INFO | GLOBAL GRADIENT NORM: 0.12018501
138
+ 2026-11-01 23:11:47 | INFO | MAX ABSOLUTE GRADIENT: 3.15666199e-03
139
+ 2026-11-01 23:11:47 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
140
+ 2026-11-01 23:11:47 | INFO | MISSING GRADIENT TENSORS: 0
141
+ 2026-11-01 23:11:47 | INFO | GPU0 FORWARD: 2.448s
142
+ 2026-11-01 23:11:47 | INFO | GPU1 FORWARD: 2.381s
143
+ 2026-11-01 23:11:47 | INFO | GPU1 BACKWARD: 6.240s
144
+ 2026-11-01 23:11:47 | INFO | GPU0 BACKWARD: 6.148s
145
+ 2026-11-01 23:11:47 | INFO | UPDATE WALL TIME: 17.415s
146
+ 2026-11-01 23:11:47 | INFO | ELAPSED TRAINING TIME: 89.55s
147
+ 2026-11-01 23:11:47 | INFO | STEP 5 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.882 GB | free=1.429 GB
148
+ 2026-11-01 23:11:47 | INFO | STEP 5 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.788 GB | free=1.503 GB
149
+ 2026-11-01 23:12:05 | INFO | --------------------------------------------------------------------------------
150
+ 2026-11-01 23:12:05 | INFO | STEP 6/70 | epoch=1/10 | position=6/7 | dataset_item=0
151
+ 2026-11-01 23:12:05 | INFO | DATASET FILE: 00000000.pt
152
+ 2026-11-01 23:12:05 | INFO | LOSS: 1.17809653
153
+ 2026-11-01 23:12:05 | INFO | LR: 4.81884058e-05
154
+ 2026-11-01 23:12:05 | INFO | TIMESTEP: 0.14347191
155
+ 2026-11-01 23:12:05 | INFO | GLOBAL GRADIENT NORM: 0.09921148
156
+ 2026-11-01 23:12:05 | INFO | MAX ABSOLUTE GRADIENT: 3.68690491e-03
157
+ 2026-11-01 23:12:05 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
158
+ 2026-11-01 23:12:05 | INFO | MISSING GRADIENT TENSORS: 0
159
+ 2026-11-01 23:12:05 | INFO | GPU0 FORWARD: 2.442s
160
+ 2026-11-01 23:12:05 | INFO | GPU1 FORWARD: 2.346s
161
+ 2026-11-01 23:12:05 | INFO | GPU1 BACKWARD: 6.138s
162
+ 2026-11-01 23:12:05 | INFO | GPU0 BACKWARD: 6.269s
163
+ 2026-11-01 23:12:05 | INFO | UPDATE WALL TIME: 17.383s
164
+ 2026-11-01 23:12:05 | INFO | ELAPSED TRAINING TIME: 107.29s
165
+ 2026-11-01 23:12:05 | INFO | STEP 6 | GPU0 | allocated=5.070 GB | reserved=12.996 GB | peak=12.821 GB | free=1.429 GB
166
+ 2026-11-01 23:12:05 | INFO | STEP 6 | GPU1 | allocated=4.984 GB | reserved=12.922 GB | peak=12.728 GB | free=1.503 GB
167
+ 2026-11-01 23:12:23 | INFO | --------------------------------------------------------------------------------
168
+ 2026-11-01 23:12:23 | INFO | STEP 7/70 | epoch=1/10 | position=7/7 | dataset_item=5
169
+ 2026-11-01 23:12:23 | INFO | DATASET FILE: 00000005.pt
170
+ 2026-11-01 23:12:23 | INFO | LOSS: 0.63783616
171
+ 2026-11-01 23:12:23 | INFO | LR: 4.78260870e-05
172
+ 2026-11-01 23:12:23 | INFO | TIMESTEP: 0.57760578
173
+ 2026-11-01 23:12:23 | INFO | GLOBAL GRADIENT NORM: 0.08457435
174
+ 2026-11-01 23:12:23 | INFO | MAX ABSOLUTE GRADIENT: 2.31361389e-03
175
+ 2026-11-01 23:12:23 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
176
+ 2026-11-01 23:12:23 | INFO | MISSING GRADIENT TENSORS: 0
177
+ 2026-11-01 23:12:23 | INFO | GPU0 FORWARD: 2.627s
178
+ 2026-11-01 23:12:23 | INFO | GPU1 FORWARD: 2.445s
179
+ 2026-11-01 23:12:23 | INFO | GPU1 BACKWARD: 6.427s
180
+ 2026-11-01 23:12:23 | INFO | GPU0 BACKWARD: 6.464s
181
+ 2026-11-01 23:12:23 | INFO | UPDATE WALL TIME: 18.150s
182
+ 2026-11-01 23:12:23 | INFO | ELAPSED TRAINING TIME: 125.75s
183
+ 2026-11-01 23:12:23 | INFO | STEP 7 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.935 GB | free=1.389 GB
184
+ 2026-11-01 23:12:23 | INFO | STEP 7 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.842 GB | free=1.464 GB
185
+ 2026-11-01 23:12:41 | INFO | --------------------------------------------------------------------------------
186
+ 2026-11-01 23:12:41 | INFO | STEP 8/70 | epoch=2/10 | position=1/7 | dataset_item=3
187
+ 2026-11-01 23:12:41 | INFO | DATASET FILE: 00000003.pt
188
+ 2026-11-01 23:12:41 | INFO | LOSS: 1.11423683
189
+ 2026-11-01 23:12:41 | INFO | LR: 4.74637681e-05
190
+ 2026-11-01 23:12:41 | INFO | TIMESTEP: 0.27784425
191
+ 2026-11-01 23:12:41 | INFO | GLOBAL GRADIENT NORM: 0.08444677
192
+ 2026-11-01 23:12:41 | INFO | MAX ABSOLUTE GRADIENT: 2.31170654e-03
193
+ 2026-11-01 23:12:41 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
194
+ 2026-11-01 23:12:41 | INFO | MISSING GRADIENT TENSORS: 0
195
+ 2026-11-01 23:12:41 | INFO | GPU0 FORWARD: 2.535s
196
+ 2026-11-01 23:12:41 | INFO | GPU1 FORWARD: 2.413s
197
+ 2026-11-01 23:12:41 | INFO | GPU1 BACKWARD: 6.291s
198
+ 2026-11-01 23:12:41 | INFO | GPU0 BACKWARD: 6.266s
199
+ 2026-11-01 23:12:41 | INFO | UPDATE WALL TIME: 17.697s
200
+ 2026-11-01 23:12:41 | INFO | ELAPSED TRAINING TIME: 143.76s
201
+ 2026-11-01 23:12:41 | INFO | STEP 8 | GPU0 | allocated=5.070 GB | reserved=13.035 GB | peak=12.898 GB | free=1.389 GB
202
+ 2026-11-01 23:12:41 | INFO | STEP 8 | GPU1 | allocated=4.984 GB | reserved=12.961 GB | peak=12.805 GB | free=1.464 GB