Hugging Face's logo

codeShare
/
kromoss-lora-exports

Safetensors
Model card Files
xet
 Community
codeShare commited on
Commit
b9c1738
·
verified ·
1 Parent(s): bc66c35

Upload Kromoss dual-T4 training log

Browse files
Files changed (1) hide show
  1. training_logs/kromoss_training_log.txt +424 -126
training_logs/kromoss_training_log.txt CHANGED
@@ -1,126 +1,424 @@
1
- 2026-11-01 23:47:22 | INFO | ================================================================================
2
- 2026-11-01 23:47:22 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
- 2026-11-01 23:47:22 | INFO | ================================================================================
4
- 2026-11-01 23:47:22 | INFO | PyTorch version: 2.11.0+cu128
5
- 2026-11-01 23:47:22 | INFO | CUDA version: 12.8
6
- 2026-11-01 23:47:22 | INFO | Dataset size: 7
7
- 2026-11-01 23:47:22 | INFO | Presentations per image: 10
8
- 2026-11-01 23:47:22 | INFO | Total optimizer updates: 70
9
- 2026-11-01 23:47:22 | INFO | Checkpoint steps: [14, 35, 70]
10
- 2026-11-01 23:47:22 | INFO | Initial global learning rate: 5.00000000e-05
11
- 2026-11-01 23:47:22 | INFO | Final global learning rate: 2.50000000e-05
12
- 2026-11-01 23:47:22 | INFO | Weight decay: 0.01
13
- 2026-11-01 23:47:22 | INFO | Gradient clipping: 0.5
14
- 2026-11-01 23:47:22 | INFO | Loss scale: 128.0
15
- 2026-11-01 23:47:22 | INFO | LoRA rank: 16
16
- 2026-11-01 23:47:22 | INFO | LoRA alpha: 16
17
- 2026-11-01 23:47:22 | INFO | Trainable parameters: 37,158,912
18
- 2026-11-01 23:47:22 | INFO | GPU0 LoRA parameters: 17,694,720
19
- 2026-11-01 23:47:22 | INFO | GPU1 LoRA parameters: 19,464,192
20
- 2026-11-01 23:47:22 | INFO | Continuation loaded: True
21
- 2026-11-01 23:47:22 | INFO | Continuation loading enabled: True
22
- 2026-11-01 23:47:22 | INFO | HF repository: codeShare/kromoss-lora-exports
23
- 2026-11-01 23:47:22 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
- 2026-11-01 23:47:22 | INFO | GPU0: Tesla T4 | 14.562 GB
25
- 2026-11-01 23:47:22 | INFO | GPU1: Tesla T4 | 14.562 GB
26
- 2026-11-01 23:47:22 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
- 2026-11-01 23:47:22 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
- 2026-11-01 23:47:22 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
- 2026-11-01 23:47:22 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
- 2026-11-01 23:47:22 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
- 2026-11-01 23:47:22 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
- 2026-11-01 23:47:22 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
- 2026-11-01 23:47:22 | INFO | --------------------------------------------------------------------------------
34
- 2026-11-01 23:47:22 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
- 2026-11-01 23:47:22 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
- 2026-11-01 23:47:22 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
- 2026-11-01 23:47:22 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
- 2026-11-01 23:47:22 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
- 2026-11-01 23:47:22 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
- 2026-11-01 23:47:22 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
- 2026-11-01 23:47:22 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
- 2026-11-01 23:47:22 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
- 2026-11-01 23:47:22 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
- 2026-11-01 23:47:22 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
- 2026-11-01 23:47:22 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
- 2026-11-01 23:47:22 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
- 2026-11-01 23:47:22 | INFO | ================================================================================
48
- 2026-11-01 23:47:22 | INFO | STARTING TRAINING
49
- 2026-11-01 23:47:22 | INFO | ================================================================================
50
- 2026-11-01 23:47:22 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
- 2026-11-01 23:47:22 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
- 2026-11-01 23:47:22 | INFO | Two AdamW optimizers are used: optimizer0 for GPU0 LoRA parameters and optimizer1 for GPU1 LoRA parameters.
53
- 2026-11-01 23:47:22 | INFO | Both AdamW optimizer states persist across all 70 updates.
54
- 2026-11-01 23:47:22 | INFO | Learning rate does NOT reset when the dataset image changes.
55
- 2026-11-01 23:47:22 | INFO | Gradient clipping threshold: 0.5
56
- 2026-11-01 23:47:22 | INFO | Noise is generated in FP32 and cast to FP16.
57
- 2026-11-01 23:47:22 | INFO | Timestep is logged for every optimizer update.
58
- 2026-11-01 23:47:22 | INFO | Continuation loading enabled: True
59
- 2026-11-01 23:47:22 | INFO | Continuation actually loaded: True
60
- 2026-11-01 23:47:28 | ERROR | ================================================================================
61
- 2026-11-01 23:47:28 | ERROR | TRAINING FAILED
62
- 2026-11-01 23:47:28 | ERROR | Traceback (most recent call last):
63
- File "/tmp/ipykernel_934/333819112.py", line 1181, in <cell line: 0>
64
- result = train_one_image(
65
- dataset_index=dataset_index,
66
- step=step
67
- )
68
- File "/tmp/ipykernel_934/333819112.py", line 957, in train_one_image
69
- grad_x1 = stage1_backward(result)
70
- File "/tmp/ipykernel_934/333819112.py", line 593, in stage1_backward
71
- gradients = torch.autograd.grad(
72
- scaled_loss,
73
- [result["x1"]] + stage1_params,
74
- allow_unused=True
75
- )
76
- File "/usr/local/lib/python3.13/dist-packages/torch/autograd/__init__.py", line 532, in grad
77
- result = _engine_run_backward(
78
- outputs,
79
- ...<5 lines>...
80
- accumulate_grad=False,
81
- )
82
- File "/usr/local/lib/python3.13/dist-packages/torch/autograd/graph.py", line 869, in _engine_run_backward
83
- return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
84
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
85
- t_outputs, *args, **kwargs
86
- ^^^^^^^^^^^^^^^^^^^^^^^^^^
87
- ) # Calls into the C++ engine to run the backward pass
88
- ^
89
- File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1173, in unpack_hook
90
- _run_fn_with_dynamo_disabled(frame.recompute_fn, *args)
91
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
92
- File "/usr/local/lib/python3.13/dist-packages/torch/_compile.py", line 54, in inner
93
- return disable_fn(*args, **kwargs)
94
- File "/usr/local/lib/python3.13/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
95
- return fn(*args, **kwargs)
96
- File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1139, in _run_fn_with_dynamo_disabled
97
- return fn(*args, **kwargs)
98
- File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1607, in recompute_fn
99
- fn(*args, **kwargs)
100
- ~~^^^^^^^^^^^^^^^^^
101
- File "/tmp/ipykernel_934/333819112.py", line 362, in run_group
102
- x = base.layers[layer_index](
103
- x,
104
- rotary,
105
- mod
106
- )
107
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
108
- return self._call_impl(*args, **kwargs)
109
- ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
110
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
111
- return forward_call(*args, **kwargs)
112
- File "/kaggle/working/pipeline_moss_image_patched.py", line 147, in forward
113
- hidden, gate = self.proj['ffn_up_gate'](y.to(dtype)).chunk(2, -1)
114
- ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
115
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
116
- return self._call_impl(*args, **kwargs)
117
- ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
118
- File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
119
- return forward_call(*args, **kwargs)
120
- File "/usr/local/lib/python3.13/dist-packages/peft/tuners/lora/layer.py", line 1102, in forward
121
- result = result + lora_B(lora_A(dropout(x))) * scaling
122
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~
123
- torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 146.00 MiB. GPU 1 has a total capacity of 14.56 GiB of which 116.81 MiB is free. Including non-PyTorch memory, this process has 14.45 GiB memory in use. Of the allocated memory 14.20 GiB is allocated by PyTorch, and 111.55 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
124
-
125
- 2026-11-01 23:47:28 | INFO | ERROR STATE | GPU0 | allocated=4.927 GB | reserved=5.678 GB | peak=5.575 GB | free=8.755 GB
126
- 2026-11-01 23:47:28 | INFO | ERROR STATE | GPU1 | allocated=14.000 GB | reserved=14.311 GB | peak=14.202 GB | free=0.114 GB
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-11-01 23:49:16 | INFO | ================================================================================
2
+ 2026-11-01 23:49:16 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
+ 2026-11-01 23:49:16 | INFO | ================================================================================
4
+ 2026-11-01 23:49:16 | INFO | PyTorch version: 2.11.0+cu128
5
+ 2026-11-01 23:49:16 | INFO | CUDA version: 12.8
6
+ 2026-11-01 23:49:16 | INFO | Dataset size: 7
7
+ 2026-11-01 23:49:16 | INFO | Presentations per image: 10
8
+ 2026-11-01 23:49:16 | INFO | Total optimizer updates: 70
9
+ 2026-11-01 23:49:16 | INFO | Checkpoint steps: [14, 35, 70]
10
+ 2026-11-01 23:49:16 | INFO | Initial global learning rate: 5.00000000e-05
11
+ 2026-11-01 23:49:16 | INFO | Final global learning rate: 2.50000000e-05
12
+ 2026-11-01 23:49:16 | INFO | Weight decay: 0.01
13
+ 2026-11-01 23:49:16 | INFO | Gradient clipping: 0.5
14
+ 2026-11-01 23:49:16 | INFO | Loss scale: 128.0
15
+ 2026-11-01 23:49:16 | INFO | LoRA rank: 16
16
+ 2026-11-01 23:49:16 | INFO | LoRA alpha: 16
17
+ 2026-11-01 23:49:16 | INFO | Trainable parameters: 37,158,912
18
+ 2026-11-01 23:49:16 | INFO | GPU0 LoRA parameters: 17,694,720
19
+ 2026-11-01 23:49:16 | INFO | GPU1 LoRA parameters: 19,464,192
20
+ 2026-11-01 23:49:16 | INFO | Continuation loaded: True
21
+ 2026-11-01 23:49:16 | INFO | Continuation loading enabled: True
22
+ 2026-11-01 23:49:16 | INFO | HF repository: codeShare/kromoss-lora-exports
23
+ 2026-11-01 23:49:16 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
+ 2026-11-01 23:49:16 | INFO | GPU0: Tesla T4 | 14.562 GB
25
+ 2026-11-01 23:49:16 | INFO | GPU1: Tesla T4 | 14.562 GB
26
+ 2026-11-01 23:49:16 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
+ 2026-11-01 23:49:16 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
+ 2026-11-01 23:49:16 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
+ 2026-11-01 23:49:16 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
+ 2026-11-01 23:49:16 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
+ 2026-11-01 23:49:16 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
+ 2026-11-01 23:49:16 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
+ 2026-11-01 23:49:16 | INFO | --------------------------------------------------------------------------------
34
+ 2026-11-01 23:49:16 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
+ 2026-11-01 23:49:16 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
+ 2026-11-01 23:49:16 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
+ 2026-11-01 23:49:16 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
+ 2026-11-01 23:49:16 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
+ 2026-11-01 23:49:16 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
+ 2026-11-01 23:49:16 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
+ 2026-11-01 23:49:16 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
+ 2026-11-01 23:49:16 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
+ 2026-11-01 23:49:16 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
+ 2026-11-01 23:49:16 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
+ 2026-11-01 23:49:16 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
+ 2026-11-01 23:49:16 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
+ 2026-11-01 23:49:16 | INFO | ================================================================================
48
+ 2026-11-01 23:49:16 | INFO | STARTING TRAINING
49
+ 2026-11-01 23:49:16 | INFO | ================================================================================
50
+ 2026-11-01 23:49:16 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
+ 2026-11-01 23:49:16 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
+ 2026-11-01 23:49:16 | INFO | Two AdamW optimizers are used: optimizer0 for GPU0 LoRA parameters and optimizer1 for GPU1 LoRA parameters.
53
+ 2026-11-01 23:49:16 | INFO | Both AdamW optimizer states persist across all 70 updates.
54
+ 2026-11-01 23:49:16 | INFO | Learning rate does NOT reset when the dataset image changes.
55
+ 2026-11-01 23:49:16 | INFO | Gradient clipping threshold: 0.5
56
+ 2026-11-01 23:49:16 | INFO | Noise is generated in FP32 and cast to FP16.
57
+ 2026-11-01 23:49:16 | INFO | Timestep is logged for every optimizer update.
58
+ 2026-11-01 23:49:16 | INFO | Continuation loading enabled: True
59
+ 2026-11-01 23:49:16 | INFO | Continuation actually loaded: True
60
+ 2026-11-01 23:49:34 | INFO | --------------------------------------------------------------------------------
61
+ 2026-11-01 23:49:34 | INFO | STEP 1/70 | epoch=1/10 | position=1/7 | dataset_item=1
62
+ 2026-11-01 23:49:34 | INFO | DATASET FILE: 00000001.pt
63
+ 2026-11-01 23:49:34 | INFO | LOSS: 1.17529559
64
+ 2026-11-01 23:49:34 | INFO | LR: 5.00000000e-05
65
+ 2026-11-01 23:49:34 | INFO | TIMESTEP: 0.32641813
66
+ 2026-11-01 23:49:34 | INFO | GLOBAL GRADIENT NORM: 0.44340117
67
+ 2026-11-01 23:49:34 | INFO | MAX ABSOLUTE GRADIENT: 2.37274170e-02
68
+ 2026-11-01 23:49:34 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
69
+ 2026-11-01 23:49:34 | INFO | MISSING GRADIENT TENSORS: 0
70
+ 2026-11-01 23:49:34 | INFO | GPU0 FORWARD: 2.471s
71
+ 2026-11-01 23:49:34 | INFO | GPU1 FORWARD: 2.353s
72
+ 2026-11-01 23:49:34 | INFO | GPU1 BACKWARD: 6.042s
73
+ 2026-11-01 23:49:34 | INFO | GPU0 BACKWARD: 6.124s
74
+ 2026-11-01 23:49:34 | INFO | UPDATE WALL TIME: 17.371s
75
+ 2026-11-01 23:49:34 | INFO | ELAPSED TRAINING TIME: 17.42s
76
+ 2026-11-01 23:49:34 | INFO | STEP 1 | GPU0 | allocated=5.070 GB | reserved=12.922 GB | peak=12.624 GB | free=1.503 GB
77
+ 2026-11-01 23:49:34 | INFO | STEP 1 | GPU1 | allocated=4.984 GB | reserved=12.855 GB | peak=12.519 GB | free=1.569 GB
78
+ 2026-11-01 23:49:52 | INFO | --------------------------------------------------------------------------------
79
+ 2026-11-01 23:49:52 | INFO | STEP 2/70 | epoch=1/10 | position=2/7 | dataset_item=3
80
+ 2026-11-01 23:49:52 | INFO | DATASET FILE: 00000003.pt
81
+ 2026-11-01 23:49:52 | INFO | LOSS: 1.27852416
82
+ 2026-11-01 23:49:52 | INFO | LR: 4.96376812e-05
83
+ 2026-11-01 23:49:52 | INFO | TIMESTEP: 0.26634458
84
+ 2026-11-01 23:49:52 | INFO | GLOBAL GRADIENT NORM: 0.56801107
85
+ 2026-11-01 23:49:52 | INFO | MAX ABSOLUTE GRADIENT: 2.33612061e-02
86
+ 2026-11-01 23:49:52 | INFO | GRADIENT CLIP COEFFICIENT: 0.88026299
87
+ 2026-11-01 23:49:52 | INFO | MISSING GRADIENT TENSORS: 0
88
+ 2026-11-01 23:49:52 | INFO | GPU0 FORWARD: 2.445s
89
+ 2026-11-01 23:49:52 | INFO | GPU1 FORWARD: 2.403s
90
+ 2026-11-01 23:49:52 | INFO | GPU1 BACKWARD: 6.323s
91
+ 2026-11-01 23:49:52 | INFO | GPU0 BACKWARD: 6.450s
92
+ 2026-11-01 23:49:52 | INFO | UPDATE WALL TIME: 17.736s
93
+ 2026-11-01 23:49:52 | INFO | ELAPSED TRAINING TIME: 35.17s
94
+ 2026-11-01 23:49:52 | INFO | STEP 2 | GPU0 | allocated=5.070 GB | reserved=13.061 GB | peak=12.898 GB | free=1.364 GB
95
+ 2026-11-01 23:49:52 | INFO | STEP 2 | GPU1 | allocated=4.984 GB | reserved=12.992 GB | peak=12.805 GB | free=1.432 GB
96
+ 2026-11-01 23:50:10 | INFO | --------------------------------------------------------------------------------
97
+ 2026-11-01 23:50:10 | INFO | STEP 3/70 | epoch=1/10 | position=3/7 | dataset_item=4
98
+ 2026-11-01 23:50:10 | INFO | DATASET FILE: 00000004.pt
99
+ 2026-11-01 23:50:10 | INFO | LOSS: 1.32079029
100
+ 2026-11-01 23:50:10 | INFO | LR: 4.92753623e-05
101
+ 2026-11-01 23:50:10 | INFO | TIMESTEP: 0.25900331
102
+ 2026-11-01 23:50:10 | INFO | GLOBAL GRADIENT NORM: 0.68435715
103
+ 2026-11-01 23:50:10 | INFO | MAX ABSOLUTE GRADIENT: 2.75268555e-02
104
+ 2026-11-01 23:50:10 | INFO | GRADIENT CLIP COEFFICIENT: 0.73061160
105
+ 2026-11-01 23:50:10 | INFO | MISSING GRADIENT TENSORS: 0
106
+ 2026-11-01 23:50:10 | INFO | GPU0 FORWARD: 2.583s
107
+ 2026-11-01 23:50:10 | INFO | GPU1 FORWARD: 2.421s
108
+ 2026-11-01 23:50:10 | INFO | GPU1 BACKWARD: 6.376s
109
+ 2026-11-01 23:50:10 | INFO | GPU0 BACKWARD: 6.521s
110
+ 2026-11-01 23:50:10 | INFO | UPDATE WALL TIME: 18.014s
111
+ 2026-11-01 23:50:10 | INFO | ELAPSED TRAINING TIME: 53.19s
112
+ 2026-11-01 23:50:10 | INFO | STEP 3 | GPU0 | allocated=5.070 GB | reserved=13.061 GB | peak=12.862 GB | free=1.364 GB
113
+ 2026-11-01 23:50:10 | INFO | STEP 3 | GPU1 | allocated=4.984 GB | reserved=12.992 GB | peak=12.768 GB | free=1.432 GB
114
+ 2026-11-01 23:50:27 | INFO | --------------------------------------------------------------------------------
115
+ 2026-11-01 23:50:27 | INFO | STEP 4/70 | epoch=1/10 | position=4/7 | dataset_item=2
116
+ 2026-11-01 23:50:27 | INFO | DATASET FILE: 00000002.pt
117
+ 2026-11-01 23:50:27 | INFO | LOSS: 1.20202696
118
+ 2026-11-01 23:50:27 | INFO | LR: 4.89130435e-05
119
+ 2026-11-01 23:50:27 | INFO | TIMESTEP: 0.32806167
120
+ 2026-11-01 23:50:27 | INFO | GLOBAL GRADIENT NORM: 0.34136441
121
+ 2026-11-01 23:50:27 | INFO | MAX ABSOLUTE GRADIENT: 2.55126953e-02
122
+ 2026-11-01 23:50:27 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
123
+ 2026-11-01 23:50:27 | INFO | MISSING GRADIENT TENSORS: 0
124
+ 2026-11-01 23:50:27 | INFO | GPU0 FORWARD: 2.520s
125
+ 2026-11-01 23:50:27 | INFO | GPU1 FORWARD: 2.368s
126
+ 2026-11-01 23:50:27 | INFO | GPU1 BACKWARD: 6.184s
127
+ 2026-11-01 23:50:27 | INFO | GPU0 BACKWARD: 6.228s
128
+ 2026-11-01 23:50:27 | INFO | UPDATE WALL TIME: 17.411s
129
+ 2026-11-01 23:50:27 | INFO | ELAPSED TRAINING TIME: 70.61s
130
+ 2026-11-01 23:50:27 | INFO | STEP 4 | GPU0 | allocated=5.070 GB | reserved=13.061 GB | peak=12.835 GB | free=1.364 GB
131
+ 2026-11-01 23:50:27 | INFO | STEP 4 | GPU1 | allocated=4.984 GB | reserved=12.992 GB | peak=12.742 GB | free=1.432 GB
132
+ 2026-11-01 23:50:44 | INFO | --------------------------------------------------------------------------------
133
+ 2026-11-01 23:50:44 | INFO | STEP 5/70 | epoch=1/10 | position=5/7 | dataset_item=6
134
+ 2026-11-01 23:50:44 | INFO | DATASET FILE: 00000006.pt
135
+ 2026-11-01 23:50:44 | INFO | LOSS: 1.38897181
136
+ 2026-11-01 23:50:44 | INFO | LR: 4.85507246e-05
137
+ 2026-11-01 23:50:44 | INFO | TIMESTEP: 0.22171448
138
+ 2026-11-01 23:50:44 | INFO | GLOBAL GRADIENT NORM: 1.69970331
139
+ 2026-11-01 23:50:44 | INFO | MAX ABSOLUTE GRADIENT: 9.50927734e-02
140
+ 2026-11-01 23:50:44 | INFO | GRADIENT CLIP COEFFICIENT: 0.29416881
141
+ 2026-11-01 23:50:44 | INFO | MISSING GRADIENT TENSORS: 0
142
+ 2026-11-01 23:50:44 | INFO | GPU0 FORWARD: 2.442s
143
+ 2026-11-01 23:50:44 | INFO | GPU1 FORWARD: 2.374s
144
+ 2026-11-01 23:50:44 | INFO | GPU1 BACKWARD: 6.268s
145
+ 2026-11-01 23:50:44 | INFO | GPU0 BACKWARD: 6.246s
146
+ 2026-11-01 23:50:44 | INFO | UPDATE WALL TIME: 17.440s
147
+ 2026-11-01 23:50:44 | INFO | ELAPSED TRAINING TIME: 88.07s
148
+ 2026-11-01 23:50:44 | INFO | STEP 5 | GPU0 | allocated=5.070 GB | reserved=13.061 GB | peak=12.882 GB | free=1.364 GB
149
+ 2026-11-01 23:50:44 | INFO | STEP 5 | GPU1 | allocated=4.984 GB | reserved=12.992 GB | peak=12.788 GB | free=1.432 GB
150
+ 2026-11-01 23:51:02 | INFO | --------------------------------------------------------------------------------
151
+ 2026-11-01 23:51:02 | INFO | STEP 6/70 | epoch=1/10 | position=6/7 | dataset_item=0
152
+ 2026-11-01 23:51:02 | INFO | DATASET FILE: 00000000.pt
153
+ 2026-11-01 23:51:02 | INFO | LOSS: 1.43497896
154
+ 2026-11-01 23:51:02 | INFO | LR: 4.81884058e-05
155
+ 2026-11-01 23:51:02 | INFO | TIMESTEP: 0.14347191
156
+ 2026-11-01 23:51:02 | INFO | GLOBAL GRADIENT NORM: 2.54458823
157
+ 2026-11-01 23:51:02 | INFO | MAX ABSOLUTE GRADIENT: 1.45141602e-01
158
+ 2026-11-01 23:51:02 | INFO | GRADIENT CLIP COEFFICIENT: 0.19649537
159
+ 2026-11-01 23:51:02 | INFO | MISSING GRADIENT TENSORS: 0
160
+ 2026-11-01 23:51:02 | INFO | GPU0 FORWARD: 2.438s
161
+ 2026-11-01 23:51:02 | INFO | GPU1 FORWARD: 2.382s
162
+ 2026-11-01 23:51:02 | INFO | GPU1 BACKWARD: 6.253s
163
+ 2026-11-01 23:51:02 | INFO | GPU0 BACKWARD: 6.345s
164
+ 2026-11-01 23:51:02 | INFO | UPDATE WALL TIME: 17.534s
165
+ 2026-11-01 23:51:02 | INFO | ELAPSED TRAINING TIME: 105.61s
166
+ 2026-11-01 23:51:02 | INFO | STEP 6 | GPU0 | allocated=5.070 GB | reserved=13.061 GB | peak=12.821 GB | free=1.364 GB
167
+ 2026-11-01 23:51:02 | INFO | STEP 6 | GPU1 | allocated=4.984 GB | reserved=12.992 GB | peak=12.728 GB | free=1.432 GB
168
+ 2026-11-01 23:51:20 | INFO | --------------------------------------------------------------------------------
169
+ 2026-11-01 23:51:20 | INFO | STEP 7/70 | epoch=1/10 | position=7/7 | dataset_item=5
170
+ 2026-11-01 23:51:20 | INFO | DATASET FILE: 00000005.pt
171
+ 2026-11-01 23:51:20 | INFO | LOSS: 0.72653091
172
+ 2026-11-01 23:51:20 | INFO | LR: 4.78260870e-05
173
+ 2026-11-01 23:51:20 | INFO | TIMESTEP: 0.57760578
174
+ 2026-11-01 23:51:20 | INFO | GLOBAL GRADIENT NORM: 0.37167326
175
+ 2026-11-01 23:51:20 | INFO | MAX ABSOLUTE GRADIENT: 1.80664062e-02
176
+ 2026-11-01 23:51:20 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
177
+ 2026-11-01 23:51:20 | INFO | MISSING GRADIENT TENSORS: 0
178
+ 2026-11-01 23:51:20 | INFO | GPU0 FORWARD: 2.588s
179
+ 2026-11-01 23:51:20 | INFO | GPU1 FORWARD: 2.456s
180
+ 2026-11-01 23:51:20 | INFO | GPU1 BACKWARD: 6.403s
181
+ 2026-11-01 23:51:20 | INFO | GPU0 BACKWARD: 6.455s
182
+ 2026-11-01 23:51:20 | INFO | UPDATE WALL TIME: 18.011s
183
+ 2026-11-01 23:51:20 | INFO | ELAPSED TRAINING TIME: 123.63s
184
+ 2026-11-01 23:51:20 | INFO | STEP 7 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.935 GB | free=1.325 GB
185
+ 2026-11-01 23:51:20 | INFO | STEP 7 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.842 GB | free=1.393 GB
186
+ 2026-11-01 23:51:38 | INFO | --------------------------------------------------------------------------------
187
+ 2026-11-01 23:51:38 | INFO | STEP 8/70 | epoch=2/10 | position=1/7 | dataset_item=3
188
+ 2026-11-01 23:51:38 | INFO | DATASET FILE: 00000003.pt
189
+ 2026-11-01 23:51:38 | INFO | LOSS: 1.27394319
190
+ 2026-11-01 23:51:38 | INFO | LR: 4.74637681e-05
191
+ 2026-11-01 23:51:38 | INFO | TIMESTEP: 0.27784425
192
+ 2026-11-01 23:51:38 | INFO | GLOBAL GRADIENT NORM: 0.55349499
193
+ 2026-11-01 23:51:38 | INFO | MAX ABSOLUTE GRADIENT: 3.02734375e-02
194
+ 2026-11-01 23:51:38 | INFO | GRADIENT CLIP COEFFICIENT: 0.90334891
195
+ 2026-11-01 23:51:38 | INFO | MISSING GRADIENT TENSORS: 0
196
+ 2026-11-01 23:51:38 | INFO | GPU0 FORWARD: 2.514s
197
+ 2026-11-01 23:51:38 | INFO | GPU1 FORWARD: 2.423s
198
+ 2026-11-01 23:51:38 | INFO | GPU1 BACKWARD: 6.299s
199
+ 2026-11-01 23:51:38 | INFO | GPU0 BACKWARD: 6.331s
200
+ 2026-11-01 23:51:38 | INFO | UPDATE WALL TIME: 17.678s
201
+ 2026-11-01 23:51:38 | INFO | ELAPSED TRAINING TIME: 141.32s
202
+ 2026-11-01 23:51:38 | INFO | STEP 8 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.898 GB | free=1.325 GB
203
+ 2026-11-01 23:51:38 | INFO | STEP 8 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.805 GB | free=1.393 GB
204
+ 2026-11-01 23:51:55 | INFO | --------------------------------------------------------------------------------
205
+ 2026-11-01 23:51:55 | INFO | STEP 9/70 | epoch=2/10 | position=2/7 | dataset_item=6
206
+ 2026-11-01 23:51:55 | INFO | DATASET FILE: 00000006.pt
207
+ 2026-11-01 23:51:55 | INFO | LOSS: 0.58841467
208
+ 2026-11-01 23:51:55 | INFO | LR: 4.71014493e-05
209
+ 2026-11-01 23:51:55 | INFO | TIMESTEP: 0.73534703
210
+ 2026-11-01 23:51:55 | INFO | GLOBAL GRADIENT NORM: 0.18841203
211
+ 2026-11-01 23:51:55 | INFO | MAX ABSOLUTE GRADIENT: 6.60324097e-03
212
+ 2026-11-01 23:51:55 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
213
+ 2026-11-01 23:51:55 | INFO | MISSING GRADIENT TENSORS: 0
214
+ 2026-11-01 23:51:55 | INFO | GPU0 FORWARD: 2.485s
215
+ 2026-11-01 23:51:55 | INFO | GPU1 FORWARD: 2.398s
216
+ 2026-11-01 23:51:55 | INFO | GPU1 BACKWARD: 6.312s
217
+ 2026-11-01 23:51:55 | INFO | GPU0 BACKWARD: 6.171s
218
+ 2026-11-01 23:51:55 | INFO | UPDATE WALL TIME: 17.475s
219
+ 2026-11-01 23:51:55 | INFO | ELAPSED TRAINING TIME: 158.81s
220
+ 2026-11-01 23:51:55 | INFO | STEP 9 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.882 GB | free=1.325 GB
221
+ 2026-11-01 23:51:55 | INFO | STEP 9 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.788 GB | free=1.393 GB
222
+ 2026-11-01 23:52:13 | INFO | --------------------------------------------------------------------------------
223
+ 2026-11-01 23:52:13 | INFO | STEP 10/70 | epoch=2/10 | position=3/7 | dataset_item=2
224
+ 2026-11-01 23:52:13 | INFO | DATASET FILE: 00000002.pt
225
+ 2026-11-01 23:52:13 | INFO | LOSS: 0.76026332
226
+ 2026-11-01 23:52:13 | INFO | LR: 4.67391304e-05
227
+ 2026-11-01 23:52:13 | INFO | TIMESTEP: 0.57242364
228
+ 2026-11-01 23:52:13 | INFO | GLOBAL GRADIENT NORM: 0.26225442
229
+ 2026-11-01 23:52:13 | INFO | MAX ABSOLUTE GRADIENT: 1.77917480e-02
230
+ 2026-11-01 23:52:13 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
231
+ 2026-11-01 23:52:13 | INFO | MISSING GRADIENT TENSORS: 0
232
+ 2026-11-01 23:52:13 | INFO | GPU0 FORWARD: 2.436s
233
+ 2026-11-01 23:52:13 | INFO | GPU1 FORWARD: 2.398s
234
+ 2026-11-01 23:52:13 | INFO | GPU1 BACKWARD: 6.254s
235
+ 2026-11-01 23:52:13 | INFO | GPU0 BACKWARD: 6.302s
236
+ 2026-11-01 23:52:13 | INFO | UPDATE WALL TIME: 17.501s
237
+ 2026-11-01 23:52:13 | INFO | ELAPSED TRAINING TIME: 176.32s
238
+ 2026-11-01 23:52:13 | INFO | STEP 10 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.835 GB | free=1.325 GB
239
+ 2026-11-01 23:52:13 | INFO | STEP 10 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.742 GB | free=1.393 GB
240
+ 2026-11-01 23:52:30 | INFO | --------------------------------------------------------------------------------
241
+ 2026-11-01 23:52:30 | INFO | STEP 11/70 | epoch=2/10 | position=4/7 | dataset_item=4
242
+ 2026-11-01 23:52:30 | INFO | DATASET FILE: 00000004.pt
243
+ 2026-11-01 23:52:30 | INFO | LOSS: 0.56975603
244
+ 2026-11-01 23:52:30 | INFO | LR: 4.63768116e-05
245
+ 2026-11-01 23:52:30 | INFO | TIMESTEP: 0.72895420
246
+ 2026-11-01 23:52:30 | INFO | GLOBAL GRADIENT NORM: 0.16596660
247
+ 2026-11-01 23:52:30 | INFO | MAX ABSOLUTE GRADIENT: 8.97979736e-03
248
+ 2026-11-01 23:52:30 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
249
+ 2026-11-01 23:52:30 | INFO | MISSING GRADIENT TENSORS: 0
250
+ 2026-11-01 23:52:30 | INFO | GPU0 FORWARD: 2.528s
251
+ 2026-11-01 23:52:30 | INFO | GPU1 FORWARD: 2.388s
252
+ 2026-11-01 23:52:30 | INFO | GPU1 BACKWARD: 6.322s
253
+ 2026-11-01 23:52:30 | INFO | GPU0 BACKWARD: 6.213s
254
+ 2026-11-01 23:52:30 | INFO | UPDATE WALL TIME: 17.560s
255
+ 2026-11-01 23:52:30 | INFO | ELAPSED TRAINING TIME: 193.89s
256
+ 2026-11-01 23:52:30 | INFO | STEP 11 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.862 GB | free=1.325 GB
257
+ 2026-11-01 23:52:30 | INFO | STEP 11 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.768 GB | free=1.393 GB
258
+ 2026-11-01 23:52:48 | INFO | --------------------------------------------------------------------------------
259
+ 2026-11-01 23:52:48 | INFO | STEP 12/70 | epoch=2/10 | position=5/7 | dataset_item=0
260
+ 2026-11-01 23:52:48 | INFO | DATASET FILE: 00000000.pt
261
+ 2026-11-01 23:52:48 | INFO | LOSS: 1.42430818
262
+ 2026-11-01 23:52:48 | INFO | LR: 4.60144928e-05
263
+ 2026-11-01 23:52:48 | INFO | TIMESTEP: 0.10170523
264
+ 2026-11-01 23:52:48 | INFO | GLOBAL GRADIENT NORM: 2.14350095
265
+ 2026-11-01 23:52:48 | INFO | MAX ABSOLUTE GRADIENT: 9.80834961e-02
266
+ 2026-11-01 23:52:48 | INFO | GRADIENT CLIP COEFFICIENT: 0.23326314
267
+ 2026-11-01 23:52:48 | INFO | MISSING GRADIENT TENSORS: 0
268
+ 2026-11-01 23:52:48 | INFO | GPU0 FORWARD: 2.428s
269
+ 2026-11-01 23:52:48 | INFO | GPU1 FORWARD: 2.407s
270
+ 2026-11-01 23:52:48 | INFO | GPU1 BACKWARD: 6.261s
271
+ 2026-11-01 23:52:48 | INFO | GPU0 BACKWARD: 6.368s
272
+ 2026-11-01 23:52:48 | INFO | UPDATE WALL TIME: 17.581s
273
+ 2026-11-01 23:52:48 | INFO | ELAPSED TRAINING TIME: 211.49s
274
+ 2026-11-01 23:52:48 | INFO | STEP 12 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.821 GB | free=1.325 GB
275
+ 2026-11-01 23:52:48 | INFO | STEP 12 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.728 GB | free=1.393 GB
276
+ 2026-11-01 23:53:06 | INFO | --------------------------------------------------------------------------------
277
+ 2026-11-01 23:53:06 | INFO | STEP 13/70 | epoch=2/10 | position=6/7 | dataset_item=5
278
+ 2026-11-01 23:53:06 | INFO | DATASET FILE: 00000005.pt
279
+ 2026-11-01 23:53:06 | INFO | LOSS: 0.53494245
280
+ 2026-11-01 23:53:06 | INFO | LR: 4.56521739e-05
281
+ 2026-11-01 23:53:06 | INFO | TIMESTEP: 0.83249700
282
+ 2026-11-01 23:53:06 | INFO | GLOBAL GRADIENT NORM: 0.17331558
283
+ 2026-11-01 23:53:06 | INFO | MAX ABSOLUTE GRADIENT: 1.04598999e-02
284
+ 2026-11-01 23:53:06 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
285
+ 2026-11-01 23:53:06 | INFO | MISSING GRADIENT TENSORS: 0
286
+ 2026-11-01 23:53:06 | INFO | GPU0 FORWARD: 2.590s
287
+ 2026-11-01 23:53:06 | INFO | GPU1 FORWARD: 2.448s
288
+ 2026-11-01 23:53:06 | INFO | GPU1 BACKWARD: 6.391s
289
+ 2026-11-01 23:53:06 | INFO | GPU0 BACKWARD: 6.281s
290
+ 2026-11-01 23:53:06 | INFO | UPDATE WALL TIME: 17.818s
291
+ 2026-11-01 23:53:06 | INFO | ELAPSED TRAINING TIME: 229.32s
292
+ 2026-11-01 23:53:06 | INFO | STEP 13 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.935 GB | free=1.325 GB
293
+ 2026-11-01 23:53:06 | INFO | STEP 13 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.842 GB | free=1.393 GB
294
+ 2026-11-01 23:53:23 | INFO | --------------------------------------------------------------------------------
295
+ 2026-11-01 23:53:23 | INFO | STEP 14/70 | epoch=2/10 | position=7/7 | dataset_item=1
296
+ 2026-11-01 23:53:23 | INFO | DATASET FILE: 00000001.pt
297
+ 2026-11-01 23:53:23 | INFO | LOSS: 0.60880160
298
+ 2026-11-01 23:53:23 | INFO | LR: 4.52898551e-05
299
+ 2026-11-01 23:53:23 | INFO | TIMESTEP: 0.68447441
300
+ 2026-11-01 23:53:23 | INFO | GLOBAL GRADIENT NORM: 0.17715210
301
+ 2026-11-01 23:53:23 | INFO | MAX ABSOLUTE GRADIENT: 9.48333740e-03
302
+ 2026-11-01 23:53:23 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
303
+ 2026-11-01 23:53:23 | INFO | MISSING GRADIENT TENSORS: 0
304
+ 2026-11-01 23:53:23 | INFO | GPU0 FORWARD: 2.420s
305
+ 2026-11-01 23:53:23 | INFO | GPU1 FORWARD: 2.365s
306
+ 2026-11-01 23:53:23 | INFO | GPU1 BACKWARD: 6.250s
307
+ 2026-11-01 23:53:23 | INFO | GPU0 BACKWARD: 6.220s
308
+ 2026-11-01 23:53:23 | INFO | UPDATE WALL TIME: 17.364s
309
+ 2026-11-01 23:53:23 | INFO | ELAPSED TRAINING TIME: 246.69s
310
+ 2026-11-01 23:53:23 | INFO | STEP 14 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.756 GB | free=1.325 GB
311
+ 2026-11-01 23:53:23 | INFO | STEP 14 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.664 GB | free=1.393 GB
312
+ 2026-11-01 23:53:23 | INFO | ================================================================================
313
+ 2026-11-01 23:53:23 | INFO | CHECKPOINT TRIGGERED AT OPTIMIZER STEP 14
314
+ 2026-11-01 23:53:27 | INFO | CHECKPOINT SUCCESS: kromoss_lora_step_000014.safetensors
315
+ 2026-11-01 23:53:27 | INFO | CHECKPOINT SIZE: 70.93 MB
316
+ 2026-11-01 23:53:27 | INFO | ================================================================================
317
+ 2026-11-01 23:53:44 | INFO | --------------------------------------------------------------------------------
318
+ 2026-11-01 23:53:44 | INFO | STEP 15/70 | epoch=3/10 | position=1/7 | dataset_item=3
319
+ 2026-11-01 23:53:44 | INFO | DATASET FILE: 00000003.pt
320
+ 2026-11-01 23:53:44 | INFO | LOSS: 1.31025481
321
+ 2026-11-01 23:53:44 | INFO | LR: 4.49275362e-05
322
+ 2026-11-01 23:53:44 | INFO | TIMESTEP: 0.27099666
323
+ 2026-11-01 23:53:44 | INFO | GLOBAL GRADIENT NORM: 0.79218926
324
+ 2026-11-01 23:53:44 | INFO | MAX ABSOLUTE GRADIENT: 2.90374756e-02
325
+ 2026-11-01 23:53:44 | INFO | GRADIENT CLIP COEFFICIENT: 0.63116151
326
+ 2026-11-01 23:53:44 | INFO | MISSING GRADIENT TENSORS: 0
327
+ 2026-11-01 23:53:44 | INFO | GPU0 FORWARD: 2.503s
328
+ 2026-11-01 23:53:44 | INFO | GPU1 FORWARD: 2.377s
329
+ 2026-11-01 23:53:44 | INFO | GPU1 BACKWARD: 6.222s
330
+ 2026-11-01 23:53:44 | INFO | GPU0 BACKWARD: 6.279s
331
+ 2026-11-01 23:53:44 | INFO | UPDATE WALL TIME: 17.500s
332
+ 2026-11-01 23:53:44 | INFO | ELAPSED TRAINING TIME: 268.02s
333
+ 2026-11-01 23:53:44 | INFO | STEP 15 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.898 GB | free=1.325 GB
334
+ 2026-11-01 23:53:44 | INFO | STEP 15 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.805 GB | free=1.393 GB
335
+ 2026-11-01 23:54:02 | INFO | --------------------------------------------------------------------------------
336
+ 2026-11-01 23:54:02 | INFO | STEP 16/70 | epoch=3/10 | position=2/7 | dataset_item=2
337
+ 2026-11-01 23:54:02 | INFO | DATASET FILE: 00000002.pt
338
+ 2026-11-01 23:54:02 | INFO | LOSS: 1.20411885
339
+ 2026-11-01 23:54:02 | INFO | LR: 4.45652174e-05
340
+ 2026-11-01 23:54:02 | INFO | TIMESTEP: 0.34128180
341
+ 2026-11-01 23:54:02 | INFO | GLOBAL GRADIENT NORM: 0.56528637
342
+ 2026-11-01 23:54:02 | INFO | MAX ABSOLUTE GRADIENT: 2.06604004e-02
343
+ 2026-11-01 23:54:02 | INFO | GRADIENT CLIP COEFFICIENT: 0.88450588
344
+ 2026-11-01 23:54:02 | INFO | MISSING GRADIENT TENSORS: 0
345
+ 2026-11-01 23:54:02 | INFO | GPU0 FORWARD: 2.443s
346
+ 2026-11-01 23:54:02 | INFO | GPU1 FORWARD: 2.390s
347
+ 2026-11-01 23:54:02 | INFO | GPU1 BACKWARD: 6.286s
348
+ 2026-11-01 23:54:02 | INFO | GPU0 BACKWARD: 6.387s
349
+ 2026-11-01 23:54:02 | INFO | UPDATE WALL TIME: 17.621s
350
+ 2026-11-01 23:54:02 | INFO | ELAPSED TRAINING TIME: 285.65s
351
+ 2026-11-01 23:54:02 | INFO | STEP 16 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.835 GB | free=1.325 GB
352
+ 2026-11-01 23:54:02 | INFO | STEP 16 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.742 GB | free=1.393 GB
353
+ 2026-11-01 23:54:20 | INFO | --------------------------------------------------------------------------------
354
+ 2026-11-01 23:54:20 | INFO | STEP 17/70 | epoch=3/10 | position=3/7 | dataset_item=4
355
+ 2026-11-01 23:54:20 | INFO | DATASET FILE: 00000004.pt
356
+ 2026-11-01 23:54:20 | INFO | LOSS: 1.85224771
357
+ 2026-11-01 23:54:20 | INFO | LR: 4.42028986e-05
358
+ 2026-11-01 23:54:20 | INFO | TIMESTEP: 0.00303028
359
+ 2026-11-01 23:54:20 | INFO | GLOBAL GRADIENT NORM: 5.27555570
360
+ 2026-11-01 23:54:20 | INFO | MAX ABSOLUTE GRADIENT: 7.31933594e-01
361
+ 2026-11-01 23:54:20 | INFO | GRADIENT CLIP COEFFICIENT: 0.09477673
362
+ 2026-11-01 23:54:20 | INFO | MISSING GRADIENT TENSORS: 0
363
+ 2026-11-01 23:54:20 | INFO | GPU0 FORWARD: 2.551s
364
+ 2026-11-01 23:54:20 | INFO | GPU1 FORWARD: 2.385s
365
+ 2026-11-01 23:54:20 | INFO | GPU1 BACKWARD: 6.357s
366
+ 2026-11-01 23:54:20 | INFO | GPU0 BACKWARD: 6.393s
367
+ 2026-11-01 23:54:20 | INFO | UPDATE WALL TIME: 17.807s
368
+ 2026-11-01 23:54:20 | INFO | ELAPSED TRAINING TIME: 303.47s
369
+ 2026-11-01 23:54:20 | INFO | STEP 17 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.862 GB | free=1.325 GB
370
+ 2026-11-01 23:54:20 | INFO | STEP 17 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.768 GB | free=1.393 GB
371
+ 2026-11-01 23:54:37 | INFO | --------------------------------------------------------------------------------
372
+ 2026-11-01 23:54:37 | INFO | STEP 18/70 | epoch=3/10 | position=4/7 | dataset_item=1
373
+ 2026-11-01 23:54:37 | INFO | DATASET FILE: 00000001.pt
374
+ 2026-11-01 23:54:37 | INFO | LOSS: 0.71637017
375
+ 2026-11-01 23:54:37 | INFO | LR: 4.38405797e-05
376
+ 2026-11-01 23:54:37 | INFO | TIMESTEP: 0.98469913
377
+ 2026-11-01 23:54:37 | INFO | GLOBAL GRADIENT NORM: 0.23142415
378
+ 2026-11-01 23:54:37 | INFO | MAX ABSOLUTE GRADIENT: 2.11181641e-02
379
+ 2026-11-01 23:54:37 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
380
+ 2026-11-01 23:54:37 | INFO | MISSING GRADIENT TENSORS: 0
381
+ 2026-11-01 23:54:37 | INFO | GPU0 FORWARD: 2.443s
382
+ 2026-11-01 23:54:37 | INFO | GPU1 FORWARD: 2.355s
383
+ 2026-11-01 23:54:37 | INFO | GPU1 BACKWARD: 6.258s
384
+ 2026-11-01 23:54:37 | INFO | GPU0 BACKWARD: 6.098s
385
+ 2026-11-01 23:54:37 | INFO | UPDATE WALL TIME: 17.273s
386
+ 2026-11-01 23:54:37 | INFO | ELAPSED TRAINING TIME: 320.75s
387
+ 2026-11-01 23:54:37 | INFO | STEP 18 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.756 GB | free=1.325 GB
388
+ 2026-11-01 23:54:37 | INFO | STEP 18 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.664 GB | free=1.393 GB
389
+ 2026-11-01 23:54:55 | INFO | --------------------------------------------------------------------------------
390
+ 2026-11-01 23:54:55 | INFO | STEP 19/70 | epoch=3/10 | position=5/7 | dataset_item=5
391
+ 2026-11-01 23:54:55 | INFO | DATASET FILE: 00000005.pt
392
+ 2026-11-01 23:54:55 | INFO | LOSS: 0.67044747
393
+ 2026-11-01 23:54:55 | INFO | LR: 4.34782609e-05
394
+ 2026-11-01 23:54:55 | INFO | TIMESTEP: 0.60690606
395
+ 2026-11-01 23:54:55 | INFO | GLOBAL GRADIENT NORM: 0.20816813
396
+ 2026-11-01 23:54:55 | INFO | MAX ABSOLUTE GRADIENT: 8.88061523e-03
397
+ 2026-11-01 23:54:55 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
398
+ 2026-11-01 23:54:55 | INFO | MISSING GRADIENT TENSORS: 0
399
+ 2026-11-01 23:54:55 | INFO | GPU0 FORWARD: 2.508s
400
+ 2026-11-01 23:54:55 | INFO | GPU1 FORWARD: 2.416s
401
+ 2026-11-01 23:54:55 | INFO | GPU1 BACKWARD: 6.381s
402
+ 2026-11-01 23:54:55 | INFO | GPU0 BACKWARD: 6.349s
403
+ 2026-11-01 23:54:55 | INFO | UPDATE WALL TIME: 17.760s
404
+ 2026-11-01 23:54:55 | INFO | ELAPSED TRAINING TIME: 338.52s
405
+ 2026-11-01 23:54:55 | INFO | STEP 19 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.935 GB | free=1.325 GB
406
+ 2026-11-01 23:54:55 | INFO | STEP 19 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.842 GB | free=1.393 GB
407
+ 2026-11-01 23:55:13 | INFO | --------------------------------------------------------------------------------
408
+ 2026-11-01 23:55:13 | INFO | STEP 20/70 | epoch=3/10 | position=6/7 | dataset_item=6
409
+ 2026-11-01 23:55:13 | INFO | DATASET FILE: 00000006.pt
410
+ 2026-11-01 23:55:13 | INFO | LOSS: 0.74476588
411
+ 2026-11-01 23:55:13 | INFO | LR: 4.31159420e-05
412
+ 2026-11-01 23:55:13 | INFO | TIMESTEP: 0.59872955
413
+ 2026-11-01 23:55:13 | INFO | GLOBAL GRADIENT NORM: 0.23598037
414
+ 2026-11-01 23:55:13 | INFO | MAX ABSOLUTE GRADIENT: 1.42745972e-02
415
+ 2026-11-01 23:55:13 | INFO | GRADIENT CLIP COEFFICIENT: 1.00000000
416
+ 2026-11-01 23:55:13 | INFO | MISSING GRADIENT TENSORS: 0
417
+ 2026-11-01 23:55:13 | INFO | GPU0 FORWARD: 2.485s
418
+ 2026-11-01 23:55:13 | INFO | GPU1 FORWARD: 2.402s
419
+ 2026-11-01 23:55:13 | INFO | GPU1 BACKWARD: 6.303s
420
+ 2026-11-01 23:55:13 | INFO | GPU0 BACKWARD: 6.332s
421
+ 2026-11-01 23:55:13 | INFO | UPDATE WALL TIME: 17.632s
422
+ 2026-11-01 23:55:13 | INFO | ELAPSED TRAINING TIME: 356.17s
423
+ 2026-11-01 23:55:13 | INFO | STEP 20 | GPU0 | allocated=5.070 GB | reserved=13.100 GB | peak=12.882 GB | free=1.325 GB
424
+ 2026-11-01 23:55:13 | INFO | STEP 20 | GPU1 | allocated=4.984 GB | reserved=13.031 GB | peak=12.788 GB | free=1.393 GB