Hugging Face's logo

codeShare
/
kromoss-lora-exports

Safetensors
Model card Files
xet
 Community
codeShare commited on
Commit
dd92fdc
·
verified ·
1 Parent(s): eb3242b

Upload Kromoss dual-T4 training log

Browse files
Files changed (1) hide show
  1. training_logs/kromoss_training_log.txt +121 -68
training_logs/kromoss_training_log.txt CHANGED
@@ -1,73 +1,126 @@
1
- 2026-11-01 23:41:22 | INFO | ================================================================================
2
- 2026-11-01 23:41:22 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
- 2026-11-01 23:41:22 | INFO | ================================================================================
4
- 2026-11-01 23:41:22 | INFO | PyTorch version: 2.11.0+cu128
5
- 2026-11-01 23:41:22 | INFO | CUDA version: 12.8
6
- 2026-11-01 23:41:22 | INFO | Dataset size: 7
7
- 2026-11-01 23:41:22 | INFO | Presentations per image: 10
8
- 2026-11-01 23:41:22 | INFO | Total optimizer updates: 70
9
- 2026-11-01 23:41:22 | INFO | Checkpoint steps: [14, 35, 70]
10
- 2026-11-01 23:41:22 | INFO | Initial global learning rate: 5.00000000e-05
11
- 2026-11-01 23:41:22 | INFO | Final global learning rate: 2.50000000e-05
12
- 2026-11-01 23:41:22 | INFO | Weight decay: 0.01
13
- 2026-11-01 23:41:22 | INFO | Gradient clipping: 0.5
14
- 2026-11-01 23:41:22 | INFO | Loss scale: 128.0
15
- 2026-11-01 23:41:22 | INFO | LoRA rank: 16
16
- 2026-11-01 23:41:22 | INFO | LoRA alpha: 16
17
- 2026-11-01 23:41:22 | INFO | Trainable parameters: 37,158,912
18
- 2026-11-01 23:41:22 | INFO | GPU0 LoRA parameters: 17,694,720
19
- 2026-11-01 23:41:22 | INFO | GPU1 LoRA parameters: 19,464,192
20
- 2026-11-01 23:41:22 | INFO | Continuation loaded: True
21
- 2026-11-01 23:41:22 | INFO | Continuation loading enabled: True
22
- 2026-11-01 23:41:22 | INFO | HF repository: codeShare/kromoss-lora-exports
23
- 2026-11-01 23:41:22 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
- 2026-11-01 23:41:22 | INFO | GPU0: Tesla T4 | 14.562 GB
25
- 2026-11-01 23:41:22 | INFO | GPU1: Tesla T4 | 14.562 GB
26
- 2026-11-01 23:41:22 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
- 2026-11-01 23:41:22 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
- 2026-11-01 23:41:22 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
- 2026-11-01 23:41:22 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
- 2026-11-01 23:41:22 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
- 2026-11-01 23:41:22 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
- 2026-11-01 23:41:22 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
- 2026-11-01 23:41:22 | INFO | --------------------------------------------------------------------------------
34
- 2026-11-01 23:41:22 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
- 2026-11-01 23:41:22 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
- 2026-11-01 23:41:22 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
- 2026-11-01 23:41:22 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
- 2026-11-01 23:41:22 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
- 2026-11-01 23:41:22 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
- 2026-11-01 23:41:22 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
- 2026-11-01 23:41:22 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
- 2026-11-01 23:41:22 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
- 2026-11-01 23:41:22 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
- 2026-11-01 23:41:22 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
- 2026-11-01 23:41:22 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
- 2026-11-01 23:41:22 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
- 2026-11-01 23:41:22 | INFO | ================================================================================
48
- 2026-11-01 23:41:22 | INFO | STARTING TRAINING
49
- 2026-11-01 23:41:22 | INFO | ================================================================================
50
- 2026-11-01 23:41:22 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
- 2026-11-01 23:41:22 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
- 2026-11-01 23:41:22 | INFO | AdamW optimizer state is retained across all 70 updates.
53
- 2026-11-01 23:41:22 | INFO | Learning rate does NOT reset when the dataset image changes.
54
- 2026-11-01 23:41:22 | INFO | Gradient clipping threshold: 0.5
55
- 2026-11-01 23:41:22 | INFO | Noise is generated in FP32 and cast to FP16.
56
- 2026-11-01 23:41:22 | INFO | Timestep is logged for every optimizer update.
57
- 2026-11-01 23:41:22 | INFO | Continuation loading enabled: True
58
- 2026-11-01 23:41:22 | INFO | Continuation actually loaded: True
59
- 2026-11-01 23:41:22 | ERROR | ================================================================================
60
- 2026-11-01 23:41:22 | ERROR | TRAINING FAILED
61
- 2026-11-01 23:41:22 | ERROR | Traceback (most recent call last):
62
- File "/tmp/ipykernel_841/4050970492.py", line 1153, in <cell line: 0>
 
63
  result = train_one_image(
64
  dataset_index=dataset_index,
65
  step=step
66
  )
67
- File "/tmp/ipykernel_841/4050970492.py", line 809, in train_one_image
68
- for group in optimizer.param_groups:
69
- ^^^^^^^^^
70
- NameError: name 'optimizer' is not defined. Did you mean: 'optimizer0'?
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
71
 
72
- 2026-11-01 23:41:22 | INFO | ERROR STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.856 GB | free=9.587 GB
73
- 2026-11-01 23:41:22 | INFO | ERROR STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.749 GB | free=9.696 GB
 
1
+ 2026-11-01 23:47:22 | INFO | ================================================================================
2
+ 2026-11-01 23:47:22 | INFO | KROMOSS DUAL-T4 TWO-STAGE TRAINING
3
+ 2026-11-01 23:47:22 | INFO | ================================================================================
4
+ 2026-11-01 23:47:22 | INFO | PyTorch version: 2.11.0+cu128
5
+ 2026-11-01 23:47:22 | INFO | CUDA version: 12.8
6
+ 2026-11-01 23:47:22 | INFO | Dataset size: 7
7
+ 2026-11-01 23:47:22 | INFO | Presentations per image: 10
8
+ 2026-11-01 23:47:22 | INFO | Total optimizer updates: 70
9
+ 2026-11-01 23:47:22 | INFO | Checkpoint steps: [14, 35, 70]
10
+ 2026-11-01 23:47:22 | INFO | Initial global learning rate: 5.00000000e-05
11
+ 2026-11-01 23:47:22 | INFO | Final global learning rate: 2.50000000e-05
12
+ 2026-11-01 23:47:22 | INFO | Weight decay: 0.01
13
+ 2026-11-01 23:47:22 | INFO | Gradient clipping: 0.5
14
+ 2026-11-01 23:47:22 | INFO | Loss scale: 128.0
15
+ 2026-11-01 23:47:22 | INFO | LoRA rank: 16
16
+ 2026-11-01 23:47:22 | INFO | LoRA alpha: 16
17
+ 2026-11-01 23:47:22 | INFO | Trainable parameters: 37,158,912
18
+ 2026-11-01 23:47:22 | INFO | GPU0 LoRA parameters: 17,694,720
19
+ 2026-11-01 23:47:22 | INFO | GPU1 LoRA parameters: 19,464,192
20
+ 2026-11-01 23:47:22 | INFO | Continuation loaded: True
21
+ 2026-11-01 23:47:22 | INFO | Continuation loading enabled: True
22
+ 2026-11-01 23:47:22 | INFO | HF repository: codeShare/kromoss-lora-exports
23
+ 2026-11-01 23:47:22 | INFO | HF log destination: training_logs/kromoss_training_log.txt
24
+ 2026-11-01 23:47:22 | INFO | GPU0: Tesla T4 | 14.562 GB
25
+ 2026-11-01 23:47:22 | INFO | GPU1: Tesla T4 | 14.562 GB
26
+ 2026-11-01 23:47:22 | INFO | DATASET ITEM 0: 00000000.pt | presentations=10
27
+ 2026-11-01 23:47:22 | INFO | DATASET ITEM 1: 00000001.pt | presentations=10
28
+ 2026-11-01 23:47:22 | INFO | DATASET ITEM 2: 00000002.pt | presentations=10
29
+ 2026-11-01 23:47:22 | INFO | DATASET ITEM 3: 00000003.pt | presentations=10
30
+ 2026-11-01 23:47:22 | INFO | DATASET ITEM 4: 00000004.pt | presentations=10
31
+ 2026-11-01 23:47:22 | INFO | DATASET ITEM 5: 00000005.pt | presentations=10
32
+ 2026-11-01 23:47:22 | INFO | DATASET ITEM 6: 00000006.pt | presentations=10
33
+ 2026-11-01 23:47:22 | INFO | --------------------------------------------------------------------------------
34
+ 2026-11-01 23:47:22 | INFO | DETERMINISTIC SHUFFLED SAMPLE PLAN
35
+ 2026-11-01 23:47:22 | INFO | EPOCH 01: 1 3 4 2 6 0 5
36
+ 2026-11-01 23:47:22 | INFO | EPOCH 02: 3 6 2 4 0 5 1
37
+ 2026-11-01 23:47:22 | INFO | EPOCH 03: 3 2 4 1 5 6 0
38
+ 2026-11-01 23:47:22 | INFO | EPOCH 04: 2 5 0 3 6 1 4
39
+ 2026-11-01 23:47:22 | INFO | EPOCH 05: 6 3 5 1 0 2 4
40
+ 2026-11-01 23:47:22 | INFO | EPOCH 06: 5 0 3 4 1 6 2
41
+ 2026-11-01 23:47:22 | INFO | EPOCH 07: 4 5 1 2 6 3 0
42
+ 2026-11-01 23:47:22 | INFO | EPOCH 08: 2 4 1 5 3 0 6
43
+ 2026-11-01 23:47:22 | INFO | EPOCH 09: 1 0 3 5 6 2 4
44
+ 2026-11-01 23:47:22 | INFO | EPOCH 10: 2 4 3 1 6 0 5
45
+ 2026-11-01 23:47:22 | INFO | INITIAL MODEL STATE | GPU0 | allocated=4.856 GB | reserved=4.873 GB | peak=4.857 GB | free=9.587 GB
46
+ 2026-11-01 23:47:22 | INFO | INITIAL MODEL STATE | GPU1 | allocated=4.749 GB | reserved=4.764 GB | peak=4.750 GB | free=9.696 GB
47
+ 2026-11-01 23:47:22 | INFO | ================================================================================
48
+ 2026-11-01 23:47:22 | INFO | STARTING TRAINING
49
+ 2026-11-01 23:47:22 | INFO | ================================================================================
50
+ 2026-11-01 23:47:22 | INFO | Schedule: ten shuffled seven-image epochs, one optimizer update per image presentation.
51
+ 2026-11-01 23:47:22 | INFO | Global LR schedule: 5.00000000e-05 -> 2.50000000e-05 across all 70 optimizer steps.
52
+ 2026-11-01 23:47:22 | INFO | Two AdamW optimizers are used: optimizer0 for GPU0 LoRA parameters and optimizer1 for GPU1 LoRA parameters.
53
+ 2026-11-01 23:47:22 | INFO | Both AdamW optimizer states persist across all 70 updates.
54
+ 2026-11-01 23:47:22 | INFO | Learning rate does NOT reset when the dataset image changes.
55
+ 2026-11-01 23:47:22 | INFO | Gradient clipping threshold: 0.5
56
+ 2026-11-01 23:47:22 | INFO | Noise is generated in FP32 and cast to FP16.
57
+ 2026-11-01 23:47:22 | INFO | Timestep is logged for every optimizer update.
58
+ 2026-11-01 23:47:22 | INFO | Continuation loading enabled: True
59
+ 2026-11-01 23:47:22 | INFO | Continuation actually loaded: True
60
+ 2026-11-01 23:47:28 | ERROR | ================================================================================
61
+ 2026-11-01 23:47:28 | ERROR | TRAINING FAILED
62
+ 2026-11-01 23:47:28 | ERROR | Traceback (most recent call last):
63
+ File "/tmp/ipykernel_934/333819112.py", line 1181, in <cell line: 0>
64
  result = train_one_image(
65
  dataset_index=dataset_index,
66
  step=step
67
  )
68
+ File "/tmp/ipykernel_934/333819112.py", line 957, in train_one_image
69
+ grad_x1 = stage1_backward(result)
70
+ File "/tmp/ipykernel_934/333819112.py", line 593, in stage1_backward
71
+ gradients = torch.autograd.grad(
72
+ scaled_loss,
73
+ [result["x1"]] + stage1_params,
74
+ allow_unused=True
75
+ )
76
+ File "/usr/local/lib/python3.13/dist-packages/torch/autograd/__init__.py", line 532, in grad
77
+ result = _engine_run_backward(
78
+ outputs,
79
+ ...<5 lines>...
80
+ accumulate_grad=False,
81
+ )
82
+ File "/usr/local/lib/python3.13/dist-packages/torch/autograd/graph.py", line 869, in _engine_run_backward
83
+ return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
84
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
85
+ t_outputs, *args, **kwargs
86
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^
87
+ ) # Calls into the C++ engine to run the backward pass
88
+ ^
89
+ File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1173, in unpack_hook
90
+ _run_fn_with_dynamo_disabled(frame.recompute_fn, *args)
91
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
92
+ File "/usr/local/lib/python3.13/dist-packages/torch/_compile.py", line 54, in inner
93
+ return disable_fn(*args, **kwargs)
94
+ File "/usr/local/lib/python3.13/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
95
+ return fn(*args, **kwargs)
96
+ File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1139, in _run_fn_with_dynamo_disabled
97
+ return fn(*args, **kwargs)
98
+ File "/usr/local/lib/python3.13/dist-packages/torch/utils/checkpoint.py", line 1607, in recompute_fn
99
+ fn(*args, **kwargs)
100
+ ~~^^^^^^^^^^^^^^^^^
101
+ File "/tmp/ipykernel_934/333819112.py", line 362, in run_group
102
+ x = base.layers[layer_index](
103
+ x,
104
+ rotary,
105
+ mod
106
+ )
107
+ File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
108
+ return self._call_impl(*args, **kwargs)
109
+ ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
110
+ File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
111
+ return forward_call(*args, **kwargs)
112
+ File "/kaggle/working/pipeline_moss_image_patched.py", line 147, in forward
113
+ hidden, gate = self.proj['ffn_up_gate'](y.to(dtype)).chunk(2, -1)
114
+ ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
115
+ File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
116
+ return self._call_impl(*args, **kwargs)
117
+ ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
118
+ File "/usr/local/lib/python3.13/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
119
+ return forward_call(*args, **kwargs)
120
+ File "/usr/local/lib/python3.13/dist-packages/peft/tuners/lora/layer.py", line 1102, in forward
121
+ result = result + lora_B(lora_A(dropout(x))) * scaling
122
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~
123
+ torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 146.00 MiB. GPU 1 has a total capacity of 14.56 GiB of which 116.81 MiB is free. Including non-PyTorch memory, this process has 14.45 GiB memory in use. Of the allocated memory 14.20 GiB is allocated by PyTorch, and 111.55 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
124
 
125
+ 2026-11-01 23:47:28 | INFO | ERROR STATE | GPU0 | allocated=4.927 GB | reserved=5.678 GB | peak=5.575 GB | free=8.755 GB
126
+ 2026-11-01 23:47:28 | INFO | ERROR STATE | GPU1 | allocated=14.000 GB | reserved=14.311 GB | peak=14.202 GB | free=0.114 GB