Upload train log after epoch 4
Browse files
checkpoints/h200_phase3_mtp_self_distill/train.log
CHANGED
|
@@ -151,3 +151,60 @@ Frozen params: 1,294,878,720
|
|
| 151 |
2026-06-22 21:53:36,746 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/config.yaml -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/config.yaml
|
| 152 |
2026-06-22 21:53:36,820 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 153 |
2026-06-22 21:53:36,879 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/train.log -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/train.log
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
2026-06-22 21:53:36,746 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/config.yaml -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/config.yaml
|
| 152 |
2026-06-22 21:53:36,820 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 153 |
2026-06-22 21:53:36,879 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/train.log -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/train.log
|
| 154 |
+
2026-06-22 21:53:37,410 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/checkpoint_manifest.json -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/checkpoint_manifest.json
|
| 155 |
+
2026-06-22 21:53:37,482 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 156 |
+
2026-06-22 21:53:37,570 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/best.pt -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/best.pt
|
| 157 |
+
2026-06-22 21:53:39,742 [INFO] Deleted old local epoch checkpoint: runs/h200_llm2seq_phase3_mtp_self_distill/epoch_002.pt
|
| 158 |
+
2026-06-22 21:53:40,029 [INFO] Saved trainable-only checkpoint to runs/h200_llm2seq_phase3_mtp_self_distill/checkpoint_438.pt (52 model tensors, stores_base_encoder_weights=False)
|
| 159 |
+
2026-06-22 21:53:56,355 [INFO] Step 440/876 | Loss: 1.5153 | CE: 0.6751 | MTP: 2.4065 | MTP_CE: 2.4065 | MTP_KL: 1.5867 | LR: 5.20e-05 | Epoch: 3/4
|
| 160 |
+
2026-06-22 21:54:36,542 [INFO] Step 445/876 | Loss: 1.5160 | CE: 0.6918 | MTP: 2.4198 | MTP_CE: 2.4198 | MTP_KL: 1.5801 | LR: 5.11e-05 | Epoch: 3/4
|
| 161 |
+
2026-06-22 21:55:16,759 [INFO] Step 450/876 | Loss: 1.5638 | CE: 0.6825 | MTP: 2.4840 | MTP_CE: 2.4840 | MTP_KL: 1.6373 | LR: 5.02e-05 | Epoch: 3/4
|
| 162 |
+
2026-06-22 21:55:56,849 [INFO] Step 455/876 | Loss: 1.5493 | CE: 0.6850 | MTP: 2.4607 | MTP_CE: 2.4607 | MTP_KL: 1.6222 | LR: 4.93e-05 | Epoch: 3/4
|
| 163 |
+
2026-06-22 21:56:37,223 [INFO] Step 460/876 | Loss: 1.5276 | CE: 0.6844 | MTP: 2.4356 | MTP_CE: 2.4356 | MTP_KL: 1.5938 | LR: 4.83e-05 | Epoch: 3/4
|
| 164 |
+
2026-06-22 21:57:17,718 [INFO] Step 465/876 | Loss: 1.5278 | CE: 0.6973 | MTP: 2.4475 | MTP_CE: 2.4475 | MTP_KL: 1.5870 | LR: 4.74e-05 | Epoch: 3/4
|
| 165 |
+
2026-06-22 21:57:57,985 [INFO] Step 470/876 | Loss: 1.5402 | CE: 0.6863 | MTP: 2.4563 | MTP_CE: 2.4563 | MTP_KL: 1.6066 | LR: 4.65e-05 | Epoch: 3/4
|
| 166 |
+
2026-06-22 21:58:38,431 [INFO] Step 475/876 | Loss: 1.4981 | CE: 0.6733 | MTP: 2.3888 | MTP_CE: 2.3888 | MTP_KL: 1.5629 | LR: 4.56e-05 | Epoch: 3/4
|
| 167 |
+
2026-06-22 21:59:19,062 [INFO] Step 480/876 | Loss: 1.4884 | CE: 0.6686 | MTP: 2.3759 | MTP_CE: 2.3759 | MTP_KL: 1.5512 | LR: 4.47e-05 | Epoch: 3/4
|
| 168 |
+
2026-06-22 21:59:59,725 [INFO] Step 485/876 | Loss: 1.4854 | CE: 0.6685 | MTP: 2.3688 | MTP_CE: 2.3688 | MTP_KL: 1.5495 | LR: 4.37e-05 | Epoch: 3/4
|
| 169 |
+
2026-06-22 22:00:39,934 [INFO] Step 490/876 | Loss: 1.4982 | CE: 0.6790 | MTP: 2.3858 | MTP_CE: 2.3858 | MTP_KL: 1.5650 | LR: 4.28e-05 | Epoch: 3/4
|
| 170 |
+
2026-06-22 22:01:20,665 [INFO] Step 495/876 | Loss: 1.4765 | CE: 0.6687 | MTP: 2.3643 | MTP_CE: 2.3643 | MTP_KL: 1.5345 | LR: 4.19e-05 | Epoch: 3/4
|
| 171 |
+
2026-06-22 22:02:01,147 [INFO] Step 500/876 | Loss: 1.4983 | CE: 0.6996 | MTP: 2.4048 | MTP_CE: 2.4048 | MTP_KL: 1.5538 | LR: 4.10e-05 | Epoch: 3/4
|
| 172 |
+
2026-06-22 22:02:41,707 [INFO] Step 505/876 | Loss: 1.4863 | CE: 0.6686 | MTP: 2.3728 | MTP_CE: 2.3728 | MTP_KL: 1.5489 | LR: 4.01e-05 | Epoch: 3/4
|
| 173 |
+
2026-06-22 22:03:22,062 [INFO] Step 510/876 | Loss: 1.4951 | CE: 0.6956 | MTP: 2.4000 | MTP_CE: 2.4000 | MTP_KL: 1.5501 | LR: 3.92e-05 | Epoch: 3/4
|
| 174 |
+
2026-06-22 22:04:02,392 [INFO] Step 515/876 | Loss: 1.4765 | CE: 0.6817 | MTP: 2.3638 | MTP_CE: 2.3638 | MTP_KL: 1.5348 | LR: 3.83e-05 | Epoch: 3/4
|
| 175 |
+
2026-06-22 22:04:42,756 [INFO] Step 520/876 | Loss: 1.4683 | CE: 0.6848 | MTP: 2.3618 | MTP_CE: 2.3618 | MTP_KL: 1.5196 | LR: 3.74e-05 | Epoch: 3/4
|
| 176 |
+
2026-06-22 22:05:22,829 [INFO] Step 525/876 | Loss: 1.5100 | CE: 0.6957 | MTP: 2.4224 | MTP_CE: 2.4224 | MTP_KL: 1.5666 | LR: 3.65e-05 | Epoch: 3/4
|
| 177 |
+
2026-06-22 22:06:02,882 [INFO] Step 530/876 | Loss: 1.4483 | CE: 0.6684 | MTP: 2.3246 | MTP_CE: 2.3246 | MTP_KL: 1.5020 | LR: 3.56e-05 | Epoch: 3/4
|
| 178 |
+
2026-06-22 22:06:43,450 [INFO] Step 535/876 | Loss: 1.4845 | CE: 0.6829 | MTP: 2.3844 | MTP_CE: 2.3844 | MTP_KL: 1.5383 | LR: 3.47e-05 | Epoch: 3/4
|
| 179 |
+
2026-06-22 22:07:24,286 [INFO] Step 540/876 | Loss: 1.4647 | CE: 0.6857 | MTP: 2.3600 | MTP_CE: 2.3600 | MTP_KL: 1.5134 | LR: 3.38e-05 | Epoch: 3/4
|
| 180 |
+
2026-06-22 22:08:04,982 [INFO] Step 545/876 | Loss: 1.4801 | CE: 0.6864 | MTP: 2.3887 | MTP_CE: 2.3887 | MTP_KL: 1.5271 | LR: 3.30e-05 | Epoch: 3/4
|
| 181 |
+
2026-06-22 22:08:45,149 [INFO] Step 550/876 | Loss: 1.4560 | CE: 0.6720 | MTP: 2.3385 | MTP_CE: 2.3385 | MTP_KL: 1.5089 | LR: 3.21e-05 | Epoch: 3/4
|
| 182 |
+
2026-06-22 22:09:24,610 [INFO] Step 555/876 | Loss: 1.4789 | CE: 0.7064 | MTP: 2.3924 | MTP_CE: 2.3924 | MTP_KL: 1.5223 | LR: 3.13e-05 | Epoch: 3/4
|
| 183 |
+
2026-06-22 22:10:05,242 [INFO] Step 560/876 | Loss: 1.4798 | CE: 0.6813 | MTP: 2.3786 | MTP_CE: 2.3786 | MTP_KL: 1.5325 | LR: 3.04e-05 | Epoch: 3/4
|
| 184 |
+
2026-06-22 22:10:45,750 [INFO] Step 565/876 | Loss: 1.4719 | CE: 0.6850 | MTP: 2.3680 | MTP_CE: 2.3680 | MTP_KL: 1.5230 | LR: 2.96e-05 | Epoch: 3/4
|
| 185 |
+
2026-06-22 22:11:26,565 [INFO] Step 570/876 | Loss: 1.4426 | CE: 0.6779 | MTP: 2.3244 | MTP_CE: 2.3244 | MTP_KL: 1.4905 | LR: 2.87e-05 | Epoch: 3/4
|
| 186 |
+
2026-06-22 22:12:07,263 [INFO] Step 575/876 | Loss: 1.4479 | CE: 0.6899 | MTP: 2.3332 | MTP_CE: 2.3332 | MTP_KL: 1.4959 | LR: 2.79e-05 | Epoch: 3/4
|
| 187 |
+
2026-06-22 22:12:47,247 [INFO] Step 580/876 | Loss: 1.4986 | CE: 0.6914 | MTP: 2.4137 | MTP_CE: 2.4137 | MTP_KL: 1.5490 | LR: 2.71e-05 | Epoch: 3/4
|
| 188 |
+
2026-06-22 22:13:27,864 [INFO] Step 585/876 | Loss: 1.4712 | CE: 0.6935 | MTP: 2.3751 | MTP_CE: 2.3751 | MTP_KL: 1.5173 | LR: 2.62e-05 | Epoch: 3/4
|
| 189 |
+
2026-06-22 22:14:07,671 [INFO] Step 590/876 | Loss: 1.4509 | CE: 0.6821 | MTP: 2.3381 | MTP_CE: 2.3381 | MTP_KL: 1.4989 | LR: 2.54e-05 | Epoch: 3/4
|
| 190 |
+
2026-06-22 22:14:48,120 [INFO] Step 595/876 | Loss: 1.4309 | CE: 0.6892 | MTP: 2.3173 | MTP_CE: 2.3173 | MTP_KL: 1.4715 | LR: 2.46e-05 | Epoch: 3/4
|
| 191 |
+
2026-06-22 22:15:28,596 [INFO] Step 600/876 | Loss: 1.4467 | CE: 0.6892 | MTP: 2.3413 | MTP_CE: 2.3413 | MTP_KL: 1.4885 | LR: 2.38e-05 | Epoch: 3/4
|
| 192 |
+
2026-06-22 22:16:08,741 [INFO] Step 605/876 | Loss: 1.4428 | CE: 0.6806 | MTP: 2.3271 | MTP_CE: 2.3271 | MTP_KL: 1.4894 | LR: 2.31e-05 | Epoch: 3/4
|
| 193 |
+
2026-06-22 22:16:49,212 [INFO] Step 610/876 | Loss: 1.4517 | CE: 0.6760 | MTP: 2.3352 | MTP_CE: 2.3352 | MTP_KL: 1.5023 | LR: 2.23e-05 | Epoch: 3/4
|
| 194 |
+
2026-06-22 22:17:29,303 [INFO] Step 615/876 | Loss: 1.4481 | CE: 0.6832 | MTP: 2.3376 | MTP_CE: 2.3376 | MTP_KL: 1.4937 | LR: 2.15e-05 | Epoch: 3/4
|
| 195 |
+
2026-06-22 22:18:09,923 [INFO] Step 620/876 | Loss: 1.4349 | CE: 0.6736 | MTP: 2.3149 | MTP_CE: 2.3149 | MTP_KL: 1.4808 | LR: 2.08e-05 | Epoch: 3/4
|
| 196 |
+
2026-06-22 22:18:50,788 [INFO] Step 625/876 | Loss: 1.4468 | CE: 0.6935 | MTP: 2.3493 | MTP_CE: 2.3493 | MTP_KL: 1.4839 | LR: 2.00e-05 | Epoch: 3/4
|
| 197 |
+
2026-06-22 22:19:30,741 [INFO] Step 630/876 | Loss: 1.4669 | CE: 0.7052 | MTP: 2.3712 | MTP_CE: 2.3712 | MTP_KL: 1.5111 | LR: 1.93e-05 | Epoch: 3/4
|
| 198 |
+
2026-06-22 22:20:11,199 [INFO] Step 635/876 | Loss: 1.4464 | CE: 0.6788 | MTP: 2.3322 | MTP_CE: 2.3322 | MTP_KL: 1.4935 | LR: 1.86e-05 | Epoch: 3/4
|
| 199 |
+
2026-06-22 22:20:51,546 [INFO] Step 640/876 | Loss: 1.4294 | CE: 0.6826 | MTP: 2.3106 | MTP_CE: 2.3106 | MTP_KL: 1.4724 | LR: 1.78e-05 | Epoch: 3/4
|
| 200 |
+
2026-06-22 22:21:31,759 [INFO] Step 645/876 | Loss: 1.4389 | CE: 0.6852 | MTP: 2.3249 | MTP_CE: 2.3249 | MTP_KL: 1.4827 | LR: 1.71e-05 | Epoch: 3/4
|
| 201 |
+
2026-06-22 22:22:12,129 [INFO] Step 650/876 | Loss: 1.4498 | CE: 0.7051 | MTP: 2.3539 | MTP_CE: 2.3539 | MTP_KL: 1.4873 | LR: 1.65e-05 | Epoch: 3/4
|
| 202 |
+
2026-06-22 22:22:52,484 [INFO] Step 655/876 | Loss: 1.4594 | CE: 0.6897 | MTP: 2.3564 | MTP_CE: 2.3564 | MTP_KL: 1.5050 | LR: 1.58e-05 | Epoch: 3/4
|
| 203 |
+
2026-06-22 22:26:30,815 [INFO] Step 657 | Eval Loss: 1.3571
|
| 204 |
+
2026-06-22 22:26:31,129 [INFO] Saved trainable-only checkpoint to runs/h200_llm2seq_phase3_mtp_self_distill/best.pt (52 model tensors, stores_base_encoder_weights=False)
|
| 205 |
+
2026-06-22 22:26:31,244 [INFO] New best eval loss: 1.3571
|
| 206 |
+
2026-06-22 22:26:31,337 [INFO] Saved trainable-only checkpoint to runs/h200_llm2seq_phase3_mtp_self_distill/epoch_004.pt (52 model tensors, stores_base_encoder_weights=False)
|
| 207 |
+
2026-06-22 22:26:31,470 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/epoch_004.pt -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/epochs/epoch_004_step_657.pt
|
| 208 |
+
2026-06-22 22:26:46,352 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/config.yaml -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/config.yaml
|
| 209 |
+
2026-06-22 22:26:46,430 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 210 |
+
2026-06-22 22:26:46,487 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/train.log -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/train.log
|