Upload train log after epoch 5
Browse files
checkpoints/h200_phase3_mtp_self_distill/train.log
CHANGED
|
@@ -208,3 +208,60 @@ Frozen params: 1,294,878,720
|
|
| 208 |
2026-06-22 22:26:46,352 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/config.yaml -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/config.yaml
|
| 209 |
2026-06-22 22:26:46,430 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 210 |
2026-06-22 22:26:46,487 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/train.log -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/train.log
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 208 |
2026-06-22 22:26:46,352 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/config.yaml -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/config.yaml
|
| 209 |
2026-06-22 22:26:46,430 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 210 |
2026-06-22 22:26:46,487 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/train.log -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/train.log
|
| 211 |
+
2026-06-22 22:26:46,919 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/checkpoint_manifest.json -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/checkpoint_manifest.json
|
| 212 |
+
2026-06-22 22:26:47,001 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 213 |
+
2026-06-22 22:26:47,078 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/best.pt -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/best.pt
|
| 214 |
+
2026-06-22 22:26:49,059 [INFO] Deleted old local epoch checkpoint: runs/h200_llm2seq_phase3_mtp_self_distill/epoch_003.pt
|
| 215 |
+
2026-06-22 22:26:49,335 [INFO] Saved trainable-only checkpoint to runs/h200_llm2seq_phase3_mtp_self_distill/checkpoint_657.pt (52 model tensors, stores_base_encoder_weights=False)
|
| 216 |
+
2026-06-22 22:27:13,585 [INFO] Step 660/876 | Loss: 1.4068 | CE: 0.6744 | MTP: 2.2824 | MTP_CE: 2.2824 | MTP_KL: 1.4442 | LR: 1.51e-05 | Epoch: 4/4
|
| 217 |
+
2026-06-22 22:27:54,013 [INFO] Step 665/876 | Loss: 1.4350 | CE: 0.7074 | MTP: 2.3320 | MTP_CE: 2.3320 | MTP_KL: 1.4708 | LR: 1.44e-05 | Epoch: 4/4
|
| 218 |
+
2026-06-22 22:28:34,006 [INFO] Step 670/876 | Loss: 1.4259 | CE: 0.6819 | MTP: 2.3082 | MTP_CE: 2.3082 | MTP_KL: 1.4669 | LR: 1.38e-05 | Epoch: 4/4
|
| 219 |
+
2026-06-22 22:29:14,548 [INFO] Step 675/876 | Loss: 1.3974 | CE: 0.6575 | MTP: 2.2527 | MTP_CE: 2.2527 | MTP_KL: 1.4432 | LR: 1.32e-05 | Epoch: 4/4
|
| 220 |
+
2026-06-22 22:29:55,057 [INFO] Step 680/876 | Loss: 1.4142 | CE: 0.6929 | MTP: 2.2930 | MTP_CE: 2.2930 | MTP_KL: 1.4527 | LR: 1.26e-05 | Epoch: 4/4
|
| 221 |
+
2026-06-22 22:30:35,529 [INFO] Step 685/876 | Loss: 1.4157 | CE: 0.6761 | MTP: 2.2873 | MTP_CE: 2.2873 | MTP_KL: 1.4591 | LR: 1.19e-05 | Epoch: 4/4
|
| 222 |
+
2026-06-22 22:31:16,080 [INFO] Step 690/876 | Loss: 1.4387 | CE: 0.6873 | MTP: 2.3294 | MTP_CE: 2.3294 | MTP_KL: 1.4798 | LR: 1.14e-05 | Epoch: 4/4
|
| 223 |
+
2026-06-22 22:31:56,272 [INFO] Step 695/876 | Loss: 1.4408 | CE: 0.6921 | MTP: 2.3379 | MTP_CE: 2.3379 | MTP_KL: 1.4788 | LR: 1.08e-05 | Epoch: 4/4
|
| 224 |
+
2026-06-22 22:32:36,698 [INFO] Step 700/876 | Loss: 1.4132 | CE: 0.6966 | MTP: 2.2958 | MTP_CE: 2.2958 | MTP_KL: 1.4490 | LR: 1.02e-05 | Epoch: 4/4
|
| 225 |
+
2026-06-22 22:33:17,503 [INFO] Step 705/876 | Loss: 1.3693 | CE: 0.6649 | MTP: 2.2245 | MTP_CE: 2.2245 | MTP_KL: 1.4038 | LR: 1.00e-05 | Epoch: 4/4
|
| 226 |
+
2026-06-22 22:33:57,122 [INFO] Step 710/876 | Loss: 1.4115 | CE: 0.7039 | MTP: 2.3025 | MTP_CE: 2.3025 | MTP_KL: 1.4415 | LR: 1.00e-05 | Epoch: 4/4
|
| 227 |
+
2026-06-22 22:34:37,491 [INFO] Step 715/876 | Loss: 1.4079 | CE: 0.6886 | MTP: 2.2973 | MTP_CE: 2.2973 | MTP_KL: 1.4373 | LR: 1.00e-05 | Epoch: 4/4
|
| 228 |
+
2026-06-22 22:35:18,312 [INFO] Step 720/876 | Loss: 1.3934 | CE: 0.6775 | MTP: 2.2638 | MTP_CE: 2.2638 | MTP_KL: 1.4284 | LR: 1.00e-05 | Epoch: 4/4
|
| 229 |
+
2026-06-22 22:35:58,576 [INFO] Step 725/876 | Loss: 1.4131 | CE: 0.6973 | MTP: 2.3034 | MTP_CE: 2.3034 | MTP_KL: 1.4442 | LR: 1.00e-05 | Epoch: 4/4
|
| 230 |
+
2026-06-22 22:36:39,078 [INFO] Step 730/876 | Loss: 1.4145 | CE: 0.7024 | MTP: 2.3071 | MTP_CE: 2.3071 | MTP_KL: 1.4447 | LR: 1.00e-05 | Epoch: 4/4
|
| 231 |
+
2026-06-22 22:37:19,231 [INFO] Step 735/876 | Loss: 1.4072 | CE: 0.6802 | MTP: 2.2774 | MTP_CE: 2.2774 | MTP_KL: 1.4480 | LR: 1.00e-05 | Epoch: 4/4
|
| 232 |
+
2026-06-22 22:37:59,325 [INFO] Step 740/876 | Loss: 1.4077 | CE: 0.6786 | MTP: 2.2842 | MTP_CE: 2.2842 | MTP_KL: 1.4450 | LR: 1.00e-05 | Epoch: 4/4
|
| 233 |
+
2026-06-22 22:38:39,769 [INFO] Step 745/876 | Loss: 1.3765 | CE: 0.6614 | MTP: 2.2267 | MTP_CE: 2.2267 | MTP_KL: 1.4170 | LR: 1.00e-05 | Epoch: 4/4
|
| 234 |
+
2026-06-22 22:39:20,288 [INFO] Step 750/876 | Loss: 1.4103 | CE: 0.7016 | MTP: 2.2943 | MTP_CE: 2.2943 | MTP_KL: 1.4440 | LR: 1.00e-05 | Epoch: 4/4
|
| 235 |
+
2026-06-22 22:40:00,632 [INFO] Step 755/876 | Loss: 1.4123 | CE: 0.6880 | MTP: 2.2937 | MTP_CE: 2.2937 | MTP_KL: 1.4484 | LR: 1.00e-05 | Epoch: 4/4
|
| 236 |
+
2026-06-22 22:40:40,689 [INFO] Step 760/876 | Loss: 1.4153 | CE: 0.6711 | MTP: 2.2876 | MTP_CE: 2.2876 | MTP_KL: 1.4580 | LR: 1.00e-05 | Epoch: 4/4
|
| 237 |
+
2026-06-22 22:41:21,289 [INFO] Step 765/876 | Loss: 1.4127 | CE: 0.6861 | MTP: 2.2900 | MTP_CE: 2.2900 | MTP_KL: 1.4513 | LR: 1.00e-05 | Epoch: 4/4
|
| 238 |
+
2026-06-22 22:42:00,767 [INFO] Step 770/876 | Loss: 1.4227 | CE: 0.6853 | MTP: 2.3106 | MTP_CE: 2.3106 | MTP_KL: 1.4590 | LR: 1.00e-05 | Epoch: 4/4
|
| 239 |
+
2026-06-22 22:42:40,897 [INFO] Step 775/876 | Loss: 1.4078 | CE: 0.6967 | MTP: 2.2920 | MTP_CE: 2.2920 | MTP_KL: 1.4405 | LR: 1.00e-05 | Epoch: 4/4
|
| 240 |
+
2026-06-22 22:43:20,839 [INFO] Step 780/876 | Loss: 1.4341 | CE: 0.6926 | MTP: 2.3309 | MTP_CE: 2.3309 | MTP_KL: 1.4696 | LR: 1.00e-05 | Epoch: 4/4
|
| 241 |
+
2026-06-22 22:44:01,363 [INFO] Step 785/876 | Loss: 1.4203 | CE: 0.6994 | MTP: 2.3161 | MTP_CE: 2.3161 | MTP_KL: 1.4509 | LR: 1.00e-05 | Epoch: 4/4
|
| 242 |
+
2026-06-22 22:44:41,467 [INFO] Step 790/876 | Loss: 1.4053 | CE: 0.6935 | MTP: 2.2885 | MTP_CE: 2.2885 | MTP_KL: 1.4375 | LR: 1.00e-05 | Epoch: 4/4
|
| 243 |
+
2026-06-22 22:45:21,751 [INFO] Step 795/876 | Loss: 1.3889 | CE: 0.6799 | MTP: 2.2668 | MTP_CE: 2.2668 | MTP_KL: 1.4177 | LR: 1.00e-05 | Epoch: 4/4
|
| 244 |
+
2026-06-22 22:46:01,252 [INFO] Step 800/876 | Loss: 1.3756 | CE: 0.6753 | MTP: 2.2388 | MTP_CE: 2.2388 | MTP_KL: 1.4079 | LR: 1.00e-05 | Epoch: 4/4
|
| 245 |
+
2026-06-22 22:46:41,675 [INFO] Step 805/876 | Loss: 1.4006 | CE: 0.6852 | MTP: 2.2785 | MTP_CE: 2.2785 | MTP_KL: 1.4340 | LR: 1.00e-05 | Epoch: 4/4
|
| 246 |
+
2026-06-22 22:47:22,106 [INFO] Step 810/876 | Loss: 1.3835 | CE: 0.6750 | MTP: 2.2517 | MTP_CE: 2.2517 | MTP_KL: 1.4160 | LR: 1.00e-05 | Epoch: 4/4
|
| 247 |
+
2026-06-22 22:48:01,992 [INFO] Step 815/876 | Loss: 1.4435 | CE: 0.6940 | MTP: 2.3453 | MTP_CE: 2.3453 | MTP_KL: 1.4798 | LR: 1.00e-05 | Epoch: 4/4
|
| 248 |
+
2026-06-22 22:48:42,094 [INFO] Step 820/876 | Loss: 1.3960 | CE: 0.6682 | MTP: 2.2594 | MTP_CE: 2.2594 | MTP_KL: 1.4363 | LR: 1.00e-05 | Epoch: 4/4
|
| 249 |
+
2026-06-22 22:49:22,381 [INFO] Step 825/876 | Loss: 1.4081 | CE: 0.6911 | MTP: 2.2846 | MTP_CE: 2.2846 | MTP_KL: 1.4455 | LR: 1.00e-05 | Epoch: 4/4
|
| 250 |
+
2026-06-22 22:50:02,548 [INFO] Step 830/876 | Loss: 1.3850 | CE: 0.6792 | MTP: 2.2482 | MTP_CE: 2.2482 | MTP_KL: 1.4211 | LR: 1.00e-05 | Epoch: 4/4
|
| 251 |
+
2026-06-22 22:50:43,209 [INFO] Step 835/876 | Loss: 1.3953 | CE: 0.6830 | MTP: 2.2707 | MTP_CE: 2.2707 | MTP_KL: 1.4282 | LR: 1.00e-05 | Epoch: 4/4
|
| 252 |
+
2026-06-22 22:51:23,917 [INFO] Step 840/876 | Loss: 1.3927 | CE: 0.6862 | MTP: 2.2645 | MTP_CE: 2.2645 | MTP_KL: 1.4267 | LR: 1.00e-05 | Epoch: 4/4
|
| 253 |
+
2026-06-22 22:52:04,226 [INFO] Step 845/876 | Loss: 1.4028 | CE: 0.6859 | MTP: 2.2836 | MTP_CE: 2.2836 | MTP_KL: 1.4354 | LR: 1.00e-05 | Epoch: 4/4
|
| 254 |
+
2026-06-22 22:52:44,463 [INFO] Step 850/876 | Loss: 1.3748 | CE: 0.6792 | MTP: 2.2397 | MTP_CE: 2.2397 | MTP_KL: 1.4057 | LR: 1.00e-05 | Epoch: 4/4
|
| 255 |
+
2026-06-22 22:53:24,902 [INFO] Step 855/876 | Loss: 1.3817 | CE: 0.6942 | MTP: 2.2584 | MTP_CE: 2.2584 | MTP_KL: 1.4084 | LR: 1.00e-05 | Epoch: 4/4
|
| 256 |
+
2026-06-22 22:54:05,557 [INFO] Step 860/876 | Loss: 1.3848 | CE: 0.6865 | MTP: 2.2581 | MTP_CE: 2.2581 | MTP_KL: 1.4147 | LR: 1.00e-05 | Epoch: 4/4
|
| 257 |
+
2026-06-22 22:54:45,622 [INFO] Step 865/876 | Loss: 1.3827 | CE: 0.6922 | MTP: 2.2569 | MTP_CE: 2.2569 | MTP_KL: 1.4111 | LR: 1.00e-05 | Epoch: 4/4
|
| 258 |
+
2026-06-22 22:55:25,630 [INFO] Step 870/876 | Loss: 1.3977 | CE: 0.6692 | MTP: 2.2665 | MTP_CE: 2.2665 | MTP_KL: 1.4354 | LR: 1.00e-05 | Epoch: 4/4
|
| 259 |
+
2026-06-22 22:56:05,746 [INFO] Step 875/876 | Loss: 1.3864 | CE: 0.6748 | MTP: 2.2475 | MTP_CE: 2.2475 | MTP_KL: 1.4243 | LR: 1.00e-05 | Epoch: 5/4
|
| 260 |
+
2026-06-22 22:59:35,633 [INFO] Step 876 | Eval Loss: 1.3235
|
| 261 |
+
2026-06-22 22:59:35,933 [INFO] Saved trainable-only checkpoint to runs/h200_llm2seq_phase3_mtp_self_distill/best.pt (52 model tensors, stores_base_encoder_weights=False)
|
| 262 |
+
2026-06-22 22:59:36,056 [INFO] New best eval loss: 1.3235
|
| 263 |
+
2026-06-22 22:59:36,148 [INFO] Saved trainable-only checkpoint to runs/h200_llm2seq_phase3_mtp_self_distill/epoch_005.pt (52 model tensors, stores_base_encoder_weights=False)
|
| 264 |
+
2026-06-22 22:59:36,286 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/epoch_005.pt -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/epochs/epoch_005_step_876.pt
|
| 265 |
+
2026-06-22 22:59:51,959 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/config.yaml -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/config.yaml
|
| 266 |
+
2026-06-22 22:59:52,039 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
|
| 267 |
+
2026-06-22 22:59:52,105 [INFO] Uploading to HF: runs/h200_llm2seq_phase3_mtp_self_distill/train.log -> BienKieu/llm2seq-wikilingua/checkpoints/h200_phase3_mtp_self_distill/train.log
|