[2026-10-23 19:32:34][INFO] Starting Qwen2.5-3B-Instruct fine-tuning on 4 GPUs [2026-10-23 19:32:34][INFO] Base model: Qwen/Qwen2.5-3B-Instruct [2026-10-23 19:32:34][INFO] Hyperparameters: lr=0.00015, micro_batch=4, accum=2, seq_len=2048 [2026-10-23 19:32:53][INFO] Trainable params: 119,734,272 (3.74% of total 3,205,672,960) [2026-10-23 19:32:53][INFO] Total dataset samples: 33029, Total training steps: 1032 [2026-10-23 19:33:17][INFO] Step 10/1032 | Loss 28.5524 | LR 3.00e-05 | tok/s 23,620 | ETA 40.7m [2026-10-23 19:33:40][INFO] Step 20/1032 | Loss 27.9757 | LR 6.00e-05 | tok/s 24,804 | ETA 39.5m [2026-10-23 19:34:03][INFO] Step 30/1032 | Loss 27.8679 | LR 9.00e-05 | tok/s 23,923 | ETA 39.1m [2026-10-23 19:34:27][INFO] Step 40/1032 | Loss 27.6360 | LR 1.20e-04 | tok/s 24,376 | ETA 38.6m [2026-10-23 19:34:50][INFO] Step 50/1032 | Loss 28.0021 | LR 1.50e-04 | tok/s 23,701 | ETA 38.2m [2026-10-23 19:35:13][INFO] Step 60/1032 | Loss 27.9318 | LR 1.50e-04 | tok/s 23,982 | ETA 37.8m [2026-10-23 19:35:36][INFO] Step 70/1032 | Loss 27.7594 | LR 1.50e-04 | tok/s 24,030 | ETA 37.4m [2026-10-23 19:36:00][INFO] Step 80/1032 | Loss 28.0117 | LR 1.50e-04 | tok/s 23,720 | ETA 37.0m [2026-10-23 19:36:23][INFO] Step 90/1032 | Loss 27.7231 | LR 1.49e-04 | tok/s 23,723 | ETA 36.5m [2026-10-23 19:36:46][INFO] Step 100/1032 | Loss 27.5048 | LR 1.49e-04 | tok/s 24,182 | ETA 36.2m [2026-10-23 19:37:09][INFO] Step 110/1032 | Loss 27.7824 | LR 1.49e-04 | tok/s 24,599 | ETA 35.8m [2026-10-23 19:37:33][INFO] Step 120/1032 | Loss 27.7583 | LR 1.48e-04 | tok/s 24,337 | ETA 35.4m [2026-10-23 19:37:56][INFO] Step 130/1032 | Loss 27.2896 | LR 1.48e-04 | tok/s 23,486 | ETA 35.0m [2026-10-23 19:38:19][INFO] Step 140/1032 | Loss 27.5444 | LR 1.47e-04 | tok/s 24,457 | ETA 34.6m [2026-10-23 19:38:42][INFO] Step 150/1032 | Loss 27.2957 | LR 1.46e-04 | tok/s 23,674 | ETA 34.2m [2026-10-23 19:39:06][INFO] Step 160/1032 | Loss 27.4005 | LR 1.45e-04 | tok/s 23,854 | ETA 33.8m [2026-10-23 19:39:29][INFO] Step 170/1032 | Loss 27.5391 | LR 1.45e-04 | tok/s 25,137 | ETA 33.4m [2026-10-23 19:39:52][INFO] Step 180/1032 | Loss 27.3353 | LR 1.44e-04 | tok/s 23,941 | ETA 33.0m [2026-10-23 19:40:15][INFO] Step 190/1032 | Loss 27.4519 | LR 1.43e-04 | tok/s 24,092 | ETA 32.6m [2026-10-23 19:40:38][INFO] Step 200/1032 | Loss 27.8107 | LR 1.42e-04 | tok/s 24,862 | ETA 32.2m [2026-10-23 19:41:01][INFO] Step 210/1032 | Loss 27.5011 | LR 1.40e-04 | tok/s 24,722 | ETA 31.9m [2026-10-23 19:41:25][INFO] Step 220/1032 | Loss 26.9358 | LR 1.39e-04 | tok/s 24,404 | ETA 31.5m [2026-10-23 19:41:48][INFO] Step 230/1032 | Loss 27.4704 | LR 1.38e-04 | tok/s 24,662 | ETA 31.1m [2026-10-23 19:42:11][INFO] Step 240/1032 | Loss 27.0211 | LR 1.37e-04 | tok/s 23,745 | ETA 30.7m [2026-10-23 19:42:35][INFO] Step 250/1032 | Loss 27.6586 | LR 1.35e-04 | tok/s 24,295 | ETA 30.3m [2026-10-23 19:42:58][INFO] Step 260/1032 | Loss 26.9126 | LR 1.34e-04 | tok/s 24,078 | ETA 29.9m [2026-10-23 19:43:22][INFO] Step 270/1032 | Loss 27.4019 | LR 1.32e-04 | tok/s 24,265 | ETA 29.6m [2026-10-23 19:43:45][INFO] Step 280/1032 | Loss 26.9057 | LR 1.31e-04 | tok/s 24,261 | ETA 29.2m [2026-10-23 19:44:08][INFO] Step 290/1032 | Loss 27.0456 | LR 1.29e-04 | tok/s 23,598 | ETA 28.8m [2026-10-23 19:44:32][INFO] Step 300/1032 | Loss 27.1177 | LR 1.27e-04 | tok/s 24,195 | ETA 28.4m [2026-10-23 19:44:55][INFO] Step 310/1032 | Loss 27.1114 | LR 1.26e-04 | tok/s 24,146 | ETA 28.0m [2026-10-23 19:45:18][INFO] Step 320/1032 | Loss 27.4643 | LR 1.24e-04 | tok/s 24,460 | ETA 27.6m [2026-10-23 19:45:42][INFO] Step 330/1032 | Loss 27.0991 | LR 1.22e-04 | tok/s 24,520 | ETA 27.2m [2026-10-23 19:46:05][INFO] Step 340/1032 | Loss 27.2437 | LR 1.20e-04 | tok/s 24,311 | ETA 26.9m [2026-10-23 19:46:28][INFO] Step 350/1032 | Loss 26.9910 | LR 1.18e-04 | tok/s 25,078 | ETA 26.5m [2026-10-23 19:46:51][INFO] Step 360/1032 | Loss 27.1137 | LR 1.16e-04 | tok/s 24,816 | ETA 26.1m [2026-10-23 19:47:14][INFO] Step 370/1032 | Loss 26.9701 | LR 1.14e-04 | tok/s 24,222 | ETA 25.7m [2026-10-23 19:47:38][INFO] Step 380/1032 | Loss 27.0585 | LR 1.12e-04 | tok/s 23,265 | ETA 25.3m [2026-10-23 19:48:01][INFO] Step 390/1032 | Loss 27.1018 | LR 1.10e-04 | tok/s 24,020 | ETA 24.9m [2026-10-23 19:48:24][INFO] Step 400/1032 | Loss 27.3676 | LR 1.08e-04 | tok/s 24,142 | ETA 24.5m [2026-10-23 19:48:48][INFO] Step 410/1032 | Loss 27.0343 | LR 1.06e-04 | tok/s 24,018 | ETA 24.1m [2026-10-23 19:49:11][INFO] Step 420/1032 | Loss 27.2474 | LR 1.03e-04 | tok/s 24,618 | ETA 23.7m [2026-10-23 19:49:34][INFO] Step 430/1032 | Loss 27.1402 | LR 1.01e-04 | tok/s 24,487 | ETA 23.4m [2026-10-23 19:49:57][INFO] Step 440/1032 | Loss 26.8032 | LR 9.88e-05 | tok/s 24,037 | ETA 23.0m [2026-10-23 19:50:21][INFO] Step 450/1032 | Loss 26.4634 | LR 9.65e-05 | tok/s 23,837 | ETA 22.6m [2026-10-23 19:50:44][INFO] Step 460/1032 | Loss 26.8807 | LR 9.42e-05 | tok/s 24,175 | ETA 22.2m [2026-10-23 19:51:08][INFO] Step 470/1032 | Loss 26.6797 | LR 9.19e-05 | tok/s 24,001 | ETA 21.8m [2026-10-23 19:51:31][INFO] Step 480/1032 | Loss 26.8361 | LR 8.95e-05 | tok/s 24,343 | ETA 21.4m [2026-10-23 19:51:54][INFO] Step 490/1032 | Loss 26.9694 | LR 8.72e-05 | tok/s 24,140 | ETA 21.0m [2026-10-23 19:52:17][INFO] Step 500/1032 | Loss 26.7539 | LR 8.48e-05 | tok/s 24,235 | ETA 20.6m [2026-10-23 19:52:40][INFO] Step 510/1032 | Loss 26.9967 | LR 8.24e-05 | tok/s 24,532 | ETA 20.2m [2026-10-23 19:53:03][INFO] Step 520/1032 | Loss 26.8844 | LR 8.00e-05 | tok/s 24,119 | ETA 19.9m [2026-10-23 19:53:26][INFO] Step 530/1032 | Loss 26.7523 | LR 7.76e-05 | tok/s 24,572 | ETA 19.5m [2026-10-23 19:53:50][INFO] Step 540/1032 | Loss 27.2568 | LR 7.52e-05 | tok/s 24,102 | ETA 19.1m [2026-10-23 19:54:13][INFO] Step 550/1032 | Loss 27.0184 | LR 7.28e-05 | tok/s 24,493 | ETA 18.7m [2026-10-23 19:54:37][INFO] Step 560/1032 | Loss 26.6201 | LR 7.04e-05 | tok/s 23,366 | ETA 18.3m [2026-10-23 19:55:00][INFO] Step 570/1032 | Loss 26.8437 | LR 6.81e-05 | tok/s 24,769 | ETA 17.9m [2026-10-23 19:55:23][INFO] Step 580/1032 | Loss 26.5909 | LR 6.57e-05 | tok/s 24,244 | ETA 17.5m [2026-10-23 19:55:46][INFO] Step 590/1032 | Loss 27.3601 | LR 6.33e-05 | tok/s 24,151 | ETA 17.1m [2026-10-23 19:56:10][INFO] Step 600/1032 | Loss 26.7513 | LR 6.09e-05 | tok/s 24,441 | ETA 16.8m [2026-10-23 19:56:33][INFO] Step 610/1032 | Loss 27.1416 | LR 5.86e-05 | tok/s 24,760 | ETA 16.4m [2026-10-23 19:56:56][INFO] Step 620/1032 | Loss 26.5967 | LR 5.62e-05 | tok/s 24,622 | ETA 16.0m [2026-10-23 19:57:19][INFO] Step 630/1032 | Loss 27.1756 | LR 5.39e-05 | tok/s 24,083 | ETA 15.6m [2026-10-23 19:57:42][INFO] Step 640/1032 | Loss 26.9370 | LR 5.16e-05 | tok/s 24,238 | ETA 15.2m [2026-10-23 19:58:05][INFO] Step 650/1032 | Loss 26.8583 | LR 4.94e-05 | tok/s 24,865 | ETA 14.8m [2026-10-23 19:58:28][INFO] Step 660/1032 | Loss 26.6109 | LR 4.71e-05 | tok/s 24,287 | ETA 14.4m [2026-10-23 19:58:52][INFO] Step 670/1032 | Loss 26.7998 | LR 4.49e-05 | tok/s 24,374 | ETA 14.0m [2026-10-23 19:59:15][INFO] Step 680/1032 | Loss 26.5955 | LR 4.27e-05 | tok/s 24,237 | ETA 13.6m [2026-10-23 19:59:38][INFO] Step 690/1032 | Loss 27.2092 | LR 4.06e-05 | tok/s 24,257 | ETA 13.3m [2026-10-23 20:00:01][INFO] Step 700/1032 | Loss 26.6470 | LR 3.85e-05 | tok/s 23,689 | ETA 12.9m [2026-10-23 20:00:25][INFO] Step 710/1032 | Loss 26.8061 | LR 3.64e-05 | tok/s 23,955 | ETA 12.5m [2026-10-23 20:00:48][INFO] Step 720/1032 | Loss 26.9421 | LR 3.44e-05 | tok/s 24,016 | ETA 12.1m [2026-10-23 20:01:11][INFO] Step 730/1032 | Loss 26.9930 | LR 3.24e-05 | tok/s 24,816 | ETA 11.7m [2026-10-23 20:01:34][INFO] Step 740/1032 | Loss 26.8245 | LR 3.04e-05 | tok/s 24,334 | ETA 11.3m [2026-10-23 20:01:58][INFO] Step 750/1032 | Loss 26.9708 | LR 2.85e-05 | tok/s 23,594 | ETA 10.9m [2026-10-23 20:02:21][INFO] Step 760/1032 | Loss 26.9284 | LR 2.66e-05 | tok/s 23,981 | ETA 10.5m [2026-10-23 20:02:45][INFO] Step 770/1032 | Loss 26.9276 | LR 2.48e-05 | tok/s 23,468 | ETA 10.2m [2026-10-23 20:03:08][INFO] Step 780/1032 | Loss 26.5787 | LR 2.31e-05 | tok/s 24,687 | ETA 9.8m [2026-10-23 20:03:31][INFO] Step 790/1032 | Loss 26.8419 | LR 2.14e-05 | tok/s 24,191 | ETA 9.4m [2026-10-23 20:03:54][INFO] Step 800/1032 | Loss 26.3311 | LR 1.97e-05 | tok/s 24,417 | ETA 9.0m [2026-10-23 20:04:18][INFO] Step 810/1032 | Loss 26.5421 | LR 1.81e-05 | tok/s 23,242 | ETA 8.6m [2026-10-23 20:04:41][INFO] Step 820/1032 | Loss 26.5327 | LR 1.66e-05 | tok/s 24,212 | ETA 8.2m [2026-10-23 20:05:04][INFO] Step 830/1032 | Loss 26.6862 | LR 1.51e-05 | tok/s 23,012 | ETA 7.8m [2026-10-23 20:05:28][INFO] Step 840/1032 | Loss 26.5199 | LR 1.37e-05 | tok/s 23,981 | ETA 7.4m [2026-10-23 20:05:51][INFO] Step 850/1032 | Loss 26.9158 | LR 1.24e-05 | tok/s 24,203 | ETA 7.1m [2026-10-23 20:06:14][INFO] Step 860/1032 | Loss 26.9329 | LR 1.11e-05 | tok/s 24,655 | ETA 6.7m [2026-10-23 20:06:38][INFO] Step 870/1032 | Loss 26.4377 | LR 9.85e-06 | tok/s 23,438 | ETA 6.3m [2026-10-23 20:07:01][INFO] Step 880/1032 | Loss 26.6710 | LR 8.69e-06 | tok/s 24,163 | ETA 5.9m [2026-10-23 20:07:24][INFO] Step 890/1032 | Loss 26.4497 | LR 7.61e-06 | tok/s 24,188 | ETA 5.5m [2026-10-23 20:07:47][INFO] Step 900/1032 | Loss 26.8202 | LR 6.59e-06 | tok/s 23,924 | ETA 5.1m [2026-10-23 20:08:11][INFO] Step 910/1032 | Loss 26.6761 | LR 5.64e-06 | tok/s 24,141 | ETA 4.7m [2026-10-23 20:08:34][INFO] Step 920/1032 | Loss 26.4439 | LR 4.76e-06 | tok/s 23,845 | ETA 4.3m [2026-10-23 20:08:57][INFO] Step 930/1032 | Loss 26.3741 | LR 3.96e-06 | tok/s 23,789 | ETA 4.0m [2026-10-23 20:09:21][INFO] Step 940/1032 | Loss 26.7890 | LR 3.23e-06 | tok/s 24,078 | ETA 3.6m [2026-10-23 20:09:44][INFO] Step 950/1032 | Loss 26.7921 | LR 2.57e-06 | tok/s 24,208 | ETA 3.2m [2026-10-23 20:10:07][INFO] Step 960/1032 | Loss 26.7195 | LR 1.98e-06 | tok/s 23,989 | ETA 2.8m [2026-10-23 20:10:30][INFO] Step 970/1032 | Loss 26.4671 | LR 1.47e-06 | tok/s 24,200 | ETA 2.4m [2026-10-23 20:10:54][INFO] Step 980/1032 | Loss 26.2816 | LR 1.04e-06 | tok/s 23,237 | ETA 2.0m [2026-10-23 20:11:17][INFO] Step 990/1032 | Loss 26.9578 | LR 6.76e-07 | tok/s 24,326 | ETA 1.6m [2026-10-23 20:11:40][INFO] Step 1000/1032 | Loss 27.0065 | LR 3.93e-07 | tok/s 24,277 | ETA 1.2m [2026-10-23 20:12:03][INFO] Step 1010/1032 | Loss 26.5352 | LR 1.86e-07 | tok/s 24,255 | ETA 0.9m [2026-10-23 20:12:27][INFO] Step 1020/1032 | Loss 26.7942 | LR 5.53e-08 | tok/s 23,840 | ETA 0.5m [2026-10-23 20:12:50][INFO] Step 1030/1032 | Loss 26.6643 | LR 1.54e-09 | tok/s 24,119 | ETA 0.1m [2026-10-23 20:12:55][INFO] Step 1032/1032 | Loss 5.5751 | LR 0.00e+00 | tok/s 25,316 | ETA 0.0m [2026-10-23 20:12:57][INFO] Training complete! Final adapter saved to /workspace/checkpoints/qwen_ft_test/final_adapter