File size: 32,915 Bytes
f7625f2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 | Failed to load /usr/local/lib/python3.12/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /usr/local/lib/python3.12/site-packages/torchao/_C_cutlass_90a.abi3.so
Failed to load /usr/local/lib/python3.12/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /usr/local/lib/python3.12/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so
/usr/local/lib/python3.12/site-packages/torchao/quantization/quant_api.py:1745: SyntaxWarning: invalid escape sequence '\.'
* regex for parameter names, must start with `re:`, e.g. `re:language\.layers\..+\.q_proj.weight`.
config.json: 0%| | 0.00/1.96k [00:00<?, ?B/s]
config.json: 100%|██████████| 1.96k/1.96k [00:00<00:00, 7.29MB/s]
model.safetensors.index.json: 0%| | 0.00/23.6k [00:00<?, ?B/s]
model.safetensors.index.json: 100%|██████████| 23.6k/23.6k [00:00<00:00, 62.1MB/s]
Downloading bytes: | 0.00B
Reconstructing (incomplete total...): | | 0.00B / 0.00B
Fetching 2 files: 0%| | 0/2 [00:00<?, ?it/s]
Reconstructing (incomplete total...): 0%| | 0.00B / 6.70GB
Reconstructing (incomplete total...): 0%| | 0.00B / 6.70GB
Reconstructing (incomplete total...): 4%|▍ | 256MB / 6.70GB, ???B/s
Downloading bytes: ▍ | 301MB, ???B/s
Downloading bytes: █▏ | 781MB, 282MB/s
Reconstructing (incomplete total...): 14%|█▍ | 969MB / 6.70GB, 240MB/s
Downloading bytes: █▊ | 1.19GB, 394MB/s
Downloading bytes: ██▎ | 1.55GB, 346MB/s
Reconstructing (incomplete total...): 20%|██ | 1.34GB / 6.70GB, 344MB/s
Downloading bytes: ██▊ | 1.90GB, 337MB/s
Reconstructing (incomplete total...): 27%|██▋ | 1.81GB / 6.70GB, 289MB/s
Downloading bytes: ███▍ | 2.28GB, 337MB/s
Reconstructing (incomplete total...): 33%|███▎ | 2.21GB / 6.70GB, 340MB/s
Downloading bytes: ███▉ | 2.65GB, 349MB/s
Downloading bytes: ████▌ | 3.02GB, 352MB/s
Reconstructing (incomplete total...): 40%|████ | 2.68GB / 6.70GB, 316MB/s
Downloading bytes: █████ | 3.38GB, 354MB/s
Reconstructing (incomplete total...): 45%|████▌ | 3.05GB / 6.70GB, 285MB/s
Fetching 2 files: 50%|█████ | 1/2 [00:10<00:10, 10.36s/it]
Reconstructing (incomplete total...): 50%|█████ | 3.38GB / 6.70GB, 290MB/s
Downloading bytes: █████▌ | 3.76GB, 355MB/s
Reconstructing (incomplete total...): 64%|██████▎ | 4.25GB / 6.70GB, 257MB/s
Downloading bytes: ██████ | 4.08GB, 254MB/s
Downloading bytes: ██████▋ | 4.50GB, 138MB/s
Reconstructing (incomplete total...): 69%|██████▊ | 4.59GB / 6.70GB, 207MB/s
Downloading bytes: ███████▏ | 4.81GB, 178MB/s
Reconstructing (incomplete total...): 75%|███████▍ | 4.99GB / 6.70GB, 204MB/s
Downloading bytes: ███████▋ | 5.11GB, 200MB/s
Reconstructing (incomplete total...): 88%|████████▊ | 5.86GB / 6.70GB, 185MB/s
Downloading bytes: ████████▏ | 5.45GB, 177MB/s
Reconstructing (incomplete total...): 93%|█████████▎| 6.20GB / 6.70GB, 173MB/s
Reconstructing (incomplete total...): 100%|██████████| 6.70GB / 6.70GB, 179MB/s
Fetching 2 files: 100%|██████████| 2/2 [00:31<00:00, 16.78s/it]
Fetching 2 files: 100%|██████████| 2/2 [00:31<00:00, 15.82s/it]
Downloading bytes: ██████████| 5.84GB, 113MB/s
Download complete: ██████████| 5.84GB, 113MB/s
Reconstruction complete: 100%|██████████| 6.70GB / 6.70GB, 172MB/s
Download complete: ██████████| 5.84GB, 113MB/s
Reconstruction complete: 100%|██████████| 6.70GB / 6.70GB, 172MB/s
Loading weights: 0%| | 0/290 [00:00<?, ?it/s]
Loading weights: 0%| | 1/290 [00:01<05:37, 1.17s/it]
Loading weights: 18%|█▊ | 51/290 [00:02<00:08, 27.53it/s]
Loading weights: 37%|███▋ | 108/290 [00:03<00:04, 39.86it/s]
Loading weights: 57%|█████▋ | 164/290 [00:04<00:02, 45.94it/s]
Loading weights: 76%|███████▌ | 220/290 [00:05<00:01, 49.18it/s]
Loading weights: 93%|█████████▎| 270/290 [00:07<00:00, 33.14it/s]
Loading weights: 100%|██████████| 290/290 [00:08<00:00, 33.42it/s]
generation_config.json: 0%| | 0.00/132 [00:00<?, ?B/s]
generation_config.json: 100%|██████████| 132/132 [00:00<00:00, 481kB/s]
{"gpu_memory": {"time_utc": "2026-11-10T08:45:06.045931+00:00", "phase": "train-b", "event": "after_model_load", "optimizer_step": null, "allocated_bytes": 3720019968, "reserved_bytes": 3871342592, "peak_allocated_bytes": 4672696320, "peak_reserved_bytes": 4697620480, "device_free_bytes": 11830231040, "device_total_bytes": 15828320256}}
0%| | 0/40 [00:00<?, ?it/s]{"gpu_memory": {"time_utc": "2026-11-10T08:46:29.026714+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 1, "skipped": false, "allocated_bytes": 3761222656, "reserved_bytes": 6767509504, "peak_allocated_bytes": 6315941376, "peak_reserved_bytes": 6767509504, "device_free_bytes": 8776777728, "device_total_bytes": 15828320256}}
{"optimizer_step": 1, "target_tokens": 2480}
2%|▎ | 1/40 [01:08<44:48, 68.95s/it]
{'loss': '0.02113', 'grad_norm': '0.09704', 'learning_rate': '0.0001', 'epoch': '0.05229'}
2%|▎ | 1/40 [01:08<44:48, 68.95s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:47:30.617100+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 2, "skipped": false, "allocated_bytes": 3761170432, "reserved_bytes": 6790578176, "peak_allocated_bytes": 6315941376, "peak_reserved_bytes": 6790578176, "device_free_bytes": 8753709056, "device_total_bytes": 15828320256}}
{"optimizer_step": 2, "target_tokens": 4532}
5%|▌ | 2/40 [02:10<40:55, 64.62s/it]
{'loss': '0.01378', 'grad_norm': '0.1328', 'learning_rate': '9.985e-05', 'epoch': '0.1046'}
5%|▌ | 2/40 [02:10<40:55, 64.62s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:48:41.111748+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 3, "skipped": false, "allocated_bytes": 3761233408, "reserved_bytes": 7084179456, "peak_allocated_bytes": 6569742336, "peak_reserved_bytes": 7084179456, "device_free_bytes": 8460107776, "device_total_bytes": 15828320256}}
{"optimizer_step": 3, "target_tokens": 6995}
8%|▊ | 3/40 [03:21<41:30, 67.30s/it]
{'loss': '0.03266', 'grad_norm': '0.1694', 'learning_rate': '9.938e-05', 'epoch': '0.1569'}
8%|▊ | 3/40 [03:21<41:30, 67.30s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:49:55.602673+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 4, "skipped": false, "allocated_bytes": 3761254912, "reserved_bytes": 7797211136, "peak_allocated_bytes": 7070235648, "peak_reserved_bytes": 7797211136, "device_free_bytes": 7747076096, "device_total_bytes": 15828320256}}
{"optimizer_step": 4, "target_tokens": 9681}
10%|█ | 4/40 [04:35<42:05, 70.14s/it]
{'loss': '0.02937', 'grad_norm': '0.1407', 'learning_rate': '9.862e-05', 'epoch': '0.2092'}
10%|█ | 4/40 [04:35<42:05, 70.14s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:51:11.625509+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 5, "skipped": false, "allocated_bytes": 3761261056, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 5, "target_tokens": 12218}
12%|█▎ | 5/40 [05:51<42:09, 72.26s/it]
{'loss': '0.03006', 'grad_norm': '0.1919', 'learning_rate': '9.755e-05', 'epoch': '0.2614'}
12%|█▎ | 5/40 [05:51<42:09, 72.26s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:52:19.480362+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 6, "skipped": false, "allocated_bytes": 3761213440, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 6, "target_tokens": 14598}
15%|█▌ | 6/40 [06:59<40:05, 70.76s/it]
{'loss': '0.01909', 'grad_norm': '0.1366', 'learning_rate': '9.619e-05', 'epoch': '0.3137'}
15%|█▌ | 6/40 [06:59<40:05, 70.76s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:53:30.386437+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 7, "skipped": false, "allocated_bytes": 3761239552, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 7, "target_tokens": 17708}
18%|█▊ | 7/40 [08:10<38:56, 70.81s/it]
{'loss': '0.03284', 'grad_norm': '0.1497', 'learning_rate': '9.455e-05', 'epoch': '0.366'}
18%|█▊ | 7/40 [08:10<38:56, 70.81s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:54:43.191489+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 8, "skipped": false, "allocated_bytes": 3761250304, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 8, "target_tokens": 20634}
20%|██ | 8/40 [09:23<38:06, 71.45s/it]
{'loss': '0.03365', 'grad_norm': '0.1679', 'learning_rate': '9.263e-05', 'epoch': '0.4183'}
20%|██ | 8/40 [09:23<38:06, 71.45s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:55:52.220189+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 9, "skipped": false, "allocated_bytes": 3761225728, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 9, "target_tokens": 23256}
22%|██▎ | 9/40 [10:32<36:31, 70.69s/it]
{'loss': '0.02218', 'grad_norm': '0.1704', 'learning_rate': '9.045e-05', 'epoch': '0.4706'}
22%|██▎ | 9/40 [10:32<36:31, 70.69s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:56:57.635173+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 10, "skipped": false, "allocated_bytes": 3761202688, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 10, "target_tokens": 25984}
25%|██▌ | 10/40 [11:37<34:31, 69.06s/it]
{'loss': '0.03839', 'grad_norm': '0.192', 'learning_rate': '8.802e-05', 'epoch': '0.5229'}
25%|██▌ | 10/40 [11:37<34:31, 69.06s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:58:10.876833+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 11, "skipped": false, "allocated_bytes": 3761244160, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 11, "target_tokens": 28787}
28%|██▊ | 11/40 [12:50<33:59, 70.34s/it]
{'loss': '0.03552', 'grad_norm': '0.1539', 'learning_rate': '8.536e-05', 'epoch': '0.5752'}
28%|██▊ | 11/40 [12:50<33:59, 70.34s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:59:24.219687+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 12, "skipped": false, "allocated_bytes": 3761250304, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 12, "target_tokens": 31529}
30%|███ | 12/40 [14:04<33:15, 71.25s/it]
{'loss': '0.02618', 'grad_norm': '0.1348', 'learning_rate': '8.247e-05', 'epoch': '0.6275'}
30%|███ | 12/40 [14:04<33:15, 71.25s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:00:33.305950+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 13, "skipped": false, "allocated_bytes": 3761225728, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 13, "target_tokens": 34461}
32%|███▎ | 13/40 [15:13<31:46, 70.60s/it]
{'loss': '0.03279', 'grad_norm': '0.1605', 'learning_rate': '7.939e-05', 'epoch': '0.6797'}
32%|███▎ | 13/40 [15:13<31:46, 70.60s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:01:51.666007+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 14, "skipped": false, "allocated_bytes": 3761284096, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 14, "target_tokens": 37195}
35%|███▌ | 14/40 [16:31<31:36, 72.94s/it]
{'loss': '0.03119', 'grad_norm': '0.1518', 'learning_rate': '7.612e-05', 'epoch': '0.732'}
35%|███▌ | 14/40 [16:31<31:36, 72.94s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:02:55.904736+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 15, "skipped": false, "allocated_bytes": 3761187328, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 15, "target_tokens": 39663}
38%|███▊ | 15/40 [17:35<29:17, 70.32s/it]
{'loss': '0.02935', 'grad_norm': '0.1228', 'learning_rate': '7.27e-05', 'epoch': '0.7843'}
38%|███▊ | 15/40 [17:35<29:17, 70.32s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:04:06.236991+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 16, "skipped": false, "allocated_bytes": 3761230336, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 16, "target_tokens": 42044}
40%|████ | 16/40 [18:46<28:07, 70.32s/it]
{'loss': '0.02547', 'grad_norm': '0.1606', 'learning_rate': '6.913e-05', 'epoch': '0.8366'}
40%|████ | 16/40 [18:46<28:07, 70.32s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:05:17.756024+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 17, "skipped": false, "allocated_bytes": 3761242624, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 17, "target_tokens": 44642}
42%|████▎ | 17/40 [19:57<27:05, 70.68s/it]
{'loss': '0.01997', 'grad_norm': '0.1152', 'learning_rate': '6.545e-05', 'epoch': '0.8889'}
42%|████▎ | 17/40 [19:57<27:05, 70.68s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:06:34.909208+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 18, "skipped": false, "allocated_bytes": 3761276416, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 18, "target_tokens": 46858}
45%|████▌ | 18/40 [21:14<26:37, 72.63s/it]
{'loss': '0.02441', 'grad_norm': '0.2005', 'learning_rate': '6.167e-05', 'epoch': '0.9412'}
45%|████▌ | 18/40 [21:14<26:37, 72.63s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:07:43.352261+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 19, "skipped": false, "allocated_bytes": 3761218048, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 19, "target_tokens": 49459}
48%|████▊ | 19/40 [22:23<24:58, 71.37s/it]
{'loss': '0.02846', 'grad_norm': '0.1578', 'learning_rate': '5.782e-05', 'epoch': '0.9935'}
48%|████▊ | 19/40 [22:23<24:58, 71.37s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:07:50.396405+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 20, "skipped": false, "allocated_bytes": 3760619008, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 20, "target_tokens": 49717}
50%|█████ | 20/40 [22:30<17:21, 52.06s/it]
{'loss': '0.00365', 'grad_norm': '0.08222', 'learning_rate': '5.392e-05', 'epoch': '1'}
50%|█████ | 20/40 [22:30<17:21, 52.06s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:08:58.159079+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 21, "skipped": false, "allocated_bytes": 3761208832, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 21, "target_tokens": 51949}
52%|█████▎ | 21/40 [23:38<17:58, 56.77s/it]
{'loss': '0.01505', 'grad_norm': '0.1053', 'learning_rate': '5e-05', 'epoch': '1.052'}
52%|█████▎ | 21/40 [23:38<17:58, 56.77s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:10:11.572590+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 22, "skipped": false, "allocated_bytes": 3761254912, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 22, "target_tokens": 54619}
55%|█████▌ | 22/40 [24:51<18:31, 61.77s/it]
{'loss': '0.01629', 'grad_norm': '0.08077', 'learning_rate': '4.608e-05', 'epoch': '1.105'}
55%|█████▌ | 22/40 [24:51<18:31, 61.77s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:11:21.083954+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 23, "skipped": false, "allocated_bytes": 3761224192, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 23, "target_tokens": 57221}
57%|█████▊ | 23/40 [26:01<18:09, 64.09s/it]
{'loss': '0.01787', 'grad_norm': '0.1102', 'learning_rate': '4.218e-05', 'epoch': '1.157'}
57%|█████▊ | 23/40 [26:01<18:09, 64.09s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:12:28.645796+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 24, "skipped": false, "allocated_bytes": 3761211904, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 24, "target_tokens": 59677}
60%|██████ | 24/40 [27:08<17:22, 65.13s/it]
{'loss': '0.01582', 'grad_norm': '0.08077', 'learning_rate': '3.833e-05', 'epoch': '1.209'}
60%|██████ | 24/40 [27:08<17:22, 65.13s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:13:38.041051+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 25, "skipped": false, "allocated_bytes": 3761227264, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 25, "target_tokens": 62132}
62%|██████▎ | 25/40 [28:17<16:36, 66.41s/it]
{'loss': '0.01388', 'grad_norm': '0.09024', 'learning_rate': '3.455e-05', 'epoch': '1.261'}
62%|██████▎ | 25/40 [28:17<16:36, 66.41s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:14:58.055885+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 26, "skipped": false, "allocated_bytes": 3761300992, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 26, "target_tokens": 65370}
65%|██████▌ | 26/40 [29:37<16:26, 70.49s/it]
{'loss': '0.0243', 'grad_norm': '0.09814', 'learning_rate': '3.087e-05', 'epoch': '1.314'}
65%|██████▌ | 26/40 [29:37<16:26, 70.49s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:16:03.059862+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 27, "skipped": false, "allocated_bytes": 3761191936, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 27, "target_tokens": 67760}
68%|██████▊ | 27/40 [30:42<14:54, 68.85s/it]
{'loss': '0.01308', 'grad_norm': '0.07287', 'learning_rate': '2.73e-05', 'epoch': '1.366'}
68%|██████▊ | 27/40 [30:42<14:54, 68.85s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:17:15.610234+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 28, "skipped": false, "allocated_bytes": 3761250304, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 28, "target_tokens": 70528}
70%|███████ | 28/40 [31:55<13:59, 69.96s/it]
{'loss': '0.01675', 'grad_norm': '0.09151', 'learning_rate': '2.388e-05', 'epoch': '1.418'}
70%|███████ | 28/40 [31:55<13:59, 69.96s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:18:30.060486+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 29, "skipped": false, "allocated_bytes": 3761264128, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 29, "target_tokens": 73416}
72%|███████▎ | 29/40 [33:09<13:04, 71.31s/it]
{'loss': '0.01473', 'grad_norm': '0.0698', 'learning_rate': '2.061e-05', 'epoch': '1.471'}
72%|███████▎ | 29/40 [33:09<13:04, 71.31s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:19:48.080322+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 30, "skipped": false, "allocated_bytes": 3761279488, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 30, "target_tokens": 75817}
75%|███████▌ | 30/40 [34:28<12:13, 73.32s/it]
{'loss': '0.01624', 'grad_norm': '0.07827', 'learning_rate': '1.753e-05', 'epoch': '1.523'}
75%|███████▌ | 30/40 [34:28<12:13, 73.32s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:20:49.787597+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 31, "skipped": false, "allocated_bytes": 3761165824, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 31, "target_tokens": 78066}
78%|███████▊ | 31/40 [35:29<10:28, 69.84s/it]
{'loss': '0.0145', 'grad_norm': '0.08211', 'learning_rate': '1.464e-05', 'epoch': '1.575'}
78%|███████▊ | 31/40 [35:29<10:28, 69.84s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:22:14.043506+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 32, "skipped": false, "allocated_bytes": 3761324032, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 32, "target_tokens": 80792}
80%|████████ | 32/40 [36:53<09:53, 74.16s/it]
{'loss': '0.01556', 'grad_norm': '0.0854', 'learning_rate': '1.198e-05', 'epoch': '1.627'}
80%|████████ | 32/40 [36:53<09:53, 74.16s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:23:18.245283+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 33, "skipped": false, "allocated_bytes": 3761190400, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 33, "target_tokens": 83505}
82%|████████▎ | 33/40 [37:58<08:18, 71.17s/it]
{'loss': '0.01741', 'grad_norm': '0.08092', 'learning_rate': '9.549e-06', 'epoch': '1.68'}
82%|████████▎ | 33/40 [37:58<08:18, 71.17s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:24:31.174354+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 34, "skipped": false, "allocated_bytes": 3761248768, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 34, "target_tokens": 86405}
85%|████████▌ | 34/40 [39:11<07:10, 71.70s/it]
{'loss': '0.01475', 'grad_norm': '0.06094', 'learning_rate': '7.368e-06', 'epoch': '1.732'}
85%|████████▌ | 34/40 [39:11<07:10, 71.70s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:25:38.441164+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 35, "skipped": false, "allocated_bytes": 3761213440, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 35, "target_tokens": 89117}
88%|████████▊ | 35/40 [40:18<05:51, 70.37s/it]
{'loss': '0.02319', 'grad_norm': '0.1096', 'learning_rate': '5.45e-06', 'epoch': '1.784'}
88%|████████▊ | 35/40 [40:18<05:51, 70.37s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:26:45.860962+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 36, "skipped": false, "allocated_bytes": 3761213440, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 36, "target_tokens": 91548}
90%|█████████ | 36/40 [41:25<04:37, 69.49s/it]
{'loss': '0.01488', 'grad_norm': '0.06782', 'learning_rate': '3.806e-06', 'epoch': '1.837'}
90%|█████████ | 36/40 [41:25<04:37, 69.49s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:27:57.962323+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 37, "skipped": false, "allocated_bytes": 3761248768, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 37, "target_tokens": 94312}
92%|█████████▎| 37/40 [42:37<03:30, 70.27s/it]
{'loss': '0.01667', 'grad_norm': '0.08227', 'learning_rate': '2.447e-06', 'epoch': '1.889'}
92%|█████████▎| 37/40 [42:37<03:30, 70.27s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:29:08.215742+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 38, "skipped": false, "allocated_bytes": 3761230336, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 38, "target_tokens": 96769}
95%|█████████▌| 38/40 [43:48<02:20, 70.27s/it]
{'loss': '0.01294', 'grad_norm': '0.07646', 'learning_rate': '1.382e-06', 'epoch': '1.941'}
95%|█████████▌| 38/40 [43:48<02:20, 70.27s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:30:12.050732+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 39, "skipped": false, "allocated_bytes": 3761185792, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 39, "target_tokens": 99157}
98%|█████████▊| 39/40 [44:51<01:08, 68.34s/it]
{'loss': '0.01309', 'grad_norm': '0.08377', 'learning_rate': '6.156e-07', 'epoch': '1.993'}
98%|█████████▊| 39/40 [44:51<01:08, 68.34s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:30:18.906469+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 40, "skipped": false, "allocated_bytes": 3760617472, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"optimizer_step": 40, "target_tokens": 99434}
100%|██████████| 40/40 [44:58<00:00, 49.89s/it]
{'loss': '0.01022', 'grad_norm': '0.2026', 'learning_rate': '1.541e-07', 'epoch': '2'}
100%|██████████| 40/40 [44:58<00:00, 49.89s/it]
{'train_runtime': '2699', 'train_samples_per_second': '0.227', 'train_steps_per_second': '0.015', 'train_loss': '0.02118', 'epoch': '2'}
100%|██████████| 40/40 [44:59<00:00, 49.89s/it]
100%|██████████| 40/40 [44:59<00:00, 67.48s/it]
{"gpu_memory": {"time_utc": "2026-11-10T09:30:20.022761+00:00", "phase": "train-b", "event": "training_complete", "optimizer_step": 40, "allocated_bytes": 3760545280, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}}
{"backend": "AgentTune.create_distill_trainer/Project.distill/AuditedSFTFactory", "global_step": 40, "loss": 0.021183327259495854, "runtime_seconds": 2713.295591436, "adapter_sha256": "90b04823368f50d6ef5ce85c171fffff317b11a3bf43a591cf3952f01b076837"}
{"step": "train-b", "completed": true}
|