| Failed to load /usr/local/lib/python3.12/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /usr/local/lib/python3.12/site-packages/torchao/_C_cutlass_90a.abi3.so |
| Failed to load /usr/local/lib/python3.12/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /usr/local/lib/python3.12/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so |
| /usr/local/lib/python3.12/site-packages/torchao/quantization/quant_api.py:1745: SyntaxWarning: invalid escape sequence '\.' |
| * regex for parameter names, must start with `re:`, e.g. `re:language\.layers\..+\.q_proj.weight`. |
|
|
|
|
| config.json: 0%| | 0.00/1.96k [00:00<?, ?B/s] |
| config.json: 100%|██████████| 1.96k/1.96k [00:00<00:00, 7.29MB/s] |
| |
| |
| model.safetensors.index.json: 0%| | 0.00/23.6k [00:00<?, ?B/s] |
| model.safetensors.index.json: 100%|██████████| 23.6k/23.6k [00:00<00:00, 62.1MB/s] |
| |
| |
| Downloading bytes: | 0.00B |
| |
| Reconstructing (incomplete total...): | | 0.00B / 0.00B |
| |
| Fetching 2 files: 0%| | 0/2 [00:00<?, ?it/s] |
| |
| Reconstructing (incomplete total...): 0%| | 0.00B / 6.70GB |
| |
| Reconstructing (incomplete total...): 0%| | 0.00B / 6.70GB |
| |
| Reconstructing (incomplete total...): 4%|▍ | 256MB / 6.70GB, ???B/s |
| |
| Downloading bytes: ▍ | 301MB, ???B/s |
| |
| Downloading bytes: █▏ | 781MB, 282MB/s |
| |
| Reconstructing (incomplete total...): 14%|█▍ | 969MB / 6.70GB, 240MB/s |
| |
| Downloading bytes: █▊ | 1.19GB, 394MB/s |
| |
| Downloading bytes: ██▎ | 1.55GB, 346MB/s |
| |
| Reconstructing (incomplete total...): 20%|██ | 1.34GB / 6.70GB, 344MB/s |
| |
| Downloading bytes: ██▊ | 1.90GB, 337MB/s |
| |
| Reconstructing (incomplete total...): 27%|██▋ | 1.81GB / 6.70GB, 289MB/s |
| |
| Downloading bytes: ███▍ | 2.28GB, 337MB/s |
| |
| Reconstructing (incomplete total...): 33%|███▎ | 2.21GB / 6.70GB, 340MB/s |
| |
| Downloading bytes: ███▉ | 2.65GB, 349MB/s |
| |
| Downloading bytes: ████▌ | 3.02GB, 352MB/s |
| |
| Reconstructing (incomplete total...): 40%|████ | 2.68GB / 6.70GB, 316MB/s |
| |
| Downloading bytes: █████ | 3.38GB, 354MB/s |
| |
| Reconstructing (incomplete total...): 45%|████▌ | 3.05GB / 6.70GB, 285MB/s |
| |
| Fetching 2 files: 50%|█████ | 1/2 [00:10<00:10, 10.36s/it] |
| |
| Reconstructing (incomplete total...): 50%|█████ | 3.38GB / 6.70GB, 290MB/s |
| |
| Downloading bytes: █████▌ | 3.76GB, 355MB/s |
| |
| Reconstructing (incomplete total...): 64%|██████▎ | 4.25GB / 6.70GB, 257MB/s |
| |
| Downloading bytes: ██████ | 4.08GB, 254MB/s |
| |
| Downloading bytes: ██████▋ | 4.50GB, 138MB/s |
| |
| Reconstructing (incomplete total...): 69%|██████▊ | 4.59GB / 6.70GB, 207MB/s |
| |
| Downloading bytes: ███████▏ | 4.81GB, 178MB/s |
| |
| Reconstructing (incomplete total...): 75%|███████▍ | 4.99GB / 6.70GB, 204MB/s |
| |
| Downloading bytes: ███████▋ | 5.11GB, 200MB/s |
| |
| Reconstructing (incomplete total...): 88%|████████▊ | 5.86GB / 6.70GB, 185MB/s |
| |
| Downloading bytes: ████████▏ | 5.45GB, 177MB/s |
| |
| Reconstructing (incomplete total...): 93%|█████████▎| 6.20GB / 6.70GB, 173MB/s |
| |
| Reconstructing (incomplete total...): 100%|██████████| 6.70GB / 6.70GB, 179MB/s |
| |
| Fetching 2 files: 100%|██████████| 2/2 [00:31<00:00, 16.78s/it] |
| Fetching 2 files: 100%|██████████| 2/2 [00:31<00:00, 15.82s/it] |
| |
| |
| Downloading bytes: ██████████| 5.84GB, 113MB/s |
| |
| Download complete: ██████████| 5.84GB, 113MB/s |
| |
| Reconstruction complete: 100%|██████████| 6.70GB / 6.70GB, 172MB/s |
| Download complete: ██████████| 5.84GB, 113MB/s |
| |
| Reconstruction complete: 100%|██████████| 6.70GB / 6.70GB, 172MB/s |
| |
| |
| Loading weights: 0%| | 0/290 [00:00<?, ?it/s] |
| |
| Loading weights: 0%| | 1/290 [00:01<05:37, 1.17s/it] |
| |
| Loading weights: 18%|█▊ | 51/290 [00:02<00:08, 27.53it/s] |
| |
| Loading weights: 37%|███▋ | 108/290 [00:03<00:04, 39.86it/s] |
| |
| Loading weights: 57%|█████▋ | 164/290 [00:04<00:02, 45.94it/s] |
| |
| Loading weights: 76%|███████▌ | 220/290 [00:05<00:01, 49.18it/s] |
| |
| Loading weights: 93%|█████████▎| 270/290 [00:07<00:00, 33.14it/s] |
| Loading weights: 100%|██████████| 290/290 [00:08<00:00, 33.42it/s] |
| |
| |
| generation_config.json: 0%| | 0.00/132 [00:00<?, ?B/s] |
| generation_config.json: 100%|██████████| 132/132 [00:00<00:00, 481kB/s] |
| {"gpu_memory": {"time_utc": "2026-11-10T08:45:06.045931+00:00", "phase": "train-b", "event": "after_model_load", "optimizer_step": null, "allocated_bytes": 3720019968, "reserved_bytes": 3871342592, "peak_allocated_bytes": 4672696320, "peak_reserved_bytes": 4697620480, "device_free_bytes": 11830231040, "device_total_bytes": 15828320256}} |
| |
| |
| 0%| | 0/40 [00:00<?, ?it/s]{"gpu_memory": {"time_utc": "2026-11-10T08:46:29.026714+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 1, "skipped": false, "allocated_bytes": 3761222656, "reserved_bytes": 6767509504, "peak_allocated_bytes": 6315941376, "peak_reserved_bytes": 6767509504, "device_free_bytes": 8776777728, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 1, "target_tokens": 2480} |
| |
| |
| 2%|▎ | 1/40 [01:08<44:48, 68.95s/it] |
| |
| |
| {'loss': '0.02113', 'grad_norm': '0.09704', 'learning_rate': '0.0001', 'epoch': '0.05229'} |
| |
| |
| 2%|▎ | 1/40 [01:08<44:48, 68.95s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:47:30.617100+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 2, "skipped": false, "allocated_bytes": 3761170432, "reserved_bytes": 6790578176, "peak_allocated_bytes": 6315941376, "peak_reserved_bytes": 6790578176, "device_free_bytes": 8753709056, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 2, "target_tokens": 4532} |
| |
| |
| 5%|▌ | 2/40 [02:10<40:55, 64.62s/it] |
| |
| |
| {'loss': '0.01378', 'grad_norm': '0.1328', 'learning_rate': '9.985e-05', 'epoch': '0.1046'} |
| |
| |
| 5%|▌ | 2/40 [02:10<40:55, 64.62s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:48:41.111748+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 3, "skipped": false, "allocated_bytes": 3761233408, "reserved_bytes": 7084179456, "peak_allocated_bytes": 6569742336, "peak_reserved_bytes": 7084179456, "device_free_bytes": 8460107776, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 3, "target_tokens": 6995} |
| |
| |
| 8%|▊ | 3/40 [03:21<41:30, 67.30s/it] |
| |
| |
| {'loss': '0.03266', 'grad_norm': '0.1694', 'learning_rate': '9.938e-05', 'epoch': '0.1569'} |
| |
| |
| 8%|▊ | 3/40 [03:21<41:30, 67.30s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:49:55.602673+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 4, "skipped": false, "allocated_bytes": 3761254912, "reserved_bytes": 7797211136, "peak_allocated_bytes": 7070235648, "peak_reserved_bytes": 7797211136, "device_free_bytes": 7747076096, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 4, "target_tokens": 9681} |
| |
| |
| 10%|█ | 4/40 [04:35<42:05, 70.14s/it] |
| |
| |
| {'loss': '0.02937', 'grad_norm': '0.1407', 'learning_rate': '9.862e-05', 'epoch': '0.2092'} |
| |
| |
| 10%|█ | 4/40 [04:35<42:05, 70.14s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:51:11.625509+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 5, "skipped": false, "allocated_bytes": 3761261056, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 5, "target_tokens": 12218} |
| |
| |
| 12%|█▎ | 5/40 [05:51<42:09, 72.26s/it] |
| |
| |
| {'loss': '0.03006', 'grad_norm': '0.1919', 'learning_rate': '9.755e-05', 'epoch': '0.2614'} |
| |
| |
| 12%|█▎ | 5/40 [05:51<42:09, 72.26s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:52:19.480362+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 6, "skipped": false, "allocated_bytes": 3761213440, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 6, "target_tokens": 14598} |
| |
| |
| 15%|█▌ | 6/40 [06:59<40:05, 70.76s/it] |
| |
| |
| {'loss': '0.01909', 'grad_norm': '0.1366', 'learning_rate': '9.619e-05', 'epoch': '0.3137'} |
| |
| |
| 15%|█▌ | 6/40 [06:59<40:05, 70.76s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:53:30.386437+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 7, "skipped": false, "allocated_bytes": 3761239552, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 7, "target_tokens": 17708} |
| |
| |
| 18%|█▊ | 7/40 [08:10<38:56, 70.81s/it] |
| |
| |
| {'loss': '0.03284', 'grad_norm': '0.1497', 'learning_rate': '9.455e-05', 'epoch': '0.366'} |
| |
| |
| 18%|█▊ | 7/40 [08:10<38:56, 70.81s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:54:43.191489+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 8, "skipped": false, "allocated_bytes": 3761250304, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 8, "target_tokens": 20634} |
| |
| |
| 20%|██ | 8/40 [09:23<38:06, 71.45s/it] |
| |
| |
| {'loss': '0.03365', 'grad_norm': '0.1679', 'learning_rate': '9.263e-05', 'epoch': '0.4183'} |
| |
| |
| 20%|██ | 8/40 [09:23<38:06, 71.45s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:55:52.220189+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 9, "skipped": false, "allocated_bytes": 3761225728, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 9, "target_tokens": 23256} |
| |
| |
| 22%|██▎ | 9/40 [10:32<36:31, 70.69s/it] |
| |
| |
| {'loss': '0.02218', 'grad_norm': '0.1704', 'learning_rate': '9.045e-05', 'epoch': '0.4706'} |
| |
| |
| 22%|██▎ | 9/40 [10:32<36:31, 70.69s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:56:57.635173+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 10, "skipped": false, "allocated_bytes": 3761202688, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 10, "target_tokens": 25984} |
| |
| |
| 25%|██▌ | 10/40 [11:37<34:31, 69.06s/it] |
| |
| |
| {'loss': '0.03839', 'grad_norm': '0.192', 'learning_rate': '8.802e-05', 'epoch': '0.5229'} |
| |
| |
| 25%|██▌ | 10/40 [11:37<34:31, 69.06s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:58:10.876833+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 11, "skipped": false, "allocated_bytes": 3761244160, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 11, "target_tokens": 28787} |
| |
| |
| 28%|██▊ | 11/40 [12:50<33:59, 70.34s/it] |
| |
| |
| {'loss': '0.03552', 'grad_norm': '0.1539', 'learning_rate': '8.536e-05', 'epoch': '0.5752'} |
| |
| |
| 28%|██▊ | 11/40 [12:50<33:59, 70.34s/it]{"gpu_memory": {"time_utc": "2026-11-10T08:59:24.219687+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 12, "skipped": false, "allocated_bytes": 3761250304, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 12, "target_tokens": 31529} |
| |
| |
| 30%|███ | 12/40 [14:04<33:15, 71.25s/it] |
| |
| |
| {'loss': '0.02618', 'grad_norm': '0.1348', 'learning_rate': '8.247e-05', 'epoch': '0.6275'} |
| |
| |
| 30%|███ | 12/40 [14:04<33:15, 71.25s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:00:33.305950+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 13, "skipped": false, "allocated_bytes": 3761225728, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 13, "target_tokens": 34461} |
| |
| |
| 32%|███▎ | 13/40 [15:13<31:46, 70.60s/it] |
| |
| |
| {'loss': '0.03279', 'grad_norm': '0.1605', 'learning_rate': '7.939e-05', 'epoch': '0.6797'} |
| |
| |
| 32%|███▎ | 13/40 [15:13<31:46, 70.60s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:01:51.666007+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 14, "skipped": false, "allocated_bytes": 3761284096, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 14, "target_tokens": 37195} |
| |
| |
| 35%|███▌ | 14/40 [16:31<31:36, 72.94s/it] |
| |
| |
| {'loss': '0.03119', 'grad_norm': '0.1518', 'learning_rate': '7.612e-05', 'epoch': '0.732'} |
| |
| |
| 35%|███▌ | 14/40 [16:31<31:36, 72.94s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:02:55.904736+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 15, "skipped": false, "allocated_bytes": 3761187328, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 15, "target_tokens": 39663} |
| |
| |
| 38%|███▊ | 15/40 [17:35<29:17, 70.32s/it] |
| |
| |
| {'loss': '0.02935', 'grad_norm': '0.1228', 'learning_rate': '7.27e-05', 'epoch': '0.7843'} |
| |
| |
| 38%|███▊ | 15/40 [17:35<29:17, 70.32s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:04:06.236991+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 16, "skipped": false, "allocated_bytes": 3761230336, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 16, "target_tokens": 42044} |
| |
| |
| 40%|████ | 16/40 [18:46<28:07, 70.32s/it] |
| |
| |
| {'loss': '0.02547', 'grad_norm': '0.1606', 'learning_rate': '6.913e-05', 'epoch': '0.8366'} |
| |
| |
| 40%|████ | 16/40 [18:46<28:07, 70.32s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:05:17.756024+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 17, "skipped": false, "allocated_bytes": 3761242624, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 17, "target_tokens": 44642} |
| |
| |
| 42%|████▎ | 17/40 [19:57<27:05, 70.68s/it] |
| |
| |
| {'loss': '0.01997', 'grad_norm': '0.1152', 'learning_rate': '6.545e-05', 'epoch': '0.8889'} |
| |
| |
| 42%|████▎ | 17/40 [19:57<27:05, 70.68s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:06:34.909208+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 18, "skipped": false, "allocated_bytes": 3761276416, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 18, "target_tokens": 46858} |
| |
| |
| 45%|████▌ | 18/40 [21:14<26:37, 72.63s/it] |
| |
| |
| {'loss': '0.02441', 'grad_norm': '0.2005', 'learning_rate': '6.167e-05', 'epoch': '0.9412'} |
| |
| |
| 45%|████▌ | 18/40 [21:14<26:37, 72.63s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:07:43.352261+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 19, "skipped": false, "allocated_bytes": 3761218048, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 19, "target_tokens": 49459} |
| |
| |
| 48%|████▊ | 19/40 [22:23<24:58, 71.37s/it] |
| |
| |
| {'loss': '0.02846', 'grad_norm': '0.1578', 'learning_rate': '5.782e-05', 'epoch': '0.9935'} |
| |
| |
| 48%|████▊ | 19/40 [22:23<24:58, 71.37s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:07:50.396405+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 20, "skipped": false, "allocated_bytes": 3760619008, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 20, "target_tokens": 49717} |
| |
| |
| 50%|█████ | 20/40 [22:30<17:21, 52.06s/it] |
| |
| |
| {'loss': '0.00365', 'grad_norm': '0.08222', 'learning_rate': '5.392e-05', 'epoch': '1'} |
| |
| |
| 50%|█████ | 20/40 [22:30<17:21, 52.06s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:08:58.159079+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 21, "skipped": false, "allocated_bytes": 3761208832, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 21, "target_tokens": 51949} |
| |
| |
| 52%|█████▎ | 21/40 [23:38<17:58, 56.77s/it] |
| |
| |
| {'loss': '0.01505', 'grad_norm': '0.1053', 'learning_rate': '5e-05', 'epoch': '1.052'} |
| |
| |
| 52%|█████▎ | 21/40 [23:38<17:58, 56.77s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:10:11.572590+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 22, "skipped": false, "allocated_bytes": 3761254912, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 22, "target_tokens": 54619} |
| |
| |
| 55%|█████▌ | 22/40 [24:51<18:31, 61.77s/it] |
| |
| |
| {'loss': '0.01629', 'grad_norm': '0.08077', 'learning_rate': '4.608e-05', 'epoch': '1.105'} |
| |
| |
| 55%|█████▌ | 22/40 [24:51<18:31, 61.77s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:11:21.083954+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 23, "skipped": false, "allocated_bytes": 3761224192, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 23, "target_tokens": 57221} |
| |
| |
| 57%|█████▊ | 23/40 [26:01<18:09, 64.09s/it] |
| |
| |
| {'loss': '0.01787', 'grad_norm': '0.1102', 'learning_rate': '4.218e-05', 'epoch': '1.157'} |
| |
| |
| 57%|█████▊ | 23/40 [26:01<18:09, 64.09s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:12:28.645796+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 24, "skipped": false, "allocated_bytes": 3761211904, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 24, "target_tokens": 59677} |
| |
| |
| 60%|██████ | 24/40 [27:08<17:22, 65.13s/it] |
| |
| |
| {'loss': '0.01582', 'grad_norm': '0.08077', 'learning_rate': '3.833e-05', 'epoch': '1.209'} |
| |
| |
| 60%|██████ | 24/40 [27:08<17:22, 65.13s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:13:38.041051+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 25, "skipped": false, "allocated_bytes": 3761227264, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 25, "target_tokens": 62132} |
| |
| |
| 62%|██████▎ | 25/40 [28:17<16:36, 66.41s/it] |
| |
| |
| {'loss': '0.01388', 'grad_norm': '0.09024', 'learning_rate': '3.455e-05', 'epoch': '1.261'} |
| |
| |
| 62%|██████▎ | 25/40 [28:17<16:36, 66.41s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:14:58.055885+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 26, "skipped": false, "allocated_bytes": 3761300992, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 26, "target_tokens": 65370} |
| |
| |
| 65%|██████▌ | 26/40 [29:37<16:26, 70.49s/it] |
| |
| |
| {'loss': '0.0243', 'grad_norm': '0.09814', 'learning_rate': '3.087e-05', 'epoch': '1.314'} |
| |
| |
| 65%|██████▌ | 26/40 [29:37<16:26, 70.49s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:16:03.059862+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 27, "skipped": false, "allocated_bytes": 3761191936, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 27, "target_tokens": 67760} |
| |
| |
| 68%|██████▊ | 27/40 [30:42<14:54, 68.85s/it] |
| |
| |
| {'loss': '0.01308', 'grad_norm': '0.07287', 'learning_rate': '2.73e-05', 'epoch': '1.366'} |
| |
| |
| 68%|██████▊ | 27/40 [30:42<14:54, 68.85s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:17:15.610234+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 28, "skipped": false, "allocated_bytes": 3761250304, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 28, "target_tokens": 70528} |
| |
| |
| 70%|███████ | 28/40 [31:55<13:59, 69.96s/it] |
| |
| |
| {'loss': '0.01675', 'grad_norm': '0.09151', 'learning_rate': '2.388e-05', 'epoch': '1.418'} |
| |
| |
| 70%|███████ | 28/40 [31:55<13:59, 69.96s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:18:30.060486+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 29, "skipped": false, "allocated_bytes": 3761264128, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 29, "target_tokens": 73416} |
| |
| |
| 72%|███████▎ | 29/40 [33:09<13:04, 71.31s/it] |
| |
| |
| {'loss': '0.01473', 'grad_norm': '0.0698', 'learning_rate': '2.061e-05', 'epoch': '1.471'} |
| |
| |
| 72%|███████▎ | 29/40 [33:09<13:04, 71.31s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:19:48.080322+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 30, "skipped": false, "allocated_bytes": 3761279488, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 30, "target_tokens": 75817} |
| |
| |
| 75%|███████▌ | 30/40 [34:28<12:13, 73.32s/it] |
| |
| |
| {'loss': '0.01624', 'grad_norm': '0.07827', 'learning_rate': '1.753e-05', 'epoch': '1.523'} |
| |
| |
| 75%|███████▌ | 30/40 [34:28<12:13, 73.32s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:20:49.787597+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 31, "skipped": false, "allocated_bytes": 3761165824, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374102528, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 31, "target_tokens": 78066} |
| |
| |
| 78%|███████▊ | 31/40 [35:29<10:28, 69.84s/it] |
| |
| |
| {'loss': '0.0145', 'grad_norm': '0.08211', 'learning_rate': '1.464e-05', 'epoch': '1.575'} |
| |
| |
| 78%|███████▊ | 31/40 [35:29<10:28, 69.84s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:22:14.043506+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 32, "skipped": false, "allocated_bytes": 3761324032, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 32, "target_tokens": 80792} |
| |
| |
| 80%|████████ | 32/40 [36:53<09:53, 74.16s/it] |
| |
| |
| {'loss': '0.01556', 'grad_norm': '0.0854', 'learning_rate': '1.198e-05', 'epoch': '1.627'} |
| |
| |
| 80%|████████ | 32/40 [36:53<09:53, 74.16s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:23:18.245283+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 33, "skipped": false, "allocated_bytes": 3761190400, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 33, "target_tokens": 83505} |
| |
| |
| 82%|████████▎ | 33/40 [37:58<08:18, 71.17s/it] |
| |
| |
| {'loss': '0.01741', 'grad_norm': '0.08092', 'learning_rate': '9.549e-06', 'epoch': '1.68'} |
| |
| |
| 82%|████████▎ | 33/40 [37:58<08:18, 71.17s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:24:31.174354+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 34, "skipped": false, "allocated_bytes": 3761248768, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 34, "target_tokens": 86405} |
| |
| |
| 85%|████████▌ | 34/40 [39:11<07:10, 71.70s/it] |
| |
| |
| {'loss': '0.01475', 'grad_norm': '0.06094', 'learning_rate': '7.368e-06', 'epoch': '1.732'} |
| |
| |
| 85%|████████▌ | 34/40 [39:11<07:10, 71.70s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:25:38.441164+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 35, "skipped": false, "allocated_bytes": 3761213440, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 35, "target_tokens": 89117} |
| |
| |
| 88%|████████▊ | 35/40 [40:18<05:51, 70.37s/it] |
| |
| |
| {'loss': '0.02319', 'grad_norm': '0.1096', 'learning_rate': '5.45e-06', 'epoch': '1.784'} |
| |
| |
| 88%|████████▊ | 35/40 [40:18<05:51, 70.37s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:26:45.860962+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 36, "skipped": false, "allocated_bytes": 3761213440, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 36, "target_tokens": 91548} |
| |
| |
| 90%|█████████ | 36/40 [41:25<04:37, 69.49s/it] |
| |
| |
| {'loss': '0.01488', 'grad_norm': '0.06782', 'learning_rate': '3.806e-06', 'epoch': '1.837'} |
| |
| |
| 90%|█████████ | 36/40 [41:25<04:37, 69.49s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:27:57.962323+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 37, "skipped": false, "allocated_bytes": 3761248768, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 37, "target_tokens": 94312} |
| |
| |
| 92%|█████████▎| 37/40 [42:37<03:30, 70.27s/it] |
| |
| |
| {'loss': '0.01667', 'grad_norm': '0.08227', 'learning_rate': '2.447e-06', 'epoch': '1.889'} |
| |
| |
| 92%|█████████▎| 37/40 [42:37<03:30, 70.27s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:29:08.215742+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 38, "skipped": false, "allocated_bytes": 3761230336, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 38, "target_tokens": 96769} |
| |
| |
| 95%|█████████▌| 38/40 [43:48<02:20, 70.27s/it] |
| |
| |
| {'loss': '0.01294', 'grad_norm': '0.07646', 'learning_rate': '1.382e-06', 'epoch': '1.941'} |
| |
| |
| 95%|█████████▌| 38/40 [43:48<02:20, 70.27s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:30:12.050732+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 39, "skipped": false, "allocated_bytes": 3761185792, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 39, "target_tokens": 99157} |
| |
| |
| 98%|█████████▊| 39/40 [44:51<01:08, 68.34s/it] |
| |
| |
| {'loss': '0.01309', 'grad_norm': '0.08377', 'learning_rate': '6.156e-07', 'epoch': '1.993'} |
| |
| |
| 98%|█████████▊| 39/40 [44:51<01:08, 68.34s/it]{"gpu_memory": {"time_utc": "2026-11-10T09:30:18.906469+00:00", "phase": "train-b", "event": "after_optimizer_step", "optimizer_step": 40, "skipped": false, "allocated_bytes": 3760617472, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"optimizer_step": 40, "target_tokens": 99434} |
| |
| |
| 100%|██████████| 40/40 [44:58<00:00, 49.89s/it] |
| |
| |
| {'loss': '0.01022', 'grad_norm': '0.2026', 'learning_rate': '1.541e-07', 'epoch': '2'} |
| |
| |
| 100%|██████████| 40/40 [44:58<00:00, 49.89s/it] |
| |
| |
| {'train_runtime': '2699', 'train_samples_per_second': '0.227', 'train_steps_per_second': '0.015', 'train_loss': '0.02118', 'epoch': '2'} |
| |
| |
| 100%|██████████| 40/40 [44:59<00:00, 49.89s/it] |
| 100%|██████████| 40/40 [44:59<00:00, 67.48s/it] |
| {"gpu_memory": {"time_utc": "2026-11-10T09:30:20.022761+00:00", "phase": "train-b", "event": "training_complete", "optimizer_step": 40, "allocated_bytes": 3760545280, "reserved_bytes": 9456058368, "peak_allocated_bytes": 8374165504, "peak_reserved_bytes": 9456058368, "device_free_bytes": 6088228864, "device_total_bytes": 15828320256}} |
| {"backend": "AgentTune.create_distill_trainer/Project.distill/AuditedSFTFactory", "global_step": 40, "loss": 0.021183327259495854, "runtime_seconds": 2713.295591436, "adapter_sha256": "90b04823368f50d6ef5ce85c171fffff317b11a3bf43a591cf3952f01b076837"} |
| {"step": "train-b", "completed": true} |
| |
