| 2026-09-12 11:11:02 [all] ============================================================================== |
| 2026-09-12 11:11:02 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-12 11:11:02 [all] sampling 1.5BB of text to train the vocabulary on |
| 2026-09-12 11:11:14 [all] ============================================================================== |
| 2026-09-12 11:11:14 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-12 11:11:14 [all] sampling 1.5BB of text to train the vocabulary on |
| 2026-09-12 11:14:32 [all] ============================================================================== |
| 2026-09-12 11:14:32 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-12 11:14:51 [all] ============================================================================== |
| 2026-09-12 11:14:51 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-12 11:14:52 [all] sampling 1.5BB of text to train the vocabulary on |
| 2026-09-12 11:15:07 [all] sample 95.3MB / 1.5BB (20K docs, 6.05MB/s) |
| 2026-09-12 11:15:12 [all] sample 192MB / 1.5BB (40K docs, 9.34MB/s) |
| 2026-09-12 11:15:17 [all] sample 290MB / 1.5BB (60K docs, 11.4MB/s) |
| 2026-09-12 11:15:22 [all] sample 387MB / 1.5BB (80K docs, 12.6MB/s) |
| 2026-09-12 11:15:27 [all] sample 480MB / 1.5BB (100K docs, 13.5MB/s) |
| 2026-09-12 11:15:32 [all] sample 575MB / 1.5BB (120K docs, 14.2MB/s) |
| 2026-09-12 11:15:37 [all] sample 667MB / 1.5BB (140K docs, 14.7MB/s) |
| 2026-09-12 11:15:42 [all] sample 761MB / 1.5BB (160K docs, 15.2MB/s) |
| 2026-09-12 11:15:47 [all] sample 855MB / 1.5BB (180K docs, 15.5MB/s) |
| 2026-09-12 11:15:52 [all] sample 948MB / 1.5BB (200K docs, 15.8MB/s) |
| 2026-09-12 11:15:57 [all] sample 1.04BB / 1.5BB (220K docs, 16MB/s) |
| 2026-09-12 11:16:02 [all] sample 1.14BB / 1.5BB (240K docs, 16.2MB/s) |
| 2026-09-12 11:16:07 [all] sample 1.23BB / 1.5BB (260K docs, 16.4MB/s) |
| 2026-09-12 11:16:11 [all] sample 1.32BB / 1.5BB (280K docs, 16.6MB/s) |
| 2026-09-12 11:16:16 [all] sample 1.42BB / 1.5BB (300K docs, 16.7MB/s) |
| 2026-09-12 11:16:21 [all] sample written: 1.5BB in 318K docs |
| 2026-09-12 11:16:21 [all] training byte-level BPE — vocab_size=32000, specials=<|endoftext|>, <|pad|>, <fim_prefix>, <fim_middle>, <fim_suffix> |
| 2026-09-12 11:17:58 [all] vocabulary trained in 00h01m36s -> runs/hypernix.3-mini/tokenizer/tokenizer.json (32000 tokens) |
| 2026-09-12 11:17:58 [all] round-trip ok — 75 chars -> 39 tokens (1.92 chars/token) |
| 2026-09-12 11:17:58 [all] packing 1.21B tokens -> 2.42 GB (uint16); 183.8 GB free on this volume |
| 2026-09-12 11:18:29 [all] packed 29.1M / 1.21B tokens (2.4%) 939K tok/s eta 00h20m57s |
| 2026-09-12 11:19:00 [all] packed 65.6M / 1.21B tokens (5.4%) 1.19M tok/s eta 00h16m04s |
| 2026-09-12 11:19:30 [all] packed 103M / 1.21B tokens (8.5%) 1.24M tok/s eta 00h14m52s |
| 2026-09-12 11:20:00 [all] packed 141M / 1.21B tokens (11.6%) 1.25M tok/s eta 00h14m15s |
| 2026-09-12 11:20:30 [all] packed 178M / 1.21B tokens (14.7%) 1.25M tok/s eta 00h13m48s |
| 2026-09-12 11:21:00 [all] packed 216M / 1.21B tokens (17.8%) 1.24M tok/s eta 00h13m22s |
| 2026-09-12 11:21:31 [all] packed 252M / 1.21B tokens (20.8%) 1.19M tok/s eta 00h13m25s |
| 2026-09-12 11:22:01 [all] packed 286M / 1.21B tokens (23.7%) 1.13M tok/s eta 00h13m34s |
| 2026-09-12 11:22:31 [all] packed 323M / 1.21B tokens (26.7%) 1.22M tok/s eta 00h12m07s |
| 2026-09-12 11:23:02 [all] packed 361M / 1.21B tokens (29.8%) 1.23M tok/s eta 00h11m31s |
| 2026-09-12 11:23:33 [all] packed 399M / 1.21B tokens (32.9%) 1.22M tok/s eta 00h11m03s |
| 2026-09-12 11:24:03 [all] packed 435M / 1.21B tokens (36.0%) 1.22M tok/s eta 00h10m36s |
| 2026-09-12 11:24:33 [all] packed 471M / 1.21B tokens (38.9%) 1.17M tok/s eta 00h10m32s |
| 2026-09-12 11:25:03 [all] packed 504M / 1.21B tokens (41.7%) 1.12M tok/s eta 00h10m29s |
| 2026-09-12 11:25:34 [all] packed 539M / 1.21B tokens (44.5%) 1.12M tok/s eta 00h10m01s |
| 2026-09-12 11:26:04 [all] packed 562M / 1.21B tokens (46.5%) 785K tok/s eta 00h13m45s |
| 2026-09-12 11:26:34 [all] packed 585M / 1.21B tokens (48.4%) 759K tok/s eta 00h13m43s |
| 2026-09-12 11:27:04 [all] packed 608M / 1.21B tokens (50.3%) 767K tok/s eta 00h13m04s |
| 2026-09-12 11:27:35 [all] packed 631M / 1.21B tokens (52.2%) 745K tok/s eta 00h12m57s |
| 2026-09-12 11:28:05 [all] packed 654M / 1.21B tokens (54.0%) 751K tok/s eta 00h12m21s |
| 2026-09-12 11:28:36 [all] packed 676M / 1.21B tokens (55.9%) 717K tok/s eta 00h12m24s |
| 2026-09-12 11:29:07 [all] packed 699M / 1.21B tokens (57.8%) 751K tok/s eta 00h11m20s |
| 2026-09-12 11:29:38 [all] packed 721M / 1.21B tokens (59.6%) 730K tok/s eta 00h11m09s |
| 2026-09-12 11:30:08 [all] packed 743M / 1.21B tokens (61.4%) 717K tok/s eta 00h10m50s |
| 2026-09-12 11:30:40 [all] packed 758M / 1.21B tokens (62.6%) 443K tok/s eta 00h17m00s |
| 2026-09-12 11:31:11 [all] packed 792M / 1.21B tokens (65.5%) 1.12M tok/s eta 00h06m11s |
| 2026-09-12 11:31:42 [all] packed 819M / 1.21B tokens (67.7%) 862K tok/s eta 00h07m33s |
| 2026-09-12 11:32:13 [all] packed 843M / 1.21B tokens (69.7%) 772K tok/s eta 00h07m55s |
| 2026-09-12 11:32:43 [all] packed 878M / 1.21B tokens (72.6%) 1.16M tok/s eta 00h04m46s |
| 2026-09-12 11:33:14 [all] packed 914M / 1.21B tokens (75.6%) 1.2M tok/s eta 00h04m07s |
| 2026-09-12 11:33:44 [all] packed 950M / 1.21B tokens (78.5%) 1.18M tok/s eta 00h03m40s |
| 2026-09-12 11:34:15 [all] packed 987M / 1.21B tokens (81.6%) 1.19M tok/s eta 00h03m07s |
| 2026-09-12 11:34:45 [all] packed 1.02B / 1.21B tokens (84.3%) 1.11M tok/s eta 00h02m51s |
| 2026-09-12 11:35:16 [all] packed 1.06B / 1.21B tokens (87.3%) 1.16M tok/s eta 00h02m12s |
| 2026-09-12 11:35:47 [all] packed 1.09B / 1.21B tokens (90.3%) 1.19M tok/s eta 00h01m38s |
| 2026-09-12 11:36:17 [all] packed 1.13B / 1.21B tokens (93.3%) 1.2M tok/s eta 00h01m07s |
| 2026-09-12 11:36:48 [all] packed 1.17B / 1.21B tokens (96.3%) 1.2M tok/s eta 00h00m37s |
| 2026-09-12 11:37:18 [all] packed 1.2B / 1.21B tokens (99.4%) 1.21M tok/s eta 00h00m06s |
| 2026-09-12 11:37:26 [all] packed 1.21B tokens from 1.17M documents in 00h19m28s |
| 2026-09-12 11:37:26 [all] dataset footprint: 2.42 GB on disk, ceiling 100 GB — clear |
| 2026-09-12 11:37:26 [all] removed the temporary tokenizer sample |
| 2026-09-12 11:37:26 [all] gpu=NVIDIA GeForce GTX 1080 capability=sm_61 torch=2.7.1+cu118 arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_37,sm_90 |
| 2026-09-12 15:02:33 [all] ============================================================================== |
| 2026-09-12 15:02:33 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-12 15:02:33 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-12 15:02:33 [all] packing 1.21B tokens -> 2.42 GB (uint16); 173.8 GB free on this volume |
| 2026-09-12 15:02:33 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-12 15:28:18 [all] ============================================================================== |
| 2026-09-12 15:28:18 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-12 15:28:18 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-12 15:28:18 [all] packing 1.21B tokens -> 2.42 GB (uint16); 172.1 GB free on this volume |
| 2026-09-12 15:28:18 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-12 15:28:19 [all] gpu=NVIDIA GeForce GTX 1080 capability=sm_61 torch=2.14.0+cu126 arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 |
| 2026-09-12 15:28:19 [all] patched BrewerAttention.forward (causal/window mask composition + is_causal fast path) |
| 2026-09-12 15:28:30 [all] ============================================================================== |
| 2026-09-12 15:28:30 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-12 15:28:30 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-12 15:28:30 [all] packing 1.21B tokens -> 2.42 GB (uint16); 172.1 GB free on this volume |
| 2026-09-12 15:28:30 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-12 15:28:32 [all] gpu=NVIDIA GeForce GTX 1080 capability=sm_61 torch=2.14.0+cu126 arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 |
| 2026-09-12 15:28:32 [all] patched BrewerAttention.forward (causal/window mask composition + is_causal fast path) |
| 2026-09-12 15:28:39 [all] built hypernix.3-mini — 48,706,048 parameters (48.7060 M) | 8L d_model=512 heads=8/2 d_ff=2203 ctx=1024 vocab=32000 tied=True sliding_window=False |
| 2026-09-12 15:28:39 [all] 33.6% of parameters are the (tied) embedding matrix; 32,322,048 are 'body' parameters |
| 2026-09-12 15:28:39 [all] parameter memory: total=48,706,048 trainable=48,706,048 frozen=0 (185.8 MB) |
| 2026-09-12 15:28:39 [all] token bin: 1.21B tokens (2.42 GB), val tail = 2M tokens |
| 2026-09-12 15:28:39 [all] plan: 1.21B tokens = 18464 steps x 65.5K tokens/step (micro=8 ctx=1024 accum=8) | peak_lr=6.00e-04 warmup=369 | deadline=168h |
| 2026-09-12 15:28:39 [all] pre-flight (GTX 1080 = 8.87 TFLOPS fp32): 1d16h05m at 30% MFU, 3d08h10m at 15%, 6d06h20m at 8% — the budget needs better than 7.2% to land inside 168h. The real number replaces this at step 100. |
| 2026-09-12 15:28:39 [all] CUDA OOM at step 0 (micro 0/8) |
| 2026-09-12 15:28:39 [all] VRAM ladder -> rung 1: checkpointed cross-entropy (chunk=2048 rows); ~1 GB saved, ~10% slower |
| 2026-09-12 15:32:34 [all] step 10/18464 loss=10.4848 ppl=35767.70 lr=1.626e-05 | 2.79K tok/s | 655K done | eta 5d00h22m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 15:40:25 [all] step 20/18464 loss=10.4770 ppl=35488.21 lr=3.252e-05 | 1.39K tok/s | 1.31M done | eta 10d01h25m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 15:40:25 [all] projected finish 10d01h37m exceeds the 168h deadline |
| 2026-09-12 15:49:09 [all] step 30/18464 loss=10.4610 ppl=34924.85 lr=4.878e-05 | 1.25K tok/s | 1.97M done | eta 11d04h18m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 15:49:09 [all] projected finish 11d04h38m exceeds the 168h deadline |
| 2026-09-12 15:57:40 [all] step 40/18464 loss=10.4368 ppl=34092.00 lr=6.504e-05 | 1.28K tok/s | 2.62M done | eta 10d21h24m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 15:57:40 [all] projected finish 10d21h53m exceeds the 168h deadline |
| 2026-09-12 15:59:20 [all] checkpoint @ step 42 (2.75M tokens, 00h30m40s elapsed) |
| 2026-09-12 16:06:22 [all] step 50/18464 loss=10.4073 ppl=33099.68 lr=8.130e-05 | 1.26K tok/s | 3.28M done | eta 11d03h00m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 16:06:22 [all] projected finish 11d03h38m exceeds the 168h deadline |
| 2026-09-12 16:15:23 [all] step 60/18464 loss=10.3676 ppl=31813.04 lr=9.756e-05 | 1.21K tok/s | 3.93M done | eta 11d12h50m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 16:15:23 [all] projected finish 11d13h37m exceeds the 168h deadline |
| 2026-09-12 16:23:42 [all] step 70/18464 loss=10.3174 ppl=30253.44 lr=1.138e-04 | 1.31K tok/s | 4.59M done | eta 10d14h55m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 16:23:42 [all] projected finish 10d15h51m exceeds the 168h deadline |
| 2026-09-12 16:29:28 [all] checkpoint @ step 77 (5.05M tokens, 01h00m48s elapsed) |
| 2026-09-12 16:31:59 [all] step 80/18464 loss=10.2599 ppl=28564.22 lr=1.301e-04 | 1.32K tok/s | 5.24M done | eta 10d13h41m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 16:31:59 [all] projected finish 10d14h44m exceeds the 168h deadline |
| 2026-09-12 16:41:16 [all] step 90/18464 loss=10.1928 ppl=26710.53 lr=1.463e-04 | 1.18K tok/s | 5.9M done | eta 11d20h03m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 16:41:16 [all] projected finish 11d21h16m exceeds the 168h deadline |
| 2026-09-12 16:51:47 [all] step 100/18464 loss=10.1192 ppl=24814.49 lr=1.626e-04 | 1.04K tok/s | 6.55M done | eta 13d09h58m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 16:51:47 [all] projected finish 13d11h21m exceeds the 168h deadline |
| 2026-09-12 16:51:47 [all] calibration: 1.31K tok/s sustained -> 10d15h48m for the full 1.21B tokens (deadline 7d00h00m, 00h00m00s headroom) |
| 2026-09-12 16:51:47 [all] this run will overrun the deadline. Options: --enforce-deadline (trim tokens), --ctx 512 (~1.7x faster), or fewer tokens via --tokens |
| 2026-09-12 17:00:16 [all] checkpoint @ step 108 (7.08M tokens, 01h31m36s elapsed) |
| 2026-09-12 17:02:18 [all] step 110/18464 loss=10.0435 ppl=23005.49 lr=1.789e-04 | 1.04K tok/s | 7.21M done | eta 13d09h52m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 17:02:18 [all] projected finish 13d11h26m exceeds the 168h deadline |
| 2026-09-12 17:12:09 [all] step 120/18464 loss=9.9765 ppl=21514.00 lr=1.951e-04 | 1.11K tok/s | 7.86M done | eta 12d13h20m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 17:12:09 [all] projected finish 12d15h04m exceeds the 168h deadline |
| 2026-09-12 17:21:30 [all] step 130/18464 loss=9.8873 ppl=19678.61 lr=2.114e-04 | 1.17K tok/s | 8.52M done | eta 11d21h43m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 17:21:30 [all] projected finish 11d23h35m exceeds the 168h deadline |
| 2026-09-12 17:30:36 [all] step 140/18464 loss=9.8260 ppl=18508.24 lr=2.276e-04 | 1.2K tok/s | 9.18M done | eta 11d13h25m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 17:30:36 [all] projected finish 11d15h27m exceeds the 168h deadline |
| 2026-09-12 17:30:36 [all] checkpoint @ step 140 (9.18M tokens, 02h01m56s elapsed) |
| 2026-09-12 17:39:35 [all] step 150/18464 loss=9.7628 ppl=17374.47 lr=2.439e-04 | 1.22K tok/s | 9.83M done | eta 11d10h19m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 17:39:35 [all] projected finish 11d12h29m exceeds the 168h deadline |
| 2026-09-12 17:48:25 [all] step 160/18464 loss=9.7012 ppl=16336.98 lr=2.602e-04 | 1.24K tok/s | 10.5M done | eta 11d05h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 17:48:25 [all] projected finish 11d07h56m exceeds the 168h deadline |
| 2026-09-12 17:57:43 [all] step 170/18464 loss=9.6367 ppl=15316.29 lr=2.764e-04 | 1.17K tok/s | 11.1M done | eta 11d19h32m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 17:57:43 [all] projected finish 11d22h01m exceeds the 168h deadline |
| 2026-09-12 18:01:35 [all] checkpoint @ step 174 (11.4M tokens, 02h32m55s elapsed) |
| 2026-09-12 18:07:29 [all] step 180/18464 loss=9.5922 ppl=14650.22 lr=2.927e-04 | 1.12K tok/s | 11.8M done | eta 12d09h39m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 18:07:29 [all] projected finish 12d12h18m exceeds the 168h deadline |
| 2026-09-12 18:17:26 [all] step 190/18464 loss=9.5293 ppl=13756.33 lr=3.089e-04 | 1.1K tok/s | 12.5M done | eta 12d14h55m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 18:17:26 [all] projected finish 12d17h44m exceeds the 168h deadline |
| 2026-09-12 18:27:31 [all] step 200/18464 loss=9.4827 ppl=13130.36 lr=3.252e-04 | 1.08K tok/s | 13.1M done | eta 12d19h10m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 18:27:31 [all] projected finish 12d22h09m exceeds the 168h deadline |
| 2026-09-12 18:31:35 [all] checkpoint @ step 204 (13.4M tokens, 03h02m55s elapsed) |
| 2026-09-12 18:37:41 [all] step 210/18464 loss=9.4410 ppl=12593.85 lr=3.415e-04 | 1.07K tok/s | 13.8M done | eta 12d21h12m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 18:37:41 [all] projected finish 13d00h21m exceeds the 168h deadline |
| 2026-09-12 18:47:22 [all] step 220/18464 loss=9.3752 ppl=11792.37 lr=3.577e-04 | 1.13K tok/s | 14.4M done | eta 12d06h25m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 18:47:22 [all] projected finish 12d09h43m exceeds the 168h deadline |
| 2026-09-12 18:56:35 [all] step 230/18464 loss=9.3274 ppl=11241.97 lr=3.740e-04 | 1.19K tok/s | 15.1M done | eta 11d15h54m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 18:56:35 [all] projected finish 11d19h22m exceeds the 168h deadline |
| 2026-09-12 19:02:15 [all] checkpoint @ step 236 (15.5M tokens, 03h33m35s elapsed) |
| 2026-09-12 19:06:08 [all] step 240/18464 loss=9.2889 ppl=10817.42 lr=3.902e-04 | 1.14K tok/s | 15.7M done | eta 12d02h19m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 19:06:08 [all] projected finish 12d05h56m exceeds the 168h deadline |
| 2026-09-12 19:16:00 [all] step 250/18464 loss=9.2471 ppl=10374.27 lr=4.065e-04 | 1.11K tok/s | 16.4M done | eta 12d11h39m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 19:16:00 [all] projected finish 12d15h26m exceeds the 168h deadline |
| 2026-09-12 19:26:00 [all] step 260/18464 loss=9.2203 ppl=10099.83 lr=4.228e-04 | 1.09K tok/s | 17M done | eta 12d15h11m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 19:26:00 [all] projected finish 12d19h08m exceeds the 168h deadline |
| 2026-09-12 19:33:06 [all] checkpoint @ step 267 (17.5M tokens, 04h04m27s elapsed) |
| 2026-09-12 19:36:08 [all] step 270/18464 loss=9.1636 ppl=9542.89 lr=4.390e-04 | 1.08K tok/s | 17.7M done | eta 12d19h02m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 19:36:08 [all] projected finish 12d23h10m exceeds the 168h deadline |
| 2026-09-12 19:46:14 [all] step 280/18464 loss=9.1313 ppl=9240.29 lr=4.553e-04 | 1.08K tok/s | 18.4M done | eta 12d18h26m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 19:46:14 [all] projected finish 12d22h44m exceeds the 168h deadline |
| 2026-09-12 19:56:21 [all] step 290/18464 loss=9.0790 ppl=8769.03 lr=4.715e-04 | 1.08K tok/s | 19M done | eta 12d18h23m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 19:56:21 [all] projected finish 12d22h50m exceeds the 168h deadline |
| 2026-09-12 20:03:58 [all] checkpoint @ step 298 (19.5M tokens, 04h35m18s elapsed) |
| 2026-09-12 20:05:46 [all] step 300/18464 loss=9.0588 ppl=8593.41 lr=4.878e-04 | 1.16K tok/s | 19.7M done | eta 11d20h45m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 20:05:46 [all] projected finish 12d01h22m exceeds the 168h deadline |
| 2026-09-12 20:15:27 [all] step 310/18464 loss=9.0193 ppl=8261.13 lr=5.041e-04 | 1.13K tok/s | 20.3M done | eta 12d05h17m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 20:15:27 [all] projected finish 12d10h03m exceeds the 168h deadline |
| 2026-09-12 20:27:05 [all] step 320/18464 loss=8.9875 ppl=8002.12 lr=5.203e-04 | 939 tok/s | 21M done | eta 14d15h47m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 20:27:05 [all] projected finish 14d20h45m exceeds the 168h deadline |
| 2026-09-12 20:35:09 [all] checkpoint @ step 326 (21.4M tokens, 05h06m29s elapsed) |
| 2026-09-12 20:40:40 [all] step 330/18464 loss=8.9799 ppl=7941.88 lr=5.366e-04 | 805 tok/s | 21.6M done | eta 17d02h15m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 20:40:40 [all] projected finish 17d07h27m exceeds the 168h deadline |
| 2026-09-12 20:54:33 [all] step 340/18464 loss=8.9371 ppl=7608.87 lr=5.528e-04 | 786 tok/s | 22.3M done | eta 17d11h31m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 20:54:33 [all] projected finish 17d16h57m exceeds the 168h deadline |
| 2026-09-12 21:05:34 [all] checkpoint @ step 348 (22.8M tokens, 05h36m54s elapsed) |
| 2026-09-12 21:08:07 [all] step 350/18464 loss=8.9153 ppl=7444.90 lr=5.691e-04 | 806 tok/s | 22.9M done | eta 17d01h22m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 21:08:07 [all] projected finish 17d07h02m exceeds the 168h deadline |
| 2026-09-12 21:20:04 [all] step 360/18464 loss=8.8956 ppl=7300.06 lr=5.854e-04 | 913 tok/s | 23.6M done | eta 15d00h52m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 21:20:04 [all] projected finish 15d06h44m exceeds the 168h deadline |
| 2026-09-12 21:31:12 [all] step 370/18464 loss=8.8714 ppl=7125.09 lr=6.000e-04 | 982 tok/s | 24.2M done | eta 13d23h31m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 21:31:12 [all] projected finish 14d05h33m exceeds the 168h deadline |
| 2026-09-12 21:36:35 [all] checkpoint @ step 375 (24.6M tokens, 06h07m55s elapsed) |
| 2026-09-12 21:41:54 [all] step 380/18464 loss=8.8467 ppl=6951.44 lr=6.000e-04 | 1.02K tok/s | 24.9M done | eta 13d10h38m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 21:41:54 [all] projected finish 13d16h52m exceeds the 168h deadline |
| 2026-09-12 21:52:18 [all] step 390/18464 loss=8.8137 ppl=6725.70 lr=6.000e-04 | 1.05K tok/s | 25.6M done | eta 13d01h26m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 21:52:18 [all] projected finish 13d07h50m exceeds the 168h deadline |
| 2026-09-12 22:02:28 [all] step 400/18464 loss=8.8076 ppl=6684.98 lr=6.000e-04 | 1.07K tok/s | 26.2M done | eta 12d18h10m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 22:02:28 [all] projected finish 13d00h43m exceeds the 168h deadline |
| 2026-09-12 22:07:29 [all] checkpoint @ step 405 (26.5M tokens, 06h38m49s elapsed) |
| 2026-09-12 22:12:22 [all] step 410/18464 loss=8.7954 ppl=6603.55 lr=6.000e-04 | 1.1K tok/s | 26.9M done | eta 12d09h36m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 22:12:22 [all] projected finish 12d16h20m exceeds the 168h deadline |
| 2026-09-12 22:22:03 [all] step 420/18464 loss=8.7610 ppl=6380.40 lr=6.000e-04 | 1.13K tok/s | 27.5M done | eta 12d03h23m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 22:22:03 [all] projected finish 12d10h16m exceeds the 168h deadline |
| 2026-09-12 22:31:39 [all] step 430/18464 loss=8.7692 ppl=6432.79 lr=6.000e-04 | 1.14K tok/s | 28.2M done | eta 12d00h31m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 22:31:39 [all] projected finish 12d07h34m exceeds the 168h deadline |
| 2026-09-12 22:38:19 [all] checkpoint @ step 437 (28.6M tokens, 07h09m39s elapsed) |
| 2026-09-12 22:41:10 [all] step 440/18464 loss=8.7227 ppl=6141.04 lr=6.000e-04 | 1.15K tok/s | 28.8M done | eta 11d21h56m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 22:41:10 [all] projected finish 12d05h08m exceeds the 168h deadline |
| 2026-09-12 22:51:24 [all] step 450/18464 loss=8.7250 ppl=6154.67 lr=6.000e-04 | 1.07K tok/s | 29.5M done | eta 12d19h07m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 22:51:24 [all] projected finish 13d02h30m exceeds the 168h deadline |
| 2026-09-12 23:02:18 [all] step 460/18464 loss=8.7076 ppl=6048.77 lr=6.000e-04 | 1K tok/s | 30.1M done | eta 13d14h47m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 23:02:18 [all] projected finish 13d22h20m exceeds the 168h deadline |
| 2026-09-12 23:08:56 [all] checkpoint @ step 466 (30.5M tokens, 07h40m16s elapsed) |
| 2026-09-12 23:13:29 [all] step 470/18464 loss=8.7062 ppl=6040.37 lr=6.000e-04 | 977 tok/s | 30.8M done | eta 13d23h22m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 23:13:29 [all] projected finish 14d07h07m exceeds the 168h deadline |
| 2026-09-12 23:25:11 [all] step 480/18464 loss=8.6784 ppl=5874.67 lr=6.000e-04 | 932 tok/s | 31.5M done | eta 14d15h08m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 23:25:11 [all] projected finish 14d23h05m exceeds the 168h deadline |
| 2026-09-12 23:37:10 [all] step 490/18464 loss=8.6565 ppl=5747.35 lr=5.999e-04 | 912 tok/s | 32.1M done | eta 14d22h43m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 23:37:10 [all] projected finish 15d06h51m exceeds the 168h deadline |
| 2026-09-12 23:39:36 [all] checkpoint @ step 492 (32.2M tokens, 08h10m56s elapsed) |
| 2026-09-12 23:49:29 [all] step 500/18464 loss=8.6437 ppl=5674.12 lr=5.999e-04 | 887 tok/s | 32.8M done | eta 15d08h50m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-12 23:49:29 [all] projected finish 15d17h10m exceeds the 168h deadline |
| 2026-09-12 23:50:26 [all] eval step 500/18464 loss=8.6352 ppl=5626.24 (best 8.6352) |
| 2026-09-13 00:03:13 [all] step 510/18464 loss=8.6350 ppl=5625.37 lr=5.999e-04 | 795 tok/s | 33.4M done | eta 17d02h59m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 00:03:13 [all] projected finish 17d11h33m exceeds the 168h deadline |
| 2026-09-13 00:09:44 [all] checkpoint @ step 515 (33.8M tokens, 08h41m04s elapsed) |
| 2026-09-13 00:16:58 [all] step 520/18464 loss=8.6321 ppl=5608.61 lr=5.999e-04 | 795 tok/s | 34.1M done | eta 17d02h57m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 00:16:58 [all] projected finish 17d11h45m exceeds the 168h deadline |
| 2026-09-13 00:32:34 [all] step 530/18464 loss=8.6351 ppl=5625.44 lr=5.999e-04 | 700 tok/s | 34.7M done | eta 19d10h23m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 00:32:34 [all] projected finish 19d19h26m exceeds the 168h deadline |
| 2026-09-13 00:40:51 [all] checkpoint @ step 535 (35.1M tokens, 09h12m11s elapsed) |
| 2026-09-13 00:49:33 [all] step 540/18464 loss=8.5965 ppl=5412.90 lr=5.999e-04 | 643 tok/s | 35.4M done | eta 21d03h36m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 00:49:33 [all] projected finish 21d12h57m exceeds the 168h deadline |
| 2026-09-13 01:06:32 [all] step 550/18464 loss=8.5909 ppl=5382.31 lr=5.999e-04 | 643 tok/s | 36M done | eta 21d03h00m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 01:06:32 [all] projected finish 21d12h38m exceeds the 168h deadline |
| 2026-09-13 01:11:30 [all] checkpoint @ step 553 (36.2M tokens, 09h42m50s elapsed) |
| 2026-09-13 01:22:45 [all] step 560/18464 loss=8.5812 ppl=5330.45 lr=5.999e-04 | 673 tok/s | 36.7M done | eta 20d03h57m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 01:22:45 [all] projected finish 20d13h51m exceeds the 168h deadline |
| 2026-09-13 01:38:42 [all] step 570/18464 loss=8.5715 ppl=5278.98 lr=5.998e-04 | 685 tok/s | 37.4M done | eta 19d19h32m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 01:38:42 [all] projected finish 20d05h42m exceeds the 168h deadline |
| 2026-09-13 01:41:47 [all] checkpoint @ step 572 (37.5M tokens, 10h13m07s elapsed) |
| 2026-09-13 01:54:10 [all] step 580/18464 loss=8.5628 ppl=5233.12 lr=5.998e-04 | 706 tok/s | 38M done | eta 19d05h14m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 01:54:10 [all] projected finish 19d15h39m exceeds the 168h deadline |
| 2026-09-13 02:09:35 [all] step 590/18464 loss=8.5380 ppl=5105.18 lr=5.998e-04 | 709 tok/s | 38.7M done | eta 19d03h00m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 02:09:35 [all] projected finish 19d13h41m exceeds the 168h deadline |
| 2026-09-13 02:12:39 [all] checkpoint @ step 592 (38.8M tokens, 10h44m00s elapsed) |
| 2026-09-13 02:24:58 [all] step 600/18464 loss=8.5425 ppl=5127.94 lr=5.998e-04 | 710 tok/s | 39.3M done | eta 19d01h47m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 02:24:58 [all] projected finish 19d12h44m exceeds the 168h deadline |
| 2026-09-13 02:40:07 [all] step 610/18464 loss=8.5205 ppl=5016.77 lr=5.998e-04 | 720 tok/s | 40M done | eta 18d19h12m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 02:40:07 [all] projected finish 19d06h24m exceeds the 168h deadline |
| 2026-09-13 02:43:10 [all] checkpoint @ step 612 (40.1M tokens, 11h14m30s elapsed) |
| 2026-09-13 02:55:23 [all] step 620/18464 loss=8.5169 ppl=4998.37 lr=5.997e-04 | 715 tok/s | 40.6M done | eta 18d22h06m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 02:55:23 [all] projected finish 19d09h32m exceeds the 168h deadline |
| 2026-09-13 03:10:38 [all] step 630/18464 loss=8.5035 ppl=4931.88 lr=5.997e-04 | 717 tok/s | 41.3M done | eta 18d21h01m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 03:10:38 [all] projected finish 19d08h43m exceeds the 168h deadline |
| 2026-09-13 03:13:41 [all] checkpoint @ step 632 (41.4M tokens, 11h45m01s elapsed) |
| 2026-09-13 03:25:57 [all] step 640/18464 loss=8.5054 ppl=4941.52 lr=5.997e-04 | 713 tok/s | 41.9M done | eta 18d23h06m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 03:25:57 [all] projected finish 19d11h04m exceeds the 168h deadline |
| 2026-09-13 03:41:18 [all] step 650/18464 loss=8.4723 ppl=4780.72 lr=5.997e-04 | 712 tok/s | 42.6M done | eta 18d23h34m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 03:41:18 [all] projected finish 19d11h47m exceeds the 168h deadline |
| 2026-09-13 03:44:22 [all] checkpoint @ step 652 (42.7M tokens, 12h15m42s elapsed) |
| 2026-09-13 03:56:43 [all] step 660/18464 loss=8.4710 ppl=4774.13 lr=5.997e-04 | 708 tok/s | 43.3M done | eta 19d01h41m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 03:56:43 [all] projected finish 19d14h10m exceeds the 168h deadline |
| 2026-09-13 04:12:05 [all] step 670/18464 loss=8.4865 ppl=4848.92 lr=5.996e-04 | 711 tok/s | 43.9M done | eta 18d23h22m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 04:12:05 [all] projected finish 19d12h05m exceeds the 168h deadline |
| 2026-09-13 04:15:11 [all] checkpoint @ step 672 (44M tokens, 12h46m31s elapsed) |
| 2026-09-13 04:27:30 [all] step 680/18464 loss=8.4491 ppl=4670.68 lr=5.996e-04 | 708 tok/s | 44.6M done | eta 19d01h14m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 04:27:30 [all] projected finish 19d14h13m exceeds the 168h deadline |
| 2026-09-13 04:42:54 [all] step 690/18464 loss=8.4521 ppl=4684.91 lr=5.996e-04 | 710 tok/s | 45.2M done | eta 18d23h59m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 04:42:54 [all] projected finish 19d13h13m exceeds the 168h deadline |
| 2026-09-13 04:45:57 [all] checkpoint @ step 692 (45.4M tokens, 13h17m17s elapsed) |
| 2026-09-13 04:58:17 [all] step 700/18464 loss=8.4486 ppl=4668.65 lr=5.996e-04 | 710 tok/s | 45.9M done | eta 18d23h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 04:58:17 [all] projected finish 19d13h07m exceeds the 168h deadline |
| 2026-09-13 05:13:44 [all] step 710/18464 loss=8.4394 ppl=4625.64 lr=5.995e-04 | 707 tok/s | 46.5M done | eta 19d01h09m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 05:13:44 [all] projected finish 19d14h55m exceeds the 168h deadline |
| 2026-09-13 05:16:49 [all] checkpoint @ step 712 (46.7M tokens, 13h48m09s elapsed) |
| 2026-09-13 05:29:07 [all] step 720/18464 loss=8.4310 ppl=4586.96 lr=5.995e-04 | 710 tok/s | 47.2M done | eta 18d22h55m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 05:29:07 [all] projected finish 19d12h56m exceeds the 168h deadline |
| 2026-09-13 05:44:34 [all] step 730/18464 loss=8.4124 ppl=4502.54 lr=5.995e-04 | 707 tok/s | 47.8M done | eta 19d00h30m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 05:44:34 [all] projected finish 19d14h46m exceeds the 168h deadline |
| 2026-09-13 05:47:39 [all] checkpoint @ step 732 (48M tokens, 14h18m59s elapsed) |
| 2026-09-13 05:59:56 [all] step 740/18464 loss=8.4059 ppl=4473.47 lr=5.994e-04 | 711 tok/s | 48.5M done | eta 18d21h54m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 05:59:56 [all] projected finish 19d12h25m exceeds the 168h deadline |
| 2026-09-13 06:15:16 [all] step 750/18464 loss=8.3896 ppl=4401.15 lr=5.994e-04 | 712 tok/s | 49.2M done | eta 18d20h53m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 06:15:16 [all] projected finish 19d11h39m exceeds the 168h deadline |
| 2026-09-13 06:18:20 [all] checkpoint @ step 752 (49.3M tokens, 14h49m40s elapsed) |
| 2026-09-13 06:30:33 [all] step 760/18464 loss=8.3892 ppl=4399.49 lr=5.994e-04 | 714 tok/s | 49.8M done | eta 18d19h05m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 06:30:33 [all] projected finish 19d10h07m exceeds the 168h deadline |
| 2026-09-13 06:45:55 [all] step 770/18464 loss=8.3919 ppl=4411.01 lr=5.993e-04 | 712 tok/s | 50.5M done | eta 18d20h42m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 06:45:55 [all] projected finish 19d12h00m exceeds the 168h deadline |
| 2026-09-13 06:48:59 [all] checkpoint @ step 772 (50.6M tokens, 15h20m20s elapsed) |
| 2026-09-13 07:01:12 [all] step 780/18464 loss=8.3763 ppl=4342.76 lr=5.993e-04 | 714 tok/s | 51.1M done | eta 18d18h40m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 07:01:12 [all] projected finish 19d10h12m exceeds the 168h deadline |
| 2026-09-13 07:16:34 [all] step 790/18464 loss=8.3720 ppl=4324.37 lr=5.993e-04 | 711 tok/s | 51.8M done | eta 18d20h50m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 07:16:34 [all] projected finish 19d12h38m exceeds the 168h deadline |
| 2026-09-13 07:19:39 [all] checkpoint @ step 792 (51.9M tokens, 15h50m59s elapsed) |
| 2026-09-13 07:31:51 [all] step 800/18464 loss=8.3675 ppl=4304.74 lr=5.992e-04 | 715 tok/s | 52.4M done | eta 18d17h50m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 07:31:51 [all] projected finish 19d09h53m exceeds the 168h deadline |
| 2026-09-13 07:47:09 [all] step 810/18464 loss=8.3736 ppl=4331.31 lr=5.992e-04 | 714 tok/s | 53.1M done | eta 18d18h15m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 07:47:09 [all] projected finish 19d10h33m exceeds the 168h deadline |
| 2026-09-13 07:50:13 [all] checkpoint @ step 812 (53.2M tokens, 16h21m33s elapsed) |
| 2026-09-13 08:02:29 [all] step 820/18464 loss=8.3476 ppl=4220.22 lr=5.992e-04 | 713 tok/s | 53.7M done | eta 18d18h36m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 08:02:29 [all] projected finish 19d11h10m exceeds the 168h deadline |
| 2026-09-13 08:17:57 [all] step 830/18464 loss=8.3170 ppl=4092.75 lr=5.991e-04 | 706 tok/s | 54.4M done | eta 18d22h56m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 08:17:57 [all] projected finish 19d15h45m exceeds the 168h deadline |
| 2026-09-13 08:21:10 [all] checkpoint @ step 832 (54.5M tokens, 16h52m30s elapsed) |
| 2026-09-13 08:34:00 [all] step 840/18464 loss=8.3378 ppl=4178.93 lr=5.991e-04 | 681 tok/s | 55.1M done | eta 19d15h10m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 08:34:00 [all] projected finish 20d08h16m exceeds the 168h deadline |
| 2026-09-13 08:50:56 [all] step 850/18464 loss=8.3234 ppl=4119.31 lr=5.991e-04 | 645 tok/s | 55.7M done | eta 20d17h20m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 08:50:56 [all] projected finish 21d10h42m exceeds the 168h deadline |
| 2026-09-13 08:52:43 [all] checkpoint @ step 851 (55.8M tokens, 17h24m03s elapsed) |
| 2026-09-13 09:06:36 [all] step 860/18464 loss=8.3208 ppl=4108.52 lr=5.990e-04 | 697 tok/s | 56.4M done | eta 19d03h40m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 09:06:36 [all] projected finish 19d21h18m exceeds the 168h deadline |
| 2026-09-13 09:16:42 [all] step 870/18464 loss=8.3200 ppl=4105.06 lr=5.990e-04 | 1.08K tok/s | 57M done | eta 12d07h50m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 09:16:42 [all] projected finish 13d01h38m exceeds the 168h deadline |
| 2026-09-13 09:23:40 [all] checkpoint @ step 877 (57.5M tokens, 17h55m00s elapsed) |
| 2026-09-13 09:26:37 [all] step 880/18464 loss=8.3048 ppl=4043.18 lr=5.989e-04 | 1.1K tok/s | 57.7M done | eta 12d02h29m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 09:26:37 [all] projected finish 12d20h27m exceeds the 168h deadline |
| 2026-09-13 09:36:31 [all] step 890/18464 loss=8.2937 ppl=3998.43 lr=5.989e-04 | 1.1K tok/s | 58.3M done | eta 12d02h00m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 09:36:31 [all] projected finish 12d20h08m exceeds the 168h deadline |
| 2026-09-13 09:46:26 [all] step 900/18464 loss=8.3100 ppl=4064.22 lr=5.989e-04 | 1.1K tok/s | 59M done | eta 12d02h43m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 09:46:26 [all] projected finish 12d21h01m exceeds the 168h deadline |
| 2026-09-13 09:54:22 [all] checkpoint @ step 908 (59.5M tokens, 18h25m42s elapsed) |
| 2026-09-13 09:56:22 [all] step 910/18464 loss=8.3060 ppl=4048.16 lr=5.988e-04 | 1.1K tok/s | 59.6M done | eta 12d02h24m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 09:56:22 [all] projected finish 12d20h51m exceeds the 168h deadline |
| 2026-09-13 10:06:20 [all] step 920/18464 loss=8.3017 ppl=4030.87 lr=5.988e-04 | 1.1K tok/s | 60.3M done | eta 12d03h14m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 10:06:20 [all] projected finish 12d21h52m exceeds the 168h deadline |
| 2026-09-13 10:16:05 [all] step 930/18464 loss=8.2760 ppl=3928.52 lr=5.987e-04 | 1.12K tok/s | 60.9M done | eta 11d21h14m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 10:16:05 [all] projected finish 12d16h01m exceeds the 168h deadline |
| 2026-09-13 10:24:56 [all] checkpoint @ step 939 (61.5M tokens, 18h56m16s elapsed) |
| 2026-09-13 10:25:54 [all] step 940/18464 loss=8.2754 ppl=3925.96 lr=5.987e-04 | 1.11K tok/s | 61.6M done | eta 11d22h42m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 10:25:54 [all] projected finish 12d17h39m exceeds the 168h deadline |
| 2026-09-13 10:37:13 [all] step 950/18464 loss=8.2683 ppl=3898.28 lr=5.986e-04 | 965 tok/s | 62.3M done | eta 13d18h20m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 10:37:13 [all] projected finish 14d13h28m exceeds the 168h deadline |
| 2026-09-13 10:47:06 [all] step 960/18464 loss=8.2797 ppl=3943.21 lr=5.986e-04 | 1.11K tok/s | 62.9M done | eta 12d00h15m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 10:47:06 [all] projected finish 12d19h33m exceeds the 168h deadline |
| 2026-09-13 10:55:53 [all] checkpoint @ step 969 (63.5M tokens, 19h27m13s elapsed) |
| 2026-09-13 10:56:52 [all] step 970/18464 loss=8.2646 ppl=3884.05 lr=5.985e-04 | 1.12K tok/s | 63.6M done | eta 11d20h48m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 10:56:52 [all] projected finish 12d16h16m exceeds the 168h deadline |
| 2026-09-13 11:06:35 [all] step 980/18464 loss=8.2318 ppl=3758.72 lr=5.985e-04 | 1.12K tok/s | 64.2M done | eta 11d19h03m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 11:06:35 [all] projected finish 12d14h41m exceeds the 168h deadline |
| 2026-09-13 11:16:21 [all] step 990/18464 loss=8.2403 ppl=3790.84 lr=5.984e-04 | 1.12K tok/s | 64.9M done | eta 11d20h21m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 11:16:21 [all] projected finish 12d16h09m exceeds the 168h deadline |
| 2026-09-13 11:26:02 [all] step 1000/18464 loss=8.2361 ppl=3774.65 lr=5.984e-04 | 1.13K tok/s | 65.5M done | eta 11d18h05m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 11:26:02 [all] projected finish 12d14h03m exceeds the 168h deadline |
| 2026-09-13 11:26:48 [all] eval step 1000/18464 loss=8.2387 ppl=3784.69 (best 8.2387) |
| 2026-09-13 11:26:49 [all] checkpoint @ step 1000 (65.5M tokens, 19h58m09s elapsed) |
| 2026-09-13 11:36:33 [all] step 1010/18464 loss=8.2517 ppl=3834.31 lr=5.983e-04 | 1.04K tok/s | 66.2M done | eta 12d17h45m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 11:36:33 [all] projected finish 13d13h53m exceeds the 168h deadline |
| 2026-09-13 11:46:11 [all] step 1020/18464 loss=8.2217 ppl=3720.79 lr=5.983e-04 | 1.13K tok/s | 66.8M done | eta 11d15h57m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 11:46:11 [all] projected finish 12d12h14m exceeds the 168h deadline |
| 2026-09-13 11:55:50 [all] step 1030/18464 loss=8.2297 ppl=3750.70 lr=5.982e-04 | 1.13K tok/s | 67.5M done | eta 11d16h42m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 11:55:50 [all] projected finish 12d13h09m exceeds the 168h deadline |
| 2026-09-13 11:56:50 [all] checkpoint @ step 1031 (67.6M tokens, 20h28m10s elapsed) |
| 2026-09-13 12:05:23 [all] step 1040/18464 loss=8.2255 ppl=3734.86 lr=5.982e-04 | 1.14K tok/s | 68.2M done | eta 11d13h17m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 12:05:23 [all] projected finish 12d09h54m exceeds the 168h deadline |
| 2026-09-13 12:14:59 [all] step 1050/18464 loss=8.2176 ppl=3705.56 lr=5.981e-04 | 1.14K tok/s | 68.8M done | eta 11d14h14m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 12:14:59 [all] projected finish 12d11h00m exceeds the 168h deadline |
| 2026-09-13 12:24:27 [all] step 1060/18464 loss=8.2107 ppl=3680.02 lr=5.981e-04 | 1.15K tok/s | 69.5M done | eta 11d10h41m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 12:24:27 [all] projected finish 12d07h37m exceeds the 168h deadline |
| 2026-09-13 12:27:18 [all] checkpoint @ step 1063 (69.7M tokens, 20h58m39s elapsed) |
| 2026-09-13 12:33:58 [all] step 1070/18464 loss=8.2278 ppl=3743.63 lr=5.980e-04 | 1.15K tok/s | 70.1M done | eta 11d11h45m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 12:33:58 [all] projected finish 12d08h50m exceeds the 168h deadline |
| 2026-09-13 12:43:29 [all] step 1080/18464 loss=8.2037 ppl=3654.53 lr=5.980e-04 | 1.15K tok/s | 70.8M done | eta 11d12h04m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 12:43:29 [all] projected finish 12d09h18m exceeds the 168h deadline |
| 2026-09-13 12:53:00 [all] step 1090/18464 loss=8.2076 ppl=3668.56 lr=5.979e-04 | 1.15K tok/s | 71.4M done | eta 11d11h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 12:53:00 [all] projected finish 12d09h01m exceeds the 168h deadline |
| 2026-09-13 12:57:41 [all] checkpoint @ step 1095 (71.8M tokens, 21h29m01s elapsed) |
| 2026-09-13 13:02:26 [all] step 1100/18464 loss=8.2025 ppl=3650.11 lr=5.978e-04 | 1.16K tok/s | 72.1M done | eta 11d08h56m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 13:02:26 [all] projected finish 12d06h30m exceeds the 168h deadline |
| 2026-09-13 13:11:58 [all] step 1110/18464 loss=8.2032 ppl=3652.66 lr=5.978e-04 | 1.15K tok/s | 72.7M done | eta 11d11h40m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 13:11:58 [all] projected finish 12d09h23m exceeds the 168h deadline |
| 2026-09-13 13:21:33 [all] step 1120/18464 loss=8.1870 ppl=3593.96 lr=5.977e-04 | 1.14K tok/s | 73.4M done | eta 11d12h45m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 13:21:33 [all] projected finish 12d10h38m exceeds the 168h deadline |
| 2026-09-13 13:28:16 [all] checkpoint @ step 1127 (73.9M tokens, 21h59m36s elapsed) |
| 2026-09-13 13:31:07 [all] step 1130/18464 loss=8.1728 ppl=3543.34 lr=5.977e-04 | 1.14K tok/s | 74.1M done | eta 11d12h47m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 13:31:07 [all] projected finish 12d10h49m exceeds the 168h deadline |
| 2026-09-13 13:40:39 [all] step 1140/18464 loss=8.1770 ppl=3558.28 lr=5.976e-04 | 1.15K tok/s | 74.7M done | eta 11d11h04m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 13:40:39 [all] projected finish 12d09h16m exceeds the 168h deadline |
| 2026-09-13 13:50:13 [all] step 1150/18464 loss=8.1785 ppl=3563.60 lr=5.975e-04 | 1.14K tok/s | 75.4M done | eta 11d12h01m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 13:50:13 [all] projected finish 12d10h23m exceeds the 168h deadline |
| 2026-09-13 13:58:50 [all] checkpoint @ step 1159 (76M tokens, 22h30m10s elapsed) |
| 2026-09-13 13:59:47 [all] step 1160/18464 loss=8.1778 ppl=3560.90 lr=5.975e-04 | 1.14K tok/s | 76M done | eta 11d11h56m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 13:59:47 [all] projected finish 12d10h27m exceeds the 168h deadline |
| 2026-09-13 14:09:26 [all] step 1170/18464 loss=8.1606 ppl=3500.15 lr=5.974e-04 | 1.13K tok/s | 76.7M done | eta 11d13h57m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 14:09:26 [all] projected finish 12d12h37m exceeds the 168h deadline |
| 2026-09-13 14:19:07 [all] step 1180/18464 loss=8.1576 ppl=3489.96 lr=5.973e-04 | 1.13K tok/s | 77.3M done | eta 11d15h10m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 14:19:07 [all] projected finish 12d14h00m exceeds the 168h deadline |
| 2026-09-13 14:28:52 [all] step 1190/18464 loss=8.1708 ppl=3536.18 lr=5.973e-04 | 1.12K tok/s | 78M done | eta 11d16h36m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 14:28:52 [all] projected finish 12d15h36m exceeds the 168h deadline |
| 2026-09-13 14:28:52 [all] checkpoint @ step 1190 (78M tokens, 23h00m13s elapsed) |
| 2026-09-13 14:38:34 [all] step 1200/18464 loss=8.1766 ppl=3556.69 lr=5.972e-04 | 1.13K tok/s | 78.6M done | eta 11d14h56m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 14:38:34 [all] projected finish 12d14h05m exceeds the 168h deadline |
| 2026-09-13 14:48:16 [all] step 1210/18464 loss=8.1513 ppl=3467.79 lr=5.971e-04 | 1.13K tok/s | 79.3M done | eta 11d15h11m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 14:48:16 [all] projected finish 12d14h30m exceeds the 168h deadline |
| 2026-09-13 14:58:05 [all] step 1220/18464 loss=8.1471 ppl=3453.22 lr=5.971e-04 | 1.11K tok/s | 80M done | eta 11d17h54m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 14:58:05 [all] projected finish 12d17h23m exceeds the 168h deadline |
| 2026-09-13 14:59:05 [all] checkpoint @ step 1221 (80M tokens, 23h30m25s elapsed) |
| 2026-09-13 15:07:57 [all] step 1230/18464 loss=8.1525 ppl=3472.15 lr=5.970e-04 | 1.11K tok/s | 80.6M done | eta 11d19h40m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 15:07:57 [all] projected finish 12d19h19m exceeds the 168h deadline |
| 2026-09-13 15:17:30 [all] step 1240/18464 loss=8.1548 ppl=3480.03 lr=5.969e-04 | 1.15K tok/s | 81.3M done | eta 11d09h50m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 15:17:30 [all] projected finish 12d09h39m exceeds the 168h deadline |
| 2026-09-13 15:26:49 [all] step 1250/18464 loss=8.1448 ppl=3445.34 lr=5.969e-04 | 1.17K tok/s | 81.9M done | eta 11d03h33m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 15:26:49 [all] projected finish 12d03h31m exceeds the 168h deadline |
| 2026-09-13 15:29:07 [all] checkpoint @ step 1252 (82.1M tokens, 1d00h00m elapsed) |
| 2026-09-13 15:38:11 [all] step 1260/18464 loss=8.1339 ppl=3408.24 lr=5.968e-04 | 961 tok/s | 82.6M done | eta 13d13h47m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 15:38:11 [all] projected finish 14d13h56m exceeds the 168h deadline |
| 2026-09-13 15:47:48 [all] step 1270/18464 loss=8.1467 ppl=3452.10 lr=5.967e-04 | 1.14K tok/s | 83.2M done | eta 11d11h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 15:47:48 [all] projected finish 12d11h56m exceeds the 168h deadline |
| 2026-09-13 15:59:34 [all] step 1280/18464 loss=8.1322 ppl=3402.40 lr=5.966e-04 | 928 tok/s | 83.9M done | eta 14d00h57m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 15:59:34 [all] projected finish 15d01h28m exceeds the 168h deadline |
| 2026-09-13 15:59:34 [all] checkpoint @ step 1280 (83.9M tokens, 1d00h30m elapsed) |
| 2026-09-13 16:08:12 [all] step 1290/18464 loss=8.1321 ppl=3402.09 lr=5.966e-04 | 1.27K tok/s | 84.5M done | eta 10d06h59m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 16:08:12 [all] projected finish 11d07h38m exceeds the 168h deadline |
| 2026-09-13 16:16:32 [all] step 1300/18464 loss=8.1252 ppl=3378.43 lr=5.965e-04 | 1.31K tok/s | 85.2M done | eta 9d22h30m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 16:16:32 [all] projected finish 10d23h18m exceeds the 168h deadline |
| 2026-09-13 16:25:11 [all] step 1310/18464 loss=8.1225 ppl=3369.35 lr=5.964e-04 | 1.26K tok/s | 85.9M done | eta 10d07h15m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 16:25:11 [all] projected finish 11d08h12m exceeds the 168h deadline |
| 2026-09-13 16:30:00 [all] checkpoint @ step 1315 (86.2M tokens, 1d01h01m elapsed) |
| 2026-09-13 16:34:56 [all] step 1320/18464 loss=8.1223 ppl=3368.66 lr=5.963e-04 | 1.12K tok/s | 86.5M done | eta 11d14h31m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 16:34:56 [all] projected finish 12d15h37m exceeds the 168h deadline |
| 2026-09-13 16:44:07 [all] step 1330/18464 loss=8.1161 ppl=3347.91 lr=5.963e-04 | 1.19K tok/s | 87.2M done | eta 10d22h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 16:44:07 [all] projected finish 11d23h53m exceeds the 168h deadline |
| 2026-09-13 16:53:15 [all] step 1340/18464 loss=8.1249 ppl=3377.70 lr=5.962e-04 | 1.2K tok/s | 87.8M done | eta 10d20h16m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 16:53:15 [all] projected finish 11d21h41m exceeds the 168h deadline |
| 2026-09-13 17:00:10 [all] checkpoint @ step 1348 (88.3M tokens, 1d01h31m elapsed) |
| 2026-09-13 17:01:49 [all] step 1350/18464 loss=8.1318 ppl=3400.83 lr=5.961e-04 | 1.27K tok/s | 88.5M done | eta 10d04h38m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 17:01:49 [all] projected finish 11d06h12m exceeds the 168h deadline |
| 2026-09-13 17:10:05 [all] step 1360/18464 loss=8.1428 ppl=3438.62 lr=5.960e-04 | 1.32K tok/s | 89.1M done | eta 9d19h29m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 17:10:05 [all] projected finish 10d21h11m exceeds the 168h deadline |
| 2026-09-13 17:18:37 [all] step 1370/18464 loss=8.1257 ppl=3380.24 lr=5.959e-04 | 1.28K tok/s | 89.8M done | eta 10d03h10m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 17:18:37 [all] projected finish 11d05h00m exceeds the 168h deadline |
| 2026-09-13 17:27:26 [all] step 1380/18464 loss=8.1380 ppl=3422.21 lr=5.959e-04 | 1.24K tok/s | 90.4M done | eta 10d11h00m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 17:27:26 [all] projected finish 11d12h59m exceeds the 168h deadline |
| 2026-09-13 17:31:02 [all] checkpoint @ step 1384 (90.7M tokens, 1d02h02m elapsed) |
| 2026-09-13 17:36:27 [all] step 1390/18464 loss=8.1259 ppl=3381.08 lr=5.958e-04 | 1.21K tok/s | 91.1M done | eta 10d16h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 17:36:27 [all] projected finish 11d18h45m exceeds the 168h deadline |
| 2026-09-13 17:45:25 [all] step 1400/18464 loss=8.1169 ppl=3350.67 lr=5.957e-04 | 1.22K tok/s | 91.8M done | eta 10d14h54m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 17:45:25 [all] projected finish 11d17h10m exceeds the 168h deadline |
| 2026-09-13 17:53:49 [all] step 1410/18464 loss=8.1347 ppl=3410.78 lr=5.956e-04 | 1.3K tok/s | 92.4M done | eta 9d22h47m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 17:53:49 [all] projected finish 11d01h13m exceeds the 168h deadline |
| 2026-09-13 18:01:09 [all] checkpoint @ step 1419 (93M tokens, 1d02h32m elapsed) |
| 2026-09-13 18:01:56 [all] step 1420/18464 loss=8.1377 ppl=3421.02 lr=5.955e-04 | 1.35K tok/s | 93.1M done | eta 9d14h35m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:01:56 [all] projected finish 10d17h08m exceeds the 168h deadline |
| 2026-09-13 18:09:52 [all] step 1430/18464 loss=8.1382 ppl=3422.92 lr=5.954e-04 | 1.38K tok/s | 93.7M done | eta 9d09h05m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:09:52 [all] projected finish 10d11h46m exceeds the 168h deadline |
| 2026-09-13 18:17:39 [all] step 1440/18464 loss=8.1139 ppl=3340.50 lr=5.954e-04 | 1.4K tok/s | 94.4M done | eta 9d05h06m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:17:39 [all] projected finish 10d07h55m exceeds the 168h deadline |
| 2026-09-13 18:25:19 [all] step 1450/18464 loss=8.1350 ppl=3411.92 lr=5.953e-04 | 1.43K tok/s | 95M done | eta 9d01h18m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:25:19 [all] projected finish 10d04h15m exceeds the 168h deadline |
| 2026-09-13 18:31:42 [all] checkpoint @ step 1458 (95.6M tokens, 1d03h03m elapsed) |
| 2026-09-13 18:33:20 [all] step 1460/18464 loss=8.1348 ppl=3411.07 lr=5.952e-04 | 1.36K tok/s | 95.7M done | eta 9d11h09m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:33:20 [all] projected finish 10d14h14m exceeds the 168h deadline |
| 2026-09-13 18:41:40 [all] step 1470/18464 loss=8.1252 ppl=3378.60 lr=5.951e-04 | 1.31K tok/s | 96.3M done | eta 9d19h51m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:41:40 [all] projected finish 10d23h04m exceeds the 168h deadline |
| 2026-09-13 18:50:10 [all] step 1480/18464 loss=8.1425 ppl=3437.41 lr=5.950e-04 | 1.28K tok/s | 97M done | eta 10d00h51m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:50:10 [all] projected finish 11d04h12m exceeds the 168h deadline |
| 2026-09-13 18:58:45 [all] step 1490/18464 loss=8.1168 ppl=3350.13 lr=5.949e-04 | 1.27K tok/s | 97.6M done | eta 10d02h56m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 18:58:45 [all] projected finish 11d06h26m exceeds the 168h deadline |
| 2026-09-13 19:02:15 [all] checkpoint @ step 1494 (97.9M tokens, 1d03h33m elapsed) |
| 2026-09-13 19:07:23 [all] step 1500/18464 loss=8.1188 ppl=3357.10 lr=5.948e-04 | 1.27K tok/s | 98.3M done | eta 10d03h51m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 19:07:23 [all] projected finish 11d07h30m exceeds the 168h deadline |
| 2026-09-13 19:08:04 [all] eval step 1500/18464 loss=8.1247 ppl=3376.74 (best 8.1247) |
| 2026-09-13 19:16:44 [all] step 1510/18464 loss=8.1302 ppl=3395.40 lr=5.947e-04 | 1.17K tok/s | 99M done | eta 11d00h19m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 19:16:44 [all] projected finish 12d04h08m exceeds the 168h deadline |
| 2026-09-13 19:25:28 [all] step 1520/18464 loss=8.1405 ppl=3430.64 lr=5.946e-04 | 1.25K tok/s | 99.6M done | eta 10d06h26m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 19:25:28 [all] projected finish 11d10h23m exceeds the 168h deadline |
| 2026-09-13 19:32:31 [all] checkpoint @ step 1528 (100M tokens, 1d04h03m elapsed) |
| 2026-09-13 19:34:15 [all] step 1530/18464 loss=8.1326 ppl=3403.49 lr=5.945e-04 | 1.24K tok/s | 100M done | eta 10d08h01m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 19:34:15 [all] projected finish 11d12h07m exceeds the 168h deadline |
| 2026-09-13 19:43:03 [all] step 1540/18464 loss=8.1252 ppl=3378.41 lr=5.944e-04 | 1.24K tok/s | 101M done | eta 10d08h20m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 19:43:03 [all] projected finish 11d12h34m exceeds the 168h deadline |
| 2026-09-13 19:51:52 [all] step 1550/18464 loss=8.1176 ppl=3353.10 lr=5.944e-04 | 1.24K tok/s | 102M done | eta 10d08h12m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 19:51:52 [all] projected finish 11d12h35m exceeds the 168h deadline |
| 2026-09-13 20:00:44 [all] step 1560/18464 loss=8.1459 ppl=3449.09 lr=5.943e-04 | 1.23K tok/s | 102M done | eta 10d09h49m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 20:00:44 [all] projected finish 11d14h21m exceeds the 168h deadline |
| 2026-09-13 20:02:32 [all] checkpoint @ step 1562 (102M tokens, 1d04h33m elapsed) |
| 2026-09-13 20:09:34 [all] step 1570/18464 loss=8.1194 ppl=3358.88 lr=5.942e-04 | 1.23K tok/s | 103M done | eta 10d09h02m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 20:09:34 [all] projected finish 11d13h43m exceeds the 168h deadline |
| 2026-09-13 20:18:25 [all] step 1580/18464 loss=8.1445 ppl=3444.41 lr=5.941e-04 | 1.24K tok/s | 104M done | eta 10d08h48m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 20:18:25 [all] projected finish 11d13h37m exceeds the 168h deadline |
| 2026-09-13 20:27:16 [all] step 1590/18464 loss=8.1050 ppl=3310.90 lr=5.940e-04 | 1.23K tok/s | 104M done | eta 10d08h47m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 20:27:16 [all] projected finish 11d13h46m exceeds the 168h deadline |
| 2026-09-13 20:33:25 [all] checkpoint @ step 1597 (105M tokens, 1d05h04m elapsed) |
| 2026-09-13 20:36:04 [all] step 1600/18464 loss=8.1204 ppl=3362.53 lr=5.939e-04 | 1.24K tok/s | 105M done | eta 10d07h34m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 20:36:04 [all] projected finish 11d12h41m exceeds the 168h deadline |
| 2026-09-13 20:44:52 [all] step 1610/18464 loss=8.1249 ppl=3377.39 lr=5.938e-04 | 1.24K tok/s | 106M done | eta 10d07h18m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 20:44:52 [all] projected finish 11d12h34m exceeds the 168h deadline |
| 2026-09-13 20:53:40 [all] step 1620/18464 loss=8.1052 ppl=3311.70 lr=5.937e-04 | 1.24K tok/s | 106M done | eta 10d06h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 20:53:40 [all] projected finish 11d12h02m exceeds the 168h deadline |
| 2026-09-13 21:02:23 [all] step 1630/18464 loss=8.1301 ppl=3394.97 lr=5.936e-04 | 1.25K tok/s | 107M done | eta 10d04h55m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 21:02:23 [all] projected finish 11d10h28m exceeds the 168h deadline |
| 2026-09-13 21:04:10 [all] checkpoint @ step 1632 (107M tokens, 1d05h35m elapsed) |
| 2026-09-13 21:11:08 [all] step 1640/18464 loss=8.1312 ppl=3398.79 lr=5.935e-04 | 1.25K tok/s | 107M done | eta 10d05h03m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 21:11:08 [all] projected finish 11d10h45m exceeds the 168h deadline |
| 2026-09-13 21:19:55 [all] step 1650/18464 loss=8.1248 ppl=3377.31 lr=5.934e-04 | 1.24K tok/s | 108M done | eta 10d06h17m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 21:19:55 [all] projected finish 11d12h08m exceeds the 168h deadline |
| 2026-09-13 21:28:37 [all] step 1660/18464 loss=8.1301 ppl=3395.18 lr=5.933e-04 | 1.25K tok/s | 109M done | eta 10d03h48m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 21:28:37 [all] projected finish 11d09h48m exceeds the 168h deadline |
| 2026-09-13 21:34:47 [all] checkpoint @ step 1667 (109M tokens, 1d06h06m elapsed) |
| 2026-09-13 21:37:22 [all] step 1670/18464 loss=8.1189 ppl=3357.30 lr=5.932e-04 | 1.25K tok/s | 109M done | eta 10d04h46m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 21:37:22 [all] projected finish 11d10h54m exceeds the 168h deadline |
| 2026-09-13 21:46:10 [all] step 1680/18464 loss=8.1178 ppl=3353.78 lr=5.930e-04 | 1.24K tok/s | 110M done | eta 10d06h12m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 21:46:10 [all] projected finish 11d12h30m exceeds the 168h deadline |
| 2026-09-13 21:54:48 [all] step 1690/18464 loss=8.1273 ppl=3385.50 lr=5.929e-04 | 1.26K tok/s | 111M done | eta 10d01h25m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 21:54:48 [all] projected finish 11d07h51m exceeds the 168h deadline |
| 2026-09-13 22:03:31 [all] step 1700/18464 loss=8.1328 ppl=3404.34 lr=5.928e-04 | 1.25K tok/s | 111M done | eta 10d03h30m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 22:03:31 [all] projected finish 11d10h05m exceeds the 168h deadline |
| 2026-09-13 22:05:15 [all] checkpoint @ step 1702 (112M tokens, 1d06h36m elapsed) |
| 2026-09-13 22:12:10 [all] step 1710/18464 loss=8.1284 ppl=3389.49 lr=5.927e-04 | 1.26K tok/s | 112M done | eta 10d01h36m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 22:12:10 [all] projected finish 11d08h20m exceeds the 168h deadline |
| 2026-09-13 22:20:55 [all] step 1720/18464 loss=8.1469 ppl=3452.76 lr=5.926e-04 | 1.25K tok/s | 113M done | eta 10d03h57m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 22:20:55 [all] projected finish 11d10h49m exceeds the 168h deadline |
| 2026-09-13 22:29:53 [all] step 1730/18464 loss=8.1158 ppl=3347.09 lr=5.925e-04 | 1.22K tok/s | 113M done | eta 10d10h00m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 22:29:53 [all] projected finish 11d17h01m exceeds the 168h deadline |
| 2026-09-13 22:35:19 [all] checkpoint @ step 1736 (114M tokens, 1d07h06m elapsed) |
| 2026-09-13 22:38:59 [all] step 1740/18464 loss=8.1222 ppl=3368.32 lr=5.924e-04 | 1.2K tok/s | 114M done | eta 10d13h39m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 22:38:59 [all] projected finish 11d20h50m exceeds the 168h deadline |
| 2026-09-13 22:48:11 [all] step 1750/18464 loss=8.1307 ppl=3397.16 lr=5.923e-04 | 1.19K tok/s | 115M done | eta 10d16h36m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 22:48:11 [all] projected finish 11d23h55m exceeds the 168h deadline |
| 2026-09-13 22:57:31 [all] step 1760/18464 loss=8.1228 ppl=3370.50 lr=5.922e-04 | 1.17K tok/s | 115M done | eta 10d19h33m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 22:57:31 [all] projected finish 12d03h02m exceeds the 168h deadline |
| 2026-09-13 23:05:54 [all] checkpoint @ step 1769 (116M tokens, 1d07h37m elapsed) |
| 2026-09-13 23:06:47 [all] step 1770/18464 loss=8.1245 ppl=3376.14 lr=5.921e-04 | 1.18K tok/s | 116M done | eta 10d17h50m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 23:06:47 [all] projected finish 12d01h28m exceeds the 168h deadline |
| 2026-09-13 23:15:28 [all] step 1780/18464 loss=8.1132 ppl=3338.17 lr=5.920e-04 | 1.26K tok/s | 117M done | eta 10d01h37m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 23:15:28 [all] projected finish 11d09h23m exceeds the 168h deadline |
| 2026-09-13 23:23:44 [all] step 1790/18464 loss=8.1276 ppl=3386.67 lr=5.918e-04 | 1.32K tok/s | 117M done | eta 9d13h36m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 23:23:44 [all] projected finish 10d21h31m exceeds the 168h deadline |
| 2026-09-13 23:31:40 [all] step 1800/18464 loss=8.1171 ppl=3351.37 lr=5.917e-04 | 1.38K tok/s | 118M done | eta 9d04h30m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 23:31:40 [all] projected finish 10d12h33m exceeds the 168h deadline |
| 2026-09-13 23:36:22 [all] checkpoint @ step 1806 (118M tokens, 1d08h07m elapsed) |
| 2026-09-13 23:39:28 [all] step 1810/18464 loss=8.1370 ppl=3418.75 lr=5.916e-04 | 1.4K tok/s | 119M done | eta 9d00h32m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 23:39:28 [all] projected finish 10d08h42m exceeds the 168h deadline |
| 2026-09-13 23:47:04 [all] step 1820/18464 loss=8.1327 ppl=3403.96 lr=5.915e-04 | 1.44K tok/s | 119M done | eta 8d18h26m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 23:47:04 [all] projected finish 10d02h44m exceeds the 168h deadline |
| 2026-09-13 23:54:31 [all] step 1830/18464 loss=8.1399 ppl=3428.44 lr=5.914e-04 | 1.46K tok/s | 120M done | eta 8d14h43m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-13 23:54:31 [all] projected finish 9d23h09m exceeds the 168h deadline |
| 2026-09-14 00:01:49 [all] step 1840/18464 loss=8.1343 ppl=3409.49 lr=5.913e-04 | 1.49K tok/s | 121M done | eta 8d10h29m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:01:49 [all] projected finish 9d19h02m exceeds the 168h deadline |
| 2026-09-14 00:06:54 [all] checkpoint @ step 1847 (121M tokens, 1d08h38m elapsed) |
| 2026-09-14 00:09:02 [all] step 1850/18464 loss=8.1298 ppl=3394.24 lr=5.911e-04 | 1.51K tok/s | 121M done | eta 8d07h48m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:09:02 [all] projected finish 9d16h29m exceeds the 168h deadline |
| 2026-09-14 00:16:09 [all] step 1860/18464 loss=8.1199 ppl=3360.53 lr=5.910e-04 | 1.54K tok/s | 122M done | eta 8d04h33m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:16:09 [all] projected finish 9d13h21m exceeds the 168h deadline |
| 2026-09-14 00:23:10 [all] step 1870/18464 loss=8.1363 ppl=3416.40 lr=5.909e-04 | 1.56K tok/s | 123M done | eta 8d02h06m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:23:10 [all] projected finish 9d11h00m exceeds the 168h deadline |
| 2026-09-14 00:30:06 [all] step 1880/18464 loss=8.1216 ppl=3366.28 lr=5.908e-04 | 1.57K tok/s | 123M done | eta 7d23h51m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:30:06 [all] projected finish 9d08h53m exceeds the 168h deadline |
| 2026-09-14 00:36:58 [all] step 1890/18464 loss=8.1122 ppl=3334.95 lr=5.907e-04 | 1.59K tok/s | 124M done | eta 7d21h44m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:36:58 [all] projected finish 9d06h52m exceeds the 168h deadline |
| 2026-09-14 00:36:59 [all] checkpoint @ step 1890 (124M tokens, 1d09h08m elapsed) |
| 2026-09-14 00:43:49 [all] step 1900/18464 loss=8.1316 ppl=3400.20 lr=5.905e-04 | 1.59K tok/s | 125M done | eta 7d21h08m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:43:49 [all] projected finish 9d06h24m exceeds the 168h deadline |
| 2026-09-14 00:50:33 [all] step 1910/18464 loss=8.1182 ppl=3354.91 lr=5.904e-04 | 1.63K tok/s | 125M done | eta 7d17h24m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:50:33 [all] projected finish 9d02h46m exceeds the 168h deadline |
| 2026-09-14 00:57:14 [all] step 1920/18464 loss=8.1311 ppl=3398.45 lr=5.903e-04 | 1.63K tok/s | 126M done | eta 7d16h25m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 00:57:14 [all] projected finish 9d01h54m exceeds the 168h deadline |
| 2026-09-14 01:03:48 [all] step 1930/18464 loss=8.1258 ppl=3380.71 lr=5.902e-04 | 1.66K tok/s | 126M done | eta 7d13h10m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:03:48 [all] projected finish 8d22h45m exceeds the 168h deadline |
| 2026-09-14 01:07:08 [all] checkpoint @ step 1935 (127M tokens, 1d09h38m elapsed) |
| 2026-09-14 01:10:23 [all] step 1940/18464 loss=8.1129 ppl=3337.35 lr=5.900e-04 | 1.66K tok/s | 127M done | eta 7d13h13m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:10:23 [all] projected finish 8d22h55m exceeds the 168h deadline |
| 2026-09-14 01:16:54 [all] step 1950/18464 loss=8.1233 ppl=3372.07 lr=5.899e-04 | 1.68K tok/s | 128M done | eta 7d11h16m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:16:54 [all] projected finish 8d21h04m exceeds the 168h deadline |
| 2026-09-14 01:23:35 [all] step 1960/18464 loss=8.1067 ppl=3316.53 lr=5.898e-04 | 1.63K tok/s | 128M done | eta 7d15h49m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:23:35 [all] projected finish 9d01h44m exceeds the 168h deadline |
| 2026-09-14 01:30:30 [all] step 1970/18464 loss=8.1179 ppl=3354.10 lr=5.896e-04 | 1.58K tok/s | 129M done | eta 7d22h07m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:30:30 [all] projected finish 9d08h09m exceeds the 168h deadline |
| 2026-09-14 01:37:35 [all] step 1980/18464 loss=8.1177 ppl=3353.27 lr=5.895e-04 | 1.54K tok/s | 130M done | eta 8d02h45m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:37:35 [all] projected finish 9d12h54m exceeds the 168h deadline |
| 2026-09-14 01:37:36 [all] checkpoint @ step 1980 (130M tokens, 1d10h08m elapsed) |
| 2026-09-14 01:44:45 [all] step 1990/18464 loss=8.1374 ppl=3419.93 lr=5.894e-04 | 1.53K tok/s | 130M done | eta 8d04h30m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:44:45 [all] projected finish 9d14h46m exceeds the 168h deadline |
| 2026-09-14 01:51:58 [all] step 2000/18464 loss=8.1094 ppl=3325.59 lr=5.893e-04 | 1.51K tok/s | 131M done | eta 8d06h13m | vram 0.56/5.01GB peak 5.60GB | rung 1 |
| 2026-09-14 01:51:58 [all] projected finish 9d16h36m exceeds the 168h deadline |
| 2026-09-14 01:52:32 [all] eval step 2000/18464 loss=8.1244 ppl=3376.00 (best 8.1244) |
| 2026-09-14 01:52:33 [all] sample @ 2000: ' in the- a a and, that with the the a and and the a a to the the the the the the the the the the the a in the the the the the the the the the the the the the the the the the' |
| 2026-09-14 01:52:34 [all] checkpoint @ step 2000 (131M tokens, 1d10h23m elapsed) |
| 2026-09-14 01:59:50 [all] step 2010/18464 loss=8.1141 ppl=3341.23 lr=5.891e-04 | 1.39K tok/s | 132M done | eta 8d23h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 01:59:50 [all] projected finish 10d10h17m exceeds the 168h deadline |
| 2026-09-14 02:07:09 [all] step 2020/18464 loss=8.1134 ppl=3339.02 lr=5.890e-04 | 1.49K tok/s | 132M done | eta 8d08h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:07:09 [all] projected finish 9d19h12m exceeds the 168h deadline |
| 2026-09-14 02:14:29 [all] step 2030/18464 loss=8.1315 ppl=3400.06 lr=5.889e-04 | 1.49K tok/s | 133M done | eta 8d08h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:14:29 [all] projected finish 9d19h28m exceeds the 168h deadline |
| 2026-09-14 02:21:48 [all] step 2040/18464 loss=8.1242 ppl=3375.14 lr=5.887e-04 | 1.49K tok/s | 134M done | eta 8d08h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:21:48 [all] projected finish 9d19h13m exceeds the 168h deadline |
| 2026-09-14 02:23:16 [all] checkpoint @ step 2042 (134M tokens, 1d10h54m elapsed) |
| 2026-09-14 02:29:10 [all] step 2050/18464 loss=8.1180 ppl=3354.24 lr=5.886e-04 | 1.48K tok/s | 134M done | eta 8d09h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:29:10 [all] projected finish 9d20h30m exceeds the 168h deadline |
| 2026-09-14 02:36:34 [all] step 2060/18464 loss=8.1221 ppl=3368.22 lr=5.885e-04 | 1.48K tok/s | 135M done | eta 8d10h24m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:36:34 [all] projected finish 9d21h32m exceeds the 168h deadline |
| 2026-09-14 02:43:58 [all] step 2070/18464 loss=8.1271 ppl=3384.84 lr=5.883e-04 | 1.48K tok/s | 136M done | eta 8d10h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:43:58 [all] projected finish 9d21h19m exceeds the 168h deadline |
| 2026-09-14 02:51:21 [all] step 2080/18464 loss=8.1156 ppl=3346.20 lr=5.882e-04 | 1.48K tok/s | 136M done | eta 8d09h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:51:21 [all] projected finish 9d21h07m exceeds the 168h deadline |
| 2026-09-14 02:53:35 [all] checkpoint @ step 2083 (137M tokens, 1d11h24m elapsed) |
| 2026-09-14 02:58:45 [all] step 2090/18464 loss=8.1236 ppl=3373.12 lr=5.881e-04 | 1.48K tok/s | 137M done | eta 8d09h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 02:58:45 [all] projected finish 9d21h21m exceeds the 168h deadline |
| 2026-09-14 03:06:08 [all] step 2100/18464 loss=8.1305 ppl=3396.65 lr=5.879e-04 | 1.48K tok/s | 138M done | eta 8d09h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:06:08 [all] projected finish 9d20h49m exceeds the 168h deadline |
| 2026-09-14 03:13:33 [all] step 2110/18464 loss=8.1352 ppl=3412.33 lr=5.878e-04 | 1.47K tok/s | 138M done | eta 8d10h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:13:33 [all] projected finish 9d21h51m exceeds the 168h deadline |
| 2026-09-14 03:20:57 [all] step 2120/18464 loss=8.1308 ppl=3397.53 lr=5.876e-04 | 1.47K tok/s | 139M done | eta 8d09h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:20:57 [all] projected finish 9d21h49m exceeds the 168h deadline |
| 2026-09-14 03:23:56 [all] checkpoint @ step 2124 (139M tokens, 1d11h55m elapsed) |
| 2026-09-14 03:28:22 [all] step 2130/18464 loss=8.1259 ppl=3380.99 lr=5.875e-04 | 1.47K tok/s | 140M done | eta 8d09h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:28:22 [all] projected finish 9d21h41m exceeds the 168h deadline |
| 2026-09-14 03:35:47 [all] step 2140/18464 loss=8.1438 ppl=3441.99 lr=5.874e-04 | 1.47K tok/s | 140M done | eta 8d09h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:35:47 [all] projected finish 9d21h53m exceeds the 168h deadline |
| 2026-09-14 03:43:12 [all] step 2150/18464 loss=8.1307 ppl=3397.25 lr=5.872e-04 | 1.47K tok/s | 141M done | eta 8d09h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:43:12 [all] projected finish 9d22h01m exceeds the 168h deadline |
| 2026-09-14 03:50:35 [all] step 2160/18464 loss=8.1107 ppl=3329.98 lr=5.871e-04 | 1.48K tok/s | 142M done | eta 8d08h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:50:35 [all] projected finish 9d21h03m exceeds the 168h deadline |
| 2026-09-14 03:54:20 [all] checkpoint @ step 2165 (142M tokens, 1d12h25m elapsed) |
| 2026-09-14 03:58:02 [all] step 2170/18464 loss=8.1320 ppl=3401.57 lr=5.869e-04 | 1.47K tok/s | 142M done | eta 8d10h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 03:58:02 [all] projected finish 9d22h37m exceeds the 168h deadline |
| 2026-09-14 04:05:35 [all] step 2180/18464 loss=8.1184 ppl=3355.69 lr=5.868e-04 | 1.45K tok/s | 143M done | eta 8d12h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:05:35 [all] projected finish 10d01h24m exceeds the 168h deadline |
| 2026-09-14 04:12:58 [all] step 2190/18464 loss=8.1135 ppl=3339.34 lr=5.866e-04 | 1.48K tok/s | 144M done | eta 8d08h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:12:58 [all] projected finish 9d20h53m exceeds the 168h deadline |
| 2026-09-14 04:20:22 [all] step 2200/18464 loss=8.1309 ppl=3397.94 lr=5.865e-04 | 1.47K tok/s | 144M done | eta 8d08h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:20:22 [all] projected finish 9d21h44m exceeds the 168h deadline |
| 2026-09-14 04:24:49 [all] checkpoint @ step 2206 (145M tokens, 1d12h56m elapsed) |
| 2026-09-14 04:27:47 [all] step 2210/18464 loss=8.1288 ppl=3390.77 lr=5.863e-04 | 1.47K tok/s | 145M done | eta 8d08h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:27:47 [all] projected finish 9d21h37m exceeds the 168h deadline |
| 2026-09-14 04:35:09 [all] step 2220/18464 loss=8.1399 ppl=3428.62 lr=5.862e-04 | 1.48K tok/s | 145M done | eta 8d07h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:35:09 [all] projected finish 9d20h34m exceeds the 168h deadline |
| 2026-09-14 04:42:30 [all] step 2230/18464 loss=8.1176 ppl=3352.91 lr=5.860e-04 | 1.48K tok/s | 146M done | eta 8d07h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:42:30 [all] projected finish 9d20h25m exceeds the 168h deadline |
| 2026-09-14 04:49:49 [all] step 2240/18464 loss=8.1256 ppl=3379.86 lr=5.859e-04 | 1.49K tok/s | 147M done | eta 8d05h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:49:49 [all] projected finish 9d19h16m exceeds the 168h deadline |
| 2026-09-14 04:54:57 [all] checkpoint @ step 2247 (147M tokens, 1d13h26m elapsed) |
| 2026-09-14 04:57:08 [all] step 2250/18464 loss=8.1085 ppl=3322.63 lr=5.857e-04 | 1.5K tok/s | 147M done | eta 8d05h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 04:57:08 [all] projected finish 9d18h49m exceeds the 168h deadline |
| 2026-09-14 05:04:25 [all] step 2260/18464 loss=8.1277 ppl=3386.90 lr=5.856e-04 | 1.5K tok/s | 148M done | eta 8d04h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:04:25 [all] projected finish 9d18h28m exceeds the 168h deadline |
| 2026-09-14 05:11:40 [all] step 2270/18464 loss=8.1199 ppl=3360.67 lr=5.854e-04 | 1.51K tok/s | 149M done | eta 8d03h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:11:40 [all] projected finish 9d17h26m exceeds the 168h deadline |
| 2026-09-14 05:18:56 [all] step 2280/18464 loss=8.1154 ppl=3345.52 lr=5.853e-04 | 1.51K tok/s | 149M done | eta 8d03h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:18:56 [all] projected finish 9d17h35m exceeds the 168h deadline |
| 2026-09-14 05:25:28 [all] checkpoint @ step 2289 (150M tokens, 1d13h56m elapsed) |
| 2026-09-14 05:26:11 [all] step 2290/18464 loss=8.1241 ppl=3374.90 lr=5.851e-04 | 1.51K tok/s | 150M done | eta 8d03h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:26:11 [all] projected finish 9d17h27m exceeds the 168h deadline |
| 2026-09-14 05:33:25 [all] step 2300/18464 loss=8.1460 ppl=3449.40 lr=5.850e-04 | 1.51K tok/s | 151M done | eta 8d02h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:33:25 [all] projected finish 9d16h50m exceeds the 168h deadline |
| 2026-09-14 05:40:39 [all] step 2310/18464 loss=8.1232 ppl=3371.81 lr=5.848e-04 | 1.51K tok/s | 151M done | eta 8d02h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:40:39 [all] projected finish 9d17h11m exceeds the 168h deadline |
| 2026-09-14 05:47:54 [all] step 2320/18464 loss=8.1403 ppl=3430.08 lr=5.847e-04 | 1.51K tok/s | 152M done | eta 8d03h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:47:54 [all] projected finish 9d17h20m exceeds the 168h deadline |
| 2026-09-14 05:55:06 [all] step 2330/18464 loss=8.1207 ppl=3363.35 lr=5.845e-04 | 1.52K tok/s | 153M done | eta 8d01h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 05:55:06 [all] projected finish 9d15h59m exceeds the 168h deadline |
| 2026-09-14 05:55:50 [all] checkpoint @ step 2331 (153M tokens, 1d14h27m elapsed) |
| 2026-09-14 06:02:18 [all] step 2340/18464 loss=8.1181 ppl=3354.77 lr=5.844e-04 | 1.52K tok/s | 153M done | eta 8d01h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:02:18 [all] projected finish 9d16h08m exceeds the 168h deadline |
| 2026-09-14 06:09:30 [all] step 2350/18464 loss=8.1157 ppl=3346.61 lr=5.842e-04 | 1.52K tok/s | 154M done | eta 8d01h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:09:30 [all] projected finish 9d16h01m exceeds the 168h deadline |
| 2026-09-14 06:16:40 [all] step 2360/18464 loss=8.1330 ppl=3405.02 lr=5.840e-04 | 1.52K tok/s | 155M done | eta 8d00h16m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:16:40 [all] projected finish 9d15h04m exceeds the 168h deadline |
| 2026-09-14 06:23:50 [all] step 2370/18464 loss=8.1385 ppl=3423.62 lr=5.839e-04 | 1.52K tok/s | 155M done | eta 8d00h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:23:50 [all] projected finish 9d15h21m exceeds the 168h deadline |
| 2026-09-14 06:26:00 [all] checkpoint @ step 2373 (156M tokens, 1d14h57m elapsed) |
| 2026-09-14 06:31:03 [all] step 2380/18464 loss=8.1281 ppl=3388.51 lr=5.837e-04 | 1.51K tok/s | 156M done | eta 8d01h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:31:03 [all] projected finish 9d16h34m exceeds the 168h deadline |
| 2026-09-14 06:38:21 [all] step 2390/18464 loss=8.1157 ppl=3346.75 lr=5.836e-04 | 1.5K tok/s | 157M done | eta 8d03h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:38:21 [all] projected finish 9d18h20m exceeds the 168h deadline |
| 2026-09-14 06:45:42 [all] step 2400/18464 loss=8.1349 ppl=3411.39 lr=5.834e-04 | 1.48K tok/s | 157M done | eta 8d05h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:45:42 [all] projected finish 9d20h21m exceeds the 168h deadline |
| 2026-09-14 06:52:58 [all] step 2410/18464 loss=8.1257 ppl=3380.30 lr=5.832e-04 | 1.5K tok/s | 158M done | eta 8d02h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 06:52:58 [all] projected finish 9d17h37m exceeds the 168h deadline |
| 2026-09-14 06:56:32 [all] checkpoint @ step 2415 (158M tokens, 1d15h27m elapsed) |
| 2026-09-14 07:00:05 [all] step 2420/18464 loss=8.1232 ppl=3371.95 lr=5.831e-04 | 1.53K tok/s | 159M done | eta 7d22h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:00:05 [all] projected finish 9d14h07m exceeds the 168h deadline |
| 2026-09-14 07:07:13 [all] step 2430/18464 loss=8.1269 ppl=3384.24 lr=5.829e-04 | 1.53K tok/s | 159M done | eta 7d22h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:07:13 [all] projected finish 9d14h17m exceeds the 168h deadline |
| 2026-09-14 07:14:19 [all] step 2440/18464 loss=8.1313 ppl=3399.25 lr=5.828e-04 | 1.54K tok/s | 160M done | eta 7d21h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:14:19 [all] projected finish 9d13h18m exceeds the 168h deadline |
| 2026-09-14 07:21:22 [all] step 2450/18464 loss=8.1312 ppl=3398.85 lr=5.826e-04 | 1.55K tok/s | 161M done | eta 7d19h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:21:22 [all] projected finish 9d11h48m exceeds the 168h deadline |
| 2026-09-14 07:27:01 [all] checkpoint @ step 2458 (161M tokens, 1d15h58m elapsed) |
| 2026-09-14 07:28:25 [all] step 2460/18464 loss=8.1086 ppl=3322.84 lr=5.824e-04 | 1.55K tok/s | 161M done | eta 7d20h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:28:25 [all] projected finish 9d12h01m exceeds the 168h deadline |
| 2026-09-14 07:35:24 [all] step 2470/18464 loss=8.1157 ppl=3346.63 lr=5.823e-04 | 1.56K tok/s | 162M done | eta 7d18h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:35:24 [all] projected finish 9d10h17m exceeds the 168h deadline |
| 2026-09-14 07:42:24 [all] step 2480/18464 loss=8.1196 ppl=3359.79 lr=5.821e-04 | 1.56K tok/s | 163M done | eta 7d18h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:42:24 [all] projected finish 9d10h46m exceeds the 168h deadline |
| 2026-09-14 07:49:25 [all] step 2490/18464 loss=8.1226 ppl=3369.74 lr=5.819e-04 | 1.56K tok/s | 163M done | eta 7d18h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:49:25 [all] projected finish 9d11h14m exceeds the 168h deadline |
| 2026-09-14 07:56:28 [all] step 2500/18464 loss=8.1240 ppl=3374.32 lr=5.817e-04 | 1.55K tok/s | 164M done | eta 7d19h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 07:56:28 [all] projected finish 9d12h01m exceeds the 168h deadline |
| 2026-09-14 07:57:01 [all] eval step 2500/18464 loss=8.1242 ppl=3375.22 (best 8.1242) |
| 2026-09-14 07:57:02 [all] checkpoint @ step 2500 (164M tokens, 1d16h28m elapsed) |
| 2026-09-14 08:04:02 [all] step 2510/18464 loss=8.1191 ppl=3357.90 lr=5.816e-04 | 1.44K tok/s | 164M done | eta 8d09h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:04:02 [all] projected finish 10d01h48m exceeds the 168h deadline |
| 2026-09-14 08:11:02 [all] step 2520/18464 loss=8.1267 ppl=3383.45 lr=5.814e-04 | 1.56K tok/s | 165M done | eta 7d18h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:11:02 [all] projected finish 9d10h46m exceeds the 168h deadline |
| 2026-09-14 08:18:02 [all] step 2530/18464 loss=8.1176 ppl=3353.06 lr=5.812e-04 | 1.56K tok/s | 166M done | eta 7d17h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:18:02 [all] projected finish 9d10h31m exceeds the 168h deadline |
| 2026-09-14 08:25:02 [all] step 2540/18464 loss=8.1186 ppl=3356.29 lr=5.811e-04 | 1.56K tok/s | 166M done | eta 7d18h00m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:25:02 [all] projected finish 9d10h57m exceeds the 168h deadline |
| 2026-09-14 08:27:10 [all] checkpoint @ step 2543 (167M tokens, 1d16h58m elapsed) |
| 2026-09-14 08:32:09 [all] step 2550/18464 loss=8.1258 ppl=3380.58 lr=5.809e-04 | 1.53K tok/s | 167M done | eta 7d20h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:32:09 [all] projected finish 9d13h50m exceeds the 168h deadline |
| 2026-09-14 08:39:22 [all] step 2560/18464 loss=8.1122 ppl=3334.86 lr=5.807e-04 | 1.52K tok/s | 168M done | eta 7d22h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:39:22 [all] projected finish 9d16h06m exceeds the 168h deadline |
| 2026-09-14 08:46:39 [all] step 2570/18464 loss=8.1206 ppl=3362.99 lr=5.805e-04 | 1.5K tok/s | 168M done | eta 8d01h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:46:39 [all] projected finish 9d18h24m exceeds the 168h deadline |
| 2026-09-14 08:54:02 [all] step 2580/18464 loss=8.1168 ppl=3350.30 lr=5.804e-04 | 1.48K tok/s | 169M done | eta 8d03h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 08:54:02 [all] projected finish 9d20h51m exceeds the 168h deadline |
| 2026-09-14 08:57:48 [all] checkpoint @ step 2585 (169M tokens, 1d17h29m elapsed) |
| 2026-09-14 09:01:34 [all] step 2590/18464 loss=8.1302 ppl=3395.48 lr=5.802e-04 | 1.45K tok/s | 170M done | eta 8d07h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:01:34 [all] projected finish 10d00h48m exceeds the 168h deadline |
| 2026-09-14 09:09:12 [all] step 2600/18464 loss=8.0976 ppl=3286.46 lr=5.800e-04 | 1.43K tok/s | 170M done | eta 8d10h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:09:12 [all] projected finish 10d03h43m exceeds the 168h deadline |
| 2026-09-14 09:16:59 [all] step 2610/18464 loss=8.1140 ppl=3341.02 lr=5.798e-04 | 1.41K tok/s | 171M done | eta 8d13h23m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:16:59 [all] projected finish 10d07h12m exceeds the 168h deadline |
| 2026-09-14 09:24:52 [all] step 2620/18464 loss=8.1270 ppl=3384.78 lr=5.797e-04 | 1.38K tok/s | 172M done | eta 8d16h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:24:52 [all] projected finish 10d10h16m exceeds the 168h deadline |
| 2026-09-14 09:28:03 [all] checkpoint @ step 2624 (172M tokens, 1d17h59m elapsed) |
| 2026-09-14 09:32:51 [all] step 2630/18464 loss=8.1230 ppl=3370.99 lr=5.795e-04 | 1.37K tok/s | 172M done | eta 8d18h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:32:51 [all] projected finish 10d12h41m exceeds the 168h deadline |
| 2026-09-14 09:40:58 [all] step 2640/18464 loss=8.1227 ppl=3370.03 lr=5.793e-04 | 1.35K tok/s | 173M done | eta 8d22h05m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:40:58 [all] projected finish 10d16h17m exceeds the 168h deadline |
| 2026-09-14 09:49:09 [all] step 2650/18464 loss=8.1146 ppl=3342.78 lr=5.791e-04 | 1.34K tok/s | 174M done | eta 8d23h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:49:09 [all] projected finish 10d17h56m exceeds the 168h deadline |
| 2026-09-14 09:57:22 [all] step 2660/18464 loss=8.1392 ppl=3426.33 lr=5.789e-04 | 1.33K tok/s | 174M done | eta 9d00h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 09:57:22 [all] projected finish 10d19h07m exceeds the 168h deadline |
| 2026-09-14 09:58:13 [all] checkpoint @ step 2661 (174M tokens, 1d18h29m elapsed) |
| 2026-09-14 10:05:41 [all] step 2670/18464 loss=8.1304 ppl=3396.28 lr=5.788e-04 | 1.31K tok/s | 175M done | eta 9d02h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 10:05:41 [all] projected finish 10d21h24m exceeds the 168h deadline |
| 2026-09-14 10:14:06 [all] step 2680/18464 loss=8.1184 ppl=3355.70 lr=5.786e-04 | 1.3K tok/s | 176M done | eta 9d05h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 10:14:06 [all] projected finish 11d00h15m exceeds the 168h deadline |
| 2026-09-14 10:22:33 [all] step 2690/18464 loss=8.1358 ppl=3414.67 lr=5.784e-04 | 1.29K tok/s | 176M done | eta 9d06h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 10:22:33 [all] projected finish 11d00h55m exceeds the 168h deadline |
| 2026-09-14 10:28:30 [all] checkpoint @ step 2697 (177M tokens, 1d18h59m elapsed) |
| 2026-09-14 10:31:03 [all] step 2700/18464 loss=8.1242 ppl=3375.26 lr=5.782e-04 | 1.28K tok/s | 177M done | eta 9d07h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 10:31:03 [all] projected finish 11d02h23m exceeds the 168h deadline |
| 2026-09-14 10:39:38 [all] step 2710/18464 loss=8.1138 ppl=3340.26 lr=5.780e-04 | 1.27K tok/s | 178M done | eta 9d09h31m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 10:39:38 [all] projected finish 11d04h42m exceeds the 168h deadline |
| 2026-09-14 10:48:15 [all] step 2720/18464 loss=8.1367 ppl=3417.78 lr=5.778e-04 | 1.27K tok/s | 178M done | eta 9d10h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 10:48:15 [all] projected finish 11d05h27m exceeds the 168h deadline |
| 2026-09-14 10:56:54 [all] step 2730/18464 loss=8.1116 ppl=3332.83 lr=5.777e-04 | 1.26K tok/s | 179M done | eta 9d10h43m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 10:56:54 [all] projected finish 11d06h12m exceeds the 168h deadline |
| 2026-09-14 10:58:38 [all] checkpoint @ step 2732 (179M tokens, 1d19h29m elapsed) |
| 2026-09-14 11:05:31 [all] step 2740/18464 loss=8.1419 ppl=3435.58 lr=5.775e-04 | 1.27K tok/s | 180M done | eta 9d09h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 11:05:31 [all] projected finish 11d05h29m exceeds the 168h deadline |
| 2026-09-14 11:14:13 [all] step 2750/18464 loss=8.1283 ppl=3389.14 lr=5.773e-04 | 1.26K tok/s | 180M done | eta 9d11h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 11:14:13 [all] projected finish 11d07h16m exceeds the 168h deadline |
| 2026-09-14 11:22:54 [all] step 2760/18464 loss=8.1280 ppl=3388.05 lr=5.771e-04 | 1.26K tok/s | 181M done | eta 9d11h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 11:22:54 [all] projected finish 11d07h13m exceeds the 168h deadline |
| 2026-09-14 11:29:01 [all] checkpoint @ step 2767 (181M tokens, 1d20h00m elapsed) |
| 2026-09-14 11:31:38 [all] step 2770/18464 loss=8.1105 ppl=3329.21 lr=5.769e-04 | 1.25K tok/s | 182M done | eta 9d12h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 11:31:38 [all] projected finish 11d08h37m exceeds the 168h deadline |
| 2026-09-14 11:40:22 [all] step 2780/18464 loss=8.1218 ppl=3367.21 lr=5.767e-04 | 1.25K tok/s | 182M done | eta 9d12h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 11:40:22 [all] projected finish 11d08h39m exceeds the 168h deadline |
| 2026-09-14 11:49:10 [all] step 2790/18464 loss=8.1124 ppl=3335.41 lr=5.765e-04 | 1.24K tok/s | 183M done | eta 9d13h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 11:49:10 [all] projected finish 11d09h51m exceeds the 168h deadline |
| 2026-09-14 11:57:54 [all] step 2800/18464 loss=8.1246 ppl=3376.67 lr=5.763e-04 | 1.25K tok/s | 184M done | eta 9d12h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 11:57:54 [all] projected finish 11d08h43m exceeds the 168h deadline |
| 2026-09-14 11:59:38 [all] checkpoint @ step 2802 (184M tokens, 1d20h30m elapsed) |
| 2026-09-14 12:06:34 [all] step 2810/18464 loss=8.1078 ppl=3320.29 lr=5.761e-04 | 1.26K tok/s | 184M done | eta 9d09h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 12:06:34 [all] projected finish 11d06h31m exceeds the 168h deadline |
| 2026-09-14 12:15:14 [all] step 2820/18464 loss=8.1175 ppl=3352.48 lr=5.759e-04 | 1.26K tok/s | 185M done | eta 9d10h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 12:15:14 [all] projected finish 11d07h04m exceeds the 168h deadline |
| 2026-09-14 12:23:54 [all] step 2830/18464 loss=8.1169 ppl=3350.77 lr=5.757e-04 | 1.26K tok/s | 185M done | eta 9d09h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 12:23:54 [all] projected finish 11d06h26m exceeds the 168h deadline |
| 2026-09-14 12:29:59 [all] checkpoint @ step 2837 (186M tokens, 1d21h01m elapsed) |
| 2026-09-14 12:32:36 [all] step 2840/18464 loss=8.1193 ppl=3358.50 lr=5.756e-04 | 1.25K tok/s | 186M done | eta 9d10h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 12:32:36 [all] projected finish 11d07h54m exceeds the 168h deadline |
| 2026-09-14 12:41:21 [all] step 2850/18464 loss=8.1190 ppl=3357.59 lr=5.754e-04 | 1.25K tok/s | 187M done | eta 9d11h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 12:41:21 [all] projected finish 11d08h54m exceeds the 168h deadline |
| 2026-09-14 12:50:10 [all] step 2860/18464 loss=8.1390 ppl=3425.57 lr=5.752e-04 | 1.24K tok/s | 187M done | eta 9d13h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 12:50:10 [all] projected finish 11d10h30m exceeds the 168h deadline |
| 2026-09-14 12:58:58 [all] step 2870/18464 loss=8.1236 ppl=3373.04 lr=5.750e-04 | 1.24K tok/s | 188M done | eta 9d12h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 12:58:58 [all] projected finish 11d10h21m exceeds the 168h deadline |
| 2026-09-14 13:00:45 [all] checkpoint @ step 2872 (188M tokens, 1d21h32m elapsed) |
| 2026-09-14 13:07:49 [all] step 2880/18464 loss=8.1167 ppl=3349.88 lr=5.748e-04 | 1.24K tok/s | 189M done | eta 9d13h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 13:07:49 [all] projected finish 11d11h20m exceeds the 168h deadline |
| 2026-09-14 13:16:37 [all] step 2890/18464 loss=8.1279 ppl=3387.66 lr=5.746e-04 | 1.24K tok/s | 189M done | eta 9d12h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 13:16:37 [all] projected finish 11d10h27m exceeds the 168h deadline |
| 2026-09-14 13:25:31 [all] step 2900/18464 loss=8.1283 ppl=3389.05 lr=5.744e-04 | 1.23K tok/s | 190M done | eta 9d14h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 13:25:31 [all] projected finish 11d12h26m exceeds the 168h deadline |
| 2026-09-14 13:30:53 [all] checkpoint @ step 2906 (190M tokens, 1d22h02m elapsed) |
| 2026-09-14 13:34:25 [all] step 2910/18464 loss=8.1322 ppl=3402.19 lr=5.742e-04 | 1.23K tok/s | 191M done | eta 9d14h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 13:34:25 [all] projected finish 11d13h02m exceeds the 168h deadline |
| 2026-09-14 13:43:20 [all] step 2920/18464 loss=8.1337 ppl=3407.38 lr=5.740e-04 | 1.23K tok/s | 191M done | eta 9d14h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 13:43:20 [all] projected finish 11d13h06m exceeds the 168h deadline |
| 2026-09-14 13:52:19 [all] step 2930/18464 loss=8.1135 ppl=3339.10 lr=5.738e-04 | 1.21K tok/s | 192M done | eta 9d16h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 13:52:19 [all] projected finish 11d15h08m exceeds the 168h deadline |
| 2026-09-14 14:01:18 [all] step 2940/18464 loss=8.1239 ppl=3374.14 lr=5.736e-04 | 1.22K tok/s | 193M done | eta 9d16h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 14:01:18 [all] projected finish 11d14h59m exceeds the 168h deadline |
| 2026-09-14 14:01:19 [all] checkpoint @ step 2940 (193M tokens, 1d22h32m elapsed) |
| 2026-09-14 14:10:22 [all] step 2950/18464 loss=8.1255 ppl=3379.70 lr=5.734e-04 | 1.21K tok/s | 193M done | eta 9d18h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 14:10:22 [all] projected finish 11d16h59m exceeds the 168h deadline |
| 2026-09-14 14:19:31 [all] step 2960/18464 loss=8.1325 ppl=3403.14 lr=5.732e-04 | 1.19K tok/s | 194M done | eta 9d20h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 14:19:31 [all] projected finish 11d19h12m exceeds the 168h deadline |
| 2026-09-14 14:28:42 [all] step 2970/18464 loss=8.1296 ppl=3393.35 lr=5.730e-04 | 1.19K tok/s | 195M done | eta 9d21h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 14:28:42 [all] projected finish 11d20h14m exceeds the 168h deadline |
| 2026-09-14 14:31:30 [all] checkpoint @ step 2973 (195M tokens, 1d23h02m elapsed) |
| 2026-09-14 14:37:59 [all] step 2980/18464 loss=8.1273 ppl=3385.63 lr=5.728e-04 | 1.18K tok/s | 195M done | eta 9d23h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 14:37:59 [all] projected finish 11d22h41m exceeds the 168h deadline |
| 2026-09-14 14:47:21 [all] step 2990/18464 loss=8.1151 ppl=3344.50 lr=5.725e-04 | 1.17K tok/s | 196M done | eta 10d01h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 14:47:21 [all] projected finish 12d00h59m exceeds the 168h deadline |
| 2026-09-14 14:56:47 [all] step 3000/18464 loss=8.1198 ppl=3360.23 lr=5.723e-04 | 1.16K tok/s | 197M done | eta 10d03h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 14:56:47 [all] projected finish 12d02h37m exceeds the 168h deadline |
| 2026-09-14 14:57:32 [all] eval step 3000/18464 loss=8.1240 ppl=3374.47 (best 8.1240) |
| 2026-09-14 14:57:32 [all] checkpoint @ step 3000 (197M tokens, 1d23h28m elapsed) |
| 2026-09-14 15:07:06 [all] step 3010/18464 loss=8.1169 ppl=3350.77 lr=5.721e-04 | 1.06K tok/s | 197M done | eta 11d01h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 15:07:06 [all] projected finish 13d01h07m exceeds the 168h deadline |
| 2026-09-14 15:16:41 [all] step 3020/18464 loss=8.1304 ppl=3396.19 lr=5.719e-04 | 1.14K tok/s | 198M done | eta 10d06h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 15:16:41 [all] projected finish 12d06h40m exceeds the 168h deadline |
| 2026-09-14 15:26:22 [all] step 3030/18464 loss=8.1322 ppl=3402.45 lr=5.717e-04 | 1.13K tok/s | 199M done | eta 10d08h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 15:26:22 [all] projected finish 12d08h53m exceeds the 168h deadline |
| 2026-09-14 15:28:19 [all] checkpoint @ step 3032 (199M tokens, 1d23h59m elapsed) |
| 2026-09-14 15:36:10 [all] step 3040/18464 loss=8.1255 ppl=3379.51 lr=5.715e-04 | 1.11K tok/s | 199M done | eta 10d12h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 15:36:10 [all] projected finish 12d12h18m exceeds the 168h deadline |
| 2026-09-14 15:45:54 [all] step 3050/18464 loss=8.1178 ppl=3353.50 lr=5.713e-04 | 1.12K tok/s | 200M done | eta 10d09h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 15:45:54 [all] projected finish 12d10h02m exceeds the 168h deadline |
| 2026-09-14 15:55:44 [all] step 3060/18464 loss=8.1333 ppl=3405.87 lr=5.711e-04 | 1.11K tok/s | 201M done | eta 10d12h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 15:55:44 [all] projected finish 12d13h05m exceeds the 168h deadline |
| 2026-09-14 15:58:41 [all] checkpoint @ step 3063 (201M tokens, 2d00h30m elapsed) |
| 2026-09-14 16:05:21 [all] step 3070/18464 loss=8.1227 ppl=3369.96 lr=5.709e-04 | 1.14K tok/s | 201M done | eta 10d06h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 16:05:21 [all] projected finish 12d07h10m exceeds the 168h deadline |
| 2026-09-14 16:14:21 [all] step 3080/18464 loss=8.1234 ppl=3372.36 lr=5.707e-04 | 1.21K tok/s | 202M done | eta 9d14h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 16:14:21 [all] projected finish 11d15h42m exceeds the 168h deadline |
| 2026-09-14 16:23:07 [all] step 3090/18464 loss=8.1300 ppl=3394.79 lr=5.704e-04 | 1.25K tok/s | 203M done | eta 9d08h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 16:23:07 [all] projected finish 11d09h34m exceeds the 168h deadline |
| 2026-09-14 16:29:27 [all] checkpoint @ step 3097 (203M tokens, 2d01h00m elapsed) |
| 2026-09-14 16:32:05 [all] step 3100/18464 loss=8.1361 ppl=3415.73 lr=5.702e-04 | 1.22K tok/s | 203M done | eta 9d13h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 16:32:05 [all] projected finish 11d14h42m exceeds the 168h deadline |
| 2026-09-14 16:40:51 [all] step 3110/18464 loss=8.1276 ppl=3386.53 lr=5.700e-04 | 1.25K tok/s | 204M done | eta 9d08h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 16:40:51 [all] projected finish 11d09h22m exceeds the 168h deadline |
| 2026-09-14 16:49:07 [all] step 3120/18464 loss=8.1238 ppl=3373.68 lr=5.698e-04 | 1.32K tok/s | 204M done | eta 8d19h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 16:49:07 [all] projected finish 10d20h56m exceeds the 168h deadline |
| 2026-09-14 16:57:23 [all] step 3130/18464 loss=8.1290 ppl=3391.31 lr=5.696e-04 | 1.32K tok/s | 205M done | eta 8d19h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 16:57:23 [all] projected finish 10d20h33m exceeds the 168h deadline |
| 2026-09-14 16:59:52 [all] checkpoint @ step 3133 (205M tokens, 2d01h31m elapsed) |
| 2026-09-14 17:05:52 [all] step 3140/18464 loss=8.1346 ppl=3410.29 lr=5.694e-04 | 1.29K tok/s | 206M done | eta 9d00h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 17:05:52 [all] projected finish 11d02h28m exceeds the 168h deadline |
| 2026-09-14 17:14:01 [all] step 3150/18464 loss=8.1336 ppl=3407.04 lr=5.692e-04 | 1.34K tok/s | 206M done | eta 8d15h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 17:14:01 [all] projected finish 10d17h41m exceeds the 168h deadline |
| 2026-09-14 17:22:10 [all] step 3160/18464 loss=8.1169 ppl=3350.52 lr=5.689e-04 | 1.34K tok/s | 207M done | eta 8d15h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 17:22:10 [all] projected finish 10d17h40m exceeds the 168h deadline |
| 2026-09-14 17:29:58 [all] checkpoint @ step 3168 (208M tokens, 2d02h01m elapsed) |
| 2026-09-14 17:31:44 [all] step 3170/18464 loss=8.1087 ppl=3323.36 lr=5.687e-04 | 1.14K tok/s | 208M done | eta 10d03h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 17:31:44 [all] projected finish 12d05h44m exceeds the 168h deadline |
| 2026-09-14 17:40:33 [all] step 3180/18464 loss=8.1294 ppl=3392.78 lr=5.685e-04 | 1.24K tok/s | 208M done | eta 9d08h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 17:40:33 [all] projected finish 11d11h05m exceeds the 168h deadline |
| 2026-09-14 17:49:04 [all] step 3190/18464 loss=8.1379 ppl=3421.87 lr=5.683e-04 | 1.28K tok/s | 209M done | eta 9d00h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 17:49:04 [all] projected finish 11d03h00m exceeds the 168h deadline |
| 2026-09-14 17:57:30 [all] step 3200/18464 loss=8.1238 ppl=3373.86 lr=5.681e-04 | 1.3K tok/s | 210M done | eta 8d22h31m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 17:57:30 [all] projected finish 11d01h00m exceeds the 168h deadline |
| 2026-09-14 18:00:05 [all] checkpoint @ step 3203 (210M tokens, 2d02h31m elapsed) |
| 2026-09-14 18:06:10 [all] step 3210/18464 loss=8.1153 ppl=3345.31 lr=5.678e-04 | 1.26K tok/s | 210M done | eta 9d04h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 18:06:10 [all] projected finish 11d07h13m exceeds the 168h deadline |
| 2026-09-14 18:15:04 [all] step 3220/18464 loss=8.1398 ppl=3428.20 lr=5.676e-04 | 1.23K tok/s | 211M done | eta 9d09h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 18:15:04 [all] projected finish 11d12h44m exceeds the 168h deadline |
| 2026-09-14 18:23:51 [all] step 3230/18464 loss=8.1331 ppl=3405.42 lr=5.674e-04 | 1.24K tok/s | 212M done | eta 9d06h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 18:23:51 [all] projected finish 11d09h40m exceeds the 168h deadline |
| 2026-09-14 18:30:30 [all] checkpoint @ step 3238 (212M tokens, 2d03h01m elapsed) |
| 2026-09-14 18:32:08 [all] step 3240/18464 loss=8.1171 ppl=3351.12 lr=5.672e-04 | 1.32K tok/s | 212M done | eta 8d18h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 18:32:08 [all] projected finish 10d21h22m exceeds the 168h deadline |
| 2026-09-14 18:40:07 [all] step 3250/18464 loss=8.1021 ppl=3301.47 lr=5.669e-04 | 1.37K tok/s | 213M done | eta 8d10h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 18:40:07 [all] projected finish 10d13h47m exceeds the 168h deadline |
| 2026-09-14 18:47:58 [all] step 3260/18464 loss=8.1113 ppl=3331.75 lr=5.667e-04 | 1.39K tok/s | 214M done | eta 8d06h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 18:47:58 [all] projected finish 10d10h09m exceeds the 168h deadline |
| 2026-09-14 18:55:42 [all] step 3270/18464 loss=8.1202 ppl=3361.68 lr=5.665e-04 | 1.41K tok/s | 214M done | eta 8d03h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 18:55:42 [all] projected finish 10d07h02m exceeds the 168h deadline |
| 2026-09-14 19:01:01 [all] checkpoint @ step 3277 (215M tokens, 2d03h32m elapsed) |
| 2026-09-14 19:03:19 [all] step 3280/18464 loss=8.1197 ppl=3360.16 lr=5.663e-04 | 1.43K tok/s | 215M done | eta 8d00h58m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 19:03:19 [all] projected finish 10d04h32m exceeds the 168h deadline |
| 2026-09-14 19:10:50 [all] step 3290/18464 loss=8.1106 ppl=3329.49 lr=5.660e-04 | 1.45K tok/s | 216M done | eta 7d22h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 19:10:50 [all] projected finish 10d01h44m exceeds the 168h deadline |
| 2026-09-14 19:18:19 [all] step 3300/18464 loss=8.1135 ppl=3339.34 lr=5.658e-04 | 1.46K tok/s | 216M done | eta 7d21h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 19:18:19 [all] projected finish 10d01h04m exceeds the 168h deadline |
| 2026-09-14 19:26:09 [all] step 3310/18464 loss=8.1314 ppl=3399.60 lr=5.656e-04 | 1.39K tok/s | 217M done | eta 8d05h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 19:26:09 [all] projected finish 10d09h48m exceeds the 168h deadline |
| 2026-09-14 19:31:07 [all] checkpoint @ step 3316 (217M tokens, 2d04h02m elapsed) |
| 2026-09-14 19:35:07 [all] step 3320/18464 loss=8.1196 ppl=3359.71 lr=5.654e-04 | 1.22K tok/s | 218M done | eta 9d10h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 19:35:07 [all] projected finish 11d14h18m exceeds the 168h deadline |
| 2026-09-14 19:45:49 [all] step 3330/18464 loss=8.1233 ppl=3372.26 lr=5.651e-04 | 1.02K tok/s | 218M done | eta 11d06h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 19:45:49 [all] projected finish 13d10h21m exceeds the 168h deadline |
| 2026-09-14 19:58:51 [all] step 3340/18464 loss=8.1296 ppl=3393.38 lr=5.649e-04 | 839 tok/s | 219M done | eta 13d16h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 19:58:51 [all] projected finish 15d20h50m exceeds the 168h deadline |
| 2026-09-14 20:01:23 [all] checkpoint @ step 3342 (219M tokens, 2d04h32m elapsed) |
| 2026-09-14 20:12:24 [all] step 3350/18464 loss=8.1172 ppl=3351.57 lr=5.647e-04 | 806 tok/s | 220M done | eta 14d05h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 20:12:24 [all] projected finish 16d10h03m exceeds the 168h deadline |
| 2026-09-14 20:29:54 [all] step 3360/18464 loss=8.1373 ppl=3419.67 lr=5.644e-04 | 624 tok/s | 220M done | eta 18d08h22m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 20:29:54 [all] projected finish 20d13h23m exceeds the 168h deadline |
| 2026-09-14 20:31:43 [all] checkpoint @ step 3361 (220M tokens, 2d05h03m elapsed) |
| 2026-09-14 20:48:25 [all] step 3370/18464 loss=8.1233 ppl=3372.15 lr=5.642e-04 | 590 tok/s | 221M done | eta 19d10h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 20:48:25 [all] projected finish 21d15h26m exceeds the 168h deadline |
| 2026-09-14 21:03:00 [all] step 3380/18464 loss=8.1169 ppl=3350.76 lr=5.640e-04 | 749 tok/s | 222M done | eta 15d06h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 21:03:00 [all] projected finish 17d12h07m exceeds the 168h deadline |
| 2026-09-14 21:03:00 [all] checkpoint @ step 3380 (222M tokens, 2d05h34m elapsed) |
| 2026-09-14 21:16:20 [all] step 3390/18464 loss=8.1157 ppl=3346.52 lr=5.637e-04 | 819 tok/s | 222M done | eta 13d23h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 21:16:20 [all] projected finish 16d04h49m exceeds the 168h deadline |
| 2026-09-14 21:29:32 [all] step 3400/18464 loss=8.1545 ppl=3478.96 lr=5.635e-04 | 828 tok/s | 223M done | eta 13d19h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 21:29:32 [all] projected finish 16d01h16m exceeds the 168h deadline |
| 2026-09-14 21:33:31 [all] checkpoint @ step 3403 (223M tokens, 2d06h04m elapsed) |
| 2026-09-14 21:42:48 [all] step 3410/18464 loss=8.1359 ppl=3415.05 lr=5.633e-04 | 823 tok/s | 223M done | eta 13d20h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 21:42:48 [all] projected finish 16d03h11m exceeds the 168h deadline |
| 2026-09-14 21:57:11 [all] step 3420/18464 loss=8.1138 ppl=3340.35 lr=5.630e-04 | 759 tok/s | 224M done | eta 15d00h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 21:57:11 [all] projected finish 17d07h10m exceeds the 168h deadline |
| 2026-09-14 22:04:54 [all] checkpoint @ step 3425 (224M tokens, 2d06h36m elapsed) |
| 2026-09-14 22:12:41 [all] step 3430/18464 loss=8.1149 ppl=3343.81 lr=5.628e-04 | 705 tok/s | 225M done | eta 16d04h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 22:12:41 [all] projected finish 18d10h51m exceeds the 168h deadline |
| 2026-09-14 22:28:05 [all] step 3440/18464 loss=8.1106 ppl=3329.57 lr=5.625e-04 | 709 tok/s | 225M done | eta 16d01h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 22:28:05 [all] projected finish 18d08h34m exceeds the 168h deadline |
| 2026-09-14 22:35:44 [all] checkpoint @ step 3445 (226M tokens, 2d07h07m elapsed) |
| 2026-09-14 22:43:15 [all] step 3450/18464 loss=8.1225 ppl=3369.48 lr=5.623e-04 | 720 tok/s | 226M done | eta 15d19h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 22:43:15 [all] projected finish 18d02h56m exceeds the 168h deadline |
| 2026-09-14 22:58:13 [all] step 3460/18464 loss=8.1151 ppl=3344.62 lr=5.621e-04 | 730 tok/s | 227M done | eta 15d14h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 22:58:13 [all] projected finish 17d21h51m exceeds the 168h deadline |
| 2026-09-14 23:05:46 [all] checkpoint @ step 3465 (227M tokens, 2d07h37m elapsed) |
| 2026-09-14 23:13:21 [all] step 3470/18464 loss=8.1184 ppl=3355.80 lr=5.618e-04 | 722 tok/s | 227M done | eta 15d18h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 23:13:21 [all] projected finish 18d01h51m exceeds the 168h deadline |
| 2026-09-14 23:28:40 [all] step 3480/18464 loss=8.1258 ppl=3380.56 lr=5.616e-04 | 713 tok/s | 228M done | eta 15d22h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 23:28:40 [all] projected finish 18d06h40m exceeds the 168h deadline |
| 2026-09-14 23:36:23 [all] checkpoint @ step 3485 (228M tokens, 2d08h07m elapsed) |
| 2026-09-14 23:43:59 [all] step 3490/18464 loss=8.1167 ppl=3349.89 lr=5.613e-04 | 714 tok/s | 229M done | eta 15d21h58m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 23:43:59 [all] projected finish 18d06h14m exceeds the 168h deadline |
| 2026-09-14 23:59:14 [all] step 3500/18464 loss=8.1261 ppl=3381.56 lr=5.611e-04 | 716 tok/s | 229M done | eta 15d20h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-14 23:59:14 [all] projected finish 18d04h57m exceeds the 168h deadline |
| 2026-09-15 00:00:23 [all] eval step 3500/18464 loss=8.1238 ppl=3373.66 (best 8.1238) |
| 2026-09-15 00:06:29 [all] checkpoint @ step 3504 (230M tokens, 2d08h37m elapsed) |
| 2026-09-15 00:15:38 [all] step 3510/18464 loss=8.1224 ppl=3369.02 lr=5.609e-04 | 666 tok/s | 230M done | eta 17d00h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 00:15:38 [all] projected finish 19d09h42m exceeds the 168h deadline |
| 2026-09-15 00:30:53 [all] step 3520/18464 loss=8.1338 ppl=3407.89 lr=5.606e-04 | 716 tok/s | 231M done | eta 15d19h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 00:30:53 [all] projected finish 18d04h47m exceeds the 168h deadline |
| 2026-09-15 00:36:59 [all] checkpoint @ step 3524 (231M tokens, 2d09h08m elapsed) |
| 2026-09-15 00:46:07 [all] step 3530/18464 loss=8.1180 ppl=3354.22 lr=5.604e-04 | 717 tok/s | 231M done | eta 15d18h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 00:46:07 [all] projected finish 18d04h15m exceeds the 168h deadline |
| 2026-09-15 01:01:23 [all] step 3540/18464 loss=8.1274 ppl=3386.09 lr=5.601e-04 | 715 tok/s | 232M done | eta 15d19h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 01:01:23 [all] projected finish 18d05h29m exceeds the 168h deadline |
| 2026-09-15 01:07:29 [all] checkpoint @ step 3544 (232M tokens, 2d09h38m elapsed) |
| 2026-09-15 01:16:41 [all] step 3550/18464 loss=8.1223 ppl=3368.77 lr=5.599e-04 | 714 tok/s | 233M done | eta 15d20h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 01:16:41 [all] projected finish 18d05h49m exceeds the 168h deadline |
| 2026-09-15 01:31:51 [all] step 3560/18464 loss=8.1224 ppl=3369.20 lr=5.596e-04 | 720 tok/s | 233M done | eta 15d16h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 01:31:51 [all] projected finish 18d02h49m exceeds the 168h deadline |
| 2026-09-15 01:37:54 [all] checkpoint @ step 3564 (234M tokens, 2d10h09m elapsed) |
| 2026-09-15 01:47:01 [all] step 3570/18464 loss=8.1214 ppl=3365.60 lr=5.594e-04 | 720 tok/s | 234M done | eta 15d16h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 01:47:01 [all] projected finish 18d02h43m exceeds the 168h deadline |
| 2026-09-15 02:02:12 [all] step 3580/18464 loss=8.1062 ppl=3314.83 lr=5.591e-04 | 719 tok/s | 235M done | eta 15d16h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 02:02:12 [all] projected finish 18d03h20m exceeds the 168h deadline |
| 2026-09-15 02:08:14 [all] checkpoint @ step 3584 (235M tokens, 2d10h39m elapsed) |
| 2026-09-15 02:17:22 [all] step 3590/18464 loss=8.1432 ppl=3439.88 lr=5.589e-04 | 720 tok/s | 235M done | eta 15d15h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 02:17:22 [all] projected finish 18d02h39m exceeds the 168h deadline |
| 2026-09-15 02:32:25 [all] step 3600/18464 loss=8.1231 ppl=3371.36 lr=5.586e-04 | 726 tok/s | 236M done | eta 15d12h54m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 02:32:25 [all] projected finish 17d23h58m exceeds the 168h deadline |
| 2026-09-15 02:38:28 [all] checkpoint @ step 3604 (236M tokens, 2d11h09m elapsed) |
| 2026-09-15 02:47:32 [all] step 3610/18464 loss=8.1185 ppl=3356.06 lr=5.584e-04 | 722 tok/s | 237M done | eta 15d14h22m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 02:47:32 [all] projected finish 18d01h41m exceeds the 168h deadline |
| 2026-09-15 03:02:38 [all] step 3620/18464 loss=8.1202 ppl=3361.71 lr=5.581e-04 | 723 tok/s | 237M done | eta 15d13h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 03:02:38 [all] projected finish 18d01h06m exceeds the 168h deadline |
| 2026-09-15 03:08:38 [all] checkpoint @ step 3624 (238M tokens, 2d11h39m elapsed) |
| 2026-09-15 03:17:36 [all] step 3630/18464 loss=8.1213 ppl=3365.31 lr=5.579e-04 | 730 tok/s | 238M done | eta 15d10h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 03:17:36 [all] projected finish 17d21h51m exceeds the 168h deadline |
| 2026-09-15 03:32:35 [all] step 3640/18464 loss=8.1086 ppl=3322.97 lr=5.576e-04 | 729 tok/s | 239M done | eta 15d10h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 03:32:35 [all] projected finish 17d22h10m exceeds the 168h deadline |
| 2026-09-15 03:40:06 [all] checkpoint @ step 3645 (239M tokens, 2d12h11m elapsed) |
| 2026-09-15 03:47:33 [all] step 3650/18464 loss=8.1227 ppl=3370.26 lr=5.574e-04 | 730 tok/s | 239M done | eta 15d09h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 03:47:33 [all] projected finish 17d21h44m exceeds the 168h deadline |
| 2026-09-15 04:02:25 [all] step 3660/18464 loss=8.1211 ppl=3364.80 lr=5.571e-04 | 735 tok/s | 240M done | eta 15d06h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 04:02:25 [all] projected finish 17d19h27m exceeds the 168h deadline |
| 2026-09-15 04:11:21 [all] checkpoint @ step 3666 (240M tokens, 2d12h42m elapsed) |
| 2026-09-15 04:17:17 [all] step 3670/18464 loss=8.1122 ppl=3334.79 lr=5.569e-04 | 735 tok/s | 241M done | eta 15d06h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 04:17:17 [all] projected finish 17d19h21m exceeds the 168h deadline |
| 2026-09-15 04:32:07 [all] step 3680/18464 loss=8.1238 ppl=3373.69 lr=5.566e-04 | 736 tok/s | 241M done | eta 15d05h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 04:32:07 [all] projected finish 17d18h48m exceeds the 168h deadline |
| 2026-09-15 04:42:25 [all] checkpoint @ step 3687 (242M tokens, 2d13h13m elapsed) |
| 2026-09-15 04:46:52 [all] step 3690/18464 loss=8.1188 ppl=3357.01 lr=5.564e-04 | 741 tok/s | 242M done | eta 15d02h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 04:46:52 [all] projected finish 17d16h17m exceeds the 168h deadline |
| 2026-09-15 05:01:28 [all] step 3700/18464 loss=8.1299 ppl=3394.39 lr=5.561e-04 | 748 tok/s | 242M done | eta 14d23h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 05:01:28 [all] projected finish 17d12h42m exceeds the 168h deadline |
| 2026-09-15 05:13:09 [all] checkpoint @ step 3708 (243M tokens, 2d13h44m elapsed) |
| 2026-09-15 05:16:03 [all] step 3710/18464 loss=8.1254 ppl=3379.35 lr=5.559e-04 | 748 tok/s | 243M done | eta 14d22h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 05:16:03 [all] projected finish 17d12h37m exceeds the 168h deadline |
| 2026-09-15 05:30:34 [all] step 3720/18464 loss=8.1174 ppl=3352.25 lr=5.556e-04 | 753 tok/s | 244M done | eta 14d20h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 05:30:34 [all] projected finish 17d10h29m exceeds the 168h deadline |
| 2026-09-15 05:43:36 [all] checkpoint @ step 3729 (244M tokens, 2d14h14m elapsed) |
| 2026-09-15 05:45:03 [all] step 3730/18464 loss=8.1100 ppl=3327.43 lr=5.553e-04 | 754 tok/s | 244M done | eta 14d19h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 05:45:03 [all] projected finish 17d10h01m exceeds the 168h deadline |
| 2026-09-15 05:59:25 [all] step 3740/18464 loss=8.1002 ppl=3295.18 lr=5.551e-04 | 760 tok/s | 245M done | eta 14d16h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 05:59:25 [all] projected finish 17d06h58m exceeds the 168h deadline |
| 2026-09-15 06:13:45 [all] step 3750/18464 loss=8.1256 ppl=3380.01 lr=5.548e-04 | 762 tok/s | 246M done | eta 14d15h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 06:13:45 [all] projected finish 17d06h22m exceeds the 168h deadline |
| 2026-09-15 06:13:45 [all] checkpoint @ step 3750 (246M tokens, 2d14h45m elapsed) |
| 2026-09-15 06:27:53 [all] step 3760/18464 loss=8.1068 ppl=3317.08 lr=5.546e-04 | 773 tok/s | 246M done | eta 14d10h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 06:27:53 [all] projected finish 17d01h13m exceeds the 168h deadline |
| 2026-09-15 06:41:56 [all] step 3770/18464 loss=8.1257 ppl=3380.31 lr=5.543e-04 | 777 tok/s | 247M done | eta 14d08h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 06:41:56 [all] projected finish 16d23h21m exceeds the 168h deadline |
| 2026-09-15 06:44:40 [all] checkpoint @ step 3772 (247M tokens, 2d15h16m elapsed) |
| 2026-09-15 06:55:22 [all] step 3780/18464 loss=8.1148 ppl=3343.53 lr=5.540e-04 | 813 tok/s | 248M done | eta 13d16h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 06:55:22 [all] projected finish 16d08h06m exceeds the 168h deadline |
| 2026-09-15 07:08:27 [all] step 3790/18464 loss=8.1395 ppl=3427.06 lr=5.538e-04 | 834 tok/s | 248M done | eta 13d08h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 07:08:27 [all] projected finish 15d23h52m exceeds the 168h deadline |
| 2026-09-15 07:14:51 [all] checkpoint @ step 3795 (249M tokens, 2d15h46m elapsed) |
| 2026-09-15 07:21:13 [all] step 3800/18464 loss=8.1369 ppl=3418.43 lr=5.535e-04 | 856 tok/s | 249M done | eta 12d23h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 07:21:13 [all] projected finish 15d15h46m exceeds the 168h deadline |
| 2026-09-15 07:33:53 [all] step 3810/18464 loss=8.1213 ppl=3365.35 lr=5.533e-04 | 862 tok/s | 250M done | eta 12d21h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 07:33:53 [all] projected finish 15d13h24m exceeds the 168h deadline |
| 2026-09-15 07:45:13 [all] checkpoint @ step 3819 (250M tokens, 2d16h16m elapsed) |
| 2026-09-15 07:46:28 [all] step 3820/18464 loss=8.1196 ppl=3359.54 lr=5.530e-04 | 868 tok/s | 250M done | eta 12d19h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 07:46:28 [all] projected finish 15d11h26m exceeds the 168h deadline |
| 2026-09-15 07:58:57 [all] step 3830/18464 loss=8.1350 ppl=3411.76 lr=5.527e-04 | 875 tok/s | 251M done | eta 12d16h22m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 07:58:57 [all] projected finish 15d08h53m exceeds the 168h deadline |
| 2026-09-15 08:11:22 [all] step 3840/18464 loss=8.1249 ppl=3377.40 lr=5.525e-04 | 879 tok/s | 252M done | eta 12d14h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 08:11:22 [all] projected finish 15d07h35m exceeds the 168h deadline |
| 2026-09-15 08:16:18 [all] checkpoint @ step 3844 (252M tokens, 2d16h47m elapsed) |
| 2026-09-15 08:23:41 [all] step 3850/18464 loss=8.1334 ppl=3406.25 lr=5.522e-04 | 887 tok/s | 252M done | eta 12d11h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 08:23:41 [all] projected finish 15d04h42m exceeds the 168h deadline |
| 2026-09-15 08:35:55 [all] step 3860/18464 loss=8.1281 ppl=3388.29 lr=5.519e-04 | 893 tok/s | 253M done | eta 12d09h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 08:35:55 [all] projected finish 15d02h52m exceeds the 168h deadline |
| 2026-09-15 08:46:56 [all] checkpoint @ step 3869 (254M tokens, 2d17h18m elapsed) |
| 2026-09-15 08:48:09 [all] step 3870/18464 loss=8.1081 ppl=3321.18 lr=5.517e-04 | 892 tok/s | 254M done | eta 12d09h49m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 08:48:09 [all] projected finish 15d03h08m exceeds the 168h deadline |
| 2026-09-15 09:00:33 [all] step 3880/18464 loss=8.1156 ppl=3346.10 lr=5.514e-04 | 882 tok/s | 254M done | eta 12d13h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 09:00:33 [all] projected finish 15d06h40m exceeds the 168h deadline |
| 2026-09-15 09:13:06 [all] step 3890/18464 loss=8.1208 ppl=3363.83 lr=5.511e-04 | 870 tok/s | 255M done | eta 12d16h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 09:13:06 [all] projected finish 15d10h43m exceeds the 168h deadline |
| 2026-09-15 09:17:00 [all] checkpoint @ step 3893 (255M tokens, 2d17h48m elapsed) |
| 2026-09-15 09:26:08 [all] step 3900/18464 loss=8.1246 ppl=3376.37 lr=5.509e-04 | 838 tok/s | 256M done | eta 13d04h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 09:26:08 [all] projected finish 15d22h12m exceeds the 168h deadline |
| 2026-09-15 09:39:41 [all] step 3910/18464 loss=8.1124 ppl=3335.60 lr=5.506e-04 | 806 tok/s | 256M done | eta 13d16h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 09:39:41 [all] projected finish 16d10h46m exceeds the 168h deadline |
| 2026-09-15 09:47:57 [all] checkpoint @ step 3916 (257M tokens, 2d18h19m elapsed) |
| 2026-09-15 09:53:33 [all] step 3920/18464 loss=8.1146 ppl=3342.82 lr=5.503e-04 | 787 tok/s | 257M done | eta 14d00h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 09:53:33 [all] projected finish 16d18h54m exceeds the 168h deadline |
| 2026-09-15 10:07:48 [all] step 3930/18464 loss=8.1291 ppl=3391.68 lr=5.500e-04 | 767 tok/s | 258M done | eta 14d09h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 10:07:48 [all] projected finish 17d03h43m exceeds the 168h deadline |
| 2026-09-15 10:19:22 [all] checkpoint @ step 3938 (258M tokens, 2d18h50m elapsed) |
| 2026-09-15 10:22:19 [all] step 3940/18464 loss=8.1433 ppl=3440.14 lr=5.498e-04 | 753 tok/s | 258M done | eta 14d15h16m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 10:22:19 [all] projected finish 17d10h10m exceeds the 168h deadline |
| 2026-09-15 10:37:06 [all] step 3950/18464 loss=8.1255 ppl=3379.71 lr=5.495e-04 | 739 tok/s | 259M done | eta 14d21h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 10:37:06 [all] projected finish 17d16h54m exceeds the 168h deadline |
| 2026-09-15 10:50:27 [all] checkpoint @ step 3959 (259M tokens, 2d19h21m elapsed) |
| 2026-09-15 10:51:57 [all] step 3960/18464 loss=8.1248 ppl=3377.25 lr=5.492e-04 | 736 tok/s | 260M done | eta 14d22h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 10:51:57 [all] projected finish 17d18h04m exceeds the 168h deadline |
| 2026-09-15 11:06:54 [all] step 3970/18464 loss=8.1082 ppl=3321.45 lr=5.490e-04 | 731 tok/s | 260M done | eta 15d01h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 11:06:54 [all] projected finish 17d20h47m exceeds the 168h deadline |
| 2026-09-15 11:21:50 [all] step 3980/18464 loss=8.1139 ppl=3340.61 lr=5.487e-04 | 731 tok/s | 261M done | eta 15d00h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 11:21:50 [all] projected finish 17d20h44m exceeds the 168h deadline |
| 2026-09-15 11:21:51 [all] checkpoint @ step 3980 (261M tokens, 2d19h53m elapsed) |
| 2026-09-15 11:36:48 [all] step 3990/18464 loss=8.1271 ppl=3384.89 lr=5.484e-04 | 730 tok/s | 261M done | eta 15d01h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 11:36:48 [all] projected finish 17d21h11m exceeds the 168h deadline |
| 2026-09-15 11:51:41 [all] step 4000/18464 loss=8.1193 ppl=3358.50 lr=5.481e-04 | 734 tok/s | 262M done | eta 14d22h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 11:51:41 [all] projected finish 17d18h52m exceeds the 168h deadline |
| 2026-09-15 11:52:48 [all] eval step 4000/18464 loss=8.1235 ppl=3372.93 (best 8.1235) |
| 2026-09-15 11:52:50 [all] sample @ 4000: ' to the to the the the the the the the the the a the the the the a the the the a to the the the the the the the the the the the the the the the the the the the a the the the the the' |
| 2026-09-15 11:52:51 [all] checkpoint @ step 4000 (262M tokens, 2d20h24m elapsed) |
| 2026-09-15 12:07:49 [all] step 4010/18464 loss=8.1052 ppl=3311.54 lr=5.479e-04 | 677 tok/s | 263M done | eta 16d04h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 12:07:49 [all] projected finish 19d01h30m exceeds the 168h deadline |
| 2026-09-15 12:22:50 [all] step 4020/18464 loss=8.1268 ppl=3384.10 lr=5.476e-04 | 728 tok/s | 263M done | eta 15d01h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 12:22:50 [all] projected finish 17d22h08m exceeds the 168h deadline |
| 2026-09-15 12:24:20 [all] checkpoint @ step 4021 (264M tokens, 2d20h55m elapsed) |
| 2026-09-15 12:37:51 [all] step 4030/18464 loss=8.1162 ppl=3348.20 lr=5.473e-04 | 727 tok/s | 264M done | eta 15d01h23m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 12:37:51 [all] projected finish 17d22h32m exceeds the 168h deadline |
| 2026-09-15 12:52:53 [all] step 4040/18464 loss=8.1270 ppl=3384.55 lr=5.470e-04 | 726 tok/s | 265M done | eta 15d01h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 12:52:53 [all] projected finish 17d22h58m exceeds the 168h deadline |
| 2026-09-15 12:54:24 [all] checkpoint @ step 4041 (265M tokens, 2d21h25m elapsed) |
| 2026-09-15 13:07:57 [all] step 4050/18464 loss=8.1338 ppl=3407.75 lr=5.467e-04 | 725 tok/s | 265M done | eta 15d01h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 13:07:57 [all] projected finish 17d23h26m exceeds the 168h deadline |
| 2026-09-15 13:23:10 [all] step 4060/18464 loss=8.1196 ppl=3359.66 lr=5.465e-04 | 717 tok/s | 266M done | eta 15d05h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 13:23:10 [all] projected finish 18d03h24m exceeds the 168h deadline |
| 2026-09-15 13:24:43 [all] checkpoint @ step 4061 (266M tokens, 2d21h56m elapsed) |
| 2026-09-15 13:38:34 [all] step 4070/18464 loss=8.1250 ppl=3377.96 lr=5.462e-04 | 710 tok/s | 267M done | eta 15d09h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 13:38:34 [all] projected finish 18d07h28m exceeds the 168h deadline |
| 2026-09-15 13:53:59 [all] step 4080/18464 loss=8.1120 ppl=3334.28 lr=5.459e-04 | 708 tok/s | 267M done | eta 15d09h43m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 13:53:59 [all] projected finish 18d08h08m exceeds the 168h deadline |
| 2026-09-15 13:55:37 [all] checkpoint @ step 4081 (267M tokens, 2d22h26m elapsed) |
| 2026-09-15 14:09:30 [all] step 4090/18464 loss=8.1340 ppl=3408.48 lr=5.456e-04 | 705 tok/s | 268M done | eta 15d11h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 14:09:30 [all] projected finish 18d10h06m exceeds the 168h deadline |
| 2026-09-15 14:25:13 [all] step 4100/18464 loss=8.1153 ppl=3345.30 lr=5.453e-04 | 695 tok/s | 269M done | eta 15d16h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 14:25:13 [all] projected finish 18d15h11m exceeds the 168h deadline |
| 2026-09-15 14:26:47 [all] checkpoint @ step 4101 (269M tokens, 2d22h58m elapsed) |
| 2026-09-15 14:41:05 [all] step 4110/18464 loss=8.1145 ppl=3342.67 lr=5.451e-04 | 688 tok/s | 269M done | eta 15d19h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 14:41:05 [all] projected finish 18d19h06m exceeds the 168h deadline |
| 2026-09-15 14:56:16 [all] step 4120/18464 loss=8.1308 ppl=3397.53 lr=5.448e-04 | 720 tok/s | 270M done | eta 15d02h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 14:56:16 [all] projected finish 18d02h18m exceeds the 168h deadline |
| 2026-09-15 14:57:42 [all] checkpoint @ step 4121 (270M tokens, 2d23h29m elapsed) |
| 2026-09-15 15:10:24 [all] step 4130/18464 loss=8.1110 ppl=3330.82 lr=5.445e-04 | 773 tok/s | 271M done | eta 14d01h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 15:10:24 [all] projected finish 17d01h07m exceeds the 168h deadline |
| 2026-09-15 15:24:24 [all] step 4140/18464 loss=8.1243 ppl=3375.62 lr=5.442e-04 | 780 tok/s | 271M done | eta 13d22h16m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 15:24:24 [all] projected finish 16d22h12m exceeds the 168h deadline |
| 2026-09-15 15:28:29 [all] checkpoint @ step 4143 (272M tokens, 2d23h59m elapsed) |
| 2026-09-15 15:37:54 [all] step 4150/18464 loss=8.1185 ppl=3356.14 lr=5.439e-04 | 809 tok/s | 272M done | eta 13d10h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 15:37:54 [all] projected finish 16d10h17m exceeds the 168h deadline |
| 2026-09-15 15:51:40 [all] step 4160/18464 loss=8.1088 ppl=3323.58 lr=5.436e-04 | 793 tok/s | 273M done | eta 13d16h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 15:51:40 [all] projected finish 16d16h41m exceeds the 168h deadline |
| 2026-09-15 15:59:27 [all] checkpoint @ step 4166 (273M tokens, 3d00h30m elapsed) |
| 2026-09-15 16:04:28 [all] step 4170/18464 loss=8.1199 ppl=3360.62 lr=5.433e-04 | 853 tok/s | 273M done | eta 12d16h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 16:04:28 [all] projected finish 15d17h29m exceeds the 168h deadline |
| 2026-09-15 16:16:51 [all] step 4180/18464 loss=8.1174 ppl=3352.42 lr=5.431e-04 | 882 tok/s | 274M done | eta 12d06h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 16:16:51 [all] projected finish 15d07h28m exceeds the 168h deadline |
| 2026-09-15 16:29:36 [all] step 4190/18464 loss=8.1226 ppl=3369.72 lr=5.428e-04 | 856 tok/s | 275M done | eta 12d15h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 16:29:36 [all] projected finish 15d16h34m exceeds the 168h deadline |
| 2026-09-15 16:29:37 [all] checkpoint @ step 4190 (275M tokens, 3d01h00m elapsed) |
| 2026-09-15 16:41:56 [all] step 4200/18464 loss=8.1118 ppl=3333.59 lr=5.425e-04 | 886 tok/s | 275M done | eta 12d05h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 16:41:56 [all] projected finish 15d06h18m exceeds the 168h deadline |
| 2026-09-15 16:54:18 [all] step 4210/18464 loss=8.1088 ppl=3323.57 lr=5.422e-04 | 883 tok/s | 276M done | eta 12d05h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 16:54:18 [all] projected finish 15d07h14m exceeds the 168h deadline |
| 2026-09-15 17:00:29 [all] checkpoint @ step 4215 (276M tokens, 3d01h31m elapsed) |
| 2026-09-15 17:06:39 [all] step 4220/18464 loss=8.1225 ppl=3369.47 lr=5.419e-04 | 885 tok/s | 277M done | eta 12d04h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 17:06:39 [all] projected finish 15d06h37m exceeds the 168h deadline |
| 2026-09-15 17:19:08 [all] step 4230/18464 loss=8.1232 ppl=3371.82 lr=5.416e-04 | 875 tok/s | 277M done | eta 12d08h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 17:19:08 [all] projected finish 15d10h06m exceeds the 168h deadline |
| 2026-09-15 17:31:28 [all] step 4240/18464 loss=8.1095 ppl=3325.82 lr=5.413e-04 | 885 tok/s | 278M done | eta 12d04h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 17:31:28 [all] projected finish 15d06h32m exceeds the 168h deadline |
| 2026-09-15 17:31:29 [all] checkpoint @ step 4240 (278M tokens, 3d02h02m elapsed) |
| 2026-09-15 17:43:40 [all] step 4250/18464 loss=8.1161 ppl=3347.97 lr=5.410e-04 | 895 tok/s | 279M done | eta 12d00h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 17:43:40 [all] projected finish 15d03h14m exceeds the 168h deadline |
| 2026-09-15 17:55:54 [all] step 4260/18464 loss=8.1212 ppl=3365.00 lr=5.407e-04 | 893 tok/s | 279M done | eta 12d01h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 17:55:54 [all] projected finish 15d04h02m exceeds the 168h deadline |
| 2026-09-15 18:01:55 [all] checkpoint @ step 4265 (280M tokens, 3d02h33m elapsed) |
| 2026-09-15 18:07:46 [all] step 4270/18464 loss=8.1149 ppl=3344.03 lr=5.404e-04 | 920 tok/s | 280M done | eta 11d16h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 18:07:46 [all] projected finish 14d19h26m exceeds the 168h deadline |
| 2026-09-15 18:19:14 [all] step 4280/18464 loss=8.1158 ppl=3346.97 lr=5.401e-04 | 953 tok/s | 280M done | eta 11d06h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 18:19:14 [all] projected finish 14d09h50m exceeds the 168h deadline |
| 2026-09-15 18:30:37 [all] step 4290/18464 loss=8.1280 ppl=3388.05 lr=5.398e-04 | 959 tok/s | 281M done | eta 11d05h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 18:30:37 [all] projected finish 14d08h05m exceeds the 168h deadline |
| 2026-09-15 18:32:53 [all] checkpoint @ step 4292 (281M tokens, 3d03h04m elapsed) |
| 2026-09-15 18:41:52 [all] step 4300/18464 loss=8.1306 ppl=3396.82 lr=5.396e-04 | 971 tok/s | 282M done | eta 11d01h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 18:41:52 [all] projected finish 14d04h41m exceeds the 168h deadline |
| 2026-09-15 18:53:11 [all] step 4310/18464 loss=8.1200 ppl=3360.87 lr=5.393e-04 | 966 tok/s | 282M done | eta 11d02h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 18:53:11 [all] projected finish 14d06h15m exceeds the 168h deadline |
| 2026-09-15 19:03:57 [all] checkpoint @ step 4319 (283M tokens, 3d03h35m elapsed) |
| 2026-09-15 19:05:11 [all] step 4320/18464 loss=8.1263 ppl=3382.30 lr=5.390e-04 | 910 tok/s | 283M done | eta 11d18h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 19:05:11 [all] projected finish 14d22h33m exceeds the 168h deadline |
| 2026-09-15 19:18:03 [all] step 4330/18464 loss=8.1111 ppl=3331.26 lr=5.387e-04 | 849 tok/s | 284M done | eta 12d15h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 19:18:03 [all] projected finish 15d18h50m exceeds the 168h deadline |
| 2026-09-15 19:31:29 [all] step 4340/18464 loss=8.1074 ppl=3319.01 lr=5.384e-04 | 813 tok/s | 284M done | eta 13d04h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 19:31:29 [all] projected finish 16d08h16m exceeds the 168h deadline |
| 2026-09-15 19:34:15 [all] checkpoint @ step 4342 (285M tokens, 3d04h05m elapsed) |
| 2026-09-15 19:45:36 [all] step 4350/18464 loss=8.1142 ppl=3341.56 lr=5.381e-04 | 774 tok/s | 285M done | eta 13d20h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 19:45:36 [all] projected finish 17d00h21m exceeds the 168h deadline |
| 2026-09-15 19:59:41 [all] step 4360/18464 loss=8.1191 ppl=3357.94 lr=5.378e-04 | 775 tok/s | 286M done | eta 13d19h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 19:59:41 [all] projected finish 16d23h45m exceeds the 168h deadline |
| 2026-09-15 20:05:25 [all] checkpoint @ step 4364 (286M tokens, 3d04h36m elapsed) |
| 2026-09-15 20:14:10 [all] step 4370/18464 loss=8.1202 ppl=3361.65 lr=5.375e-04 | 754 tok/s | 286M done | eta 14d04h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 20:14:10 [all] projected finish 17d08h50m exceeds the 168h deadline |
| 2026-09-15 20:29:03 [all] step 4380/18464 loss=8.1176 ppl=3352.89 lr=5.372e-04 | 734 tok/s | 287M done | eta 14d13h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 20:29:03 [all] projected finish 17d18h14m exceeds the 168h deadline |
| 2026-09-15 20:36:22 [all] checkpoint @ step 4385 (287M tokens, 3d05h07m elapsed) |
| 2026-09-15 20:43:25 [all] step 4390/18464 loss=8.1228 ppl=3370.36 lr=5.369e-04 | 760 tok/s | 288M done | eta 14d01h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 20:43:25 [all] projected finish 17d06h22m exceeds the 168h deadline |
| 2026-09-15 20:57:22 [all] step 4400/18464 loss=8.1219 ppl=3367.46 lr=5.366e-04 | 783 tok/s | 288M done | eta 13d14h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 20:57:22 [all] projected finish 16d20h19m exceeds the 168h deadline |
| 2026-09-15 21:07:10 [all] checkpoint @ step 4407 (289M tokens, 3d05h38m elapsed) |
| 2026-09-15 21:11:26 [all] step 4410/18464 loss=8.1149 ppl=3343.79 lr=5.363e-04 | 777 tok/s | 289M done | eta 13d17h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 21:11:26 [all] projected finish 16d23h09m exceeds the 168h deadline |
| 2026-09-15 21:25:54 [all] step 4420/18464 loss=8.1465 ppl=3451.18 lr=5.360e-04 | 755 tok/s | 290M done | eta 14d02h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 21:25:54 [all] projected finish 17d08h45m exceeds the 168h deadline |
| 2026-09-15 21:37:48 [all] checkpoint @ step 4428 (290M tokens, 3d06h09m elapsed) |
| 2026-09-15 21:40:47 [all] step 4430/18464 loss=8.1046 ppl=3309.53 lr=5.357e-04 | 734 tok/s | 290M done | eta 14d11h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 21:40:47 [all] projected finish 17d18h11m exceeds the 168h deadline |
| 2026-09-15 21:55:50 [all] step 4440/18464 loss=8.1116 ppl=3332.88 lr=5.354e-04 | 726 tok/s | 291M done | eta 14d15h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 21:55:50 [all] projected finish 17d22h09m exceeds the 168h deadline |
| 2026-09-15 22:08:06 [all] checkpoint @ step 4448 (292M tokens, 3d06h39m elapsed) |
| 2026-09-15 22:11:15 [all] step 4450/18464 loss=8.1155 ppl=3345.86 lr=5.350e-04 | 708 tok/s | 292M done | eta 15d00h23m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 22:11:15 [all] projected finish 18d07h06m exceeds the 168h deadline |
| 2026-09-15 22:27:03 [all] step 4460/18464 loss=8.1085 ppl=3322.62 lr=5.347e-04 | 691 tok/s | 292M done | eta 15d08h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 22:27:03 [all] projected finish 18d15h38m exceeds the 168h deadline |
| 2026-09-15 22:39:41 [all] checkpoint @ step 4468 (293M tokens, 3d07h11m elapsed) |
| 2026-09-15 22:42:52 [all] step 4470/18464 loss=8.1112 ppl=3331.60 lr=5.344e-04 | 691 tok/s | 293M done | eta 15d08h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 22:42:52 [all] projected finish 18d16h10m exceeds the 168h deadline |
| 2026-09-15 22:58:53 [all] step 4480/18464 loss=8.1106 ppl=3329.70 lr=5.341e-04 | 682 tok/s | 294M done | eta 15d13h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 22:58:53 [all] projected finish 18d20h45m exceeds the 168h deadline |
| 2026-09-15 23:10:06 [all] checkpoint @ step 4487 (294M tokens, 3d07h41m elapsed) |
| 2026-09-15 23:14:53 [all] step 4490/18464 loss=8.1145 ppl=3342.73 lr=5.338e-04 | 683 tok/s | 294M done | eta 15d12h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 23:14:53 [all] projected finish 18d20h22m exceeds the 168h deadline |
| 2026-09-15 23:30:57 [all] step 4500/18464 loss=8.1297 ppl=3393.69 lr=5.335e-04 | 680 tok/s | 295M done | eta 15d13h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 23:30:57 [all] projected finish 18d21h50m exceeds the 168h deadline |
| 2026-09-15 23:32:11 [all] eval step 4500/18464 loss=8.1233 ppl=3372.15 (best 8.1233) |
| 2026-09-15 23:40:21 [all] checkpoint @ step 4505 (295M tokens, 3d08h11m elapsed) |
| 2026-09-15 23:48:25 [all] step 4510/18464 loss=8.1091 ppl=3324.63 lr=5.332e-04 | 625 tok/s | 296M done | eta 16d22h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-15 23:48:25 [all] projected finish 20d06h41m exceeds the 168h deadline |
| 2026-09-16 00:04:41 [all] step 4520/18464 loss=8.1048 ppl=3310.35 lr=5.329e-04 | 671 tok/s | 296M done | eta 15d18h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 00:04:41 [all] projected finish 19d02h39m exceeds the 168h deadline |
| 2026-09-16 00:11:06 [all] checkpoint @ step 4524 (296M tokens, 3d08h42m elapsed) |
| 2026-09-16 00:20:38 [all] step 4530/18464 loss=8.1113 ppl=3331.77 lr=5.326e-04 | 685 tok/s | 297M done | eta 15d10h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 00:20:38 [all] projected finish 18d19h26m exceeds the 168h deadline |
| 2026-09-16 00:36:25 [all] step 4540/18464 loss=8.1153 ppl=3345.31 lr=5.323e-04 | 693 tok/s | 298M done | eta 15d06h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 00:36:25 [all] projected finish 18d15h09m exceeds the 168h deadline |
| 2026-09-16 00:41:09 [all] checkpoint @ step 4543 (298M tokens, 3d09h12m elapsed) |
| 2026-09-16 00:52:02 [all] step 4550/18464 loss=8.1233 ppl=3372.04 lr=5.320e-04 | 699 tok/s | 298M done | eta 15d02h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 00:52:02 [all] projected finish 18d11h43m exceeds the 168h deadline |
| 2026-09-16 01:07:34 [all] step 4560/18464 loss=8.1216 ppl=3366.34 lr=5.317e-04 | 704 tok/s | 299M done | eta 14d23h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 01:07:34 [all] projected finish 18d09h23m exceeds the 168h deadline |
| 2026-09-16 01:12:15 [all] checkpoint @ step 4563 (299M tokens, 3d09h43m elapsed) |
| 2026-09-16 01:23:02 [all] step 4570/18464 loss=8.1103 ppl=3328.64 lr=5.313e-04 | 706 tok/s | 299M done | eta 14d22h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 01:23:02 [all] projected finish 18d08h07m exceeds the 168h deadline |
| 2026-09-16 01:38:24 [all] step 4580/18464 loss=8.1291 ppl=3391.86 lr=5.310e-04 | 711 tok/s | 300M done | eta 14d19h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 01:38:24 [all] projected finish 18d05h38m exceeds the 168h deadline |
| 2026-09-16 01:43:02 [all] checkpoint @ step 4583 (300M tokens, 3d10h14m elapsed) |
| 2026-09-16 01:53:47 [all] step 4590/18464 loss=8.1225 ppl=3369.44 lr=5.307e-04 | 710 tok/s | 301M done | eta 14d19h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 01:53:47 [all] projected finish 18d06h16m exceeds the 168h deadline |
| 2026-09-16 02:09:04 [all] step 4600/18464 loss=8.1260 ppl=3381.39 lr=5.304e-04 | 714 tok/s | 301M done | eta 14d17h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 02:09:04 [all] projected finish 18d03h58m exceeds the 168h deadline |
| 2026-09-16 02:13:43 [all] checkpoint @ step 4603 (302M tokens, 3d10h45m elapsed) |
| 2026-09-16 02:24:27 [all] step 4610/18464 loss=8.1243 ppl=3375.46 lr=5.301e-04 | 710 tok/s | 302M done | eta 14d19h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 02:24:27 [all] projected finish 18d06h03m exceeds the 168h deadline |
| 2026-09-16 02:39:51 [all] step 4620/18464 loss=8.1213 ppl=3365.40 lr=5.298e-04 | 709 tok/s | 303M done | eta 14d19h23m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 02:39:51 [all] projected finish 18d06h34m exceeds the 168h deadline |
| 2026-09-16 02:44:27 [all] checkpoint @ step 4623 (303M tokens, 3d11h15m elapsed) |
| 2026-09-16 02:55:11 [all] step 4630/18464 loss=8.1171 ppl=3351.38 lr=5.295e-04 | 712 tok/s | 303M done | eta 14d17h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 02:55:11 [all] projected finish 18d05h02m exceeds the 168h deadline |
| 2026-09-16 03:10:28 [all] step 4640/18464 loss=8.1192 ppl=3358.41 lr=5.291e-04 | 715 tok/s | 304M done | eta 14d15h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 03:10:28 [all] projected finish 18d03h27m exceeds the 168h deadline |
| 2026-09-16 03:15:02 [all] checkpoint @ step 4643 (304M tokens, 3d11h46m elapsed) |
| 2026-09-16 03:25:44 [all] step 4650/18464 loss=8.0932 ppl=3272.29 lr=5.288e-04 | 715 tok/s | 305M done | eta 14d15h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 03:25:44 [all] projected finish 18d03h30m exceeds the 168h deadline |
| 2026-09-16 03:40:57 [all] step 4660/18464 loss=8.1255 ppl=3379.57 lr=5.285e-04 | 717 tok/s | 305M done | eta 14d14h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 03:40:57 [all] projected finish 18d02h29m exceeds the 168h deadline |
| 2026-09-16 03:45:31 [all] checkpoint @ step 4663 (306M tokens, 3d12h16m elapsed) |
| 2026-09-16 03:56:06 [all] step 4670/18464 loss=8.1410 ppl=3432.34 lr=5.282e-04 | 721 tok/s | 306M done | eta 14d12h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 03:56:06 [all] projected finish 18d00h38m exceeds the 168h deadline |
| 2026-09-16 04:11:15 [all] step 4680/18464 loss=8.1313 ppl=3399.20 lr=5.279e-04 | 721 tok/s | 307M done | eta 14d11h58m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 04:11:15 [all] projected finish 18d00h40m exceeds the 168h deadline |
| 2026-09-16 04:15:47 [all] checkpoint @ step 4683 (307M tokens, 3d12h47m elapsed) |
| 2026-09-16 04:26:20 [all] step 4690/18464 loss=8.1222 ppl=3368.47 lr=5.276e-04 | 724 tok/s | 307M done | eta 14d10h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 04:26:20 [all] projected finish 17d23h28m exceeds the 168h deadline |
| 2026-09-16 04:41:30 [all] step 4700/18464 loss=8.1198 ppl=3360.37 lr=5.272e-04 | 720 tok/s | 308M done | eta 14d11h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 04:41:30 [all] projected finish 18d00h58m exceeds the 168h deadline |
| 2026-09-16 04:45:59 [all] checkpoint @ step 4703 (308M tokens, 3d13h17m elapsed) |
| 2026-09-16 04:56:31 [all] step 4710/18464 loss=8.1275 ppl=3386.24 lr=5.269e-04 | 727 tok/s | 309M done | eta 14d08h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 04:56:31 [all] projected finish 17d21h46m exceeds the 168h deadline |
| 2026-09-16 05:11:36 [all] step 4720/18464 loss=8.1193 ppl=3358.72 lr=5.266e-04 | 725 tok/s | 309M done | eta 14d09h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 05:11:36 [all] projected finish 17d23h00m exceeds the 168h deadline |
| 2026-09-16 05:16:08 [all] checkpoint @ step 4723 (310M tokens, 3d13h47m elapsed) |
| 2026-09-16 05:26:39 [all] step 4730/18464 loss=8.1074 ppl=3319.10 lr=5.263e-04 | 726 tok/s | 310M done | eta 14d08h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 05:26:39 [all] projected finish 17d22h32m exceeds the 168h deadline |
| 2026-09-16 05:41:36 [all] step 4740/18464 loss=8.1155 ppl=3345.97 lr=5.259e-04 | 730 tok/s | 311M done | eta 14d06h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 05:41:36 [all] projected finish 17d20h16m exceeds the 168h deadline |
| 2026-09-16 05:47:34 [all] checkpoint @ step 4744 (311M tokens, 3d14h18m elapsed) |
| 2026-09-16 05:56:29 [all] step 4750/18464 loss=8.1213 ppl=3365.30 lr=5.256e-04 | 734 tok/s | 311M done | eta 14d04h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 05:56:29 [all] projected finish 17d18h35m exceeds the 168h deadline |
| 2026-09-16 06:11:29 [all] step 4760/18464 loss=8.1258 ppl=3380.52 lr=5.253e-04 | 728 tok/s | 312M done | eta 14d06h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 06:11:29 [all] projected finish 17d21h25m exceeds the 168h deadline |
| 2026-09-16 06:18:50 [all] checkpoint @ step 4765 (312M tokens, 3d14h50m elapsed) |
| 2026-09-16 06:26:12 [all] step 4770/18464 loss=8.1096 ppl=3326.20 lr=5.250e-04 | 742 tok/s | 313M done | eta 13d23h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 06:26:12 [all] projected finish 17d14h54m exceeds the 168h deadline |
| 2026-09-16 06:40:56 [all] step 4780/18464 loss=8.1330 ppl=3404.84 lr=5.247e-04 | 742 tok/s | 313M done | eta 13d23h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 06:40:56 [all] projected finish 17d15h03m exceeds the 168h deadline |
| 2026-09-16 06:49:20 [all] checkpoint @ step 4786 (314M tokens, 3d15h20m elapsed) |
| 2026-09-16 06:54:49 [all] step 4790/18464 loss=8.1199 ppl=3360.61 lr=5.243e-04 | 787 tok/s | 314M done | eta 13d04h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 06:54:49 [all] projected finish 16d19h56m exceeds the 168h deadline |
| 2026-09-16 07:08:20 [all] step 4800/18464 loss=8.1257 ppl=3380.29 lr=5.240e-04 | 808 tok/s | 315M done | eta 12d19h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 07:08:20 [all] projected finish 16d11h21m exceeds the 168h deadline |
| 2026-09-16 07:20:16 [all] checkpoint @ step 4809 (315M tokens, 3d15h51m elapsed) |
| 2026-09-16 07:21:35 [all] step 4810/18464 loss=8.1150 ppl=3344.34 lr=5.237e-04 | 824 tok/s | 315M done | eta 12d13h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 07:21:35 [all] projected finish 16d05h22m exceeds the 168h deadline |
| 2026-09-16 07:34:36 [all] step 4820/18464 loss=8.1171 ppl=3351.17 lr=5.233e-04 | 839 tok/s | 316M done | eta 12d08h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 07:34:36 [all] projected finish 16d00h16m exceeds the 168h deadline |
| 2026-09-16 07:47:28 [all] step 4830/18464 loss=8.1343 ppl=3409.39 lr=5.230e-04 | 849 tok/s | 317M done | eta 12d04h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 07:47:28 [all] projected finish 15d20h38m exceeds the 168h deadline |
| 2026-09-16 07:51:21 [all] checkpoint @ step 4833 (317M tokens, 3d16h22m elapsed) |
| 2026-09-16 08:00:22 [all] step 4840/18464 loss=8.1109 ppl=3330.45 lr=5.227e-04 | 847 tok/s | 317M done | eta 12d04h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 08:00:22 [all] projected finish 15d21h17m exceeds the 168h deadline |
| 2026-09-16 08:13:08 [all] step 4850/18464 loss=8.1286 ppl=3389.91 lr=5.224e-04 | 855 tok/s | 318M done | eta 12d01h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 08:13:08 [all] projected finish 15d18h31m exceeds the 168h deadline |
| 2026-09-16 08:22:03 [all] checkpoint @ step 4857 (318M tokens, 3d16h53m elapsed) |
| 2026-09-16 08:25:52 [all] step 4860/18464 loss=8.1343 ppl=3409.57 lr=5.220e-04 | 858 tok/s | 319M done | eta 12d00h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 08:25:52 [all] projected finish 15d17h38m exceeds the 168h deadline |
| 2026-09-16 08:38:43 [all] step 4870/18464 loss=8.1092 ppl=3324.80 lr=5.217e-04 | 850 tok/s | 319M done | eta 12d03h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 08:38:43 [all] projected finish 15d20h19m exceeds the 168h deadline |
| 2026-09-16 08:51:44 [all] step 4880/18464 loss=8.1293 ppl=3392.58 lr=5.214e-04 | 839 tok/s | 320M done | eta 12d06h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 08:51:44 [all] projected finish 16d00h04m exceeds the 168h deadline |
| 2026-09-16 08:53:03 [all] checkpoint @ step 4881 (320M tokens, 3d17h24m elapsed) |
| 2026-09-16 09:05:24 [all] step 4890/18464 loss=8.1361 ppl=3415.67 lr=5.210e-04 | 799 tok/s | 320M done | eta 12d21h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 09:05:24 [all] projected finish 16d14h43m exceeds the 168h deadline |
| 2026-09-16 09:19:24 [all] step 4900/18464 loss=8.1219 ppl=3367.48 lr=5.207e-04 | 780 tok/s | 321M done | eta 13d04h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 09:19:24 [all] projected finish 16d22h33m exceeds the 168h deadline |
| 2026-09-16 09:23:36 [all] checkpoint @ step 4903 (321M tokens, 3d17h54m elapsed) |
| 2026-09-16 09:33:30 [all] step 4910/18464 loss=8.1143 ppl=3342.01 lr=5.204e-04 | 775 tok/s | 322M done | eta 13d06h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 09:33:30 [all] projected finish 17d00h35m exceeds the 168h deadline |
| 2026-09-16 09:47:49 [all] step 4920/18464 loss=8.1231 ppl=3371.60 lr=5.200e-04 | 763 tok/s | 322M done | eta 13d11h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 09:47:49 [all] projected finish 17d05h27m exceeds the 168h deadline |
| 2026-09-16 09:55:02 [all] checkpoint @ step 4925 (323M tokens, 3d18h26m elapsed) |
| 2026-09-16 10:02:16 [all] step 4930/18464 loss=8.1284 ppl=3389.25 lr=5.197e-04 | 756 tok/s | 323M done | eta 13d13h49m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 10:02:16 [all] projected finish 17d08h22m exceeds the 168h deadline |
| 2026-09-16 10:16:52 [all] step 4940/18464 loss=8.1186 ppl=3356.46 lr=5.194e-04 | 748 tok/s | 324M done | eta 13d17h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 10:16:52 [all] projected finish 17d11h57m exceeds the 168h deadline |
| 2026-09-16 10:25:41 [all] checkpoint @ step 4946 (324M tokens, 3d18h57m elapsed) |
| 2026-09-16 10:31:37 [all] step 4950/18464 loss=8.1339 ppl=3408.00 lr=5.190e-04 | 741 tok/s | 324M done | eta 13d20h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 10:31:37 [all] projected finish 17d15h13m exceeds the 168h deadline |
| 2026-09-16 10:46:33 [all] step 4960/18464 loss=8.1214 ppl=3365.72 lr=5.187e-04 | 732 tok/s | 325M done | eta 13d23h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 10:46:33 [all] projected finish 17d19h13m exceeds the 168h deadline |
| 2026-09-16 10:56:59 [all] checkpoint @ step 4967 (326M tokens, 3d19h28m elapsed) |
| 2026-09-16 11:01:27 [all] step 4970/18464 loss=8.1146 ppl=3342.91 lr=5.184e-04 | 733 tok/s | 326M done | eta 13d23h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 11:01:27 [all] projected finish 17d18h42m exceeds the 168h deadline |
| 2026-09-16 11:16:28 [all] step 4980/18464 loss=8.1385 ppl=3423.68 lr=5.180e-04 | 728 tok/s | 326M done | eta 14d01h23m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 11:16:28 [all] projected finish 17d21h11m exceeds the 168h deadline |
| 2026-09-16 11:27:02 [all] checkpoint @ step 4987 (327M tokens, 3d19h58m elapsed) |
| 2026-09-16 11:31:34 [all] step 4990/18464 loss=8.1222 ppl=3368.43 lr=5.177e-04 | 723 tok/s | 327M done | eta 14d03h24m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 11:31:34 [all] projected finish 17d23h27m exceeds the 168h deadline |
| 2026-09-16 11:46:33 [all] step 5000/18464 loss=8.1223 ppl=3368.72 lr=5.174e-04 | 729 tok/s | 328M done | eta 14d00h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 11:46:33 [all] projected finish 17d20h25m exceeds the 168h deadline |
| 2026-09-16 11:47:40 [all] eval step 5000/18464 loss=8.1231 ppl=3371.41 (best 8.1231) |
| 2026-09-16 11:47:41 [all] checkpoint @ step 5000 (328M tokens, 3d20h19m elapsed) |
| 2026-09-16 12:02:35 [all] step 5010/18464 loss=8.1240 ppl=3374.66 lr=5.170e-04 | 682 tok/s | 328M done | eta 14d23h16m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 12:02:35 [all] projected finish 18d19h50m exceeds the 168h deadline |
| 2026-09-16 12:17:19 [all] step 5020/18464 loss=8.1240 ppl=3374.55 lr=5.167e-04 | 741 tok/s | 329M done | eta 13d18h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 12:17:19 [all] projected finish 17d15h15m exceeds the 168h deadline |
| 2026-09-16 12:18:50 [all] checkpoint @ step 5021 (329M tokens, 3d20h50m elapsed) |
| 2026-09-16 12:32:04 [all] step 5030/18464 loss=8.1165 ppl=3349.36 lr=5.163e-04 | 741 tok/s | 330M done | eta 13d18h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 12:32:04 [all] projected finish 17d15h15m exceeds the 168h deadline |
| 2026-09-16 12:46:48 [all] step 5040/18464 loss=8.1360 ppl=3415.34 lr=5.160e-04 | 741 tok/s | 330M done | eta 13d17h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 12:46:48 [all] projected finish 17d14h58m exceeds the 168h deadline |
| 2026-09-16 12:49:48 [all] checkpoint @ step 5042 (330M tokens, 3d21h21m elapsed) |
| 2026-09-16 13:01:49 [all] step 5050/18464 loss=8.1314 ppl=3399.59 lr=5.157e-04 | 727 tok/s | 331M done | eta 13d23h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 13:01:49 [all] projected finish 17d21h14m exceeds the 168h deadline |
| 2026-09-16 13:16:45 [all] step 5060/18464 loss=8.1152 ppl=3344.80 lr=5.153e-04 | 731 tok/s | 332M done | eta 13d21h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 13:16:45 [all] projected finish 17d19h27m exceeds the 168h deadline |
| 2026-09-16 13:21:14 [all] checkpoint @ step 5063 (332M tokens, 3d21h52m elapsed) |
| 2026-09-16 13:31:46 [all] step 5070/18464 loss=8.1161 ppl=3347.90 lr=5.150e-04 | 728 tok/s | 332M done | eta 13d22h58m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 13:31:46 [all] projected finish 17d21h01m exceeds the 168h deadline |
| 2026-09-16 13:46:43 [all] step 5080/18464 loss=8.1346 ppl=3410.61 lr=5.146e-04 | 730 tok/s | 333M done | eta 13d21h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 13:46:43 [all] projected finish 17d19h50m exceeds the 168h deadline |
| 2026-09-16 13:52:43 [all] checkpoint @ step 5084 (333M tokens, 3d22h24m elapsed) |
| 2026-09-16 14:01:38 [all] step 5090/18464 loss=8.1254 ppl=3379.38 lr=5.143e-04 | 732 tok/s | 334M done | eta 13d20h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 14:01:38 [all] projected finish 17d19h08m exceeds the 168h deadline |
| 2026-09-16 14:15:28 [all] step 5100/18464 loss=8.1309 ppl=3398.01 lr=5.140e-04 | 790 tok/s | 334M done | eta 12d20h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 14:15:28 [all] projected finish 16d18h47m exceeds the 168h deadline |
| 2026-09-16 14:23:22 [all] checkpoint @ step 5106 (335M tokens, 3d22h54m elapsed) |
| 2026-09-16 14:28:31 [all] step 5110/18464 loss=8.1324 ppl=3402.79 lr=5.136e-04 | 837 tok/s | 335M done | eta 12d02h23m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 14:28:31 [all] projected finish 16d01h23m exceeds the 168h deadline |
| 2026-09-16 14:41:07 [all] step 5120/18464 loss=8.1297 ppl=3393.73 lr=5.133e-04 | 867 tok/s | 336M done | eta 11d16h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 14:41:07 [all] projected finish 15d15h25m exceeds the 168h deadline |
| 2026-09-16 14:53:35 [all] step 5130/18464 loss=8.1256 ppl=3380.03 lr=5.129e-04 | 876 tok/s | 336M done | eta 11d13h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 14:53:35 [all] projected finish 15d12h40m exceeds the 168h deadline |
| 2026-09-16 14:53:36 [all] checkpoint @ step 5130 (336M tokens, 3d23h24m elapsed) |
| 2026-09-16 15:05:52 [all] step 5140/18464 loss=8.1253 ppl=3378.79 lr=5.126e-04 | 889 tok/s | 337M done | eta 11d08h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 15:05:52 [all] projected finish 15d08h26m exceeds the 168h deadline |
| 2026-09-16 15:17:54 [all] step 5150/18464 loss=8.1295 ppl=3393.27 lr=5.122e-04 | 909 tok/s | 338M done | eta 11d02h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 15:17:54 [all] projected finish 15d02h33m exceeds the 168h deadline |
| 2026-09-16 15:23:56 [all] checkpoint @ step 5155 (338M tokens, 3d23h55m elapsed) |
| 2026-09-16 15:29:54 [all] step 5160/18464 loss=8.1249 ppl=3377.43 lr=5.119e-04 | 909 tok/s | 338M done | eta 11d02h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 15:29:54 [all] projected finish 15d02h22m exceeds the 168h deadline |
| 2026-09-16 15:41:42 [all] step 5170/18464 loss=8.1086 ppl=3323.05 lr=5.115e-04 | 926 tok/s | 339M done | eta 10d21h16m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 15:41:42 [all] projected finish 14d21h29m exceeds the 168h deadline |
| 2026-09-16 15:53:16 [all] step 5180/18464 loss=8.1177 ppl=3353.19 lr=5.112e-04 | 944 tok/s | 339M done | eta 10d16h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 15:53:16 [all] projected finish 14d16h42m exceeds the 168h deadline |
| 2026-09-16 15:54:26 [all] checkpoint @ step 5181 (340M tokens, 4d00h25m elapsed) |
| 2026-09-16 16:04:52 [all] step 5190/18464 loss=8.1195 ppl=3359.44 lr=5.109e-04 | 942 tok/s | 340M done | eta 10d16h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 16:04:52 [all] projected finish 14d17h06m exceeds the 168h deadline |
| 2026-09-16 16:16:13 [all] step 5200/18464 loss=8.1241 ppl=3374.83 lr=5.105e-04 | 963 tok/s | 341M done | eta 10d10h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 16:16:13 [all] projected finish 14d11h31m exceeds the 168h deadline |
| 2026-09-16 16:24:28 [all] checkpoint @ step 5207 (341M tokens, 4d00h55m elapsed) |
| 2026-09-16 16:28:07 [all] step 5210/18464 loss=8.1343 ppl=3409.40 lr=5.102e-04 | 917 tok/s | 341M done | eta 10d23h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 16:28:07 [all] projected finish 15d00h02m exceeds the 168h deadline |
| 2026-09-16 16:40:35 [all] step 5220/18464 loss=8.1522 ppl=3470.90 lr=5.098e-04 | 877 tok/s | 342M done | eta 11d11h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 16:40:35 [all] projected finish 15d12h13m exceeds the 168h deadline |
| 2026-09-16 16:53:18 [all] step 5230/18464 loss=8.1260 ppl=3381.41 lr=5.095e-04 | 859 tok/s | 343M done | eta 11d16h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 16:53:18 [all] projected finish 15d17h52m exceeds the 168h deadline |
| 2026-09-16 16:54:33 [all] checkpoint @ step 5231 (343M tokens, 4d01h25m elapsed) |
| 2026-09-16 17:05:59 [all] step 5240/18464 loss=8.1329 ppl=3404.54 lr=5.091e-04 | 861 tok/s | 343M done | eta 11d15h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 17:05:59 [all] projected finish 15d17h17m exceeds the 168h deadline |
| 2026-09-16 17:19:27 [all] step 5250/18464 loss=8.1126 ppl=3336.31 lr=5.088e-04 | 812 tok/s | 344M done | eta 12d08h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 17:19:27 [all] projected finish 16d10h16m exceeds the 168h deadline |
| 2026-09-16 17:24:56 [all] checkpoint @ step 5254 (344M tokens, 4d01h56m elapsed) |
| 2026-09-16 17:33:18 [all] step 5260/18464 loss=8.1379 ppl=3421.70 lr=5.084e-04 | 789 tok/s | 345M done | eta 12d16h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 17:33:18 [all] projected finish 16d18h52m exceeds the 168h deadline |
| 2026-09-16 17:47:36 [all] step 5270/18464 loss=8.1360 ppl=3415.09 lr=5.080e-04 | 763 tok/s | 345M done | eta 13d02h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 17:47:36 [all] projected finish 17d05h00m exceeds the 168h deadline |
| 2026-09-16 17:56:24 [all] checkpoint @ step 5276 (346M tokens, 4d02h27m elapsed) |
| 2026-09-16 18:02:13 [all] step 5280/18464 loss=8.1251 ppl=3378.32 lr=5.077e-04 | 747 tok/s | 346M done | eta 13d09h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 18:02:13 [all] projected finish 17d11h41m exceeds the 168h deadline |
| 2026-09-16 18:16:40 [all] step 5290/18464 loss=8.1150 ppl=3344.34 lr=5.073e-04 | 756 tok/s | 347M done | eta 13d05h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 18:16:40 [all] projected finish 17d07h59m exceeds the 168h deadline |
| 2026-09-16 18:26:49 [all] checkpoint @ step 5297 (347M tokens, 4d02h58m elapsed) |
| 2026-09-16 18:31:09 [all] step 5300/18464 loss=8.1403 ppl=3429.85 lr=5.070e-04 | 754 tok/s | 347M done | eta 13d05h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 18:31:09 [all] projected finish 17d08h56m exceeds the 168h deadline |
| 2026-09-16 18:45:31 [all] step 5310/18464 loss=8.1019 ppl=3300.86 lr=5.066e-04 | 760 tok/s | 348M done | eta 13d02h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 18:45:31 [all] projected finish 17d06h16m exceeds the 168h deadline |
| 2026-09-16 18:57:01 [all] checkpoint @ step 5318 (349M tokens, 4d03h28m elapsed) |
| 2026-09-16 18:59:52 [all] step 5320/18464 loss=8.1180 ppl=3354.35 lr=5.063e-04 | 762 tok/s | 349M done | eta 13d02h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 18:59:52 [all] projected finish 17d05h42m exceeds the 168h deadline |
| 2026-09-16 19:14:17 [all] step 5330/18464 loss=8.1145 ppl=3342.61 lr=5.059e-04 | 758 tok/s | 349M done | eta 13d03h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 19:14:17 [all] projected finish 17d07h12m exceeds the 168h deadline |
| 2026-09-16 19:28:22 [all] step 5340/18464 loss=8.1281 ppl=3388.51 lr=5.056e-04 | 775 tok/s | 350M done | eta 12d20h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 19:28:22 [all] projected finish 17d00h08m exceeds the 168h deadline |
| 2026-09-16 19:28:22 [all] checkpoint @ step 5340 (350M tokens, 4d03h59m elapsed) |
| 2026-09-16 19:42:18 [all] step 5350/18464 loss=8.1171 ppl=3351.24 lr=5.052e-04 | 784 tok/s | 351M done | eta 12d16h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 19:42:18 [all] projected finish 16d20h40m exceeds the 168h deadline |
| 2026-09-16 19:56:54 [all] step 5360/18464 loss=8.1167 ppl=3349.79 lr=5.049e-04 | 748 tok/s | 351M done | eta 13d06h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 19:56:54 [all] projected finish 17d11h20m exceeds the 168h deadline |
| 2026-09-16 19:58:26 [all] checkpoint @ step 5361 (351M tokens, 4d04h29m elapsed) |
| 2026-09-16 20:11:20 [all] step 5370/18464 loss=8.1292 ppl=3391.98 lr=5.045e-04 | 757 tok/s | 352M done | eta 13d02h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 20:11:20 [all] projected finish 17d07h42m exceeds the 168h deadline |
| 2026-09-16 20:25:18 [all] step 5380/18464 loss=8.1316 ppl=3400.12 lr=5.041e-04 | 782 tok/s | 353M done | eta 12d16h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 20:25:18 [all] projected finish 16d21h36m exceeds the 168h deadline |
| 2026-09-16 20:29:34 [all] checkpoint @ step 5383 (353M tokens, 4d05h00m elapsed) |
| 2026-09-16 20:39:23 [all] step 5390/18464 loss=8.1068 ppl=3316.91 lr=5.038e-04 | 775 tok/s | 353M done | eta 12d19h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 20:39:23 [all] projected finish 17d00h12m exceeds the 168h deadline |
| 2026-09-16 20:53:24 [all] step 5400/18464 loss=8.1145 ppl=3342.63 lr=5.034e-04 | 780 tok/s | 354M done | eta 12d17h00m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 20:53:24 [all] projected finish 16d22h24m exceeds the 168h deadline |
| 2026-09-16 21:00:28 [all] checkpoint @ step 5405 (354M tokens, 4d05h31m elapsed) |
| 2026-09-16 21:07:37 [all] step 5410/18464 loss=8.1113 ppl=3331.85 lr=5.031e-04 | 768 tok/s | 355M done | eta 12d21h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 21:07:37 [all] projected finish 17d03h06m exceeds the 168h deadline |
| 2026-09-16 21:22:18 [all] step 5420/18464 loss=8.1185 ppl=3355.85 lr=5.027e-04 | 744 tok/s | 355M done | eta 13d07h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 21:22:18 [all] projected finish 17d13h05m exceeds the 168h deadline |
| 2026-09-16 21:31:10 [all] checkpoint @ step 5426 (356M tokens, 4d06h02m elapsed) |
| 2026-09-16 21:37:12 [all] step 5430/18464 loss=8.1109 ppl=3330.66 lr=5.023e-04 | 733 tok/s | 356M done | eta 13d11h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 21:37:12 [all] projected finish 17d17h49m exceeds the 168h deadline |
| 2026-09-16 21:53:49 [all] step 5440/18464 loss=8.1405 ppl=3430.74 lr=5.020e-04 | 657 tok/s | 357M done | eta 15d00h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 21:53:49 [all] projected finish 19d07h04m exceeds the 168h deadline |
| 2026-09-16 22:01:44 [all] checkpoint @ step 5445 (357M tokens, 4d06h33m elapsed) |
| 2026-09-16 22:09:31 [all] step 5450/18464 loss=8.0926 ppl=3270.32 lr=5.016e-04 | 696 tok/s | 357M done | eta 14d04h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 22:09:31 [all] projected finish 18d11h17m exceeds the 168h deadline |
| 2026-09-16 22:24:51 [all] step 5460/18464 loss=8.1110 ppl=3330.99 lr=5.013e-04 | 713 tok/s | 358M done | eta 13d20h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 22:24:51 [all] projected finish 18d03h07m exceeds the 168h deadline |
| 2026-09-16 22:33:05 [all] checkpoint @ step 5465 (358M tokens, 4d07h04m elapsed) |
| 2026-09-16 22:41:12 [all] step 5470/18464 loss=8.1118 ppl=3333.54 lr=5.009e-04 | 668 tok/s | 358M done | eta 14d18h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 22:41:12 [all] projected finish 19d01h24m exceeds the 168h deadline |
| 2026-09-16 22:56:24 [all] step 5480/18464 loss=8.1294 ppl=3392.63 lr=5.005e-04 | 719 tok/s | 359M done | eta 13d16h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 22:56:24 [all] projected finish 18d00h18m exceeds the 168h deadline |
| 2026-09-16 23:04:02 [all] checkpoint @ step 5485 (359M tokens, 4d07h35m elapsed) |
| 2026-09-16 23:11:10 [all] step 5490/18464 loss=8.1084 ppl=3322.27 lr=5.002e-04 | 740 tok/s | 360M done | eta 13d07h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 23:11:10 [all] projected finish 17d14h54m exceeds the 168h deadline |
| 2026-09-16 23:24:15 [all] step 5500/18464 loss=8.1119 ppl=3334.00 lr=4.998e-04 | 835 tok/s | 360M done | eta 11d18h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 23:24:15 [all] projected finish 16d02h42m exceeds the 168h deadline |
| 2026-09-16 23:25:11 [all] eval step 5500/18464 loss=8.1229 ppl=3370.63 (best 8.1229) |
| 2026-09-16 23:35:00 [all] checkpoint @ step 5508 (361M tokens, 4d08h06m elapsed) |
| 2026-09-16 23:37:25 [all] step 5510/18464 loss=8.1169 ppl=3350.74 lr=4.994e-04 | 830 tok/s | 361M done | eta 11d20h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 23:37:25 [all] projected finish 16d04h15m exceeds the 168h deadline |
| 2026-09-16 23:49:53 [all] step 5520/18464 loss=8.1257 ppl=3380.11 lr=4.991e-04 | 876 tok/s | 362M done | eta 11d05h01m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-16 23:49:53 [all] projected finish 15d13h23m exceeds the 168h deadline |
| 2026-09-17 00:02:48 [all] step 5530/18464 loss=8.1155 ppl=3345.88 lr=4.987e-04 | 845 tok/s | 362M done | eta 11d14h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 00:02:48 [all] projected finish 15d23h12m exceeds the 168h deadline |
| 2026-09-17 00:05:28 [all] checkpoint @ step 5532 (363M tokens, 4d08h36m elapsed) |
| 2026-09-17 00:15:57 [all] step 5540/18464 loss=8.1259 ppl=3380.90 lr=4.983e-04 | 831 tok/s | 363M done | eta 11d19h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 00:15:57 [all] projected finish 16d03h59m exceeds the 168h deadline |
| 2026-09-17 00:29:18 [all] step 5550/18464 loss=8.1212 ppl=3364.98 lr=4.980e-04 | 818 tok/s | 364M done | eta 11d23h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 00:29:18 [all] projected finish 16d08h20m exceeds the 168h deadline |
| 2026-09-17 00:36:11 [all] checkpoint @ step 5555 (364M tokens, 4d09h07m elapsed) |
| 2026-09-17 00:42:52 [all] step 5560/18464 loss=8.1204 ppl=3362.36 lr=4.976e-04 | 805 tok/s | 364M done | eta 12d03h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 00:42:52 [all] projected finish 16d13h00m exceeds the 168h deadline |
| 2026-09-17 00:56:09 [all] step 5570/18464 loss=8.1226 ppl=3369.80 lr=4.972e-04 | 823 tok/s | 365M done | eta 11d21h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 00:56:09 [all] projected finish 16d06h43m exceeds the 168h deadline |
| 2026-09-17 01:06:48 [all] checkpoint @ step 5578 (366M tokens, 4d09h38m elapsed) |
| 2026-09-17 01:09:26 [all] step 5580/18464 loss=8.1210 ppl=3364.50 lr=4.969e-04 | 822 tok/s | 366M done | eta 11d21h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 01:09:26 [all] projected finish 16d06h59m exceeds the 168h deadline |
| 2026-09-17 01:22:44 [all] step 5590/18464 loss=8.1091 ppl=3324.45 lr=4.965e-04 | 821 tok/s | 366M done | eta 11d21h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 01:22:44 [all] projected finish 16d07h19m exceeds the 168h deadline |
| 2026-09-17 01:35:54 [all] step 5600/18464 loss=8.1195 ppl=3359.38 lr=4.961e-04 | 830 tok/s | 367M done | eta 11d18h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 01:35:54 [all] projected finish 16d04h18m exceeds the 168h deadline |
| 2026-09-17 01:37:11 [all] checkpoint @ step 5601 (367M tokens, 4d10h08m elapsed) |
| 2026-09-17 01:48:56 [all] step 5610/18464 loss=8.1087 ppl=3323.40 lr=4.958e-04 | 837 tok/s | 368M done | eta 11d15h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 01:48:56 [all] projected finish 16d01h45m exceeds the 168h deadline |
| 2026-09-17 02:01:53 [all] step 5620/18464 loss=8.1203 ppl=3362.00 lr=4.954e-04 | 844 tok/s | 368M done | eta 11d13h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 02:01:53 [all] projected finish 15d23h37m exceeds the 168h deadline |
| 2026-09-17 02:08:23 [all] checkpoint @ step 5625 (369M tokens, 4d10h39m elapsed) |
| 2026-09-17 02:14:51 [all] step 5630/18464 loss=8.1069 ppl=3317.11 lr=4.950e-04 | 842 tok/s | 369M done | eta 11d13h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 02:14:51 [all] projected finish 16d00h12m exceeds the 168h deadline |
| 2026-09-17 02:27:46 [all] step 5640/18464 loss=8.1302 ppl=3395.40 lr=4.947e-04 | 846 tok/s | 370M done | eta 11d11h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 02:27:46 [all] projected finish 15d22h52m exceeds the 168h deadline |
| 2026-09-17 02:39:21 [all] checkpoint @ step 5649 (370M tokens, 4d11h10m elapsed) |
| 2026-09-17 02:40:38 [all] step 5650/18464 loss=8.1143 ppl=3341.78 lr=4.943e-04 | 849 tok/s | 370M done | eta 11d10h49m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 02:40:38 [all] projected finish 15d22h01m exceeds the 168h deadline |
| 2026-09-17 02:53:29 [all] step 5660/18464 loss=8.1217 ppl=3366.69 lr=4.939e-04 | 850 tok/s | 371M done | eta 11d10h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 02:53:29 [all] projected finish 15d21h36m exceeds the 168h deadline |
| 2026-09-17 03:06:28 [all] step 5670/18464 loss=8.1146 ppl=3342.98 lr=4.935e-04 | 840 tok/s | 372M done | eta 11d13h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 03:06:28 [all] projected finish 16d00h44m exceeds the 168h deadline |
| 2026-09-17 03:10:23 [all] checkpoint @ step 5673 (372M tokens, 4d11h41m elapsed) |
| 2026-09-17 03:19:27 [all] step 5680/18464 loss=8.1176 ppl=3353.04 lr=4.932e-04 | 841 tok/s | 372M done | eta 11d12h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 03:19:27 [all] projected finish 16d00h28m exceeds the 168h deadline |
| 2026-09-17 03:32:22 [all] step 5690/18464 loss=8.1068 ppl=3316.99 lr=4.928e-04 | 846 tok/s | 373M done | eta 11d10h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 03:32:22 [all] projected finish 15d23h03m exceeds the 168h deadline |
| 2026-09-17 03:41:22 [all] checkpoint @ step 5697 (373M tokens, 4d12h12m elapsed) |
| 2026-09-17 03:45:13 [all] step 5700/18464 loss=8.1266 ppl=3383.30 lr=4.924e-04 | 850 tok/s | 374M done | eta 11d09h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 03:45:13 [all] projected finish 15d21h37m exceeds the 168h deadline |
| 2026-09-17 03:57:59 [all] step 5710/18464 loss=8.1400 ppl=3428.85 lr=4.920e-04 | 856 tok/s | 374M done | eta 11d07h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 03:57:59 [all] projected finish 15d19h35m exceeds the 168h deadline |
| 2026-09-17 04:10:40 [all] step 5720/18464 loss=8.1140 ppl=3340.77 lr=4.917e-04 | 861 tok/s | 375M done | eta 11d05h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 04:10:40 [all] projected finish 15d18h16m exceeds the 168h deadline |
| 2026-09-17 04:11:57 [all] checkpoint @ step 5721 (375M tokens, 4d12h43m elapsed) |
| 2026-09-17 04:23:22 [all] step 5730/18464 loss=8.1107 ppl=3330.07 lr=4.913e-04 | 860 tok/s | 376M done | eta 11d05h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 04:23:22 [all] projected finish 15d18h30m exceeds the 168h deadline |
| 2026-09-17 04:35:54 [all] step 5740/18464 loss=8.1189 ppl=3357.32 lr=4.909e-04 | 872 tok/s | 376M done | eta 11d01h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 04:35:54 [all] projected finish 15d14h52m exceeds the 168h deadline |
| 2026-09-17 04:42:09 [all] checkpoint @ step 5745 (377M tokens, 4d13h13m elapsed) |
| 2026-09-17 04:48:27 [all] step 5750/18464 loss=8.1448 ppl=3445.54 lr=4.905e-04 | 871 tok/s | 377M done | eta 11d01h43m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 04:48:27 [all] projected finish 15d15h02m exceeds the 168h deadline |
| 2026-09-17 05:00:57 [all] step 5760/18464 loss=8.1054 ppl=3312.20 lr=4.902e-04 | 873 tok/s | 377M done | eta 11d00h54m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 05:00:57 [all] projected finish 15d14h26m exceeds the 168h deadline |
| 2026-09-17 05:13:19 [all] step 5770/18464 loss=8.1239 ppl=3374.22 lr=4.898e-04 | 883 tok/s | 378M done | eta 10d21h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 05:13:19 [all] projected finish 15d11h24m exceeds the 168h deadline |
| 2026-09-17 05:13:20 [all] checkpoint @ step 5770 (378M tokens, 4d13h44m elapsed) |
| 2026-09-17 05:25:42 [all] step 5780/18464 loss=8.1210 ppl=3364.43 lr=4.894e-04 | 882 tok/s | 379M done | eta 10d21h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 05:25:42 [all] projected finish 15d11h45m exceeds the 168h deadline |
| 2026-09-17 05:38:06 [all] step 5790/18464 loss=8.1100 ppl=3327.46 lr=4.890e-04 | 881 tok/s | 379M done | eta 10d21h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 05:38:06 [all] projected finish 15d12h00m exceeds the 168h deadline |
| 2026-09-17 05:44:16 [all] checkpoint @ step 5795 (380M tokens, 4d14h15m elapsed) |
| 2026-09-17 05:50:25 [all] step 5800/18464 loss=8.1251 ppl=3378.37 lr=4.886e-04 | 887 tok/s | 380M done | eta 10d19h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 05:50:25 [all] projected finish 15d10h15m exceeds the 168h deadline |
| 2026-09-17 06:02:47 [all] step 5810/18464 loss=8.1341 ppl=3408.67 lr=4.883e-04 | 884 tok/s | 381M done | eta 10d20h43m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 06:02:47 [all] projected finish 15d11h17m exceeds the 168h deadline |
| 2026-09-17 06:15:05 [all] step 5820/18464 loss=8.1236 ppl=3373.09 lr=4.879e-04 | 888 tok/s | 381M done | eta 10d19h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 06:15:05 [all] projected finish 15d09h58m exceeds the 168h deadline |
| 2026-09-17 06:15:05 [all] checkpoint @ step 5820 (381M tokens, 4d14h46m elapsed) |
| 2026-09-17 06:27:23 [all] step 5830/18464 loss=8.1285 ppl=3389.86 lr=4.875e-04 | 887 tok/s | 382M done | eta 10d19h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 06:27:23 [all] projected finish 15d10h11m exceeds the 168h deadline |
| 2026-09-17 06:39:39 [all] step 5840/18464 loss=8.1003 ppl=3295.38 lr=4.871e-04 | 891 tok/s | 383M done | eta 10d17h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 06:39:39 [all] projected finish 15d09h04m exceeds the 168h deadline |
| 2026-09-17 06:45:49 [all] checkpoint @ step 5845 (383M tokens, 4d15h17m elapsed) |
| 2026-09-17 06:51:58 [all] step 5850/18464 loss=8.1216 ppl=3366.28 lr=4.867e-04 | 887 tok/s | 383M done | eta 10d18h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 06:51:58 [all] projected finish 15d10h19m exceeds the 168h deadline |
| 2026-09-17 07:04:08 [all] step 5860/18464 loss=8.1365 ppl=3416.83 lr=4.864e-04 | 898 tok/s | 384M done | eta 10d15h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 07:04:08 [all] projected finish 15d07h12m exceeds the 168h deadline |
| 2026-09-17 07:16:00 [all] step 5870/18464 loss=8.1049 ppl=3310.63 lr=4.860e-04 | 920 tok/s | 385M done | eta 10d09h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 07:16:00 [all] projected finish 15d00h52m exceeds the 168h deadline |
| 2026-09-17 07:16:00 [all] checkpoint @ step 5870 (385M tokens, 4d15h47m elapsed) |
| 2026-09-17 07:27:46 [all] step 5880/18464 loss=8.1234 ppl=3372.39 lr=4.856e-04 | 928 tok/s | 385M done | eta 10d06h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 07:27:46 [all] projected finish 14d22h44m exceeds the 168h deadline |
| 2026-09-17 07:39:29 [all] step 5890/18464 loss=8.1149 ppl=3343.79 lr=4.852e-04 | 932 tok/s | 386M done | eta 10d05h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 07:39:29 [all] projected finish 14d21h45m exceeds the 168h deadline |
| 2026-09-17 07:46:26 [all] checkpoint @ step 5896 (386M tokens, 4d16h17m elapsed) |
| 2026-09-17 07:51:05 [all] step 5900/18464 loss=8.1195 ppl=3359.47 lr=4.848e-04 | 942 tok/s | 387M done | eta 10d02h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 07:51:05 [all] projected finish 14d19h16m exceeds the 168h deadline |
| 2026-09-17 08:02:46 [all] step 5910/18464 loss=8.1313 ppl=3399.12 lr=4.844e-04 | 935 tok/s | 387M done | eta 10d04h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 08:02:46 [all] projected finish 14d21h03m exceeds the 168h deadline |
| 2026-09-17 08:14:28 [all] step 5920/18464 loss=8.1201 ppl=3361.42 lr=4.841e-04 | 934 tok/s | 388M done | eta 10d04h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 08:14:28 [all] projected finish 14d21h18m exceeds the 168h deadline |
| 2026-09-17 08:16:48 [all] checkpoint @ step 5922 (388M tokens, 4d16h48m elapsed) |
| 2026-09-17 08:26:09 [all] step 5930/18464 loss=8.1050 ppl=3310.99 lr=4.837e-04 | 935 tok/s | 389M done | eta 10d04h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 08:26:09 [all] projected finish 14d21h03m exceeds the 168h deadline |
| 2026-09-17 08:37:42 [all] step 5940/18464 loss=8.1023 ppl=3302.15 lr=4.833e-04 | 945 tok/s | 389M done | eta 10d01h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 08:37:42 [all] projected finish 14d18h19m exceeds the 168h deadline |
| 2026-09-17 08:46:51 [all] checkpoint @ step 5948 (390M tokens, 4d17h18m elapsed) |
| 2026-09-17 08:49:09 [all] step 5950/18464 loss=8.1322 ppl=3402.11 lr=4.829e-04 | 954 tok/s | 390M done | eta 9d22h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 08:49:09 [all] projected finish 14d16h05m exceeds the 168h deadline |
| 2026-09-17 09:00:32 [all] step 5960/18464 loss=8.1305 ppl=3396.34 lr=4.825e-04 | 959 tok/s | 391M done | eta 9d21h16m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 09:00:32 [all] projected finish 14d14h48m exceeds the 168h deadline |
| 2026-09-17 09:11:56 [all] step 5970/18464 loss=8.1141 ppl=3341.33 lr=4.821e-04 | 958 tok/s | 391M done | eta 9d21h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 09:11:56 [all] projected finish 14d15h11m exceeds the 168h deadline |
| 2026-09-17 09:17:41 [all] checkpoint @ step 5975 (392M tokens, 4d17h49m elapsed) |
| 2026-09-17 09:23:23 [all] step 5980/18464 loss=8.1313 ppl=3399.24 lr=4.817e-04 | 955 tok/s | 392M done | eta 9d22h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 09:23:23 [all] projected finish 14d15h57m exceeds the 168h deadline |
| 2026-09-17 09:34:48 [all] step 5990/18464 loss=8.1167 ppl=3350.00 lr=4.814e-04 | 957 tok/s | 393M done | eta 9d21h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 09:34:48 [all] projected finish 14d15h23m exceeds the 168h deadline |
| 2026-09-17 09:46:15 [all] step 6000/18464 loss=8.1291 ppl=3391.87 lr=4.810e-04 | 954 tok/s | 393M done | eta 9d21h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 09:46:15 [all] projected finish 14d16h08m exceeds the 168h deadline |
| 2026-09-17 09:47:06 [all] eval step 6000/18464 loss=8.1226 ppl=3369.89 (best 8.1226) |
| 2026-09-17 09:47:08 [all] sample @ 6000: ' (\n\n for the the for the the the the to the the the the the,-- the be the the the the the the the the the the the the the the the the the the the the the the a the of the' |
| 2026-09-17 09:47:08 [all] checkpoint @ step 6000 (393M tokens, 4d18h18m elapsed) |
| 2026-09-17 09:58:38 [all] step 6010/18464 loss=8.1144 ppl=3342.15 lr=4.806e-04 | 882 tok/s | 394M done | eta 10d17h05m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 09:58:38 [all] projected finish 15d11h35m exceeds the 168h deadline |
| 2026-09-17 10:10:11 [all] step 6020/18464 loss=8.1419 ppl=3435.48 lr=4.802e-04 | 946 tok/s | 395M done | eta 9d23h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 10:10:11 [all] projected finish 14d18h16m exceeds the 168h deadline |
| 2026-09-17 10:17:09 [all] checkpoint @ step 6026 (395M tokens, 4d18h48m elapsed) |
| 2026-09-17 10:21:46 [all] step 6030/18464 loss=8.1204 ppl=3362.48 lr=4.798e-04 | 942 tok/s | 395M done | eta 10d00h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 10:21:46 [all] projected finish 14d19h06m exceeds the 168h deadline |
| 2026-09-17 10:33:20 [all] step 6040/18464 loss=8.1257 ppl=3380.24 lr=4.794e-04 | 945 tok/s | 396M done | eta 9d23h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 10:33:20 [all] projected finish 14d18h23m exceeds the 168h deadline |
| 2026-09-17 10:44:51 [all] step 6050/18464 loss=8.1326 ppl=3403.61 lr=4.790e-04 | 948 tok/s | 396M done | eta 9d22h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 10:44:51 [all] projected finish 14d17h35m exceeds the 168h deadline |
| 2026-09-17 10:47:11 [all] checkpoint @ step 6052 (397M tokens, 4d19h18m elapsed) |
| 2026-09-17 10:56:26 [all] step 6060/18464 loss=8.1106 ppl=3329.71 lr=4.786e-04 | 943 tok/s | 397M done | eta 9d23h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 10:56:26 [all] projected finish 14d18h55m exceeds the 168h deadline |
| 2026-09-17 11:08:05 [all] step 6070/18464 loss=8.1207 ppl=3363.30 lr=4.782e-04 | 937 tok/s | 398M done | eta 10d00h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 11:08:05 [all] projected finish 14d20h20m exceeds the 168h deadline |
| 2026-09-17 11:17:26 [all] checkpoint @ step 6078 (398M tokens, 4d19h48m elapsed) |
| 2026-09-17 11:19:46 [all] step 6080/18464 loss=8.1279 ppl=3387.62 lr=4.778e-04 | 936 tok/s | 398M done | eta 10d00h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 11:19:46 [all] projected finish 14d20h48m exceeds the 168h deadline |
| 2026-09-17 11:31:23 [all] step 6090/18464 loss=8.1248 ppl=3377.16 lr=4.774e-04 | 940 tok/s | 399M done | eta 9d23h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 11:31:23 [all] projected finish 14d19h39m exceeds the 168h deadline |
| 2026-09-17 11:42:58 [all] step 6100/18464 loss=8.1257 ppl=3380.27 lr=4.771e-04 | 942 tok/s | 400M done | eta 9d22h49m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 11:42:58 [all] projected finish 14d19h03m exceeds the 168h deadline |
| 2026-09-17 11:47:38 [all] checkpoint @ step 6104 (400M tokens, 4d20h18m elapsed) |
| 2026-09-17 11:54:37 [all] step 6110/18464 loss=8.1113 ppl=3331.95 lr=4.767e-04 | 938 tok/s | 400M done | eta 9d23h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 11:54:37 [all] projected finish 14d20h11m exceeds the 168h deadline |
| 2026-09-17 12:06:18 [all] step 6120/18464 loss=8.1178 ppl=3353.61 lr=4.763e-04 | 934 tok/s | 401M done | eta 10d00h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 12:06:18 [all] projected finish 14d21h06m exceeds the 168h deadline |
| 2026-09-17 12:18:00 [all] step 6130/18464 loss=8.1328 ppl=3404.45 lr=4.759e-04 | 933 tok/s | 402M done | eta 10d00h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 12:18:00 [all] projected finish 14d21h25m exceeds the 168h deadline |
| 2026-09-17 12:18:01 [all] checkpoint @ step 6130 (402M tokens, 4d20h49m elapsed) |
| 2026-09-17 12:29:44 [all] step 6140/18464 loss=8.1409 ppl=3431.88 lr=4.755e-04 | 931 tok/s | 402M done | eta 10d00h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 12:29:44 [all] projected finish 14d22h00m exceeds the 168h deadline |
| 2026-09-17 12:41:30 [all] step 6150/18464 loss=8.1168 ppl=3350.26 lr=4.751e-04 | 929 tok/s | 403M done | eta 10d01h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 12:41:30 [all] projected finish 14d22h34m exceeds the 168h deadline |
| 2026-09-17 12:48:32 [all] checkpoint @ step 6156 (403M tokens, 4d21h19m elapsed) |
| 2026-09-17 12:53:15 [all] step 6160/18464 loss=8.1237 ppl=3373.60 lr=4.747e-04 | 930 tok/s | 404M done | eta 10d00h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 12:53:15 [all] projected finish 14d22h21m exceeds the 168h deadline |
| 2026-09-17 13:05:03 [all] step 6170/18464 loss=8.1340 ppl=3408.27 lr=4.743e-04 | 926 tok/s | 404M done | eta 10d01h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 13:05:03 [all] projected finish 14d23h25m exceeds the 168h deadline |
| 2026-09-17 13:16:57 [all] step 6180/18464 loss=8.1258 ppl=3380.56 lr=4.739e-04 | 917 tok/s | 405M done | eta 10d03h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 13:16:57 [all] projected finish 15d01h33m exceeds the 168h deadline |
| 2026-09-17 13:19:24 [all] checkpoint @ step 6182 (405M tokens, 4d21h50m elapsed) |
| 2026-09-17 13:29:02 [all] step 6190/18464 loss=8.1329 ppl=3404.70 lr=4.735e-04 | 904 tok/s | 406M done | eta 10d07h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 13:29:02 [all] projected finish 15d05h08m exceeds the 168h deadline |
| 2026-09-17 13:41:03 [all] step 6200/18464 loss=8.1269 ppl=3384.28 lr=4.731e-04 | 910 tok/s | 406M done | eta 10d05h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 13:41:03 [all] projected finish 15d03h38m exceeds the 168h deadline |
| 2026-09-17 13:49:29 [all] checkpoint @ step 6207 (407M tokens, 4d22h20m elapsed) |
| 2026-09-17 13:53:06 [all] step 6210/18464 loss=8.1197 ppl=3359.95 lr=4.727e-04 | 906 tok/s | 407M done | eta 10d06h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 13:53:06 [all] projected finish 15d04h44m exceeds the 168h deadline |
| 2026-09-17 14:05:11 [all] step 6220/18464 loss=8.1253 ppl=3378.79 lr=4.723e-04 | 904 tok/s | 408M done | eta 10d06h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 14:05:11 [all] projected finish 15d05h05m exceeds the 168h deadline |
| 2026-09-17 14:17:21 [all] step 6230/18464 loss=8.1130 ppl=3337.49 lr=4.719e-04 | 898 tok/s | 408M done | eta 10d08h08m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 14:17:21 [all] projected finish 15d06h57m exceeds the 168h deadline |
| 2026-09-17 14:19:49 [all] checkpoint @ step 6232 (408M tokens, 4d22h51m elapsed) |
| 2026-09-17 14:29:35 [all] step 6240/18464 loss=8.1198 ppl=3360.24 lr=4.715e-04 | 893 tok/s | 409M done | eta 10d09h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 14:29:35 [all] projected finish 15d08h15m exceeds the 168h deadline |
| 2026-09-17 14:41:49 [all] step 6250/18464 loss=8.1135 ppl=3339.39 lr=4.711e-04 | 893 tok/s | 410M done | eta 10d09h05m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 14:41:49 [all] projected finish 15d08h19m exceeds the 168h deadline |
| 2026-09-17 14:50:29 [all] checkpoint @ step 6257 (410M tokens, 4d23h21m elapsed) |
| 2026-09-17 14:54:11 [all] step 6260/18464 loss=8.1336 ppl=3406.96 lr=4.707e-04 | 884 tok/s | 410M done | eta 10d11h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 14:54:11 [all] projected finish 15d10h40m exceeds the 168h deadline |
| 2026-09-17 15:06:38 [all] step 6270/18464 loss=8.1431 ppl=3439.58 lr=4.703e-04 | 876 tok/s | 411M done | eta 10d13h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 15:06:38 [all] projected finish 15d12h55m exceeds the 168h deadline |
| 2026-09-17 15:19:11 [all] step 6280/18464 loss=8.1102 ppl=3328.17 lr=4.699e-04 | 871 tok/s | 412M done | eta 10d14h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 15:19:11 [all] projected finish 15d14h26m exceeds the 168h deadline |
| 2026-09-17 15:21:42 [all] checkpoint @ step 6282 (412M tokens, 4d23h53m elapsed) |
| 2026-09-17 15:31:45 [all] step 6290/18464 loss=8.1323 ppl=3402.75 lr=4.695e-04 | 868 tok/s | 412M done | eta 10d15h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 15:31:45 [all] projected finish 15d15h14m exceeds the 168h deadline |
| 2026-09-17 15:44:25 [all] step 6300/18464 loss=8.1154 ppl=3345.57 lr=4.691e-04 | 863 tok/s | 413M done | eta 10d16h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 15:44:25 [all] projected finish 15d16h58m exceeds the 168h deadline |
| 2026-09-17 15:52:03 [all] checkpoint @ step 6306 (413M tokens, 5d00h23m elapsed) |
| 2026-09-17 15:57:05 [all] step 6310/18464 loss=8.1229 ppl=3370.81 lr=4.687e-04 | 862 tok/s | 414M done | eta 10d16h43m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 15:57:05 [all] projected finish 15d17h12m exceeds the 168h deadline |
| 2026-09-17 16:09:49 [all] step 6320/18464 loss=8.1195 ppl=3359.43 lr=4.683e-04 | 859 tok/s | 414M done | eta 10d17h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 16:09:49 [all] projected finish 15d18h10m exceeds the 168h deadline |
| 2026-09-17 16:22:47 [all] step 6330/18464 loss=8.1331 ppl=3405.22 lr=4.679e-04 | 842 tok/s | 415M done | eta 10d22h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 16:22:47 [all] projected finish 15d23h06m exceeds the 168h deadline |
| 2026-09-17 16:22:47 [all] checkpoint @ step 6330 (415M tokens, 5d00h54m elapsed) |
| 2026-09-17 16:35:44 [all] step 6340/18464 loss=8.1277 ppl=3387.11 lr=4.675e-04 | 843 tok/s | 415M done | eta 10d21h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 16:35:44 [all] projected finish 15d22h58m exceeds the 168h deadline |
| 2026-09-17 16:48:43 [all] step 6350/18464 loss=8.1286 ppl=3390.12 lr=4.671e-04 | 841 tok/s | 416M done | eta 10d22h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 16:48:43 [all] projected finish 15d23h31m exceeds the 168h deadline |
| 2026-09-17 16:53:57 [all] checkpoint @ step 6354 (416M tokens, 5d01h25m elapsed) |
| 2026-09-17 17:01:48 [all] step 6360/18464 loss=8.1288 ppl=3390.85 lr=4.667e-04 | 835 tok/s | 417M done | eta 10d23h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 17:01:48 [all] projected finish 16d01h20m exceeds the 168h deadline |
| 2026-09-17 17:14:54 [all] step 6370/18464 loss=8.1192 ppl=3358.41 lr=4.663e-04 | 833 tok/s | 417M done | eta 11d00h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 17:14:54 [all] projected finish 16d01h55m exceeds the 168h deadline |
| 2026-09-17 17:24:11 [all] checkpoint @ step 6377 (418M tokens, 5d01h55m elapsed) |
| 2026-09-17 17:28:10 [all] step 6380/18464 loss=8.1286 ppl=3390.15 lr=4.659e-04 | 824 tok/s | 418M done | eta 11d02h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 17:28:10 [all] projected finish 16d04h57m exceeds the 168h deadline |
| 2026-09-17 17:41:30 [all] step 6390/18464 loss=8.1244 ppl=3375.75 lr=4.655e-04 | 819 tok/s | 419M done | eta 11d04h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 17:41:30 [all] projected finish 16d06h33m exceeds the 168h deadline |
| 2026-09-17 17:55:05 [all] step 6400/18464 loss=8.1407 ppl=3431.47 lr=4.651e-04 | 804 tok/s | 419M done | eta 11d09h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 17:55:05 [all] projected finish 16d11h43m exceeds the 168h deadline |
| 2026-09-17 17:55:06 [all] checkpoint @ step 6400 (419M tokens, 5d02h26m elapsed) |
| 2026-09-17 18:08:44 [all] step 6410/18464 loss=8.1179 ppl=3353.98 lr=4.647e-04 | 801 tok/s | 420M done | eta 11d10h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 18:08:44 [all] projected finish 16d12h43m exceeds the 168h deadline |
| 2026-09-17 18:22:25 [all] step 6420/18464 loss=8.1322 ppl=3402.15 lr=4.643e-04 | 798 tok/s | 421M done | eta 11d10h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 18:22:25 [all] projected finish 16d13h35m exceeds the 168h deadline |
| 2026-09-17 18:25:11 [all] checkpoint @ step 6422 (421M tokens, 5d02h56m elapsed) |
| 2026-09-17 18:36:05 [all] step 6430/18464 loss=8.1316 ppl=3400.08 lr=4.638e-04 | 799 tok/s | 421M done | eta 11d10h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 18:36:05 [all] projected finish 16d13h19m exceeds the 168h deadline |
| 2026-09-17 18:49:46 [all] step 6440/18464 loss=8.1306 ppl=3396.98 lr=4.634e-04 | 798 tok/s | 422M done | eta 11d10h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 18:49:46 [all] projected finish 16d13h38m exceeds the 168h deadline |
| 2026-09-17 18:55:13 [all] checkpoint @ step 6444 (422M tokens, 5d03h26m elapsed) |
| 2026-09-17 19:03:26 [all] step 6450/18464 loss=8.1152 ppl=3344.79 lr=4.630e-04 | 800 tok/s | 423M done | eta 11d09h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 19:03:26 [all] projected finish 16d13h01m exceeds the 168h deadline |
| 2026-09-17 19:17:09 [all] step 6460/18464 loss=8.1315 ppl=3399.82 lr=4.626e-04 | 796 tok/s | 423M done | eta 11d10h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 19:17:09 [all] projected finish 16d14h13m exceeds the 168h deadline |
| 2026-09-17 19:25:24 [all] checkpoint @ step 6466 (424M tokens, 5d03h56m elapsed) |
| 2026-09-17 19:30:55 [all] step 6470/18464 loss=8.1353 ppl=3412.92 lr=4.622e-04 | 793 tok/s | 424M done | eta 11d11h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 19:30:55 [all] projected finish 16d15h23m exceeds the 168h deadline |
| 2026-09-17 19:44:49 [all] step 6480/18464 loss=8.1470 ppl=3453.17 lr=4.618e-04 | 786 tok/s | 425M done | eta 11d13h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 19:44:49 [all] projected finish 16d17h50m exceeds the 168h deadline |
| 2026-09-17 19:56:01 [all] checkpoint @ step 6488 (425M tokens, 5d04h27m elapsed) |
| 2026-09-17 19:58:51 [all] step 6490/18464 loss=8.1214 ppl=3365.73 lr=4.614e-04 | 778 tok/s | 425M done | eta 11d16h16m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 19:58:51 [all] projected finish 16d20h46m exceeds the 168h deadline |
| 2026-09-17 20:12:50 [all] step 6500/18464 loss=8.1363 ppl=3416.38 lr=4.610e-04 | 782 tok/s | 426M done | eta 11d14h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 20:12:50 [all] projected finish 16d19h22m exceeds the 168h deadline |
| 2026-09-17 20:13:53 [all] eval step 6500/18464 loss=8.1224 ppl=3369.15 (best 8.1224) |
| 2026-09-17 20:26:22 [all] checkpoint @ step 6509 (427M tokens, 5d04h57m elapsed) |
| 2026-09-17 20:27:51 [all] step 6510/18464 loss=8.1264 ppl=3382.73 lr=4.606e-04 | 727 tok/s | 427M done | eta 12d11h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 20:27:51 [all] projected finish 17d16h13m exceeds the 168h deadline |
| 2026-09-17 20:42:45 [all] step 6520/18464 loss=8.1311 ppl=3398.42 lr=4.602e-04 | 733 tok/s | 427M done | eta 12d08h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 20:42:45 [all] projected finish 17d13h42m exceeds the 168h deadline |
| 2026-09-17 20:57:31 [all] step 6530/18464 loss=8.1163 ppl=3348.56 lr=4.598e-04 | 739 tok/s | 428M done | eta 12d05h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 20:57:31 [all] projected finish 17d11h20m exceeds the 168h deadline |
| 2026-09-17 20:57:32 [all] checkpoint @ step 6530 (428M tokens, 5d05h28m elapsed) |
| 2026-09-17 21:12:13 [all] step 6540/18464 loss=8.1217 ppl=3366.64 lr=4.593e-04 | 743 tok/s | 429M done | eta 12d04h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 21:12:13 [all] projected finish 17d09h46m exceeds the 168h deadline |
| 2026-09-17 21:26:54 [all] step 6550/18464 loss=8.1196 ppl=3359.53 lr=4.589e-04 | 744 tok/s | 429M done | eta 12d03h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 21:26:54 [all] projected finish 17d09h38m exceeds the 168h deadline |
| 2026-09-17 21:28:23 [all] checkpoint @ step 6551 (429M tokens, 5d05h59m elapsed) |
| 2026-09-17 21:41:34 [all] step 6560/18464 loss=8.1246 ppl=3376.67 lr=4.585e-04 | 745 tok/s | 430M done | eta 12d02h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 21:41:34 [all] projected finish 17d09h03m exceeds the 168h deadline |
| 2026-09-17 21:56:07 [all] step 6570/18464 loss=8.1292 ppl=3392.15 lr=4.581e-04 | 751 tok/s | 431M done | eta 12d00h26m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 21:56:07 [all] projected finish 17d06h53m exceeds the 168h deadline |
| 2026-09-17 21:59:02 [all] checkpoint @ step 6572 (431M tokens, 5d06h30m elapsed) |
| 2026-09-17 22:10:40 [all] step 6580/18464 loss=8.1357 ppl=3414.29 lr=4.577e-04 | 750 tok/s | 431M done | eta 12d00h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 22:10:40 [all] projected finish 17d07h00m exceeds the 168h deadline |
| 2026-09-17 22:25:22 [all] step 6590/18464 loss=8.1139 ppl=3340.54 lr=4.573e-04 | 743 tok/s | 432M done | eta 12d03h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 22:25:22 [all] projected finish 17d09h58m exceeds the 168h deadline |
| 2026-09-17 22:29:54 [all] checkpoint @ step 6593 (432M tokens, 5d07h01m elapsed) |
| 2026-09-17 22:40:29 [all] step 6600/18464 loss=8.1035 ppl=3306.04 lr=4.569e-04 | 723 tok/s | 433M done | eta 12d10h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 22:40:29 [all] projected finish 17d17h51m exceeds the 168h deadline |
| 2026-09-17 22:55:49 [all] step 6610/18464 loss=8.1358 ppl=3414.51 lr=4.565e-04 | 712 tok/s | 433M done | eta 12d15h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 22:55:49 [all] projected finish 17d22h38m exceeds the 168h deadline |
| 2026-09-17 23:00:26 [all] checkpoint @ step 6613 (433M tokens, 5d07h31m elapsed) |
| 2026-09-17 23:11:20 [all] step 6620/18464 loss=8.1069 ppl=3317.28 lr=4.560e-04 | 704 tok/s | 434M done | eta 12d18h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 23:11:20 [all] projected finish 18d01h46m exceeds the 168h deadline |
| 2026-09-17 23:27:04 [all] step 6630/18464 loss=8.1290 ppl=3391.56 lr=4.556e-04 | 694 tok/s | 435M done | eta 12d22h31m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 23:27:04 [all] projected finish 18d06h29m exceeds the 168h deadline |
| 2026-09-17 23:31:53 [all] checkpoint @ step 6633 (435M tokens, 5d08h03m elapsed) |
| 2026-09-17 23:43:05 [all] step 6640/18464 loss=8.1260 ppl=3381.34 lr=4.552e-04 | 682 tok/s | 435M done | eta 13d03h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 23:43:05 [all] projected finish 18d11h49m exceeds the 168h deadline |
| 2026-09-17 23:59:08 [all] step 6650/18464 loss=8.1074 ppl=3318.93 lr=4.548e-04 | 681 tok/s | 436M done | eta 13d03h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-17 23:59:08 [all] projected finish 18d12h20m exceeds the 168h deadline |
| 2026-09-18 00:02:20 [all] checkpoint @ step 6652 (436M tokens, 5d08h33m elapsed) |
| 2026-09-18 00:15:13 [all] step 6660/18464 loss=8.1009 ppl=3297.43 lr=4.544e-04 | 679 tok/s | 436M done | eta 13d04h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 00:15:13 [all] projected finish 18d13h08m exceeds the 168h deadline |
| 2026-09-18 00:31:20 [all] step 6670/18464 loss=8.1205 ppl=3362.63 lr=4.540e-04 | 677 tok/s | 437M done | eta 13d05h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 00:31:20 [all] projected finish 18d14h07m exceeds the 168h deadline |
| 2026-09-18 00:32:58 [all] checkpoint @ step 6671 (437M tokens, 5d09h04m elapsed) |
| 2026-09-18 00:47:35 [all] step 6680/18464 loss=8.1274 ppl=3385.91 lr=4.535e-04 | 673 tok/s | 438M done | eta 13d06h58m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 00:47:35 [all] projected finish 18d16h17m exceeds the 168h deadline |
| 2026-09-18 01:03:47 [all] step 6690/18464 loss=8.1186 ppl=3356.31 lr=4.531e-04 | 674 tok/s | 438M done | eta 13d06h00m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 01:03:47 [all] projected finish 18d15h36m exceeds the 168h deadline |
| 2026-09-18 01:03:48 [all] checkpoint @ step 6690 (438M tokens, 5d09h35m elapsed) |
| 2026-09-18 01:19:56 [all] step 6700/18464 loss=8.1154 ppl=3345.58 lr=4.527e-04 | 676 tok/s | 439M done | eta 13d04h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 01:19:56 [all] projected finish 18d14h28m exceeds the 168h deadline |
| 2026-09-18 01:34:23 [all] checkpoint @ step 6709 (440M tokens, 5d10h05m elapsed) |
| 2026-09-18 01:36:00 [all] step 6710/18464 loss=8.1141 ppl=3341.16 lr=4.523e-04 | 680 tok/s | 440M done | eta 13d02h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 01:36:00 [all] projected finish 18d12h57m exceeds the 168h deadline |
| 2026-09-18 01:52:09 [all] step 6720/18464 loss=8.1193 ppl=3358.76 lr=4.519e-04 | 676 tok/s | 440M done | eta 13d04h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 01:52:09 [all] projected finish 18d14h27m exceeds the 168h deadline |
| 2026-09-18 02:05:08 [all] checkpoint @ step 6728 (441M tokens, 5d10h36m elapsed) |
| 2026-09-18 02:08:22 [all] step 6730/18464 loss=8.1299 ppl=3394.42 lr=4.515e-04 | 674 tok/s | 441M done | eta 13d05h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 02:08:22 [all] projected finish 18d15h43m exceeds the 168h deadline |
| 2026-09-18 02:24:41 [all] step 6740/18464 loss=8.1302 ppl=3395.31 lr=4.510e-04 | 670 tok/s | 442M done | eta 13d06h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 02:24:41 [all] projected finish 18d17h37m exceeds the 168h deadline |
| 2026-09-18 02:36:05 [all] checkpoint @ step 6747 (442M tokens, 5d11h07m elapsed) |
| 2026-09-18 02:40:58 [all] step 6750/18464 loss=8.1326 ppl=3403.70 lr=4.506e-04 | 670 tok/s | 442M done | eta 13d06h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 02:40:58 [all] projected finish 18d17h14m exceeds the 168h deadline |
| 2026-09-18 02:57:07 [all] step 6760/18464 loss=8.1266 ppl=3383.44 lr=4.502e-04 | 677 tok/s | 443M done | eta 13d02h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 02:57:07 [all] projected finish 18d14h24m exceeds the 168h deadline |
| 2026-09-18 03:06:47 [all] checkpoint @ step 6766 (443M tokens, 5d11h38m elapsed) |
| 2026-09-18 03:13:11 [all] step 6770/18464 loss=8.1178 ppl=3353.77 lr=4.498e-04 | 680 tok/s | 444M done | eta 13d01h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 03:13:11 [all] projected finish 18d12h59m exceeds the 168h deadline |
| 2026-09-18 03:29:14 [all] step 6780/18464 loss=8.1280 ppl=3388.18 lr=4.494e-04 | 680 tok/s | 444M done | eta 13d00h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 03:29:14 [all] projected finish 18d12h37m exceeds the 168h deadline |
| 2026-09-18 03:37:15 [all] checkpoint @ step 6785 (445M tokens, 5d12h08m elapsed) |
| 2026-09-18 03:45:14 [all] step 6790/18464 loss=8.1319 ppl=3401.13 lr=4.489e-04 | 683 tok/s | 445M done | eta 12d23h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 03:45:14 [all] projected finish 18d11h29m exceeds the 168h deadline |
| 2026-09-18 04:01:11 [all] step 6800/18464 loss=8.1063 ppl=3315.35 lr=4.485e-04 | 684 tok/s | 446M done | eta 12d22h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 04:01:11 [all] projected finish 18d10h45m exceeds the 168h deadline |
| 2026-09-18 04:07:36 [all] checkpoint @ step 6804 (446M tokens, 5d12h38m elapsed) |
| 2026-09-18 04:17:16 [all] step 6810/18464 loss=8.1170 ppl=3350.84 lr=4.481e-04 | 680 tok/s | 446M done | eta 13d00h12m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 04:17:16 [all] projected finish 18d13h01m exceeds the 168h deadline |
| 2026-09-18 04:33:21 [all] step 6820/18464 loss=8.1105 ppl=3329.26 lr=4.477e-04 | 679 tok/s | 447M done | eta 13d00h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 04:33:21 [all] projected finish 18d13h07m exceeds the 168h deadline |
| 2026-09-18 04:38:08 [all] checkpoint @ step 6823 (447M tokens, 5d13h09m elapsed) |
| 2026-09-18 04:49:19 [all] step 6830/18464 loss=8.1253 ppl=3379.03 lr=4.473e-04 | 684 tok/s | 448M done | eta 12d21h49m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 04:49:19 [all] projected finish 18d11h10m exceeds the 168h deadline |
| 2026-09-18 05:05:13 [all] step 6840/18464 loss=8.1065 ppl=3315.85 lr=4.468e-04 | 687 tok/s | 448M done | eta 12d19h48m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 05:05:13 [all] projected finish 18d09h25m exceeds the 168h deadline |
| 2026-09-18 05:08:23 [all] checkpoint @ step 6842 (448M tokens, 5d13h39m elapsed) |
| 2026-09-18 05:21:07 [all] step 6850/18464 loss=8.1119 ppl=3334.02 lr=4.464e-04 | 687 tok/s | 449M done | eta 12d19h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 05:21:07 [all] projected finish 18d09h38m exceeds the 168h deadline |
| 2026-09-18 05:37:00 [all] step 6860/18464 loss=8.1297 ppl=3393.87 lr=4.460e-04 | 687 tok/s | 450M done | eta 12d19h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 05:37:00 [all] projected finish 18d09h30m exceeds the 168h deadline |
| 2026-09-18 05:38:35 [all] checkpoint @ step 6861 (450M tokens, 5d14h09m elapsed) |
| 2026-09-18 05:52:48 [all] step 6870/18464 loss=8.1279 ppl=3387.57 lr=4.456e-04 | 692 tok/s | 450M done | eta 12d17h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 05:52:48 [all] projected finish 18d07h30m exceeds the 168h deadline |
| 2026-09-18 06:08:32 [all] step 6880/18464 loss=8.1094 ppl=3325.66 lr=4.451e-04 | 694 tok/s | 451M done | eta 12d15h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 06:08:32 [all] projected finish 18d06h37m exceeds the 168h deadline |
| 2026-09-18 06:10:06 [all] checkpoint @ step 6881 (451M tokens, 5d14h41m elapsed) |
| 2026-09-18 06:23:57 [all] step 6890/18464 loss=8.1075 ppl=3319.27 lr=4.447e-04 | 709 tok/s | 452M done | eta 12d09h21m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 06:23:57 [all] projected finish 18d00h17m exceeds the 168h deadline |
| 2026-09-18 06:39:20 [all] step 6900/18464 loss=8.1190 ppl=3357.64 lr=4.443e-04 | 710 tok/s | 452M done | eta 12d08h34m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 06:39:20 [all] projected finish 17d23h45m exceeds the 168h deadline |
| 2026-09-18 06:40:49 [all] checkpoint @ step 6901 (452M tokens, 5d15h12m elapsed) |
| 2026-09-18 06:53:53 [all] step 6910/18464 loss=8.1224 ppl=3369.11 lr=4.439e-04 | 751 tok/s | 453M done | eta 11d16h05m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 06:53:53 [all] projected finish 17d07h31m exceeds the 168h deadline |
| 2026-09-18 07:08:00 [all] step 6920/18464 loss=8.1244 ppl=3375.82 lr=4.434e-04 | 774 tok/s | 454M done | eta 11d07h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 07:08:00 [all] projected finish 16d23h18m exceeds the 168h deadline |
| 2026-09-18 07:12:11 [all] checkpoint @ step 6923 (454M tokens, 5d15h43m elapsed) |
| 2026-09-18 07:21:53 [all] step 6930/18464 loss=8.1246 ppl=3376.56 lr=4.430e-04 | 787 tok/s | 454M done | eta 11d02h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 07:21:53 [all] projected finish 16d18h45m exceeds the 168h deadline |
| 2026-09-18 07:35:41 [all] step 6940/18464 loss=8.1128 ppl=3336.89 lr=4.426e-04 | 792 tok/s | 455M done | eta 11d00h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 07:35:41 [all] projected finish 16d17h00m exceeds the 168h deadline |
| 2026-09-18 07:42:32 [all] checkpoint @ step 6945 (455M tokens, 5d16h13m elapsed) |
| 2026-09-18 07:49:23 [all] step 6950/18464 loss=8.1344 ppl=3409.66 lr=4.422e-04 | 797 tok/s | 455M done | eta 10d22h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 07:49:23 [all] projected finish 16d15h13m exceeds the 168h deadline |
| 2026-09-18 08:02:53 [all] step 6960/18464 loss=8.1125 ppl=3335.94 lr=4.417e-04 | 809 tok/s | 456M done | eta 10d18h49m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 08:02:53 [all] projected finish 16d11h23m exceeds the 168h deadline |
| 2026-09-18 08:13:36 [all] checkpoint @ step 6968 (457M tokens, 5d16h44m elapsed) |
| 2026-09-18 08:16:18 [all] step 6970/18464 loss=8.1148 ppl=3343.72 lr=4.413e-04 | 813 tok/s | 457M done | eta 10d17h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 08:16:18 [all] projected finish 16d10h02m exceeds the 168h deadline |
| 2026-09-18 08:29:39 [all] step 6980/18464 loss=8.1239 ppl=3374.07 lr=4.409e-04 | 819 tok/s | 457M done | eta 10d15h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 08:29:39 [all] projected finish 16d08h20m exceeds the 168h deadline |
| 2026-09-18 08:42:54 [all] step 6990/18464 loss=8.1019 ppl=3300.88 lr=4.405e-04 | 824 tok/s | 458M done | eta 10d13h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 08:42:54 [all] projected finish 16d06h42m exceeds the 168h deadline |
| 2026-09-18 08:44:13 [all] checkpoint @ step 6991 (458M tokens, 5d17h15m elapsed) |
| 2026-09-18 08:56:07 [all] step 7000/18464 loss=8.1271 ppl=3385.09 lr=4.400e-04 | 827 tok/s | 459M done | eta 10d12h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 08:56:07 [all] projected finish 16d05h54m exceeds the 168h deadline |
| 2026-09-18 08:57:05 [all] eval step 7000/18464 loss=8.1222 ppl=3368.37 (best 8.1222) |
| 2026-09-18 08:57:06 [all] checkpoint @ step 7000 (459M tokens, 5d17h28m elapsed) |
| 2026-09-18 09:10:17 [all] step 7010/18464 loss=8.1223 ppl=3368.70 lr=4.396e-04 | 771 tok/s | 459M done | eta 11d06h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 09:10:17 [all] projected finish 17d00h14m exceeds the 168h deadline |
| 2026-09-18 09:23:29 [all] step 7020/18464 loss=8.1293 ppl=3392.46 lr=4.392e-04 | 827 tok/s | 460M done | eta 10d11h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 09:23:29 [all] projected finish 16d05h41m exceeds the 168h deadline |
| 2026-09-18 09:27:25 [all] checkpoint @ step 7023 (460M tokens, 5d17h58m elapsed) |
| 2026-09-18 09:36:39 [all] step 7030/18464 loss=8.1150 ppl=3344.10 lr=4.387e-04 | 829 tok/s | 461M done | eta 10d10h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 09:36:39 [all] projected finish 16d05h05m exceeds the 168h deadline |
| 2026-09-18 09:49:51 [all] step 7040/18464 loss=8.1145 ppl=3342.75 lr=4.383e-04 | 827 tok/s | 461M done | eta 10d11h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 09:49:51 [all] projected finish 16d05h40m exceeds the 168h deadline |
| 2026-09-18 09:57:47 [all] checkpoint @ step 7046 (462M tokens, 5d18h29m elapsed) |
| 2026-09-18 10:03:07 [all] step 7050/18464 loss=8.1348 ppl=3411.28 lr=4.379e-04 | 823 tok/s | 462M done | eta 10d12h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 10:03:07 [all] projected finish 16d07h01m exceeds the 168h deadline |
| 2026-09-18 10:16:32 [all] step 7060/18464 loss=8.1211 ppl=3364.87 lr=4.375e-04 | 814 tok/s | 463M done | eta 10d15h00m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 10:16:32 [all] projected finish 16d09h48m exceeds the 168h deadline |
| 2026-09-18 10:28:57 [all] checkpoint @ step 7069 (463M tokens, 5d19h00m elapsed) |
| 2026-09-18 10:30:22 [all] step 7070/18464 loss=8.1131 ppl=3337.85 lr=4.370e-04 | 790 tok/s | 463M done | eta 10d22h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 10:30:22 [all] projected finish 16d17h35m exceeds the 168h deadline |
| 2026-09-18 10:44:27 [all] step 7080/18464 loss=8.1294 ppl=3392.89 lr=4.366e-04 | 776 tok/s | 464M done | eta 11d03h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 10:44:27 [all] projected finish 16d22h18m exceeds the 168h deadline |
| 2026-09-18 10:57:57 [all] step 7090/18464 loss=8.1136 ppl=3339.55 lr=4.362e-04 | 809 tok/s | 465M done | eta 10d15h58m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 10:57:57 [all] projected finish 16d11h28m exceeds the 168h deadline |
| 2026-09-18 10:59:19 [all] checkpoint @ step 7091 (465M tokens, 5d19h30m elapsed) |
| 2026-09-18 11:11:27 [all] step 7100/18464 loss=8.1146 ppl=3342.92 lr=4.357e-04 | 809 tok/s | 465M done | eta 10d15h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 11:11:27 [all] projected finish 16d11h23m exceeds the 168h deadline |
| 2026-09-18 11:24:57 [all] step 7110/18464 loss=8.1175 ppl=3352.78 lr=4.353e-04 | 809 tok/s | 466M done | eta 10d15h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 11:24:57 [all] projected finish 16d11h26m exceeds the 168h deadline |
| 2026-09-18 11:30:22 [all] checkpoint @ step 7114 (466M tokens, 5d20h01m elapsed) |
| 2026-09-18 11:38:35 [all] step 7120/18464 loss=8.1306 ppl=3396.83 lr=4.349e-04 | 801 tok/s | 467M done | eta 10d17h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 11:38:35 [all] projected finish 16d14h00m exceeds the 168h deadline |
| 2026-09-18 11:52:19 [all] step 7130/18464 loss=8.1170 ppl=3350.79 lr=4.344e-04 | 795 tok/s | 467M done | eta 10d19h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 11:52:19 [all] projected finish 16d15h57m exceeds the 168h deadline |
| 2026-09-18 12:00:33 [all] checkpoint @ step 7136 (468M tokens, 5d20h31m elapsed) |
| 2026-09-18 12:06:06 [all] step 7140/18464 loss=8.1410 ppl=3432.35 lr=4.340e-04 | 793 tok/s | 468M done | eta 10d19h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 12:06:06 [all] projected finish 16d16h37m exceeds the 168h deadline |
| 2026-09-18 12:20:12 [all] step 7150/18464 loss=8.1375 ppl=3420.42 lr=4.336e-04 | 775 tok/s | 469M done | eta 11d01h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 12:20:12 [all] projected finish 16d22h44m exceeds the 168h deadline |
| 2026-09-18 12:31:29 [all] checkpoint @ step 7158 (469M tokens, 5d21h02m elapsed) |
| 2026-09-18 12:34:19 [all] step 7160/18464 loss=8.1432 ppl=3439.83 lr=4.331e-04 | 774 tok/s | 469M done | eta 11d01h57m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 12:34:19 [all] projected finish 16d23h02m exceeds the 168h deadline |
| 2026-09-18 12:48:32 [all] step 7170/18464 loss=8.1156 ppl=3346.16 lr=4.327e-04 | 768 tok/s | 470M done | eta 11d03h42m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 12:48:32 [all] projected finish 17d01h02m exceeds the 168h deadline |
| 2026-09-18 13:02:50 [all] step 7180/18464 loss=8.1241 ppl=3374.93 lr=4.323e-04 | 764 tok/s | 471M done | eta 11d04h54m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 13:02:50 [all] projected finish 17d02h29m exceeds the 168h deadline |
| 2026-09-18 13:02:51 [all] checkpoint @ step 7180 (471M tokens, 5d21h34m elapsed) |
| 2026-09-18 13:17:16 [all] step 7190/18464 loss=8.1332 ppl=3405.72 lr=4.318e-04 | 757 tok/s | 471M done | eta 11d07h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 13:17:16 [all] projected finish 17d05h00m exceeds the 168h deadline |
| 2026-09-18 13:31:50 [all] step 7200/18464 loss=8.1172 ppl=3351.77 lr=4.314e-04 | 750 tok/s | 472M done | eta 11d09h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 13:31:50 [all] projected finish 17d07h17m exceeds the 168h deadline |
| 2026-09-18 13:33:17 [all] checkpoint @ step 7201 (472M tokens, 5d22h04m elapsed) |
| 2026-09-18 13:46:28 [all] step 7210/18464 loss=8.1236 ppl=3373.02 lr=4.310e-04 | 746 tok/s | 473M done | eta 11d10h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 13:46:28 [all] projected finish 17d08h47m exceeds the 168h deadline |
| 2026-09-18 14:01:11 [all] step 7220/18464 loss=8.1142 ppl=3341.46 lr=4.305e-04 | 742 tok/s | 473M done | eta 11d11h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 14:01:11 [all] projected finish 17d10h23m exceeds the 168h deadline |
| 2026-09-18 14:04:10 [all] checkpoint @ step 7222 (473M tokens, 5d22h35m elapsed) |
| 2026-09-18 14:16:02 [all] step 7230/18464 loss=8.1424 ppl=3437.27 lr=4.301e-04 | 735 tok/s | 474M done | eta 11d14h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 14:16:02 [all] projected finish 17d12h54m exceeds the 168h deadline |
| 2026-09-18 14:31:09 [all] step 7240/18464 loss=8.1340 ppl=3408.40 lr=4.297e-04 | 723 tok/s | 474M done | eta 11d18h43m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 14:31:09 [all] projected finish 17d17h45m exceeds the 168h deadline |
| 2026-09-18 14:35:42 [all] checkpoint @ step 7243 (475M tokens, 5d23h07m elapsed) |
| 2026-09-18 14:46:27 [all] step 7250/18464 loss=8.1223 ppl=3368.77 lr=4.292e-04 | 714 tok/s | 475M done | eta 11d21h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 14:46:27 [all] projected finish 17d21h12m exceeds the 168h deadline |
| 2026-09-18 15:02:02 [all] step 7260/18464 loss=8.1116 ppl=3332.77 lr=4.288e-04 | 701 tok/s | 476M done | eta 12d02h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 15:02:02 [all] projected finish 18d02h29m exceeds the 168h deadline |
| 2026-09-18 15:06:55 [all] checkpoint @ step 7263 (476M tokens, 5d23h38m elapsed) |
| 2026-09-18 15:18:06 [all] step 7270/18464 loss=8.1295 ppl=3393.24 lr=4.284e-04 | 679 tok/s | 476M done | eta 12d11h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 15:18:06 [all] projected finish 18d11h49m exceeds the 168h deadline |
| 2026-09-18 15:34:00 [all] step 7280/18464 loss=8.1157 ppl=3346.55 lr=4.279e-04 | 687 tok/s | 477M done | eta 12d08h11m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 15:34:00 [all] projected finish 18d08h17m exceeds the 168h deadline |
| 2026-09-18 15:37:09 [all] checkpoint @ step 7282 (477M tokens, 6d00h08m elapsed) |
| 2026-09-18 15:49:07 [all] step 7290/18464 loss=8.1117 ppl=3333.15 lr=4.275e-04 | 722 tok/s | 478M done | eta 11d17h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 15:49:07 [all] projected finish 17d17h53m exceeds the 168h deadline |
| 2026-09-18 16:03:21 [all] step 7300/18464 loss=8.1226 ppl=3369.84 lr=4.270e-04 | 767 tok/s | 478M done | eta 11d01h00m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 16:03:21 [all] projected finish 17d01h35m exceeds the 168h deadline |
| 2026-09-18 16:07:33 [all] checkpoint @ step 7303 (479M tokens, 6d00h38m elapsed) |
| 2026-09-18 16:17:20 [all] step 7310/18464 loss=8.1276 ppl=3386.74 lr=4.266e-04 | 782 tok/s | 479M done | eta 10d19h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 16:17:20 [all] projected finish 16d20h33m exceeds the 168h deadline |
| 2026-09-18 16:30:52 [all] step 7320/18464 loss=8.1376 ppl=3420.85 lr=4.262e-04 | 807 tok/s | 480M done | eta 10d11h31m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 16:30:52 [all] projected finish 16d12h33m exceeds the 168h deadline |
| 2026-09-18 16:38:52 [all] checkpoint @ step 7326 (480M tokens, 6d01h10m elapsed) |
| 2026-09-18 16:44:12 [all] step 7330/18464 loss=8.1173 ppl=3351.83 lr=4.257e-04 | 819 tok/s | 480M done | eta 10d07h23m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 16:44:12 [all] projected finish 16d08h39m exceeds the 168h deadline |
| 2026-09-18 16:57:18 [all] step 7340/18464 loss=8.1216 ppl=3366.49 lr=4.253e-04 | 834 tok/s | 481M done | eta 10d02h49m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 16:57:18 [all] projected finish 16d04h17m exceeds the 168h deadline |
| 2026-09-18 17:08:56 [all] checkpoint @ step 7349 (482M tokens, 6d01h40m elapsed) |
| 2026-09-18 17:10:13 [all] step 7350/18464 loss=8.1291 ppl=3391.66 lr=4.248e-04 | 846 tok/s | 482M done | eta 9d23h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 17:10:13 [all] projected finish 16d00h51m exceeds the 168h deadline |
| 2026-09-18 17:22:59 [all] step 7360/18464 loss=8.1262 ppl=3381.77 lr=4.244e-04 | 855 tok/s | 482M done | eta 9d20h24m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 17:22:59 [all] projected finish 15d22h18m exceeds the 168h deadline |
| 2026-09-18 17:35:55 [all] step 7370/18464 loss=8.1133 ppl=3338.48 lr=4.240e-04 | 845 tok/s | 483M done | eta 9d23h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 17:35:55 [all] projected finish 16d01h14m exceeds the 168h deadline |
| 2026-09-18 17:40:02 [all] checkpoint @ step 7373 (483M tokens, 6d02h11m elapsed) |
| 2026-09-18 17:49:48 [all] step 7380/18464 loss=8.1423 ppl=3436.97 lr=4.235e-04 | 787 tok/s | 484M done | eta 10d16h31m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 17:49:48 [all] projected finish 16d18h52m exceeds the 168h deadline |
| 2026-09-18 18:04:03 [all] step 7390/18464 loss=8.1169 ppl=3350.64 lr=4.231e-04 | 767 tok/s | 484M done | eta 10d22h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 18:04:03 [all] projected finish 17d01h28m exceeds the 168h deadline |
| 2026-09-18 18:10:58 [all] checkpoint @ step 7395 (485M tokens, 6d02h42m elapsed) |
| 2026-09-18 18:17:39 [all] step 7400/18464 loss=8.1198 ppl=3360.35 lr=4.227e-04 | 803 tok/s | 485M done | eta 10d10h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 18:17:39 [all] projected finish 16d13h35m exceeds the 168h deadline |
| 2026-09-18 18:30:34 [all] step 7410/18464 loss=8.1252 ppl=3378.62 lr=4.222e-04 | 845 tok/s | 486M done | eta 9d22h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 18:30:34 [all] projected finish 16d01h12m exceeds the 168h deadline |
| 2026-09-18 18:41:56 [all] checkpoint @ step 7419 (486M tokens, 6d03h13m elapsed) |
| 2026-09-18 18:43:11 [all] step 7420/18464 loss=8.1070 ppl=3317.47 lr=4.218e-04 | 866 tok/s | 486M done | eta 9d16h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 18:43:11 [all] projected finish 15d19h21m exceeds the 168h deadline |
| 2026-09-18 18:56:17 [all] step 7430/18464 loss=8.1219 ppl=3367.43 lr=4.213e-04 | 834 tok/s | 487M done | eta 10d00h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 18:56:17 [all] projected finish 16d04h26m exceeds the 168h deadline |
| 2026-09-18 19:09:57 [all] step 7440/18464 loss=8.1224 ppl=3369.15 lr=4.209e-04 | 800 tok/s | 488M done | eta 10d10h55m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 19:09:57 [all] projected finish 16d14h36m exceeds the 168h deadline |
| 2026-09-18 19:12:44 [all] checkpoint @ step 7442 (488M tokens, 6d03h44m elapsed) |
| 2026-09-18 19:24:01 [all] step 7450/18464 loss=8.1216 ppl=3366.26 lr=4.204e-04 | 776 tok/s | 488M done | eta 10d18h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 19:24:01 [all] projected finish 16d22h12m exceeds the 168h deadline |
| 2026-09-18 19:38:31 [all] step 7460/18464 loss=8.1154 ppl=3345.62 lr=4.200e-04 | 753 tok/s | 489M done | eta 11d01h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 19:38:31 [all] projected finish 17d06h06m exceeds the 168h deadline |
| 2026-09-18 19:42:48 [all] checkpoint @ step 7463 (489M tokens, 6d04h14m elapsed) |
| 2026-09-18 19:52:47 [all] step 7470/18464 loss=8.1205 ppl=3362.61 lr=4.196e-04 | 765 tok/s | 490M done | eta 10d21h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 19:52:47 [all] projected finish 17d01h54m exceeds the 168h deadline |
| 2026-09-18 20:07:05 [all] step 7480/18464 loss=8.1196 ppl=3359.62 lr=4.191e-04 | 764 tok/s | 490M done | eta 10d21h44m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 20:07:05 [all] projected finish 17d02h23m exceeds the 168h deadline |
| 2026-09-18 20:12:54 [all] checkpoint @ step 7484 (490M tokens, 6d04h44m elapsed) |
| 2026-09-18 20:21:32 [all] step 7490/18464 loss=8.1274 ppl=3386.13 lr=4.187e-04 | 756 tok/s | 491M done | eta 11d00h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 20:21:32 [all] projected finish 17d05h07m exceeds the 168h deadline |
| 2026-09-18 20:36:03 [all] step 7500/18464 loss=8.1160 ppl=3347.52 lr=4.182e-04 | 753 tok/s | 492M done | eta 11d01h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 20:36:03 [all] projected finish 17d06h14m exceeds the 168h deadline |
| 2026-09-18 20:37:08 [all] eval step 7500/18464 loss=8.1220 ppl=3367.61 (best 8.1220) |
| 2026-09-18 20:42:58 [all] checkpoint @ step 7504 (492M tokens, 6d05h14m elapsed) |
| 2026-09-18 20:51:45 [all] step 7510/18464 loss=8.1244 ppl=3375.90 lr=4.178e-04 | 696 tok/s | 492M done | eta 11d22h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 20:51:45 [all] projected finish 18d04h03m exceeds the 168h deadline |
| 2026-09-18 21:06:18 [all] step 7520/18464 loss=8.1234 ppl=3372.48 lr=4.173e-04 | 750 tok/s | 493M done | eta 11d01h28m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 21:06:18 [all] projected finish 17d07h06m exceeds the 168h deadline |
| 2026-09-18 21:13:28 [all] checkpoint @ step 7525 (493M tokens, 6d05h44m elapsed) |
| 2026-09-18 21:20:42 [all] step 7530/18464 loss=8.1367 ppl=3417.76 lr=4.169e-04 | 758 tok/s | 493M done | eta 10d22h31m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 21:20:42 [all] projected finish 17d04h23m exceeds the 168h deadline |
| 2026-09-18 21:35:16 [all] step 7540/18464 loss=8.1307 ppl=3397.09 lr=4.165e-04 | 750 tok/s | 494M done | eta 11d01h14m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 21:35:16 [all] projected finish 17d07h20m exceeds the 168h deadline |
| 2026-09-18 21:44:00 [all] checkpoint @ step 7546 (495M tokens, 6d06h15m elapsed) |
| 2026-09-18 21:49:55 [all] step 7550/18464 loss=8.1194 ppl=3358.94 lr=4.160e-04 | 746 tok/s | 495M done | eta 11d02h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 21:49:55 [all] projected finish 17d08h51m exceeds the 168h deadline |
| 2026-09-18 22:05:07 [all] step 7560/18464 loss=8.1185 ppl=3355.92 lr=4.156e-04 | 719 tok/s | 495M done | eta 11d12h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 22:05:07 [all] projected finish 17d18h41m exceeds the 168h deadline |
| 2026-09-18 22:14:27 [all] checkpoint @ step 7566 (496M tokens, 6d06h45m elapsed) |
| 2026-09-18 22:20:43 [all] step 7570/18464 loss=8.1159 ppl=3347.35 lr=4.151e-04 | 700 tok/s | 496M done | eta 11d19h13m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 22:20:43 [all] projected finish 18d02h05m exceeds the 168h deadline |
| 2026-09-18 22:36:29 [all] step 7580/18464 loss=8.1240 ppl=3374.54 lr=4.147e-04 | 692 tok/s | 497M done | eta 11d22h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 22:36:29 [all] projected finish 18d05h17m exceeds the 168h deadline |
| 2026-09-18 22:45:56 [all] checkpoint @ step 7586 (497M tokens, 6d07h17m elapsed) |
| 2026-09-18 22:52:13 [all] step 7590/18464 loss=8.1333 ppl=3406.18 lr=4.142e-04 | 694 tok/s | 497M done | eta 11d21h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 22:52:13 [all] projected finish 18d04h26m exceeds the 168h deadline |
| 2026-09-18 23:08:02 [all] step 7600/18464 loss=8.1279 ppl=3387.68 lr=4.138e-04 | 690 tok/s | 498M done | eta 11d22h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 23:08:02 [all] projected finish 18d06h08m exceeds the 168h deadline |
| 2026-09-18 23:17:31 [all] checkpoint @ step 7606 (498M tokens, 6d07h48m elapsed) |
| 2026-09-18 23:23:51 [all] step 7610/18464 loss=8.1346 ppl=3410.46 lr=4.133e-04 | 691 tok/s | 499M done | eta 11d21h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 23:23:51 [all] projected finish 18d05h48m exceeds the 168h deadline |
| 2026-09-18 23:39:48 [all] step 7620/18464 loss=8.1350 ppl=3411.70 lr=4.129e-04 | 685 tok/s | 499M done | eta 12d00h17m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 23:39:48 [all] projected finish 18d08h29m exceeds the 168h deadline |
| 2026-09-18 23:47:45 [all] checkpoint @ step 7625 (500M tokens, 6d08h19m elapsed) |
| 2026-09-18 23:55:40 [all] step 7630/18464 loss=8.1105 ppl=3329.10 lr=4.125e-04 | 689 tok/s | 500M done | eta 11d22h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-18 23:55:40 [all] projected finish 18d06h52m exceeds the 168h deadline |
| 2026-09-19 00:11:39 [all] step 7640/18464 loss=8.1193 ppl=3358.76 lr=4.120e-04 | 683 tok/s | 501M done | eta 12d00h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 00:11:39 [all] projected finish 18d09h15m exceeds the 168h deadline |
| 2026-09-19 00:18:08 [all] checkpoint @ step 7644 (501M tokens, 6d08h49m elapsed) |
| 2026-09-19 00:27:42 [all] step 7650/18464 loss=8.1381 ppl=3422.26 lr=4.116e-04 | 680 tok/s | 501M done | eta 12d01h19m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 00:27:42 [all] projected finish 18d10h18m exceeds the 168h deadline |
| 2026-09-19 00:43:48 [all] step 7660/18464 loss=8.1166 ppl=3349.63 lr=4.111e-04 | 679 tok/s | 502M done | eta 12d01h46m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 00:43:48 [all] projected finish 18d11h01m exceeds the 168h deadline |
| 2026-09-19 00:48:34 [all] checkpoint @ step 7663 (502M tokens, 6d09h19m elapsed) |
| 2026-09-19 00:59:50 [all] step 7670/18464 loss=8.1053 ppl=3311.81 lr=4.107e-04 | 681 tok/s | 503M done | eta 12d00h24m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 00:59:50 [all] projected finish 18d09h55m exceeds the 168h deadline |
| 2026-09-19 01:15:53 [all] step 7680/18464 loss=8.1310 ppl=3398.04 lr=4.102e-04 | 680 tok/s | 503M done | eta 12d00h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 01:15:53 [all] projected finish 18d10h19m exceeds the 168h deadline |
| 2026-09-19 01:19:07 [all] checkpoint @ step 7682 (503M tokens, 6d09h50m elapsed) |
| 2026-09-19 01:31:59 [all] step 7690/18464 loss=8.1125 ppl=3335.75 lr=4.098e-04 | 679 tok/s | 504M done | eta 12d01h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 01:31:59 [all] projected finish 18d11h05m exceeds the 168h deadline |
| 2026-09-19 01:48:07 [all] step 7700/18464 loss=8.1446 ppl=3444.81 lr=4.093e-04 | 677 tok/s | 505M done | eta 12d01h32m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 01:48:07 [all] projected finish 18d11h52m exceeds the 168h deadline |
| 2026-09-19 01:49:46 [all] checkpoint @ step 7701 (505M tokens, 6d10h21m elapsed) |
| 2026-09-19 02:04:13 [all] step 7710/18464 loss=8.1119 ppl=3333.84 lr=4.089e-04 | 679 tok/s | 505M done | eta 12d00h25m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 02:04:13 [all] projected finish 18d11h00m exceeds the 168h deadline |
| 2026-09-19 02:20:22 [all] step 7720/18464 loss=8.1265 ppl=3383.11 lr=4.084e-04 | 676 tok/s | 506M done | eta 12d01h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 02:20:22 [all] projected finish 18d12h01m exceeds the 168h deadline |
| 2026-09-19 02:20:22 [all] checkpoint @ step 7720 (506M tokens, 6d10h51m elapsed) |
| 2026-09-19 02:36:37 [all] step 7730/18464 loss=8.1346 ppl=3410.43 lr=4.080e-04 | 672 tok/s | 507M done | eta 12d02h45m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 02:36:37 [all] projected finish 18d13h53m exceeds the 168h deadline |
| 2026-09-19 02:51:12 [all] checkpoint @ step 7739 (507M tokens, 6d11h22m elapsed) |
| 2026-09-19 02:52:49 [all] step 7740/18464 loss=8.1343 ppl=3409.27 lr=4.075e-04 | 674 tok/s | 507M done | eta 12d01h37m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 02:52:49 [all] projected finish 18d13h01m exceeds the 168h deadline |
| 2026-09-19 03:09:03 [all] step 7750/18464 loss=8.1226 ppl=3369.70 lr=4.071e-04 | 673 tok/s | 508M done | eta 12d01h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 03:09:03 [all] projected finish 18d13h31m exceeds the 168h deadline |
| 2026-09-19 03:22:01 [all] checkpoint @ step 7758 (508M tokens, 6d11h53m elapsed) |
| 2026-09-19 03:25:15 [all] step 7760/18464 loss=8.1406 ppl=3431.13 lr=4.066e-04 | 674 tok/s | 509M done | eta 12d00h59m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 03:25:15 [all] projected finish 18d12h56m exceeds the 168h deadline |
| 2026-09-19 03:41:28 [all] step 7770/18464 loss=8.1280 ppl=3387.90 lr=4.062e-04 | 673 tok/s | 509M done | eta 12d01h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 03:41:28 [all] projected finish 18d13h16m exceeds the 168h deadline |
| 2026-09-19 03:52:50 [all] checkpoint @ step 7777 (510M tokens, 6d12h24m elapsed) |
| 2026-09-19 03:57:44 [all] step 7780/18464 loss=8.1110 ppl=3330.96 lr=4.057e-04 | 672 tok/s | 510M done | eta 12d01h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 03:57:44 [all] projected finish 18d13h59m exceeds the 168h deadline |
| 2026-09-19 04:14:00 [all] step 7790/18464 loss=8.1129 ppl=3337.37 lr=4.053e-04 | 671 tok/s | 511M done | eta 12d01h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 04:14:00 [all] projected finish 18d14h25m exceeds the 168h deadline |
| 2026-09-19 04:23:50 [all] checkpoint @ step 7796 (511M tokens, 6d12h55m elapsed) |
| 2026-09-19 04:30:18 [all] step 7800/18464 loss=8.1344 ppl=3409.91 lr=4.048e-04 | 670 tok/s | 511M done | eta 12d01h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 04:30:18 [all] projected finish 18d14h42m exceeds the 168h deadline |
| 2026-09-19 04:46:31 [all] step 7810/18464 loss=8.1112 ppl=3331.52 lr=4.044e-04 | 674 tok/s | 512M done | eta 11d23h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 04:46:31 [all] projected finish 18d13h05m exceeds the 168h deadline |
| 2026-09-19 04:54:36 [all] checkpoint @ step 7815 (512M tokens, 6d13h25m elapsed) |
| 2026-09-19 05:02:42 [all] step 7820/18464 loss=8.1282 ppl=3388.74 lr=4.039e-04 | 675 tok/s | 512M done | eta 11d23h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 05:02:42 [all] projected finish 18d12h38m exceeds the 168h deadline |
| 2026-09-19 05:18:55 [all] step 7830/18464 loss=8.1135 ppl=3339.40 lr=4.035e-04 | 674 tok/s | 513M done | eta 11d23h20m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 05:18:55 [all] projected finish 18d13h10m exceeds the 168h deadline |
| 2026-09-19 05:25:28 [all] checkpoint @ step 7834 (513M tokens, 6d13h56m elapsed) |
| 2026-09-19 05:35:13 [all] step 7840/18464 loss=8.1228 ppl=3370.35 lr=4.030e-04 | 670 tok/s | 514M done | eta 12d00h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 05:35:13 [all] projected finish 18d14h57m exceeds the 168h deadline |
| 2026-09-19 05:51:24 [all] step 7850/18464 loss=8.1138 ppl=3340.22 lr=4.026e-04 | 675 tok/s | 514M done | eta 11d22h18m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 05:51:24 [all] projected finish 18d12h41m exceeds the 168h deadline |
| 2026-09-19 05:56:15 [all] checkpoint @ step 7853 (515M tokens, 6d14h27m elapsed) |
| 2026-09-19 06:07:36 [all] step 7860/18464 loss=8.1158 ppl=3346.83 lr=4.021e-04 | 675 tok/s | 515M done | eta 11d22h03m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 06:07:36 [all] projected finish 18d12h42m exceeds the 168h deadline |
| 2026-09-19 06:23:46 [all] step 7870/18464 loss=8.1256 ppl=3379.81 lr=4.017e-04 | 676 tok/s | 516M done | eta 11d21h29m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 06:23:46 [all] projected finish 18d12h24m exceeds the 168h deadline |
| 2026-09-19 06:27:02 [all] checkpoint @ step 7872 (516M tokens, 6d14h58m elapsed) |
| 2026-09-19 06:40:02 [all] step 7880/18464 loss=8.1219 ppl=3367.26 lr=4.012e-04 | 671 tok/s | 516M done | eta 11d23h02m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 06:40:02 [all] projected finish 18d14h13m exceeds the 168h deadline |
| 2026-09-19 06:56:11 [all] step 7890/18464 loss=8.1128 ppl=3336.80 lr=4.008e-04 | 676 tok/s | 517M done | eta 11d20h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 06:56:11 [all] projected finish 18d12h08m exceeds the 168h deadline |
| 2026-09-19 06:57:48 [all] checkpoint @ step 7891 (517M tokens, 6d15h29m elapsed) |
| 2026-09-19 07:12:24 [all] step 7900/18464 loss=8.1231 ppl=3371.30 lr=4.003e-04 | 673 tok/s | 518M done | eta 11d21h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 07:12:24 [all] projected finish 18d13h17m exceeds the 168h deadline |
| 2026-09-19 07:28:39 [all] step 7910/18464 loss=8.1249 ppl=3377.47 lr=3.999e-04 | 672 tok/s | 518M done | eta 11d21h51m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 07:28:39 [all] projected finish 18d13h51m exceeds the 168h deadline |
| 2026-09-19 07:28:40 [all] checkpoint @ step 7910 (518M tokens, 6d16h00m elapsed) |
| 2026-09-19 07:44:58 [all] step 7920/18464 loss=8.1272 ppl=3385.33 lr=3.994e-04 | 670 tok/s | 519M done | eta 11d22h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 07:44:58 [all] projected finish 18d14h46m exceeds the 168h deadline |
| 2026-09-19 07:59:36 [all] checkpoint @ step 7929 (520M tokens, 6d16h30m elapsed) |
| 2026-09-19 08:01:11 [all] step 7930/18464 loss=8.1203 ppl=3362.14 lr=3.990e-04 | 673 tok/s | 520M done | eta 11d20h52m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 08:01:11 [all] projected finish 18d13h25m exceeds the 168h deadline |
| 2026-09-19 08:17:21 [all] step 7940/18464 loss=8.1395 ppl=3427.30 lr=3.985e-04 | 676 tok/s | 520M done | eta 11d19h36m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 08:17:21 [all] projected finish 18d12h25m exceeds the 168h deadline |
| 2026-09-19 08:30:21 [all] checkpoint @ step 7948 (521M tokens, 6d17h01m elapsed) |
| 2026-09-19 08:33:33 [all] step 7950/18464 loss=8.1187 ppl=3356.65 lr=3.980e-04 | 674 tok/s | 521M done | eta 11d19h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 08:33:33 [all] projected finish 18d12h55m exceeds the 168h deadline |
| 2026-09-19 08:49:50 [all] step 7960/18464 loss=8.1209 ppl=3363.96 lr=3.976e-04 | 671 tok/s | 522M done | eta 11d21h00m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 08:49:50 [all] projected finish 18d14h21m exceeds the 168h deadline |
| 2026-09-19 09:01:11 [all] checkpoint @ step 7967 (522M tokens, 6d17h32m elapsed) |
| 2026-09-19 09:06:06 [all] step 7970/18464 loss=8.1335 ppl=3406.82 lr=3.971e-04 | 671 tok/s | 522M done | eta 11d20h33m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 09:06:06 [all] projected finish 18d14h10m exceeds the 168h deadline |
| 2026-09-19 09:22:07 [all] step 7980/18464 loss=8.1186 ppl=3356.37 lr=3.967e-04 | 682 tok/s | 523M done | eta 11d15h50m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 09:22:07 [all] projected finish 18d09h43m exceeds the 168h deadline |
| 2026-09-19 09:31:48 [all] checkpoint @ step 7986 (523M tokens, 6d18h03m elapsed) |
| 2026-09-19 09:38:16 [all] step 7990/18464 loss=8.1195 ppl=3359.38 lr=3.962e-04 | 676 tok/s | 524M done | eta 11d17h53m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 09:38:16 [all] projected finish 18d12h03m exceeds the 168h deadline |
| 2026-09-19 09:54:41 [all] step 8000/18464 loss=8.1248 ppl=3377.28 lr=3.958e-04 | 665 tok/s | 524M done | eta 11d22h24m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 09:54:41 [all] projected finish 18d16h50m exceeds the 168h deadline |
| 2026-09-19 09:55:57 [all] eval step 8000/18464 loss=8.1217 ppl=3366.86 (best 8.1217) |
| 2026-09-19 09:55:59 [all] sample @ 8000: ' the a a a a a are the the the the the is and the the the a a the to the the a a the a a in the the the the the and the.. the the the and a, the the the the' |
| 2026-09-19 09:56:00 [all] checkpoint @ step 8000 (524M tokens, 6d18h27m elapsed) |
| 2026-09-19 10:12:55 [all] step 8010/18464 loss=8.1333 ppl=3406.14 lr=3.953e-04 | 599 tok/s | 525M done | eta 13d05h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 10:12:55 [all] projected finish 20d00h22m exceeds the 168h deadline |
| 2026-09-19 10:26:00 [all] step 8020/18464 loss=8.1083 ppl=3322.00 lr=3.949e-04 | 835 tok/s | 526M done | eta 9d11h47m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 10:26:00 [all] projected finish 16d06h45m exceeds the 168h deadline |
| 2026-09-19 10:26:01 [all] checkpoint @ step 8020 (526M tokens, 6d18h57m elapsed) |
| 2026-09-19 10:39:35 [all] step 8030/18464 loss=8.1212 ppl=3365.03 lr=3.944e-04 | 804 tok/s | 526M done | eta 9d20h07m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 10:39:35 [all] projected finish 16d15h18m exceeds the 168h deadline |
| 2026-09-19 10:54:43 [all] step 8040/18464 loss=8.1141 ppl=3341.13 lr=3.940e-04 | 721 tok/s | 527M done | eta 10d23h00m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 10:54:43 [all] projected finish 17d18h27m exceeds the 168h deadline |
| 2026-09-19 10:56:35 [all] checkpoint @ step 8041 (527M tokens, 6d19h27m elapsed) |
| 2026-09-19 11:11:35 [all] step 8050/18464 loss=8.1368 ppl=3417.82 lr=3.935e-04 | 648 tok/s | 528M done | eta 12d04h31m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 11:11:35 [all] projected finish 19d00h14m exceeds the 168h deadline |
| 2026-09-19 11:27:22 [all] step 8060/18464 loss=8.1338 ppl=3407.56 lr=3.930e-04 | 692 tok/s | 528M done | eta 11d09h38m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 11:27:22 [all] projected finish 18d05h37m exceeds the 168h deadline |
| 2026-09-19 11:27:22 [all] checkpoint @ step 8060 (528M tokens, 6d19h58m elapsed) |
| 2026-09-19 11:41:38 [all] step 8070/18464 loss=8.1248 ppl=3377.03 lr=3.926e-04 | 766 tok/s | 529M done | eta 10d07h10m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 11:41:38 [all] projected finish 17d03h23m exceeds the 168h deadline |
| 2026-09-19 11:57:45 [all] checkpoint @ step 8078 (529M tokens, 6d20h29m elapsed) |
| 2026-09-19 12:01:46 [all] step 8080/18464 loss=8.1108 ppl=3330.27 lr=3.921e-04 | 542 tok/s | 530M done | eta 14d12h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 12:01:46 [all] projected finish 21d09h03m exceeds the 168h deadline |
| 2026-09-19 12:22:18 [all] step 8090/18464 loss=8.1335 ppl=3406.75 lr=3.917e-04 | 532 tok/s | 530M done | eta 14d19h09m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 12:22:18 [all] projected finish 21d16h03m exceeds the 168h deadline |
| 2026-09-19 12:28:26 [all] checkpoint @ step 8093 (530M tokens, 6d20h59m elapsed) |
| 2026-09-19 12:42:10 [all] step 8100/18464 loss=8.1259 ppl=3380.80 lr=3.912e-04 | 550 tok/s | 531M done | eta 14d07h06m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 12:42:10 [all] projected finish 21d04h20m exceeds the 168h deadline |
| 2026-09-19 12:58:41 [all] checkpoint @ step 8109 (531M tokens, 6d21h30m elapsed) |
| 2026-09-19 13:00:17 [all] step 8110/18464 loss=8.1115 ppl=3332.57 lr=3.908e-04 | 603 tok/s | 531M done | eta 13d00h35m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 13:00:17 [all] projected finish 19d22h06m exceeds the 168h deadline |
| 2026-09-19 13:16:02 [all] step 8120/18464 loss=8.1207 ppl=3363.34 lr=3.903e-04 | 694 tok/s | 532M done | eta 11d07h30m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 13:16:02 [all] projected finish 18d05h18m exceeds the 168h deadline |
| 2026-09-19 13:29:27 [all] checkpoint @ step 8129 (533M tokens, 6d22h00m elapsed) |
| 2026-09-19 13:30:55 [all] step 8130/18464 loss=8.1306 ppl=3396.91 lr=3.899e-04 | 734 tok/s | 533M done | eta 10d16h15m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 13:30:55 [all] projected finish 17d14h17m exceeds the 168h deadline |
| 2026-09-19 13:45:09 [all] step 8140/18464 loss=8.1200 ppl=3360.99 lr=3.894e-04 | 767 tok/s | 533M done | eta 10d05h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 13:45:09 [all] projected finish 17d03h20m exceeds the 168h deadline |
| 2026-09-19 13:58:55 [all] step 8150/18464 loss=8.1430 ppl=3439.20 lr=3.889e-04 | 793 tok/s | 534M done | eta 9d20h39m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 13:58:55 [all] projected finish 16d19h09m exceeds the 168h deadline |
| 2026-09-19 14:00:18 [all] checkpoint @ step 8151 (534M tokens, 6d22h31m elapsed) |
| 2026-09-19 14:12:26 [all] step 8160/18464 loss=8.1461 ppl=3449.75 lr=3.885e-04 | 808 tok/s | 535M done | eta 9d16h04m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 14:12:26 [all] projected finish 16d14h48m exceeds the 168h deadline |
| 2026-09-19 14:25:39 [all] step 8170/18464 loss=8.1301 ppl=3395.09 lr=3.880e-04 | 827 tok/s | 535M done | eta 9d10h43m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 14:25:39 [all] projected finish 16d09h40m exceeds the 168h deadline |
| 2026-09-19 14:30:54 [all] checkpoint @ step 8174 (536M tokens, 6d23h02m elapsed) |
| 2026-09-19 14:38:45 [all] step 8180/18464 loss=8.1302 ppl=3395.59 lr=3.876e-04 | 834 tok/s | 536M done | eta 9d08h27m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 14:38:45 [all] projected finish 16d07h37m exceeds the 168h deadline |
| 2026-09-19 14:51:42 [all] step 8190/18464 loss=8.1240 ppl=3374.45 lr=3.871e-04 | 844 tok/s | 537M done | eta 9d05h40m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 14:51:42 [all] projected finish 16d05h03m exceeds the 168h deadline |
| 2026-09-19 15:02:01 [all] checkpoint @ step 8198 (537M tokens, 6d23h33m elapsed) |
| 2026-09-19 15:04:34 [all] step 8200/18464 loss=8.1168 ppl=3350.23 lr=3.866e-04 | 848 tok/s | 537M done | eta 9d04h22m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 15:04:34 [all] projected finish 16d03h58m exceeds the 168h deadline |
| 2026-09-19 15:17:20 [all] step 8210/18464 loss=8.1217 ppl=3366.86 lr=3.862e-04 | 856 tok/s | 538M done | eta 9d01h56m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 15:17:20 [all] projected finish 16d01h45m exceeds the 168h deadline |
| 2026-09-19 15:29:54 [all] step 8220/18464 loss=8.1080 ppl=3320.96 lr=3.857e-04 | 869 tok/s | 539M done | eta 8d22h41m | vram 0.57/5.02GB peak 5.60GB | rung 1 |
| 2026-09-19 15:29:54 [all] projected finish 15d22h42m exceeds the 168h deadline |
| 2026-09-19 15:29:54 [all] wall-clock deadline of 168h reached at step 8220 — stopping cleanly; --resume will continue if you want more |
| 2026-09-19 15:29:54 [all] final model exported -> runs/hypernix.3-mini/final (load it with neo_oven.preheat_brewed(path, tokenizer_source=path)) |
| 2026-09-19 15:29:54 [all] DONE — 539M tokens in 7d00h01m (deadline 168h, OVER BUDGET). best val ppl 3366.86 |
| 2026-09-19 16:35:45 [all] ============================================================================== |
| 2026-09-19 16:35:45 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-19 16:35:45 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-19 16:35:45 [all] packing 1.21B tokens -> 2.42 GB (uint16); 301.5 GB free on this volume |
| 2026-09-19 16:35:45 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-19 16:35:47 [all] gpu=NVIDIA GeForce GTX 1080 capability=sm_61 torch=2.14.0+cu126 arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 |
| 2026-09-19 16:35:47 [all] patched BrewerAttention.forward (causal/window mask composition + is_causal fast path) |
| 2026-09-19 16:35:57 [all] built hypernix.3-mini — 48,706,048 parameters (48.7060 M) | 8L d_model=512 heads=8/2 d_ff=2203 ctx=1024 vocab=32000 tied=True sliding_window=False |
| 2026-09-19 16:35:57 [all] 33.6% of parameters are the (tied) embedding matrix; 32,322,048 are 'body' parameters |
| 2026-09-19 16:35:57 [all] parameter memory: total=48,706,048 trainable=48,706,048 frozen=0 (185.8 MB) |
| 2026-09-19 16:35:57 [all] token bin: 1.21B tokens (2.42 GB), val tail = 2M tokens |
| 2026-09-19 16:35:57 [all] plan: 1.21B tokens = 18464 steps x 65.5K tokens/step (micro=8 ctx=1024 accum=8) | peak_lr=6.00e-04 warmup=369 | deadline=168h |
| 2026-09-19 16:35:57 [all] pre-flight (GTX 1080 = 8.87 TFLOPS fp32): 1d16h05m at 30% MFU, 3d08h10m at 15%, 6d06h20m at 8% — the budget needs better than 7.2% to land inside 168h. The real number replaces this at step 100. |
| 2026-09-19 16:35:57 [all] resumed at step 8198/18464 (537M tokens, 6d23h33m already spent) |
| 2026-09-19 16:36:05 [all] calibration: 891 tok/s sustained -> 15d17h19m for the full 1.21B tokens (deadline 7d00h00m, 00h00m00s headroom) |
| 2026-09-19 16:36:05 [all] this run will overrun the deadline. Options: --enforce-deadline (trim tokens), --ctx 512 (~1.7x faster), or fewer tokens via --tokens |
| 2026-09-19 16:36:13 [all] step 8200/18464 loss=8.1227 ppl=3370.19 lr=3.866e-04 | 8.12K tok/s | 537M done | eta 22h59m52s | vram 0.75/5.16GB peak 5.01GB | rung 1 |
| 2026-09-19 16:36:13 [all] projected finish 7d22h33m exceeds the 168h deadline |
| 2026-09-19 16:36:34 [all] ============================================================================== |
| 2026-09-19 16:36:34 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-19 16:36:34 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-19 16:36:34 [all] packing 1.21B tokens -> 2.42 GB (uint16); 301.5 GB free on this volume |
| 2026-09-19 16:36:34 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-19 17:48:48 [all] ============================================================================== |
| 2026-09-19 17:48:48 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-19 17:48:48 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-19 17:48:48 [all] packing 1.21B tokens -> 2.42 GB (uint16); 301.6 GB free on this volume |
| 2026-09-19 17:48:48 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-19 17:48:51 [all] gpu=NVIDIA GeForce GTX 1080 capability=sm_61 torch=2.14.0+cu126 arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 |
| 2026-09-19 17:48:51 [all] patched BrewerAttention.forward (causal/window mask composition + is_causal fast path) |
| 2026-09-19 17:48:59 [all] built hypernix.3-mini — 48,706,048 parameters (48.7060 M) | 8L d_model=512 heads=8/2 d_ff=2203 ctx=512 vocab=32000 tied=True sliding_window=False |
| 2026-09-19 17:48:59 [all] 33.6% of parameters are the (tied) embedding matrix; 32,322,048 are 'body' parameters |
| 2026-09-19 17:48:59 [all] parameter memory: total=48,706,048 trainable=48,706,048 frozen=0 (185.8 MB) |
| 2026-09-19 17:48:59 [all] token bin: 1.21B tokens (2.42 GB), val tail = 2M tokens |
| 2026-09-19 17:48:59 [all] plan: 1.21B tokens = 18464 steps x 65.5K tokens/step (micro=8 ctx=512 accum=16) | peak_lr=6.00e-04 warmup=369 | deadline=168h |
| 2026-09-19 17:48:59 [all] pre-flight (GTX 1080 = 8.87 TFLOPS fp32): 1d14h30m at 30% MFU, 3d05h00m at 15%, 6d00h22m at 8% — the budget needs better than 6.9% to land inside 168h. The real number replaces this at step 100. |
| 2026-09-19 17:48:59 [all] resumed at step 8198/18464 (537M tokens, 6d23h33m already spent) |
| 2026-09-19 17:49:03 [all] calibration: 891 tok/s sustained -> 15d17h20m for the full 1.21B tokens (deadline 7d00h00m, 00h00m00s headroom) |
| 2026-09-19 17:49:03 [all] this run will overrun the deadline. Options: --enforce-deadline (trim tokens), --ctx 512 (~1.7x faster), or fewer tokens via --tokens |
| 2026-09-19 17:49:06 [all] step 8200/18464 loss=8.1208 ppl=3363.81 lr=3.866e-04 | 10.3K tok/s | 537M done | eta 09h05m36s | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:49:06 [all] projected finish 7d08h39m exceeds the 168h deadline |
| 2026-09-19 17:49:36 [all] step 8210/18464 loss=8.1299 ppl=3394.49 lr=3.862e-04 | 10.7K tok/s | 538M done | eta 08h43m23s | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:49:36 [all] projected finish 7d08h17m exceeds the 168h deadline |
| 2026-09-19 17:50:07 [all] step 8220/18464 loss=8.1327 ppl=3403.94 lr=3.857e-04 | 10.7K tok/s | 538M done | eta 08h42m47s | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:50:07 [all] projected finish 7d08h17m exceeds the 168h deadline |
| 2026-09-19 17:50:40 [all] step 8230/18464 loss=8.1247 ppl=3376.73 lr=3.853e-04 | 9.81K tok/s | 538M done | eta 09h29m38s | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:50:40 [all] projected finish 7d09h04m exceeds the 168h deadline |
| 2026-09-19 17:51:42 [all] step 8240/18464 loss=8.1044 ppl=3309.14 lr=3.848e-04 | 5.3K tok/s | 539M done | eta 17h33m44s | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:51:42 [all] projected finish 7d17h09m exceeds the 168h deadline |
| 2026-09-19 17:53:17 [all] step 8250/18464 loss=8.1252 ppl=3378.63 lr=3.844e-04 | 3.44K tok/s | 539M done | eta 1d03h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:53:17 [all] projected finish 8d02h37m exceeds the 168h deadline |
| 2026-09-19 17:55:40 [all] step 8260/18464 loss=8.1202 ppl=3361.62 lr=3.839e-04 | 2.29K tok/s | 539M done | eta 1d16h29m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:55:40 [all] projected finish 8d16h09m exceeds the 168h deadline |
| 2026-09-19 17:58:08 [all] step 8270/18464 loss=8.1258 ppl=3380.56 lr=3.834e-04 | 2.21K tok/s | 540M done | eta 1d17h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 17:58:08 [all] projected finish 8d17h36m exceeds the 168h deadline |
| 2026-09-19 18:01:05 [all] step 8280/18464 loss=8.1165 ppl=3349.44 lr=3.830e-04 | 1.86K tok/s | 540M done | eta 2d01h52m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:01:05 [all] projected finish 9d01h38m exceeds the 168h deadline |
| 2026-09-19 18:04:15 [all] step 8290/18464 loss=8.1137 ppl=3340.07 lr=3.825e-04 | 1.72K tok/s | 540M done | eta 2d05h45m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:04:15 [all] projected finish 9d05h34m exceeds the 168h deadline |
| 2026-09-19 18:07:32 [all] step 8300/18464 loss=8.1303 ppl=3395.87 lr=3.821e-04 | 1.66K tok/s | 541M done | eta 2d07h44m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:07:32 [all] projected finish 9d07h36m exceeds the 168h deadline |
| 2026-09-19 18:11:01 [all] step 8310/18464 loss=8.1114 ppl=3332.37 lr=3.816e-04 | 1.57K tok/s | 541M done | eta 2d10h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:11:01 [all] projected finish 9d10h49m exceeds the 168h deadline |
| 2026-09-19 18:14:37 [all] step 8320/18464 loss=8.1050 ppl=3310.88 lr=3.811e-04 | 1.52K tok/s | 541M done | eta 2d12h51m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:14:37 [all] projected finish 9d12h50m exceeds the 168h deadline |
| 2026-09-19 18:15:45 [all] wall-clock deadline of 168h reached at step 8323 — stopping cleanly; --resume will continue if you want more |
| 2026-09-19 18:15:45 [all] final model exported -> runs/hypernix.3-mini/final (load it with neo_oven.preheat_brewed(path, tokenizer_source=path)) |
| 2026-09-19 18:15:45 [all] DONE — 541M tokens in 7d00h00m (deadline 168h, OVER BUDGET). best val ppl 3366.86 |
| 2026-09-19 18:23:28 [all] ============================================================================== |
| 2026-09-19 18:23:28 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-19 18:23:28 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-19 18:23:28 [all] packing 1.21B tokens -> 2.42 GB (uint16); 301.6 GB free on this volume |
| 2026-09-19 18:23:28 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-19 18:23:29 [all] gpu=NVIDIA GeForce GTX 1080 capability=sm_61 torch=2.14.0+cu126 arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 |
| 2026-09-19 18:23:29 [all] patched BrewerAttention.forward (causal/window mask composition + is_causal fast path) |
| 2026-09-19 18:23:35 [all] built hypernix.3-mini — 48,706,048 parameters (48.7060 M) | 8L d_model=512 heads=8/2 d_ff=2203 ctx=512 vocab=32000 tied=True sliding_window=False |
| 2026-09-19 18:23:35 [all] 33.6% of parameters are the (tied) embedding matrix; 32,322,048 are 'body' parameters |
| 2026-09-19 18:23:35 [all] parameter memory: total=48,706,048 trainable=48,706,048 frozen=0 (185.8 MB) |
| 2026-09-19 18:23:35 [all] token bin: 1.21B tokens (2.42 GB), val tail = 2M tokens |
| 2026-09-19 18:23:35 [all] plan: 1.21B tokens = 18464 steps x 65.5K tokens/step (micro=8 ctx=512 accum=16) | peak_lr=6.00e-04 warmup=369 | deadline=9999h |
| 2026-09-19 18:23:35 [all] pre-flight (GTX 1080 = 8.87 TFLOPS fp32): 1d14h30m at 30% MFU, 3d05h00m at 15%, 6d00h22m at 8% — the budget needs better than 0.1% to land inside 9999h. The real number replaces this at step 100. |
| 2026-09-19 18:23:35 [all] resumed at step 8198/18464 (537M tokens, 6d23h33m already spent) |
| 2026-09-19 18:23:39 [all] calibration: 891 tok/s sustained -> 15d17h20m for the full 1.21B tokens (deadline 416d15h00m, 400d21h39m headroom) |
| 2026-09-19 18:23:45 [all] step 8200/18464 loss=8.1208 ppl=3363.81 lr=3.866e-04 | 6.91K tok/s | 537M done | eta 13h30m46s | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:25:18 [all] step 8210/18464 loss=8.1299 ppl=3394.49 lr=3.862e-04 | 3.52K tok/s | 538M done | eta 1d02h30m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:27:48 [all] step 8220/18464 loss=8.1327 ppl=3403.94 lr=3.857e-04 | 2.18K tok/s | 538M done | eta 1d18h43m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:31:14 [all] step 8230/18464 loss=8.1247 ppl=3376.73 lr=3.853e-04 | 1.59K tok/s | 538M done | eta 2d10h44m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:34:59 [all] step 8240/18464 loss=8.1044 ppl=3309.14 lr=3.848e-04 | 1.46K tok/s | 539M done | eta 2d15h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:38:55 [all] step 8250/18464 loss=8.1252 ppl=3378.63 lr=3.844e-04 | 1.39K tok/s | 539M done | eta 2d19h05m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:42:56 [all] step 8260/18464 loss=8.1202 ppl=3361.62 lr=3.839e-04 | 1.36K tok/s | 539M done | eta 2d20h13m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:47:04 [all] step 8270/18464 loss=8.1258 ppl=3380.56 lr=3.834e-04 | 1.32K tok/s | 540M done | eta 2d22h10m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:51:13 [all] step 8280/18464 loss=8.1165 ppl=3349.44 lr=3.830e-04 | 1.31K tok/s | 540M done | eta 2d22h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:53:43 [all] checkpoint @ step 8286 (540M tokens, 7d00h03m elapsed) |
| 2026-09-19 18:55:24 [all] step 8290/18464 loss=8.1137 ppl=3340.07 lr=3.825e-04 | 1.31K tok/s | 540M done | eta 2d22h51m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 18:59:37 [all] step 8300/18464 loss=8.1303 ppl=3395.87 lr=3.821e-04 | 1.29K tok/s | 541M done | eta 2d23h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:03:50 [all] step 8310/18464 loss=8.1114 ppl=3332.37 lr=3.816e-04 | 1.29K tok/s | 541M done | eta 2d23h25m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:08:09 [all] step 8320/18464 loss=8.1050 ppl=3310.88 lr=3.811e-04 | 1.27K tok/s | 541M done | eta 3d00h50m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:12:26 [all] step 8330/18464 loss=8.0863 ppl=3249.74 lr=3.807e-04 | 1.27K tok/s | 542M done | eta 3d00h28m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:16:43 [all] step 8340/18464 loss=8.1168 ppl=3350.22 lr=3.802e-04 | 1.28K tok/s | 542M done | eta 3d00h15m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:21:00 [all] step 8350/18464 loss=8.1170 ppl=3350.84 lr=3.798e-04 | 1.27K tok/s | 542M done | eta 3d00h17m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:24:03 [all] checkpoint @ step 8357 (542M tokens, 7d00h33m elapsed) |
| 2026-09-19 19:25:19 [all] step 8360/18464 loss=8.0899 ppl=3261.50 lr=3.793e-04 | 1.27K tok/s | 543M done | eta 3d00h40m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:29:38 [all] step 8370/18464 loss=8.1078 ppl=3320.28 lr=3.788e-04 | 1.27K tok/s | 543M done | eta 3d00h26m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:33:58 [all] step 8380/18464 loss=8.1230 ppl=3371.21 lr=3.784e-04 | 1.26K tok/s | 543M done | eta 3d00h47m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:38:20 [all] step 8390/18464 loss=8.0938 ppl=3273.96 lr=3.779e-04 | 1.25K tok/s | 544M done | eta 3d01h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:42:41 [all] step 8400/18464 loss=8.1049 ppl=3310.55 lr=3.774e-04 | 1.25K tok/s | 544M done | eta 3d01h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:47:05 [all] step 8410/18464 loss=8.0918 ppl=3267.69 lr=3.770e-04 | 1.24K tok/s | 544M done | eta 3d01h47m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:51:28 [all] step 8420/18464 loss=8.0792 ppl=3226.60 lr=3.765e-04 | 1.25K tok/s | 545M done | eta 3d01h16m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 19:54:07 [all] checkpoint @ step 8426 (545M tokens, 7d01h03m elapsed) |
| 2026-09-19 19:55:51 [all] step 8430/18464 loss=8.0689 ppl=3193.59 lr=3.761e-04 | 1.24K tok/s | 545M done | eta 3d01h24m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:00:16 [all] step 8440/18464 loss=8.1002 ppl=3295.06 lr=3.756e-04 | 1.24K tok/s | 545M done | eta 3d01h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:04:45 [all] step 8450/18464 loss=8.0737 ppl=3208.90 lr=3.751e-04 | 1.22K tok/s | 546M done | eta 3d02h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:09:17 [all] step 8460/18464 loss=8.0873 ppl=3252.94 lr=3.747e-04 | 1.21K tok/s | 546M done | eta 3d03h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:13:48 [all] step 8470/18464 loss=8.0896 ppl=3260.45 lr=3.742e-04 | 1.21K tok/s | 546M done | eta 3d03h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:18:26 [all] step 8480/18464 loss=8.0794 ppl=3227.44 lr=3.738e-04 | 1.18K tok/s | 547M done | eta 3d05h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:22:58 [all] step 8490/18464 loss=8.0701 ppl=3197.32 lr=3.733e-04 | 1.2K tok/s | 547M done | eta 3d03h29m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:24:22 [all] checkpoint @ step 8493 (547M tokens, 7d01h34m elapsed) |
| 2026-09-19 20:27:33 [all] step 8500/18464 loss=8.0690 ppl=3193.82 lr=3.728e-04 | 1.19K tok/s | 547M done | eta 3d03h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:27:55 [all] eval step 8500/18464 loss=8.0849 ppl=3245.12 (best 8.0849) |
| 2026-09-19 20:32:27 [all] step 8510/18464 loss=8.0680 ppl=3190.80 lr=3.724e-04 | 1.11K tok/s | 547M done | eta 3d09h16m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:37:02 [all] step 8520/18464 loss=8.0726 ppl=3205.44 lr=3.719e-04 | 1.19K tok/s | 548M done | eta 3d04h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:41:39 [all] step 8530/18464 loss=8.0840 ppl=3242.08 lr=3.714e-04 | 1.19K tok/s | 548M done | eta 3d04h16m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:46:16 [all] step 8540/18464 loss=8.0635 ppl=3176.39 lr=3.710e-04 | 1.18K tok/s | 548M done | eta 3d04h25m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:50:54 [all] step 8550/18464 loss=8.0781 ppl=3223.05 lr=3.705e-04 | 1.18K tok/s | 549M done | eta 3d04h26m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 20:54:38 [all] checkpoint @ step 8558 (549M tokens, 7d02h04m elapsed) |
| 2026-09-19 20:55:33 [all] step 8560/18464 loss=8.0680 ppl=3190.65 lr=3.700e-04 | 1.17K tok/s | 549M done | eta 3d04h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:00:13 [all] step 8570/18464 loss=8.0617 ppl=3170.76 lr=3.696e-04 | 1.17K tok/s | 549M done | eta 3d04h47m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:04:48 [all] step 8580/18464 loss=8.0445 ppl=3116.66 lr=3.691e-04 | 1.19K tok/s | 550M done | eta 3d03h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:09:15 [all] step 8590/18464 loss=8.0502 ppl=3134.32 lr=3.687e-04 | 1.23K tok/s | 550M done | eta 3d01h19m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:13:32 [all] step 8600/18464 loss=8.0555 ppl=3151.07 lr=3.682e-04 | 1.28K tok/s | 550M done | eta 2d22h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:17:45 [all] step 8610/18464 loss=8.0489 ppl=3130.27 lr=3.677e-04 | 1.29K tok/s | 551M done | eta 2d21h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:21:56 [all] step 8620/18464 loss=8.0460 ppl=3121.38 lr=3.673e-04 | 1.31K tok/s | 551M done | eta 2d20h36m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:24:47 [all] checkpoint @ step 8627 (551M tokens, 7d02h34m elapsed) |
| 2026-09-19 21:26:00 [all] step 8630/18464 loss=8.0497 ppl=3132.95 lr=3.668e-04 | 1.34K tok/s | 551M done | eta 2d18h41m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:30:07 [all] step 8640/18464 loss=8.0403 ppl=3103.58 lr=3.663e-04 | 1.33K tok/s | 552M done | eta 2d19h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:34:10 [all] step 8650/18464 loss=8.0539 ppl=3146.07 lr=3.659e-04 | 1.35K tok/s | 552M done | eta 2d18h16m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:38:08 [all] step 8660/18464 loss=8.0248 ppl=3055.95 lr=3.654e-04 | 1.37K tok/s | 552M done | eta 2d17h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:42:04 [all] step 8670/18464 loss=8.0439 ppl=3114.74 lr=3.649e-04 | 1.39K tok/s | 553M done | eta 2d16h06m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:45:55 [all] step 8680/18464 loss=8.0190 ppl=3038.06 lr=3.645e-04 | 1.42K tok/s | 553M done | eta 2d14h47m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:49:54 [all] step 8690/18464 loss=8.0243 ppl=3054.14 lr=3.640e-04 | 1.37K tok/s | 553M done | eta 2d16h49m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:53:56 [all] step 8700/18464 loss=8.0017 ppl=2986.00 lr=3.635e-04 | 1.36K tok/s | 554M done | eta 2d17h34m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 21:55:10 [all] checkpoint @ step 8703 (554M tokens, 7d03h04m elapsed) |
| 2026-09-19 21:57:58 [all] step 8710/18464 loss=8.0340 ppl=3084.20 lr=3.631e-04 | 1.35K tok/s | 554M done | eta 2d17h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:01:54 [all] step 8720/18464 loss=8.0342 ppl=3084.67 lr=3.626e-04 | 1.39K tok/s | 554M done | eta 2d15h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:05:46 [all] step 8730/18464 loss=8.0396 ppl=3101.36 lr=3.622e-04 | 1.41K tok/s | 555M done | eta 2d14h41m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:09:34 [all] step 8740/18464 loss=8.0254 ppl=3057.71 lr=3.617e-04 | 1.44K tok/s | 555M done | eta 2d13h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:13:20 [all] step 8750/18464 loss=8.0069 ppl=3001.58 lr=3.612e-04 | 1.45K tok/s | 555M done | eta 2d12h59m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:17:05 [all] step 8760/18464 loss=8.0242 ppl=3054.08 lr=3.608e-04 | 1.46K tok/s | 556M done | eta 2d12h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:20:46 [all] step 8770/18464 loss=8.0114 ppl=3015.02 lr=3.603e-04 | 1.48K tok/s | 556M done | eta 2d11h26m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:24:29 [all] step 8780/18464 loss=8.0099 ppl=3010.62 lr=3.598e-04 | 1.47K tok/s | 556M done | eta 2d12h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:25:14 [all] checkpoint @ step 8782 (556M tokens, 7d03h35m elapsed) |
| 2026-09-19 22:28:09 [all] step 8790/18464 loss=8.0011 ppl=2984.11 lr=3.594e-04 | 1.49K tok/s | 557M done | eta 2d11h07m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:31:48 [all] step 8800/18464 loss=8.0118 ppl=3016.25 lr=3.589e-04 | 1.5K tok/s | 557M done | eta 2d10h45m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:35:25 [all] step 8810/18464 loss=8.0159 ppl=3028.62 lr=3.584e-04 | 1.51K tok/s | 557M done | eta 2d10h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:39:03 [all] step 8820/18464 loss=8.0212 ppl=3044.85 lr=3.580e-04 | 1.51K tok/s | 558M done | eta 2d10h17m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:42:39 [all] step 8830/18464 loss=8.0093 ppl=3008.78 lr=3.575e-04 | 1.52K tok/s | 558M done | eta 2d09h47m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:46:12 [all] step 8840/18464 loss=8.0113 ppl=3014.74 lr=3.570e-04 | 1.53K tok/s | 558M done | eta 2d09h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:49:49 [all] step 8850/18464 loss=8.0041 ppl=2993.16 lr=3.566e-04 | 1.52K tok/s | 559M done | eta 2d09h45m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:53:25 [all] step 8860/18464 loss=8.0187 ppl=3037.37 lr=3.561e-04 | 1.52K tok/s | 559M done | eta 2d09h36m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 22:55:36 [all] checkpoint @ step 8866 (559M tokens, 7d04h05m elapsed) |
| 2026-09-19 22:57:01 [all] step 8870/18464 loss=8.0111 ppl=3014.13 lr=3.556e-04 | 1.52K tok/s | 559M done | eta 2d09h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:00:35 [all] step 8880/18464 loss=7.9902 ppl=2951.84 lr=3.552e-04 | 1.53K tok/s | 560M done | eta 2d09h05m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:04:08 [all] step 8890/18464 loss=7.9938 ppl=2962.67 lr=3.547e-04 | 1.54K tok/s | 560M done | eta 2d08h30m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:07:41 [all] step 8900/18464 loss=8.0058 ppl=2998.18 lr=3.542e-04 | 1.54K tok/s | 560M done | eta 2d08h32m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:11:14 [all] step 8910/18464 loss=7.9874 ppl=2943.55 lr=3.538e-04 | 1.53K tok/s | 561M done | eta 2d08h43m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:14:47 [all] step 8920/18464 loss=7.9924 ppl=2958.35 lr=3.533e-04 | 1.54K tok/s | 561M done | eta 2d08h17m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:18:17 [all] step 8930/18464 loss=7.9976 ppl=2973.72 lr=3.528e-04 | 1.56K tok/s | 561M done | eta 2d07h44m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:21:47 [all] step 8940/18464 loss=7.9877 ppl=2944.62 lr=3.524e-04 | 1.56K tok/s | 562M done | eta 2d07h37m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:25:16 [all] step 8950/18464 loss=7.9793 ppl=2919.84 lr=3.519e-04 | 1.57K tok/s | 562M done | eta 2d07h12m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:25:37 [all] checkpoint @ step 8951 (562M tokens, 7d04h35m elapsed) |
| 2026-09-19 23:28:42 [all] step 8960/18464 loss=7.9954 ppl=2967.15 lr=3.514e-04 | 1.6K tok/s | 562M done | eta 2d06h12m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:32:06 [all] step 8970/18464 loss=7.9860 ppl=2939.40 lr=3.510e-04 | 1.6K tok/s | 563M done | eta 2d05h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:35:29 [all] step 8980/18464 loss=7.9815 ppl=2926.45 lr=3.505e-04 | 1.62K tok/s | 563M done | eta 2d05h20m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:38:55 [all] step 8990/18464 loss=7.9841 ppl=2933.84 lr=3.500e-04 | 1.58K tok/s | 563M done | eta 2d06h25m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:42:27 [all] step 9000/18464 loss=7.9833 ppl=2931.70 lr=3.496e-04 | 1.55K tok/s | 564M done | eta 2d07h38m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:42:44 [all] eval step 9000/18464 loss=7.9952 ppl=2966.75 (best 7.9952) |
| 2026-09-19 23:42:44 [all] checkpoint @ step 9000 (564M tokens, 7d04h52m elapsed) |
| 2026-09-19 23:46:21 [all] step 9010/18464 loss=8.0058 ppl=2998.25 lr=3.491e-04 | 1.4K tok/s | 564M done | eta 2d13h32m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:50:05 [all] step 9020/18464 loss=7.9763 ppl=2911.15 lr=3.486e-04 | 1.47K tok/s | 564M done | eta 2d10h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:53:51 [all] step 9030/18464 loss=7.9537 ppl=2846.22 lr=3.482e-04 | 1.45K tok/s | 565M done | eta 2d11h12m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-19 23:57:38 [all] step 9040/18464 loss=7.9801 ppl=2922.22 lr=3.477e-04 | 1.44K tok/s | 565M done | eta 2d11h34m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:01:30 [all] step 9050/18464 loss=7.9677 ppl=2886.21 lr=3.472e-04 | 1.41K tok/s | 565M done | eta 2d12h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:05:24 [all] step 9060/18464 loss=7.9787 ppl=2918.03 lr=3.467e-04 | 1.4K tok/s | 566M done | eta 2d13h06m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:09:20 [all] step 9070/18464 loss=7.9636 ppl=2874.29 lr=3.463e-04 | 1.39K tok/s | 566M done | eta 2d13h37m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:12:53 [all] checkpoint @ step 9079 (566M tokens, 7d05h22m elapsed) |
| 2026-09-20 00:13:17 [all] step 9080/18464 loss=7.9648 ppl=2877.85 lr=3.458e-04 | 1.38K tok/s | 566M done | eta 2d13h49m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:17:14 [all] step 9090/18464 loss=7.9642 ppl=2876.05 lr=3.453e-04 | 1.39K tok/s | 566M done | eta 2d13h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:21:12 [all] step 9100/18464 loss=7.9747 ppl=2906.54 lr=3.449e-04 | 1.37K tok/s | 567M done | eta 2d14h05m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:25:14 [all] step 9110/18464 loss=7.9718 ppl=2898.08 lr=3.444e-04 | 1.35K tok/s | 567M done | eta 2d14h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:29:19 [all] step 9120/18464 loss=7.9726 ppl=2900.25 lr=3.439e-04 | 1.34K tok/s | 567M done | eta 2d15h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:33:26 [all] step 9130/18464 loss=7.9575 ppl=2857.00 lr=3.435e-04 | 1.32K tok/s | 568M done | eta 2d16h13m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:37:32 [all] step 9140/18464 loss=7.9517 ppl=2840.39 lr=3.430e-04 | 1.34K tok/s | 568M done | eta 2d15h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:41:37 [all] step 9150/18464 loss=7.9626 ppl=2871.65 lr=3.425e-04 | 1.33K tok/s | 568M done | eta 2d15h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:43:17 [all] checkpoint @ step 9154 (569M tokens, 7d05h53m elapsed) |
| 2026-09-20 00:45:45 [all] step 9160/18464 loss=7.9664 ppl=2882.51 lr=3.421e-04 | 1.33K tok/s | 569M done | eta 2d15h52m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:49:52 [all] step 9170/18464 loss=7.9571 ppl=2855.91 lr=3.416e-04 | 1.33K tok/s | 569M done | eta 2d15h47m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:53:59 [all] step 9180/18464 loss=7.9352 ppl=2793.93 lr=3.411e-04 | 1.32K tok/s | 569M done | eta 2d15h50m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 00:58:06 [all] step 9190/18464 loss=7.9577 ppl=2857.55 lr=3.407e-04 | 1.33K tok/s | 570M done | eta 2d15h34m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:02:17 [all] step 9200/18464 loss=7.9518 ppl=2840.63 lr=3.402e-04 | 1.3K tok/s | 570M done | eta 2d16h40m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:06:25 [all] step 9210/18464 loss=7.9382 ppl=2802.26 lr=3.397e-04 | 1.32K tok/s | 570M done | eta 2d15h46m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:10:35 [all] step 9220/18464 loss=7.9547 ppl=2849.01 lr=3.393e-04 | 1.31K tok/s | 571M done | eta 2d16h09m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:13:32 [all] checkpoint @ step 9227 (571M tokens, 7d06h23m elapsed) |
| 2026-09-20 01:14:48 [all] step 9230/18464 loss=7.9508 ppl=2837.82 lr=3.388e-04 | 1.3K tok/s | 571M done | eta 2d16h49m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:18:59 [all] step 9240/18464 loss=7.9330 ppl=2787.70 lr=3.383e-04 | 1.3K tok/s | 571M done | eta 2d16h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:23:11 [all] step 9250/18464 loss=7.9563 ppl=2853.57 lr=3.379e-04 | 1.3K tok/s | 572M done | eta 2d16h27m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:27:24 [all] step 9260/18464 loss=7.9440 ppl=2818.65 lr=3.374e-04 | 1.3K tok/s | 572M done | eta 2d16h34m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:31:35 [all] step 9270/18464 loss=7.9641 ppl=2875.76 lr=3.369e-04 | 1.31K tok/s | 572M done | eta 2d16h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:35:50 [all] step 9280/18464 loss=7.9332 ppl=2788.41 lr=3.364e-04 | 1.28K tok/s | 573M done | eta 2d17h06m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:40:06 [all] step 9290/18464 loss=7.9231 ppl=2760.35 lr=3.360e-04 | 1.28K tok/s | 573M done | eta 2d17h20m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:43:57 [all] checkpoint @ step 9299 (573M tokens, 7d06h53m elapsed) |
| 2026-09-20 01:44:23 [all] step 9300/18464 loss=7.9348 ppl=2792.91 lr=3.355e-04 | 1.28K tok/s | 573M done | eta 2d17h21m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:48:40 [all] step 9310/18464 loss=7.9475 ppl=2828.49 lr=3.350e-04 | 1.28K tok/s | 574M done | eta 2d17h14m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:52:56 [all] step 9320/18464 loss=7.9425 ppl=2814.51 lr=3.346e-04 | 1.28K tok/s | 574M done | eta 2d17h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 01:57:15 [all] step 9330/18464 loss=7.9193 ppl=2749.72 lr=3.341e-04 | 1.27K tok/s | 574M done | eta 2d17h41m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:01:34 [all] step 9340/18464 loss=7.9294 ppl=2777.87 lr=3.336e-04 | 1.26K tok/s | 575M done | eta 2d17h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:05:54 [all] step 9350/18464 loss=7.9405 ppl=2808.73 lr=3.332e-04 | 1.26K tok/s | 575M done | eta 2d17h56m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:10:16 [all] step 9360/18464 loss=7.9265 ppl=2769.76 lr=3.327e-04 | 1.25K tok/s | 575M done | eta 2d18h13m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:14:11 [all] checkpoint @ step 9369 (576M tokens, 7d07h23m elapsed) |
| 2026-09-20 02:14:36 [all] step 9370/18464 loss=7.9207 ppl=2753.71 lr=3.322e-04 | 1.26K tok/s | 576M done | eta 2d17h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:18:58 [all] step 9380/18464 loss=7.9293 ppl=2777.47 lr=3.318e-04 | 1.25K tok/s | 576M done | eta 2d18h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:23:18 [all] step 9390/18464 loss=7.9329 ppl=2787.62 lr=3.313e-04 | 1.26K tok/s | 576M done | eta 2d17h43m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:27:39 [all] step 9400/18464 loss=7.9461 ppl=2824.41 lr=3.308e-04 | 1.26K tok/s | 577M done | eta 2d17h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:31:59 [all] step 9410/18464 loss=7.9129 ppl=2732.27 lr=3.304e-04 | 1.26K tok/s | 577M done | eta 2d17h22m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:36:22 [all] step 9420/18464 loss=7.9238 ppl=2762.20 lr=3.299e-04 | 1.25K tok/s | 577M done | eta 2d18h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:40:41 [all] step 9430/18464 loss=7.9390 ppl=2804.67 lr=3.294e-04 | 1.26K tok/s | 578M done | eta 2d17h08m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:44:37 [all] checkpoint @ step 9439 (578M tokens, 7d07h54m elapsed) |
| 2026-09-20 02:45:03 [all] step 9440/18464 loss=7.9277 ppl=2773.00 lr=3.289e-04 | 1.25K tok/s | 578M done | eta 2d17h30m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:49:25 [all] step 9450/18464 loss=7.9445 ppl=2820.00 lr=3.285e-04 | 1.25K tok/s | 578M done | eta 2d17h36m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:53:47 [all] step 9460/18464 loss=7.9004 ppl=2698.47 lr=3.280e-04 | 1.25K tok/s | 579M done | eta 2d17h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 02:58:08 [all] step 9470/18464 loss=7.9401 ppl=2807.60 lr=3.275e-04 | 1.26K tok/s | 579M done | eta 2d17h12m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:02:33 [all] step 9480/18464 loss=7.9215 ppl=2756.03 lr=3.271e-04 | 1.24K tok/s | 579M done | eta 2d18h12m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:06:58 [all] step 9490/18464 loss=7.9242 ppl=2763.41 lr=3.266e-04 | 1.24K tok/s | 580M done | eta 2d17h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:11:21 [all] step 9500/18464 loss=7.9194 ppl=2750.20 lr=3.261e-04 | 1.25K tok/s | 580M done | eta 2d17h32m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:11:41 [all] eval step 9500/18464 loss=7.9341 ppl=2790.95 (best 7.9341) |
| 2026-09-20 03:14:48 [all] checkpoint @ step 9507 (580M tokens, 7d08h24m elapsed) |
| 2026-09-20 03:16:06 [all] step 9510/18464 loss=7.9019 ppl=2702.50 lr=3.257e-04 | 1.15K tok/s | 580M done | eta 2d22h57m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:20:32 [all] step 9520/18464 loss=7.9309 ppl=2781.92 lr=3.252e-04 | 1.23K tok/s | 581M done | eta 2d18h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:24:56 [all] step 9530/18464 loss=7.9181 ppl=2746.43 lr=3.247e-04 | 1.24K tok/s | 581M done | eta 2d17h32m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:29:19 [all] step 9540/18464 loss=7.9259 ppl=2768.05 lr=3.243e-04 | 1.24K tok/s | 581M done | eta 2d17h15m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:33:45 [all] step 9550/18464 loss=7.9290 ppl=2776.74 lr=3.238e-04 | 1.23K tok/s | 582M done | eta 2d17h51m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:38:11 [all] step 9560/18464 loss=7.9322 ppl=2785.47 lr=3.233e-04 | 1.24K tok/s | 582M done | eta 2d17h37m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:42:36 [all] step 9570/18464 loss=7.9276 ppl=2772.64 lr=3.229e-04 | 1.23K tok/s | 582M done | eta 2d17h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:44:50 [all] checkpoint @ step 9575 (582M tokens, 7d08h54m elapsed) |
| 2026-09-20 03:47:04 [all] step 9580/18464 loss=7.9228 ppl=2759.40 lr=3.224e-04 | 1.22K tok/s | 583M done | eta 2d18h09m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:51:31 [all] step 9590/18464 loss=7.9148 ppl=2737.62 lr=3.219e-04 | 1.23K tok/s | 583M done | eta 2d17h51m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 03:56:00 [all] step 9600/18464 loss=7.9115 ppl=2728.59 lr=3.214e-04 | 1.22K tok/s | 583M done | eta 2d18h07m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:00:28 [all] step 9610/18464 loss=7.9389 ppl=2804.30 lr=3.210e-04 | 1.22K tok/s | 584M done | eta 2d17h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:04:57 [all] step 9620/18464 loss=7.9343 ppl=2791.40 lr=3.205e-04 | 1.22K tok/s | 584M done | eta 2d18h02m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:09:23 [all] step 9630/18464 loss=7.9407 ppl=2809.39 lr=3.200e-04 | 1.23K tok/s | 584M done | eta 2d17h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:13:51 [all] step 9640/18464 loss=7.9113 ppl=2727.80 lr=3.196e-04 | 1.22K tok/s | 585M done | eta 2d17h43m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:15:12 [all] checkpoint @ step 9643 (585M tokens, 7d09h24m elapsed) |
| 2026-09-20 04:18:19 [all] step 9650/18464 loss=7.9405 ppl=2808.67 lr=3.191e-04 | 1.22K tok/s | 585M done | eta 2d17h32m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:22:47 [all] step 9660/18464 loss=7.9213 ppl=2755.47 lr=3.186e-04 | 1.22K tok/s | 585M done | eta 2d17h32m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:27:15 [all] step 9670/18464 loss=7.9260 ppl=2768.36 lr=3.182e-04 | 1.22K tok/s | 585M done | eta 2d17h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:31:45 [all] step 9680/18464 loss=7.9358 ppl=2795.66 lr=3.177e-04 | 1.21K tok/s | 586M done | eta 2d17h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:36:13 [all] step 9690/18464 loss=7.9143 ppl=2736.02 lr=3.172e-04 | 1.22K tok/s | 586M done | eta 2d17h17m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:40:42 [all] step 9700/18464 loss=7.9083 ppl=2719.72 lr=3.168e-04 | 1.22K tok/s | 586M done | eta 2d17h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:45:13 [all] step 9710/18464 loss=7.9127 ppl=2731.65 lr=3.163e-04 | 1.21K tok/s | 587M done | eta 2d17h48m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:45:13 [all] checkpoint @ step 9710 (587M tokens, 7d09h54m elapsed) |
| 2026-09-20 04:49:44 [all] step 9720/18464 loss=7.9377 ppl=2800.93 lr=3.158e-04 | 1.21K tok/s | 587M done | eta 2d17h46m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:54:12 [all] step 9730/18464 loss=7.9280 ppl=2773.94 lr=3.154e-04 | 1.22K tok/s | 587M done | eta 2d17h09m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 04:58:43 [all] step 9740/18464 loss=7.9008 ppl=2699.33 lr=3.149e-04 | 1.21K tok/s | 588M done | eta 2d17h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:03:12 [all] step 9750/18464 loss=7.9135 ppl=2734.01 lr=3.144e-04 | 1.22K tok/s | 588M done | eta 2d17h04m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:07:42 [all] step 9760/18464 loss=7.9272 ppl=2771.62 lr=3.140e-04 | 1.21K tok/s | 588M done | eta 2d17h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:12:12 [all] step 9770/18464 loss=7.9232 ppl=2760.59 lr=3.135e-04 | 1.22K tok/s | 589M done | eta 2d17h07m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:15:21 [all] checkpoint @ step 9777 (589M tokens, 7d10h25m elapsed) |
| 2026-09-20 05:16:42 [all] step 9780/18464 loss=7.9409 ppl=2810.01 lr=3.130e-04 | 1.21K tok/s | 589M done | eta 2d17h05m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:21:13 [all] step 9790/18464 loss=7.9082 ppl=2719.37 lr=3.126e-04 | 1.21K tok/s | 589M done | eta 2d17h15m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:25:42 [all] step 9800/18464 loss=7.9008 ppl=2699.35 lr=3.121e-04 | 1.22K tok/s | 590M done | eta 2d16h46m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:30:13 [all] step 9810/18464 loss=7.9110 ppl=2727.25 lr=3.116e-04 | 1.21K tok/s | 590M done | eta 2d17h18m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:34:43 [all] step 9820/18464 loss=7.8887 ppl=2666.85 lr=3.111e-04 | 1.22K tok/s | 590M done | eta 2d16h38m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:39:13 [all] step 9830/18464 loss=7.9219 ppl=2757.08 lr=3.107e-04 | 1.21K tok/s | 591M done | eta 2d16h57m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:43:42 [all] step 9840/18464 loss=7.9236 ppl=2761.81 lr=3.102e-04 | 1.22K tok/s | 591M done | eta 2d16h22m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:45:31 [all] checkpoint @ step 9844 (591M tokens, 7d10h55m elapsed) |
| 2026-09-20 05:48:13 [all] step 9850/18464 loss=7.9134 ppl=2733.55 lr=3.097e-04 | 1.21K tok/s | 591M done | eta 2d16h48m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:52:47 [all] step 9860/18464 loss=7.9326 ppl=2786.70 lr=3.093e-04 | 1.2K tok/s | 592M done | eta 2d17h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 05:57:17 [all] step 9870/18464 loss=7.8955 ppl=2685.08 lr=3.088e-04 | 1.21K tok/s | 592M done | eta 2d16h30m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:01:48 [all] step 9880/18464 loss=7.9027 ppl=2704.63 lr=3.083e-04 | 1.21K tok/s | 592M done | eta 2d16h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:06:17 [all] step 9890/18464 loss=7.9071 ppl=2716.54 lr=3.079e-04 | 1.22K tok/s | 593M done | eta 2d16h02m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:10:47 [all] step 9900/18464 loss=7.9143 ppl=2736.20 lr=3.074e-04 | 1.21K tok/s | 593M done | eta 2d16h10m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:15:16 [all] step 9910/18464 loss=7.9208 ppl=2754.04 lr=3.069e-04 | 1.22K tok/s | 593M done | eta 2d16h01m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:15:44 [all] checkpoint @ step 9911 (593M tokens, 7d11h25m elapsed) |
| 2026-09-20 06:19:46 [all] step 9920/18464 loss=7.9337 ppl=2789.74 lr=3.065e-04 | 1.22K tok/s | 594M done | eta 2d15h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:24:13 [all] step 9930/18464 loss=7.9042 ppl=2708.63 lr=3.060e-04 | 1.22K tok/s | 594M done | eta 2d15h27m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:28:41 [all] step 9940/18464 loss=7.9026 ppl=2704.27 lr=3.055e-04 | 1.23K tok/s | 594M done | eta 2d15h18m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:33:07 [all] step 9950/18464 loss=7.9345 ppl=2791.90 lr=3.051e-04 | 1.23K tok/s | 595M done | eta 2d15h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:37:39 [all] step 9960/18464 loss=7.9081 ppl=2719.33 lr=3.046e-04 | 1.21K tok/s | 595M done | eta 2d16h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:42:03 [all] step 9970/18464 loss=7.9258 ppl=2767.76 lr=3.041e-04 | 1.24K tok/s | 595M done | eta 2d14h15m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:45:55 [all] checkpoint @ step 9979 (596M tokens, 7d11h55m elapsed) |
| 2026-09-20 06:46:20 [all] step 9980/18464 loss=7.9162 ppl=2741.31 lr=3.037e-04 | 1.28K tok/s | 596M done | eta 2d12h30m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:50:35 [all] step 9990/18464 loss=7.9201 ppl=2751.96 lr=3.032e-04 | 1.28K tok/s | 596M done | eta 2d12h05m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:54:49 [all] step 10000/18464 loss=7.9153 ppl=2738.97 lr=3.027e-04 | 1.29K tok/s | 596M done | eta 2d11h37m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 06:55:08 [all] eval step 10000/18464 loss=7.9320 ppl=2784.87 (best 7.9320) |
| 2026-09-20 06:55:09 [all] sample @ 10000: ' in the a a a and, that with the more was and and the a have to the the the the the the the the the the the a in the the the a the this to the the this it the thes, the be' |
| 2026-09-20 06:55:10 [all] checkpoint @ step 10000 (596M tokens, 7d12h04m elapsed) |
| 2026-09-20 06:59:18 [all] step 10010/18464 loss=7.9300 ppl=2779.40 lr=3.023e-04 | 1.21K tok/s | 597M done | eta 2d15h22m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:03:26 [all] step 10020/18464 loss=7.9277 ppl=2772.97 lr=3.018e-04 | 1.33K tok/s | 597M done | eta 2d09h59m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:07:33 [all] step 10030/18464 loss=7.9351 ppl=2793.51 lr=3.013e-04 | 1.32K tok/s | 597M done | eta 2d10h00m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:11:36 [all] step 10040/18464 loss=7.9220 ppl=2757.17 lr=3.009e-04 | 1.35K tok/s | 598M done | eta 2d08h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:15:37 [all] step 10050/18464 loss=7.9141 ppl=2735.70 lr=3.004e-04 | 1.36K tok/s | 598M done | eta 2d08h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:19:38 [all] step 10060/18464 loss=7.9061 ppl=2713.69 lr=2.999e-04 | 1.36K tok/s | 598M done | eta 2d08h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:23:35 [all] step 10070/18464 loss=7.9174 ppl=2744.67 lr=2.995e-04 | 1.38K tok/s | 599M done | eta 2d07h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:25:33 [all] checkpoint @ step 10075 (599M tokens, 7d12h35m elapsed) |
| 2026-09-20 07:27:32 [all] step 10080/18464 loss=7.9046 ppl=2709.74 lr=2.990e-04 | 1.38K tok/s | 599M done | eta 2d07h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:31:30 [all] step 10090/18464 loss=7.9094 ppl=2722.76 lr=2.985e-04 | 1.38K tok/s | 599M done | eta 2d07h21m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:35:26 [all] step 10100/18464 loss=7.9036 ppl=2706.91 lr=2.981e-04 | 1.39K tok/s | 600M done | eta 2d06h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:39:24 [all] step 10110/18464 loss=7.9094 ppl=2722.74 lr=2.976e-04 | 1.38K tok/s | 600M done | eta 2d07h07m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:43:21 [all] step 10120/18464 loss=7.9205 ppl=2753.17 lr=2.972e-04 | 1.38K tok/s | 600M done | eta 2d07h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:47:16 [all] step 10130/18464 loss=7.9200 ppl=2751.70 lr=2.967e-04 | 1.39K tok/s | 601M done | eta 2d06h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:51:12 [all] step 10140/18464 loss=7.9202 ppl=2752.44 lr=2.962e-04 | 1.39K tok/s | 601M done | eta 2d06h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:55:06 [all] step 10150/18464 loss=7.9183 ppl=2747.20 lr=2.958e-04 | 1.4K tok/s | 601M done | eta 2d06h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 07:55:54 [all] checkpoint @ step 10152 (601M tokens, 7d13h05m elapsed) |
| 2026-09-20 07:59:01 [all] step 10160/18464 loss=7.8999 ppl=2696.96 lr=2.953e-04 | 1.39K tok/s | 602M done | eta 2d06h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:02:57 [all] step 10170/18464 loss=7.9092 ppl=2722.26 lr=2.948e-04 | 1.39K tok/s | 602M done | eta 2d06h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:06:51 [all] step 10180/18464 loss=7.8986 ppl=2693.41 lr=2.944e-04 | 1.4K tok/s | 602M done | eta 2d05h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:10:45 [all] step 10190/18464 loss=7.9215 ppl=2755.99 lr=2.939e-04 | 1.4K tok/s | 603M done | eta 2d05h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:14:39 [all] step 10200/18464 loss=7.9124 ppl=2730.81 lr=2.934e-04 | 1.4K tok/s | 603M done | eta 2d05h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:18:31 [all] step 10210/18464 loss=7.9144 ppl=2736.30 lr=2.930e-04 | 1.41K tok/s | 603M done | eta 2d05h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:22:24 [all] step 10220/18464 loss=7.8970 ppl=2689.08 lr=2.925e-04 | 1.41K tok/s | 604M done | eta 2d05h14m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:26:18 [all] step 10230/18464 loss=7.9106 ppl=2725.98 lr=2.920e-04 | 1.4K tok/s | 604M done | eta 2d05h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:26:18 [all] checkpoint @ step 10230 (604M tokens, 7d13h36m elapsed) |
| 2026-09-20 08:30:11 [all] step 10240/18464 loss=7.9195 ppl=2750.40 lr=2.916e-04 | 1.4K tok/s | 604M done | eta 2d05h21m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:34:03 [all] step 10250/18464 loss=7.9054 ppl=2711.90 lr=2.911e-04 | 1.41K tok/s | 605M done | eta 2d04h56m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:37:58 [all] step 10260/18464 loss=7.9075 ppl=2717.63 lr=2.906e-04 | 1.4K tok/s | 605M done | eta 2d05h28m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:41:49 [all] step 10270/18464 loss=7.9152 ppl=2738.52 lr=2.902e-04 | 1.42K tok/s | 605M done | eta 2d04h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:45:40 [all] step 10280/18464 loss=7.9083 ppl=2719.68 lr=2.897e-04 | 1.42K tok/s | 605M done | eta 2d04h34m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:49:30 [all] step 10290/18464 loss=7.9274 ppl=2772.08 lr=2.893e-04 | 1.42K tok/s | 606M done | eta 2d04h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:53:20 [all] step 10300/18464 loss=7.9265 ppl=2769.60 lr=2.888e-04 | 1.42K tok/s | 606M done | eta 2d04h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 08:56:25 [all] checkpoint @ step 10308 (606M tokens, 7d14h06m elapsed) |
| 2026-09-20 08:57:11 [all] step 10310/18464 loss=7.9129 ppl=2732.37 lr=2.883e-04 | 1.42K tok/s | 606M done | eta 2d04h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:01:01 [all] step 10320/18464 loss=7.9230 ppl=2759.93 lr=2.879e-04 | 1.42K tok/s | 607M done | eta 2d04h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:04:50 [all] step 10330/18464 loss=7.9281 ppl=2774.27 lr=2.874e-04 | 1.43K tok/s | 607M done | eta 2d03h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:08:42 [all] step 10340/18464 loss=7.9546 ppl=2848.76 lr=2.869e-04 | 1.41K tok/s | 607M done | eta 2d04h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:12:32 [all] step 10350/18464 loss=7.9136 ppl=2734.17 lr=2.865e-04 | 1.43K tok/s | 608M done | eta 2d03h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:16:21 [all] step 10360/18464 loss=7.9190 ppl=2749.10 lr=2.860e-04 | 1.43K tok/s | 608M done | eta 2d03h31m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:20:12 [all] step 10370/18464 loss=7.9325 ppl=2786.44 lr=2.856e-04 | 1.42K tok/s | 608M done | eta 2d04h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:24:00 [all] step 10380/18464 loss=7.9119 ppl=2729.51 lr=2.851e-04 | 1.44K tok/s | 609M done | eta 2d03h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:26:42 [all] checkpoint @ step 10387 (609M tokens, 7d14h36m elapsed) |
| 2026-09-20 09:27:49 [all] step 10390/18464 loss=7.9095 ppl=2722.93 lr=2.846e-04 | 1.43K tok/s | 609M done | eta 2d03h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:31:40 [all] step 10400/18464 loss=7.9212 ppl=2755.04 lr=2.842e-04 | 1.42K tok/s | 609M done | eta 2d03h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:35:28 [all] step 10410/18464 loss=7.9185 ppl=2747.55 lr=2.837e-04 | 1.44K tok/s | 610M done | eta 2d02h53m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:39:16 [all] step 10420/18464 loss=7.9405 ppl=2808.81 lr=2.832e-04 | 1.43K tok/s | 610M done | eta 2d03h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:43:06 [all] step 10430/18464 loss=7.9103 ppl=2725.12 lr=2.828e-04 | 1.43K tok/s | 610M done | eta 2d03h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:46:57 [all] step 10440/18464 loss=7.9088 ppl=2721.22 lr=2.823e-04 | 1.42K tok/s | 611M done | eta 2d03h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:50:50 [all] step 10450/18464 loss=7.8879 ppl=2664.96 lr=2.819e-04 | 1.4K tok/s | 611M done | eta 2d03h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:55:21 [all] step 10460/18464 loss=7.9199 ppl=2751.42 lr=2.814e-04 | 1.21K tok/s | 611M done | eta 2d12h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 09:57:10 [all] checkpoint @ step 10463 (611M tokens, 7d15h06m elapsed) |
| 2026-09-20 10:01:35 [all] step 10470/18464 loss=7.9041 ppl=2708.38 lr=2.809e-04 | 876 tok/s | 612M done | eta 3d11h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 10:08:25 [all] step 10480/18464 loss=7.9132 ppl=2733.10 lr=2.805e-04 | 799 tok/s | 612M done | eta 3d18h57m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 10:15:12 [all] step 10490/18464 loss=7.9265 ppl=2769.65 lr=2.800e-04 | 805 tok/s | 612M done | eta 3d18h07m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 10:21:49 [all] step 10500/18464 loss=7.9275 ppl=2772.38 lr=2.796e-04 | 825 tok/s | 613M done | eta 3d15h49m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 10:22:20 [all] eval step 10500/18464 loss=7.9319 ppl=2784.63 (best 7.9319) |
| 2026-09-20 10:27:41 [all] checkpoint @ step 10508 (613M tokens, 7d15h37m elapsed) |
| 2026-09-20 10:29:03 [all] step 10510/18464 loss=7.9146 ppl=2736.97 lr=2.791e-04 | 754 tok/s | 613M done | eta 3d23h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 10:35:57 [all] step 10520/18464 loss=7.9399 ppl=2807.07 lr=2.786e-04 | 793 tok/s | 613M done | eta 3d19h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-20 10:43:39 [all] ============================================================================== |
| 2026-09-20 10:43:39 [all] hypernix0xv2 pretrain — stage=all run=runs/hypernix.3-mini |
| 2026-09-20 10:43:39 [all] tokenizer already at runs/hypernix.3-mini/tokenizer/tokenizer.json (use --force to retrain) |
| 2026-09-20 10:43:39 [all] packing 1.21B tokens -> 2.42 GB (uint16); 295.5 GB free on this volume |
| 2026-09-20 10:43:39 [all] already packed 1.21B tokens — nothing to do |
| 2026-09-20 10:43:41 [all] gpu=NVIDIA GeForce GTX 1080 capability=sm_61 torch=2.14.0+cu126 arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 |
| 2026-09-20 10:43:41 [all] patched BrewerAttention.forward (causal/window mask composition + is_causal fast path) |
| 2026-09-20 10:43:51 [all] built hypernix.3-mini — 48,706,048 parameters (48.7060 M) | 8L d_model=512 heads=8/2 d_ff=2203 ctx=512 vocab=32000 tied=True sliding_window=False |
| 2026-09-20 10:43:51 [all] 33.6% of parameters are the (tied) embedding matrix; 32,322,048 are 'body' parameters |
| 2026-09-20 10:43:51 [all] parameter memory: total=48,706,048 trainable=48,706,048 frozen=0 (185.8 MB) |
| 2026-09-20 10:43:51 [all] token bin: 1.21B tokens (2.42 GB), val tail = 2M tokens |
| 2026-09-20 10:43:51 [all] plan: 1.21B tokens = 18464 steps x 65.5K tokens/step (micro=8 ctx=512 accum=16) | peak_lr=6.00e-04 warmup=369 | deadline=9999h |
| 2026-09-20 10:43:51 [all] pre-flight (GTX 1080 = 8.87 TFLOPS fp32): 1d14h30m at 30% MFU, 3d05h00m at 15%, 6d00h22m at 8% — the budget needs better than 0.1% to land inside 9999h. The real number replaces this at step 100. |
| 2026-09-20 10:43:51 [all] resumed at step 10508/18464 (613M tokens, 7d15h37m already spent) |
| 2026-09-20 10:44:03 [all] calibration: 927 tok/s sustained -> 15d02h28m for the full 1.21B tokens (deadline 416d15h00m, 401d12h31m headroom) |
| 2026-09-20 10:44:18 [all] step 10510/18464 loss=7.9050 ppl=2710.92 lr=2.791e-04 | 2.44K tok/s | 613M done | eta 1d05h37m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 10:47:27 [all] step 10520/18464 loss=7.9216 ppl=2756.05 lr=2.786e-04 | 1.73K tok/s | 613M done | eta 1d17h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 10:51:53 [all] step 10530/18464 loss=7.9240 ppl=2762.73 lr=2.782e-04 | 1.23K tok/s | 614M done | eta 2d10h44m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 10:56:22 [all] step 10540/18464 loss=7.9141 ppl=2735.61 lr=2.777e-04 | 1.22K tok/s | 614M done | eta 2d11h06m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:00:47 [all] step 10550/18464 loss=7.8931 ppl=2678.65 lr=2.773e-04 | 1.24K tok/s | 614M done | eta 2d10h14m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:05:18 [all] step 10560/18464 loss=7.9192 ppl=2749.49 lr=2.768e-04 | 1.21K tok/s | 615M done | eta 2d11h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:10:17 [all] step 10570/18464 loss=7.9109 ppl=2726.75 lr=2.763e-04 | 1.1K tok/s | 615M done | eta 2d17h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:14:21 [all] checkpoint @ step 10577 (615M tokens, 7d16h07m elapsed) |
| 2026-09-20 11:16:20 [all] step 10580/18464 loss=7.9186 ppl=2747.96 lr=2.759e-04 | 902 tok/s | 615M done | eta 3d07h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:22:17 [all] step 10590/18464 loss=7.9103 ppl=2725.08 lr=2.754e-04 | 918 tok/s | 616M done | eta 3d06h02m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:27:45 [all] step 10600/18464 loss=7.9095 ppl=2723.09 lr=2.750e-04 | 999 tok/s | 616M done | eta 2d23h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:32:55 [all] step 10610/18464 loss=7.9265 ppl=2769.73 lr=2.745e-04 | 1.05K tok/s | 616M done | eta 2d19h45m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:38:30 [all] step 10620/18464 loss=7.9075 ppl=2717.67 lr=2.740e-04 | 979 tok/s | 617M done | eta 3d00h56m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:44:48 [all] checkpoint @ step 10629 (617M tokens, 7d16h38m elapsed) |
| 2026-09-20 11:45:30 [all] step 10630/18464 loss=7.8986 ppl=2693.50 lr=2.736e-04 | 779 tok/s | 617M done | eta 3d19h29m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:52:23 [all] step 10640/18464 loss=7.8844 ppl=2655.55 lr=2.731e-04 | 794 tok/s | 617M done | eta 3d17h41m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 11:59:16 [all] step 10650/18464 loss=7.9185 ppl=2747.59 lr=2.727e-04 | 793 tok/s | 618M done | eta 3d17h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:06:26 [all] step 10660/18464 loss=7.9193 ppl=2749.86 lr=2.722e-04 | 763 tok/s | 618M done | eta 3d21h06m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:13:39 [all] step 10670/18464 loss=7.8918 ppl=2675.19 lr=2.717e-04 | 757 tok/s | 618M done | eta 3d21h44m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:15:06 [all] checkpoint @ step 10672 (618M tokens, 7d17h08m elapsed) |
| 2026-09-20 12:20:50 [all] step 10680/18464 loss=7.9104 ppl=2725.47 lr=2.713e-04 | 761 tok/s | 619M done | eta 3d21h08m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:28:04 [all] step 10690/18464 loss=7.9284 ppl=2775.06 lr=2.708e-04 | 753 tok/s | 619M done | eta 3d21h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:35:17 [all] step 10700/18464 loss=7.9009 ppl=2699.84 lr=2.704e-04 | 758 tok/s | 619M done | eta 3d21h17m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:42:30 [all] step 10710/18464 loss=7.9152 ppl=2738.58 lr=2.699e-04 | 756 tok/s | 620M done | eta 3d21h18m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:45:26 [all] checkpoint @ step 10714 (620M tokens, 7d17h39m elapsed) |
| 2026-09-20 12:49:40 [all] step 10720/18464 loss=7.8996 ppl=2696.09 lr=2.695e-04 | 762 tok/s | 620M done | eta 3d20h27m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 12:56:53 [all] step 10730/18464 loss=7.8881 ppl=2665.28 lr=2.690e-04 | 757 tok/s | 620M done | eta 3d20h59m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:04:05 [all] step 10740/18464 loss=7.8811 ppl=2646.81 lr=2.685e-04 | 759 tok/s | 621M done | eta 3d20h38m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:11:16 [all] step 10750/18464 loss=7.9111 ppl=2727.36 lr=2.681e-04 | 759 tok/s | 621M done | eta 3d20h29m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:15:39 [all] checkpoint @ step 10756 (621M tokens, 7d18h09m elapsed) |
| 2026-09-20 13:18:32 [all] step 10760/18464 loss=7.8869 ppl=2662.20 lr=2.676e-04 | 752 tok/s | 621M done | eta 3d21h14m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:25:51 [all] step 10770/18464 loss=7.9032 ppl=2705.98 lr=2.672e-04 | 747 tok/s | 622M done | eta 3d21h48m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:33:06 [all] step 10780/18464 loss=7.9087 ppl=2720.85 lr=2.667e-04 | 753 tok/s | 622M done | eta 3d20h50m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:40:22 [all] step 10790/18464 loss=7.9008 ppl=2699.43 lr=2.663e-04 | 751 tok/s | 622M done | eta 3d21h02m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:45:51 [all] checkpoint @ step 10798 (622M tokens, 7d18h39m elapsed) |
| 2026-09-20 13:47:12 [all] step 10800/18464 loss=7.8913 ppl=2674.03 lr=2.658e-04 | 800 tok/s | 623M done | eta 3d15h12m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:53:26 [all] step 10810/18464 loss=7.8943 ppl=2682.01 lr=2.654e-04 | 876 tok/s | 623M done | eta 3d07h34m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 13:59:23 [all] step 10820/18464 loss=7.8932 ppl=2678.90 lr=2.649e-04 | 919 tok/s | 623M done | eta 3d03h41m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:07:06 [all] step 10830/18464 loss=7.8983 ppl=2692.60 lr=2.644e-04 | 707 tok/s | 624M done | eta 4d02h14m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:15:06 [all] step 10840/18464 loss=7.9136 ppl=2734.13 lr=2.640e-04 | 682 tok/s | 624M done | eta 4d05h43m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:15:53 [all] checkpoint @ step 10841 (624M tokens, 7d19h09m elapsed) |
| 2026-09-20 14:22:51 [all] step 10850/18464 loss=7.8906 ppl=2671.96 lr=2.635e-04 | 705 tok/s | 624M done | eta 4d02h17m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:30:30 [all] step 10860/18464 loss=7.9075 ppl=2717.63 lr=2.631e-04 | 714 tok/s | 624M done | eta 4d00h59m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:37:49 [all] step 10870/18464 loss=7.8991 ppl=2694.93 lr=2.626e-04 | 747 tok/s | 625M done | eta 3d20h30m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:43:36 [all] step 10880/18464 loss=7.8937 ppl=2680.22 lr=2.622e-04 | 944 tok/s | 625M done | eta 3d01h09m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:46:07 [all] checkpoint @ step 10884 (625M tokens, 7d19h39m elapsed) |
| 2026-09-20 14:50:47 [all] step 10890/18464 loss=7.8777 ppl=2637.70 lr=2.617e-04 | 761 tok/s | 625M done | eta 3d18h38m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 14:58:35 [all] step 10900/18464 loss=7.8862 ppl=2660.37 lr=2.613e-04 | 700 tok/s | 626M done | eta 4d02h22m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:06:06 [all] step 10910/18464 loss=7.8904 ppl=2671.59 lr=2.608e-04 | 727 tok/s | 626M done | eta 3d22h34m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:13:25 [all] step 10920/18464 loss=7.8857 ppl=2658.98 lr=2.604e-04 | 746 tok/s | 626M done | eta 3d20h05m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:16:21 [all] checkpoint @ step 10924 (627M tokens, 7d20h09m elapsed) |
| 2026-09-20 15:20:48 [all] step 10930/18464 loss=7.8857 ppl=2658.94 lr=2.599e-04 | 740 tok/s | 627M done | eta 3d20h38m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:28:22 [all] step 10940/18464 loss=7.8896 ppl=2669.38 lr=2.595e-04 | 721 tok/s | 627M done | eta 3d22h58m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:36:03 [all] step 10950/18464 loss=7.8807 ppl=2645.81 lr=2.590e-04 | 712 tok/s | 627M done | eta 4d00h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:43:50 [all] step 10960/18464 loss=7.8961 ppl=2686.77 lr=2.586e-04 | 701 tok/s | 628M done | eta 4d01h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:47:02 [all] checkpoint @ step 10964 (628M tokens, 7d20h40m elapsed) |
| 2026-09-20 15:51:50 [all] step 10970/18464 loss=7.8669 ppl=2609.54 lr=2.581e-04 | 683 tok/s | 628M done | eta 4d03h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 15:59:55 [all] step 10980/18464 loss=7.8875 ppl=2663.89 lr=2.577e-04 | 675 tok/s | 628M done | eta 4d04h57m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 16:07:56 [all] step 10990/18464 loss=7.8633 ppl=2600.22 lr=2.572e-04 | 681 tok/s | 629M done | eta 4d03h52m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 16:15:47 [all] step 11000/18464 loss=7.8707 ppl=2619.37 lr=2.567e-04 | 696 tok/s | 629M done | eta 4d01h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 16:16:20 [all] eval step 11000/18464 loss=7.8941 ppl=2681.30 (best 7.8941) |
| 2026-09-20 16:16:20 [all] checkpoint @ step 11000 (629M tokens, 7d21h09m elapsed) |
| 2026-09-20 16:23:19 [all] step 11010/18464 loss=7.8497 ppl=2564.97 lr=2.563e-04 | 724 tok/s | 629M done | eta 3d21h40m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 16:30:37 [all] step 11020/18464 loss=7.8836 ppl=2653.47 lr=2.558e-04 | 749 tok/s | 630M done | eta 3d18h25m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 16:39:03 [all] step 11030/18464 loss=7.8852 ppl=2657.77 lr=2.554e-04 | 647 tok/s | 630M done | eta 4d08h37m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 16:46:31 [all] checkpoint @ step 11039 (630M tokens, 7d21h40m elapsed) |
| 2026-09-20 16:47:21 [all] step 11040/18464 loss=7.8911 ppl=2673.36 lr=2.549e-04 | 659 tok/s | 630M done | eta 4d06h30m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 16:55:35 [all] step 11050/18464 loss=7.8766 ppl=2634.89 lr=2.545e-04 | 663 tok/s | 631M done | eta 4d05h50m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 17:03:57 [all] step 11060/18464 loss=7.8589 ppl=2588.72 lr=2.540e-04 | 652 tok/s | 631M done | eta 4d07h20m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 17:12:14 [all] step 11070/18464 loss=7.8794 ppl=2642.28 lr=2.536e-04 | 660 tok/s | 631M done | eta 4d05h55m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 17:17:06 [all] checkpoint @ step 11076 (632M tokens, 7d22h10m elapsed) |
| 2026-09-20 17:20:21 [all] step 11080/18464 loss=7.8675 ppl=2611.13 lr=2.531e-04 | 673 tok/s | 632M done | eta 4d03h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 17:28:31 [all] step 11090/18464 loss=7.8662 ppl=2607.63 lr=2.527e-04 | 669 tok/s | 632M done | eta 4d04h20m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 17:36:35 [all] step 11100/18464 loss=7.8591 ppl=2589.08 lr=2.522e-04 | 677 tok/s | 632M done | eta 4d02h58m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 17:44:41 [all] step 11110/18464 loss=7.8689 ppl=2614.70 lr=2.518e-04 | 674 tok/s | 633M done | eta 4d03h22m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 17:47:07 [all] checkpoint @ step 11113 (633M tokens, 7d22h40m elapsed) |
| 2026-09-20 17:52:49 [all] step 11120/18464 loss=7.8763 ppl=2634.04 lr=2.513e-04 | 672 tok/s | 633M done | eta 4d03h28m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:00:56 [all] step 11130/18464 loss=7.8827 ppl=2651.08 lr=2.509e-04 | 672 tok/s | 633M done | eta 4d03h22m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:09:26 [all] step 11140/18464 loss=7.8712 ppl=2620.61 lr=2.505e-04 | 643 tok/s | 634M done | eta 4d07h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:17:35 [all] step 11150/18464 loss=7.8726 ppl=2624.26 lr=2.500e-04 | 671 tok/s | 634M done | eta 4d03h16m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:17:35 [all] checkpoint @ step 11150 (634M tokens, 7d23h11m elapsed) |
| 2026-09-20 18:25:36 [all] step 11160/18464 loss=7.8658 ppl=2606.69 lr=2.496e-04 | 681 tok/s | 634M done | eta 4d01h38m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:33:39 [all] step 11170/18464 loss=7.8819 ppl=2648.84 lr=2.491e-04 | 678 tok/s | 635M done | eta 4d01h51m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:41:42 [all] step 11180/18464 loss=7.8767 ppl=2635.24 lr=2.487e-04 | 679 tok/s | 635M done | eta 4d01h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:48:06 [all] checkpoint @ step 11188 (635M tokens, 7d23h41m elapsed) |
| 2026-09-20 18:49:41 [all] step 11190/18464 loss=7.8563 ppl=2581.92 lr=2.482e-04 | 683 tok/s | 635M done | eta 4d00h52m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 18:57:48 [all] step 11200/18464 loss=7.8609 ppl=2593.92 lr=2.478e-04 | 673 tok/s | 636M done | eta 4d02h12m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 19:06:09 [all] step 11210/18464 loss=7.8745 ppl=2629.42 lr=2.473e-04 | 654 tok/s | 636M done | eta 4d04h59m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 19:14:14 [all] step 11220/18464 loss=7.8555 ppl=2579.89 lr=2.469e-04 | 676 tok/s | 636M done | eta 4d01h32m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 19:18:13 [all] checkpoint @ step 11225 (636M tokens, 8d00h11m elapsed) |
| 2026-09-20 19:22:12 [all] step 11230/18464 loss=7.8614 ppl=2595.04 lr=2.464e-04 | 686 tok/s | 637M done | eta 4d00h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 19:30:10 [all] step 11240/18464 loss=7.8669 ppl=2609.45 lr=2.460e-04 | 686 tok/s | 637M done | eta 3d23h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 19:38:09 [all] step 11250/18464 loss=7.8574 ppl=2584.73 lr=2.455e-04 | 684 tok/s | 637M done | eta 3d23h57m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 19:46:04 [all] step 11260/18464 loss=7.8474 ppl=2559.16 lr=2.451e-04 | 689 tok/s | 638M done | eta 3d23h07m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 19:48:27 [all] checkpoint @ step 11263 (638M tokens, 8d00h42m elapsed) |
| 2026-09-20 19:53:58 [all] step 11270/18464 loss=7.8683 ppl=2613.09 lr=2.447e-04 | 691 tok/s | 638M done | eta 3d22h46m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:01:55 [all] step 11280/18464 loss=7.8576 ppl=2585.22 lr=2.442e-04 | 688 tok/s | 638M done | eta 3d23h04m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:09:49 [all] step 11290/18464 loss=7.8547 ppl=2577.83 lr=2.438e-04 | 691 tok/s | 639M done | eta 3d22h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:17:43 [all] step 11300/18464 loss=7.8584 ppl=2587.46 lr=2.433e-04 | 691 tok/s | 639M done | eta 3d22h21m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:18:32 [all] checkpoint @ step 11301 (639M tokens, 8d01h12m elapsed) |
| 2026-09-20 20:25:40 [all] step 11310/18464 loss=7.8584 ppl=2587.47 lr=2.429e-04 | 688 tok/s | 639M done | eta 3d22h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:33:29 [all] step 11320/18464 loss=7.8830 ppl=2651.69 lr=2.424e-04 | 699 tok/s | 640M done | eta 3d21h04m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:41:20 [all] step 11330/18464 loss=7.8516 ppl=2569.75 lr=2.420e-04 | 695 tok/s | 640M done | eta 3d21h26m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:49:12 [all] step 11340/18464 loss=7.8308 ppl=2516.88 lr=2.415e-04 | 694 tok/s | 640M done | eta 3d21h23m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 20:49:12 [all] checkpoint @ step 11340 (640M tokens, 8d01h42m elapsed) |
| 2026-09-20 20:57:03 [all] step 11350/18464 loss=7.8579 ppl=2586.08 lr=2.411e-04 | 696 tok/s | 641M done | eta 3d20h58m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:04:52 [all] step 11360/18464 loss=7.8464 ppl=2556.46 lr=2.407e-04 | 698 tok/s | 641M done | eta 3d20h42m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:12:42 [all] step 11370/18464 loss=7.8576 ppl=2585.18 lr=2.402e-04 | 698 tok/s | 641M done | eta 3d20h28m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:19:44 [all] checkpoint @ step 11379 (641M tokens, 8d02h13m elapsed) |
| 2026-09-20 21:20:32 [all] step 11380/18464 loss=7.8443 ppl=2551.08 lr=2.398e-04 | 697 tok/s | 642M done | eta 3d20h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:28:24 [all] step 11390/18464 loss=7.8462 ppl=2556.01 lr=2.393e-04 | 694 tok/s | 642M done | eta 3d20h45m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:36:14 [all] step 11400/18464 loss=7.8470 ppl=2558.06 lr=2.389e-04 | 697 tok/s | 642M done | eta 3d20h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:44:05 [all] step 11410/18464 loss=7.8573 ppl=2584.49 lr=2.385e-04 | 696 tok/s | 643M done | eta 3d20h14m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:50:27 [all] checkpoint @ step 11418 (643M tokens, 8d02h44m elapsed) |
| 2026-09-20 21:52:00 [all] step 11420/18464 loss=7.8554 ppl=2579.72 lr=2.380e-04 | 690 tok/s | 643M done | eta 3d20h55m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 21:59:51 [all] step 11430/18464 loss=7.8566 ppl=2582.82 lr=2.376e-04 | 695 tok/s | 643M done | eta 3d20h07m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:07:45 [all] step 11440/18464 loss=7.8425 ppl=2546.68 lr=2.371e-04 | 692 tok/s | 643M done | eta 3d20h24m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:15:38 [all] step 11450/18464 loss=7.8378 ppl=2534.53 lr=2.367e-04 | 692 tok/s | 644M done | eta 3d20h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:21:11 [all] checkpoint @ step 11457 (644M tokens, 8d03h14m elapsed) |
| 2026-09-20 22:23:36 [all] step 11460/18464 loss=7.8484 ppl=2561.72 lr=2.362e-04 | 685 tok/s | 644M done | eta 3d21h06m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:31:42 [all] step 11470/18464 loss=7.8541 ppl=2576.19 lr=2.358e-04 | 674 tok/s | 644M done | eta 3d22h26m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:39:48 [all] step 11480/18464 loss=7.8447 ppl=2552.19 lr=2.354e-04 | 674 tok/s | 645M done | eta 3d22h17m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:47:53 [all] step 11490/18464 loss=7.8240 ppl=2499.97 lr=2.349e-04 | 677 tok/s | 645M done | eta 3d21h47m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:51:59 [all] checkpoint @ step 11495 (645M tokens, 8d03h45m elapsed) |
| 2026-09-20 22:56:06 [all] step 11500/18464 loss=7.8474 ppl=2559.19 lr=2.345e-04 | 664 tok/s | 645M done | eta 3d23h27m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 22:56:44 [all] eval step 11500/18464 loss=7.8593 ppl=2589.77 (best 7.8593) |
| 2026-09-20 23:04:59 [all] step 11510/18464 loss=7.8415 ppl=2544.11 lr=2.341e-04 | 615 tok/s | 646M done | eta 4d07h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 23:12:37 [all] step 11520/18464 loss=7.8293 ppl=2513.14 lr=2.336e-04 | 717 tok/s | 646M done | eta 3d16h10m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 23:20:38 [all] step 11530/18464 loss=7.8461 ppl=2555.63 lr=2.332e-04 | 681 tok/s | 646M done | eta 3d20h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 23:22:14 [all] checkpoint @ step 11532 (647M tokens, 8d04h15m elapsed) |
| 2026-09-20 23:28:56 [all] step 11540/18464 loss=7.8422 ppl=2545.83 lr=2.327e-04 | 657 tok/s | 647M done | eta 3d23h54m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 23:37:23 [all] step 11550/18464 loss=7.8260 ppl=2504.85 lr=2.323e-04 | 647 tok/s | 647M done | eta 4d01h20m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 23:45:54 [all] step 11560/18464 loss=7.8491 ppl=2563.49 lr=2.319e-04 | 642 tok/s | 647M done | eta 4d01h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-20 23:52:50 [all] checkpoint @ step 11568 (648M tokens, 8d04h46m elapsed) |
| 2026-09-20 23:54:33 [all] step 11570/18464 loss=7.8385 ppl=2536.50 lr=2.314e-04 | 631 tok/s | 648M done | eta 4d03h24m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 00:03:07 [all] step 11580/18464 loss=7.8595 ppl=2590.14 lr=2.310e-04 | 637 tok/s | 648M done | eta 4d02h24m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 00:11:42 [all] step 11590/18464 loss=7.8279 ppl=2509.56 lr=2.306e-04 | 637 tok/s | 648M done | eta 4d02h13m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 00:20:19 [all] step 11600/18464 loss=7.8183 ppl=2485.75 lr=2.301e-04 | 633 tok/s | 649M done | eta 4d02h40m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 00:22:54 [all] checkpoint @ step 11603 (649M tokens, 8d05h16m elapsed) |
| 2026-09-21 00:28:56 [all] step 11610/18464 loss=7.8316 ppl=2518.91 lr=2.297e-04 | 634 tok/s | 649M done | eta 4d02h27m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 00:37:39 [all] step 11620/18464 loss=7.8453 ppl=2553.62 lr=2.293e-04 | 627 tok/s | 649M done | eta 4d03h25m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 00:46:17 [all] step 11630/18464 loss=7.8400 ppl=2540.23 lr=2.288e-04 | 633 tok/s | 650M done | eta 4d02h15m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 00:53:17 [all] checkpoint @ step 11638 (650M tokens, 8d05h46m elapsed) |
| 2026-09-21 00:55:02 [all] step 11640/18464 loss=7.8173 ppl=2483.12 lr=2.284e-04 | 625 tok/s | 650M done | eta 4d03h26m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 01:03:44 [all] step 11650/18464 loss=7.8298 ppl=2514.43 lr=2.279e-04 | 627 tok/s | 650M done | eta 4d02h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 01:12:31 [all] step 11660/18464 loss=7.8414 ppl=2543.74 lr=2.275e-04 | 622 tok/s | 651M done | eta 4d03h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 01:21:16 [all] step 11670/18464 loss=7.8272 ppl=2507.94 lr=2.271e-04 | 623 tok/s | 651M done | eta 4d03h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 01:23:55 [all] checkpoint @ step 11673 (651M tokens, 8d06h17m elapsed) |
| 2026-09-21 01:30:04 [all] step 11680/18464 loss=7.8225 ppl=2496.02 lr=2.266e-04 | 621 tok/s | 651M done | eta 4d03h29m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 01:38:50 [all] step 11690/18464 loss=7.8320 ppl=2519.85 lr=2.262e-04 | 623 tok/s | 652M done | eta 4d02h58m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 01:47:31 [all] step 11700/18464 loss=7.8339 ppl=2524.70 lr=2.258e-04 | 630 tok/s | 652M done | eta 4d01h45m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 01:54:27 [all] checkpoint @ step 11708 (652M tokens, 8d06h48m elapsed) |
| 2026-09-21 01:56:12 [all] step 11710/18464 loss=7.8485 ppl=2561.97 lr=2.254e-04 | 629 tok/s | 652M done | eta 4d01h43m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:04:44 [all] step 11720/18464 loss=7.8164 ppl=2480.84 lr=2.249e-04 | 639 tok/s | 653M done | eta 4d00h01m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:13:10 [all] step 11730/18464 loss=7.8276 ppl=2508.96 lr=2.245e-04 | 648 tok/s | 653M done | eta 3d22h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:21:41 [all] step 11740/18464 loss=7.8440 ppl=2550.28 lr=2.241e-04 | 640 tok/s | 653M done | eta 3d23h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:25:06 [all] checkpoint @ step 11744 (653M tokens, 8d07h18m elapsed) |
| 2026-09-21 02:30:12 [all] step 11750/18464 loss=7.8319 ppl=2519.73 lr=2.236e-04 | 642 tok/s | 654M done | eta 3d23h11m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:38:40 [all] step 11760/18464 loss=7.8503 ppl=2566.47 lr=2.232e-04 | 645 tok/s | 654M done | eta 3d22h36m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:47:04 [all] step 11770/18464 loss=7.8061 ppl=2455.44 lr=2.228e-04 | 651 tok/s | 654M done | eta 3d21h38m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:55:35 [all] step 11780/18464 loss=7.8463 ppl=2556.26 lr=2.223e-04 | 640 tok/s | 655M done | eta 3d23h01m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 02:55:36 [all] checkpoint @ step 11780 (655M tokens, 8d07h49m elapsed) |
| 2026-09-21 03:04:05 [all] step 11790/18464 loss=7.8283 ppl=2510.54 lr=2.219e-04 | 643 tok/s | 655M done | eta 3d22h29m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 03:12:36 [all] step 11800/18464 loss=7.8305 ppl=2516.10 lr=2.215e-04 | 641 tok/s | 655M done | eta 3d22h35m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 03:21:08 [all] step 11810/18464 loss=7.8268 ppl=2506.98 lr=2.210e-04 | 640 tok/s | 656M done | eta 3d22h39m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 03:26:17 [all] checkpoint @ step 11816 (656M tokens, 8d08h19m elapsed) |
| 2026-09-21 03:29:42 [all] step 11820/18464 loss=7.8094 ppl=2463.64 lr=2.206e-04 | 638 tok/s | 656M done | eta 3d22h50m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 03:38:14 [all] step 11830/18464 loss=7.8391 ppl=2537.90 lr=2.202e-04 | 640 tok/s | 656M done | eta 3d22h25m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 03:46:48 [all] step 11840/18464 loss=7.8263 ppl=2505.61 lr=2.198e-04 | 638 tok/s | 657M done | eta 3d22h27m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 03:55:20 [all] step 11850/18464 loss=7.8344 ppl=2526.08 lr=2.193e-04 | 640 tok/s | 657M done | eta 3d22h05m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 03:57:02 [all] checkpoint @ step 11852 (657M tokens, 8d08h50m elapsed) |
| 2026-09-21 04:03:52 [all] step 11860/18464 loss=7.8370 ppl=2532.58 lr=2.189e-04 | 640 tok/s | 657M done | eta 3d21h53m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 04:12:23 [all] step 11870/18464 loss=7.8400 ppl=2540.17 lr=2.185e-04 | 641 tok/s | 658M done | eta 3d21h40m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 04:20:55 [all] step 11880/18464 loss=7.8362 ppl=2530.47 lr=2.181e-04 | 641 tok/s | 658M done | eta 3d21h31m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 04:27:46 [all] checkpoint @ step 11888 (658M tokens, 8d09h21m elapsed) |
| 2026-09-21 04:29:29 [all] step 11890/18464 loss=7.8321 ppl=2520.27 lr=2.176e-04 | 637 tok/s | 658M done | eta 3d21h52m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 04:37:59 [all] step 11900/18464 loss=7.8231 ppl=2497.62 lr=2.172e-04 | 642 tok/s | 659M done | eta 3d21h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 04:46:27 [all] step 11910/18464 loss=7.8195 ppl=2488.70 lr=2.168e-04 | 645 tok/s | 659M done | eta 3d20h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 04:55:00 [all] step 11920/18464 loss=7.8467 ppl=2557.20 lr=2.163e-04 | 639 tok/s | 659M done | eta 3d21h08m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 04:58:25 [all] checkpoint @ step 11924 (659M tokens, 8d09h52m elapsed) |
| 2026-09-21 05:03:35 [all] step 11930/18464 loss=7.8424 ppl=2546.34 lr=2.159e-04 | 636 tok/s | 660M done | eta 3d21h28m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 05:12:03 [all] step 11940/18464 loss=7.8498 ppl=2565.28 lr=2.155e-04 | 645 tok/s | 660M done | eta 3d20h03m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 05:20:32 [all] step 11950/18464 loss=7.8199 ppl=2489.61 lr=2.151e-04 | 643 tok/s | 660M done | eta 3d20h10m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 05:29:00 [all] step 11960/18464 loss=7.8494 ppl=2564.11 lr=2.146e-04 | 645 tok/s | 661M done | eta 3d19h49m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 05:29:01 [all] checkpoint @ step 11960 (661M tokens, 8d10h22m elapsed) |
| 2026-09-21 05:37:33 [all] step 11970/18464 loss=7.8306 ppl=2516.51 lr=2.142e-04 | 639 tok/s | 661M done | eta 3d20h33m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 05:46:03 [all] step 11980/18464 loss=7.8341 ppl=2525.31 lr=2.138e-04 | 643 tok/s | 661M done | eta 3d19h49m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 05:54:35 [all] step 11990/18464 loss=7.8460 ppl=2555.43 lr=2.134e-04 | 640 tok/s | 662M done | eta 3d20h00m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 05:59:41 [all] checkpoint @ step 11996 (662M tokens, 8d10h53m elapsed) |
| 2026-09-21 06:03:05 [all] step 12000/18464 loss=7.8233 ppl=2498.05 lr=2.130e-04 | 642 tok/s | 662M done | eta 3d19h40m | vram 0.74/3.37GB peak 3.27GB | rung 1 |
| 2026-09-21 06:03:44 [all] eval step 12000/18464 loss=7.8403 ppl=2540.86 (best 7.8403) |
| 2026-09-21 06:03:46 [all] sample @ 12000: ' in the or a which and as that with the more was and and the a have to the in the the an a to the most that the a in the they, a be this to the have.\nThe in the other of the' |
| 2026-09-21 06:03:47 [all] checkpoint @ step 12000 (662M tokens, 8d10h57m elapsed) |
| 2026-09-21 06:12:22 [all] step 12010/18464 loss=7.8160 ppl=2479.89 lr=2.125e-04 | 588 tok/s | 662M done | eta 4d03h52m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 06:20:54 [all] step 12020/18464 loss=7.8201 ppl=2490.03 lr=2.121e-04 | 641 tok/s | 663M done | eta 3d19h33m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 06:29:04 [all] step 12030/18464 loss=7.8470 ppl=2558.02 lr=2.117e-04 | 668 tok/s | 663M done | eta 3d15h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 06:34:00 [all] checkpoint @ step 12036 (663M tokens, 8d11h27m elapsed) |
| 2026-09-21 06:37:15 [all] step 12040/18464 loss=7.8373 ppl=2533.25 lr=2.113e-04 | 668 tok/s | 663M done | eta 3d15h30m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 06:45:18 [all] step 12050/18464 loss=7.8092 ppl=2463.09 lr=2.108e-04 | 678 tok/s | 663M done | eta 3d14h03m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 06:53:18 [all] step 12060/18464 loss=7.8221 ppl=2495.25 lr=2.104e-04 | 682 tok/s | 664M done | eta 3d13h25m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:00:59 [all] step 12070/18464 loss=7.8362 ppl=2530.60 lr=2.100e-04 | 711 tok/s | 664M done | eta 3d09h53m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:04:12 [all] checkpoint @ step 12074 (664M tokens, 8d11h57m elapsed) |
| 2026-09-21 07:09:27 [all] step 12080/18464 loss=7.8309 ppl=2517.23 lr=2.096e-04 | 645 tok/s | 664M done | eta 3d18h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:17:38 [all] step 12090/18464 loss=7.8500 ppl=2565.65 lr=2.092e-04 | 668 tok/s | 665M done | eta 3d14h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:25:27 [all] step 12100/18464 loss=7.8164 ppl=2480.97 lr=2.088e-04 | 698 tok/s | 665M done | eta 3d10h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:33:07 [all] step 12110/18464 loss=7.8085 ppl=2461.56 lr=2.083e-04 | 712 tok/s | 665M done | eta 3d09h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:34:40 [all] checkpoint @ step 12112 (666M tokens, 8d12h28m elapsed) |
| 2026-09-21 07:40:43 [all] step 12120/18464 loss=7.8192 ppl=2487.95 lr=2.079e-04 | 718 tok/s | 666M done | eta 3d08h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:48:16 [all] step 12130/18464 loss=7.7977 ppl=2434.90 lr=2.075e-04 | 724 tok/s | 666M done | eta 3d07h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 07:55:47 [all] step 12140/18464 loss=7.8303 ppl=2515.72 lr=2.071e-04 | 727 tok/s | 666M done | eta 3d07h07m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:02:55 [all] step 12150/18464 loss=7.8328 ppl=2522.07 lr=2.067e-04 | 764 tok/s | 667M done | eta 3d03h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:05:04 [all] checkpoint @ step 12153 (667M tokens, 8d12h58m elapsed) |
| 2026-09-21 08:10:03 [all] step 12160/18464 loss=7.8211 ppl=2492.72 lr=2.062e-04 | 767 tok/s | 667M done | eta 3d02h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:17:10 [all] step 12170/18464 loss=7.8421 ppl=2545.50 lr=2.058e-04 | 766 tok/s | 667M done | eta 3d02h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:24:16 [all] step 12180/18464 loss=7.8038 ppl=2449.90 lr=2.054e-04 | 770 tok/s | 668M done | eta 3d02h17m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:31:22 [all] step 12190/18464 loss=7.8114 ppl=2468.53 lr=2.050e-04 | 769 tok/s | 668M done | eta 3d02h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:35:35 [all] checkpoint @ step 12196 (668M tokens, 8d13h29m elapsed) |
| 2026-09-21 08:38:26 [all] step 12200/18464 loss=7.8158 ppl=2479.39 lr=2.046e-04 | 772 tok/s | 668M done | eta 3d01h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:45:28 [all] step 12210/18464 loss=7.8227 ppl=2496.72 lr=2.042e-04 | 777 tok/s | 669M done | eta 3d01h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:52:28 [all] step 12220/18464 loss=7.8300 ppl=2514.93 lr=2.038e-04 | 781 tok/s | 669M done | eta 3d00h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 08:59:34 [all] step 12230/18464 loss=7.8414 ppl=2543.81 lr=2.033e-04 | 768 tok/s | 669M done | eta 3d01h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:05:57 [all] checkpoint @ step 12239 (670M tokens, 8d13h59m elapsed) |
| 2026-09-21 09:06:38 [all] step 12240/18464 loss=7.8132 ppl=2473.09 lr=2.029e-04 | 773 tok/s | 670M done | eta 3d01h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:13:39 [all] step 12250/18464 loss=7.8112 ppl=2468.02 lr=2.025e-04 | 779 tok/s | 670M done | eta 3d00h36m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:20:32 [all] step 12260/18464 loss=7.8425 ppl=2546.47 lr=2.021e-04 | 793 tok/s | 670M done | eta 2d23h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:27:32 [all] step 12270/18464 loss=7.8170 ppl=2482.46 lr=2.017e-04 | 780 tok/s | 671M done | eta 3d00h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:34:28 [all] step 12280/18464 loss=7.8324 ppl=2521.07 lr=2.013e-04 | 789 tok/s | 671M done | eta 2d23h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:36:32 [all] checkpoint @ step 12283 (671M tokens, 8d14h30m elapsed) |
| 2026-09-21 09:41:21 [all] step 12290/18464 loss=7.8248 ppl=2501.88 lr=2.009e-04 | 793 tok/s | 671M done | eta 2d22h53m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:48:14 [all] step 12300/18464 loss=7.8279 ppl=2509.61 lr=2.004e-04 | 793 tok/s | 672M done | eta 2d22h47m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 09:55:08 [all] step 12310/18464 loss=7.8237 ppl=2499.05 lr=2.000e-04 | 793 tok/s | 672M done | eta 2d22h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:02:06 [all] step 12320/18464 loss=7.8381 ppl=2535.46 lr=1.996e-04 | 784 tok/s | 672M done | eta 2d23h21m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:06:53 [all] checkpoint @ step 12327 (673M tokens, 8d15h00m elapsed) |
| 2026-09-21 10:08:57 [all] step 12330/18464 loss=7.8371 ppl=2532.91 lr=1.992e-04 | 797 tok/s | 673M done | eta 2d22h03m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:15:51 [all] step 12340/18464 loss=7.8427 ppl=2546.98 lr=1.988e-04 | 792 tok/s | 673M done | eta 2d22h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:22:45 [all] step 12350/18464 loss=7.8303 ppl=2515.67 lr=1.984e-04 | 791 tok/s | 673M done | eta 2d22h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:29:39 [all] step 12360/18464 loss=7.8243 ppl=2500.53 lr=1.980e-04 | 791 tok/s | 674M done | eta 2d22h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:36:39 [all] step 12370/18464 loss=7.8137 ppl=2474.22 lr=1.976e-04 | 780 tok/s | 674M done | eta 2d23h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:37:21 [all] checkpoint @ step 12371 (674M tokens, 8d15h30m elapsed) |
| 2026-09-21 10:43:36 [all] step 12380/18464 loss=7.8248 ppl=2501.97 lr=1.972e-04 | 785 tok/s | 674M done | eta 2d22h34m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:50:32 [all] step 12390/18464 loss=7.8132 ppl=2473.09 lr=1.968e-04 | 788 tok/s | 675M done | eta 2d22h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 10:57:26 [all] step 12400/18464 loss=7.8181 ppl=2485.28 lr=1.964e-04 | 791 tok/s | 675M done | eta 2d21h49m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:04:20 [all] step 12410/18464 loss=7.8113 ppl=2468.27 lr=1.959e-04 | 792 tok/s | 675M done | eta 2d21h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:07:52 [all] checkpoint @ step 12415 (675M tokens, 8d16h01m elapsed) |
| 2026-09-21 11:11:20 [all] step 12420/18464 loss=7.8170 ppl=2482.42 lr=1.955e-04 | 780 tok/s | 676M done | eta 2d22h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:18:27 [all] step 12430/18464 loss=7.8288 ppl=2511.85 lr=1.951e-04 | 768 tok/s | 676M done | eta 2d23h30m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:25:48 [all] step 12440/18464 loss=7.8288 ppl=2511.96 lr=1.947e-04 | 743 tok/s | 676M done | eta 3d01h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:33:38 [all] step 12450/18464 loss=7.8287 ppl=2511.68 lr=1.943e-04 | 697 tok/s | 677M done | eta 3d06h33m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:38:09 [all] checkpoint @ step 12456 (677M tokens, 8d16h31m elapsed) |
| 2026-09-21 11:41:08 [all] step 12460/18464 loss=7.8255 ppl=2503.57 lr=1.939e-04 | 729 tok/s | 677M done | eta 3d02h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:48:58 [all] step 12470/18464 loss=7.8087 ppl=2461.93 lr=1.935e-04 | 697 tok/s | 677M done | eta 3d06h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 11:56:52 [all] step 12480/18464 loss=7.8174 ppl=2483.56 lr=1.931e-04 | 691 tok/s | 678M done | eta 3d06h52m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:04:19 [all] step 12490/18464 loss=7.8065 ppl=2456.64 lr=1.927e-04 | 734 tok/s | 678M done | eta 3d02h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:08:48 [all] checkpoint @ step 12496 (678M tokens, 8d17h02m elapsed) |
| 2026-09-21 12:11:43 [all] step 12500/18464 loss=7.8300 ppl=2514.82 lr=1.923e-04 | 737 tok/s | 678M done | eta 3d01h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:12:17 [all] eval step 12500/18464 loss=7.8402 ppl=2540.78 (best 7.8402) |
| 2026-09-21 12:19:41 [all] step 12510/18464 loss=7.8213 ppl=2493.11 lr=1.919e-04 | 686 tok/s | 679M done | eta 3d07h00m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:27:21 [all] step 12520/18464 loss=7.8239 ppl=2499.52 lr=1.915e-04 | 713 tok/s | 679M done | eta 3d03h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:34:51 [all] step 12530/18464 loss=7.8051 ppl=2453.14 lr=1.911e-04 | 728 tok/s | 679M done | eta 3d02h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:39:20 [all] checkpoint @ step 12536 (679M tokens, 8d17h32m elapsed) |
| 2026-09-21 12:42:20 [all] step 12540/18464 loss=7.8195 ppl=2488.59 lr=1.907e-04 | 730 tok/s | 680M done | eta 3d01h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:50:02 [all] step 12550/18464 loss=7.8281 ppl=2510.05 lr=1.903e-04 | 709 tok/s | 680M done | eta 3d03h54m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 12:58:32 [all] step 12560/18464 loss=7.8138 ppl=2474.42 lr=1.899e-04 | 642 tok/s | 680M done | eta 3d11h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 13:06:35 [all] step 12570/18464 loss=7.8154 ppl=2478.58 lr=1.895e-04 | 679 tok/s | 681M done | eta 3d07h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 13:09:46 [all] checkpoint @ step 12574 (681M tokens, 8d18h03m elapsed) |
| 2026-09-21 13:14:32 [all] step 12580/18464 loss=7.8235 ppl=2498.75 lr=1.891e-04 | 686 tok/s | 681M done | eta 3d06h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 13:22:30 [all] step 12590/18464 loss=7.8161 ppl=2480.16 lr=1.887e-04 | 686 tok/s | 681M done | eta 3d05h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 13:30:31 [all] step 12600/18464 loss=7.8365 ppl=2531.29 lr=1.883e-04 | 682 tok/s | 682M done | eta 3d06h14m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 13:38:34 [all] step 12610/18464 loss=7.8347 ppl=2526.87 lr=1.879e-04 | 678 tok/s | 682M done | eta 3d06h38m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 13:40:12 [all] checkpoint @ step 12612 (682M tokens, 8d18h33m elapsed) |
| 2026-09-21 13:46:41 [all] step 12620/18464 loss=7.8219 ppl=2494.73 lr=1.875e-04 | 673 tok/s | 682M done | eta 3d07h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 13:54:50 [all] step 12630/18464 loss=7.8317 ppl=2519.22 lr=1.871e-04 | 670 tok/s | 682M done | eta 3d07h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 14:02:59 [all] step 12640/18464 loss=7.8370 ppl=2532.72 lr=1.867e-04 | 670 tok/s | 683M done | eta 3d07h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 14:10:22 [all] checkpoint @ step 12649 (683M tokens, 8d19h03m elapsed) |
| 2026-09-21 14:11:12 [all] step 12650/18464 loss=7.8637 ppl=2601.00 lr=1.863e-04 | 665 tok/s | 683M done | eta 3d07h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 14:19:26 [all] step 12660/18464 loss=7.8219 ppl=2494.65 lr=1.859e-04 | 663 tok/s | 683M done | eta 3d07h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 14:27:44 [all] step 12670/18464 loss=7.8279 ppl=2509.60 lr=1.855e-04 | 658 tok/s | 684M done | eta 3d08h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 14:36:06 [all] step 12680/18464 loss=7.8414 ppl=2543.84 lr=1.851e-04 | 654 tok/s | 684M done | eta 3d08h30m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 14:41:07 [all] checkpoint @ step 12686 (684M tokens, 8d19h34m elapsed) |
| 2026-09-21 14:44:35 [all] step 12690/18464 loss=7.8204 ppl=2491.01 lr=1.847e-04 | 643 tok/s | 684M done | eta 3d09h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 14:53:23 [all] step 12700/18464 loss=7.8191 ppl=2487.69 lr=1.843e-04 | 621 tok/s | 685M done | eta 3d12h26m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 15:02:06 [all] step 12710/18464 loss=7.8293 ppl=2513.05 lr=1.839e-04 | 627 tok/s | 685M done | eta 3d11h35m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 15:10:52 [all] step 12720/18464 loss=7.8274 ppl=2508.44 lr=1.835e-04 | 623 tok/s | 685M done | eta 3d11h53m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 15:11:46 [all] checkpoint @ step 12721 (685M tokens, 8d20h05m elapsed) |
| 2026-09-21 15:19:38 [all] step 12730/18464 loss=7.8488 ppl=2562.68 lr=1.831e-04 | 622 tok/s | 686M done | eta 3d11h53m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 15:28:31 [all] step 12740/18464 loss=7.8184 ppl=2485.91 lr=1.827e-04 | 616 tok/s | 686M done | eta 3d12h38m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 15:37:25 [all] step 12750/18464 loss=7.8170 ppl=2482.45 lr=1.824e-04 | 613 tok/s | 686M done | eta 3d12h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 15:41:55 [all] checkpoint @ step 12755 (687M tokens, 8d20h35m elapsed) |
| 2026-09-21 15:46:28 [all] step 12760/18464 loss=7.7967 ppl=2432.49 lr=1.820e-04 | 603 tok/s | 687M done | eta 3d14h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 15:55:38 [all] step 12770/18464 loss=7.8291 ppl=2512.58 lr=1.816e-04 | 596 tok/s | 687M done | eta 3d14h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 16:04:38 [all] step 12780/18464 loss=7.8137 ppl=2474.16 lr=1.812e-04 | 607 tok/s | 687M done | eta 3d13h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 16:12:39 [all] checkpoint @ step 12789 (688M tokens, 8d21h06m elapsed) |
| 2026-09-21 16:13:30 [all] step 12790/18464 loss=7.8221 ppl=2495.02 lr=1.808e-04 | 615 tok/s | 688M done | eta 3d11h56m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 16:22:25 [all] step 12800/18464 loss=7.8338 ppl=2524.59 lr=1.804e-04 | 613 tok/s | 688M done | eta 3d12h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 16:31:34 [all] step 12810/18464 loss=7.8371 ppl=2532.83 lr=1.800e-04 | 597 tok/s | 688M done | eta 3d14h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 16:40:59 [all] step 12820/18464 loss=7.8226 ppl=2496.42 lr=1.796e-04 | 580 tok/s | 689M done | eta 3d16h31m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 16:42:54 [all] checkpoint @ step 12822 (689M tokens, 8d21h36m elapsed) |
| 2026-09-21 16:50:29 [all] step 12830/18464 loss=7.8483 ppl=2561.28 lr=1.792e-04 | 574 tok/s | 689M done | eta 3d17h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 16:59:59 [all] step 12840/18464 loss=7.8019 ppl=2445.29 lr=1.788e-04 | 575 tok/s | 689M done | eta 3d16h59m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 17:09:28 [all] step 12850/18464 loss=7.8161 ppl=2480.26 lr=1.785e-04 | 575 tok/s | 690M done | eta 3d16h47m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 17:13:14 [all] checkpoint @ step 12854 (690M tokens, 8d22h06m elapsed) |
| 2026-09-21 17:18:47 [all] step 12860/18464 loss=7.7996 ppl=2439.54 lr=1.781e-04 | 586 tok/s | 690M done | eta 3d15h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 17:28:16 [all] step 12870/18464 loss=7.8443 ppl=2551.05 lr=1.777e-04 | 577 tok/s | 690M done | eta 3d16h17m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 17:37:55 [all] step 12880/18464 loss=7.8400 ppl=2540.25 lr=1.773e-04 | 566 tok/s | 691M done | eta 3d17h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 17:43:35 [all] checkpoint @ step 12886 (691M tokens, 8d22h37m elapsed) |
| 2026-09-21 17:47:15 [all] step 12890/18464 loss=7.8217 ppl=2494.11 lr=1.769e-04 | 585 tok/s | 691M done | eta 3d14h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 17:56:33 [all] step 12900/18464 loss=7.8208 ppl=2491.97 lr=1.765e-04 | 587 tok/s | 691M done | eta 3d14h14m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:05:39 [all] step 12910/18464 loss=7.8439 ppl=2550.16 lr=1.761e-04 | 600 tok/s | 692M done | eta 3d12h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:13:58 [all] step 12920/18464 loss=7.8293 ppl=2513.06 lr=1.757e-04 | 657 tok/s | 692M done | eta 3d04h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:13:58 [all] checkpoint @ step 12920 (692M tokens, 8d23h07m elapsed) |
| 2026-09-21 18:21:50 [all] step 12930/18464 loss=7.8211 ppl=2492.61 lr=1.754e-04 | 694 tok/s | 692M done | eta 3d00h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:29:22 [all] step 12940/18464 loss=7.8268 ppl=2507.01 lr=1.750e-04 | 725 tok/s | 693M done | eta 2d21h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:36:35 [all] step 12950/18464 loss=7.8353 ppl=2528.25 lr=1.746e-04 | 757 tok/s | 693M done | eta 2d18h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:43:40 [all] step 12960/18464 loss=7.8283 ppl=2510.63 lr=1.742e-04 | 771 tok/s | 693M done | eta 2d16h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:44:23 [all] checkpoint @ step 12961 (693M tokens, 8d23h37m elapsed) |
| 2026-09-21 18:50:52 [all] step 12970/18464 loss=7.8137 ppl=2474.24 lr=1.738e-04 | 759 tok/s | 694M done | eta 2d17h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 18:58:16 [all] step 12980/18464 loss=7.8239 ppl=2499.57 lr=1.734e-04 | 737 tok/s | 694M done | eta 2d19h41m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 19:06:00 [all] step 12990/18464 loss=7.8224 ppl=2495.90 lr=1.731e-04 | 706 tok/s | 694M done | eta 2d22h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 19:13:54 [all] step 13000/18464 loss=7.8390 ppl=2537.67 lr=1.727e-04 | 691 tok/s | 695M done | eta 2d23h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 19:14:30 [all] eval step 13000/18464 loss=7.8402 ppl=2540.67 (best 7.8402) |
| 2026-09-21 19:14:31 [all] checkpoint @ step 13000 (695M tokens, 9d00h08m elapsed) |
| 2026-09-21 19:22:17 [all] step 13010/18464 loss=7.8019 ppl=2445.20 lr=1.723e-04 | 652 tok/s | 695M done | eta 3d04h09m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 19:30:01 [all] step 13020/18464 loss=7.8230 ppl=2497.29 lr=1.719e-04 | 707 tok/s | 695M done | eta 2d22h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 19:37:45 [all] step 13030/18464 loss=7.8317 ppl=2519.26 lr=1.715e-04 | 705 tok/s | 696M done | eta 2d22h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 19:45:04 [all] checkpoint @ step 13039 (696M tokens, 9d00h38m elapsed) |
| 2026-09-21 19:45:53 [all] step 13040/18464 loss=7.8135 ppl=2473.83 lr=1.712e-04 | 672 tok/s | 696M done | eta 3d01h25m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 19:53:52 [all] step 13050/18464 loss=7.8182 ppl=2485.53 lr=1.708e-04 | 684 tok/s | 696M done | eta 3d00h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:01:59 [all] step 13060/18464 loss=7.8241 ppl=2500.17 lr=1.704e-04 | 672 tok/s | 697M done | eta 3d01h14m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:10:05 [all] step 13070/18464 loss=7.8106 ppl=2466.73 lr=1.700e-04 | 675 tok/s | 697M done | eta 3d00h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:15:45 [all] checkpoint @ step 13077 (697M tokens, 9d01h09m elapsed) |
| 2026-09-21 20:18:14 [all] step 13080/18464 loss=7.8128 ppl=2472.15 lr=1.697e-04 | 671 tok/s | 697M done | eta 3d01h03m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:26:09 [all] step 13090/18464 loss=7.8100 ppl=2465.02 lr=1.693e-04 | 690 tok/s | 698M done | eta 2d22h56m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:34:12 [all] step 13100/18464 loss=7.8463 ppl=2556.21 lr=1.689e-04 | 678 tok/s | 698M done | eta 3d00h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:42:29 [all] step 13110/18464 loss=7.7998 ppl=2440.13 lr=1.685e-04 | 660 tok/s | 698M done | eta 3d01h47m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:46:33 [all] checkpoint @ step 13115 (698M tokens, 9d01h40m elapsed) |
| 2026-09-21 20:50:42 [all] step 13120/18464 loss=7.8327 ppl=2521.62 lr=1.681e-04 | 663 tok/s | 699M done | eta 3d01h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 20:58:52 [all] step 13130/18464 loss=7.8203 ppl=2490.66 lr=1.678e-04 | 670 tok/s | 699M done | eta 3d00h28m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 21:07:00 [all] step 13140/18464 loss=7.8176 ppl=2483.99 lr=1.674e-04 | 670 tok/s | 699M done | eta 3d00h17m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 21:15:09 [all] step 13150/18464 loss=7.8254 ppl=2503.48 lr=1.670e-04 | 671 tok/s | 700M done | eta 3d00h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 21:16:47 [all] checkpoint @ step 13152 (700M tokens, 9d02h10m elapsed) |
| 2026-09-21 21:23:20 [all] step 13160/18464 loss=7.8202 ppl=2490.44 lr=1.667e-04 | 668 tok/s | 700M done | eta 3d00h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 21:31:50 [all] step 13170/18464 loss=7.8493 ppl=2563.87 lr=1.663e-04 | 642 tok/s | 700M done | eta 3d03h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 21:40:23 [all] step 13180/18464 loss=7.8436 ppl=2549.48 lr=1.659e-04 | 639 tok/s | 701M done | eta 3d03h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 21:47:08 [all] checkpoint @ step 13188 (701M tokens, 9d02h40m elapsed) |
| 2026-09-21 21:48:45 [all] step 13190/18464 loss=7.8301 ppl=2515.21 lr=1.655e-04 | 653 tok/s | 701M done | eta 3d01h33m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 21:57:00 [all] step 13200/18464 loss=7.8383 ppl=2535.98 lr=1.652e-04 | 662 tok/s | 701M done | eta 3d00h21m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 22:05:23 [all] step 13210/18464 loss=7.8247 ppl=2501.70 lr=1.648e-04 | 651 tok/s | 701M done | eta 3d01h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 22:13:45 [all] step 13220/18464 loss=7.8259 ppl=2504.69 lr=1.644e-04 | 653 tok/s | 702M done | eta 3d01h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 22:17:56 [all] checkpoint @ step 13225 (702M tokens, 9d03h11m elapsed) |
| 2026-09-21 22:22:14 [all] step 13230/18464 loss=7.8461 ppl=2555.83 lr=1.641e-04 | 644 tok/s | 702M done | eta 3d01h57m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 22:31:06 [all] step 13240/18464 loss=7.8179 ppl=2484.76 lr=1.637e-04 | 616 tok/s | 702M done | eta 3d05h14m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 22:39:52 [all] step 13250/18464 loss=7.8540 ppl=2576.07 lr=1.633e-04 | 624 tok/s | 703M done | eta 3d04h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 22:48:40 [all] step 13260/18464 loss=7.8226 ppl=2496.31 lr=1.629e-04 | 620 tok/s | 703M done | eta 3d04h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 22:48:41 [all] checkpoint @ step 13260 (703M tokens, 9d03h42m elapsed) |
| 2026-09-21 22:57:27 [all] step 13270/18464 loss=7.8183 ppl=2485.75 lr=1.626e-04 | 622 tok/s | 703M done | eta 3d03h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 23:06:15 [all] step 13280/18464 loss=7.8167 ppl=2481.70 lr=1.622e-04 | 621 tok/s | 704M done | eta 3d04h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 23:15:16 [all] step 13290/18464 loss=7.8128 ppl=2471.97 lr=1.618e-04 | 605 tok/s | 704M done | eta 3d05h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 23:18:57 [all] checkpoint @ step 13294 (704M tokens, 9d04h12m elapsed) |
| 2026-09-21 23:24:27 [all] step 13300/18464 loss=7.8244 ppl=2500.96 lr=1.615e-04 | 595 tok/s | 704M done | eta 3d06h56m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 23:33:46 [all] step 13310/18464 loss=7.8111 ppl=2467.78 lr=1.611e-04 | 586 tok/s | 705M done | eta 3d08h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 23:42:58 [all] step 13320/18464 loss=7.8184 ppl=2485.95 lr=1.607e-04 | 593 tok/s | 705M done | eta 3d06h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-21 23:49:27 [all] checkpoint @ step 13327 (705M tokens, 9d04h43m elapsed) |
| 2026-09-21 23:52:10 [all] step 13330/18464 loss=7.7954 ppl=2429.47 lr=1.604e-04 | 593 tok/s | 705M done | eta 3d06h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 00:01:23 [all] step 13340/18464 loss=7.8292 ppl=2512.95 lr=1.600e-04 | 593 tok/s | 706M done | eta 3d06h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 00:10:37 [all] step 13350/18464 loss=7.8207 ppl=2491.53 lr=1.596e-04 | 591 tok/s | 706M done | eta 3d06h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 00:19:44 [all] step 13360/18464 loss=7.8178 ppl=2484.50 lr=1.593e-04 | 599 tok/s | 706M done | eta 3d05h34m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 00:19:45 [all] checkpoint @ step 13360 (706M tokens, 9d05h13m elapsed) |
| 2026-09-22 00:28:55 [all] step 13370/18464 loss=7.8401 ppl=2540.38 lr=1.589e-04 | 595 tok/s | 707M done | eta 3d05h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 00:38:06 [all] step 13380/18464 loss=7.8248 ppl=2501.87 lr=1.586e-04 | 595 tok/s | 707M done | eta 3d05h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 00:47:21 [all] step 13390/18464 loss=7.8097 ppl=2464.49 lr=1.582e-04 | 591 tok/s | 707M done | eta 3d06h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 00:50:07 [all] checkpoint @ step 13393 (707M tokens, 9d05h43m elapsed) |
| 2026-09-22 00:56:33 [all] step 13400/18464 loss=7.8209 ppl=2492.26 lr=1.578e-04 | 593 tok/s | 708M done | eta 3d05h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 01:05:39 [all] step 13410/18464 loss=7.8329 ppl=2522.35 lr=1.575e-04 | 600 tok/s | 708M done | eta 3d04h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 01:14:55 [all] step 13420/18464 loss=7.8577 ppl=2585.55 lr=1.571e-04 | 590 tok/s | 708M done | eta 3d05h47m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 01:20:30 [all] checkpoint @ step 13426 (709M tokens, 9d06h14m elapsed) |
| 2026-09-22 01:24:10 [all] step 13430/18464 loss=7.8145 ppl=2476.13 lr=1.568e-04 | 590 tok/s | 709M done | eta 3d05h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 01:33:20 [all] step 13440/18464 loss=7.8183 ppl=2485.69 lr=1.564e-04 | 596 tok/s | 709M done | eta 3d04h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 01:42:31 [all] step 13450/18464 loss=7.8279 ppl=2509.53 lr=1.560e-04 | 595 tok/s | 709M done | eta 3d04h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 01:50:47 [all] checkpoint @ step 13459 (710M tokens, 9d06h44m elapsed) |
| 2026-09-22 01:51:41 [all] step 13460/18464 loss=7.8480 ppl=2560.67 lr=1.557e-04 | 596 tok/s | 710M done | eta 3d04h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 02:00:42 [all] step 13470/18464 loss=7.8171 ppl=2482.77 lr=1.553e-04 | 605 tok/s | 710M done | eta 3d03h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 02:09:48 [all] step 13480/18464 loss=7.8396 ppl=2539.16 lr=1.550e-04 | 600 tok/s | 710M done | eta 3d03h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 02:18:53 [all] step 13490/18464 loss=7.8196 ppl=2488.97 lr=1.546e-04 | 602 tok/s | 711M done | eta 3d03h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 02:21:34 [all] checkpoint @ step 13493 (711M tokens, 9d07h15m elapsed) |
| 2026-09-22 02:27:54 [all] step 13500/18464 loss=7.8252 ppl=2502.91 lr=1.543e-04 | 605 tok/s | 711M done | eta 3d02h40m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 02:28:35 [all] eval step 13500/18464 loss=7.8401 ppl=2540.58 (best 7.8401) |
| 2026-09-22 02:37:37 [all] step 13510/18464 loss=7.8140 ppl=2474.91 lr=1.539e-04 | 563 tok/s | 711M done | eta 3d08h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 02:46:37 [all] step 13520/18464 loss=7.8178 ppl=2484.49 lr=1.535e-04 | 606 tok/s | 712M done | eta 3d02h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 02:52:05 [all] checkpoint @ step 13526 (712M tokens, 9d07h45m elapsed) |
| 2026-09-22 02:55:43 [all] step 13530/18464 loss=7.8383 ppl=2535.99 lr=1.532e-04 | 600 tok/s | 712M done | eta 3d02h49m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 03:04:48 [all] step 13540/18464 loss=7.8287 ppl=2511.63 lr=1.528e-04 | 601 tok/s | 712M done | eta 3d02h33m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 03:13:54 [all] step 13550/18464 loss=7.8232 ppl=2498.01 lr=1.525e-04 | 600 tok/s | 713M done | eta 3d02h30m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 03:22:57 [all] step 13560/18464 loss=7.8148 ppl=2477.10 lr=1.521e-04 | 604 tok/s | 713M done | eta 3d01h56m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 03:22:57 [all] checkpoint @ step 13560 (713M tokens, 9d08h16m elapsed) |
| 2026-09-22 03:32:01 [all] step 13570/18464 loss=7.8276 ppl=2508.90 lr=1.518e-04 | 602 tok/s | 713M done | eta 3d01h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 03:41:02 [all] step 13580/18464 loss=7.8282 ppl=2510.44 lr=1.514e-04 | 606 tok/s | 714M done | eta 3d01h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 03:50:05 [all] step 13590/18464 loss=7.8114 ppl=2468.67 lr=1.511e-04 | 603 tok/s | 714M done | eta 3d01h36m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 03:53:44 [all] checkpoint @ step 13594 (714M tokens, 9d08h47m elapsed) |
| 2026-09-22 03:59:08 [all] step 13600/18464 loss=7.8184 ppl=2485.86 lr=1.507e-04 | 604 tok/s | 714M done | eta 3d01h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 04:08:08 [all] step 13610/18464 loss=7.8495 ppl=2564.35 lr=1.504e-04 | 606 tok/s | 715M done | eta 3d00h52m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 04:17:11 [all] step 13620/18464 loss=7.8435 ppl=2549.20 lr=1.500e-04 | 604 tok/s | 715M done | eta 3d01h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 04:24:24 [all] checkpoint @ step 13628 (715M tokens, 9d09h17m elapsed) |
| 2026-09-22 04:26:14 [all] step 13630/18464 loss=7.8314 ppl=2518.47 lr=1.497e-04 | 604 tok/s | 715M done | eta 3d00h53m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 04:35:15 [all] step 13640/18464 loss=7.8302 ppl=2515.56 lr=1.493e-04 | 606 tok/s | 716M done | eta 3d00h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 04:44:10 [all] step 13650/18464 loss=7.8444 ppl=2551.35 lr=1.490e-04 | 613 tok/s | 716M done | eta 2d23h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 04:53:05 [all] step 13660/18464 loss=7.8084 ppl=2461.13 lr=1.486e-04 | 612 tok/s | 716M done | eta 2d23h26m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 04:54:56 [all] checkpoint @ step 13662 (716M tokens, 9d09h48m elapsed) |
| 2026-09-22 05:02:01 [all] step 13670/18464 loss=7.8151 ppl=2477.75 lr=1.483e-04 | 612 tok/s | 717M done | eta 2d23h20m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 05:10:57 [all] step 13680/18464 loss=7.8140 ppl=2475.07 lr=1.479e-04 | 612 tok/s | 717M done | eta 2d23h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 05:19:50 [all] step 13690/18464 loss=7.8333 ppl=2523.34 lr=1.476e-04 | 614 tok/s | 717M done | eta 2d22h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 05:25:12 [all] checkpoint @ step 13696 (717M tokens, 9d10h18m elapsed) |
| 2026-09-22 05:28:44 [all] step 13700/18464 loss=7.8351 ppl=2527.73 lr=1.472e-04 | 614 tok/s | 718M done | eta 2d22h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 05:37:32 [all] step 13710/18464 loss=7.7944 ppl=2426.91 lr=1.469e-04 | 620 tok/s | 718M done | eta 2d21h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 05:46:24 [all] step 13720/18464 loss=7.8095 ppl=2463.85 lr=1.466e-04 | 616 tok/s | 718M done | eta 2d22h03m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 05:55:15 [all] step 13730/18464 loss=7.8391 ppl=2538.03 lr=1.462e-04 | 617 tok/s | 719M done | eta 2d21h52m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 05:55:15 [all] checkpoint @ step 13730 (719M tokens, 9d10h48m elapsed) |
| 2026-09-22 06:04:07 [all] step 13740/18464 loss=7.8334 ppl=2523.54 lr=1.459e-04 | 616 tok/s | 719M done | eta 2d21h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 06:12:57 [all] step 13750/18464 loss=7.8441 ppl=2550.61 lr=1.455e-04 | 618 tok/s | 719M done | eta 2d21h24m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 06:21:47 [all] step 13760/18464 loss=7.7826 ppl=2398.48 lr=1.452e-04 | 618 tok/s | 720M done | eta 2d21h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 06:25:18 [all] checkpoint @ step 13764 (720M tokens, 9d11h18m elapsed) |
| 2026-09-22 06:30:36 [all] step 13770/18464 loss=7.8212 ppl=2492.98 lr=1.448e-04 | 620 tok/s | 720M done | eta 2d20h54m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 06:39:12 [all] step 13780/18464 loss=7.8295 ppl=2513.77 lr=1.445e-04 | 634 tok/s | 720M done | eta 2d19h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 06:47:37 [all] step 13790/18464 loss=7.8126 ppl=2471.56 lr=1.442e-04 | 649 tok/s | 721M done | eta 2d17h30m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 06:55:50 [all] step 13800/18464 loss=7.8209 ppl=2492.13 lr=1.438e-04 | 665 tok/s | 721M done | eta 2d15h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 06:55:50 [all] checkpoint @ step 13800 (721M tokens, 9d11h49m elapsed) |
| 2026-09-22 07:03:59 [all] step 13810/18464 loss=7.8077 ppl=2459.37 lr=1.435e-04 | 670 tok/s | 721M done | eta 2d15h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:11:53 [all] step 13820/18464 loss=7.8218 ppl=2494.50 lr=1.431e-04 | 691 tok/s | 721M done | eta 2d13h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:19:23 [all] step 13830/18464 loss=7.8349 ppl=2527.29 lr=1.428e-04 | 728 tok/s | 722M done | eta 2d09h54m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:26:03 [all] checkpoint @ step 13839 (722M tokens, 9d12h19m elapsed) |
| 2026-09-22 07:26:48 [all] step 13840/18464 loss=7.8259 ppl=2504.63 lr=1.425e-04 | 736 tok/s | 722M done | eta 2d09h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:34:25 [all] step 13850/18464 loss=7.8253 ppl=2503.16 lr=1.421e-04 | 718 tok/s | 722M done | eta 2d10h28m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:41:56 [all] step 13860/18464 loss=7.8188 ppl=2486.89 lr=1.418e-04 | 727 tok/s | 723M done | eta 2d09h40m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:49:22 [all] step 13870/18464 loss=7.8478 ppl=2560.21 lr=1.415e-04 | 734 tok/s | 723M done | eta 2d09h00m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:56:41 [all] step 13880/18464 loss=7.8268 ppl=2506.83 lr=1.411e-04 | 747 tok/s | 723M done | eta 2d07h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 07:56:41 [all] checkpoint @ step 13880 (723M tokens, 9d12h50m elapsed) |
| 2026-09-22 08:04:01 [all] step 13890/18464 loss=7.8065 ppl=2456.40 lr=1.408e-04 | 744 tok/s | 724M done | eta 2d07h57m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:11:17 [all] step 13900/18464 loss=7.8298 ppl=2514.42 lr=1.405e-04 | 752 tok/s | 724M done | eta 2d07h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:18:29 [all] step 13910/18464 loss=7.8485 ppl=2561.96 lr=1.401e-04 | 759 tok/s | 724M done | eta 2d06h38m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:25:40 [all] step 13920/18464 loss=7.8166 ppl=2481.35 lr=1.398e-04 | 760 tok/s | 725M done | eta 2d06h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:27:08 [all] checkpoint @ step 13922 (725M tokens, 9d13h20m elapsed) |
| 2026-09-22 08:32:53 [all] step 13930/18464 loss=7.8164 ppl=2481.08 lr=1.395e-04 | 757 tok/s | 725M done | eta 2d06h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:40:03 [all] step 13940/18464 loss=7.8274 ppl=2508.31 lr=1.391e-04 | 762 tok/s | 725M done | eta 2d06h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:47:11 [all] step 13950/18464 loss=7.8161 ppl=2480.18 lr=1.388e-04 | 765 tok/s | 726M done | eta 2d05h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:54:18 [all] step 13960/18464 loss=7.8151 ppl=2477.83 lr=1.385e-04 | 768 tok/s | 726M done | eta 2d05h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 08:57:09 [all] checkpoint @ step 13964 (726M tokens, 9d13h50m elapsed) |
| 2026-09-22 09:01:25 [all] step 13970/18464 loss=7.8077 ppl=2459.56 lr=1.381e-04 | 768 tok/s | 726M done | eta 2d05h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:08:32 [all] step 13980/18464 loss=7.8291 ppl=2512.55 lr=1.378e-04 | 768 tok/s | 727M done | eta 2d05h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:15:38 [all] step 13990/18464 loss=7.8302 ppl=2515.47 lr=1.375e-04 | 769 tok/s | 727M done | eta 2d04h56m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:22:43 [all] step 14000/18464 loss=7.8172 ppl=2482.99 lr=1.371e-04 | 770 tok/s | 727M done | eta 2d04h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:23:15 [all] eval step 14000/18464 loss=7.8401 ppl=2540.48 (best 7.8401) |
| 2026-09-22 09:23:17 [all] sample @ 14000: ' to the to the the the the this the the the the a as of of the a that the a a a the.\nThe which a a a the and the the the the it a of in a a a the the the.' |
| 2026-09-22 09:23:18 [all] checkpoint @ step 14000 (727M tokens, 9d14h16m elapsed) |
| 2026-09-22 09:30:24 [all] step 14010/18464 loss=7.8117 ppl=2469.40 lr=1.368e-04 | 710 tok/s | 728M done | eta 2d09h03m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:37:30 [all] step 14020/18464 loss=7.8277 ppl=2509.23 lr=1.365e-04 | 770 tok/s | 728M done | eta 2d04h31m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:44:34 [all] step 14030/18464 loss=7.8155 ppl=2478.63 lr=1.362e-04 | 771 tok/s | 728M done | eta 2d04h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:51:43 [all] step 14040/18464 loss=7.8362 ppl=2530.68 lr=1.358e-04 | 765 tok/s | 729M done | eta 2d04h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 09:53:52 [all] checkpoint @ step 14043 (729M tokens, 9d14h47m elapsed) |
| 2026-09-22 09:58:54 [all] step 14050/18464 loss=7.8223 ppl=2495.56 lr=1.355e-04 | 759 tok/s | 729M done | eta 2d04h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:06:13 [all] step 14060/18464 loss=7.8215 ppl=2493.71 lr=1.352e-04 | 746 tok/s | 729M done | eta 2d05h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:13:34 [all] step 14070/18464 loss=7.8262 ppl=2505.43 lr=1.349e-04 | 744 tok/s | 730M done | eta 2d05h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:20:57 [all] step 14080/18464 loss=7.8113 ppl=2468.29 lr=1.345e-04 | 739 tok/s | 730M done | eta 2d05h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:23:58 [all] checkpoint @ step 14084 (730M tokens, 9d15h17m elapsed) |
| 2026-09-22 10:28:34 [all] step 14090/18464 loss=7.8289 ppl=2512.04 lr=1.342e-04 | 718 tok/s | 730M done | eta 2d07h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:36:24 [all] step 14100/18464 loss=7.8225 ppl=2496.14 lr=1.339e-04 | 696 tok/s | 731M done | eta 2d09h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:44:30 [all] step 14110/18464 loss=7.8374 ppl=2533.53 lr=1.336e-04 | 675 tok/s | 731M done | eta 2d10h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:52:44 [all] step 14120/18464 loss=7.8156 ppl=2478.98 lr=1.332e-04 | 662 tok/s | 731M done | eta 2d11h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 10:54:24 [all] checkpoint @ step 14122 (731M tokens, 9d15h47m elapsed) |
| 2026-09-22 11:00:58 [all] step 14130/18464 loss=7.8151 ppl=2477.84 lr=1.329e-04 | 664 tok/s | 732M done | eta 2d11h26m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 11:09:00 [all] step 14140/18464 loss=7.8069 ppl=2457.59 lr=1.326e-04 | 680 tok/s | 732M done | eta 2d09h54m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 11:17:13 [all] step 14150/18464 loss=7.8163 ppl=2480.79 lr=1.323e-04 | 665 tok/s | 732M done | eta 2d11h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 11:24:53 [all] checkpoint @ step 14159 (733M tokens, 9d16h18m elapsed) |
| 2026-09-22 11:25:46 [all] step 14160/18464 loss=7.8384 ppl=2536.20 lr=1.320e-04 | 639 tok/s | 733M done | eta 2d13h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 11:34:26 [all] step 14170/18464 loss=7.8153 ppl=2478.13 lr=1.317e-04 | 630 tok/s | 733M done | eta 2d14h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 11:43:05 [all] step 14180/18464 loss=7.8264 ppl=2505.94 lr=1.313e-04 | 631 tok/s | 733M done | eta 2d13h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 11:51:46 [all] step 14190/18464 loss=7.8289 ppl=2512.24 lr=1.310e-04 | 629 tok/s | 734M done | eta 2d13h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 11:55:15 [all] checkpoint @ step 14194 (734M tokens, 9d16h48m elapsed) |
| 2026-09-22 12:00:27 [all] step 14200/18464 loss=7.8276 ppl=2508.85 lr=1.307e-04 | 628 tok/s | 734M done | eta 2d13h47m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 12:09:04 [all] step 14210/18464 loss=7.8279 ppl=2509.56 lr=1.304e-04 | 635 tok/s | 734M done | eta 2d13h00m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 12:17:39 [all] step 14220/18464 loss=7.8328 ppl=2522.00 lr=1.301e-04 | 636 tok/s | 735M done | eta 2d12h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 12:25:21 [all] checkpoint @ step 14229 (735M tokens, 9d17h18m elapsed) |
| 2026-09-22 12:26:12 [all] step 14230/18464 loss=7.8206 ppl=2491.30 lr=1.298e-04 | 639 tok/s | 735M done | eta 2d12h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 12:34:46 [all] step 14240/18464 loss=7.8140 ppl=2475.03 lr=1.294e-04 | 638 tok/s | 735M done | eta 2d12h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 12:43:16 [all] step 14250/18464 loss=7.8200 ppl=2489.88 lr=1.291e-04 | 642 tok/s | 736M done | eta 2d11h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 12:51:51 [all] step 14260/18464 loss=7.8122 ppl=2470.64 lr=1.288e-04 | 637 tok/s | 736M done | eta 2d12h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 12:56:08 [all] checkpoint @ step 14265 (736M tokens, 9d17h49m elapsed) |
| 2026-09-22 13:00:27 [all] step 14270/18464 loss=7.8175 ppl=2483.72 lr=1.285e-04 | 635 tok/s | 736M done | eta 2d12h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 13:09:03 [all] step 14280/18464 loss=7.8283 ppl=2510.58 lr=1.282e-04 | 635 tok/s | 737M done | eta 2d11h59m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 13:17:43 [all] step 14290/18464 loss=7.8314 ppl=2518.38 lr=1.279e-04 | 630 tok/s | 737M done | eta 2d12h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 13:26:25 [all] step 14300/18464 loss=7.8371 ppl=2532.88 lr=1.276e-04 | 627 tok/s | 737M done | eta 2d12h26m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 13:26:26 [all] checkpoint @ step 14300 (737M tokens, 9d18h20m elapsed) |
| 2026-09-22 13:35:10 [all] step 14310/18464 loss=7.8366 ppl=2531.57 lr=1.273e-04 | 624 tok/s | 738M done | eta 2d12h33m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 13:44:00 [all] step 14320/18464 loss=7.8212 ppl=2492.78 lr=1.269e-04 | 618 tok/s | 738M done | eta 2d13h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 13:52:50 [all] step 14330/18464 loss=7.7993 ppl=2438.83 lr=1.266e-04 | 619 tok/s | 738M done | eta 2d12h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 13:57:17 [all] checkpoint @ step 14335 (738M tokens, 9d18h50m elapsed) |
| 2026-09-22 14:01:43 [all] step 14340/18464 loss=7.8477 ppl=2559.93 lr=1.263e-04 | 614 tok/s | 739M done | eta 2d13h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 14:10:40 [all] step 14350/18464 loss=7.8302 ppl=2515.39 lr=1.260e-04 | 611 tok/s | 739M done | eta 2d13h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 14:19:46 [all] step 14360/18464 loss=7.8319 ppl=2519.74 lr=1.257e-04 | 600 tok/s | 739M done | eta 2d14h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 14:28:02 [all] checkpoint @ step 14369 (739M tokens, 9d19h21m elapsed) |
| 2026-09-22 14:28:58 [all] step 14370/18464 loss=7.8219 ppl=2494.75 lr=1.254e-04 | 594 tok/s | 740M done | eta 2d14h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 14:38:14 [all] step 14380/18464 loss=7.8162 ppl=2480.52 lr=1.251e-04 | 589 tok/s | 740M done | eta 2d15h09m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 14:47:29 [all] step 14390/18464 loss=7.8054 ppl=2453.88 lr=1.248e-04 | 591 tok/s | 740M done | eta 2d14h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 14:56:49 [all] step 14400/18464 loss=7.8280 ppl=2509.89 lr=1.245e-04 | 585 tok/s | 740M done | eta 2d15h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 14:58:43 [all] checkpoint @ step 14402 (741M tokens, 9d19h52m elapsed) |
| 2026-09-22 15:06:09 [all] step 14410/18464 loss=7.8197 ppl=2489.08 lr=1.242e-04 | 585 tok/s | 741M done | eta 2d15h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 15:15:34 [all] step 14420/18464 loss=7.8038 ppl=2449.91 lr=1.239e-04 | 580 tok/s | 741M done | eta 2d15h25m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 15:25:09 [all] step 14430/18464 loss=7.8287 ppl=2511.54 lr=1.236e-04 | 569 tok/s | 741M done | eta 2d16h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 15:29:00 [all] checkpoint @ step 14434 (742M tokens, 9d20h22m elapsed) |
| 2026-09-22 15:34:48 [all] step 14440/18464 loss=7.8346 ppl=2526.64 lr=1.233e-04 | 566 tok/s | 742M done | eta 2d16h41m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 15:44:06 [all] step 14450/18464 loss=7.8153 ppl=2478.21 lr=1.230e-04 | 587 tok/s | 742M done | eta 2d14h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 15:52:38 [all] step 14460/18464 loss=7.8178 ppl=2484.52 lr=1.227e-04 | 640 tok/s | 742M done | eta 2d08h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 15:59:10 [all] checkpoint @ step 14468 (743M tokens, 9d20h52m elapsed) |
| 2026-09-22 16:00:46 [all] step 14470/18464 loss=7.8161 ppl=2480.28 lr=1.224e-04 | 671 tok/s | 743M done | eta 2d06h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:08:28 [all] step 14480/18464 loss=7.7993 ppl=2438.85 lr=1.221e-04 | 708 tok/s | 743M done | eta 2d03h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:16:01 [all] step 14490/18464 loss=7.8410 ppl=2542.83 lr=1.218e-04 | 724 tok/s | 743M done | eta 2d01h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:23:37 [all] step 14500/18464 loss=7.8166 ppl=2481.54 lr=1.215e-04 | 718 tok/s | 744M done | eta 2d02h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:24:13 [all] eval step 14500/18464 loss=7.8401 ppl=2540.41 (best 7.8401) |
| 2026-09-22 16:29:27 [all] checkpoint @ step 14507 (744M tokens, 9d21h23m elapsed) |
| 2026-09-22 16:31:40 [all] step 14510/18464 loss=7.8143 ppl=2475.69 lr=1.212e-04 | 679 tok/s | 744M done | eta 2d05h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:38:58 [all] step 14520/18464 loss=7.8108 ppl=2467.03 lr=1.209e-04 | 749 tok/s | 744M done | eta 1d23h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:46:06 [all] step 14530/18464 loss=7.8374 ppl=2533.52 lr=1.206e-04 | 764 tok/s | 745M done | eta 1d22h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:53:07 [all] step 14540/18464 loss=7.8072 ppl=2458.21 lr=1.203e-04 | 779 tok/s | 745M done | eta 1d21h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:59:58 [all] step 14550/18464 loss=7.8282 ppl=2510.49 lr=1.200e-04 | 798 tok/s | 745M done | eta 1d20h38m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 16:59:58 [all] checkpoint @ step 14550 (745M tokens, 9d21h53m elapsed) |
| 2026-09-22 17:06:47 [all] step 14560/18464 loss=7.8163 ppl=2480.68 lr=1.197e-04 | 801 tok/s | 746M done | eta 1d20h22m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 17:14:07 [all] step 14570/18464 loss=7.8331 ppl=2522.76 lr=1.194e-04 | 745 tok/s | 746M done | eta 1d23h34m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 17:21:47 [all] step 14580/18464 loss=7.8188 ppl=2487.02 lr=1.191e-04 | 712 tok/s | 746M done | eta 2d01h40m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 17:29:38 [all] step 14590/18464 loss=7.8145 ppl=2476.37 lr=1.188e-04 | 696 tok/s | 747M done | eta 2d02h38m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 17:30:28 [all] checkpoint @ step 14591 (747M tokens, 9d22h24m elapsed) |
| 2026-09-22 17:37:57 [all] step 14600/18464 loss=7.8067 ppl=2457.11 lr=1.185e-04 | 656 tok/s | 747M done | eta 2d05h36m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 17:46:29 [all] step 14610/18464 loss=7.8236 ppl=2498.93 lr=1.182e-04 | 641 tok/s | 747M done | eta 2d06h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 17:55:01 [all] step 14620/18464 loss=7.8285 ppl=2511.06 lr=1.180e-04 | 639 tok/s | 748M done | eta 2d06h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:01:02 [all] checkpoint @ step 14627 (748M tokens, 9d22h54m elapsed) |
| 2026-09-22 18:03:30 [all] step 14630/18464 loss=7.8329 ppl=2522.17 lr=1.177e-04 | 644 tok/s | 748M done | eta 2d06h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:11:27 [all] step 14640/18464 loss=7.7989 ppl=2437.80 lr=1.174e-04 | 687 tok/s | 748M done | eta 2d02h41m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:18:48 [all] step 14650/18464 loss=7.8164 ppl=2480.93 lr=1.171e-04 | 743 tok/s | 749M done | eta 1d22h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:25:51 [all] step 14660/18464 loss=7.8052 ppl=2453.24 lr=1.168e-04 | 775 tok/s | 749M done | eta 1d20h41m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:31:21 [all] checkpoint @ step 14668 (749M tokens, 9d23h24m elapsed) |
| 2026-09-22 18:32:42 [all] step 14670/18464 loss=7.8360 ppl=2530.14 lr=1.165e-04 | 798 tok/s | 749M done | eta 1d19h17m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:39:21 [all] step 14680/18464 loss=7.8213 ppl=2493.20 lr=1.162e-04 | 822 tok/s | 750M done | eta 1d17h54m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:45:56 [all] step 14690/18464 loss=7.8168 ppl=2481.87 lr=1.159e-04 | 830 tok/s | 750M done | eta 1d17h24m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:52:25 [all] step 14700/18464 loss=7.8131 ppl=2472.78 lr=1.157e-04 | 840 tok/s | 750M done | eta 1d16h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 18:58:59 [all] step 14710/18464 loss=7.7944 ppl=2426.88 lr=1.154e-04 | 832 tok/s | 751M done | eta 1d17h03m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 19:01:45 [all] checkpoint @ step 14714 (751M tokens, 9d23h55m elapsed) |
| 2026-09-22 19:06:18 [all] step 14720/18464 loss=7.8310 ppl=2517.38 lr=1.151e-04 | 747 tok/s | 751M done | eta 1d21h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 19:14:47 [all] step 14730/18464 loss=7.8201 ppl=2490.12 lr=1.148e-04 | 644 tok/s | 751M done | eta 2d04h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 19:22:53 [all] step 14740/18464 loss=7.8237 ppl=2499.13 lr=1.145e-04 | 674 tok/s | 752M done | eta 2d02h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 19:30:42 [all] step 14750/18464 loss=7.8121 ppl=2470.27 lr=1.142e-04 | 699 tok/s | 752M done | eta 2d00h20m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 19:32:15 [all] checkpoint @ step 14752 (752M tokens, 10d00h25m elapsed) |
| 2026-09-22 19:38:39 [all] step 14760/18464 loss=7.8168 ppl=2481.94 lr=1.140e-04 | 687 tok/s | 752M done | eta 2d01h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 19:46:49 [all] step 14770/18464 loss=7.8258 ppl=2504.49 lr=1.137e-04 | 669 tok/s | 753M done | eta 2d02h17m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 19:55:04 [all] step 14780/18464 loss=7.8043 ppl=2451.12 lr=1.134e-04 | 662 tok/s | 753M done | eta 2d02h41m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 20:02:48 [all] checkpoint @ step 14789 (753M tokens, 10d00h56m elapsed) |
| 2026-09-22 20:03:40 [all] step 14790/18464 loss=7.8185 ppl=2486.20 lr=1.131e-04 | 635 tok/s | 753M done | eta 2d04h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 20:12:27 [all] step 14800/18464 loss=7.8298 ppl=2514.47 lr=1.128e-04 | 622 tok/s | 754M done | eta 2d05h35m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 20:21:19 [all] step 14810/18464 loss=7.8259 ppl=2504.62 lr=1.126e-04 | 616 tok/s | 754M done | eta 2d06h00m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 20:29:57 [all] step 14820/18464 loss=7.8154 ppl=2478.48 lr=1.123e-04 | 632 tok/s | 754M done | eta 2d04h28m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 20:33:27 [all] checkpoint @ step 14824 (754M tokens, 10d01h27m elapsed) |
| 2026-09-22 20:38:40 [all] step 14830/18464 loss=7.8167 ppl=2481.76 lr=1.120e-04 | 627 tok/s | 755M done | eta 2d04h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 20:47:26 [all] step 14840/18464 loss=7.8113 ppl=2468.46 lr=1.117e-04 | 623 tok/s | 755M done | eta 2d04h57m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 20:56:11 [all] step 14850/18464 loss=7.8156 ppl=2479.05 lr=1.115e-04 | 624 tok/s | 755M done | eta 2d04h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 21:03:58 [all] checkpoint @ step 14859 (756M tokens, 10d01h57m elapsed) |
| 2026-09-22 21:04:49 [all] step 14860/18464 loss=7.8088 ppl=2462.05 lr=1.112e-04 | 632 tok/s | 756M done | eta 2d03h54m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 21:13:22 [all] step 14870/18464 loss=7.8269 ppl=2507.13 lr=1.109e-04 | 639 tok/s | 756M done | eta 2d03h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 21:21:52 [all] step 14880/18464 loss=7.8312 ppl=2517.99 lr=1.106e-04 | 642 tok/s | 756M done | eta 2d02h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 21:30:27 [all] step 14890/18464 loss=7.8102 ppl=2465.68 lr=1.104e-04 | 637 tok/s | 757M done | eta 2d03h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 21:34:49 [all] checkpoint @ step 14895 (757M tokens, 10d02h28m elapsed) |
| 2026-09-22 21:39:15 [all] step 14900/18464 loss=7.8205 ppl=2491.09 lr=1.101e-04 | 621 tok/s | 757M done | eta 2d04h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 21:48:09 [all] step 14910/18464 loss=7.8210 ppl=2492.37 lr=1.098e-04 | 613 tok/s | 757M done | eta 2d04h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 21:57:14 [all] step 14920/18464 loss=7.8256 ppl=2503.89 lr=1.095e-04 | 601 tok/s | 758M done | eta 2d05h40m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 22:05:34 [all] checkpoint @ step 14929 (758M tokens, 10d02h59m elapsed) |
| 2026-09-22 22:06:31 [all] step 14930/18464 loss=7.8283 ppl=2510.71 lr=1.093e-04 | 589 tok/s | 758M done | eta 2d06h38m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 22:15:49 [all] step 14940/18464 loss=7.8130 ppl=2472.60 lr=1.090e-04 | 587 tok/s | 758M done | eta 2d06h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 22:25:28 [all] step 14950/18464 loss=7.8100 ppl=2465.06 lr=1.087e-04 | 566 tok/s | 759M done | eta 2d08h30m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 22:35:00 [all] step 14960/18464 loss=7.8087 ppl=2461.97 lr=1.085e-04 | 573 tok/s | 759M done | eta 2d07h40m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 22:35:58 [all] checkpoint @ step 14961 (759M tokens, 10d03h29m elapsed) |
| 2026-09-22 22:44:19 [all] step 14970/18464 loss=7.8258 ppl=2504.39 lr=1.082e-04 | 586 tok/s | 759M done | eta 2d06h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 22:53:31 [all] step 14980/18464 loss=7.8100 ppl=2465.05 lr=1.079e-04 | 594 tok/s | 759M done | eta 2d05h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 23:03:03 [all] step 14990/18464 loss=7.8323 ppl=2520.63 lr=1.077e-04 | 573 tok/s | 760M done | eta 2d07h09m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 23:06:49 [all] checkpoint @ step 14994 (760M tokens, 10d04h00m elapsed) |
| 2026-09-22 23:12:29 [all] step 15000/18464 loss=7.8154 ppl=2478.46 lr=1.074e-04 | 579 tok/s | 760M done | eta 2d06h26m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 23:13:13 [all] eval step 15000/18464 loss=7.8400 ppl=2540.31 (best 7.8400) |
| 2026-09-22 23:13:14 [all] checkpoint @ step 15000 (760M tokens, 10d04h06m elapsed) |
| 2026-09-22 23:22:37 [all] step 15010/18464 loss=7.8234 ppl=2498.40 lr=1.071e-04 | 538 tok/s | 760M done | eta 2d10h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 23:32:00 [all] step 15020/18464 loss=7.8138 ppl=2474.44 lr=1.069e-04 | 583 tok/s | 761M done | eta 2d05h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 23:41:22 [all] step 15030/18464 loss=7.8119 ppl=2469.71 lr=1.066e-04 | 583 tok/s | 761M done | eta 2d05h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-22 23:43:14 [all] checkpoint @ step 15032 (761M tokens, 10d04h36m elapsed) |
| 2026-09-22 23:50:44 [all] step 15040/18464 loss=7.8103 ppl=2465.81 lr=1.063e-04 | 583 tok/s | 761M done | eta 2d05h28m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 00:00:09 [all] step 15050/18464 loss=7.8256 ppl=2503.77 lr=1.061e-04 | 580 tok/s | 762M done | eta 2d05h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 00:09:29 [all] step 15060/18464 loss=7.8289 ppl=2512.29 lr=1.058e-04 | 585 tok/s | 762M done | eta 2d04h57m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 00:14:09 [all] checkpoint @ step 15065 (762M tokens, 10d05h07m elapsed) |
| 2026-09-23 00:18:51 [all] step 15070/18464 loss=7.8120 ppl=2470.02 lr=1.056e-04 | 583 tok/s | 762M done | eta 2d04h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 00:28:08 [all] step 15080/18464 loss=7.8263 ppl=2505.58 lr=1.053e-04 | 588 tok/s | 763M done | eta 2d04h22m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 00:37:31 [all] step 15090/18464 loss=7.8231 ppl=2497.56 lr=1.050e-04 | 582 tok/s | 763M done | eta 2d04h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 00:44:56 [all] checkpoint @ step 15098 (763M tokens, 10d05h38m elapsed) |
| 2026-09-23 00:46:48 [all] step 15100/18464 loss=7.8056 ppl=2454.39 lr=1.048e-04 | 588 tok/s | 763M done | eta 2d04h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 00:56:04 [all] step 15110/18464 loss=7.8407 ppl=2541.95 lr=1.045e-04 | 590 tok/s | 764M done | eta 2d03h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 01:05:25 [all] step 15120/18464 loss=7.8183 ppl=2485.76 lr=1.043e-04 | 583 tok/s | 764M done | eta 2d04h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 01:14:44 [all] step 15130/18464 loss=7.8195 ppl=2488.60 lr=1.040e-04 | 587 tok/s | 764M done | eta 2d03h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 01:15:41 [all] checkpoint @ step 15131 (764M tokens, 10d06h09m elapsed) |
| 2026-09-23 01:24:06 [all] step 15140/18464 loss=7.8114 ppl=2468.66 lr=1.038e-04 | 583 tok/s | 765M done | eta 2d03h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 01:33:14 [all] step 15150/18464 loss=7.8138 ppl=2474.62 lr=1.035e-04 | 598 tok/s | 765M done | eta 2d02h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 01:42:23 [all] step 15160/18464 loss=7.8099 ppl=2464.84 lr=1.032e-04 | 598 tok/s | 765M done | eta 2d02h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 01:46:03 [all] checkpoint @ step 15164 (766M tokens, 10d06h39m elapsed) |
| 2026-09-23 01:51:31 [all] step 15170/18464 loss=7.8435 ppl=2549.05 lr=1.030e-04 | 597 tok/s | 766M done | eta 2d02h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 02:00:38 [all] step 15180/18464 loss=7.8380 ppl=2535.18 lr=1.027e-04 | 600 tok/s | 766M done | eta 2d01h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 02:09:38 [all] step 15190/18464 loss=7.8190 ppl=2487.45 lr=1.025e-04 | 607 tok/s | 766M done | eta 2d01h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 02:16:57 [all] checkpoint @ step 15198 (767M tokens, 10d07h10m elapsed) |
| 2026-09-23 02:18:46 [all] step 15200/18464 loss=7.8214 ppl=2493.34 lr=1.022e-04 | 597 tok/s | 767M done | eta 2d01h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 02:27:52 [all] step 15210/18464 loss=7.8248 ppl=2501.81 lr=1.020e-04 | 600 tok/s | 767M done | eta 2d01h20m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 02:37:01 [all] step 15220/18464 loss=7.8209 ppl=2492.05 lr=1.017e-04 | 598 tok/s | 767M done | eta 2d01h24m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 02:46:10 [all] step 15230/18464 loss=7.8218 ppl=2494.51 lr=1.015e-04 | 596 tok/s | 768M done | eta 2d01h21m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 02:47:04 [all] checkpoint @ step 15231 (768M tokens, 10d07h40m elapsed) |
| 2026-09-23 02:55:16 [all] step 15240/18464 loss=7.8304 ppl=2515.82 lr=1.012e-04 | 600 tok/s | 768M done | eta 2d00h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 03:04:19 [all] step 15250/18464 loss=7.8283 ppl=2510.76 lr=1.010e-04 | 603 tok/s | 768M done | eta 2d00h28m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 03:13:24 [all] step 15260/18464 loss=7.8208 ppl=2491.78 lr=1.007e-04 | 602 tok/s | 769M done | eta 2d00h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 03:17:56 [all] checkpoint @ step 15265 (769M tokens, 10d08h11m elapsed) |
| 2026-09-23 03:22:27 [all] step 15270/18464 loss=7.8324 ppl=2520.96 lr=1.005e-04 | 603 tok/s | 769M done | eta 2d00h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 03:31:26 [all] step 15280/18464 loss=7.8113 ppl=2468.23 lr=1.002e-04 | 609 tok/s | 769M done | eta 1d23h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 03:40:21 [all] step 15290/18464 loss=7.8118 ppl=2469.67 lr=9.999e-05 | 612 tok/s | 770M done | eta 1d23h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 03:48:22 [all] checkpoint @ step 15299 (770M tokens, 10d08h41m elapsed) |
| 2026-09-23 03:49:17 [all] step 15300/18464 loss=7.8265 ppl=2506.18 lr=9.975e-05 | 611 tok/s | 770M done | eta 1d23h07m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 03:58:10 [all] step 15310/18464 loss=7.8255 ppl=2503.71 lr=9.950e-05 | 615 tok/s | 770M done | eta 1d22h41m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 04:07:05 [all] step 15320/18464 loss=7.8163 ppl=2480.75 lr=9.926e-05 | 613 tok/s | 771M done | eta 1d22h41m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 04:15:58 [all] step 15330/18464 loss=7.8414 ppl=2543.70 lr=9.902e-05 | 614 tok/s | 771M done | eta 1d22h25m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 04:18:42 [all] checkpoint @ step 15333 (771M tokens, 10d09h12m elapsed) |
| 2026-09-23 04:24:54 [all] step 15340/18464 loss=7.8307 ppl=2516.57 lr=9.877e-05 | 612 tok/s | 771M done | eta 1d22h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 04:33:44 [all] step 15350/18464 loss=7.8208 ppl=2491.98 lr=9.853e-05 | 619 tok/s | 772M done | eta 1d21h49m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 04:42:33 [all] step 15360/18464 loss=7.8583 ppl=2587.08 lr=9.829e-05 | 619 tok/s | 772M done | eta 1d21h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 04:48:43 [all] checkpoint @ step 15367 (772M tokens, 10d09h42m elapsed) |
| 2026-09-23 04:51:20 [all] step 15370/18464 loss=7.8209 ppl=2492.04 lr=9.805e-05 | 621 tok/s | 772M done | eta 1d21h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 05:00:08 [all] step 15380/18464 loss=7.8208 ppl=2492.02 lr=9.781e-05 | 621 tok/s | 773M done | eta 1d21h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 05:08:50 [all] step 15390/18464 loss=7.8291 ppl=2512.55 lr=9.757e-05 | 628 tok/s | 773M done | eta 1d20h33m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 05:17:28 [all] step 15400/18464 loss=7.8200 ppl=2490.01 lr=9.733e-05 | 632 tok/s | 773M done | eta 1d20h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 05:19:12 [all] checkpoint @ step 15402 (773M tokens, 10d10h12m elapsed) |
| 2026-09-23 05:26:08 [all] step 15410/18464 loss=7.8364 ppl=2531.17 lr=9.710e-05 | 630 tok/s | 774M done | eta 1d20h07m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 05:34:45 [all] step 15420/18464 loss=7.8087 ppl=2461.94 lr=9.686e-05 | 633 tok/s | 774M done | eta 1d19h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 05:43:21 [all] step 15430/18464 loss=7.8366 ppl=2531.60 lr=9.662e-05 | 635 tok/s | 774M done | eta 1d19h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 05:49:21 [all] checkpoint @ step 15437 (774M tokens, 10d10h42m elapsed) |
| 2026-09-23 05:51:56 [all] step 15440/18464 loss=7.8260 ppl=2504.79 lr=9.639e-05 | 636 tok/s | 775M done | eta 1d19h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:00:33 [all] step 15450/18464 loss=7.8264 ppl=2505.85 lr=9.615e-05 | 634 tok/s | 775M done | eta 1d19h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:09:05 [all] step 15460/18464 loss=7.8237 ppl=2499.19 lr=9.592e-05 | 640 tok/s | 775M done | eta 1d18h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:17:34 [all] step 15470/18464 loss=7.8410 ppl=2542.80 lr=9.569e-05 | 644 tok/s | 776M done | eta 1d18h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:20:09 [all] checkpoint @ step 15473 (776M tokens, 10d11h13m elapsed) |
| 2026-09-23 06:26:03 [all] step 15480/18464 loss=7.8256 ppl=2503.98 lr=9.545e-05 | 643 tok/s | 776M done | eta 1d18h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:34:34 [all] step 15490/18464 loss=7.8324 ppl=2520.95 lr=9.522e-05 | 642 tok/s | 776M done | eta 1d18h10m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:42:59 [all] step 15500/18464 loss=7.8270 ppl=2507.47 lr=9.499e-05 | 648 tok/s | 777M done | eta 1d17h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:43:37 [all] eval step 15500/18464 loss=7.8400 ppl=2540.26 (best 7.8400) |
| 2026-09-23 06:50:10 [all] checkpoint @ step 15508 (777M tokens, 10d11h43m elapsed) |
| 2026-09-23 06:51:45 [all] step 15510/18464 loss=7.8242 ppl=2500.28 lr=9.476e-05 | 623 tok/s | 777M done | eta 1d19h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 06:59:35 [all] step 15520/18464 loss=7.8230 ppl=2497.32 lr=9.453e-05 | 697 tok/s | 777M done | eta 1d14h25m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:07:29 [all] step 15530/18464 loss=7.8113 ppl=2468.24 lr=9.430e-05 | 692 tok/s | 778M done | eta 1d14h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:15:07 [all] step 15540/18464 loss=7.8260 ppl=2504.83 lr=9.407e-05 | 715 tok/s | 778M done | eta 1d13h14m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:20:17 [all] checkpoint @ step 15547 (778M tokens, 10d12h13m elapsed) |
| 2026-09-23 07:22:29 [all] step 15550/18464 loss=7.8432 ppl=2548.31 lr=9.385e-05 | 741 tok/s | 778M done | eta 1d11h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:29:44 [all] step 15560/18464 loss=7.8249 ppl=2502.16 lr=9.362e-05 | 754 tok/s | 779M done | eta 1d11h03m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:36:55 [all] step 15570/18464 loss=7.8127 ppl=2471.82 lr=9.339e-05 | 761 tok/s | 779M done | eta 1d10h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:44:02 [all] step 15580/18464 loss=7.8375 ppl=2533.75 lr=9.317e-05 | 767 tok/s | 779M done | eta 1d10h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:50:26 [all] checkpoint @ step 15589 (779M tokens, 10d12h44m elapsed) |
| 2026-09-23 07:51:09 [all] step 15590/18464 loss=7.8030 ppl=2448.05 lr=9.294e-05 | 767 tok/s | 779M done | eta 1d10h05m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 07:58:27 [all] step 15600/18464 loss=7.8376 ppl=2534.02 lr=9.272e-05 | 749 tok/s | 780M done | eta 1d10h49m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:05:45 [all] step 15610/18464 loss=7.8276 ppl=2508.86 lr=9.250e-05 | 748 tok/s | 780M done | eta 1d10h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:13:00 [all] step 15620/18464 loss=7.8441 ppl=2550.72 lr=9.227e-05 | 753 tok/s | 780M done | eta 1d10h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:20:15 [all] step 15630/18464 loss=7.8415 ppl=2544.06 lr=9.205e-05 | 754 tok/s | 781M done | eta 1d10h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:20:58 [all] checkpoint @ step 15631 (781M tokens, 10d13h14m elapsed) |
| 2026-09-23 08:27:27 [all] step 15640/18464 loss=7.8296 ppl=2513.98 lr=9.183e-05 | 759 tok/s | 781M done | eta 1d09h52m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:34:38 [all] step 15650/18464 loss=7.8308 ppl=2516.93 lr=9.161e-05 | 760 tok/s | 781M done | eta 1d09h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:41:47 [all] step 15660/18464 loss=7.8415 ppl=2544.05 lr=9.139e-05 | 764 tok/s | 782M done | eta 1d09h24m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:48:54 [all] step 15670/18464 loss=7.8151 ppl=2477.63 lr=9.117e-05 | 767 tok/s | 782M done | eta 1d09h09m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 08:51:02 [all] checkpoint @ step 15673 (782M tokens, 10d13h44m elapsed) |
| 2026-09-23 08:55:58 [all] step 15680/18464 loss=7.7928 ppl=2423.03 lr=9.095e-05 | 774 tok/s | 782M done | eta 1d08h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:03:03 [all] step 15690/18464 loss=7.8111 ppl=2467.75 lr=9.073e-05 | 771 tok/s | 783M done | eta 1d08h43m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:10:07 [all] step 15700/18464 loss=7.8475 ppl=2559.36 lr=9.052e-05 | 771 tok/s | 783M done | eta 1d08h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:17:08 [all] step 15710/18464 loss=7.8371 ppl=2532.92 lr=9.030e-05 | 778 tok/s | 783M done | eta 1d08h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:21:23 [all] checkpoint @ step 15716 (784M tokens, 10d14h14m elapsed) |
| 2026-09-23 09:24:11 [all] step 15720/18464 loss=7.8263 ppl=2505.54 lr=9.009e-05 | 775 tok/s | 784M done | eta 1d08h12m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:31:12 [all] step 15730/18464 loss=7.8656 ppl=2606.03 lr=8.987e-05 | 778 tok/s | 784M done | eta 1d07h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:38:15 [all] step 15740/18464 loss=7.8229 ppl=2497.16 lr=8.966e-05 | 774 tok/s | 784M done | eta 1d08h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:45:20 [all] step 15750/18464 loss=7.8289 ppl=2512.28 lr=8.944e-05 | 772 tok/s | 785M done | eta 1d08h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:51:46 [all] checkpoint @ step 15759 (785M tokens, 10d14h45m elapsed) |
| 2026-09-23 09:52:29 [all] step 15760/18464 loss=7.8422 ppl=2545.69 lr=8.923e-05 | 763 tok/s | 785M done | eta 1d08h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 09:59:44 [all] step 15770/18464 loss=7.8438 ppl=2549.98 lr=8.902e-05 | 754 tok/s | 785M done | eta 1d08h31m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 10:07:07 [all] step 15780/18464 loss=7.8356 ppl=2529.03 lr=8.881e-05 | 740 tok/s | 786M done | eta 1d09h01m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 10:14:25 [all] step 15790/18464 loss=7.8293 ppl=2513.28 lr=8.860e-05 | 748 tok/s | 786M done | eta 1d08h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 10:21:52 [all] step 15800/18464 loss=7.8269 ppl=2507.13 lr=8.839e-05 | 733 tok/s | 786M done | eta 1d09h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 10:21:53 [all] checkpoint @ step 15800 (786M tokens, 10d15h15m elapsed) |
| 2026-09-23 10:29:39 [all] step 15810/18464 loss=7.8383 ppl=2535.96 lr=8.818e-05 | 702 tok/s | 787M done | eta 1d10h24m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 10:37:29 [all] step 15820/18464 loss=7.8238 ppl=2499.42 lr=8.797e-05 | 697 tok/s | 787M done | eta 1d10h31m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 10:45:11 [all] step 15830/18464 loss=7.8311 ppl=2517.66 lr=8.776e-05 | 709 tok/s | 787M done | eta 1d09h49m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 10:52:13 [all] checkpoint @ step 15839 (788M tokens, 10d15h45m elapsed) |
| 2026-09-23 10:52:59 [all] step 15840/18464 loss=7.8187 ppl=2486.77 lr=8.756e-05 | 700 tok/s | 788M done | eta 1d10h07m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:00:51 [all] step 15850/18464 loss=7.8122 ppl=2470.67 lr=8.735e-05 | 695 tok/s | 788M done | eta 1d10h15m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:08:50 [all] step 15860/18464 loss=7.8227 ppl=2496.68 lr=8.715e-05 | 685 tok/s | 788M done | eta 1d10h36m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:17:04 [all] step 15870/18464 loss=7.8353 ppl=2528.21 lr=8.694e-05 | 663 tok/s | 789M done | eta 1d11h36m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:22:51 [all] checkpoint @ step 15877 (789M tokens, 10d16h16m elapsed) |
| 2026-09-23 11:25:19 [all] step 15880/18464 loss=7.8375 ppl=2533.95 lr=8.674e-05 | 662 tok/s | 789M done | eta 1d11h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:33:35 [all] step 15890/18464 loss=7.8114 ppl=2468.71 lr=8.654e-05 | 660 tok/s | 789M done | eta 1d11h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:41:59 [all] step 15900/18464 loss=7.8259 ppl=2504.73 lr=8.633e-05 | 651 tok/s | 790M done | eta 1d11h52m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:50:13 [all] step 15910/18464 loss=7.8113 ppl=2468.33 lr=8.613e-05 | 664 tok/s | 790M done | eta 1d11h00m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 11:53:33 [all] checkpoint @ step 15914 (790M tokens, 10d16h47m elapsed) |
| 2026-09-23 11:58:33 [all] step 15920/18464 loss=7.8269 ppl=2507.09 lr=8.593e-05 | 654 tok/s | 790M done | eta 1d11h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 12:06:55 [all] step 15930/18464 loss=7.8065 ppl=2456.42 lr=8.573e-05 | 653 tok/s | 791M done | eta 1d11h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 12:15:14 [all] step 15940/18464 loss=7.8333 ppl=2523.33 lr=8.553e-05 | 657 tok/s | 791M done | eta 1d10h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 12:23:33 [all] step 15950/18464 loss=7.8318 ppl=2519.34 lr=8.533e-05 | 657 tok/s | 791M done | eta 1d10h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 12:23:33 [all] checkpoint @ step 15950 (791M tokens, 10d17h17m elapsed) |
| 2026-09-23 12:31:53 [all] step 15960/18464 loss=7.7929 ppl=2423.35 lr=8.513e-05 | 656 tok/s | 792M done | eta 1d10h44m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 12:40:12 [all] step 15970/18464 loss=7.8057 ppl=2454.64 lr=8.494e-05 | 656 tok/s | 792M done | eta 1d10h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 12:48:35 [all] step 15980/18464 loss=7.8350 ppl=2527.46 lr=8.474e-05 | 652 tok/s | 792M done | eta 1d10h40m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 12:53:39 [all] checkpoint @ step 15986 (792M tokens, 10d17h47m elapsed) |
| 2026-09-23 12:57:02 [all] step 15990/18464 loss=7.8199 ppl=2489.67 lr=8.455e-05 | 646 tok/s | 793M done | eta 1d10h50m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 13:05:33 [all] step 16000/18464 loss=7.8178 ppl=2484.35 lr=8.435e-05 | 641 tok/s | 793M done | eta 1d10h58m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 13:06:11 [all] eval step 16000/18464 loss=7.8400 ppl=2540.16 (best 7.8400) |
| 2026-09-23 13:06:14 [all] sample @ 16000: ' the your this for the the for the the the was to the the the thes, to the the be the the the the the the the the the it the the the to thes the of the the the the have the of the' |
| 2026-09-23 13:06:14 [all] checkpoint @ step 16000 (793M tokens, 10d17h59m elapsed) |
| 2026-09-23 13:14:47 [all] step 16010/18464 loss=7.8247 ppl=2501.74 lr=8.416e-05 | 591 tok/s | 793M done | eta 1d13h46m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 13:23:20 [all] step 16020/18464 loss=7.8166 ppl=2481.57 lr=8.396e-05 | 639 tok/s | 794M done | eta 1d10h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 13:31:56 [all] step 16030/18464 loss=7.8271 ppl=2507.68 lr=8.377e-05 | 634 tok/s | 794M done | eta 1d10h55m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 13:36:17 [all] checkpoint @ step 16035 (794M tokens, 10d18h29m elapsed) |
| 2026-09-23 13:40:40 [all] step 16040/18464 loss=7.8346 ppl=2526.57 lr=8.358e-05 | 626 tok/s | 794M done | eta 1d11h16m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 13:49:23 [all] step 16050/18464 loss=7.8146 ppl=2476.51 lr=8.339e-05 | 627 tok/s | 795M done | eta 1d11h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 13:58:08 [all] step 16060/18464 loss=7.8349 ppl=2527.31 lr=8.320e-05 | 624 tok/s | 795M done | eta 1d11h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 14:06:59 [all] step 16070/18464 loss=7.8202 ppl=2490.31 lr=8.301e-05 | 617 tok/s | 795M done | eta 1d11h19m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 14:07:00 [all] checkpoint @ step 16070 (795M tokens, 10d19h00m elapsed) |
| 2026-09-23 14:15:56 [all] step 16080/18464 loss=7.8242 ppl=2500.39 lr=8.282e-05 | 610 tok/s | 796M done | eta 1d11h34m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 14:24:59 [all] step 16090/18464 loss=7.8174 ppl=2483.40 lr=8.263e-05 | 604 tok/s | 796M done | eta 1d11h48m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 14:34:05 [all] step 16100/18464 loss=7.8247 ppl=2501.71 lr=8.244e-05 | 600 tok/s | 796M done | eta 1d11h51m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 14:37:46 [all] checkpoint @ step 16104 (796M tokens, 10d19h31m elapsed) |
| 2026-09-23 14:43:20 [all] step 16110/18464 loss=7.8192 ppl=2487.98 lr=8.226e-05 | 591 tok/s | 797M done | eta 1d12h14m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 14:52:46 [all] step 16120/18464 loss=7.8286 ppl=2511.40 lr=8.207e-05 | 578 tok/s | 797M done | eta 1d12h53m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 15:02:20 [all] step 16130/18464 loss=7.8441 ppl=2550.64 lr=8.188e-05 | 571 tok/s | 797M done | eta 1d13h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 15:08:13 [all] checkpoint @ step 16136 (797M tokens, 10d20h01m elapsed) |
| 2026-09-23 15:12:07 [all] step 16140/18464 loss=7.8098 ppl=2464.75 lr=8.170e-05 | 559 tok/s | 798M done | eta 1d13h52m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 15:21:53 [all] step 16150/18464 loss=7.8233 ppl=2498.02 lr=8.152e-05 | 559 tok/s | 798M done | eta 1d13h39m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 15:31:46 [all] step 16160/18464 loss=7.8207 ppl=2491.62 lr=8.133e-05 | 552 tok/s | 798M done | eta 1d13h59m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 15:38:46 [all] checkpoint @ step 16167 (798M tokens, 10d20h32m elapsed) |
| 2026-09-23 15:41:38 [all] step 16170/18464 loss=7.8216 ppl=2493.79 lr=8.115e-05 | 554 tok/s | 798M done | eta 1d13h42m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 15:50:42 [all] step 16180/18464 loss=7.8232 ppl=2497.99 lr=8.097e-05 | 603 tok/s | 799M done | eta 1d10h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 15:59:14 [all] step 16190/18464 loss=7.8158 ppl=2479.39 lr=8.079e-05 | 640 tok/s | 799M done | eta 1d08h21m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 16:07:35 [all] step 16200/18464 loss=7.8179 ppl=2484.57 lr=8.061e-05 | 655 tok/s | 799M done | eta 1d07h27m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 16:09:13 [all] checkpoint @ step 16202 (800M tokens, 10d21h02m elapsed) |
| 2026-09-23 16:15:42 [all] step 16210/18464 loss=7.8413 ppl=2543.43 lr=8.043e-05 | 673 tok/s | 800M done | eta 1d06h29m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 16:23:43 [all] step 16220/18464 loss=7.8237 ppl=2499.05 lr=8.025e-05 | 681 tok/s | 800M done | eta 1d05h59m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 16:31:35 [all] step 16230/18464 loss=7.8129 ppl=2472.30 lr=8.007e-05 | 694 tok/s | 800M done | eta 1d05h17m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 16:39:28 [all] step 16240/18464 loss=7.8241 ppl=2500.20 lr=7.990e-05 | 693 tok/s | 801M done | eta 1d05h13m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 16:39:28 [all] checkpoint @ step 16240 (801M tokens, 10d21h33m elapsed) |
| 2026-09-23 16:47:19 [all] step 16250/18464 loss=7.8259 ppl=2504.72 lr=7.972e-05 | 695 tok/s | 801M done | eta 1d04h59m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 16:54:59 [all] step 16260/18464 loss=7.8431 ppl=2548.01 lr=7.954e-05 | 713 tok/s | 801M done | eta 1d04h08m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:02:33 [all] step 16270/18464 loss=7.8126 ppl=2471.61 lr=7.937e-05 | 723 tok/s | 802M done | eta 1d03h38m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:10:04 [all] step 16280/18464 loss=7.8154 ppl=2478.59 lr=7.920e-05 | 726 tok/s | 802M done | eta 1d03h23m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:10:04 [all] checkpoint @ step 16280 (802M tokens, 10d22h03m elapsed) |
| 2026-09-23 17:17:30 [all] step 16290/18464 loss=7.8217 ppl=2494.04 lr=7.902e-05 | 734 tok/s | 802M done | eta 1d02h57m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:24:42 [all] step 16300/18464 loss=7.8086 ppl=2461.72 lr=7.885e-05 | 759 tok/s | 803M done | eta 1d01h56m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:31:45 [all] step 16310/18464 loss=7.8312 ppl=2517.89 lr=7.868e-05 | 775 tok/s | 803M done | eta 1d01h18m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:38:46 [all] step 16320/18464 loss=7.8403 ppl=2540.95 lr=7.851e-05 | 777 tok/s | 803M done | eta 1d01h06m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:40:11 [all] checkpoint @ step 16322 (803M tokens, 10d22h33m elapsed) |
| 2026-09-23 17:45:55 [all] step 16330/18464 loss=7.8302 ppl=2515.44 lr=7.834e-05 | 764 tok/s | 804M done | eta 1d01h26m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 17:53:19 [all] step 16340/18464 loss=7.8299 ppl=2514.68 lr=7.817e-05 | 738 tok/s | 804M done | eta 1d02h11m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:00:19 [all] step 16350/18464 loss=7.8027 ppl=2447.27 lr=7.800e-05 | 782 tok/s | 804M done | eta 1d00h37m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:07:18 [all] step 16360/18464 loss=7.8283 ppl=2510.63 lr=7.783e-05 | 780 tok/s | 805M done | eta 1d00h32m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:10:44 [all] checkpoint @ step 16365 (805M tokens, 10d23h04m elapsed) |
| 2026-09-23 18:14:05 [all] step 16370/18464 loss=7.8377 ppl=2534.35 lr=7.766e-05 | 805 tok/s | 805M done | eta 23h40m27s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:20:49 [all] step 16380/18464 loss=7.8066 ppl=2456.71 lr=7.750e-05 | 811 tok/s | 805M done | eta 23h22m35s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:27:30 [all] step 16390/18464 loss=7.8157 ppl=2479.26 lr=7.733e-05 | 818 tok/s | 806M done | eta 23h04m58s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:34:08 [all] step 16400/18464 loss=7.8338 ppl=2524.47 lr=7.717e-05 | 823 tok/s | 806M done | eta 22h48m52s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:40:50 [all] step 16410/18464 loss=7.8336 ppl=2524.05 lr=7.700e-05 | 814 tok/s | 806M done | eta 22h58m16s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:40:51 [all] checkpoint @ step 16410 (806M tokens, 10d23h34m elapsed) |
| 2026-09-23 18:47:24 [all] step 16420/18464 loss=7.8250 ppl=2502.38 lr=7.684e-05 | 833 tok/s | 807M done | eta 22h20m47s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 18:53:52 [all] step 16430/18464 loss=7.8277 ppl=2509.07 lr=7.668e-05 | 844 tok/s | 807M done | eta 21h56m36s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:00:29 [all] step 16440/18464 loss=7.7826 ppl=2398.60 lr=7.652e-05 | 826 tok/s | 807M done | eta 22h18m27s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:07:09 [all] step 16450/18464 loss=7.8126 ppl=2471.64 lr=7.635e-05 | 820 tok/s | 808M done | eta 22h22m01s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:11:08 [all] checkpoint @ step 16456 (808M tokens, 11d00h04m elapsed) |
| 2026-09-23 19:13:46 [all] step 16460/18464 loss=7.8272 ppl=2507.91 lr=7.619e-05 | 825 tok/s | 808M done | eta 22h06m52s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:20:22 [all] step 16470/18464 loss=7.8208 ppl=2491.94 lr=7.603e-05 | 829 tok/s | 808M done | eta 21h53m32s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:26:52 [all] step 16480/18464 loss=7.8519 ppl=2570.63 lr=7.588e-05 | 840 tok/s | 809M done | eta 21h30m06s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:33:19 [all] step 16490/18464 loss=7.8516 ppl=2569.75 lr=7.572e-05 | 847 tok/s | 809M done | eta 21h12m48s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:39:46 [all] step 16500/18464 loss=7.8089 ppl=2462.47 lr=7.556e-05 | 846 tok/s | 809M done | eta 21h07m53s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:40:15 [all] eval step 16500/18464 loss=7.8400 ppl=2540.09 (best 7.8400) |
| 2026-09-23 19:41:33 [all] checkpoint @ step 16502 (809M tokens, 11d00h35m elapsed) |
| 2026-09-23 19:46:47 [all] step 16510/18464 loss=7.8328 ppl=2522.02 lr=7.540e-05 | 777 tok/s | 810M done | eta 22h52m37s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:53:04 [all] step 16520/18464 loss=7.8193 ppl=2488.11 lr=7.525e-05 | 869 tok/s | 810M done | eta 20h21m53s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 19:59:17 [all] step 16530/18464 loss=7.8336 ppl=2523.95 lr=7.509e-05 | 879 tok/s | 810M done | eta 20h02m08s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:05:27 [all] step 16540/18464 loss=7.8283 ppl=2510.60 lr=7.494e-05 | 888 tok/s | 811M done | eta 19h43m47s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:11:39 [all] step 16550/18464 loss=7.8236 ppl=2498.84 lr=7.479e-05 | 879 tok/s | 811M done | eta 19h49m07s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:11:40 [all] checkpoint @ step 16550 (811M tokens, 11d01h05m elapsed) |
| 2026-09-23 20:17:56 [all] step 16560/18464 loss=7.8469 ppl=2557.83 lr=7.463e-05 | 870 tok/s | 811M done | eta 19h55m46s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:24:13 [all] step 16570/18464 loss=7.8195 ppl=2488.73 lr=7.448e-05 | 870 tok/s | 812M done | eta 19h49m17s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:30:29 [all] step 16580/18464 loss=7.8323 ppl=2520.65 lr=7.433e-05 | 871 tok/s | 812M done | eta 19h41m55s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:37:06 [all] step 16590/18464 loss=7.8165 ppl=2481.22 lr=7.418e-05 | 826 tok/s | 812M done | eta 20h38m57s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:42:17 [all] checkpoint @ step 16597 (812M tokens, 11d01h35m elapsed) |
| 2026-09-23 20:44:32 [all] step 16600/18464 loss=7.8401 ppl=2540.34 lr=7.403e-05 | 734 tok/s | 813M done | eta 23h06m46s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:52:00 [all] step 16610/18464 loss=7.8246 ppl=2501.27 lr=7.388e-05 | 733 tok/s | 813M done | eta 23h02m11s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 20:59:26 [all] step 16620/18464 loss=7.8040 ppl=2450.44 lr=7.373e-05 | 735 tok/s | 813M done | eta 22h51m02s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 21:06:46 [all] step 16630/18464 loss=7.8165 ppl=2481.27 lr=7.359e-05 | 744 tok/s | 814M done | eta 22h25m23s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 21:12:25 [all] checkpoint @ step 16637 (814M tokens, 11d02h06m elapsed) |
| 2026-09-23 21:14:55 [all] step 16640/18464 loss=7.8270 ppl=2507.39 lr=7.344e-05 | 671 tok/s | 814M done | eta 1d00h45m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 21:22:47 [all] step 16650/18464 loss=7.8462 ppl=2556.02 lr=7.329e-05 | 693 tok/s | 814M done | eta 23h49m00s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 21:30:33 [all] step 16660/18464 loss=7.8176 ppl=2483.96 lr=7.315e-05 | 704 tok/s | 815M done | eta 23h19m48s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 21:38:24 [all] step 16670/18464 loss=7.8097 ppl=2464.29 lr=7.301e-05 | 695 tok/s | 815M done | eta 23h30m07s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 21:43:07 [all] checkpoint @ step 16676 (815M tokens, 11d02h36m elapsed) |
| 2026-09-23 21:46:16 [all] step 16680/18464 loss=7.8412 ppl=2543.18 lr=7.286e-05 | 694 tok/s | 815M done | eta 23h23m35s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 21:54:08 [all] step 16690/18464 loss=7.8371 ppl=2532.89 lr=7.272e-05 | 695 tok/s | 816M done | eta 23h14m10s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:02:06 [all] step 16700/18464 loss=7.8356 ppl=2529.00 lr=7.258e-05 | 686 tok/s | 816M done | eta 23h24m37s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:10:03 [all] step 16710/18464 loss=7.8235 ppl=2498.73 lr=7.244e-05 | 686 tok/s | 816M done | eta 23h15m58s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:13:20 [all] checkpoint @ step 16714 (816M tokens, 11d03h06m elapsed) |
| 2026-09-23 22:18:14 [all] step 16720/18464 loss=7.8214 ppl=2493.51 lr=7.230e-05 | 668 tok/s | 817M done | eta 23h46m39s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:26:22 [all] step 16730/18464 loss=7.8159 ppl=2479.82 lr=7.216e-05 | 672 tok/s | 817M done | eta 23h29m07s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:34:05 [all] step 16740/18464 loss=7.8352 ppl=2528.10 lr=7.202e-05 | 707 tok/s | 817M done | eta 22h11m37s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:41:56 [all] step 16750/18464 loss=7.8076 ppl=2459.31 lr=7.188e-05 | 696 tok/s | 817M done | eta 22h25m51s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:43:30 [all] checkpoint @ step 16752 (818M tokens, 11d03h37m elapsed) |
| 2026-09-23 22:49:52 [all] step 16760/18464 loss=7.8266 ppl=2506.47 lr=7.174e-05 | 688 tok/s | 818M done | eta 22h31m53s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 22:57:56 [all] step 16770/18464 loss=7.8347 ppl=2526.67 lr=7.161e-05 | 677 tok/s | 818M done | eta 22h45m38s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 23:05:26 [all] step 16780/18464 loss=7.8337 ppl=2524.25 lr=7.147e-05 | 728 tok/s | 818M done | eta 21h02m28s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 23:12:19 [all] step 16790/18464 loss=7.8254 ppl=2503.32 lr=7.134e-05 | 793 tok/s | 819M done | eta 19h12m22s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 23:13:37 [all] checkpoint @ step 16792 (819M tokens, 11d04h07m elapsed) |
| 2026-09-23 23:19:03 [all] step 16800/18464 loss=7.8127 ppl=2471.82 lr=7.120e-05 | 810 tok/s | 819M done | eta 18h41m14s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 23:27:55 [all] step 16810/18464 loss=7.8201 ppl=2490.22 lr=7.107e-05 | 616 tok/s | 819M done | eta 1d00h26m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 23:37:25 [all] step 16820/18464 loss=7.8237 ppl=2499.23 lr=7.094e-05 | 574 tok/s | 820M done | eta 1d02h02m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 23:44:21 [all] checkpoint @ step 16827 (820M tokens, 11d04h37m elapsed) |
| 2026-09-23 23:47:22 [all] step 16830/18464 loss=7.8265 ppl=2506.21 lr=7.081e-05 | 549 tok/s | 820M done | eta 1d03h04m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-23 23:56:54 [all] step 16840/18464 loss=7.8649 ppl=2604.24 lr=7.067e-05 | 573 tok/s | 820M done | eta 1d01h49m | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 00:04:55 [all] step 16850/18464 loss=7.8140 ppl=2474.89 lr=7.054e-05 | 681 tok/s | 821M done | eta 21h34m42s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 00:13:03 [all] step 16860/18464 loss=7.8368 ppl=2532.10 lr=7.041e-05 | 672 tok/s | 821M done | eta 21h43m08s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 00:14:42 [all] checkpoint @ step 16862 (821M tokens, 11d05h08m elapsed) |
| 2026-09-24 00:21:10 [all] step 16870/18464 loss=7.8297 ppl=2514.24 lr=7.029e-05 | 673 tok/s | 821M done | eta 21h34m15s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 00:29:17 [all] step 16880/18464 loss=7.8201 ppl=2490.12 lr=7.016e-05 | 673 tok/s | 822M done | eta 21h25m41s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 00:37:23 [all] step 16890/18464 loss=7.8184 ppl=2486.04 lr=7.003e-05 | 675 tok/s | 822M done | eta 21h14m15s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 00:45:31 [all] step 16900/18464 loss=7.8342 ppl=2525.48 lr=6.991e-05 | 671 tok/s | 822M done | eta 21h12m25s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 00:45:31 [all] checkpoint @ step 16900 (822M tokens, 11d05h39m elapsed) |
| 2026-09-24 00:53:43 [all] step 16910/18464 loss=7.8380 ppl=2535.12 lr=6.978e-05 | 666 tok/s | 823M done | eta 21h14m19s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 01:01:54 [all] step 16920/18464 loss=7.8306 ppl=2516.48 lr=6.966e-05 | 668 tok/s | 823M done | eta 21h02m32s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 01:10:10 [all] step 16930/18464 loss=7.8053 ppl=2453.68 lr=6.953e-05 | 660 tok/s | 823M done | eta 21h09m54s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 01:16:02 [all] checkpoint @ step 16937 (824M tokens, 11d06h09m elapsed) |
| 2026-09-24 01:18:33 [all] step 16940/18464 loss=7.8559 ppl=2580.79 lr=6.941e-05 | 652 tok/s | 824M done | eta 21h16m24s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 01:26:50 [all] step 16950/18464 loss=7.8427 ppl=2546.99 lr=6.929e-05 | 659 tok/s | 824M done | eta 20h55m12s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 01:35:07 [all] step 16960/18464 loss=7.8164 ppl=2480.97 lr=6.916e-05 | 660 tok/s | 824M done | eta 20h45m23s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 01:43:27 [all] step 16970/18464 loss=7.8519 ppl=2570.69 lr=6.904e-05 | 655 tok/s | 825M done | eta 20h45m40s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 01:46:49 [all] checkpoint @ step 16974 (825M tokens, 11d06h40m elapsed) |
| 2026-09-24 01:51:53 [all] step 16980/18464 loss=7.8318 ppl=2519.38 lr=6.892e-05 | 649 tok/s | 825M done | eta 20h49m21s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:00:14 [all] step 16990/18464 loss=7.8341 ppl=2525.24 lr=6.880e-05 | 654 tok/s | 825M done | eta 20h31m35s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:08:35 [all] step 17000/18464 loss=7.8209 ppl=2492.11 lr=6.869e-05 | 654 tok/s | 826M done | eta 20h21m49s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:09:12 [all] eval step 17000/18464 loss=7.8399 ppl=2540.04 (best 7.8399) |
| 2026-09-24 02:09:13 [all] checkpoint @ step 17000 (826M tokens, 11d07h02m elapsed) |
| 2026-09-24 02:17:31 [all] step 17010/18464 loss=7.8336 ppl=2524.11 lr=6.857e-05 | 611 tok/s | 826M done | eta 21h40m25s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:25:55 [all] step 17020/18464 loss=7.8242 ppl=2500.48 lr=6.845e-05 | 650 tok/s | 826M done | eta 20h12m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:34:15 [all] step 17030/18464 loss=7.8249 ppl=2502.11 lr=6.834e-05 | 655 tok/s | 827M done | eta 19h55m02s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:39:19 [all] checkpoint @ step 17036 (827M tokens, 11d07h32m elapsed) |
| 2026-09-24 02:42:39 [all] step 17040/18464 loss=7.8165 ppl=2481.32 lr=6.822e-05 | 650 tok/s | 827M done | eta 19h55m51s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:50:57 [all] step 17050/18464 loss=7.8336 ppl=2523.97 lr=6.811e-05 | 658 tok/s | 827M done | eta 19h32m46s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 02:59:22 [all] step 17060/18464 loss=7.8234 ppl=2498.40 lr=6.799e-05 | 648 tok/s | 828M done | eta 19h43m22s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 03:07:43 [all] step 17070/18464 loss=7.8471 ppl=2558.34 lr=6.788e-05 | 654 tok/s | 828M done | eta 19h23m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 03:09:25 [all] checkpoint @ step 17072 (828M tokens, 11d08h03m elapsed) |
| 2026-09-24 03:16:11 [all] step 17080/18464 loss=7.8245 ppl=2501.20 lr=6.777e-05 | 646 tok/s | 828M done | eta 19h29m57s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 03:24:39 [all] step 17090/18464 loss=7.8465 ppl=2556.82 lr=6.766e-05 | 645 tok/s | 829M done | eta 19h23m37s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 03:33:04 [all] step 17100/18464 loss=7.8134 ppl=2473.56 lr=6.755e-05 | 649 tok/s | 829M done | eta 19h08m27s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 03:39:47 [all] checkpoint @ step 17108 (829M tokens, 11d08h33m elapsed) |
| 2026-09-24 03:41:28 [all] step 17110/18464 loss=7.8118 ppl=2469.54 lr=6.744e-05 | 650 tok/s | 829M done | eta 18h57m05s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 03:49:51 [all] step 17120/18464 loss=7.8411 ppl=2542.90 lr=6.733e-05 | 650 tok/s | 830M done | eta 18h48m27s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 03:58:18 [all] step 17130/18464 loss=7.8114 ppl=2468.64 lr=6.722e-05 | 647 tok/s | 830M done | eta 18h46m32s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 04:06:49 [all] step 17140/18464 loss=7.8192 ppl=2488.04 lr=6.711e-05 | 641 tok/s | 830M done | eta 18h47m17s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 04:10:08 [all] checkpoint @ step 17144 (830M tokens, 11d09h03m elapsed) |
| 2026-09-24 04:15:12 [all] step 17150/18464 loss=7.8081 ppl=2460.56 lr=6.701e-05 | 651 tok/s | 831M done | eta 18h21m41s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 04:23:32 [all] step 17160/18464 loss=7.8046 ppl=2451.78 lr=6.690e-05 | 656 tok/s | 831M done | eta 18h05m26s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 04:31:59 [all] step 17170/18464 loss=7.8128 ppl=2472.05 lr=6.680e-05 | 646 tok/s | 831M done | eta 18h13m34s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 04:40:24 [all] step 17180/18464 loss=7.8226 ppl=2496.35 lr=6.669e-05 | 648 tok/s | 832M done | eta 18h01m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 04:40:24 [all] checkpoint @ step 17180 (832M tokens, 11d09h34m elapsed) |
| 2026-09-24 04:48:49 [all] step 17190/18464 loss=7.8354 ppl=2528.52 lr=6.659e-05 | 649 tok/s | 832M done | eta 17h51m41s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 04:57:11 [all] step 17200/18464 loss=7.8315 ppl=2518.63 lr=6.649e-05 | 653 tok/s | 832M done | eta 17h37m14s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 05:05:36 [all] step 17210/18464 loss=7.8224 ppl=2495.81 lr=6.638e-05 | 649 tok/s | 833M done | eta 17h35m26s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 05:10:39 [all] checkpoint @ step 17216 (833M tokens, 11d10h04m elapsed) |
| 2026-09-24 05:14:00 [all] step 17220/18464 loss=7.8317 ppl=2519.16 lr=6.628e-05 | 650 tok/s | 833M done | eta 17h25m53s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 05:22:19 [all] step 17230/18464 loss=7.8154 ppl=2478.51 lr=6.618e-05 | 657 tok/s | 833M done | eta 17h05m57s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 05:30:38 [all] step 17240/18464 loss=7.8379 ppl=2534.87 lr=6.608e-05 | 656 tok/s | 834M done | eta 16h58m59s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 05:39:02 [all] step 17250/18464 loss=7.8125 ppl=2471.26 lr=6.598e-05 | 651 tok/s | 834M done | eta 16h58m05s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 05:40:43 [all] checkpoint @ step 17252 (834M tokens, 11d10h34m elapsed) |
| 2026-09-24 05:47:22 [all] step 17260/18464 loss=7.8269 ppl=2507.12 lr=6.589e-05 | 655 tok/s | 834M done | eta 16h43m58s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 05:55:44 [all] step 17270/18464 loss=7.8057 ppl=2454.53 lr=6.579e-05 | 652 tok/s | 835M done | eta 16h40m03s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:04:11 [all] step 17280/18464 loss=7.8316 ppl=2519.02 lr=6.569e-05 | 647 tok/s | 835M done | eta 16h40m02s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:10:55 [all] checkpoint @ step 17288 (835M tokens, 11d11h04m elapsed) |
| 2026-09-24 06:12:37 [all] step 17290/18464 loss=7.8346 ppl=2526.41 lr=6.560e-05 | 648 tok/s | 835M done | eta 16h29m28s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:20:58 [all] step 17300/18464 loss=7.8193 ppl=2488.12 lr=6.550e-05 | 654 tok/s | 836M done | eta 16h11m42s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:29:04 [all] step 17310/18464 loss=7.8205 ppl=2491.07 lr=6.541e-05 | 674 tok/s | 836M done | eta 15h35m33s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:36:43 [all] step 17320/18464 loss=7.8316 ppl=2518.97 lr=6.532e-05 | 715 tok/s | 836M done | eta 14h34m21s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:41:19 [all] checkpoint @ step 17326 (836M tokens, 11d11h34m elapsed) |
| 2026-09-24 06:44:18 [all] step 17330/18464 loss=7.8173 ppl=2483.23 lr=6.523e-05 | 720 tok/s | 837M done | eta 14h20m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:51:30 [all] step 17340/18464 loss=7.8154 ppl=2478.41 lr=6.513e-05 | 758 tok/s | 837M done | eta 13h29m33s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 06:58:26 [all] step 17350/18464 loss=7.8129 ppl=2472.33 lr=6.504e-05 | 788 tok/s | 837M done | eta 12h52m20s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:05:18 [all] step 17360/18464 loss=7.8211 ppl=2492.54 lr=6.495e-05 | 795 tok/s | 837M done | eta 12h37m57s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:11:24 [all] checkpoint @ step 17369 (838M tokens, 11d12h05m elapsed) |
| 2026-09-24 07:12:05 [all] step 17370/18464 loss=7.8384 ppl=2536.11 lr=6.486e-05 | 806 tok/s | 838M done | eta 12h21m23s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:18:50 [all] step 17380/18464 loss=7.8318 ppl=2519.54 lr=6.478e-05 | 810 tok/s | 838M done | eta 12h11m17s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:25:30 [all] step 17390/18464 loss=7.7907 ppl=2417.94 lr=6.469e-05 | 819 tok/s | 838M done | eta 11h56m08s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:32:17 [all] step 17400/18464 loss=7.8138 ppl=2474.56 lr=6.460e-05 | 804 tok/s | 839M done | eta 12h02m58s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:39:20 [all] step 17410/18464 loss=7.8208 ppl=2491.78 lr=6.452e-05 | 776 tok/s | 839M done | eta 12h22m05s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:42:00 [all] checkpoint @ step 17414 (839M tokens, 11d12h35m elapsed) |
| 2026-09-24 07:46:01 [all] step 17420/18464 loss=7.8356 ppl=2529.09 lr=6.443e-05 | 817 tok/s | 839M done | eta 11h38m06s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:52:34 [all] step 17430/18464 loss=7.8304 ppl=2516.05 lr=6.435e-05 | 835 tok/s | 840M done | eta 11h16m15s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 07:59:02 [all] step 17440/18464 loss=7.8128 ppl=2472.00 lr=6.426e-05 | 843 tok/s | 840M done | eta 11h03m41s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:05:28 [all] step 17450/18464 loss=7.8354 ppl=2528.56 lr=6.418e-05 | 850 tok/s | 840M done | eta 10h51m47s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:11:52 [all] step 17460/18464 loss=7.8281 ppl=2510.19 lr=6.410e-05 | 854 tok/s | 841M done | eta 10h41m47s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:12:30 [all] checkpoint @ step 17461 (841M tokens, 11d13h06m elapsed) |
| 2026-09-24 08:18:16 [all] step 17470/18464 loss=7.8068 ppl=2457.30 lr=6.402e-05 | 853 tok/s | 841M done | eta 10h36m45s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:24:42 [all] step 17480/18464 loss=7.8084 ppl=2461.17 lr=6.394e-05 | 850 tok/s | 841M done | eta 10h32m33s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:31:10 [all] step 17490/18464 loss=7.8270 ppl=2507.51 lr=6.386e-05 | 844 tok/s | 842M done | eta 10h30m11s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:37:43 [all] step 17500/18464 loss=7.8238 ppl=2499.35 lr=6.378e-05 | 833 tok/s | 842M done | eta 10h32m12s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:38:14 [all] eval step 17500/18464 loss=7.8399 ppl=2539.93 (best 7.8399) |
| 2026-09-24 08:42:56 [all] checkpoint @ step 17507 (842M tokens, 11d13h36m elapsed) |
| 2026-09-24 08:44:55 [all] step 17510/18464 loss=7.8272 ppl=2508.02 lr=6.370e-05 | 759 tok/s | 842M done | eta 11h26m11s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:51:37 [all] step 17520/18464 loss=7.8194 ppl=2488.44 lr=6.363e-05 | 815 tok/s | 843M done | eta 10h32m35s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 08:58:25 [all] step 17530/18464 loss=7.8418 ppl=2544.78 lr=6.355e-05 | 804 tok/s | 843M done | eta 10h34m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 09:05:17 [all] step 17540/18464 loss=7.8273 ppl=2508.13 lr=6.347e-05 | 796 tok/s | 843M done | eta 10h34m04s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 09:12:17 [all] step 17550/18464 loss=7.8177 ppl=2484.11 lr=6.340e-05 | 779 tok/s | 844M done | eta 10h40m36s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 09:13:01 [all] checkpoint @ step 17551 (844M tokens, 11d14h06m elapsed) |
| 2026-09-24 09:19:41 [all] step 17560/18464 loss=7.8454 ppl=2554.02 lr=6.333e-05 | 739 tok/s | 844M done | eta 11h08m24s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 09:27:26 [all] step 17570/18464 loss=7.8163 ppl=2480.75 lr=6.325e-05 | 704 tok/s | 844M done | eta 11h33m39s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 09:35:24 [all] step 17580/18464 loss=7.8311 ppl=2517.78 lr=6.318e-05 | 685 tok/s | 845M done | eta 11h44m37s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 09:43:34 [all] step 17590/18464 loss=7.8305 ppl=2516.23 lr=6.311e-05 | 670 tok/s | 845M done | eta 11h52m50s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 09:43:34 [all] checkpoint @ step 17590 (845M tokens, 11d14h37m elapsed) |
| 2026-09-24 09:51:43 [all] step 17600/18464 loss=7.8240 ppl=2499.99 lr=6.304e-05 | 670 tok/s | 845M done | eta 11h44m03s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 10:00:06 [all] step 17610/18464 loss=7.8103 ppl=2465.77 lr=6.297e-05 | 652 tok/s | 846M done | eta 11h55m52s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 10:08:59 [all] step 17620/18464 loss=7.8331 ppl=2522.79 lr=6.290e-05 | 615 tok/s | 846M done | eta 12h29m28s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 10:13:43 [all] checkpoint @ step 17625 (846M tokens, 11d15h07m elapsed) |
| 2026-09-24 10:19:14 [all] step 17630/18464 loss=7.8396 ppl=2539.27 lr=6.283e-05 | 532 tok/s | 846M done | eta 14h15m34s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 10:28:53 [all] step 17640/18464 loss=7.8141 ppl=2475.33 lr=6.276e-05 | 566 tok/s | 847M done | eta 13h14m29s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 10:37:43 [all] step 17650/18464 loss=7.8457 ppl=2554.73 lr=6.270e-05 | 618 tok/s | 847M done | eta 11h59m53s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 10:43:48 [all] checkpoint @ step 17657 (847M tokens, 11d15h37m elapsed) |
| 2026-09-24 10:46:24 [all] step 17660/18464 loss=7.8511 ppl=2568.59 lr=6.263e-05 | 630 tok/s | 847M done | eta 11h37m19s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 10:54:59 [all] step 17670/18464 loss=7.8391 ppl=2537.91 lr=6.257e-05 | 635 tok/s | 848M done | eta 11h22m24s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 11:03:37 [all] step 17680/18464 loss=7.8217 ppl=2494.15 lr=6.250e-05 | 633 tok/s | 848M done | eta 11h16m40s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 11:12:20 [all] step 17690/18464 loss=7.8244 ppl=2500.82 lr=6.244e-05 | 626 tok/s | 848M done | eta 11h15m07s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 11:14:06 [all] checkpoint @ step 17692 (848M tokens, 11d16h07m elapsed) |
| 2026-09-24 11:21:02 [all] step 17700/18464 loss=7.8253 ppl=2503.19 lr=6.238e-05 | 629 tok/s | 849M done | eta 11h03m48s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 11:29:45 [all] step 17710/18464 loss=7.8196 ppl=2488.94 lr=6.232e-05 | 626 tok/s | 849M done | eta 10h57m19s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 11:38:29 [all] step 17720/18464 loss=7.8213 ppl=2493.14 lr=6.226e-05 | 625 tok/s | 849M done | eta 10h49m41s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 11:44:38 [all] checkpoint @ step 17727 (850M tokens, 11d16h38m elapsed) |
| 2026-09-24 11:47:13 [all] step 17730/18464 loss=7.8193 ppl=2488.07 lr=6.220e-05 | 625 tok/s | 850M done | eta 10h41m07s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 11:55:56 [all] step 17740/18464 loss=7.8521 ppl=2571.10 lr=6.214e-05 | 626 tok/s | 850M done | eta 10h31m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 12:04:37 [all] step 17750/18464 loss=7.8305 ppl=2516.11 lr=6.208e-05 | 629 tok/s | 850M done | eta 10h19m29s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 12:13:18 [all] step 17760/18464 loss=7.8144 ppl=2475.99 lr=6.202e-05 | 629 tok/s | 851M done | eta 10h11m07s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 12:15:04 [all] checkpoint @ step 17762 (851M tokens, 11d17h08m elapsed) |
| 2026-09-24 12:22:01 [all] step 17770/18464 loss=7.8232 ppl=2497.98 lr=6.196e-05 | 626 tok/s | 851M done | eta 10h05m38s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 12:30:52 [all] step 17780/18464 loss=7.8429 ppl=2547.60 lr=6.191e-05 | 618 tok/s | 851M done | eta 10h04m51s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 12:39:43 [all] step 17790/18464 loss=7.8019 ppl=2445.24 lr=6.185e-05 | 618 tok/s | 852M done | eta 09h56m04s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 12:45:57 [all] checkpoint @ step 17797 (852M tokens, 11d17h39m elapsed) |
| 2026-09-24 12:48:37 [all] step 17800/18464 loss=7.8246 ppl=2501.43 lr=6.180e-05 | 613 tok/s | 852M done | eta 09h51m18s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 12:57:35 [all] step 17810/18464 loss=7.8281 ppl=2510.10 lr=6.174e-05 | 609 tok/s | 852M done | eta 09h46m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 13:06:43 [all] step 17820/18464 loss=7.8389 ppl=2537.39 lr=6.169e-05 | 598 tok/s | 853M done | eta 09h48m01s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 13:15:46 [all] step 17830/18464 loss=7.8301 ppl=2515.11 lr=6.164e-05 | 604 tok/s | 853M done | eta 09h33m19s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 13:16:40 [all] checkpoint @ step 17831 (853M tokens, 11d18h10m elapsed) |
| 2026-09-24 13:25:02 [all] step 17840/18464 loss=7.8213 ppl=2493.10 lr=6.159e-05 | 589 tok/s | 853M done | eta 09h38m47s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 13:34:21 [all] step 17850/18464 loss=7.8212 ppl=2492.96 lr=6.154e-05 | 586 tok/s | 854M done | eta 09h32m19s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 13:43:49 [all] step 17860/18464 loss=7.8267 ppl=2506.74 lr=6.149e-05 | 578 tok/s | 854M done | eta 09h31m07s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 13:47:34 [all] checkpoint @ step 17864 (854M tokens, 11d18h41m elapsed) |
| 2026-09-24 13:53:10 [all] step 17870/18464 loss=7.8258 ppl=2504.41 lr=6.144e-05 | 584 tok/s | 854M done | eta 09h15m34s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:01:52 [all] step 17880/18464 loss=7.8191 ppl=2487.61 lr=6.139e-05 | 627 tok/s | 855M done | eta 08h28m32s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:10:18 [all] step 17890/18464 loss=7.8473 ppl=2558.80 lr=6.134e-05 | 648 tok/s | 855M done | eta 08h03m33s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:17:40 [all] checkpoint @ step 17899 (855M tokens, 11d19h11m elapsed) |
| 2026-09-24 14:18:29 [all] step 17900/18464 loss=7.8305 ppl=2516.25 lr=6.130e-05 | 667 tok/s | 855M done | eta 07h41m36s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:26:33 [all] step 17910/18464 loss=7.8139 ppl=2474.87 lr=6.125e-05 | 677 tok/s | 856M done | eta 07h27m09s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:34:36 [all] step 17920/18464 loss=7.8443 ppl=2551.18 lr=6.121e-05 | 678 tok/s | 856M done | eta 07h17m57s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:42:34 [all] step 17930/18464 loss=7.8189 ppl=2487.25 lr=6.116e-05 | 686 tok/s | 856M done | eta 07h05m08s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:48:13 [all] checkpoint @ step 17937 (856M tokens, 11d19h41m elapsed) |
| 2026-09-24 14:50:33 [all] step 17940/18464 loss=7.8336 ppl=2523.94 lr=6.112e-05 | 684 tok/s | 856M done | eta 06h58m13s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 14:58:29 [all] step 17950/18464 loss=7.8233 ppl=2498.06 lr=6.108e-05 | 688 tok/s | 857M done | eta 06h48m14s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 15:06:26 [all] step 17960/18464 loss=7.8334 ppl=2523.60 lr=6.104e-05 | 688 tok/s | 857M done | eta 06h40m20s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 15:14:20 [all] step 17970/18464 loss=7.8207 ppl=2491.61 lr=6.100e-05 | 692 tok/s | 857M done | eta 06h29m57s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 15:18:17 [all] checkpoint @ step 17975 (858M tokens, 11d20h11m elapsed) |
| 2026-09-24 15:22:09 [all] step 17980/18464 loss=7.8171 ppl=2482.68 lr=6.096e-05 | 699 tok/s | 858M done | eta 06h18m10s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 15:30:04 [all] step 17990/18464 loss=7.8246 ppl=2501.43 lr=6.092e-05 | 689 tok/s | 858M done | eta 06h15m38s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 15:37:50 [all] step 18000/18464 loss=7.8226 ppl=2496.47 lr=6.088e-05 | 703 tok/s | 858M done | eta 06h00m30s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 15:38:25 [all] eval step 18000/18464 loss=7.8399 ppl=2539.84 (best 7.8399) |
| 2026-09-24 15:38:27 [all] sample @ 18000: ' the a was the the an are the or the not of the the the the by a a their to the the a your the a not in the the\nThe and and the.\n--- and a from the a that to' |
| 2026-09-24 15:38:28 [all] checkpoint @ step 18000 (858M tokens, 11d20h32m elapsed) |
| 2026-09-24 15:46:18 [all] step 18010/18464 loss=7.8326 ppl=2521.44 lr=6.084e-05 | 646 tok/s | 859M done | eta 06h23m56s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 15:53:57 [all] step 18020/18464 loss=7.8281 ppl=2510.17 lr=6.081e-05 | 714 tok/s | 859M done | eta 05h39m46s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:01:43 [all] step 18030/18464 loss=7.8379 ppl=2534.95 lr=6.077e-05 | 703 tok/s | 859M done | eta 05h37m01s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:08:42 [all] checkpoint @ step 18039 (860M tokens, 11d21h02m elapsed) |
| 2026-09-24 16:09:28 [all] step 18040/18464 loss=7.8461 ppl=2555.75 lr=6.073e-05 | 704 tok/s | 860M done | eta 05h29m04s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:17:16 [all] step 18050/18464 loss=7.8270 ppl=2507.48 lr=6.070e-05 | 700 tok/s | 860M done | eta 05h22m47s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:25:04 [all] step 18060/18464 loss=7.8395 ppl=2539.04 lr=6.067e-05 | 700 tok/s | 860M done | eta 05h15m01s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:32:53 [all] step 18070/18464 loss=7.8201 ppl=2490.21 lr=6.063e-05 | 699 tok/s | 861M done | eta 05h07m54s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:39:01 [all] checkpoint @ step 18078 (861M tokens, 11d21h32m elapsed) |
| 2026-09-24 16:40:33 [all] step 18080/18464 loss=7.8030 ppl=2447.83 lr=6.060e-05 | 712 tok/s | 861M done | eta 04h54m24s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:48:09 [all] step 18090/18464 loss=7.8287 ppl=2511.76 lr=6.057e-05 | 718 tok/s | 861M done | eta 04h44m31s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 16:55:50 [all] step 18100/18464 loss=7.8283 ppl=2510.58 lr=6.054e-05 | 711 tok/s | 862M done | eta 04h39m41s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:03:20 [all] step 18110/18464 loss=7.8345 ppl=2526.37 lr=6.051e-05 | 728 tok/s | 862M done | eta 04h25m25s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:09:22 [all] checkpoint @ step 18118 (862M tokens, 11d22h02m elapsed) |
| 2026-09-24 17:10:50 [all] step 18120/18464 loss=7.8390 ppl=2537.68 lr=6.048e-05 | 728 tok/s | 862M done | eta 04h18m04s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:18:16 [all] step 18130/18464 loss=7.8535 ppl=2574.76 lr=6.046e-05 | 735 tok/s | 863M done | eta 04h08m16s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:25:35 [all] step 18140/18464 loss=7.8296 ppl=2513.82 lr=6.043e-05 | 747 tok/s | 863M done | eta 03h56m43s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:32:44 [all] step 18150/18464 loss=7.8222 ppl=2495.31 lr=6.040e-05 | 764 tok/s | 863M done | eta 03h44m29s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:39:35 [all] step 18160/18464 loss=7.8405 ppl=2541.51 lr=6.038e-05 | 796 tok/s | 864M done | eta 03h28m31s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:39:36 [all] checkpoint @ step 18160 (864M tokens, 11d22h33m elapsed) |
| 2026-09-24 17:46:37 [all] step 18170/18464 loss=7.8192 ppl=2487.79 lr=6.035e-05 | 777 tok/s | 864M done | eta 03h26m45s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 17:53:43 [all] step 18180/18464 loss=7.8128 ppl=2471.95 lr=6.033e-05 | 770 tok/s | 864M done | eta 03h21m32s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:00:52 [all] step 18190/18464 loss=7.8289 ppl=2512.20 lr=6.031e-05 | 763 tok/s | 865M done | eta 03h16m01s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:07:58 [all] step 18200/18464 loss=7.8246 ppl=2501.47 lr=6.029e-05 | 770 tok/s | 865M done | eta 03h07m21s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:10:04 [all] checkpoint @ step 18203 (865M tokens, 11d23h03m elapsed) |
| 2026-09-24 18:14:57 [all] step 18210/18464 loss=7.8415 ppl=2544.05 lr=6.026e-05 | 782 tok/s | 865M done | eta 02h57m21s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:22:24 [all] step 18220/18464 loss=7.8167 ppl=2481.73 lr=6.024e-05 | 732 tok/s | 866M done | eta 03h01m55s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:30:19 [all] step 18230/18464 loss=7.8324 ppl=2521.02 lr=6.022e-05 | 690 tok/s | 866M done | eta 03h05m10s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:38:00 [all] step 18240/18464 loss=7.8415 ppl=2544.01 lr=6.021e-05 | 711 tok/s | 866M done | eta 02h52m09s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:40:11 [all] checkpoint @ step 18243 (866M tokens, 11d23h33m elapsed) |
| 2026-09-24 18:45:00 [all] step 18250/18464 loss=7.8425 ppl=2546.62 lr=6.019e-05 | 781 tok/s | 867M done | eta 02h29m33s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 18:52:43 [all] step 18260/18464 loss=7.8299 ppl=2514.69 lr=6.017e-05 | 708 tok/s | 867M done | eta 02h37m23s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 19:00:20 [all] step 18270/18464 loss=7.8365 ppl=2531.26 lr=6.015e-05 | 716 tok/s | 867M done | eta 02h27m52s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 19:08:27 [all] step 18280/18464 loss=7.8248 ppl=2501.89 lr=6.014e-05 | 673 tok/s | 868M done | eta 02h29m22s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 19:10:55 [all] checkpoint @ step 18283 (868M tokens, 12d00h04m elapsed) |
| 2026-09-24 19:17:19 [all] step 18290/18464 loss=7.8185 ppl=2486.21 lr=6.012e-05 | 616 tok/s | 868M done | eta 02h34m15s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 19:26:38 [all] step 18300/18464 loss=7.8142 ppl=2475.47 lr=6.011e-05 | 586 tok/s | 868M done | eta 02h32m52s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 19:36:09 [all] step 18310/18464 loss=7.8510 ppl=2568.30 lr=6.010e-05 | 574 tok/s | 869M done | eta 02h26m30s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 19:41:02 [all] checkpoint @ step 18315 (869M tokens, 12d00h34m elapsed) |
| 2026-09-24 19:45:53 [all] step 18320/18464 loss=7.8127 ppl=2471.72 lr=6.009e-05 | 561 tok/s | 869M done | eta 02h20m08s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 19:55:15 [all] step 18330/18464 loss=7.8304 ppl=2516.00 lr=6.007e-05 | 583 tok/s | 869M done | eta 02h05m26s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 20:04:15 [all] step 18340/18464 loss=7.8115 ppl=2468.87 lr=6.006e-05 | 607 tok/s | 870M done | eta 01h51m33s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 20:11:57 [all] checkpoint @ step 18348 (870M tokens, 12d01h05m elapsed) |
| 2026-09-24 20:13:53 [all] step 18350/18464 loss=7.8260 ppl=2504.89 lr=6.005e-05 | 566 tok/s | 870M done | eta 01h49m54s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 20:23:28 [all] step 18360/18464 loss=7.8220 ppl=2494.91 lr=6.004e-05 | 570 tok/s | 870M done | eta 01h39m35s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 20:32:59 [all] step 18370/18464 loss=7.8178 ppl=2484.42 lr=6.004e-05 | 574 tok/s | 871M done | eta 01h29m28s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 20:42:28 [all] step 18380/18464 loss=7.8252 ppl=2502.96 lr=6.003e-05 | 575 tok/s | 871M done | eta 01h19m45s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 20:42:29 [all] checkpoint @ step 18380 (871M tokens, 12d01h36m elapsed) |
| 2026-09-24 20:51:42 [all] step 18390/18464 loss=7.8226 ppl=2496.43 lr=6.002e-05 | 592 tok/s | 871M done | eta 01h08m16s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 21:00:47 [all] step 18400/18464 loss=7.8138 ppl=2474.45 lr=6.002e-05 | 602 tok/s | 872M done | eta 00h58m06s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 21:11:18 [all] step 18410/18464 loss=7.8360 ppl=2530.06 lr=6.001e-05 | 519 tok/s | 872M done | eta 00h56m49s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 21:13:31 [all] checkpoint @ step 18412 (872M tokens, 12d02h07m elapsed) |
| 2026-09-24 21:22:11 [all] step 18420/18464 loss=7.8399 ppl=2539.98 lr=6.001e-05 | 502 tok/s | 872M done | eta 00h47m51s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 21:32:35 [all] step 18430/18464 loss=7.8422 ppl=2545.70 lr=6.000e-05 | 525 tok/s | 873M done | eta 00h35m23s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 21:42:40 [all] step 18440/18464 loss=7.8224 ppl=2495.94 lr=6.000e-05 | 541 tok/s | 873M done | eta 00h24m12s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 21:43:45 [all] checkpoint @ step 18441 (873M tokens, 12d02h37m elapsed) |
| 2026-09-24 21:53:55 [all] step 18450/18464 loss=7.8388 ppl=2537.24 lr=6.000e-05 | 486 tok/s | 873M done | eta 00h15m44s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 22:05:04 [all] step 18460/18464 loss=7.8178 ppl=2484.48 lr=6.000e-05 | 490 tok/s | 874M done | eta 00h04m27s | vram 0.75/3.37GB peak 3.28GB | rung 1 |
| 2026-09-24 22:10:04 [all] eval step 18464/18464 loss=7.8398 ppl=2539.79 (best 7.8398) |
| 2026-09-24 22:10:04 [all] checkpoint @ step 18464 (874M tokens, 12d03h03m elapsed) |
| 2026-09-24 22:10:05 [all] final model exported -> runs/hypernix.3-mini/final (load it with neo_oven.preheat_brewed(path, tokenizer_source=path)) |
| 2026-09-24 22:10:05 [all] DONE — 874M tokens in 12d03h03m (deadline 9999h, within budget). best val ppl 2539.79 |
|
|
