| R102 start 2026-10-23T22:17:25Z z /root/pilot1b/start/ckpt_2250.pt sha aeddfb023abb11c8654b47c728c9680c232d1b6568697e3a1eee8128efd37054 steps 5720 S3 3594 S4 5389 ckpt-at 4576 |
| [22:17:31] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 5090 world=8 |
| [22:17:31] dane: train=1,889,519,627B block=4096 batch=256 tok/step=1,048,576 |
| [22:17:42] model GPT-ref: 1073.9M param (L20 d2048 h16) |
| [22:17:43] optimizer=muon muon_lr=0.02 (mult=33.3x) |
| [22:17:43] r11 pamiec: act_ckpt=True loss_chunk=1024 |
| [22:17:43] r13: mixer_layout=AAAAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved |
| [22:17:46] torch.compile enabled |
| [22:17:52] RESUME @ 2250 |
| [22:17:52] START gpt-ref: steps=5720 (od 2250) lr=0.0006->6e-05 |
| [22:17:52] sampler epoch: 461,307 okien/epoke (offset0 2245), start od kroku 2250 = epoka 1.249 |
| [22:17:52] data-dir-2: /root/pilot1b/data_p01 1,880,520,445 tok, 459,110 okien/epoke = 1793 krokow, od kroku strumienia 1801 |
| [22:17:52] r24 train3: 1,883,249,429 tok, 459,777 okien/epoke = 1796 krokow, od kroku strumienia 3594 |
| [22:17:52] r24 train4: 1,890,850,536 tok, 461,632 okien/epoke = 1803 krokow, od kroku strumienia 5389 |
| [22:21:05] step 2260/5720 loss 2.6379 lr 6.00e-04 gnorm 0.10 54,164 tok/s peakVRAM 27.0GB |
| [22:22:38] step 2270/5720 loss 2.6461 lr 6.00e-04 gnorm 0.08 113,489 tok/s peakVRAM 27.0GB |
| [22:24:10] step 2280/5720 loss 2.6532 lr 6.00e-04 gnorm 0.08 113,231 tok/s peakVRAM 27.0GB |
| [22:25:43] step 2290/5720 loss 2.6472 lr 6.00e-04 gnorm 0.08 113,191 tok/s peakVRAM 27.0GB |
| [22:27:16] step 2300/5720 loss 2.6436 lr 6.00e-04 gnorm 0.08 113,099 tok/s peakVRAM 27.0GB |
| [22:28:48] step 2310/5720 loss 2.6365 lr 6.00e-04 gnorm 0.10 113,222 tok/s peakVRAM 27.0GB |
| [22:30:21] step 2320/5720 loss 2.6297 lr 6.00e-04 gnorm 0.07 113,237 tok/s peakVRAM 27.0GB |
| [22:31:54] step 2330/5720 loss 2.6421 lr 6.00e-04 gnorm 0.09 113,292 tok/s peakVRAM 27.0GB |
| [22:33:26] step 2340/5720 loss 2.6351 lr 6.00e-04 gnorm 0.09 113,238 tok/s peakVRAM 27.0GB |
| [22:34:59] step 2350/5720 loss 2.6249 lr 6.00e-04 gnorm 0.07 113,263 tok/s peakVRAM 27.0GB |
| [22:36:31] step 2360/5720 loss 2.6175 lr 6.00e-04 gnorm 0.08 113,301 tok/s peakVRAM 27.0GB |
| [22:38:04] step 2370/5720 loss 2.6197 lr 6.00e-04 gnorm 0.07 113,345 tok/s peakVRAM 27.0GB |
| [22:39:36] step 2380/5720 loss 2.6259 lr 6.00e-04 gnorm 0.08 113,443 tok/s peakVRAM 27.0GB |
| [22:41:09] step 2390/5720 loss 2.6177 lr 6.00e-04 gnorm 0.07 113,353 tok/s peakVRAM 27.0GB |
| [22:42:41] step 2400/5720 loss 2.6315 lr 6.00e-04 gnorm 0.08 113,380 tok/s peakVRAM 27.0GB |
| [22:42:50] ckpt @ 2400 |
| [22:44:22] step 2410/5720 loss 2.6364 lr 6.00e-04 gnorm 0.09 103,710 tok/s peakVRAM 27.0GB |
| [22:45:55] step 2420/5720 loss 2.6255 lr 6.00e-04 gnorm 0.09 113,315 tok/s peakVRAM 27.0GB |
| [22:47:27] step 2430/5720 loss 2.6181 lr 6.00e-04 gnorm 0.08 113,253 tok/s peakVRAM 27.0GB |
| [22:49:00] step 2440/5720 loss 2.6159 lr 6.00e-04 gnorm 0.08 113,251 tok/s peakVRAM 27.0GB |
| [22:50:33] step 2450/5720 loss 2.6271 lr 6.00e-04 gnorm 0.08 113,260 tok/s peakVRAM 27.0GB |
| [22:52:05] step 2460/5720 loss 2.6269 lr 6.00e-04 gnorm 0.09 113,266 tok/s peakVRAM 27.0GB |
| [22:53:38] step 2470/5720 loss 2.6160 lr 6.00e-04 gnorm 0.08 113,419 tok/s peakVRAM 27.0GB |
| [22:55:10] step 2480/5720 loss 2.6256 lr 6.00e-04 gnorm 0.08 113,388 tok/s peakVRAM 27.0GB |
| [22:56:43] step 2490/5720 loss 2.6332 lr 6.00e-04 gnorm 0.07 113,374 tok/s peakVRAM 27.0GB |
| [22:58:15] step 2500/5720 loss 2.6236 lr 6.00e-04 gnorm 0.07 113,430 tok/s peakVRAM 27.0GB |
| [22:59:48] step 2510/5720 loss 2.6180 lr 6.00e-04 gnorm 0.08 113,347 tok/s peakVRAM 27.0GB |
| [23:01:20] step 2520/5720 loss 2.6165 lr 6.00e-04 gnorm 0.07 113,325 tok/s peakVRAM 27.0GB |
| [23:02:53] step 2530/5720 loss 2.6012 lr 6.00e-04 gnorm 0.07 113,268 tok/s peakVRAM 27.0GB |
| [23:04:25] step 2540/5720 loss 2.6064 lr 6.00e-04 gnorm 0.09 113,220 tok/s peakVRAM 27.0GB |
| [23:05:58] step 2550/5720 loss 2.6060 lr 6.00e-04 gnorm 0.08 113,163 tok/s peakVRAM 27.0GB |
| [23:06:06] ckpt @ 2550 |
| [23:07:38] step 2560/5720 loss 2.5938 lr 6.00e-04 gnorm 0.07 104,295 tok/s peakVRAM 27.0GB |
| [23:09:11] step 2570/5720 loss 2.6062 lr 6.00e-04 gnorm 0.08 113,333 tok/s peakVRAM 27.0GB |
| [23:10:44] step 2580/5720 loss 2.6028 lr 6.00e-04 gnorm 0.08 113,215 tok/s peakVRAM 27.0GB |
| [23:12:16] step 2590/5720 loss 2.6091 lr 6.00e-04 gnorm 0.08 113,178 tok/s peakVRAM 27.0GB |
| [23:13:49] step 2600/5720 loss 2.6079 lr 6.00e-04 gnorm 0.08 113,194 tok/s peakVRAM 27.0GB |
| [23:15:22] step 2610/5720 loss 2.6177 lr 6.00e-04 gnorm 0.08 113,199 tok/s peakVRAM 27.0GB |
| [23:16:54] step 2620/5720 loss 2.5998 lr 6.00e-04 gnorm 0.08 113,183 tok/s peakVRAM 27.0GB |
| [23:18:27] step 2630/5720 loss 2.5994 lr 6.00e-04 gnorm 0.08 112,992 tok/s peakVRAM 27.0GB |
| [23:20:00] step 2640/5720 loss 2.6001 lr 6.00e-04 gnorm 0.08 113,121 tok/s peakVRAM 27.0GB |
| [23:21:32] step 2650/5720 loss 2.6212 lr 6.00e-04 gnorm 0.08 113,194 tok/s peakVRAM 27.0GB |
| [23:23:05] step 2660/5720 loss 2.5814 lr 6.00e-04 gnorm 0.08 113,228 tok/s peakVRAM 27.0GB |
| [23:24:37] step 2670/5720 loss 2.6000 lr 6.00e-04 gnorm 0.09 113,302 tok/s peakVRAM 27.0GB |
| [23:26:10] step 2680/5720 loss 2.5943 lr 6.00e-04 gnorm 0.07 113,406 tok/s peakVRAM 27.0GB |
| [23:27:42] step 2690/5720 loss 2.5914 lr 6.00e-04 gnorm 0.09 113,344 tok/s peakVRAM 27.0GB |
| [23:29:15] step 2700/5720 loss 2.5959 lr 6.00e-04 gnorm 0.09 113,295 tok/s peakVRAM 27.0GB |
| [23:29:23] ckpt @ 2700 |
| [23:30:56] step 2710/5720 loss 2.6002 lr 6.00e-04 gnorm 0.07 104,284 tok/s peakVRAM 27.0GB |
| [23:32:28] step 2720/5720 loss 2.5896 lr 6.00e-04 gnorm 0.08 113,130 tok/s peakVRAM 27.0GB |
| [23:34:01] step 2730/5720 loss 2.6118 lr 6.00e-04 gnorm 0.07 113,009 tok/s peakVRAM 27.0GB |
| [23:35:34] step 2740/5720 loss 2.5942 lr 6.00e-04 gnorm 0.07 113,004 tok/s peakVRAM 27.0GB |
| [23:37:07] step 2750/5720 loss 2.5820 lr 6.00e-04 gnorm 0.07 113,043 tok/s peakVRAM 27.0GB |
| [23:38:39] step 2760/5720 loss 2.5676 lr 6.00e-04 gnorm 0.08 113,155 tok/s peakVRAM 27.0GB |
| [23:40:12] step 2770/5720 loss 2.5999 lr 6.00e-04 gnorm 0.07 113,241 tok/s peakVRAM 27.0GB |
| [23:41:45] step 2780/5720 loss 2.5972 lr 6.00e-04 gnorm 0.09 113,110 tok/s peakVRAM 27.0GB |
| [23:43:17] step 2790/5720 loss 2.5851 lr 6.00e-04 gnorm 0.08 113,233 tok/s peakVRAM 27.0GB |
| [23:44:50] step 2800/5720 loss 2.5806 lr 6.00e-04 gnorm 0.08 113,205 tok/s peakVRAM 27.0GB |
| [23:46:22] step 2810/5720 loss 2.5821 lr 6.00e-04 gnorm 0.07 113,123 tok/s peakVRAM 27.0GB |
| [23:47:55] step 2820/5720 loss 2.5823 lr 6.00e-04 gnorm 0.08 113,254 tok/s peakVRAM 27.0GB |
| [23:49:28] step 2830/5720 loss 2.5675 lr 6.00e-04 gnorm 0.07 113,404 tok/s peakVRAM 27.0GB |
| [23:51:00] step 2840/5720 loss 2.5864 lr 6.00e-04 gnorm 0.07 113,377 tok/s peakVRAM 27.0GB |
| [23:52:32] step 2850/5720 loss 2.5572 lr 6.00e-04 gnorm 0.07 113,451 tok/s peakVRAM 27.0GB |
| [23:52:41] ckpt @ 2850 |
| [23:54:13] step 2860/5720 loss 2.5817 lr 6.00e-04 gnorm 0.08 103,743 tok/s peakVRAM 27.0GB |
| [23:55:46] step 2870/5720 loss 2.5747 lr 6.00e-04 gnorm 0.07 113,433 tok/s peakVRAM 27.0GB |
| [23:57:19] step 2880/5720 loss 2.5715 lr 6.00e-04 gnorm 0.08 113,227 tok/s peakVRAM 27.0GB |
| [23:58:51] step 2890/5720 loss 2.5776 lr 6.00e-04 gnorm 0.07 113,315 tok/s peakVRAM 27.0GB |
| [00:00:24] step 2900/5720 loss 2.5869 lr 6.00e-04 gnorm 0.09 113,436 tok/s peakVRAM 27.0GB |
| [00:01:56] step 2910/5720 loss 2.5714 lr 6.00e-04 gnorm 0.07 113,418 tok/s peakVRAM 27.0GB |
| [00:03:28] step 2920/5720 loss 2.5774 lr 6.00e-04 gnorm 0.07 113,436 tok/s peakVRAM 27.0GB |
| [00:05:01] step 2930/5720 loss 2.5815 lr 6.00e-04 gnorm 0.08 113,427 tok/s peakVRAM 27.0GB |
| [00:06:33] step 2940/5720 loss 2.5607 lr 6.00e-04 gnorm 0.07 113,484 tok/s peakVRAM 27.0GB |
| [00:08:06] step 2950/5720 loss 2.5818 lr 6.00e-04 gnorm 0.09 113,380 tok/s peakVRAM 27.0GB |
| [00:09:38] step 2960/5720 loss 2.5656 lr 6.00e-04 gnorm 0.07 113,392 tok/s peakVRAM 27.0GB |
| [00:11:11] step 2970/5720 loss 2.5829 lr 6.00e-04 gnorm 0.08 113,329 tok/s peakVRAM 27.0GB |
| [00:12:43] step 2980/5720 loss 2.5874 lr 6.00e-04 gnorm 0.08 113,352 tok/s peakVRAM 27.0GB |
| [00:14:16] step 2990/5720 loss 2.5585 lr 6.00e-04 gnorm 0.06 113,331 tok/s peakVRAM 27.0GB |
| [00:15:48] step 3000/5720 loss 2.5749 lr 6.00e-04 gnorm 0.07 113,327 tok/s peakVRAM 27.0GB |
| [00:15:57] ckpt @ 3000 |
| [00:17:29] step 3010/5720 loss 2.5600 lr 6.00e-04 gnorm 0.08 104,129 tok/s peakVRAM 27.0GB |
| [00:19:02] step 3020/5720 loss 2.5549 lr 6.00e-04 gnorm 0.07 113,342 tok/s peakVRAM 27.0GB |
| [00:20:34] step 3030/5720 loss 2.5535 lr 6.00e-04 gnorm 0.08 113,236 tok/s peakVRAM 27.0GB |
| [00:22:07] step 3040/5720 loss 2.5648 lr 6.00e-04 gnorm 0.07 113,214 tok/s peakVRAM 27.0GB |
| [00:23:39] step 3050/5720 loss 2.5730 lr 6.00e-04 gnorm 0.08 113,294 tok/s peakVRAM 27.0GB |
| [00:25:12] step 3060/5720 loss 2.5516 lr 6.00e-04 gnorm 0.06 113,304 tok/s peakVRAM 27.0GB |
| [00:26:44] step 3070/5720 loss 2.5735 lr 6.00e-04 gnorm 0.08 113,346 tok/s peakVRAM 27.0GB |
| [00:28:17] step 3080/5720 loss 2.5782 lr 6.00e-04 gnorm 0.08 113,366 tok/s peakVRAM 27.0GB |
| [00:29:49] step 3090/5720 loss 2.5622 lr 6.00e-04 gnorm 0.07 113,345 tok/s peakVRAM 27.0GB |
| [00:31:22] step 3100/5720 loss 2.5571 lr 6.00e-04 gnorm 0.07 113,372 tok/s peakVRAM 27.0GB |
| [00:32:54] step 3110/5720 loss 2.5389 lr 6.00e-04 gnorm 0.07 113,364 tok/s peakVRAM 27.0GB |
| [00:34:27] step 3120/5720 loss 2.5615 lr 6.00e-04 gnorm 0.07 113,394 tok/s peakVRAM 27.0GB |
| [00:35:59] step 3130/5720 loss 2.5639 lr 6.00e-04 gnorm 0.08 113,415 tok/s peakVRAM 27.0GB |
| [00:37:32] step 3140/5720 loss 2.5503 lr 6.00e-04 gnorm 0.07 113,358 tok/s peakVRAM 27.0GB |
| [00:39:04] step 3150/5720 loss 2.5463 lr 6.00e-04 gnorm 0.07 113,405 tok/s peakVRAM 27.0GB |
| [00:39:13] ckpt @ 3150 |
| [00:40:45] step 3160/5720 loss 2.5686 lr 6.00e-04 gnorm 0.07 104,167 tok/s peakVRAM 27.0GB |
| [00:42:17] step 3170/5720 loss 2.5492 lr 6.00e-04 gnorm 0.08 113,506 tok/s peakVRAM 27.0GB |
| [00:43:50] step 3180/5720 loss 2.5450 lr 6.00e-04 gnorm 0.08 113,475 tok/s peakVRAM 27.0GB |
| [00:45:22] step 3190/5720 loss 2.5459 lr 6.00e-04 gnorm 0.07 113,522 tok/s peakVRAM 27.0GB |
| [00:46:55] step 3200/5720 loss 2.5286 lr 6.00e-04 gnorm 0.07 113,402 tok/s peakVRAM 27.0GB |
| [00:48:27] step 3210/5720 loss 2.5580 lr 6.00e-04 gnorm 0.07 113,391 tok/s peakVRAM 27.0GB |
| [00:49:59] step 3220/5720 loss 2.5494 lr 6.00e-04 gnorm 0.08 113,361 tok/s peakVRAM 27.0GB |
| [00:51:32] step 3230/5720 loss 2.5754 lr 6.00e-04 gnorm 0.07 113,407 tok/s peakVRAM 27.0GB |
| [00:53:04] step 3240/5720 loss 2.5362 lr 6.00e-04 gnorm 0.07 113,385 tok/s peakVRAM 27.0GB |
| [00:54:37] step 3250/5720 loss 2.5611 lr 6.00e-04 gnorm 0.06 113,386 tok/s peakVRAM 27.0GB |
| [00:56:09] step 3260/5720 loss 2.5556 lr 6.00e-04 gnorm 0.07 113,390 tok/s peakVRAM 27.0GB |
| [00:57:42] step 3270/5720 loss 2.5300 lr 6.00e-04 gnorm 0.07 113,442 tok/s peakVRAM 27.0GB |
| [00:59:14] step 3280/5720 loss 2.5741 lr 6.00e-04 gnorm 0.07 113,384 tok/s peakVRAM 27.0GB |
| [01:00:47] step 3290/5720 loss 2.5494 lr 6.00e-04 gnorm 0.06 113,408 tok/s peakVRAM 27.0GB |
| [01:02:19] step 3300/5720 loss 2.5484 lr 6.00e-04 gnorm 0.06 113,464 tok/s peakVRAM 27.0GB |
| [01:02:28] ckpt @ 3300 |
| [01:04:00] step 3310/5720 loss 2.5522 lr 6.00e-04 gnorm 0.08 103,995 tok/s peakVRAM 27.0GB |
|
|
