|
|
| █████ █████ |
| ░░███ ░░███ |
| ████████ ██████ ████████ ██████ ██████ ░███████ ██████ ███████ |
| ░░███░░███ ░░░░░███ ░░███░░███ ███░░███ ███░░███ ░███░░███ ░░░░░███░░░███░ |
| ░███ ░███ ███████ ░███ ░███ ░███ ░███░███ ░░░ ░███ ░███ ███████ ░███ |
| ░███ ░███ ███░░███ ░███ ░███ ░███ ░███░███ ███ ░███ ░███ ███░░███ ░███ ███ |
| ████ █████░░████████ ████ █████░░██████ ░░██████ ████ █████░░███████ ░░█████ |
| ░░░░ ░░░░░ ░░░░░░░░ ░░░░ ░░░░░ ░░░░░░ ░░░░░░ ░░░░ ░░░░░ ░░░░░░░░ ░░░░░ |
| |
| Autodetected device type: cuda |
| 2026-08-10 21:12:28,190 - nanochat.common - [32m[1mINFO[0m - Distributed world size: 8 |
| GPU: NVIDIA H200 | Peak FLOPS (BF16): 9.89e+14 |
| COMPUTE_DTYPE: torch.bfloat16 (auto-detected: CUDA SM 90 (bf16 supported)) |
| ✓ Using Flash Attention 3: efficient, new and awesome. |
| Vocab size: 65,536 |
| Model config: |
| { |
| "sequence_len": 2048, |
| "vocab_size": 65536, |
| "n_layer": 32, |
| "n_head": 16, |
| "n_kv_head": 16, |
| "n_embd": 2048, |
| "window_pattern": "L", |
| "attn_variant": "hmap", |
| "hmap_alpha": 1.0, |
| "hmap_beta": 1.0, |
| "witten": false, |
| "bidirectional": false |
| } |
| [d32ft] Weights-only warm start from /root/.cache/nanochat-d32ft/bootstrap/karpathy-d32/model_000650.pt |
| [d32ft] New optimizer + new dataloader state; local step starts at 0 |
| [d32ft] missing from checkpoint: backout_lambda |
| [d32ft] missing from checkpoint: resid_lambdas |
| [d32ft] missing from checkpoint: smear_gate.weight |
| [d32ft] missing from checkpoint: smear_lambda |
| [d32ft] missing from checkpoint: transformer.h.1.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.11.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.13.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.15.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.17.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.19.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.21.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.23.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.25.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.27.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.29.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.3.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.31.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.5.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.7.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: transformer.h.9.attn.ve_gate.weight |
| [d32ft] missing from checkpoint: value_embeds.1.weight |
| [d32ft] missing from checkpoint: value_embeds.11.weight |
| [d32ft] missing from checkpoint: value_embeds.13.weight |
| [d32ft] missing from checkpoint: value_embeds.15.weight |
| [d32ft] missing from checkpoint: value_embeds.17.weight |
| [d32ft] missing from checkpoint: value_embeds.19.weight |
| [d32ft] missing from checkpoint: value_embeds.21.weight |
| [d32ft] missing from checkpoint: value_embeds.23.weight |
| [d32ft] missing from checkpoint: value_embeds.25.weight |
| [d32ft] missing from checkpoint: value_embeds.27.weight |
| [d32ft] missing from checkpoint: value_embeds.29.weight |
| [d32ft] missing from checkpoint: value_embeds.3.weight |
| [d32ft] missing from checkpoint: value_embeds.31.weight |
| [d32ft] missing from checkpoint: value_embeds.5.weight |
| [d32ft] missing from checkpoint: value_embeds.7.weight |
| [d32ft] missing from checkpoint: value_embeds.9.weight |
| [d32ft] missing from checkpoint: x0_lambdas |
| [d32ft] graft-neutral override: resid_lambdas -> 1.0 (was from-scratch init ~1.13) |
| [d32ft] graft-neutral override: x0_lambdas -> 0.0 (was from-scratch init) |
| [d32ft] kept init (verify neutral!): smear_lambda shape=(1,) mean=0.0000 values=[0.000] |
| [d32ft] graft-neutral override: backout_lambda -> 0.0 (was from-scratch init) |
| [d32ft] kept init (verify neutral!): transformer.h.1.attn.ve_gate.weight shape=(16, 12) mean=0.0098 values=[0.014, 0.000, 0.015, 0.001, 0.002, 0.010, 0.009, 0.010, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.3.attn.ve_gate.weight shape=(16, 12) mean=0.0101 values=[0.007, 0.005, 0.002, 0.012, 0.010, 0.002, 0.018, 0.012, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.5.attn.ve_gate.weight shape=(16, 12) mean=0.0101 values=[0.003, 0.019, 0.008, 0.014, 0.018, 0.019, 0.007, 0.013, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.7.attn.ve_gate.weight shape=(16, 12) mean=0.0096 values=[0.002, 0.010, 0.006, 0.009, 0.003, 0.007, 0.018, 0.005, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.9.attn.ve_gate.weight shape=(16, 12) mean=0.0097 values=[0.013, 0.015, 0.014, 0.018, 0.001, 0.017, 0.009, 0.017, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.11.attn.ve_gate.weight shape=(16, 12) mean=0.0100 values=[0.008, 0.004, 0.007, 0.014, 0.013, 0.009, 0.003, 0.003, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.13.attn.ve_gate.weight shape=(16, 12) mean=0.0102 values=[0.000, 0.003, 0.004, 0.005, 0.008, 0.018, 0.011, 0.008, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.15.attn.ve_gate.weight shape=(16, 12) mean=0.0103 values=[0.017, 0.008, 0.002, 0.014, 0.001, 0.011, 0.003, 0.011, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.17.attn.ve_gate.weight shape=(16, 12) mean=0.0098 values=[0.012, 0.008, 0.019, 0.007, 0.014, 0.006, 0.002, 0.005, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.19.attn.ve_gate.weight shape=(16, 12) mean=0.0107 values=[0.010, 0.010, 0.020, 0.014, 0.019, 0.019, 0.003, 0.006, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.21.attn.ve_gate.weight shape=(16, 12) mean=0.0095 values=[0.007, 0.000, 0.016, 0.005, 0.014, 0.010, 0.003, 0.003, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.23.attn.ve_gate.weight shape=(16, 12) mean=0.0099 values=[0.019, 0.017, 0.020, 0.001, 0.009, 0.018, 0.013, 0.006, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.25.attn.ve_gate.weight shape=(16, 12) mean=0.0099 values=[0.013, 0.014, 0.010, 0.004, 0.005, 0.004, 0.009, 0.002, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.27.attn.ve_gate.weight shape=(16, 12) mean=0.0099 values=[0.003, 0.005, 0.013, 0.002, 0.010, 0.009, 0.009, 0.012, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.29.attn.ve_gate.weight shape=(16, 12) mean=0.0104 values=[0.016, 0.014, 0.018, 0.018, 0.007, 0.018, 0.019, 0.002, ...] |
| [d32ft] kept init (verify neutral!): transformer.h.31.attn.ve_gate.weight shape=(16, 12) mean=0.0098 values=[0.019, 0.011, 0.020, 0.008, 0.017, 0.020, 0.001, 0.008, ...] |
| [d32ft] kept init (verify neutral!): smear_gate.weight shape=(1, 24) mean=0.0102 values=[0.008, 0.013, 0.017, 0.004, 0.015, 0.017, 0.017, 0.002, ...] |
| [d32ft] zero-initialized VE table: value_embeds.1.weight |
| [d32ft] zero-initialized VE table: value_embeds.3.weight |
| [d32ft] zero-initialized VE table: value_embeds.5.weight |
| [d32ft] zero-initialized VE table: value_embeds.7.weight |
| [d32ft] zero-initialized VE table: value_embeds.9.weight |
| [d32ft] zero-initialized VE table: value_embeds.11.weight |
| [d32ft] zero-initialized VE table: value_embeds.13.weight |
| [d32ft] zero-initialized VE table: value_embeds.15.weight |
| [d32ft] zero-initialized VE table: value_embeds.17.weight |
| [d32ft] zero-initialized VE table: value_embeds.19.weight |
| [d32ft] zero-initialized VE table: value_embeds.21.weight |
| [d32ft] zero-initialized VE table: value_embeds.23.weight |
| [d32ft] zero-initialized VE table: value_embeds.25.weight |
| [d32ft] zero-initialized VE table: value_embeds.27.weight |
| [d32ft] zero-initialized VE table: value_embeds.29.weight |
| [d32ft] zero-initialized VE table: value_embeds.31.weight |
| Parameter counts: |
| wte : 134,217,728 |
| value_embeds : 2,147,483,648 |
| lm_head : 134,217,728 |
| transformer_matrices : 1,610,615,808 |
| scalars : 90 |
| total : 4,026,535,002 |
| Estimated FLOPs per token: 1.207961e+10 |
| Scaling the LR for the AdamW parameters ∝1/√(2048/768) = 0.612372 |
| Using user-provided number of iterations: 1,500 |
| Total number of training tokens: 786,432,000 |
| Tokens : Scaling params ratio: 0.45 |
| Total training FLOPs estimate: 9.499795e+18 |
| Tokens / micro-batch / rank: 2 x 2048 = 4,096 |
| Tokens / micro-batch: 32,768 |
| Total batch size 524,288 => gradient accumulation steps: 16 |
| Step 00000 | Validation bpb: 1.795972 |
| Step 00000 | ICL early: 5.1304 late: 5.8076 score: +0.6772 | Induction loss: 15.0148 acc: 0.0000 (random-half: 15.1274) |
| step 00000/01500 (0.00%) | loss: 5.939279 | lrm: 0.03 | dt: 173819.10ms | tok/sec: 3,016 | bf16_mfu: 0.46 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00001/01500 (0.07%) | loss: 5.956664 | lrm: 0.05 | dt: 2685.29ms | tok/sec: 195,244 | bf16_mfu: 29.81 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00002/01500 (0.13%) | loss: 5.883790 | lrm: 0.07 | dt: 2861.30ms | tok/sec: 183,234 | bf16_mfu: 27.98 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00003/01500 (0.20%) | loss: 5.876734 | lrm: 0.10 | dt: 2482.99ms | tok/sec: 211,152 | bf16_mfu: 32.24 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00004/01500 (0.27%) | loss: 5.805432 | lrm: 0.12 | dt: 2494.62ms | tok/sec: 210,167 | bf16_mfu: 32.09 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00005/01500 (0.33%) | loss: 5.792202 | lrm: 0.15 | dt: 2494.67ms | tok/sec: 210,162 | bf16_mfu: 32.09 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00006/01500 (0.40%) | loss: 5.767968 | lrm: 0.17 | dt: 2502.11ms | tok/sec: 209,538 | bf16_mfu: 31.99 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00007/01500 (0.47%) | loss: 5.776086 | lrm: 0.20 | dt: 2500.57ms | tok/sec: 209,667 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00008/01500 (0.53%) | loss: 5.763400 | lrm: 0.23 | dt: 2490.52ms | tok/sec: 210,513 | bf16_mfu: 32.14 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00009/01500 (0.60%) | loss: 5.676017 | lrm: 0.25 | dt: 2494.58ms | tok/sec: 210,170 | bf16_mfu: 32.09 | epoch: 1 pq: 0 rg: 8 | total time: 0.00m |
| step 00010/01500 (0.67%) | loss: 5.621683 | lrm: 0.28 | dt: 2497.56ms | tok/sec: 209,919 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 16 | total time: 0.00m |
| step 00011/01500 (0.73%) | loss: 5.571587 | lrm: 0.30 | dt: 2499.24ms | tok/sec: 209,779 | bf16_mfu: 32.03 | epoch: 1 pq: 0 rg: 16 | total time: 0.04m | eta: 62.0m |
| step 00012/01500 (0.80%) | loss: 5.548217 | lrm: 0.33 | dt: 2498.47ms | tok/sec: 209,843 | bf16_mfu: 32.04 | epoch: 1 pq: 0 rg: 16 | total time: 0.08m | eta: 62.0m |
| step 00013/01500 (0.87%) | loss: 5.470431 | lrm: 0.35 | dt: 2498.29ms | tok/sec: 209,858 | bf16_mfu: 32.04 | epoch: 1 pq: 0 rg: 16 | total time: 0.12m | eta: 61.9m |
| step 00014/01500 (0.93%) | loss: 5.430450 | lrm: 0.38 | dt: 2501.25ms | tok/sec: 209,610 | bf16_mfu: 32.00 | epoch: 1 pq: 0 rg: 16 | total time: 0.17m | eta: 61.9m |
| step 00015/01500 (1.00%) | loss: 5.347769 | lrm: 0.40 | dt: 2499.86ms | tok/sec: 209,726 | bf16_mfu: 32.02 | epoch: 1 pq: 0 rg: 16 | total time: 0.21m | eta: 61.9m |
| step 00016/01500 (1.07%) | loss: 5.246905 | lrm: 0.42 | dt: 2499.73ms | tok/sec: 209,737 | bf16_mfu: 32.02 | epoch: 1 pq: 0 rg: 16 | total time: 0.25m | eta: 61.8m |
| step 00017/01500 (1.13%) | loss: 5.172430 | lrm: 0.45 | dt: 2493.68ms | tok/sec: 210,246 | bf16_mfu: 32.10 | epoch: 1 pq: 0 rg: 16 | total time: 0.29m | eta: 61.8m |
| step 00018/01500 (1.20%) | loss: 5.120843 | lrm: 0.47 | dt: 2499.59ms | tok/sec: 209,749 | bf16_mfu: 32.02 | epoch: 1 pq: 0 rg: 24 | total time: 0.33m | eta: 61.7m |
| step 00019/01500 (1.27%) | loss: 5.091685 | lrm: 0.50 | dt: 2494.79ms | tok/sec: 210,153 | bf16_mfu: 32.09 | epoch: 1 pq: 0 rg: 24 | total time: 0.37m | eta: 61.7m |
| step 00020/01500 (1.33%) | loss: 5.031218 | lrm: 0.53 | dt: 2497.78ms | tok/sec: 209,901 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 24 | total time: 0.42m | eta: 61.6m |
| step 00021/01500 (1.40%) | loss: 4.924438 | lrm: 0.55 | dt: 2500.80ms | tok/sec: 209,648 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 24 | total time: 0.46m | eta: 61.6m |
| step 00022/01500 (1.47%) | loss: 4.878394 | lrm: 0.57 | dt: 2500.97ms | tok/sec: 209,633 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 24 | total time: 0.50m | eta: 61.6m |
| step 00023/01500 (1.53%) | loss: 4.790773 | lrm: 0.60 | dt: 2500.81ms | tok/sec: 209,647 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 24 | total time: 0.54m | eta: 61.5m |
| step 00024/01500 (1.60%) | loss: 4.750170 | lrm: 0.62 | dt: 2502.89ms | tok/sec: 209,473 | bf16_mfu: 31.98 | epoch: 1 pq: 0 rg: 24 | total time: 0.58m | eta: 61.5m |
| step 00025/01500 (1.67%) | loss: 4.698532 | lrm: 0.65 | dt: 2500.15ms | tok/sec: 209,703 | bf16_mfu: 32.02 | epoch: 1 pq: 0 rg: 24 | total time: 0.62m | eta: 61.4m |
| step 00026/01500 (1.73%) | loss: 4.655308 | lrm: 0.68 | dt: 2497.32ms | tok/sec: 209,939 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 32 | total time: 0.67m | eta: 61.4m |
| step 00027/01500 (1.80%) | loss: 4.655427 | lrm: 0.70 | dt: 2501.75ms | tok/sec: 209,568 | bf16_mfu: 32.00 | epoch: 1 pq: 0 rg: 32 | total time: 0.71m | eta: 61.4m |
| step 00028/01500 (1.87%) | loss: 4.569183 | lrm: 0.72 | dt: 2500.24ms | tok/sec: 209,695 | bf16_mfu: 32.02 | epoch: 1 pq: 0 rg: 32 | total time: 0.75m | eta: 61.3m |
| step 00029/01500 (1.93%) | loss: 4.494453 | lrm: 0.75 | dt: 2495.72ms | tok/sec: 210,074 | bf16_mfu: 32.07 | epoch: 1 pq: 0 rg: 32 | total time: 0.79m | eta: 61.3m |
| step 00030/01500 (2.00%) | loss: 4.472533 | lrm: 0.78 | dt: 2500.82ms | tok/sec: 209,646 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 32 | total time: 0.83m | eta: 61.2m |
| step 00031/01500 (2.07%) | loss: 4.447625 | lrm: 0.80 | dt: 2502.08ms | tok/sec: 209,541 | bf16_mfu: 31.99 | epoch: 1 pq: 0 rg: 32 | total time: 0.87m | eta: 61.2m |
| step 00032/01500 (2.13%) | loss: 4.363409 | lrm: 0.82 | dt: 2502.24ms | tok/sec: 209,527 | bf16_mfu: 31.99 | epoch: 1 pq: 0 rg: 32 | total time: 0.92m | eta: 61.2m |
| step 00033/01500 (2.20%) | loss: 4.322013 | lrm: 0.85 | dt: 2499.29ms | tok/sec: 209,775 | bf16_mfu: 32.03 | epoch: 1 pq: 0 rg: 32 | total time: 0.96m | eta: 61.1m |
| step 00034/01500 (2.27%) | loss: 4.241499 | lrm: 0.88 | dt: 2501.64ms | tok/sec: 209,577 | bf16_mfu: 32.00 | epoch: 1 pq: 0 rg: 40 | total time: 1.00m | eta: 61.1m |
| step 00035/01500 (2.33%) | loss: 4.149794 | lrm: 0.90 | dt: 2498.59ms | tok/sec: 209,833 | bf16_mfu: 32.04 | epoch: 1 pq: 0 rg: 40 | total time: 1.04m | eta: 61.0m |
| step 00036/01500 (2.40%) | loss: 4.086062 | lrm: 0.93 | dt: 2500.66ms | tok/sec: 209,659 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 40 | total time: 1.08m | eta: 61.0m |
| step 00037/01500 (2.47%) | loss: 4.060564 | lrm: 0.95 | dt: 2495.85ms | tok/sec: 210,063 | bf16_mfu: 32.07 | epoch: 1 pq: 0 rg: 40 | total time: 1.12m | eta: 60.9m |
| step 00038/01500 (2.53%) | loss: 4.015661 | lrm: 0.97 | dt: 2498.20ms | tok/sec: 209,866 | bf16_mfu: 32.04 | epoch: 1 pq: 0 rg: 40 | total time: 1.17m | eta: 60.9m |
| step 00039/01500 (2.60%) | loss: 3.966935 | lrm: 1.00 | dt: 2496.40ms | tok/sec: 210,017 | bf16_mfu: 32.06 | epoch: 1 pq: 0 rg: 40 | total time: 1.21m | eta: 60.9m |
| step 00040/01500 (2.67%) | loss: 3.943025 | lrm: 1.00 | dt: 2495.80ms | tok/sec: 210,067 | bf16_mfu: 32.07 | epoch: 1 pq: 0 rg: 40 | total time: 1.25m | eta: 60.8m |
| step 00041/01500 (2.73%) | loss: 3.921660 | lrm: 1.00 | dt: 2504.60ms | tok/sec: 209,329 | bf16_mfu: 31.96 | epoch: 1 pq: 0 rg: 48 | total time: 1.29m | eta: 60.8m |
| step 00042/01500 (2.80%) | loss: 3.918710 | lrm: 1.00 | dt: 2497.40ms | tok/sec: 209,933 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 48 | total time: 1.33m | eta: 60.7m |
| step 00043/01500 (2.87%) | loss: 3.887281 | lrm: 1.00 | dt: 2502.83ms | tok/sec: 209,477 | bf16_mfu: 31.98 | epoch: 1 pq: 0 rg: 48 | total time: 1.37m | eta: 60.7m |
| step 00044/01500 (2.93%) | loss: 3.897078 | lrm: 1.00 | dt: 2499.14ms | tok/sec: 209,787 | bf16_mfu: 32.03 | epoch: 1 pq: 0 rg: 48 | total time: 1.42m | eta: 60.7m |
| step 00045/01500 (3.00%) | loss: 3.824480 | lrm: 1.00 | dt: 2500.70ms | tok/sec: 209,656 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 48 | total time: 1.46m | eta: 60.6m |
| step 00046/01500 (3.07%) | loss: 3.721530 | lrm: 1.00 | dt: 2497.85ms | tok/sec: 209,895 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 48 | total time: 1.50m | eta: 60.6m |
| step 00047/01500 (3.13%) | loss: 3.684740 | lrm: 1.00 | dt: 2500.95ms | tok/sec: 209,635 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 48 | total time: 1.54m | eta: 60.5m |
| step 00048/01500 (3.20%) | loss: 3.662864 | lrm: 1.00 | dt: 2497.91ms | tok/sec: 209,890 | bf16_mfu: 32.04 | epoch: 1 pq: 0 rg: 48 | total time: 1.58m | eta: 60.5m |
| step 00049/01500 (3.27%) | loss: 3.596936 | lrm: 1.00 | dt: 2497.21ms | tok/sec: 209,949 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 56 | total time: 1.62m | eta: 60.4m |
| step 00050/01500 (3.33%) | loss: 3.584375 | lrm: 1.00 | dt: 2501.48ms | tok/sec: 209,591 | bf16_mfu: 32.00 | epoch: 1 pq: 0 rg: 56 | total time: 1.67m | eta: 60.4m |
| step 00051/01500 (3.40%) | loss: 3.537943 | lrm: 1.00 | dt: 2501.91ms | tok/sec: 209,555 | bf16_mfu: 31.99 | epoch: 1 pq: 0 rg: 56 | total time: 1.71m | eta: 60.4m |
| step 00052/01500 (3.47%) | loss: 3.532923 | lrm: 1.00 | dt: 2494.04ms | tok/sec: 210,216 | bf16_mfu: 32.09 | epoch: 1 pq: 0 rg: 56 | total time: 1.75m | eta: 60.3m |
| step 00053/01500 (3.53%) | loss: 3.510258 | lrm: 1.00 | dt: 2500.43ms | tok/sec: 209,679 | bf16_mfu: 32.01 | epoch: 1 pq: 0 rg: 56 | total time: 1.79m | eta: 60.3m |
| step 00054/01500 (3.60%) | loss: 3.510891 | lrm: 1.00 | dt: 2497.17ms | tok/sec: 209,952 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 56 | total time: 1.83m | eta: 60.2m |
| step 00055/01500 (3.67%) | loss: 3.529868 | lrm: 1.00 | dt: 2499.21ms | tok/sec: 209,781 | bf16_mfu: 32.03 | epoch: 1 pq: 0 rg: 56 | total time: 1.87m | eta: 60.2m |
| step 00056/01500 (3.73%) | loss: 3.606472 | lrm: 1.00 | dt: 2502.32ms | tok/sec: 209,521 | bf16_mfu: 31.99 | epoch: 1 pq: 0 rg: 64 | total time: 1.92m | eta: 60.2m |
| step 00057/01500 (3.80%) | loss: 3.568959 | lrm: 1.00 | dt: 2497.56ms | tok/sec: 209,919 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 64 | total time: 1.96m | eta: 60.1m |
| step 00058/01500 (3.87%) | loss: 3.569934 | lrm: 1.00 | dt: 2496.95ms | tok/sec: 209,971 | bf16_mfu: 32.06 | epoch: 1 pq: 0 rg: 64 | total time: 2.00m | eta: 60.1m |
| step 00059/01500 (3.93%) | loss: 3.554943 | lrm: 1.00 | dt: 2496.78ms | tok/sec: 209,985 | bf16_mfu: 32.06 | epoch: 1 pq: 0 rg: 64 | total time: 2.04m | eta: 60.0m |
| step 00060/01500 (4.00%) | loss: 3.484738 | lrm: 1.00 | dt: 2496.90ms | tok/sec: 209,975 | bf16_mfu: 32.06 | epoch: 1 pq: 0 rg: 64 | total time: 2.08m | eta: 60.0m |
| step 00061/01500 (4.07%) | loss: 3.424122 | lrm: 1.00 | dt: 2496.32ms | tok/sec: 210,024 | bf16_mfu: 32.07 | epoch: 1 pq: 0 rg: 64 | total time: 2.12m | eta: 59.9m |
| step 00062/01500 (4.13%) | loss: 3.383661 | lrm: 1.00 | dt: 2502.71ms | tok/sec: 209,488 | bf16_mfu: 31.98 | epoch: 1 pq: 0 rg: 64 | total time: 2.17m | eta: 59.9m |
| step 00063/01500 (4.20%) | loss: 3.327162 | lrm: 1.00 | dt: 2498.35ms | tok/sec: 209,854 | bf16_mfu: 32.04 | epoch: 1 pq: 0 rg: 64 | total time: 2.21m | eta: 59.9m |
| step 00064/01500 (4.27%) | loss: 3.325502 | lrm: 1.00 | dt: 2501.26ms | tok/sec: 209,609 | bf16_mfu: 32.00 | epoch: 1 pq: 0 rg: 72 | total time: 2.25m | eta: 59.8m |
| step 00065/01500 (4.33%) | loss: 3.307940 | lrm: 1.00 | dt: 2499.18ms | tok/sec: 209,783 | bf16_mfu: 32.03 | epoch: 1 pq: 0 rg: 72 | total time: 2.29m | eta: 59.8m |
| step 00066/01500 (4.40%) | loss: 3.332836 | lrm: 1.00 | dt: 2500.05ms | tok/sec: 209,711 | bf16_mfu: 32.02 | epoch: 1 pq: 0 rg: 72 | total time: 2.33m | eta: 59.7m |
| step 00067/01500 (4.47%) | loss: 3.303473 | lrm: 1.00 | dt: 2494.88ms | tok/sec: 210,145 | bf16_mfu: 32.08 | epoch: 1 pq: 0 rg: 72 | total time: 2.37m | eta: 59.7m |
| step 00068/01500 (4.53%) | loss: 3.301317 | lrm: 1.00 | dt: 2499.16ms | tok/sec: 209,785 | bf16_mfu: 32.03 | epoch: 1 pq: 0 rg: 72 | total time: 2.42m | eta: 59.6m |
| step 00069/01500 (4.60%) | loss: 3.300453 | lrm: 1.00 | dt: 2497.43ms | tok/sec: 209,931 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 72 | total time: 2.46m | eta: 59.6m |
| step 00070/01500 (4.67%) | loss: 3.303051 | lrm: 1.00 | dt: 2503.44ms | tok/sec: 209,427 | bf16_mfu: 31.97 | epoch: 1 pq: 0 rg: 72 | total time: 2.50m | eta: 59.6m |
| step 00071/01500 (4.73%) | loss: 3.346494 | lrm: 1.00 | dt: 2647.30ms | tok/sec: 198,046 | bf16_mfu: 30.24 | epoch: 1 pq: 0 rg: 80 | total time: 2.54m | eta: 59.6m |
| step 00072/01500 (4.80%) | loss: 3.331093 | lrm: 1.00 | dt: 2501.62ms | tok/sec: 209,579 | bf16_mfu: 32.00 | epoch: 1 pq: 0 rg: 80 | total time: 2.58m | eta: 59.5m |
| step 00073/01500 (4.87%) | loss: 3.307411 | lrm: 1.00 | dt: 2497.78ms | tok/sec: 209,901 | bf16_mfu: 32.05 | epoch: 1 pq: 0 rg: 80 | total time: 2.63m | eta: 59.5m |
| step 00074/01500 (4.93%) | loss: 3.285279 | lrm: 1.00 | dt: 2502.72ms | tok/sec: 209,486 | bf16_mfu: 31.98 | epoch: 1 pq: 0 rg: 80 | total time: 2.67m | eta: 59.5m |
| step 00075/01500 (5.00%) | loss: 3.246838 | lrm: 1.00 | dt: 2496.13ms | tok/sec: 210,040 | bf16_mfu: 32.07 | epoch: 1 pq: 0 rg: 80 | total time: 2.71m | eta: 59.4m |
| step 00076/01500 (5.07%) | loss: 3.204915 | lrm: 1.00 | dt: 2496.94ms | tok/sec: 209,972 | bf16_mfu: 32.06 | epoch: 1 pq: 0 rg: 80 | total time: 2.75m | eta: 59.4m |
| step 00077/01500 (5.13%) | loss: 3.173970 | lrm: 1.00 | dt: 2501.91ms | tok/sec: 209,555 | bf16_mfu: 31.99 | epoch: 1 pq: 0 rg: 80 | total time: 2.79m | eta: 59.3m |
| step 00078/01500 (5.20%) | loss: 3.154224 | lrm: 1.00 | dt: 2505.64ms | tok/sec: 209,243 | bf16_mfu: 31.95 | epoch: 1 pq: 0 rg: 80 | total time: 2.84m | eta: 59.3m |
| step 00079/01500 (5.27%) | loss: 3.160177 | lrm: 1.00 | dt: 2500.09ms | tok/sec: 209,707 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 0 | total time: 2.88m | eta: 59.2m |
| step 00080/01500 (5.33%) | loss: 3.208149 | lrm: 1.00 | dt: 2505.87ms | tok/sec: 209,224 | bf16_mfu: 31.94 | epoch: 1 pq: 1 rg: 0 | total time: 2.92m | eta: 59.2m |
| step 00081/01500 (5.40%) | loss: 3.227993 | lrm: 1.00 | dt: 2497.36ms | tok/sec: 209,936 | bf16_mfu: 32.05 | epoch: 1 pq: 1 rg: 0 | total time: 2.96m | eta: 59.2m |
| step 00082/01500 (5.47%) | loss: 3.244446 | lrm: 1.00 | dt: 2503.82ms | tok/sec: 209,395 | bf16_mfu: 31.97 | epoch: 1 pq: 1 rg: 0 | total time: 3.00m | eta: 59.1m |
| step 00083/01500 (5.53%) | loss: 3.228024 | lrm: 1.00 | dt: 2497.13ms | tok/sec: 209,956 | bf16_mfu: 32.05 | epoch: 1 pq: 1 rg: 0 | total time: 3.04m | eta: 59.1m |
| step 00084/01500 (5.60%) | loss: 3.237289 | lrm: 1.00 | dt: 2495.61ms | tok/sec: 210,084 | bf16_mfu: 32.07 | epoch: 1 pq: 1 rg: 0 | total time: 3.09m | eta: 59.0m |
| step 00085/01500 (5.67%) | loss: 3.227059 | lrm: 1.00 | dt: 2499.86ms | tok/sec: 209,727 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 0 | total time: 3.13m | eta: 59.0m |
| step 00086/01500 (5.73%) | loss: 3.235147 | lrm: 1.00 | dt: 2498.19ms | tok/sec: 209,866 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 8 | total time: 3.17m | eta: 58.9m |
| step 00087/01500 (5.80%) | loss: 3.183798 | lrm: 1.00 | dt: 2497.28ms | tok/sec: 209,943 | bf16_mfu: 32.05 | epoch: 1 pq: 1 rg: 8 | total time: 3.21m | eta: 58.9m |
| step 00088/01500 (5.87%) | loss: 3.160538 | lrm: 1.00 | dt: 2500.68ms | tok/sec: 209,658 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 8 | total time: 3.25m | eta: 58.9m |
| step 00089/01500 (5.93%) | loss: 3.142993 | lrm: 1.00 | dt: 2493.69ms | tok/sec: 210,245 | bf16_mfu: 32.10 | epoch: 1 pq: 1 rg: 8 | total time: 3.29m | eta: 58.8m |
| step 00090/01500 (6.00%) | loss: 3.148296 | lrm: 1.00 | dt: 2500.75ms | tok/sec: 209,652 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 8 | total time: 3.33m | eta: 58.8m |
| step 00091/01500 (6.07%) | loss: 3.148940 | lrm: 1.00 | dt: 2497.71ms | tok/sec: 209,907 | bf16_mfu: 32.05 | epoch: 1 pq: 1 rg: 8 | total time: 3.38m | eta: 58.7m |
| step 00092/01500 (6.13%) | loss: 3.138557 | lrm: 1.00 | dt: 2495.00ms | tok/sec: 210,135 | bf16_mfu: 32.08 | epoch: 1 pq: 1 rg: 8 | total time: 3.42m | eta: 58.7m |
| step 00093/01500 (6.20%) | loss: 3.047582 | lrm: 1.00 | dt: 2499.25ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 16 | total time: 3.46m | eta: 58.6m |
| step 00094/01500 (6.27%) | loss: 3.023392 | lrm: 1.00 | dt: 2498.33ms | tok/sec: 209,855 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 16 | total time: 3.50m | eta: 58.6m |
| step 00095/01500 (6.33%) | loss: 2.988202 | lrm: 1.00 | dt: 2499.87ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 16 | total time: 3.54m | eta: 58.6m |
| step 00096/01500 (6.40%) | loss: 2.978244 | lrm: 1.00 | dt: 2502.08ms | tok/sec: 209,540 | bf16_mfu: 31.99 | epoch: 1 pq: 1 rg: 16 | total time: 3.58m | eta: 58.5m |
| step 00097/01500 (6.47%) | loss: 2.943740 | lrm: 1.00 | dt: 2498.33ms | tok/sec: 209,855 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 16 | total time: 3.63m | eta: 58.5m |
| step 00098/01500 (6.53%) | loss: 2.956456 | lrm: 1.00 | dt: 2501.04ms | tok/sec: 209,628 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 16 | total time: 3.67m | eta: 58.4m |
| step 00099/01500 (6.60%) | loss: 2.917940 | lrm: 1.00 | dt: 2500.95ms | tok/sec: 209,635 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 16 | total time: 3.71m | eta: 58.4m |
| Step 00100 | Validation bpb: 0.877004 |
| Step 00100 | ICL early: 3.2338 late: 3.3452 score: +0.1114 | Induction loss: 13.4303 acc: 0.0038 (random-half: 14.8965) |
| step 00100/01500 (6.67%) | loss: 2.915524 | lrm: 1.00 | dt: 2497.35ms | tok/sec: 209,937 | bf16_mfu: 32.05 | epoch: 1 pq: 1 rg: 16 | total time: 3.75m | eta: 58.4m |
| step 00101/01500 (6.73%) | loss: 2.898875 | lrm: 1.00 | dt: 2496.59ms | tok/sec: 210,001 | bf16_mfu: 32.06 | epoch: 1 pq: 1 rg: 24 | total time: 3.79m | eta: 58.3m |
| step 00102/01500 (6.80%) | loss: 2.925061 | lrm: 1.00 | dt: 2505.95ms | tok/sec: 209,217 | bf16_mfu: 31.94 | epoch: 1 pq: 1 rg: 24 | total time: 3.83m | eta: 58.3m |
| step 00103/01500 (6.87%) | loss: 2.932929 | lrm: 1.00 | dt: 2496.37ms | tok/sec: 210,020 | bf16_mfu: 32.06 | epoch: 1 pq: 1 rg: 24 | total time: 3.88m | eta: 58.2m |
| step 00104/01500 (6.93%) | loss: 2.906146 | lrm: 1.00 | dt: 2497.89ms | tok/sec: 209,892 | bf16_mfu: 32.05 | epoch: 1 pq: 1 rg: 24 | total time: 3.92m | eta: 58.2m |
| step 00105/01500 (7.00%) | loss: 2.907918 | lrm: 1.00 | dt: 2502.32ms | tok/sec: 209,520 | bf16_mfu: 31.99 | epoch: 1 pq: 1 rg: 24 | total time: 3.96m | eta: 58.1m |
| step 00106/01500 (7.07%) | loss: 2.924005 | lrm: 1.00 | dt: 2506.34ms | tok/sec: 209,184 | bf16_mfu: 31.94 | epoch: 1 pq: 1 rg: 24 | total time: 4.00m | eta: 58.1m |
| step 00107/01500 (7.13%) | loss: 2.904355 | lrm: 1.00 | dt: 2505.20ms | tok/sec: 209,279 | bf16_mfu: 31.95 | epoch: 1 pq: 1 rg: 24 | total time: 4.04m | eta: 58.1m |
| step 00108/01500 (7.20%) | loss: 2.956387 | lrm: 1.00 | dt: 2500.24ms | tok/sec: 209,695 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 32 | total time: 4.08m | eta: 58.0m |
| step 00109/01500 (7.27%) | loss: 2.921634 | lrm: 1.00 | dt: 2498.37ms | tok/sec: 209,852 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 32 | total time: 4.13m | eta: 58.0m |
| step 00110/01500 (7.33%) | loss: 2.921770 | lrm: 1.00 | dt: 2498.71ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 32 | total time: 4.17m | eta: 57.9m |
| step 00111/01500 (7.40%) | loss: 2.914309 | lrm: 1.00 | dt: 2499.01ms | tok/sec: 209,798 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 32 | total time: 4.21m | eta: 57.9m |
| step 00112/01500 (7.47%) | loss: 2.934415 | lrm: 1.00 | dt: 2499.80ms | tok/sec: 209,731 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 32 | total time: 4.25m | eta: 57.9m |
| step 00113/01500 (7.53%) | loss: 2.912347 | lrm: 1.00 | dt: 2499.10ms | tok/sec: 209,790 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 32 | total time: 4.29m | eta: 57.8m |
| step 00114/01500 (7.60%) | loss: 2.897675 | lrm: 1.00 | dt: 2498.62ms | tok/sec: 209,830 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 32 | total time: 4.33m | eta: 57.8m |
| step 00115/01500 (7.67%) | loss: 2.918555 | lrm: 1.00 | dt: 2496.98ms | tok/sec: 209,968 | bf16_mfu: 32.06 | epoch: 1 pq: 1 rg: 32 | total time: 4.38m | eta: 57.7m |
| step 00116/01500 (7.73%) | loss: 2.921530 | lrm: 1.00 | dt: 2502.70ms | tok/sec: 209,489 | bf16_mfu: 31.98 | epoch: 1 pq: 1 rg: 40 | total time: 4.42m | eta: 57.7m |
| step 00117/01500 (7.80%) | loss: 2.926781 | lrm: 1.00 | dt: 2501.53ms | tok/sec: 209,586 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 40 | total time: 4.46m | eta: 57.6m |
| step 00118/01500 (7.87%) | loss: 2.944181 | lrm: 1.00 | dt: 2503.88ms | tok/sec: 209,390 | bf16_mfu: 31.97 | epoch: 1 pq: 1 rg: 40 | total time: 4.50m | eta: 57.6m |
| step 00119/01500 (7.93%) | loss: 2.963816 | lrm: 1.00 | dt: 2494.88ms | tok/sec: 210,145 | bf16_mfu: 32.08 | epoch: 1 pq: 1 rg: 40 | total time: 4.54m | eta: 57.6m |
| step 00120/01500 (8.00%) | loss: 2.978842 | lrm: 1.00 | dt: 2499.18ms | tok/sec: 209,784 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 40 | total time: 4.58m | eta: 57.5m |
| step 00121/01500 (8.07%) | loss: 2.982057 | lrm: 1.00 | dt: 2500.73ms | tok/sec: 209,653 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 40 | total time: 4.63m | eta: 57.5m |
| step 00122/01500 (8.13%) | loss: 2.973189 | lrm: 1.00 | dt: 2500.84ms | tok/sec: 209,644 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 40 | total time: 4.67m | eta: 57.4m |
| step 00123/01500 (8.20%) | loss: 2.962652 | lrm: 1.00 | dt: 2499.48ms | tok/sec: 209,758 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 48 | total time: 4.71m | eta: 57.4m |
| step 00124/01500 (8.27%) | loss: 2.924283 | lrm: 1.00 | dt: 2503.58ms | tok/sec: 209,415 | bf16_mfu: 31.97 | epoch: 1 pq: 1 rg: 48 | total time: 4.75m | eta: 57.4m |
| step 00125/01500 (8.33%) | loss: 2.920081 | lrm: 1.00 | dt: 2501.28ms | tok/sec: 209,607 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 48 | total time: 4.79m | eta: 57.3m |
| step 00126/01500 (8.40%) | loss: 2.920181 | lrm: 1.00 | dt: 2500.93ms | tok/sec: 209,637 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 48 | total time: 4.83m | eta: 57.3m |
| step 00127/01500 (8.47%) | loss: 2.946619 | lrm: 1.00 | dt: 2500.60ms | tok/sec: 209,665 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 48 | total time: 4.88m | eta: 57.2m |
| step 00128/01500 (8.53%) | loss: 2.953652 | lrm: 1.00 | dt: 2501.46ms | tok/sec: 209,592 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 48 | total time: 4.92m | eta: 57.2m |
| step 00129/01500 (8.60%) | loss: 2.943835 | lrm: 1.00 | dt: 2497.84ms | tok/sec: 209,896 | bf16_mfu: 32.05 | epoch: 1 pq: 1 rg: 48 | total time: 4.96m | eta: 57.1m |
| step 00130/01500 (8.67%) | loss: 2.915199 | lrm: 1.00 | dt: 2508.27ms | tok/sec: 209,023 | bf16_mfu: 31.91 | epoch: 1 pq: 1 rg: 56 | total time: 5.00m | eta: 57.1m |
| step 00131/01500 (8.73%) | loss: 2.835930 | lrm: 1.00 | dt: 2498.84ms | tok/sec: 209,812 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 56 | total time: 5.04m | eta: 57.1m |
| step 00132/01500 (8.80%) | loss: 2.796642 | lrm: 1.00 | dt: 2500.51ms | tok/sec: 209,672 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 56 | total time: 5.08m | eta: 57.0m |
| step 00133/01500 (8.87%) | loss: 2.824124 | lrm: 1.00 | dt: 2496.34ms | tok/sec: 210,022 | bf16_mfu: 32.07 | epoch: 1 pq: 1 rg: 56 | total time: 5.13m | eta: 57.0m |
| step 00134/01500 (8.93%) | loss: 2.820834 | lrm: 1.00 | dt: 2502.11ms | tok/sec: 209,538 | bf16_mfu: 31.99 | epoch: 1 pq: 1 rg: 56 | total time: 5.17m | eta: 56.9m |
| step 00135/01500 (9.00%) | loss: 2.827388 | lrm: 1.00 | dt: 2504.91ms | tok/sec: 209,304 | bf16_mfu: 31.96 | epoch: 1 pq: 1 rg: 56 | total time: 5.21m | eta: 56.9m |
| step 00136/01500 (9.07%) | loss: 2.829499 | lrm: 1.00 | dt: 2501.80ms | tok/sec: 209,564 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 56 | total time: 5.25m | eta: 56.9m |
| step 00137/01500 (9.13%) | loss: 2.877256 | lrm: 1.00 | dt: 2499.21ms | tok/sec: 209,781 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 56 | total time: 5.29m | eta: 56.8m |
| step 00138/01500 (9.20%) | loss: 2.871995 | lrm: 1.00 | dt: 2499.98ms | tok/sec: 209,717 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 64 | total time: 5.34m | eta: 56.8m |
| step 00139/01500 (9.27%) | loss: 2.857608 | lrm: 1.00 | dt: 2498.79ms | tok/sec: 209,817 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 64 | total time: 5.38m | eta: 56.7m |
| step 00140/01500 (9.33%) | loss: 2.870178 | lrm: 1.00 | dt: 2502.12ms | tok/sec: 209,537 | bf16_mfu: 31.99 | epoch: 1 pq: 1 rg: 64 | total time: 5.42m | eta: 56.7m |
| step 00141/01500 (9.40%) | loss: 2.891083 | lrm: 1.00 | dt: 2502.49ms | tok/sec: 209,506 | bf16_mfu: 31.99 | epoch: 1 pq: 1 rg: 64 | total time: 5.46m | eta: 56.6m |
| step 00142/01500 (9.47%) | loss: 2.938610 | lrm: 1.00 | dt: 2501.14ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 64 | total time: 5.50m | eta: 56.6m |
| step 00143/01500 (9.53%) | loss: 2.966213 | lrm: 1.00 | dt: 2498.90ms | tok/sec: 209,807 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 64 | total time: 5.54m | eta: 56.6m |
| step 00144/01500 (9.60%) | loss: 2.975140 | lrm: 1.00 | dt: 2498.56ms | tok/sec: 209,835 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 64 | total time: 5.59m | eta: 56.5m |
| step 00145/01500 (9.67%) | loss: 2.964519 | lrm: 1.00 | dt: 2749.59ms | tok/sec: 190,678 | bf16_mfu: 29.11 | epoch: 1 pq: 1 rg: 72 | total time: 5.63m | eta: 56.5m |
| step 00146/01500 (9.73%) | loss: 2.953216 | lrm: 1.00 | dt: 2499.90ms | tok/sec: 209,723 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 72 | total time: 5.67m | eta: 56.5m |
| step 00147/01500 (9.80%) | loss: 2.914044 | lrm: 1.00 | dt: 2500.78ms | tok/sec: 209,649 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 72 | total time: 5.71m | eta: 56.4m |
| step 00148/01500 (9.87%) | loss: 2.920703 | lrm: 1.00 | dt: 2499.11ms | tok/sec: 209,789 | bf16_mfu: 32.03 | epoch: 1 pq: 1 rg: 72 | total time: 5.76m | eta: 56.4m |
| step 00149/01500 (9.93%) | loss: 2.899524 | lrm: 1.00 | dt: 2498.52ms | tok/sec: 209,839 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 72 | total time: 5.80m | eta: 56.3m |
| step 00150/01500 (10.00%) | loss: 2.917274 | lrm: 1.00 | dt: 2502.08ms | tok/sec: 209,540 | bf16_mfu: 31.99 | epoch: 1 pq: 1 rg: 72 | total time: 5.84m | eta: 56.3m |
| step 00151/01500 (10.07%) | loss: 2.939549 | lrm: 1.00 | dt: 2498.40ms | tok/sec: 209,849 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 72 | total time: 5.88m | eta: 56.3m |
| step 00152/01500 (10.13%) | loss: 2.917839 | lrm: 1.00 | dt: 2499.72ms | tok/sec: 209,738 | bf16_mfu: 32.02 | epoch: 1 pq: 1 rg: 80 | total time: 5.92m | eta: 56.2m |
| step 00153/01500 (10.20%) | loss: 2.856035 | lrm: 1.00 | dt: 2500.77ms | tok/sec: 209,650 | bf16_mfu: 32.01 | epoch: 1 pq: 1 rg: 80 | total time: 5.96m | eta: 56.2m |
| step 00154/01500 (10.27%) | loss: 2.805288 | lrm: 1.00 | dt: 2503.66ms | tok/sec: 209,408 | bf16_mfu: 31.97 | epoch: 1 pq: 1 rg: 80 | total time: 6.01m | eta: 56.1m |
| step 00155/01500 (10.33%) | loss: 2.785939 | lrm: 1.00 | dt: 2501.78ms | tok/sec: 209,566 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 80 | total time: 6.05m | eta: 56.1m |
| step 00156/01500 (10.40%) | loss: 2.786126 | lrm: 1.00 | dt: 2501.44ms | tok/sec: 209,594 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 80 | total time: 6.09m | eta: 56.1m |
| step 00157/01500 (10.47%) | loss: 2.739186 | lrm: 1.00 | dt: 2501.45ms | tok/sec: 209,593 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 80 | total time: 6.13m | eta: 56.0m |
| step 00158/01500 (10.53%) | loss: 2.744949 | lrm: 1.00 | dt: 2501.34ms | tok/sec: 209,603 | bf16_mfu: 32.00 | epoch: 1 pq: 1 rg: 80 | total time: 6.17m | eta: 56.0m |
| step 00159/01500 (10.60%) | loss: 2.774082 | lrm: 1.00 | dt: 2498.28ms | tok/sec: 209,859 | bf16_mfu: 32.04 | epoch: 1 pq: 1 rg: 80 | total time: 6.21m | eta: 55.9m |
| step 00160/01500 (10.67%) | loss: 2.714753 | lrm: 1.00 | dt: 2510.83ms | tok/sec: 208,810 | bf16_mfu: 31.88 | epoch: 1 pq: 2 rg: 0 | total time: 6.26m | eta: 55.9m |
| step 00161/01500 (10.73%) | loss: 2.688827 | lrm: 1.00 | dt: 2498.04ms | tok/sec: 209,879 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 0 | total time: 6.30m | eta: 55.8m |
| step 00162/01500 (10.80%) | loss: 2.656541 | lrm: 1.00 | dt: 2501.99ms | tok/sec: 209,548 | bf16_mfu: 31.99 | epoch: 1 pq: 2 rg: 0 | total time: 6.34m | eta: 55.8m |
| step 00163/01500 (10.87%) | loss: 2.671044 | lrm: 1.00 | dt: 2506.87ms | tok/sec: 209,140 | bf16_mfu: 31.93 | epoch: 1 pq: 2 rg: 0 | total time: 6.38m | eta: 55.8m |
| step 00164/01500 (10.93%) | loss: 2.712461 | lrm: 1.00 | dt: 2503.64ms | tok/sec: 209,410 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 0 | total time: 6.42m | eta: 55.7m |
| step 00165/01500 (11.00%) | loss: 2.730720 | lrm: 1.00 | dt: 2504.83ms | tok/sec: 209,310 | bf16_mfu: 31.96 | epoch: 1 pq: 2 rg: 0 | total time: 6.46m | eta: 55.7m |
| step 00166/01500 (11.07%) | loss: 2.751532 | lrm: 1.00 | dt: 2500.21ms | tok/sec: 209,697 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 0 | total time: 6.51m | eta: 55.6m |
| step 00167/01500 (11.13%) | loss: 2.798152 | lrm: 1.00 | dt: 2501.45ms | tok/sec: 209,593 | bf16_mfu: 32.00 | epoch: 1 pq: 2 rg: 8 | total time: 6.55m | eta: 55.6m |
| step 00168/01500 (11.20%) | loss: 2.865066 | lrm: 1.00 | dt: 2502.38ms | tok/sec: 209,515 | bf16_mfu: 31.99 | epoch: 1 pq: 2 rg: 8 | total time: 6.59m | eta: 55.6m |
| step 00169/01500 (11.27%) | loss: 2.860980 | lrm: 1.00 | dt: 2504.08ms | tok/sec: 209,373 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 8 | total time: 6.63m | eta: 55.5m |
| step 00170/01500 (11.33%) | loss: 2.853965 | lrm: 1.00 | dt: 2497.51ms | tok/sec: 209,924 | bf16_mfu: 32.05 | epoch: 1 pq: 2 rg: 8 | total time: 6.67m | eta: 55.5m |
| step 00171/01500 (11.40%) | loss: 2.882350 | lrm: 1.00 | dt: 2503.40ms | tok/sec: 209,430 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 8 | total time: 6.71m | eta: 55.4m |
| step 00172/01500 (11.47%) | loss: 2.881416 | lrm: 1.00 | dt: 2501.18ms | tok/sec: 209,616 | bf16_mfu: 32.00 | epoch: 1 pq: 2 rg: 8 | total time: 6.76m | eta: 55.4m |
| step 00173/01500 (11.53%) | loss: 2.835252 | lrm: 1.00 | dt: 2500.53ms | tok/sec: 209,670 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 8 | total time: 6.80m | eta: 55.3m |
| step 00174/01500 (11.60%) | loss: 2.866169 | lrm: 1.00 | dt: 2499.69ms | tok/sec: 209,741 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 8 | total time: 6.84m | eta: 55.3m |
| step 00175/01500 (11.67%) | loss: 2.884370 | lrm: 1.00 | dt: 2500.46ms | tok/sec: 209,676 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 16 | total time: 6.88m | eta: 55.3m |
| step 00176/01500 (11.73%) | loss: 2.859987 | lrm: 1.00 | dt: 2504.46ms | tok/sec: 209,341 | bf16_mfu: 31.96 | epoch: 1 pq: 2 rg: 16 | total time: 6.92m | eta: 55.2m |
| step 00177/01500 (11.80%) | loss: 2.856667 | lrm: 1.00 | dt: 2503.79ms | tok/sec: 209,397 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 16 | total time: 6.97m | eta: 55.2m |
| step 00178/01500 (11.87%) | loss: 2.856164 | lrm: 1.00 | dt: 2504.08ms | tok/sec: 209,373 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 16 | total time: 7.01m | eta: 55.1m |
| step 00179/01500 (11.93%) | loss: 2.863707 | lrm: 1.00 | dt: 2501.25ms | tok/sec: 209,610 | bf16_mfu: 32.00 | epoch: 1 pq: 2 rg: 16 | total time: 7.05m | eta: 55.1m |
| step 00180/01500 (12.00%) | loss: 2.833860 | lrm: 1.00 | dt: 2499.77ms | tok/sec: 209,734 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 16 | total time: 7.09m | eta: 55.1m |
| step 00181/01500 (12.07%) | loss: 2.809028 | lrm: 1.00 | dt: 2500.77ms | tok/sec: 209,650 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 16 | total time: 7.13m | eta: 55.0m |
| step 00182/01500 (12.13%) | loss: 2.818415 | lrm: 1.00 | dt: 2499.14ms | tok/sec: 209,787 | bf16_mfu: 32.03 | epoch: 1 pq: 2 rg: 24 | total time: 7.17m | eta: 55.0m |
| step 00183/01500 (12.20%) | loss: 2.826522 | lrm: 1.00 | dt: 2502.62ms | tok/sec: 209,495 | bf16_mfu: 31.98 | epoch: 1 pq: 2 rg: 24 | total time: 7.22m | eta: 54.9m |
| step 00184/01500 (12.27%) | loss: 2.807088 | lrm: 1.00 | dt: 2502.90ms | tok/sec: 209,472 | bf16_mfu: 31.98 | epoch: 1 pq: 2 rg: 24 | total time: 7.26m | eta: 54.9m |
| step 00185/01500 (12.33%) | loss: 2.785449 | lrm: 1.00 | dt: 2498.21ms | tok/sec: 209,865 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 24 | total time: 7.30m | eta: 54.8m |
| step 00186/01500 (12.40%) | loss: 2.777982 | lrm: 1.00 | dt: 2502.86ms | tok/sec: 209,475 | bf16_mfu: 31.98 | epoch: 1 pq: 2 rg: 24 | total time: 7.34m | eta: 54.8m |
| step 00187/01500 (12.47%) | loss: 2.793644 | lrm: 1.00 | dt: 2499.91ms | tok/sec: 209,722 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 24 | total time: 7.38m | eta: 54.8m |
| step 00188/01500 (12.53%) | loss: 2.772648 | lrm: 1.00 | dt: 2500.69ms | tok/sec: 209,656 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 24 | total time: 7.42m | eta: 54.7m |
| step 00189/01500 (12.60%) | loss: 2.773812 | lrm: 1.00 | dt: 2505.01ms | tok/sec: 209,295 | bf16_mfu: 31.95 | epoch: 1 pq: 2 rg: 32 | total time: 7.47m | eta: 54.7m |
| step 00190/01500 (12.67%) | loss: 2.774943 | lrm: 1.00 | dt: 2503.36ms | tok/sec: 209,433 | bf16_mfu: 31.98 | epoch: 1 pq: 2 rg: 32 | total time: 7.51m | eta: 54.6m |
| step 00191/01500 (12.73%) | loss: 2.763321 | lrm: 1.00 | dt: 2500.16ms | tok/sec: 209,701 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 32 | total time: 7.55m | eta: 54.6m |
| step 00192/01500 (12.80%) | loss: 2.728710 | lrm: 1.00 | dt: 2502.47ms | tok/sec: 209,508 | bf16_mfu: 31.99 | epoch: 1 pq: 2 rg: 32 | total time: 7.59m | eta: 54.6m |
| step 00193/01500 (12.87%) | loss: 2.780058 | lrm: 1.00 | dt: 2499.87ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 32 | total time: 7.63m | eta: 54.5m |
| step 00194/01500 (12.93%) | loss: 2.777232 | lrm: 1.00 | dt: 2500.12ms | tok/sec: 209,705 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 32 | total time: 7.67m | eta: 54.5m |
| step 00195/01500 (13.00%) | loss: 2.808145 | lrm: 1.00 | dt: 2500.63ms | tok/sec: 209,662 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 32 | total time: 7.72m | eta: 54.4m |
| step 00196/01500 (13.07%) | loss: 2.792706 | lrm: 1.00 | dt: 2497.46ms | tok/sec: 209,928 | bf16_mfu: 32.05 | epoch: 1 pq: 2 rg: 32 | total time: 7.76m | eta: 54.4m |
| step 00197/01500 (13.13%) | loss: 2.789577 | lrm: 1.00 | dt: 2500.91ms | tok/sec: 209,638 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 40 | total time: 7.80m | eta: 54.3m |
| step 00198/01500 (13.20%) | loss: 2.842831 | lrm: 1.00 | dt: 2499.25ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 2 rg: 40 | total time: 7.84m | eta: 54.3m |
| step 00199/01500 (13.27%) | loss: 2.854176 | lrm: 1.00 | dt: 2496.81ms | tok/sec: 209,983 | bf16_mfu: 32.06 | epoch: 1 pq: 2 rg: 40 | total time: 7.88m | eta: 54.3m |
| Step 00200 | Validation bpb: 0.809731 |
| Step 00200 | ICL early: 3.0344 late: 3.0982 score: +0.0638 | Induction loss: 12.2065 acc: 0.0161 (random-half: 14.5413) |
| step 00200/01500 (13.33%) | loss: 2.840186 | lrm: 1.00 | dt: 2499.73ms | tok/sec: 209,738 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 40 | total time: 7.92m | eta: 54.2m |
| step 00201/01500 (13.40%) | loss: 2.777129 | lrm: 1.00 | dt: 2497.93ms | tok/sec: 209,888 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 40 | total time: 7.97m | eta: 54.2m |
| step 00202/01500 (13.47%) | loss: 2.757498 | lrm: 1.00 | dt: 2500.63ms | tok/sec: 209,662 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 40 | total time: 8.01m | eta: 54.1m |
| step 00203/01500 (13.53%) | loss: 2.731559 | lrm: 1.00 | dt: 2501.13ms | tok/sec: 209,620 | bf16_mfu: 32.00 | epoch: 1 pq: 2 rg: 40 | total time: 8.05m | eta: 54.1m |
| step 00204/01500 (13.60%) | loss: 2.736987 | lrm: 1.00 | dt: 2499.95ms | tok/sec: 209,719 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 48 | total time: 8.09m | eta: 54.0m |
| step 00205/01500 (13.67%) | loss: 2.719595 | lrm: 1.00 | dt: 2495.74ms | tok/sec: 210,073 | bf16_mfu: 32.07 | epoch: 1 pq: 2 rg: 48 | total time: 8.13m | eta: 54.0m |
| step 00206/01500 (13.73%) | loss: 2.702856 | lrm: 1.00 | dt: 2503.57ms | tok/sec: 209,416 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 48 | total time: 8.17m | eta: 54.0m |
| step 00207/01500 (13.80%) | loss: 2.743100 | lrm: 1.00 | dt: 2500.33ms | tok/sec: 209,687 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 48 | total time: 8.22m | eta: 53.9m |
| step 00208/01500 (13.87%) | loss: 2.731707 | lrm: 1.00 | dt: 2498.31ms | tok/sec: 209,857 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 48 | total time: 8.26m | eta: 53.9m |
| step 00209/01500 (13.93%) | loss: 2.750448 | lrm: 1.00 | dt: 2505.58ms | tok/sec: 209,248 | bf16_mfu: 31.95 | epoch: 1 pq: 2 rg: 48 | total time: 8.30m | eta: 53.8m |
| step 00210/01500 (14.00%) | loss: 2.651502 | lrm: 1.00 | dt: 2498.41ms | tok/sec: 209,848 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 48 | total time: 8.34m | eta: 53.8m |
| step 00211/01500 (14.07%) | loss: 2.643204 | lrm: 1.00 | dt: 2507.21ms | tok/sec: 209,112 | bf16_mfu: 31.93 | epoch: 1 pq: 2 rg: 56 | total time: 8.38m | eta: 53.8m |
| step 00212/01500 (14.13%) | loss: 2.636580 | lrm: 1.00 | dt: 2502.34ms | tok/sec: 209,518 | bf16_mfu: 31.99 | epoch: 1 pq: 2 rg: 56 | total time: 8.42m | eta: 53.7m |
| step 00213/01500 (14.20%) | loss: 2.646286 | lrm: 1.00 | dt: 2501.99ms | tok/sec: 209,548 | bf16_mfu: 31.99 | epoch: 1 pq: 2 rg: 56 | total time: 8.47m | eta: 53.7m |
| step 00214/01500 (14.27%) | loss: 2.657232 | lrm: 1.00 | dt: 2501.48ms | tok/sec: 209,591 | bf16_mfu: 32.00 | epoch: 1 pq: 2 rg: 56 | total time: 8.51m | eta: 53.6m |
| step 00215/01500 (14.33%) | loss: 2.670429 | lrm: 1.00 | dt: 2499.40ms | tok/sec: 209,765 | bf16_mfu: 32.03 | epoch: 1 pq: 2 rg: 56 | total time: 8.55m | eta: 53.6m |
| step 00216/01500 (14.40%) | loss: 2.685346 | lrm: 1.00 | dt: 2502.41ms | tok/sec: 209,513 | bf16_mfu: 31.99 | epoch: 1 pq: 2 rg: 56 | total time: 8.59m | eta: 53.5m |
| step 00217/01500 (14.47%) | loss: 2.687898 | lrm: 1.00 | dt: 2503.66ms | tok/sec: 209,408 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 56 | total time: 8.63m | eta: 53.5m |
| step 00218/01500 (14.53%) | loss: 2.686027 | lrm: 1.00 | dt: 2499.84ms | tok/sec: 209,728 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 64 | total time: 8.67m | eta: 53.5m |
| step 00219/01500 (14.60%) | loss: 2.746372 | lrm: 1.00 | dt: 2905.22ms | tok/sec: 180,464 | bf16_mfu: 27.55 | epoch: 1 pq: 2 rg: 64 | total time: 8.72m | eta: 53.5m |
| step 00220/01500 (14.67%) | loss: 2.746497 | lrm: 1.00 | dt: 2499.56ms | tok/sec: 209,752 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 64 | total time: 8.76m | eta: 53.4m |
| step 00221/01500 (14.73%) | loss: 2.755573 | lrm: 1.00 | dt: 2504.56ms | tok/sec: 209,333 | bf16_mfu: 31.96 | epoch: 1 pq: 2 rg: 64 | total time: 8.81m | eta: 53.4m |
| step 00222/01500 (14.80%) | loss: 2.775914 | lrm: 1.00 | dt: 2496.10ms | tok/sec: 210,043 | bf16_mfu: 32.07 | epoch: 1 pq: 2 rg: 64 | total time: 8.85m | eta: 53.3m |
| step 00223/01500 (14.87%) | loss: 2.712128 | lrm: 1.00 | dt: 2497.29ms | tok/sec: 209,942 | bf16_mfu: 32.05 | epoch: 1 pq: 2 rg: 64 | total time: 8.89m | eta: 53.3m |
| step 00224/01500 (14.93%) | loss: 2.699746 | lrm: 1.00 | dt: 2504.78ms | tok/sec: 209,314 | bf16_mfu: 31.96 | epoch: 1 pq: 2 rg: 64 | total time: 8.93m | eta: 53.3m |
| step 00225/01500 (15.00%) | loss: 2.695812 | lrm: 1.00 | dt: 2500.45ms | tok/sec: 209,677 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 64 | total time: 8.97m | eta: 53.2m |
| step 00226/01500 (15.07%) | loss: 2.651124 | lrm: 1.00 | dt: 2501.00ms | tok/sec: 209,631 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 72 | total time: 9.01m | eta: 53.2m |
| step 00227/01500 (15.13%) | loss: 2.630231 | lrm: 1.00 | dt: 2496.61ms | tok/sec: 210,000 | bf16_mfu: 32.06 | epoch: 1 pq: 2 rg: 72 | total time: 9.06m | eta: 53.1m |
| step 00228/01500 (15.20%) | loss: 2.631078 | lrm: 1.00 | dt: 2503.45ms | tok/sec: 209,425 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 72 | total time: 9.10m | eta: 53.1m |
| step 00229/01500 (15.27%) | loss: 2.615961 | lrm: 1.00 | dt: 2503.66ms | tok/sec: 209,408 | bf16_mfu: 31.97 | epoch: 1 pq: 2 rg: 72 | total time: 9.14m | eta: 53.0m |
| step 00230/01500 (15.33%) | loss: 2.634345 | lrm: 1.00 | dt: 2494.33ms | tok/sec: 210,191 | bf16_mfu: 32.09 | epoch: 1 pq: 2 rg: 72 | total time: 9.18m | eta: 53.0m |
| step 00231/01500 (15.40%) | loss: 2.658159 | lrm: 1.00 | dt: 2498.41ms | tok/sec: 209,848 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 72 | total time: 9.22m | eta: 53.0m |
| step 00232/01500 (15.47%) | loss: 2.665212 | lrm: 1.00 | dt: 2499.67ms | tok/sec: 209,743 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 72 | total time: 9.26m | eta: 52.9m |
| step 00233/01500 (15.53%) | loss: 2.702525 | lrm: 1.00 | dt: 2499.89ms | tok/sec: 209,724 | bf16_mfu: 32.02 | epoch: 1 pq: 2 rg: 72 | total time: 9.31m | eta: 52.9m |
| step 00234/01500 (15.60%) | loss: 2.688055 | lrm: 1.00 | dt: 2498.00ms | tok/sec: 209,883 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 80 | total time: 9.35m | eta: 52.8m |
| step 00235/01500 (15.67%) | loss: 2.677379 | lrm: 1.00 | dt: 2500.49ms | tok/sec: 209,674 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 80 | total time: 9.39m | eta: 52.8m |
| step 00236/01500 (15.73%) | loss: 2.665520 | lrm: 1.00 | dt: 2499.24ms | tok/sec: 209,778 | bf16_mfu: 32.03 | epoch: 1 pq: 2 rg: 80 | total time: 9.43m | eta: 52.7m |
| step 00237/01500 (15.80%) | loss: 2.635362 | lrm: 1.00 | dt: 2498.00ms | tok/sec: 209,883 | bf16_mfu: 32.04 | epoch: 1 pq: 2 rg: 80 | total time: 9.47m | eta: 52.7m |
| step 00238/01500 (15.87%) | loss: 2.681989 | lrm: 1.00 | dt: 2500.71ms | tok/sec: 209,656 | bf16_mfu: 32.01 | epoch: 1 pq: 2 rg: 80 | total time: 9.51m | eta: 52.7m |
| step 00239/01500 (15.93%) | loss: 2.665659 | lrm: 1.00 | dt: 2497.48ms | tok/sec: 209,926 | bf16_mfu: 32.05 | epoch: 1 pq: 2 rg: 80 | total time: 9.56m | eta: 52.6m |
| step 00240/01500 (16.00%) | loss: 2.653508 | lrm: 1.00 | dt: 2498.78ms | tok/sec: 209,817 | bf16_mfu: 32.03 | epoch: 1 pq: 2 rg: 80 | total time: 9.60m | eta: 52.6m |
| step 00241/01500 (16.07%) | loss: 2.631025 | lrm: 1.00 | dt: 2498.98ms | tok/sec: 209,800 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 0 | total time: 9.64m | eta: 52.5m |
| step 00242/01500 (16.13%) | loss: 2.629892 | lrm: 1.00 | dt: 2498.83ms | tok/sec: 209,813 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 0 | total time: 9.68m | eta: 52.5m |
| step 00243/01500 (16.20%) | loss: 2.642400 | lrm: 1.00 | dt: 2498.98ms | tok/sec: 209,800 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 0 | total time: 9.72m | eta: 52.5m |
| step 00244/01500 (16.27%) | loss: 2.664730 | lrm: 1.00 | dt: 2497.75ms | tok/sec: 209,903 | bf16_mfu: 32.05 | epoch: 1 pq: 3 rg: 0 | total time: 9.76m | eta: 52.4m |
| step 00245/01500 (16.33%) | loss: 2.677625 | lrm: 1.00 | dt: 2496.70ms | tok/sec: 209,992 | bf16_mfu: 32.06 | epoch: 1 pq: 3 rg: 0 | total time: 9.81m | eta: 52.4m |
| step 00246/01500 (16.40%) | loss: 2.619270 | lrm: 1.00 | dt: 2499.02ms | tok/sec: 209,797 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 0 | total time: 9.85m | eta: 52.3m |
| step 00247/01500 (16.47%) | loss: 2.646338 | lrm: 1.00 | dt: 2500.25ms | tok/sec: 209,693 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 0 | total time: 9.89m | eta: 52.3m |
| step 00248/01500 (16.53%) | loss: 2.684204 | lrm: 1.00 | dt: 2505.23ms | tok/sec: 209,277 | bf16_mfu: 31.95 | epoch: 1 pq: 3 rg: 0 | total time: 9.93m | eta: 52.2m |
| step 00249/01500 (16.60%) | loss: 2.700333 | lrm: 1.00 | dt: 2505.08ms | tok/sec: 209,290 | bf16_mfu: 31.95 | epoch: 1 pq: 3 rg: 8 | total time: 9.97m | eta: 52.2m |
| <|bos|>The capital of France is Paris. It is the capital of the Île-de-France region. It |
| <|bos|>The chemical symbol of gold is Au, and the atomic number of gold is 10. The atomic number of |
| <|bos|>If yesterday was Friday, then tomorrow will be Saturday. If today is Saturday, tomorrow will be Sunday. If today is Sunday |
| <|bos|>The opposite of hot is cold. Hot is a temperature that is above the boiling point of water. Cold |
| <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, |
| <|bos|>My favorite color is blue. I like blue because it makes me feel happy. I like blue because |
| <|bos|>If 5*x + 3 = 13, then x is 3. If x = 3, then x = 3. If |
| 2026-08-10 21:35:19,358 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank4.pt |
| 2026-08-10 21:35:37,688 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank3.pt |
| 2026-08-10 21:35:39,857 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank5.pt |
| 2026-08-10 21:35:50,438 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank1.pt |
| 2026-08-10 21:35:58,952 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank7.pt |
| 2026-08-10 21:35:59,184 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank2.pt |
| 2026-08-10 21:35:59,240 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank6.pt |
| 2026-08-10 21:37:00,283 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved model parameters to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/model_000250.pt |
| 2026-08-10 21:37:00,285 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved metadata to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/meta_000250.json |
| 2026-08-10 21:37:07,539 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000250_rank0.pt |
| step 00250/01500 (16.67%) | loss: 2.701050 | lrm: 1.00 | dt: 2475.05ms | tok/sec: 211,828 | bf16_mfu: 32.34 | epoch: 1 pq: 3 rg: 8 | total time: 10.01m | eta: 52.2m |
| step 00251/01500 (16.73%) | loss: 2.694390 | lrm: 1.00 | dt: 2482.94ms | tok/sec: 211,156 | bf16_mfu: 32.24 | epoch: 1 pq: 3 rg: 8 | total time: 10.06m | eta: 52.1m |
| step 00252/01500 (16.80%) | loss: 2.676868 | lrm: 1.00 | dt: 2497.56ms | tok/sec: 209,920 | bf16_mfu: 32.05 | epoch: 1 pq: 3 rg: 8 | total time: 10.10m | eta: 52.1m |
| step 00253/01500 (16.87%) | loss: 2.672160 | lrm: 1.00 | dt: 2491.61ms | tok/sec: 210,421 | bf16_mfu: 32.13 | epoch: 1 pq: 3 rg: 8 | total time: 10.14m | eta: 52.0m |
| step 00254/01500 (16.93%) | loss: 2.669649 | lrm: 1.00 | dt: 2496.71ms | tok/sec: 209,991 | bf16_mfu: 32.06 | epoch: 1 pq: 3 rg: 8 | total time: 10.18m | eta: 52.0m |
| step 00255/01500 (17.00%) | loss: 2.682889 | lrm: 1.00 | dt: 2498.70ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 8 | total time: 10.22m | eta: 51.9m |
| step 00256/01500 (17.07%) | loss: 2.690229 | lrm: 1.00 | dt: 2500.71ms | tok/sec: 209,655 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 8 | total time: 10.26m | eta: 51.9m |
| step 00257/01500 (17.13%) | loss: 2.713785 | lrm: 1.00 | dt: 2500.17ms | tok/sec: 209,701 | bf16_mfu: 32.02 | epoch: 1 pq: 3 rg: 16 | total time: 10.30m | eta: 51.9m |
| step 00258/01500 (17.20%) | loss: 2.744359 | lrm: 1.00 | dt: 2495.70ms | tok/sec: 210,076 | bf16_mfu: 32.07 | epoch: 1 pq: 3 rg: 16 | total time: 10.35m | eta: 51.8m |
| step 00259/01500 (17.27%) | loss: 2.716034 | lrm: 1.00 | dt: 2494.01ms | tok/sec: 210,219 | bf16_mfu: 32.10 | epoch: 1 pq: 3 rg: 16 | total time: 10.39m | eta: 51.8m |
| step 00260/01500 (17.33%) | loss: 2.699451 | lrm: 1.00 | dt: 2492.83ms | tok/sec: 210,318 | bf16_mfu: 32.11 | epoch: 1 pq: 3 rg: 16 | total time: 10.43m | eta: 51.7m |
| step 00261/01500 (17.40%) | loss: 2.692689 | lrm: 1.00 | dt: 2498.95ms | tok/sec: 209,803 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 16 | total time: 10.47m | eta: 51.7m |
| step 00262/01500 (17.47%) | loss: 2.710181 | lrm: 1.00 | dt: 2498.71ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 16 | total time: 10.51m | eta: 51.6m |
| step 00263/01500 (17.53%) | loss: 2.706901 | lrm: 1.00 | dt: 2495.49ms | tok/sec: 210,093 | bf16_mfu: 32.08 | epoch: 1 pq: 3 rg: 16 | total time: 10.55m | eta: 51.6m |
| step 00264/01500 (17.60%) | loss: 2.721221 | lrm: 1.00 | dt: 2496.82ms | tok/sec: 209,981 | bf16_mfu: 32.06 | epoch: 1 pq: 3 rg: 16 | total time: 10.60m | eta: 51.6m |
| step 00265/01500 (17.67%) | loss: 2.756175 | lrm: 1.00 | dt: 2499.27ms | tok/sec: 209,776 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 24 | total time: 10.64m | eta: 51.5m |
| step 00266/01500 (17.73%) | loss: 2.752914 | lrm: 1.00 | dt: 2495.08ms | tok/sec: 210,129 | bf16_mfu: 32.08 | epoch: 1 pq: 3 rg: 24 | total time: 10.68m | eta: 51.5m |
| step 00267/01500 (17.80%) | loss: 2.741282 | lrm: 1.00 | dt: 2497.50ms | tok/sec: 209,925 | bf16_mfu: 32.05 | epoch: 1 pq: 3 rg: 24 | total time: 10.72m | eta: 51.4m |
| step 00268/01500 (17.87%) | loss: 2.730078 | lrm: 1.00 | dt: 2498.35ms | tok/sec: 209,853 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 24 | total time: 10.76m | eta: 51.4m |
| step 00269/01500 (17.93%) | loss: 2.656281 | lrm: 1.00 | dt: 2490.78ms | tok/sec: 210,491 | bf16_mfu: 32.14 | epoch: 1 pq: 3 rg: 24 | total time: 10.80m | eta: 51.4m |
| step 00270/01500 (18.00%) | loss: 2.609942 | lrm: 1.00 | dt: 2491.76ms | tok/sec: 210,408 | bf16_mfu: 32.12 | epoch: 1 pq: 3 rg: 24 | total time: 10.85m | eta: 51.3m |
| step 00271/01500 (18.07%) | loss: 2.616283 | lrm: 1.00 | dt: 2497.39ms | tok/sec: 209,934 | bf16_mfu: 32.05 | epoch: 1 pq: 3 rg: 24 | total time: 10.89m | eta: 51.3m |
| step 00272/01500 (18.13%) | loss: 2.665154 | lrm: 1.00 | dt: 2495.60ms | tok/sec: 210,084 | bf16_mfu: 32.07 | epoch: 1 pq: 3 rg: 24 | total time: 10.93m | eta: 51.2m |
| step 00273/01500 (18.20%) | loss: 2.688141 | lrm: 1.00 | dt: 2504.76ms | tok/sec: 209,316 | bf16_mfu: 31.96 | epoch: 1 pq: 3 rg: 32 | total time: 10.97m | eta: 51.2m |
| step 00274/01500 (18.27%) | loss: 2.728167 | lrm: 1.00 | dt: 2500.42ms | tok/sec: 209,679 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 32 | total time: 11.01m | eta: 51.1m |
| step 00275/01500 (18.33%) | loss: 2.761949 | lrm: 1.00 | dt: 2499.35ms | tok/sec: 209,770 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 32 | total time: 11.05m | eta: 51.1m |
| step 00276/01500 (18.40%) | loss: 2.759468 | lrm: 1.00 | dt: 2501.02ms | tok/sec: 209,629 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 32 | total time: 11.10m | eta: 51.1m |
| step 00277/01500 (18.47%) | loss: 2.734210 | lrm: 1.00 | dt: 2505.11ms | tok/sec: 209,287 | bf16_mfu: 31.95 | epoch: 1 pq: 3 rg: 32 | total time: 11.14m | eta: 51.0m |
| step 00278/01500 (18.53%) | loss: 2.710704 | lrm: 1.00 | dt: 2498.71ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 32 | total time: 11.18m | eta: 51.0m |
| step 00279/01500 (18.60%) | loss: 2.695947 | lrm: 1.00 | dt: 2500.23ms | tok/sec: 209,695 | bf16_mfu: 32.02 | epoch: 1 pq: 3 rg: 32 | total time: 11.22m | eta: 50.9m |
| step 00280/01500 (18.67%) | loss: 2.661022 | lrm: 1.00 | dt: 2502.46ms | tok/sec: 209,509 | bf16_mfu: 31.99 | epoch: 1 pq: 3 rg: 32 | total time: 11.26m | eta: 50.9m |
| step 00281/01500 (18.73%) | loss: 2.661178 | lrm: 1.00 | dt: 2500.31ms | tok/sec: 209,689 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 40 | total time: 11.30m | eta: 50.8m |
| step 00282/01500 (18.80%) | loss: 2.656587 | lrm: 1.00 | dt: 2496.01ms | tok/sec: 210,050 | bf16_mfu: 32.07 | epoch: 1 pq: 3 rg: 40 | total time: 11.35m | eta: 50.8m |
| step 00283/01500 (18.87%) | loss: 2.667393 | lrm: 1.00 | dt: 2497.85ms | tok/sec: 209,895 | bf16_mfu: 32.05 | epoch: 1 pq: 3 rg: 40 | total time: 11.39m | eta: 50.8m |
| step 00284/01500 (18.93%) | loss: 2.645016 | lrm: 1.00 | dt: 2499.81ms | tok/sec: 209,731 | bf16_mfu: 32.02 | epoch: 1 pq: 3 rg: 40 | total time: 11.43m | eta: 50.7m |
| step 00285/01500 (19.00%) | loss: 2.658358 | lrm: 1.00 | dt: 2497.54ms | tok/sec: 209,921 | bf16_mfu: 32.05 | epoch: 1 pq: 3 rg: 40 | total time: 11.47m | eta: 50.7m |
| step 00286/01500 (19.07%) | loss: 2.648874 | lrm: 1.00 | dt: 2499.08ms | tok/sec: 209,792 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 40 | total time: 11.51m | eta: 50.6m |
| step 00287/01500 (19.13%) | loss: 2.643574 | lrm: 1.00 | dt: 2501.42ms | tok/sec: 209,595 | bf16_mfu: 32.00 | epoch: 1 pq: 3 rg: 40 | total time: 11.55m | eta: 50.6m |
| step 00288/01500 (19.20%) | loss: 2.632357 | lrm: 1.00 | dt: 2498.17ms | tok/sec: 209,868 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 40 | total time: 11.60m | eta: 50.6m |
| step 00289/01500 (19.27%) | loss: 2.605199 | lrm: 1.00 | dt: 2500.52ms | tok/sec: 209,671 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 48 | total time: 11.64m | eta: 50.5m |
| step 00290/01500 (19.33%) | loss: 2.599687 | lrm: 1.00 | dt: 2500.76ms | tok/sec: 209,651 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 48 | total time: 11.68m | eta: 50.5m |
| step 00291/01500 (19.40%) | loss: 2.565944 | lrm: 1.00 | dt: 2498.25ms | tok/sec: 209,862 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 48 | total time: 11.72m | eta: 50.4m |
| step 00292/01500 (19.47%) | loss: 2.578022 | lrm: 1.00 | dt: 2498.69ms | tok/sec: 209,825 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 48 | total time: 11.76m | eta: 50.4m |
| step 00293/01500 (19.53%) | loss: 2.594464 | lrm: 1.00 | dt: 2498.45ms | tok/sec: 209,845 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 48 | total time: 11.80m | eta: 50.3m |
| step 00294/01500 (19.60%) | loss: 2.617919 | lrm: 1.00 | dt: 2497.29ms | tok/sec: 209,943 | bf16_mfu: 32.05 | epoch: 1 pq: 3 rg: 48 | total time: 11.85m | eta: 50.3m |
| step 00295/01500 (19.67%) | loss: 2.630398 | lrm: 1.00 | dt: 2500.08ms | tok/sec: 209,708 | bf16_mfu: 32.02 | epoch: 1 pq: 3 rg: 48 | total time: 11.89m | eta: 50.3m |
| step 00296/01500 (19.73%) | loss: 2.588017 | lrm: 1.00 | dt: 2498.17ms | tok/sec: 209,868 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 48 | total time: 11.93m | eta: 50.2m |
| step 00297/01500 (19.80%) | loss: 2.613355 | lrm: 1.00 | dt: 2498.17ms | tok/sec: 209,868 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 56 | total time: 11.97m | eta: 50.2m |
| step 00298/01500 (19.87%) | loss: 2.601538 | lrm: 1.00 | dt: 2764.71ms | tok/sec: 189,635 | bf16_mfu: 28.95 | epoch: 1 pq: 3 rg: 56 | total time: 12.02m | eta: 50.2m |
| step 00299/01500 (19.93%) | loss: 2.582510 | lrm: 1.00 | dt: 2498.32ms | tok/sec: 209,855 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 56 | total time: 12.06m | eta: 50.1m |
| Step 00300 | Validation bpb: 0.784707 |
| Step 00300 | ICL early: 2.9330 late: 3.0051 score: +0.0720 | Induction loss: 11.5097 acc: 0.0289 (random-half: 14.3222) |
| step 00300/01500 (20.00%) | loss: 2.581956 | lrm: 1.00 | dt: 2493.35ms | tok/sec: 210,274 | bf16_mfu: 32.10 | epoch: 1 pq: 3 rg: 56 | total time: 12.10m | eta: 50.1m |
| step 00301/01500 (20.07%) | loss: 2.570770 | lrm: 1.00 | dt: 2504.38ms | tok/sec: 209,348 | bf16_mfu: 31.96 | epoch: 1 pq: 3 rg: 56 | total time: 12.14m | eta: 50.0m |
| step 00302/01500 (20.13%) | loss: 2.582061 | lrm: 1.00 | dt: 2504.64ms | tok/sec: 209,326 | bf16_mfu: 31.96 | epoch: 1 pq: 3 rg: 56 | total time: 12.18m | eta: 50.0m |
| step 00303/01500 (20.20%) | loss: 2.599564 | lrm: 1.00 | dt: 2496.45ms | tok/sec: 210,013 | bf16_mfu: 32.06 | epoch: 1 pq: 3 rg: 56 | total time: 12.22m | eta: 49.9m |
| step 00304/01500 (20.27%) | loss: 2.617667 | lrm: 1.00 | dt: 2502.67ms | tok/sec: 209,491 | bf16_mfu: 31.98 | epoch: 1 pq: 3 rg: 56 | total time: 12.27m | eta: 49.9m |
| step 00305/01500 (20.33%) | loss: 2.607164 | lrm: 1.00 | dt: 2499.88ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 3 rg: 64 | total time: 12.31m | eta: 49.9m |
| step 00306/01500 (20.40%) | loss: 2.613627 | lrm: 1.00 | dt: 2509.25ms | tok/sec: 208,942 | bf16_mfu: 31.90 | epoch: 1 pq: 3 rg: 64 | total time: 12.35m | eta: 49.8m |
| step 00307/01500 (20.47%) | loss: 2.584020 | lrm: 1.00 | dt: 2498.86ms | tok/sec: 209,810 | bf16_mfu: 32.03 | epoch: 1 pq: 3 rg: 64 | total time: 12.39m | eta: 49.8m |
| step 00308/01500 (20.53%) | loss: 2.639379 | lrm: 1.00 | dt: 2504.32ms | tok/sec: 209,353 | bf16_mfu: 31.96 | epoch: 1 pq: 3 rg: 64 | total time: 12.43m | eta: 49.7m |
| step 00309/01500 (20.60%) | loss: 2.666293 | lrm: 1.00 | dt: 2499.62ms | tok/sec: 209,746 | bf16_mfu: 32.02 | epoch: 1 pq: 3 rg: 64 | total time: 12.47m | eta: 49.7m |
| step 00310/01500 (20.67%) | loss: 2.689088 | lrm: 1.00 | dt: 2504.00ms | tok/sec: 209,379 | bf16_mfu: 31.97 | epoch: 1 pq: 3 rg: 64 | total time: 12.52m | eta: 49.6m |
| step 00311/01500 (20.73%) | loss: 2.672490 | lrm: 1.00 | dt: 2504.04ms | tok/sec: 209,377 | bf16_mfu: 31.97 | epoch: 1 pq: 3 rg: 64 | total time: 12.56m | eta: 49.6m |
| step 00312/01500 (20.80%) | loss: 2.624428 | lrm: 1.00 | dt: 2499.85ms | tok/sec: 209,727 | bf16_mfu: 32.02 | epoch: 1 pq: 3 rg: 64 | total time: 12.60m | eta: 49.6m |
| step 00313/01500 (20.87%) | loss: 2.686193 | lrm: 1.00 | dt: 2501.23ms | tok/sec: 209,612 | bf16_mfu: 32.00 | epoch: 1 pq: 3 rg: 64 | total time: 12.64m | eta: 49.5m |
| step 00314/01500 (20.93%) | loss: 2.658987 | lrm: 1.00 | dt: 2503.55ms | tok/sec: 209,418 | bf16_mfu: 31.97 | epoch: 1 pq: 3 rg: 72 | total time: 12.68m | eta: 49.5m |
| step 00315/01500 (21.00%) | loss: 2.668251 | lrm: 1.00 | dt: 2496.38ms | tok/sec: 210,019 | bf16_mfu: 32.06 | epoch: 1 pq: 3 rg: 72 | total time: 12.73m | eta: 49.4m |
| step 00316/01500 (21.07%) | loss: 2.661666 | lrm: 1.00 | dt: 2503.78ms | tok/sec: 209,398 | bf16_mfu: 31.97 | epoch: 1 pq: 3 rg: 72 | total time: 12.77m | eta: 49.4m |
| step 00317/01500 (21.13%) | loss: 2.642533 | lrm: 1.00 | dt: 2502.36ms | tok/sec: 209,517 | bf16_mfu: 31.99 | epoch: 1 pq: 3 rg: 72 | total time: 12.81m | eta: 49.4m |
| step 00318/01500 (21.20%) | loss: 2.658486 | lrm: 1.00 | dt: 2501.67ms | tok/sec: 209,575 | bf16_mfu: 32.00 | epoch: 1 pq: 3 rg: 72 | total time: 12.85m | eta: 49.3m |
| step 00319/01500 (21.27%) | loss: 2.665368 | lrm: 1.00 | dt: 2501.86ms | tok/sec: 209,558 | bf16_mfu: 31.99 | epoch: 1 pq: 3 rg: 72 | total time: 12.89m | eta: 49.3m |
| step 00320/01500 (21.33%) | loss: 2.652778 | lrm: 1.00 | dt: 2503.00ms | tok/sec: 209,463 | bf16_mfu: 31.98 | epoch: 1 pq: 3 rg: 72 | total time: 12.93m | eta: 49.2m |
| step 00321/01500 (21.40%) | loss: 2.658043 | lrm: 1.00 | dt: 2498.38ms | tok/sec: 209,851 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 72 | total time: 12.98m | eta: 49.2m |
| step 00322/01500 (21.47%) | loss: 2.681621 | lrm: 1.00 | dt: 2501.26ms | tok/sec: 209,609 | bf16_mfu: 32.00 | epoch: 1 pq: 3 rg: 80 | total time: 13.02m | eta: 49.1m |
| step 00323/01500 (21.53%) | loss: 2.668596 | lrm: 1.00 | dt: 2502.51ms | tok/sec: 209,504 | bf16_mfu: 31.99 | epoch: 1 pq: 3 rg: 80 | total time: 13.06m | eta: 49.1m |
| step 00324/01500 (21.60%) | loss: 2.632350 | lrm: 1.00 | dt: 2501.00ms | tok/sec: 209,631 | bf16_mfu: 32.01 | epoch: 1 pq: 3 rg: 80 | total time: 13.10m | eta: 49.1m |
| step 00325/01500 (21.67%) | loss: 2.624731 | lrm: 1.00 | dt: 2504.15ms | tok/sec: 209,367 | bf16_mfu: 31.97 | epoch: 1 pq: 3 rg: 80 | total time: 13.14m | eta: 49.0m |
| step 00326/01500 (21.73%) | loss: 2.591722 | lrm: 1.00 | dt: 2503.18ms | tok/sec: 209,448 | bf16_mfu: 31.98 | epoch: 1 pq: 3 rg: 80 | total time: 13.18m | eta: 49.0m |
| step 00327/01500 (21.80%) | loss: 2.547419 | lrm: 1.00 | dt: 2505.53ms | tok/sec: 209,252 | bf16_mfu: 31.95 | epoch: 1 pq: 3 rg: 80 | total time: 13.23m | eta: 48.9m |
| step 00328/01500 (21.87%) | loss: 2.578428 | lrm: 1.00 | dt: 2498.67ms | tok/sec: 209,827 | bf16_mfu: 32.04 | epoch: 1 pq: 3 rg: 80 | total time: 13.27m | eta: 48.9m |
| step 00329/01500 (21.93%) | loss: 2.560731 | lrm: 1.00 | dt: 2500.75ms | tok/sec: 209,652 | bf16_mfu: 32.01 | epoch: 1 pq: 4 rg: 0 | total time: 13.31m | eta: 48.9m |
| step 00330/01500 (22.00%) | loss: 2.585317 | lrm: 1.00 | dt: 2497.66ms | tok/sec: 209,912 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 0 | total time: 13.35m | eta: 48.8m |
| step 00331/01500 (22.07%) | loss: 2.612563 | lrm: 1.00 | dt: 2501.03ms | tok/sec: 209,628 | bf16_mfu: 32.00 | epoch: 1 pq: 4 rg: 0 | total time: 13.39m | eta: 48.8m |
| step 00332/01500 (22.13%) | loss: 2.622800 | lrm: 1.00 | dt: 2499.40ms | tok/sec: 209,765 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 0 | total time: 13.43m | eta: 48.7m |
| step 00333/01500 (22.20%) | loss: 2.609684 | lrm: 1.00 | dt: 2503.44ms | tok/sec: 209,426 | bf16_mfu: 31.97 | epoch: 1 pq: 4 rg: 0 | total time: 13.48m | eta: 48.7m |
| step 00334/01500 (22.27%) | loss: 2.639728 | lrm: 1.00 | dt: 2501.09ms | tok/sec: 209,624 | bf16_mfu: 32.00 | epoch: 1 pq: 4 rg: 0 | total time: 13.52m | eta: 48.6m |
| step 00335/01500 (22.33%) | loss: 2.655841 | lrm: 1.00 | dt: 2496.66ms | tok/sec: 209,995 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 0 | total time: 13.56m | eta: 48.6m |
| step 00336/01500 (22.40%) | loss: 2.605947 | lrm: 1.00 | dt: 2501.92ms | tok/sec: 209,554 | bf16_mfu: 31.99 | epoch: 1 pq: 4 rg: 0 | total time: 13.60m | eta: 48.6m |
| step 00337/01500 (22.47%) | loss: 2.620740 | lrm: 1.00 | dt: 2503.39ms | tok/sec: 209,431 | bf16_mfu: 31.97 | epoch: 1 pq: 4 rg: 0 | total time: 13.64m | eta: 48.5m |
| step 00338/01500 (22.53%) | loss: 2.611902 | lrm: 1.00 | dt: 2496.71ms | tok/sec: 209,991 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 8 | total time: 13.68m | eta: 48.5m |
| step 00339/01500 (22.60%) | loss: 2.629633 | lrm: 1.00 | dt: 2504.56ms | tok/sec: 209,333 | bf16_mfu: 31.96 | epoch: 1 pq: 4 rg: 8 | total time: 13.73m | eta: 48.4m |
| step 00340/01500 (22.67%) | loss: 2.595688 | lrm: 1.00 | dt: 2498.92ms | tok/sec: 209,805 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 8 | total time: 13.77m | eta: 48.4m |
| step 00341/01500 (22.73%) | loss: 2.599709 | lrm: 1.00 | dt: 2501.73ms | tok/sec: 209,570 | bf16_mfu: 32.00 | epoch: 1 pq: 4 rg: 8 | total time: 13.81m | eta: 48.4m |
| step 00342/01500 (22.80%) | loss: 2.642176 | lrm: 1.00 | dt: 2498.07ms | tok/sec: 209,877 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 8 | total time: 13.85m | eta: 48.3m |
| step 00343/01500 (22.87%) | loss: 2.618690 | lrm: 1.00 | dt: 2497.95ms | tok/sec: 209,886 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 8 | total time: 13.89m | eta: 48.3m |
| step 00344/01500 (22.93%) | loss: 2.640408 | lrm: 1.00 | dt: 2500.23ms | tok/sec: 209,696 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 8 | total time: 13.93m | eta: 48.2m |
| step 00345/01500 (23.00%) | loss: 2.651068 | lrm: 1.00 | dt: 2499.25ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 8 | total time: 13.98m | eta: 48.2m |
| step 00346/01500 (23.07%) | loss: 2.640402 | lrm: 1.00 | dt: 2496.41ms | tok/sec: 210,016 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 16 | total time: 14.02m | eta: 48.1m |
| step 00347/01500 (23.13%) | loss: 2.640979 | lrm: 1.00 | dt: 2496.47ms | tok/sec: 210,011 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 16 | total time: 14.06m | eta: 48.1m |
| step 00348/01500 (23.20%) | loss: 2.625766 | lrm: 1.00 | dt: 2505.88ms | tok/sec: 209,223 | bf16_mfu: 31.94 | epoch: 1 pq: 4 rg: 16 | total time: 14.10m | eta: 48.1m |
| step 00349/01500 (23.27%) | loss: 2.638305 | lrm: 1.00 | dt: 2504.49ms | tok/sec: 209,339 | bf16_mfu: 31.96 | epoch: 1 pq: 4 rg: 16 | total time: 14.14m | eta: 48.0m |
| step 00350/01500 (23.33%) | loss: 2.631730 | lrm: 1.00 | dt: 2502.65ms | tok/sec: 209,492 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 16 | total time: 14.18m | eta: 48.0m |
| step 00351/01500 (23.40%) | loss: 2.678804 | lrm: 1.00 | dt: 2499.11ms | tok/sec: 209,789 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 16 | total time: 14.23m | eta: 47.9m |
| step 00352/01500 (23.47%) | loss: 2.662850 | lrm: 1.00 | dt: 2499.22ms | tok/sec: 209,780 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 16 | total time: 14.27m | eta: 47.9m |
| step 00353/01500 (23.53%) | loss: 2.642324 | lrm: 1.00 | dt: 2500.14ms | tok/sec: 209,703 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 16 | total time: 14.31m | eta: 47.8m |
| step 00354/01500 (23.60%) | loss: 2.685310 | lrm: 1.00 | dt: 2504.34ms | tok/sec: 209,351 | bf16_mfu: 31.96 | epoch: 1 pq: 4 rg: 24 | total time: 14.35m | eta: 47.8m |
| step 00355/01500 (23.67%) | loss: 2.695756 | lrm: 1.00 | dt: 2500.28ms | tok/sec: 209,692 | bf16_mfu: 32.01 | epoch: 1 pq: 4 rg: 24 | total time: 14.39m | eta: 47.8m |
| step 00356/01500 (23.73%) | loss: 2.614161 | lrm: 1.00 | dt: 2501.99ms | tok/sec: 209,548 | bf16_mfu: 31.99 | epoch: 1 pq: 4 rg: 24 | total time: 14.43m | eta: 47.7m |
| step 00357/01500 (23.80%) | loss: 2.627071 | lrm: 1.00 | dt: 2500.09ms | tok/sec: 209,707 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 24 | total time: 14.48m | eta: 47.7m |
| step 00358/01500 (23.87%) | loss: 2.638768 | lrm: 1.00 | dt: 2497.49ms | tok/sec: 209,926 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 24 | total time: 14.52m | eta: 47.6m |
| step 00359/01500 (23.93%) | loss: 2.608650 | lrm: 1.00 | dt: 2499.26ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 24 | total time: 14.56m | eta: 47.6m |
| step 00360/01500 (24.00%) | loss: 2.622549 | lrm: 1.00 | dt: 2499.18ms | tok/sec: 209,783 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 24 | total time: 14.60m | eta: 47.6m |
| step 00361/01500 (24.07%) | loss: 2.607933 | lrm: 1.00 | dt: 2496.72ms | tok/sec: 209,991 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 24 | total time: 14.64m | eta: 47.5m |
| step 00362/01500 (24.13%) | loss: 2.607696 | lrm: 1.00 | dt: 2496.08ms | tok/sec: 210,044 | bf16_mfu: 32.07 | epoch: 1 pq: 4 rg: 24 | total time: 14.68m | eta: 47.5m |
| step 00363/01500 (24.20%) | loss: 2.595557 | lrm: 1.00 | dt: 2498.71ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 32 | total time: 14.73m | eta: 47.4m |
| step 00364/01500 (24.27%) | loss: 2.650946 | lrm: 1.00 | dt: 2498.88ms | tok/sec: 209,809 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 32 | total time: 14.77m | eta: 47.4m |
| step 00365/01500 (24.33%) | loss: 2.641341 | lrm: 1.00 | dt: 2502.68ms | tok/sec: 209,490 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 32 | total time: 14.81m | eta: 47.3m |
| step 00366/01500 (24.40%) | loss: 2.646917 | lrm: 1.00 | dt: 2502.62ms | tok/sec: 209,495 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 32 | total time: 14.85m | eta: 47.3m |
| step 00367/01500 (24.47%) | loss: 2.635313 | lrm: 1.00 | dt: 2496.63ms | tok/sec: 209,998 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 32 | total time: 14.89m | eta: 47.3m |
| step 00368/01500 (24.53%) | loss: 2.604769 | lrm: 1.00 | dt: 2498.86ms | tok/sec: 209,811 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 32 | total time: 14.93m | eta: 47.2m |
| step 00369/01500 (24.60%) | loss: 2.614996 | lrm: 1.00 | dt: 2497.13ms | tok/sec: 209,956 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 32 | total time: 14.98m | eta: 47.2m |
| step 00370/01500 (24.67%) | loss: 2.614470 | lrm: 1.00 | dt: 2499.68ms | tok/sec: 209,742 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 32 | total time: 15.02m | eta: 47.1m |
| step 00371/01500 (24.73%) | loss: 2.626087 | lrm: 1.00 | dt: 2503.00ms | tok/sec: 209,463 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 40 | total time: 15.06m | eta: 47.1m |
| step 00372/01500 (24.80%) | loss: 2.616861 | lrm: 1.00 | dt: 2502.16ms | tok/sec: 209,534 | bf16_mfu: 31.99 | epoch: 1 pq: 4 rg: 40 | total time: 15.10m | eta: 47.1m |
| step 00373/01500 (24.87%) | loss: 2.583940 | lrm: 1.00 | dt: 2499.57ms | tok/sec: 209,751 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 40 | total time: 15.14m | eta: 47.0m |
| step 00374/01500 (24.93%) | loss: 2.535830 | lrm: 1.00 | dt: 2499.88ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 40 | total time: 15.18m | eta: 47.0m |
| step 00375/01500 (25.00%) | loss: 2.537096 | lrm: 1.00 | dt: 2502.59ms | tok/sec: 209,498 | bf16_mfu: 31.99 | epoch: 1 pq: 4 rg: 40 | total time: 15.23m | eta: 46.9m |
| step 00376/01500 (25.07%) | loss: 2.543708 | lrm: 1.00 | dt: 2497.40ms | tok/sec: 209,933 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 40 | total time: 15.27m | eta: 46.9m |
| step 00377/01500 (25.13%) | loss: 2.540690 | lrm: 1.00 | dt: 2499.48ms | tok/sec: 209,758 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 40 | total time: 15.31m | eta: 46.8m |
| step 00378/01500 (25.20%) | loss: 2.534647 | lrm: 1.00 | dt: 2500.09ms | tok/sec: 209,707 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 40 | total time: 15.35m | eta: 46.8m |
| step 00379/01500 (25.27%) | loss: 2.458322 | lrm: 1.00 | dt: 3008.53ms | tok/sec: 174,266 | bf16_mfu: 26.61 | epoch: 1 pq: 4 rg: 48 | total time: 15.40m | eta: 46.8m |
| step 00380/01500 (25.33%) | loss: 2.478253 | lrm: 1.00 | dt: 2498.42ms | tok/sec: 209,847 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 48 | total time: 15.44m | eta: 46.7m |
| step 00381/01500 (25.40%) | loss: 2.439136 | lrm: 1.00 | dt: 2497.76ms | tok/sec: 209,903 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 48 | total time: 15.48m | eta: 46.7m |
| step 00382/01500 (25.47%) | loss: 2.367838 | lrm: 1.00 | dt: 2499.43ms | tok/sec: 209,762 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 48 | total time: 15.53m | eta: 46.7m |
| step 00383/01500 (25.53%) | loss: 2.388532 | lrm: 1.00 | dt: 2498.49ms | tok/sec: 209,842 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 48 | total time: 15.57m | eta: 46.6m |
| step 00384/01500 (25.60%) | loss: 2.437780 | lrm: 1.00 | dt: 2500.15ms | tok/sec: 209,702 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 48 | total time: 15.61m | eta: 46.6m |
| step 00385/01500 (25.67%) | loss: 2.414677 | lrm: 1.00 | dt: 2497.87ms | tok/sec: 209,893 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 48 | total time: 15.65m | eta: 46.5m |
| step 00386/01500 (25.73%) | loss: 2.424039 | lrm: 1.00 | dt: 2502.17ms | tok/sec: 209,533 | bf16_mfu: 31.99 | epoch: 1 pq: 4 rg: 48 | total time: 15.69m | eta: 46.5m |
| step 00387/01500 (25.80%) | loss: 2.392342 | lrm: 1.00 | dt: 2495.37ms | tok/sec: 210,104 | bf16_mfu: 32.08 | epoch: 1 pq: 4 rg: 56 | total time: 15.73m | eta: 46.5m |
| step 00388/01500 (25.87%) | loss: 2.331812 | lrm: 1.00 | dt: 2497.23ms | tok/sec: 209,947 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 56 | total time: 15.78m | eta: 46.4m |
| step 00389/01500 (25.93%) | loss: 2.366487 | lrm: 1.00 | dt: 2502.65ms | tok/sec: 209,493 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 56 | total time: 15.82m | eta: 46.4m |
| step 00390/01500 (26.00%) | loss: 2.327390 | lrm: 1.00 | dt: 2497.37ms | tok/sec: 209,935 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 56 | total time: 15.86m | eta: 46.3m |
| step 00391/01500 (26.07%) | loss: 2.420736 | lrm: 1.00 | dt: 2498.26ms | tok/sec: 209,861 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 56 | total time: 15.90m | eta: 46.3m |
| step 00392/01500 (26.13%) | loss: 2.435264 | lrm: 1.00 | dt: 2501.12ms | tok/sec: 209,620 | bf16_mfu: 32.00 | epoch: 1 pq: 4 rg: 56 | total time: 15.94m | eta: 46.2m |
| step 00393/01500 (26.20%) | loss: 2.430283 | lrm: 1.00 | dt: 2502.73ms | tok/sec: 209,486 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 56 | total time: 15.98m | eta: 46.2m |
| step 00394/01500 (26.27%) | loss: 2.462935 | lrm: 1.00 | dt: 2497.48ms | tok/sec: 209,927 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 56 | total time: 16.03m | eta: 46.2m |
| step 00395/01500 (26.33%) | loss: 2.477935 | lrm: 1.00 | dt: 2504.88ms | tok/sec: 209,306 | bf16_mfu: 31.96 | epoch: 1 pq: 4 rg: 56 | total time: 16.07m | eta: 46.1m |
| step 00396/01500 (26.40%) | loss: 2.477937 | lrm: 1.00 | dt: 2499.35ms | tok/sec: 209,769 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 64 | total time: 16.11m | eta: 46.1m |
| step 00397/01500 (26.47%) | loss: 2.507707 | lrm: 1.00 | dt: 2505.31ms | tok/sec: 209,270 | bf16_mfu: 31.95 | epoch: 1 pq: 4 rg: 64 | total time: 16.15m | eta: 46.0m |
| step 00398/01500 (26.53%) | loss: 2.508839 | lrm: 1.00 | dt: 2501.14ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 4 rg: 64 | total time: 16.19m | eta: 46.0m |
| step 00399/01500 (26.60%) | loss: 2.461043 | lrm: 1.00 | dt: 2498.06ms | tok/sec: 209,877 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 64 | total time: 16.23m | eta: 45.9m |
| Step 00400 | Validation bpb: 0.769512 |
| Step 00400 | ICL early: 2.8437 late: 2.9914 score: +0.1477 | Induction loss: 10.8087 acc: 0.0461 (random-half: 14.1634) |
| step 00400/01500 (26.67%) | loss: 2.516207 | lrm: 1.00 | dt: 2497.98ms | tok/sec: 209,885 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 64 | total time: 16.28m | eta: 45.9m |
| step 00401/01500 (26.73%) | loss: 2.519900 | lrm: 1.00 | dt: 2504.08ms | tok/sec: 209,373 | bf16_mfu: 31.97 | epoch: 1 pq: 4 rg: 64 | total time: 16.32m | eta: 45.9m |
| step 00402/01500 (26.80%) | loss: 2.501173 | lrm: 1.00 | dt: 2502.29ms | tok/sec: 209,523 | bf16_mfu: 31.99 | epoch: 1 pq: 4 rg: 64 | total time: 16.36m | eta: 45.8m |
| step 00403/01500 (26.87%) | loss: 2.533427 | lrm: 1.00 | dt: 2501.77ms | tok/sec: 209,566 | bf16_mfu: 32.00 | epoch: 1 pq: 4 rg: 64 | total time: 16.40m | eta: 45.8m |
| step 00404/01500 (26.93%) | loss: 2.571604 | lrm: 1.00 | dt: 2503.99ms | tok/sec: 209,381 | bf16_mfu: 31.97 | epoch: 1 pq: 4 rg: 72 | total time: 16.44m | eta: 45.7m |
| step 00405/01500 (27.00%) | loss: 2.573077 | lrm: 1.00 | dt: 2500.14ms | tok/sec: 209,703 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 72 | total time: 16.48m | eta: 45.7m |
| step 00406/01500 (27.07%) | loss: 2.582443 | lrm: 1.00 | dt: 2504.71ms | tok/sec: 209,321 | bf16_mfu: 31.96 | epoch: 1 pq: 4 rg: 72 | total time: 16.53m | eta: 45.7m |
| step 00407/01500 (27.13%) | loss: 2.595983 | lrm: 1.00 | dt: 2499.64ms | tok/sec: 209,745 | bf16_mfu: 32.02 | epoch: 1 pq: 4 rg: 72 | total time: 16.57m | eta: 45.6m |
| step 00408/01500 (27.20%) | loss: 2.582707 | lrm: 1.00 | dt: 2502.59ms | tok/sec: 209,498 | bf16_mfu: 31.99 | epoch: 1 pq: 4 rg: 72 | total time: 16.61m | eta: 45.6m |
| step 00409/01500 (27.27%) | loss: 2.591550 | lrm: 1.00 | dt: 2497.45ms | tok/sec: 209,929 | bf16_mfu: 32.05 | epoch: 1 pq: 4 rg: 72 | total time: 16.65m | eta: 45.5m |
| step 00410/01500 (27.33%) | loss: 2.597649 | lrm: 1.00 | dt: 2500.90ms | tok/sec: 209,640 | bf16_mfu: 32.01 | epoch: 1 pq: 4 rg: 72 | total time: 16.69m | eta: 45.5m |
| step 00411/01500 (27.40%) | loss: 2.615967 | lrm: 1.00 | dt: 2501.65ms | tok/sec: 209,576 | bf16_mfu: 32.00 | epoch: 1 pq: 4 rg: 72 | total time: 16.73m | eta: 45.4m |
| step 00412/01500 (27.47%) | loss: 2.592289 | lrm: 1.00 | dt: 2498.68ms | tok/sec: 209,826 | bf16_mfu: 32.04 | epoch: 1 pq: 4 rg: 80 | total time: 16.78m | eta: 45.4m |
| step 00413/01500 (27.53%) | loss: 2.633285 | lrm: 1.00 | dt: 2503.42ms | tok/sec: 209,428 | bf16_mfu: 31.97 | epoch: 1 pq: 4 rg: 80 | total time: 16.82m | eta: 45.4m |
| step 00414/01500 (27.60%) | loss: 2.634857 | lrm: 1.00 | dt: 2502.98ms | tok/sec: 209,465 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 80 | total time: 16.86m | eta: 45.3m |
| step 00415/01500 (27.67%) | loss: 2.646015 | lrm: 1.00 | dt: 2496.70ms | tok/sec: 209,992 | bf16_mfu: 32.06 | epoch: 1 pq: 4 rg: 80 | total time: 16.90m | eta: 45.3m |
| step 00416/01500 (27.73%) | loss: 2.659769 | lrm: 1.00 | dt: 2498.94ms | tok/sec: 209,804 | bf16_mfu: 32.03 | epoch: 1 pq: 4 rg: 80 | total time: 16.94m | eta: 45.2m |
| step 00417/01500 (27.80%) | loss: 2.644014 | lrm: 1.00 | dt: 2504.60ms | tok/sec: 209,329 | bf16_mfu: 31.96 | epoch: 1 pq: 4 rg: 80 | total time: 16.98m | eta: 45.2m |
| step 00418/01500 (27.87%) | loss: 2.642324 | lrm: 1.00 | dt: 2503.14ms | tok/sec: 209,452 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 80 | total time: 17.03m | eta: 45.2m |
| step 00419/01500 (27.93%) | loss: 2.610305 | lrm: 1.00 | dt: 2503.30ms | tok/sec: 209,438 | bf16_mfu: 31.98 | epoch: 1 pq: 4 rg: 80 | total time: 17.07m | eta: 45.1m |
| step 00420/01500 (28.00%) | loss: 2.601027 | lrm: 1.00 | dt: 2500.63ms | tok/sec: 209,662 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 0 | total time: 17.11m | eta: 45.1m |
| step 00421/01500 (28.07%) | loss: 2.611735 | lrm: 1.00 | dt: 2502.53ms | tok/sec: 209,503 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 0 | total time: 17.15m | eta: 45.0m |
| step 00422/01500 (28.13%) | loss: 2.596965 | lrm: 1.00 | dt: 2500.54ms | tok/sec: 209,670 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 0 | total time: 17.19m | eta: 45.0m |
| step 00423/01500 (28.20%) | loss: 2.610565 | lrm: 1.00 | dt: 2498.29ms | tok/sec: 209,858 | bf16_mfu: 32.04 | epoch: 1 pq: 5 rg: 0 | total time: 17.23m | eta: 44.9m |
| step 00424/01500 (28.27%) | loss: 2.614224 | lrm: 1.00 | dt: 2498.66ms | tok/sec: 209,828 | bf16_mfu: 32.04 | epoch: 1 pq: 5 rg: 0 | total time: 17.28m | eta: 44.9m |
| step 00425/01500 (28.33%) | loss: 2.633454 | lrm: 1.00 | dt: 2500.31ms | tok/sec: 209,689 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 0 | total time: 17.32m | eta: 44.9m |
| step 00426/01500 (28.40%) | loss: 2.653873 | lrm: 1.00 | dt: 2495.96ms | tok/sec: 210,054 | bf16_mfu: 32.07 | epoch: 1 pq: 5 rg: 0 | total time: 17.36m | eta: 44.8m |
| step 00427/01500 (28.47%) | loss: 2.624991 | lrm: 1.00 | dt: 2498.74ms | tok/sec: 209,821 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 0 | total time: 17.40m | eta: 44.8m |
| step 00428/01500 (28.53%) | loss: 2.600636 | lrm: 1.00 | dt: 2504.57ms | tok/sec: 209,332 | bf16_mfu: 31.96 | epoch: 1 pq: 5 rg: 8 | total time: 17.44m | eta: 44.7m |
| step 00429/01500 (28.60%) | loss: 2.589331 | lrm: 1.00 | dt: 2496.92ms | tok/sec: 209,973 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 8 | total time: 17.48m | eta: 44.7m |
| step 00430/01500 (28.67%) | loss: 2.603567 | lrm: 1.00 | dt: 2502.68ms | tok/sec: 209,490 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 8 | total time: 17.53m | eta: 44.6m |
| step 00431/01500 (28.73%) | loss: 2.654011 | lrm: 1.00 | dt: 2498.71ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 8 | total time: 17.57m | eta: 44.6m |
| step 00432/01500 (28.80%) | loss: 2.552250 | lrm: 1.00 | dt: 2499.25ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 8 | total time: 17.61m | eta: 44.6m |
| step 00433/01500 (28.87%) | loss: 2.576312 | lrm: 1.00 | dt: 2501.86ms | tok/sec: 209,559 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 8 | total time: 17.65m | eta: 44.5m |
| step 00434/01500 (28.93%) | loss: 2.587704 | lrm: 1.00 | dt: 2500.07ms | tok/sec: 209,709 | bf16_mfu: 32.02 | epoch: 1 pq: 5 rg: 8 | total time: 17.69m | eta: 44.5m |
| step 00435/01500 (29.00%) | loss: 2.696486 | lrm: 1.00 | dt: 2502.16ms | tok/sec: 209,534 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 8 | total time: 17.73m | eta: 44.4m |
| step 00436/01500 (29.07%) | loss: 2.713219 | lrm: 1.00 | dt: 2498.66ms | tok/sec: 209,827 | bf16_mfu: 32.04 | epoch: 1 pq: 5 rg: 8 | total time: 17.78m | eta: 44.4m |
| step 00437/01500 (29.13%) | loss: 2.742426 | lrm: 1.00 | dt: 2500.29ms | tok/sec: 209,691 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 16 | total time: 17.82m | eta: 44.4m |
| step 00438/01500 (29.20%) | loss: 2.716522 | lrm: 1.00 | dt: 2502.57ms | tok/sec: 209,499 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 16 | total time: 17.86m | eta: 44.3m |
| step 00439/01500 (29.27%) | loss: 2.714635 | lrm: 1.00 | dt: 2499.53ms | tok/sec: 209,754 | bf16_mfu: 32.02 | epoch: 1 pq: 5 rg: 16 | total time: 17.90m | eta: 44.3m |
| step 00440/01500 (29.33%) | loss: 2.735734 | lrm: 1.00 | dt: 2503.85ms | tok/sec: 209,392 | bf16_mfu: 31.97 | epoch: 1 pq: 5 rg: 16 | total time: 17.94m | eta: 44.2m |
| step 00441/01500 (29.40%) | loss: 2.716592 | lrm: 1.00 | dt: 2502.81ms | tok/sec: 209,479 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 16 | total time: 17.98m | eta: 44.2m |
| step 00442/01500 (29.47%) | loss: 2.679167 | lrm: 1.00 | dt: 2504.39ms | tok/sec: 209,347 | bf16_mfu: 31.96 | epoch: 1 pq: 5 rg: 16 | total time: 18.03m | eta: 44.1m |
| step 00443/01500 (29.53%) | loss: 2.652363 | lrm: 1.00 | dt: 2503.08ms | tok/sec: 209,456 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 16 | total time: 18.07m | eta: 44.1m |
| step 00444/01500 (29.60%) | loss: 2.668071 | lrm: 1.00 | dt: 2496.79ms | tok/sec: 209,984 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 16 | total time: 18.11m | eta: 44.1m |
| step 00445/01500 (29.67%) | loss: 2.665459 | lrm: 1.00 | dt: 2502.05ms | tok/sec: 209,543 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 24 | total time: 18.15m | eta: 44.0m |
| step 00446/01500 (29.73%) | loss: 2.679841 | lrm: 1.00 | dt: 2501.16ms | tok/sec: 209,617 | bf16_mfu: 32.00 | epoch: 1 pq: 5 rg: 24 | total time: 18.19m | eta: 44.0m |
| step 00447/01500 (29.80%) | loss: 2.669771 | lrm: 1.00 | dt: 2502.88ms | tok/sec: 209,473 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 24 | total time: 18.23m | eta: 43.9m |
| step 00448/01500 (29.87%) | loss: 2.672466 | lrm: 1.00 | dt: 2500.79ms | tok/sec: 209,648 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 24 | total time: 18.28m | eta: 43.9m |
| step 00449/01500 (29.93%) | loss: 2.641235 | lrm: 1.00 | dt: 2503.18ms | tok/sec: 209,448 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 24 | total time: 18.32m | eta: 43.9m |
| step 00450/01500 (30.00%) | loss: 2.633718 | lrm: 1.00 | dt: 2502.95ms | tok/sec: 209,467 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 24 | total time: 18.36m | eta: 43.8m |
| step 00451/01500 (30.07%) | loss: 2.610007 | lrm: 1.00 | dt: 2504.55ms | tok/sec: 209,334 | bf16_mfu: 31.96 | epoch: 1 pq: 5 rg: 24 | total time: 18.40m | eta: 43.8m |
| step 00452/01500 (30.13%) | loss: 2.558630 | lrm: 1.00 | dt: 2494.24ms | tok/sec: 210,199 | bf16_mfu: 32.09 | epoch: 1 pq: 5 rg: 24 | total time: 18.44m | eta: 43.7m |
| step 00453/01500 (30.20%) | loss: 2.532510 | lrm: 1.00 | dt: 2500.48ms | tok/sec: 209,674 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 32 | total time: 18.48m | eta: 43.7m |
| step 00454/01500 (30.27%) | loss: 2.527563 | lrm: 1.00 | dt: 2501.22ms | tok/sec: 209,612 | bf16_mfu: 32.00 | epoch: 1 pq: 5 rg: 32 | total time: 18.53m | eta: 43.6m |
| step 00455/01500 (30.33%) | loss: 2.534919 | lrm: 1.00 | dt: 2504.73ms | tok/sec: 209,319 | bf16_mfu: 31.96 | epoch: 1 pq: 5 rg: 32 | total time: 18.57m | eta: 43.6m |
| step 00456/01500 (30.40%) | loss: 2.523789 | lrm: 1.00 | dt: 2499.57ms | tok/sec: 209,750 | bf16_mfu: 32.02 | epoch: 1 pq: 5 rg: 32 | total time: 18.61m | eta: 43.6m |
| step 00457/01500 (30.47%) | loss: 2.497783 | lrm: 1.00 | dt: 2501.66ms | tok/sec: 209,575 | bf16_mfu: 32.00 | epoch: 1 pq: 5 rg: 32 | total time: 18.65m | eta: 43.5m |
| step 00458/01500 (30.53%) | loss: 2.515548 | lrm: 1.00 | dt: 2502.16ms | tok/sec: 209,534 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 32 | total time: 18.69m | eta: 43.5m |
| step 00459/01500 (30.60%) | loss: 2.558479 | lrm: 1.00 | dt: 2500.55ms | tok/sec: 209,668 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 32 | total time: 18.74m | eta: 43.4m |
| step 00460/01500 (30.67%) | loss: 2.568665 | lrm: 1.00 | dt: 2601.98ms | tok/sec: 201,495 | bf16_mfu: 30.76 | epoch: 1 pq: 5 rg: 32 | total time: 18.78m | eta: 43.4m |
| step 00461/01500 (30.73%) | loss: 2.549469 | lrm: 1.00 | dt: 2497.65ms | tok/sec: 209,912 | bf16_mfu: 32.05 | epoch: 1 pq: 5 rg: 40 | total time: 18.82m | eta: 43.4m |
| step 00462/01500 (30.80%) | loss: 2.527152 | lrm: 1.00 | dt: 2497.69ms | tok/sec: 209,908 | bf16_mfu: 32.05 | epoch: 1 pq: 5 rg: 40 | total time: 18.86m | eta: 43.3m |
| step 00463/01500 (30.87%) | loss: 2.562973 | lrm: 1.00 | dt: 2499.69ms | tok/sec: 209,741 | bf16_mfu: 32.02 | epoch: 1 pq: 5 rg: 40 | total time: 18.90m | eta: 43.3m |
| step 00464/01500 (30.93%) | loss: 2.558161 | lrm: 1.00 | dt: 2496.52ms | tok/sec: 210,007 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 40 | total time: 18.94m | eta: 43.2m |
| step 00465/01500 (31.00%) | loss: 2.654085 | lrm: 1.00 | dt: 2503.38ms | tok/sec: 209,432 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 40 | total time: 18.99m | eta: 43.2m |
| step 00466/01500 (31.07%) | loss: 2.631398 | lrm: 1.00 | dt: 2498.73ms | tok/sec: 209,821 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 40 | total time: 19.03m | eta: 43.1m |
| step 00467/01500 (31.13%) | loss: 2.654613 | lrm: 1.00 | dt: 2506.62ms | tok/sec: 209,161 | bf16_mfu: 31.93 | epoch: 1 pq: 5 rg: 40 | total time: 19.07m | eta: 43.1m |
| step 00468/01500 (31.20%) | loss: 2.650343 | lrm: 1.00 | dt: 2500.46ms | tok/sec: 209,676 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 40 | total time: 19.11m | eta: 43.1m |
| step 00469/01500 (31.27%) | loss: 2.615141 | lrm: 1.00 | dt: 2498.96ms | tok/sec: 209,802 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 48 | total time: 19.15m | eta: 43.0m |
| step 00470/01500 (31.33%) | loss: 2.604841 | lrm: 1.00 | dt: 2500.30ms | tok/sec: 209,689 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 48 | total time: 19.20m | eta: 43.0m |
| step 00471/01500 (31.40%) | loss: 2.579015 | lrm: 1.00 | dt: 2496.60ms | tok/sec: 210,001 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 48 | total time: 19.24m | eta: 42.9m |
| step 00472/01500 (31.47%) | loss: 2.538458 | lrm: 1.00 | dt: 2496.98ms | tok/sec: 209,968 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 48 | total time: 19.28m | eta: 42.9m |
| step 00473/01500 (31.53%) | loss: 2.529463 | lrm: 1.00 | dt: 2498.70ms | tok/sec: 209,824 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 48 | total time: 19.32m | eta: 42.9m |
| step 00474/01500 (31.60%) | loss: 2.578395 | lrm: 1.00 | dt: 2499.59ms | tok/sec: 209,749 | bf16_mfu: 32.02 | epoch: 1 pq: 5 rg: 48 | total time: 19.36m | eta: 42.8m |
| step 00475/01500 (31.67%) | loss: 2.591929 | lrm: 1.00 | dt: 2501.67ms | tok/sec: 209,575 | bf16_mfu: 32.00 | epoch: 1 pq: 5 rg: 48 | total time: 19.40m | eta: 42.8m |
| step 00476/01500 (31.73%) | loss: 2.588667 | lrm: 1.00 | dt: 2504.15ms | tok/sec: 209,367 | bf16_mfu: 31.97 | epoch: 1 pq: 5 rg: 48 | total time: 19.45m | eta: 42.7m |
| step 00477/01500 (31.80%) | loss: 2.549180 | lrm: 1.00 | dt: 2498.01ms | tok/sec: 209,882 | bf16_mfu: 32.04 | epoch: 1 pq: 5 rg: 48 | total time: 19.49m | eta: 42.7m |
| step 00478/01500 (31.87%) | loss: 2.571854 | lrm: 1.00 | dt: 2501.14ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 5 rg: 56 | total time: 19.53m | eta: 42.6m |
| step 00479/01500 (31.93%) | loss: 2.581863 | lrm: 1.00 | dt: 2495.07ms | tok/sec: 210,129 | bf16_mfu: 32.08 | epoch: 1 pq: 5 rg: 56 | total time: 19.57m | eta: 42.6m |
| step 00480/01500 (32.00%) | loss: 2.540622 | lrm: 1.00 | dt: 2502.05ms | tok/sec: 209,543 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 56 | total time: 19.61m | eta: 42.6m |
| step 00481/01500 (32.07%) | loss: 2.452574 | lrm: 1.00 | dt: 2504.77ms | tok/sec: 209,315 | bf16_mfu: 31.96 | epoch: 1 pq: 5 rg: 56 | total time: 19.65m | eta: 42.5m |
| step 00482/01500 (32.13%) | loss: 2.525343 | lrm: 1.00 | dt: 2500.68ms | tok/sec: 209,658 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 56 | total time: 19.70m | eta: 42.5m |
| step 00483/01500 (32.20%) | loss: 2.550240 | lrm: 1.00 | dt: 2496.04ms | tok/sec: 210,048 | bf16_mfu: 32.07 | epoch: 1 pq: 5 rg: 56 | total time: 19.74m | eta: 42.4m |
| step 00484/01500 (32.27%) | loss: 2.497811 | lrm: 1.00 | dt: 2498.28ms | tok/sec: 209,859 | bf16_mfu: 32.04 | epoch: 1 pq: 5 rg: 56 | total time: 19.78m | eta: 42.4m |
| step 00485/01500 (32.33%) | loss: 2.496545 | lrm: 1.00 | dt: 2499.34ms | tok/sec: 209,770 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 56 | total time: 19.82m | eta: 42.4m |
| step 00486/01500 (32.40%) | loss: 2.525746 | lrm: 1.00 | dt: 2503.49ms | tok/sec: 209,423 | bf16_mfu: 31.97 | epoch: 1 pq: 5 rg: 64 | total time: 19.86m | eta: 42.3m |
| step 00487/01500 (32.47%) | loss: 2.539668 | lrm: 1.00 | dt: 2497.55ms | tok/sec: 209,921 | bf16_mfu: 32.05 | epoch: 1 pq: 5 rg: 64 | total time: 19.90m | eta: 42.3m |
| step 00488/01500 (32.53%) | loss: 2.530516 | lrm: 1.00 | dt: 2500.24ms | tok/sec: 209,694 | bf16_mfu: 32.02 | epoch: 1 pq: 5 rg: 64 | total time: 19.94m | eta: 42.2m |
| step 00489/01500 (32.60%) | loss: 2.537329 | lrm: 1.00 | dt: 2501.13ms | tok/sec: 209,620 | bf16_mfu: 32.00 | epoch: 1 pq: 5 rg: 64 | total time: 19.99m | eta: 42.2m |
| step 00490/01500 (32.67%) | loss: 2.534464 | lrm: 1.00 | dt: 2503.14ms | tok/sec: 209,452 | bf16_mfu: 31.98 | epoch: 1 pq: 5 rg: 64 | total time: 20.03m | eta: 42.1m |
| step 00491/01500 (32.73%) | loss: 2.530831 | lrm: 1.00 | dt: 2503.65ms | tok/sec: 209,409 | bf16_mfu: 31.97 | epoch: 1 pq: 5 rg: 64 | total time: 20.07m | eta: 42.1m |
| step 00492/01500 (32.80%) | loss: 2.498196 | lrm: 1.00 | dt: 2500.75ms | tok/sec: 209,652 | bf16_mfu: 32.01 | epoch: 1 pq: 5 rg: 64 | total time: 20.11m | eta: 42.1m |
| step 00493/01500 (32.87%) | loss: 2.483597 | lrm: 1.00 | dt: 2497.92ms | tok/sec: 209,889 | bf16_mfu: 32.04 | epoch: 1 pq: 5 rg: 64 | total time: 20.15m | eta: 42.0m |
| step 00494/01500 (32.93%) | loss: 2.509084 | lrm: 1.00 | dt: 2502.00ms | tok/sec: 209,547 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 72 | total time: 20.20m | eta: 42.0m |
| step 00495/01500 (33.00%) | loss: 2.470687 | lrm: 1.00 | dt: 2506.07ms | tok/sec: 209,207 | bf16_mfu: 31.94 | epoch: 1 pq: 5 rg: 72 | total time: 20.24m | eta: 41.9m |
| step 00496/01500 (33.07%) | loss: 2.479860 | lrm: 1.00 | dt: 2502.48ms | tok/sec: 209,507 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 72 | total time: 20.28m | eta: 41.9m |
| step 00497/01500 (33.13%) | loss: 2.529020 | lrm: 1.00 | dt: 2496.55ms | tok/sec: 210,005 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 72 | total time: 20.32m | eta: 41.9m |
| step 00498/01500 (33.20%) | loss: 2.506069 | lrm: 1.00 | dt: 2496.61ms | tok/sec: 209,999 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 72 | total time: 20.36m | eta: 41.8m |
| step 00499/01500 (33.27%) | loss: 2.503669 | lrm: 1.00 | dt: 2502.22ms | tok/sec: 209,528 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 72 | total time: 20.40m | eta: 41.8m |
| Step 00500 | Validation bpb: 0.758676 |
| Step 00500 | ICL early: 2.7840 late: 2.9436 score: +0.1596 | Induction loss: 10.4020 acc: 0.0576 (random-half: 14.0550) |
| <|bos|>The capital of France is Paris. It is the capital of the Île-de-France region. It |
| <|bos|>The chemical symbol of gold is Au. The chemical symbol of silver is Ag. The chemical symbol of copper is |
| <|bos|>If yesterday was Friday, then tomorrow will be Saturday. Today is Saturday, and tomorrow is Sunday. If today is Saturday, |
| <|bos|>The opposite of hot is cold. Hot is a color that is associated with the sun and the sun's |
| <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, |
| <|bos|>My favorite color is blue. I like to wear blue jeans and I like to wear blue jeans. |
| <|bos|>If 5*x + 3 = 13, then x is 3. If 5*x + 3 = 13, then |
| 2026-08-10 21:53:12,881 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank5.pt |
| 2026-08-10 21:53:14,092 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank3.pt |
| 2026-08-10 21:53:17,130 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank4.pt |
| 2026-08-10 21:53:18,463 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank1.pt |
| 2026-08-10 21:53:18,846 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank7.pt |
| 2026-08-10 21:53:19,073 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank2.pt |
| 2026-08-10 21:53:19,269 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank6.pt |
| 2026-08-10 21:54:18,230 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved model parameters to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/model_000500.pt |
| 2026-08-10 21:54:18,232 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved metadata to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/meta_000500.json |
| 2026-08-10 21:54:25,608 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000500_rank0.pt |
| step 00500/01500 (33.33%) | loss: 2.500904 | lrm: 1.00 | dt: 2475.99ms | tok/sec: 211,748 | bf16_mfu: 32.33 | epoch: 1 pq: 5 rg: 72 | total time: 20.44m | eta: 41.7m |
| step 00501/01500 (33.40%) | loss: 2.463526 | lrm: 1.00 | dt: 2481.54ms | tok/sec: 211,275 | bf16_mfu: 32.26 | epoch: 1 pq: 5 rg: 72 | total time: 20.49m | eta: 41.7m |
| step 00502/01500 (33.47%) | loss: 2.518413 | lrm: 1.00 | dt: 2484.65ms | tok/sec: 211,011 | bf16_mfu: 32.22 | epoch: 1 pq: 5 rg: 80 | total time: 20.53m | eta: 41.6m |
| step 00503/01500 (33.53%) | loss: 2.577312 | lrm: 1.00 | dt: 2492.33ms | tok/sec: 210,360 | bf16_mfu: 32.12 | epoch: 1 pq: 5 rg: 80 | total time: 20.57m | eta: 41.6m |
| step 00504/01500 (33.60%) | loss: 2.613699 | lrm: 1.00 | dt: 2491.09ms | tok/sec: 210,465 | bf16_mfu: 32.13 | epoch: 1 pq: 5 rg: 80 | total time: 20.61m | eta: 41.6m |
| step 00505/01500 (33.67%) | loss: 2.619416 | lrm: 1.00 | dt: 2496.58ms | tok/sec: 210,002 | bf16_mfu: 32.06 | epoch: 1 pq: 5 rg: 80 | total time: 20.65m | eta: 41.5m |
| step 00506/01500 (33.73%) | loss: 2.636152 | lrm: 1.00 | dt: 2498.08ms | tok/sec: 209,876 | bf16_mfu: 32.04 | epoch: 1 pq: 5 rg: 80 | total time: 20.69m | eta: 41.5m |
| step 00507/01500 (33.80%) | loss: 2.624096 | lrm: 1.00 | dt: 2502.37ms | tok/sec: 209,516 | bf16_mfu: 31.99 | epoch: 1 pq: 5 rg: 80 | total time: 20.74m | eta: 41.4m |
| step 00508/01500 (33.87%) | loss: 2.641861 | lrm: 1.00 | dt: 2499.42ms | tok/sec: 209,763 | bf16_mfu: 32.03 | epoch: 1 pq: 5 rg: 80 | total time: 20.78m | eta: 41.4m |
| step 00509/01500 (33.93%) | loss: 2.601222 | lrm: 1.00 | dt: 2505.30ms | tok/sec: 209,271 | bf16_mfu: 31.95 | epoch: 1 pq: 5 rg: 80 | total time: 20.82m | eta: 41.3m |
| step 00510/01500 (34.00%) | loss: 2.568798 | lrm: 1.00 | dt: 2501.78ms | tok/sec: 209,565 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 0 | total time: 20.86m | eta: 41.3m |
| step 00511/01500 (34.07%) | loss: 2.529884 | lrm: 1.00 | dt: 2493.74ms | tok/sec: 210,241 | bf16_mfu: 32.10 | epoch: 1 pq: 6 rg: 0 | total time: 20.90m | eta: 41.3m |
| step 00512/01500 (34.13%) | loss: 2.572685 | lrm: 1.00 | dt: 2497.54ms | tok/sec: 209,922 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 0 | total time: 20.94m | eta: 41.2m |
| step 00513/01500 (34.20%) | loss: 2.588677 | lrm: 1.00 | dt: 2498.74ms | tok/sec: 209,821 | bf16_mfu: 32.03 | epoch: 1 pq: 6 rg: 0 | total time: 20.99m | eta: 41.2m |
| step 00514/01500 (34.27%) | loss: 2.607854 | lrm: 1.00 | dt: 2497.14ms | tok/sec: 209,955 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 0 | total time: 21.03m | eta: 41.1m |
| step 00515/01500 (34.33%) | loss: 2.605475 | lrm: 1.00 | dt: 2506.18ms | tok/sec: 209,198 | bf16_mfu: 31.94 | epoch: 1 pq: 6 rg: 0 | total time: 21.07m | eta: 41.1m |
| step 00516/01500 (34.40%) | loss: 2.625303 | lrm: 1.00 | dt: 2497.45ms | tok/sec: 209,929 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 0 | total time: 21.11m | eta: 41.1m |
| step 00517/01500 (34.47%) | loss: 2.591898 | lrm: 1.00 | dt: 2498.23ms | tok/sec: 209,864 | bf16_mfu: 32.04 | epoch: 1 pq: 6 rg: 0 | total time: 21.15m | eta: 41.0m |
| step 00518/01500 (34.53%) | loss: 2.607237 | lrm: 1.00 | dt: 2509.66ms | tok/sec: 208,907 | bf16_mfu: 31.89 | epoch: 1 pq: 6 rg: 0 | total time: 21.19m | eta: 41.0m |
| step 00519/01500 (34.60%) | loss: 2.591363 | lrm: 1.00 | dt: 2502.36ms | tok/sec: 209,517 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 8 | total time: 21.24m | eta: 40.9m |
| step 00520/01500 (34.67%) | loss: 2.605786 | lrm: 1.00 | dt: 2501.47ms | tok/sec: 209,592 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 8 | total time: 21.28m | eta: 40.9m |
| step 00521/01500 (34.73%) | loss: 2.601866 | lrm: 1.00 | dt: 2498.37ms | tok/sec: 209,851 | bf16_mfu: 32.04 | epoch: 1 pq: 6 rg: 8 | total time: 21.32m | eta: 40.8m |
| step 00522/01500 (34.80%) | loss: 2.550722 | lrm: 1.00 | dt: 2501.55ms | tok/sec: 209,585 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 8 | total time: 21.36m | eta: 40.8m |
| step 00523/01500 (34.87%) | loss: 2.519845 | lrm: 1.00 | dt: 2502.27ms | tok/sec: 209,524 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 8 | total time: 21.40m | eta: 40.8m |
| step 00524/01500 (34.93%) | loss: 2.545520 | lrm: 1.00 | dt: 2502.08ms | tok/sec: 209,540 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 8 | total time: 21.44m | eta: 40.7m |
| step 00525/01500 (35.00%) | loss: 2.555268 | lrm: 1.00 | dt: 2502.19ms | tok/sec: 209,531 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 8 | total time: 21.49m | eta: 40.7m |
| step 00526/01500 (35.07%) | loss: 2.545308 | lrm: 1.00 | dt: 2495.97ms | tok/sec: 210,053 | bf16_mfu: 32.07 | epoch: 1 pq: 6 rg: 8 | total time: 21.53m | eta: 40.6m |
| step 00527/01500 (35.13%) | loss: 2.570636 | lrm: 1.00 | dt: 2502.35ms | tok/sec: 209,518 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 16 | total time: 21.57m | eta: 40.6m |
| step 00528/01500 (35.20%) | loss: 2.579141 | lrm: 1.00 | dt: 2495.69ms | tok/sec: 210,077 | bf16_mfu: 32.07 | epoch: 1 pq: 6 rg: 16 | total time: 21.61m | eta: 40.6m |
| step 00529/01500 (35.27%) | loss: 2.580171 | lrm: 1.00 | dt: 2498.88ms | tok/sec: 209,808 | bf16_mfu: 32.03 | epoch: 1 pq: 6 rg: 16 | total time: 21.65m | eta: 40.5m |
| step 00530/01500 (35.33%) | loss: 2.548039 | lrm: 1.00 | dt: 2500.94ms | tok/sec: 209,636 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 16 | total time: 21.69m | eta: 40.5m |
| step 00531/01500 (35.40%) | loss: 2.502980 | lrm: 1.00 | dt: 2500.77ms | tok/sec: 209,650 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 16 | total time: 21.74m | eta: 40.4m |
| step 00532/01500 (35.47%) | loss: 2.525493 | lrm: 1.00 | dt: 2501.45ms | tok/sec: 209,594 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 16 | total time: 21.78m | eta: 40.4m |
| step 00533/01500 (35.53%) | loss: 2.506324 | lrm: 1.00 | dt: 2499.94ms | tok/sec: 209,720 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 16 | total time: 21.82m | eta: 40.3m |
| step 00534/01500 (35.60%) | loss: 2.501816 | lrm: 1.00 | dt: 2501.13ms | tok/sec: 209,620 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 16 | total time: 21.86m | eta: 40.3m |
| step 00535/01500 (35.67%) | loss: 2.491864 | lrm: 1.00 | dt: 2504.63ms | tok/sec: 209,327 | bf16_mfu: 31.96 | epoch: 1 pq: 6 rg: 24 | total time: 21.90m | eta: 40.3m |
| step 00536/01500 (35.73%) | loss: 2.486440 | lrm: 1.00 | dt: 2500.08ms | tok/sec: 209,708 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 24 | total time: 21.94m | eta: 40.2m |
| step 00537/01500 (35.80%) | loss: 2.434391 | lrm: 1.00 | dt: 2500.46ms | tok/sec: 209,676 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 24 | total time: 21.99m | eta: 40.2m |
| step 00538/01500 (35.87%) | loss: 2.410299 | lrm: 1.00 | dt: 2500.80ms | tok/sec: 209,648 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 24 | total time: 22.03m | eta: 40.1m |
| step 00539/01500 (35.93%) | loss: 2.355827 | lrm: 1.00 | dt: 2500.74ms | tok/sec: 209,652 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 24 | total time: 22.07m | eta: 40.1m |
| step 00540/01500 (36.00%) | loss: 2.366697 | lrm: 1.00 | dt: 2501.53ms | tok/sec: 209,586 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 24 | total time: 22.11m | eta: 40.0m |
| step 00541/01500 (36.07%) | loss: 2.375723 | lrm: 1.00 | dt: 3375.97ms | tok/sec: 155,299 | bf16_mfu: 23.71 | epoch: 1 pq: 6 rg: 24 | total time: 22.17m | eta: 40.0m |
| step 00542/01500 (36.13%) | loss: 2.350408 | lrm: 1.00 | dt: 2494.44ms | tok/sec: 210,182 | bf16_mfu: 32.09 | epoch: 1 pq: 6 rg: 24 | total time: 22.21m | eta: 40.0m |
| step 00543/01500 (36.20%) | loss: 2.399915 | lrm: 1.00 | dt: 2495.23ms | tok/sec: 210,116 | bf16_mfu: 32.08 | epoch: 1 pq: 6 rg: 32 | total time: 22.25m | eta: 40.0m |
| step 00544/01500 (36.27%) | loss: 2.404802 | lrm: 1.00 | dt: 2497.91ms | tok/sec: 209,890 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 32 | total time: 22.29m | eta: 39.9m |
| step 00545/01500 (36.33%) | loss: 2.411226 | lrm: 1.00 | dt: 2502.94ms | tok/sec: 209,469 | bf16_mfu: 31.98 | epoch: 1 pq: 6 rg: 32 | total time: 22.33m | eta: 39.9m |
| step 00546/01500 (36.40%) | loss: 2.450549 | lrm: 1.00 | dt: 2499.50ms | tok/sec: 209,757 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 32 | total time: 22.38m | eta: 39.8m |
| step 00547/01500 (36.47%) | loss: 2.512951 | lrm: 1.00 | dt: 2501.35ms | tok/sec: 209,601 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 32 | total time: 22.42m | eta: 39.8m |
| step 00548/01500 (36.53%) | loss: 2.452717 | lrm: 1.00 | dt: 2497.34ms | tok/sec: 209,938 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 32 | total time: 22.46m | eta: 39.7m |
| step 00549/01500 (36.60%) | loss: 2.430202 | lrm: 1.00 | dt: 2501.50ms | tok/sec: 209,589 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 32 | total time: 22.50m | eta: 39.7m |
| step 00550/01500 (36.67%) | loss: 2.483732 | lrm: 1.00 | dt: 2498.78ms | tok/sec: 209,817 | bf16_mfu: 32.03 | epoch: 1 pq: 6 rg: 32 | total time: 22.54m | eta: 39.7m |
| step 00551/01500 (36.73%) | loss: 2.436023 | lrm: 1.00 | dt: 2502.61ms | tok/sec: 209,496 | bf16_mfu: 31.98 | epoch: 1 pq: 6 rg: 40 | total time: 22.58m | eta: 39.6m |
| step 00552/01500 (36.80%) | loss: 2.370890 | lrm: 1.00 | dt: 2501.94ms | tok/sec: 209,552 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 40 | total time: 22.63m | eta: 39.6m |
| step 00553/01500 (36.87%) | loss: 2.398080 | lrm: 1.00 | dt: 2499.48ms | tok/sec: 209,759 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 40 | total time: 22.67m | eta: 39.5m |
| step 00554/01500 (36.93%) | loss: 2.441161 | lrm: 1.00 | dt: 2499.22ms | tok/sec: 209,780 | bf16_mfu: 32.03 | epoch: 1 pq: 6 rg: 40 | total time: 22.71m | eta: 39.5m |
| step 00555/01500 (37.00%) | loss: 2.442970 | lrm: 1.00 | dt: 2515.28ms | tok/sec: 208,440 | bf16_mfu: 31.82 | epoch: 1 pq: 6 rg: 40 | total time: 22.75m | eta: 39.4m |
| step 00556/01500 (37.07%) | loss: 2.426135 | lrm: 1.00 | dt: 2497.45ms | tok/sec: 209,928 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 40 | total time: 22.79m | eta: 39.4m |
| step 00557/01500 (37.13%) | loss: 2.446479 | lrm: 1.00 | dt: 2499.78ms | tok/sec: 209,733 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 40 | total time: 22.83m | eta: 39.4m |
| step 00558/01500 (37.20%) | loss: 2.498584 | lrm: 1.00 | dt: 2495.45ms | tok/sec: 210,097 | bf16_mfu: 32.08 | epoch: 1 pq: 6 rg: 40 | total time: 22.88m | eta: 39.3m |
| step 00559/01500 (37.27%) | loss: 2.485985 | lrm: 1.00 | dt: 2502.52ms | tok/sec: 209,503 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 48 | total time: 22.92m | eta: 39.3m |
| step 00560/01500 (37.33%) | loss: 2.494606 | lrm: 1.00 | dt: 2500.85ms | tok/sec: 209,644 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 48 | total time: 22.96m | eta: 39.2m |
| step 00561/01500 (37.40%) | loss: 2.515181 | lrm: 1.00 | dt: 2504.71ms | tok/sec: 209,321 | bf16_mfu: 31.96 | epoch: 1 pq: 6 rg: 48 | total time: 23.00m | eta: 39.2m |
| step 00562/01500 (37.47%) | loss: 2.476007 | lrm: 1.00 | dt: 2500.38ms | tok/sec: 209,682 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 48 | total time: 23.04m | eta: 39.2m |
| step 00563/01500 (37.53%) | loss: 2.465159 | lrm: 1.00 | dt: 2500.36ms | tok/sec: 209,685 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 48 | total time: 23.08m | eta: 39.1m |
| step 00564/01500 (37.60%) | loss: 2.490591 | lrm: 1.00 | dt: 2500.24ms | tok/sec: 209,695 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 48 | total time: 23.13m | eta: 39.1m |
| step 00565/01500 (37.67%) | loss: 2.511817 | lrm: 1.00 | dt: 2498.87ms | tok/sec: 209,810 | bf16_mfu: 32.03 | epoch: 1 pq: 6 rg: 48 | total time: 23.17m | eta: 39.0m |
| step 00566/01500 (37.73%) | loss: 2.482246 | lrm: 1.00 | dt: 2497.28ms | tok/sec: 209,943 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 48 | total time: 23.21m | eta: 39.0m |
| step 00567/01500 (37.80%) | loss: 2.484301 | lrm: 1.00 | dt: 2500.44ms | tok/sec: 209,678 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 56 | total time: 23.25m | eta: 38.9m |
| step 00568/01500 (37.87%) | loss: 2.439863 | lrm: 1.00 | dt: 2499.93ms | tok/sec: 209,720 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 56 | total time: 23.29m | eta: 38.9m |
| step 00569/01500 (37.93%) | loss: 2.414052 | lrm: 1.00 | dt: 2504.69ms | tok/sec: 209,322 | bf16_mfu: 31.96 | epoch: 1 pq: 6 rg: 56 | total time: 23.33m | eta: 38.9m |
| step 00570/01500 (38.00%) | loss: 2.429948 | lrm: 1.00 | dt: 2497.33ms | tok/sec: 209,939 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 56 | total time: 23.38m | eta: 38.8m |
| step 00571/01500 (38.07%) | loss: 2.440057 | lrm: 1.00 | dt: 2500.07ms | tok/sec: 209,709 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 56 | total time: 23.42m | eta: 38.8m |
| step 00572/01500 (38.13%) | loss: 2.492334 | lrm: 1.00 | dt: 2498.79ms | tok/sec: 209,817 | bf16_mfu: 32.03 | epoch: 1 pq: 6 rg: 56 | total time: 23.46m | eta: 38.7m |
| step 00573/01500 (38.20%) | loss: 2.540238 | lrm: 1.00 | dt: 2500.43ms | tok/sec: 209,679 | bf16_mfu: 32.01 | epoch: 1 pq: 6 rg: 56 | total time: 23.50m | eta: 38.7m |
| step 00574/01500 (38.27%) | loss: 2.579401 | lrm: 1.00 | dt: 2499.69ms | tok/sec: 209,740 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 56 | total time: 23.54m | eta: 38.7m |
| step 00575/01500 (38.33%) | loss: 2.539988 | lrm: 1.00 | dt: 2497.77ms | tok/sec: 209,902 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 64 | total time: 23.58m | eta: 38.6m |
| step 00576/01500 (38.40%) | loss: 2.479109 | lrm: 1.00 | dt: 2501.94ms | tok/sec: 209,552 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 64 | total time: 23.63m | eta: 38.6m |
| step 00577/01500 (38.47%) | loss: 2.549384 | lrm: 1.00 | dt: 2506.03ms | tok/sec: 209,210 | bf16_mfu: 31.94 | epoch: 1 pq: 6 rg: 64 | total time: 23.67m | eta: 38.5m |
| step 00578/01500 (38.53%) | loss: 2.503151 | lrm: 1.00 | dt: 2496.43ms | tok/sec: 210,015 | bf16_mfu: 32.06 | epoch: 1 pq: 6 rg: 64 | total time: 23.71m | eta: 38.5m |
| step 00579/01500 (38.60%) | loss: 2.483110 | lrm: 1.00 | dt: 2503.21ms | tok/sec: 209,446 | bf16_mfu: 31.98 | epoch: 1 pq: 6 rg: 64 | total time: 23.75m | eta: 38.4m |
| step 00580/01500 (38.67%) | loss: 2.403492 | lrm: 1.00 | dt: 2499.16ms | tok/sec: 209,785 | bf16_mfu: 32.03 | epoch: 1 pq: 6 rg: 64 | total time: 23.79m | eta: 38.4m |
| step 00581/01500 (38.73%) | loss: 2.474327 | lrm: 1.00 | dt: 2498.07ms | tok/sec: 209,877 | bf16_mfu: 32.04 | epoch: 1 pq: 6 rg: 64 | total time: 23.83m | eta: 38.4m |
| step 00582/01500 (38.80%) | loss: 2.492671 | lrm: 1.00 | dt: 2500.23ms | tok/sec: 209,696 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 64 | total time: 23.88m | eta: 38.3m |
| step 00583/01500 (38.87%) | loss: 2.491891 | lrm: 1.00 | dt: 2502.13ms | tok/sec: 209,536 | bf16_mfu: 31.99 | epoch: 1 pq: 6 rg: 72 | total time: 23.92m | eta: 38.3m |
| step 00584/01500 (38.93%) | loss: 2.457806 | lrm: 1.00 | dt: 2497.86ms | tok/sec: 209,894 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 72 | total time: 23.96m | eta: 38.2m |
| step 00585/01500 (39.00%) | loss: 2.450195 | lrm: 1.00 | dt: 2506.96ms | tok/sec: 209,132 | bf16_mfu: 31.93 | epoch: 1 pq: 6 rg: 72 | total time: 24.00m | eta: 38.2m |
| step 00586/01500 (39.07%) | loss: 2.478761 | lrm: 1.00 | dt: 2497.61ms | tok/sec: 209,915 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 72 | total time: 24.04m | eta: 38.2m |
| step 00587/01500 (39.13%) | loss: 2.517311 | lrm: 1.00 | dt: 2500.22ms | tok/sec: 209,696 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 72 | total time: 24.08m | eta: 38.1m |
| step 00588/01500 (39.20%) | loss: 2.535973 | lrm: 1.00 | dt: 2503.20ms | tok/sec: 209,446 | bf16_mfu: 31.98 | epoch: 1 pq: 6 rg: 72 | total time: 24.13m | eta: 38.1m |
| step 00589/01500 (39.27%) | loss: 2.500935 | lrm: 1.00 | dt: 2501.28ms | tok/sec: 209,608 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 72 | total time: 24.17m | eta: 38.0m |
| step 00590/01500 (39.33%) | loss: 2.467282 | lrm: 1.00 | dt: 2497.19ms | tok/sec: 209,951 | bf16_mfu: 32.05 | epoch: 1 pq: 6 rg: 72 | total time: 24.21m | eta: 38.0m |
| step 00591/01500 (39.40%) | loss: 2.462404 | lrm: 1.00 | dt: 2495.94ms | tok/sec: 210,056 | bf16_mfu: 32.07 | epoch: 1 pq: 6 rg: 72 | total time: 24.25m | eta: 37.9m |
| step 00592/01500 (39.47%) | loss: 2.442026 | lrm: 1.00 | dt: 2499.57ms | tok/sec: 209,751 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 80 | total time: 24.29m | eta: 37.9m |
| step 00593/01500 (39.53%) | loss: 2.445343 | lrm: 1.00 | dt: 2499.51ms | tok/sec: 209,756 | bf16_mfu: 32.02 | epoch: 1 pq: 6 rg: 80 | total time: 24.33m | eta: 37.9m |
| step 00594/01500 (39.60%) | loss: 2.444952 | lrm: 1.00 | dt: 2501.17ms | tok/sec: 209,616 | bf16_mfu: 32.00 | epoch: 1 pq: 6 rg: 80 | total time: 24.38m | eta: 37.8m |
| step 00595/01500 (39.67%) | loss: 2.503910 | lrm: 1.00 | dt: 2505.30ms | tok/sec: 209,271 | bf16_mfu: 31.95 | epoch: 1 pq: 6 rg: 80 | total time: 24.42m | eta: 37.8m |
| step 00596/01500 (39.73%) | loss: 2.517616 | lrm: 1.00 | dt: 2495.79ms | tok/sec: 210,069 | bf16_mfu: 32.07 | epoch: 1 pq: 6 rg: 80 | total time: 24.46m | eta: 37.7m |
| step 00597/01500 (39.80%) | loss: 2.523563 | lrm: 1.00 | dt: 2504.03ms | tok/sec: 209,377 | bf16_mfu: 31.97 | epoch: 1 pq: 6 rg: 80 | total time: 24.50m | eta: 37.7m |
| step 00598/01500 (39.87%) | loss: 2.520262 | lrm: 1.00 | dt: 2498.34ms | tok/sec: 209,854 | bf16_mfu: 32.04 | epoch: 1 pq: 6 rg: 80 | total time: 24.54m | eta: 37.6m |
| step 00599/01500 (39.93%) | loss: 2.546318 | lrm: 1.00 | dt: 2503.60ms | tok/sec: 209,413 | bf16_mfu: 31.97 | epoch: 1 pq: 6 rg: 80 | total time: 24.58m | eta: 37.6m |
| Step 00600 | Validation bpb: 0.751316 |
| Step 00600 | ICL early: 2.7678 late: 2.9164 score: +0.1486 | Induction loss: 10.1360 acc: 0.0637 (random-half: 13.9930) |
| step 00600/01500 (40.00%) | loss: 2.573231 | lrm: 1.00 | dt: 2495.06ms | tok/sec: 210,130 | bf16_mfu: 32.08 | epoch: 1 pq: 7 rg: 0 | total time: 24.63m | eta: 37.6m |
| step 00601/01500 (40.07%) | loss: 2.590959 | lrm: 1.00 | dt: 2499.26ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 0 | total time: 24.67m | eta: 37.5m |
| step 00602/01500 (40.13%) | loss: 2.623053 | lrm: 1.00 | dt: 2497.92ms | tok/sec: 209,890 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 0 | total time: 24.71m | eta: 37.5m |
| step 00603/01500 (40.20%) | loss: 2.657384 | lrm: 1.00 | dt: 2497.67ms | tok/sec: 209,911 | bf16_mfu: 32.05 | epoch: 1 pq: 7 rg: 0 | total time: 24.75m | eta: 37.4m |
| step 00604/01500 (40.27%) | loss: 2.634753 | lrm: 1.00 | dt: 2502.15ms | tok/sec: 209,534 | bf16_mfu: 31.99 | epoch: 1 pq: 7 rg: 0 | total time: 24.79m | eta: 37.4m |
| step 00605/01500 (40.33%) | loss: 2.635326 | lrm: 1.00 | dt: 2498.93ms | tok/sec: 209,804 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 0 | total time: 24.83m | eta: 37.4m |
| step 00606/01500 (40.40%) | loss: 2.613039 | lrm: 1.00 | dt: 2503.49ms | tok/sec: 209,422 | bf16_mfu: 31.97 | epoch: 1 pq: 7 rg: 0 | total time: 24.88m | eta: 37.3m |
| step 00607/01500 (40.47%) | loss: 2.566059 | lrm: 1.00 | dt: 2497.07ms | tok/sec: 209,961 | bf16_mfu: 32.06 | epoch: 1 pq: 7 rg: 0 | total time: 24.92m | eta: 37.3m |
| step 00608/01500 (40.53%) | loss: 2.564715 | lrm: 1.00 | dt: 2498.68ms | tok/sec: 209,825 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 8 | total time: 24.96m | eta: 37.2m |
| step 00609/01500 (40.60%) | loss: 2.534109 | lrm: 1.00 | dt: 2499.28ms | tok/sec: 209,775 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 8 | total time: 25.00m | eta: 37.2m |
| step 00610/01500 (40.67%) | loss: 2.484601 | lrm: 1.00 | dt: 2500.25ms | tok/sec: 209,694 | bf16_mfu: 32.01 | epoch: 1 pq: 7 rg: 8 | total time: 25.04m | eta: 37.1m |
| step 00611/01500 (40.73%) | loss: 2.481609 | lrm: 1.00 | dt: 2499.37ms | tok/sec: 209,767 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 8 | total time: 25.08m | eta: 37.1m |
| step 00612/01500 (40.80%) | loss: 2.523672 | lrm: 1.00 | dt: 2498.71ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 8 | total time: 25.13m | eta: 37.1m |
| step 00613/01500 (40.87%) | loss: 2.490175 | lrm: 1.00 | dt: 2502.44ms | tok/sec: 209,510 | bf16_mfu: 31.99 | epoch: 1 pq: 7 rg: 8 | total time: 25.17m | eta: 37.0m |
| step 00614/01500 (40.93%) | loss: 2.499527 | lrm: 1.00 | dt: 2502.15ms | tok/sec: 209,534 | bf16_mfu: 31.99 | epoch: 1 pq: 7 rg: 8 | total time: 25.21m | eta: 37.0m |
| step 00615/01500 (41.00%) | loss: 2.466889 | lrm: 1.00 | dt: 2503.91ms | tok/sec: 209,387 | bf16_mfu: 31.97 | epoch: 1 pq: 7 rg: 8 | total time: 25.25m | eta: 36.9m |
| step 00616/01500 (41.07%) | loss: 2.544195 | lrm: 1.00 | dt: 2503.06ms | tok/sec: 209,459 | bf16_mfu: 31.98 | epoch: 1 pq: 7 rg: 16 | total time: 25.29m | eta: 36.9m |
| step 00617/01500 (41.13%) | loss: 2.536221 | lrm: 1.00 | dt: 2499.84ms | tok/sec: 209,728 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 16 | total time: 25.33m | eta: 36.9m |
| step 00618/01500 (41.20%) | loss: 2.591457 | lrm: 1.00 | dt: 2498.25ms | tok/sec: 209,861 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 16 | total time: 25.38m | eta: 36.8m |
| step 00619/01500 (41.27%) | loss: 2.605541 | lrm: 1.00 | dt: 2502.96ms | tok/sec: 209,467 | bf16_mfu: 31.98 | epoch: 1 pq: 7 rg: 16 | total time: 25.42m | eta: 36.8m |
| step 00620/01500 (41.33%) | loss: 2.558934 | lrm: 1.00 | dt: 2501.61ms | tok/sec: 209,580 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 16 | total time: 25.46m | eta: 36.7m |
| step 00621/01500 (41.40%) | loss: 2.563063 | lrm: 1.00 | dt: 2645.23ms | tok/sec: 198,201 | bf16_mfu: 30.26 | epoch: 1 pq: 7 rg: 16 | total time: 25.50m | eta: 36.7m |
| step 00622/01500 (41.47%) | loss: 2.549821 | lrm: 1.00 | dt: 2501.61ms | tok/sec: 209,580 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 16 | total time: 25.54m | eta: 36.6m |
| step 00623/01500 (41.53%) | loss: 2.532256 | lrm: 1.00 | dt: 2500.50ms | tok/sec: 209,673 | bf16_mfu: 32.01 | epoch: 1 pq: 7 rg: 16 | total time: 25.59m | eta: 36.6m |
| step 00624/01500 (41.60%) | loss: 2.536579 | lrm: 1.00 | dt: 2506.06ms | tok/sec: 209,207 | bf16_mfu: 31.94 | epoch: 1 pq: 7 rg: 24 | total time: 25.63m | eta: 36.6m |
| step 00625/01500 (41.67%) | loss: 2.582130 | lrm: 1.00 | dt: 2497.51ms | tok/sec: 209,924 | bf16_mfu: 32.05 | epoch: 1 pq: 7 rg: 24 | total time: 25.67m | eta: 36.5m |
| step 00626/01500 (41.73%) | loss: 2.585435 | lrm: 1.00 | dt: 2497.79ms | tok/sec: 209,900 | bf16_mfu: 32.05 | epoch: 1 pq: 7 rg: 24 | total time: 25.71m | eta: 36.5m |
| step 00627/01500 (41.80%) | loss: 2.578101 | lrm: 1.00 | dt: 2499.72ms | tok/sec: 209,739 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 24 | total time: 25.75m | eta: 36.4m |
| step 00628/01500 (41.87%) | loss: 2.586965 | lrm: 1.00 | dt: 2499.31ms | tok/sec: 209,773 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 24 | total time: 25.79m | eta: 36.4m |
| step 00629/01500 (41.93%) | loss: 2.558031 | lrm: 1.00 | dt: 2503.92ms | tok/sec: 209,386 | bf16_mfu: 31.97 | epoch: 1 pq: 7 rg: 24 | total time: 25.84m | eta: 36.4m |
| step 00630/01500 (42.00%) | loss: 2.555793 | lrm: 1.00 | dt: 2499.98ms | tok/sec: 209,717 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 24 | total time: 25.88m | eta: 36.3m |
| step 00631/01500 (42.07%) | loss: 2.562231 | lrm: 1.00 | dt: 2503.98ms | tok/sec: 209,382 | bf16_mfu: 31.97 | epoch: 1 pq: 7 rg: 24 | total time: 25.92m | eta: 36.3m |
| step 00632/01500 (42.13%) | loss: 2.553953 | lrm: 1.00 | dt: 2498.23ms | tok/sec: 209,863 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 24 | total time: 25.96m | eta: 36.2m |
| step 00633/01500 (42.20%) | loss: 2.562541 | lrm: 1.00 | dt: 2496.87ms | tok/sec: 209,978 | bf16_mfu: 32.06 | epoch: 1 pq: 7 rg: 32 | total time: 26.00m | eta: 36.2m |
| step 00634/01500 (42.27%) | loss: 2.567383 | lrm: 1.00 | dt: 2499.62ms | tok/sec: 209,747 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 32 | total time: 26.05m | eta: 36.1m |
| step 00635/01500 (42.33%) | loss: 2.529419 | lrm: 1.00 | dt: 2504.54ms | tok/sec: 209,334 | bf16_mfu: 31.96 | epoch: 1 pq: 7 rg: 32 | total time: 26.09m | eta: 36.1m |
| step 00636/01500 (42.40%) | loss: 2.557127 | lrm: 1.00 | dt: 2500.53ms | tok/sec: 209,670 | bf16_mfu: 32.01 | epoch: 1 pq: 7 rg: 32 | total time: 26.13m | eta: 36.1m |
| step 00637/01500 (42.47%) | loss: 2.555620 | lrm: 1.00 | dt: 2499.21ms | tok/sec: 209,781 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 32 | total time: 26.17m | eta: 36.0m |
| step 00638/01500 (42.53%) | loss: 2.502341 | lrm: 1.00 | dt: 2498.16ms | tok/sec: 209,869 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 32 | total time: 26.21m | eta: 36.0m |
| step 00639/01500 (42.60%) | loss: 2.480235 | lrm: 1.00 | dt: 2497.12ms | tok/sec: 209,957 | bf16_mfu: 32.06 | epoch: 1 pq: 7 rg: 32 | total time: 26.25m | eta: 35.9m |
| step 00640/01500 (42.67%) | loss: 2.470096 | lrm: 1.00 | dt: 2499.12ms | tok/sec: 209,788 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 32 | total time: 26.30m | eta: 35.9m |
| step 00641/01500 (42.73%) | loss: 2.460140 | lrm: 1.00 | dt: 2494.71ms | tok/sec: 210,159 | bf16_mfu: 32.09 | epoch: 1 pq: 7 rg: 40 | total time: 26.34m | eta: 35.9m |
| step 00642/01500 (42.80%) | loss: 2.454784 | lrm: 1.00 | dt: 2506.16ms | tok/sec: 209,199 | bf16_mfu: 31.94 | epoch: 1 pq: 7 rg: 40 | total time: 26.38m | eta: 35.8m |
| step 00643/01500 (42.87%) | loss: 2.453636 | lrm: 1.00 | dt: 2504.26ms | tok/sec: 209,358 | bf16_mfu: 31.96 | epoch: 1 pq: 7 rg: 40 | total time: 26.42m | eta: 35.8m |
| step 00644/01500 (42.93%) | loss: 2.493934 | lrm: 1.00 | dt: 2498.31ms | tok/sec: 209,857 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 40 | total time: 26.46m | eta: 35.7m |
| step 00645/01500 (43.00%) | loss: 2.482507 | lrm: 1.00 | dt: 2497.23ms | tok/sec: 209,948 | bf16_mfu: 32.05 | epoch: 1 pq: 7 rg: 40 | total time: 26.50m | eta: 35.7m |
| step 00646/01500 (43.07%) | loss: 2.500603 | lrm: 1.00 | dt: 2501.29ms | tok/sec: 209,607 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 40 | total time: 26.55m | eta: 35.6m |
| step 00647/01500 (43.13%) | loss: 2.483368 | lrm: 1.00 | dt: 2503.37ms | tok/sec: 209,432 | bf16_mfu: 31.98 | epoch: 1 pq: 7 rg: 40 | total time: 26.59m | eta: 35.6m |
| step 00648/01500 (43.20%) | loss: 2.496198 | lrm: 1.00 | dt: 2500.98ms | tok/sec: 209,632 | bf16_mfu: 32.01 | epoch: 1 pq: 7 rg: 40 | total time: 26.63m | eta: 35.6m |
| step 00649/01500 (43.27%) | loss: 2.470514 | lrm: 1.00 | dt: 2499.53ms | tok/sec: 209,754 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 48 | total time: 26.67m | eta: 35.5m |
| step 00650/01500 (43.33%) | loss: 2.483934 | lrm: 1.00 | dt: 2501.27ms | tok/sec: 209,608 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 48 | total time: 26.71m | eta: 35.5m |
| step 00651/01500 (43.40%) | loss: 2.463478 | lrm: 1.00 | dt: 2505.83ms | tok/sec: 209,227 | bf16_mfu: 31.94 | epoch: 1 pq: 7 rg: 48 | total time: 26.75m | eta: 35.4m |
| step 00652/01500 (43.47%) | loss: 2.462173 | lrm: 1.00 | dt: 2504.54ms | tok/sec: 209,335 | bf16_mfu: 31.96 | epoch: 1 pq: 7 rg: 48 | total time: 26.80m | eta: 35.4m |
| step 00653/01500 (43.53%) | loss: 2.452519 | lrm: 1.00 | dt: 2499.08ms | tok/sec: 209,792 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 48 | total time: 26.84m | eta: 35.4m |
| step 00654/01500 (43.60%) | loss: 2.466888 | lrm: 1.00 | dt: 2501.52ms | tok/sec: 209,587 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 48 | total time: 26.88m | eta: 35.3m |
| step 00655/01500 (43.67%) | loss: 2.389489 | lrm: 1.00 | dt: 2502.10ms | tok/sec: 209,539 | bf16_mfu: 31.99 | epoch: 1 pq: 7 rg: 48 | total time: 26.92m | eta: 35.3m |
| step 00656/01500 (43.73%) | loss: 2.398192 | lrm: 1.00 | dt: 2500.36ms | tok/sec: 209,685 | bf16_mfu: 32.01 | epoch: 1 pq: 7 rg: 48 | total time: 26.96m | eta: 35.2m |
| step 00657/01500 (43.80%) | loss: 2.424800 | lrm: 1.00 | dt: 2498.61ms | tok/sec: 209,832 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 56 | total time: 27.00m | eta: 35.2m |
| step 00658/01500 (43.87%) | loss: 2.439903 | lrm: 1.00 | dt: 2501.29ms | tok/sec: 209,606 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 56 | total time: 27.05m | eta: 35.1m |
| step 00659/01500 (43.93%) | loss: 2.458521 | lrm: 1.00 | dt: 2500.78ms | tok/sec: 209,649 | bf16_mfu: 32.01 | epoch: 1 pq: 7 rg: 56 | total time: 27.09m | eta: 35.1m |
| step 00660/01500 (44.00%) | loss: 2.462703 | lrm: 1.00 | dt: 2509.84ms | tok/sec: 208,892 | bf16_mfu: 31.89 | epoch: 1 pq: 7 rg: 56 | total time: 27.13m | eta: 35.1m |
| step 00661/01500 (44.07%) | loss: 2.472215 | lrm: 1.00 | dt: 2502.80ms | tok/sec: 209,480 | bf16_mfu: 31.98 | epoch: 1 pq: 7 rg: 56 | total time: 27.17m | eta: 35.0m |
| step 00662/01500 (44.13%) | loss: 2.471677 | lrm: 1.00 | dt: 2504.38ms | tok/sec: 209,348 | bf16_mfu: 31.96 | epoch: 1 pq: 7 rg: 56 | total time: 27.21m | eta: 35.0m |
| step 00663/01500 (44.20%) | loss: 2.462579 | lrm: 1.00 | dt: 2502.03ms | tok/sec: 209,544 | bf16_mfu: 31.99 | epoch: 1 pq: 7 rg: 56 | total time: 27.25m | eta: 34.9m |
| step 00664/01500 (44.27%) | loss: 2.475935 | lrm: 1.00 | dt: 2501.32ms | tok/sec: 209,604 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 56 | total time: 27.30m | eta: 34.9m |
| step 00665/01500 (44.33%) | loss: 2.480156 | lrm: 1.00 | dt: 2495.91ms | tok/sec: 210,058 | bf16_mfu: 32.07 | epoch: 1 pq: 7 rg: 64 | total time: 27.34m | eta: 34.8m |
| step 00666/01500 (44.40%) | loss: 2.508779 | lrm: 1.00 | dt: 2496.96ms | tok/sec: 209,970 | bf16_mfu: 32.06 | epoch: 1 pq: 7 rg: 64 | total time: 27.38m | eta: 34.8m |
| step 00667/01500 (44.47%) | loss: 2.471590 | lrm: 1.00 | dt: 2506.19ms | tok/sec: 209,197 | bf16_mfu: 31.94 | epoch: 1 pq: 7 rg: 64 | total time: 27.42m | eta: 34.8m |
| step 00668/01500 (44.53%) | loss: 2.491536 | lrm: 1.00 | dt: 2499.73ms | tok/sec: 209,738 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 64 | total time: 27.46m | eta: 34.7m |
| step 00669/01500 (44.60%) | loss: 2.513735 | lrm: 1.00 | dt: 2501.13ms | tok/sec: 209,620 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 64 | total time: 27.50m | eta: 34.7m |
| step 00670/01500 (44.67%) | loss: 2.491293 | lrm: 1.00 | dt: 2498.09ms | tok/sec: 209,875 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 64 | total time: 27.55m | eta: 34.6m |
| step 00671/01500 (44.73%) | loss: 2.493976 | lrm: 1.00 | dt: 2495.46ms | tok/sec: 210,096 | bf16_mfu: 32.08 | epoch: 1 pq: 7 rg: 64 | total time: 27.59m | eta: 34.6m |
| step 00672/01500 (44.80%) | loss: 2.483517 | lrm: 1.00 | dt: 2496.53ms | tok/sec: 210,006 | bf16_mfu: 32.06 | epoch: 1 pq: 7 rg: 64 | total time: 27.63m | eta: 34.6m |
| step 00673/01500 (44.87%) | loss: 2.500287 | lrm: 1.00 | dt: 2499.84ms | tok/sec: 209,728 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 72 | total time: 27.67m | eta: 34.5m |
| step 00674/01500 (44.93%) | loss: 2.473263 | lrm: 1.00 | dt: 2500.21ms | tok/sec: 209,697 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 72 | total time: 27.71m | eta: 34.5m |
| step 00675/01500 (45.00%) | loss: 2.487736 | lrm: 1.00 | dt: 2500.47ms | tok/sec: 209,675 | bf16_mfu: 32.01 | epoch: 1 pq: 7 rg: 72 | total time: 27.75m | eta: 34.4m |
| step 00676/01500 (45.07%) | loss: 2.477532 | lrm: 1.00 | dt: 2504.80ms | tok/sec: 209,313 | bf16_mfu: 31.96 | epoch: 1 pq: 7 rg: 72 | total time: 27.80m | eta: 34.4m |
| step 00677/01500 (45.13%) | loss: 2.454688 | lrm: 1.00 | dt: 2499.50ms | tok/sec: 209,757 | bf16_mfu: 32.02 | epoch: 1 pq: 7 rg: 72 | total time: 27.84m | eta: 34.3m |
| step 00678/01500 (45.20%) | loss: 2.430004 | lrm: 1.00 | dt: 2498.98ms | tok/sec: 209,800 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 72 | total time: 27.88m | eta: 34.3m |
| step 00679/01500 (45.27%) | loss: 2.466982 | lrm: 1.00 | dt: 2505.25ms | tok/sec: 209,276 | bf16_mfu: 31.95 | epoch: 1 pq: 7 rg: 72 | total time: 27.92m | eta: 34.3m |
| step 00680/01500 (45.33%) | loss: 2.479966 | lrm: 1.00 | dt: 2495.83ms | tok/sec: 210,066 | bf16_mfu: 32.07 | epoch: 1 pq: 7 rg: 72 | total time: 27.96m | eta: 34.2m |
| step 00681/01500 (45.40%) | loss: 2.485721 | lrm: 1.00 | dt: 2498.94ms | tok/sec: 209,804 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 80 | total time: 28.00m | eta: 34.2m |
| step 00682/01500 (45.47%) | loss: 2.496000 | lrm: 1.00 | dt: 2501.68ms | tok/sec: 209,574 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 80 | total time: 28.05m | eta: 34.1m |
| step 00683/01500 (45.53%) | loss: 2.475892 | lrm: 1.00 | dt: 2499.28ms | tok/sec: 209,775 | bf16_mfu: 32.03 | epoch: 1 pq: 7 rg: 80 | total time: 28.09m | eta: 34.1m |
| step 00684/01500 (45.60%) | loss: 2.461710 | lrm: 1.00 | dt: 2503.92ms | tok/sec: 209,386 | bf16_mfu: 31.97 | epoch: 1 pq: 7 rg: 80 | total time: 28.13m | eta: 34.1m |
| step 00685/01500 (45.67%) | loss: 2.436354 | lrm: 1.00 | dt: 2495.77ms | tok/sec: 210,070 | bf16_mfu: 32.07 | epoch: 1 pq: 7 rg: 80 | total time: 28.17m | eta: 34.0m |
| step 00686/01500 (45.73%) | loss: 2.415804 | lrm: 1.00 | dt: 2501.18ms | tok/sec: 209,616 | bf16_mfu: 32.00 | epoch: 1 pq: 7 rg: 80 | total time: 28.21m | eta: 34.0m |
| step 00687/01500 (45.80%) | loss: 2.399857 | lrm: 1.00 | dt: 2498.45ms | tok/sec: 209,845 | bf16_mfu: 32.04 | epoch: 1 pq: 7 rg: 80 | total time: 28.25m | eta: 33.9m |
| step 00688/01500 (45.87%) | loss: 2.421625 | lrm: 1.00 | dt: 2502.31ms | tok/sec: 209,521 | bf16_mfu: 31.99 | epoch: 1 pq: 7 rg: 80 | total time: 28.30m | eta: 33.9m |
| step 00689/01500 (45.93%) | loss: 2.439345 | lrm: 1.00 | dt: 2501.34ms | tok/sec: 209,602 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 0 | total time: 28.34m | eta: 33.8m |
| step 00690/01500 (46.00%) | loss: 2.447331 | lrm: 1.00 | dt: 2498.95ms | tok/sec: 209,803 | bf16_mfu: 32.03 | epoch: 1 pq: 8 rg: 0 | total time: 28.38m | eta: 33.8m |
| step 00691/01500 (46.07%) | loss: 2.425838 | lrm: 1.00 | dt: 2500.67ms | tok/sec: 209,658 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 0 | total time: 28.42m | eta: 33.8m |
| step 00692/01500 (46.13%) | loss: 2.469981 | lrm: 1.00 | dt: 2496.11ms | tok/sec: 210,041 | bf16_mfu: 32.07 | epoch: 1 pq: 8 rg: 0 | total time: 28.46m | eta: 33.7m |
| step 00693/01500 (46.20%) | loss: 2.510274 | lrm: 1.00 | dt: 2501.90ms | tok/sec: 209,555 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 0 | total time: 28.50m | eta: 33.7m |
| step 00694/01500 (46.27%) | loss: 2.501991 | lrm: 1.00 | dt: 2503.50ms | tok/sec: 209,422 | bf16_mfu: 31.97 | epoch: 1 pq: 8 rg: 0 | total time: 28.55m | eta: 33.6m |
| step 00695/01500 (46.33%) | loss: 2.533423 | lrm: 1.00 | dt: 2502.78ms | tok/sec: 209,482 | bf16_mfu: 31.98 | epoch: 1 pq: 8 rg: 0 | total time: 28.59m | eta: 33.6m |
| step 00696/01500 (46.40%) | loss: 2.516100 | lrm: 1.00 | dt: 2501.67ms | tok/sec: 209,575 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 8 | total time: 28.63m | eta: 33.6m |
| step 00697/01500 (46.47%) | loss: 2.490330 | lrm: 1.00 | dt: 2502.44ms | tok/sec: 209,510 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 8 | total time: 28.67m | eta: 33.5m |
| step 00698/01500 (46.53%) | loss: 2.467871 | lrm: 1.00 | dt: 2501.35ms | tok/sec: 209,601 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 8 | total time: 28.71m | eta: 33.5m |
| step 00699/01500 (46.60%) | loss: 2.510379 | lrm: 1.00 | dt: 2501.73ms | tok/sec: 209,570 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 8 | total time: 28.75m | eta: 33.4m |
| Step 00700 | Validation bpb: 0.745837 |
| Step 00700 | ICL early: 2.7512 late: 2.9013 score: +0.1501 | Induction loss: 9.7585 acc: 0.0784 (random-half: 13.9678) |
| step 00700/01500 (46.67%) | loss: 2.376646 | lrm: 1.00 | dt: 2498.71ms | tok/sec: 209,823 | bf16_mfu: 32.03 | epoch: 1 pq: 8 rg: 8 | total time: 28.80m | eta: 33.4m |
| step 00701/01500 (46.73%) | loss: 2.384216 | lrm: 1.00 | dt: 2501.50ms | tok/sec: 209,589 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 8 | total time: 28.84m | eta: 33.3m |
| step 00702/01500 (46.80%) | loss: 2.357306 | lrm: 1.00 | dt: 2856.88ms | tok/sec: 183,517 | bf16_mfu: 28.02 | epoch: 1 pq: 8 rg: 8 | total time: 28.89m | eta: 33.3m |
| step 00703/01500 (46.87%) | loss: 2.364936 | lrm: 1.00 | dt: 2500.41ms | tok/sec: 209,680 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 8 | total time: 28.93m | eta: 33.3m |
| step 00704/01500 (46.93%) | loss: 2.426741 | lrm: 1.00 | dt: 2499.82ms | tok/sec: 209,730 | bf16_mfu: 32.02 | epoch: 1 pq: 8 rg: 8 | total time: 28.97m | eta: 33.2m |
| step 00705/01500 (47.00%) | loss: 2.429270 | lrm: 1.00 | dt: 2499.99ms | tok/sec: 209,716 | bf16_mfu: 32.02 | epoch: 1 pq: 8 rg: 16 | total time: 29.01m | eta: 33.2m |
| step 00706/01500 (47.07%) | loss: 2.431152 | lrm: 1.00 | dt: 2499.87ms | tok/sec: 209,726 | bf16_mfu: 32.02 | epoch: 1 pq: 8 rg: 16 | total time: 29.05m | eta: 33.1m |
| step 00707/01500 (47.13%) | loss: 2.440887 | lrm: 1.00 | dt: 2501.34ms | tok/sec: 209,603 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 16 | total time: 29.09m | eta: 33.1m |
| step 00708/01500 (47.20%) | loss: 2.427637 | lrm: 1.00 | dt: 2502.40ms | tok/sec: 209,514 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 16 | total time: 29.14m | eta: 33.1m |
| step 00709/01500 (47.27%) | loss: 2.426681 | lrm: 1.00 | dt: 2507.04ms | tok/sec: 209,125 | bf16_mfu: 31.93 | epoch: 1 pq: 8 rg: 16 | total time: 29.18m | eta: 33.0m |
| step 00710/01500 (47.33%) | loss: 2.447408 | lrm: 1.00 | dt: 2499.23ms | tok/sec: 209,780 | bf16_mfu: 32.03 | epoch: 1 pq: 8 rg: 16 | total time: 29.22m | eta: 33.0m |
| step 00711/01500 (47.40%) | loss: 2.485365 | lrm: 1.00 | dt: 2502.19ms | tok/sec: 209,531 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 16 | total time: 29.26m | eta: 32.9m |
| step 00712/01500 (47.47%) | loss: 2.525821 | lrm: 1.00 | dt: 2506.10ms | tok/sec: 209,204 | bf16_mfu: 31.94 | epoch: 1 pq: 8 rg: 24 | total time: 29.30m | eta: 32.9m |
| step 00713/01500 (47.53%) | loss: 2.563213 | lrm: 1.00 | dt: 2503.93ms | tok/sec: 209,385 | bf16_mfu: 31.97 | epoch: 1 pq: 8 rg: 24 | total time: 29.34m | eta: 32.9m |
| step 00714/01500 (47.60%) | loss: 2.450297 | lrm: 1.00 | dt: 2497.82ms | tok/sec: 209,898 | bf16_mfu: 32.05 | epoch: 1 pq: 8 rg: 24 | total time: 29.39m | eta: 32.8m |
| step 00715/01500 (47.67%) | loss: 2.448795 | lrm: 1.00 | dt: 2497.33ms | tok/sec: 209,939 | bf16_mfu: 32.05 | epoch: 1 pq: 8 rg: 24 | total time: 29.43m | eta: 32.8m |
| step 00716/01500 (47.73%) | loss: 2.484999 | lrm: 1.00 | dt: 2501.28ms | tok/sec: 209,607 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 24 | total time: 29.47m | eta: 32.7m |
| step 00717/01500 (47.80%) | loss: 2.458955 | lrm: 1.00 | dt: 2502.92ms | tok/sec: 209,470 | bf16_mfu: 31.98 | epoch: 1 pq: 8 rg: 24 | total time: 29.51m | eta: 32.7m |
| step 00718/01500 (47.87%) | loss: 2.459789 | lrm: 1.00 | dt: 2496.35ms | tok/sec: 210,021 | bf16_mfu: 32.06 | epoch: 1 pq: 8 rg: 24 | total time: 29.55m | eta: 32.6m |
| step 00719/01500 (47.93%) | loss: 2.461536 | lrm: 1.00 | dt: 2501.18ms | tok/sec: 209,615 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 24 | total time: 29.59m | eta: 32.6m |
| step 00720/01500 (48.00%) | loss: 2.508784 | lrm: 1.00 | dt: 2502.23ms | tok/sec: 209,527 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 24 | total time: 29.64m | eta: 32.6m |
| step 00721/01500 (48.07%) | loss: 2.546022 | lrm: 1.00 | dt: 2499.04ms | tok/sec: 209,795 | bf16_mfu: 32.03 | epoch: 1 pq: 8 rg: 32 | total time: 29.68m | eta: 32.5m |
| step 00722/01500 (48.13%) | loss: 2.569562 | lrm: 1.00 | dt: 2496.47ms | tok/sec: 210,011 | bf16_mfu: 32.06 | epoch: 1 pq: 8 rg: 32 | total time: 29.72m | eta: 32.5m |
| step 00723/01500 (48.20%) | loss: 2.549892 | lrm: 1.00 | dt: 2498.75ms | tok/sec: 209,820 | bf16_mfu: 32.03 | epoch: 1 pq: 8 rg: 32 | total time: 29.76m | eta: 32.4m |
| step 00724/01500 (48.27%) | loss: 2.553147 | lrm: 1.00 | dt: 2497.75ms | tok/sec: 209,904 | bf16_mfu: 32.05 | epoch: 1 pq: 8 rg: 32 | total time: 29.80m | eta: 32.4m |
| step 00725/01500 (48.33%) | loss: 2.513919 | lrm: 1.00 | dt: 2502.86ms | tok/sec: 209,475 | bf16_mfu: 31.98 | epoch: 1 pq: 8 rg: 32 | total time: 29.84m | eta: 32.3m |
| step 00726/01500 (48.40%) | loss: 2.511291 | lrm: 1.00 | dt: 2501.91ms | tok/sec: 209,555 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 32 | total time: 29.89m | eta: 32.3m |
| step 00727/01500 (48.47%) | loss: 2.526622 | lrm: 1.00 | dt: 2502.54ms | tok/sec: 209,502 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 32 | total time: 29.93m | eta: 32.3m |
| step 00728/01500 (48.53%) | loss: 2.583192 | lrm: 1.00 | dt: 2501.89ms | tok/sec: 209,556 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 40 | total time: 29.97m | eta: 32.2m |
| step 00729/01500 (48.60%) | loss: 2.570209 | lrm: 1.00 | dt: 2502.26ms | tok/sec: 209,525 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 40 | total time: 30.01m | eta: 32.2m |
| step 00730/01500 (48.67%) | loss: 2.555771 | lrm: 1.00 | dt: 2501.04ms | tok/sec: 209,627 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 40 | total time: 30.05m | eta: 32.1m |
| step 00731/01500 (48.73%) | loss: 2.594396 | lrm: 1.00 | dt: 2498.95ms | tok/sec: 209,803 | bf16_mfu: 32.03 | epoch: 1 pq: 8 rg: 40 | total time: 30.09m | eta: 32.1m |
| step 00732/01500 (48.80%) | loss: 2.598548 | lrm: 1.00 | dt: 2499.45ms | tok/sec: 209,761 | bf16_mfu: 32.03 | epoch: 1 pq: 8 rg: 40 | total time: 30.14m | eta: 32.1m |
| step 00733/01500 (48.87%) | loss: 2.575133 | lrm: 1.00 | dt: 2503.75ms | tok/sec: 209,401 | bf16_mfu: 31.97 | epoch: 1 pq: 8 rg: 40 | total time: 30.18m | eta: 32.0m |
| step 00734/01500 (48.93%) | loss: 2.570469 | lrm: 1.00 | dt: 2498.43ms | tok/sec: 209,847 | bf16_mfu: 32.04 | epoch: 1 pq: 8 rg: 40 | total time: 30.22m | eta: 32.0m |
| step 00735/01500 (49.00%) | loss: 2.577838 | lrm: 1.00 | dt: 2497.93ms | tok/sec: 209,889 | bf16_mfu: 32.04 | epoch: 1 pq: 8 rg: 40 | total time: 30.26m | eta: 31.9m |
| step 00736/01500 (49.07%) | loss: 2.562257 | lrm: 1.00 | dt: 2501.84ms | tok/sec: 209,560 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 48 | total time: 30.30m | eta: 31.9m |
| step 00737/01500 (49.13%) | loss: 2.512280 | lrm: 1.00 | dt: 2497.68ms | tok/sec: 209,910 | bf16_mfu: 32.05 | epoch: 1 pq: 8 rg: 48 | total time: 30.34m | eta: 31.8m |
| step 00738/01500 (49.20%) | loss: 2.525939 | lrm: 1.00 | dt: 2500.16ms | tok/sec: 209,701 | bf16_mfu: 32.02 | epoch: 1 pq: 8 rg: 48 | total time: 30.39m | eta: 31.8m |
| step 00739/01500 (49.27%) | loss: 2.487640 | lrm: 1.00 | dt: 2500.39ms | tok/sec: 209,682 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 48 | total time: 30.43m | eta: 31.8m |
| step 00740/01500 (49.33%) | loss: 2.484023 | lrm: 1.00 | dt: 2503.10ms | tok/sec: 209,455 | bf16_mfu: 31.98 | epoch: 1 pq: 8 rg: 48 | total time: 30.47m | eta: 31.7m |
| step 00741/01500 (49.40%) | loss: 2.495674 | lrm: 1.00 | dt: 2505.57ms | tok/sec: 209,249 | bf16_mfu: 31.95 | epoch: 1 pq: 8 rg: 48 | total time: 30.51m | eta: 31.7m |
| step 00742/01500 (49.47%) | loss: 2.493286 | lrm: 1.00 | dt: 2500.44ms | tok/sec: 209,678 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 48 | total time: 30.55m | eta: 31.6m |
| step 00743/01500 (49.53%) | loss: 2.512838 | lrm: 1.00 | dt: 2506.23ms | tok/sec: 209,193 | bf16_mfu: 31.94 | epoch: 1 pq: 8 rg: 48 | total time: 30.59m | eta: 31.6m |
| step 00744/01500 (49.60%) | loss: 2.526144 | lrm: 1.00 | dt: 2499.91ms | tok/sec: 209,722 | bf16_mfu: 32.02 | epoch: 1 pq: 8 rg: 48 | total time: 30.64m | eta: 31.6m |
| step 00745/01500 (49.67%) | loss: 2.485605 | lrm: 1.00 | dt: 2502.44ms | tok/sec: 209,511 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 56 | total time: 30.68m | eta: 31.5m |
| step 00746/01500 (49.73%) | loss: 2.502808 | lrm: 1.00 | dt: 2496.03ms | tok/sec: 210,048 | bf16_mfu: 32.07 | epoch: 1 pq: 8 rg: 56 | total time: 30.72m | eta: 31.5m |
| step 00747/01500 (49.80%) | loss: 2.541489 | lrm: 1.00 | dt: 2500.48ms | tok/sec: 209,674 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 56 | total time: 30.76m | eta: 31.4m |
| step 00748/01500 (49.87%) | loss: 2.580649 | lrm: 1.00 | dt: 2500.65ms | tok/sec: 209,660 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 56 | total time: 30.80m | eta: 31.4m |
| step 00749/01500 (49.93%) | loss: 2.583109 | lrm: 1.00 | dt: 2500.45ms | tok/sec: 209,677 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 56 | total time: 30.84m | eta: 31.3m |
| <|bos|>The capital of France is Paris. The city is the capital of France and is the capital of the Î |
| <|bos|>The chemical symbol of gold is Au. It is a chemical element with the symbol Au. It is a chemical |
| <|bos|>If yesterday was Friday, then tomorrow will be Saturday. If today was Saturday, then tomorrow will be Sunday. If today was |
| <|bos|>The opposite of hot is cold. Hot is a state of high temperature, while cold is a state of |
| <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, |
| <|bos|>My favorite color is blue. I love the sky and the ocean. I love the ocean because it |
| <|bos|>If 5*x + 3 = 13, then x is 3. If 5*x + 3 = 13, then |
| 2026-08-10 22:10:01,490 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank1.pt |
| 2026-08-10 22:10:14,777 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank6.pt |
| 2026-08-10 22:10:18,203 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank3.pt |
| 2026-08-10 22:10:19,319 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank7.pt |
| 2026-08-10 22:10:20,280 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank2.pt |
| 2026-08-10 22:10:23,249 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank4.pt |
| 2026-08-10 22:10:24,164 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank5.pt |
| 2026-08-10 22:11:24,553 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved model parameters to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/model_000750.pt |
| 2026-08-10 22:11:24,556 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved metadata to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/meta_000750.json |
| 2026-08-10 22:11:31,788 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_000750_rank0.pt |
| step 00750/01500 (50.00%) | loss: 2.613416 | lrm: 1.00 | dt: 2475.46ms | tok/sec: 211,794 | bf16_mfu: 32.34 | epoch: 1 pq: 8 rg: 56 | total time: 30.89m | eta: 31.3m |
| step 00751/01500 (50.07%) | loss: 2.575135 | lrm: 1.00 | dt: 2483.76ms | tok/sec: 211,086 | bf16_mfu: 32.23 | epoch: 1 pq: 8 rg: 56 | total time: 30.93m | eta: 31.3m |
| step 00752/01500 (50.13%) | loss: 2.561207 | lrm: 1.00 | dt: 2488.51ms | tok/sec: 210,683 | bf16_mfu: 32.17 | epoch: 1 pq: 8 rg: 56 | total time: 30.97m | eta: 31.2m |
| step 00753/01500 (50.20%) | loss: 2.551033 | lrm: 1.00 | dt: 2489.22ms | tok/sec: 210,623 | bf16_mfu: 32.16 | epoch: 1 pq: 8 rg: 64 | total time: 31.01m | eta: 31.2m |
| step 00754/01500 (50.27%) | loss: 2.522961 | lrm: 1.00 | dt: 2494.46ms | tok/sec: 210,181 | bf16_mfu: 32.09 | epoch: 1 pq: 8 rg: 64 | total time: 31.05m | eta: 31.1m |
| step 00755/01500 (50.33%) | loss: 2.514381 | lrm: 1.00 | dt: 2496.97ms | tok/sec: 209,969 | bf16_mfu: 32.06 | epoch: 1 pq: 8 rg: 64 | total time: 31.09m | eta: 31.1m |
| step 00756/01500 (50.40%) | loss: 2.539011 | lrm: 1.00 | dt: 2498.62ms | tok/sec: 209,831 | bf16_mfu: 32.04 | epoch: 1 pq: 8 rg: 64 | total time: 31.13m | eta: 31.1m |
| step 00757/01500 (50.47%) | loss: 2.512881 | lrm: 1.00 | dt: 2496.05ms | tok/sec: 210,046 | bf16_mfu: 32.07 | epoch: 1 pq: 8 rg: 64 | total time: 31.18m | eta: 31.0m |
| step 00758/01500 (50.53%) | loss: 2.500824 | lrm: 1.00 | dt: 2506.59ms | tok/sec: 209,164 | bf16_mfu: 31.93 | epoch: 1 pq: 8 rg: 64 | total time: 31.22m | eta: 31.0m |
| step 00759/01500 (50.60%) | loss: 2.456874 | lrm: 1.00 | dt: 2510.10ms | tok/sec: 208,870 | bf16_mfu: 31.89 | epoch: 1 pq: 8 rg: 64 | total time: 31.26m | eta: 30.9m |
| step 00760/01500 (50.67%) | loss: 2.449276 | lrm: 1.00 | dt: 2505.61ms | tok/sec: 209,245 | bf16_mfu: 31.95 | epoch: 1 pq: 8 rg: 64 | total time: 31.30m | eta: 30.9m |
| step 00761/01500 (50.73%) | loss: 2.460873 | lrm: 1.00 | dt: 2511.35ms | tok/sec: 208,767 | bf16_mfu: 31.87 | epoch: 1 pq: 8 rg: 72 | total time: 31.34m | eta: 30.8m |
| step 00762/01500 (50.80%) | loss: 2.463210 | lrm: 1.00 | dt: 2503.80ms | tok/sec: 209,397 | bf16_mfu: 31.97 | epoch: 1 pq: 8 rg: 72 | total time: 31.39m | eta: 30.8m |
| step 00763/01500 (50.87%) | loss: 2.438271 | lrm: 1.00 | dt: 2504.37ms | tok/sec: 209,349 | bf16_mfu: 31.96 | epoch: 1 pq: 8 rg: 72 | total time: 31.43m | eta: 30.8m |
| step 00764/01500 (50.93%) | loss: 2.398702 | lrm: 1.00 | dt: 2511.76ms | tok/sec: 208,733 | bf16_mfu: 31.87 | epoch: 1 pq: 8 rg: 72 | total time: 31.47m | eta: 30.7m |
| step 00765/01500 (51.00%) | loss: 2.378540 | lrm: 1.00 | dt: 2504.58ms | tok/sec: 209,331 | bf16_mfu: 31.96 | epoch: 1 pq: 8 rg: 72 | total time: 31.51m | eta: 30.7m |
| step 00766/01500 (51.07%) | loss: 2.396469 | lrm: 1.00 | dt: 2506.18ms | tok/sec: 209,198 | bf16_mfu: 31.94 | epoch: 1 pq: 8 rg: 72 | total time: 31.55m | eta: 30.6m |
| step 00767/01500 (51.13%) | loss: 2.431147 | lrm: 1.00 | dt: 2501.81ms | tok/sec: 209,563 | bf16_mfu: 32.00 | epoch: 1 pq: 8 rg: 72 | total time: 31.59m | eta: 30.6m |
| step 00768/01500 (51.20%) | loss: 2.426193 | lrm: 1.00 | dt: 2500.47ms | tok/sec: 209,675 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 72 | total time: 31.64m | eta: 30.6m |
| step 00769/01500 (51.27%) | loss: 2.418622 | lrm: 1.00 | dt: 2503.90ms | tok/sec: 209,388 | bf16_mfu: 31.97 | epoch: 1 pq: 8 rg: 80 | total time: 31.68m | eta: 30.5m |
| step 00770/01500 (51.33%) | loss: 2.455873 | lrm: 1.00 | dt: 2497.98ms | tok/sec: 209,884 | bf16_mfu: 32.04 | epoch: 1 pq: 8 rg: 80 | total time: 31.72m | eta: 30.5m |
| step 00771/01500 (51.40%) | loss: 2.489749 | lrm: 1.00 | dt: 2500.25ms | tok/sec: 209,694 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 80 | total time: 31.76m | eta: 30.4m |
| step 00772/01500 (51.47%) | loss: 2.477514 | lrm: 1.00 | dt: 2502.03ms | tok/sec: 209,544 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 80 | total time: 31.80m | eta: 30.4m |
| step 00773/01500 (51.53%) | loss: 2.490943 | lrm: 1.00 | dt: 2499.93ms | tok/sec: 209,721 | bf16_mfu: 32.02 | epoch: 1 pq: 8 rg: 80 | total time: 31.84m | eta: 30.3m |
| step 00774/01500 (51.60%) | loss: 2.454402 | lrm: 1.00 | dt: 2500.35ms | tok/sec: 209,685 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 80 | total time: 31.89m | eta: 30.3m |
| step 00775/01500 (51.67%) | loss: 2.453161 | lrm: 1.00 | dt: 2500.67ms | tok/sec: 209,659 | bf16_mfu: 32.01 | epoch: 1 pq: 8 rg: 80 | total time: 31.93m | eta: 30.3m |
| step 00776/01500 (51.73%) | loss: 2.457556 | lrm: 1.00 | dt: 2502.05ms | tok/sec: 209,543 | bf16_mfu: 31.99 | epoch: 1 pq: 8 rg: 80 | total time: 31.97m | eta: 30.2m |
| step 00777/01500 (51.80%) | loss: 2.470391 | lrm: 1.00 | dt: 2496.88ms | tok/sec: 209,976 | bf16_mfu: 32.06 | epoch: 1 pq: 9 rg: 0 | total time: 32.01m | eta: 30.2m |
| step 00778/01500 (51.87%) | loss: 2.436660 | lrm: 1.00 | dt: 2500.80ms | tok/sec: 209,648 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 0 | total time: 32.05m | eta: 30.1m |
| step 00779/01500 (51.93%) | loss: 2.393297 | lrm: 1.00 | dt: 2497.66ms | tok/sec: 209,912 | bf16_mfu: 32.05 | epoch: 1 pq: 9 rg: 0 | total time: 32.09m | eta: 30.1m |
| step 00780/01500 (52.00%) | loss: 2.393302 | lrm: 1.00 | dt: 2496.59ms | tok/sec: 210,001 | bf16_mfu: 32.06 | epoch: 1 pq: 9 rg: 0 | total time: 32.14m | eta: 30.0m |
| step 00781/01500 (52.07%) | loss: 2.438059 | lrm: 1.00 | dt: 2499.50ms | tok/sec: 209,757 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 0 | total time: 32.18m | eta: 30.0m |
| step 00782/01500 (52.13%) | loss: 2.459424 | lrm: 1.00 | dt: 2503.23ms | tok/sec: 209,444 | bf16_mfu: 31.98 | epoch: 1 pq: 9 rg: 0 | total time: 32.22m | eta: 30.0m |
| step 00783/01500 (52.20%) | loss: 2.507362 | lrm: 1.00 | dt: 2499.13ms | tok/sec: 209,788 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 0 | total time: 32.26m | eta: 29.9m |
| step 00784/01500 (52.27%) | loss: 2.479795 | lrm: 1.00 | dt: 2796.38ms | tok/sec: 187,487 | bf16_mfu: 28.62 | epoch: 1 pq: 9 rg: 0 | total time: 32.31m | eta: 29.9m |
| step 00785/01500 (52.33%) | loss: 2.443444 | lrm: 1.00 | dt: 2500.97ms | tok/sec: 209,634 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 8 | total time: 32.35m | eta: 29.8m |
| step 00786/01500 (52.40%) | loss: 2.376552 | lrm: 1.00 | dt: 2499.26ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 8 | total time: 32.39m | eta: 29.8m |
| step 00787/01500 (52.47%) | loss: 2.346382 | lrm: 1.00 | dt: 2496.73ms | tok/sec: 209,989 | bf16_mfu: 32.06 | epoch: 1 pq: 9 rg: 8 | total time: 32.43m | eta: 29.8m |
| step 00788/01500 (52.53%) | loss: 2.339653 | lrm: 1.00 | dt: 2501.14ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 9 rg: 8 | total time: 32.47m | eta: 29.7m |
| step 00789/01500 (52.60%) | loss: 2.385250 | lrm: 1.00 | dt: 2503.26ms | tok/sec: 209,441 | bf16_mfu: 31.98 | epoch: 1 pq: 9 rg: 8 | total time: 32.52m | eta: 29.7m |
| step 00790/01500 (52.67%) | loss: 2.442078 | lrm: 1.00 | dt: 2499.15ms | tok/sec: 209,786 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 8 | total time: 32.56m | eta: 29.6m |
| step 00791/01500 (52.73%) | loss: 2.409874 | lrm: 1.00 | dt: 2499.38ms | tok/sec: 209,767 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 8 | total time: 32.60m | eta: 29.6m |
| step 00792/01500 (52.80%) | loss: 2.374510 | lrm: 1.00 | dt: 2498.75ms | tok/sec: 209,820 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 8 | total time: 32.64m | eta: 29.6m |
| step 00793/01500 (52.87%) | loss: 2.398126 | lrm: 1.00 | dt: 2498.92ms | tok/sec: 209,805 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 16 | total time: 32.68m | eta: 29.5m |
| step 00794/01500 (52.93%) | loss: 2.397676 | lrm: 1.00 | dt: 2499.77ms | tok/sec: 209,734 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 16 | total time: 32.72m | eta: 29.5m |
| step 00795/01500 (53.00%) | loss: 2.424432 | lrm: 1.00 | dt: 2500.29ms | tok/sec: 209,690 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 16 | total time: 32.77m | eta: 29.4m |
| step 00796/01500 (53.07%) | loss: 2.421531 | lrm: 1.00 | dt: 2500.98ms | tok/sec: 209,633 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 16 | total time: 32.81m | eta: 29.4m |
| step 00797/01500 (53.13%) | loss: 2.430554 | lrm: 1.00 | dt: 2500.53ms | tok/sec: 209,670 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 16 | total time: 32.85m | eta: 29.3m |
| step 00798/01500 (53.20%) | loss: 2.479384 | lrm: 1.00 | dt: 2496.81ms | tok/sec: 209,982 | bf16_mfu: 32.06 | epoch: 1 pq: 9 rg: 16 | total time: 32.89m | eta: 29.3m |
| step 00799/01500 (53.27%) | loss: 2.448715 | lrm: 1.00 | dt: 2501.10ms | tok/sec: 209,623 | bf16_mfu: 32.00 | epoch: 1 pq: 9 rg: 16 | total time: 32.93m | eta: 29.3m |
| Step 00800 | Validation bpb: 0.741616 |
| Step 00800 | ICL early: 2.7298 late: 2.8802 score: +0.1504 | Induction loss: 9.6485 acc: 0.0788 (random-half: 13.9446) |
| step 00800/01500 (53.33%) | loss: 2.464734 | lrm: 1.00 | dt: 2497.15ms | tok/sec: 209,954 | bf16_mfu: 32.05 | epoch: 1 pq: 9 rg: 16 | total time: 32.97m | eta: 29.2m |
| step 00801/01500 (53.40%) | loss: 2.485478 | lrm: 1.00 | dt: 2497.07ms | tok/sec: 209,961 | bf16_mfu: 32.06 | epoch: 1 pq: 9 rg: 24 | total time: 33.02m | eta: 29.2m |
| step 00802/01500 (53.47%) | loss: 2.503597 | lrm: 1.00 | dt: 2512.89ms | tok/sec: 208,639 | bf16_mfu: 31.85 | epoch: 1 pq: 9 rg: 24 | total time: 33.06m | eta: 29.1m |
| step 00803/01500 (53.53%) | loss: 2.497169 | lrm: 1.00 | dt: 2495.10ms | tok/sec: 210,126 | bf16_mfu: 32.08 | epoch: 1 pq: 9 rg: 24 | total time: 33.10m | eta: 29.1m |
| step 00804/01500 (53.60%) | loss: 2.495256 | lrm: 1.00 | dt: 2498.36ms | tok/sec: 209,853 | bf16_mfu: 32.04 | epoch: 1 pq: 9 rg: 24 | total time: 33.14m | eta: 29.1m |
| step 00805/01500 (53.67%) | loss: 2.459168 | lrm: 1.00 | dt: 2496.95ms | tok/sec: 209,971 | bf16_mfu: 32.06 | epoch: 1 pq: 9 rg: 24 | total time: 33.18m | eta: 29.0m |
| step 00806/01500 (53.73%) | loss: 2.542786 | lrm: 1.00 | dt: 2499.63ms | tok/sec: 209,746 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 24 | total time: 33.22m | eta: 29.0m |
| step 00807/01500 (53.80%) | loss: 2.502023 | lrm: 1.00 | dt: 2498.68ms | tok/sec: 209,826 | bf16_mfu: 32.04 | epoch: 1 pq: 9 rg: 24 | total time: 33.27m | eta: 28.9m |
| step 00808/01500 (53.87%) | loss: 2.530503 | lrm: 1.00 | dt: 2498.87ms | tok/sec: 209,810 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 24 | total time: 33.31m | eta: 28.9m |
| step 00809/01500 (53.93%) | loss: 2.506526 | lrm: 1.00 | dt: 2499.72ms | tok/sec: 209,738 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 24 | total time: 33.35m | eta: 28.8m |
| step 00810/01500 (54.00%) | loss: 2.499926 | lrm: 1.00 | dt: 2499.17ms | tok/sec: 209,784 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 32 | total time: 33.39m | eta: 28.8m |
| step 00811/01500 (54.07%) | loss: 2.511620 | lrm: 1.00 | dt: 2499.70ms | tok/sec: 209,740 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 32 | total time: 33.43m | eta: 28.8m |
| step 00812/01500 (54.13%) | loss: 2.518845 | lrm: 1.00 | dt: 2500.26ms | tok/sec: 209,693 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 32 | total time: 33.47m | eta: 28.7m |
| step 00813/01500 (54.20%) | loss: 2.486244 | lrm: 1.00 | dt: 2497.50ms | tok/sec: 209,925 | bf16_mfu: 32.05 | epoch: 1 pq: 9 rg: 32 | total time: 33.52m | eta: 28.7m |
| step 00814/01500 (54.27%) | loss: 2.483264 | lrm: 1.00 | dt: 2502.02ms | tok/sec: 209,545 | bf16_mfu: 31.99 | epoch: 1 pq: 9 rg: 32 | total time: 33.56m | eta: 28.6m |
| step 00815/01500 (54.33%) | loss: 2.496015 | lrm: 1.00 | dt: 2500.60ms | tok/sec: 209,665 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 32 | total time: 33.60m | eta: 28.6m |
| step 00816/01500 (54.40%) | loss: 2.470131 | lrm: 1.00 | dt: 2501.89ms | tok/sec: 209,556 | bf16_mfu: 31.99 | epoch: 1 pq: 9 rg: 32 | total time: 33.64m | eta: 28.5m |
| step 00817/01500 (54.47%) | loss: 2.521402 | lrm: 1.00 | dt: 2499.69ms | tok/sec: 209,741 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 32 | total time: 33.68m | eta: 28.5m |
| step 00818/01500 (54.53%) | loss: 2.530285 | lrm: 1.00 | dt: 2497.64ms | tok/sec: 209,913 | bf16_mfu: 32.05 | epoch: 1 pq: 9 rg: 40 | total time: 33.72m | eta: 28.5m |
| step 00819/01500 (54.60%) | loss: 2.537301 | lrm: 1.00 | dt: 2504.44ms | tok/sec: 209,343 | bf16_mfu: 31.96 | epoch: 1 pq: 9 rg: 40 | total time: 33.77m | eta: 28.4m |
| step 00820/01500 (54.67%) | loss: 2.521476 | lrm: 1.00 | dt: 2498.51ms | tok/sec: 209,840 | bf16_mfu: 32.04 | epoch: 1 pq: 9 rg: 40 | total time: 33.81m | eta: 28.4m |
| step 00821/01500 (54.73%) | loss: 2.536437 | lrm: 1.00 | dt: 2500.13ms | tok/sec: 209,704 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 40 | total time: 33.85m | eta: 28.3m |
| step 00822/01500 (54.80%) | loss: 2.523154 | lrm: 1.00 | dt: 2501.26ms | tok/sec: 209,609 | bf16_mfu: 32.00 | epoch: 1 pq: 9 rg: 40 | total time: 33.89m | eta: 28.3m |
| step 00823/01500 (54.87%) | loss: 2.503628 | lrm: 1.00 | dt: 2505.43ms | tok/sec: 209,260 | bf16_mfu: 31.95 | epoch: 1 pq: 9 rg: 40 | total time: 33.93m | eta: 28.3m |
| step 00824/01500 (54.93%) | loss: 2.489208 | lrm: 1.00 | dt: 2497.59ms | tok/sec: 209,917 | bf16_mfu: 32.05 | epoch: 1 pq: 9 rg: 40 | total time: 33.97m | eta: 28.2m |
| step 00825/01500 (55.00%) | loss: 2.499214 | lrm: 1.00 | dt: 2499.67ms | tok/sec: 209,742 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 40 | total time: 34.02m | eta: 28.2m |
| step 00826/01500 (55.07%) | loss: 2.467831 | lrm: 1.00 | dt: 2499.25ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 48 | total time: 34.06m | eta: 28.1m |
| step 00827/01500 (55.13%) | loss: 2.434501 | lrm: 1.00 | dt: 2498.28ms | tok/sec: 209,859 | bf16_mfu: 32.04 | epoch: 1 pq: 9 rg: 48 | total time: 34.10m | eta: 28.1m |
| step 00828/01500 (55.20%) | loss: 2.413911 | lrm: 1.00 | dt: 2500.52ms | tok/sec: 209,671 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 48 | total time: 34.14m | eta: 28.0m |
| step 00829/01500 (55.27%) | loss: 2.417626 | lrm: 1.00 | dt: 2497.00ms | tok/sec: 209,966 | bf16_mfu: 32.06 | epoch: 1 pq: 9 rg: 48 | total time: 34.18m | eta: 28.0m |
| step 00830/01500 (55.33%) | loss: 2.412850 | lrm: 1.00 | dt: 2500.08ms | tok/sec: 209,708 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 48 | total time: 34.22m | eta: 28.0m |
| step 00831/01500 (55.40%) | loss: 2.401505 | lrm: 1.00 | dt: 2501.14ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 9 rg: 48 | total time: 34.27m | eta: 27.9m |
| step 00832/01500 (55.47%) | loss: 2.387107 | lrm: 1.00 | dt: 2500.33ms | tok/sec: 209,687 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 48 | total time: 34.31m | eta: 27.9m |
| step 00833/01500 (55.53%) | loss: 2.388825 | lrm: 1.00 | dt: 2503.41ms | tok/sec: 209,429 | bf16_mfu: 31.97 | epoch: 1 pq: 9 rg: 48 | total time: 34.35m | eta: 27.8m |
| step 00834/01500 (55.60%) | loss: 2.321134 | lrm: 1.00 | dt: 2502.12ms | tok/sec: 209,537 | bf16_mfu: 31.99 | epoch: 1 pq: 9 rg: 56 | total time: 34.39m | eta: 27.8m |
| step 00835/01500 (55.67%) | loss: 2.310482 | lrm: 1.00 | dt: 2498.86ms | tok/sec: 209,811 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 56 | total time: 34.43m | eta: 27.8m |
| step 00836/01500 (55.73%) | loss: 2.296415 | lrm: 1.00 | dt: 2498.59ms | tok/sec: 209,833 | bf16_mfu: 32.04 | epoch: 1 pq: 9 rg: 56 | total time: 34.47m | eta: 27.7m |
| step 00837/01500 (55.80%) | loss: 2.241724 | lrm: 1.00 | dt: 2497.20ms | tok/sec: 209,950 | bf16_mfu: 32.05 | epoch: 1 pq: 9 rg: 56 | total time: 34.52m | eta: 27.7m |
| step 00838/01500 (55.87%) | loss: 2.274436 | lrm: 1.00 | dt: 2497.66ms | tok/sec: 209,911 | bf16_mfu: 32.05 | epoch: 1 pq: 9 rg: 56 | total time: 34.56m | eta: 27.6m |
| step 00839/01500 (55.93%) | loss: 2.280051 | lrm: 1.00 | dt: 2498.69ms | tok/sec: 209,825 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 56 | total time: 34.60m | eta: 27.6m |
| step 00840/01500 (56.00%) | loss: 2.259181 | lrm: 1.00 | dt: 2501.94ms | tok/sec: 209,552 | bf16_mfu: 31.99 | epoch: 1 pq: 9 rg: 56 | total time: 34.64m | eta: 27.5m |
| step 00841/01500 (56.07%) | loss: 2.319050 | lrm: 1.00 | dt: 2498.99ms | tok/sec: 209,799 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 56 | total time: 34.68m | eta: 27.5m |
| step 00842/01500 (56.13%) | loss: 2.335775 | lrm: 1.00 | dt: 2498.74ms | tok/sec: 209,820 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 64 | total time: 34.72m | eta: 27.5m |
| step 00843/01500 (56.20%) | loss: 2.365811 | lrm: 1.00 | dt: 2500.41ms | tok/sec: 209,680 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 64 | total time: 34.77m | eta: 27.4m |
| step 00844/01500 (56.27%) | loss: 2.361053 | lrm: 1.00 | dt: 2501.03ms | tok/sec: 209,628 | bf16_mfu: 32.00 | epoch: 1 pq: 9 rg: 64 | total time: 34.81m | eta: 27.4m |
| step 00845/01500 (56.33%) | loss: 2.376585 | lrm: 1.00 | dt: 2505.31ms | tok/sec: 209,270 | bf16_mfu: 31.95 | epoch: 1 pq: 9 rg: 64 | total time: 34.85m | eta: 27.3m |
| step 00846/01500 (56.40%) | loss: 2.384495 | lrm: 1.00 | dt: 2502.76ms | tok/sec: 209,483 | bf16_mfu: 31.98 | epoch: 1 pq: 9 rg: 64 | total time: 34.89m | eta: 27.3m |
| step 00847/01500 (56.47%) | loss: 2.378119 | lrm: 1.00 | dt: 2505.06ms | tok/sec: 209,291 | bf16_mfu: 31.95 | epoch: 1 pq: 9 rg: 64 | total time: 34.93m | eta: 27.3m |
| step 00848/01500 (56.53%) | loss: 2.429506 | lrm: 1.00 | dt: 2501.69ms | tok/sec: 209,573 | bf16_mfu: 32.00 | epoch: 1 pq: 9 rg: 64 | total time: 34.97m | eta: 27.2m |
| step 00849/01500 (56.60%) | loss: 2.447713 | lrm: 1.00 | dt: 2503.76ms | tok/sec: 209,400 | bf16_mfu: 31.97 | epoch: 1 pq: 9 rg: 64 | total time: 35.02m | eta: 27.2m |
| step 00850/01500 (56.67%) | loss: 2.429006 | lrm: 1.00 | dt: 2502.71ms | tok/sec: 209,488 | bf16_mfu: 31.98 | epoch: 1 pq: 9 rg: 72 | total time: 35.06m | eta: 27.1m |
| step 00851/01500 (56.73%) | loss: 2.430386 | lrm: 1.00 | dt: 2501.07ms | tok/sec: 209,625 | bf16_mfu: 32.00 | epoch: 1 pq: 9 rg: 72 | total time: 35.10m | eta: 27.1m |
| step 00852/01500 (56.80%) | loss: 2.461675 | lrm: 1.00 | dt: 2500.50ms | tok/sec: 209,672 | bf16_mfu: 32.01 | epoch: 1 pq: 9 rg: 72 | total time: 35.14m | eta: 27.0m |
| step 00853/01500 (56.87%) | loss: 2.427366 | lrm: 1.00 | dt: 2497.99ms | tok/sec: 209,883 | bf16_mfu: 32.04 | epoch: 1 pq: 9 rg: 72 | total time: 35.18m | eta: 27.0m |
| step 00854/01500 (56.93%) | loss: 2.420453 | lrm: 1.00 | dt: 2500.12ms | tok/sec: 209,705 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 72 | total time: 35.22m | eta: 27.0m |
| step 00855/01500 (57.00%) | loss: 2.422431 | lrm: 1.00 | dt: 2506.84ms | tok/sec: 209,142 | bf16_mfu: 31.93 | epoch: 1 pq: 9 rg: 72 | total time: 35.27m | eta: 26.9m |
| step 00856/01500 (57.07%) | loss: 2.393103 | lrm: 1.00 | dt: 2504.09ms | tok/sec: 209,372 | bf16_mfu: 31.97 | epoch: 1 pq: 9 rg: 72 | total time: 35.31m | eta: 26.9m |
| step 00857/01500 (57.13%) | loss: 2.413194 | lrm: 1.00 | dt: 2499.06ms | tok/sec: 209,794 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 72 | total time: 35.35m | eta: 26.8m |
| step 00858/01500 (57.20%) | loss: 2.432611 | lrm: 1.00 | dt: 2498.59ms | tok/sec: 209,833 | bf16_mfu: 32.04 | epoch: 1 pq: 9 rg: 80 | total time: 35.39m | eta: 26.8m |
| step 00859/01500 (57.27%) | loss: 2.446908 | lrm: 1.00 | dt: 2505.13ms | tok/sec: 209,286 | bf16_mfu: 31.95 | epoch: 1 pq: 9 rg: 80 | total time: 35.43m | eta: 26.8m |
| step 00860/01500 (57.33%) | loss: 2.430805 | lrm: 1.00 | dt: 2503.85ms | tok/sec: 209,392 | bf16_mfu: 31.97 | epoch: 1 pq: 9 rg: 80 | total time: 35.47m | eta: 26.7m |
| step 00861/01500 (57.40%) | loss: 2.454606 | lrm: 1.00 | dt: 2499.66ms | tok/sec: 209,743 | bf16_mfu: 32.02 | epoch: 1 pq: 9 rg: 80 | total time: 35.52m | eta: 26.7m |
| step 00862/01500 (57.47%) | loss: 2.432909 | lrm: 1.00 | dt: 2504.19ms | tok/sec: 209,364 | bf16_mfu: 31.96 | epoch: 1 pq: 9 rg: 80 | total time: 35.56m | eta: 26.6m |
| step 00863/01500 (57.53%) | loss: 2.460640 | lrm: 1.00 | dt: 2498.95ms | tok/sec: 209,803 | bf16_mfu: 32.03 | epoch: 1 pq: 9 rg: 80 | total time: 35.60m | eta: 26.6m |
| step 00864/01500 (57.60%) | loss: 2.466855 | lrm: 1.00 | dt: 2495.95ms | tok/sec: 210,055 | bf16_mfu: 32.07 | epoch: 1 pq: 9 rg: 80 | total time: 35.64m | eta: 26.5m |
| step 00865/01500 (57.67%) | loss: 2.488819 | lrm: 1.00 | dt: 2751.02ms | tok/sec: 190,579 | bf16_mfu: 29.10 | epoch: 1 pq: 9 rg: 80 | total time: 35.69m | eta: 26.5m |
| step 00866/01500 (57.73%) | loss: 2.460920 | lrm: 1.00 | dt: 2502.21ms | tok/sec: 209,529 | bf16_mfu: 31.99 | epoch: 1 pq: 10 rg: 0 | total time: 35.73m | eta: 26.5m |
| step 00867/01500 (57.80%) | loss: 2.434904 | lrm: 1.00 | dt: 2503.90ms | tok/sec: 209,388 | bf16_mfu: 31.97 | epoch: 1 pq: 10 rg: 0 | total time: 35.77m | eta: 26.4m |
| step 00868/01500 (57.87%) | loss: 2.458098 | lrm: 1.00 | dt: 2513.55ms | tok/sec: 208,584 | bf16_mfu: 31.85 | epoch: 1 pq: 10 rg: 0 | total time: 35.81m | eta: 26.4m |
| step 00869/01500 (57.93%) | loss: 2.449237 | lrm: 1.00 | dt: 2497.80ms | tok/sec: 209,900 | bf16_mfu: 32.05 | epoch: 1 pq: 10 rg: 0 | total time: 35.85m | eta: 26.3m |
| step 00870/01500 (58.00%) | loss: 2.448500 | lrm: 1.00 | dt: 2497.95ms | tok/sec: 209,887 | bf16_mfu: 32.04 | epoch: 1 pq: 10 rg: 0 | total time: 35.90m | eta: 26.3m |
| step 00871/01500 (58.07%) | loss: 2.446891 | lrm: 1.00 | dt: 2500.76ms | tok/sec: 209,651 | bf16_mfu: 32.01 | epoch: 1 pq: 10 rg: 0 | total time: 35.94m | eta: 26.3m |
| step 00872/01500 (58.13%) | loss: 2.394001 | lrm: 1.00 | dt: 2496.83ms | tok/sec: 209,981 | bf16_mfu: 32.06 | epoch: 1 pq: 10 rg: 0 | total time: 35.98m | eta: 26.2m |
| step 00873/01500 (58.20%) | loss: 2.374766 | lrm: 1.00 | dt: 2502.69ms | tok/sec: 209,489 | bf16_mfu: 31.98 | epoch: 1 pq: 10 rg: 0 | total time: 36.02m | eta: 26.2m |
| step 00874/01500 (58.27%) | loss: 2.393871 | lrm: 1.00 | dt: 2499.73ms | tok/sec: 209,737 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 8 | total time: 36.06m | eta: 26.1m |
| step 00875/01500 (58.33%) | loss: 2.388008 | lrm: 1.00 | dt: 2502.98ms | tok/sec: 209,465 | bf16_mfu: 31.98 | epoch: 1 pq: 10 rg: 8 | total time: 36.10m | eta: 26.1m |
| step 00876/01500 (58.40%) | loss: 2.403612 | lrm: 1.00 | dt: 2499.08ms | tok/sec: 209,792 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 8 | total time: 36.15m | eta: 26.0m |
| step 00877/01500 (58.47%) | loss: 2.423884 | lrm: 1.00 | dt: 2497.25ms | tok/sec: 209,945 | bf16_mfu: 32.05 | epoch: 1 pq: 10 rg: 8 | total time: 36.19m | eta: 26.0m |
| step 00878/01500 (58.53%) | loss: 2.443265 | lrm: 1.00 | dt: 2504.74ms | tok/sec: 209,318 | bf16_mfu: 31.96 | epoch: 1 pq: 10 rg: 8 | total time: 36.23m | eta: 26.0m |
| step 00879/01500 (58.60%) | loss: 2.426598 | lrm: 1.00 | dt: 2495.43ms | tok/sec: 210,099 | bf16_mfu: 32.08 | epoch: 1 pq: 10 rg: 8 | total time: 36.27m | eta: 25.9m |
| step 00880/01500 (58.67%) | loss: 2.472516 | lrm: 1.00 | dt: 2498.87ms | tok/sec: 209,810 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 8 | total time: 36.31m | eta: 25.9m |
| step 00881/01500 (58.73%) | loss: 2.499515 | lrm: 1.00 | dt: 2500.67ms | tok/sec: 209,658 | bf16_mfu: 32.01 | epoch: 1 pq: 10 rg: 8 | total time: 36.35m | eta: 25.8m |
| step 00882/01500 (58.80%) | loss: 2.513423 | lrm: 1.00 | dt: 2499.99ms | tok/sec: 209,715 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 8 | total time: 36.40m | eta: 25.8m |
| step 00883/01500 (58.87%) | loss: 2.552547 | lrm: 1.00 | dt: 2502.99ms | tok/sec: 209,465 | bf16_mfu: 31.98 | epoch: 1 pq: 10 rg: 16 | total time: 36.44m | eta: 25.8m |
| step 00884/01500 (58.93%) | loss: 2.552526 | lrm: 1.00 | dt: 2504.21ms | tok/sec: 209,362 | bf16_mfu: 31.96 | epoch: 1 pq: 10 rg: 16 | total time: 36.48m | eta: 25.7m |
| step 00885/01500 (59.00%) | loss: 2.534397 | lrm: 1.00 | dt: 2501.25ms | tok/sec: 209,610 | bf16_mfu: 32.00 | epoch: 1 pq: 10 rg: 16 | total time: 36.52m | eta: 25.7m |
| step 00886/01500 (59.07%) | loss: 2.541211 | lrm: 1.00 | dt: 2500.06ms | tok/sec: 209,710 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 16 | total time: 36.56m | eta: 25.6m |
| step 00887/01500 (59.13%) | loss: 2.519362 | lrm: 1.00 | dt: 2499.03ms | tok/sec: 209,796 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 16 | total time: 36.60m | eta: 25.6m |
| step 00888/01500 (59.20%) | loss: 2.499014 | lrm: 1.00 | dt: 2501.62ms | tok/sec: 209,579 | bf16_mfu: 32.00 | epoch: 1 pq: 10 rg: 16 | total time: 36.65m | eta: 25.5m |
| step 00889/01500 (59.27%) | loss: 2.477072 | lrm: 1.00 | dt: 2503.42ms | tok/sec: 209,428 | bf16_mfu: 31.97 | epoch: 1 pq: 10 rg: 16 | total time: 36.69m | eta: 25.5m |
| step 00890/01500 (59.33%) | loss: 2.486483 | lrm: 1.00 | dt: 2500.29ms | tok/sec: 209,690 | bf16_mfu: 32.01 | epoch: 1 pq: 10 rg: 16 | total time: 36.73m | eta: 25.5m |
| step 00891/01500 (59.40%) | loss: 2.499196 | lrm: 1.00 | dt: 2498.86ms | tok/sec: 209,810 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 24 | total time: 36.77m | eta: 25.4m |
| step 00892/01500 (59.47%) | loss: 2.511819 | lrm: 1.00 | dt: 2497.76ms | tok/sec: 209,903 | bf16_mfu: 32.05 | epoch: 1 pq: 10 rg: 24 | total time: 36.81m | eta: 25.4m |
| step 00893/01500 (59.53%) | loss: 2.506047 | lrm: 1.00 | dt: 2498.67ms | tok/sec: 209,826 | bf16_mfu: 32.04 | epoch: 1 pq: 10 rg: 24 | total time: 36.85m | eta: 25.3m |
| step 00894/01500 (59.60%) | loss: 2.506997 | lrm: 1.00 | dt: 2504.83ms | tok/sec: 209,310 | bf16_mfu: 31.96 | epoch: 1 pq: 10 rg: 24 | total time: 36.90m | eta: 25.3m |
| step 00895/01500 (59.67%) | loss: 2.437824 | lrm: 1.00 | dt: 2494.00ms | tok/sec: 210,219 | bf16_mfu: 32.10 | epoch: 1 pq: 10 rg: 24 | total time: 36.94m | eta: 25.3m |
| step 00896/01500 (59.73%) | loss: 2.491216 | lrm: 1.00 | dt: 2500.11ms | tok/sec: 209,706 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 24 | total time: 36.98m | eta: 25.2m |
| step 00897/01500 (59.80%) | loss: 2.459149 | lrm: 1.00 | dt: 2506.67ms | tok/sec: 209,157 | bf16_mfu: 31.93 | epoch: 1 pq: 10 rg: 24 | total time: 37.02m | eta: 25.2m |
| step 00898/01500 (59.87%) | loss: 2.434871 | lrm: 1.00 | dt: 2497.20ms | tok/sec: 209,950 | bf16_mfu: 32.05 | epoch: 1 pq: 10 rg: 24 | total time: 37.06m | eta: 25.1m |
| step 00899/01500 (59.93%) | loss: 2.421553 | lrm: 1.00 | dt: 2498.61ms | tok/sec: 209,831 | bf16_mfu: 32.04 | epoch: 1 pq: 10 rg: 32 | total time: 37.10m | eta: 25.1m |
| Step 00900 | Validation bpb: 0.738064 |
| Step 00900 | ICL early: 2.7088 late: 2.8737 score: +0.1649 | Induction loss: 9.3294 acc: 0.0943 (random-half: 13.9594) |
| step 00900/01500 (60.00%) | loss: 2.437313 | lrm: 1.00 | dt: 2495.98ms | tok/sec: 210,053 | bf16_mfu: 32.07 | epoch: 1 pq: 10 rg: 32 | total time: 37.15m | eta: 25.0m |
| step 00901/01500 (60.07%) | loss: 2.452271 | lrm: 1.00 | dt: 2499.60ms | tok/sec: 209,748 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 32 | total time: 37.19m | eta: 25.0m |
| step 00902/01500 (60.13%) | loss: 2.442853 | lrm: 1.00 | dt: 2496.67ms | tok/sec: 209,995 | bf16_mfu: 32.06 | epoch: 1 pq: 10 rg: 32 | total time: 37.23m | eta: 25.0m |
| step 00903/01500 (60.20%) | loss: 2.442931 | lrm: 1.00 | dt: 2499.68ms | tok/sec: 209,742 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 32 | total time: 37.27m | eta: 24.9m |
| step 00904/01500 (60.27%) | loss: 2.466908 | lrm: 1.00 | dt: 2496.53ms | tok/sec: 210,006 | bf16_mfu: 32.06 | epoch: 1 pq: 10 rg: 32 | total time: 37.31m | eta: 24.9m |
| step 00905/01500 (60.33%) | loss: 2.434947 | lrm: 1.00 | dt: 2499.20ms | tok/sec: 209,782 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 32 | total time: 37.35m | eta: 24.8m |
| step 00906/01500 (60.40%) | loss: 2.480730 | lrm: 1.00 | dt: 2498.09ms | tok/sec: 209,875 | bf16_mfu: 32.04 | epoch: 1 pq: 10 rg: 32 | total time: 37.40m | eta: 24.8m |
| step 00907/01500 (60.47%) | loss: 2.504384 | lrm: 1.00 | dt: 2499.45ms | tok/sec: 209,761 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 40 | total time: 37.44m | eta: 24.7m |
| step 00908/01500 (60.53%) | loss: 2.495713 | lrm: 1.00 | dt: 2505.00ms | tok/sec: 209,296 | bf16_mfu: 31.95 | epoch: 1 pq: 10 rg: 40 | total time: 37.48m | eta: 24.7m |
| step 00909/01500 (60.60%) | loss: 2.462858 | lrm: 1.00 | dt: 2499.99ms | tok/sec: 209,716 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 40 | total time: 37.52m | eta: 24.7m |
| step 00910/01500 (60.67%) | loss: 2.435030 | lrm: 1.00 | dt: 2505.34ms | tok/sec: 209,268 | bf16_mfu: 31.95 | epoch: 1 pq: 10 rg: 40 | total time: 37.56m | eta: 24.6m |
| step 00911/01500 (60.73%) | loss: 2.447943 | lrm: 1.00 | dt: 2502.82ms | tok/sec: 209,478 | bf16_mfu: 31.98 | epoch: 1 pq: 10 rg: 40 | total time: 37.60m | eta: 24.6m |
| step 00912/01500 (60.80%) | loss: 2.454640 | lrm: 1.00 | dt: 2493.91ms | tok/sec: 210,227 | bf16_mfu: 32.10 | epoch: 1 pq: 10 rg: 40 | total time: 37.65m | eta: 24.5m |
| step 00913/01500 (60.87%) | loss: 2.445828 | lrm: 1.00 | dt: 2498.86ms | tok/sec: 209,811 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 40 | total time: 37.69m | eta: 24.5m |
| step 00914/01500 (60.93%) | loss: 2.422156 | lrm: 1.00 | dt: 2495.24ms | tok/sec: 210,115 | bf16_mfu: 32.08 | epoch: 1 pq: 10 rg: 40 | total time: 37.73m | eta: 24.5m |
| step 00915/01500 (61.00%) | loss: 2.371485 | lrm: 1.00 | dt: 2498.23ms | tok/sec: 209,863 | bf16_mfu: 32.04 | epoch: 1 pq: 10 rg: 48 | total time: 37.77m | eta: 24.4m |
| step 00916/01500 (61.07%) | loss: 2.345092 | lrm: 1.00 | dt: 2501.57ms | tok/sec: 209,583 | bf16_mfu: 32.00 | epoch: 1 pq: 10 rg: 48 | total time: 37.81m | eta: 24.4m |
| step 00917/01500 (61.13%) | loss: 2.322704 | lrm: 1.00 | dt: 2503.62ms | tok/sec: 209,411 | bf16_mfu: 31.97 | epoch: 1 pq: 10 rg: 48 | total time: 37.85m | eta: 24.3m |
| step 00918/01500 (61.20%) | loss: 2.368564 | lrm: 1.00 | dt: 2499.05ms | tok/sec: 209,794 | bf16_mfu: 32.03 | epoch: 1 pq: 10 rg: 48 | total time: 37.90m | eta: 24.3m |
| step 00919/01500 (61.27%) | loss: 2.370071 | lrm: 1.00 | dt: 2501.15ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 10 rg: 48 | total time: 37.94m | eta: 24.2m |
| step 00920/01500 (61.33%) | loss: 2.356302 | lrm: 1.00 | dt: 2502.72ms | tok/sec: 209,487 | bf16_mfu: 31.98 | epoch: 1 pq: 10 rg: 48 | total time: 37.98m | eta: 24.2m |
| step 00921/01500 (61.40%) | loss: 2.315991 | lrm: 1.00 | dt: 2504.31ms | tok/sec: 209,354 | bf16_mfu: 31.96 | epoch: 1 pq: 10 rg: 48 | total time: 38.02m | eta: 24.2m |
| step 00922/01500 (61.47%) | loss: 2.319314 | lrm: 1.00 | dt: 2499.49ms | tok/sec: 209,758 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 48 | total time: 38.06m | eta: 24.1m |
| step 00923/01500 (61.53%) | loss: 2.356224 | lrm: 1.00 | dt: 2499.80ms | tok/sec: 209,731 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 56 | total time: 38.10m | eta: 24.1m |
| step 00924/01500 (61.60%) | loss: 2.359322 | lrm: 1.00 | dt: 2498.15ms | tok/sec: 209,870 | bf16_mfu: 32.04 | epoch: 1 pq: 10 rg: 56 | total time: 38.15m | eta: 24.0m |
| step 00925/01500 (61.67%) | loss: 2.411088 | lrm: 1.00 | dt: 2497.91ms | tok/sec: 209,890 | bf16_mfu: 32.04 | epoch: 1 pq: 10 rg: 56 | total time: 38.19m | eta: 24.0m |
| step 00926/01500 (61.73%) | loss: 2.339008 | lrm: 1.00 | dt: 2499.73ms | tok/sec: 209,738 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 56 | total time: 38.23m | eta: 24.0m |
| step 00927/01500 (61.80%) | loss: 2.375406 | lrm: 1.00 | dt: 2503.46ms | tok/sec: 209,424 | bf16_mfu: 31.97 | epoch: 1 pq: 10 rg: 56 | total time: 38.27m | eta: 23.9m |
| step 00928/01500 (61.87%) | loss: 2.395494 | lrm: 1.00 | dt: 2501.10ms | tok/sec: 209,622 | bf16_mfu: 32.00 | epoch: 1 pq: 10 rg: 56 | total time: 38.31m | eta: 23.9m |
| step 00929/01500 (61.93%) | loss: 2.427484 | lrm: 1.00 | dt: 2500.83ms | tok/sec: 209,645 | bf16_mfu: 32.01 | epoch: 1 pq: 10 rg: 56 | total time: 38.35m | eta: 23.8m |
| step 00930/01500 (62.00%) | loss: 2.435807 | lrm: 1.00 | dt: 2502.10ms | tok/sec: 209,539 | bf16_mfu: 31.99 | epoch: 1 pq: 10 rg: 56 | total time: 38.40m | eta: 23.8m |
| step 00931/01500 (62.07%) | loss: 2.426410 | lrm: 1.00 | dt: 2504.83ms | tok/sec: 209,310 | bf16_mfu: 31.96 | epoch: 1 pq: 10 rg: 56 | total time: 38.44m | eta: 23.7m |
| step 00932/01500 (62.13%) | loss: 2.424321 | lrm: 1.00 | dt: 2494.62ms | tok/sec: 210,167 | bf16_mfu: 32.09 | epoch: 1 pq: 10 rg: 64 | total time: 38.48m | eta: 23.7m |
| step 00933/01500 (62.20%) | loss: 2.443905 | lrm: 1.00 | dt: 2501.26ms | tok/sec: 209,609 | bf16_mfu: 32.00 | epoch: 1 pq: 10 rg: 64 | total time: 38.52m | eta: 23.7m |
| step 00934/01500 (62.27%) | loss: 2.477102 | lrm: 1.00 | dt: 2502.26ms | tok/sec: 209,525 | bf16_mfu: 31.99 | epoch: 1 pq: 10 rg: 64 | total time: 38.56m | eta: 23.6m |
| step 00935/01500 (62.33%) | loss: 2.472756 | lrm: 1.00 | dt: 2503.48ms | tok/sec: 209,424 | bf16_mfu: 31.97 | epoch: 1 pq: 10 rg: 64 | total time: 38.60m | eta: 23.6m |
| step 00936/01500 (62.40%) | loss: 2.477611 | lrm: 1.00 | dt: 2495.53ms | tok/sec: 210,090 | bf16_mfu: 32.08 | epoch: 1 pq: 10 rg: 64 | total time: 38.65m | eta: 23.5m |
| step 00937/01500 (62.47%) | loss: 2.486416 | lrm: 1.00 | dt: 2504.99ms | tok/sec: 209,297 | bf16_mfu: 31.95 | epoch: 1 pq: 10 rg: 64 | total time: 38.69m | eta: 23.5m |
| step 00938/01500 (62.53%) | loss: 2.504975 | lrm: 1.00 | dt: 2502.44ms | tok/sec: 209,510 | bf16_mfu: 31.99 | epoch: 1 pq: 10 rg: 64 | total time: 38.73m | eta: 23.5m |
| step 00939/01500 (62.60%) | loss: 2.523929 | lrm: 1.00 | dt: 2512.75ms | tok/sec: 208,650 | bf16_mfu: 31.86 | epoch: 1 pq: 10 rg: 64 | total time: 38.77m | eta: 23.4m |
| step 00940/01500 (62.67%) | loss: 2.517792 | lrm: 1.00 | dt: 2499.65ms | tok/sec: 209,744 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 72 | total time: 38.81m | eta: 23.4m |
| step 00941/01500 (62.73%) | loss: 2.534914 | lrm: 1.00 | dt: 2499.52ms | tok/sec: 209,755 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 72 | total time: 38.85m | eta: 23.3m |
| step 00942/01500 (62.80%) | loss: 2.592099 | lrm: 1.00 | dt: 2508.99ms | tok/sec: 208,963 | bf16_mfu: 31.90 | epoch: 1 pq: 10 rg: 72 | total time: 38.90m | eta: 23.3m |
| step 00943/01500 (62.87%) | loss: 2.642259 | lrm: 1.00 | dt: 2506.59ms | tok/sec: 209,164 | bf16_mfu: 31.93 | epoch: 1 pq: 10 rg: 72 | total time: 38.94m | eta: 23.2m |
| step 00944/01500 (62.93%) | loss: 2.654288 | lrm: 1.00 | dt: 2500.12ms | tok/sec: 209,704 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 72 | total time: 38.98m | eta: 23.2m |
| step 00945/01500 (63.00%) | loss: 2.667749 | lrm: 1.00 | dt: 2504.46ms | tok/sec: 209,341 | bf16_mfu: 31.96 | epoch: 1 pq: 10 rg: 72 | total time: 39.02m | eta: 23.2m |
| step 00946/01500 (63.07%) | loss: 2.651784 | lrm: 1.00 | dt: 2864.18ms | tok/sec: 183,049 | bf16_mfu: 27.95 | epoch: 1 pq: 10 rg: 72 | total time: 39.07m | eta: 23.1m |
| step 00947/01500 (63.13%) | loss: 2.667027 | lrm: 1.00 | dt: 2502.78ms | tok/sec: 209,482 | bf16_mfu: 31.98 | epoch: 1 pq: 10 rg: 72 | total time: 39.11m | eta: 23.1m |
| step 00948/01500 (63.20%) | loss: 2.621160 | lrm: 1.00 | dt: 2505.13ms | tok/sec: 209,286 | bf16_mfu: 31.95 | epoch: 1 pq: 10 rg: 80 | total time: 39.15m | eta: 23.0m |
| step 00949/01500 (63.27%) | loss: 2.607871 | lrm: 1.00 | dt: 2499.78ms | tok/sec: 209,733 | bf16_mfu: 32.02 | epoch: 1 pq: 10 rg: 80 | total time: 39.19m | eta: 23.0m |
| step 00950/01500 (63.33%) | loss: 2.592276 | lrm: 1.00 | dt: 2497.56ms | tok/sec: 209,919 | bf16_mfu: 32.05 | epoch: 1 pq: 10 rg: 80 | total time: 39.24m | eta: 23.0m |
| step 00951/01500 (63.40%) | loss: 2.583517 | lrm: 1.00 | dt: 2507.85ms | tok/sec: 209,058 | bf16_mfu: 31.92 | epoch: 1 pq: 10 rg: 80 | total time: 39.28m | eta: 22.9m |
| step 00952/01500 (63.47%) | loss: 2.591586 | lrm: 1.00 | dt: 2495.67ms | tok/sec: 210,079 | bf16_mfu: 32.07 | epoch: 1 pq: 10 rg: 80 | total time: 39.32m | eta: 22.9m |
| step 00953/01500 (63.53%) | loss: 2.588482 | lrm: 1.00 | dt: 2496.82ms | tok/sec: 209,982 | bf16_mfu: 32.06 | epoch: 1 pq: 10 rg: 80 | total time: 39.36m | eta: 22.8m |
| step 00954/01500 (63.60%) | loss: 2.538744 | lrm: 1.00 | dt: 2500.56ms | tok/sec: 209,668 | bf16_mfu: 32.01 | epoch: 1 pq: 10 rg: 80 | total time: 39.40m | eta: 22.8m |
| step 00955/01500 (63.67%) | loss: 2.597563 | lrm: 1.00 | dt: 2499.85ms | tok/sec: 209,727 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 0 | total time: 39.44m | eta: 22.7m |
| step 00956/01500 (63.73%) | loss: 2.552908 | lrm: 1.00 | dt: 2500.05ms | tok/sec: 209,710 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 0 | total time: 39.49m | eta: 22.7m |
| step 00957/01500 (63.80%) | loss: 2.587846 | lrm: 1.00 | dt: 2499.13ms | tok/sec: 209,788 | bf16_mfu: 32.03 | epoch: 1 pq: 11 rg: 0 | total time: 39.53m | eta: 22.7m |
| step 00958/01500 (63.87%) | loss: 2.517173 | lrm: 1.00 | dt: 2499.85ms | tok/sec: 209,727 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 0 | total time: 39.57m | eta: 22.6m |
| step 00959/01500 (63.93%) | loss: 2.483435 | lrm: 1.00 | dt: 2500.10ms | tok/sec: 209,707 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 0 | total time: 39.61m | eta: 22.6m |
| step 00960/01500 (64.00%) | loss: 2.511735 | lrm: 1.00 | dt: 2497.28ms | tok/sec: 209,943 | bf16_mfu: 32.05 | epoch: 1 pq: 11 rg: 0 | total time: 39.65m | eta: 22.5m |
| step 00961/01500 (64.07%) | loss: 2.485906 | lrm: 1.00 | dt: 2499.28ms | tok/sec: 209,775 | bf16_mfu: 32.03 | epoch: 1 pq: 11 rg: 0 | total time: 39.69m | eta: 22.5m |
| step 00962/01500 (64.13%) | loss: 2.495403 | lrm: 1.00 | dt: 2500.50ms | tok/sec: 209,673 | bf16_mfu: 32.01 | epoch: 1 pq: 11 rg: 0 | total time: 39.74m | eta: 22.5m |
| step 00963/01500 (64.20%) | loss: 2.427355 | lrm: 1.00 | dt: 2501.12ms | tok/sec: 209,621 | bf16_mfu: 32.00 | epoch: 1 pq: 11 rg: 0 | total time: 39.78m | eta: 22.4m |
| step 00964/01500 (64.27%) | loss: 2.373301 | lrm: 1.00 | dt: 2503.56ms | tok/sec: 209,417 | bf16_mfu: 31.97 | epoch: 1 pq: 11 rg: 8 | total time: 39.82m | eta: 22.4m |
| step 00965/01500 (64.33%) | loss: 2.403538 | lrm: 1.00 | dt: 2497.58ms | tok/sec: 209,918 | bf16_mfu: 32.05 | epoch: 1 pq: 11 rg: 8 | total time: 39.86m | eta: 22.3m |
| step 00966/01500 (64.40%) | loss: 2.396511 | lrm: 1.00 | dt: 2499.82ms | tok/sec: 209,730 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 8 | total time: 39.90m | eta: 22.3m |
| step 00967/01500 (64.47%) | loss: 2.421103 | lrm: 1.00 | dt: 2502.89ms | tok/sec: 209,472 | bf16_mfu: 31.98 | epoch: 1 pq: 11 rg: 8 | total time: 39.94m | eta: 22.2m |
| step 00968/01500 (64.53%) | loss: 2.371453 | lrm: 1.00 | dt: 2506.20ms | tok/sec: 209,196 | bf16_mfu: 31.94 | epoch: 1 pq: 11 rg: 8 | total time: 39.99m | eta: 22.2m |
| step 00969/01500 (64.60%) | loss: 2.408201 | lrm: 1.00 | dt: 2500.77ms | tok/sec: 209,650 | bf16_mfu: 32.01 | epoch: 1 pq: 11 rg: 8 | total time: 40.03m | eta: 22.2m |
| step 00970/01500 (64.67%) | loss: 2.475795 | lrm: 1.00 | dt: 2500.10ms | tok/sec: 209,706 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 8 | total time: 40.07m | eta: 22.1m |
| step 00971/01500 (64.73%) | loss: 2.476792 | lrm: 1.00 | dt: 2498.16ms | tok/sec: 209,870 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 8 | total time: 40.11m | eta: 22.1m |
| step 00972/01500 (64.80%) | loss: 2.493392 | lrm: 1.00 | dt: 2502.77ms | tok/sec: 209,482 | bf16_mfu: 31.98 | epoch: 1 pq: 11 rg: 16 | total time: 40.15m | eta: 22.0m |
| step 00973/01500 (64.87%) | loss: 2.530999 | lrm: 1.00 | dt: 2504.69ms | tok/sec: 209,322 | bf16_mfu: 31.96 | epoch: 1 pq: 11 rg: 16 | total time: 40.19m | eta: 22.0m |
| step 00974/01500 (64.93%) | loss: 2.523076 | lrm: 1.00 | dt: 2498.97ms | tok/sec: 209,801 | bf16_mfu: 32.03 | epoch: 1 pq: 11 rg: 16 | total time: 40.24m | eta: 22.0m |
| step 00975/01500 (65.00%) | loss: 2.548861 | lrm: 1.00 | dt: 2503.95ms | tok/sec: 209,384 | bf16_mfu: 31.97 | epoch: 1 pq: 11 rg: 16 | total time: 40.28m | eta: 21.9m |
| step 00976/01500 (65.07%) | loss: 2.541862 | lrm: 1.00 | dt: 2500.11ms | tok/sec: 209,705 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 16 | total time: 40.32m | eta: 21.9m |
| step 00977/01500 (65.13%) | loss: 2.530597 | lrm: 1.00 | dt: 2501.88ms | tok/sec: 209,557 | bf16_mfu: 31.99 | epoch: 1 pq: 11 rg: 16 | total time: 40.36m | eta: 21.8m |
| step 00978/01500 (65.20%) | loss: 2.512620 | lrm: 1.00 | dt: 2500.03ms | tok/sec: 209,712 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 16 | total time: 40.40m | eta: 21.8m |
| step 00979/01500 (65.27%) | loss: 2.513045 | lrm: 1.00 | dt: 2501.07ms | tok/sec: 209,625 | bf16_mfu: 32.00 | epoch: 1 pq: 11 rg: 16 | total time: 40.44m | eta: 21.7m |
| step 00980/01500 (65.33%) | loss: 2.520401 | lrm: 1.00 | dt: 2500.33ms | tok/sec: 209,687 | bf16_mfu: 32.01 | epoch: 1 pq: 11 rg: 24 | total time: 40.49m | eta: 21.7m |
| step 00981/01500 (65.40%) | loss: 2.507651 | lrm: 1.00 | dt: 2498.06ms | tok/sec: 209,877 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 24 | total time: 40.53m | eta: 21.7m |
| step 00982/01500 (65.47%) | loss: 2.498525 | lrm: 1.00 | dt: 2498.41ms | tok/sec: 209,848 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 24 | total time: 40.57m | eta: 21.6m |
| step 00983/01500 (65.53%) | loss: 2.495252 | lrm: 1.00 | dt: 2504.27ms | tok/sec: 209,357 | bf16_mfu: 31.96 | epoch: 1 pq: 11 rg: 24 | total time: 40.61m | eta: 21.6m |
| step 00984/01500 (65.60%) | loss: 2.509092 | lrm: 1.00 | dt: 2504.26ms | tok/sec: 209,358 | bf16_mfu: 31.96 | epoch: 1 pq: 11 rg: 24 | total time: 40.65m | eta: 21.5m |
| step 00985/01500 (65.67%) | loss: 2.470145 | lrm: 1.00 | dt: 2498.52ms | tok/sec: 209,839 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 24 | total time: 40.69m | eta: 21.5m |
| step 00986/01500 (65.73%) | loss: 2.498084 | lrm: 1.00 | dt: 2504.01ms | tok/sec: 209,379 | bf16_mfu: 31.97 | epoch: 1 pq: 11 rg: 24 | total time: 40.74m | eta: 21.5m |
| step 00987/01500 (65.80%) | loss: 2.484797 | lrm: 1.00 | dt: 2503.29ms | tok/sec: 209,439 | bf16_mfu: 31.98 | epoch: 1 pq: 11 rg: 24 | total time: 40.78m | eta: 21.4m |
| step 00988/01500 (65.87%) | loss: 2.476182 | lrm: 1.00 | dt: 2522.53ms | tok/sec: 207,841 | bf16_mfu: 31.73 | epoch: 1 pq: 11 rg: 32 | total time: 40.82m | eta: 21.4m |
| step 00989/01500 (65.93%) | loss: 2.454756 | lrm: 1.00 | dt: 2502.31ms | tok/sec: 209,521 | bf16_mfu: 31.99 | epoch: 1 pq: 11 rg: 32 | total time: 40.86m | eta: 21.3m |
| step 00990/01500 (66.00%) | loss: 2.453343 | lrm: 1.00 | dt: 2504.37ms | tok/sec: 209,349 | bf16_mfu: 31.96 | epoch: 1 pq: 11 rg: 32 | total time: 40.90m | eta: 21.3m |
| step 00991/01500 (66.07%) | loss: 2.431266 | lrm: 1.00 | dt: 2498.48ms | tok/sec: 209,843 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 32 | total time: 40.95m | eta: 21.2m |
| step 00992/01500 (66.13%) | loss: 2.458795 | lrm: 1.00 | dt: 2502.59ms | tok/sec: 209,498 | bf16_mfu: 31.99 | epoch: 1 pq: 11 rg: 32 | total time: 40.99m | eta: 21.2m |
| step 00993/01500 (66.20%) | loss: 2.454704 | lrm: 1.00 | dt: 2504.36ms | tok/sec: 209,350 | bf16_mfu: 31.96 | epoch: 1 pq: 11 rg: 32 | total time: 41.03m | eta: 21.2m |
| step 00994/01500 (66.27%) | loss: 2.408088 | lrm: 1.00 | dt: 2496.91ms | tok/sec: 209,974 | bf16_mfu: 32.06 | epoch: 1 pq: 11 rg: 32 | total time: 41.07m | eta: 21.1m |
| step 00995/01500 (66.33%) | loss: 2.413461 | lrm: 1.00 | dt: 2498.20ms | tok/sec: 209,866 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 32 | total time: 41.11m | eta: 21.1m |
| step 00996/01500 (66.40%) | loss: 2.423102 | lrm: 1.00 | dt: 2515.28ms | tok/sec: 208,441 | bf16_mfu: 31.82 | epoch: 1 pq: 11 rg: 40 | total time: 41.15m | eta: 21.0m |
| step 00997/01500 (66.47%) | loss: 2.420640 | lrm: 1.00 | dt: 2495.56ms | tok/sec: 210,088 | bf16_mfu: 32.08 | epoch: 1 pq: 11 rg: 40 | total time: 41.20m | eta: 21.0m |
| step 00998/01500 (66.53%) | loss: 2.447795 | lrm: 1.00 | dt: 2500.33ms | tok/sec: 209,687 | bf16_mfu: 32.01 | epoch: 1 pq: 11 rg: 40 | total time: 41.24m | eta: 21.0m |
| step 00999/01500 (66.60%) | loss: 2.445551 | lrm: 1.00 | dt: 2499.24ms | tok/sec: 209,778 | bf16_mfu: 32.03 | epoch: 1 pq: 11 rg: 40 | total time: 41.28m | eta: 20.9m |
| Step 01000 | Validation bpb: 0.735113 |
| Step 01000 | ICL early: 2.6780 late: 2.8650 score: +0.1870 | Induction loss: 9.4661 acc: 0.0879 (random-half: 14.0060) |
| <|bos|>The capital of France is Paris. It is the capital of the Île-de-France region. It |
| <|bos|>The chemical symbol of gold is Au. It is a chemical element with the symbol Au. It is a chemical |
| <|bos|>If yesterday was Friday, then tomorrow will be Saturday. Today is the 14th day of the month. The 14 |
| <|bos|>The opposite of hot is cold. Hot is a state of high temperature, while cold is a state of |
| <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, |
| <|bos|>My favorite color is blue. I love the sky, the ocean, and the sky. I love |
| <|bos|>If 5*x + 3 = 13, then x is 3. If 5*x + 3 = 13, then |
| 2026-08-10 22:31:14,048 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank6.pt |
| 2026-08-10 22:31:14,754 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank3.pt |
| 2026-08-10 22:31:15,333 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank4.pt |
| 2026-08-10 22:31:17,835 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank5.pt |
| 2026-08-10 22:31:18,694 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank1.pt |
| 2026-08-10 22:31:19,271 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank2.pt |
| 2026-08-10 22:32:18,578 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank7.pt |
| 2026-08-10 22:32:34,385 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved model parameters to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/model_001000.pt |
| 2026-08-10 22:32:34,390 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved metadata to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/meta_001000.json |
| 2026-08-10 22:32:41,351 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001000_rank0.pt |
| step 01000/01500 (66.67%) | loss: 2.425055 | lrm: 1.00 | dt: 2472.32ms | tok/sec: 212,063 | bf16_mfu: 32.38 | epoch: 1 pq: 11 rg: 40 | total time: 41.32m | eta: 20.9m |
| step 01001/01500 (66.73%) | loss: 2.437340 | lrm: 1.00 | dt: 2483.08ms | tok/sec: 211,144 | bf16_mfu: 32.24 | epoch: 1 pq: 11 rg: 40 | total time: 41.36m | eta: 20.8m |
| step 01002/01500 (66.80%) | loss: 2.402506 | lrm: 1.00 | dt: 2487.48ms | tok/sec: 210,770 | bf16_mfu: 32.18 | epoch: 1 pq: 11 rg: 40 | total time: 41.40m | eta: 20.8m |
| step 01003/01500 (66.87%) | loss: 2.402290 | lrm: 1.00 | dt: 2491.41ms | tok/sec: 210,438 | bf16_mfu: 32.13 | epoch: 1 pq: 11 rg: 40 | total time: 41.44m | eta: 20.7m |
| step 01004/01500 (66.93%) | loss: 2.383145 | lrm: 1.00 | dt: 2490.94ms | tok/sec: 210,478 | bf16_mfu: 32.13 | epoch: 1 pq: 11 rg: 48 | total time: 41.49m | eta: 20.7m |
| step 01005/01500 (67.00%) | loss: 2.388188 | lrm: 1.00 | dt: 2496.07ms | tok/sec: 210,045 | bf16_mfu: 32.07 | epoch: 1 pq: 11 rg: 48 | total time: 41.53m | eta: 20.7m |
| step 01006/01500 (67.07%) | loss: 2.405605 | lrm: 1.00 | dt: 2495.65ms | tok/sec: 210,080 | bf16_mfu: 32.07 | epoch: 1 pq: 11 rg: 48 | total time: 41.57m | eta: 20.6m |
| step 01007/01500 (67.13%) | loss: 2.419697 | lrm: 1.00 | dt: 2500.89ms | tok/sec: 209,640 | bf16_mfu: 32.01 | epoch: 1 pq: 11 rg: 48 | total time: 41.61m | eta: 20.6m |
| step 01008/01500 (67.20%) | loss: 2.427633 | lrm: 1.00 | dt: 2499.88ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 48 | total time: 41.65m | eta: 20.5m |
| step 01009/01500 (67.27%) | loss: 2.466596 | lrm: 1.00 | dt: 2500.15ms | tok/sec: 209,702 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 48 | total time: 41.69m | eta: 20.5m |
| step 01010/01500 (67.33%) | loss: 2.444322 | lrm: 1.00 | dt: 2496.52ms | tok/sec: 210,007 | bf16_mfu: 32.06 | epoch: 1 pq: 11 rg: 48 | total time: 41.74m | eta: 20.5m |
| step 01011/01500 (67.40%) | loss: 2.402679 | lrm: 1.00 | dt: 2498.62ms | tok/sec: 209,831 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 48 | total time: 41.78m | eta: 20.4m |
| step 01012/01500 (67.47%) | loss: 2.347197 | lrm: 1.00 | dt: 2498.42ms | tok/sec: 209,847 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 56 | total time: 41.82m | eta: 20.4m |
| step 01013/01500 (67.53%) | loss: 2.349528 | lrm: 1.00 | dt: 2498.35ms | tok/sec: 209,853 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 56 | total time: 41.86m | eta: 20.3m |
| step 01014/01500 (67.60%) | loss: 2.385504 | lrm: 1.00 | dt: 2496.70ms | tok/sec: 209,991 | bf16_mfu: 32.06 | epoch: 1 pq: 11 rg: 56 | total time: 41.90m | eta: 20.3m |
| step 01015/01500 (67.67%) | loss: 2.412254 | lrm: 1.00 | dt: 2505.98ms | tok/sec: 209,214 | bf16_mfu: 31.94 | epoch: 1 pq: 11 rg: 56 | total time: 41.94m | eta: 20.2m |
| step 01016/01500 (67.73%) | loss: 2.448342 | lrm: 1.00 | dt: 2499.72ms | tok/sec: 209,739 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 56 | total time: 41.99m | eta: 20.2m |
| step 01017/01500 (67.80%) | loss: 2.467616 | lrm: 1.00 | dt: 2504.20ms | tok/sec: 209,363 | bf16_mfu: 31.96 | epoch: 1 pq: 11 rg: 56 | total time: 42.03m | eta: 20.2m |
| step 01018/01500 (67.87%) | loss: 2.465448 | lrm: 1.00 | dt: 2494.68ms | tok/sec: 210,162 | bf16_mfu: 32.09 | epoch: 1 pq: 11 rg: 56 | total time: 42.07m | eta: 20.1m |
| step 01019/01500 (67.93%) | loss: 2.379567 | lrm: 1.00 | dt: 2499.98ms | tok/sec: 209,716 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 56 | total time: 42.11m | eta: 20.1m |
| step 01020/01500 (68.00%) | loss: 2.391365 | lrm: 1.00 | dt: 2501.41ms | tok/sec: 209,596 | bf16_mfu: 32.00 | epoch: 1 pq: 11 rg: 56 | total time: 42.15m | eta: 20.0m |
| step 01021/01500 (68.07%) | loss: 2.410615 | lrm: 1.00 | dt: 2496.32ms | tok/sec: 210,024 | bf16_mfu: 32.07 | epoch: 1 pq: 11 rg: 64 | total time: 42.19m | eta: 20.0m |
| step 01022/01500 (68.13%) | loss: 2.457156 | lrm: 1.00 | dt: 2503.78ms | tok/sec: 209,398 | bf16_mfu: 31.97 | epoch: 1 pq: 11 rg: 64 | total time: 42.24m | eta: 19.9m |
| step 01023/01500 (68.20%) | loss: 2.501823 | lrm: 1.00 | dt: 2500.79ms | tok/sec: 209,649 | bf16_mfu: 32.01 | epoch: 1 pq: 11 rg: 64 | total time: 42.28m | eta: 19.9m |
| step 01024/01500 (68.27%) | loss: 2.546373 | lrm: 1.00 | dt: 2496.48ms | tok/sec: 210,010 | bf16_mfu: 32.06 | epoch: 1 pq: 11 rg: 64 | total time: 42.32m | eta: 19.9m |
| step 01025/01500 (68.33%) | loss: 2.576119 | lrm: 1.00 | dt: 2496.50ms | tok/sec: 210,008 | bf16_mfu: 32.06 | epoch: 1 pq: 11 rg: 64 | total time: 42.36m | eta: 19.8m |
| step 01026/01500 (68.40%) | loss: 2.607259 | lrm: 1.00 | dt: 2502.13ms | tok/sec: 209,536 | bf16_mfu: 31.99 | epoch: 1 pq: 11 rg: 64 | total time: 42.40m | eta: 19.8m |
| step 01027/01500 (68.47%) | loss: 2.595531 | lrm: 1.00 | dt: 3246.22ms | tok/sec: 161,507 | bf16_mfu: 24.66 | epoch: 1 pq: 11 rg: 64 | total time: 42.46m | eta: 19.7m |
| step 01028/01500 (68.53%) | loss: 2.574413 | lrm: 1.00 | dt: 2493.00ms | tok/sec: 210,303 | bf16_mfu: 32.11 | epoch: 1 pq: 11 rg: 64 | total time: 42.50m | eta: 19.7m |
| step 01029/01500 (68.60%) | loss: 2.571716 | lrm: 1.00 | dt: 2496.43ms | tok/sec: 210,015 | bf16_mfu: 32.06 | epoch: 1 pq: 11 rg: 72 | total time: 42.54m | eta: 19.7m |
| step 01030/01500 (68.67%) | loss: 2.690301 | lrm: 1.00 | dt: 2499.53ms | tok/sec: 209,754 | bf16_mfu: 32.02 | epoch: 1 pq: 11 rg: 72 | total time: 42.58m | eta: 19.6m |
| step 01031/01500 (68.73%) | loss: 2.722240 | lrm: 1.00 | dt: 2501.88ms | tok/sec: 209,557 | bf16_mfu: 31.99 | epoch: 1 pq: 11 rg: 72 | total time: 42.62m | eta: 19.6m |
| step 01032/01500 (68.80%) | loss: 2.717924 | lrm: 1.00 | dt: 2501.72ms | tok/sec: 209,570 | bf16_mfu: 32.00 | epoch: 1 pq: 11 rg: 72 | total time: 42.66m | eta: 19.5m |
| step 01033/01500 (68.87%) | loss: 2.668257 | lrm: 1.00 | dt: 2496.20ms | tok/sec: 210,034 | bf16_mfu: 32.07 | epoch: 1 pq: 11 rg: 72 | total time: 42.71m | eta: 19.5m |
| step 01034/01500 (68.93%) | loss: 2.682798 | lrm: 1.00 | dt: 2495.25ms | tok/sec: 210,114 | bf16_mfu: 32.08 | epoch: 1 pq: 11 rg: 72 | total time: 42.75m | eta: 19.5m |
| step 01035/01500 (69.00%) | loss: 2.604531 | lrm: 1.00 | dt: 2503.97ms | tok/sec: 209,382 | bf16_mfu: 31.97 | epoch: 1 pq: 11 rg: 72 | total time: 42.79m | eta: 19.4m |
| step 01036/01500 (69.07%) | loss: 2.600583 | lrm: 1.00 | dt: 2498.61ms | tok/sec: 209,832 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 72 | total time: 42.83m | eta: 19.4m |
| step 01037/01500 (69.13%) | loss: 2.602779 | lrm: 1.00 | dt: 2494.94ms | tok/sec: 210,140 | bf16_mfu: 32.08 | epoch: 1 pq: 11 rg: 80 | total time: 42.87m | eta: 19.3m |
| step 01038/01500 (69.20%) | loss: 2.547325 | lrm: 1.00 | dt: 2502.10ms | tok/sec: 209,539 | bf16_mfu: 31.99 | epoch: 1 pq: 11 rg: 80 | total time: 42.91m | eta: 19.3m |
| step 01039/01500 (69.27%) | loss: 2.541426 | lrm: 1.00 | dt: 2507.52ms | tok/sec: 209,086 | bf16_mfu: 31.92 | epoch: 1 pq: 11 rg: 80 | total time: 42.96m | eta: 19.2m |
| step 01040/01500 (69.33%) | loss: 2.568093 | lrm: 1.00 | dt: 2500.88ms | tok/sec: 209,641 | bf16_mfu: 32.01 | epoch: 1 pq: 11 rg: 80 | total time: 43.00m | eta: 19.2m |
| step 01041/01500 (69.40%) | loss: 2.592260 | lrm: 1.00 | dt: 2505.36ms | tok/sec: 209,266 | bf16_mfu: 31.95 | epoch: 1 pq: 11 rg: 80 | total time: 43.04m | eta: 19.2m |
| step 01042/01500 (69.47%) | loss: 2.612483 | lrm: 1.00 | dt: 2501.88ms | tok/sec: 209,557 | bf16_mfu: 31.99 | epoch: 1 pq: 11 rg: 80 | total time: 43.08m | eta: 19.1m |
| step 01043/01500 (69.53%) | loss: 2.596725 | lrm: 1.00 | dt: 2499.36ms | tok/sec: 209,768 | bf16_mfu: 32.03 | epoch: 1 pq: 11 rg: 80 | total time: 43.12m | eta: 19.1m |
| step 01044/01500 (69.60%) | loss: 2.567148 | lrm: 1.00 | dt: 2498.40ms | tok/sec: 209,849 | bf16_mfu: 32.04 | epoch: 1 pq: 11 rg: 80 | total time: 43.16m | eta: 19.0m |
| step 01045/01500 (69.67%) | loss: 2.558828 | lrm: 1.00 | dt: 2500.85ms | tok/sec: 209,644 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 0 | total time: 43.21m | eta: 19.0m |
| step 01046/01500 (69.73%) | loss: 2.535682 | lrm: 1.00 | dt: 2503.67ms | tok/sec: 209,408 | bf16_mfu: 31.97 | epoch: 1 pq: 12 rg: 0 | total time: 43.25m | eta: 19.0m |
| step 01047/01500 (69.80%) | loss: 2.538711 | lrm: 1.00 | dt: 2501.11ms | tok/sec: 209,622 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 0 | total time: 43.29m | eta: 18.9m |
| step 01048/01500 (69.87%) | loss: 2.531904 | lrm: 1.00 | dt: 2497.34ms | tok/sec: 209,938 | bf16_mfu: 32.05 | epoch: 1 pq: 12 rg: 0 | total time: 43.33m | eta: 18.9m |
| step 01049/01500 (69.93%) | loss: 2.528915 | lrm: 1.00 | dt: 2499.77ms | tok/sec: 209,734 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 0 | total time: 43.37m | eta: 18.8m |
| step 01050/01500 (70.00%) | loss: 2.543039 | lrm: 1.00 | dt: 2500.74ms | tok/sec: 209,652 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 0 | total time: 43.41m | eta: 18.8m |
| step 01051/01500 (70.07%) | loss: 2.557352 | lrm: 1.00 | dt: 2504.41ms | tok/sec: 209,346 | bf16_mfu: 31.96 | epoch: 1 pq: 12 rg: 0 | total time: 43.46m | eta: 18.7m |
| step 01052/01500 (70.13%) | loss: 2.538566 | lrm: 1.00 | dt: 2494.94ms | tok/sec: 210,140 | bf16_mfu: 32.08 | epoch: 1 pq: 12 rg: 0 | total time: 43.50m | eta: 18.7m |
| step 01053/01500 (70.20%) | loss: 2.524232 | lrm: 1.00 | dt: 2497.16ms | tok/sec: 209,953 | bf16_mfu: 32.05 | epoch: 1 pq: 12 rg: 8 | total time: 43.54m | eta: 18.7m |
| step 01054/01500 (70.27%) | loss: 2.535897 | lrm: 1.00 | dt: 2502.53ms | tok/sec: 209,503 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 8 | total time: 43.58m | eta: 18.6m |
| step 01055/01500 (70.33%) | loss: 2.511785 | lrm: 1.00 | dt: 2497.90ms | tok/sec: 209,891 | bf16_mfu: 32.05 | epoch: 1 pq: 12 rg: 8 | total time: 43.62m | eta: 18.6m |
| step 01056/01500 (70.40%) | loss: 2.540567 | lrm: 1.00 | dt: 2499.79ms | tok/sec: 209,733 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 8 | total time: 43.66m | eta: 18.5m |
| step 01057/01500 (70.47%) | loss: 2.489501 | lrm: 1.00 | dt: 2500.49ms | tok/sec: 209,673 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 8 | total time: 43.71m | eta: 18.5m |
| step 01058/01500 (70.53%) | loss: 2.484187 | lrm: 1.00 | dt: 2502.51ms | tok/sec: 209,504 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 8 | total time: 43.75m | eta: 18.5m |
| step 01059/01500 (70.60%) | loss: 2.487643 | lrm: 1.00 | dt: 2503.50ms | tok/sec: 209,421 | bf16_mfu: 31.97 | epoch: 1 pq: 12 rg: 8 | total time: 43.79m | eta: 18.4m |
| step 01060/01500 (70.67%) | loss: 2.465699 | lrm: 1.00 | dt: 2500.88ms | tok/sec: 209,641 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 8 | total time: 43.83m | eta: 18.4m |
| step 01061/01500 (70.73%) | loss: 2.440185 | lrm: 1.00 | dt: 2499.36ms | tok/sec: 209,768 | bf16_mfu: 32.03 | epoch: 1 pq: 12 rg: 16 | total time: 43.87m | eta: 18.3m |
| step 01062/01500 (70.80%) | loss: 2.421542 | lrm: 1.00 | dt: 2498.22ms | tok/sec: 209,864 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 16 | total time: 43.91m | eta: 18.3m |
| step 01063/01500 (70.87%) | loss: 2.416919 | lrm: 1.00 | dt: 2499.87ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 16 | total time: 43.96m | eta: 18.2m |
| step 01064/01500 (70.93%) | loss: 2.430497 | lrm: 1.00 | dt: 2498.25ms | tok/sec: 209,862 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 16 | total time: 44.00m | eta: 18.2m |
| step 01065/01500 (71.00%) | loss: 2.405876 | lrm: 1.00 | dt: 2499.87ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 16 | total time: 44.04m | eta: 18.2m |
| step 01066/01500 (71.07%) | loss: 2.428573 | lrm: 1.00 | dt: 2498.56ms | tok/sec: 209,836 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 16 | total time: 44.08m | eta: 18.1m |
| step 01067/01500 (71.13%) | loss: 2.404212 | lrm: 1.00 | dt: 2499.30ms | tok/sec: 209,774 | bf16_mfu: 32.03 | epoch: 1 pq: 12 rg: 16 | total time: 44.12m | eta: 18.1m |
| step 01068/01500 (71.20%) | loss: 2.394564 | lrm: 1.00 | dt: 2497.63ms | tok/sec: 209,914 | bf16_mfu: 32.05 | epoch: 1 pq: 12 rg: 16 | total time: 44.16m | eta: 18.0m |
| step 01069/01500 (71.27%) | loss: 2.417578 | lrm: 1.00 | dt: 2501.42ms | tok/sec: 209,596 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 24 | total time: 44.21m | eta: 18.0m |
| step 01070/01500 (71.33%) | loss: 2.422496 | lrm: 1.00 | dt: 2501.65ms | tok/sec: 209,576 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 24 | total time: 44.25m | eta: 17.9m |
| step 01071/01500 (71.40%) | loss: 2.426493 | lrm: 1.00 | dt: 2500.84ms | tok/sec: 209,644 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 24 | total time: 44.29m | eta: 17.9m |
| step 01072/01500 (71.47%) | loss: 2.430037 | lrm: 1.00 | dt: 2502.49ms | tok/sec: 209,506 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 24 | total time: 44.33m | eta: 17.9m |
| step 01073/01500 (71.53%) | loss: 2.414623 | lrm: 1.00 | dt: 2501.72ms | tok/sec: 209,571 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 24 | total time: 44.37m | eta: 17.8m |
| step 01074/01500 (71.60%) | loss: 2.452738 | lrm: 1.00 | dt: 2502.09ms | tok/sec: 209,540 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 24 | total time: 44.41m | eta: 17.8m |
| step 01075/01500 (71.67%) | loss: 2.483931 | lrm: 1.00 | dt: 2504.53ms | tok/sec: 209,336 | bf16_mfu: 31.96 | epoch: 1 pq: 12 rg: 24 | total time: 44.46m | eta: 17.7m |
| step 01076/01500 (71.73%) | loss: 2.512765 | lrm: 1.00 | dt: 2503.36ms | tok/sec: 209,434 | bf16_mfu: 31.98 | epoch: 1 pq: 12 rg: 24 | total time: 44.50m | eta: 17.7m |
| step 01077/01500 (71.80%) | loss: 2.489914 | lrm: 1.00 | dt: 2501.77ms | tok/sec: 209,566 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 32 | total time: 44.54m | eta: 17.7m |
| step 01078/01500 (71.87%) | loss: 2.481657 | lrm: 1.00 | dt: 2502.19ms | tok/sec: 209,531 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 32 | total time: 44.58m | eta: 17.6m |
| step 01079/01500 (71.93%) | loss: 2.432451 | lrm: 1.00 | dt: 2501.46ms | tok/sec: 209,593 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 32 | total time: 44.62m | eta: 17.6m |
| step 01080/01500 (72.00%) | loss: 2.459475 | lrm: 1.00 | dt: 2501.02ms | tok/sec: 209,629 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 32 | total time: 44.66m | eta: 17.5m |
| step 01081/01500 (72.07%) | loss: 2.503668 | lrm: 1.00 | dt: 2501.72ms | tok/sec: 209,570 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 32 | total time: 44.71m | eta: 17.5m |
| step 01082/01500 (72.13%) | loss: 2.481614 | lrm: 1.00 | dt: 2500.24ms | tok/sec: 209,694 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 32 | total time: 44.75m | eta: 17.4m |
| step 01083/01500 (72.20%) | loss: 2.539859 | lrm: 1.00 | dt: 2499.89ms | tok/sec: 209,724 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 32 | total time: 44.79m | eta: 17.4m |
| step 01084/01500 (72.27%) | loss: 2.565516 | lrm: 1.00 | dt: 2501.42ms | tok/sec: 209,596 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 32 | total time: 44.83m | eta: 17.4m |
| step 01085/01500 (72.33%) | loss: 2.591217 | lrm: 1.00 | dt: 2497.61ms | tok/sec: 209,915 | bf16_mfu: 32.05 | epoch: 1 pq: 12 rg: 40 | total time: 44.87m | eta: 17.3m |
| step 01086/01500 (72.40%) | loss: 2.577020 | lrm: 1.00 | dt: 2499.82ms | tok/sec: 209,730 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 40 | total time: 44.91m | eta: 17.3m |
| step 01087/01500 (72.47%) | loss: 2.575139 | lrm: 1.00 | dt: 2498.06ms | tok/sec: 209,878 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 40 | total time: 44.96m | eta: 17.2m |
| step 01088/01500 (72.53%) | loss: 2.565641 | lrm: 1.00 | dt: 2499.84ms | tok/sec: 209,728 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 40 | total time: 45.00m | eta: 17.2m |
| step 01089/01500 (72.60%) | loss: 2.549572 | lrm: 1.00 | dt: 2499.69ms | tok/sec: 209,741 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 40 | total time: 45.04m | eta: 17.2m |
| step 01090/01500 (72.67%) | loss: 2.563537 | lrm: 1.00 | dt: 2499.82ms | tok/sec: 209,730 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 40 | total time: 45.08m | eta: 17.1m |
| step 01091/01500 (72.73%) | loss: 2.590664 | lrm: 1.00 | dt: 2498.18ms | tok/sec: 209,868 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 40 | total time: 45.12m | eta: 17.1m |
| step 01092/01500 (72.80%) | loss: 2.578195 | lrm: 1.00 | dt: 2504.73ms | tok/sec: 209,319 | bf16_mfu: 31.96 | epoch: 1 pq: 12 rg: 40 | total time: 45.16m | eta: 17.0m |
| step 01093/01500 (72.87%) | loss: 2.571242 | lrm: 1.00 | dt: 2502.18ms | tok/sec: 209,532 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 48 | total time: 45.21m | eta: 17.0m |
| step 01094/01500 (72.93%) | loss: 2.579990 | lrm: 1.00 | dt: 2501.77ms | tok/sec: 209,567 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 48 | total time: 45.25m | eta: 16.9m |
| step 01095/01500 (73.00%) | loss: 2.537517 | lrm: 1.00 | dt: 2505.12ms | tok/sec: 209,286 | bf16_mfu: 31.95 | epoch: 1 pq: 12 rg: 48 | total time: 45.29m | eta: 16.9m |
| step 01096/01500 (73.07%) | loss: 2.553105 | lrm: 1.00 | dt: 2500.22ms | tok/sec: 209,696 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 48 | total time: 45.33m | eta: 16.9m |
| step 01097/01500 (73.13%) | loss: 2.558843 | lrm: 1.00 | dt: 2501.56ms | tok/sec: 209,584 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 48 | total time: 45.37m | eta: 16.8m |
| step 01098/01500 (73.20%) | loss: 2.570094 | lrm: 1.00 | dt: 2499.03ms | tok/sec: 209,796 | bf16_mfu: 32.03 | epoch: 1 pq: 12 rg: 48 | total time: 45.42m | eta: 16.8m |
| step 01099/01500 (73.27%) | loss: 2.582346 | lrm: 1.00 | dt: 2503.95ms | tok/sec: 209,384 | bf16_mfu: 31.97 | epoch: 1 pq: 12 rg: 48 | total time: 45.46m | eta: 16.7m |
| Step 01100 | Validation bpb: 0.732576 |
| Step 01100 | ICL early: 2.6734 late: 2.8559 score: +0.1825 | Induction loss: 9.4218 acc: 0.0877 (random-half: 13.9912) |
| step 01100/01500 (73.33%) | loss: 2.586440 | lrm: 1.00 | dt: 2496.63ms | tok/sec: 209,997 | bf16_mfu: 32.06 | epoch: 1 pq: 12 rg: 48 | total time: 45.50m | eta: 16.7m |
| step 01101/01500 (73.40%) | loss: 2.580091 | lrm: 1.00 | dt: 2499.67ms | tok/sec: 209,742 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 56 | total time: 45.54m | eta: 16.7m |
| step 01102/01500 (73.47%) | loss: 2.545139 | lrm: 1.00 | dt: 2499.03ms | tok/sec: 209,796 | bf16_mfu: 32.03 | epoch: 1 pq: 12 rg: 56 | total time: 45.58m | eta: 16.6m |
| step 01103/01500 (73.53%) | loss: 2.540063 | lrm: 1.00 | dt: 2500.63ms | tok/sec: 209,661 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 56 | total time: 45.62m | eta: 16.6m |
| step 01104/01500 (73.60%) | loss: 2.527436 | lrm: 1.00 | dt: 2502.48ms | tok/sec: 209,506 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 56 | total time: 45.67m | eta: 16.5m |
| step 01105/01500 (73.67%) | loss: 2.461445 | lrm: 1.00 | dt: 2500.80ms | tok/sec: 209,647 | bf16_mfu: 32.01 | epoch: 1 pq: 12 rg: 56 | total time: 45.71m | eta: 16.5m |
| step 01106/01500 (73.73%) | loss: 2.489434 | lrm: 1.00 | dt: 2504.12ms | tok/sec: 209,370 | bf16_mfu: 31.97 | epoch: 1 pq: 12 rg: 56 | total time: 45.75m | eta: 16.4m |
| step 01107/01500 (73.80%) | loss: 2.519811 | lrm: 1.00 | dt: 2499.64ms | tok/sec: 209,745 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 56 | total time: 45.79m | eta: 16.4m |
| step 01108/01500 (73.87%) | loss: 2.539433 | lrm: 1.00 | dt: 3348.74ms | tok/sec: 156,562 | bf16_mfu: 23.90 | epoch: 1 pq: 12 rg: 56 | total time: 45.85m | eta: 16.4m |
| step 01109/01500 (73.93%) | loss: 2.525969 | lrm: 1.00 | dt: 2499.52ms | tok/sec: 209,755 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 64 | total time: 45.89m | eta: 16.3m |
| step 01110/01500 (74.00%) | loss: 2.483961 | lrm: 1.00 | dt: 2501.22ms | tok/sec: 209,613 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 64 | total time: 45.93m | eta: 16.3m |
| step 01111/01500 (74.07%) | loss: 2.447209 | lrm: 1.00 | dt: 2508.64ms | tok/sec: 208,992 | bf16_mfu: 31.91 | epoch: 1 pq: 12 rg: 64 | total time: 45.97m | eta: 16.2m |
| step 01112/01500 (74.13%) | loss: 2.446468 | lrm: 1.00 | dt: 2502.01ms | tok/sec: 209,546 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 64 | total time: 46.01m | eta: 16.2m |
| step 01113/01500 (74.20%) | loss: 2.491553 | lrm: 1.00 | dt: 2506.43ms | tok/sec: 209,177 | bf16_mfu: 31.94 | epoch: 1 pq: 12 rg: 64 | total time: 46.05m | eta: 16.2m |
| step 01114/01500 (74.27%) | loss: 2.459502 | lrm: 1.00 | dt: 2497.85ms | tok/sec: 209,895 | bf16_mfu: 32.05 | epoch: 1 pq: 12 rg: 64 | total time: 46.10m | eta: 16.1m |
| step 01115/01500 (74.33%) | loss: 2.445792 | lrm: 1.00 | dt: 2501.11ms | tok/sec: 209,622 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 64 | total time: 46.14m | eta: 16.1m |
| step 01116/01500 (74.40%) | loss: 2.443582 | lrm: 1.00 | dt: 2502.09ms | tok/sec: 209,540 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 64 | total time: 46.18m | eta: 16.0m |
| step 01117/01500 (74.47%) | loss: 2.398858 | lrm: 1.00 | dt: 2497.99ms | tok/sec: 209,883 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 72 | total time: 46.22m | eta: 16.0m |
| step 01118/01500 (74.53%) | loss: 2.435793 | lrm: 1.00 | dt: 2502.90ms | tok/sec: 209,472 | bf16_mfu: 31.98 | epoch: 1 pq: 12 rg: 72 | total time: 46.26m | eta: 15.9m |
| step 01119/01500 (74.60%) | loss: 2.450739 | lrm: 1.00 | dt: 2501.60ms | tok/sec: 209,580 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 72 | total time: 46.30m | eta: 15.9m |
| step 01120/01500 (74.67%) | loss: 2.453562 | lrm: 1.00 | dt: 2501.61ms | tok/sec: 209,580 | bf16_mfu: 32.00 | epoch: 1 pq: 12 rg: 72 | total time: 46.35m | eta: 15.9m |
| step 01121/01500 (74.73%) | loss: 2.458797 | lrm: 1.00 | dt: 2503.79ms | tok/sec: 209,397 | bf16_mfu: 31.97 | epoch: 1 pq: 12 rg: 72 | total time: 46.39m | eta: 15.8m |
| step 01122/01500 (74.80%) | loss: 2.506607 | lrm: 1.00 | dt: 2503.03ms | tok/sec: 209,461 | bf16_mfu: 31.98 | epoch: 1 pq: 12 rg: 72 | total time: 46.43m | eta: 15.8m |
| step 01123/01500 (74.87%) | loss: 2.531235 | lrm: 1.00 | dt: 2499.23ms | tok/sec: 209,779 | bf16_mfu: 32.03 | epoch: 1 pq: 12 rg: 72 | total time: 46.47m | eta: 15.7m |
| step 01124/01500 (74.93%) | loss: 2.536931 | lrm: 1.00 | dt: 2506.69ms | tok/sec: 209,155 | bf16_mfu: 31.93 | epoch: 1 pq: 12 rg: 72 | total time: 46.51m | eta: 15.7m |
| step 01125/01500 (75.00%) | loss: 2.524225 | lrm: 1.00 | dt: 2503.65ms | tok/sec: 209,409 | bf16_mfu: 31.97 | epoch: 1 pq: 12 rg: 80 | total time: 46.56m | eta: 15.7m |
| step 01126/01500 (75.07%) | loss: 2.552255 | lrm: 1.00 | dt: 2498.45ms | tok/sec: 209,845 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 80 | total time: 46.60m | eta: 15.6m |
| step 01127/01500 (75.13%) | loss: 2.552342 | lrm: 1.00 | dt: 2498.11ms | tok/sec: 209,873 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 80 | total time: 46.64m | eta: 15.6m |
| step 01128/01500 (75.20%) | loss: 2.509837 | lrm: 1.00 | dt: 2502.03ms | tok/sec: 209,544 | bf16_mfu: 31.99 | epoch: 1 pq: 12 rg: 80 | total time: 46.68m | eta: 15.5m |
| step 01129/01500 (75.27%) | loss: 2.483425 | lrm: 1.00 | dt: 2499.42ms | tok/sec: 209,764 | bf16_mfu: 32.03 | epoch: 1 pq: 12 rg: 80 | total time: 46.72m | eta: 15.5m |
| step 01130/01500 (75.33%) | loss: 2.458530 | lrm: 1.00 | dt: 2498.58ms | tok/sec: 209,834 | bf16_mfu: 32.04 | epoch: 1 pq: 12 rg: 80 | total time: 46.76m | eta: 15.4m |
| step 01131/01500 (75.40%) | loss: 2.472561 | lrm: 1.00 | dt: 2497.88ms | tok/sec: 209,893 | bf16_mfu: 32.05 | epoch: 1 pq: 12 rg: 80 | total time: 46.80m | eta: 15.4m |
| step 01132/01500 (75.47%) | loss: 2.425508 | lrm: 1.00 | dt: 2499.91ms | tok/sec: 209,722 | bf16_mfu: 32.02 | epoch: 1 pq: 12 rg: 80 | total time: 46.85m | eta: 15.4m |
| step 01133/01500 (75.53%) | loss: 2.371697 | lrm: 1.00 | dt: 2494.87ms | tok/sec: 210,146 | bf16_mfu: 32.08 | epoch: 1 pq: 13 rg: 0 | total time: 46.89m | eta: 15.3m |
| step 01134/01500 (75.60%) | loss: 2.387674 | lrm: 1.00 | dt: 2495.54ms | tok/sec: 210,090 | bf16_mfu: 32.08 | epoch: 1 pq: 13 rg: 0 | total time: 46.93m | eta: 15.3m |
| step 01135/01500 (75.67%) | loss: 2.403613 | lrm: 1.00 | dt: 2497.57ms | tok/sec: 209,919 | bf16_mfu: 32.05 | epoch: 1 pq: 13 rg: 0 | total time: 46.97m | eta: 15.2m |
| step 01136/01500 (75.73%) | loss: 2.420161 | lrm: 1.00 | dt: 2504.45ms | tok/sec: 209,342 | bf16_mfu: 31.96 | epoch: 1 pq: 13 rg: 0 | total time: 47.01m | eta: 15.2m |
| step 01137/01500 (75.80%) | loss: 2.432826 | lrm: 1.00 | dt: 2500.64ms | tok/sec: 209,661 | bf16_mfu: 32.01 | epoch: 1 pq: 13 rg: 0 | total time: 47.05m | eta: 15.2m |
| step 01138/01500 (75.87%) | loss: 2.469571 | lrm: 1.00 | dt: 2499.01ms | tok/sec: 209,798 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 0 | total time: 47.10m | eta: 15.1m |
| step 01139/01500 (75.93%) | loss: 2.512893 | lrm: 1.00 | dt: 2501.99ms | tok/sec: 209,548 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 0 | total time: 47.14m | eta: 15.1m |
| step 01140/01500 (76.00%) | loss: 2.490199 | lrm: 1.00 | dt: 2499.42ms | tok/sec: 209,763 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 0 | total time: 47.18m | eta: 15.0m |
| step 01141/01500 (76.07%) | loss: 2.503078 | lrm: 1.00 | dt: 2498.09ms | tok/sec: 209,875 | bf16_mfu: 32.04 | epoch: 1 pq: 13 rg: 8 | total time: 47.22m | eta: 15.0m |
| step 01142/01500 (76.13%) | loss: 2.508764 | lrm: 1.00 | dt: 2496.40ms | tok/sec: 210,017 | bf16_mfu: 32.06 | epoch: 1 pq: 13 rg: 8 | total time: 47.26m | eta: 14.9m |
| step 01143/01500 (76.20%) | loss: 2.497010 | lrm: 1.00 | dt: 2496.84ms | tok/sec: 209,980 | bf16_mfu: 32.06 | epoch: 1 pq: 13 rg: 8 | total time: 47.30m | eta: 14.9m |
| step 01144/01500 (76.27%) | loss: 2.474529 | lrm: 1.00 | dt: 2502.73ms | tok/sec: 209,486 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 8 | total time: 47.35m | eta: 14.9m |
| step 01145/01500 (76.33%) | loss: 2.409986 | lrm: 1.00 | dt: 2502.98ms | tok/sec: 209,465 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 8 | total time: 47.39m | eta: 14.8m |
| step 01146/01500 (76.40%) | loss: 2.436207 | lrm: 1.00 | dt: 2499.66ms | tok/sec: 209,743 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 8 | total time: 47.43m | eta: 14.8m |
| step 01147/01500 (76.47%) | loss: 2.360981 | lrm: 1.00 | dt: 2500.45ms | tok/sec: 209,677 | bf16_mfu: 32.01 | epoch: 1 pq: 13 rg: 8 | total time: 47.47m | eta: 14.7m |
| step 01148/01500 (76.53%) | loss: 2.366878 | lrm: 1.00 | dt: 2498.45ms | tok/sec: 209,845 | bf16_mfu: 32.04 | epoch: 1 pq: 13 rg: 8 | total time: 47.51m | eta: 14.7m |
| step 01149/01500 (76.60%) | loss: 2.446126 | lrm: 1.00 | dt: 2500.18ms | tok/sec: 209,699 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 16 | total time: 47.55m | eta: 14.7m |
| step 01150/01500 (76.67%) | loss: 2.529810 | lrm: 1.00 | dt: 2496.55ms | tok/sec: 210,004 | bf16_mfu: 32.06 | epoch: 1 pq: 13 rg: 16 | total time: 47.60m | eta: 14.6m |
| step 01151/01500 (76.73%) | loss: 2.563946 | lrm: 1.00 | dt: 2503.01ms | tok/sec: 209,462 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 16 | total time: 47.64m | eta: 14.6m |
| step 01152/01500 (76.80%) | loss: 2.584806 | lrm: 1.00 | dt: 2500.47ms | tok/sec: 209,675 | bf16_mfu: 32.01 | epoch: 1 pq: 13 rg: 16 | total time: 47.68m | eta: 14.5m |
| step 01153/01500 (76.87%) | loss: 2.592696 | lrm: 1.00 | dt: 2499.77ms | tok/sec: 209,734 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 16 | total time: 47.72m | eta: 14.5m |
| step 01154/01500 (76.93%) | loss: 2.579082 | lrm: 1.00 | dt: 2498.89ms | tok/sec: 209,808 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 16 | total time: 47.76m | eta: 14.4m |
| step 01155/01500 (77.00%) | loss: 2.558637 | lrm: 1.00 | dt: 2501.73ms | tok/sec: 209,570 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 16 | total time: 47.80m | eta: 14.4m |
| step 01156/01500 (77.07%) | loss: 2.556868 | lrm: 1.00 | dt: 2503.23ms | tok/sec: 209,444 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 16 | total time: 47.85m | eta: 14.4m |
| step 01157/01500 (77.13%) | loss: 2.561193 | lrm: 1.00 | dt: 2501.57ms | tok/sec: 209,583 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 16 | total time: 47.89m | eta: 14.3m |
| step 01158/01500 (77.20%) | loss: 2.567330 | lrm: 1.00 | dt: 2499.27ms | tok/sec: 209,776 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 24 | total time: 47.93m | eta: 14.3m |
| step 01159/01500 (77.27%) | loss: 2.534040 | lrm: 1.00 | dt: 2502.83ms | tok/sec: 209,477 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 24 | total time: 47.97m | eta: 14.2m |
| step 01160/01500 (77.33%) | loss: 2.520394 | lrm: 1.00 | dt: 2502.64ms | tok/sec: 209,493 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 24 | total time: 48.01m | eta: 14.2m |
| step 01161/01500 (77.40%) | loss: 2.539431 | lrm: 1.00 | dt: 2499.65ms | tok/sec: 209,744 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 24 | total time: 48.05m | eta: 14.2m |
| step 01162/01500 (77.47%) | loss: 2.542988 | lrm: 1.00 | dt: 2504.93ms | tok/sec: 209,302 | bf16_mfu: 31.96 | epoch: 1 pq: 13 rg: 24 | total time: 48.10m | eta: 14.1m |
| step 01163/01500 (77.53%) | loss: 2.557495 | lrm: 1.00 | dt: 2500.02ms | tok/sec: 209,713 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 24 | total time: 48.14m | eta: 14.1m |
| step 01164/01500 (77.60%) | loss: 2.583253 | lrm: 1.00 | dt: 2499.44ms | tok/sec: 209,762 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 24 | total time: 48.18m | eta: 14.0m |
| step 01165/01500 (77.67%) | loss: 2.549616 | lrm: 1.00 | dt: 2496.71ms | tok/sec: 209,991 | bf16_mfu: 32.06 | epoch: 1 pq: 13 rg: 24 | total time: 48.22m | eta: 14.0m |
| step 01166/01500 (77.73%) | loss: 2.587962 | lrm: 1.00 | dt: 2498.82ms | tok/sec: 209,813 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 32 | total time: 48.26m | eta: 13.9m |
| step 01167/01500 (77.80%) | loss: 2.556298 | lrm: 1.00 | dt: 2501.40ms | tok/sec: 209,598 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 32 | total time: 48.30m | eta: 13.9m |
| step 01168/01500 (77.87%) | loss: 2.576640 | lrm: 1.00 | dt: 2496.13ms | tok/sec: 210,040 | bf16_mfu: 32.07 | epoch: 1 pq: 13 rg: 32 | total time: 48.35m | eta: 13.9m |
| step 01169/01500 (77.93%) | loss: 2.577357 | lrm: 1.00 | dt: 2501.96ms | tok/sec: 209,550 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 32 | total time: 48.39m | eta: 13.8m |
| step 01170/01500 (78.00%) | loss: 2.579842 | lrm: 1.00 | dt: 2506.18ms | tok/sec: 209,197 | bf16_mfu: 31.94 | epoch: 1 pq: 13 rg: 32 | total time: 48.43m | eta: 13.8m |
| step 01171/01500 (78.07%) | loss: 2.553053 | lrm: 1.00 | dt: 2501.56ms | tok/sec: 209,584 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 32 | total time: 48.47m | eta: 13.7m |
| step 01172/01500 (78.13%) | loss: 2.505223 | lrm: 1.00 | dt: 2500.23ms | tok/sec: 209,696 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 32 | total time: 48.51m | eta: 13.7m |
| step 01173/01500 (78.20%) | loss: 2.533689 | lrm: 1.00 | dt: 2501.21ms | tok/sec: 209,614 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 32 | total time: 48.56m | eta: 13.7m |
| step 01174/01500 (78.27%) | loss: 2.529751 | lrm: 1.00 | dt: 2504.64ms | tok/sec: 209,326 | bf16_mfu: 31.96 | epoch: 1 pq: 13 rg: 40 | total time: 48.60m | eta: 13.6m |
| step 01175/01500 (78.33%) | loss: 2.530845 | lrm: 1.00 | dt: 2502.42ms | tok/sec: 209,512 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 40 | total time: 48.64m | eta: 13.6m |
| step 01176/01500 (78.40%) | loss: 2.569716 | lrm: 1.00 | dt: 2498.02ms | tok/sec: 209,881 | bf16_mfu: 32.04 | epoch: 1 pq: 13 rg: 40 | total time: 48.68m | eta: 13.5m |
| step 01177/01500 (78.47%) | loss: 2.557234 | lrm: 1.00 | dt: 2504.77ms | tok/sec: 209,316 | bf16_mfu: 31.96 | epoch: 1 pq: 13 rg: 40 | total time: 48.72m | eta: 13.5m |
| step 01178/01500 (78.53%) | loss: 2.550491 | lrm: 1.00 | dt: 2505.66ms | tok/sec: 209,241 | bf16_mfu: 31.95 | epoch: 1 pq: 13 rg: 40 | total time: 48.76m | eta: 13.4m |
| step 01179/01500 (78.60%) | loss: 2.539480 | lrm: 1.00 | dt: 2504.82ms | tok/sec: 209,311 | bf16_mfu: 31.96 | epoch: 1 pq: 13 rg: 40 | total time: 48.81m | eta: 13.4m |
| step 01180/01500 (78.67%) | loss: 2.553053 | lrm: 1.00 | dt: 2498.57ms | tok/sec: 209,835 | bf16_mfu: 32.04 | epoch: 1 pq: 13 rg: 40 | total time: 48.85m | eta: 13.4m |
| step 01181/01500 (78.73%) | loss: 2.548203 | lrm: 1.00 | dt: 2503.52ms | tok/sec: 209,420 | bf16_mfu: 31.97 | epoch: 1 pq: 13 rg: 40 | total time: 48.89m | eta: 13.3m |
| step 01182/01500 (78.80%) | loss: 2.497362 | lrm: 1.00 | dt: 2500.40ms | tok/sec: 209,681 | bf16_mfu: 32.01 | epoch: 1 pq: 13 rg: 48 | total time: 48.93m | eta: 13.3m |
| step 01183/01500 (78.87%) | loss: 2.533041 | lrm: 1.00 | dt: 2500.14ms | tok/sec: 209,703 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 48 | total time: 48.97m | eta: 13.2m |
| step 01184/01500 (78.93%) | loss: 2.502415 | lrm: 1.00 | dt: 2499.05ms | tok/sec: 209,794 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 48 | total time: 49.01m | eta: 13.2m |
| step 01185/01500 (79.00%) | loss: 2.432266 | lrm: 1.00 | dt: 2499.68ms | tok/sec: 209,742 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 48 | total time: 49.06m | eta: 13.2m |
| step 01186/01500 (79.07%) | loss: 2.429734 | lrm: 1.00 | dt: 2499.95ms | tok/sec: 209,719 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 48 | total time: 49.10m | eta: 13.1m |
| step 01187/01500 (79.13%) | loss: 2.438783 | lrm: 1.00 | dt: 2503.01ms | tok/sec: 209,462 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 48 | total time: 49.14m | eta: 13.1m |
| step 01188/01500 (79.20%) | loss: 2.412814 | lrm: 1.00 | dt: 2503.67ms | tok/sec: 209,407 | bf16_mfu: 31.97 | epoch: 1 pq: 13 rg: 48 | total time: 49.18m | eta: 13.0m |
| step 01189/01500 (79.27%) | loss: 2.417956 | lrm: 1.00 | dt: 3166.04ms | tok/sec: 165,597 | bf16_mfu: 25.28 | epoch: 1 pq: 13 rg: 48 | total time: 49.23m | eta: 13.0m |
| step 01190/01500 (79.33%) | loss: 2.391732 | lrm: 1.00 | dt: 2495.70ms | tok/sec: 210,076 | bf16_mfu: 32.07 | epoch: 1 pq: 13 rg: 56 | total time: 49.27m | eta: 12.9m |
| step 01191/01500 (79.40%) | loss: 2.413218 | lrm: 1.00 | dt: 2500.86ms | tok/sec: 209,643 | bf16_mfu: 32.01 | epoch: 1 pq: 13 rg: 56 | total time: 49.32m | eta: 12.9m |
| step 01192/01500 (79.47%) | loss: 2.409355 | lrm: 1.00 | dt: 2498.83ms | tok/sec: 209,813 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 56 | total time: 49.36m | eta: 12.9m |
| step 01193/01500 (79.53%) | loss: 2.403850 | lrm: 1.00 | dt: 2502.16ms | tok/sec: 209,534 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 56 | total time: 49.40m | eta: 12.8m |
| step 01194/01500 (79.60%) | loss: 2.383438 | lrm: 1.00 | dt: 2501.48ms | tok/sec: 209,590 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 56 | total time: 49.44m | eta: 12.8m |
| step 01195/01500 (79.67%) | loss: 2.384559 | lrm: 1.00 | dt: 2499.75ms | tok/sec: 209,735 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 56 | total time: 49.48m | eta: 12.7m |
| step 01196/01500 (79.73%) | loss: 2.327422 | lrm: 1.00 | dt: 2497.07ms | tok/sec: 209,961 | bf16_mfu: 32.06 | epoch: 1 pq: 13 rg: 56 | total time: 49.52m | eta: 12.7m |
| step 01197/01500 (79.80%) | loss: 2.384903 | lrm: 1.00 | dt: 2504.17ms | tok/sec: 209,365 | bf16_mfu: 31.96 | epoch: 1 pq: 13 rg: 56 | total time: 49.57m | eta: 12.7m |
| step 01198/01500 (79.87%) | loss: 2.424343 | lrm: 1.00 | dt: 2494.40ms | tok/sec: 210,186 | bf16_mfu: 32.09 | epoch: 1 pq: 13 rg: 64 | total time: 49.61m | eta: 12.6m |
| step 01199/01500 (79.93%) | loss: 2.430876 | lrm: 1.00 | dt: 2497.48ms | tok/sec: 209,926 | bf16_mfu: 32.05 | epoch: 1 pq: 13 rg: 64 | total time: 49.65m | eta: 12.6m |
| Step 01200 | Validation bpb: 0.730429 |
| Step 01200 | ICL early: 2.6633 late: 2.8423 score: +0.1790 | Induction loss: 9.2954 acc: 0.0939 (random-half: 14.1187) |
| step 01200/01500 (80.00%) | loss: 2.440996 | lrm: 1.00 | dt: 2493.94ms | tok/sec: 210,224 | bf16_mfu: 32.10 | epoch: 1 pq: 13 rg: 64 | total time: 49.69m | eta: 12.5m |
| step 01201/01500 (80.07%) | loss: 2.432617 | lrm: 1.00 | dt: 2497.17ms | tok/sec: 209,952 | bf16_mfu: 32.05 | epoch: 1 pq: 13 rg: 64 | total time: 49.73m | eta: 12.5m |
| step 01202/01500 (80.13%) | loss: 2.445783 | lrm: 1.00 | dt: 2498.38ms | tok/sec: 209,851 | bf16_mfu: 32.04 | epoch: 1 pq: 13 rg: 64 | total time: 49.77m | eta: 12.4m |
| step 01203/01500 (80.20%) | loss: 2.470045 | lrm: 1.00 | dt: 2500.16ms | tok/sec: 209,702 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 64 | total time: 49.82m | eta: 12.4m |
| step 01204/01500 (80.27%) | loss: 2.446939 | lrm: 1.00 | dt: 2498.92ms | tok/sec: 209,806 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 64 | total time: 49.86m | eta: 12.4m |
| step 01205/01500 (80.33%) | loss: 2.424146 | lrm: 1.00 | dt: 2500.20ms | tok/sec: 209,698 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 64 | total time: 49.90m | eta: 12.3m |
| step 01206/01500 (80.40%) | loss: 2.387269 | lrm: 1.00 | dt: 2503.97ms | tok/sec: 209,382 | bf16_mfu: 31.97 | epoch: 1 pq: 13 rg: 72 | total time: 49.94m | eta: 12.3m |
| step 01207/01500 (80.47%) | loss: 2.356603 | lrm: 1.00 | dt: 2503.11ms | tok/sec: 209,454 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 72 | total time: 49.98m | eta: 12.2m |
| step 01208/01500 (80.53%) | loss: 2.381411 | lrm: 1.00 | dt: 2502.20ms | tok/sec: 209,531 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 72 | total time: 50.02m | eta: 12.2m |
| step 01209/01500 (80.60%) | loss: 2.398901 | lrm: 1.00 | dt: 2498.46ms | tok/sec: 209,844 | bf16_mfu: 32.04 | epoch: 1 pq: 13 rg: 72 | total time: 50.07m | eta: 12.2m |
| step 01210/01500 (80.67%) | loss: 2.424929 | lrm: 1.00 | dt: 2502.33ms | tok/sec: 209,519 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 72 | total time: 50.11m | eta: 12.1m |
| step 01211/01500 (80.73%) | loss: 2.432458 | lrm: 1.00 | dt: 2502.75ms | tok/sec: 209,484 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 72 | total time: 50.15m | eta: 12.1m |
| step 01212/01500 (80.80%) | loss: 2.416121 | lrm: 1.00 | dt: 2500.12ms | tok/sec: 209,704 | bf16_mfu: 32.02 | epoch: 1 pq: 13 rg: 72 | total time: 50.19m | eta: 12.0m |
| step 01213/01500 (80.87%) | loss: 2.403905 | lrm: 1.00 | dt: 2502.53ms | tok/sec: 209,503 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 72 | total time: 50.23m | eta: 12.0m |
| step 01214/01500 (80.93%) | loss: 2.445069 | lrm: 1.00 | dt: 2499.27ms | tok/sec: 209,776 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 80 | total time: 50.27m | eta: 11.9m |
| step 01215/01500 (81.00%) | loss: 2.472159 | lrm: 1.00 | dt: 2499.01ms | tok/sec: 209,798 | bf16_mfu: 32.03 | epoch: 1 pq: 13 rg: 80 | total time: 50.32m | eta: 11.9m |
| step 01216/01500 (81.07%) | loss: 2.488684 | lrm: 1.00 | dt: 2502.64ms | tok/sec: 209,493 | bf16_mfu: 31.98 | epoch: 1 pq: 13 rg: 80 | total time: 50.36m | eta: 11.9m |
| step 01217/01500 (81.13%) | loss: 2.488064 | lrm: 1.00 | dt: 2500.84ms | tok/sec: 209,644 | bf16_mfu: 32.01 | epoch: 1 pq: 13 rg: 80 | total time: 50.40m | eta: 11.8m |
| step 01218/01500 (81.20%) | loss: 2.464704 | lrm: 1.00 | dt: 2502.59ms | tok/sec: 209,497 | bf16_mfu: 31.99 | epoch: 1 pq: 13 rg: 80 | total time: 50.44m | eta: 11.8m |
| step 01219/01500 (81.27%) | loss: 2.457899 | lrm: 1.00 | dt: 2498.21ms | tok/sec: 209,865 | bf16_mfu: 32.04 | epoch: 1 pq: 13 rg: 80 | total time: 50.48m | eta: 11.7m |
| step 01220/01500 (81.33%) | loss: 2.463797 | lrm: 1.00 | dt: 2501.32ms | tok/sec: 209,604 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 80 | total time: 50.52m | eta: 11.7m |
| step 01221/01500 (81.40%) | loss: 2.421569 | lrm: 1.00 | dt: 2501.07ms | tok/sec: 209,625 | bf16_mfu: 32.00 | epoch: 1 pq: 13 rg: 80 | total time: 50.57m | eta: 11.6m |
| step 01222/01500 (81.47%) | loss: 2.387758 | lrm: 1.00 | dt: 2501.59ms | tok/sec: 209,582 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 0 | total time: 50.61m | eta: 11.6m |
| step 01223/01500 (81.53%) | loss: 2.403879 | lrm: 1.00 | dt: 2502.32ms | tok/sec: 209,521 | bf16_mfu: 31.99 | epoch: 1 pq: 14 rg: 0 | total time: 50.65m | eta: 11.6m |
| step 01224/01500 (81.60%) | loss: 2.443035 | lrm: 1.00 | dt: 2496.38ms | tok/sec: 210,019 | bf16_mfu: 32.06 | epoch: 1 pq: 14 rg: 0 | total time: 50.69m | eta: 11.5m |
| step 01225/01500 (81.67%) | loss: 2.418260 | lrm: 1.00 | dt: 2499.23ms | tok/sec: 209,779 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 0 | total time: 50.73m | eta: 11.5m |
| step 01226/01500 (81.73%) | loss: 2.448829 | lrm: 1.00 | dt: 2499.69ms | tok/sec: 209,741 | bf16_mfu: 32.02 | epoch: 1 pq: 14 rg: 0 | total time: 50.77m | eta: 11.4m |
| step 01227/01500 (81.80%) | loss: 2.460668 | lrm: 1.00 | dt: 2503.39ms | tok/sec: 209,431 | bf16_mfu: 31.97 | epoch: 1 pq: 14 rg: 0 | total time: 50.82m | eta: 11.4m |
| step 01228/01500 (81.87%) | loss: 2.476715 | lrm: 1.00 | dt: 2501.51ms | tok/sec: 209,588 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 0 | total time: 50.86m | eta: 11.4m |
| step 01229/01500 (81.93%) | loss: 2.423908 | lrm: 1.00 | dt: 2500.45ms | tok/sec: 209,677 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 0 | total time: 50.90m | eta: 11.3m |
| step 01230/01500 (82.00%) | loss: 2.413463 | lrm: 1.00 | dt: 2501.00ms | tok/sec: 209,631 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 8 | total time: 50.94m | eta: 11.3m |
| step 01231/01500 (82.07%) | loss: 2.394710 | lrm: 1.00 | dt: 2504.12ms | tok/sec: 209,369 | bf16_mfu: 31.97 | epoch: 1 pq: 14 rg: 8 | total time: 50.98m | eta: 11.2m |
| step 01232/01500 (82.13%) | loss: 2.354343 | lrm: 1.00 | dt: 2502.03ms | tok/sec: 209,544 | bf16_mfu: 31.99 | epoch: 1 pq: 14 rg: 8 | total time: 51.03m | eta: 11.2m |
| step 01233/01500 (82.20%) | loss: 2.371003 | lrm: 1.00 | dt: 2497.85ms | tok/sec: 209,895 | bf16_mfu: 32.05 | epoch: 1 pq: 14 rg: 8 | total time: 51.07m | eta: 11.1m |
| step 01234/01500 (82.27%) | loss: 2.352386 | lrm: 1.00 | dt: 2499.29ms | tok/sec: 209,774 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 8 | total time: 51.11m | eta: 11.1m |
| step 01235/01500 (82.33%) | loss: 2.389660 | lrm: 1.00 | dt: 2503.97ms | tok/sec: 209,382 | bf16_mfu: 31.97 | epoch: 1 pq: 14 rg: 8 | total time: 51.15m | eta: 11.1m |
| step 01236/01500 (82.40%) | loss: 2.360726 | lrm: 1.00 | dt: 2502.77ms | tok/sec: 209,483 | bf16_mfu: 31.98 | epoch: 1 pq: 14 rg: 8 | total time: 51.19m | eta: 11.0m |
| step 01237/01500 (82.47%) | loss: 2.362755 | lrm: 1.00 | dt: 2497.18ms | tok/sec: 209,951 | bf16_mfu: 32.05 | epoch: 1 pq: 14 rg: 8 | total time: 51.23m | eta: 11.0m |
| step 01238/01500 (82.53%) | loss: 2.368698 | lrm: 1.00 | dt: 2500.44ms | tok/sec: 209,678 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 16 | total time: 51.28m | eta: 10.9m |
| step 01239/01500 (82.60%) | loss: 2.393627 | lrm: 1.00 | dt: 2496.97ms | tok/sec: 209,969 | bf16_mfu: 32.06 | epoch: 1 pq: 14 rg: 16 | total time: 51.32m | eta: 10.9m |
| step 01240/01500 (82.67%) | loss: 2.452025 | lrm: 1.00 | dt: 2499.03ms | tok/sec: 209,796 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 16 | total time: 51.36m | eta: 10.9m |
| step 01241/01500 (82.73%) | loss: 2.425301 | lrm: 1.00 | dt: 2503.15ms | tok/sec: 209,451 | bf16_mfu: 31.98 | epoch: 1 pq: 14 rg: 16 | total time: 51.40m | eta: 10.8m |
| step 01242/01500 (82.80%) | loss: 2.406693 | lrm: 1.00 | dt: 2504.33ms | tok/sec: 209,352 | bf16_mfu: 31.96 | epoch: 1 pq: 14 rg: 16 | total time: 51.44m | eta: 10.8m |
| step 01243/01500 (82.87%) | loss: 2.389649 | lrm: 1.00 | dt: 2503.29ms | tok/sec: 209,439 | bf16_mfu: 31.98 | epoch: 1 pq: 14 rg: 16 | total time: 51.48m | eta: 10.7m |
| step 01244/01500 (82.93%) | loss: 2.383802 | lrm: 1.00 | dt: 2501.70ms | tok/sec: 209,572 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 16 | total time: 51.53m | eta: 10.7m |
| step 01245/01500 (83.00%) | loss: 2.383217 | lrm: 1.00 | dt: 2501.24ms | tok/sec: 209,611 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 16 | total time: 51.57m | eta: 10.6m |
| step 01246/01500 (83.07%) | loss: 2.355744 | lrm: 1.00 | dt: 2496.78ms | tok/sec: 209,985 | bf16_mfu: 32.06 | epoch: 1 pq: 14 rg: 24 | total time: 51.61m | eta: 10.6m |
| step 01247/01500 (83.13%) | loss: 2.357943 | lrm: 1.00 | dt: 2506.83ms | tok/sec: 209,144 | bf16_mfu: 31.93 | epoch: 1 pq: 14 rg: 24 | total time: 51.65m | eta: 10.6m |
| step 01248/01500 (83.20%) | loss: 2.360747 | lrm: 1.00 | dt: 2504.34ms | tok/sec: 209,351 | bf16_mfu: 31.96 | epoch: 1 pq: 14 rg: 24 | total time: 51.69m | eta: 10.5m |
| step 01249/01500 (83.27%) | loss: 2.363689 | lrm: 1.00 | dt: 2499.03ms | tok/sec: 209,796 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 24 | total time: 51.73m | eta: 10.5m |
| <|bos|>The capital of France is Paris. The city is known for its historical landmarks, such as the Eiffel Tower |
| <|bos|>The chemical symbol of gold is Au. It is a soft, yellow, dense, ductile, malleable, and |
| <|bos|>If yesterday was Friday, then tomorrow will be Saturday. If today was Saturday, then tomorrow will be Sunday. If today was |
| <|bos|>The opposite of hot is cold. Hot is a state of energy, a state of motion, a state |
| <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, |
| <|bos|>My favorite color is blue. I like to wear blue jeans and a blue shirt. I like to |
| <|bos|>If 5*x + 3 = 13, then x is 3. If 5*x + 3 = 13, then |
| 2026-08-10 22:49:26,478 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank3.pt |
| 2026-08-10 22:49:47,566 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank1.pt |
| 2026-08-10 22:50:08,904 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank5.pt |
| 2026-08-10 22:50:20,188 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank7.pt |
| 2026-08-10 22:50:24,906 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank2.pt |
| 2026-08-10 22:50:24,907 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank4.pt |
| 2026-08-10 22:50:24,982 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank6.pt |
| 2026-08-10 22:51:45,997 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved model parameters to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/model_001250.pt |
| 2026-08-10 22:51:46,001 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved metadata to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/meta_001250.json |
| 2026-08-10 22:51:54,408 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001250_rank0.pt |
| step 01250/01500 (83.33%) | loss: 2.361200 | lrm: 1.00 | dt: 2476.27ms | tok/sec: 211,724 | bf16_mfu: 32.32 | epoch: 1 pq: 14 rg: 24 | total time: 51.78m | eta: 10.4m |
| step 01251/01500 (83.40%) | loss: 2.330253 | lrm: 1.00 | dt: 2489.12ms | tok/sec: 210,631 | bf16_mfu: 32.16 | epoch: 1 pq: 14 rg: 24 | total time: 51.82m | eta: 10.4m |
| step 01252/01500 (83.47%) | loss: 2.327421 | lrm: 1.00 | dt: 2492.38ms | tok/sec: 210,355 | bf16_mfu: 32.12 | epoch: 1 pq: 14 rg: 24 | total time: 51.86m | eta: 10.4m |
| step 01253/01500 (83.53%) | loss: 2.342858 | lrm: 1.00 | dt: 2493.54ms | tok/sec: 210,258 | bf16_mfu: 32.10 | epoch: 1 pq: 14 rg: 24 | total time: 51.90m | eta: 10.3m |
| step 01254/01500 (83.60%) | loss: 2.348323 | lrm: 1.00 | dt: 2496.53ms | tok/sec: 210,006 | bf16_mfu: 32.06 | epoch: 1 pq: 14 rg: 32 | total time: 51.94m | eta: 10.3m |
| step 01255/01500 (83.67%) | loss: 2.349545 | lrm: 1.00 | dt: 2496.69ms | tok/sec: 209,993 | bf16_mfu: 32.06 | epoch: 1 pq: 14 rg: 32 | total time: 51.98m | eta: 10.2m |
| step 01256/01500 (83.73%) | loss: 2.331776 | lrm: 1.00 | dt: 2498.68ms | tok/sec: 209,826 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 32 | total time: 52.02m | eta: 10.2m |
| step 01257/01500 (83.80%) | loss: 2.300347 | lrm: 1.00 | dt: 2501.04ms | tok/sec: 209,628 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 32 | total time: 52.07m | eta: 10.1m |
| step 01258/01500 (83.87%) | loss: 2.317687 | lrm: 1.00 | dt: 2502.44ms | tok/sec: 209,510 | bf16_mfu: 31.99 | epoch: 1 pq: 14 rg: 32 | total time: 52.11m | eta: 10.1m |
| step 01259/01500 (83.93%) | loss: 2.330505 | lrm: 1.00 | dt: 2510.40ms | tok/sec: 208,846 | bf16_mfu: 31.89 | epoch: 1 pq: 14 rg: 32 | total time: 52.15m | eta: 10.1m |
| step 01260/01500 (84.00%) | loss: 2.323666 | lrm: 1.00 | dt: 2506.75ms | tok/sec: 209,150 | bf16_mfu: 31.93 | epoch: 1 pq: 14 rg: 32 | total time: 52.19m | eta: 10.0m |
| step 01261/01500 (84.07%) | loss: 2.324743 | lrm: 1.00 | dt: 2510.75ms | tok/sec: 208,817 | bf16_mfu: 31.88 | epoch: 1 pq: 14 rg: 32 | total time: 52.23m | eta: 10.0m |
| step 01262/01500 (84.13%) | loss: 2.364188 | lrm: 1.00 | dt: 2508.82ms | tok/sec: 208,977 | bf16_mfu: 31.91 | epoch: 1 pq: 14 rg: 40 | total time: 52.28m | eta: 9.9m |
| step 01263/01500 (84.20%) | loss: 2.375115 | lrm: 1.00 | dt: 2503.92ms | tok/sec: 209,386 | bf16_mfu: 31.97 | epoch: 1 pq: 14 rg: 40 | total time: 52.32m | eta: 9.9m |
| step 01264/01500 (84.27%) | loss: 2.383553 | lrm: 1.00 | dt: 2499.87ms | tok/sec: 209,726 | bf16_mfu: 32.02 | epoch: 1 pq: 14 rg: 40 | total time: 52.36m | eta: 9.9m |
| step 01265/01500 (84.33%) | loss: 2.397808 | lrm: 1.00 | dt: 2505.98ms | tok/sec: 209,214 | bf16_mfu: 31.94 | epoch: 1 pq: 14 rg: 40 | total time: 52.40m | eta: 9.8m |
| step 01266/01500 (84.40%) | loss: 2.385561 | lrm: 1.00 | dt: 2501.65ms | tok/sec: 209,577 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 40 | total time: 52.44m | eta: 9.8m |
| step 01267/01500 (84.47%) | loss: 2.382359 | lrm: 1.00 | dt: 2498.54ms | tok/sec: 209,837 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 40 | total time: 52.48m | eta: 9.7m |
| step 01268/01500 (84.53%) | loss: 2.390636 | lrm: 1.00 | dt: 2503.71ms | tok/sec: 209,404 | bf16_mfu: 31.97 | epoch: 1 pq: 14 rg: 40 | total time: 52.53m | eta: 9.7m |
| step 01269/01500 (84.60%) | loss: 2.394031 | lrm: 1.00 | dt: 2498.86ms | tok/sec: 209,811 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 40 | total time: 52.57m | eta: 9.6m |
| step 01270/01500 (84.67%) | loss: 2.343721 | lrm: 1.00 | dt: 3570.65ms | tok/sec: 146,832 | bf16_mfu: 22.42 | epoch: 1 pq: 14 rg: 48 | total time: 52.63m | eta: 9.6m |
| step 01271/01500 (84.73%) | loss: 2.345929 | lrm: 1.00 | dt: 2494.67ms | tok/sec: 210,163 | bf16_mfu: 32.09 | epoch: 1 pq: 14 rg: 48 | total time: 52.67m | eta: 9.6m |
| step 01272/01500 (84.80%) | loss: 2.334842 | lrm: 1.00 | dt: 2494.59ms | tok/sec: 210,170 | bf16_mfu: 32.09 | epoch: 1 pq: 14 rg: 48 | total time: 52.71m | eta: 9.5m |
| step 01273/01500 (84.87%) | loss: 2.354259 | lrm: 1.00 | dt: 2505.26ms | tok/sec: 209,274 | bf16_mfu: 31.95 | epoch: 1 pq: 14 rg: 48 | total time: 52.75m | eta: 9.5m |
| step 01274/01500 (84.93%) | loss: 2.321031 | lrm: 1.00 | dt: 2498.69ms | tok/sec: 209,825 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 48 | total time: 52.79m | eta: 9.4m |
| step 01275/01500 (85.00%) | loss: 2.290331 | lrm: 1.00 | dt: 2501.70ms | tok/sec: 209,572 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 48 | total time: 52.83m | eta: 9.4m |
| step 01276/01500 (85.07%) | loss: 2.316285 | lrm: 1.00 | dt: 2497.73ms | tok/sec: 209,906 | bf16_mfu: 32.05 | epoch: 1 pq: 14 rg: 48 | total time: 52.88m | eta: 9.4m |
| step 01277/01500 (85.13%) | loss: 2.356199 | lrm: 1.00 | dt: 2503.38ms | tok/sec: 209,432 | bf16_mfu: 31.97 | epoch: 1 pq: 14 rg: 48 | total time: 52.92m | eta: 9.3m |
| step 01278/01500 (85.20%) | loss: 2.315412 | lrm: 1.00 | dt: 2502.47ms | tok/sec: 209,507 | bf16_mfu: 31.99 | epoch: 1 pq: 14 rg: 56 | total time: 52.96m | eta: 9.3m |
| step 01279/01500 (85.27%) | loss: 2.356142 | lrm: 1.00 | dt: 2500.31ms | tok/sec: 209,689 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 56 | total time: 53.00m | eta: 9.2m |
| step 01280/01500 (85.33%) | loss: 2.386494 | lrm: 1.00 | dt: 2498.09ms | tok/sec: 209,875 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 56 | total time: 53.04m | eta: 9.2m |
| step 01281/01500 (85.40%) | loss: 2.414202 | lrm: 1.00 | dt: 2498.91ms | tok/sec: 209,806 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 56 | total time: 53.08m | eta: 9.1m |
| step 01282/01500 (85.47%) | loss: 2.393893 | lrm: 1.00 | dt: 2500.78ms | tok/sec: 209,649 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 56 | total time: 53.13m | eta: 9.1m |
| step 01283/01500 (85.53%) | loss: 2.400295 | lrm: 1.00 | dt: 2500.76ms | tok/sec: 209,651 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 56 | total time: 53.17m | eta: 9.1m |
| step 01284/01500 (85.60%) | loss: 2.415913 | lrm: 1.00 | dt: 2498.74ms | tok/sec: 209,820 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 56 | total time: 53.21m | eta: 9.0m |
| step 01285/01500 (85.67%) | loss: 2.393784 | lrm: 1.00 | dt: 2500.66ms | tok/sec: 209,660 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 56 | total time: 53.25m | eta: 9.0m |
| step 01286/01500 (85.73%) | loss: 2.422404 | lrm: 1.00 | dt: 2507.51ms | tok/sec: 209,087 | bf16_mfu: 31.92 | epoch: 1 pq: 14 rg: 64 | total time: 53.29m | eta: 8.9m |
| step 01287/01500 (85.80%) | loss: 2.362867 | lrm: 1.00 | dt: 2499.79ms | tok/sec: 209,732 | bf16_mfu: 32.02 | epoch: 1 pq: 14 rg: 64 | total time: 53.33m | eta: 8.9m |
| step 01288/01500 (85.87%) | loss: 2.369990 | lrm: 1.00 | dt: 2502.51ms | tok/sec: 209,504 | bf16_mfu: 31.99 | epoch: 1 pq: 14 rg: 64 | total time: 53.38m | eta: 8.9m |
| step 01289/01500 (85.93%) | loss: 2.387106 | lrm: 1.00 | dt: 2497.97ms | tok/sec: 209,885 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 64 | total time: 53.42m | eta: 8.8m |
| step 01290/01500 (86.00%) | loss: 2.441367 | lrm: 1.00 | dt: 2498.72ms | tok/sec: 209,822 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 64 | total time: 53.46m | eta: 8.8m |
| step 01291/01500 (86.07%) | loss: 2.421063 | lrm: 1.00 | dt: 2498.61ms | tok/sec: 209,831 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 64 | total time: 53.50m | eta: 8.7m |
| step 01292/01500 (86.13%) | loss: 2.404441 | lrm: 1.00 | dt: 2497.75ms | tok/sec: 209,904 | bf16_mfu: 32.05 | epoch: 1 pq: 14 rg: 64 | total time: 53.54m | eta: 8.7m |
| step 01293/01500 (86.20%) | loss: 2.396020 | lrm: 1.00 | dt: 2498.47ms | tok/sec: 209,843 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 64 | total time: 53.58m | eta: 8.6m |
| step 01294/01500 (86.27%) | loss: 2.396105 | lrm: 1.00 | dt: 2498.06ms | tok/sec: 209,877 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 64 | total time: 53.63m | eta: 8.6m |
| step 01295/01500 (86.33%) | loss: 2.414203 | lrm: 1.00 | dt: 2499.37ms | tok/sec: 209,768 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 72 | total time: 53.67m | eta: 8.6m |
| step 01296/01500 (86.40%) | loss: 2.407398 | lrm: 1.00 | dt: 2501.14ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 72 | total time: 53.71m | eta: 8.5m |
| step 01297/01500 (86.47%) | loss: 2.449826 | lrm: 1.00 | dt: 2504.61ms | tok/sec: 209,329 | bf16_mfu: 31.96 | epoch: 1 pq: 14 rg: 72 | total time: 53.75m | eta: 8.5m |
| step 01298/01500 (86.53%) | loss: 2.424689 | lrm: 1.00 | dt: 2501.53ms | tok/sec: 209,586 | bf16_mfu: 32.00 | epoch: 1 pq: 14 rg: 72 | total time: 53.79m | eta: 8.4m |
| step 01299/01500 (86.60%) | loss: 2.431997 | lrm: 1.00 | dt: 2500.21ms | tok/sec: 209,697 | bf16_mfu: 32.02 | epoch: 1 pq: 14 rg: 72 | total time: 53.83m | eta: 8.4m |
| Step 01300 | Validation bpb: 0.728475 |
| Step 01300 | ICL early: 2.6509 late: 2.8619 score: +0.2110 | Induction loss: 9.3552 acc: 0.0929 (random-half: 14.1283) |
| step 01300/01500 (86.67%) | loss: 2.454086 | lrm: 1.00 | dt: 2497.64ms | tok/sec: 209,913 | bf16_mfu: 32.05 | epoch: 1 pq: 14 rg: 72 | total time: 53.88m | eta: 8.4m |
| step 01301/01500 (86.73%) | loss: 2.479593 | lrm: 1.00 | dt: 2500.61ms | tok/sec: 209,663 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 72 | total time: 53.92m | eta: 8.3m |
| step 01302/01500 (86.80%) | loss: 2.504229 | lrm: 1.00 | dt: 2498.93ms | tok/sec: 209,805 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 72 | total time: 53.96m | eta: 8.3m |
| step 01303/01500 (86.87%) | loss: 2.524538 | lrm: 1.00 | dt: 2500.66ms | tok/sec: 209,660 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 80 | total time: 54.00m | eta: 8.2m |
| step 01304/01500 (86.93%) | loss: 2.528756 | lrm: 1.00 | dt: 2498.22ms | tok/sec: 209,864 | bf16_mfu: 32.04 | epoch: 1 pq: 14 rg: 80 | total time: 54.04m | eta: 8.2m |
| step 01305/01500 (87.00%) | loss: 2.495952 | lrm: 1.00 | dt: 2503.92ms | tok/sec: 209,386 | bf16_mfu: 31.97 | epoch: 1 pq: 14 rg: 80 | total time: 54.08m | eta: 8.1m |
| step 01306/01500 (87.07%) | loss: 2.466344 | lrm: 1.00 | dt: 2505.80ms | tok/sec: 209,230 | bf16_mfu: 31.94 | epoch: 1 pq: 14 rg: 80 | total time: 54.13m | eta: 8.1m |
| step 01307/01500 (87.13%) | loss: 2.455489 | lrm: 1.00 | dt: 2499.37ms | tok/sec: 209,767 | bf16_mfu: 32.03 | epoch: 1 pq: 14 rg: 80 | total time: 54.17m | eta: 8.1m |
| step 01308/01500 (87.20%) | loss: 2.468556 | lrm: 1.00 | dt: 2506.67ms | tok/sec: 209,157 | bf16_mfu: 31.93 | epoch: 1 pq: 14 rg: 80 | total time: 54.21m | eta: 8.0m |
| step 01309/01500 (87.27%) | loss: 2.513764 | lrm: 1.00 | dt: 2500.46ms | tok/sec: 209,676 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 80 | total time: 54.25m | eta: 8.0m |
| step 01310/01500 (87.33%) | loss: 2.485792 | lrm: 1.00 | dt: 2500.41ms | tok/sec: 209,680 | bf16_mfu: 32.01 | epoch: 1 pq: 14 rg: 80 | total time: 54.29m | eta: 7.9m |
| step 01311/01500 (87.40%) | loss: 2.482397 | lrm: 1.00 | dt: 2499.60ms | tok/sec: 209,748 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 0 | total time: 54.34m | eta: 7.9m |
| step 01312/01500 (87.47%) | loss: 2.471948 | lrm: 1.00 | dt: 2499.55ms | tok/sec: 209,752 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 0 | total time: 54.38m | eta: 7.9m |
| step 01313/01500 (87.53%) | loss: 2.466685 | lrm: 1.00 | dt: 2500.53ms | tok/sec: 209,670 | bf16_mfu: 32.01 | epoch: 1 pq: 15 rg: 0 | total time: 54.42m | eta: 7.8m |
| step 01314/01500 (87.60%) | loss: 2.418181 | lrm: 1.00 | dt: 2506.89ms | tok/sec: 209,138 | bf16_mfu: 31.93 | epoch: 1 pq: 15 rg: 0 | total time: 54.46m | eta: 7.8m |
| step 01315/01500 (87.67%) | loss: 2.433514 | lrm: 1.00 | dt: 2501.43ms | tok/sec: 209,595 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 0 | total time: 54.50m | eta: 7.7m |
| step 01316/01500 (87.73%) | loss: 2.432581 | lrm: 1.00 | dt: 2498.72ms | tok/sec: 209,822 | bf16_mfu: 32.03 | epoch: 1 pq: 15 rg: 0 | total time: 54.54m | eta: 7.7m |
| step 01317/01500 (87.80%) | loss: 2.454954 | lrm: 1.00 | dt: 2495.22ms | tok/sec: 210,117 | bf16_mfu: 32.08 | epoch: 1 pq: 15 rg: 0 | total time: 54.59m | eta: 7.6m |
| step 01318/01500 (87.87%) | loss: 2.456899 | lrm: 1.00 | dt: 2501.90ms | tok/sec: 209,556 | bf16_mfu: 31.99 | epoch: 1 pq: 15 rg: 0 | total time: 54.63m | eta: 7.6m |
| step 01319/01500 (87.93%) | loss: 2.471288 | lrm: 1.00 | dt: 2507.48ms | tok/sec: 209,089 | bf16_mfu: 31.92 | epoch: 1 pq: 15 rg: 8 | total time: 54.67m | eta: 7.6m |
| step 01320/01500 (88.00%) | loss: 2.497319 | lrm: 1.00 | dt: 2499.71ms | tok/sec: 209,739 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 8 | total time: 54.71m | eta: 7.5m |
| step 01321/01500 (88.07%) | loss: 2.521943 | lrm: 1.00 | dt: 2497.67ms | tok/sec: 209,910 | bf16_mfu: 32.05 | epoch: 1 pq: 15 rg: 8 | total time: 54.75m | eta: 7.5m |
| step 01322/01500 (88.13%) | loss: 2.468437 | lrm: 1.00 | dt: 2501.28ms | tok/sec: 209,607 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 8 | total time: 54.79m | eta: 7.4m |
| step 01323/01500 (88.20%) | loss: 2.497319 | lrm: 1.00 | dt: 2498.47ms | tok/sec: 209,843 | bf16_mfu: 32.04 | epoch: 1 pq: 15 rg: 8 | total time: 54.84m | eta: 7.4m |
| step 01324/01500 (88.27%) | loss: 2.500568 | lrm: 1.00 | dt: 2505.22ms | tok/sec: 209,278 | bf16_mfu: 31.95 | epoch: 1 pq: 15 rg: 8 | total time: 54.88m | eta: 7.4m |
| step 01325/01500 (88.33%) | loss: 2.524526 | lrm: 1.00 | dt: 2498.36ms | tok/sec: 209,853 | bf16_mfu: 32.04 | epoch: 1 pq: 15 rg: 8 | total time: 54.92m | eta: 7.3m |
| step 01326/01500 (88.40%) | loss: 2.541899 | lrm: 1.00 | dt: 2497.00ms | tok/sec: 209,966 | bf16_mfu: 32.06 | epoch: 1 pq: 15 rg: 8 | total time: 54.96m | eta: 7.3m |
| step 01327/01500 (88.47%) | loss: 2.579647 | lrm: 1.00 | dt: 2506.93ms | tok/sec: 209,135 | bf16_mfu: 31.93 | epoch: 1 pq: 15 rg: 16 | total time: 55.00m | eta: 7.2m |
| step 01328/01500 (88.53%) | loss: 2.554086 | lrm: 1.00 | dt: 2502.26ms | tok/sec: 209,525 | bf16_mfu: 31.99 | epoch: 1 pq: 15 rg: 16 | total time: 55.04m | eta: 7.2m |
| step 01329/01500 (88.60%) | loss: 2.548375 | lrm: 1.00 | dt: 2506.95ms | tok/sec: 209,133 | bf16_mfu: 31.93 | epoch: 1 pq: 15 rg: 16 | total time: 55.09m | eta: 7.1m |
| step 01330/01500 (88.67%) | loss: 2.498161 | lrm: 1.00 | dt: 2503.39ms | tok/sec: 209,431 | bf16_mfu: 31.97 | epoch: 1 pq: 15 rg: 16 | total time: 55.13m | eta: 7.1m |
| step 01331/01500 (88.73%) | loss: 2.498972 | lrm: 1.00 | dt: 2503.71ms | tok/sec: 209,404 | bf16_mfu: 31.97 | epoch: 1 pq: 15 rg: 16 | total time: 55.17m | eta: 7.1m |
| step 01332/01500 (88.80%) | loss: 2.508089 | lrm: 1.00 | dt: 2501.13ms | tok/sec: 209,620 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 16 | total time: 55.21m | eta: 7.0m |
| step 01333/01500 (88.87%) | loss: 2.511404 | lrm: 1.00 | dt: 2499.18ms | tok/sec: 209,783 | bf16_mfu: 32.03 | epoch: 1 pq: 15 rg: 16 | total time: 55.25m | eta: 7.0m |
| step 01334/01500 (88.93%) | loss: 2.510940 | lrm: 1.00 | dt: 2501.09ms | tok/sec: 209,624 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 16 | total time: 55.29m | eta: 6.9m |
| step 01335/01500 (89.00%) | loss: 2.536113 | lrm: 1.00 | dt: 2495.40ms | tok/sec: 210,101 | bf16_mfu: 32.08 | epoch: 1 pq: 15 rg: 24 | total time: 55.34m | eta: 6.9m |
| step 01336/01500 (89.07%) | loss: 2.531636 | lrm: 1.00 | dt: 2505.76ms | tok/sec: 209,233 | bf16_mfu: 31.94 | epoch: 1 pq: 15 rg: 24 | total time: 55.38m | eta: 6.8m |
| step 01337/01500 (89.13%) | loss: 2.538024 | lrm: 1.00 | dt: 2499.30ms | tok/sec: 209,773 | bf16_mfu: 32.03 | epoch: 1 pq: 15 rg: 24 | total time: 55.42m | eta: 6.8m |
| step 01338/01500 (89.20%) | loss: 2.515331 | lrm: 1.00 | dt: 2501.75ms | tok/sec: 209,568 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 24 | total time: 55.46m | eta: 6.8m |
| step 01339/01500 (89.27%) | loss: 2.507357 | lrm: 1.00 | dt: 2503.18ms | tok/sec: 209,448 | bf16_mfu: 31.98 | epoch: 1 pq: 15 rg: 24 | total time: 55.50m | eta: 6.7m |
| step 01340/01500 (89.33%) | loss: 2.503695 | lrm: 1.00 | dt: 2495.13ms | tok/sec: 210,124 | bf16_mfu: 32.08 | epoch: 1 pq: 15 rg: 24 | total time: 55.54m | eta: 6.7m |
| step 01341/01500 (89.40%) | loss: 2.507368 | lrm: 1.00 | dt: 2500.79ms | tok/sec: 209,648 | bf16_mfu: 32.01 | epoch: 1 pq: 15 rg: 24 | total time: 55.59m | eta: 6.6m |
| step 01342/01500 (89.47%) | loss: 2.531401 | lrm: 1.00 | dt: 2502.42ms | tok/sec: 209,512 | bf16_mfu: 31.99 | epoch: 1 pq: 15 rg: 24 | total time: 55.63m | eta: 6.6m |
| step 01343/01500 (89.53%) | loss: 2.550755 | lrm: 1.00 | dt: 2501.14ms | tok/sec: 209,619 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 32 | total time: 55.67m | eta: 6.6m |
| step 01344/01500 (89.60%) | loss: 2.541343 | lrm: 1.00 | dt: 2499.26ms | tok/sec: 209,777 | bf16_mfu: 32.03 | epoch: 1 pq: 15 rg: 32 | total time: 55.71m | eta: 6.5m |
| step 01345/01500 (89.67%) | loss: 2.531321 | lrm: 1.00 | dt: 2498.66ms | tok/sec: 209,827 | bf16_mfu: 32.04 | epoch: 1 pq: 15 rg: 32 | total time: 55.75m | eta: 6.5m |
| step 01346/01500 (89.73%) | loss: 2.550525 | lrm: 1.00 | dt: 2505.34ms | tok/sec: 209,267 | bf16_mfu: 31.95 | epoch: 1 pq: 15 rg: 32 | total time: 55.79m | eta: 6.4m |
| step 01347/01500 (89.80%) | loss: 2.572877 | lrm: 1.00 | dt: 2501.74ms | tok/sec: 209,569 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 32 | total time: 55.84m | eta: 6.4m |
| step 01348/01500 (89.87%) | loss: 2.542516 | lrm: 1.00 | dt: 2501.66ms | tok/sec: 209,575 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 32 | total time: 55.88m | eta: 6.3m |
| step 01349/01500 (89.93%) | loss: 2.543491 | lrm: 1.00 | dt: 2500.14ms | tok/sec: 209,703 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 32 | total time: 55.92m | eta: 6.3m |
| step 01350/01500 (90.00%) | loss: 2.531649 | lrm: 1.00 | dt: 2498.62ms | tok/sec: 209,830 | bf16_mfu: 32.04 | epoch: 1 pq: 15 rg: 40 | total time: 55.96m | eta: 6.3m |
| step 01351/01500 (90.07%) | loss: 2.496322 | lrm: 1.00 | dt: 3533.98ms | tok/sec: 148,356 | bf16_mfu: 22.65 | epoch: 1 pq: 15 rg: 40 | total time: 56.02m | eta: 6.2m |
| step 01352/01500 (90.13%) | loss: 2.514389 | lrm: 1.00 | dt: 2497.03ms | tok/sec: 209,964 | bf16_mfu: 32.06 | epoch: 1 pq: 15 rg: 40 | total time: 56.06m | eta: 6.2m |
| step 01353/01500 (90.20%) | loss: 2.539929 | lrm: 1.00 | dt: 2500.11ms | tok/sec: 209,706 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 40 | total time: 56.10m | eta: 6.1m |
| step 01354/01500 (90.27%) | loss: 2.512404 | lrm: 1.00 | dt: 2501.89ms | tok/sec: 209,556 | bf16_mfu: 31.99 | epoch: 1 pq: 15 rg: 40 | total time: 56.14m | eta: 6.1m |
| step 01355/01500 (90.33%) | loss: 2.554807 | lrm: 1.00 | dt: 2498.80ms | tok/sec: 209,816 | bf16_mfu: 32.03 | epoch: 1 pq: 15 rg: 40 | total time: 56.19m | eta: 6.1m |
| step 01356/01500 (90.40%) | loss: 2.572568 | lrm: 1.00 | dt: 2498.51ms | tok/sec: 209,840 | bf16_mfu: 32.04 | epoch: 1 pq: 15 rg: 40 | total time: 56.23m | eta: 6.0m |
| step 01357/01500 (90.47%) | loss: 2.558059 | lrm: 1.00 | dt: 2499.78ms | tok/sec: 209,733 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 40 | total time: 56.27m | eta: 6.0m |
| step 01358/01500 (90.53%) | loss: 2.559718 | lrm: 1.00 | dt: 2502.75ms | tok/sec: 209,484 | bf16_mfu: 31.98 | epoch: 1 pq: 15 rg: 40 | total time: 56.31m | eta: 5.9m |
| step 01359/01500 (90.60%) | loss: 2.585213 | lrm: 1.00 | dt: 2504.48ms | tok/sec: 209,340 | bf16_mfu: 31.96 | epoch: 1 pq: 15 rg: 48 | total time: 56.35m | eta: 5.9m |
| step 01360/01500 (90.67%) | loss: 2.567475 | lrm: 1.00 | dt: 2505.23ms | tok/sec: 209,277 | bf16_mfu: 31.95 | epoch: 1 pq: 15 rg: 48 | total time: 56.39m | eta: 5.8m |
| step 01361/01500 (90.73%) | loss: 2.560613 | lrm: 1.00 | dt: 2497.82ms | tok/sec: 209,898 | bf16_mfu: 32.05 | epoch: 1 pq: 15 rg: 48 | total time: 56.44m | eta: 5.8m |
| step 01362/01500 (90.80%) | loss: 2.522151 | lrm: 1.00 | dt: 2497.24ms | tok/sec: 209,947 | bf16_mfu: 32.05 | epoch: 1 pq: 15 rg: 48 | total time: 56.48m | eta: 5.8m |
| step 01363/01500 (90.87%) | loss: 2.515407 | lrm: 1.00 | dt: 2499.59ms | tok/sec: 209,749 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 48 | total time: 56.52m | eta: 5.7m |
| step 01364/01500 (90.93%) | loss: 2.463668 | lrm: 1.00 | dt: 2499.60ms | tok/sec: 209,749 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 48 | total time: 56.56m | eta: 5.7m |
| step 01365/01500 (91.00%) | loss: 2.453724 | lrm: 1.00 | dt: 2502.13ms | tok/sec: 209,536 | bf16_mfu: 31.99 | epoch: 1 pq: 15 rg: 48 | total time: 56.60m | eta: 5.6m |
| step 01366/01500 (91.07%) | loss: 2.412913 | lrm: 1.00 | dt: 2499.58ms | tok/sec: 209,750 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 48 | total time: 56.64m | eta: 5.6m |
| step 01367/01500 (91.13%) | loss: 2.417648 | lrm: 1.00 | dt: 2499.88ms | tok/sec: 209,725 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 56 | total time: 56.69m | eta: 5.6m |
| step 01368/01500 (91.20%) | loss: 2.410528 | lrm: 1.00 | dt: 2497.42ms | tok/sec: 209,932 | bf16_mfu: 32.05 | epoch: 1 pq: 15 rg: 56 | total time: 56.73m | eta: 5.5m |
| step 01369/01500 (91.27%) | loss: 2.407004 | lrm: 1.00 | dt: 2498.31ms | tok/sec: 209,857 | bf16_mfu: 32.04 | epoch: 1 pq: 15 rg: 56 | total time: 56.77m | eta: 5.5m |
| step 01370/01500 (91.33%) | loss: 2.421171 | lrm: 1.00 | dt: 2497.22ms | tok/sec: 209,948 | bf16_mfu: 32.05 | epoch: 1 pq: 15 rg: 56 | total time: 56.81m | eta: 5.4m |
| step 01371/01500 (91.40%) | loss: 2.404136 | lrm: 1.00 | dt: 2501.38ms | tok/sec: 209,599 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 56 | total time: 56.85m | eta: 5.4m |
| step 01372/01500 (91.47%) | loss: 2.425430 | lrm: 1.00 | dt: 2498.14ms | tok/sec: 209,870 | bf16_mfu: 32.04 | epoch: 1 pq: 15 rg: 56 | total time: 56.89m | eta: 5.3m |
| step 01373/01500 (91.53%) | loss: 2.435518 | lrm: 1.00 | dt: 2500.87ms | tok/sec: 209,642 | bf16_mfu: 32.01 | epoch: 1 pq: 15 rg: 56 | total time: 56.94m | eta: 5.3m |
| step 01374/01500 (91.60%) | loss: 2.433409 | lrm: 1.00 | dt: 2499.43ms | tok/sec: 209,763 | bf16_mfu: 32.03 | epoch: 1 pq: 15 rg: 56 | total time: 56.98m | eta: 5.3m |
| step 01375/01500 (91.67%) | loss: 2.415074 | lrm: 1.00 | dt: 2505.46ms | tok/sec: 209,258 | bf16_mfu: 31.95 | epoch: 1 pq: 15 rg: 64 | total time: 57.02m | eta: 5.2m |
| step 01376/01500 (91.73%) | loss: 2.424119 | lrm: 1.00 | dt: 2499.90ms | tok/sec: 209,723 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 64 | total time: 57.06m | eta: 5.2m |
| step 01377/01500 (91.80%) | loss: 2.424437 | lrm: 1.00 | dt: 2502.73ms | tok/sec: 209,486 | bf16_mfu: 31.98 | epoch: 1 pq: 15 rg: 64 | total time: 57.10m | eta: 5.1m |
| step 01378/01500 (91.87%) | loss: 2.426031 | lrm: 1.00 | dt: 2503.73ms | tok/sec: 209,402 | bf16_mfu: 31.97 | epoch: 1 pq: 15 rg: 64 | total time: 57.14m | eta: 5.1m |
| step 01379/01500 (91.93%) | loss: 2.419726 | lrm: 1.00 | dt: 2506.55ms | tok/sec: 209,167 | bf16_mfu: 31.93 | epoch: 1 pq: 15 rg: 64 | total time: 57.19m | eta: 5.1m |
| step 01380/01500 (92.00%) | loss: 2.448142 | lrm: 1.00 | dt: 2499.58ms | tok/sec: 209,750 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 64 | total time: 57.23m | eta: 5.0m |
| step 01381/01500 (92.07%) | loss: 2.399573 | lrm: 1.00 | dt: 2497.10ms | tok/sec: 209,958 | bf16_mfu: 32.06 | epoch: 1 pq: 15 rg: 64 | total time: 57.27m | eta: 5.0m |
| step 01382/01500 (92.13%) | loss: 2.383089 | lrm: 1.00 | dt: 2495.66ms | tok/sec: 210,079 | bf16_mfu: 32.07 | epoch: 1 pq: 15 rg: 64 | total time: 57.31m | eta: 4.9m |
| step 01383/01500 (92.20%) | loss: 2.388769 | lrm: 1.00 | dt: 2500.54ms | tok/sec: 209,670 | bf16_mfu: 32.01 | epoch: 1 pq: 15 rg: 72 | total time: 57.35m | eta: 4.9m |
| step 01384/01500 (92.27%) | loss: 2.380467 | lrm: 1.00 | dt: 2497.57ms | tok/sec: 209,919 | bf16_mfu: 32.05 | epoch: 1 pq: 15 rg: 72 | total time: 57.39m | eta: 4.8m |
| step 01385/01500 (92.33%) | loss: 2.401750 | lrm: 1.00 | dt: 2501.86ms | tok/sec: 209,558 | bf16_mfu: 31.99 | epoch: 1 pq: 15 rg: 72 | total time: 57.44m | eta: 4.8m |
| step 01386/01500 (92.40%) | loss: 2.434829 | lrm: 1.00 | dt: 2504.21ms | tok/sec: 209,362 | bf16_mfu: 31.96 | epoch: 1 pq: 15 rg: 72 | total time: 57.48m | eta: 4.8m |
| step 01387/01500 (92.47%) | loss: 2.422441 | lrm: 1.00 | dt: 2496.20ms | tok/sec: 210,034 | bf16_mfu: 32.07 | epoch: 1 pq: 15 rg: 72 | total time: 57.52m | eta: 4.7m |
| step 01388/01500 (92.53%) | loss: 2.441884 | lrm: 1.00 | dt: 2501.04ms | tok/sec: 209,627 | bf16_mfu: 32.00 | epoch: 1 pq: 15 rg: 72 | total time: 57.56m | eta: 4.7m |
| step 01389/01500 (92.60%) | loss: 2.432226 | lrm: 1.00 | dt: 2499.63ms | tok/sec: 209,745 | bf16_mfu: 32.02 | epoch: 1 pq: 15 rg: 72 | total time: 57.60m | eta: 4.6m |
| step 01390/01500 (92.67%) | loss: 2.427714 | lrm: 1.00 | dt: 2504.70ms | tok/sec: 209,321 | bf16_mfu: 31.96 | epoch: 1 pq: 15 rg: 72 | total time: 57.65m | eta: 4.6m |
| step 01391/01500 (92.73%) | loss: 2.449333 | lrm: 1.00 | dt: 2505.90ms | tok/sec: 209,221 | bf16_mfu: 31.94 | epoch: 1 pq: 15 rg: 80 | total time: 57.69m | eta: 4.6m |
| step 01392/01500 (92.80%) | loss: 2.421657 | lrm: 1.00 | dt: 2503.30ms | tok/sec: 209,438 | bf16_mfu: 31.98 | epoch: 1 pq: 15 rg: 80 | total time: 57.73m | eta: 4.5m |
| step 01393/01500 (92.87%) | loss: 2.388789 | lrm: 1.00 | dt: 2502.54ms | tok/sec: 209,502 | bf16_mfu: 31.99 | epoch: 1 pq: 15 rg: 80 | total time: 57.77m | eta: 4.5m |
| step 01394/01500 (92.93%) | loss: 2.394633 | lrm: 1.00 | dt: 2495.98ms | tok/sec: 210,053 | bf16_mfu: 32.07 | epoch: 1 pq: 15 rg: 80 | total time: 57.81m | eta: 4.4m |
| step 01395/01500 (93.00%) | loss: 2.400391 | lrm: 1.00 | dt: 2503.12ms | tok/sec: 209,453 | bf16_mfu: 31.98 | epoch: 1 pq: 15 rg: 80 | total time: 57.85m | eta: 4.4m |
| step 01396/01500 (93.07%) | loss: 2.408024 | lrm: 1.00 | dt: 2505.66ms | tok/sec: 209,241 | bf16_mfu: 31.95 | epoch: 1 pq: 15 rg: 80 | total time: 57.90m | eta: 4.3m |
| step 01397/01500 (93.13%) | loss: 2.436805 | lrm: 1.00 | dt: 2496.85ms | tok/sec: 209,979 | bf16_mfu: 32.06 | epoch: 1 pq: 15 rg: 80 | total time: 57.94m | eta: 4.3m |
| step 01398/01500 (93.20%) | loss: 2.449819 | lrm: 1.00 | dt: 2498.86ms | tok/sec: 209,810 | bf16_mfu: 32.03 | epoch: 1 pq: 15 rg: 80 | total time: 57.98m | eta: 4.3m |
| step 01399/01500 (93.27%) | loss: 2.409758 | lrm: 1.00 | dt: 2498.56ms | tok/sec: 209,836 | bf16_mfu: 32.04 | epoch: 1 pq: 16 rg: 0 | total time: 58.02m | eta: 4.2m |
| Step 01400 | Validation bpb: 0.726742 |
| Step 01400 | ICL early: 2.6447 late: 2.8352 score: +0.1905 | Induction loss: 9.4753 acc: 0.0817 (random-half: 14.0049) |
| step 01400/01500 (93.33%) | loss: 2.407156 | lrm: 1.00 | dt: 2496.82ms | tok/sec: 209,981 | bf16_mfu: 32.06 | epoch: 1 pq: 16 rg: 0 | total time: 58.06m | eta: 4.2m |
| step 01401/01500 (93.40%) | loss: 2.403219 | lrm: 1.00 | dt: 2495.31ms | tok/sec: 210,109 | bf16_mfu: 32.08 | epoch: 1 pq: 16 rg: 0 | total time: 58.10m | eta: 4.1m |
| step 01402/01500 (93.47%) | loss: 2.389298 | lrm: 1.00 | dt: 2502.02ms | tok/sec: 209,545 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 0 | total time: 58.15m | eta: 4.1m |
| step 01403/01500 (93.53%) | loss: 2.374202 | lrm: 1.00 | dt: 2506.42ms | tok/sec: 209,178 | bf16_mfu: 31.94 | epoch: 1 pq: 16 rg: 0 | total time: 58.19m | eta: 4.1m |
| step 01404/01500 (93.60%) | loss: 2.410423 | lrm: 1.00 | dt: 2495.39ms | tok/sec: 210,102 | bf16_mfu: 32.08 | epoch: 1 pq: 16 rg: 0 | total time: 58.23m | eta: 4.0m |
| step 01405/01500 (93.67%) | loss: 2.426619 | lrm: 1.00 | dt: 2500.29ms | tok/sec: 209,691 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 0 | total time: 58.27m | eta: 4.0m |
| step 01406/01500 (93.73%) | loss: 2.397991 | lrm: 1.00 | dt: 2498.97ms | tok/sec: 209,801 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 0 | total time: 58.31m | eta: 3.9m |
| step 01407/01500 (93.80%) | loss: 2.382572 | lrm: 1.00 | dt: 2501.25ms | tok/sec: 209,610 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 8 | total time: 58.35m | eta: 3.9m |
| step 01408/01500 (93.87%) | loss: 2.413051 | lrm: 1.00 | dt: 2500.11ms | tok/sec: 209,706 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 8 | total time: 58.40m | eta: 3.8m |
| step 01409/01500 (93.93%) | loss: 2.382012 | lrm: 1.00 | dt: 2502.67ms | tok/sec: 209,491 | bf16_mfu: 31.98 | epoch: 1 pq: 16 rg: 8 | total time: 58.44m | eta: 3.8m |
| step 01410/01500 (94.00%) | loss: 2.365499 | lrm: 1.00 | dt: 2497.09ms | tok/sec: 209,959 | bf16_mfu: 32.06 | epoch: 1 pq: 16 rg: 8 | total time: 58.48m | eta: 3.8m |
| step 01411/01500 (94.07%) | loss: 2.380447 | lrm: 1.00 | dt: 2506.13ms | tok/sec: 209,202 | bf16_mfu: 31.94 | epoch: 1 pq: 16 rg: 8 | total time: 58.52m | eta: 3.7m |
| step 01412/01500 (94.13%) | loss: 2.356123 | lrm: 1.00 | dt: 2501.11ms | tok/sec: 209,621 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 8 | total time: 58.56m | eta: 3.7m |
| step 01413/01500 (94.20%) | loss: 2.395847 | lrm: 1.00 | dt: 2504.82ms | tok/sec: 209,311 | bf16_mfu: 31.96 | epoch: 1 pq: 16 rg: 8 | total time: 58.60m | eta: 3.6m |
| step 01414/01500 (94.27%) | loss: 2.478718 | lrm: 1.00 | dt: 2503.01ms | tok/sec: 209,463 | bf16_mfu: 31.98 | epoch: 1 pq: 16 rg: 8 | total time: 58.65m | eta: 3.6m |
| step 01415/01500 (94.33%) | loss: 2.453347 | lrm: 1.00 | dt: 2501.11ms | tok/sec: 209,621 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 16 | total time: 58.69m | eta: 3.6m |
| step 01416/01500 (94.40%) | loss: 2.473875 | lrm: 1.00 | dt: 2505.13ms | tok/sec: 209,285 | bf16_mfu: 31.95 | epoch: 1 pq: 16 rg: 16 | total time: 58.73m | eta: 3.5m |
| step 01417/01500 (94.47%) | loss: 2.475766 | lrm: 1.00 | dt: 2501.04ms | tok/sec: 209,627 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 16 | total time: 58.77m | eta: 3.5m |
| step 01418/01500 (94.53%) | loss: 2.443631 | lrm: 1.00 | dt: 2499.46ms | tok/sec: 209,760 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 16 | total time: 58.81m | eta: 3.4m |
| step 01419/01500 (94.60%) | loss: 2.397742 | lrm: 1.00 | dt: 2499.47ms | tok/sec: 209,759 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 16 | total time: 58.85m | eta: 3.4m |
| step 01420/01500 (94.67%) | loss: 2.433936 | lrm: 1.00 | dt: 2496.71ms | tok/sec: 209,991 | bf16_mfu: 32.06 | epoch: 1 pq: 16 rg: 16 | total time: 58.90m | eta: 3.3m |
| step 01421/01500 (94.73%) | loss: 2.418765 | lrm: 1.00 | dt: 2502.30ms | tok/sec: 209,522 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 16 | total time: 58.94m | eta: 3.3m |
| step 01422/01500 (94.80%) | loss: 2.397046 | lrm: 1.00 | dt: 2497.44ms | tok/sec: 209,929 | bf16_mfu: 32.05 | epoch: 1 pq: 16 rg: 16 | total time: 58.98m | eta: 3.3m |
| step 01423/01500 (94.87%) | loss: 2.391084 | lrm: 1.00 | dt: 2499.23ms | tok/sec: 209,779 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 24 | total time: 59.02m | eta: 3.2m |
| step 01424/01500 (94.93%) | loss: 2.398878 | lrm: 1.00 | dt: 2498.95ms | tok/sec: 209,803 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 24 | total time: 59.06m | eta: 3.2m |
| step 01425/01500 (95.00%) | loss: 2.419595 | lrm: 1.00 | dt: 2499.91ms | tok/sec: 209,723 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 24 | total time: 59.10m | eta: 3.1m |
| step 01426/01500 (95.07%) | loss: 2.457590 | lrm: 1.00 | dt: 2502.04ms | tok/sec: 209,544 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 24 | total time: 59.15m | eta: 3.1m |
| step 01427/01500 (95.13%) | loss: 2.493086 | lrm: 1.00 | dt: 2502.18ms | tok/sec: 209,532 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 24 | total time: 59.19m | eta: 3.0m |
| step 01428/01500 (95.20%) | loss: 2.485929 | lrm: 1.00 | dt: 2498.83ms | tok/sec: 209,812 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 24 | total time: 59.23m | eta: 3.0m |
| step 01429/01500 (95.27%) | loss: 2.487202 | lrm: 1.00 | dt: 2499.08ms | tok/sec: 209,792 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 24 | total time: 59.27m | eta: 3.0m |
| step 01430/01500 (95.33%) | loss: 2.471029 | lrm: 1.00 | dt: 2502.32ms | tok/sec: 209,520 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 24 | total time: 59.31m | eta: 2.9m |
| step 01431/01500 (95.40%) | loss: 2.456697 | lrm: 1.00 | dt: 2505.03ms | tok/sec: 209,294 | bf16_mfu: 31.95 | epoch: 1 pq: 16 rg: 24 | total time: 59.35m | eta: 2.9m |
| step 01432/01500 (95.47%) | loss: 2.437899 | lrm: 1.00 | dt: 2747.31ms | tok/sec: 190,836 | bf16_mfu: 29.14 | epoch: 1 pq: 16 rg: 32 | total time: 59.40m | eta: 2.8m |
| step 01433/01500 (95.53%) | loss: 2.431286 | lrm: 1.00 | dt: 2500.94ms | tok/sec: 209,636 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 32 | total time: 59.44m | eta: 2.8m |
| step 01434/01500 (95.60%) | loss: 2.451353 | lrm: 1.00 | dt: 2500.11ms | tok/sec: 209,706 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 32 | total time: 59.48m | eta: 2.8m |
| step 01435/01500 (95.67%) | loss: 2.444935 | lrm: 1.00 | dt: 2507.48ms | tok/sec: 209,089 | bf16_mfu: 31.92 | epoch: 1 pq: 16 rg: 32 | total time: 59.52m | eta: 2.7m |
| step 01436/01500 (95.73%) | loss: 2.460388 | lrm: 1.00 | dt: 2500.79ms | tok/sec: 209,648 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 32 | total time: 59.57m | eta: 2.7m |
| step 01437/01500 (95.80%) | loss: 2.473801 | lrm: 1.00 | dt: 2506.70ms | tok/sec: 209,154 | bf16_mfu: 31.93 | epoch: 1 pq: 16 rg: 32 | total time: 59.61m | eta: 2.6m |
| step 01438/01500 (95.87%) | loss: 2.483231 | lrm: 1.00 | dt: 2501.54ms | tok/sec: 209,586 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 32 | total time: 59.65m | eta: 2.6m |
| step 01439/01500 (95.93%) | loss: 2.443377 | lrm: 1.00 | dt: 2502.14ms | tok/sec: 209,536 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 32 | total time: 59.69m | eta: 2.5m |
| step 01440/01500 (96.00%) | loss: 2.462744 | lrm: 1.00 | dt: 2504.65ms | tok/sec: 209,325 | bf16_mfu: 31.96 | epoch: 1 pq: 16 rg: 40 | total time: 59.73m | eta: 2.5m |
| step 01441/01500 (96.07%) | loss: 2.448690 | lrm: 1.00 | dt: 2502.88ms | tok/sec: 209,473 | bf16_mfu: 31.98 | epoch: 1 pq: 16 rg: 40 | total time: 59.78m | eta: 2.5m |
| step 01442/01500 (96.13%) | loss: 2.443707 | lrm: 1.00 | dt: 2497.90ms | tok/sec: 209,891 | bf16_mfu: 32.05 | epoch: 1 pq: 16 rg: 40 | total time: 59.82m | eta: 2.4m |
| step 01443/01500 (96.20%) | loss: 2.447738 | lrm: 1.00 | dt: 2503.93ms | tok/sec: 209,386 | bf16_mfu: 31.97 | epoch: 1 pq: 16 rg: 40 | total time: 59.86m | eta: 2.4m |
| step 01444/01500 (96.27%) | loss: 2.430013 | lrm: 1.00 | dt: 2498.28ms | tok/sec: 209,859 | bf16_mfu: 32.04 | epoch: 1 pq: 16 rg: 40 | total time: 59.90m | eta: 2.3m |
| step 01445/01500 (96.33%) | loss: 2.428550 | lrm: 1.00 | dt: 2503.70ms | tok/sec: 209,405 | bf16_mfu: 31.97 | epoch: 1 pq: 16 rg: 40 | total time: 59.94m | eta: 2.3m |
| step 01446/01500 (96.40%) | loss: 2.428568 | lrm: 1.00 | dt: 2501.13ms | tok/sec: 209,620 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 40 | total time: 59.98m | eta: 2.3m |
| step 01447/01500 (96.47%) | loss: 2.453339 | lrm: 1.00 | dt: 2501.42ms | tok/sec: 209,596 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 40 | total time: 60.03m | eta: 2.2m |
| step 01448/01500 (96.53%) | loss: 2.472597 | lrm: 1.00 | dt: 2503.40ms | tok/sec: 209,430 | bf16_mfu: 31.97 | epoch: 1 pq: 16 rg: 48 | total time: 60.07m | eta: 2.2m |
| step 01449/01500 (96.60%) | loss: 2.449282 | lrm: 1.00 | dt: 2499.85ms | tok/sec: 209,727 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 48 | total time: 60.11m | eta: 2.1m |
| step 01450/01500 (96.67%) | loss: 2.456129 | lrm: 1.00 | dt: 2505.50ms | tok/sec: 209,254 | bf16_mfu: 31.95 | epoch: 1 pq: 16 rg: 48 | total time: 60.15m | eta: 2.1m |
| step 01451/01500 (96.73%) | loss: 2.478262 | lrm: 1.00 | dt: 2495.35ms | tok/sec: 210,105 | bf16_mfu: 32.08 | epoch: 1 pq: 16 rg: 48 | total time: 60.19m | eta: 2.0m |
| step 01452/01500 (96.80%) | loss: 2.469788 | lrm: 1.00 | dt: 2500.53ms | tok/sec: 209,670 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 48 | total time: 60.23m | eta: 2.0m |
| step 01453/01500 (96.87%) | loss: 2.490518 | lrm: 1.00 | dt: 2507.76ms | tok/sec: 209,066 | bf16_mfu: 31.92 | epoch: 1 pq: 16 rg: 48 | total time: 60.28m | eta: 2.0m |
| step 01454/01500 (96.93%) | loss: 2.480713 | lrm: 1.00 | dt: 2501.42ms | tok/sec: 209,595 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 48 | total time: 60.32m | eta: 1.9m |
| step 01455/01500 (97.00%) | loss: 2.483244 | lrm: 1.00 | dt: 2500.85ms | tok/sec: 209,643 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 48 | total time: 60.36m | eta: 1.9m |
| step 01456/01500 (97.07%) | loss: 2.475816 | lrm: 1.00 | dt: 2499.12ms | tok/sec: 209,789 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 56 | total time: 60.40m | eta: 1.8m |
| step 01457/01500 (97.13%) | loss: 2.505269 | lrm: 1.00 | dt: 2502.45ms | tok/sec: 209,509 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 56 | total time: 60.44m | eta: 1.8m |
| step 01458/01500 (97.20%) | loss: 2.408040 | lrm: 1.00 | dt: 2500.37ms | tok/sec: 209,684 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 56 | total time: 60.48m | eta: 1.8m |
| step 01459/01500 (97.27%) | loss: 2.404778 | lrm: 1.00 | dt: 2501.87ms | tok/sec: 209,558 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 56 | total time: 60.53m | eta: 1.7m |
| step 01460/01500 (97.33%) | loss: 2.401158 | lrm: 1.00 | dt: 2505.31ms | tok/sec: 209,270 | bf16_mfu: 31.95 | epoch: 1 pq: 16 rg: 56 | total time: 60.57m | eta: 1.7m |
| step 01461/01500 (97.40%) | loss: 2.393260 | lrm: 1.00 | dt: 2499.41ms | tok/sec: 209,764 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 56 | total time: 60.61m | eta: 1.6m |
| step 01462/01500 (97.47%) | loss: 2.377952 | lrm: 1.00 | dt: 2501.03ms | tok/sec: 209,628 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 56 | total time: 60.65m | eta: 1.6m |
| step 01463/01500 (97.53%) | loss: 2.363887 | lrm: 1.00 | dt: 2497.56ms | tok/sec: 209,920 | bf16_mfu: 32.05 | epoch: 1 pq: 16 rg: 56 | total time: 60.69m | eta: 1.5m |
| step 01464/01500 (97.60%) | loss: 2.300430 | lrm: 1.00 | dt: 2496.16ms | tok/sec: 210,038 | bf16_mfu: 32.07 | epoch: 1 pq: 16 rg: 64 | total time: 60.73m | eta: 1.5m |
| step 01465/01500 (97.67%) | loss: 2.296305 | lrm: 1.00 | dt: 2504.82ms | tok/sec: 209,311 | bf16_mfu: 31.96 | epoch: 1 pq: 16 rg: 64 | total time: 60.78m | eta: 1.5m |
| step 01466/01500 (97.73%) | loss: 2.294729 | lrm: 1.00 | dt: 2505.22ms | tok/sec: 209,278 | bf16_mfu: 31.95 | epoch: 1 pq: 16 rg: 64 | total time: 60.82m | eta: 1.4m |
| step 01467/01500 (97.80%) | loss: 2.273269 | lrm: 1.00 | dt: 2500.66ms | tok/sec: 209,659 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 64 | total time: 60.86m | eta: 1.4m |
| step 01468/01500 (97.87%) | loss: 2.291700 | lrm: 1.00 | dt: 2498.76ms | tok/sec: 209,819 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 64 | total time: 60.90m | eta: 1.3m |
| step 01469/01500 (97.93%) | loss: 2.234252 | lrm: 1.00 | dt: 2499.27ms | tok/sec: 209,776 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 64 | total time: 60.94m | eta: 1.3m |
| step 01470/01500 (98.00%) | loss: 2.249146 | lrm: 1.00 | dt: 2503.34ms | tok/sec: 209,435 | bf16_mfu: 31.98 | epoch: 1 pq: 16 rg: 64 | total time: 60.98m | eta: 1.3m |
| step 01471/01500 (98.07%) | loss: 2.286775 | lrm: 1.00 | dt: 2504.94ms | tok/sec: 209,301 | bf16_mfu: 31.96 | epoch: 1 pq: 16 rg: 64 | total time: 61.03m | eta: 1.2m |
| step 01472/01500 (98.13%) | loss: 2.296955 | lrm: 1.00 | dt: 2497.88ms | tok/sec: 209,892 | bf16_mfu: 32.05 | epoch: 1 pq: 16 rg: 64 | total time: 61.07m | eta: 1.2m |
| step 01473/01500 (98.20%) | loss: 2.349969 | lrm: 1.00 | dt: 2500.89ms | tok/sec: 209,640 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 72 | total time: 61.11m | eta: 1.1m |
| step 01474/01500 (98.27%) | loss: 2.382145 | lrm: 1.00 | dt: 2502.49ms | tok/sec: 209,506 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 72 | total time: 61.15m | eta: 1.1m |
| step 01475/01500 (98.33%) | loss: 2.374149 | lrm: 1.00 | dt: 2497.38ms | tok/sec: 209,935 | bf16_mfu: 32.05 | epoch: 1 pq: 16 rg: 72 | total time: 61.19m | eta: 1.0m |
| step 01476/01500 (98.40%) | loss: 2.375506 | lrm: 1.00 | dt: 2501.69ms | tok/sec: 209,573 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 72 | total time: 61.23m | eta: 1.0m |
| step 01477/01500 (98.47%) | loss: 2.387930 | lrm: 1.00 | dt: 2501.36ms | tok/sec: 209,601 | bf16_mfu: 32.00 | epoch: 1 pq: 16 rg: 72 | total time: 61.28m | eta: 1.0m |
| step 01478/01500 (98.53%) | loss: 2.390028 | lrm: 1.00 | dt: 2499.69ms | tok/sec: 209,741 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 72 | total time: 61.32m | eta: 0.9m |
| step 01479/01500 (98.60%) | loss: 2.376601 | lrm: 1.00 | dt: 2499.56ms | tok/sec: 209,752 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 72 | total time: 61.36m | eta: 0.9m |
| step 01480/01500 (98.67%) | loss: 2.330563 | lrm: 1.00 | dt: 2504.31ms | tok/sec: 209,353 | bf16_mfu: 31.96 | epoch: 1 pq: 16 rg: 72 | total time: 61.40m | eta: 0.8m |
| step 01481/01500 (98.73%) | loss: 2.326115 | lrm: 1.00 | dt: 2498.09ms | tok/sec: 209,875 | bf16_mfu: 32.04 | epoch: 1 pq: 16 rg: 80 | total time: 61.44m | eta: 0.8m |
| step 01482/01500 (98.80%) | loss: 2.331603 | lrm: 1.00 | dt: 2500.58ms | tok/sec: 209,666 | bf16_mfu: 32.01 | epoch: 1 pq: 16 rg: 80 | total time: 61.48m | eta: 0.8m |
| step 01483/01500 (98.87%) | loss: 2.368139 | lrm: 1.00 | dt: 2503.89ms | tok/sec: 209,389 | bf16_mfu: 31.97 | epoch: 1 pq: 16 rg: 80 | total time: 61.53m | eta: 0.7m |
| step 01484/01500 (98.93%) | loss: 2.375422 | lrm: 1.00 | dt: 2498.38ms | tok/sec: 209,850 | bf16_mfu: 32.04 | epoch: 1 pq: 16 rg: 80 | total time: 61.57m | eta: 0.7m |
| step 01485/01500 (99.00%) | loss: 2.354445 | lrm: 1.00 | dt: 2502.87ms | tok/sec: 209,475 | bf16_mfu: 31.98 | epoch: 1 pq: 16 rg: 80 | total time: 61.61m | eta: 0.6m |
| step 01486/01500 (99.07%) | loss: 2.357608 | lrm: 1.00 | dt: 2499.33ms | tok/sec: 209,771 | bf16_mfu: 32.03 | epoch: 1 pq: 16 rg: 80 | total time: 61.65m | eta: 0.6m |
| step 01487/01500 (99.13%) | loss: 2.380119 | lrm: 1.00 | dt: 2502.01ms | tok/sec: 209,546 | bf16_mfu: 31.99 | epoch: 1 pq: 16 rg: 80 | total time: 61.69m | eta: 0.5m |
| step 01488/01500 (99.20%) | loss: 2.400317 | lrm: 1.00 | dt: 2500.08ms | tok/sec: 209,708 | bf16_mfu: 32.02 | epoch: 1 pq: 16 rg: 80 | total time: 61.73m | eta: 0.5m |
| step 01489/01500 (99.27%) | loss: 2.360456 | lrm: 1.00 | dt: 2500.68ms | tok/sec: 209,658 | bf16_mfu: 32.01 | epoch: 1 pq: 17 rg: 0 | total time: 61.78m | eta: 0.5m |
| step 01490/01500 (99.33%) | loss: 2.400154 | lrm: 1.00 | dt: 2504.74ms | tok/sec: 209,317 | bf16_mfu: 31.96 | epoch: 1 pq: 17 rg: 0 | total time: 61.82m | eta: 0.4m |
| step 01491/01500 (99.40%) | loss: 2.391098 | lrm: 1.00 | dt: 2508.42ms | tok/sec: 209,011 | bf16_mfu: 31.91 | epoch: 1 pq: 17 rg: 0 | total time: 61.86m | eta: 0.4m |
| step 01492/01500 (99.47%) | loss: 2.382892 | lrm: 1.00 | dt: 2498.70ms | tok/sec: 209,824 | bf16_mfu: 32.03 | epoch: 1 pq: 17 rg: 0 | total time: 61.90m | eta: 0.3m |
| step 01493/01500 (99.53%) | loss: 2.386637 | lrm: 1.00 | dt: 2501.30ms | tok/sec: 209,606 | bf16_mfu: 32.00 | epoch: 1 pq: 17 rg: 0 | total time: 61.94m | eta: 0.3m |
| step 01494/01500 (99.60%) | loss: 2.416373 | lrm: 1.00 | dt: 2503.63ms | tok/sec: 209,411 | bf16_mfu: 31.97 | epoch: 1 pq: 17 rg: 0 | total time: 61.98m | eta: 0.3m |
| step 01495/01500 (99.67%) | loss: 2.397640 | lrm: 1.00 | dt: 2505.17ms | tok/sec: 209,282 | bf16_mfu: 31.95 | epoch: 1 pq: 17 rg: 0 | total time: 62.03m | eta: 0.2m |
| step 01496/01500 (99.73%) | loss: 2.419714 | lrm: 1.00 | dt: 2499.40ms | tok/sec: 209,765 | bf16_mfu: 32.03 | epoch: 1 pq: 17 rg: 0 | total time: 62.07m | eta: 0.2m |
| step 01497/01500 (99.80%) | loss: 2.435781 | lrm: 1.00 | dt: 2501.58ms | tok/sec: 209,582 | bf16_mfu: 32.00 | epoch: 1 pq: 17 rg: 8 | total time: 62.11m | eta: 0.1m |
| step 01498/01500 (99.87%) | loss: 2.414833 | lrm: 1.00 | dt: 2502.76ms | tok/sec: 209,483 | bf16_mfu: 31.98 | epoch: 1 pq: 17 rg: 8 | total time: 62.15m | eta: 0.1m |
| step 01499/01500 (99.93%) | loss: 2.365202 | lrm: 1.00 | dt: 2500.78ms | tok/sec: 209,649 | bf16_mfu: 32.01 | epoch: 1 pq: 17 rg: 8 | total time: 62.19m | eta: 0.0m |
| Step 01500 | Validation bpb: 0.725184 |
| Step 01500 | ICL early: 2.6255 late: 2.8190 score: +0.1936 | Induction loss: 9.1172 acc: 0.1010 (random-half: 14.1163) |
| <|bos|>The capital of France is Paris. The capital of France is Paris. The capital of France is Paris. |
| <|bos|>The chemical symbol of gold is Au. It is a soft, yellow, lustrous metal. It is one |
| <|bos|>If yesterday was Friday, then tomorrow will be Saturday. If today was Saturday, then tomorrow will be Sunday. If today was |
| <|bos|>The opposite of hot is cold. Hot is a state of high temperature, while cold is a state of |
| <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, |
| <|bos|>My favorite color is blue. I love the sky, the ocean, and the sky. I love |
| <|bos|>If 5*x + 3 = 13, then x is 3. If 5*x + 3 = 13, then |
| 2026-08-10 23:12:52,520 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank3.pt |
| 2026-08-10 23:13:13,943 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank7.pt |
| 2026-08-10 23:13:27,026 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank4.pt |
| 2026-08-10 23:13:31,204 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank1.pt |
| 2026-08-10 23:13:33,896 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank6.pt |
| 2026-08-10 23:13:34,565 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank5.pt |
| 2026-08-10 23:15:27,234 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank2.pt |
| 2026-08-10 23:15:33,909 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved model parameters to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/model_001500.pt |
| 2026-08-10 23:15:33,913 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved metadata to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/meta_001500.json |
| 2026-08-10 23:15:42,503 - nanochat.checkpoint_manager - [32m[1mINFO[0m - Saved optimizer state to: /root/.cache/nanochat-d32ft/base_checkpoints/d32cmap/optim_001500_rank0.pt |
| Peak memory usage: 60713.63MiB |
| Total training time: 62.19m |
| Minimum validation bpb: 0.725184 |
| [W810 23:15:45.815974576 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator()) |
| |
