=== Llama Context Parallel Training Script (bluscriptCP) === [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) ================ Configuration (bluscriptCP) ================ d_model/layers : 1536 / 36 q/kv heads : 12 / 4 (G=3, hd=128) ffn_hidden : 4096 (SwiGLU) context_length : 65536 vocab_size : 50304 tying=0 B x T : 1 x 65536 world_size : 4 (dp=1 x cp=4) CP_ATTN_MODE : ring (load_balance=HeadTail) global_batch : 524288 (grad_accum=8) precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] max_lr/min_lr : 0.0006 / 6e-05 warmup/steps : 100 / 695 (WSD: stable=593, decay tail=2) checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp) ============================================================= Initializing model on CUDA device 0... [YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693 [InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/b4epoch32k_step2227.ckpt (base step 2227, loss 2.75788) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP Parameters: 1060624896 Parameters per GPU: 1060624896 max_steps: 695 warmup_steps: 100 [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=983236608 padded=983236608 W=4 shard=245809152 | 38 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 2813.06 MB (vs 11252.2 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.41 MB (vs 885.639 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] [ZeRO-1] DataParallel is DISABLED (both would double-reduce) [precision] bf16 params=217, all with fp32 master weights found 5 shards for split train found 1 shards for split val Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log3.csv (run 3) [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base) Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run3_step_.ckpt (every 50 steps, keep 2) [re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp) Starting training... [CP ring P2P] OVERLAP: dedicated non-blocking stream validation loss: 2.9859 [CP bwd] overlap=ON step 0 | loss: 2.903530 | lr 1.8000e-06 | norm: 0.1801 | dt: 23631.38ms | tok/sec: 22186.1 | mem: 107021MB | Time Left: 04 hrs : 33 mins [TIMING] data: 3.9ms | fwd: 4560.4ms | loss: 6.5ms | bwd: 9806.2ms | clip: 0.0ms | optim: 22.7ms step 1 | loss: 3.082012 | lr 3.6000e-06 | norm: 0.2005 | dt: 14108.96ms | tok/sec: 37159.9 | mem: 108493MB | Time Left: 02 hrs : 43 mins [TIMING] data: 3.9ms | fwd: 4437.0ms | loss: 6.5ms | bwd: 9631.4ms | clip: 0.0ms | optim: 22.3ms step 2 | loss: 2.886171 | lr 5.4000e-06 | norm: 0.2042 | dt: 14237.14ms | tok/sec: 36825.4 | mem: 108493MB | Time Left: 02 hrs : 44 mins [TIMING] data: 3.2ms | fwd: 4538.6ms | loss: 6.5ms | bwd: 9658.4ms | clip: 0.0ms | optim: 22.3ms step 3 | loss: 3.095930 | lr 7.2000e-06 | norm: 0.2075 | dt: 12776.49ms | tok/sec: 41035.4 | mem: 108493MB | Time Left: 02 hrs : 27 mins [TIMING] data: 3.5ms | fwd: 3143.4ms | loss: 6.6ms | bwd: 9592.5ms | clip: 0.0ms | optim: 22.4ms step 4 | loss: 3.027580 | lr 9.0000e-06 | norm: 0.1897 | dt: 12813.96ms | tok/sec: 40915.4 | mem: 108493MB | Time Left: 02 hrs : 27 mins [TIMING] data: 3.4ms | fwd: 3148.6ms | loss: 6.5ms | bwd: 9625.0ms | clip: 0.0ms | optim: 22.5ms step 5 | loss: 3.112786 | lr 1.0800e-05 | norm: 0.2091 | dt: 12833.49ms | tok/sec: 40853.1 | mem: 108493MB | Time Left: 02 hrs : 27 mins [TIMING] data: 3.5ms | fwd: 3155.0ms | loss: 6.5ms | bwd: 9638.4ms | clip: 0.0ms | optim: 22.2ms step 6 | loss: 2.922808 | lr 1.2600e-05 | norm: 0.2177 | dt: 12850.11ms | tok/sec: 40800.3 | mem: 108493MB | Time Left: 02 hrs : 27 mins [TIMING] data: 3.5ms | fwd: 3156.9ms | loss: 6.5ms | bwd: 9653.0ms | clip: 0.0ms | optim: 22.3ms step 7 | loss: 3.302711 | lr 1.4400e-05 | norm: 0.2319 | dt: 12852.26ms | tok/sec: 40793.4 | mem: 108493MB | Time Left: 02 hrs : 27 mins [TIMING] data: 3.4ms | fwd: 3160.4ms | loss: 6.5ms | bwd: 9651.7ms | clip: 0.0ms | optim: 22.3ms step 8 | loss: 2.801080 | lr 1.6200e-05 | norm: 0.1830 | dt: 12863.23ms | tok/sec: 40758.7 | mem: 108493MB | Time Left: 02 hrs : 27 mins [TIMING] data: 3.5ms | fwd: 3158.6ms | loss: 6.5ms | bwd: 9664.2ms | clip: 0.0ms | optim: 22.4ms step 9 | loss: 2.836718 | lr 1.8000e-05 | norm: 0.1788 | dt: 12862.86ms | tok/sec: 40759.8 | mem: 108493MB | Time Left: 02 hrs : 27 mins [TIMING] data: 3.4ms | fwd: 3160.1ms | loss: 6.5ms | bwd: 9662.5ms | clip: 0.0ms | optim: 22.4ms step 10 | loss: 3.119102 | lr 1.9800e-05 | norm: 0.1894 | dt: 12829.80ms | tok/sec: 40864.9 | mem: 108493MB | Time Left: 02 hrs : 26 mins [TIMING] data: 3.5ms | fwd: 3155.5ms | loss: 6.5ms | bwd: 9634.1ms | clip: 0.0ms | optim: 22.3ms step 11 | loss: 2.847482 | lr 2.1600e-05 | norm: 0.2169 | dt: 12833.71ms | tok/sec: 40852.4 | mem: 108493MB | Time Left: 02 hrs : 26 mins [TIMING] data: 3.6ms | fwd: 3155.0ms | loss: 6.5ms | bwd: 9638.4ms | clip: 0.0ms | optim: 22.4ms step 12 | loss: 3.016829 | lr 2.3400e-05 | norm: 0.1826 | dt: 12826.60ms | tok/sec: 40875.1 | mem: 108493MB | Time Left: 02 hrs : 26 mins [TIMING] data: 3.5ms | fwd: 3150.5ms | loss: 6.5ms | bwd: 9636.0ms | clip: 0.0ms | optim: 22.2ms step 13 | loss: 2.562746 | lr 2.5200e-05 | norm: 0.1944 | dt: 12836.58ms | tok/sec: 40843.3 | mem: 108493MB | Time Left: 02 hrs : 25 mins [TIMING] data: 3.5ms | fwd: 3159.0ms | loss: 6.5ms | bwd: 9637.4ms | clip: 0.0ms | optim: 22.2ms step 14 | loss: 3.239070 | lr 2.7000e-05 | norm: 0.2126 | dt: 12838.56ms | tok/sec: 40837.0 | mem: 108493MB | Time Left: 02 hrs : 25 mins [TIMING] data: 3.5ms | fwd: 3154.5ms | loss: 6.5ms | bwd: 9644.0ms | clip: 0.0ms | optim: 22.2ms step 15 | loss: 2.958668 | lr 2.8800e-05 | norm: 0.2004 | dt: 12845.82ms | tok/sec: 40813.9 | mem: 108493MB | Time Left: 02 hrs : 25 mins [TIMING] data: 3.5ms | fwd: 3154.9ms | loss: 6.5ms | bwd: 9650.5ms | clip: 0.0ms | optim: 22.4ms step 16 | loss: 2.994854 | lr 3.0600e-05 | norm: 0.1526 | dt: 12846.40ms | tok/sec: 40812.1 | mem: 108493MB | Time Left: 02 hrs : 25 mins [TIMING] data: 3.4ms | fwd: 3154.4ms | loss: 6.5ms | bwd: 9651.9ms | clip: 0.0ms | optim: 22.3ms step 17 | loss: 3.034569 | lr 3.2400e-05 | norm: 0.1725 | dt: 12825.72ms | tok/sec: 40877.9 | mem: 108493MB | Time Left: 02 hrs : 24 mins [TIMING] data: 3.5ms | fwd: 3156.0ms | loss: 6.5ms | bwd: 9629.5ms | clip: 0.0ms | optim: 22.2ms step 18 | loss: 2.835390 | lr 3.4200e-05 | norm: 0.1914 | dt: 12839.76ms | tok/sec: 40833.2 | mem: 108493MB | Time Left: 02 hrs : 24 mins [TIMING] data: 3.5ms | fwd: 3157.4ms | loss: 6.5ms | bwd: 9642.1ms | clip: 0.0ms | optim: 22.2ms step 19 | loss: 2.831429 | lr 3.6000e-05 | norm: 0.1710 | dt: 12834.16ms | tok/sec: 40851.0 | mem: 108493MB | Time Left: 02 hrs : 24 mins [TIMING] data: 3.5ms | fwd: 3153.1ms | loss: 6.5ms | bwd: 9640.9ms | clip: 0.0ms | optim: 22.3ms step 20 | loss: 3.005372 | lr 3.7800e-05 | norm: 0.1801 | dt: 12827.37ms | tok/sec: 40872.6 | mem: 108493MB | Time Left: 02 hrs : 24 mins [TIMING] data: 3.4ms | fwd: 3154.7ms | loss: 6.5ms | bwd: 9632.5ms | clip: 0.0ms | optim: 22.4ms step 21 | loss: 2.661791 | lr 3.9600e-05 | norm: 0.2038 | dt: 12836.89ms | tok/sec: 40842.3 | mem: 108493MB | Time Left: 02 hrs : 24 mins [TIMING] data: 3.5ms | fwd: 3154.2ms | loss: 6.5ms | bwd: 9642.4ms | clip: 0.0ms | optim: 22.3ms step 22 | loss: 3.171215 | lr 4.1400e-05 | norm: 0.1923 | dt: 12845.54ms | tok/sec: 40814.8 | mem: 108493MB | Time Left: 02 hrs : 24 mins [TIMING] data: 3.5ms | fwd: 3158.5ms | loss: 6.5ms | bwd: 9646.5ms | clip: 0.0ms | optim: 22.5ms step 23 | loss: 2.713984 | lr 4.3200e-05 | norm: 0.1618 | dt: 12837.71ms | tok/sec: 40839.7 | mem: 108493MB | Time Left: 02 hrs : 23 mins [TIMING] data: 3.4ms | fwd: 3153.3ms | loss: 6.5ms | bwd: 9644.3ms | clip: 0.0ms | optim: 22.2ms step 24 | loss: 2.585613 | lr 4.5000e-05 | norm: 0.1588 | dt: 12842.00ms | tok/sec: 40826.0 | mem: 108493MB | Time Left: 02 hrs : 23 mins [TIMING] data: 3.4ms | fwd: 3156.3ms | loss: 6.5ms | bwd: 9645.6ms | clip: 0.0ms | optim: 22.3ms step 25 | loss: 3.209198 | lr 4.6800e-05 | norm: 0.2091 | dt: 12832.96ms | tok/sec: 40854.8 | mem: 108493MB | Time Left: 02 hrs : 23 mins [TIMING] data: 3.5ms | fwd: 3153.1ms | loss: 6.5ms | bwd: 9639.4ms | clip: 0.0ms | optim: 22.5ms step 26 | loss: 2.917761 | lr 4.8600e-05 | norm: 0.1782 | dt: 12827.72ms | tok/sec: 40871.5 | mem: 108493MB | Time Left: 02 hrs : 23 mins [TIMING] data: 3.6ms | fwd: 3156.2ms | loss: 6.5ms | bwd: 9631.2ms | clip: 0.0ms | optim: 22.2ms step 27 | loss: 2.914461 | lr 5.0400e-05 | norm: 0.1650 | dt: 12837.71ms | tok/sec: 40839.7 | mem: 108493MB | Time Left: 02 hrs : 22 mins [TIMING] data: 3.4ms | fwd: 3155.4ms | loss: 6.5ms | bwd: 9642.2ms | clip: 0.0ms | optim: 22.2ms step 28 | loss: 3.068598 | lr 5.2200e-05 | norm: 0.1900 | dt: 12846.50ms | tok/sec: 40811.7 | mem: 108493MB | Time Left: 02 hrs : 22 mins [TIMING] data: 3.6ms | fwd: 3153.6ms | loss: 6.5ms | bwd: 9652.5ms | clip: 0.0ms | optim: 22.4ms step 29 | loss: 2.561096 | lr 5.4000e-05 | norm: 0.1590 | dt: 12846.44ms | tok/sec: 40811.9 | mem: 108493MB | Time Left: 02 hrs : 22 mins [TIMING] data: 3.5ms | fwd: 3155.9ms | loss: 6.5ms | bwd: 9650.1ms | clip: 0.0ms | optim: 22.4ms step 30 | loss: 3.219699 | lr 5.5800e-05 | norm: 0.1910 | dt: 12841.82ms | tok/sec: 40826.6 | mem: 108493MB | Time Left: 02 hrs : 22 mins [TIMING] data: 3.5ms | fwd: 3155.8ms | loss: 6.5ms | bwd: 9645.9ms | clip: 0.0ms | optim: 22.2ms step 31 | loss: 2.912727 | lr 5.7600e-05 | norm: 0.1738 | dt: 12851.01ms | tok/sec: 40797.4 | mem: 108493MB | Time Left: 02 hrs : 22 mins [TIMING] data: 3.4ms | fwd: 3157.0ms | loss: 6.5ms | bwd: 9653.6ms | clip: 0.0ms | optim: 22.5ms step 32 | loss: 2.874547 | lr 5.9400e-05 | norm: 0.1922 | dt: 12845.46ms | tok/sec: 40815.1 | mem: 108493MB | Time Left: 02 hrs : 21 mins [TIMING] data: 3.5ms | fwd: 3156.0ms | loss: 6.5ms | bwd: 9649.2ms | clip: 0.0ms | optim: 22.3ms step 33 | loss: 3.051065 | lr 6.1200e-05 | norm: 0.1766 | dt: 12831.79ms | tok/sec: 40858.5 | mem: 108493MB | Time Left: 02 hrs : 21 mins [TIMING] data: 3.4ms | fwd: 3156.5ms | loss: 6.5ms | bwd: 9635.1ms | clip: 0.0ms | optim: 22.3ms step 34 | loss: 2.595632 | lr 6.3000e-05 | norm: 0.1927 | dt: 12834.02ms | tok/sec: 40851.4 | mem: 108493MB | Time Left: 02 hrs : 21 mins [TIMING] data: 3.3ms | fwd: 3152.6ms | loss: 6.5ms | bwd: 9641.3ms | clip: 0.0ms | optim: 22.3ms step 35 | loss: 2.962336 | lr 6.4800e-05 | norm: 0.2375 | dt: 12834.84ms | tok/sec: 40848.8 | mem: 108493MB | Time Left: 02 hrs : 21 mins [TIMING] data: 3.6ms | fwd: 3153.2ms | loss: 6.5ms | bwd: 9641.1ms | clip: 0.0ms | optim: 22.3ms step 36 | loss: 2.889726 | lr 6.6600e-05 | norm: 0.1728 | dt: 12832.89ms | tok/sec: 40855.0 | mem: 108493MB | Time Left: 02 hrs : 20 mins [TIMING] data: 3.5ms | fwd: 3157.5ms | loss: 6.5ms | bwd: 9634.9ms | clip: 0.0ms | optim: 22.5ms step 37 | loss: 3.078321 | lr 6.8400e-05 | norm: 0.1735 | dt: 12833.26ms | tok/sec: 40853.8 | mem: 108493MB | Time Left: 02 hrs : 20 mins [TIMING] data: 3.6ms | fwd: 3155.0ms | loss: 6.5ms | bwd: 9637.5ms | clip: 0.0ms | optim: 22.3ms step 38 | loss: 2.628988 | lr 7.0200e-05 | norm: 0.1595 | dt: 12831.97ms | tok/sec: 40858.0 | mem: 108493MB | Time Left: 02 hrs : 20 mins [TIMING] data: 3.4ms | fwd: 3153.5ms | loss: 6.5ms | bwd: 9638.3ms | clip: 0.0ms | optim: 22.3ms step 39 | loss: 3.080482 | lr 7.2000e-05 | norm: 0.1762 | dt: 12833.94ms | tok/sec: 40851.7 | mem: 108493MB | Time Left: 02 hrs : 20 mins [TIMING] data: 3.6ms | fwd: 3153.1ms | loss: 6.5ms | bwd: 9640.3ms | clip: 0.0ms | optim: 22.5ms step 40 | loss: 2.664310 | lr 7.3800e-05 | norm: 0.1670 | dt: 12846.38ms | tok/sec: 40812.1 | mem: 108493MB | Time Left: 02 hrs : 20 mins [TIMING] data: 3.4ms | fwd: 3154.4ms | loss: 6.5ms | bwd: 9651.6ms | clip: 0.0ms | optim: 22.4ms step 41 | loss: 3.105606 | lr 7.5600e-05 | norm: 0.2467 | dt: 12840.46ms | tok/sec: 40830.9 | mem: 108493MB | Time Left: 02 hrs : 19 mins [TIMING] data: 3.5ms | fwd: 3155.3ms | loss: 6.5ms | bwd: 9644.9ms | clip: 0.0ms | optim: 22.3ms step 42 | loss: 3.199048 | lr 7.7400e-05 | norm: 0.2412 | dt: 12843.91ms | tok/sec: 40820.0 | mem: 108493MB | Time Left: 02 hrs : 19 mins [TIMING] data: 3.5ms | fwd: 3154.2ms | loss: 6.5ms | bwd: 9649.4ms | clip: 0.0ms | optim: 22.3ms step 43 | loss: 3.075292 | lr 7.9200e-05 | norm: 0.1668 | dt: 12841.48ms | tok/sec: 40827.7 | mem: 108493MB | Time Left: 02 hrs : 19 mins [TIMING] data: 3.5ms | fwd: 3156.8ms | loss: 6.5ms | bwd: 9644.4ms | clip: 0.0ms | optim: 22.3ms step 44 | loss: 3.191494 | lr 8.1000e-05 | norm: 0.2026 | dt: 12839.01ms | tok/sec: 40835.5 | mem: 108493MB | Time Left: 02 hrs : 19 mins [TIMING] data: 3.4ms | fwd: 3158.3ms | loss: 6.5ms | bwd: 9640.4ms | clip: 0.0ms | optim: 22.4ms step 45 | loss: 2.884108 | lr 8.2800e-05 | norm: 0.2083 | dt: 12840.81ms | tok/sec: 40829.8 | mem: 108493MB | Time Left: 02 hrs : 19 mins [TIMING] data: 3.5ms | fwd: 3158.2ms | loss: 6.5ms | bwd: 9641.9ms | clip: 0.0ms | optim: 22.5ms step 46 | loss: 2.951002 | lr 8.4600e-05 | norm: 0.2034 | dt: 12839.59ms | tok/sec: 40833.7 | mem: 108493MB | Time Left: 02 hrs : 18 mins [TIMING] data: 3.5ms | fwd: 3154.9ms | loss: 6.5ms | bwd: 9644.4ms | clip: 0.0ms | optim: 22.3ms step 47 | loss: 2.536675 | lr 8.6400e-05 | norm: 0.2095 | dt: 12831.62ms | tok/sec: 40859.1 | mem: 108493MB | Time Left: 02 hrs : 18 mins [TIMING] data: 3.4ms | fwd: 3153.1ms | loss: 6.5ms | bwd: 9638.3ms | clip: 0.0ms | optim: 22.3ms step 48 | loss: 3.390801 | lr 8.8200e-05 | norm: 0.2302 | dt: 12845.30ms | tok/sec: 40815.5 | mem: 108493MB | Time Left: 02 hrs : 18 mins [TIMING] data: 3.4ms | fwd: 3155.6ms | loss: 6.6ms | bwd: 9649.0ms | clip: 0.0ms | optim: 22.5ms step 49 | loss: 3.319005 | lr 9.0000e-05 | norm: 0.2102 | dt: 12844.47ms | tok/sec: 40818.2 | mem: 108493MB | Time Left: 02 hrs : 18 mins [TIMING] data: 3.4ms | fwd: 3154.3ms | loss: 6.6ms | bwd: 9649.9ms | clip: 0.0ms | optim: 22.2ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_50.ckpt (safely) step 50 | loss: 2.984474 | lr 9.1800e-05 | norm: 0.2410 | dt: 12905.04ms | tok/sec: 40626.6 | mem: 108493MB | Time Left: 02 hrs : 18 mins [TIMING] data: 3.8ms | fwd: 3295.1ms | loss: 6.5ms | bwd: 9569.2ms | clip: 0.0ms | optim: 22.3ms step 51 | loss: 2.659516 | lr 9.3600e-05 | norm: 0.1773 | dt: 12783.76ms | tok/sec: 41012.0 | mem: 108493MB | Time Left: 02 hrs : 17 mins [TIMING] data: 3.8ms | fwd: 3145.8ms | loss: 6.5ms | bwd: 9597.0ms | clip: 0.0ms | optim: 22.4ms step 52 | loss: 2.831632 | lr 9.5400e-05 | norm: 0.2110 | dt: 12805.58ms | tok/sec: 40942.1 | mem: 108493MB | Time Left: 02 hrs : 17 mins [TIMING] data: 3.9ms | fwd: 3151.3ms | loss: 6.5ms | bwd: 9613.5ms | clip: 0.0ms | optim: 22.4ms step 53 | loss: 3.096663 | lr 9.7200e-05 | norm: 0.2115 | dt: 12822.60ms | tok/sec: 40887.8 | mem: 108493MB | Time Left: 02 hrs : 17 mins [TIMING] data: 3.9ms | fwd: 3151.0ms | loss: 6.5ms | bwd: 9631.0ms | clip: 0.0ms | optim: 22.3ms step 54 | loss: 3.215490 | lr 9.9000e-05 | norm: 0.2688 | dt: 12819.07ms | tok/sec: 40899.1 | mem: 108493MB | Time Left: 02 hrs : 16 mins [TIMING] data: 3.8ms | fwd: 3155.4ms | loss: 6.5ms | bwd: 9622.9ms | clip: 0.0ms | optim: 22.4ms step 55 | loss: 3.335117 | lr 1.0080e-04 | norm: 0.2198 | dt: 12820.34ms | tok/sec: 40895.0 | mem: 108493MB | Time Left: 02 hrs : 16 mins [TIMING] data: 4.0ms | fwd: 3154.9ms | loss: 6.5ms | bwd: 9624.5ms | clip: 0.0ms | optim: 22.4ms step 56 | loss: 3.083021 | lr 1.0260e-04 | norm: 0.2047 | dt: 12829.01ms | tok/sec: 40867.4 | mem: 108493MB | Time Left: 02 hrs : 16 mins [TIMING] data: 3.8ms | fwd: 3153.0ms | loss: 6.5ms | bwd: 9635.2ms | clip: 0.0ms | optim: 22.4ms step 57 | loss: 3.000520 | lr 1.0440e-04 | norm: 0.2083 | dt: 12825.42ms | tok/sec: 40878.8 | mem: 108493MB | Time Left: 02 hrs : 16 mins [TIMING] data: 3.8ms | fwd: 3155.3ms | loss: 6.5ms | bwd: 9629.6ms | clip: 0.0ms | optim: 22.2ms step 58 | loss: 2.896519 | lr 1.0620e-04 | norm: 0.1941 | dt: 12839.27ms | tok/sec: 40834.7 | mem: 108493MB | Time Left: 02 hrs : 16 mins [TIMING] data: 3.9ms | fwd: 3158.8ms | loss: 6.5ms | bwd: 9639.8ms | clip: 0.0ms | optim: 22.3ms step 59 | loss: 2.811779 | lr 1.0800e-04 | norm: 0.1955 | dt: 12819.30ms | tok/sec: 40898.3 | mem: 108493MB | Time Left: 02 hrs : 15 mins [TIMING] data: 3.9ms | fwd: 3149.8ms | loss: 6.5ms | bwd: 9628.9ms | clip: 0.0ms | optim: 22.2ms step 60 | loss: 2.900863 | lr 1.0980e-04 | norm: 0.2027 | dt: 12844.11ms | tok/sec: 40819.3 | mem: 108493MB | Time Left: 02 hrs : 15 mins [TIMING] data: 3.9ms | fwd: 3158.2ms | loss: 6.5ms | bwd: 9645.1ms | clip: 0.0ms | optim: 22.4ms step 61 | loss: 2.592367 | lr 1.1160e-04 | norm: 0.2087 | dt: 12841.27ms | tok/sec: 40828.4 | mem: 108493MB | Time Left: 02 hrs : 15 mins [TIMING] data: 3.9ms | fwd: 3155.1ms | loss: 6.5ms | bwd: 9645.4ms | clip: 0.0ms | optim: 22.3ms step 62 | loss: 2.987607 | lr 1.1340e-04 | norm: 0.2137 | dt: 12847.37ms | tok/sec: 40809.0 | mem: 108493MB | Time Left: 02 hrs : 15 mins [TIMING] data: 3.9ms | fwd: 3156.3ms | loss: 6.5ms | bwd: 9650.1ms | clip: 0.0ms | optim: 22.5ms step 63 | loss: 2.935050 | lr 1.1520e-04 | norm: 0.1913 | dt: 12833.47ms | tok/sec: 40853.2 | mem: 108493MB | Time Left: 02 hrs : 15 mins [TIMING] data: 3.9ms | fwd: 3155.1ms | loss: 6.5ms | bwd: 9637.5ms | clip: 0.0ms | optim: 22.5ms step 64 | loss: 3.112875 | lr 1.1700e-04 | norm: 0.2164 | dt: 12849.45ms | tok/sec: 40802.4 | mem: 108493MB | Time Left: 02 hrs : 15 mins [TIMING] data: 4.0ms | fwd: 3156.7ms | loss: 6.5ms | bwd: 9651.8ms | clip: 0.0ms | optim: 22.4ms step 65 | loss: 3.012614 | lr 1.1880e-04 | norm: 0.2268 | dt: 12840.04ms | tok/sec: 40832.3 | mem: 108493MB | Time Left: 02 hrs : 14 mins [TIMING] data: 3.8ms | fwd: 3154.0ms | loss: 6.5ms | bwd: 9645.4ms | clip: 0.0ms | optim: 22.3ms step 66 | loss: 2.867213 | lr 1.2060e-04 | norm: 0.2190 | dt: 12839.49ms | tok/sec: 40834.0 | mem: 108493MB | Time Left: 02 hrs : 14 mins [TIMING] data: 3.9ms | fwd: 3154.6ms | loss: 6.5ms | bwd: 9644.2ms | clip: 0.0ms | optim: 22.2ms step 67 | loss: 2.888515 | lr 1.2240e-04 | norm: 0.1782 | dt: 12850.66ms | tok/sec: 40798.5 | mem: 108493MB | Time Left: 02 hrs : 14 mins [TIMING] data: 3.9ms | fwd: 3157.3ms | loss: 6.5ms | bwd: 9652.5ms | clip: 0.0ms | optim: 22.3ms step 68 | loss: 2.922676 | lr 1.2420e-04 | norm: 0.2005 | dt: 12834.56ms | tok/sec: 40849.7 | mem: 108493MB | Time Left: 02 hrs : 14 mins [TIMING] data: 3.9ms | fwd: 3153.8ms | loss: 6.5ms | bwd: 9640.1ms | clip: 0.0ms | optim: 22.3ms step 69 | loss: 2.991062 | lr 1.2600e-04 | norm: 0.2351 | dt: 12839.20ms | tok/sec: 40835.0 | mem: 108493MB | Time Left: 02 hrs : 13 mins [TIMING] data: 3.8ms | fwd: 3155.3ms | loss: 6.5ms | bwd: 9643.3ms | clip: 0.0ms | optim: 22.3ms step 70 | loss: 3.104146 | lr 1.2780e-04 | norm: 0.2553 | dt: 12851.11ms | tok/sec: 40797.1 | mem: 108493MB | Time Left: 02 hrs : 13 mins [TIMING] data: 3.9ms | fwd: 3154.1ms | loss: 6.6ms | bwd: 9656.3ms | clip: 0.0ms | optim: 22.2ms step 71 | loss: 2.747528 | lr 1.2960e-04 | norm: 0.2340 | dt: 12848.00ms | tok/sec: 40807.0 | mem: 108493MB | Time Left: 02 hrs : 13 mins [TIMING] data: 4.1ms | fwd: 3155.5ms | loss: 6.5ms | bwd: 9651.2ms | clip: 0.0ms | optim: 22.4ms step 72 | loss: 3.235977 | lr 1.3140e-04 | norm: 0.2331 | dt: 12834.68ms | tok/sec: 40849.3 | mem: 108493MB | Time Left: 02 hrs : 13 mins [TIMING] data: 3.9ms | fwd: 3154.4ms | loss: 6.5ms | bwd: 9639.5ms | clip: 0.0ms | optim: 22.4ms step 73 | loss: 3.080717 | lr 1.3320e-04 | norm: 0.2517 | dt: 12850.37ms | tok/sec: 40799.4 | mem: 108493MB | Time Left: 02 hrs : 13 mins [TIMING] data: 3.8ms | fwd: 3157.1ms | loss: 6.5ms | bwd: 9652.3ms | clip: 0.0ms | optim: 22.5ms step 74 | loss: 2.635970 | lr 1.3500e-04 | norm: 0.2467 | dt: 12843.64ms | tok/sec: 40820.8 | mem: 108493MB | Time Left: 02 hrs : 12 mins [TIMING] data: 3.8ms | fwd: 3156.8ms | loss: 6.5ms | bwd: 9646.2ms | clip: 0.0ms | optim: 22.4ms step 75 | loss: 3.182057 | lr 1.3680e-04 | norm: 0.3033 | dt: 12839.64ms | tok/sec: 40833.5 | mem: 108493MB | Time Left: 02 hrs : 12 mins [TIMING] data: 3.9ms | fwd: 3157.8ms | loss: 6.5ms | bwd: 9640.9ms | clip: 0.0ms | optim: 22.5ms step 76 | loss: 2.485651 | lr 1.3860e-04 | norm: 0.2289 | dt: 12850.08ms | tok/sec: 40800.4 | mem: 108493MB | Time Left: 02 hrs : 12 mins [TIMING] data: 3.8ms | fwd: 3155.3ms | loss: 6.5ms | bwd: 9654.3ms | clip: 0.0ms | optim: 22.2ms step 77 | loss: 2.885402 | lr 1.4040e-04 | norm: 0.2301 | dt: 12840.16ms | tok/sec: 40831.9 | mem: 108493MB | Time Left: 02 hrs : 12 mins [TIMING] data: 3.8ms | fwd: 3155.7ms | loss: 6.5ms | bwd: 9643.9ms | clip: 0.0ms | optim: 22.3ms step 78 | loss: 2.785285 | lr 1.4220e-04 | norm: 0.2261 | dt: 12839.96ms | tok/sec: 40832.5 | mem: 108493MB | Time Left: 02 hrs : 12 mins [TIMING] data: 3.8ms | fwd: 3153.8ms | loss: 6.5ms | bwd: 9645.7ms | clip: 0.0ms | optim: 22.2ms step 79 | loss: 2.703315 | lr 1.4400e-04 | norm: 0.2775 | dt: 12846.00ms | tok/sec: 40813.3 | mem: 108493MB | Time Left: 02 hrs : 11 mins [TIMING] data: 3.9ms | fwd: 3158.3ms | loss: 6.5ms | bwd: 9646.9ms | clip: 0.0ms | optim: 22.4ms step 80 | loss: 2.648544 | lr 1.4580e-04 | norm: 0.2414 | dt: 12843.10ms | tok/sec: 40822.5 | mem: 108493MB | Time Left: 02 hrs : 11 mins [TIMING] data: 3.8ms | fwd: 3158.2ms | loss: 6.5ms | bwd: 9644.3ms | clip: 0.0ms | optim: 22.3ms step 81 | loss: 3.120618 | lr 1.4760e-04 | norm: 0.2386 | dt: 12842.90ms | tok/sec: 40823.2 | mem: 108493MB | Time Left: 02 hrs : 11 mins [TIMING] data: 3.9ms | fwd: 3155.3ms | loss: 6.5ms | bwd: 9647.0ms | clip: 0.0ms | optim: 22.3ms step 82 | loss: 3.148950 | lr 1.4940e-04 | norm: 0.2165 | dt: 12838.38ms | tok/sec: 40837.5 | mem: 108493MB | Time Left: 02 hrs : 11 mins [TIMING] data: 3.8ms | fwd: 3156.6ms | loss: 6.5ms | bwd: 9641.1ms | clip: 0.0ms | optim: 22.5ms step 83 | loss: 3.023019 | lr 1.5120e-04 | norm: 0.2480 | dt: 12886.14ms | tok/sec: 40686.2 | mem: 108493MB | Time Left: 02 hrs : 11 mins [TIMING] data: 3.9ms | fwd: 3198.3ms | loss: 6.5ms | bwd: 9647.3ms | clip: 0.0ms | optim: 22.2ms step 84 | loss: 3.070096 | lr 1.5300e-04 | norm: 0.2094 | dt: 12840.12ms | tok/sec: 40832.0 | mem: 108493MB | Time Left: 02 hrs : 10 mins [TIMING] data: 3.9ms | fwd: 3157.5ms | loss: 6.5ms | bwd: 9642.1ms | clip: 0.0ms | optim: 22.2ms step 85 | loss: 3.041272 | lr 1.5480e-04 | norm: 0.2413 | dt: 12843.78ms | tok/sec: 40820.4 | mem: 108493MB | Time Left: 02 hrs : 10 mins [TIMING] data: 3.9ms | fwd: 3155.7ms | loss: 6.5ms | bwd: 9647.4ms | clip: 0.0ms | optim: 22.4ms step 86 | loss: 3.213121 | lr 1.5660e-04 | norm: 0.2444 | dt: 12841.16ms | tok/sec: 40828.7 | mem: 108493MB | Time Left: 02 hrs : 10 mins [TIMING] data: 4.1ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9643.6ms | clip: 0.0ms | optim: 22.3ms step 87 | loss: 2.808081 | lr 1.5840e-04 | norm: 0.2678 | dt: 12846.34ms | tok/sec: 40812.2 | mem: 108493MB | Time Left: 02 hrs : 10 mins [TIMING] data: 4.0ms | fwd: 3155.7ms | loss: 6.6ms | bwd: 9649.7ms | clip: 0.0ms | optim: 22.3ms step 88 | loss: 3.074635 | lr 1.6020e-04 | norm: 0.2309 | dt: 12838.58ms | tok/sec: 40836.9 | mem: 108493MB | Time Left: 02 hrs : 09 mins [TIMING] data: 4.0ms | fwd: 3158.5ms | loss: 6.6ms | bwd: 9639.1ms | clip: 0.0ms | optim: 22.3ms step 89 | loss: 3.138343 | lr 1.6200e-04 | norm: 0.2798 | dt: 12835.38ms | tok/sec: 40847.1 | mem: 108493MB | Time Left: 02 hrs : 09 mins [TIMING] data: 4.0ms | fwd: 3157.8ms | loss: 6.6ms | bwd: 9636.6ms | clip: 0.0ms | optim: 22.3ms step 90 | loss: 2.973195 | lr 1.6380e-04 | norm: 0.2145 | dt: 12855.77ms | tok/sec: 40782.3 | mem: 108493MB | Time Left: 02 hrs : 09 mins [TIMING] data: 4.0ms | fwd: 3155.0ms | loss: 6.6ms | bwd: 9659.9ms | clip: 0.0ms | optim: 22.3ms step 91 | loss: 2.877568 | lr 1.6560e-04 | norm: 0.2802 | dt: 12842.70ms | tok/sec: 40823.8 | mem: 108493MB | Time Left: 02 hrs : 09 mins [TIMING] data: 4.0ms | fwd: 3155.1ms | loss: 6.6ms | bwd: 9646.6ms | clip: 0.0ms | optim: 22.4ms step 92 | loss: 3.100367 | lr 1.6740e-04 | norm: 0.2492 | dt: 12844.39ms | tok/sec: 40818.4 | mem: 108493MB | Time Left: 02 hrs : 09 mins [TIMING] data: 3.9ms | fwd: 3153.4ms | loss: 6.6ms | bwd: 9650.2ms | clip: 0.0ms | optim: 22.3ms step 93 | loss: 2.953257 | lr 1.6920e-04 | norm: 0.2021 | dt: 12839.42ms | tok/sec: 40834.2 | mem: 108493MB | Time Left: 02 hrs : 08 mins [TIMING] data: 3.9ms | fwd: 3155.3ms | loss: 6.5ms | bwd: 9643.5ms | clip: 0.0ms | optim: 22.3ms step 94 | loss: 3.030771 | lr 1.7100e-04 | norm: 0.2423 | dt: 12912.73ms | tok/sec: 40602.4 | mem: 108493MB | Time Left: 02 hrs : 09 mins [TIMING] data: 3.8ms | fwd: 3219.1ms | loss: 6.5ms | bwd: 9653.1ms | clip: 0.0ms | optim: 22.3ms step 95 | loss: 3.062978 | lr 1.7280e-04 | norm: 0.2224 | dt: 12847.08ms | tok/sec: 40809.9 | mem: 108493MB | Time Left: 02 hrs : 08 mins [TIMING] data: 3.9ms | fwd: 3169.9ms | loss: 6.5ms | bwd: 9636.5ms | clip: 0.0ms | optim: 22.2ms step 96 | loss: 2.832562 | lr 1.7460e-04 | norm: 0.2609 | dt: 12843.92ms | tok/sec: 40819.9 | mem: 108493MB | Time Left: 02 hrs : 08 mins [TIMING] data: 3.8ms | fwd: 3155.2ms | loss: 6.5ms | bwd: 9648.0ms | clip: 0.0ms | optim: 22.4ms step 97 | loss: 2.938385 | lr 1.7640e-04 | norm: 0.3094 | dt: 12894.38ms | tok/sec: 40660.2 | mem: 108493MB | Time Left: 02 hrs : 08 mins [TIMING] data: 3.9ms | fwd: 3201.1ms | loss: 6.5ms | bwd: 9652.6ms | clip: 0.0ms | optim: 22.3ms step 98 | loss: 3.069011 | lr 1.7820e-04 | norm: 0.2168 | dt: 12834.43ms | tok/sec: 40850.1 | mem: 108493MB | Time Left: 02 hrs : 07 mins [TIMING] data: 3.9ms | fwd: 3150.9ms | loss: 6.4ms | bwd: 9642.9ms | clip: 0.0ms | optim: 22.5ms step 99 | loss: 2.947042 | lr 1.8000e-04 | norm: 0.2201 | dt: 12846.80ms | tok/sec: 40810.8 | mem: 108493MB | Time Left: 02 hrs : 07 mins [TIMING] data: 3.9ms | fwd: 3157.9ms | loss: 6.5ms | bwd: 9648.2ms | clip: 0.0ms | optim: 22.3ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_100.ckpt (safely) step 100 | loss: 2.705538 | lr 1.8000e-04 | norm: 0.2804 | dt: 12747.77ms | tok/sec: 41127.8 | mem: 108493MB | Time Left: 02 hrs : 06 mins [TIMING] data: 3.8ms | fwd: 3134.7ms | loss: 6.6ms | bwd: 9572.2ms | clip: 0.0ms | optim: 22.2ms step 101 | loss: 3.020739 | lr 1.8000e-04 | norm: 0.2204 | dt: 12806.05ms | tok/sec: 40940.6 | mem: 108493MB | Time Left: 02 hrs : 06 mins [TIMING] data: 3.9ms | fwd: 3169.1ms | loss: 6.6ms | bwd: 9596.1ms | clip: 0.0ms | optim: 22.2ms step 102 | loss: 3.315276 | lr 1.8000e-04 | norm: 0.2764 | dt: 12808.44ms | tok/sec: 40933.0 | mem: 108493MB | Time Left: 02 hrs : 06 mins [TIMING] data: 3.8ms | fwd: 3152.5ms | loss: 6.6ms | bwd: 9615.1ms | clip: 0.0ms | optim: 22.4ms step 103 | loss: 3.159712 | lr 1.8000e-04 | norm: 0.2189 | dt: 12810.39ms | tok/sec: 40926.8 | mem: 108493MB | Time Left: 02 hrs : 06 mins [TIMING] data: 3.9ms | fwd: 3146.0ms | loss: 6.6ms | bwd: 9623.6ms | clip: 0.0ms | optim: 22.3ms step 104 | loss: 3.056949 | lr 1.8000e-04 | norm: 0.2173 | dt: 12815.62ms | tok/sec: 40910.1 | mem: 108493MB | Time Left: 02 hrs : 06 mins [TIMING] data: 3.8ms | fwd: 3151.8ms | loss: 6.6ms | bwd: 9622.9ms | clip: 0.0ms | optim: 22.4ms step 105 | loss: 3.169673 | lr 1.8000e-04 | norm: 0.2315 | dt: 12825.06ms | tok/sec: 40880.0 | mem: 108493MB | Time Left: 02 hrs : 06 mins [TIMING] data: 3.9ms | fwd: 3149.7ms | loss: 6.6ms | bwd: 9634.6ms | clip: 0.0ms | optim: 22.3ms step 106 | loss: 2.693388 | lr 1.8000e-04 | norm: 0.3161 | dt: 12892.94ms | tok/sec: 40664.7 | mem: 108493MB | Time Left: 02 hrs : 06 mins [TIMING] data: 3.8ms | fwd: 3215.1ms | loss: 6.6ms | bwd: 9637.0ms | clip: 0.0ms | optim: 22.4ms step 107 | loss: 2.819462 | lr 1.8000e-04 | norm: 0.2578 | dt: 12837.78ms | tok/sec: 40839.5 | mem: 108493MB | Time Left: 02 hrs : 05 mins [TIMING] data: 3.8ms | fwd: 3147.3ms | loss: 6.6ms | bwd: 9649.6ms | clip: 0.0ms | optim: 22.4ms step 108 | loss: 2.951405 | lr 1.8000e-04 | norm: 0.2482 | dt: 12834.52ms | tok/sec: 40849.8 | mem: 108493MB | Time Left: 02 hrs : 05 mins [TIMING] data: 3.8ms | fwd: 3150.2ms | loss: 6.6ms | bwd: 9643.4ms | clip: 0.0ms | optim: 22.5ms step 109 | loss: 2.995142 | lr 1.8000e-04 | norm: 0.2143 | dt: 12834.08ms | tok/sec: 40851.2 | mem: 108493MB | Time Left: 02 hrs : 05 mins [TIMING] data: 3.9ms | fwd: 3151.6ms | loss: 6.6ms | bwd: 9641.6ms | clip: 0.0ms | optim: 22.3ms step 110 | loss: 2.866556 | lr 1.8000e-04 | norm: 0.2455 | dt: 12841.06ms | tok/sec: 40829.0 | mem: 108493MB | Time Left: 02 hrs : 05 mins [TIMING] data: 3.9ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9645.4ms | clip: 0.0ms | optim: 22.5ms step 111 | loss: 3.066834 | lr 1.8000e-04 | norm: 0.3069 | dt: 12838.08ms | tok/sec: 40838.5 | mem: 108493MB | Time Left: 02 hrs : 04 mins [TIMING] data: 3.8ms | fwd: 3157.9ms | loss: 6.7ms | bwd: 9639.2ms | clip: 0.0ms | optim: 22.3ms step 112 | loss: 3.013091 | lr 1.8000e-04 | norm: 0.2994 | dt: 12833.51ms | tok/sec: 40853.1 | mem: 108493MB | Time Left: 02 hrs : 04 mins [TIMING] data: 3.8ms | fwd: 3149.6ms | loss: 6.6ms | bwd: 9643.2ms | clip: 0.0ms | optim: 22.2ms step 113 | loss: 3.023381 | lr 1.8000e-04 | norm: 0.3220 | dt: 12885.09ms | tok/sec: 40689.5 | mem: 108493MB | Time Left: 02 hrs : 04 mins [TIMING] data: 3.8ms | fwd: 3202.7ms | loss: 6.6ms | bwd: 9641.4ms | clip: 0.0ms | optim: 22.4ms step 114 | loss: 2.990218 | lr 1.8000e-04 | norm: 0.2328 | dt: 12848.47ms | tok/sec: 40805.5 | mem: 108493MB | Time Left: 02 hrs : 04 mins [TIMING] data: 3.8ms | fwd: 3155.2ms | loss: 6.6ms | bwd: 9652.5ms | clip: 0.0ms | optim: 22.3ms step 115 | loss: 3.055565 | lr 1.8000e-04 | norm: 0.2739 | dt: 12830.71ms | tok/sec: 40862.0 | mem: 108493MB | Time Left: 02 hrs : 04 mins [TIMING] data: 3.9ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9637.5ms | clip: 0.0ms | optim: 22.3ms step 116 | loss: 3.122262 | lr 1.8000e-04 | norm: 0.3118 | dt: 12826.27ms | tok/sec: 40876.1 | mem: 108493MB | Time Left: 02 hrs : 03 mins [TIMING] data: 3.8ms | fwd: 3154.7ms | loss: 6.6ms | bwd: 9630.5ms | clip: 0.0ms | optim: 22.4ms step 117 | loss: 3.173700 | lr 1.8000e-04 | norm: 0.1969 | dt: 12843.75ms | tok/sec: 40820.5 | mem: 108493MB | Time Left: 02 hrs : 03 mins [TIMING] data: 3.8ms | fwd: 3156.2ms | loss: 6.6ms | bwd: 9646.5ms | clip: 0.0ms | optim: 22.4ms step 118 | loss: 3.384058 | lr 1.8000e-04 | norm: 0.2793 | dt: 12831.10ms | tok/sec: 40860.7 | mem: 108493MB | Time Left: 02 hrs : 03 mins [TIMING] data: 3.8ms | fwd: 3156.1ms | loss: 6.6ms | bwd: 9634.1ms | clip: 0.0ms | optim: 22.4ms step 119 | loss: 2.728273 | lr 1.8000e-04 | norm: 0.2479 | dt: 12846.12ms | tok/sec: 40812.9 | mem: 108493MB | Time Left: 02 hrs : 03 mins [TIMING] data: 3.8ms | fwd: 3158.3ms | loss: 6.6ms | bwd: 9646.8ms | clip: 0.0ms | optim: 22.5ms step 120 | loss: 3.242033 | lr 1.8000e-04 | norm: 0.3015 | dt: 12857.61ms | tok/sec: 40776.5 | mem: 108493MB | Time Left: 02 hrs : 03 mins [TIMING] data: 3.9ms | fwd: 3175.1ms | loss: 6.6ms | bwd: 9641.6ms | clip: 0.0ms | optim: 22.4ms step 121 | loss: 3.104766 | lr 1.8000e-04 | norm: 0.2494 | dt: 12843.24ms | tok/sec: 40822.1 | mem: 108493MB | Time Left: 02 hrs : 02 mins [TIMING] data: 3.8ms | fwd: 3153.6ms | loss: 6.6ms | bwd: 9648.8ms | clip: 0.0ms | optim: 22.4ms step 122 | loss: 3.218400 | lr 1.8000e-04 | norm: 0.2508 | dt: 12839.52ms | tok/sec: 40833.9 | mem: 108493MB | Time Left: 02 hrs : 02 mins [TIMING] data: 3.9ms | fwd: 3153.2ms | loss: 6.6ms | bwd: 9645.5ms | clip: 0.0ms | optim: 22.3ms step 123 | loss: 2.849234 | lr 1.8000e-04 | norm: 0.2582 | dt: 12837.50ms | tok/sec: 40840.4 | mem: 108493MB | Time Left: 02 hrs : 02 mins [TIMING] data: 3.9ms | fwd: 3152.2ms | loss: 6.6ms | bwd: 9644.4ms | clip: 0.0ms | optim: 22.3ms step 124 | loss: 2.884363 | lr 1.8000e-04 | norm: 0.4986 | dt: 12846.66ms | tok/sec: 40811.2 | mem: 108493MB | Time Left: 02 hrs : 02 mins [TIMING] data: 3.8ms | fwd: 3157.8ms | loss: 6.6ms | bwd: 9647.9ms | clip: 0.0ms | optim: 22.4ms step 125 | loss: 2.866528 | lr 1.8000e-04 | norm: 0.2593 | dt: 12908.55ms | tok/sec: 40615.6 | mem: 108493MB | Time Left: 02 hrs : 02 mins [TIMING] data: 3.9ms | fwd: 3224.4ms | loss: 6.6ms | bwd: 9643.2ms | clip: 0.0ms | optim: 22.4ms step 126 | loss: 3.087130 | lr 1.8000e-04 | norm: 0.2256 | dt: 12845.11ms | tok/sec: 40816.2 | mem: 108493MB | Time Left: 02 hrs : 01 mins [TIMING] data: 3.8ms | fwd: 3158.8ms | loss: 6.6ms | bwd: 9645.5ms | clip: 0.0ms | optim: 22.3ms step 127 | loss: 3.074619 | lr 1.8000e-04 | norm: 0.3011 | dt: 12851.02ms | tok/sec: 40797.4 | mem: 108493MB | Time Left: 02 hrs : 01 mins [TIMING] data: 3.8ms | fwd: 3159.2ms | loss: 6.6ms | bwd: 9650.8ms | clip: 0.0ms | optim: 22.5ms step 128 | loss: 3.197137 | lr 1.8000e-04 | norm: 0.3453 | dt: 12833.45ms | tok/sec: 40853.2 | mem: 108493MB | Time Left: 02 hrs : 01 mins [TIMING] data: 3.9ms | fwd: 3152.1ms | loss: 6.6ms | bwd: 9640.7ms | clip: 0.0ms | optim: 22.2ms step 129 | loss: 2.801103 | lr 1.8000e-04 | norm: 0.2427 | dt: 12844.48ms | tok/sec: 40818.2 | mem: 108493MB | Time Left: 02 hrs : 01 mins [TIMING] data: 3.9ms | fwd: 3157.0ms | loss: 6.6ms | bwd: 9646.4ms | clip: 0.0ms | optim: 22.6ms step 130 | loss: 3.101458 | lr 1.8000e-04 | norm: 0.3357 | dt: 12836.25ms | tok/sec: 40844.3 | mem: 108493MB | Time Left: 02 hrs : 00 mins [TIMING] data: 3.9ms | fwd: 3150.6ms | loss: 6.6ms | bwd: 9644.9ms | clip: 0.0ms | optim: 22.2ms step 131 | loss: 2.970423 | lr 1.8000e-04 | norm: 0.2876 | dt: 12842.03ms | tok/sec: 40825.9 | mem: 108493MB | Time Left: 02 hrs : 00 mins [TIMING] data: 3.9ms | fwd: 3157.1ms | loss: 6.6ms | bwd: 9644.1ms | clip: 0.0ms | optim: 22.3ms step 132 | loss: 2.790962 | lr 1.8000e-04 | norm: 0.3288 | dt: 12839.28ms | tok/sec: 40834.7 | mem: 108493MB | Time Left: 02 hrs : 00 mins [TIMING] data: 3.8ms | fwd: 3156.2ms | loss: 6.6ms | bwd: 9642.1ms | clip: 0.0ms | optim: 22.4ms step 133 | loss: 3.253783 | lr 1.8000e-04 | norm: 0.2843 | dt: 12840.16ms | tok/sec: 40831.9 | mem: 108493MB | Time Left: 02 hrs : 00 mins [TIMING] data: 3.8ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9642.9ms | clip: 0.0ms | optim: 22.3ms step 134 | loss: 2.888616 | lr 1.8000e-04 | norm: 0.3250 | dt: 12841.19ms | tok/sec: 40828.6 | mem: 108493MB | Time Left: 02 hrs : 00 mins [TIMING] data: 3.8ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9644.0ms | clip: 0.0ms | optim: 22.3ms step 135 | loss: 2.735237 | lr 1.8000e-04 | norm: 0.2995 | dt: 12832.84ms | tok/sec: 40855.2 | mem: 108493MB | Time Left: 01 hrs : 59 mins [TIMING] data: 3.8ms | fwd: 3158.6ms | loss: 6.7ms | bwd: 9633.3ms | clip: 0.0ms | optim: 22.4ms step 136 | loss: 2.964885 | lr 1.8000e-04 | norm: 0.2873 | dt: 12830.18ms | tok/sec: 40863.6 | mem: 108493MB | Time Left: 01 hrs : 59 mins [TIMING] data: 3.8ms | fwd: 3153.6ms | loss: 6.6ms | bwd: 9635.7ms | clip: 0.0ms | optim: 22.4ms step 137 | loss: 3.233908 | lr 1.8000e-04 | norm: 0.2971 | dt: 12832.46ms | tok/sec: 40856.4 | mem: 108493MB | Time Left: 01 hrs : 59 mins [TIMING] data: 3.9ms | fwd: 3152.2ms | loss: 6.6ms | bwd: 9639.0ms | clip: 0.0ms | optim: 22.7ms step 138 | loss: 2.829647 | lr 1.8000e-04 | norm: 0.2788 | dt: 12845.54ms | tok/sec: 40814.8 | mem: 108493MB | Time Left: 01 hrs : 59 mins [TIMING] data: 3.8ms | fwd: 3152.7ms | loss: 6.6ms | bwd: 9652.1ms | clip: 0.0ms | optim: 22.3ms step 139 | loss: 3.086826 | lr 1.8000e-04 | norm: 0.2928 | dt: 12867.83ms | tok/sec: 40744.1 | mem: 108493MB | Time Left: 01 hrs : 59 mins [TIMING] data: 3.8ms | fwd: 3182.1ms | loss: 6.6ms | bwd: 9644.8ms | clip: 0.0ms | optim: 22.4ms step 140 | loss: 3.019313 | lr 1.8000e-04 | norm: 0.3258 | dt: 12853.44ms | tok/sec: 40789.7 | mem: 108493MB | Time Left: 01 hrs : 58 mins [TIMING] data: 3.9ms | fwd: 3158.8ms | loss: 6.6ms | bwd: 9653.9ms | clip: 0.0ms | optim: 22.3ms step 141 | loss: 3.068764 | lr 1.8000e-04 | norm: 0.2516 | dt: 12834.02ms | tok/sec: 40851.4 | mem: 108493MB | Time Left: 01 hrs : 58 mins [TIMING] data: 3.8ms | fwd: 3158.4ms | loss: 6.6ms | bwd: 9634.7ms | clip: 0.0ms | optim: 22.5ms step 142 | loss: 3.231549 | lr 1.8000e-04 | norm: 0.2908 | dt: 12850.89ms | tok/sec: 40797.8 | mem: 108493MB | Time Left: 01 hrs : 58 mins [TIMING] data: 3.8ms | fwd: 3154.4ms | loss: 6.6ms | bwd: 9655.5ms | clip: 0.0ms | optim: 22.5ms step 143 | loss: 3.023292 | lr 1.8000e-04 | norm: 0.3781 | dt: 12847.69ms | tok/sec: 40808.0 | mem: 108493MB | Time Left: 01 hrs : 58 mins [TIMING] data: 4.1ms | fwd: 3157.7ms | loss: 6.6ms | bwd: 9648.7ms | clip: 0.0ms | optim: 22.4ms step 144 | loss: 2.909716 | lr 1.8000e-04 | norm: 0.2826 | dt: 12831.14ms | tok/sec: 40860.6 | mem: 108493MB | Time Left: 01 hrs : 57 mins [TIMING] data: 3.8ms | fwd: 3154.0ms | loss: 6.6ms | bwd: 9636.2ms | clip: 0.0ms | optim: 22.4ms step 145 | loss: 3.210708 | lr 1.8000e-04 | norm: 0.3493 | dt: 12838.14ms | tok/sec: 40838.3 | mem: 108493MB | Time Left: 01 hrs : 57 mins [TIMING] data: 3.8ms | fwd: 3153.2ms | loss: 6.6ms | bwd: 9644.2ms | clip: 0.0ms | optim: 22.3ms step 146 | loss: 2.759732 | lr 1.8000e-04 | norm: 0.2406 | dt: 12912.30ms | tok/sec: 40603.8 | mem: 108493MB | Time Left: 01 hrs : 58 mins [TIMING] data: 3.8ms | fwd: 3223.4ms | loss: 6.6ms | bwd: 9647.9ms | clip: 0.0ms | optim: 22.6ms step 147 | loss: 2.949819 | lr 1.8000e-04 | norm: 0.2444 | dt: 12843.61ms | tok/sec: 40820.9 | mem: 108493MB | Time Left: 01 hrs : 57 mins [TIMING] data: 3.8ms | fwd: 3155.9ms | loss: 6.6ms | bwd: 9647.0ms | clip: 0.0ms | optim: 22.3ms step 148 | loss: 3.290678 | lr 1.8000e-04 | norm: 0.2614 | dt: 12840.84ms | tok/sec: 40829.7 | mem: 108493MB | Time Left: 01 hrs : 57 mins [TIMING] data: 3.9ms | fwd: 3155.4ms | loss: 6.6ms | bwd: 9644.6ms | clip: 0.0ms | optim: 22.2ms step 149 | loss: 2.864918 | lr 1.8000e-04 | norm: 0.3275 | dt: 12905.63ms | tok/sec: 40624.7 | mem: 108493MB | Time Left: 01 hrs : 57 mins [TIMING] data: 3.9ms | fwd: 3216.5ms | loss: 6.6ms | bwd: 9648.1ms | clip: 0.0ms | optim: 22.5ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_150.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_50.ckpt step 150 | loss: 2.556189 | lr 1.8000e-04 | norm: 0.2991 | dt: 12790.72ms | tok/sec: 40989.7 | mem: 108493MB | Time Left: 01 hrs : 56 mins [TIMING] data: 3.7ms | fwd: 3190.2ms | loss: 6.6ms | bwd: 9559.4ms | clip: 0.0ms | optim: 22.5ms step 151 | loss: 3.116900 | lr 1.8000e-04 | norm: 0.3917 | dt: 12770.97ms | tok/sec: 41053.1 | mem: 108493MB | Time Left: 01 hrs : 55 mins [TIMING] data: 3.7ms | fwd: 3142.3ms | loss: 6.6ms | bwd: 9588.0ms | clip: 0.0ms | optim: 22.2ms step 152 | loss: 2.857841 | lr 1.8000e-04 | norm: 0.3148 | dt: 12799.54ms | tok/sec: 40961.5 | mem: 108493MB | Time Left: 01 hrs : 55 mins [TIMING] data: 3.7ms | fwd: 3151.7ms | loss: 6.6ms | bwd: 9607.2ms | clip: 0.0ms | optim: 22.3ms step 153 | loss: 2.722872 | lr 1.8000e-04 | norm: 0.2463 | dt: 12812.25ms | tok/sec: 40920.8 | mem: 108493MB | Time Left: 01 hrs : 55 mins [TIMING] data: 3.7ms | fwd: 3152.1ms | loss: 6.6ms | bwd: 9619.6ms | clip: 0.0ms | optim: 22.3ms step 154 | loss: 3.052043 | lr 1.8000e-04 | norm: 0.3388 | dt: 12840.58ms | tok/sec: 40830.6 | mem: 108493MB | Time Left: 01 hrs : 55 mins [TIMING] data: 3.7ms | fwd: 3174.0ms | loss: 6.6ms | bwd: 9625.8ms | clip: 0.0ms | optim: 22.4ms step 155 | loss: 2.583287 | lr 1.8000e-04 | norm: 0.2301 | dt: 12825.06ms | tok/sec: 40880.0 | mem: 108493MB | Time Left: 01 hrs : 55 mins [TIMING] data: 3.7ms | fwd: 3153.1ms | loss: 6.6ms | bwd: 9631.3ms | clip: 0.0ms | optim: 22.3ms step 156 | loss: 3.212246 | lr 1.8000e-04 | norm: 0.2945 | dt: 12811.07ms | tok/sec: 40924.6 | mem: 108493MB | Time Left: 01 hrs : 55 mins [TIMING] data: 3.7ms | fwd: 3148.9ms | loss: 6.6ms | bwd: 9621.6ms | clip: 0.0ms | optim: 22.3ms step 157 | loss: 2.792709 | lr 1.8000e-04 | norm: 0.3222 | dt: 12823.20ms | tok/sec: 40885.9 | mem: 108493MB | Time Left: 01 hrs : 54 mins [TIMING] data: 3.7ms | fwd: 3158.3ms | loss: 6.6ms | bwd: 9624.3ms | clip: 0.0ms | optim: 22.3ms step 158 | loss: 2.980895 | lr 1.8000e-04 | norm: 0.2192 | dt: 12854.33ms | tok/sec: 40786.9 | mem: 108493MB | Time Left: 01 hrs : 55 mins [TIMING] data: 3.7ms | fwd: 3181.8ms | loss: 6.6ms | bwd: 9632.0ms | clip: 0.0ms | optim: 22.3ms step 159 | loss: 2.988370 | lr 1.8000e-04 | norm: 0.2802 | dt: 12833.68ms | tok/sec: 40852.5 | mem: 108493MB | Time Left: 01 hrs : 54 mins [TIMING] data: 3.7ms | fwd: 3153.5ms | loss: 6.6ms | bwd: 9639.6ms | clip: 0.0ms | optim: 22.3ms step 160 | loss: 2.928037 | lr 1.8000e-04 | norm: 0.3128 | dt: 12829.22ms | tok/sec: 40866.7 | mem: 108493MB | Time Left: 01 hrs : 54 mins [TIMING] data: 3.7ms | fwd: 3153.5ms | loss: 6.6ms | bwd: 9635.1ms | clip: 0.0ms | optim: 22.4ms step 161 | loss: 3.125691 | lr 1.8000e-04 | norm: 0.2259 | dt: 12898.12ms | tok/sec: 40648.4 | mem: 108493MB | Time Left: 01 hrs : 54 mins [TIMING] data: 3.8ms | fwd: 3215.0ms | loss: 6.6ms | bwd: 9642.5ms | clip: 0.0ms | optim: 22.2ms step 162 | loss: 3.167480 | lr 1.8000e-04 | norm: 0.3061 | dt: 12833.67ms | tok/sec: 40852.5 | mem: 108493MB | Time Left: 01 hrs : 54 mins [TIMING] data: 3.7ms | fwd: 3158.5ms | loss: 6.6ms | bwd: 9634.3ms | clip: 0.0ms | optim: 22.5ms step 163 | loss: 3.210653 | lr 1.8000e-04 | norm: 0.3382 | dt: 12840.05ms | tok/sec: 40832.2 | mem: 108493MB | Time Left: 01 hrs : 53 mins [TIMING] data: 3.7ms | fwd: 3154.5ms | loss: 6.6ms | bwd: 9644.8ms | clip: 0.0ms | optim: 22.4ms step 164 | loss: 3.074993 | lr 1.8000e-04 | norm: 0.2255 | dt: 12831.51ms | tok/sec: 40859.4 | mem: 108493MB | Time Left: 01 hrs : 53 mins [TIMING] data: 3.7ms | fwd: 3149.7ms | loss: 6.6ms | bwd: 9641.1ms | clip: 0.0ms | optim: 22.3ms step 165 | loss: 2.443204 | lr 1.8000e-04 | norm: 0.3175 | dt: 12900.87ms | tok/sec: 40639.7 | mem: 108493MB | Time Left: 01 hrs : 53 mins [TIMING] data: 3.7ms | fwd: 3213.5ms | loss: 6.6ms | bwd: 9646.7ms | clip: 0.0ms | optim: 22.3ms step 166 | loss: 2.803377 | lr 1.8000e-04 | norm: 0.3737 | dt: 12837.68ms | tok/sec: 40839.8 | mem: 108493MB | Time Left: 01 hrs : 53 mins [TIMING] data: 3.7ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9640.4ms | clip: 0.0ms | optim: 22.5ms step 167 | loss: 3.000459 | lr 1.8000e-04 | norm: 0.2256 | dt: 12837.19ms | tok/sec: 40841.3 | mem: 108493MB | Time Left: 01 hrs : 52 mins [TIMING] data: 3.7ms | fwd: 3151.8ms | loss: 6.6ms | bwd: 9644.7ms | clip: 0.0ms | optim: 22.4ms step 168 | loss: 3.151734 | lr 1.8000e-04 | norm: 0.4117 | dt: 12827.83ms | tok/sec: 40871.1 | mem: 108493MB | Time Left: 01 hrs : 52 mins [TIMING] data: 3.7ms | fwd: 3152.5ms | loss: 6.6ms | bwd: 9634.5ms | clip: 0.0ms | optim: 22.5ms step 169 | loss: 2.943547 | lr 1.8000e-04 | norm: 0.2082 | dt: 12828.60ms | tok/sec: 40868.7 | mem: 108493MB | Time Left: 01 hrs : 52 mins [TIMING] data: 3.7ms | fwd: 3156.6ms | loss: 6.6ms | bwd: 9631.4ms | clip: 0.0ms | optim: 22.2ms step 170 | loss: 3.110672 | lr 1.8000e-04 | norm: 0.2602 | dt: 12838.11ms | tok/sec: 40838.4 | mem: 108493MB | Time Left: 01 hrs : 52 mins [TIMING] data: 3.8ms | fwd: 3154.9ms | loss: 6.6ms | bwd: 9642.5ms | clip: 0.0ms | optim: 22.3ms step 171 | loss: 2.676930 | lr 1.8000e-04 | norm: 0.2148 | dt: 12892.30ms | tok/sec: 40666.7 | mem: 108493MB | Time Left: 01 hrs : 52 mins [TIMING] data: 3.8ms | fwd: 3198.1ms | loss: 6.6ms | bwd: 9653.4ms | clip: 0.0ms | optim: 22.3ms step 172 | loss: 2.927512 | lr 1.8000e-04 | norm: 0.2422 | dt: 12829.18ms | tok/sec: 40866.8 | mem: 108493MB | Time Left: 01 hrs : 51 mins [TIMING] data: 3.8ms | fwd: 3161.0ms | loss: 6.6ms | bwd: 9627.5ms | clip: 0.0ms | optim: 22.3ms step 173 | loss: 3.197856 | lr 1.8000e-04 | norm: 0.2178 | dt: 12833.45ms | tok/sec: 40853.2 | mem: 108493MB | Time Left: 01 hrs : 51 mins [TIMING] data: 3.7ms | fwd: 3153.4ms | loss: 6.6ms | bwd: 9639.3ms | clip: 0.0ms | optim: 22.4ms step 174 | loss: 3.119886 | lr 1.8000e-04 | norm: 0.2069 | dt: 12840.07ms | tok/sec: 40832.2 | mem: 108493MB | Time Left: 01 hrs : 51 mins [TIMING] data: 3.7ms | fwd: 3151.7ms | loss: 6.6ms | bwd: 9647.7ms | clip: 0.0ms | optim: 22.3ms step 175 | loss: 2.781343 | lr 1.8000e-04 | norm: 0.2845 | dt: 12835.16ms | tok/sec: 40847.8 | mem: 108493MB | Time Left: 01 hrs : 51 mins [TIMING] data: 3.7ms | fwd: 3153.1ms | loss: 6.6ms | bwd: 9641.4ms | clip: 0.0ms | optim: 22.3ms step 176 | loss: 2.989787 | lr 1.8000e-04 | norm: 0.2985 | dt: 12855.57ms | tok/sec: 40782.9 | mem: 108493MB | Time Left: 01 hrs : 51 mins [TIMING] data: 3.7ms | fwd: 3158.1ms | loss: 6.6ms | bwd: 9656.9ms | clip: 0.0ms | optim: 22.2ms step 177 | loss: 3.130120 | lr 1.8000e-04 | norm: 0.2696 | dt: 12838.31ms | tok/sec: 40837.8 | mem: 108493MB | Time Left: 01 hrs : 50 mins [TIMING] data: 3.7ms | fwd: 3157.4ms | loss: 6.6ms | bwd: 9640.3ms | clip: 0.0ms | optim: 22.3ms step 178 | loss: 2.918132 | lr 1.8000e-04 | norm: 0.2372 | dt: 12839.77ms | tok/sec: 40833.1 | mem: 108493MB | Time Left: 01 hrs : 50 mins [TIMING] data: 3.7ms | fwd: 3153.9ms | loss: 6.6ms | bwd: 9644.7ms | clip: 0.0ms | optim: 22.7ms step 179 | loss: 3.139784 | lr 1.8000e-04 | norm: 0.2202 | dt: 12854.83ms | tok/sec: 40785.3 | mem: 108493MB | Time Left: 01 hrs : 50 mins [TIMING] data: 3.7ms | fwd: 3158.1ms | loss: 6.6ms | bwd: 9656.1ms | clip: 0.0ms | optim: 22.3ms step 180 | loss: 3.084623 | lr 1.8000e-04 | norm: 0.2521 | dt: 12828.84ms | tok/sec: 40867.9 | mem: 108493MB | Time Left: 01 hrs : 50 mins [TIMING] data: 3.7ms | fwd: 3153.6ms | loss: 6.6ms | bwd: 9633.9ms | clip: 0.0ms | optim: 22.5ms step 181 | loss: 2.970140 | lr 1.8000e-04 | norm: 0.2281 | dt: 12841.27ms | tok/sec: 40828.4 | mem: 108493MB | Time Left: 01 hrs : 50 mins [TIMING] data: 3.7ms | fwd: 3163.7ms | loss: 6.6ms | bwd: 9636.6ms | clip: 0.0ms | optim: 22.6ms step 182 | loss: 2.869480 | lr 1.8000e-04 | norm: 0.4237 | dt: 12849.91ms | tok/sec: 40800.9 | mem: 108493MB | Time Left: 01 hrs : 49 mins [TIMING] data: 3.7ms | fwd: 3155.7ms | loss: 6.6ms | bwd: 9653.6ms | clip: 0.0ms | optim: 22.3ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b5_65536_131072/train_shard_00001.bin 100000000 step 183 | loss: 2.928205 | lr 1.8000e-04 | norm: 0.2575 | dt: 12914.19ms | tok/sec: 40597.8 | mem: 108493MB | Time Left: 01 hrs : 50 mins [TIMING] data: 15.2ms | fwd: 3216.6ms | loss: 6.6ms | bwd: 9645.3ms | clip: 0.0ms | optim: 22.3ms step 184 | loss: 2.973988 | lr 1.8000e-04 | norm: 0.2689 | dt: 12834.84ms | tok/sec: 40848.8 | mem: 108493MB | Time Left: 01 hrs : 49 mins [TIMING] data: 3.7ms | fwd: 3157.0ms | loss: 6.6ms | bwd: 9637.0ms | clip: 0.0ms | optim: 22.5ms step 185 | loss: 2.965532 | lr 1.8000e-04 | norm: 0.2297 | dt: 12843.69ms | tok/sec: 40820.7 | mem: 108493MB | Time Left: 01 hrs : 49 mins [TIMING] data: 3.8ms | fwd: 3152.9ms | loss: 6.6ms | bwd: 9650.2ms | clip: 0.0ms | optim: 22.2ms step 186 | loss: 2.921149 | lr 1.8000e-04 | norm: 0.3695 | dt: 12835.01ms | tok/sec: 40848.3 | mem: 108493MB | Time Left: 01 hrs : 48 mins [TIMING] data: 3.7ms | fwd: 3153.7ms | loss: 6.6ms | bwd: 9640.5ms | clip: 0.0ms | optim: 22.4ms step 187 | loss: 3.275259 | lr 1.8000e-04 | norm: 0.3722 | dt: 12841.06ms | tok/sec: 40829.0 | mem: 108493MB | Time Left: 01 hrs : 48 mins [TIMING] data: 3.7ms | fwd: 3157.9ms | loss: 6.6ms | bwd: 9642.6ms | clip: 0.0ms | optim: 22.2ms step 188 | loss: 3.130569 | lr 1.8000e-04 | norm: 0.2344 | dt: 12861.74ms | tok/sec: 40763.4 | mem: 108493MB | Time Left: 01 hrs : 48 mins [TIMING] data: 3.8ms | fwd: 3168.2ms | loss: 6.6ms | bwd: 9652.8ms | clip: 0.0ms | optim: 22.4ms step 189 | loss: 3.052735 | lr 1.8000e-04 | norm: 0.2302 | dt: 12842.54ms | tok/sec: 40824.3 | mem: 108493MB | Time Left: 01 hrs : 48 mins [TIMING] data: 3.8ms | fwd: 3155.0ms | loss: 6.6ms | bwd: 9646.7ms | clip: 0.0ms | optim: 22.4ms step 190 | loss: 3.054395 | lr 1.8000e-04 | norm: 0.2370 | dt: 12849.60ms | tok/sec: 40801.9 | mem: 108493MB | Time Left: 01 hrs : 48 mins [TIMING] data: 3.7ms | fwd: 3159.4ms | loss: 6.6ms | bwd: 9649.5ms | clip: 0.0ms | optim: 22.4ms step 191 | loss: 2.875958 | lr 1.8000e-04 | norm: 0.4113 | dt: 12907.76ms | tok/sec: 40618.0 | mem: 108493MB | Time Left: 01 hrs : 48 mins [TIMING] data: 3.8ms | fwd: 3213.5ms | loss: 6.6ms | bwd: 9653.4ms | clip: 0.0ms | optim: 22.5ms step 192 | loss: 3.279243 | lr 1.8000e-04 | norm: 0.2029 | dt: 12893.95ms | tok/sec: 40661.6 | mem: 108493MB | Time Left: 01 hrs : 48 mins [TIMING] data: 3.8ms | fwd: 3202.0ms | loss: 6.6ms | bwd: 9651.2ms | clip: 0.0ms | optim: 22.4ms step 193 | loss: 3.093401 | lr 1.8000e-04 | norm: 0.2114 | dt: 12840.85ms | tok/sec: 40829.7 | mem: 108493MB | Time Left: 01 hrs : 47 mins [TIMING] data: 3.7ms | fwd: 3154.9ms | loss: 6.6ms | bwd: 9645.2ms | clip: 0.0ms | optim: 22.5ms step 194 | loss: 2.686778 | lr 1.8000e-04 | norm: 0.3559 | dt: 12844.64ms | tok/sec: 40817.6 | mem: 108493MB | Time Left: 01 hrs : 47 mins [TIMING] data: 3.7ms | fwd: 3155.9ms | loss: 6.6ms | bwd: 9647.7ms | clip: 0.0ms | optim: 22.5ms step 195 | loss: 2.464939 | lr 1.8000e-04 | norm: 0.1973 | dt: 12897.32ms | tok/sec: 40650.9 | mem: 108493MB | Time Left: 01 hrs : 47 mins [TIMING] data: 3.7ms | fwd: 3214.1ms | loss: 6.6ms | bwd: 9642.4ms | clip: 0.0ms | optim: 22.4ms step 196 | loss: 3.021199 | lr 1.8000e-04 | norm: 0.2468 | dt: 12843.95ms | tok/sec: 40819.9 | mem: 108493MB | Time Left: 01 hrs : 46 mins [TIMING] data: 3.8ms | fwd: 3155.3ms | loss: 6.6ms | bwd: 9647.8ms | clip: 0.0ms | optim: 22.4ms step 197 | loss: 3.136552 | lr 1.8000e-04 | norm: 0.2526 | dt: 12841.89ms | tok/sec: 40826.4 | mem: 108493MB | Time Left: 01 hrs : 46 mins [TIMING] data: 3.8ms | fwd: 3156.2ms | loss: 6.6ms | bwd: 9645.1ms | clip: 0.0ms | optim: 22.2ms step 198 | loss: 2.957233 | lr 1.8000e-04 | norm: 0.2290 | dt: 12835.97ms | tok/sec: 40845.2 | mem: 108493MB | Time Left: 01 hrs : 46 mins [TIMING] data: 3.7ms | fwd: 3152.9ms | loss: 6.6ms | bwd: 9642.3ms | clip: 0.0ms | optim: 22.3ms step 199 | loss: 3.176784 | lr 1.8000e-04 | norm: 0.2605 | dt: 12829.79ms | tok/sec: 40864.9 | mem: 108493MB | Time Left: 01 hrs : 46 mins [TIMING] data: 3.7ms | fwd: 3153.7ms | loss: 6.6ms | bwd: 9635.3ms | clip: 0.0ms | optim: 22.4ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_200.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_100.ckpt step 200 | loss: 2.979025 | lr 1.8000e-04 | norm: 0.2266 | dt: 13022.53ms | tok/sec: 40260.1 | mem: 108493MB | Time Left: 01 hrs : 47 mins [TIMING] data: 4.3ms | fwd: 3420.5ms | loss: 6.5ms | bwd: 9560.5ms | clip: 0.0ms | optim: 22.6ms step 201 | loss: 2.900025 | lr 1.8000e-04 | norm: 0.2591 | dt: 12762.16ms | tok/sec: 41081.4 | mem: 108493MB | Time Left: 01 hrs : 45 mins [TIMING] data: 4.3ms | fwd: 3141.4ms | loss: 6.5ms | bwd: 9579.4ms | clip: 0.0ms | optim: 22.6ms step 202 | loss: 2.769219 | lr 1.8000e-04 | norm: 0.2822 | dt: 12789.23ms | tok/sec: 40994.5 | mem: 108493MB | Time Left: 01 hrs : 45 mins [TIMING] data: 4.4ms | fwd: 3147.0ms | loss: 6.5ms | bwd: 9600.7ms | clip: 0.0ms | optim: 22.5ms step 203 | loss: 2.729409 | lr 1.8000e-04 | norm: 0.2707 | dt: 12806.06ms | tok/sec: 40940.6 | mem: 108493MB | Time Left: 01 hrs : 45 mins [TIMING] data: 4.3ms | fwd: 3146.4ms | loss: 6.5ms | bwd: 9618.4ms | clip: 0.0ms | optim: 22.4ms step 204 | loss: 2.960144 | lr 1.8000e-04 | norm: 0.2511 | dt: 12842.23ms | tok/sec: 40825.3 | mem: 108493MB | Time Left: 01 hrs : 45 mins [TIMING] data: 4.3ms | fwd: 3173.7ms | loss: 6.5ms | bwd: 9626.9ms | clip: 0.0ms | optim: 22.5ms step 205 | loss: 2.734574 | lr 1.8000e-04 | norm: 0.2892 | dt: 12825.22ms | tok/sec: 40879.4 | mem: 108493MB | Time Left: 01 hrs : 44 mins [TIMING] data: 4.3ms | fwd: 3155.8ms | loss: 6.5ms | bwd: 9628.0ms | clip: 0.0ms | optim: 22.7ms step 206 | loss: 2.837455 | lr 1.8000e-04 | norm: 0.2863 | dt: 12835.96ms | tok/sec: 40845.3 | mem: 108493MB | Time Left: 01 hrs : 44 mins [TIMING] data: 4.4ms | fwd: 3154.9ms | loss: 6.5ms | bwd: 9639.4ms | clip: 0.0ms | optim: 22.7ms step 207 | loss: 3.129260 | lr 1.8000e-04 | norm: 0.2327 | dt: 12833.17ms | tok/sec: 40854.1 | mem: 108493MB | Time Left: 01 hrs : 44 mins [TIMING] data: 4.3ms | fwd: 3154.6ms | loss: 6.5ms | bwd: 9637.1ms | clip: 0.0ms | optim: 22.5ms step 208 | loss: 2.619639 | lr 1.8000e-04 | norm: 0.3245 | dt: 12830.05ms | tok/sec: 40864.1 | mem: 108493MB | Time Left: 01 hrs : 44 mins [TIMING] data: 4.3ms | fwd: 3152.5ms | loss: 6.5ms | bwd: 9636.2ms | clip: 0.0ms | optim: 22.5ms step 209 | loss: 3.255617 | lr 1.8000e-04 | norm: 0.3298 | dt: 12840.42ms | tok/sec: 40831.1 | mem: 108493MB | Time Left: 01 hrs : 44 mins [TIMING] data: 4.3ms | fwd: 3155.1ms | loss: 6.5ms | bwd: 9643.8ms | clip: 0.0ms | optim: 22.7ms step 210 | loss: 3.229221 | lr 1.8000e-04 | norm: 0.2216 | dt: 12850.34ms | tok/sec: 40799.5 | mem: 108493MB | Time Left: 01 hrs : 43 mins [TIMING] data: 4.3ms | fwd: 3157.5ms | loss: 6.5ms | bwd: 9651.3ms | clip: 0.0ms | optim: 22.6ms step 211 | loss: 2.983120 | lr 1.8000e-04 | norm: 0.2272 | dt: 12840.68ms | tok/sec: 40830.2 | mem: 108493MB | Time Left: 01 hrs : 43 mins [TIMING] data: 4.3ms | fwd: 3160.6ms | loss: 6.6ms | bwd: 9638.5ms | clip: 0.0ms | optim: 22.5ms step 212 | loss: 2.809622 | lr 1.8000e-04 | norm: 0.2662 | dt: 12836.75ms | tok/sec: 40842.7 | mem: 108493MB | Time Left: 01 hrs : 43 mins [TIMING] data: 4.3ms | fwd: 3156.9ms | loss: 6.6ms | bwd: 9638.3ms | clip: 0.0ms | optim: 22.5ms step 213 | loss: 2.955848 | lr 1.8000e-04 | norm: 0.2602 | dt: 12826.47ms | tok/sec: 40875.5 | mem: 108493MB | Time Left: 01 hrs : 43 mins [TIMING] data: 4.4ms | fwd: 3153.6ms | loss: 6.5ms | bwd: 9631.1ms | clip: 0.0ms | optim: 22.7ms step 214 | loss: 2.873417 | lr 1.8000e-04 | norm: 0.2121 | dt: 12833.44ms | tok/sec: 40853.3 | mem: 108493MB | Time Left: 01 hrs : 42 mins [TIMING] data: 4.3ms | fwd: 3153.6ms | loss: 6.5ms | bwd: 9638.4ms | clip: 0.0ms | optim: 22.5ms step 215 | loss: 3.047452 | lr 1.8000e-04 | norm: 0.3230 | dt: 12835.66ms | tok/sec: 40846.2 | mem: 108493MB | Time Left: 01 hrs : 42 mins [TIMING] data: 4.3ms | fwd: 3154.3ms | loss: 6.6ms | bwd: 9639.8ms | clip: 0.0ms | optim: 22.5ms step 216 | loss: 2.890471 | lr 1.8000e-04 | norm: 0.2704 | dt: 12833.32ms | tok/sec: 40853.7 | mem: 108493MB | Time Left: 01 hrs : 42 mins [TIMING] data: 4.3ms | fwd: 3158.0ms | loss: 6.6ms | bwd: 9633.7ms | clip: 0.0ms | optim: 22.6ms step 217 | loss: 2.811154 | lr 1.8000e-04 | norm: 0.2582 | dt: 12872.83ms | tok/sec: 40728.3 | mem: 108493MB | Time Left: 01 hrs : 42 mins [TIMING] data: 4.4ms | fwd: 3184.6ms | loss: 6.6ms | bwd: 9646.2ms | clip: 0.0ms | optim: 22.8ms step 218 | loss: 2.644052 | lr 1.8000e-04 | norm: 0.2513 | dt: 12850.14ms | tok/sec: 40800.2 | mem: 108493MB | Time Left: 01 hrs : 42 mins [TIMING] data: 4.3ms | fwd: 3156.8ms | loss: 6.6ms | bwd: 9651.9ms | clip: 0.0ms | optim: 22.5ms step 219 | loss: 3.070936 | lr 1.8000e-04 | norm: 0.3002 | dt: 12831.04ms | tok/sec: 40860.9 | mem: 108493MB | Time Left: 01 hrs : 41 mins [TIMING] data: 4.3ms | fwd: 3156.2ms | loss: 6.6ms | bwd: 9633.1ms | clip: 0.0ms | optim: 22.6ms step 220 | loss: 3.019344 | lr 1.8000e-04 | norm: 0.2316 | dt: 12838.84ms | tok/sec: 40836.1 | mem: 108493MB | Time Left: 01 hrs : 41 mins [TIMING] data: 4.3ms | fwd: 3153.7ms | loss: 6.6ms | bwd: 9643.5ms | clip: 0.0ms | optim: 22.6ms step 221 | loss: 3.332009 | lr 1.8000e-04 | norm: 0.2212 | dt: 12833.20ms | tok/sec: 40854.0 | mem: 108493MB | Time Left: 01 hrs : 41 mins [TIMING] data: 4.3ms | fwd: 3158.8ms | loss: 6.6ms | bwd: 9632.9ms | clip: 0.0ms | optim: 22.4ms step 222 | loss: 2.942266 | lr 1.8000e-04 | norm: 0.2633 | dt: 12834.76ms | tok/sec: 40849.1 | mem: 108493MB | Time Left: 01 hrs : 41 mins [TIMING] data: 4.3ms | fwd: 3151.8ms | loss: 6.6ms | bwd: 9641.4ms | clip: 0.0ms | optim: 22.5ms step 223 | loss: 3.051885 | lr 1.8000e-04 | norm: 0.2303 | dt: 12833.23ms | tok/sec: 40853.9 | mem: 108493MB | Time Left: 01 hrs : 40 mins [TIMING] data: 4.4ms | fwd: 3154.3ms | loss: 6.6ms | bwd: 9637.2ms | clip: 0.0ms | optim: 22.6ms step 224 | loss: 3.108602 | lr 1.8000e-04 | norm: 0.2556 | dt: 12840.34ms | tok/sec: 40831.3 | mem: 108493MB | Time Left: 01 hrs : 40 mins [TIMING] data: 4.3ms | fwd: 3151.5ms | loss: 6.6ms | bwd: 9647.4ms | clip: 0.0ms | optim: 22.4ms step 225 | loss: 3.023808 | lr 1.8000e-04 | norm: 0.2679 | dt: 12839.88ms | tok/sec: 40832.8 | mem: 108493MB | Time Left: 01 hrs : 40 mins [TIMING] data: 4.3ms | fwd: 3161.3ms | loss: 6.6ms | bwd: 9637.0ms | clip: 0.0ms | optim: 22.6ms step 226 | loss: 3.166828 | lr 1.8000e-04 | norm: 0.2652 | dt: 12824.14ms | tok/sec: 40882.9 | mem: 108493MB | Time Left: 01 hrs : 40 mins [TIMING] data: 4.3ms | fwd: 3147.5ms | loss: 6.6ms | bwd: 9634.9ms | clip: 0.0ms | optim: 22.7ms step 227 | loss: 3.389651 | lr 1.8000e-04 | norm: 0.2976 | dt: 12833.69ms | tok/sec: 40852.5 | mem: 108493MB | Time Left: 01 hrs : 40 mins [TIMING] data: 4.3ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9637.5ms | clip: 0.0ms | optim: 22.6ms step 228 | loss: 2.822129 | lr 1.8000e-04 | norm: 0.3269 | dt: 12844.77ms | tok/sec: 40817.2 | mem: 108493MB | Time Left: 01 hrs : 39 mins [TIMING] data: 4.3ms | fwd: 3158.4ms | loss: 6.6ms | bwd: 9644.7ms | clip: 0.0ms | optim: 22.8ms step 229 | loss: 2.853920 | lr 1.8000e-04 | norm: 0.2333 | dt: 12846.60ms | tok/sec: 40811.4 | mem: 108493MB | Time Left: 01 hrs : 39 mins [TIMING] data: 4.4ms | fwd: 3160.6ms | loss: 6.5ms | bwd: 9644.4ms | clip: 0.0ms | optim: 22.6ms step 230 | loss: 2.477539 | lr 1.8000e-04 | norm: 0.2457 | dt: 12832.81ms | tok/sec: 40855.3 | mem: 108493MB | Time Left: 01 hrs : 39 mins [TIMING] data: 4.3ms | fwd: 3154.3ms | loss: 6.6ms | bwd: 9636.8ms | clip: 0.0ms | optim: 22.7ms step 231 | loss: 2.947180 | lr 1.8000e-04 | norm: 0.3482 | dt: 12839.01ms | tok/sec: 40835.5 | mem: 108493MB | Time Left: 01 hrs : 39 mins [TIMING] data: 4.3ms | fwd: 3158.0ms | loss: 6.6ms | bwd: 9639.4ms | clip: 0.0ms | optim: 22.6ms step 232 | loss: 2.965376 | lr 1.8000e-04 | norm: 0.4041 | dt: 12832.94ms | tok/sec: 40854.9 | mem: 108493MB | Time Left: 01 hrs : 39 mins [TIMING] data: 4.4ms | fwd: 3152.9ms | loss: 6.6ms | bwd: 9638.4ms | clip: 0.0ms | optim: 22.5ms step 233 | loss: 2.604413 | lr 1.8000e-04 | norm: 0.2729 | dt: 12832.74ms | tok/sec: 40855.5 | mem: 108493MB | Time Left: 01 hrs : 38 mins [TIMING] data: 4.2ms | fwd: 3153.8ms | loss: 6.6ms | bwd: 9637.2ms | clip: 0.0ms | optim: 22.6ms step 234 | loss: 2.780605 | lr 1.8000e-04 | norm: 0.2837 | dt: 12841.55ms | tok/sec: 40827.5 | mem: 108493MB | Time Left: 01 hrs : 38 mins [TIMING] data: 4.3ms | fwd: 3159.7ms | loss: 6.5ms | bwd: 9640.4ms | clip: 0.0ms | optim: 22.6ms step 235 | loss: 3.075331 | lr 1.8000e-04 | norm: 0.2734 | dt: 12837.26ms | tok/sec: 40841.1 | mem: 108493MB | Time Left: 01 hrs : 38 mins [TIMING] data: 4.3ms | fwd: 3157.9ms | loss: 6.6ms | bwd: 9637.9ms | clip: 0.0ms | optim: 22.5ms step 236 | loss: 2.583430 | lr 1.8000e-04 | norm: 0.3037 | dt: 12831.28ms | tok/sec: 40860.1 | mem: 108493MB | Time Left: 01 hrs : 38 mins [TIMING] data: 4.3ms | fwd: 3151.4ms | loss: 6.6ms | bwd: 9638.2ms | clip: 0.0ms | optim: 22.6ms step 237 | loss: 3.224058 | lr 1.8000e-04 | norm: 0.2465 | dt: 12821.63ms | tok/sec: 40890.9 | mem: 108493MB | Time Left: 01 hrs : 37 mins [TIMING] data: 4.3ms | fwd: 3155.1ms | loss: 6.6ms | bwd: 9625.1ms | clip: 0.0ms | optim: 22.4ms step 238 | loss: 2.923151 | lr 1.8000e-04 | norm: 0.3149 | dt: 12844.28ms | tok/sec: 40818.8 | mem: 108493MB | Time Left: 01 hrs : 37 mins [TIMING] data: 4.3ms | fwd: 3156.8ms | loss: 6.6ms | bwd: 9645.9ms | clip: 0.0ms | optim: 22.5ms step 239 | loss: 3.255567 | lr 1.8000e-04 | norm: 0.2373 | dt: 12859.12ms | tok/sec: 40771.7 | mem: 108493MB | Time Left: 01 hrs : 37 mins [TIMING] data: 4.3ms | fwd: 3186.6ms | loss: 6.6ms | bwd: 9630.8ms | clip: 0.0ms | optim: 22.5ms step 240 | loss: 3.125548 | lr 1.8000e-04 | norm: 0.2703 | dt: 12833.40ms | tok/sec: 40853.4 | mem: 108493MB | Time Left: 01 hrs : 37 mins [TIMING] data: 4.4ms | fwd: 3155.1ms | loss: 6.6ms | bwd: 9636.5ms | clip: 0.0ms | optim: 22.6ms step 241 | loss: 2.890630 | lr 1.8000e-04 | norm: 0.3848 | dt: 12831.58ms | tok/sec: 40859.2 | mem: 108493MB | Time Left: 01 hrs : 37 mins [TIMING] data: 4.3ms | fwd: 3157.4ms | loss: 6.6ms | bwd: 9632.5ms | clip: 0.0ms | optim: 22.5ms step 242 | loss: 3.059378 | lr 1.8000e-04 | norm: 0.2390 | dt: 12839.09ms | tok/sec: 40835.3 | mem: 108493MB | Time Left: 01 hrs : 36 mins [TIMING] data: 4.3ms | fwd: 3156.6ms | loss: 6.6ms | bwd: 9640.9ms | clip: 0.0ms | optim: 22.6ms step 243 | loss: 2.862886 | lr 1.8000e-04 | norm: 0.3377 | dt: 12834.90ms | tok/sec: 40848.6 | mem: 108493MB | Time Left: 01 hrs : 36 mins [TIMING] data: 4.3ms | fwd: 3154.7ms | loss: 6.6ms | bwd: 9638.6ms | clip: 0.0ms | optim: 22.5ms step 244 | loss: 2.902782 | lr 1.8000e-04 | norm: 0.2445 | dt: 12901.81ms | tok/sec: 40636.8 | mem: 108493MB | Time Left: 01 hrs : 36 mins [TIMING] data: 4.3ms | fwd: 3221.5ms | loss: 6.6ms | bwd: 9638.8ms | clip: 0.0ms | optim: 22.5ms step 245 | loss: 2.713846 | lr 1.8000e-04 | norm: 0.5346 | dt: 12832.48ms | tok/sec: 40856.3 | mem: 108493MB | Time Left: 01 hrs : 36 mins [TIMING] data: 4.3ms | fwd: 3152.9ms | loss: 6.6ms | bwd: 9638.1ms | clip: 0.0ms | optim: 22.5ms step 246 | loss: 3.172978 | lr 1.8000e-04 | norm: 0.2442 | dt: 12905.38ms | tok/sec: 40625.5 | mem: 108493MB | Time Left: 01 hrs : 36 mins [TIMING] data: 4.3ms | fwd: 3217.8ms | loss: 6.6ms | bwd: 9645.9ms | clip: 0.0ms | optim: 22.6ms step 247 | loss: 2.833372 | lr 1.8000e-04 | norm: 0.3009 | dt: 12851.12ms | tok/sec: 40797.1 | mem: 108493MB | Time Left: 01 hrs : 35 mins [TIMING] data: 4.3ms | fwd: 3153.0ms | loss: 6.6ms | bwd: 9656.5ms | clip: 0.0ms | optim: 22.5ms step 248 | loss: 2.932739 | lr 1.8000e-04 | norm: 0.2928 | dt: 12836.27ms | tok/sec: 40844.3 | mem: 108493MB | Time Left: 01 hrs : 35 mins [TIMING] data: 4.4ms | fwd: 3153.7ms | loss: 6.6ms | bwd: 9640.7ms | clip: 0.0ms | optim: 22.6ms step 249 | loss: 2.983198 | lr 1.8000e-04 | norm: 0.3191 | dt: 12840.05ms | tok/sec: 40832.2 | mem: 108493MB | Time Left: 01 hrs : 35 mins [TIMING] data: 4.4ms | fwd: 3156.2ms | loss: 6.6ms | bwd: 9642.1ms | clip: 0.0ms | optim: 22.7ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_250.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_150.ckpt validation loss: 3.0088 step 250 | loss: 3.107552 | lr 1.8000e-04 | norm: 0.2572 | dt: 23174.89ms | tok/sec: 22623.1 | mem: 108493MB | Time Left: 02 hrs : 51 mins [TIMING] data: 4.4ms | fwd: 3149.1ms | loss: 6.6ms | bwd: 9571.2ms | clip: 0.0ms | optim: 22.5ms step 251 | loss: 2.820528 | lr 1.8000e-04 | norm: 0.2812 | dt: 12772.71ms | tok/sec: 41047.5 | mem: 108493MB | Time Left: 01 hrs : 34 mins [TIMING] data: 4.3ms | fwd: 3140.6ms | loss: 6.6ms | bwd: 9590.4ms | clip: 0.0ms | optim: 22.7ms step 252 | loss: 3.234959 | lr 1.8000e-04 | norm: 0.2519 | dt: 12795.03ms | tok/sec: 40975.9 | mem: 108493MB | Time Left: 01 hrs : 34 mins [TIMING] data: 4.3ms | fwd: 3155.0ms | loss: 6.6ms | bwd: 9598.1ms | clip: 0.0ms | optim: 22.7ms step 253 | loss: 3.131260 | lr 1.8000e-04 | norm: 0.3062 | dt: 12806.70ms | tok/sec: 40938.6 | mem: 108493MB | Time Left: 01 hrs : 34 mins [TIMING] data: 4.3ms | fwd: 3151.8ms | loss: 6.6ms | bwd: 9613.2ms | clip: 0.0ms | optim: 22.6ms step 254 | loss: 2.727864 | lr 1.8000e-04 | norm: 0.2615 | dt: 12811.15ms | tok/sec: 40924.3 | mem: 108493MB | Time Left: 01 hrs : 34 mins [TIMING] data: 4.3ms | fwd: 3150.6ms | loss: 6.6ms | bwd: 9619.0ms | clip: 0.0ms | optim: 22.4ms step 255 | loss: 2.917562 | lr 1.8000e-04 | norm: 0.2574 | dt: 12830.13ms | tok/sec: 40863.8 | mem: 108493MB | Time Left: 01 hrs : 34 mins [TIMING] data: 4.3ms | fwd: 3153.8ms | loss: 6.6ms | bwd: 9634.5ms | clip: 0.0ms | optim: 22.6ms step 256 | loss: 3.143251 | lr 1.8000e-04 | norm: 0.2415 | dt: 12822.27ms | tok/sec: 40888.9 | mem: 108493MB | Time Left: 01 hrs : 33 mins [TIMING] data: 4.3ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9625.9ms | clip: 0.0ms | optim: 22.5ms step 257 | loss: 2.765154 | lr 1.8000e-04 | norm: 0.2715 | dt: 12834.15ms | tok/sec: 40851.0 | mem: 108493MB | Time Left: 01 hrs : 33 mins [TIMING] data: 4.3ms | fwd: 3152.4ms | loss: 6.6ms | bwd: 9640.2ms | clip: 0.0ms | optim: 22.5ms step 258 | loss: 2.941906 | lr 1.8000e-04 | norm: 0.3420 | dt: 12820.79ms | tok/sec: 40893.6 | mem: 108493MB | Time Left: 01 hrs : 33 mins [TIMING] data: 4.4ms | fwd: 3154.1ms | loss: 6.6ms | bwd: 9624.9ms | clip: 0.0ms | optim: 22.5ms step 259 | loss: 2.832259 | lr 1.8000e-04 | norm: 0.2910 | dt: 12822.11ms | tok/sec: 40889.4 | mem: 108493MB | Time Left: 01 hrs : 33 mins [TIMING] data: 4.3ms | fwd: 3153.5ms | loss: 6.6ms | bwd: 9627.0ms | clip: 0.0ms | optim: 22.5ms step 260 | loss: 2.849274 | lr 1.8000e-04 | norm: 0.3670 | dt: 12887.83ms | tok/sec: 40680.9 | mem: 108493MB | Time Left: 01 hrs : 33 mins [TIMING] data: 4.3ms | fwd: 3195.8ms | loss: 6.6ms | bwd: 9650.4ms | clip: 0.0ms | optim: 22.6ms step 261 | loss: 2.745932 | lr 1.8000e-04 | norm: 0.2887 | dt: 12835.50ms | tok/sec: 40846.7 | mem: 108493MB | Time Left: 01 hrs : 32 mins [TIMING] data: 4.3ms | fwd: 3154.1ms | loss: 6.6ms | bwd: 9639.7ms | clip: 0.0ms | optim: 22.7ms step 262 | loss: 3.045666 | lr 1.8000e-04 | norm: 0.2749 | dt: 12837.23ms | tok/sec: 40841.2 | mem: 108493MB | Time Left: 01 hrs : 32 mins [TIMING] data: 4.4ms | fwd: 3153.0ms | loss: 6.6ms | bwd: 9642.0ms | clip: 0.0ms | optim: 23.0ms step 263 | loss: 2.809128 | lr 1.8000e-04 | norm: 0.6620 | dt: 12864.37ms | tok/sec: 40755.0 | mem: 108493MB | Time Left: 01 hrs : 32 mins [TIMING] data: 4.3ms | fwd: 3195.9ms | loss: 6.6ms | bwd: 9626.9ms | clip: 0.0ms | optim: 22.4ms step 264 | loss: 3.241755 | lr 1.8000e-04 | norm: 0.2741 | dt: 12823.37ms | tok/sec: 40885.4 | mem: 108493MB | Time Left: 01 hrs : 32 mins [TIMING] data: 4.4ms | fwd: 3149.0ms | loss: 6.6ms | bwd: 9632.6ms | clip: 0.0ms | optim: 22.4ms step 265 | loss: 3.022745 | lr 1.8000e-04 | norm: 0.2599 | dt: 12831.27ms | tok/sec: 40860.2 | mem: 108493MB | Time Left: 01 hrs : 31 mins [TIMING] data: 4.3ms | fwd: 3157.1ms | loss: 6.6ms | bwd: 9632.5ms | clip: 0.0ms | optim: 22.6ms step 266 | loss: 3.041460 | lr 1.8000e-04 | norm: 0.2750 | dt: 12832.31ms | tok/sec: 40856.9 | mem: 108493MB | Time Left: 01 hrs : 31 mins [TIMING] data: 4.3ms | fwd: 3156.9ms | loss: 6.6ms | bwd: 9633.9ms | clip: 0.0ms | optim: 22.5ms step 267 | loss: 2.954383 | lr 1.8000e-04 | norm: 0.2369 | dt: 12834.42ms | tok/sec: 40850.1 | mem: 108493MB | Time Left: 01 hrs : 31 mins [TIMING] data: 4.3ms | fwd: 3154.7ms | loss: 6.6ms | bwd: 9638.2ms | clip: 0.0ms | optim: 22.5ms step 268 | loss: 2.530771 | lr 1.8000e-04 | norm: 0.2621 | dt: 12826.88ms | tok/sec: 40874.2 | mem: 108493MB | Time Left: 01 hrs : 31 mins [TIMING] data: 4.3ms | fwd: 3155.2ms | loss: 6.6ms | bwd: 9630.1ms | clip: 0.0ms | optim: 22.6ms step 269 | loss: 2.917572 | lr 1.8000e-04 | norm: 0.3910 | dt: 12843.03ms | tok/sec: 40822.8 | mem: 108493MB | Time Left: 01 hrs : 31 mins [TIMING] data: 4.3ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9645.1ms | clip: 0.0ms | optim: 22.5ms step 270 | loss: 3.098329 | lr 1.8000e-04 | norm: 0.2193 | dt: 12837.30ms | tok/sec: 40841.0 | mem: 108493MB | Time Left: 01 hrs : 30 mins [TIMING] data: 4.3ms | fwd: 3153.5ms | loss: 6.6ms | bwd: 9641.6ms | clip: 0.0ms | optim: 22.5ms step 271 | loss: 2.768176 | lr 1.8000e-04 | norm: 0.2396 | dt: 12820.95ms | tok/sec: 40893.1 | mem: 108493MB | Time Left: 01 hrs : 30 mins [TIMING] data: 4.3ms | fwd: 3153.6ms | loss: 6.6ms | bwd: 9625.6ms | clip: 0.0ms | optim: 22.4ms step 272 | loss: 3.324354 | lr 1.8000e-04 | norm: 0.2885 | dt: 12837.43ms | tok/sec: 40840.6 | mem: 108493MB | Time Left: 01 hrs : 30 mins [TIMING] data: 4.3ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9641.2ms | clip: 0.0ms | optim: 22.4ms step 273 | loss: 2.759171 | lr 1.8000e-04 | norm: 0.2524 | dt: 12879.08ms | tok/sec: 40708.5 | mem: 108493MB | Time Left: 01 hrs : 30 mins [TIMING] data: 4.3ms | fwd: 3203.7ms | loss: 6.6ms | bwd: 9633.9ms | clip: 0.0ms | optim: 22.6ms step 274 | loss: 3.118215 | lr 1.8000e-04 | norm: 0.2225 | dt: 12821.67ms | tok/sec: 40890.8 | mem: 108493MB | Time Left: 01 hrs : 29 mins [TIMING] data: 4.3ms | fwd: 3155.9ms | loss: 6.6ms | bwd: 9624.2ms | clip: 0.0ms | optim: 22.4ms step 275 | loss: 3.024614 | lr 1.8000e-04 | norm: 0.2873 | dt: 12810.96ms | tok/sec: 40925.0 | mem: 108493MB | Time Left: 01 hrs : 29 mins [TIMING] data: 4.3ms | fwd: 3147.6ms | loss: 6.6ms | bwd: 9621.8ms | clip: 0.0ms | optim: 22.5ms step 276 | loss: 2.833103 | lr 1.8000e-04 | norm: 0.3349 | dt: 12838.02ms | tok/sec: 40838.7 | mem: 108493MB | Time Left: 01 hrs : 29 mins [TIMING] data: 4.3ms | fwd: 3157.0ms | loss: 6.6ms | bwd: 9639.2ms | clip: 0.0ms | optim: 22.6ms step 277 | loss: 3.184470 | lr 1.8000e-04 | norm: 0.2515 | dt: 12824.80ms | tok/sec: 40880.8 | mem: 108493MB | Time Left: 01 hrs : 29 mins [TIMING] data: 4.4ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9630.8ms | clip: 0.0ms | optim: 22.4ms step 278 | loss: 3.151454 | lr 1.8000e-04 | norm: 0.2657 | dt: 12842.92ms | tok/sec: 40823.1 | mem: 108493MB | Time Left: 01 hrs : 29 mins [TIMING] data: 4.3ms | fwd: 3159.5ms | loss: 6.6ms | bwd: 9641.9ms | clip: 0.0ms | optim: 22.5ms step 279 | loss: 3.182845 | lr 1.8000e-04 | norm: 0.2684 | dt: 12833.27ms | tok/sec: 40853.8 | mem: 108493MB | Time Left: 01 hrs : 28 mins [TIMING] data: 4.3ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9636.9ms | clip: 0.0ms | optim: 22.5ms step 280 | loss: 2.869285 | lr 1.8000e-04 | norm: 0.3202 | dt: 12836.32ms | tok/sec: 40844.1 | mem: 108493MB | Time Left: 01 hrs : 28 mins [TIMING] data: 4.3ms | fwd: 3152.7ms | loss: 6.6ms | bwd: 9642.0ms | clip: 0.0ms | optim: 22.7ms step 281 | loss: 2.848287 | lr 1.8000e-04 | norm: 0.2171 | dt: 12828.52ms | tok/sec: 40868.9 | mem: 108493MB | Time Left: 01 hrs : 28 mins [TIMING] data: 4.4ms | fwd: 3157.1ms | loss: 6.6ms | bwd: 9629.6ms | clip: 0.0ms | optim: 22.6ms step 282 | loss: 3.079267 | lr 1.8000e-04 | norm: 0.1889 | dt: 12826.55ms | tok/sec: 40875.2 | mem: 108493MB | Time Left: 01 hrs : 28 mins [TIMING] data: 4.4ms | fwd: 3156.0ms | loss: 6.6ms | bwd: 9628.9ms | clip: 0.0ms | optim: 22.5ms step 283 | loss: 3.007225 | lr 1.8000e-04 | norm: 0.3008 | dt: 12844.70ms | tok/sec: 40817.5 | mem: 108493MB | Time Left: 01 hrs : 28 mins [TIMING] data: 4.3ms | fwd: 3158.1ms | loss: 6.6ms | bwd: 9644.8ms | clip: 0.0ms | optim: 22.6ms step 284 | loss: 2.667108 | lr 1.8000e-04 | norm: 0.2450 | dt: 12831.61ms | tok/sec: 40859.1 | mem: 108493MB | Time Left: 01 hrs : 27 mins [TIMING] data: 4.3ms | fwd: 3156.1ms | loss: 6.6ms | bwd: 9633.9ms | clip: 0.0ms | optim: 22.5ms step 285 | loss: 2.799646 | lr 1.8000e-04 | norm: 0.3099 | dt: 12835.11ms | tok/sec: 40848.0 | mem: 108493MB | Time Left: 01 hrs : 27 mins [TIMING] data: 4.4ms | fwd: 3151.2ms | loss: 6.6ms | bwd: 9642.2ms | clip: 0.0ms | optim: 22.5ms step 286 | loss: 2.754669 | lr 1.8000e-04 | norm: 0.2983 | dt: 12826.87ms | tok/sec: 40874.2 | mem: 108493MB | Time Left: 01 hrs : 27 mins [TIMING] data: 4.4ms | fwd: 3150.5ms | loss: 6.6ms | bwd: 9634.7ms | clip: 0.0ms | optim: 22.4ms step 287 | loss: 2.945090 | lr 1.8000e-04 | norm: 0.2115 | dt: 12828.59ms | tok/sec: 40868.7 | mem: 108493MB | Time Left: 01 hrs : 27 mins [TIMING] data: 4.3ms | fwd: 3154.9ms | loss: 6.6ms | bwd: 9632.1ms | clip: 0.0ms | optim: 22.4ms step 288 | loss: 3.121429 | lr 1.8000e-04 | norm: 0.2138 | dt: 12839.49ms | tok/sec: 40834.0 | mem: 108493MB | Time Left: 01 hrs : 27 mins [TIMING] data: 4.3ms | fwd: 3157.6ms | loss: 6.6ms | bwd: 9640.4ms | clip: 0.0ms | optim: 22.5ms step 289 | loss: 2.808765 | lr 1.8000e-04 | norm: 0.2282 | dt: 12839.95ms | tok/sec: 40832.6 | mem: 108493MB | Time Left: 01 hrs : 26 mins [TIMING] data: 4.3ms | fwd: 3151.8ms | loss: 6.6ms | bwd: 9646.7ms | clip: 0.0ms | optim: 22.5ms step 290 | loss: 2.765847 | lr 1.8000e-04 | norm: 0.2624 | dt: 12828.57ms | tok/sec: 40868.8 | mem: 108493MB | Time Left: 01 hrs : 26 mins [TIMING] data: 4.3ms | fwd: 3156.3ms | loss: 6.6ms | bwd: 9630.8ms | clip: 0.0ms | optim: 22.5ms step 291 | loss: 2.921622 | lr 1.8000e-04 | norm: 0.3013 | dt: 12835.32ms | tok/sec: 40847.3 | mem: 108493MB | Time Left: 01 hrs : 26 mins [TIMING] data: 4.3ms | fwd: 3155.7ms | loss: 6.6ms | bwd: 9638.2ms | clip: 0.0ms | optim: 22.5ms step 292 | loss: 3.177122 | lr 1.8000e-04 | norm: 0.2674 | dt: 12879.12ms | tok/sec: 40708.4 | mem: 108493MB | Time Left: 01 hrs : 26 mins [TIMING] data: 4.3ms | fwd: 3207.6ms | loss: 6.6ms | bwd: 9629.9ms | clip: 0.0ms | optim: 22.6ms step 293 | loss: 3.141580 | lr 1.8000e-04 | norm: 0.2342 | dt: 12827.78ms | tok/sec: 40871.3 | mem: 108493MB | Time Left: 01 hrs : 25 mins [TIMING] data: 4.3ms | fwd: 3150.7ms | loss: 6.6ms | bwd: 9635.3ms | clip: 0.0ms | optim: 22.7ms step 294 | loss: 2.790162 | lr 1.8000e-04 | norm: 0.2867 | dt: 12892.64ms | tok/sec: 40665.7 | mem: 108493MB | Time Left: 01 hrs : 26 mins [TIMING] data: 4.3ms | fwd: 3209.5ms | loss: 6.6ms | bwd: 9641.6ms | clip: 0.0ms | optim: 22.5ms step 295 | loss: 2.861574 | lr 1.8000e-04 | norm: 0.2589 | dt: 12833.70ms | tok/sec: 40852.4 | mem: 108493MB | Time Left: 01 hrs : 25 mins [TIMING] data: 4.3ms | fwd: 3149.2ms | loss: 6.6ms | bwd: 9643.1ms | clip: 0.0ms | optim: 22.4ms step 296 | loss: 2.600466 | lr 1.8000e-04 | norm: 0.2694 | dt: 12857.03ms | tok/sec: 40778.3 | mem: 108493MB | Time Left: 01 hrs : 25 mins [TIMING] data: 4.3ms | fwd: 3159.7ms | loss: 6.6ms | bwd: 9655.7ms | clip: 0.0ms | optim: 22.6ms step 297 | loss: 2.895202 | lr 1.8000e-04 | norm: 0.2633 | dt: 12835.80ms | tok/sec: 40845.8 | mem: 108493MB | Time Left: 01 hrs : 25 mins [TIMING] data: 4.3ms | fwd: 3157.0ms | loss: 6.6ms | bwd: 9637.3ms | clip: 0.0ms | optim: 22.5ms step 298 | loss: 2.968583 | lr 1.8000e-04 | norm: 0.3245 | dt: 12845.47ms | tok/sec: 40815.0 | mem: 108493MB | Time Left: 01 hrs : 24 mins [TIMING] data: 4.4ms | fwd: 3156.1ms | loss: 6.6ms | bwd: 9647.7ms | clip: 0.0ms | optim: 22.5ms step 299 | loss: 3.039231 | lr 1.8000e-04 | norm: 0.2856 | dt: 12837.39ms | tok/sec: 40840.7 | mem: 108493MB | Time Left: 01 hrs : 24 mins [TIMING] data: 4.3ms | fwd: 3159.5ms | loss: 6.6ms | bwd: 9636.3ms | clip: 0.0ms | optim: 22.4ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_300.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_200.ckpt step 300 | loss: 2.760157 | lr 1.8000e-04 | norm: 0.2046 | dt: 12762.33ms | tok/sec: 41080.9 | mem: 108493MB | Time Left: 01 hrs : 24 mins [TIMING] data: 4.4ms | fwd: 3175.5ms | loss: 6.6ms | bwd: 9545.3ms | clip: 0.0ms | optim: 22.3ms step 301 | loss: 2.904699 | lr 1.8000e-04 | norm: 0.2314 | dt: 12774.45ms | tok/sec: 41041.9 | mem: 108493MB | Time Left: 01 hrs : 23 mins [TIMING] data: 4.4ms | fwd: 3143.6ms | loss: 6.6ms | bwd: 9589.7ms | clip: 0.0ms | optim: 22.2ms step 302 | loss: 2.975716 | lr 1.8000e-04 | norm: 0.2280 | dt: 12789.54ms | tok/sec: 40993.5 | mem: 108493MB | Time Left: 01 hrs : 23 mins [TIMING] data: 4.4ms | fwd: 3149.5ms | loss: 6.6ms | bwd: 9598.7ms | clip: 0.0ms | optim: 22.3ms step 303 | loss: 3.147175 | lr 1.8000e-04 | norm: 0.2003 | dt: 12814.15ms | tok/sec: 40914.8 | mem: 108493MB | Time Left: 01 hrs : 23 mins [TIMING] data: 4.3ms | fwd: 3152.4ms | loss: 6.6ms | bwd: 9620.4ms | clip: 0.0ms | optim: 22.4ms step 304 | loss: 2.575998 | lr 1.8000e-04 | norm: 0.2181 | dt: 12806.86ms | tok/sec: 40938.1 | mem: 108493MB | Time Left: 01 hrs : 23 mins [TIMING] data: 4.4ms | fwd: 3148.5ms | loss: 6.6ms | bwd: 9617.0ms | clip: 0.0ms | optim: 22.4ms step 305 | loss: 2.906385 | lr 1.8000e-04 | norm: 0.2431 | dt: 12826.90ms | tok/sec: 40874.1 | mem: 108493MB | Time Left: 01 hrs : 23 mins [TIMING] data: 4.3ms | fwd: 3152.7ms | loss: 6.6ms | bwd: 9632.8ms | clip: 0.0ms | optim: 22.5ms step 306 | loss: 3.146288 | lr 1.8000e-04 | norm: 0.2146 | dt: 12828.18ms | tok/sec: 40870.0 | mem: 108493MB | Time Left: 01 hrs : 23 mins [TIMING] data: 4.3ms | fwd: 3153.0ms | loss: 6.6ms | bwd: 9633.8ms | clip: 0.0ms | optim: 22.4ms step 307 | loss: 2.971466 | lr 1.8000e-04 | norm: 0.2682 | dt: 12880.33ms | tok/sec: 40704.5 | mem: 108493MB | Time Left: 01 hrs : 23 mins [TIMING] data: 4.4ms | fwd: 3217.2ms | loss: 6.6ms | bwd: 9621.7ms | clip: 0.0ms | optim: 22.3ms step 308 | loss: 2.985111 | lr 1.8000e-04 | norm: 0.2863 | dt: 12828.90ms | tok/sec: 40867.7 | mem: 108493MB | Time Left: 01 hrs : 22 mins [TIMING] data: 4.4ms | fwd: 3158.9ms | loss: 6.6ms | bwd: 9628.6ms | clip: 0.0ms | optim: 22.3ms step 309 | loss: 3.122300 | lr 1.8000e-04 | norm: 0.2144 | dt: 12818.19ms | tok/sec: 40901.9 | mem: 108493MB | Time Left: 01 hrs : 22 mins [TIMING] data: 4.4ms | fwd: 3151.2ms | loss: 6.6ms | bwd: 9625.8ms | clip: 0.0ms | optim: 22.2ms step 310 | loss: 3.086428 | lr 1.8000e-04 | norm: 0.2480 | dt: 12823.41ms | tok/sec: 40885.2 | mem: 108493MB | Time Left: 01 hrs : 22 mins [TIMING] data: 4.3ms | fwd: 3149.0ms | loss: 6.6ms | bwd: 9633.0ms | clip: 0.0ms | optim: 22.4ms step 311 | loss: 3.165472 | lr 1.8000e-04 | norm: 0.2564 | dt: 12813.17ms | tok/sec: 40917.9 | mem: 108493MB | Time Left: 01 hrs : 22 mins [TIMING] data: 4.4ms | fwd: 3152.8ms | loss: 6.5ms | bwd: 9619.2ms | clip: 0.0ms | optim: 22.3ms step 312 | loss: 2.858697 | lr 1.8000e-04 | norm: 0.2078 | dt: 12822.74ms | tok/sec: 40887.3 | mem: 108493MB | Time Left: 01 hrs : 21 mins [TIMING] data: 4.4ms | fwd: 3147.8ms | loss: 6.6ms | bwd: 9633.6ms | clip: 0.0ms | optim: 22.4ms step 313 | loss: 3.140396 | lr 1.8000e-04 | norm: 0.2701 | dt: 12840.47ms | tok/sec: 40830.9 | mem: 108493MB | Time Left: 01 hrs : 21 mins [TIMING] data: 4.3ms | fwd: 3157.4ms | loss: 6.5ms | bwd: 9642.0ms | clip: 0.0ms | optim: 22.2ms step 314 | loss: 3.110787 | lr 1.8000e-04 | norm: 0.1967 | dt: 12825.87ms | tok/sec: 40877.4 | mem: 108493MB | Time Left: 01 hrs : 21 mins [TIMING] data: 4.4ms | fwd: 3152.9ms | loss: 6.5ms | bwd: 9631.9ms | clip: 0.0ms | optim: 22.2ms step 315 | loss: 2.968996 | lr 1.8000e-04 | norm: 0.2097 | dt: 12825.77ms | tok/sec: 40877.7 | mem: 108493MB | Time Left: 01 hrs : 21 mins [TIMING] data: 4.4ms | fwd: 3155.4ms | loss: 6.6ms | bwd: 9629.0ms | clip: 0.0ms | optim: 22.5ms step 316 | loss: 2.889165 | lr 1.8000e-04 | norm: 0.2447 | dt: 12834.87ms | tok/sec: 40848.7 | mem: 108493MB | Time Left: 01 hrs : 21 mins [TIMING] data: 4.3ms | fwd: 3150.0ms | loss: 6.6ms | bwd: 9643.6ms | clip: 0.0ms | optim: 22.3ms step 317 | loss: 3.285981 | lr 1.8000e-04 | norm: 0.2696 | dt: 12819.50ms | tok/sec: 40897.7 | mem: 108493MB | Time Left: 01 hrs : 20 mins [TIMING] data: 4.3ms | fwd: 3147.7ms | loss: 6.6ms | bwd: 9630.6ms | clip: 0.0ms | optim: 22.3ms step 318 | loss: 2.932148 | lr 1.8000e-04 | norm: 0.2321 | dt: 12826.08ms | tok/sec: 40876.7 | mem: 108493MB | Time Left: 01 hrs : 20 mins [TIMING] data: 4.4ms | fwd: 3147.8ms | loss: 6.6ms | bwd: 9637.0ms | clip: 0.0ms | optim: 22.3ms step 319 | loss: 3.188653 | lr 1.8000e-04 | norm: 0.2591 | dt: 12832.14ms | tok/sec: 40857.4 | mem: 108493MB | Time Left: 01 hrs : 20 mins [TIMING] data: 4.4ms | fwd: 3154.9ms | loss: 6.6ms | bwd: 9635.9ms | clip: 0.0ms | optim: 22.4ms step 320 | loss: 3.008488 | lr 1.8000e-04 | norm: 0.4271 | dt: 12845.30ms | tok/sec: 40815.6 | mem: 108493MB | Time Left: 01 hrs : 20 mins [TIMING] data: 4.4ms | fwd: 3167.1ms | loss: 6.6ms | bwd: 9636.9ms | clip: 0.0ms | optim: 22.5ms step 321 | loss: 2.896602 | lr 1.8000e-04 | norm: 0.2016 | dt: 12872.13ms | tok/sec: 40730.5 | mem: 108493MB | Time Left: 01 hrs : 20 mins [TIMING] data: 4.4ms | fwd: 3191.8ms | loss: 6.6ms | bwd: 9638.9ms | clip: 0.0ms | optim: 22.3ms step 322 | loss: 3.189522 | lr 1.8000e-04 | norm: 0.2572 | dt: 12835.17ms | tok/sec: 40847.8 | mem: 108493MB | Time Left: 01 hrs : 19 mins [TIMING] data: 4.4ms | fwd: 3155.9ms | loss: 6.6ms | bwd: 9638.0ms | clip: 0.0ms | optim: 22.3ms step 323 | loss: 3.149186 | lr 1.8000e-04 | norm: 0.2511 | dt: 12824.17ms | tok/sec: 40882.8 | mem: 108493MB | Time Left: 01 hrs : 19 mins [TIMING] data: 4.4ms | fwd: 3156.9ms | loss: 6.6ms | bwd: 9625.9ms | clip: 0.0ms | optim: 22.3ms step 324 | loss: 2.954212 | lr 1.8000e-04 | norm: 0.1997 | dt: 12827.67ms | tok/sec: 40871.6 | mem: 108493MB | Time Left: 01 hrs : 19 mins [TIMING] data: 4.3ms | fwd: 3149.1ms | loss: 6.6ms | bwd: 9637.3ms | clip: 0.0ms | optim: 22.2ms step 325 | loss: 3.018676 | lr 1.8000e-04 | norm: 0.1991 | dt: 12820.33ms | tok/sec: 40895.1 | mem: 108493MB | Time Left: 01 hrs : 19 mins [TIMING] data: 4.4ms | fwd: 3148.8ms | loss: 6.6ms | bwd: 9630.1ms | clip: 0.0ms | optim: 22.4ms step 326 | loss: 3.157910 | lr 1.8000e-04 | norm: 0.2711 | dt: 12828.24ms | tok/sec: 40869.8 | mem: 108493MB | Time Left: 01 hrs : 18 mins [TIMING] data: 4.4ms | fwd: 3155.2ms | loss: 6.6ms | bwd: 9631.5ms | clip: 0.0ms | optim: 22.5ms step 327 | loss: 2.704699 | lr 1.8000e-04 | norm: 0.2725 | dt: 12836.85ms | tok/sec: 40842.4 | mem: 108493MB | Time Left: 01 hrs : 18 mins [TIMING] data: 4.4ms | fwd: 3154.0ms | loss: 6.6ms | bwd: 9641.5ms | clip: 0.0ms | optim: 22.4ms step 328 | loss: 3.078629 | lr 1.8000e-04 | norm: 0.4254 | dt: 12832.93ms | tok/sec: 40854.9 | mem: 108493MB | Time Left: 01 hrs : 18 mins [TIMING] data: 4.4ms | fwd: 3153.2ms | loss: 6.6ms | bwd: 9638.4ms | clip: 0.0ms | optim: 22.3ms step 329 | loss: 2.932401 | lr 1.8000e-04 | norm: 0.2388 | dt: 12839.39ms | tok/sec: 40834.3 | mem: 108493MB | Time Left: 01 hrs : 18 mins [TIMING] data: 4.4ms | fwd: 3156.3ms | loss: 6.6ms | bwd: 9641.6ms | clip: 0.0ms | optim: 22.4ms step 330 | loss: 3.151811 | lr 1.8000e-04 | norm: 0.2396 | dt: 12829.38ms | tok/sec: 40866.2 | mem: 108493MB | Time Left: 01 hrs : 18 mins [TIMING] data: 4.4ms | fwd: 3149.0ms | loss: 6.6ms | bwd: 9639.1ms | clip: 0.0ms | optim: 22.2ms step 331 | loss: 2.906637 | lr 1.8000e-04 | norm: 0.2447 | dt: 12832.44ms | tok/sec: 40856.4 | mem: 108493MB | Time Left: 01 hrs : 17 mins [TIMING] data: 4.4ms | fwd: 3155.6ms | loss: 6.6ms | bwd: 9635.5ms | clip: 0.0ms | optim: 22.3ms step 332 | loss: 3.002137 | lr 1.8000e-04 | norm: 0.2772 | dt: 12832.50ms | tok/sec: 40856.2 | mem: 108493MB | Time Left: 01 hrs : 17 mins [TIMING] data: 4.4ms | fwd: 3158.2ms | loss: 6.6ms | bwd: 9632.6ms | clip: 0.0ms | optim: 22.5ms step 333 | loss: 3.146642 | lr 1.8000e-04 | norm: 0.2335 | dt: 12825.28ms | tok/sec: 40879.3 | mem: 108493MB | Time Left: 01 hrs : 17 mins [TIMING] data: 4.4ms | fwd: 3156.0ms | loss: 6.6ms | bwd: 9628.0ms | clip: 0.0ms | optim: 22.3ms step 334 | loss: 3.052756 | lr 1.8000e-04 | norm: 0.2570 | dt: 12832.99ms | tok/sec: 40854.7 | mem: 108493MB | Time Left: 01 hrs : 17 mins [TIMING] data: 4.4ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9639.1ms | clip: 0.0ms | optim: 22.4ms step 335 | loss: 2.930621 | lr 1.8000e-04 | norm: 0.2335 | dt: 12834.63ms | tok/sec: 40849.5 | mem: 108493MB | Time Left: 01 hrs : 17 mins [TIMING] data: 4.3ms | fwd: 3153.7ms | loss: 6.6ms | bwd: 9639.6ms | clip: 0.0ms | optim: 22.4ms step 336 | loss: 2.958940 | lr 1.8000e-04 | norm: 0.1865 | dt: 12833.19ms | tok/sec: 40854.1 | mem: 108493MB | Time Left: 01 hrs : 16 mins [TIMING] data: 4.4ms | fwd: 3152.1ms | loss: 6.6ms | bwd: 9639.8ms | clip: 0.0ms | optim: 22.3ms step 337 | loss: 3.039662 | lr 1.8000e-04 | norm: 0.2222 | dt: 12820.93ms | tok/sec: 40893.1 | mem: 108493MB | Time Left: 01 hrs : 16 mins [TIMING] data: 4.4ms | fwd: 3149.5ms | loss: 6.6ms | bwd: 9629.9ms | clip: 0.0ms | optim: 22.4ms step 338 | loss: 2.788351 | lr 1.8000e-04 | norm: 0.3067 | dt: 12874.74ms | tok/sec: 40722.2 | mem: 108493MB | Time Left: 01 hrs : 16 mins [TIMING] data: 4.3ms | fwd: 3190.3ms | loss: 6.6ms | bwd: 9643.2ms | clip: 0.0ms | optim: 22.3ms step 339 | loss: 2.622558 | lr 1.8000e-04 | norm: 0.2439 | dt: 12822.99ms | tok/sec: 40886.6 | mem: 108493MB | Time Left: 01 hrs : 16 mins [TIMING] data: 4.4ms | fwd: 3152.5ms | loss: 6.6ms | bwd: 9629.2ms | clip: 0.0ms | optim: 22.2ms step 340 | loss: 2.987433 | lr 1.8000e-04 | norm: 0.2398 | dt: 12845.51ms | tok/sec: 40814.9 | mem: 108493MB | Time Left: 01 hrs : 16 mins [TIMING] data: 4.3ms | fwd: 3158.1ms | loss: 6.6ms | bwd: 9646.1ms | clip: 0.0ms | optim: 22.4ms step 341 | loss: 3.092014 | lr 1.8000e-04 | norm: 0.2938 | dt: 12837.20ms | tok/sec: 40841.3 | mem: 108493MB | Time Left: 01 hrs : 15 mins [TIMING] data: 4.4ms | fwd: 3156.8ms | loss: 6.6ms | bwd: 9638.9ms | clip: 0.0ms | optim: 22.5ms step 342 | loss: 3.138996 | lr 1.8000e-04 | norm: 0.2327 | dt: 12826.80ms | tok/sec: 40874.4 | mem: 108493MB | Time Left: 01 hrs : 15 mins [TIMING] data: 4.4ms | fwd: 3152.0ms | loss: 6.6ms | bwd: 9633.2ms | clip: 0.0ms | optim: 22.5ms step 343 | loss: 2.971185 | lr 1.8000e-04 | norm: 0.2086 | dt: 12845.02ms | tok/sec: 40816.4 | mem: 108493MB | Time Left: 01 hrs : 15 mins [TIMING] data: 4.5ms | fwd: 3157.9ms | loss: 6.6ms | bwd: 9645.7ms | clip: 0.0ms | optim: 22.3ms step 344 | loss: 3.165244 | lr 1.8000e-04 | norm: 0.2241 | dt: 12829.52ms | tok/sec: 40865.8 | mem: 108493MB | Time Left: 01 hrs : 15 mins [TIMING] data: 4.4ms | fwd: 3150.6ms | loss: 6.6ms | bwd: 9637.5ms | clip: 0.0ms | optim: 22.3ms step 345 | loss: 2.993647 | lr 1.8000e-04 | norm: 0.2048 | dt: 12843.64ms | tok/sec: 40820.8 | mem: 108493MB | Time Left: 01 hrs : 14 mins [TIMING] data: 4.4ms | fwd: 3158.4ms | loss: 6.6ms | bwd: 9643.8ms | clip: 0.0ms | optim: 22.4ms step 346 | loss: 3.285748 | lr 1.8000e-04 | norm: 0.2666 | dt: 12834.66ms | tok/sec: 40849.4 | mem: 108493MB | Time Left: 01 hrs : 14 mins [TIMING] data: 4.4ms | fwd: 3154.0ms | loss: 6.6ms | bwd: 9639.1ms | clip: 0.0ms | optim: 22.4ms step 347 | loss: 3.158352 | lr 1.8000e-04 | norm: 0.2245 | dt: 12839.19ms | tok/sec: 40835.0 | mem: 108493MB | Time Left: 01 hrs : 14 mins [TIMING] data: 4.4ms | fwd: 3154.3ms | loss: 6.6ms | bwd: 9643.3ms | clip: 0.0ms | optim: 22.6ms step 348 | loss: 3.159408 | lr 1.8000e-04 | norm: 1.6247 | dt: 12852.65ms | tok/sec: 40792.2 | mem: 108493MB | Time Left: 01 hrs : 14 mins [TIMING] data: 4.4ms | fwd: 3158.2ms | loss: 6.6ms | bwd: 9653.1ms | clip: 0.0ms | optim: 22.3ms step 349 | loss: 3.190736 | lr 1.8000e-04 | norm: 0.2766 | dt: 12836.65ms | tok/sec: 40843.0 | mem: 108493MB | Time Left: 01 hrs : 14 mins [TIMING] data: 4.4ms | fwd: 3158.4ms | loss: 6.6ms | bwd: 9637.0ms | clip: 0.0ms | optim: 22.3ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_350.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_250.ckpt step 350 | loss: 2.983176 | lr 1.8000e-04 | norm: 0.2060 | dt: 12722.38ms | tok/sec: 41209.9 | mem: 108493MB | Time Left: 01 hrs : 13 mins [TIMING] data: 4.5ms | fwd: 3137.7ms | loss: 6.6ms | bwd: 9543.0ms | clip: 0.0ms | optim: 22.4ms step 351 | loss: 3.016051 | lr 1.8000e-04 | norm: 0.2377 | dt: 12793.70ms | tok/sec: 40980.2 | mem: 108493MB | Time Left: 01 hrs : 13 mins [TIMING] data: 4.3ms | fwd: 3163.9ms | loss: 6.5ms | bwd: 9588.7ms | clip: 0.0ms | optim: 22.3ms step 352 | loss: 3.188255 | lr 1.8000e-04 | norm: 0.2125 | dt: 12778.26ms | tok/sec: 41029.7 | mem: 108493MB | Time Left: 01 hrs : 13 mins [TIMING] data: 4.3ms | fwd: 3147.9ms | loss: 6.6ms | bwd: 9589.2ms | clip: 0.0ms | optim: 22.3ms step 353 | loss: 3.045575 | lr 1.8000e-04 | norm: 0.1942 | dt: 12800.74ms | tok/sec: 40957.6 | mem: 108493MB | Time Left: 01 hrs : 12 mins [TIMING] data: 4.3ms | fwd: 3147.5ms | loss: 6.5ms | bwd: 9612.3ms | clip: 0.0ms | optim: 22.3ms step 354 | loss: 2.844591 | lr 1.8000e-04 | norm: 0.2343 | dt: 12857.47ms | tok/sec: 40776.9 | mem: 108493MB | Time Left: 01 hrs : 13 mins [TIMING] data: 4.5ms | fwd: 3198.6ms | loss: 6.5ms | bwd: 9617.7ms | clip: 0.0ms | optim: 22.2ms step 355 | loss: 3.083425 | lr 1.8000e-04 | norm: 0.2249 | dt: 12824.20ms | tok/sec: 40882.7 | mem: 108493MB | Time Left: 01 hrs : 12 mins [TIMING] data: 4.4ms | fwd: 3151.0ms | loss: 6.6ms | bwd: 9631.5ms | clip: 0.0ms | optim: 22.6ms step 356 | loss: 2.968467 | lr 1.8000e-04 | norm: 0.2915 | dt: 12824.08ms | tok/sec: 40883.1 | mem: 108493MB | Time Left: 01 hrs : 12 mins [TIMING] data: 4.3ms | fwd: 3148.5ms | loss: 6.5ms | bwd: 9634.6ms | clip: 0.0ms | optim: 22.2ms step 357 | loss: 2.656154 | lr 1.8000e-04 | norm: 0.5843 | dt: 12815.75ms | tok/sec: 40909.6 | mem: 108493MB | Time Left: 01 hrs : 12 mins [TIMING] data: 4.4ms | fwd: 3146.1ms | loss: 6.5ms | bwd: 9628.6ms | clip: 0.0ms | optim: 22.2ms step 358 | loss: 2.559948 | lr 1.8000e-04 | norm: 0.2721 | dt: 12822.07ms | tok/sec: 40889.5 | mem: 108493MB | Time Left: 01 hrs : 12 mins [TIMING] data: 4.4ms | fwd: 3153.3ms | loss: 6.5ms | bwd: 9627.6ms | clip: 0.0ms | optim: 22.3ms step 359 | loss: 2.930913 | lr 1.8000e-04 | norm: 0.2777 | dt: 12830.90ms | tok/sec: 40861.4 | mem: 108493MB | Time Left: 01 hrs : 11 mins [TIMING] data: 4.4ms | fwd: 3151.6ms | loss: 6.5ms | bwd: 9638.2ms | clip: 0.0ms | optim: 22.3ms step 360 | loss: 2.995180 | lr 1.8000e-04 | norm: 0.2555 | dt: 12825.54ms | tok/sec: 40878.4 | mem: 108493MB | Time Left: 01 hrs : 11 mins [TIMING] data: 4.4ms | fwd: 3145.1ms | loss: 6.5ms | bwd: 9639.3ms | clip: 0.0ms | optim: 22.3ms step 361 | loss: 3.116910 | lr 1.8000e-04 | norm: 0.2704 | dt: 12824.53ms | tok/sec: 40881.6 | mem: 108493MB | Time Left: 01 hrs : 11 mins [TIMING] data: 4.4ms | fwd: 3152.1ms | loss: 6.6ms | bwd: 9631.2ms | clip: 0.0ms | optim: 22.4ms step 362 | loss: 2.631602 | lr 1.8000e-04 | norm: 0.2479 | dt: 12830.84ms | tok/sec: 40861.5 | mem: 108493MB | Time Left: 01 hrs : 11 mins [TIMING] data: 4.3ms | fwd: 3151.2ms | loss: 6.5ms | bwd: 9638.3ms | clip: 0.0ms | optim: 22.5ms step 363 | loss: 3.235361 | lr 1.8000e-04 | norm: 0.2329 | dt: 12828.56ms | tok/sec: 40868.8 | mem: 108493MB | Time Left: 01 hrs : 10 mins [TIMING] data: 4.3ms | fwd: 3152.6ms | loss: 6.5ms | bwd: 9634.8ms | clip: 0.0ms | optim: 22.4ms step 364 | loss: 2.705560 | lr 1.8000e-04 | norm: 0.2398 | dt: 12842.77ms | tok/sec: 40823.6 | mem: 108493MB | Time Left: 01 hrs : 10 mins [TIMING] data: 4.3ms | fwd: 3154.5ms | loss: 6.5ms | bwd: 9647.1ms | clip: 0.0ms | optim: 22.4ms step 365 | loss: 2.910263 | lr 1.8000e-04 | norm: 0.2483 | dt: 12830.55ms | tok/sec: 40862.5 | mem: 108493MB | Time Left: 01 hrs : 10 mins [TIMING] data: 4.3ms | fwd: 3151.2ms | loss: 6.5ms | bwd: 9638.4ms | clip: 0.0ms | optim: 22.3ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b5_65536_131072/train_shard_00002.bin 100000000 step 366 | loss: 3.005711 | lr 1.8000e-04 | norm: 0.2788 | dt: 12881.58ms | tok/sec: 40700.6 | mem: 108493MB | Time Left: 01 hrs : 10 mins [TIMING] data: 9.0ms | fwd: 3196.7ms | loss: 6.5ms | bwd: 9639.1ms | clip: 0.0ms | optim: 22.3ms step 367 | loss: 3.079184 | lr 1.8000e-04 | norm: 0.2483 | dt: 12905.64ms | tok/sec: 40624.7 | mem: 108493MB | Time Left: 01 hrs : 10 mins [TIMING] data: 4.3ms | fwd: 3219.4ms | loss: 6.5ms | bwd: 9644.9ms | clip: 0.0ms | optim: 22.5ms step 368 | loss: 3.324134 | lr 1.8000e-04 | norm: 0.2532 | dt: 12836.45ms | tok/sec: 40843.7 | mem: 108493MB | Time Left: 01 hrs : 09 mins [TIMING] data: 4.3ms | fwd: 3154.9ms | loss: 6.5ms | bwd: 9640.3ms | clip: 0.0ms | optim: 22.4ms step 369 | loss: 3.307187 | lr 1.8000e-04 | norm: 0.3266 | dt: 12827.05ms | tok/sec: 40873.6 | mem: 108493MB | Time Left: 01 hrs : 09 mins [TIMING] data: 4.3ms | fwd: 3151.6ms | loss: 6.5ms | bwd: 9634.4ms | clip: 0.0ms | optim: 22.3ms step 370 | loss: 2.936333 | lr 1.8000e-04 | norm: 0.2327 | dt: 12836.27ms | tok/sec: 40844.3 | mem: 108493MB | Time Left: 01 hrs : 09 mins [TIMING] data: 4.4ms | fwd: 3150.9ms | loss: 6.5ms | bwd: 9644.2ms | clip: 0.0ms | optim: 22.3ms step 371 | loss: 2.637288 | lr 1.8000e-04 | norm: 0.2640 | dt: 12832.80ms | tok/sec: 40855.3 | mem: 108493MB | Time Left: 01 hrs : 09 mins [TIMING] data: 4.4ms | fwd: 3152.1ms | loss: 6.5ms | bwd: 9639.1ms | clip: 0.0ms | optim: 22.6ms step 372 | loss: 2.574068 | lr 1.8000e-04 | norm: 0.2538 | dt: 12835.52ms | tok/sec: 40846.6 | mem: 108493MB | Time Left: 01 hrs : 09 mins [TIMING] data: 4.4ms | fwd: 3153.1ms | loss: 6.5ms | bwd: 9641.4ms | clip: 0.0ms | optim: 22.3ms step 373 | loss: 3.058856 | lr 1.8000e-04 | norm: 0.2829 | dt: 12843.78ms | tok/sec: 40820.4 | mem: 108493MB | Time Left: 01 hrs : 08 mins [TIMING] data: 4.3ms | fwd: 3157.6ms | loss: 6.5ms | bwd: 9644.9ms | clip: 0.0ms | optim: 22.5ms step 374 | loss: 2.826590 | lr 1.8000e-04 | norm: 0.2753 | dt: 12863.39ms | tok/sec: 40758.1 | mem: 108493MB | Time Left: 01 hrs : 08 mins [TIMING] data: 4.4ms | fwd: 3177.4ms | loss: 6.5ms | bwd: 9644.7ms | clip: 0.0ms | optim: 22.4ms step 375 | loss: 2.763584 | lr 1.8000e-04 | norm: 0.2261 | dt: 12832.87ms | tok/sec: 40855.1 | mem: 108493MB | Time Left: 01 hrs : 08 mins [TIMING] data: 4.4ms | fwd: 3152.9ms | loss: 6.5ms | bwd: 9638.7ms | clip: 0.0ms | optim: 22.4ms step 376 | loss: 2.988871 | lr 1.8000e-04 | norm: 0.2078 | dt: 12840.25ms | tok/sec: 40831.6 | mem: 108493MB | Time Left: 01 hrs : 08 mins [TIMING] data: 4.4ms | fwd: 3153.8ms | loss: 6.5ms | bwd: 9645.2ms | clip: 0.0ms | optim: 22.4ms step 377 | loss: 3.028848 | lr 1.8000e-04 | norm: 0.2476 | dt: 12893.87ms | tok/sec: 40661.8 | mem: 108493MB | Time Left: 01 hrs : 08 mins [TIMING] data: 4.4ms | fwd: 3219.2ms | loss: 6.5ms | bwd: 9633.3ms | clip: 0.0ms | optim: 22.4ms step 378 | loss: 3.006437 | lr 1.8000e-04 | norm: 0.3196 | dt: 12853.78ms | tok/sec: 40788.6 | mem: 108493MB | Time Left: 01 hrs : 07 mins [TIMING] data: 4.4ms | fwd: 3161.3ms | loss: 6.6ms | bwd: 9651.2ms | clip: 0.0ms | optim: 22.5ms step 379 | loss: 3.077123 | lr 1.8000e-04 | norm: 0.5137 | dt: 12837.52ms | tok/sec: 40840.3 | mem: 108493MB | Time Left: 01 hrs : 07 mins [TIMING] data: 4.4ms | fwd: 3155.3ms | loss: 6.5ms | bwd: 9641.1ms | clip: 0.0ms | optim: 22.2ms step 380 | loss: 3.020652 | lr 1.8000e-04 | norm: 0.2343 | dt: 12840.48ms | tok/sec: 40830.9 | mem: 108493MB | Time Left: 01 hrs : 07 mins [TIMING] data: 4.4ms | fwd: 3153.4ms | loss: 6.5ms | bwd: 9645.7ms | clip: 0.0ms | optim: 22.4ms step 381 | loss: 2.856817 | lr 1.8000e-04 | norm: 0.3365 | dt: 12836.79ms | tok/sec: 40842.6 | mem: 108493MB | Time Left: 01 hrs : 07 mins [TIMING] data: 4.3ms | fwd: 3155.2ms | loss: 6.5ms | bwd: 9640.5ms | clip: 0.0ms | optim: 22.3ms step 382 | loss: 3.079731 | lr 1.8000e-04 | norm: 0.2188 | dt: 12833.69ms | tok/sec: 40852.5 | mem: 108493MB | Time Left: 01 hrs : 06 mins [TIMING] data: 4.4ms | fwd: 3152.2ms | loss: 6.5ms | bwd: 9640.3ms | clip: 0.0ms | optim: 22.4ms step 383 | loss: 3.070147 | lr 1.8000e-04 | norm: 0.2345 | dt: 12847.45ms | tok/sec: 40808.7 | mem: 108493MB | Time Left: 01 hrs : 06 mins [TIMING] data: 4.3ms | fwd: 3158.1ms | loss: 6.5ms | bwd: 9648.3ms | clip: 0.0ms | optim: 22.3ms step 384 | loss: 3.026050 | lr 1.8000e-04 | norm: 0.2531 | dt: 12831.68ms | tok/sec: 40858.9 | mem: 108493MB | Time Left: 01 hrs : 06 mins [TIMING] data: 4.4ms | fwd: 3154.3ms | loss: 6.5ms | bwd: 9636.0ms | clip: 0.0ms | optim: 22.5ms step 385 | loss: 2.841217 | lr 1.8000e-04 | norm: 0.4300 | dt: 12837.97ms | tok/sec: 40838.8 | mem: 108493MB | Time Left: 01 hrs : 06 mins [TIMING] data: 4.4ms | fwd: 3156.3ms | loss: 6.6ms | bwd: 9640.5ms | clip: 0.0ms | optim: 22.3ms step 386 | loss: 2.987332 | lr 1.8000e-04 | norm: 0.2024 | dt: 12852.93ms | tok/sec: 40791.3 | mem: 108493MB | Time Left: 01 hrs : 06 mins [TIMING] data: 4.4ms | fwd: 3151.1ms | loss: 6.5ms | bwd: 9660.8ms | clip: 0.0ms | optim: 22.3ms step 387 | loss: 3.138704 | lr 1.8000e-04 | norm: 0.2608 | dt: 12833.55ms | tok/sec: 40852.9 | mem: 108493MB | Time Left: 01 hrs : 05 mins [TIMING] data: 4.3ms | fwd: 3154.8ms | loss: 6.5ms | bwd: 9637.5ms | clip: 0.0ms | optim: 22.5ms step 388 | loss: 2.846701 | lr 1.8000e-04 | norm: 0.2347 | dt: 12885.66ms | tok/sec: 40687.7 | mem: 108493MB | Time Left: 01 hrs : 05 mins [TIMING] data: 4.4ms | fwd: 3191.4ms | loss: 6.5ms | bwd: 9653.1ms | clip: 0.0ms | optim: 22.3ms step 389 | loss: 3.131402 | lr 1.8000e-04 | norm: 0.2060 | dt: 12840.17ms | tok/sec: 40831.9 | mem: 108493MB | Time Left: 01 hrs : 05 mins [TIMING] data: 4.4ms | fwd: 3155.6ms | loss: 6.6ms | bwd: 9643.1ms | clip: 0.0ms | optim: 22.4ms step 390 | loss: 2.926301 | lr 1.8000e-04 | norm: 0.2965 | dt: 12859.70ms | tok/sec: 40769.8 | mem: 108493MB | Time Left: 01 hrs : 05 mins [TIMING] data: 4.4ms | fwd: 3165.4ms | loss: 6.6ms | bwd: 9653.0ms | clip: 0.0ms | optim: 22.4ms step 391 | loss: 2.978170 | lr 1.8000e-04 | norm: 0.2238 | dt: 12838.73ms | tok/sec: 40836.4 | mem: 108493MB | Time Left: 01 hrs : 05 mins [TIMING] data: 4.4ms | fwd: 3151.8ms | loss: 6.6ms | bwd: 9645.6ms | clip: 0.0ms | optim: 22.3ms step 392 | loss: 2.775031 | lr 1.8000e-04 | norm: 0.2429 | dt: 12846.40ms | tok/sec: 40812.1 | mem: 108493MB | Time Left: 01 hrs : 04 mins [TIMING] data: 4.4ms | fwd: 3157.0ms | loss: 6.6ms | bwd: 9648.0ms | clip: 0.0ms | optim: 22.5ms step 393 | loss: 3.049076 | lr 1.8000e-04 | norm: 0.2437 | dt: 12852.33ms | tok/sec: 40793.2 | mem: 108493MB | Time Left: 01 hrs : 04 mins [TIMING] data: 4.4ms | fwd: 3155.6ms | loss: 6.5ms | bwd: 9655.6ms | clip: 0.0ms | optim: 22.3ms step 394 | loss: 3.268913 | lr 1.8000e-04 | norm: 0.2347 | dt: 12850.61ms | tok/sec: 40798.7 | mem: 108493MB | Time Left: 01 hrs : 04 mins [TIMING] data: 4.4ms | fwd: 3158.8ms | loss: 6.6ms | bwd: 9650.5ms | clip: 0.0ms | optim: 22.4ms step 395 | loss: 2.742187 | lr 1.8000e-04 | norm: 0.2195 | dt: 12879.19ms | tok/sec: 40708.1 | mem: 108493MB | Time Left: 01 hrs : 04 mins [TIMING] data: 4.4ms | fwd: 3185.9ms | loss: 6.5ms | bwd: 9652.2ms | clip: 0.0ms | optim: 22.3ms step 396 | loss: 3.056838 | lr 1.8000e-04 | norm: 0.2322 | dt: 12840.52ms | tok/sec: 40830.7 | mem: 108493MB | Time Left: 01 hrs : 03 mins [TIMING] data: 4.4ms | fwd: 3155.9ms | loss: 6.6ms | bwd: 9643.5ms | clip: 0.0ms | optim: 22.3ms step 397 | loss: 3.309800 | lr 1.8000e-04 | norm: 0.2168 | dt: 12841.13ms | tok/sec: 40828.8 | mem: 108493MB | Time Left: 01 hrs : 03 mins [TIMING] data: 4.3ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9645.2ms | clip: 0.0ms | optim: 22.3ms step 398 | loss: 2.789086 | lr 1.8000e-04 | norm: 0.2098 | dt: 12833.53ms | tok/sec: 40853.0 | mem: 108493MB | Time Left: 01 hrs : 03 mins [TIMING] data: 4.4ms | fwd: 3151.2ms | loss: 6.6ms | bwd: 9641.0ms | clip: 0.0ms | optim: 22.4ms step 399 | loss: 3.111372 | lr 1.8000e-04 | norm: 0.1778 | dt: 12892.18ms | tok/sec: 40667.1 | mem: 108493MB | Time Left: 01 hrs : 03 mins [TIMING] data: 4.4ms | fwd: 3209.8ms | loss: 6.5ms | bwd: 9641.2ms | clip: 0.0ms | optim: 22.3ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_400.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_300.ckpt step 400 | loss: 2.905966 | lr 1.8000e-04 | norm: 0.3060 | dt: 13003.61ms | tok/sec: 40318.6 | mem: 108493MB | Time Left: 01 hrs : 03 mins [TIMING] data: 4.6ms | fwd: 3405.3ms | loss: 6.6ms | bwd: 9556.5ms | clip: 0.0ms | optim: 22.3ms step 401 | loss: 2.941042 | lr 1.8000e-04 | norm: 0.2152 | dt: 12779.13ms | tok/sec: 41026.9 | mem: 108493MB | Time Left: 01 hrs : 02 mins [TIMING] data: 4.5ms | fwd: 3144.3ms | loss: 6.6ms | bwd: 9593.4ms | clip: 0.0ms | optim: 22.3ms step 402 | loss: 3.325269 | lr 1.8000e-04 | norm: 0.2346 | dt: 12800.40ms | tok/sec: 40958.7 | mem: 108493MB | Time Left: 01 hrs : 02 mins [TIMING] data: 4.6ms | fwd: 3144.0ms | loss: 6.6ms | bwd: 9614.8ms | clip: 0.0ms | optim: 22.3ms step 403 | loss: 3.108928 | lr 1.8000e-04 | norm: 0.1988 | dt: 12818.12ms | tok/sec: 40902.1 | mem: 108493MB | Time Left: 01 hrs : 02 mins [TIMING] data: 4.6ms | fwd: 3150.4ms | loss: 6.6ms | bwd: 9626.1ms | clip: 0.0ms | optim: 22.3ms step 404 | loss: 2.864557 | lr 1.8000e-04 | norm: 0.2082 | dt: 12813.34ms | tok/sec: 40917.4 | mem: 108493MB | Time Left: 01 hrs : 02 mins [TIMING] data: 4.6ms | fwd: 3150.7ms | loss: 6.6ms | bwd: 9620.8ms | clip: 0.0ms | optim: 22.6ms step 405 | loss: 2.912993 | lr 1.8000e-04 | norm: 0.2214 | dt: 12818.43ms | tok/sec: 40901.1 | mem: 108493MB | Time Left: 01 hrs : 01 mins [TIMING] data: 4.5ms | fwd: 3152.4ms | loss: 6.6ms | bwd: 9624.6ms | clip: 0.0ms | optim: 22.3ms step 406 | loss: 3.033185 | lr 1.8000e-04 | norm: 0.2296 | dt: 12821.96ms | tok/sec: 40889.9 | mem: 108493MB | Time Left: 01 hrs : 01 mins [TIMING] data: 4.6ms | fwd: 3149.4ms | loss: 6.6ms | bwd: 9631.1ms | clip: 0.0ms | optim: 22.3ms step 407 | loss: 3.261375 | lr 1.8000e-04 | norm: 0.2424 | dt: 12827.93ms | tok/sec: 40870.8 | mem: 108493MB | Time Left: 01 hrs : 01 mins [TIMING] data: 4.5ms | fwd: 3156.5ms | loss: 6.6ms | bwd: 9629.9ms | clip: 0.0ms | optim: 22.4ms step 408 | loss: 2.997370 | lr 1.8000e-04 | norm: 0.2667 | dt: 12824.20ms | tok/sec: 40882.7 | mem: 108493MB | Time Left: 01 hrs : 01 mins [TIMING] data: 4.6ms | fwd: 3150.9ms | loss: 6.6ms | bwd: 9631.8ms | clip: 0.0ms | optim: 22.3ms step 409 | loss: 2.875690 | lr 1.8000e-04 | norm: 0.2545 | dt: 12874.32ms | tok/sec: 40723.6 | mem: 108493MB | Time Left: 01 hrs : 01 mins [TIMING] data: 4.5ms | fwd: 3179.3ms | loss: 6.6ms | bwd: 9653.5ms | clip: 0.0ms | optim: 22.3ms step 410 | loss: 3.041541 | lr 1.8000e-04 | norm: 0.2311 | dt: 12838.92ms | tok/sec: 40835.8 | mem: 108493MB | Time Left: 01 hrs : 00 mins [TIMING] data: 4.5ms | fwd: 3159.1ms | loss: 6.6ms | bwd: 9638.1ms | clip: 0.0ms | optim: 22.5ms step 411 | loss: 2.997994 | lr 1.8000e-04 | norm: 0.2480 | dt: 12833.61ms | tok/sec: 40852.7 | mem: 108493MB | Time Left: 01 hrs : 00 mins [TIMING] data: 4.6ms | fwd: 3158.3ms | loss: 6.6ms | bwd: 9633.5ms | clip: 0.0ms | optim: 22.6ms step 412 | loss: 3.080277 | lr 1.8000e-04 | norm: 0.2214 | dt: 12847.42ms | tok/sec: 40808.8 | mem: 108493MB | Time Left: 01 hrs : 00 mins [TIMING] data: 4.6ms | fwd: 3156.0ms | loss: 6.6ms | bwd: 9649.9ms | clip: 0.0ms | optim: 22.3ms step 413 | loss: 2.809402 | lr 1.8000e-04 | norm: 0.2841 | dt: 12833.77ms | tok/sec: 40852.2 | mem: 108493MB | Time Left: 01 hrs : 00 mins [TIMING] data: 4.6ms | fwd: 3156.8ms | loss: 6.6ms | bwd: 9635.4ms | clip: 0.0ms | optim: 22.3ms step 414 | loss: 2.429967 | lr 1.8000e-04 | norm: 0.2571 | dt: 12839.10ms | tok/sec: 40835.2 | mem: 108493MB | Time Left: 01 hrs : 00 mins [TIMING] data: 4.7ms | fwd: 3157.1ms | loss: 6.6ms | bwd: 9640.3ms | clip: 0.0ms | optim: 22.3ms step 415 | loss: 2.893539 | lr 1.8000e-04 | norm: 0.2479 | dt: 12827.96ms | tok/sec: 40870.7 | mem: 108493MB | Time Left: 00 hrs : 59 mins [TIMING] data: 4.6ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9631.7ms | clip: 0.0ms | optim: 22.4ms step 416 | loss: 3.108972 | lr 1.8000e-04 | norm: 0.2362 | dt: 12844.28ms | tok/sec: 40818.8 | mem: 108493MB | Time Left: 00 hrs : 59 mins [TIMING] data: 4.6ms | fwd: 3154.2ms | loss: 6.6ms | bwd: 9648.6ms | clip: 0.0ms | optim: 22.3ms step 417 | loss: 2.566674 | lr 1.8000e-04 | norm: 0.2872 | dt: 12905.62ms | tok/sec: 40624.8 | mem: 108493MB | Time Left: 00 hrs : 59 mins [TIMING] data: 4.5ms | fwd: 3220.2ms | loss: 6.6ms | bwd: 9644.0ms | clip: 0.0ms | optim: 22.3ms step 418 | loss: 3.153097 | lr 1.8000e-04 | norm: 0.4049 | dt: 12838.44ms | tok/sec: 40837.4 | mem: 108493MB | Time Left: 00 hrs : 59 mins [TIMING] data: 4.6ms | fwd: 3155.3ms | loss: 6.6ms | bwd: 9641.4ms | clip: 0.0ms | optim: 22.5ms step 419 | loss: 3.017240 | lr 1.8000e-04 | norm: 0.2931 | dt: 12843.82ms | tok/sec: 40820.3 | mem: 108493MB | Time Left: 00 hrs : 59 mins [TIMING] data: 4.5ms | fwd: 3153.1ms | loss: 6.6ms | bwd: 9649.1ms | clip: 0.0ms | optim: 22.4ms step 420 | loss: 2.891714 | lr 1.8000e-04 | norm: 0.2820 | dt: 12841.90ms | tok/sec: 40826.4 | mem: 108493MB | Time Left: 00 hrs : 58 mins [TIMING] data: 4.5ms | fwd: 3155.4ms | loss: 6.6ms | bwd: 9644.8ms | clip: 0.0ms | optim: 22.5ms step 421 | loss: 3.057061 | lr 1.8000e-04 | norm: 0.3559 | dt: 12845.79ms | tok/sec: 40814.0 | mem: 108493MB | Time Left: 00 hrs : 58 mins [TIMING] data: 4.5ms | fwd: 3156.6ms | loss: 6.6ms | bwd: 9647.5ms | clip: 0.0ms | optim: 22.5ms step 422 | loss: 3.103817 | lr 1.8000e-04 | norm: 0.2581 | dt: 12840.17ms | tok/sec: 40831.9 | mem: 108493MB | Time Left: 00 hrs : 58 mins [TIMING] data: 4.5ms | fwd: 3155.5ms | loss: 6.6ms | bwd: 9643.1ms | clip: 0.0ms | optim: 22.4ms step 423 | loss: 2.725772 | lr 1.8000e-04 | norm: 0.2872 | dt: 12842.16ms | tok/sec: 40825.5 | mem: 108493MB | Time Left: 00 hrs : 58 mins [TIMING] data: 4.5ms | fwd: 3156.0ms | loss: 6.6ms | bwd: 9644.6ms | clip: 0.0ms | optim: 22.3ms step 424 | loss: 2.911441 | lr 1.8000e-04 | norm: 0.2470 | dt: 12832.22ms | tok/sec: 40857.2 | mem: 108493MB | Time Left: 00 hrs : 57 mins [TIMING] data: 4.6ms | fwd: 3157.4ms | loss: 6.6ms | bwd: 9633.1ms | clip: 0.0ms | optim: 22.5ms step 425 | loss: 3.222884 | lr 1.8000e-04 | norm: 0.3443 | dt: 12853.11ms | tok/sec: 40790.7 | mem: 108493MB | Time Left: 00 hrs : 57 mins [TIMING] data: 4.6ms | fwd: 3159.7ms | loss: 6.6ms | bwd: 9651.7ms | clip: 0.0ms | optim: 22.5ms step 426 | loss: 3.265219 | lr 1.8000e-04 | norm: 0.2645 | dt: 12846.94ms | tok/sec: 40810.3 | mem: 108493MB | Time Left: 00 hrs : 57 mins [TIMING] data: 4.6ms | fwd: 3156.8ms | loss: 6.6ms | bwd: 9648.4ms | clip: 0.0ms | optim: 22.5ms step 427 | loss: 2.924688 | lr 1.8000e-04 | norm: 0.2055 | dt: 12844.23ms | tok/sec: 40819.0 | mem: 108493MB | Time Left: 00 hrs : 57 mins [TIMING] data: 4.5ms | fwd: 3156.3ms | loss: 6.6ms | bwd: 9646.3ms | clip: 0.0ms | optim: 22.5ms step 428 | loss: 2.907848 | lr 1.8000e-04 | norm: 0.3003 | dt: 12835.60ms | tok/sec: 40846.4 | mem: 108493MB | Time Left: 00 hrs : 57 mins [TIMING] data: 4.6ms | fwd: 3153.1ms | loss: 6.6ms | bwd: 9640.9ms | clip: 0.0ms | optim: 22.3ms step 429 | loss: 3.006146 | lr 1.8000e-04 | norm: 0.2851 | dt: 12849.11ms | tok/sec: 40803.5 | mem: 108493MB | Time Left: 00 hrs : 56 mins [TIMING] data: 4.6ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9652.9ms | clip: 0.0ms | optim: 22.3ms step 430 | loss: 2.738014 | lr 1.8000e-04 | norm: 0.3395 | dt: 12834.78ms | tok/sec: 40849.0 | mem: 108493MB | Time Left: 00 hrs : 56 mins [TIMING] data: 4.7ms | fwd: 3156.5ms | loss: 6.6ms | bwd: 9636.6ms | clip: 0.0ms | optim: 22.4ms step 431 | loss: 3.160033 | lr 1.8000e-04 | norm: 0.3595 | dt: 12885.24ms | tok/sec: 40689.0 | mem: 108493MB | Time Left: 00 hrs : 56 mins [TIMING] data: 4.5ms | fwd: 3207.2ms | loss: 6.6ms | bwd: 9636.6ms | clip: 0.0ms | optim: 22.3ms step 432 | loss: 3.221043 | lr 1.8000e-04 | norm: 0.2816 | dt: 12844.10ms | tok/sec: 40819.4 | mem: 108493MB | Time Left: 00 hrs : 56 mins [TIMING] data: 4.6ms | fwd: 3156.0ms | loss: 6.6ms | bwd: 9646.6ms | clip: 0.0ms | optim: 22.4ms step 433 | loss: 3.010752 | lr 1.8000e-04 | norm: 0.2637 | dt: 12828.31ms | tok/sec: 40869.6 | mem: 108493MB | Time Left: 00 hrs : 56 mins [TIMING] data: 4.6ms | fwd: 3152.2ms | loss: 6.6ms | bwd: 9632.9ms | clip: 0.0ms | optim: 24.0ms step 434 | loss: 2.978500 | lr 1.8000e-04 | norm: 0.2349 | dt: 12847.26ms | tok/sec: 40809.3 | mem: 108493MB | Time Left: 00 hrs : 55 mins [TIMING] data: 4.6ms | fwd: 3157.3ms | loss: 6.6ms | bwd: 9648.4ms | clip: 0.0ms | optim: 22.3ms step 435 | loss: 3.065373 | lr 1.8000e-04 | norm: 0.3006 | dt: 12847.33ms | tok/sec: 40809.1 | mem: 108493MB | Time Left: 00 hrs : 55 mins [TIMING] data: 4.6ms | fwd: 3154.1ms | loss: 6.6ms | bwd: 9651.7ms | clip: 0.0ms | optim: 22.3ms step 436 | loss: 3.033729 | lr 1.8000e-04 | norm: 0.2306 | dt: 12850.12ms | tok/sec: 40800.2 | mem: 108493MB | Time Left: 00 hrs : 55 mins [TIMING] data: 4.6ms | fwd: 3155.1ms | loss: 6.6ms | bwd: 9653.4ms | clip: 0.0ms | optim: 22.3ms step 437 | loss: 2.670005 | lr 1.8000e-04 | norm: 0.2367 | dt: 12852.29ms | tok/sec: 40793.3 | mem: 108493MB | Time Left: 00 hrs : 55 mins [TIMING] data: 4.6ms | fwd: 3160.7ms | loss: 6.6ms | bwd: 9650.1ms | clip: 0.0ms | optim: 22.3ms step 438 | loss: 2.821919 | lr 1.8000e-04 | norm: 0.2676 | dt: 12851.60ms | tok/sec: 40795.5 | mem: 108493MB | Time Left: 00 hrs : 55 mins [TIMING] data: 4.6ms | fwd: 3155.0ms | loss: 6.6ms | bwd: 9654.9ms | clip: 0.0ms | optim: 22.4ms step 439 | loss: 3.185823 | lr 1.8000e-04 | norm: 0.2303 | dt: 12827.51ms | tok/sec: 40872.1 | mem: 108493MB | Time Left: 00 hrs : 54 mins [TIMING] data: 4.6ms | fwd: 3151.4ms | loss: 6.6ms | bwd: 9634.6ms | clip: 0.0ms | optim: 22.3ms step 440 | loss: 3.072515 | lr 1.8000e-04 | norm: 0.2184 | dt: 12837.50ms | tok/sec: 40840.3 | mem: 108493MB | Time Left: 00 hrs : 54 mins [TIMING] data: 4.6ms | fwd: 3157.3ms | loss: 6.6ms | bwd: 9638.6ms | clip: 0.0ms | optim: 22.3ms step 441 | loss: 2.665057 | lr 1.8000e-04 | norm: 0.2006 | dt: 12826.04ms | tok/sec: 40876.9 | mem: 108493MB | Time Left: 00 hrs : 54 mins [TIMING] data: 4.5ms | fwd: 3152.1ms | loss: 6.6ms | bwd: 9632.3ms | clip: 0.0ms | optim: 22.5ms step 442 | loss: 2.852482 | lr 1.8000e-04 | norm: 0.2565 | dt: 12902.31ms | tok/sec: 40635.2 | mem: 108493MB | Time Left: 00 hrs : 54 mins [TIMING] data: 4.6ms | fwd: 3215.0ms | loss: 6.6ms | bwd: 9645.7ms | clip: 0.0ms | optim: 22.4ms step 443 | loss: 2.969991 | lr 1.8000e-04 | norm: 0.2471 | dt: 12887.00ms | tok/sec: 40683.5 | mem: 108493MB | Time Left: 00 hrs : 54 mins [TIMING] data: 4.6ms | fwd: 3204.3ms | loss: 6.6ms | bwd: 9641.3ms | clip: 0.0ms | optim: 22.3ms step 444 | loss: 3.093454 | lr 1.8000e-04 | norm: 0.2599 | dt: 12872.08ms | tok/sec: 40730.7 | mem: 108493MB | Time Left: 00 hrs : 53 mins [TIMING] data: 4.6ms | fwd: 3183.2ms | loss: 6.6ms | bwd: 9647.1ms | clip: 0.0ms | optim: 22.5ms step 445 | loss: 2.857968 | lr 1.8000e-04 | norm: 0.2432 | dt: 12831.56ms | tok/sec: 40859.3 | mem: 108493MB | Time Left: 00 hrs : 53 mins [TIMING] data: 4.6ms | fwd: 3155.8ms | loss: 6.6ms | bwd: 9634.3ms | clip: 0.0ms | optim: 22.3ms step 446 | loss: 3.018646 | lr 1.8000e-04 | norm: 0.2659 | dt: 12840.36ms | tok/sec: 40831.3 | mem: 108493MB | Time Left: 00 hrs : 53 mins [TIMING] data: 4.5ms | fwd: 3157.1ms | loss: 6.6ms | bwd: 9641.6ms | clip: 0.0ms | optim: 22.7ms step 447 | loss: 2.696102 | lr 1.8000e-04 | norm: 0.2338 | dt: 12839.79ms | tok/sec: 40833.1 | mem: 108493MB | Time Left: 00 hrs : 53 mins [TIMING] data: 4.6ms | fwd: 3155.4ms | loss: 6.6ms | bwd: 9642.9ms | clip: 0.0ms | optim: 22.3ms step 448 | loss: 2.798927 | lr 1.8000e-04 | norm: 0.2355 | dt: 12837.45ms | tok/sec: 40840.5 | mem: 108493MB | Time Left: 00 hrs : 52 mins [TIMING] data: 4.6ms | fwd: 3157.2ms | loss: 6.6ms | bwd: 9638.9ms | clip: 0.0ms | optim: 22.2ms step 449 | loss: 3.108598 | lr 1.8000e-04 | norm: 0.2250 | dt: 12838.38ms | tok/sec: 40837.6 | mem: 108493MB | Time Left: 00 hrs : 52 mins [TIMING] data: 4.5ms | fwd: 3152.4ms | loss: 6.6ms | bwd: 9644.6ms | clip: 0.0ms | optim: 22.3ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_450.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_350.ckpt step 450 | loss: 3.049031 | lr 1.8000e-04 | norm: 0.2109 | dt: 12737.81ms | tok/sec: 41160.0 | mem: 108493MB | Time Left: 00 hrs : 52 mins [TIMING] data: 4.7ms | fwd: 3134.2ms | loss: 6.6ms | bwd: 9561.7ms | clip: 0.0ms | optim: 22.5ms step 451 | loss: 2.866359 | lr 1.8000e-04 | norm: 0.2226 | dt: 12771.53ms | tok/sec: 41051.3 | mem: 108493MB | Time Left: 00 hrs : 51 mins [TIMING] data: 4.5ms | fwd: 3146.5ms | loss: 6.6ms | bwd: 9583.6ms | clip: 0.0ms | optim: 22.2ms step 452 | loss: 2.978879 | lr 1.8000e-04 | norm: 0.2475 | dt: 12779.38ms | tok/sec: 41026.1 | mem: 108493MB | Time Left: 00 hrs : 51 mins [TIMING] data: 4.5ms | fwd: 3145.3ms | loss: 6.6ms | bwd: 9592.6ms | clip: 0.0ms | optim: 22.4ms step 453 | loss: 3.010677 | lr 1.8000e-04 | norm: 0.2405 | dt: 12810.54ms | tok/sec: 40926.3 | mem: 108493MB | Time Left: 00 hrs : 51 mins [TIMING] data: 4.6ms | fwd: 3147.4ms | loss: 6.6ms | bwd: 9621.6ms | clip: 0.0ms | optim: 22.4ms step 454 | loss: 2.790729 | lr 1.8000e-04 | norm: 0.2230 | dt: 12818.94ms | tok/sec: 40899.5 | mem: 108493MB | Time Left: 00 hrs : 51 mins [TIMING] data: 4.6ms | fwd: 3148.2ms | loss: 6.6ms | bwd: 9629.3ms | clip: 0.0ms | optim: 22.3ms step 455 | loss: 2.983464 | lr 1.8000e-04 | norm: 0.2095 | dt: 12827.65ms | tok/sec: 40871.7 | mem: 108493MB | Time Left: 00 hrs : 51 mins [TIMING] data: 4.6ms | fwd: 3156.6ms | loss: 6.6ms | bwd: 9629.5ms | clip: 0.0ms | optim: 22.4ms step 456 | loss: 3.208194 | lr 1.8000e-04 | norm: 0.2569 | dt: 12831.12ms | tok/sec: 40860.6 | mem: 108493MB | Time Left: 00 hrs : 51 mins [TIMING] data: 4.5ms | fwd: 3162.9ms | loss: 6.6ms | bwd: 9626.6ms | clip: 0.0ms | optim: 22.5ms step 457 | loss: 2.868066 | lr 1.8000e-04 | norm: 0.2157 | dt: 12826.18ms | tok/sec: 40876.4 | mem: 108493MB | Time Left: 00 hrs : 50 mins [TIMING] data: 4.5ms | fwd: 3151.9ms | loss: 6.6ms | bwd: 9632.9ms | clip: 0.0ms | optim: 22.3ms step 458 | loss: 3.201875 | lr 1.8000e-04 | norm: 0.2794 | dt: 12835.69ms | tok/sec: 40846.1 | mem: 108493MB | Time Left: 00 hrs : 50 mins [TIMING] data: 4.7ms | fwd: 3154.0ms | loss: 6.6ms | bwd: 9639.9ms | clip: 0.0ms | optim: 22.5ms step 459 | loss: 2.827011 | lr 1.8000e-04 | norm: 0.2326 | dt: 12818.30ms | tok/sec: 40901.5 | mem: 108493MB | Time Left: 00 hrs : 50 mins [TIMING] data: 4.5ms | fwd: 3149.0ms | loss: 6.6ms | bwd: 9628.0ms | clip: 0.0ms | optim: 22.3ms step 460 | loss: 2.838022 | lr 1.8000e-04 | norm: 0.4386 | dt: 12815.14ms | tok/sec: 40911.6 | mem: 108493MB | Time Left: 00 hrs : 50 mins [TIMING] data: 4.5ms | fwd: 3146.3ms | loss: 6.6ms | bwd: 9627.4ms | clip: 0.0ms | optim: 22.3ms step 461 | loss: 3.084715 | lr 1.8000e-04 | norm: 0.3392 | dt: 12839.69ms | tok/sec: 40833.4 | mem: 108493MB | Time Left: 00 hrs : 50 mins [TIMING] data: 4.5ms | fwd: 3157.4ms | loss: 6.6ms | bwd: 9640.9ms | clip: 0.0ms | optim: 22.3ms step 462 | loss: 2.963472 | lr 1.8000e-04 | norm: 0.2140 | dt: 12818.33ms | tok/sec: 40901.4 | mem: 108493MB | Time Left: 00 hrs : 49 mins [TIMING] data: 4.5ms | fwd: 3152.5ms | loss: 6.6ms | bwd: 9624.5ms | clip: 0.0ms | optim: 22.3ms step 463 | loss: 2.864945 | lr 1.8000e-04 | norm: 0.2122 | dt: 12830.54ms | tok/sec: 40862.5 | mem: 108493MB | Time Left: 00 hrs : 49 mins [TIMING] data: 4.6ms | fwd: 3157.3ms | loss: 6.6ms | bwd: 9631.7ms | clip: 0.0ms | optim: 22.4ms step 464 | loss: 2.704412 | lr 1.8000e-04 | norm: 0.5994 | dt: 12891.18ms | tok/sec: 40670.3 | mem: 108493MB | Time Left: 00 hrs : 49 mins [TIMING] data: 4.5ms | fwd: 3214.3ms | loss: 6.6ms | bwd: 9635.3ms | clip: 0.0ms | optim: 22.5ms step 465 | loss: 2.828480 | lr 1.8000e-04 | norm: 0.2764 | dt: 12836.20ms | tok/sec: 40844.5 | mem: 108493MB | Time Left: 00 hrs : 49 mins [TIMING] data: 4.6ms | fwd: 3155.7ms | loss: 6.6ms | bwd: 9638.9ms | clip: 0.0ms | optim: 22.4ms step 466 | loss: 3.089213 | lr 1.8000e-04 | norm: 0.2364 | dt: 12837.00ms | tok/sec: 40841.9 | mem: 108493MB | Time Left: 00 hrs : 48 mins [TIMING] data: 4.5ms | fwd: 3151.5ms | loss: 6.6ms | bwd: 9643.9ms | clip: 0.0ms | optim: 22.4ms step 467 | loss: 2.841102 | lr 1.8000e-04 | norm: 0.1828 | dt: 12845.84ms | tok/sec: 40813.8 | mem: 108493MB | Time Left: 00 hrs : 48 mins [TIMING] data: 4.5ms | fwd: 3158.1ms | loss: 6.6ms | bwd: 9645.5ms | clip: 0.0ms | optim: 23.1ms step 468 | loss: 2.865230 | lr 1.8000e-04 | norm: 0.2250 | dt: 12832.42ms | tok/sec: 40856.5 | mem: 108493MB | Time Left: 00 hrs : 48 mins [TIMING] data: 4.5ms | fwd: 3148.1ms | loss: 6.6ms | bwd: 9642.6ms | clip: 0.0ms | optim: 22.6ms step 469 | loss: 2.980935 | lr 1.8000e-04 | norm: 0.1861 | dt: 12826.88ms | tok/sec: 40874.2 | mem: 108493MB | Time Left: 00 hrs : 48 mins [TIMING] data: 4.5ms | fwd: 3151.4ms | loss: 6.6ms | bwd: 9634.2ms | clip: 0.0ms | optim: 22.3ms step 470 | loss: 2.825069 | lr 1.8000e-04 | norm: 0.2595 | dt: 12839.94ms | tok/sec: 40832.6 | mem: 108493MB | Time Left: 00 hrs : 48 mins [TIMING] data: 4.5ms | fwd: 3151.4ms | loss: 6.6ms | bwd: 9647.2ms | clip: 0.0ms | optim: 22.3ms step 471 | loss: 2.997283 | lr 1.8000e-04 | norm: 0.2199 | dt: 12830.97ms | tok/sec: 40861.1 | mem: 108493MB | Time Left: 00 hrs : 47 mins [TIMING] data: 4.5ms | fwd: 3153.2ms | loss: 6.6ms | bwd: 9636.4ms | clip: 0.0ms | optim: 22.3ms step 472 | loss: 2.852414 | lr 1.8000e-04 | norm: 0.2511 | dt: 12899.07ms | tok/sec: 40645.4 | mem: 108493MB | Time Left: 00 hrs : 47 mins [TIMING] data: 4.5ms | fwd: 3217.4ms | loss: 6.6ms | bwd: 9640.4ms | clip: 0.0ms | optim: 22.3ms step 473 | loss: 2.990280 | lr 1.8000e-04 | norm: 0.2342 | dt: 12878.95ms | tok/sec: 40708.9 | mem: 108493MB | Time Left: 00 hrs : 47 mins [TIMING] data: 4.5ms | fwd: 3204.6ms | loss: 6.6ms | bwd: 9633.0ms | clip: 0.0ms | optim: 22.3ms step 474 | loss: 2.985396 | lr 1.8000e-04 | norm: 0.2389 | dt: 12838.55ms | tok/sec: 40837.0 | mem: 108493MB | Time Left: 00 hrs : 47 mins [TIMING] data: 4.6ms | fwd: 3154.2ms | loss: 6.6ms | bwd: 9643.0ms | clip: 0.0ms | optim: 22.2ms step 475 | loss: 2.712084 | lr 1.8000e-04 | norm: 0.2034 | dt: 12817.68ms | tok/sec: 40903.5 | mem: 108493MB | Time Left: 00 hrs : 46 mins [TIMING] data: 4.5ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9623.9ms | clip: 0.0ms | optim: 22.3ms step 476 | loss: 2.818600 | lr 1.8000e-04 | norm: 0.4080 | dt: 12822.55ms | tok/sec: 40888.0 | mem: 108493MB | Time Left: 00 hrs : 46 mins [TIMING] data: 4.5ms | fwd: 3154.0ms | loss: 6.6ms | bwd: 9627.2ms | clip: 0.0ms | optim: 22.3ms step 477 | loss: 3.039415 | lr 1.8000e-04 | norm: 0.2508 | dt: 12817.59ms | tok/sec: 40903.8 | mem: 108493MB | Time Left: 00 hrs : 46 mins [TIMING] data: 4.5ms | fwd: 3151.9ms | loss: 6.6ms | bwd: 9624.2ms | clip: 0.0ms | optim: 22.4ms step 478 | loss: 2.535914 | lr 1.8000e-04 | norm: 0.2264 | dt: 12832.31ms | tok/sec: 40856.9 | mem: 108493MB | Time Left: 00 hrs : 46 mins [TIMING] data: 4.6ms | fwd: 3149.1ms | loss: 6.6ms | bwd: 9641.9ms | clip: 0.0ms | optim: 22.3ms step 479 | loss: 2.973129 | lr 1.8000e-04 | norm: 0.2547 | dt: 12834.67ms | tok/sec: 40849.4 | mem: 108493MB | Time Left: 00 hrs : 46 mins [TIMING] data: 4.5ms | fwd: 3156.5ms | loss: 6.6ms | bwd: 9636.9ms | clip: 0.0ms | optim: 22.2ms step 480 | loss: 3.043205 | lr 1.8000e-04 | norm: 0.2453 | dt: 12821.07ms | tok/sec: 40892.7 | mem: 108493MB | Time Left: 00 hrs : 45 mins [TIMING] data: 4.5ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9625.0ms | clip: 0.0ms | optim: 22.3ms step 481 | loss: 2.813926 | lr 1.8000e-04 | norm: 0.2987 | dt: 12839.86ms | tok/sec: 40832.8 | mem: 108493MB | Time Left: 00 hrs : 45 mins [TIMING] data: 4.5ms | fwd: 3152.7ms | loss: 6.6ms | bwd: 9645.6ms | clip: 0.0ms | optim: 22.5ms step 482 | loss: 2.747662 | lr 1.8000e-04 | norm: 0.2877 | dt: 12837.27ms | tok/sec: 40841.1 | mem: 108493MB | Time Left: 00 hrs : 45 mins [TIMING] data: 4.5ms | fwd: 3151.5ms | loss: 6.6ms | bwd: 9644.3ms | clip: 0.0ms | optim: 22.3ms step 483 | loss: 3.017756 | lr 1.8000e-04 | norm: 0.3703 | dt: 12860.98ms | tok/sec: 40765.8 | mem: 108493MB | Time Left: 00 hrs : 45 mins [TIMING] data: 4.5ms | fwd: 3181.9ms | loss: 6.6ms | bwd: 9637.7ms | clip: 0.0ms | optim: 22.4ms step 484 | loss: 3.044131 | lr 1.8000e-04 | norm: 0.2822 | dt: 12821.53ms | tok/sec: 40891.2 | mem: 108493MB | Time Left: 00 hrs : 45 mins [TIMING] data: 4.5ms | fwd: 3153.5ms | loss: 6.6ms | bwd: 9626.3ms | clip: 0.0ms | optim: 22.6ms step 485 | loss: 3.028036 | lr 1.8000e-04 | norm: 0.2414 | dt: 12839.05ms | tok/sec: 40835.4 | mem: 108493MB | Time Left: 00 hrs : 44 mins [TIMING] data: 4.5ms | fwd: 3159.2ms | loss: 6.6ms | bwd: 9638.6ms | clip: 0.0ms | optim: 22.3ms step 486 | loss: 3.182760 | lr 1.8000e-04 | norm: 0.3106 | dt: 12821.58ms | tok/sec: 40891.1 | mem: 108493MB | Time Left: 00 hrs : 44 mins [TIMING] data: 4.5ms | fwd: 3147.7ms | loss: 6.6ms | bwd: 9632.6ms | clip: 0.0ms | optim: 22.3ms step 487 | loss: 2.813215 | lr 1.8000e-04 | norm: 0.3059 | dt: 12840.40ms | tok/sec: 40831.1 | mem: 108493MB | Time Left: 00 hrs : 44 mins [TIMING] data: 4.5ms | fwd: 3166.8ms | loss: 6.6ms | bwd: 9632.0ms | clip: 0.0ms | optim: 22.4ms step 488 | loss: 2.909073 | lr 1.8000e-04 | norm: 0.3818 | dt: 12830.29ms | tok/sec: 40863.3 | mem: 108493MB | Time Left: 00 hrs : 44 mins [TIMING] data: 4.5ms | fwd: 3158.0ms | loss: 6.6ms | bwd: 9630.8ms | clip: 0.0ms | optim: 22.3ms step 489 | loss: 3.044792 | lr 1.8000e-04 | norm: 0.9850 | dt: 12826.79ms | tok/sec: 40874.5 | mem: 108493MB | Time Left: 00 hrs : 44 mins [TIMING] data: 4.5ms | fwd: 3154.4ms | loss: 6.6ms | bwd: 9631.1ms | clip: 0.0ms | optim: 22.3ms step 490 | loss: 3.004969 | lr 1.8000e-04 | norm: 0.6899 | dt: 12823.58ms | tok/sec: 40884.7 | mem: 108493MB | Time Left: 00 hrs : 43 mins [TIMING] data: 4.5ms | fwd: 3154.4ms | loss: 6.6ms | bwd: 9627.8ms | clip: 0.0ms | optim: 22.2ms step 491 | loss: 2.754041 | lr 1.8000e-04 | norm: 0.2351 | dt: 12850.71ms | tok/sec: 40798.4 | mem: 108493MB | Time Left: 00 hrs : 43 mins [TIMING] data: 4.5ms | fwd: 3161.7ms | loss: 6.6ms | bwd: 9647.6ms | clip: 0.0ms | optim: 22.4ms step 492 | loss: 3.183249 | lr 1.8000e-04 | norm: 0.2209 | dt: 12840.53ms | tok/sec: 40830.7 | mem: 108493MB | Time Left: 00 hrs : 43 mins [TIMING] data: 4.5ms | fwd: 3153.9ms | loss: 6.6ms | bwd: 9645.2ms | clip: 0.0ms | optim: 22.2ms step 493 | loss: 3.044974 | lr 1.8000e-04 | norm: 0.2479 | dt: 12833.17ms | tok/sec: 40854.1 | mem: 108493MB | Time Left: 00 hrs : 43 mins [TIMING] data: 4.5ms | fwd: 3153.5ms | loss: 6.6ms | bwd: 9638.2ms | clip: 0.0ms | optim: 22.4ms step 494 | loss: 2.931549 | lr 1.8000e-04 | norm: 0.2263 | dt: 12822.99ms | tok/sec: 40886.6 | mem: 108493MB | Time Left: 00 hrs : 42 mins [TIMING] data: 4.6ms | fwd: 3150.2ms | loss: 6.6ms | bwd: 9631.4ms | clip: 0.0ms | optim: 22.3ms step 495 | loss: 3.068449 | lr 1.8000e-04 | norm: 0.2125 | dt: 12827.83ms | tok/sec: 40871.1 | mem: 108493MB | Time Left: 00 hrs : 42 mins [TIMING] data: 4.5ms | fwd: 3153.7ms | loss: 6.6ms | bwd: 9632.5ms | clip: 0.0ms | optim: 22.5ms step 496 | loss: 3.143793 | lr 1.8000e-04 | norm: 0.2173 | dt: 12829.62ms | tok/sec: 40865.4 | mem: 108493MB | Time Left: 00 hrs : 42 mins [TIMING] data: 4.5ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9633.5ms | clip: 0.0ms | optim: 22.3ms step 497 | loss: 3.093701 | lr 1.8000e-04 | norm: 0.2256 | dt: 12832.09ms | tok/sec: 40857.6 | mem: 108493MB | Time Left: 00 hrs : 42 mins [TIMING] data: 4.5ms | fwd: 3150.7ms | loss: 6.7ms | bwd: 9639.8ms | clip: 0.0ms | optim: 22.3ms step 498 | loss: 3.081130 | lr 1.8000e-04 | norm: 0.2826 | dt: 12833.44ms | tok/sec: 40853.3 | mem: 108493MB | Time Left: 00 hrs : 42 mins [TIMING] data: 4.6ms | fwd: 3153.3ms | loss: 6.6ms | bwd: 9638.0ms | clip: 0.0ms | optim: 22.7ms step 499 | loss: 3.079418 | lr 1.8000e-04 | norm: 0.3634 | dt: 12844.04ms | tok/sec: 40819.5 | mem: 108493MB | Time Left: 00 hrs : 41 mins [TIMING] data: 4.5ms | fwd: 3156.6ms | loss: 6.6ms | bwd: 9645.8ms | clip: 0.0ms | optim: 22.5ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_500.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_400.ckpt validation loss: 2.9778 step 500 | loss: 2.924253 | lr 1.8000e-04 | norm: 0.2538 | dt: 20702.53ms | tok/sec: 25324.8 | mem: 108493MB | Time Left: 01 hrs : 07 mins [TIMING] data: 4.6ms | fwd: 3147.0ms | loss: 6.7ms | bwd: 9577.1ms | clip: 0.0ms | optim: 22.3ms step 501 | loss: 3.030045 | lr 1.8000e-04 | norm: 0.2724 | dt: 12766.75ms | tok/sec: 41066.7 | mem: 108493MB | Time Left: 00 hrs : 41 mins [TIMING] data: 4.5ms | fwd: 3140.1ms | loss: 6.6ms | bwd: 9585.0ms | clip: 0.0ms | optim: 22.5ms step 502 | loss: 2.702259 | lr 1.8000e-04 | norm: 0.2597 | dt: 12817.69ms | tok/sec: 40903.5 | mem: 108493MB | Time Left: 00 hrs : 41 mins [TIMING] data: 4.5ms | fwd: 3163.2ms | loss: 6.6ms | bwd: 9613.1ms | clip: 0.0ms | optim: 22.3ms step 503 | loss: 3.050541 | lr 1.8000e-04 | norm: 0.2746 | dt: 12805.08ms | tok/sec: 40943.8 | mem: 108493MB | Time Left: 00 hrs : 40 mins [TIMING] data: 4.5ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9611.0ms | clip: 0.0ms | optim: 22.4ms step 504 | loss: 3.026444 | lr 1.8000e-04 | norm: 0.2565 | dt: 12806.03ms | tok/sec: 40940.7 | mem: 108493MB | Time Left: 00 hrs : 40 mins [TIMING] data: 4.5ms | fwd: 3147.9ms | loss: 6.6ms | bwd: 9616.7ms | clip: 0.0ms | optim: 22.4ms step 505 | loss: 3.064511 | lr 1.8000e-04 | norm: 0.2232 | dt: 12821.04ms | tok/sec: 40892.8 | mem: 108493MB | Time Left: 00 hrs : 40 mins [TIMING] data: 4.6ms | fwd: 3147.2ms | loss: 6.7ms | bwd: 9632.0ms | clip: 0.0ms | optim: 22.4ms step 506 | loss: 2.746309 | lr 1.8000e-04 | norm: 0.3405 | dt: 12881.98ms | tok/sec: 40699.3 | mem: 108493MB | Time Left: 00 hrs : 40 mins [TIMING] data: 4.5ms | fwd: 3215.2ms | loss: 6.6ms | bwd: 9625.4ms | clip: 0.0ms | optim: 22.3ms step 507 | loss: 3.198687 | lr 1.8000e-04 | norm: 0.2852 | dt: 12817.53ms | tok/sec: 40904.0 | mem: 108493MB | Time Left: 00 hrs : 40 mins [TIMING] data: 4.5ms | fwd: 3152.1ms | loss: 6.6ms | bwd: 9624.0ms | clip: 0.0ms | optim: 22.4ms step 508 | loss: 2.953954 | lr 1.8000e-04 | norm: 0.2301 | dt: 12860.75ms | tok/sec: 40766.5 | mem: 108493MB | Time Left: 00 hrs : 40 mins [TIMING] data: 4.5ms | fwd: 3186.8ms | loss: 6.6ms | bwd: 9632.4ms | clip: 0.0ms | optim: 22.4ms step 509 | loss: 3.022934 | lr 1.8000e-04 | norm: 0.2584 | dt: 12820.74ms | tok/sec: 40893.7 | mem: 108493MB | Time Left: 00 hrs : 39 mins [TIMING] data: 4.5ms | fwd: 3150.2ms | loss: 6.6ms | bwd: 9629.3ms | clip: 0.0ms | optim: 22.3ms step 510 | loss: 3.070247 | lr 1.8000e-04 | norm: 0.3794 | dt: 12829.06ms | tok/sec: 40867.2 | mem: 108493MB | Time Left: 00 hrs : 39 mins [TIMING] data: 4.5ms | fwd: 3157.5ms | loss: 6.6ms | bwd: 9630.0ms | clip: 0.0ms | optim: 22.5ms step 511 | loss: 2.880010 | lr 1.8000e-04 | norm: 0.3131 | dt: 12810.66ms | tok/sec: 40925.9 | mem: 108493MB | Time Left: 00 hrs : 39 mins [TIMING] data: 4.5ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9614.7ms | clip: 0.0ms | optim: 22.3ms step 512 | loss: 3.003440 | lr 1.8000e-04 | norm: 0.2300 | dt: 12830.83ms | tok/sec: 40861.6 | mem: 108493MB | Time Left: 00 hrs : 39 mins [TIMING] data: 4.5ms | fwd: 3155.3ms | loss: 6.6ms | bwd: 9634.3ms | clip: 0.0ms | optim: 22.3ms step 513 | loss: 2.707247 | lr 1.8000e-04 | norm: 0.2571 | dt: 12831.47ms | tok/sec: 40859.5 | mem: 108493MB | Time Left: 00 hrs : 38 mins [TIMING] data: 4.5ms | fwd: 3152.7ms | loss: 6.6ms | bwd: 9637.4ms | clip: 0.0ms | optim: 22.3ms step 514 | loss: 3.086557 | lr 1.8000e-04 | norm: 0.3279 | dt: 12826.44ms | tok/sec: 40875.6 | mem: 108493MB | Time Left: 00 hrs : 38 mins [TIMING] data: 4.5ms | fwd: 3153.3ms | loss: 6.6ms | bwd: 9631.7ms | clip: 0.0ms | optim: 22.4ms step 515 | loss: 3.064240 | lr 1.8000e-04 | norm: 0.2585 | dt: 12829.15ms | tok/sec: 40866.9 | mem: 108493MB | Time Left: 00 hrs : 38 mins [TIMING] data: 4.5ms | fwd: 3149.7ms | loss: 6.6ms | bwd: 9638.1ms | clip: 0.0ms | optim: 22.3ms step 516 | loss: 3.028144 | lr 1.8000e-04 | norm: 0.2374 | dt: 12890.81ms | tok/sec: 40671.5 | mem: 108493MB | Time Left: 00 hrs : 38 mins [TIMING] data: 4.5ms | fwd: 3218.7ms | loss: 6.6ms | bwd: 9630.8ms | clip: 0.0ms | optim: 22.3ms step 517 | loss: 3.557054 | lr 1.8000e-04 | norm: 0.3703 | dt: 12834.15ms | tok/sec: 40851.0 | mem: 108493MB | Time Left: 00 hrs : 38 mins [TIMING] data: 4.5ms | fwd: 3156.6ms | loss: 6.6ms | bwd: 9636.0ms | clip: 0.0ms | optim: 22.5ms step 518 | loss: 3.130965 | lr 1.8000e-04 | norm: 0.2503 | dt: 12842.09ms | tok/sec: 40825.8 | mem: 108493MB | Time Left: 00 hrs : 37 mins [TIMING] data: 4.5ms | fwd: 3157.7ms | loss: 6.6ms | bwd: 9643.0ms | clip: 0.0ms | optim: 22.3ms step 519 | loss: 2.924809 | lr 1.8000e-04 | norm: 0.2662 | dt: 12836.02ms | tok/sec: 40845.1 | mem: 108493MB | Time Left: 00 hrs : 37 mins [TIMING] data: 4.5ms | fwd: 3151.1ms | loss: 6.7ms | bwd: 9643.2ms | clip: 0.0ms | optim: 22.4ms step 520 | loss: 2.778615 | lr 1.8000e-04 | norm: 0.2385 | dt: 12834.04ms | tok/sec: 40851.4 | mem: 108493MB | Time Left: 00 hrs : 37 mins [TIMING] data: 4.5ms | fwd: 3155.5ms | loss: 6.6ms | bwd: 9637.0ms | clip: 0.0ms | optim: 22.4ms step 521 | loss: 3.069102 | lr 1.8000e-04 | norm: 0.2405 | dt: 12826.58ms | tok/sec: 40875.1 | mem: 108493MB | Time Left: 00 hrs : 37 mins [TIMING] data: 4.5ms | fwd: 3152.4ms | loss: 6.6ms | bwd: 9632.7ms | clip: 0.0ms | optim: 22.5ms step 522 | loss: 3.033666 | lr 1.8000e-04 | norm: 0.2568 | dt: 12833.07ms | tok/sec: 40854.5 | mem: 108493MB | Time Left: 00 hrs : 37 mins [TIMING] data: 4.5ms | fwd: 3148.1ms | loss: 6.6ms | bwd: 9643.5ms | clip: 0.0ms | optim: 22.5ms step 523 | loss: 3.209324 | lr 1.8000e-04 | norm: 0.2542 | dt: 12834.48ms | tok/sec: 40850.0 | mem: 108493MB | Time Left: 00 hrs : 36 mins [TIMING] data: 4.5ms | fwd: 3156.3ms | loss: 6.6ms | bwd: 9636.9ms | clip: 0.0ms | optim: 22.2ms step 524 | loss: 3.032391 | lr 1.8000e-04 | norm: 0.2600 | dt: 12857.69ms | tok/sec: 40776.2 | mem: 108493MB | Time Left: 00 hrs : 36 mins [TIMING] data: 4.5ms | fwd: 3184.2ms | loss: 6.6ms | bwd: 9632.1ms | clip: 0.0ms | optim: 22.3ms step 525 | loss: 2.824915 | lr 1.8000e-04 | norm: 0.2634 | dt: 12829.54ms | tok/sec: 40865.7 | mem: 108493MB | Time Left: 00 hrs : 36 mins [TIMING] data: 4.5ms | fwd: 3153.9ms | loss: 6.6ms | bwd: 9634.2ms | clip: 0.0ms | optim: 22.4ms step 526 | loss: 3.011051 | lr 1.8000e-04 | norm: 0.3394 | dt: 12827.65ms | tok/sec: 40871.7 | mem: 108493MB | Time Left: 00 hrs : 36 mins [TIMING] data: 4.5ms | fwd: 3152.6ms | loss: 6.6ms | bwd: 9633.7ms | clip: 0.0ms | optim: 22.4ms step 527 | loss: 3.035379 | lr 1.8000e-04 | norm: 0.2297 | dt: 12820.29ms | tok/sec: 40895.2 | mem: 108493MB | Time Left: 00 hrs : 35 mins [TIMING] data: 4.5ms | fwd: 3152.7ms | loss: 6.6ms | bwd: 9626.4ms | clip: 0.0ms | optim: 22.2ms step 528 | loss: 2.640626 | lr 1.8000e-04 | norm: 0.2426 | dt: 12839.46ms | tok/sec: 40834.1 | mem: 108493MB | Time Left: 00 hrs : 35 mins [TIMING] data: 4.5ms | fwd: 3158.3ms | loss: 6.6ms | bwd: 9639.9ms | clip: 0.0ms | optim: 22.3ms step 529 | loss: 2.824936 | lr 1.8000e-04 | norm: 0.6644 | dt: 12838.20ms | tok/sec: 40838.1 | mem: 108493MB | Time Left: 00 hrs : 35 mins [TIMING] data: 4.5ms | fwd: 3156.1ms | loss: 6.6ms | bwd: 9640.6ms | clip: 0.0ms | optim: 22.5ms step 530 | loss: 3.119645 | lr 1.8000e-04 | norm: 0.2941 | dt: 12834.01ms | tok/sec: 40851.5 | mem: 108493MB | Time Left: 00 hrs : 35 mins [TIMING] data: 4.5ms | fwd: 3155.3ms | loss: 6.6ms | bwd: 9637.4ms | clip: 0.0ms | optim: 22.3ms step 531 | loss: 3.032253 | lr 1.8000e-04 | norm: 0.2465 | dt: 12821.52ms | tok/sec: 40891.3 | mem: 108493MB | Time Left: 00 hrs : 35 mins [TIMING] data: 4.5ms | fwd: 3152.5ms | loss: 6.6ms | bwd: 9627.7ms | clip: 0.0ms | optim: 22.3ms step 532 | loss: 3.231014 | lr 1.8000e-04 | norm: 0.2086 | dt: 12846.32ms | tok/sec: 40812.3 | mem: 108493MB | Time Left: 00 hrs : 34 mins [TIMING] data: 4.5ms | fwd: 3155.2ms | loss: 6.6ms | bwd: 9649.5ms | clip: 0.0ms | optim: 22.5ms step 533 | loss: 3.030294 | lr 1.8000e-04 | norm: 0.2408 | dt: 12829.36ms | tok/sec: 40866.3 | mem: 108493MB | Time Left: 00 hrs : 34 mins [TIMING] data: 4.5ms | fwd: 3154.5ms | loss: 6.6ms | bwd: 9633.6ms | clip: 0.0ms | optim: 22.3ms step 534 | loss: 3.244010 | lr 1.8000e-04 | norm: 0.2302 | dt: 12820.22ms | tok/sec: 40895.4 | mem: 108493MB | Time Left: 00 hrs : 34 mins [TIMING] data: 4.5ms | fwd: 3155.7ms | loss: 6.6ms | bwd: 9623.2ms | clip: 0.0ms | optim: 22.3ms step 535 | loss: 2.886804 | lr 1.8000e-04 | norm: 0.2091 | dt: 12820.22ms | tok/sec: 40895.4 | mem: 108493MB | Time Left: 00 hrs : 34 mins [TIMING] data: 4.5ms | fwd: 3150.9ms | loss: 6.6ms | bwd: 9627.9ms | clip: 0.0ms | optim: 22.3ms step 536 | loss: 2.814050 | lr 1.8000e-04 | norm: 0.2436 | dt: 12836.05ms | tok/sec: 40845.0 | mem: 108493MB | Time Left: 00 hrs : 34 mins [TIMING] data: 4.6ms | fwd: 3158.0ms | loss: 6.6ms | bwd: 9636.8ms | clip: 0.0ms | optim: 22.2ms step 537 | loss: 3.002290 | lr 1.8000e-04 | norm: 0.2170 | dt: 12821.85ms | tok/sec: 40890.2 | mem: 108493MB | Time Left: 00 hrs : 33 mins [TIMING] data: 4.5ms | fwd: 3157.7ms | loss: 6.6ms | bwd: 9622.6ms | clip: 0.0ms | optim: 22.3ms step 538 | loss: 3.172391 | lr 1.8000e-04 | norm: 0.3287 | dt: 12818.06ms | tok/sec: 40902.3 | mem: 108493MB | Time Left: 00 hrs : 33 mins [TIMING] data: 4.5ms | fwd: 3154.5ms | loss: 6.6ms | bwd: 9622.3ms | clip: 0.0ms | optim: 22.3ms step 539 | loss: 2.896319 | lr 1.8000e-04 | norm: 0.2704 | dt: 12834.16ms | tok/sec: 40851.0 | mem: 108493MB | Time Left: 00 hrs : 33 mins [TIMING] data: 4.5ms | fwd: 3152.1ms | loss: 6.6ms | bwd: 9640.7ms | clip: 0.0ms | optim: 22.3ms step 540 | loss: 2.716701 | lr 1.8000e-04 | norm: 0.2275 | dt: 12840.28ms | tok/sec: 40831.5 | mem: 108493MB | Time Left: 00 hrs : 33 mins [TIMING] data: 4.5ms | fwd: 3159.1ms | loss: 6.6ms | bwd: 9639.6ms | clip: 0.0ms | optim: 22.5ms step 541 | loss: 3.104419 | lr 1.8000e-04 | norm: 0.3213 | dt: 12834.35ms | tok/sec: 40850.4 | mem: 108493MB | Time Left: 00 hrs : 32 mins [TIMING] data: 4.5ms | fwd: 3154.7ms | loss: 6.6ms | bwd: 9638.1ms | clip: 0.0ms | optim: 22.5ms step 542 | loss: 2.700773 | lr 1.8000e-04 | norm: 0.2359 | dt: 12831.27ms | tok/sec: 40860.2 | mem: 108493MB | Time Left: 00 hrs : 32 mins [TIMING] data: 4.5ms | fwd: 3157.4ms | loss: 6.6ms | bwd: 9632.6ms | clip: 0.0ms | optim: 22.2ms step 543 | loss: 2.912658 | lr 1.8000e-04 | norm: 0.2474 | dt: 12855.28ms | tok/sec: 40783.9 | mem: 108493MB | Time Left: 00 hrs : 32 mins [TIMING] data: 4.5ms | fwd: 3176.2ms | loss: 6.6ms | bwd: 9637.7ms | clip: 0.0ms | optim: 22.2ms step 544 | loss: 2.881262 | lr 1.8000e-04 | norm: 0.2387 | dt: 12840.15ms | tok/sec: 40831.9 | mem: 108493MB | Time Left: 00 hrs : 32 mins [TIMING] data: 4.5ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9644.2ms | clip: 0.0ms | optim: 22.3ms step 545 | loss: 3.376977 | lr 1.8000e-04 | norm: 0.4802 | dt: 12822.48ms | tok/sec: 40888.2 | mem: 108493MB | Time Left: 00 hrs : 32 mins [TIMING] data: 4.5ms | fwd: 3157.0ms | loss: 6.6ms | bwd: 9624.1ms | clip: 0.0ms | optim: 22.4ms step 546 | loss: 2.949278 | lr 1.8000e-04 | norm: 0.2336 | dt: 12889.29ms | tok/sec: 40676.2 | mem: 108493MB | Time Left: 00 hrs : 32 mins [TIMING] data: 4.5ms | fwd: 3211.2ms | loss: 6.6ms | bwd: 9636.5ms | clip: 0.0ms | optim: 22.5ms step 547 | loss: 2.866865 | lr 1.8000e-04 | norm: 0.2965 | dt: 12831.99ms | tok/sec: 40857.9 | mem: 108493MB | Time Left: 00 hrs : 31 mins [TIMING] data: 4.5ms | fwd: 3154.7ms | loss: 6.6ms | bwd: 9635.6ms | clip: 0.0ms | optim: 22.6ms step 548 | loss: 2.915625 | lr 1.8000e-04 | norm: 0.2082 | dt: 12833.43ms | tok/sec: 40853.3 | mem: 108493MB | Time Left: 00 hrs : 31 mins [TIMING] data: 4.5ms | fwd: 3154.0ms | loss: 6.6ms | bwd: 9638.1ms | clip: 0.0ms | optim: 22.4ms step 549 | loss: 3.000151 | lr 1.8000e-04 | norm: 0.2629 | dt: 12831.29ms | tok/sec: 40860.1 | mem: 108493MB | Time Left: 00 hrs : 31 mins [TIMING] data: 4.5ms | fwd: 3148.7ms | loss: 6.6ms | bwd: 9641.1ms | clip: 0.0ms | optim: 22.5ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_550.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_450.ckpt Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b5_65536_131072/train_shard_00003.bin 100000000 step 550 | loss: 2.943639 | lr 1.8000e-04 | norm: 0.2548 | dt: 13337.14ms | tok/sec: 39310.4 | mem: 108493MB | Time Left: 00 hrs : 32 mins [TIMING] data: 8.1ms | fwd: 3733.9ms | loss: 6.6ms | bwd: 9558.1ms | clip: 0.0ms | optim: 22.3ms step 551 | loss: 2.872264 | lr 1.8000e-04 | norm: 0.2280 | dt: 12767.27ms | tok/sec: 41065.0 | mem: 108493MB | Time Left: 00 hrs : 30 mins [TIMING] data: 4.5ms | fwd: 3140.1ms | loss: 6.6ms | bwd: 9585.9ms | clip: 0.0ms | optim: 22.2ms step 552 | loss: 2.936829 | lr 1.8000e-04 | norm: 0.2431 | dt: 12790.87ms | tok/sec: 40989.2 | mem: 108493MB | Time Left: 00 hrs : 30 mins [TIMING] data: 4.5ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9597.2ms | clip: 0.0ms | optim: 22.4ms step 553 | loss: 2.898156 | lr 1.8000e-04 | norm: 0.2345 | dt: 12812.78ms | tok/sec: 40919.2 | mem: 108493MB | Time Left: 00 hrs : 30 mins [TIMING] data: 4.5ms | fwd: 3148.9ms | loss: 6.6ms | bwd: 9622.3ms | clip: 0.0ms | optim: 22.5ms step 554 | loss: 2.972840 | lr 1.8000e-04 | norm: 0.2754 | dt: 12814.02ms | tok/sec: 40915.2 | mem: 108493MB | Time Left: 00 hrs : 30 mins [TIMING] data: 4.6ms | fwd: 3145.7ms | loss: 6.6ms | bwd: 9626.8ms | clip: 0.0ms | optim: 22.5ms step 555 | loss: 2.805411 | lr 1.8000e-04 | norm: 0.2281 | dt: 12825.85ms | tok/sec: 40877.4 | mem: 108493MB | Time Left: 00 hrs : 29 mins [TIMING] data: 4.5ms | fwd: 3152.0ms | loss: 6.6ms | bwd: 9632.5ms | clip: 0.0ms | optim: 22.3ms step 556 | loss: 3.036297 | lr 1.8000e-04 | norm: 0.2819 | dt: 12821.64ms | tok/sec: 40890.9 | mem: 108493MB | Time Left: 00 hrs : 29 mins [TIMING] data: 4.5ms | fwd: 3151.3ms | loss: 6.6ms | bwd: 9628.9ms | clip: 0.0ms | optim: 22.5ms step 557 | loss: 2.806372 | lr 1.8000e-04 | norm: 0.2529 | dt: 12824.42ms | tok/sec: 40882.0 | mem: 108493MB | Time Left: 00 hrs : 29 mins [TIMING] data: 4.5ms | fwd: 3151.1ms | loss: 6.6ms | bwd: 9632.0ms | clip: 0.0ms | optim: 22.3ms step 558 | loss: 3.051725 | lr 1.8000e-04 | norm: 0.2046 | dt: 12811.08ms | tok/sec: 40924.6 | mem: 108493MB | Time Left: 00 hrs : 29 mins [TIMING] data: 4.4ms | fwd: 3154.0ms | loss: 6.6ms | bwd: 9615.8ms | clip: 0.0ms | optim: 22.3ms step 559 | loss: 3.160486 | lr 1.8000e-04 | norm: 0.2282 | dt: 12822.14ms | tok/sec: 40889.3 | mem: 108493MB | Time Left: 00 hrs : 29 mins [TIMING] data: 4.5ms | fwd: 3156.5ms | loss: 6.6ms | bwd: 9624.2ms | clip: 0.0ms | optim: 22.5ms step 560 | loss: 2.972435 | lr 1.8000e-04 | norm: 0.2003 | dt: 12823.38ms | tok/sec: 40885.3 | mem: 108493MB | Time Left: 00 hrs : 28 mins [TIMING] data: 4.5ms | fwd: 3156.0ms | loss: 6.6ms | bwd: 9625.8ms | clip: 0.0ms | optim: 22.5ms step 561 | loss: 3.226526 | lr 1.8000e-04 | norm: 0.2550 | dt: 12832.50ms | tok/sec: 40856.3 | mem: 108493MB | Time Left: 00 hrs : 28 mins [TIMING] data: 4.6ms | fwd: 3158.4ms | loss: 6.6ms | bwd: 9632.8ms | clip: 0.0ms | optim: 22.3ms step 562 | loss: 3.032834 | lr 1.8000e-04 | norm: 0.2255 | dt: 12812.95ms | tok/sec: 40918.6 | mem: 108493MB | Time Left: 00 hrs : 28 mins [TIMING] data: 4.6ms | fwd: 3155.8ms | loss: 6.6ms | bwd: 9615.7ms | clip: 0.0ms | optim: 22.3ms step 563 | loss: 2.319188 | lr 1.8000e-04 | norm: 0.2536 | dt: 12824.16ms | tok/sec: 40882.8 | mem: 108493MB | Time Left: 00 hrs : 28 mins [TIMING] data: 4.5ms | fwd: 3155.8ms | loss: 6.6ms | bwd: 9626.7ms | clip: 0.0ms | optim: 22.5ms step 564 | loss: 2.975928 | lr 1.8000e-04 | norm: 0.2430 | dt: 12840.85ms | tok/sec: 40829.7 | mem: 108493MB | Time Left: 00 hrs : 28 mins [TIMING] data: 4.5ms | fwd: 3154.5ms | loss: 6.6ms | bwd: 9644.8ms | clip: 0.0ms | optim: 22.4ms step 565 | loss: 2.941253 | lr 1.8000e-04 | norm: 0.2371 | dt: 12842.15ms | tok/sec: 40825.6 | mem: 108493MB | Time Left: 00 hrs : 27 mins [TIMING] data: 4.5ms | fwd: 3156.7ms | loss: 6.6ms | bwd: 9644.1ms | clip: 0.0ms | optim: 22.4ms step 566 | loss: 2.859506 | lr 1.8000e-04 | norm: 0.2055 | dt: 12828.46ms | tok/sec: 40869.1 | mem: 108493MB | Time Left: 00 hrs : 27 mins [TIMING] data: 4.5ms | fwd: 3152.9ms | loss: 6.6ms | bwd: 9634.2ms | clip: 0.0ms | optim: 22.4ms step 567 | loss: 3.067632 | lr 1.8000e-04 | norm: 0.1889 | dt: 12841.63ms | tok/sec: 40827.2 | mem: 108493MB | Time Left: 00 hrs : 27 mins [TIMING] data: 4.5ms | fwd: 3162.9ms | loss: 6.6ms | bwd: 9637.4ms | clip: 0.0ms | optim: 22.4ms step 568 | loss: 2.909504 | lr 1.8000e-04 | norm: 0.2409 | dt: 12835.21ms | tok/sec: 40847.6 | mem: 108493MB | Time Left: 00 hrs : 27 mins [TIMING] data: 4.6ms | fwd: 3158.4ms | loss: 6.6ms | bwd: 9635.5ms | clip: 0.0ms | optim: 22.3ms step 569 | loss: 2.789964 | lr 1.8000e-04 | norm: 0.2399 | dt: 12826.06ms | tok/sec: 40876.8 | mem: 108493MB | Time Left: 00 hrs : 26 mins [TIMING] data: 4.5ms | fwd: 3149.7ms | loss: 6.6ms | bwd: 9635.1ms | clip: 0.0ms | optim: 22.3ms step 570 | loss: 3.159377 | lr 1.8000e-04 | norm: 0.2150 | dt: 12826.53ms | tok/sec: 40875.3 | mem: 108493MB | Time Left: 00 hrs : 26 mins [TIMING] data: 4.6ms | fwd: 3151.5ms | loss: 6.6ms | bwd: 9633.6ms | clip: 0.0ms | optim: 22.3ms step 571 | loss: 2.882967 | lr 1.8000e-04 | norm: 0.2678 | dt: 12825.20ms | tok/sec: 40879.5 | mem: 108493MB | Time Left: 00 hrs : 26 mins [TIMING] data: 4.5ms | fwd: 3155.6ms | loss: 6.6ms | bwd: 9628.3ms | clip: 0.0ms | optim: 22.3ms step 572 | loss: 3.158074 | lr 1.8000e-04 | norm: 0.3626 | dt: 12815.63ms | tok/sec: 40910.0 | mem: 108493MB | Time Left: 00 hrs : 26 mins [TIMING] data: 4.5ms | fwd: 3153.9ms | loss: 6.6ms | bwd: 9620.5ms | clip: 0.0ms | optim: 22.3ms step 573 | loss: 2.907897 | lr 1.8000e-04 | norm: 0.3164 | dt: 12823.21ms | tok/sec: 40885.8 | mem: 108493MB | Time Left: 00 hrs : 26 mins [TIMING] data: 4.5ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9629.6ms | clip: 0.0ms | optim: 22.2ms step 574 | loss: 3.118965 | lr 1.8000e-04 | norm: 0.2415 | dt: 12830.98ms | tok/sec: 40861.1 | mem: 108493MB | Time Left: 00 hrs : 25 mins [TIMING] data: 4.6ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9634.9ms | clip: 0.0ms | optim: 22.3ms step 575 | loss: 2.969619 | lr 1.8000e-04 | norm: 0.2192 | dt: 12829.56ms | tok/sec: 40865.6 | mem: 108493MB | Time Left: 00 hrs : 25 mins [TIMING] data: 4.5ms | fwd: 3148.3ms | loss: 6.6ms | bwd: 9639.8ms | clip: 0.0ms | optim: 22.4ms step 576 | loss: 3.191441 | lr 1.8000e-04 | norm: 0.1988 | dt: 12824.52ms | tok/sec: 40881.7 | mem: 108493MB | Time Left: 00 hrs : 25 mins [TIMING] data: 4.5ms | fwd: 3151.1ms | loss: 6.6ms | bwd: 9631.9ms | clip: 0.0ms | optim: 22.4ms step 577 | loss: 2.682159 | lr 1.8000e-04 | norm: 0.1854 | dt: 12835.70ms | tok/sec: 40846.1 | mem: 108493MB | Time Left: 00 hrs : 25 mins [TIMING] data: 4.5ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9637.9ms | clip: 0.0ms | optim: 22.4ms step 578 | loss: 2.906640 | lr 1.8000e-04 | norm: 0.2707 | dt: 12825.40ms | tok/sec: 40878.9 | mem: 108493MB | Time Left: 00 hrs : 25 mins [TIMING] data: 4.5ms | fwd: 3154.5ms | loss: 6.6ms | bwd: 9629.6ms | clip: 0.0ms | optim: 22.3ms step 579 | loss: 2.976421 | lr 1.8000e-04 | norm: 0.2477 | dt: 12821.25ms | tok/sec: 40892.1 | mem: 108493MB | Time Left: 00 hrs : 24 mins [TIMING] data: 4.5ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9623.6ms | clip: 0.0ms | optim: 22.2ms step 580 | loss: 2.940933 | lr 1.8000e-04 | norm: 0.3113 | dt: 12833.38ms | tok/sec: 40853.5 | mem: 108493MB | Time Left: 00 hrs : 24 mins [TIMING] data: 4.5ms | fwd: 3160.3ms | loss: 6.6ms | bwd: 9631.7ms | clip: 0.0ms | optim: 22.3ms step 581 | loss: 3.053155 | lr 1.8000e-04 | norm: 0.1945 | dt: 12830.94ms | tok/sec: 40861.2 | mem: 108493MB | Time Left: 00 hrs : 24 mins [TIMING] data: 4.5ms | fwd: 3153.4ms | loss: 6.6ms | bwd: 9636.1ms | clip: 0.0ms | optim: 22.5ms step 582 | loss: 2.855389 | lr 1.8000e-04 | norm: 0.2143 | dt: 12836.95ms | tok/sec: 40842.1 | mem: 108493MB | Time Left: 00 hrs : 24 mins [TIMING] data: 4.4ms | fwd: 3156.7ms | loss: 6.6ms | bwd: 9638.8ms | clip: 0.0ms | optim: 22.5ms step 583 | loss: 3.317870 | lr 1.8000e-04 | norm: 0.2577 | dt: 12833.99ms | tok/sec: 40851.5 | mem: 108493MB | Time Left: 00 hrs : 23 mins [TIMING] data: 4.5ms | fwd: 3153.1ms | loss: 6.6ms | bwd: 9639.4ms | clip: 0.0ms | optim: 22.3ms step 584 | loss: 2.836640 | lr 1.8000e-04 | norm: 0.2203 | dt: 12831.46ms | tok/sec: 40859.6 | mem: 108493MB | Time Left: 00 hrs : 23 mins [TIMING] data: 4.6ms | fwd: 3150.7ms | loss: 6.6ms | bwd: 9639.3ms | clip: 0.0ms | optim: 22.3ms step 585 | loss: 2.742846 | lr 1.8000e-04 | norm: 0.2082 | dt: 12819.61ms | tok/sec: 40897.3 | mem: 108493MB | Time Left: 00 hrs : 23 mins [TIMING] data: 4.5ms | fwd: 3154.2ms | loss: 6.6ms | bwd: 9624.0ms | clip: 0.0ms | optim: 22.3ms step 586 | loss: 2.589376 | lr 1.8000e-04 | norm: 0.2210 | dt: 12835.68ms | tok/sec: 40846.1 | mem: 108493MB | Time Left: 00 hrs : 23 mins [TIMING] data: 4.5ms | fwd: 3151.4ms | loss: 6.6ms | bwd: 9643.0ms | clip: 0.0ms | optim: 22.3ms step 587 | loss: 3.204112 | lr 1.8000e-04 | norm: 0.2507 | dt: 12824.26ms | tok/sec: 40882.5 | mem: 108493MB | Time Left: 00 hrs : 23 mins [TIMING] data: 4.5ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9626.4ms | clip: 0.0ms | optim: 22.3ms step 588 | loss: 3.090109 | lr 1.8000e-04 | norm: 0.2117 | dt: 12856.64ms | tok/sec: 40779.5 | mem: 108493MB | Time Left: 00 hrs : 22 mins [TIMING] data: 4.5ms | fwd: 3187.4ms | loss: 6.6ms | bwd: 9628.0ms | clip: 0.0ms | optim: 22.2ms step 589 | loss: 2.799028 | lr 1.8000e-04 | norm: 0.3125 | dt: 12816.21ms | tok/sec: 40908.2 | mem: 108493MB | Time Left: 00 hrs : 22 mins [TIMING] data: 4.5ms | fwd: 3150.1ms | loss: 6.6ms | bwd: 9624.8ms | clip: 0.0ms | optim: 22.2ms step 590 | loss: 2.853396 | lr 1.8000e-04 | norm: 0.2364 | dt: 12833.71ms | tok/sec: 40852.4 | mem: 108493MB | Time Left: 00 hrs : 22 mins [TIMING] data: 4.5ms | fwd: 3155.0ms | loss: 6.6ms | bwd: 9637.3ms | clip: 0.0ms | optim: 22.3ms step 591 | loss: 2.956753 | lr 1.8000e-04 | norm: 0.3409 | dt: 12846.96ms | tok/sec: 40810.3 | mem: 108493MB | Time Left: 00 hrs : 22 mins [TIMING] data: 4.5ms | fwd: 3163.9ms | loss: 6.6ms | bwd: 9641.7ms | clip: 0.0ms | optim: 22.3ms step 592 | loss: 2.986943 | lr 1.8000e-04 | norm: 0.1800 | dt: 12818.27ms | tok/sec: 40901.6 | mem: 108493MB | Time Left: 00 hrs : 22 mins [TIMING] data: 4.6ms | fwd: 3154.7ms | loss: 6.6ms | bwd: 9622.2ms | clip: 0.0ms | optim: 22.3ms step 593 | loss: 2.970083 | lr 1.8000e-04 | norm: 0.2141 | dt: 12823.38ms | tok/sec: 40885.3 | mem: 108493MB | Time Left: 00 hrs : 21 mins [TIMING] data: 4.5ms | fwd: 3158.1ms | loss: 6.6ms | bwd: 9624.0ms | clip: 0.0ms | optim: 22.3ms step 594 | loss: 2.872084 | lr 1.8000e-04 | norm: 0.2311 | dt: 12832.56ms | tok/sec: 40856.1 | mem: 108493MB | Time Left: 00 hrs : 21 mins [TIMING] data: 4.5ms | fwd: 3158.5ms | loss: 6.6ms | bwd: 9632.7ms | clip: 0.0ms | optim: 22.3ms step 595 | loss: 2.956891 | lr 1.8000e-04 | norm: 0.3049 | dt: 12824.63ms | tok/sec: 40881.3 | mem: 108493MB | Time Left: 00 hrs : 21 mins [TIMING] data: 4.5ms | fwd: 3151.0ms | loss: 6.6ms | bwd: 9632.3ms | clip: 0.0ms | optim: 22.4ms step 596 | loss: 3.093858 | lr 1.8000e-04 | norm: 0.2066 | dt: 12819.37ms | tok/sec: 40898.1 | mem: 108493MB | Time Left: 00 hrs : 21 mins [TIMING] data: 4.5ms | fwd: 3148.3ms | loss: 6.6ms | bwd: 9629.8ms | clip: 0.0ms | optim: 22.3ms step 597 | loss: 3.177413 | lr 1.8000e-04 | norm: 0.2074 | dt: 12818.08ms | tok/sec: 40902.2 | mem: 108493MB | Time Left: 00 hrs : 20 mins [TIMING] data: 4.5ms | fwd: 3156.5ms | loss: 6.6ms | bwd: 9620.2ms | clip: 0.0ms | optim: 22.4ms step 598 | loss: 2.889366 | lr 1.8000e-04 | norm: 0.2547 | dt: 12837.82ms | tok/sec: 40839.3 | mem: 108493MB | Time Left: 00 hrs : 20 mins [TIMING] data: 4.4ms | fwd: 3154.9ms | loss: 6.6ms | bwd: 9641.5ms | clip: 0.0ms | optim: 22.4ms step 599 | loss: 3.066301 | lr 1.8000e-04 | norm: 0.1961 | dt: 12869.08ms | tok/sec: 40740.1 | mem: 108493MB | Time Left: 00 hrs : 20 mins [TIMING] data: 4.5ms | fwd: 3199.7ms | loss: 6.6ms | bwd: 9628.0ms | clip: 0.0ms | optim: 22.4ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_600.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_500.ckpt step 600 | loss: 3.262360 | lr 1.8000e-04 | norm: 0.6575 | dt: 12725.32ms | tok/sec: 41200.4 | mem: 108493MB | Time Left: 00 hrs : 20 mins [TIMING] data: 4.5ms | fwd: 3133.9ms | loss: 6.6ms | bwd: 9549.9ms | clip: 0.0ms | optim: 22.4ms step 601 | loss: 3.311826 | lr 1.8000e-04 | norm: 0.2590 | dt: 12757.43ms | tok/sec: 41096.7 | mem: 108493MB | Time Left: 00 hrs : 19 mins [TIMING] data: 4.5ms | fwd: 3141.7ms | loss: 6.6ms | bwd: 9574.4ms | clip: 0.0ms | optim: 22.3ms step 602 | loss: 3.233364 | lr 1.8000e-04 | norm: 0.2659 | dt: 12781.54ms | tok/sec: 41019.2 | mem: 108493MB | Time Left: 00 hrs : 19 mins [TIMING] data: 4.5ms | fwd: 3142.8ms | loss: 6.6ms | bwd: 9597.2ms | clip: 0.0ms | optim: 22.5ms step 603 | loss: 2.953822 | lr 1.8000e-04 | norm: 0.2489 | dt: 12805.34ms | tok/sec: 40942.9 | mem: 108493MB | Time Left: 00 hrs : 19 mins [TIMING] data: 4.5ms | fwd: 3149.7ms | loss: 6.6ms | bwd: 9614.3ms | clip: 0.0ms | optim: 22.4ms step 604 | loss: 3.519025 | lr 1.8000e-04 | norm: 0.2116 | dt: 12804.57ms | tok/sec: 40945.4 | mem: 108493MB | Time Left: 00 hrs : 19 mins [TIMING] data: 4.5ms | fwd: 3147.9ms | loss: 6.6ms | bwd: 9615.1ms | clip: 0.0ms | optim: 22.5ms step 605 | loss: 3.183364 | lr 1.8000e-04 | norm: 0.3202 | dt: 12829.68ms | tok/sec: 40865.2 | mem: 108493MB | Time Left: 00 hrs : 19 mins [TIMING] data: 4.5ms | fwd: 3151.9ms | loss: 6.6ms | bwd: 9635.9ms | clip: 0.0ms | optim: 22.9ms step 606 | loss: 2.871273 | lr 1.8000e-04 | norm: 0.2399 | dt: 12825.69ms | tok/sec: 40877.9 | mem: 108493MB | Time Left: 00 hrs : 19 mins [TIMING] data: 4.5ms | fwd: 3155.4ms | loss: 6.6ms | bwd: 9628.9ms | clip: 0.0ms | optim: 22.3ms step 607 | loss: 2.909288 | lr 1.8000e-04 | norm: 0.2224 | dt: 12834.31ms | tok/sec: 40850.5 | mem: 108493MB | Time Left: 00 hrs : 18 mins [TIMING] data: 4.5ms | fwd: 3152.5ms | loss: 6.6ms | bwd: 9640.2ms | clip: 0.0ms | optim: 22.5ms step 608 | loss: 3.009409 | lr 1.8000e-04 | norm: 0.2420 | dt: 12819.76ms | tok/sec: 40896.9 | mem: 108493MB | Time Left: 00 hrs : 18 mins [TIMING] data: 4.5ms | fwd: 3150.3ms | loss: 6.6ms | bwd: 9627.8ms | clip: 0.0ms | optim: 22.3ms step 609 | loss: 3.028877 | lr 1.8000e-04 | norm: 0.5228 | dt: 12820.97ms | tok/sec: 40893.0 | mem: 108493MB | Time Left: 00 hrs : 18 mins [TIMING] data: 4.6ms | fwd: 3157.3ms | loss: 6.6ms | bwd: 9622.2ms | clip: 0.0ms | optim: 22.3ms step 610 | loss: 2.612901 | lr 1.8000e-04 | norm: 0.2090 | dt: 12831.60ms | tok/sec: 40859.1 | mem: 108493MB | Time Left: 00 hrs : 18 mins [TIMING] data: 4.6ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9635.1ms | clip: 0.0ms | optim: 22.5ms step 611 | loss: 2.919166 | lr 1.8000e-04 | norm: 0.2547 | dt: 12829.63ms | tok/sec: 40865.4 | mem: 108493MB | Time Left: 00 hrs : 17 mins [TIMING] data: 4.5ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9633.3ms | clip: 0.0ms | optim: 22.4ms step 612 | loss: 2.895611 | lr 1.8000e-04 | norm: 0.2450 | dt: 12832.27ms | tok/sec: 40857.0 | mem: 108493MB | Time Left: 00 hrs : 17 mins [TIMING] data: 4.5ms | fwd: 3154.4ms | loss: 6.6ms | bwd: 9636.3ms | clip: 0.0ms | optim: 22.4ms step 613 | loss: 2.919814 | lr 1.8000e-04 | norm: 0.2180 | dt: 12820.25ms | tok/sec: 40895.3 | mem: 108493MB | Time Left: 00 hrs : 17 mins [TIMING] data: 4.5ms | fwd: 3155.6ms | loss: 6.6ms | bwd: 9623.3ms | clip: 0.0ms | optim: 22.3ms step 614 | loss: 3.367930 | lr 1.8000e-04 | norm: 0.9883 | dt: 12885.14ms | tok/sec: 40689.3 | mem: 108493MB | Time Left: 00 hrs : 17 mins [TIMING] data: 4.5ms | fwd: 3216.9ms | loss: 6.6ms | bwd: 9626.8ms | clip: 0.0ms | optim: 22.3ms step 615 | loss: 3.037163 | lr 1.8000e-04 | norm: 0.2701 | dt: 12827.46ms | tok/sec: 40872.3 | mem: 108493MB | Time Left: 00 hrs : 17 mins [TIMING] data: 4.5ms | fwd: 3156.9ms | loss: 6.6ms | bwd: 9629.1ms | clip: 0.0ms | optim: 22.5ms step 616 | loss: 2.878213 | lr 1.8000e-04 | norm: 0.3012 | dt: 12820.41ms | tok/sec: 40894.8 | mem: 108493MB | Time Left: 00 hrs : 16 mins [TIMING] data: 4.5ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9624.5ms | clip: 0.0ms | optim: 22.3ms step 617 | loss: 3.001348 | lr 1.8000e-04 | norm: 0.2480 | dt: 12828.67ms | tok/sec: 40868.5 | mem: 108493MB | Time Left: 00 hrs : 16 mins [TIMING] data: 4.5ms | fwd: 3156.4ms | loss: 6.6ms | bwd: 9630.8ms | clip: 0.0ms | optim: 22.3ms step 618 | loss: 3.113537 | lr 1.8000e-04 | norm: 0.3106 | dt: 12823.77ms | tok/sec: 40884.1 | mem: 108493MB | Time Left: 00 hrs : 16 mins [TIMING] data: 4.5ms | fwd: 3150.2ms | loss: 6.6ms | bwd: 9632.0ms | clip: 0.0ms | optim: 22.5ms step 619 | loss: 2.945028 | lr 1.8000e-04 | norm: 0.2497 | dt: 12820.35ms | tok/sec: 40895.0 | mem: 108493MB | Time Left: 00 hrs : 16 mins [TIMING] data: 4.5ms | fwd: 3153.6ms | loss: 6.6ms | bwd: 9625.4ms | clip: 0.0ms | optim: 22.3ms step 620 | loss: 2.867368 | lr 1.8000e-04 | norm: 0.2537 | dt: 12892.26ms | tok/sec: 40666.9 | mem: 108493MB | Time Left: 00 hrs : 16 mins [TIMING] data: 4.5ms | fwd: 3214.3ms | loss: 6.6ms | bwd: 9636.4ms | clip: 0.0ms | optim: 22.5ms step 621 | loss: 2.674272 | lr 1.8000e-04 | norm: 0.3790 | dt: 12823.36ms | tok/sec: 40885.4 | mem: 108493MB | Time Left: 00 hrs : 15 mins [TIMING] data: 4.5ms | fwd: 3148.6ms | loss: 6.6ms | bwd: 9633.3ms | clip: 0.0ms | optim: 22.4ms step 622 | loss: 2.995434 | lr 1.8000e-04 | norm: 0.2838 | dt: 12826.10ms | tok/sec: 40876.6 | mem: 108493MB | Time Left: 00 hrs : 15 mins [TIMING] data: 4.5ms | fwd: 3153.5ms | loss: 6.6ms | bwd: 9631.2ms | clip: 0.0ms | optim: 22.3ms step 623 | loss: 2.493079 | lr 1.8000e-04 | norm: 0.2584 | dt: 12837.83ms | tok/sec: 40839.3 | mem: 108493MB | Time Left: 00 hrs : 15 mins [TIMING] data: 4.5ms | fwd: 3153.9ms | loss: 6.6ms | bwd: 9642.6ms | clip: 0.0ms | optim: 22.3ms step 624 | loss: 2.898406 | lr 1.8000e-04 | norm: 0.3621 | dt: 12825.32ms | tok/sec: 40879.1 | mem: 108493MB | Time Left: 00 hrs : 15 mins [TIMING] data: 4.5ms | fwd: 3161.7ms | loss: 6.6ms | bwd: 9622.0ms | clip: 0.0ms | optim: 22.6ms step 625 | loss: 2.959695 | lr 1.8000e-04 | norm: 0.2998 | dt: 12860.77ms | tok/sec: 40766.5 | mem: 108493MB | Time Left: 00 hrs : 15 mins [TIMING] data: 4.5ms | fwd: 3182.1ms | loss: 6.6ms | bwd: 9637.0ms | clip: 0.0ms | optim: 22.5ms step 626 | loss: 3.032037 | lr 1.8000e-04 | norm: 0.3388 | dt: 12831.16ms | tok/sec: 40860.5 | mem: 108493MB | Time Left: 00 hrs : 14 mins [TIMING] data: 4.5ms | fwd: 3154.6ms | loss: 6.6ms | bwd: 9635.1ms | clip: 0.0ms | optim: 22.4ms step 627 | loss: 3.088195 | lr 1.8000e-04 | norm: 0.2979 | dt: 12824.94ms | tok/sec: 40880.3 | mem: 108493MB | Time Left: 00 hrs : 14 mins [TIMING] data: 4.5ms | fwd: 3149.7ms | loss: 6.6ms | bwd: 9633.8ms | clip: 0.0ms | optim: 22.4ms step 628 | loss: 3.054765 | lr 1.8000e-04 | norm: 0.2991 | dt: 12830.46ms | tok/sec: 40862.8 | mem: 108493MB | Time Left: 00 hrs : 14 mins [TIMING] data: 4.5ms | fwd: 3154.1ms | loss: 6.6ms | bwd: 9635.1ms | clip: 0.0ms | optim: 22.2ms step 629 | loss: 2.918499 | lr 1.8000e-04 | norm: 0.2317 | dt: 12824.91ms | tok/sec: 40880.5 | mem: 108493MB | Time Left: 00 hrs : 14 mins [TIMING] data: 4.5ms | fwd: 3151.8ms | loss: 6.6ms | bwd: 9631.4ms | clip: 0.0ms | optim: 22.7ms step 630 | loss: 2.715118 | lr 1.8000e-04 | norm: 0.2904 | dt: 12819.70ms | tok/sec: 40897.1 | mem: 108493MB | Time Left: 00 hrs : 13 mins [TIMING] data: 4.5ms | fwd: 3150.1ms | loss: 6.6ms | bwd: 9628.0ms | clip: 0.0ms | optim: 22.5ms step 631 | loss: 2.636536 | lr 1.8000e-04 | norm: 0.3225 | dt: 12829.16ms | tok/sec: 40866.9 | mem: 108493MB | Time Left: 00 hrs : 13 mins [TIMING] data: 4.5ms | fwd: 3153.3ms | loss: 6.6ms | bwd: 9634.5ms | clip: 0.0ms | optim: 22.3ms step 632 | loss: 2.709949 | lr 1.8000e-04 | norm: 0.2816 | dt: 12839.36ms | tok/sec: 40834.4 | mem: 108493MB | Time Left: 00 hrs : 13 mins [TIMING] data: 4.5ms | fwd: 3156.1ms | loss: 6.6ms | bwd: 9641.9ms | clip: 0.0ms | optim: 22.2ms step 633 | loss: 2.875460 | lr 1.8000e-04 | norm: 0.2482 | dt: 12844.18ms | tok/sec: 40819.1 | mem: 108493MB | Time Left: 00 hrs : 13 mins [TIMING] data: 4.5ms | fwd: 3160.0ms | loss: 6.6ms | bwd: 9642.6ms | clip: 0.0ms | optim: 22.5ms step 634 | loss: 2.933730 | lr 1.8000e-04 | norm: 0.2720 | dt: 12826.83ms | tok/sec: 40874.3 | mem: 108493MB | Time Left: 00 hrs : 13 mins [TIMING] data: 4.5ms | fwd: 3156.0ms | loss: 6.6ms | bwd: 9629.3ms | clip: 0.0ms | optim: 22.5ms step 635 | loss: 2.849545 | lr 1.8000e-04 | norm: 0.3651 | dt: 12827.70ms | tok/sec: 40871.6 | mem: 108493MB | Time Left: 00 hrs : 12 mins [TIMING] data: 4.5ms | fwd: 3151.4ms | loss: 6.6ms | bwd: 9633.9ms | clip: 0.0ms | optim: 22.3ms step 636 | loss: 2.623607 | lr 1.8000e-04 | norm: 0.2115 | dt: 12822.83ms | tok/sec: 40887.1 | mem: 108493MB | Time Left: 00 hrs : 12 mins [TIMING] data: 4.5ms | fwd: 3152.0ms | loss: 6.6ms | bwd: 9629.6ms | clip: 0.0ms | optim: 22.2ms step 637 | loss: 2.809004 | lr 1.8000e-04 | norm: 0.4146 | dt: 12837.69ms | tok/sec: 40839.7 | mem: 108493MB | Time Left: 00 hrs : 12 mins [TIMING] data: 4.6ms | fwd: 3155.9ms | loss: 6.6ms | bwd: 9640.4ms | clip: 0.0ms | optim: 22.4ms step 638 | loss: 2.987421 | lr 1.8000e-04 | norm: 0.2570 | dt: 12837.41ms | tok/sec: 40840.6 | mem: 108493MB | Time Left: 00 hrs : 12 mins [TIMING] data: 4.5ms | fwd: 3153.7ms | loss: 6.6ms | bwd: 9642.4ms | clip: 0.0ms | optim: 22.3ms step 639 | loss: 2.930867 | lr 1.8000e-04 | norm: 0.5802 | dt: 12889.62ms | tok/sec: 40675.2 | mem: 108493MB | Time Left: 00 hrs : 12 mins [TIMING] data: 4.5ms | fwd: 3215.8ms | loss: 6.6ms | bwd: 9632.2ms | clip: 0.0ms | optim: 22.5ms step 640 | loss: 3.114867 | lr 1.8000e-04 | norm: 0.2228 | dt: 12824.38ms | tok/sec: 40882.1 | mem: 108493MB | Time Left: 00 hrs : 11 mins [TIMING] data: 4.6ms | fwd: 3154.8ms | loss: 6.6ms | bwd: 9628.0ms | clip: 0.0ms | optim: 22.5ms step 641 | loss: 3.113560 | lr 1.8000e-04 | norm: 0.2224 | dt: 12816.77ms | tok/sec: 40906.4 | mem: 108493MB | Time Left: 00 hrs : 11 mins [TIMING] data: 4.5ms | fwd: 3154.5ms | loss: 6.7ms | bwd: 9620.9ms | clip: 0.0ms | optim: 22.3ms step 642 | loss: 2.964723 | lr 1.8000e-04 | norm: 0.2091 | dt: 12821.20ms | tok/sec: 40892.3 | mem: 108493MB | Time Left: 00 hrs : 11 mins [TIMING] data: 4.5ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9627.2ms | clip: 0.0ms | optim: 22.6ms step 643 | loss: 2.991478 | lr 1.8000e-04 | norm: 0.2615 | dt: 12822.93ms | tok/sec: 40886.8 | mem: 108493MB | Time Left: 00 hrs : 11 mins [TIMING] data: 4.5ms | fwd: 3151.7ms | loss: 6.6ms | bwd: 9629.8ms | clip: 0.0ms | optim: 22.3ms step 644 | loss: 3.176454 | lr 1.8000e-04 | norm: 0.2277 | dt: 12827.79ms | tok/sec: 40871.3 | mem: 108493MB | Time Left: 00 hrs : 10 mins [TIMING] data: 4.5ms | fwd: 3152.2ms | loss: 6.6ms | bwd: 9634.1ms | clip: 0.0ms | optim: 22.4ms step 645 | loss: 3.131259 | lr 1.8000e-04 | norm: 0.2182 | dt: 12828.67ms | tok/sec: 40868.5 | mem: 108493MB | Time Left: 00 hrs : 10 mins [TIMING] data: 4.5ms | fwd: 3151.7ms | loss: 6.6ms | bwd: 9635.7ms | clip: 0.0ms | optim: 22.3ms step 646 | loss: 3.024759 | lr 1.8000e-04 | norm: 0.2534 | dt: 12823.49ms | tok/sec: 40885.0 | mem: 108493MB | Time Left: 00 hrs : 10 mins [TIMING] data: 4.4ms | fwd: 3153.3ms | loss: 6.6ms | bwd: 9628.8ms | clip: 0.0ms | optim: 22.5ms step 647 | loss: 2.978769 | lr 1.8000e-04 | norm: 0.2582 | dt: 12825.43ms | tok/sec: 40878.8 | mem: 108493MB | Time Left: 00 hrs : 10 mins [TIMING] data: 4.5ms | fwd: 3154.2ms | loss: 6.6ms | bwd: 9629.8ms | clip: 0.0ms | optim: 22.3ms step 648 | loss: 3.014819 | lr 1.8000e-04 | norm: 0.2329 | dt: 12837.21ms | tok/sec: 40841.3 | mem: 108493MB | Time Left: 00 hrs : 10 mins [TIMING] data: 4.5ms | fwd: 3154.2ms | loss: 6.6ms | bwd: 9641.6ms | clip: 0.0ms | optim: 22.4ms step 649 | loss: 2.662709 | lr 1.8000e-04 | norm: 0.2510 | dt: 12824.05ms | tok/sec: 40883.2 | mem: 108493MB | Time Left: 00 hrs : 09 mins [TIMING] data: 4.5ms | fwd: 3150.5ms | loss: 6.6ms | bwd: 9632.1ms | clip: 0.0ms | optim: 22.3ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_650.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_550.ckpt step 650 | loss: 2.818289 | lr 1.8000e-04 | norm: 0.2054 | dt: 12741.03ms | tok/sec: 41149.6 | mem: 108493MB | Time Left: 00 hrs : 09 mins [TIMING] data: 4.6ms | fwd: 3148.3ms | loss: 6.7ms | bwd: 9550.7ms | clip: 0.0ms | optim: 22.5ms step 651 | loss: 3.005862 | lr 1.8000e-04 | norm: 0.3002 | dt: 12754.48ms | tok/sec: 41106.2 | mem: 108493MB | Time Left: 00 hrs : 09 mins [TIMING] data: 4.6ms | fwd: 3140.4ms | loss: 6.7ms | bwd: 9572.5ms | clip: 0.0ms | optim: 22.2ms step 652 | loss: 2.940339 | lr 1.8000e-04 | norm: 0.1990 | dt: 12784.14ms | tok/sec: 41010.8 | mem: 108493MB | Time Left: 00 hrs : 09 mins [TIMING] data: 4.6ms | fwd: 3149.9ms | loss: 6.6ms | bwd: 9592.5ms | clip: 0.0ms | optim: 22.4ms step 653 | loss: 3.141171 | lr 1.8000e-04 | norm: 0.4234 | dt: 12805.93ms | tok/sec: 40941.0 | mem: 108493MB | Time Left: 00 hrs : 08 mins [TIMING] data: 4.5ms | fwd: 3145.2ms | loss: 6.6ms | bwd: 9619.2ms | clip: 0.0ms | optim: 22.4ms step 654 | loss: 3.064949 | lr 1.8000e-04 | norm: 0.2742 | dt: 12820.38ms | tok/sec: 40894.9 | mem: 108493MB | Time Left: 00 hrs : 08 mins [TIMING] data: 4.7ms | fwd: 3148.2ms | loss: 6.7ms | bwd: 9630.2ms | clip: 0.0ms | optim: 22.4ms step 655 | loss: 2.907993 | lr 1.8000e-04 | norm: 0.3732 | dt: 12808.00ms | tok/sec: 40934.4 | mem: 108493MB | Time Left: 00 hrs : 08 mins [TIMING] data: 4.6ms | fwd: 3149.5ms | loss: 6.7ms | bwd: 9616.8ms | clip: 0.0ms | optim: 22.3ms step 656 | loss: 2.834852 | lr 1.8000e-04 | norm: 0.8992 | dt: 12820.21ms | tok/sec: 40895.4 | mem: 108493MB | Time Left: 00 hrs : 08 mins [TIMING] data: 4.6ms | fwd: 3148.0ms | loss: 6.7ms | bwd: 9630.6ms | clip: 0.0ms | optim: 22.3ms step 657 | loss: 3.147063 | lr 1.8000e-04 | norm: 0.2469 | dt: 12885.68ms | tok/sec: 40687.7 | mem: 108493MB | Time Left: 00 hrs : 08 mins [TIMING] data: 4.6ms | fwd: 3217.3ms | loss: 6.7ms | bwd: 9626.5ms | clip: 0.0ms | optim: 22.5ms step 658 | loss: 2.999230 | lr 1.8000e-04 | norm: 0.2451 | dt: 12825.07ms | tok/sec: 40879.9 | mem: 108493MB | Time Left: 00 hrs : 07 mins [TIMING] data: 4.5ms | fwd: 3153.5ms | loss: 6.7ms | bwd: 9630.0ms | clip: 0.0ms | optim: 22.2ms step 659 | loss: 2.956033 | lr 1.8000e-04 | norm: 0.2431 | dt: 12819.71ms | tok/sec: 40897.0 | mem: 108493MB | Time Left: 00 hrs : 07 mins [TIMING] data: 4.7ms | fwd: 3153.0ms | loss: 6.7ms | bwd: 9624.9ms | clip: 0.0ms | optim: 22.3ms step 660 | loss: 2.709018 | lr 1.8000e-04 | norm: 0.2567 | dt: 12822.17ms | tok/sec: 40889.2 | mem: 108493MB | Time Left: 00 hrs : 07 mins [TIMING] data: 4.5ms | fwd: 3146.5ms | loss: 6.7ms | bwd: 9634.0ms | clip: 0.0ms | optim: 22.3ms step 661 | loss: 2.743886 | lr 1.8000e-04 | norm: 0.2345 | dt: 12819.74ms | tok/sec: 40896.9 | mem: 108493MB | Time Left: 00 hrs : 07 mins [TIMING] data: 4.5ms | fwd: 3149.7ms | loss: 6.7ms | bwd: 9628.5ms | clip: 0.0ms | optim: 22.3ms step 662 | loss: 3.141176 | lr 1.8000e-04 | norm: 0.2997 | dt: 12820.78ms | tok/sec: 40893.6 | mem: 108493MB | Time Left: 00 hrs : 07 mins [TIMING] data: 4.5ms | fwd: 3153.9ms | loss: 6.7ms | bwd: 9625.3ms | clip: 0.0ms | optim: 22.3ms step 663 | loss: 3.069171 | lr 1.8000e-04 | norm: 0.2923 | dt: 12836.01ms | tok/sec: 40845.1 | mem: 108493MB | Time Left: 00 hrs : 06 mins [TIMING] data: 4.8ms | fwd: 3154.1ms | loss: 6.6ms | bwd: 9640.1ms | clip: 0.0ms | optim: 22.3ms step 664 | loss: 2.821355 | lr 1.8000e-04 | norm: 0.2055 | dt: 12818.64ms | tok/sec: 40900.4 | mem: 108493MB | Time Left: 00 hrs : 06 mins [TIMING] data: 4.8ms | fwd: 3157.9ms | loss: 6.6ms | bwd: 9619.0ms | clip: 0.0ms | optim: 22.3ms step 665 | loss: 2.656325 | lr 1.8000e-04 | norm: 0.2233 | dt: 12826.52ms | tok/sec: 40875.3 | mem: 108493MB | Time Left: 00 hrs : 06 mins [TIMING] data: 4.9ms | fwd: 3155.6ms | loss: 6.7ms | bwd: 9628.5ms | clip: 0.0ms | optim: 22.6ms step 666 | loss: 3.034221 | lr 1.8000e-04 | norm: 0.5326 | dt: 12810.56ms | tok/sec: 40926.2 | mem: 108493MB | Time Left: 00 hrs : 06 mins [TIMING] data: 4.7ms | fwd: 3150.0ms | loss: 6.7ms | bwd: 9618.6ms | clip: 0.0ms | optim: 22.4ms step 667 | loss: 3.004286 | lr 1.8000e-04 | norm: 0.2036 | dt: 12824.45ms | tok/sec: 40881.9 | mem: 108493MB | Time Left: 00 hrs : 05 mins [TIMING] data: 4.8ms | fwd: 3150.5ms | loss: 6.6ms | bwd: 9632.1ms | clip: 0.0ms | optim: 22.3ms step 668 | loss: 2.977492 | lr 1.8000e-04 | norm: 0.2604 | dt: 12815.91ms | tok/sec: 40909.2 | mem: 108493MB | Time Left: 00 hrs : 05 mins [TIMING] data: 4.8ms | fwd: 3153.8ms | loss: 6.7ms | bwd: 9620.2ms | clip: 0.0ms | optim: 22.3ms step 669 | loss: 2.866544 | lr 1.8000e-04 | norm: 0.3023 | dt: 12822.52ms | tok/sec: 40888.1 | mem: 108493MB | Time Left: 00 hrs : 05 mins [TIMING] data: 4.7ms | fwd: 3154.9ms | loss: 6.7ms | bwd: 9625.8ms | clip: 0.0ms | optim: 22.3ms step 670 | loss: 2.734171 | lr 1.8000e-04 | norm: 0.2468 | dt: 12818.13ms | tok/sec: 40902.1 | mem: 108493MB | Time Left: 00 hrs : 05 mins [TIMING] data: 4.8ms | fwd: 3156.2ms | loss: 6.7ms | bwd: 9619.7ms | clip: 0.0ms | optim: 22.6ms step 671 | loss: 3.222540 | lr 1.8000e-04 | norm: 0.2070 | dt: 12831.82ms | tok/sec: 40858.4 | mem: 108493MB | Time Left: 00 hrs : 05 mins [TIMING] data: 4.7ms | fwd: 3167.2ms | loss: 6.7ms | bwd: 9622.7ms | clip: 0.0ms | optim: 22.3ms step 672 | loss: 3.033446 | lr 1.8000e-04 | norm: 0.3280 | dt: 12828.16ms | tok/sec: 40870.1 | mem: 108493MB | Time Left: 00 hrs : 04 mins [TIMING] data: 4.8ms | fwd: 3155.8ms | loss: 6.7ms | bwd: 9630.4ms | clip: 0.0ms | optim: 22.3ms step 673 | loss: 2.413906 | lr 1.8000e-04 | norm: 0.2959 | dt: 12820.37ms | tok/sec: 40894.9 | mem: 108493MB | Time Left: 00 hrs : 04 mins [TIMING] data: 5.0ms | fwd: 3152.9ms | loss: 6.7ms | bwd: 9625.1ms | clip: 0.0ms | optim: 22.3ms step 674 | loss: 2.716961 | lr 1.8000e-04 | norm: 0.2541 | dt: 12825.54ms | tok/sec: 40878.4 | mem: 108493MB | Time Left: 00 hrs : 04 mins [TIMING] data: 4.6ms | fwd: 3153.5ms | loss: 6.7ms | bwd: 9630.3ms | clip: 0.0ms | optim: 22.3ms step 675 | loss: 3.012308 | lr 1.8000e-04 | norm: 0.2809 | dt: 12822.52ms | tok/sec: 40888.1 | mem: 108493MB | Time Left: 00 hrs : 04 mins [TIMING] data: 4.6ms | fwd: 3155.5ms | loss: 6.7ms | bwd: 9625.2ms | clip: 0.0ms | optim: 22.3ms step 676 | loss: 2.914495 | lr 1.8000e-04 | norm: 0.3585 | dt: 12831.34ms | tok/sec: 40859.9 | mem: 108493MB | Time Left: 00 hrs : 04 mins [TIMING] data: 4.7ms | fwd: 3157.0ms | loss: 6.7ms | bwd: 9632.2ms | clip: 0.0ms | optim: 22.5ms step 677 | loss: 2.839561 | lr 1.8000e-04 | norm: 0.2055 | dt: 12840.09ms | tok/sec: 40832.1 | mem: 108493MB | Time Left: 00 hrs : 03 mins [TIMING] data: 4.6ms | fwd: 3152.7ms | loss: 6.8ms | bwd: 9645.3ms | clip: 0.0ms | optim: 22.3ms step 678 | loss: 3.261472 | lr 1.8000e-04 | norm: 0.8851 | dt: 12829.20ms | tok/sec: 40866.8 | mem: 108493MB | Time Left: 00 hrs : 03 mins [TIMING] data: 4.7ms | fwd: 3154.5ms | loss: 6.8ms | bwd: 9632.7ms | clip: 0.0ms | optim: 22.3ms step 679 | loss: 2.501373 | lr 1.8000e-04 | norm: 0.5141 | dt: 12829.17ms | tok/sec: 40866.9 | mem: 108493MB | Time Left: 00 hrs : 03 mins [TIMING] data: 4.7ms | fwd: 3152.9ms | loss: 6.7ms | bwd: 9634.1ms | clip: 0.0ms | optim: 22.4ms step 680 | loss: 2.965237 | lr 1.8000e-04 | norm: 0.3309 | dt: 12874.06ms | tok/sec: 40724.4 | mem: 108493MB | Time Left: 00 hrs : 03 mins [TIMING] data: 4.7ms | fwd: 3201.6ms | loss: 6.7ms | bwd: 9630.1ms | clip: 0.0ms | optim: 22.5ms step 681 | loss: 3.080583 | lr 1.8000e-04 | norm: 0.2719 | dt: 12893.35ms | tok/sec: 40663.4 | mem: 108493MB | Time Left: 00 hrs : 03 mins [TIMING] data: 4.7ms | fwd: 3214.0ms | loss: 6.7ms | bwd: 9637.3ms | clip: 0.0ms | optim: 22.3ms step 682 | loss: 3.004236 | lr 1.8000e-04 | norm: 0.2494 | dt: 12813.51ms | tok/sec: 40916.8 | mem: 108493MB | Time Left: 00 hrs : 02 mins [TIMING] data: 4.5ms | fwd: 3156.0ms | loss: 6.7ms | bwd: 9615.8ms | clip: 0.0ms | optim: 22.3ms step 683 | loss: 3.083542 | lr 1.8000e-04 | norm: 0.2654 | dt: 12829.07ms | tok/sec: 40867.2 | mem: 108493MB | Time Left: 00 hrs : 02 mins [TIMING] data: 4.6ms | fwd: 3156.9ms | loss: 6.6ms | bwd: 9630.5ms | clip: 0.0ms | optim: 22.4ms step 684 | loss: 3.012187 | lr 1.8000e-04 | norm: 0.2331 | dt: 12828.88ms | tok/sec: 40867.8 | mem: 108493MB | Time Left: 00 hrs : 02 mins [TIMING] data: 4.5ms | fwd: 3154.3ms | loss: 6.7ms | bwd: 9632.8ms | clip: 0.0ms | optim: 22.5ms step 685 | loss: 2.688249 | lr 1.8000e-04 | norm: 0.2857 | dt: 12824.09ms | tok/sec: 40883.1 | mem: 108493MB | Time Left: 00 hrs : 02 mins [TIMING] data: 4.6ms | fwd: 3151.9ms | loss: 6.7ms | bwd: 9630.6ms | clip: 0.0ms | optim: 22.4ms step 686 | loss: 2.899711 | lr 1.8000e-04 | norm: 0.2649 | dt: 12833.43ms | tok/sec: 40853.3 | mem: 108493MB | Time Left: 00 hrs : 01 mins [TIMING] data: 4.6ms | fwd: 3155.4ms | loss: 6.7ms | bwd: 9636.4ms | clip: 0.0ms | optim: 22.3ms step 687 | loss: 2.831093 | lr 1.8000e-04 | norm: 0.3451 | dt: 12812.32ms | tok/sec: 40920.6 | mem: 108493MB | Time Left: 00 hrs : 01 mins [TIMING] data: 4.6ms | fwd: 3153.2ms | loss: 6.7ms | bwd: 9617.3ms | clip: 0.0ms | optim: 22.5ms step 688 | loss: 2.804329 | lr 1.8000e-04 | norm: 0.2076 | dt: 12829.94ms | tok/sec: 40864.4 | mem: 108493MB | Time Left: 00 hrs : 01 mins [TIMING] data: 4.6ms | fwd: 3152.3ms | loss: 6.6ms | bwd: 9636.0ms | clip: 0.0ms | optim: 22.4ms step 689 | loss: 3.039609 | lr 1.8000e-04 | norm: 0.2501 | dt: 12835.26ms | tok/sec: 40847.5 | mem: 108493MB | Time Left: 00 hrs : 01 mins [TIMING] data: 4.6ms | fwd: 3153.0ms | loss: 6.7ms | bwd: 9640.2ms | clip: 0.0ms | optim: 22.6ms step 690 | loss: 2.945518 | lr 1.8000e-04 | norm: 0.2377 | dt: 12837.73ms | tok/sec: 40839.6 | mem: 108493MB | Time Left: 00 hrs : 01 mins [TIMING] data: 4.6ms | fwd: 3164.1ms | loss: 6.7ms | bwd: 9631.8ms | clip: 0.0ms | optim: 22.4ms step 691 | loss: 2.836600 | lr 1.8000e-04 | norm: 0.2293 | dt: 12822.31ms | tok/sec: 40888.7 | mem: 108493MB | Time Left: 00 hrs : 00 mins [TIMING] data: 4.6ms | fwd: 3150.0ms | loss: 6.7ms | bwd: 9630.6ms | clip: 0.0ms | optim: 22.3ms step 692 | loss: 2.767101 | lr 1.8000e-04 | norm: 0.2063 | dt: 12820.12ms | tok/sec: 40895.7 | mem: 108493MB | Time Left: 00 hrs : 00 mins [TIMING] data: 4.6ms | fwd: 3154.5ms | loss: 6.7ms | bwd: 9623.8ms | clip: 0.0ms | optim: 22.4ms step 693 | loss: 2.680468 | lr 1.8000e-04 | norm: 0.2045 | dt: 12834.88ms | tok/sec: 40848.7 | mem: 108493MB | Time Left: 00 hrs : 00 mins [TIMING] data: 4.6ms | fwd: 3158.0ms | loss: 6.7ms | bwd: 9635.0ms | clip: 0.0ms | optim: 22.4ms validation loss: 2.9561 step 694 | loss: 2.934074 | lr 1.2000e-04 | norm: 0.2371 | dt: 20804.07ms | tok/sec: 25201.2 | mem: 108493MB | Time Left: 00 hrs : 00 mins [TIMING] data: 4.6ms | fwd: 3157.9ms | loss: 6.7ms | bwd: 9628.7ms | clip: 0.0ms | optim: 22.4ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_695.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_600.ckpt === Training complete ===