Hugging Face's logo

unparallelled
/
BluScriptCP

Safetensors
Model card Files
xet
 Community
BluScriptCP / logs /ext32k /ext.log
unparallelled's picture
unparallelled
Upload folder using huggingface_hub
cbacfbc verified
Raw History Blame Contribute Delete
189 kB
=== Llama Context Parallel Training Script (bluscriptCP) === [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) ================ Configuration (bluscriptCP) ================ d_model/layers : 1536 / 36 q/kv heads : 12 / 4 (G=3, hd=128) ffn_hidden : 4096 (SwiGLU) context_length : 32768 vocab_size : 50304 tying=0 B x T : 1 x 32768 world_size : 2 (dp=1 x cp=2) CP_ATTN_MODE : ring (load_balance=HeadTail) global_batch : 524288 (grad_accum=16) precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] max_lr/min_lr : 0.0006 / 6e-05 warmup/steps : 100 / 750 (WSD: stable=550, decay tail=100) checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp) ============================================================= Initializing model on CUDA device 0... [YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693 [InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/lenbucket_l2_blumodelcp_run2_step_400.ckpt (base step 400, loss 2.79415) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP Parameters: 1060624896 Parameters per GPU: 1060624896 max_steps: 750 warmup_steps: 100 [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] [ZeRO-1] DataParallel is DISABLED (both would double-reduce) [precision] bf16 params=217, all with fp32 master weights found 6 shards for split train found 1 shards for split val Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (run 2) [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base) Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 1) [re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp) Starting training... [CP ring P2P] OVERLAP: dedicated non-blocking stream validation loss: 3.2605 [CP bwd] overlap=ON step 0 | loss: 3.091515 | lr 1.8000e-06 | norm: 0.4821 | dt: 22251.54ms | tok/sec: 23561.9 | mem: 107275MB | Time Left: 04 hrs : 38 mins [TIMING] data: 7.2ms | fwd: 4388.1ms | loss: 13.2ms | bwd: 11046.9ms | clip: 0.0ms | optim: 24.4ms step 1 | loss: 2.851759 | lr 3.6000e-06 | norm: 0.4942 | dt: 15109.42ms | tok/sec: 34699.4 | mem: 108811MB | Time Left: 03 hrs : 08 mins [TIMING] data: 8.5ms | fwd: 4285.8ms | loss: 13.2ms | bwd: 10764.1ms | clip: 0.0ms | optim: 23.8ms step 2 | loss: 3.259181 | lr 5.4000e-06 | norm: 0.5146 | dt: 15152.84ms | tok/sec: 34600.0 | mem: 108811MB | Time Left: 03 hrs : 08 mins [TIMING] data: 7.4ms | fwd: 4317.4ms | loss: 13.1ms | bwd: 10777.0ms | clip: 0.0ms | optim: 23.9ms step 3 | loss: 3.104079 | lr 7.2000e-06 | norm: 0.4620 | dt: 14419.06ms | tok/sec: 36360.8 | mem: 108811MB | Time Left: 02 hrs : 59 mins [TIMING] data: 7.5ms | fwd: 3555.4ms | loss: 13.1ms | bwd: 10804.7ms | clip: 0.0ms | optim: 23.9ms step 4 | loss: 2.992731 | lr 9.0000e-06 | norm: 0.3655 | dt: 14449.92ms | tok/sec: 36283.1 | mem: 108811MB | Time Left: 02 hrs : 59 mins [TIMING] data: 7.4ms | fwd: 3571.0ms | loss: 13.0ms | bwd: 10820.3ms | clip: 0.0ms | optim: 23.9ms step 5 | loss: 2.949207 | lr 1.0800e-05 | norm: 0.3143 | dt: 14492.56ms | tok/sec: 36176.4 | mem: 108811MB | Time Left: 02 hrs : 59 mins [TIMING] data: 7.3ms | fwd: 3570.1ms | loss: 13.0ms | bwd: 10864.1ms | clip: 0.0ms | optim: 24.0ms step 6 | loss: 2.879876 | lr 1.2600e-05 | norm: 0.2052 | dt: 14508.82ms | tok/sec: 36135.8 | mem: 108811MB | Time Left: 02 hrs : 59 mins [TIMING] data: 7.4ms | fwd: 3592.1ms | loss: 13.0ms | bwd: 10858.0ms | clip: 0.0ms | optim: 23.9ms step 7 | loss: 3.127487 | lr 1.4400e-05 | norm: 0.1999 | dt: 14508.29ms | tok/sec: 36137.1 | mem: 108811MB | Time Left: 02 hrs : 59 mins [TIMING] data: 7.2ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10873.0ms | clip: 0.0ms | optim: 23.8ms step 8 | loss: 3.158005 | lr 1.6200e-05 | norm: 0.2220 | dt: 14501.11ms | tok/sec: 36155.0 | mem: 108811MB | Time Left: 02 hrs : 59 mins [TIMING] data: 7.2ms | fwd: 3575.5ms | loss: 13.0ms | bwd: 10867.3ms | clip: 0.0ms | optim: 24.0ms step 9 | loss: 2.938362 | lr 1.8000e-05 | norm: 0.3525 | dt: 14492.75ms | tok/sec: 36175.9 | mem: 108811MB | Time Left: 02 hrs : 58 mins [TIMING] data: 7.2ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10854.7ms | clip: 0.0ms | optim: 23.8ms step 10 | loss: 2.920319 | lr 1.9800e-05 | norm: 0.3592 | dt: 14504.87ms | tok/sec: 36145.6 | mem: 108811MB | Time Left: 02 hrs : 58 mins [TIMING] data: 7.1ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10867.4ms | clip: 0.0ms | optim: 24.0ms step 11 | loss: 2.894566 | lr 2.1600e-05 | norm: 0.2804 | dt: 14494.86ms | tok/sec: 36170.6 | mem: 108811MB | Time Left: 02 hrs : 58 mins [TIMING] data: 7.3ms | fwd: 3571.9ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 24.0ms step 12 | loss: 3.158563 | lr 2.3400e-05 | norm: 0.2971 | dt: 14498.34ms | tok/sec: 36161.9 | mem: 108811MB | Time Left: 02 hrs : 58 mins [TIMING] data: 7.1ms | fwd: 3574.5ms | loss: 13.1ms | bwd: 10865.7ms | clip: 0.0ms | optim: 23.9ms step 13 | loss: 2.949954 | lr 2.5200e-05 | norm: 0.2729 | dt: 14487.26ms | tok/sec: 36189.6 | mem: 108811MB | Time Left: 02 hrs : 57 mins [TIMING] data: 7.1ms | fwd: 3586.9ms | loss: 13.2ms | bwd: 10841.8ms | clip: 0.0ms | optim: 24.0ms step 14 | loss: 3.281897 | lr 2.7000e-05 | norm: 0.2473 | dt: 14486.78ms | tok/sec: 36190.8 | mem: 108811MB | Time Left: 02 hrs : 57 mins [TIMING] data: 7.1ms | fwd: 3596.4ms | loss: 13.1ms | bwd: 10832.2ms | clip: 0.0ms | optim: 23.9ms step 15 | loss: 2.717209 | lr 2.8800e-05 | norm: 0.2128 | dt: 14492.40ms | tok/sec: 36176.7 | mem: 108811MB | Time Left: 02 hrs : 57 mins [TIMING] data: 7.2ms | fwd: 3587.9ms | loss: 13.1ms | bwd: 10846.1ms | clip: 0.0ms | optim: 24.1ms step 16 | loss: 2.953643 | lr 3.0600e-05 | norm: 1.7032 | dt: 14493.01ms | tok/sec: 36175.2 | mem: 108811MB | Time Left: 02 hrs : 57 mins [TIMING] data: 7.1ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10858.8ms | clip: 0.0ms | optim: 23.8ms step 17 | loss: 2.865139 | lr 3.2400e-05 | norm: 0.1980 | dt: 14513.24ms | tok/sec: 36124.8 | mem: 108811MB | Time Left: 02 hrs : 57 mins [TIMING] data: 7.1ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10879.3ms | clip: 0.0ms | optim: 24.0ms step 18 | loss: 2.843351 | lr 3.4200e-05 | norm: 0.2114 | dt: 14498.66ms | tok/sec: 36161.1 | mem: 108811MB | Time Left: 02 hrs : 56 mins [TIMING] data: 7.1ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10861.1ms | clip: 0.0ms | optim: 24.0ms step 19 | loss: 3.087666 | lr 3.6000e-05 | norm: 0.2159 | dt: 14508.12ms | tok/sec: 36137.6 | mem: 108811MB | Time Left: 02 hrs : 56 mins [TIMING] data: 7.2ms | fwd: 3573.7ms | loss: 13.0ms | bwd: 10875.9ms | clip: 0.0ms | optim: 24.1ms step 20 | loss: 2.901704 | lr 3.7800e-05 | norm: 0.1994 | dt: 14478.30ms | tok/sec: 36212.0 | mem: 108811MB | Time Left: 02 hrs : 56 mins [TIMING] data: 7.1ms | fwd: 3571.5ms | loss: 13.1ms | bwd: 10848.8ms | clip: 0.0ms | optim: 23.9ms step 21 | loss: 2.904223 | lr 3.9600e-05 | norm: 0.2292 | dt: 14469.39ms | tok/sec: 36234.3 | mem: 108811MB | Time Left: 02 hrs : 55 mins [TIMING] data: 7.2ms | fwd: 3572.4ms | loss: 13.0ms | bwd: 10838.5ms | clip: 0.0ms | optim: 24.3ms step 22 | loss: 2.614633 | lr 4.1400e-05 | norm: 0.1743 | dt: 14486.56ms | tok/sec: 36191.3 | mem: 108811MB | Time Left: 02 hrs : 55 mins [TIMING] data: 7.1ms | fwd: 3576.6ms | loss: 13.0ms | bwd: 10851.9ms | clip: 0.0ms | optim: 23.9ms step 23 | loss: 2.814205 | lr 4.3200e-05 | norm: 0.1738 | dt: 14475.65ms | tok/sec: 36218.6 | mem: 108811MB | Time Left: 02 hrs : 55 mins [TIMING] data: 7.3ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10842.1ms | clip: 0.0ms | optim: 24.0ms step 24 | loss: 2.763636 | lr 4.5000e-05 | norm: 0.1923 | dt: 14475.62ms | tok/sec: 36218.7 | mem: 108811MB | Time Left: 02 hrs : 55 mins [TIMING] data: 7.1ms | fwd: 3573.3ms | loss: 13.0ms | bwd: 10844.3ms | clip: 0.0ms | optim: 23.9ms step 25 | loss: 3.117971 | lr 4.6800e-05 | norm: 0.1939 | dt: 14475.23ms | tok/sec: 36219.7 | mem: 108811MB | Time Left: 02 hrs : 54 mins [TIMING] data: 7.2ms | fwd: 3572.8ms | loss: 13.1ms | bwd: 10844.1ms | clip: 0.0ms | optim: 24.0ms step 26 | loss: 2.854959 | lr 4.8600e-05 | norm: 0.2082 | dt: 14482.13ms | tok/sec: 36202.4 | mem: 108811MB | Time Left: 02 hrs : 54 mins [TIMING] data: 7.1ms | fwd: 3574.0ms | loss: 13.0ms | bwd: 10850.1ms | clip: 0.0ms | optim: 23.9ms step 27 | loss: 2.828546 | lr 5.0400e-05 | norm: 0.2755 | dt: 14471.36ms | tok/sec: 36229.4 | mem: 108811MB | Time Left: 02 hrs : 54 mins [TIMING] data: 7.1ms | fwd: 3571.4ms | loss: 13.0ms | bwd: 10841.8ms | clip: 0.0ms | optim: 23.9ms step 28 | loss: 3.075350 | lr 5.2200e-05 | norm: 0.1779 | dt: 14476.73ms | tok/sec: 36215.9 | mem: 108811MB | Time Left: 02 hrs : 54 mins [TIMING] data: 7.1ms | fwd: 3570.0ms | loss: 13.1ms | bwd: 10848.5ms | clip: 0.0ms | optim: 24.0ms step 29 | loss: 2.964859 | lr 5.4000e-05 | norm: 0.1770 | dt: 14486.68ms | tok/sec: 36191.0 | mem: 108811MB | Time Left: 02 hrs : 54 mins [TIMING] data: 7.2ms | fwd: 3576.5ms | loss: 13.0ms | bwd: 10852.0ms | clip: 0.0ms | optim: 23.9ms step 30 | loss: 2.877312 | lr 5.5800e-05 | norm: 0.1892 | dt: 14486.66ms | tok/sec: 36191.1 | mem: 108811MB | Time Left: 02 hrs : 53 mins [TIMING] data: 7.1ms | fwd: 3572.1ms | loss: 13.0ms | bwd: 10856.4ms | clip: 0.0ms | optim: 24.0ms step 31 | loss: 2.797316 | lr 5.7600e-05 | norm: 0.1948 | dt: 14472.09ms | tok/sec: 36227.5 | mem: 108811MB | Time Left: 02 hrs : 53 mins [TIMING] data: 7.1ms | fwd: 3574.0ms | loss: 13.0ms | bwd: 10840.0ms | clip: 0.0ms | optim: 24.0ms step 32 | loss: 2.903166 | lr 5.9400e-05 | norm: 0.1901 | dt: 14474.17ms | tok/sec: 36222.3 | mem: 108811MB | Time Left: 02 hrs : 53 mins [TIMING] data: 7.2ms | fwd: 3570.2ms | loss: 13.0ms | bwd: 10845.8ms | clip: 0.0ms | optim: 23.9ms step 33 | loss: 2.820713 | lr 6.1200e-05 | norm: 0.2335 | dt: 14478.50ms | tok/sec: 36211.5 | mem: 108811MB | Time Left: 02 hrs : 53 mins [TIMING] data: 7.3ms | fwd: 3570.2ms | loss: 13.0ms | bwd: 10849.9ms | clip: 0.0ms | optim: 24.1ms step 34 | loss: 2.632634 | lr 6.3000e-05 | norm: 0.2040 | dt: 14482.02ms | tok/sec: 36202.7 | mem: 108811MB | Time Left: 02 hrs : 52 mins [TIMING] data: 7.1ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.9ms step 35 | loss: 2.763508 | lr 6.4800e-05 | norm: 0.3384 | dt: 14471.97ms | tok/sec: 36227.8 | mem: 108811MB | Time Left: 02 hrs : 52 mins [TIMING] data: 7.1ms | fwd: 3573.6ms | loss: 13.0ms | bwd: 10840.3ms | clip: 0.0ms | optim: 24.0ms step 36 | loss: 2.926087 | lr 6.6600e-05 | norm: 0.1989 | dt: 14472.83ms | tok/sec: 36225.7 | mem: 108811MB | Time Left: 02 hrs : 52 mins [TIMING] data: 7.2ms | fwd: 3575.8ms | loss: 13.0ms | bwd: 10838.8ms | clip: 0.0ms | optim: 23.9ms step 37 | loss: 2.925209 | lr 6.8400e-05 | norm: 0.1890 | dt: 14473.41ms | tok/sec: 36224.2 | mem: 108811MB | Time Left: 02 hrs : 51 mins [TIMING] data: 7.1ms | fwd: 3572.0ms | loss: 13.1ms | bwd: 10843.0ms | clip: 0.0ms | optim: 24.0ms step 38 | loss: 3.109977 | lr 7.0200e-05 | norm: 0.2166 | dt: 14476.45ms | tok/sec: 36216.6 | mem: 108811MB | Time Left: 02 hrs : 51 mins [TIMING] data: 7.2ms | fwd: 3572.1ms | loss: 13.1ms | bwd: 10846.1ms | clip: 0.0ms | optim: 24.0ms step 39 | loss: 2.732811 | lr 7.2000e-05 | norm: 0.2445 | dt: 14470.13ms | tok/sec: 36232.4 | mem: 108811MB | Time Left: 02 hrs : 51 mins [TIMING] data: 7.2ms | fwd: 3572.6ms | loss: 13.1ms | bwd: 10837.8ms | clip: 0.0ms | optim: 24.0ms step 40 | loss: 3.214739 | lr 7.3800e-05 | norm: 0.2220 | dt: 14465.75ms | tok/sec: 36243.4 | mem: 108811MB | Time Left: 02 hrs : 51 mins [TIMING] data: 7.4ms | fwd: 3575.1ms | loss: 13.2ms | bwd: 10831.5ms | clip: 0.0ms | optim: 24.0ms step 41 | loss: 2.953046 | lr 7.5600e-05 | norm: 0.2080 | dt: 14464.82ms | tok/sec: 36245.7 | mem: 108811MB | Time Left: 02 hrs : 50 mins [TIMING] data: 7.3ms | fwd: 3575.4ms | loss: 13.2ms | bwd: 10830.0ms | clip: 0.0ms | optim: 24.0ms step 42 | loss: 3.236126 | lr 7.7400e-05 | norm: 0.2003 | dt: 14476.85ms | tok/sec: 36215.6 | mem: 108811MB | Time Left: 02 hrs : 50 mins [TIMING] data: 7.2ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10841.8ms | clip: 0.0ms | optim: 23.9ms step 43 | loss: 3.199827 | lr 7.9200e-05 | norm: 0.1998 | dt: 14469.27ms | tok/sec: 36234.6 | mem: 108811MB | Time Left: 02 hrs : 50 mins [TIMING] data: 7.1ms | fwd: 3574.6ms | loss: 13.2ms | bwd: 10836.1ms | clip: 0.0ms | optim: 23.9ms step 44 | loss: 2.781108 | lr 8.1000e-05 | norm: 0.1965 | dt: 14477.67ms | tok/sec: 36213.5 | mem: 108811MB | Time Left: 02 hrs : 50 mins [TIMING] data: 7.3ms | fwd: 3573.2ms | loss: 13.3ms | bwd: 10845.6ms | clip: 0.0ms | optim: 24.0ms step 45 | loss: 3.055509 | lr 8.2800e-05 | norm: 0.1981 | dt: 14493.04ms | tok/sec: 36175.2 | mem: 108811MB | Time Left: 02 hrs : 50 mins [TIMING] data: 7.2ms | fwd: 3572.5ms | loss: 13.2ms | bwd: 10861.9ms | clip: 0.0ms | optim: 24.0ms step 46 | loss: 3.055510 | lr 8.4600e-05 | norm: 0.2144 | dt: 14475.73ms | tok/sec: 36218.4 | mem: 108811MB | Time Left: 02 hrs : 49 mins [TIMING] data: 7.2ms | fwd: 3574.3ms | loss: 13.2ms | bwd: 10842.8ms | clip: 0.0ms | optim: 23.9ms step 47 | loss: 2.951757 | lr 8.6400e-05 | norm: 0.2100 | dt: 14473.03ms | tok/sec: 36225.2 | mem: 108811MB | Time Left: 02 hrs : 49 mins [TIMING] data: 7.3ms | fwd: 3572.2ms | loss: 13.2ms | bwd: 10842.3ms | clip: 0.0ms | optim: 23.9ms step 48 | loss: 2.824059 | lr 8.8200e-05 | norm: 0.2159 | dt: 14477.98ms | tok/sec: 36212.8 | mem: 108811MB | Time Left: 02 hrs : 49 mins [TIMING] data: 7.2ms | fwd: 3576.4ms | loss: 13.2ms | bwd: 10842.9ms | clip: 0.0ms | optim: 23.9ms step 49 | loss: 3.061847 | lr 9.0000e-05 | norm: 0.2022 | dt: 14485.08ms | tok/sec: 36195.0 | mem: 108811MB | Time Left: 02 hrs : 49 mins [TIMING] data: 7.2ms | fwd: 3578.4ms | loss: 13.2ms | bwd: 10848.1ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_50.ckpt (safely) step 50 | loss: 2.572393 | lr 9.1800e-05 | norm: 0.1948 | dt: 14397.72ms | tok/sec: 36414.7 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3567.3ms | loss: 13.1ms | bwd: 10770.1ms | clip: 0.0ms | optim: 23.9ms step 51 | loss: 3.059585 | lr 9.3600e-05 | norm: 0.1986 | dt: 14420.27ms | tok/sec: 36357.7 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3561.5ms | loss: 13.1ms | bwd: 10798.7ms | clip: 0.0ms | optim: 24.0ms step 52 | loss: 3.057576 | lr 9.5400e-05 | norm: 0.1780 | dt: 14430.12ms | tok/sec: 36332.9 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3562.8ms | loss: 13.0ms | bwd: 10807.2ms | clip: 0.0ms | optim: 23.9ms step 53 | loss: 2.891887 | lr 9.7200e-05 | norm: 0.1853 | dt: 14436.10ms | tok/sec: 36317.8 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.0ms | fwd: 3567.1ms | loss: 13.1ms | bwd: 10807.6ms | clip: 0.0ms | optim: 24.5ms step 54 | loss: 2.973932 | lr 9.9000e-05 | norm: 0.1843 | dt: 14453.46ms | tok/sec: 36274.2 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.2ms | fwd: 3569.0ms | loss: 13.1ms | bwd: 10824.3ms | clip: 0.0ms | optim: 23.9ms step 55 | loss: 3.288378 | lr 1.0080e-04 | norm: 0.2130 | dt: 14468.28ms | tok/sec: 36237.1 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.3ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10831.3ms | clip: 0.0ms | optim: 23.9ms step 56 | loss: 3.121540 | lr 1.0260e-04 | norm: 0.2297 | dt: 14465.91ms | tok/sec: 36243.0 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.0ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10830.8ms | clip: 0.0ms | optim: 23.9ms step 57 | loss: 3.073643 | lr 1.0440e-04 | norm: 0.2345 | dt: 14477.42ms | tok/sec: 36214.2 | mem: 108811MB | Time Left: 02 hrs : 47 mins [TIMING] data: 9.0ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10839.7ms | clip: 0.0ms | optim: 24.0ms step 58 | loss: 3.167919 | lr 1.0620e-04 | norm: 0.3769 | dt: 14467.23ms | tok/sec: 36239.7 | mem: 108811MB | Time Left: 02 hrs : 46 mins [TIMING] data: 9.1ms | fwd: 3577.7ms | loss: 13.0ms | bwd: 10829.6ms | clip: 0.0ms | optim: 24.0ms step 59 | loss: 2.916999 | lr 1.0800e-04 | norm: 0.2414 | dt: 14473.25ms | tok/sec: 36224.6 | mem: 108811MB | Time Left: 02 hrs : 46 mins [TIMING] data: 9.1ms | fwd: 3568.1ms | loss: 13.1ms | bwd: 10843.7ms | clip: 0.0ms | optim: 24.9ms step 60 | loss: 2.866323 | lr 1.0980e-04 | norm: 0.2040 | dt: 14469.64ms | tok/sec: 36233.6 | mem: 108811MB | Time Left: 02 hrs : 46 mins [TIMING] data: 9.0ms | fwd: 3566.9ms | loss: 13.0ms | bwd: 10842.7ms | clip: 0.0ms | optim: 24.0ms step 61 | loss: 3.157508 | lr 1.1160e-04 | norm: 0.2537 | dt: 14483.29ms | tok/sec: 36199.5 | mem: 108811MB | Time Left: 02 hrs : 46 mins [TIMING] data: 9.2ms | fwd: 3574.9ms | loss: 13.1ms | bwd: 10848.1ms | clip: 0.0ms | optim: 24.0ms step 62 | loss: 2.821198 | lr 1.1340e-04 | norm: 0.2101 | dt: 14468.46ms | tok/sec: 36236.6 | mem: 108811MB | Time Left: 02 hrs : 45 mins [TIMING] data: 9.1ms | fwd: 3577.0ms | loss: 13.0ms | bwd: 10831.3ms | clip: 0.0ms | optim: 23.9ms step 63 | loss: 3.104165 | lr 1.1520e-04 | norm: 0.2779 | dt: 14470.21ms | tok/sec: 36232.2 | mem: 108811MB | Time Left: 02 hrs : 45 mins [TIMING] data: 9.3ms | fwd: 3569.5ms | loss: 13.1ms | bwd: 10839.9ms | clip: 0.0ms | optim: 24.0ms step 64 | loss: 2.965489 | lr 1.1700e-04 | norm: 0.2312 | dt: 14479.35ms | tok/sec: 36209.4 | mem: 108811MB | Time Left: 02 hrs : 45 mins [TIMING] data: 9.2ms | fwd: 3569.7ms | loss: 13.0ms | bwd: 10849.3ms | clip: 0.0ms | optim: 24.0ms step 65 | loss: 2.850994 | lr 1.1880e-04 | norm: 0.2566 | dt: 14478.88ms | tok/sec: 36210.5 | mem: 108811MB | Time Left: 02 hrs : 45 mins [TIMING] data: 9.1ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10841.0ms | clip: 0.0ms | optim: 23.9ms step 66 | loss: 2.613414 | lr 1.2060e-04 | norm: 0.2928 | dt: 14468.09ms | tok/sec: 36237.5 | mem: 108811MB | Time Left: 02 hrs : 44 mins [TIMING] data: 9.0ms | fwd: 3570.2ms | loss: 13.1ms | bwd: 10837.8ms | clip: 0.0ms | optim: 24.0ms step 67 | loss: 2.956029 | lr 1.2240e-04 | norm: 0.2368 | dt: 14478.24ms | tok/sec: 36212.1 | mem: 108811MB | Time Left: 02 hrs : 44 mins [TIMING] data: 9.5ms | fwd: 3574.6ms | loss: 13.1ms | bwd: 10842.7ms | clip: 0.0ms | optim: 24.0ms step 68 | loss: 3.065088 | lr 1.2420e-04 | norm: 0.2384 | dt: 14492.58ms | tok/sec: 36176.3 | mem: 108811MB | Time Left: 02 hrs : 44 mins [TIMING] data: 9.1ms | fwd: 3573.0ms | loss: 13.1ms | bwd: 10859.3ms | clip: 0.0ms | optim: 23.9ms step 69 | loss: 2.923152 | lr 1.2600e-04 | norm: 0.2359 | dt: 14481.93ms | tok/sec: 36202.9 | mem: 108811MB | Time Left: 02 hrs : 44 mins [TIMING] data: 9.1ms | fwd: 3576.2ms | loss: 13.0ms | bwd: 10845.8ms | clip: 0.0ms | optim: 23.9ms step 70 | loss: 3.127590 | lr 1.2780e-04 | norm: 0.3109 | dt: 14482.50ms | tok/sec: 36201.5 | mem: 108811MB | Time Left: 02 hrs : 44 mins [TIMING] data: 9.0ms | fwd: 3582.0ms | loss: 13.0ms | bwd: 10840.7ms | clip: 0.0ms | optim: 23.9ms step 71 | loss: 2.841836 | lr 1.2960e-04 | norm: 0.2507 | dt: 14483.81ms | tok/sec: 36198.2 | mem: 108811MB | Time Left: 02 hrs : 43 mins [TIMING] data: 9.1ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10849.0ms | clip: 0.0ms | optim: 23.9ms step 72 | loss: 2.963378 | lr 1.3140e-04 | norm: 0.3071 | dt: 14489.60ms | tok/sec: 36183.8 | mem: 108811MB | Time Left: 02 hrs : 43 mins [TIMING] data: 9.0ms | fwd: 3572.8ms | loss: 13.0ms | bwd: 10856.0ms | clip: 0.0ms | optim: 24.0ms step 73 | loss: 2.863774 | lr 1.3320e-04 | norm: 0.2023 | dt: 14498.36ms | tok/sec: 36161.9 | mem: 108811MB | Time Left: 02 hrs : 43 mins [TIMING] data: 9.3ms | fwd: 3578.3ms | loss: 13.2ms | bwd: 10858.8ms | clip: 0.0ms | optim: 23.9ms step 74 | loss: 2.798149 | lr 1.3500e-04 | norm: 0.2158 | dt: 14486.03ms | tok/sec: 36192.7 | mem: 108811MB | Time Left: 02 hrs : 43 mins [TIMING] data: 9.1ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10850.3ms | clip: 0.0ms | optim: 23.9ms step 75 | loss: 3.111937 | lr 1.3680e-04 | norm: 0.2757 | dt: 14494.38ms | tok/sec: 36171.8 | mem: 108811MB | Time Left: 02 hrs : 43 mins [TIMING] data: 9.1ms | fwd: 3572.9ms | loss: 13.1ms | bwd: 10861.1ms | clip: 0.0ms | optim: 23.9ms step 76 | loss: 2.770065 | lr 1.3860e-04 | norm: 0.2077 | dt: 14502.11ms | tok/sec: 36152.5 | mem: 108811MB | Time Left: 02 hrs : 42 mins [TIMING] data: 8.9ms | fwd: 3578.2ms | loss: 13.0ms | bwd: 10864.3ms | clip: 0.0ms | optim: 23.9ms step 77 | loss: 2.866266 | lr 1.4040e-04 | norm: 0.2363 | dt: 14474.14ms | tok/sec: 36222.4 | mem: 108811MB | Time Left: 02 hrs : 42 mins [TIMING] data: 9.2ms | fwd: 3591.8ms | loss: 13.1ms | bwd: 10821.9ms | clip: 0.0ms | optim: 24.0ms step 78 | loss: 2.833569 | lr 1.4220e-04 | norm: 0.2593 | dt: 14479.81ms | tok/sec: 36208.2 | mem: 108811MB | Time Left: 02 hrs : 42 mins [TIMING] data: 9.2ms | fwd: 3586.4ms | loss: 13.1ms | bwd: 10832.8ms | clip: 0.0ms | optim: 23.9ms step 79 | loss: 3.001464 | lr 1.4400e-04 | norm: 0.2451 | dt: 14486.94ms | tok/sec: 36190.4 | mem: 108811MB | Time Left: 02 hrs : 42 mins [TIMING] data: 9.2ms | fwd: 3577.0ms | loss: 13.1ms | bwd: 10849.5ms | clip: 0.0ms | optim: 23.9ms step 80 | loss: 2.868001 | lr 1.4580e-04 | norm: 0.2701 | dt: 14555.34ms | tok/sec: 36020.3 | mem: 108811MB | Time Left: 02 hrs : 42 mins [TIMING] data: 9.3ms | fwd: 3664.7ms | loss: 13.0ms | bwd: 10830.4ms | clip: 0.0ms | optim: 23.9ms step 81 | loss: 3.042478 | lr 1.4760e-04 | norm: 0.2703 | dt: 14483.56ms | tok/sec: 36198.8 | mem: 108811MB | Time Left: 02 hrs : 41 mins [TIMING] data: 9.1ms | fwd: 3572.7ms | loss: 13.0ms | bwd: 10850.7ms | clip: 0.0ms | optim: 24.0ms step 82 | loss: 2.968200 | lr 1.4940e-04 | norm: 0.2579 | dt: 14476.76ms | tok/sec: 36215.8 | mem: 108811MB | Time Left: 02 hrs : 41 mins [TIMING] data: 9.0ms | fwd: 3572.4ms | loss: 13.0ms | bwd: 10844.5ms | clip: 0.0ms | optim: 23.9ms step 83 | loss: 2.784357 | lr 1.5120e-04 | norm: 0.2504 | dt: 14553.87ms | tok/sec: 36024.0 | mem: 108811MB | Time Left: 02 hrs : 41 mins [TIMING] data: 9.7ms | fwd: 3661.6ms | loss: 13.1ms | bwd: 10830.6ms | clip: 0.0ms | optim: 23.9ms step 84 | loss: 3.115194 | lr 1.5300e-04 | norm: 0.2383 | dt: 14483.62ms | tok/sec: 36198.7 | mem: 108811MB | Time Left: 02 hrs : 40 mins [TIMING] data: 9.3ms | fwd: 3590.3ms | loss: 13.2ms | bwd: 10832.2ms | clip: 0.0ms | optim: 24.0ms step 85 | loss: 2.826023 | lr 1.5480e-04 | norm: 0.3536 | dt: 14477.77ms | tok/sec: 36213.3 | mem: 108811MB | Time Left: 02 hrs : 40 mins [TIMING] data: 9.2ms | fwd: 3573.9ms | loss: 13.1ms | bwd: 10843.5ms | clip: 0.0ms | optim: 23.9ms step 86 | loss: 3.009665 | lr 1.5660e-04 | norm: 0.2859 | dt: 14479.48ms | tok/sec: 36209.0 | mem: 108811MB | Time Left: 02 hrs : 40 mins [TIMING] data: 9.1ms | fwd: 3572.6ms | loss: 13.1ms | bwd: 10845.9ms | clip: 0.0ms | optim: 24.0ms step 87 | loss: 2.808977 | lr 1.5840e-04 | norm: 0.2653 | dt: 14469.29ms | tok/sec: 36234.5 | mem: 108811MB | Time Left: 02 hrs : 39 mins [TIMING] data: 9.0ms | fwd: 3571.8ms | loss: 13.1ms | bwd: 10837.6ms | clip: 0.0ms | optim: 23.9ms step 88 | loss: 2.802406 | lr 1.6020e-04 | norm: 0.2127 | dt: 14479.48ms | tok/sec: 36209.0 | mem: 108811MB | Time Left: 02 hrs : 39 mins [TIMING] data: 9.0ms | fwd: 3580.2ms | loss: 13.0ms | bwd: 10839.4ms | clip: 0.0ms | optim: 23.9ms step 89 | loss: 3.042083 | lr 1.6200e-04 | norm: 0.2493 | dt: 14477.77ms | tok/sec: 36213.3 | mem: 108811MB | Time Left: 02 hrs : 39 mins [TIMING] data: 9.1ms | fwd: 3569.4ms | loss: 13.1ms | bwd: 10848.3ms | clip: 0.0ms | optim: 24.0ms step 90 | loss: 2.602268 | lr 1.6380e-04 | norm: 0.2396 | dt: 14488.02ms | tok/sec: 36187.7 | mem: 108811MB | Time Left: 02 hrs : 39 mins [TIMING] data: 9.2ms | fwd: 3577.1ms | loss: 13.1ms | bwd: 10849.8ms | clip: 0.0ms | optim: 24.0ms step 91 | loss: 3.112506 | lr 1.6560e-04 | norm: 0.2776 | dt: 14487.64ms | tok/sec: 36188.6 | mem: 108811MB | Time Left: 02 hrs : 39 mins [TIMING] data: 9.0ms | fwd: 3591.0ms | loss: 13.1ms | bwd: 10836.8ms | clip: 0.0ms | optim: 23.9ms step 92 | loss: 3.119073 | lr 1.6740e-04 | norm: 0.2196 | dt: 14480.56ms | tok/sec: 36206.3 | mem: 108811MB | Time Left: 02 hrs : 38 mins [TIMING] data: 9.3ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10837.4ms | clip: 0.0ms | optim: 24.1ms step 93 | loss: 3.359677 | lr 1.6920e-04 | norm: 0.2566 | dt: 14492.00ms | tok/sec: 36177.8 | mem: 108811MB | Time Left: 02 hrs : 38 mins [TIMING] data: 9.2ms | fwd: 3576.5ms | loss: 13.1ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.9ms step 94 | loss: 2.822005 | lr 1.7100e-04 | norm: 0.2302 | dt: 14552.93ms | tok/sec: 36026.3 | mem: 108811MB | Time Left: 02 hrs : 39 mins [TIMING] data: 9.2ms | fwd: 3654.4ms | loss: 13.2ms | bwd: 10837.9ms | clip: 0.0ms | optim: 23.9ms step 95 | loss: 3.042235 | lr 1.7280e-04 | norm: 0.2242 | dt: 14476.55ms | tok/sec: 36216.4 | mem: 108811MB | Time Left: 02 hrs : 38 mins [TIMING] data: 9.1ms | fwd: 3590.8ms | loss: 13.0ms | bwd: 10825.6ms | clip: 0.0ms | optim: 24.0ms step 96 | loss: 2.946624 | lr 1.7460e-04 | norm: 0.2614 | dt: 14468.57ms | tok/sec: 36236.3 | mem: 108811MB | Time Left: 02 hrs : 37 mins [TIMING] data: 9.0ms | fwd: 3581.2ms | loss: 13.0ms | bwd: 10827.4ms | clip: 0.0ms | optim: 24.0ms step 97 | loss: 3.272624 | lr 1.7640e-04 | norm: 0.2919 | dt: 14554.24ms | tok/sec: 36023.0 | mem: 108811MB | Time Left: 02 hrs : 38 mins [TIMING] data: 9.1ms | fwd: 3661.4ms | loss: 13.1ms | bwd: 10832.0ms | clip: 0.0ms | optim: 24.0ms step 98 | loss: 2.955138 | lr 1.7820e-04 | norm: 0.2353 | dt: 14476.23ms | tok/sec: 36217.2 | mem: 108811MB | Time Left: 02 hrs : 37 mins [TIMING] data: 9.9ms | fwd: 3575.6ms | loss: 13.0ms | bwd: 10839.8ms | clip: 0.0ms | optim: 24.0ms step 99 | loss: 2.831222 | lr 1.8000e-04 | norm: 0.2629 | dt: 14487.44ms | tok/sec: 36189.2 | mem: 108811MB | Time Left: 02 hrs : 37 mins [TIMING] data: 9.2ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10848.5ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_100.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_50.ckpt step 100 | loss: 3.013002 | lr 1.8000e-04 | norm: 0.2262 | dt: 14395.20ms | tok/sec: 36421.0 | mem: 108811MB | Time Left: 02 hrs : 35 mins [TIMING] data: 9.2ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10755.1ms | clip: 0.0ms | optim: 24.0ms step 101 | loss: 3.068043 | lr 1.8000e-04 | norm: 0.2128 | dt: 14414.08ms | tok/sec: 36373.3 | mem: 108811MB | Time Left: 02 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3565.1ms | loss: 13.1ms | bwd: 10788.7ms | clip: 0.0ms | optim: 24.0ms step 102 | loss: 3.026262 | lr 1.8000e-04 | norm: 0.2226 | dt: 14429.20ms | tok/sec: 36335.2 | mem: 108811MB | Time Left: 02 hrs : 35 mins [TIMING] data: 9.0ms | fwd: 3568.7ms | loss: 13.1ms | bwd: 10800.5ms | clip: 0.0ms | optim: 23.9ms step 103 | loss: 2.801084 | lr 1.8000e-04 | norm: 0.2357 | dt: 14513.34ms | tok/sec: 36124.6 | mem: 108811MB | Time Left: 02 hrs : 36 mins [TIMING] data: 9.4ms | fwd: 3652.1ms | loss: 13.2ms | bwd: 10800.1ms | clip: 0.0ms | optim: 24.0ms step 104 | loss: 3.028031 | lr 1.8000e-04 | norm: 0.4012 | dt: 14453.55ms | tok/sec: 36274.0 | mem: 108811MB | Time Left: 02 hrs : 35 mins [TIMING] data: 9.3ms | fwd: 3574.6ms | loss: 13.2ms | bwd: 10818.2ms | clip: 0.0ms | optim: 24.0ms step 105 | loss: 2.914334 | lr 1.8000e-04 | norm: 0.2283 | dt: 14480.90ms | tok/sec: 36205.5 | mem: 108811MB | Time Left: 02 hrs : 35 mins [TIMING] data: 10.1ms | fwd: 3595.8ms | loss: 13.2ms | bwd: 10823.2ms | clip: 0.0ms | optim: 24.1ms step 106 | loss: 3.032989 | lr 1.8000e-04 | norm: 0.2933 | dt: 14473.50ms | tok/sec: 36224.0 | mem: 108811MB | Time Left: 02 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3572.4ms | loss: 13.2ms | bwd: 10840.8ms | clip: 0.0ms | optim: 23.9ms step 107 | loss: 2.987231 | lr 1.8000e-04 | norm: 0.2883 | dt: 14472.72ms | tok/sec: 36225.9 | mem: 108811MB | Time Left: 02 hrs : 35 mins [TIMING] data: 9.0ms | fwd: 3572.8ms | loss: 13.1ms | bwd: 10839.1ms | clip: 0.0ms | optim: 24.0ms step 108 | loss: 3.238133 | lr 1.8000e-04 | norm: 0.3757 | dt: 14485.06ms | tok/sec: 36195.1 | mem: 108811MB | Time Left: 02 hrs : 34 mins [TIMING] data: 9.1ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10848.3ms | clip: 0.0ms | optim: 23.9ms step 109 | loss: 3.145993 | lr 1.8000e-04 | norm: 0.3201 | dt: 14485.73ms | tok/sec: 36193.4 | mem: 108811MB | Time Left: 02 hrs : 34 mins [TIMING] data: 9.0ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10845.5ms | clip: 0.0ms | optim: 23.9ms step 110 | loss: 2.788668 | lr 1.8000e-04 | norm: 0.2600 | dt: 14487.27ms | tok/sec: 36189.6 | mem: 108811MB | Time Left: 02 hrs : 34 mins [TIMING] data: 9.1ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10852.0ms | clip: 0.0ms | optim: 23.9ms step 111 | loss: 2.545287 | lr 1.8000e-04 | norm: 0.2748 | dt: 14485.37ms | tok/sec: 36194.3 | mem: 108811MB | Time Left: 02 hrs : 34 mins [TIMING] data: 9.2ms | fwd: 3591.9ms | loss: 13.1ms | bwd: 10833.1ms | clip: 0.0ms | optim: 24.0ms step 112 | loss: 3.166708 | lr 1.8000e-04 | norm: 0.2714 | dt: 14483.69ms | tok/sec: 36198.5 | mem: 108811MB | Time Left: 02 hrs : 34 mins [TIMING] data: 9.2ms | fwd: 3590.9ms | loss: 13.1ms | bwd: 10832.5ms | clip: 0.0ms | optim: 24.0ms step 113 | loss: 3.012598 | lr 1.8000e-04 | norm: 0.2588 | dt: 14493.63ms | tok/sec: 36173.7 | mem: 108811MB | Time Left: 02 hrs : 33 mins [TIMING] data: 9.1ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10853.8ms | clip: 0.0ms | optim: 23.9ms step 114 | loss: 3.001585 | lr 1.8000e-04 | norm: 0.2338 | dt: 14480.50ms | tok/sec: 36206.5 | mem: 108811MB | Time Left: 02 hrs : 33 mins [TIMING] data: 9.1ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10841.9ms | clip: 0.0ms | optim: 23.9ms step 115 | loss: 3.408441 | lr 1.8000e-04 | norm: 0.2614 | dt: 14487.06ms | tok/sec: 36190.1 | mem: 108811MB | Time Left: 02 hrs : 33 mins [TIMING] data: 9.1ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10848.0ms | clip: 0.0ms | optim: 24.3ms step 116 | loss: 3.079810 | lr 1.8000e-04 | norm: 0.2386 | dt: 14488.57ms | tok/sec: 36186.3 | mem: 108811MB | Time Left: 02 hrs : 33 mins [TIMING] data: 9.2ms | fwd: 3592.2ms | loss: 13.1ms | bwd: 10836.0ms | clip: 0.0ms | optim: 24.0ms step 117 | loss: 3.118852 | lr 1.8000e-04 | norm: 0.2419 | dt: 14486.25ms | tok/sec: 36192.1 | mem: 108811MB | Time Left: 02 hrs : 32 mins [TIMING] data: 9.0ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10845.5ms | clip: 0.0ms | optim: 23.9ms step 118 | loss: 3.030657 | lr 1.8000e-04 | norm: 0.2544 | dt: 14478.74ms | tok/sec: 36210.9 | mem: 108811MB | Time Left: 02 hrs : 32 mins [TIMING] data: 9.9ms | fwd: 3573.3ms | loss: 13.0ms | bwd: 10844.7ms | clip: 0.0ms | optim: 24.0ms step 119 | loss: 2.870487 | lr 1.8000e-04 | norm: 0.2331 | dt: 14484.41ms | tok/sec: 36196.7 | mem: 108811MB | Time Left: 02 hrs : 32 mins [TIMING] data: 9.2ms | fwd: 3571.5ms | loss: 13.1ms | bwd: 10852.5ms | clip: 0.0ms | optim: 23.9ms step 120 | loss: 3.130050 | lr 1.8000e-04 | norm: 0.2548 | dt: 14494.87ms | tok/sec: 36170.6 | mem: 108811MB | Time Left: 02 hrs : 32 mins [TIMING] data: 9.1ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10858.2ms | clip: 0.0ms | optim: 23.9ms step 121 | loss: 3.119498 | lr 1.8000e-04 | norm: 0.2384 | dt: 14489.15ms | tok/sec: 36184.9 | mem: 108811MB | Time Left: 02 hrs : 31 mins [TIMING] data: 9.3ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10852.8ms | clip: 0.0ms | optim: 24.0ms step 122 | loss: 3.032317 | lr 1.8000e-04 | norm: 0.2198 | dt: 14537.15ms | tok/sec: 36065.4 | mem: 108811MB | Time Left: 02 hrs : 32 mins [TIMING] data: 9.1ms | fwd: 3641.1ms | loss: 13.1ms | bwd: 10836.0ms | clip: 0.0ms | optim: 24.0ms step 123 | loss: 3.228720 | lr 1.8000e-04 | norm: 0.2107 | dt: 14489.01ms | tok/sec: 36185.2 | mem: 108811MB | Time Left: 02 hrs : 31 mins [TIMING] data: 9.0ms | fwd: 3573.9ms | loss: 13.1ms | bwd: 10855.0ms | clip: 0.0ms | optim: 24.0ms step 124 | loss: 3.063460 | lr 1.8000e-04 | norm: 0.2268 | dt: 14476.42ms | tok/sec: 36216.7 | mem: 108811MB | Time Left: 02 hrs : 31 mins [TIMING] data: 9.0ms | fwd: 3577.7ms | loss: 13.8ms | bwd: 10837.9ms | clip: 0.0ms | optim: 24.0ms step 125 | loss: 3.333213 | lr 1.8000e-04 | norm: 0.2352 | dt: 14474.21ms | tok/sec: 36222.2 | mem: 108811MB | Time Left: 02 hrs : 30 mins [TIMING] data: 9.4ms | fwd: 3590.8ms | loss: 13.1ms | bwd: 10823.0ms | clip: 0.0ms | optim: 24.0ms step 126 | loss: 3.079295 | lr 1.8000e-04 | norm: 0.2471 | dt: 14480.44ms | tok/sec: 36206.6 | mem: 108811MB | Time Left: 02 hrs : 30 mins [TIMING] data: 9.1ms | fwd: 3572.7ms | loss: 13.1ms | bwd: 10847.7ms | clip: 0.0ms | optim: 23.9ms step 127 | loss: 3.088480 | lr 1.8000e-04 | norm: 0.2392 | dt: 14480.98ms | tok/sec: 36205.3 | mem: 108811MB | Time Left: 02 hrs : 30 mins [TIMING] data: 9.3ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10843.4ms | clip: 0.0ms | optim: 24.0ms step 128 | loss: 2.864865 | lr 1.8000e-04 | norm: 0.2266 | dt: 14495.06ms | tok/sec: 36170.1 | mem: 108811MB | Time Left: 02 hrs : 30 mins [TIMING] data: 9.2ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10855.8ms | clip: 0.0ms | optim: 23.9ms step 129 | loss: 2.999127 | lr 1.8000e-04 | norm: 0.2720 | dt: 14479.68ms | tok/sec: 36208.5 | mem: 108811MB | Time Left: 02 hrs : 29 mins [TIMING] data: 9.0ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10841.7ms | clip: 0.0ms | optim: 23.9ms step 130 | loss: 2.758952 | lr 1.8000e-04 | norm: 0.2089 | dt: 14481.94ms | tok/sec: 36202.9 | mem: 108811MB | Time Left: 02 hrs : 29 mins [TIMING] data: 9.0ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10837.0ms | clip: 0.0ms | optim: 24.0ms step 131 | loss: 3.265690 | lr 1.8000e-04 | norm: 0.2235 | dt: 14480.88ms | tok/sec: 36205.5 | mem: 108811MB | Time Left: 02 hrs : 29 mins [TIMING] data: 9.2ms | fwd: 3575.9ms | loss: 13.0ms | bwd: 10845.0ms | clip: 0.0ms | optim: 23.9ms step 132 | loss: 3.075686 | lr 1.8000e-04 | norm: 0.2378 | dt: 14483.25ms | tok/sec: 36199.6 | mem: 108811MB | Time Left: 02 hrs : 29 mins [TIMING] data: 9.0ms | fwd: 3576.8ms | loss: 13.0ms | bwd: 10846.0ms | clip: 0.0ms | optim: 24.4ms step 133 | loss: 3.007264 | lr 1.8000e-04 | norm: 0.2186 | dt: 14488.85ms | tok/sec: 36185.6 | mem: 108811MB | Time Left: 02 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10852.0ms | clip: 0.0ms | optim: 24.0ms step 134 | loss: 2.721003 | lr 1.8000e-04 | norm: 0.2559 | dt: 14553.75ms | tok/sec: 36024.3 | mem: 108811MB | Time Left: 02 hrs : 29 mins [TIMING] data: 9.1ms | fwd: 3660.5ms | loss: 13.1ms | bwd: 10832.9ms | clip: 0.0ms | optim: 23.9ms step 135 | loss: 2.743815 | lr 1.8000e-04 | norm: 0.2473 | dt: 14480.61ms | tok/sec: 36206.2 | mem: 108811MB | Time Left: 02 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10843.7ms | clip: 0.0ms | optim: 24.0ms step 136 | loss: 2.780344 | lr 1.8000e-04 | norm: 0.2364 | dt: 14480.82ms | tok/sec: 36205.7 | mem: 108811MB | Time Left: 02 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3574.3ms | loss: 13.1ms | bwd: 10846.3ms | clip: 0.0ms | optim: 23.9ms step 137 | loss: 2.954257 | lr 1.8000e-04 | norm: 0.3249 | dt: 14490.43ms | tok/sec: 36181.7 | mem: 108811MB | Time Left: 02 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3579.2ms | loss: 13.0ms | bwd: 10851.3ms | clip: 0.0ms | optim: 23.9ms step 138 | loss: 2.898310 | lr 1.8000e-04 | norm: 0.3576 | dt: 14495.37ms | tok/sec: 36169.4 | mem: 108811MB | Time Left: 02 hrs : 27 mins [TIMING] data: 10.0ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10856.0ms | clip: 0.0ms | optim: 24.0ms step 139 | loss: 3.004957 | lr 1.8000e-04 | norm: 0.2654 | dt: 14477.65ms | tok/sec: 36213.6 | mem: 108811MB | Time Left: 02 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10842.3ms | clip: 0.0ms | optim: 23.9ms step 140 | loss: 3.032887 | lr 1.8000e-04 | norm: 0.4472 | dt: 14486.09ms | tok/sec: 36192.5 | mem: 108811MB | Time Left: 02 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10846.8ms | clip: 0.0ms | optim: 24.0ms step 141 | loss: 3.194194 | lr 1.8000e-04 | norm: 0.3222 | dt: 14483.33ms | tok/sec: 36199.4 | mem: 108811MB | Time Left: 02 hrs : 27 mins [TIMING] data: 9.3ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.9ms step 142 | loss: 2.981706 | lr 1.8000e-04 | norm: 0.4215 | dt: 14479.10ms | tok/sec: 36210.0 | mem: 108811MB | Time Left: 02 hrs : 26 mins [TIMING] data: 9.3ms | fwd: 3590.6ms | loss: 13.1ms | bwd: 10828.1ms | clip: 0.0ms | optim: 23.9ms step 143 | loss: 3.136675 | lr 1.8000e-04 | norm: 0.2529 | dt: 14491.30ms | tok/sec: 36179.5 | mem: 108811MB | Time Left: 02 hrs : 26 mins [TIMING] data: 9.3ms | fwd: 3580.8ms | loss: 13.0ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.9ms step 144 | loss: 2.826812 | lr 1.8000e-04 | norm: 0.2397 | dt: 14479.17ms | tok/sec: 36209.8 | mem: 108811MB | Time Left: 02 hrs : 26 mins [TIMING] data: 9.3ms | fwd: 3577.3ms | loss: 13.0ms | bwd: 10841.5ms | clip: 0.0ms | optim: 23.9ms step 145 | loss: 2.739453 | lr 1.8000e-04 | norm: 0.2904 | dt: 14474.11ms | tok/sec: 36222.5 | mem: 108811MB | Time Left: 02 hrs : 25 mins [TIMING] data: 9.2ms | fwd: 3570.0ms | loss: 13.0ms | bwd: 10844.0ms | clip: 0.0ms | optim: 24.0ms step 146 | loss: 2.960023 | lr 1.8000e-04 | norm: 0.2796 | dt: 14473.56ms | tok/sec: 36223.8 | mem: 108811MB | Time Left: 02 hrs : 25 mins [TIMING] data: 9.0ms | fwd: 3584.4ms | loss: 13.0ms | bwd: 10829.4ms | clip: 0.0ms | optim: 24.0ms step 147 | loss: 2.728320 | lr 1.8000e-04 | norm: 0.2445 | dt: 14547.73ms | tok/sec: 36039.2 | mem: 108811MB | Time Left: 02 hrs : 26 mins [TIMING] data: 9.2ms | fwd: 3649.9ms | loss: 13.2ms | bwd: 10837.4ms | clip: 0.0ms | optim: 24.0ms step 148 | loss: 3.010621 | lr 1.8000e-04 | norm: 0.3048 | dt: 14483.07ms | tok/sec: 36200.1 | mem: 108811MB | Time Left: 02 hrs : 25 mins [TIMING] data: 9.1ms | fwd: 3576.4ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 24.0ms step 149 | loss: 2.636559 | lr 1.8000e-04 | norm: 0.2677 | dt: 14494.97ms | tok/sec: 36170.3 | mem: 108811MB | Time Left: 02 hrs : 25 mins [TIMING] data: 9.1ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_150.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_100.ckpt step 150 | loss: 2.907981 | lr 1.8000e-04 | norm: 0.3289 | dt: 14386.67ms | tok/sec: 36442.6 | mem: 108811MB | Time Left: 02 hrs : 23 mins [TIMING] data: 9.0ms | fwd: 3571.1ms | loss: 13.0ms | bwd: 10755.7ms | clip: 0.0ms | optim: 23.9ms step 151 | loss: 3.076273 | lr 1.8000e-04 | norm: 0.2868 | dt: 14411.13ms | tok/sec: 36380.8 | mem: 108811MB | Time Left: 02 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3577.8ms | loss: 13.0ms | bwd: 10773.0ms | clip: 0.0ms | optim: 24.3ms step 152 | loss: 3.092355 | lr 1.8000e-04 | norm: 0.2802 | dt: 14420.85ms | tok/sec: 36356.2 | mem: 108811MB | Time Left: 02 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3565.9ms | loss: 13.0ms | bwd: 10794.6ms | clip: 0.0ms | optim: 24.0ms step 153 | loss: 2.990697 | lr 1.8000e-04 | norm: 0.3577 | dt: 14449.21ms | tok/sec: 36284.9 | mem: 108811MB | Time Left: 02 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3568.2ms | loss: 13.0ms | bwd: 10821.2ms | clip: 0.0ms | optim: 23.9ms step 154 | loss: 3.243481 | lr 1.8000e-04 | norm: 0.3215 | dt: 14534.32ms | tok/sec: 36072.4 | mem: 108811MB | Time Left: 02 hrs : 24 mins [TIMING] data: 9.0ms | fwd: 3656.3ms | loss: 13.0ms | bwd: 10818.0ms | clip: 0.0ms | optim: 23.9ms step 155 | loss: 2.937765 | lr 1.8000e-04 | norm: 0.2341 | dt: 14479.38ms | tok/sec: 36209.3 | mem: 108811MB | Time Left: 02 hrs : 23 mins [TIMING] data: 9.0ms | fwd: 3595.5ms | loss: 13.0ms | bwd: 10823.9ms | clip: 0.0ms | optim: 23.9ms step 156 | loss: 2.898167 | lr 1.8000e-04 | norm: 0.2625 | dt: 14472.94ms | tok/sec: 36225.4 | mem: 108811MB | Time Left: 02 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10831.6ms | clip: 0.0ms | optim: 24.0ms step 157 | loss: 2.723282 | lr 1.8000e-04 | norm: 0.2392 | dt: 14476.93ms | tok/sec: 36215.4 | mem: 108811MB | Time Left: 02 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3586.5ms | loss: 13.1ms | bwd: 10830.1ms | clip: 0.0ms | optim: 23.9ms step 158 | loss: 3.090994 | lr 1.8000e-04 | norm: 0.2504 | dt: 14479.38ms | tok/sec: 36209.3 | mem: 108811MB | Time Left: 02 hrs : 22 mins [TIMING] data: 9.2ms | fwd: 3573.9ms | loss: 13.1ms | bwd: 10845.1ms | clip: 0.0ms | optim: 23.9ms step 159 | loss: 2.966264 | lr 1.8000e-04 | norm: 0.2331 | dt: 14490.38ms | tok/sec: 36181.8 | mem: 108811MB | Time Left: 02 hrs : 22 mins [TIMING] data: 9.1ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 160 | loss: 2.837614 | lr 1.8000e-04 | norm: 0.2493 | dt: 14477.00ms | tok/sec: 36215.2 | mem: 108811MB | Time Left: 02 hrs : 22 mins [TIMING] data: 9.1ms | fwd: 3574.1ms | loss: 13.0ms | bwd: 10843.0ms | clip: 0.0ms | optim: 23.9ms step 161 | loss: 3.126181 | lr 1.8000e-04 | norm: 0.2641 | dt: 14485.70ms | tok/sec: 36193.5 | mem: 108811MB | Time Left: 02 hrs : 22 mins [TIMING] data: 9.1ms | fwd: 3586.8ms | loss: 13.1ms | bwd: 10838.9ms | clip: 0.0ms | optim: 24.0ms step 162 | loss: 3.141209 | lr 1.8000e-04 | norm: 0.2227 | dt: 14495.63ms | tok/sec: 36168.7 | mem: 108811MB | Time Left: 02 hrs : 22 mins [TIMING] data: 9.1ms | fwd: 3581.2ms | loss: 13.0ms | bwd: 10853.8ms | clip: 0.0ms | optim: 23.9ms step 163 | loss: 3.102221 | lr 1.8000e-04 | norm: 0.3480 | dt: 14492.81ms | tok/sec: 36175.7 | mem: 108811MB | Time Left: 02 hrs : 21 mins [TIMING] data: 9.1ms | fwd: 3594.1ms | loss: 13.1ms | bwd: 10838.5ms | clip: 0.0ms | optim: 23.9ms step 164 | loss: 2.815700 | lr 1.8000e-04 | norm: 0.2286 | dt: 14487.58ms | tok/sec: 36188.8 | mem: 108811MB | Time Left: 02 hrs : 21 mins [TIMING] data: 9.1ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10849.2ms | clip: 0.0ms | optim: 23.9ms step 165 | loss: 3.107598 | lr 1.8000e-04 | norm: 0.2332 | dt: 14478.81ms | tok/sec: 36210.7 | mem: 108811MB | Time Left: 02 hrs : 21 mins [TIMING] data: 9.0ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10841.2ms | clip: 0.0ms | optim: 24.0ms step 166 | loss: 2.709398 | lr 1.8000e-04 | norm: 0.2359 | dt: 14484.61ms | tok/sec: 36196.2 | mem: 108811MB | Time Left: 02 hrs : 20 mins [TIMING] data: 9.1ms | fwd: 3574.0ms | loss: 13.1ms | bwd: 10850.5ms | clip: 0.0ms | optim: 23.9ms step 167 | loss: 3.186709 | lr 1.8000e-04 | norm: 0.2658 | dt: 14478.27ms | tok/sec: 36212.1 | mem: 108811MB | Time Left: 02 hrs : 20 mins [TIMING] data: 9.1ms | fwd: 3574.2ms | loss: 13.0ms | bwd: 10844.0ms | clip: 0.0ms | optim: 23.9ms step 168 | loss: 3.031979 | lr 1.8000e-04 | norm: 0.3784 | dt: 14488.22ms | tok/sec: 36187.2 | mem: 108811MB | Time Left: 02 hrs : 20 mins [TIMING] data: 9.0ms | fwd: 3581.6ms | loss: 13.0ms | bwd: 10846.7ms | clip: 0.0ms | optim: 24.0ms step 169 | loss: 2.880915 | lr 1.8000e-04 | norm: 0.3343 | dt: 14483.75ms | tok/sec: 36198.3 | mem: 108811MB | Time Left: 02 hrs : 20 mins [TIMING] data: 9.1ms | fwd: 3587.0ms | loss: 13.1ms | bwd: 10836.5ms | clip: 0.0ms | optim: 23.9ms step 170 | loss: 3.124794 | lr 1.8000e-04 | norm: 0.2456 | dt: 14490.50ms | tok/sec: 36181.5 | mem: 108811MB | Time Left: 02 hrs : 20 mins [TIMING] data: 9.0ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10848.2ms | clip: 0.0ms | optim: 24.0ms step 171 | loss: 3.094166 | lr 1.8000e-04 | norm: 0.2699 | dt: 14480.12ms | tok/sec: 36207.4 | mem: 108811MB | Time Left: 02 hrs : 19 mins [TIMING] data: 9.0ms | fwd: 3586.9ms | loss: 13.0ms | bwd: 10833.3ms | clip: 0.0ms | optim: 23.9ms step 172 | loss: 2.966642 | lr 1.8000e-04 | norm: 0.2506 | dt: 14560.13ms | tok/sec: 36008.5 | mem: 108811MB | Time Left: 02 hrs : 20 mins [TIMING] data: 9.1ms | fwd: 3661.9ms | loss: 13.0ms | bwd: 10838.1ms | clip: 0.0ms | optim: 24.0ms step 173 | loss: 3.017975 | lr 1.8000e-04 | norm: 0.2566 | dt: 14553.37ms | tok/sec: 36025.2 | mem: 108811MB | Time Left: 02 hrs : 19 mins [TIMING] data: 9.1ms | fwd: 3661.8ms | loss: 13.1ms | bwd: 10831.4ms | clip: 0.0ms | optim: 24.1ms step 174 | loss: 3.210958 | lr 1.8000e-04 | norm: 0.3872 | dt: 14485.19ms | tok/sec: 36194.8 | mem: 108811MB | Time Left: 02 hrs : 19 mins [TIMING] data: 9.2ms | fwd: 3574.8ms | loss: 13.0ms | bwd: 10850.3ms | clip: 0.0ms | optim: 23.9ms step 175 | loss: 2.959009 | lr 1.8000e-04 | norm: 0.2829 | dt: 14494.01ms | tok/sec: 36172.7 | mem: 108811MB | Time Left: 02 hrs : 18 mins [TIMING] data: 9.2ms | fwd: 3575.0ms | loss: 13.1ms | bwd: 10858.6ms | clip: 0.0ms | optim: 23.9ms step 176 | loss: 3.086313 | lr 1.8000e-04 | norm: 0.2623 | dt: 14544.94ms | tok/sec: 36046.1 | mem: 108811MB | Time Left: 02 hrs : 19 mins [TIMING] data: 9.0ms | fwd: 3651.3ms | loss: 13.1ms | bwd: 10833.8ms | clip: 0.0ms | optim: 23.9ms step 177 | loss: 3.211298 | lr 1.8000e-04 | norm: 0.2877 | dt: 14553.58ms | tok/sec: 36024.7 | mem: 108811MB | Time Left: 02 hrs : 18 mins [TIMING] data: 9.4ms | fwd: 3663.1ms | loss: 13.1ms | bwd: 10828.5ms | clip: 0.0ms | optim: 23.9ms step 178 | loss: 2.946225 | lr 1.8000e-04 | norm: 0.2557 | dt: 14496.95ms | tok/sec: 36165.4 | mem: 108811MB | Time Left: 02 hrs : 18 mins [TIMING] data: 9.0ms | fwd: 3582.8ms | loss: 13.0ms | bwd: 10854.3ms | clip: 0.0ms | optim: 24.0ms step 179 | loss: 2.971588 | lr 1.8000e-04 | norm: 0.2664 | dt: 14483.61ms | tok/sec: 36198.7 | mem: 108811MB | Time Left: 02 hrs : 17 mins [TIMING] data: 9.1ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10844.8ms | clip: 0.0ms | optim: 24.0ms step 180 | loss: 3.046493 | lr 1.8000e-04 | norm: 0.3031 | dt: 14542.82ms | tok/sec: 36051.3 | mem: 108811MB | Time Left: 02 hrs : 18 mins [TIMING] data: 9.1ms | fwd: 3662.0ms | loss: 13.1ms | bwd: 10820.9ms | clip: 0.0ms | optim: 23.9ms step 181 | loss: 2.849336 | lr 1.8000e-04 | norm: 0.2699 | dt: 14493.64ms | tok/sec: 36173.7 | mem: 108811MB | Time Left: 02 hrs : 17 mins [TIMING] data: 9.1ms | fwd: 3578.0ms | loss: 13.4ms | bwd: 10855.3ms | clip: 0.0ms | optim: 23.9ms step 182 | loss: 3.005729 | lr 1.8000e-04 | norm: 0.2676 | dt: 14480.78ms | tok/sec: 36205.8 | mem: 108811MB | Time Left: 02 hrs : 17 mins [TIMING] data: 9.1ms | fwd: 3580.6ms | loss: 13.1ms | bwd: 10839.8ms | clip: 0.0ms | optim: 24.0ms step 183 | loss: 3.191743 | lr 1.8000e-04 | norm: 0.3172 | dt: 14485.04ms | tok/sec: 36195.1 | mem: 108811MB | Time Left: 02 hrs : 16 mins [TIMING] data: 9.0ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10842.5ms | clip: 0.0ms | optim: 24.0ms step 184 | loss: 2.777579 | lr 1.8000e-04 | norm: 0.2406 | dt: 14478.61ms | tok/sec: 36211.2 | mem: 108811MB | Time Left: 02 hrs : 16 mins [TIMING] data: 9.0ms | fwd: 3574.6ms | loss: 13.0ms | bwd: 10844.2ms | clip: 0.0ms | optim: 23.9ms step 185 | loss: 3.199887 | lr 1.8000e-04 | norm: 0.2451 | dt: 14477.31ms | tok/sec: 36214.5 | mem: 108811MB | Time Left: 02 hrs : 16 mins [TIMING] data: 9.1ms | fwd: 3593.0ms | loss: 13.0ms | bwd: 10824.4ms | clip: 0.0ms | optim: 23.9ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00001.bin 100000000 step 186 | loss: 2.529170 | lr 1.8000e-04 | norm: 0.2333 | dt: 14492.49ms | tok/sec: 36176.5 | mem: 108811MB | Time Left: 02 hrs : 16 mins [TIMING] data: 11.2ms | fwd: 3576.1ms | loss: 13.0ms | bwd: 10854.2ms | clip: 0.0ms | optim: 23.9ms step 187 | loss: 2.824290 | lr 1.8000e-04 | norm: 0.2217 | dt: 14469.74ms | tok/sec: 36233.4 | mem: 108811MB | Time Left: 02 hrs : 15 mins [TIMING] data: 9.0ms | fwd: 3573.6ms | loss: 13.1ms | bwd: 10836.0ms | clip: 0.0ms | optim: 23.9ms step 188 | loss: 3.272614 | lr 1.8000e-04 | norm: 0.2053 | dt: 14487.83ms | tok/sec: 36188.2 | mem: 108811MB | Time Left: 02 hrs : 15 mins [TIMING] data: 9.1ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10848.7ms | clip: 0.0ms | optim: 24.0ms step 189 | loss: 2.931975 | lr 1.8000e-04 | norm: 0.2384 | dt: 14481.94ms | tok/sec: 36202.9 | mem: 108811MB | Time Left: 02 hrs : 15 mins [TIMING] data: 9.1ms | fwd: 3578.0ms | loss: 13.1ms | bwd: 10843.3ms | clip: 0.0ms | optim: 24.2ms step 190 | loss: 2.934358 | lr 1.8000e-04 | norm: 0.2110 | dt: 14492.10ms | tok/sec: 36177.5 | mem: 108811MB | Time Left: 02 hrs : 15 mins [TIMING] data: 9.1ms | fwd: 3578.0ms | loss: 13.0ms | bwd: 10853.8ms | clip: 0.0ms | optim: 24.2ms step 191 | loss: 3.029578 | lr 1.8000e-04 | norm: 0.2354 | dt: 14547.41ms | tok/sec: 36039.9 | mem: 108811MB | Time Left: 02 hrs : 15 mins [TIMING] data: 9.0ms | fwd: 3660.3ms | loss: 13.0ms | bwd: 10827.2ms | clip: 0.0ms | optim: 24.0ms step 192 | loss: 2.939940 | lr 1.8000e-04 | norm: 0.3377 | dt: 14473.26ms | tok/sec: 36224.6 | mem: 108811MB | Time Left: 02 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10833.8ms | clip: 0.0ms | optim: 23.9ms step 193 | loss: 2.838968 | lr 1.8000e-04 | norm: 0.3255 | dt: 14491.48ms | tok/sec: 36179.0 | mem: 108811MB | Time Left: 02 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3591.3ms | loss: 13.1ms | bwd: 10840.2ms | clip: 0.0ms | optim: 23.9ms step 194 | loss: 3.232436 | lr 1.8000e-04 | norm: 0.3606 | dt: 14484.42ms | tok/sec: 36196.7 | mem: 108811MB | Time Left: 02 hrs : 14 mins [TIMING] data: 9.0ms | fwd: 3577.5ms | loss: 13.0ms | bwd: 10847.0ms | clip: 0.0ms | optim: 24.0ms step 195 | loss: 3.140624 | lr 1.8000e-04 | norm: 0.2239 | dt: 14474.85ms | tok/sec: 36220.6 | mem: 108811MB | Time Left: 02 hrs : 13 mins [TIMING] data: 9.0ms | fwd: 3578.0ms | loss: 13.1ms | bwd: 10836.8ms | clip: 0.0ms | optim: 24.0ms step 196 | loss: 3.075119 | lr 1.8000e-04 | norm: 0.2905 | dt: 14492.51ms | tok/sec: 36176.5 | mem: 108811MB | Time Left: 02 hrs : 13 mins [TIMING] data: 9.0ms | fwd: 3592.5ms | loss: 13.1ms | bwd: 10839.8ms | clip: 0.0ms | optim: 23.9ms step 197 | loss: 2.706177 | lr 1.8000e-04 | norm: 0.3322 | dt: 14483.01ms | tok/sec: 36200.2 | mem: 108811MB | Time Left: 02 hrs : 13 mins [TIMING] data: 9.0ms | fwd: 3582.3ms | loss: 13.0ms | bwd: 10840.8ms | clip: 0.0ms | optim: 24.0ms step 198 | loss: 2.941463 | lr 1.8000e-04 | norm: 0.2756 | dt: 14488.18ms | tok/sec: 36187.3 | mem: 108811MB | Time Left: 02 hrs : 13 mins [TIMING] data: 9.5ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10851.7ms | clip: 0.0ms | optim: 23.9ms step 199 | loss: 3.071114 | lr 1.8000e-04 | norm: 0.2903 | dt: 14483.18ms | tok/sec: 36199.8 | mem: 108811MB | Time Left: 02 hrs : 13 mins [TIMING] data: 9.1ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10845.2ms | clip: 0.0ms | optim: 24.1ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_150.ckpt step 200 | loss: 3.042020 | lr 1.8000e-04 | norm: 0.2279 | dt: 14380.74ms | tok/sec: 36457.6 | mem: 108811MB | Time Left: 02 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3571.4ms | loss: 13.1ms | bwd: 10749.2ms | clip: 0.0ms | optim: 23.9ms step 201 | loss: 3.029318 | lr 1.8000e-04 | norm: 0.2528 | dt: 14408.70ms | tok/sec: 36386.9 | mem: 108811MB | Time Left: 02 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3558.5ms | loss: 13.1ms | bwd: 10790.1ms | clip: 0.0ms | optim: 23.9ms step 202 | loss: 2.889757 | lr 1.8000e-04 | norm: 0.2087 | dt: 14431.65ms | tok/sec: 36329.0 | mem: 108811MB | Time Left: 02 hrs : 11 mins [TIMING] data: 9.0ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10788.3ms | clip: 0.0ms | optim: 23.9ms step 203 | loss: 2.923893 | lr 1.8000e-04 | norm: 0.2536 | dt: 14438.74ms | tok/sec: 36311.2 | mem: 108811MB | Time Left: 02 hrs : 11 mins [TIMING] data: 9.0ms | fwd: 3571.1ms | loss: 13.0ms | bwd: 10807.7ms | clip: 0.0ms | optim: 23.9ms step 204 | loss: 2.673539 | lr 1.8000e-04 | norm: 0.2151 | dt: 14457.89ms | tok/sec: 36263.1 | mem: 108811MB | Time Left: 02 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3571.7ms | loss: 13.0ms | bwd: 10826.2ms | clip: 0.0ms | optim: 23.9ms step 205 | loss: 2.883733 | lr 1.8000e-04 | norm: 0.2037 | dt: 14469.99ms | tok/sec: 36232.8 | mem: 108811MB | Time Left: 02 hrs : 11 mins [TIMING] data: 9.0ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10834.2ms | clip: 0.0ms | optim: 23.9ms step 206 | loss: 3.190569 | lr 1.8000e-04 | norm: 0.2540 | dt: 14471.56ms | tok/sec: 36228.8 | mem: 108811MB | Time Left: 02 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3574.1ms | loss: 13.0ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.9ms step 207 | loss: 3.150795 | lr 1.8000e-04 | norm: 0.4066 | dt: 14467.75ms | tok/sec: 36238.4 | mem: 108811MB | Time Left: 02 hrs : 10 mins [TIMING] data: 9.0ms | fwd: 3591.6ms | loss: 13.1ms | bwd: 10816.2ms | clip: 0.0ms | optim: 24.0ms step 208 | loss: 2.914786 | lr 1.8000e-04 | norm: 0.2811 | dt: 14474.31ms | tok/sec: 36222.0 | mem: 108811MB | Time Left: 02 hrs : 10 mins [TIMING] data: 9.1ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10838.8ms | clip: 0.0ms | optim: 23.9ms step 209 | loss: 2.747725 | lr 1.8000e-04 | norm: 0.2386 | dt: 14483.46ms | tok/sec: 36199.1 | mem: 108811MB | Time Left: 02 hrs : 10 mins [TIMING] data: 9.1ms | fwd: 3573.9ms | loss: 13.0ms | bwd: 10849.6ms | clip: 0.0ms | optim: 23.9ms step 210 | loss: 3.075998 | lr 1.8000e-04 | norm: 0.3345 | dt: 14468.97ms | tok/sec: 36235.3 | mem: 108811MB | Time Left: 02 hrs : 10 mins [TIMING] data: 9.1ms | fwd: 3582.3ms | loss: 13.0ms | bwd: 10826.2ms | clip: 0.0ms | optim: 23.9ms step 211 | loss: 3.025711 | lr 1.8000e-04 | norm: 0.2343 | dt: 14499.78ms | tok/sec: 36158.3 | mem: 108811MB | Time Left: 02 hrs : 10 mins [TIMING] data: 9.0ms | fwd: 3595.9ms | loss: 13.0ms | bwd: 10844.0ms | clip: 0.0ms | optim: 23.9ms step 212 | loss: 2.922903 | lr 1.8000e-04 | norm: 0.2334 | dt: 14499.40ms | tok/sec: 36159.3 | mem: 108811MB | Time Left: 02 hrs : 10 mins [TIMING] data: 9.0ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10863.3ms | clip: 0.0ms | optim: 23.9ms step 213 | loss: 3.029754 | lr 1.8000e-04 | norm: 0.2590 | dt: 14495.09ms | tok/sec: 36170.0 | mem: 108811MB | Time Left: 02 hrs : 09 mins [TIMING] data: 9.1ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10854.9ms | clip: 0.0ms | optim: 23.9ms step 214 | loss: 2.917193 | lr 1.8000e-04 | norm: 0.2235 | dt: 14509.36ms | tok/sec: 36134.5 | mem: 108811MB | Time Left: 02 hrs : 09 mins [TIMING] data: 9.0ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10872.2ms | clip: 0.0ms | optim: 23.9ms step 215 | loss: 2.715849 | lr 1.8000e-04 | norm: 0.2053 | dt: 14491.94ms | tok/sec: 36177.9 | mem: 108811MB | Time Left: 02 hrs : 09 mins [TIMING] data: 9.0ms | fwd: 3599.1ms | loss: 13.1ms | bwd: 10832.9ms | clip: 0.0ms | optim: 23.9ms step 216 | loss: 3.177967 | lr 1.8000e-04 | norm: 0.2518 | dt: 14484.87ms | tok/sec: 36195.6 | mem: 108811MB | Time Left: 02 hrs : 08 mins [TIMING] data: 9.4ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10845.2ms | clip: 0.0ms | optim: 23.9ms step 217 | loss: 3.148716 | lr 1.8000e-04 | norm: 0.2097 | dt: 14487.06ms | tok/sec: 36190.1 | mem: 108811MB | Time Left: 02 hrs : 08 mins [TIMING] data: 9.0ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10845.7ms | clip: 0.0ms | optim: 24.0ms step 218 | loss: 2.758627 | lr 1.8000e-04 | norm: 0.3015 | dt: 14483.09ms | tok/sec: 36200.0 | mem: 108811MB | Time Left: 02 hrs : 08 mins [TIMING] data: 9.0ms | fwd: 3580.5ms | loss: 13.0ms | bwd: 10842.5ms | clip: 0.0ms | optim: 23.9ms step 219 | loss: 3.125578 | lr 1.8000e-04 | norm: 0.2161 | dt: 14490.19ms | tok/sec: 36182.3 | mem: 108811MB | Time Left: 02 hrs : 08 mins [TIMING] data: 9.4ms | fwd: 3577.1ms | loss: 13.1ms | bwd: 10852.7ms | clip: 0.0ms | optim: 24.0ms step 220 | loss: 3.051028 | lr 1.8000e-04 | norm: 0.2473 | dt: 14490.92ms | tok/sec: 36180.5 | mem: 108811MB | Time Left: 02 hrs : 08 mins [TIMING] data: 9.0ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10853.8ms | clip: 0.0ms | optim: 23.9ms step 221 | loss: 2.883252 | lr 1.8000e-04 | norm: 0.2140 | dt: 14495.48ms | tok/sec: 36169.1 | mem: 108811MB | Time Left: 02 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10853.4ms | clip: 0.0ms | optim: 24.0ms step 222 | loss: 2.982268 | lr 1.8000e-04 | norm: 0.2020 | dt: 14495.34ms | tok/sec: 36169.4 | mem: 108811MB | Time Left: 02 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3580.3ms | loss: 13.0ms | bwd: 10855.1ms | clip: 0.0ms | optim: 23.9ms step 223 | loss: 2.991250 | lr 1.8000e-04 | norm: 0.2342 | dt: 14487.20ms | tok/sec: 36189.7 | mem: 108811MB | Time Left: 02 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3579.1ms | loss: 13.0ms | bwd: 10848.1ms | clip: 0.0ms | optim: 24.0ms step 224 | loss: 3.061347 | lr 1.8000e-04 | norm: 0.2110 | dt: 14540.79ms | tok/sec: 36056.4 | mem: 108811MB | Time Left: 02 hrs : 07 mins [TIMING] data: 9.3ms | fwd: 3657.7ms | loss: 13.1ms | bwd: 10822.6ms | clip: 0.0ms | optim: 23.9ms step 225 | loss: 2.967731 | lr 1.8000e-04 | norm: 0.2028 | dt: 14496.72ms | tok/sec: 36166.0 | mem: 108811MB | Time Left: 02 hrs : 06 mins [TIMING] data: 9.1ms | fwd: 3582.0ms | loss: 13.0ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.9ms step 226 | loss: 2.972879 | lr 1.8000e-04 | norm: 0.2605 | dt: 14481.93ms | tok/sec: 36202.9 | mem: 108811MB | Time Left: 02 hrs : 06 mins [TIMING] data: 9.0ms | fwd: 3576.0ms | loss: 13.0ms | bwd: 10845.9ms | clip: 0.0ms | optim: 23.9ms step 227 | loss: 3.112542 | lr 1.8000e-04 | norm: 0.2982 | dt: 14486.84ms | tok/sec: 36190.6 | mem: 108811MB | Time Left: 02 hrs : 06 mins [TIMING] data: 9.0ms | fwd: 3578.7ms | loss: 13.0ms | bwd: 10848.2ms | clip: 0.0ms | optim: 24.0ms step 228 | loss: 2.615844 | lr 1.8000e-04 | norm: 0.1963 | dt: 14486.58ms | tok/sec: 36191.3 | mem: 108811MB | Time Left: 02 hrs : 06 mins [TIMING] data: 9.1ms | fwd: 3591.5ms | loss: 13.1ms | bwd: 10835.0ms | clip: 0.0ms | optim: 23.9ms step 229 | loss: 3.067689 | lr 1.8000e-04 | norm: 0.4074 | dt: 14560.15ms | tok/sec: 36008.4 | mem: 108811MB | Time Left: 02 hrs : 06 mins [TIMING] data: 9.2ms | fwd: 3662.7ms | loss: 13.1ms | bwd: 10837.2ms | clip: 0.0ms | optim: 24.0ms step 230 | loss: 2.998020 | lr 1.8000e-04 | norm: 0.2005 | dt: 14492.95ms | tok/sec: 36175.4 | mem: 108811MB | Time Left: 02 hrs : 05 mins [TIMING] data: 9.3ms | fwd: 3598.3ms | loss: 13.0ms | bwd: 10834.3ms | clip: 0.0ms | optim: 24.0ms step 231 | loss: 3.089678 | lr 1.8000e-04 | norm: 0.7931 | dt: 14552.43ms | tok/sec: 36027.5 | mem: 108811MB | Time Left: 02 hrs : 05 mins [TIMING] data: 9.2ms | fwd: 3659.2ms | loss: 13.1ms | bwd: 10833.0ms | clip: 0.0ms | optim: 23.9ms step 232 | loss: 3.276863 | lr 1.8000e-04 | norm: 0.2426 | dt: 14489.16ms | tok/sec: 36184.9 | mem: 108811MB | Time Left: 02 hrs : 05 mins [TIMING] data: 9.1ms | fwd: 3591.6ms | loss: 13.2ms | bwd: 10837.0ms | clip: 0.0ms | optim: 24.1ms step 233 | loss: 3.037186 | lr 1.8000e-04 | norm: 0.2278 | dt: 14505.16ms | tok/sec: 36144.9 | mem: 108811MB | Time Left: 02 hrs : 04 mins [TIMING] data: 9.0ms | fwd: 3580.4ms | loss: 13.0ms | bwd: 10864.7ms | clip: 0.0ms | optim: 24.0ms step 234 | loss: 2.967620 | lr 1.8000e-04 | norm: 0.2026 | dt: 14556.22ms | tok/sec: 36018.1 | mem: 108811MB | Time Left: 02 hrs : 05 mins [TIMING] data: 9.2ms | fwd: 3661.9ms | loss: 13.0ms | bwd: 10834.2ms | clip: 0.0ms | optim: 23.9ms step 235 | loss: 2.985307 | lr 1.8000e-04 | norm: 0.2572 | dt: 14480.01ms | tok/sec: 36207.7 | mem: 108811MB | Time Left: 02 hrs : 04 mins [TIMING] data: 9.1ms | fwd: 3572.4ms | loss: 13.1ms | bwd: 10847.3ms | clip: 0.0ms | optim: 24.1ms step 236 | loss: 3.264188 | lr 1.8000e-04 | norm: 0.2407 | dt: 14487.24ms | tok/sec: 36189.6 | mem: 108811MB | Time Left: 02 hrs : 04 mins [TIMING] data: 9.1ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10845.9ms | clip: 0.0ms | optim: 23.9ms step 237 | loss: 3.199886 | lr 1.8000e-04 | norm: 0.2442 | dt: 14509.98ms | tok/sec: 36132.9 | mem: 108811MB | Time Left: 02 hrs : 04 mins [TIMING] data: 9.0ms | fwd: 3577.3ms | loss: 13.0ms | bwd: 10872.9ms | clip: 0.0ms | optim: 23.9ms step 238 | loss: 2.971516 | lr 1.8000e-04 | norm: 0.2255 | dt: 14493.58ms | tok/sec: 36173.8 | mem: 108811MB | Time Left: 02 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3580.5ms | loss: 13.0ms | bwd: 10853.1ms | clip: 0.0ms | optim: 24.0ms step 239 | loss: 3.008428 | lr 1.8000e-04 | norm: 0.2273 | dt: 14493.89ms | tok/sec: 36173.0 | mem: 108811MB | Time Left: 02 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10850.6ms | clip: 0.0ms | optim: 23.9ms step 240 | loss: 2.786235 | lr 1.8000e-04 | norm: 0.2421 | dt: 14559.63ms | tok/sec: 36009.7 | mem: 108811MB | Time Left: 02 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3660.5ms | loss: 13.1ms | bwd: 10839.1ms | clip: 0.0ms | optim: 23.9ms step 241 | loss: 2.927471 | lr 1.8000e-04 | norm: 0.2326 | dt: 14488.32ms | tok/sec: 36186.9 | mem: 108811MB | Time Left: 02 hrs : 02 mins [TIMING] data: 9.0ms | fwd: 3580.7ms | loss: 13.0ms | bwd: 10847.7ms | clip: 0.0ms | optim: 24.0ms step 242 | loss: 3.150245 | lr 1.8000e-04 | norm: 0.2323 | dt: 14499.26ms | tok/sec: 36159.6 | mem: 108811MB | Time Left: 02 hrs : 02 mins [TIMING] data: 9.0ms | fwd: 3580.7ms | loss: 13.0ms | bwd: 10858.6ms | clip: 0.0ms | optim: 24.0ms step 243 | loss: 3.172523 | lr 1.8000e-04 | norm: 0.2449 | dt: 14496.56ms | tok/sec: 36166.4 | mem: 108811MB | Time Left: 02 hrs : 02 mins [TIMING] data: 9.3ms | fwd: 3592.8ms | loss: 13.1ms | bwd: 10843.0ms | clip: 0.0ms | optim: 24.0ms step 244 | loss: 3.044659 | lr 1.8000e-04 | norm: 0.2370 | dt: 14492.05ms | tok/sec: 36177.6 | mem: 108811MB | Time Left: 02 hrs : 02 mins [TIMING] data: 9.0ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10854.1ms | clip: 0.0ms | optim: 24.0ms step 245 | loss: 3.016348 | lr 1.8000e-04 | norm: 0.3494 | dt: 14552.73ms | tok/sec: 36026.8 | mem: 108811MB | Time Left: 02 hrs : 02 mins [TIMING] data: 9.1ms | fwd: 3662.1ms | loss: 13.1ms | bwd: 10830.5ms | clip: 0.0ms | optim: 24.0ms step 246 | loss: 2.861705 | lr 1.8000e-04 | norm: 0.2294 | dt: 14491.43ms | tok/sec: 36179.2 | mem: 108811MB | Time Left: 02 hrs : 01 mins [TIMING] data: 9.0ms | fwd: 3589.9ms | loss: 13.1ms | bwd: 10841.3ms | clip: 0.0ms | optim: 24.0ms step 247 | loss: 3.021536 | lr 1.8000e-04 | norm: 0.3134 | dt: 14477.03ms | tok/sec: 36215.2 | mem: 108811MB | Time Left: 02 hrs : 01 mins [TIMING] data: 9.0ms | fwd: 3580.8ms | loss: 13.0ms | bwd: 10836.3ms | clip: 0.0ms | optim: 23.9ms step 248 | loss: 3.003254 | lr 1.8000e-04 | norm: 0.2908 | dt: 14495.51ms | tok/sec: 36169.0 | mem: 108811MB | Time Left: 02 hrs : 01 mins [TIMING] data: 9.3ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10856.3ms | clip: 0.0ms | optim: 24.0ms step 249 | loss: 3.086852 | lr 1.8000e-04 | norm: 0.2160 | dt: 14496.03ms | tok/sec: 36167.7 | mem: 108811MB | Time Left: 02 hrs : 01 mins [TIMING] data: 9.0ms | fwd: 3577.9ms | loss: 13.0ms | bwd: 10858.2ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_250.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt validation loss: 3.1921 step 250 | loss: 2.892407 | lr 1.8000e-04 | norm: 0.2225 | dt: 18944.27ms | tok/sec: 27675.3 | mem: 108811MB | Time Left: 02 hrs : 37 mins [TIMING] data: 9.2ms | fwd: 3562.6ms | loss: 13.0ms | bwd: 10779.0ms | clip: 0.0ms | optim: 23.9ms step 251 | loss: 3.141205 | lr 1.8000e-04 | norm: 0.2273 | dt: 14420.99ms | tok/sec: 36355.9 | mem: 108811MB | Time Left: 01 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3566.1ms | loss: 13.0ms | bwd: 10794.9ms | clip: 0.0ms | optim: 23.9ms step 252 | loss: 3.021185 | lr 1.8000e-04 | norm: 0.3304 | dt: 14437.39ms | tok/sec: 36314.6 | mem: 108811MB | Time Left: 01 hrs : 59 mins [TIMING] data: 9.2ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10799.5ms | clip: 0.0ms | optim: 23.9ms step 253 | loss: 3.054901 | lr 1.8000e-04 | norm: 0.2780 | dt: 14445.55ms | tok/sec: 36294.1 | mem: 108811MB | Time Left: 01 hrs : 59 mins [TIMING] data: 9.0ms | fwd: 3573.7ms | loss: 13.1ms | bwd: 10811.8ms | clip: 0.0ms | optim: 24.0ms step 254 | loss: 2.948195 | lr 1.8000e-04 | norm: 0.2402 | dt: 14459.64ms | tok/sec: 36258.7 | mem: 108811MB | Time Left: 01 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10824.2ms | clip: 0.0ms | optim: 24.0ms step 255 | loss: 2.867964 | lr 1.8000e-04 | norm: 0.2070 | dt: 14464.35ms | tok/sec: 36246.9 | mem: 108811MB | Time Left: 01 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3575.4ms | loss: 13.0ms | bwd: 10829.1ms | clip: 0.0ms | optim: 23.9ms step 256 | loss: 2.989999 | lr 1.8000e-04 | norm: 0.2239 | dt: 14464.93ms | tok/sec: 36245.4 | mem: 108811MB | Time Left: 01 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3574.2ms | loss: 13.1ms | bwd: 10830.4ms | clip: 0.0ms | optim: 23.9ms step 257 | loss: 3.228361 | lr 1.8000e-04 | norm: 0.2761 | dt: 14491.38ms | tok/sec: 36179.3 | mem: 108811MB | Time Left: 01 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3577.3ms | loss: 13.2ms | bwd: 10853.6ms | clip: 0.0ms | optim: 24.1ms step 258 | loss: 2.995969 | lr 1.8000e-04 | norm: 0.2300 | dt: 14478.74ms | tok/sec: 36210.9 | mem: 108811MB | Time Left: 01 hrs : 58 mins [TIMING] data: 9.2ms | fwd: 3583.4ms | loss: 13.1ms | bwd: 10835.1ms | clip: 0.0ms | optim: 23.9ms step 259 | loss: 3.264681 | lr 1.8000e-04 | norm: 0.2585 | dt: 14488.21ms | tok/sec: 36187.2 | mem: 108811MB | Time Left: 01 hrs : 58 mins [TIMING] data: 9.1ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10845.2ms | clip: 0.0ms | optim: 24.0ms step 260 | loss: 3.117464 | lr 1.8000e-04 | norm: 0.5579 | dt: 14472.30ms | tok/sec: 36227.0 | mem: 108811MB | Time Left: 01 hrs : 58 mins [TIMING] data: 9.1ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10833.6ms | clip: 0.0ms | optim: 23.8ms step 261 | loss: 2.919971 | lr 1.8000e-04 | norm: 0.3116 | dt: 14486.44ms | tok/sec: 36191.6 | mem: 108811MB | Time Left: 01 hrs : 58 mins [TIMING] data: 9.1ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10848.2ms | clip: 0.0ms | optim: 23.9ms step 262 | loss: 2.766665 | lr 1.8000e-04 | norm: 0.2418 | dt: 14486.25ms | tok/sec: 36192.1 | mem: 108811MB | Time Left: 01 hrs : 57 mins [TIMING] data: 9.1ms | fwd: 3594.9ms | loss: 13.1ms | bwd: 10831.5ms | clip: 0.0ms | optim: 23.9ms step 263 | loss: 3.073060 | lr 1.8000e-04 | norm: 0.4130 | dt: 14481.77ms | tok/sec: 36203.3 | mem: 108811MB | Time Left: 01 hrs : 57 mins [TIMING] data: 9.1ms | fwd: 3579.3ms | loss: 13.0ms | bwd: 10842.4ms | clip: 0.0ms | optim: 23.9ms step 264 | loss: 2.840243 | lr 1.8000e-04 | norm: 0.3008 | dt: 14485.46ms | tok/sec: 36194.1 | mem: 108811MB | Time Left: 01 hrs : 57 mins [TIMING] data: 9.1ms | fwd: 3580.6ms | loss: 13.0ms | bwd: 10844.8ms | clip: 0.0ms | optim: 24.0ms step 265 | loss: 2.827043 | lr 1.8000e-04 | norm: 0.2231 | dt: 14483.75ms | tok/sec: 36198.4 | mem: 108811MB | Time Left: 01 hrs : 57 mins [TIMING] data: 9.0ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10840.8ms | clip: 0.0ms | optim: 24.0ms step 266 | loss: 2.766498 | lr 1.8000e-04 | norm: 0.2967 | dt: 14476.14ms | tok/sec: 36217.4 | mem: 108811MB | Time Left: 01 hrs : 56 mins [TIMING] data: 9.1ms | fwd: 3588.3ms | loss: 13.0ms | bwd: 10827.9ms | clip: 0.0ms | optim: 24.0ms step 267 | loss: 3.243832 | lr 1.8000e-04 | norm: 0.2705 | dt: 14493.96ms | tok/sec: 36172.9 | mem: 108811MB | Time Left: 01 hrs : 56 mins [TIMING] data: 9.3ms | fwd: 3579.3ms | loss: 13.0ms | bwd: 10854.1ms | clip: 0.0ms | optim: 24.0ms step 268 | loss: 3.133843 | lr 1.8000e-04 | norm: 0.2593 | dt: 14486.97ms | tok/sec: 36190.3 | mem: 108811MB | Time Left: 01 hrs : 56 mins [TIMING] data: 9.3ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10844.0ms | clip: 0.0ms | optim: 24.1ms step 269 | loss: 3.087945 | lr 1.8000e-04 | norm: 0.4101 | dt: 14489.11ms | tok/sec: 36185.0 | mem: 108811MB | Time Left: 01 hrs : 56 mins [TIMING] data: 9.2ms | fwd: 3594.9ms | loss: 13.0ms | bwd: 10834.2ms | clip: 0.0ms | optim: 23.9ms step 270 | loss: 2.942813 | lr 1.8000e-04 | norm: 0.3693 | dt: 14551.41ms | tok/sec: 36030.0 | mem: 108811MB | Time Left: 01 hrs : 56 mins [TIMING] data: 9.1ms | fwd: 3660.0ms | loss: 13.0ms | bwd: 10831.5ms | clip: 0.0ms | optim: 23.9ms step 271 | loss: 2.595963 | lr 1.8000e-04 | norm: 0.2470 | dt: 14492.98ms | tok/sec: 36175.3 | mem: 108811MB | Time Left: 01 hrs : 55 mins [TIMING] data: 9.0ms | fwd: 3597.2ms | loss: 13.1ms | bwd: 10835.8ms | clip: 0.0ms | optim: 23.9ms step 272 | loss: 2.951489 | lr 1.8000e-04 | norm: 0.3106 | dt: 14561.10ms | tok/sec: 36006.1 | mem: 108811MB | Time Left: 01 hrs : 56 mins [TIMING] data: 9.2ms | fwd: 3661.4ms | loss: 13.0ms | bwd: 10839.3ms | clip: 0.0ms | optim: 24.0ms step 273 | loss: 3.175905 | lr 1.8000e-04 | norm: 0.2603 | dt: 14505.84ms | tok/sec: 36143.2 | mem: 108811MB | Time Left: 01 hrs : 55 mins [TIMING] data: 9.2ms | fwd: 3581.1ms | loss: 13.0ms | bwd: 10864.7ms | clip: 0.0ms | optim: 23.9ms step 274 | loss: 2.937213 | lr 1.8000e-04 | norm: 0.2294 | dt: 14487.25ms | tok/sec: 36189.6 | mem: 108811MB | Time Left: 01 hrs : 54 mins [TIMING] data: 9.1ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10845.6ms | clip: 0.0ms | optim: 23.8ms step 275 | loss: 2.938621 | lr 1.8000e-04 | norm: 0.2793 | dt: 14495.95ms | tok/sec: 36167.9 | mem: 108811MB | Time Left: 01 hrs : 54 mins [TIMING] data: 9.2ms | fwd: 3582.5ms | loss: 13.0ms | bwd: 10853.5ms | clip: 0.0ms | optim: 23.9ms step 276 | loss: 2.870670 | lr 1.8000e-04 | norm: 0.2683 | dt: 14551.49ms | tok/sec: 36029.8 | mem: 108811MB | Time Left: 01 hrs : 54 mins [TIMING] data: 9.0ms | fwd: 3656.5ms | loss: 13.1ms | bwd: 10835.0ms | clip: 0.0ms | optim: 23.9ms step 277 | loss: 3.330475 | lr 1.8000e-04 | norm: 0.2330 | dt: 14564.27ms | tok/sec: 35998.2 | mem: 108811MB | Time Left: 01 hrs : 54 mins [TIMING] data: 9.2ms | fwd: 3661.7ms | loss: 13.0ms | bwd: 10842.5ms | clip: 0.0ms | optim: 23.9ms step 278 | loss: 2.728668 | lr 1.8000e-04 | norm: 0.2096 | dt: 14489.04ms | tok/sec: 36185.1 | mem: 108811MB | Time Left: 01 hrs : 53 mins [TIMING] data: 9.1ms | fwd: 3580.3ms | loss: 13.0ms | bwd: 10848.7ms | clip: 0.0ms | optim: 24.0ms step 279 | loss: 3.082459 | lr 1.8000e-04 | norm: 0.2251 | dt: 14487.11ms | tok/sec: 36190.0 | mem: 108811MB | Time Left: 01 hrs : 53 mins [TIMING] data: 9.1ms | fwd: 3591.1ms | loss: 13.1ms | bwd: 10836.0ms | clip: 0.0ms | optim: 23.9ms step 280 | loss: 2.945036 | lr 1.8000e-04 | norm: 0.2351 | dt: 14493.76ms | tok/sec: 36173.4 | mem: 108811MB | Time Left: 01 hrs : 53 mins [TIMING] data: 9.0ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10856.7ms | clip: 0.0ms | optim: 24.0ms step 281 | loss: 3.163779 | lr 1.8000e-04 | norm: 0.3161 | dt: 14548.19ms | tok/sec: 36038.0 | mem: 108811MB | Time Left: 01 hrs : 53 mins [TIMING] data: 9.4ms | fwd: 3660.5ms | loss: 13.1ms | bwd: 10827.3ms | clip: 0.0ms | optim: 23.8ms step 282 | loss: 2.807217 | lr 1.8000e-04 | norm: 0.5395 | dt: 14489.38ms | tok/sec: 36184.3 | mem: 108811MB | Time Left: 01 hrs : 53 mins [TIMING] data: 9.1ms | fwd: 3575.5ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 24.0ms step 283 | loss: 3.121996 | lr 1.8000e-04 | norm: 0.2961 | dt: 14496.29ms | tok/sec: 36167.1 | mem: 108811MB | Time Left: 01 hrs : 52 mins [TIMING] data: 9.0ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10854.5ms | clip: 0.0ms | optim: 23.9ms step 284 | loss: 3.143355 | lr 1.8000e-04 | norm: 0.2131 | dt: 14495.71ms | tok/sec: 36168.5 | mem: 108811MB | Time Left: 01 hrs : 52 mins [TIMING] data: 9.1ms | fwd: 3582.6ms | loss: 13.0ms | bwd: 10853.3ms | clip: 0.0ms | optim: 23.9ms step 285 | loss: 2.737532 | lr 1.8000e-04 | norm: 0.2363 | dt: 14490.38ms | tok/sec: 36181.8 | mem: 108811MB | Time Left: 01 hrs : 52 mins [TIMING] data: 9.1ms | fwd: 3577.5ms | loss: 13.0ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.9ms step 286 | loss: 3.161401 | lr 1.8000e-04 | norm: 0.2448 | dt: 14486.48ms | tok/sec: 36191.5 | mem: 108811MB | Time Left: 01 hrs : 52 mins [TIMING] data: 9.0ms | fwd: 3582.5ms | loss: 13.0ms | bwd: 10844.1ms | clip: 0.0ms | optim: 23.9ms step 287 | loss: 2.856423 | lr 1.8000e-04 | norm: 0.2301 | dt: 14496.76ms | tok/sec: 36165.9 | mem: 108811MB | Time Left: 01 hrs : 51 mins [TIMING] data: 9.3ms | fwd: 3584.4ms | loss: 13.0ms | bwd: 10851.9ms | clip: 0.0ms | optim: 24.1ms step 288 | loss: 3.038842 | lr 1.8000e-04 | norm: 0.3384 | dt: 14492.09ms | tok/sec: 36177.5 | mem: 108811MB | Time Left: 01 hrs : 51 mins [TIMING] data: 9.5ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10851.3ms | clip: 0.0ms | optim: 23.9ms step 289 | loss: 3.154300 | lr 1.8000e-04 | norm: 0.2229 | dt: 14495.73ms | tok/sec: 36168.5 | mem: 108811MB | Time Left: 01 hrs : 51 mins [TIMING] data: 9.1ms | fwd: 3576.7ms | loss: 13.0ms | bwd: 10859.1ms | clip: 0.0ms | optim: 23.9ms step 290 | loss: 2.730999 | lr 1.8000e-04 | norm: 0.3251 | dt: 14488.53ms | tok/sec: 36186.4 | mem: 108811MB | Time Left: 01 hrs : 51 mins [TIMING] data: 9.1ms | fwd: 3577.9ms | loss: 13.0ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms step 291 | loss: 3.117490 | lr 1.8000e-04 | norm: 0.3454 | dt: 14497.33ms | tok/sec: 36164.5 | mem: 108811MB | Time Left: 01 hrs : 50 mins [TIMING] data: 9.0ms | fwd: 3579.0ms | loss: 13.0ms | bwd: 10858.5ms | clip: 0.0ms | optim: 24.0ms step 292 | loss: 2.835977 | lr 1.8000e-04 | norm: 0.3283 | dt: 14480.23ms | tok/sec: 36207.2 | mem: 108811MB | Time Left: 01 hrs : 50 mins [TIMING] data: 8.9ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10843.6ms | clip: 0.0ms | optim: 24.0ms step 293 | loss: 3.244142 | lr 1.8000e-04 | norm: 0.2690 | dt: 14540.42ms | tok/sec: 36057.3 | mem: 108811MB | Time Left: 01 hrs : 50 mins [TIMING] data: 9.0ms | fwd: 3662.5ms | loss: 13.0ms | bwd: 10818.0ms | clip: 0.0ms | optim: 23.9ms step 294 | loss: 2.888590 | lr 1.8000e-04 | norm: 0.2016 | dt: 14481.89ms | tok/sec: 36203.0 | mem: 108811MB | Time Left: 01 hrs : 50 mins [TIMING] data: 9.1ms | fwd: 3588.6ms | loss: 13.0ms | bwd: 10833.2ms | clip: 0.0ms | optim: 23.9ms step 295 | loss: 2.932020 | lr 1.8000e-04 | norm: 0.2546 | dt: 14497.02ms | tok/sec: 36165.2 | mem: 108811MB | Time Left: 01 hrs : 49 mins [TIMING] data: 9.0ms | fwd: 3581.6ms | loss: 13.0ms | bwd: 10855.6ms | clip: 0.0ms | optim: 23.9ms step 296 | loss: 2.829576 | lr 1.8000e-04 | norm: 0.2290 | dt: 14491.79ms | tok/sec: 36178.3 | mem: 108811MB | Time Left: 01 hrs : 49 mins [TIMING] data: 9.2ms | fwd: 3578.0ms | loss: 13.0ms | bwd: 10853.6ms | clip: 0.0ms | optim: 24.2ms step 297 | loss: 2.843266 | lr 1.8000e-04 | norm: 0.2615 | dt: 14494.96ms | tok/sec: 36170.4 | mem: 108811MB | Time Left: 01 hrs : 49 mins [TIMING] data: 9.2ms | fwd: 3578.8ms | loss: 13.0ms | bwd: 10856.2ms | clip: 0.0ms | optim: 23.9ms step 298 | loss: 3.199498 | lr 1.8000e-04 | norm: 0.3021 | dt: 14490.50ms | tok/sec: 36181.5 | mem: 108811MB | Time Left: 01 hrs : 49 mins [TIMING] data: 9.2ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10850.0ms | clip: 0.0ms | optim: 24.1ms step 299 | loss: 3.054778 | lr 1.8000e-04 | norm: 0.2190 | dt: 14506.44ms | tok/sec: 36141.8 | mem: 108811MB | Time Left: 01 hrs : 49 mins [TIMING] data: 9.2ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10864.7ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_300.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_250.ckpt step 300 | loss: 2.909358 | lr 1.8000e-04 | norm: 0.2547 | dt: 14373.70ms | tok/sec: 36475.5 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.2ms | fwd: 3578.6ms | loss: 13.2ms | bwd: 10734.4ms | clip: 0.0ms | optim: 24.0ms step 301 | loss: 3.041117 | lr 1.8000e-04 | norm: 0.2191 | dt: 14408.70ms | tok/sec: 36386.9 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.4ms | fwd: 3566.7ms | loss: 13.0ms | bwd: 10781.2ms | clip: 0.0ms | optim: 24.0ms step 302 | loss: 3.017186 | lr 1.8000e-04 | norm: 0.2407 | dt: 14433.75ms | tok/sec: 36323.8 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.7ms | fwd: 3584.5ms | loss: 13.1ms | bwd: 10788.6ms | clip: 0.0ms | optim: 23.9ms step 303 | loss: 3.008958 | lr 1.8000e-04 | norm: 0.3161 | dt: 14451.70ms | tok/sec: 36278.6 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3573.1ms | loss: 13.0ms | bwd: 10818.0ms | clip: 0.0ms | optim: 24.6ms step 304 | loss: 3.097311 | lr 1.8000e-04 | norm: 0.2597 | dt: 14469.53ms | tok/sec: 36233.9 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3570.9ms | loss: 13.0ms | bwd: 10838.6ms | clip: 0.0ms | optim: 24.0ms step 305 | loss: 2.985472 | lr 1.8000e-04 | norm: 0.2459 | dt: 14478.86ms | tok/sec: 36210.6 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10838.8ms | clip: 0.0ms | optim: 23.9ms step 306 | loss: 3.049172 | lr 1.8000e-04 | norm: 0.2481 | dt: 14479.67ms | tok/sec: 36208.6 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.2ms | fwd: 3586.9ms | loss: 13.1ms | bwd: 10832.3ms | clip: 0.0ms | optim: 23.9ms step 307 | loss: 2.880740 | lr 1.8000e-04 | norm: 0.2077 | dt: 14550.26ms | tok/sec: 36032.9 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3666.3ms | loss: 13.0ms | bwd: 10824.0ms | clip: 0.0ms | optim: 23.9ms step 308 | loss: 2.943624 | lr 1.8000e-04 | norm: 0.2940 | dt: 14564.06ms | tok/sec: 35998.7 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3668.3ms | loss: 13.2ms | bwd: 10835.3ms | clip: 0.0ms | optim: 24.0ms step 309 | loss: 2.850285 | lr 1.8000e-04 | norm: 0.2077 | dt: 14571.27ms | tok/sec: 35980.9 | mem: 108811MB | Time Left: 01 hrs : 47 mins [TIMING] data: 9.2ms | fwd: 3663.8ms | loss: 13.1ms | bwd: 10847.0ms | clip: 0.0ms | optim: 24.1ms step 310 | loss: 3.135624 | lr 1.8000e-04 | norm: 0.2547 | dt: 14490.95ms | tok/sec: 36180.4 | mem: 108811MB | Time Left: 01 hrs : 46 mins [TIMING] data: 9.2ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10851.2ms | clip: 0.0ms | optim: 23.9ms step 311 | loss: 2.975355 | lr 1.8000e-04 | norm: 0.2175 | dt: 14500.73ms | tok/sec: 36156.0 | mem: 108811MB | Time Left: 01 hrs : 46 mins [TIMING] data: 10.0ms | fwd: 3597.1ms | loss: 13.2ms | bwd: 10842.4ms | clip: 0.0ms | optim: 23.9ms step 312 | loss: 2.866420 | lr 1.8000e-04 | norm: 0.2206 | dt: 14501.93ms | tok/sec: 36153.0 | mem: 108811MB | Time Left: 01 hrs : 45 mins [TIMING] data: 9.3ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10859.4ms | clip: 0.0ms | optim: 23.9ms step 313 | loss: 3.129918 | lr 1.8000e-04 | norm: 0.2226 | dt: 14496.21ms | tok/sec: 36167.2 | mem: 108811MB | Time Left: 01 hrs : 45 mins [TIMING] data: 9.3ms | fwd: 3588.4ms | loss: 13.2ms | bwd: 10847.0ms | clip: 0.0ms | optim: 24.0ms step 314 | loss: 3.116793 | lr 1.8000e-04 | norm: 0.2384 | dt: 14493.16ms | tok/sec: 36174.9 | mem: 108811MB | Time Left: 01 hrs : 45 mins [TIMING] data: 9.2ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10854.1ms | clip: 0.0ms | optim: 24.0ms step 315 | loss: 2.756485 | lr 1.8000e-04 | norm: 0.1835 | dt: 14488.28ms | tok/sec: 36187.0 | mem: 108811MB | Time Left: 01 hrs : 45 mins [TIMING] data: 9.2ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10851.4ms | clip: 0.0ms | optim: 24.0ms step 316 | loss: 3.245055 | lr 1.8000e-04 | norm: 0.2438 | dt: 14491.09ms | tok/sec: 36180.0 | mem: 108811MB | Time Left: 01 hrs : 44 mins [TIMING] data: 9.1ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10853.0ms | clip: 0.0ms | optim: 24.0ms step 317 | loss: 3.088799 | lr 1.8000e-04 | norm: 0.2923 | dt: 14503.12ms | tok/sec: 36150.0 | mem: 108811MB | Time Left: 01 hrs : 44 mins [TIMING] data: 9.1ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10865.0ms | clip: 0.0ms | optim: 24.0ms step 318 | loss: 2.981781 | lr 1.8000e-04 | norm: 0.4119 | dt: 14504.09ms | tok/sec: 36147.6 | mem: 108811MB | Time Left: 01 hrs : 44 mins [TIMING] data: 9.2ms | fwd: 3585.0ms | loss: 13.1ms | bwd: 10858.8ms | clip: 0.0ms | optim: 23.9ms step 319 | loss: 3.052352 | lr 1.8000e-04 | norm: 0.2091 | dt: 14503.84ms | tok/sec: 36148.2 | mem: 108811MB | Time Left: 01 hrs : 44 mins [TIMING] data: 9.5ms | fwd: 3586.3ms | loss: 13.1ms | bwd: 10857.0ms | clip: 0.0ms | optim: 23.9ms step 320 | loss: 3.096810 | lr 1.8000e-04 | norm: 0.1962 | dt: 14497.47ms | tok/sec: 36164.1 | mem: 108811MB | Time Left: 01 hrs : 43 mins [TIMING] data: 9.0ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10857.3ms | clip: 0.0ms | optim: 23.9ms step 321 | loss: 2.757743 | lr 1.8000e-04 | norm: 0.1996 | dt: 14507.97ms | tok/sec: 36137.9 | mem: 108811MB | Time Left: 01 hrs : 43 mins [TIMING] data: 9.4ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10867.9ms | clip: 0.0ms | optim: 23.9ms step 322 | loss: 2.812786 | lr 1.8000e-04 | norm: 0.2859 | dt: 14499.04ms | tok/sec: 36160.2 | mem: 108811MB | Time Left: 01 hrs : 43 mins [TIMING] data: 9.6ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10859.4ms | clip: 0.0ms | optim: 24.1ms step 323 | loss: 3.248081 | lr 1.8000e-04 | norm: 0.2046 | dt: 14501.53ms | tok/sec: 36154.0 | mem: 108811MB | Time Left: 01 hrs : 43 mins [TIMING] data: 9.2ms | fwd: 3592.0ms | loss: 13.2ms | bwd: 10848.7ms | clip: 0.0ms | optim: 23.9ms step 324 | loss: 2.800618 | lr 1.8000e-04 | norm: 0.3344 | dt: 14505.39ms | tok/sec: 36144.4 | mem: 108811MB | Time Left: 01 hrs : 42 mins [TIMING] data: 10.0ms | fwd: 3594.1ms | loss: 13.2ms | bwd: 10849.1ms | clip: 0.0ms | optim: 24.1ms step 325 | loss: 2.986109 | lr 1.8000e-04 | norm: 0.2092 | dt: 14503.01ms | tok/sec: 36150.3 | mem: 108811MB | Time Left: 01 hrs : 42 mins [TIMING] data: 10.1ms | fwd: 3579.9ms | loss: 13.2ms | bwd: 10859.8ms | clip: 0.0ms | optim: 24.0ms step 326 | loss: 2.672048 | lr 1.8000e-04 | norm: 0.1963 | dt: 14497.09ms | tok/sec: 36165.1 | mem: 108811MB | Time Left: 01 hrs : 42 mins [TIMING] data: 9.9ms | fwd: 3576.6ms | loss: 13.2ms | bwd: 10858.8ms | clip: 0.0ms | optim: 24.0ms step 327 | loss: 2.693366 | lr 1.8000e-04 | norm: 0.2199 | dt: 14504.77ms | tok/sec: 36145.9 | mem: 108811MB | Time Left: 01 hrs : 42 mins [TIMING] data: 10.4ms | fwd: 3602.2ms | loss: 13.2ms | bwd: 10839.7ms | clip: 0.0ms | optim: 24.0ms step 328 | loss: 2.965066 | lr 1.8000e-04 | norm: 0.2389 | dt: 14492.14ms | tok/sec: 36177.4 | mem: 108811MB | Time Left: 01 hrs : 41 mins [TIMING] data: 9.6ms | fwd: 3582.9ms | loss: 13.2ms | bwd: 10847.5ms | clip: 0.0ms | optim: 23.9ms step 329 | loss: 2.719521 | lr 1.8000e-04 | norm: 0.2312 | dt: 14496.72ms | tok/sec: 36166.0 | mem: 108811MB | Time Left: 01 hrs : 41 mins [TIMING] data: 9.2ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10858.7ms | clip: 0.0ms | optim: 24.1ms step 330 | loss: 2.882398 | lr 1.8000e-04 | norm: 0.2108 | dt: 14503.52ms | tok/sec: 36149.0 | mem: 108811MB | Time Left: 01 hrs : 41 mins [TIMING] data: 9.2ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10861.1ms | clip: 0.0ms | optim: 23.9ms step 331 | loss: 2.790061 | lr 1.8000e-04 | norm: 0.1844 | dt: 14578.06ms | tok/sec: 35964.2 | mem: 108811MB | Time Left: 01 hrs : 41 mins [TIMING] data: 9.2ms | fwd: 3667.7ms | loss: 13.1ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.9ms step 332 | loss: 2.865694 | lr 1.8000e-04 | norm: 0.2186 | dt: 14492.32ms | tok/sec: 36177.0 | mem: 108811MB | Time Left: 01 hrs : 40 mins [TIMING] data: 9.2ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10851.3ms | clip: 0.0ms | optim: 23.9ms step 333 | loss: 3.006808 | lr 1.8000e-04 | norm: 0.1956 | dt: 14499.43ms | tok/sec: 36159.2 | mem: 108811MB | Time Left: 01 hrs : 40 mins [TIMING] data: 9.2ms | fwd: 3593.3ms | loss: 13.1ms | bwd: 10845.5ms | clip: 0.0ms | optim: 24.1ms step 334 | loss: 3.108690 | lr 1.8000e-04 | norm: 0.2001 | dt: 14499.73ms | tok/sec: 36158.5 | mem: 108811MB | Time Left: 01 hrs : 40 mins [TIMING] data: 9.1ms | fwd: 3585.3ms | loss: 13.1ms | bwd: 10854.3ms | clip: 0.0ms | optim: 24.0ms step 335 | loss: 2.836473 | lr 1.8000e-04 | norm: 0.2135 | dt: 14495.13ms | tok/sec: 36169.9 | mem: 108811MB | Time Left: 01 hrs : 40 mins [TIMING] data: 9.1ms | fwd: 3580.6ms | loss: 13.1ms | bwd: 10854.5ms | clip: 0.0ms | optim: 23.9ms step 336 | loss: 2.975732 | lr 1.8000e-04 | norm: 0.2525 | dt: 14502.97ms | tok/sec: 36150.4 | mem: 108811MB | Time Left: 01 hrs : 40 mins [TIMING] data: 9.1ms | fwd: 3588.8ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 24.0ms step 337 | loss: 2.805744 | lr 1.8000e-04 | norm: 0.2986 | dt: 14501.73ms | tok/sec: 36153.5 | mem: 108811MB | Time Left: 01 hrs : 39 mins [TIMING] data: 9.1ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10859.3ms | clip: 0.0ms | optim: 24.0ms step 338 | loss: 3.027896 | lr 1.8000e-04 | norm: 0.2378 | dt: 14497.41ms | tok/sec: 36164.3 | mem: 108811MB | Time Left: 01 hrs : 39 mins [TIMING] data: 9.2ms | fwd: 3603.1ms | loss: 13.0ms | bwd: 10834.4ms | clip: 0.0ms | optim: 23.9ms step 339 | loss: 2.834209 | lr 1.8000e-04 | norm: 0.2498 | dt: 14512.25ms | tok/sec: 36127.3 | mem: 108811MB | Time Left: 01 hrs : 39 mins [TIMING] data: 9.2ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10868.3ms | clip: 0.0ms | optim: 23.8ms step 340 | loss: 3.100985 | lr 1.8000e-04 | norm: 0.2382 | dt: 14496.33ms | tok/sec: 36167.0 | mem: 108811MB | Time Left: 01 hrs : 39 mins [TIMING] data: 9.1ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10857.0ms | clip: 0.0ms | optim: 24.0ms step 341 | loss: 2.985726 | lr 1.8000e-04 | norm: 0.2138 | dt: 14495.35ms | tok/sec: 36169.4 | mem: 108811MB | Time Left: 01 hrs : 38 mins [TIMING] data: 9.1ms | fwd: 3594.0ms | loss: 13.1ms | bwd: 10840.9ms | clip: 0.0ms | optim: 24.0ms step 342 | loss: 2.949282 | lr 1.8000e-04 | norm: 0.2255 | dt: 14497.41ms | tok/sec: 36164.3 | mem: 108811MB | Time Left: 01 hrs : 38 mins [TIMING] data: 9.1ms | fwd: 3594.6ms | loss: 13.0ms | bwd: 10842.9ms | clip: 0.0ms | optim: 23.8ms step 343 | loss: 2.790857 | lr 1.8000e-04 | norm: 0.2213 | dt: 14508.12ms | tok/sec: 36137.5 | mem: 108811MB | Time Left: 01 hrs : 38 mins [TIMING] data: 9.6ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10867.5ms | clip: 0.0ms | optim: 24.2ms step 344 | loss: 2.932796 | lr 1.8000e-04 | norm: 0.2199 | dt: 14494.73ms | tok/sec: 36171.0 | mem: 108811MB | Time Left: 01 hrs : 38 mins [TIMING] data: 9.2ms | fwd: 3595.1ms | loss: 13.0ms | bwd: 10839.5ms | clip: 0.0ms | optim: 23.9ms step 345 | loss: 2.643596 | lr 1.8000e-04 | norm: 0.2646 | dt: 14508.29ms | tok/sec: 36137.1 | mem: 108811MB | Time Left: 01 hrs : 37 mins [TIMING] data: 9.1ms | fwd: 3584.6ms | loss: 13.1ms | bwd: 10863.6ms | clip: 0.0ms | optim: 23.9ms step 346 | loss: 2.721446 | lr 1.8000e-04 | norm: 0.2259 | dt: 14503.51ms | tok/sec: 36149.0 | mem: 108811MB | Time Left: 01 hrs : 37 mins [TIMING] data: 9.2ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10863.2ms | clip: 0.0ms | optim: 24.0ms step 347 | loss: 2.657533 | lr 1.8000e-04 | norm: 0.2114 | dt: 14501.47ms | tok/sec: 36154.1 | mem: 108811MB | Time Left: 01 hrs : 37 mins [TIMING] data: 9.1ms | fwd: 3595.0ms | loss: 13.1ms | bwd: 10846.2ms | clip: 0.0ms | optim: 23.9ms step 348 | loss: 3.336380 | lr 1.8000e-04 | norm: 0.2542 | dt: 14496.70ms | tok/sec: 36166.0 | mem: 108811MB | Time Left: 01 hrs : 37 mins [TIMING] data: 9.2ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10856.7ms | clip: 0.0ms | optim: 23.9ms step 349 | loss: 3.033248 | lr 1.8000e-04 | norm: 0.2122 | dt: 14511.16ms | tok/sec: 36130.0 | mem: 108811MB | Time Left: 01 hrs : 36 mins [TIMING] data: 9.1ms | fwd: 3583.4ms | loss: 13.1ms | bwd: 10867.7ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_350.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_300.ckpt step 350 | loss: 3.073126 | lr 1.8000e-04 | norm: 0.2709 | dt: 14354.57ms | tok/sec: 36524.1 | mem: 108811MB | Time Left: 01 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3568.9ms | loss: 13.1ms | bwd: 10725.3ms | clip: 0.0ms | optim: 24.0ms step 351 | loss: 2.864077 | lr 1.8000e-04 | norm: 0.2032 | dt: 14392.44ms | tok/sec: 36428.0 | mem: 108811MB | Time Left: 01 hrs : 35 mins [TIMING] data: 9.2ms | fwd: 3560.6ms | loss: 13.0ms | bwd: 10771.9ms | clip: 0.0ms | optim: 23.9ms step 352 | loss: 3.181767 | lr 1.8000e-04 | norm: 0.2261 | dt: 14430.45ms | tok/sec: 36332.1 | mem: 108811MB | Time Left: 01 hrs : 35 mins [TIMING] data: 9.2ms | fwd: 3570.7ms | loss: 13.0ms | bwd: 10799.5ms | clip: 0.0ms | optim: 24.0ms step 353 | loss: 3.081780 | lr 1.8000e-04 | norm: 0.3285 | dt: 14459.80ms | tok/sec: 36258.3 | mem: 108811MB | Time Left: 01 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3572.2ms | loss: 13.0ms | bwd: 10827.8ms | clip: 0.0ms | optim: 23.9ms step 354 | loss: 2.810652 | lr 1.8000e-04 | norm: 0.2437 | dt: 14488.62ms | tok/sec: 36186.2 | mem: 108811MB | Time Left: 01 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3579.5ms | loss: 13.0ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.9ms step 355 | loss: 3.105063 | lr 1.8000e-04 | norm: 0.2149 | dt: 14497.82ms | tok/sec: 36163.2 | mem: 108811MB | Time Left: 01 hrs : 35 mins [TIMING] data: 9.2ms | fwd: 3577.8ms | loss: 13.2ms | bwd: 10859.3ms | clip: 0.0ms | optim: 23.9ms step 356 | loss: 3.133838 | lr 1.8000e-04 | norm: 0.1992 | dt: 14507.29ms | tok/sec: 36139.6 | mem: 108811MB | Time Left: 01 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3591.3ms | loss: 13.0ms | bwd: 10855.9ms | clip: 0.0ms | optim: 24.0ms step 357 | loss: 3.123127 | lr 1.8000e-04 | norm: 0.2422 | dt: 14491.21ms | tok/sec: 36179.7 | mem: 108811MB | Time Left: 01 hrs : 34 mins [TIMING] data: 9.1ms | fwd: 3594.7ms | loss: 13.0ms | bwd: 10836.6ms | clip: 0.0ms | optim: 23.9ms step 358 | loss: 2.969609 | lr 1.8000e-04 | norm: 0.1988 | dt: 14517.88ms | tok/sec: 36113.3 | mem: 108811MB | Time Left: 01 hrs : 34 mins [TIMING] data: 9.2ms | fwd: 3584.4ms | loss: 13.1ms | bwd: 10873.4ms | clip: 0.0ms | optim: 24.0ms step 359 | loss: 2.691902 | lr 1.8000e-04 | norm: 0.1857 | dt: 14515.84ms | tok/sec: 36118.3 | mem: 108811MB | Time Left: 01 hrs : 34 mins [TIMING] data: 9.3ms | fwd: 3586.7ms | loss: 13.0ms | bwd: 10869.0ms | clip: 0.0ms | optim: 24.0ms step 360 | loss: 2.831034 | lr 1.8000e-04 | norm: 0.2068 | dt: 14521.59ms | tok/sec: 36104.0 | mem: 108811MB | Time Left: 01 hrs : 34 mins [TIMING] data: 9.1ms | fwd: 3586.8ms | loss: 13.1ms | bwd: 10874.7ms | clip: 0.0ms | optim: 24.0ms step 361 | loss: 3.076149 | lr 1.8000e-04 | norm: 0.2306 | dt: 14522.89ms | tok/sec: 36100.8 | mem: 108811MB | Time Left: 01 hrs : 34 mins [TIMING] data: 9.2ms | fwd: 3587.2ms | loss: 13.1ms | bwd: 10875.5ms | clip: 0.0ms | optim: 23.9ms step 362 | loss: 2.937885 | lr 1.8000e-04 | norm: 0.2447 | dt: 14514.74ms | tok/sec: 36121.1 | mem: 108811MB | Time Left: 01 hrs : 33 mins [TIMING] data: 9.2ms | fwd: 3584.5ms | loss: 13.0ms | bwd: 10870.1ms | clip: 0.0ms | optim: 24.0ms step 363 | loss: 2.849757 | lr 1.8000e-04 | norm: 0.3352 | dt: 14510.57ms | tok/sec: 36131.5 | mem: 108811MB | Time Left: 01 hrs : 33 mins [TIMING] data: 9.1ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10868.9ms | clip: 0.0ms | optim: 23.9ms step 364 | loss: 3.038043 | lr 1.8000e-04 | norm: 0.2883 | dt: 14511.49ms | tok/sec: 36129.2 | mem: 108811MB | Time Left: 01 hrs : 33 mins [TIMING] data: 9.2ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10869.6ms | clip: 0.0ms | optim: 24.0ms step 365 | loss: 3.111445 | lr 1.8000e-04 | norm: 0.2306 | dt: 14506.92ms | tok/sec: 36140.5 | mem: 108811MB | Time Left: 01 hrs : 33 mins [TIMING] data: 9.2ms | fwd: 3581.3ms | loss: 13.1ms | bwd: 10865.5ms | clip: 0.0ms | optim: 24.0ms step 366 | loss: 2.911431 | lr 1.8000e-04 | norm: 0.2083 | dt: 14477.77ms | tok/sec: 36213.3 | mem: 108811MB | Time Left: 01 hrs : 32 mins [TIMING] data: 9.2ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10835.5ms | clip: 0.0ms | optim: 24.0ms step 367 | loss: 2.618045 | lr 1.8000e-04 | norm: 0.2025 | dt: 14498.49ms | tok/sec: 36161.6 | mem: 108811MB | Time Left: 01 hrs : 32 mins [TIMING] data: 9.1ms | fwd: 3597.0ms | loss: 13.1ms | bwd: 10841.5ms | clip: 0.0ms | optim: 24.0ms step 368 | loss: 2.974987 | lr 1.8000e-04 | norm: 0.1916 | dt: 14475.58ms | tok/sec: 36218.8 | mem: 108811MB | Time Left: 01 hrs : 32 mins [TIMING] data: 9.2ms | fwd: 3590.6ms | loss: 13.1ms | bwd: 10825.0ms | clip: 0.0ms | optim: 23.9ms step 369 | loss: 2.897172 | lr 1.8000e-04 | norm: 0.2622 | dt: 14490.98ms | tok/sec: 36180.3 | mem: 108811MB | Time Left: 01 hrs : 32 mins [TIMING] data: 9.2ms | fwd: 3590.9ms | loss: 13.1ms | bwd: 10839.8ms | clip: 0.0ms | optim: 23.9ms step 370 | loss: 3.164998 | lr 1.8000e-04 | norm: 0.3562 | dt: 14490.10ms | tok/sec: 36182.5 | mem: 108811MB | Time Left: 01 hrs : 31 mins [TIMING] data: 9.2ms | fwd: 3585.3ms | loss: 13.1ms | bwd: 10844.7ms | clip: 0.0ms | optim: 23.9ms step 371 | loss: 3.015636 | lr 1.8000e-04 | norm: 0.1931 | dt: 14500.35ms | tok/sec: 36156.9 | mem: 108811MB | Time Left: 01 hrs : 31 mins [TIMING] data: 9.2ms | fwd: 3594.6ms | loss: 13.0ms | bwd: 10845.6ms | clip: 0.0ms | optim: 24.0ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00002.bin 100000000 step 372 | loss: 2.780494 | lr 1.8000e-04 | norm: 0.1996 | dt: 14486.33ms | tok/sec: 36191.9 | mem: 108811MB | Time Left: 01 hrs : 31 mins [TIMING] data: 14.0ms | fwd: 3577.8ms | loss: 13.0ms | bwd: 10843.5ms | clip: 0.0ms | optim: 24.0ms step 373 | loss: 3.171426 | lr 1.8000e-04 | norm: 0.2462 | dt: 14493.65ms | tok/sec: 36173.6 | mem: 108811MB | Time Left: 01 hrs : 31 mins [TIMING] data: 9.1ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10850.8ms | clip: 0.0ms | optim: 24.0ms step 374 | loss: 2.990998 | lr 1.8000e-04 | norm: 0.2032 | dt: 14489.34ms | tok/sec: 36184.4 | mem: 108811MB | Time Left: 01 hrs : 30 mins [TIMING] data: 9.2ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10849.5ms | clip: 0.0ms | optim: 24.0ms step 375 | loss: 2.918159 | lr 1.8000e-04 | norm: 0.2027 | dt: 14500.85ms | tok/sec: 36155.7 | mem: 108811MB | Time Left: 01 hrs : 30 mins [TIMING] data: 9.0ms | fwd: 3580.3ms | loss: 13.0ms | bwd: 10860.7ms | clip: 0.0ms | optim: 23.9ms step 376 | loss: 3.316640 | lr 1.8000e-04 | norm: 0.2384 | dt: 14494.87ms | tok/sec: 36170.6 | mem: 108811MB | Time Left: 01 hrs : 30 mins [TIMING] data: 9.1ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10855.8ms | clip: 0.0ms | optim: 24.0ms step 377 | loss: 2.904143 | lr 1.8000e-04 | norm: 0.2007 | dt: 14498.45ms | tok/sec: 36161.7 | mem: 108811MB | Time Left: 01 hrs : 30 mins [TIMING] data: 9.1ms | fwd: 3580.0ms | loss: 13.0ms | bwd: 10857.4ms | clip: 0.0ms | optim: 25.0ms step 378 | loss: 2.599680 | lr 1.8000e-04 | norm: 0.2298 | dt: 14490.82ms | tok/sec: 36180.7 | mem: 108811MB | Time Left: 01 hrs : 29 mins [TIMING] data: 9.1ms | fwd: 3588.4ms | loss: 13.0ms | bwd: 10842.3ms | clip: 0.0ms | optim: 24.1ms step 379 | loss: 3.095214 | lr 1.8000e-04 | norm: 0.2763 | dt: 14495.32ms | tok/sec: 36169.5 | mem: 108811MB | Time Left: 01 hrs : 29 mins [TIMING] data: 9.1ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 24.0ms step 380 | loss: 3.253443 | lr 1.8000e-04 | norm: 0.2146 | dt: 14485.72ms | tok/sec: 36193.4 | mem: 108811MB | Time Left: 01 hrs : 29 mins [TIMING] data: 9.2ms | fwd: 3580.2ms | loss: 13.0ms | bwd: 10845.5ms | clip: 0.0ms | optim: 23.9ms step 381 | loss: 3.174277 | lr 1.8000e-04 | norm: 0.2285 | dt: 14493.86ms | tok/sec: 36173.1 | mem: 108811MB | Time Left: 01 hrs : 29 mins [TIMING] data: 9.2ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10854.8ms | clip: 0.0ms | optim: 24.7ms step 382 | loss: 2.888484 | lr 1.8000e-04 | norm: 0.1808 | dt: 14491.01ms | tok/sec: 36180.2 | mem: 108811MB | Time Left: 01 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10852.7ms | clip: 0.0ms | optim: 24.0ms step 383 | loss: 3.072024 | lr 1.8000e-04 | norm: 0.2417 | dt: 14491.53ms | tok/sec: 36178.9 | mem: 108811MB | Time Left: 01 hrs : 28 mins [TIMING] data: 9.3ms | fwd: 3582.0ms | loss: 13.0ms | bwd: 10849.3ms | clip: 0.0ms | optim: 24.0ms step 384 | loss: 2.870299 | lr 1.8000e-04 | norm: 0.1751 | dt: 14498.60ms | tok/sec: 36161.3 | mem: 108811MB | Time Left: 01 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3581.9ms | loss: 13.0ms | bwd: 10856.8ms | clip: 0.0ms | optim: 24.0ms step 385 | loss: 2.951061 | lr 1.8000e-04 | norm: 0.1962 | dt: 14476.81ms | tok/sec: 36215.7 | mem: 108811MB | Time Left: 01 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3583.4ms | loss: 13.0ms | bwd: 10833.4ms | clip: 0.0ms | optim: 24.0ms step 386 | loss: 3.037984 | lr 1.8000e-04 | norm: 0.1824 | dt: 14502.69ms | tok/sec: 36151.1 | mem: 108811MB | Time Left: 01 hrs : 27 mins [TIMING] data: 9.3ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10862.4ms | clip: 0.0ms | optim: 24.0ms step 387 | loss: 3.038720 | lr 1.8000e-04 | norm: 0.2691 | dt: 14501.93ms | tok/sec: 36153.0 | mem: 108811MB | Time Left: 01 hrs : 27 mins [TIMING] data: 9.2ms | fwd: 3583.5ms | loss: 13.0ms | bwd: 10858.2ms | clip: 0.0ms | optim: 23.9ms step 388 | loss: 2.772935 | lr 1.8000e-04 | norm: 0.2255 | dt: 14492.91ms | tok/sec: 36175.5 | mem: 108811MB | Time Left: 01 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3581.7ms | loss: 13.0ms | bwd: 10851.4ms | clip: 0.0ms | optim: 23.8ms step 389 | loss: 3.013582 | lr 1.8000e-04 | norm: 0.2031 | dt: 14561.86ms | tok/sec: 36004.2 | mem: 108811MB | Time Left: 01 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3663.8ms | loss: 13.1ms | bwd: 10836.8ms | clip: 0.0ms | optim: 23.9ms step 390 | loss: 2.853286 | lr 1.8000e-04 | norm: 0.1970 | dt: 14508.58ms | tok/sec: 36136.4 | mem: 108811MB | Time Left: 01 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3601.1ms | loss: 13.0ms | bwd: 10847.3ms | clip: 0.0ms | optim: 23.9ms step 391 | loss: 2.827247 | lr 1.8000e-04 | norm: 0.2673 | dt: 14496.89ms | tok/sec: 36165.6 | mem: 108811MB | Time Left: 01 hrs : 26 mins [TIMING] data: 9.2ms | fwd: 3586.9ms | loss: 13.1ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.9ms step 392 | loss: 2.820348 | lr 1.8000e-04 | norm: 0.2618 | dt: 14490.90ms | tok/sec: 36180.5 | mem: 108811MB | Time Left: 01 hrs : 26 mins [TIMING] data: 9.1ms | fwd: 3590.0ms | loss: 13.1ms | bwd: 10841.0ms | clip: 0.0ms | optim: 23.9ms step 393 | loss: 2.966884 | lr 1.8000e-04 | norm: 0.3450 | dt: 14502.68ms | tok/sec: 36151.1 | mem: 108811MB | Time Left: 01 hrs : 26 mins [TIMING] data: 9.0ms | fwd: 3601.0ms | loss: 13.0ms | bwd: 10841.7ms | clip: 0.0ms | optim: 24.0ms step 394 | loss: 2.899376 | lr 1.8000e-04 | norm: 0.2320 | dt: 14502.52ms | tok/sec: 36151.5 | mem: 108811MB | Time Left: 01 hrs : 26 mins [TIMING] data: 9.1ms | fwd: 3595.2ms | loss: 13.1ms | bwd: 10847.1ms | clip: 0.0ms | optim: 24.1ms step 395 | loss: 2.901861 | lr 1.8000e-04 | norm: 0.3148 | dt: 14505.15ms | tok/sec: 36144.9 | mem: 108811MB | Time Left: 01 hrs : 25 mins [TIMING] data: 9.2ms | fwd: 3590.1ms | loss: 13.1ms | bwd: 10854.9ms | clip: 0.0ms | optim: 24.0ms step 396 | loss: 2.938989 | lr 1.8000e-04 | norm: 0.1797 | dt: 14507.42ms | tok/sec: 36139.3 | mem: 108811MB | Time Left: 01 hrs : 25 mins [TIMING] data: 9.1ms | fwd: 3599.7ms | loss: 13.0ms | bwd: 10847.8ms | clip: 0.0ms | optim: 23.9ms step 397 | loss: 2.997095 | lr 1.8000e-04 | norm: 0.2665 | dt: 14488.45ms | tok/sec: 36186.6 | mem: 108811MB | Time Left: 01 hrs : 25 mins [TIMING] data: 9.2ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10848.0ms | clip: 0.0ms | optim: 24.0ms step 398 | loss: 3.338165 | lr 1.8000e-04 | norm: 0.2277 | dt: 14501.68ms | tok/sec: 36153.6 | mem: 108811MB | Time Left: 01 hrs : 25 mins [TIMING] data: 9.1ms | fwd: 3596.3ms | loss: 13.1ms | bwd: 10844.6ms | clip: 0.0ms | optim: 24.4ms step 399 | loss: 2.553559 | lr 1.8000e-04 | norm: 0.2206 | dt: 14493.41ms | tok/sec: 36174.2 | mem: 108811MB | Time Left: 01 hrs : 24 mins [TIMING] data: 9.1ms | fwd: 3579.7ms | loss: 13.0ms | bwd: 10853.7ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_400.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_350.ckpt step 400 | loss: 3.178947 | lr 1.8000e-04 | norm: 0.3468 | dt: 14424.87ms | tok/sec: 36346.1 | mem: 108811MB | Time Left: 01 hrs : 24 mins [TIMING] data: 9.4ms | fwd: 3618.0ms | loss: 13.1ms | bwd: 10746.1ms | clip: 0.0ms | optim: 23.9ms step 401 | loss: 2.629553 | lr 1.8000e-04 | norm: 2.0837 | dt: 14397.70ms | tok/sec: 36414.7 | mem: 108811MB | Time Left: 01 hrs : 23 mins [TIMING] data: 9.2ms | fwd: 3578.2ms | loss: 13.0ms | bwd: 10759.4ms | clip: 0.0ms | optim: 23.9ms step 402 | loss: 2.737635 | lr 1.8000e-04 | norm: 0.2259 | dt: 14433.62ms | tok/sec: 36324.1 | mem: 108811MB | Time Left: 01 hrs : 23 mins [TIMING] data: 9.2ms | fwd: 3571.6ms | loss: 13.2ms | bwd: 10801.7ms | clip: 0.0ms | optim: 24.0ms step 403 | loss: 3.257662 | lr 1.8000e-04 | norm: 0.2774 | dt: 14451.15ms | tok/sec: 36280.0 | mem: 108811MB | Time Left: 01 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10808.0ms | clip: 0.0ms | optim: 23.9ms step 404 | loss: 2.852297 | lr 1.8000e-04 | norm: 0.2261 | dt: 14555.89ms | tok/sec: 36019.0 | mem: 108811MB | Time Left: 01 hrs : 23 mins [TIMING] data: 9.3ms | fwd: 3663.3ms | loss: 13.1ms | bwd: 10832.2ms | clip: 0.0ms | optim: 24.0ms step 405 | loss: 2.929736 | lr 1.8000e-04 | norm: 0.2624 | dt: 14499.50ms | tok/sec: 36159.0 | mem: 108811MB | Time Left: 01 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3584.5ms | loss: 13.1ms | bwd: 10854.9ms | clip: 0.0ms | optim: 23.9ms step 406 | loss: 2.756514 | lr 1.8000e-04 | norm: 0.1693 | dt: 14496.55ms | tok/sec: 36166.4 | mem: 108811MB | Time Left: 01 hrs : 23 mins [TIMING] data: 9.2ms | fwd: 3577.0ms | loss: 13.0ms | bwd: 10859.6ms | clip: 0.0ms | optim: 24.0ms step 407 | loss: 3.271858 | lr 1.8000e-04 | norm: 0.3381 | dt: 14562.39ms | tok/sec: 36002.9 | mem: 108811MB | Time Left: 01 hrs : 23 mins [TIMING] data: 9.2ms | fwd: 3661.6ms | loss: 13.0ms | bwd: 10840.8ms | clip: 0.0ms | optim: 24.1ms step 408 | loss: 3.053077 | lr 1.8000e-04 | norm: 0.7241 | dt: 14508.79ms | tok/sec: 36135.9 | mem: 108811MB | Time Left: 01 hrs : 22 mins [TIMING] data: 9.0ms | fwd: 3584.5ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 23.9ms step 409 | loss: 2.905395 | lr 1.8000e-04 | norm: 0.2201 | dt: 14521.64ms | tok/sec: 36103.9 | mem: 108811MB | Time Left: 01 hrs : 22 mins [TIMING] data: 9.1ms | fwd: 3585.6ms | loss: 13.0ms | bwd: 10876.1ms | clip: 0.0ms | optim: 23.9ms step 410 | loss: 2.843511 | lr 1.8000e-04 | norm: 0.2323 | dt: 14506.63ms | tok/sec: 36141.3 | mem: 108811MB | Time Left: 01 hrs : 22 mins [TIMING] data: 9.2ms | fwd: 3601.9ms | loss: 13.1ms | bwd: 10844.6ms | clip: 0.0ms | optim: 24.0ms step 411 | loss: 2.966667 | lr 1.8000e-04 | norm: 0.2425 | dt: 14513.13ms | tok/sec: 36125.1 | mem: 108811MB | Time Left: 01 hrs : 21 mins [TIMING] data: 9.2ms | fwd: 3599.6ms | loss: 13.0ms | bwd: 10853.6ms | clip: 0.0ms | optim: 23.9ms step 412 | loss: 2.927269 | lr 1.8000e-04 | norm: 2.4847 | dt: 14506.45ms | tok/sec: 36141.7 | mem: 108811MB | Time Left: 01 hrs : 21 mins [TIMING] data: 9.1ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10863.5ms | clip: 0.0ms | optim: 23.9ms step 413 | loss: 2.765959 | lr 1.8000e-04 | norm: 0.2330 | dt: 14522.24ms | tok/sec: 36102.4 | mem: 108811MB | Time Left: 01 hrs : 21 mins [TIMING] data: 9.2ms | fwd: 3581.6ms | loss: 13.0ms | bwd: 10880.6ms | clip: 0.0ms | optim: 23.9ms step 414 | loss: 3.211147 | lr 1.8000e-04 | norm: 0.2488 | dt: 14509.16ms | tok/sec: 36135.0 | mem: 108811MB | Time Left: 01 hrs : 21 mins [TIMING] data: 9.2ms | fwd: 3586.9ms | loss: 13.0ms | bwd: 10862.2ms | clip: 0.0ms | optim: 23.9ms step 415 | loss: 2.959682 | lr 1.8000e-04 | norm: 0.2899 | dt: 14528.26ms | tok/sec: 36087.5 | mem: 108811MB | Time Left: 01 hrs : 21 mins [TIMING] data: 9.2ms | fwd: 3582.7ms | loss: 13.0ms | bwd: 10885.5ms | clip: 0.0ms | optim: 23.9ms step 416 | loss: 3.177491 | lr 1.8000e-04 | norm: 0.2449 | dt: 14513.02ms | tok/sec: 36125.4 | mem: 108811MB | Time Left: 01 hrs : 20 mins [TIMING] data: 9.1ms | fwd: 3585.4ms | loss: 13.1ms | bwd: 10867.3ms | clip: 0.0ms | optim: 24.1ms step 417 | loss: 2.841663 | lr 1.8000e-04 | norm: 0.2205 | dt: 14510.09ms | tok/sec: 36132.7 | mem: 108811MB | Time Left: 01 hrs : 20 mins [TIMING] data: 9.3ms | fwd: 3583.2ms | loss: 13.0ms | bwd: 10866.9ms | clip: 0.0ms | optim: 23.9ms step 418 | loss: 3.043510 | lr 1.8000e-04 | norm: 0.2336 | dt: 14514.31ms | tok/sec: 36122.2 | mem: 108811MB | Time Left: 01 hrs : 20 mins [TIMING] data: 9.2ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10873.0ms | clip: 0.0ms | optim: 24.0ms step 419 | loss: 3.109581 | lr 1.8000e-04 | norm: 0.2206 | dt: 14507.40ms | tok/sec: 36139.3 | mem: 108811MB | Time Left: 01 hrs : 20 mins [TIMING] data: 9.2ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10866.3ms | clip: 0.0ms | optim: 23.9ms step 420 | loss: 2.907886 | lr 1.8000e-04 | norm: 0.2256 | dt: 14505.19ms | tok/sec: 36144.9 | mem: 108811MB | Time Left: 01 hrs : 19 mins [TIMING] data: 9.3ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10863.0ms | clip: 0.0ms | optim: 23.9ms step 421 | loss: 2.972862 | lr 1.8000e-04 | norm: 0.6469 | dt: 14502.43ms | tok/sec: 36151.7 | mem: 108811MB | Time Left: 01 hrs : 19 mins [TIMING] data: 9.3ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10860.2ms | clip: 0.0ms | optim: 23.9ms step 422 | loss: 3.041441 | lr 1.8000e-04 | norm: 0.2141 | dt: 14504.33ms | tok/sec: 36147.0 | mem: 108811MB | Time Left: 01 hrs : 19 mins [TIMING] data: 9.3ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10862.1ms | clip: 0.0ms | optim: 23.9ms step 423 | loss: 2.973965 | lr 1.8000e-04 | norm: 0.3313 | dt: 14497.94ms | tok/sec: 36162.9 | mem: 108811MB | Time Left: 01 hrs : 19 mins [TIMING] data: 9.2ms | fwd: 3578.0ms | loss: 13.1ms | bwd: 10859.8ms | clip: 0.0ms | optim: 23.9ms step 424 | loss: 2.879693 | lr 1.8000e-04 | norm: 0.5485 | dt: 14502.56ms | tok/sec: 36151.4 | mem: 108811MB | Time Left: 01 hrs : 18 mins [TIMING] data: 9.1ms | fwd: 3577.6ms | loss: 13.0ms | bwd: 10865.0ms | clip: 0.0ms | optim: 24.0ms step 425 | loss: 2.845224 | lr 1.8000e-04 | norm: 0.2153 | dt: 14500.83ms | tok/sec: 36155.7 | mem: 108811MB | Time Left: 01 hrs : 18 mins [TIMING] data: 9.1ms | fwd: 3580.6ms | loss: 13.1ms | bwd: 10860.1ms | clip: 0.0ms | optim: 24.0ms step 426 | loss: 3.082062 | lr 1.8000e-04 | norm: 0.2032 | dt: 14513.82ms | tok/sec: 36123.4 | mem: 108811MB | Time Left: 01 hrs : 18 mins [TIMING] data: 9.2ms | fwd: 3584.4ms | loss: 13.0ms | bwd: 10869.0ms | clip: 0.0ms | optim: 24.2ms step 427 | loss: 2.734458 | lr 1.8000e-04 | norm: 0.1963 | dt: 14489.10ms | tok/sec: 36185.0 | mem: 108811MB | Time Left: 01 hrs : 17 mins [TIMING] data: 9.2ms | fwd: 3592.0ms | loss: 13.1ms | bwd: 10837.0ms | clip: 0.0ms | optim: 23.9ms step 428 | loss: 2.933876 | lr 1.8000e-04 | norm: 0.2355 | dt: 14501.47ms | tok/sec: 36154.1 | mem: 108811MB | Time Left: 01 hrs : 17 mins [TIMING] data: 9.2ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10858.7ms | clip: 0.0ms | optim: 24.0ms step 429 | loss: 2.884275 | lr 1.8000e-04 | norm: 0.5047 | dt: 14504.58ms | tok/sec: 36146.4 | mem: 108811MB | Time Left: 01 hrs : 17 mins [TIMING] data: 9.1ms | fwd: 3601.8ms | loss: 13.1ms | bwd: 10842.6ms | clip: 0.0ms | optim: 24.2ms step 430 | loss: 2.869512 | lr 1.8000e-04 | norm: 0.2412 | dt: 14486.61ms | tok/sec: 36191.2 | mem: 108811MB | Time Left: 01 hrs : 17 mins [TIMING] data: 9.2ms | fwd: 3589.1ms | loss: 13.0ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.9ms step 431 | loss: 2.773499 | lr 1.8000e-04 | norm: 0.2914 | dt: 14499.51ms | tok/sec: 36159.0 | mem: 108811MB | Time Left: 01 hrs : 17 mins [TIMING] data: 9.2ms | fwd: 3580.1ms | loss: 13.0ms | bwd: 10859.1ms | clip: 0.0ms | optim: 24.0ms step 432 | loss: 2.862323 | lr 1.8000e-04 | norm: 0.2272 | dt: 14490.92ms | tok/sec: 36180.4 | mem: 108811MB | Time Left: 01 hrs : 16 mins [TIMING] data: 9.2ms | fwd: 3578.0ms | loss: 13.0ms | bwd: 10852.8ms | clip: 0.0ms | optim: 24.0ms step 433 | loss: 3.112463 | lr 1.8000e-04 | norm: 0.2397 | dt: 14492.91ms | tok/sec: 36175.5 | mem: 108811MB | Time Left: 01 hrs : 16 mins [TIMING] data: 9.1ms | fwd: 3595.5ms | loss: 13.0ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.9ms step 434 | loss: 2.996111 | lr 1.8000e-04 | norm: 0.2072 | dt: 14499.99ms | tok/sec: 36157.8 | mem: 108811MB | Time Left: 01 hrs : 16 mins [TIMING] data: 9.1ms | fwd: 3598.1ms | loss: 13.1ms | bwd: 10841.8ms | clip: 0.0ms | optim: 24.1ms step 435 | loss: 3.183437 | lr 1.8000e-04 | norm: 0.2514 | dt: 14488.62ms | tok/sec: 36186.2 | mem: 108811MB | Time Left: 01 hrs : 16 mins [TIMING] data: 9.1ms | fwd: 3579.9ms | loss: 13.0ms | bwd: 10848.6ms | clip: 0.0ms | optim: 23.9ms step 436 | loss: 3.004741 | lr 1.8000e-04 | norm: 0.2368 | dt: 14497.71ms | tok/sec: 36163.5 | mem: 108811MB | Time Left: 01 hrs : 15 mins [TIMING] data: 9.3ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.9ms step 437 | loss: 3.174088 | lr 1.8000e-04 | norm: 0.2177 | dt: 14485.99ms | tok/sec: 36192.8 | mem: 108811MB | Time Left: 01 hrs : 15 mins [TIMING] data: 9.2ms | fwd: 3596.7ms | loss: 13.0ms | bwd: 10828.7ms | clip: 0.0ms | optim: 23.9ms step 438 | loss: 3.144621 | lr 1.8000e-04 | norm: 0.2279 | dt: 14495.86ms | tok/sec: 36168.1 | mem: 108811MB | Time Left: 01 hrs : 15 mins [TIMING] data: 9.2ms | fwd: 3586.9ms | loss: 13.1ms | bwd: 10848.3ms | clip: 0.0ms | optim: 23.9ms step 439 | loss: 2.961333 | lr 1.8000e-04 | norm: 1.3521 | dt: 14486.44ms | tok/sec: 36191.6 | mem: 108811MB | Time Left: 01 hrs : 15 mins [TIMING] data: 9.2ms | fwd: 3584.0ms | loss: 13.1ms | bwd: 10842.2ms | clip: 0.0ms | optim: 23.9ms step 440 | loss: 3.042739 | lr 1.8000e-04 | norm: 0.2069 | dt: 14478.04ms | tok/sec: 36212.6 | mem: 108811MB | Time Left: 01 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3592.8ms | loss: 13.1ms | bwd: 10825.1ms | clip: 0.0ms | optim: 23.9ms step 441 | loss: 2.777756 | lr 1.8000e-04 | norm: 0.1955 | dt: 14496.79ms | tok/sec: 36165.8 | mem: 108811MB | Time Left: 01 hrs : 14 mins [TIMING] data: 9.2ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10852.5ms | clip: 0.0ms | optim: 24.0ms step 442 | loss: 3.000081 | lr 1.8000e-04 | norm: 0.2254 | dt: 14493.62ms | tok/sec: 36173.7 | mem: 108811MB | Time Left: 01 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10851.1ms | clip: 0.0ms | optim: 24.0ms step 443 | loss: 2.790348 | lr 1.8000e-04 | norm: 0.2529 | dt: 14553.08ms | tok/sec: 36025.9 | mem: 108811MB | Time Left: 01 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3661.5ms | loss: 13.0ms | bwd: 10831.6ms | clip: 0.0ms | optim: 23.9ms step 444 | loss: 2.849119 | lr 1.8000e-04 | norm: 0.2326 | dt: 14494.48ms | tok/sec: 36171.6 | mem: 108811MB | Time Left: 01 hrs : 13 mins [TIMING] data: 9.2ms | fwd: 3584.9ms | loss: 13.1ms | bwd: 10849.5ms | clip: 0.0ms | optim: 23.9ms step 445 | loss: 2.583553 | lr 1.8000e-04 | norm: 0.1818 | dt: 14488.57ms | tok/sec: 36186.3 | mem: 108811MB | Time Left: 01 hrs : 13 mins [TIMING] data: 9.3ms | fwd: 3592.4ms | loss: 13.1ms | bwd: 10835.8ms | clip: 0.0ms | optim: 24.0ms step 446 | loss: 2.929541 | lr 1.8000e-04 | norm: 0.2461 | dt: 14494.54ms | tok/sec: 36171.4 | mem: 108811MB | Time Left: 01 hrs : 13 mins [TIMING] data: 9.2ms | fwd: 3595.9ms | loss: 13.1ms | bwd: 10838.2ms | clip: 0.0ms | optim: 23.9ms step 447 | loss: 2.757211 | lr 1.8000e-04 | norm: 0.3089 | dt: 14506.78ms | tok/sec: 36140.9 | mem: 108811MB | Time Left: 01 hrs : 13 mins [TIMING] data: 9.1ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10863.0ms | clip: 0.0ms | optim: 24.0ms step 448 | loss: 3.052965 | lr 1.8000e-04 | norm: 0.2008 | dt: 14489.25ms | tok/sec: 36184.6 | mem: 108811MB | Time Left: 01 hrs : 12 mins [TIMING] data: 9.1ms | fwd: 3578.4ms | loss: 13.2ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms step 449 | loss: 3.052673 | lr 1.8000e-04 | norm: 0.1963 | dt: 14484.74ms | tok/sec: 36195.9 | mem: 108811MB | Time Left: 01 hrs : 12 mins [TIMING] data: 9.0ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10841.9ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_450.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_400.ckpt step 450 | loss: 2.907457 | lr 1.8000e-04 | norm: 0.2200 | dt: 15215.37ms | tok/sec: 34457.8 | mem: 108811MB | Time Left: 01 hrs : 16 mins [TIMING] data: 9.6ms | fwd: 4417.7ms | loss: 13.2ms | bwd: 10736.0ms | clip: 0.0ms | optim: 24.0ms step 451 | loss: 2.982053 | lr 1.8000e-04 | norm: 0.2127 | dt: 14396.33ms | tok/sec: 36418.2 | mem: 108811MB | Time Left: 01 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3571.8ms | loss: 13.1ms | bwd: 10764.5ms | clip: 0.0ms | optim: 24.0ms step 452 | loss: 2.846790 | lr 1.8000e-04 | norm: 0.2251 | dt: 14418.27ms | tok/sec: 36362.8 | mem: 108811MB | Time Left: 01 hrs : 11 mins [TIMING] data: 9.2ms | fwd: 3571.7ms | loss: 13.1ms | bwd: 10786.3ms | clip: 0.0ms | optim: 23.9ms step 453 | loss: 2.713422 | lr 1.8000e-04 | norm: 0.2382 | dt: 14458.43ms | tok/sec: 36261.7 | mem: 108811MB | Time Left: 01 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3575.9ms | loss: 13.0ms | bwd: 10822.7ms | clip: 0.0ms | optim: 23.9ms step 454 | loss: 2.676862 | lr 1.8000e-04 | norm: 0.3162 | dt: 14494.90ms | tok/sec: 36170.5 | mem: 108811MB | Time Left: 01 hrs : 11 mins [TIMING] data: 9.0ms | fwd: 3579.7ms | loss: 13.0ms | bwd: 10855.4ms | clip: 0.0ms | optim: 23.9ms step 455 | loss: 3.104548 | lr 1.8000e-04 | norm: 0.2128 | dt: 14510.19ms | tok/sec: 36132.4 | mem: 108811MB | Time Left: 01 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3588.7ms | loss: 13.0ms | bwd: 10861.5ms | clip: 0.0ms | optim: 23.9ms step 456 | loss: 3.023906 | lr 1.8000e-04 | norm: 0.1996 | dt: 14517.66ms | tok/sec: 36113.8 | mem: 108811MB | Time Left: 01 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10872.9ms | clip: 0.0ms | optim: 23.9ms step 457 | loss: 3.283628 | lr 1.8000e-04 | norm: 0.2045 | dt: 14549.79ms | tok/sec: 36034.1 | mem: 108811MB | Time Left: 01 hrs : 11 mins [TIMING] data: 9.4ms | fwd: 3638.3ms | loss: 13.1ms | bwd: 10851.2ms | clip: 0.0ms | optim: 24.0ms step 458 | loss: 2.879846 | lr 1.8000e-04 | norm: 0.4000 | dt: 14502.25ms | tok/sec: 36152.2 | mem: 108811MB | Time Left: 01 hrs : 10 mins [TIMING] data: 9.3ms | fwd: 3587.5ms | loss: 13.1ms | bwd: 10854.4ms | clip: 0.0ms | optim: 24.0ms step 459 | loss: 3.165325 | lr 1.8000e-04 | norm: 0.2699 | dt: 14550.09ms | tok/sec: 36033.3 | mem: 108811MB | Time Left: 01 hrs : 10 mins [TIMING] data: 9.1ms | fwd: 3632.9ms | loss: 13.1ms | bwd: 10857.1ms | clip: 0.0ms | optim: 23.9ms step 460 | loss: 3.089573 | lr 1.8000e-04 | norm: 0.3143 | dt: 14546.77ms | tok/sec: 36041.5 | mem: 108811MB | Time Left: 01 hrs : 10 mins [TIMING] data: 9.2ms | fwd: 3640.1ms | loss: 13.0ms | bwd: 10846.6ms | clip: 0.0ms | optim: 24.0ms step 461 | loss: 2.887126 | lr 1.8000e-04 | norm: 0.2271 | dt: 14522.03ms | tok/sec: 36102.9 | mem: 108811MB | Time Left: 01 hrs : 09 mins [TIMING] data: 9.2ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10879.5ms | clip: 0.0ms | optim: 24.0ms step 462 | loss: 3.283305 | lr 1.8000e-04 | norm: 0.2230 | dt: 14532.19ms | tok/sec: 36077.7 | mem: 108811MB | Time Left: 01 hrs : 09 mins [TIMING] data: 9.6ms | fwd: 3616.2ms | loss: 13.0ms | bwd: 10855.4ms | clip: 0.0ms | optim: 23.9ms step 463 | loss: 2.873415 | lr 1.8000e-04 | norm: 0.2218 | dt: 14522.67ms | tok/sec: 36101.4 | mem: 108811MB | Time Left: 01 hrs : 09 mins [TIMING] data: 9.1ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10878.6ms | clip: 0.0ms | optim: 23.9ms step 464 | loss: 2.751074 | lr 1.8000e-04 | norm: 0.1958 | dt: 14510.49ms | tok/sec: 36131.7 | mem: 108811MB | Time Left: 01 hrs : 09 mins [TIMING] data: 9.1ms | fwd: 3595.7ms | loss: 13.1ms | bwd: 10854.5ms | clip: 0.0ms | optim: 24.0ms step 465 | loss: 2.969721 | lr 1.8000e-04 | norm: 0.7927 | dt: 14507.54ms | tok/sec: 36139.0 | mem: 108811MB | Time Left: 01 hrs : 08 mins [TIMING] data: 9.2ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10865.5ms | clip: 0.0ms | optim: 23.9ms step 466 | loss: 3.396869 | lr 1.8000e-04 | norm: 0.2675 | dt: 14508.15ms | tok/sec: 36137.5 | mem: 108811MB | Time Left: 01 hrs : 08 mins [TIMING] data: 9.1ms | fwd: 3584.5ms | loss: 13.1ms | bwd: 10863.1ms | clip: 0.0ms | optim: 24.2ms step 467 | loss: 3.006928 | lr 1.8000e-04 | norm: 0.2402 | dt: 14503.66ms | tok/sec: 36148.7 | mem: 108811MB | Time Left: 01 hrs : 08 mins [TIMING] data: 9.0ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10863.7ms | clip: 0.0ms | optim: 24.0ms step 468 | loss: 2.904657 | lr 1.8000e-04 | norm: 0.2017 | dt: 14522.54ms | tok/sec: 36101.7 | mem: 108811MB | Time Left: 01 hrs : 08 mins [TIMING] data: 9.1ms | fwd: 3584.3ms | loss: 13.0ms | bwd: 10878.2ms | clip: 0.0ms | optim: 24.1ms step 469 | loss: 3.084258 | lr 1.8000e-04 | norm: 0.2924 | dt: 14514.20ms | tok/sec: 36122.4 | mem: 108811MB | Time Left: 01 hrs : 07 mins [TIMING] data: 9.0ms | fwd: 3582.4ms | loss: 13.0ms | bwd: 10871.9ms | clip: 0.0ms | optim: 23.9ms step 470 | loss: 2.924728 | lr 1.8000e-04 | norm: 0.2407 | dt: 14499.87ms | tok/sec: 36158.1 | mem: 108811MB | Time Left: 01 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3581.3ms | loss: 13.1ms | bwd: 10858.4ms | clip: 0.0ms | optim: 24.0ms step 471 | loss: 2.855034 | lr 1.8000e-04 | norm: 0.2180 | dt: 14509.70ms | tok/sec: 36133.6 | mem: 108811MB | Time Left: 01 hrs : 07 mins [TIMING] data: 8.9ms | fwd: 3585.7ms | loss: 13.0ms | bwd: 10864.2ms | clip: 0.0ms | optim: 23.9ms step 472 | loss: 2.748672 | lr 1.8000e-04 | norm: 0.2299 | dt: 14509.13ms | tok/sec: 36135.0 | mem: 108811MB | Time Left: 01 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3586.9ms | loss: 13.0ms | bwd: 10862.3ms | clip: 0.0ms | optim: 23.9ms step 473 | loss: 3.003809 | lr 1.8000e-04 | norm: 0.2104 | dt: 14487.32ms | tok/sec: 36189.4 | mem: 108811MB | Time Left: 01 hrs : 06 mins [TIMING] data: 9.0ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10844.2ms | clip: 0.0ms | optim: 23.9ms step 474 | loss: 2.942260 | lr 1.8000e-04 | norm: 0.3702 | dt: 14498.76ms | tok/sec: 36160.9 | mem: 108811MB | Time Left: 01 hrs : 06 mins [TIMING] data: 9.0ms | fwd: 3584.0ms | loss: 13.0ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.9ms step 475 | loss: 2.835461 | lr 1.8000e-04 | norm: 0.2068 | dt: 14496.56ms | tok/sec: 36166.4 | mem: 108811MB | Time Left: 01 hrs : 06 mins [TIMING] data: 9.0ms | fwd: 3583.3ms | loss: 13.0ms | bwd: 10853.5ms | clip: 0.0ms | optim: 23.9ms step 476 | loss: 2.849000 | lr 1.8000e-04 | norm: 0.2193 | dt: 14491.43ms | tok/sec: 36179.2 | mem: 108811MB | Time Left: 01 hrs : 06 mins [TIMING] data: 10.0ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10854.4ms | clip: 0.0ms | optim: 23.8ms step 477 | loss: 2.725057 | lr 1.8000e-04 | norm: 0.2475 | dt: 14479.97ms | tok/sec: 36207.8 | mem: 108811MB | Time Left: 01 hrs : 05 mins [TIMING] data: 9.1ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10841.1ms | clip: 0.0ms | optim: 23.9ms step 478 | loss: 3.135294 | lr 1.8000e-04 | norm: 0.3503 | dt: 14488.12ms | tok/sec: 36187.4 | mem: 108811MB | Time Left: 01 hrs : 05 mins [TIMING] data: 9.1ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10848.9ms | clip: 0.0ms | optim: 23.9ms step 479 | loss: 2.890121 | lr 1.8000e-04 | norm: 0.2158 | dt: 14493.57ms | tok/sec: 36173.8 | mem: 108811MB | Time Left: 01 hrs : 05 mins [TIMING] data: 9.0ms | fwd: 3579.3ms | loss: 13.0ms | bwd: 10854.5ms | clip: 0.0ms | optim: 23.9ms step 480 | loss: 2.927031 | lr 1.8000e-04 | norm: 0.2064 | dt: 14472.44ms | tok/sec: 36226.6 | mem: 108811MB | Time Left: 01 hrs : 05 mins [TIMING] data: 9.0ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10833.4ms | clip: 0.0ms | optim: 23.9ms step 481 | loss: 3.147365 | lr 1.8000e-04 | norm: 0.2220 | dt: 14483.44ms | tok/sec: 36199.1 | mem: 108811MB | Time Left: 01 hrs : 04 mins [TIMING] data: 9.1ms | fwd: 3581.5ms | loss: 13.0ms | bwd: 10841.5ms | clip: 0.0ms | optim: 23.9ms step 482 | loss: 3.096213 | lr 1.8000e-04 | norm: 0.2081 | dt: 14492.22ms | tok/sec: 36177.2 | mem: 108811MB | Time Left: 01 hrs : 04 mins [TIMING] data: 9.1ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10851.5ms | clip: 0.0ms | optim: 23.9ms step 483 | loss: 2.814423 | lr 1.8000e-04 | norm: 0.1870 | dt: 14485.31ms | tok/sec: 36194.5 | mem: 108811MB | Time Left: 01 hrs : 04 mins [TIMING] data: 9.1ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10848.4ms | clip: 0.0ms | optim: 23.9ms step 484 | loss: 3.030839 | lr 1.8000e-04 | norm: 0.2519 | dt: 14491.27ms | tok/sec: 36179.6 | mem: 108811MB | Time Left: 01 hrs : 04 mins [TIMING] data: 9.2ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10849.5ms | clip: 0.0ms | optim: 24.0ms step 485 | loss: 2.935967 | lr 1.8000e-04 | norm: 0.2375 | dt: 14490.28ms | tok/sec: 36182.0 | mem: 108811MB | Time Left: 01 hrs : 03 mins [TIMING] data: 9.2ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10854.1ms | clip: 0.0ms | optim: 23.9ms step 486 | loss: 3.008139 | lr 1.8000e-04 | norm: 0.2111 | dt: 14478.40ms | tok/sec: 36211.7 | mem: 108811MB | Time Left: 01 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3573.2ms | loss: 13.1ms | bwd: 10845.0ms | clip: 0.0ms | optim: 23.9ms step 487 | loss: 2.458391 | lr 1.8000e-04 | norm: 0.2198 | dt: 14486.54ms | tok/sec: 36191.4 | mem: 108811MB | Time Left: 01 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10848.3ms | clip: 0.0ms | optim: 23.9ms step 488 | loss: 2.856071 | lr 1.8000e-04 | norm: 0.2533 | dt: 14493.01ms | tok/sec: 36175.2 | mem: 108811MB | Time Left: 01 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3575.9ms | loss: 13.0ms | bwd: 10857.2ms | clip: 0.0ms | optim: 23.8ms step 489 | loss: 3.180814 | lr 1.8000e-04 | norm: 0.4400 | dt: 14486.57ms | tok/sec: 36191.3 | mem: 108811MB | Time Left: 01 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3583.3ms | loss: 13.0ms | bwd: 10843.3ms | clip: 0.0ms | optim: 24.0ms step 490 | loss: 3.347589 | lr 1.8000e-04 | norm: 0.2199 | dt: 14487.46ms | tok/sec: 36189.1 | mem: 108811MB | Time Left: 01 hrs : 02 mins [TIMING] data: 9.3ms | fwd: 3583.7ms | loss: 13.0ms | bwd: 10843.7ms | clip: 0.0ms | optim: 23.9ms step 491 | loss: 2.629587 | lr 1.8000e-04 | norm: 0.2173 | dt: 14496.26ms | tok/sec: 36167.1 | mem: 108811MB | Time Left: 01 hrs : 02 mins [TIMING] data: 9.1ms | fwd: 3582.1ms | loss: 13.0ms | bwd: 10854.1ms | clip: 0.0ms | optim: 24.0ms step 492 | loss: 3.113065 | lr 1.8000e-04 | norm: 0.2099 | dt: 14479.78ms | tok/sec: 36208.3 | mem: 108811MB | Time Left: 01 hrs : 02 mins [TIMING] data: 9.1ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10840.3ms | clip: 0.0ms | optim: 23.9ms step 493 | loss: 2.972940 | lr 1.8000e-04 | norm: 0.2411 | dt: 14493.43ms | tok/sec: 36174.2 | mem: 108811MB | Time Left: 01 hrs : 02 mins [TIMING] data: 9.1ms | fwd: 3579.0ms | loss: 13.0ms | bwd: 10854.4ms | clip: 0.0ms | optim: 24.0ms step 494 | loss: 3.075511 | lr 1.8000e-04 | norm: 0.2394 | dt: 14484.34ms | tok/sec: 36196.9 | mem: 108811MB | Time Left: 01 hrs : 01 mins [TIMING] data: 9.1ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10847.0ms | clip: 0.0ms | optim: 23.9ms step 495 | loss: 2.783476 | lr 1.8000e-04 | norm: 0.2144 | dt: 14496.15ms | tok/sec: 36167.4 | mem: 108811MB | Time Left: 01 hrs : 01 mins [TIMING] data: 9.0ms | fwd: 3581.2ms | loss: 13.0ms | bwd: 10855.2ms | clip: 0.0ms | optim: 23.9ms step 496 | loss: 2.782462 | lr 1.8000e-04 | norm: 0.2541 | dt: 14485.56ms | tok/sec: 36193.8 | mem: 108811MB | Time Left: 01 hrs : 01 mins [TIMING] data: 9.0ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10847.0ms | clip: 0.0ms | optim: 23.9ms step 497 | loss: 2.974517 | lr 1.8000e-04 | norm: 0.2397 | dt: 14490.51ms | tok/sec: 36181.5 | mem: 108811MB | Time Left: 01 hrs : 01 mins [TIMING] data: 9.1ms | fwd: 3582.0ms | loss: 13.0ms | bwd: 10848.4ms | clip: 0.0ms | optim: 23.9ms step 498 | loss: 3.011478 | lr 1.8000e-04 | norm: 0.2098 | dt: 14502.42ms | tok/sec: 36151.8 | mem: 108811MB | Time Left: 01 hrs : 00 mins [TIMING] data: 9.1ms | fwd: 3583.9ms | loss: 13.1ms | bwd: 10858.5ms | clip: 0.0ms | optim: 23.9ms step 499 | loss: 3.157937 | lr 1.8000e-04 | norm: 0.2670 | dt: 14487.35ms | tok/sec: 36189.4 | mem: 108811MB | Time Left: 01 hrs : 00 mins [TIMING] data: 9.1ms | fwd: 3581.8ms | loss: 13.0ms | bwd: 10845.6ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_500.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_450.ckpt validation loss: 3.1468 step 500 | loss: 2.840570 | lr 1.8000e-04 | norm: 0.2087 | dt: 19004.50ms | tok/sec: 27587.6 | mem: 108811MB | Time Left: 01 hrs : 19 mins [TIMING] data: 9.2ms | fwd: 3562.4ms | loss: 13.1ms | bwd: 10792.7ms | clip: 0.0ms | optim: 24.0ms step 501 | loss: 3.172596 | lr 1.8000e-04 | norm: 0.2384 | dt: 14422.51ms | tok/sec: 36352.1 | mem: 108811MB | Time Left: 00 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3573.4ms | loss: 13.1ms | bwd: 10788.9ms | clip: 0.0ms | optim: 24.0ms step 502 | loss: 3.084194 | lr 1.8000e-04 | norm: 0.2736 | dt: 14452.13ms | tok/sec: 36277.5 | mem: 108811MB | Time Left: 00 hrs : 59 mins [TIMING] data: 9.0ms | fwd: 3571.9ms | loss: 13.0ms | bwd: 10820.2ms | clip: 0.0ms | optim: 24.0ms step 503 | loss: 2.932420 | lr 1.8000e-04 | norm: 0.1937 | dt: 14475.99ms | tok/sec: 36217.8 | mem: 108811MB | Time Left: 00 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.9ms step 504 | loss: 3.173355 | lr 1.8000e-04 | norm: 0.2273 | dt: 14475.23ms | tok/sec: 36219.7 | mem: 108811MB | Time Left: 00 hrs : 59 mins [TIMING] data: 9.0ms | fwd: 3579.6ms | loss: 13.0ms | bwd: 10835.6ms | clip: 0.0ms | optim: 24.1ms step 505 | loss: 2.638260 | lr 1.8000e-04 | norm: 0.1993 | dt: 14468.70ms | tok/sec: 36236.0 | mem: 108811MB | Time Left: 00 hrs : 59 mins [TIMING] data: 9.1ms | fwd: 3571.4ms | loss: 13.0ms | bwd: 10837.3ms | clip: 0.0ms | optim: 24.0ms step 506 | loss: 2.926466 | lr 1.8000e-04 | norm: 0.2236 | dt: 14502.19ms | tok/sec: 36152.3 | mem: 108811MB | Time Left: 00 hrs : 58 mins [TIMING] data: 9.2ms | fwd: 3581.8ms | loss: 13.0ms | bwd: 10860.5ms | clip: 0.0ms | optim: 23.9ms step 507 | loss: 2.987359 | lr 1.8000e-04 | norm: 0.4543 | dt: 14494.00ms | tok/sec: 36172.8 | mem: 108811MB | Time Left: 00 hrs : 58 mins [TIMING] data: 9.9ms | fwd: 3579.2ms | loss: 13.0ms | bwd: 10854.3ms | clip: 0.0ms | optim: 23.9ms step 508 | loss: 3.189649 | lr 1.8000e-04 | norm: 0.3060 | dt: 14496.03ms | tok/sec: 36167.7 | mem: 108811MB | Time Left: 00 hrs : 58 mins [TIMING] data: 9.1ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10856.7ms | clip: 0.0ms | optim: 23.8ms step 509 | loss: 3.188657 | lr 1.8000e-04 | norm: 0.2216 | dt: 14499.55ms | tok/sec: 36158.9 | mem: 108811MB | Time Left: 00 hrs : 58 mins [TIMING] data: 9.2ms | fwd: 3582.4ms | loss: 13.0ms | bwd: 10857.1ms | clip: 0.0ms | optim: 24.0ms step 510 | loss: 2.716764 | lr 1.8000e-04 | norm: 0.3271 | dt: 14504.25ms | tok/sec: 36147.2 | mem: 108811MB | Time Left: 00 hrs : 58 mins [TIMING] data: 9.2ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10861.1ms | clip: 0.0ms | optim: 23.9ms step 511 | loss: 3.133421 | lr 1.8000e-04 | norm: 0.2551 | dt: 14504.63ms | tok/sec: 36146.3 | mem: 108811MB | Time Left: 00 hrs : 57 mins [TIMING] data: 9.0ms | fwd: 3581.7ms | loss: 13.0ms | bwd: 10863.1ms | clip: 0.0ms | optim: 23.9ms step 512 | loss: 2.983437 | lr 1.8000e-04 | norm: 0.2214 | dt: 14512.83ms | tok/sec: 36125.8 | mem: 108811MB | Time Left: 00 hrs : 57 mins [TIMING] data: 9.3ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10873.6ms | clip: 0.0ms | optim: 23.9ms step 513 | loss: 3.076455 | lr 1.8000e-04 | norm: 0.2518 | dt: 14504.37ms | tok/sec: 36146.9 | mem: 108811MB | Time Left: 00 hrs : 57 mins [TIMING] data: 9.1ms | fwd: 3580.0ms | loss: 13.0ms | bwd: 10864.4ms | clip: 0.0ms | optim: 24.0ms step 514 | loss: 3.054566 | lr 1.8000e-04 | norm: 0.2150 | dt: 14513.51ms | tok/sec: 36124.1 | mem: 108811MB | Time Left: 00 hrs : 57 mins [TIMING] data: 9.0ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10870.4ms | clip: 0.0ms | optim: 23.9ms step 515 | loss: 3.280118 | lr 1.8000e-04 | norm: 0.1965 | dt: 14504.92ms | tok/sec: 36145.5 | mem: 108811MB | Time Left: 00 hrs : 56 mins [TIMING] data: 9.3ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10860.4ms | clip: 0.0ms | optim: 24.0ms step 516 | loss: 2.896882 | lr 1.8000e-04 | norm: 0.1781 | dt: 14504.55ms | tok/sec: 36146.4 | mem: 108811MB | Time Left: 00 hrs : 56 mins [TIMING] data: 8.9ms | fwd: 3581.7ms | loss: 13.0ms | bwd: 10863.0ms | clip: 0.0ms | optim: 23.9ms step 517 | loss: 2.881300 | lr 1.8000e-04 | norm: 0.2373 | dt: 14511.06ms | tok/sec: 36130.2 | mem: 108811MB | Time Left: 00 hrs : 56 mins [TIMING] data: 9.2ms | fwd: 3583.5ms | loss: 13.0ms | bwd: 10867.5ms | clip: 0.0ms | optim: 23.8ms step 518 | loss: 2.914470 | lr 1.8000e-04 | norm: 0.2154 | dt: 14519.43ms | tok/sec: 36109.4 | mem: 108811MB | Time Left: 00 hrs : 56 mins [TIMING] data: 9.0ms | fwd: 3584.7ms | loss: 13.1ms | bwd: 10874.8ms | clip: 0.0ms | optim: 23.9ms step 519 | loss: 2.743757 | lr 1.8000e-04 | norm: 0.1876 | dt: 14526.40ms | tok/sec: 36092.1 | mem: 108811MB | Time Left: 00 hrs : 55 mins [TIMING] data: 9.1ms | fwd: 3582.4ms | loss: 13.0ms | bwd: 10884.1ms | clip: 0.0ms | optim: 24.0ms step 520 | loss: 2.632266 | lr 1.8000e-04 | norm: 0.1945 | dt: 14505.76ms | tok/sec: 36143.4 | mem: 108811MB | Time Left: 00 hrs : 55 mins [TIMING] data: 9.0ms | fwd: 3581.9ms | loss: 13.0ms | bwd: 10863.9ms | clip: 0.0ms | optim: 24.0ms step 521 | loss: 2.971760 | lr 1.8000e-04 | norm: 0.2206 | dt: 14514.14ms | tok/sec: 36122.6 | mem: 108811MB | Time Left: 00 hrs : 55 mins [TIMING] data: 9.1ms | fwd: 3579.7ms | loss: 13.0ms | bwd: 10874.6ms | clip: 0.0ms | optim: 24.0ms step 522 | loss: 2.651556 | lr 1.8000e-04 | norm: 0.1996 | dt: 14506.14ms | tok/sec: 36142.5 | mem: 108811MB | Time Left: 00 hrs : 55 mins [TIMING] data: 9.1ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10864.8ms | clip: 0.0ms | optim: 23.9ms step 523 | loss: 2.992589 | lr 1.8000e-04 | norm: 0.5744 | dt: 14513.24ms | tok/sec: 36124.8 | mem: 108811MB | Time Left: 00 hrs : 54 mins [TIMING] data: 9.1ms | fwd: 3583.2ms | loss: 13.0ms | bwd: 10870.1ms | clip: 0.0ms | optim: 23.9ms step 524 | loss: 2.932841 | lr 1.8000e-04 | norm: 0.1931 | dt: 14512.32ms | tok/sec: 36127.1 | mem: 108811MB | Time Left: 00 hrs : 54 mins [TIMING] data: 9.1ms | fwd: 3586.6ms | loss: 13.0ms | bwd: 10865.3ms | clip: 0.0ms | optim: 24.0ms step 525 | loss: 2.884559 | lr 1.8000e-04 | norm: 0.1988 | dt: 14511.40ms | tok/sec: 36129.4 | mem: 108811MB | Time Left: 00 hrs : 54 mins [TIMING] data: 9.1ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10867.2ms | clip: 0.0ms | optim: 24.0ms step 526 | loss: 2.979855 | lr 1.8000e-04 | norm: 0.2441 | dt: 14510.08ms | tok/sec: 36132.7 | mem: 108811MB | Time Left: 00 hrs : 54 mins [TIMING] data: 9.1ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10870.6ms | clip: 0.0ms | optim: 24.1ms step 527 | loss: 2.839912 | lr 1.8000e-04 | norm: 0.2098 | dt: 14516.11ms | tok/sec: 36117.7 | mem: 108811MB | Time Left: 00 hrs : 53 mins [TIMING] data: 9.1ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10874.8ms | clip: 0.0ms | optim: 23.9ms step 528 | loss: 2.973166 | lr 1.8000e-04 | norm: 0.5341 | dt: 14509.29ms | tok/sec: 36134.6 | mem: 108811MB | Time Left: 00 hrs : 53 mins [TIMING] data: 9.4ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10865.4ms | clip: 0.0ms | optim: 24.1ms step 529 | loss: 2.897523 | lr 1.8000e-04 | norm: 0.1842 | dt: 14504.04ms | tok/sec: 36147.7 | mem: 108811MB | Time Left: 00 hrs : 53 mins [TIMING] data: 9.3ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10862.1ms | clip: 0.0ms | optim: 23.9ms step 530 | loss: 3.314602 | lr 1.8000e-04 | norm: 0.3063 | dt: 14515.22ms | tok/sec: 36119.9 | mem: 108811MB | Time Left: 00 hrs : 53 mins [TIMING] data: 9.1ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10872.0ms | clip: 0.0ms | optim: 24.1ms step 531 | loss: 2.738464 | lr 1.8000e-04 | norm: 0.3517 | dt: 14514.03ms | tok/sec: 36122.8 | mem: 108811MB | Time Left: 00 hrs : 52 mins [TIMING] data: 9.1ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10871.7ms | clip: 0.0ms | optim: 24.0ms step 532 | loss: 2.875875 | lr 1.8000e-04 | norm: 0.2454 | dt: 14512.33ms | tok/sec: 36127.1 | mem: 108811MB | Time Left: 00 hrs : 52 mins [TIMING] data: 9.1ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10874.0ms | clip: 0.0ms | optim: 23.9ms step 533 | loss: 2.861120 | lr 1.8000e-04 | norm: 0.1703 | dt: 14503.66ms | tok/sec: 36148.7 | mem: 108811MB | Time Left: 00 hrs : 52 mins [TIMING] data: 9.0ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10864.9ms | clip: 0.0ms | optim: 24.0ms step 534 | loss: 3.107659 | lr 1.8000e-04 | norm: 0.2393 | dt: 14508.65ms | tok/sec: 36136.2 | mem: 108811MB | Time Left: 00 hrs : 52 mins [TIMING] data: 9.0ms | fwd: 3583.1ms | loss: 13.0ms | bwd: 10865.7ms | clip: 0.0ms | optim: 24.0ms step 535 | loss: 2.784458 | lr 1.8000e-04 | norm: 0.1935 | dt: 14507.21ms | tok/sec: 36139.8 | mem: 108811MB | Time Left: 00 hrs : 51 mins [TIMING] data: 9.4ms | fwd: 3579.1ms | loss: 13.0ms | bwd: 10867.6ms | clip: 0.0ms | optim: 23.9ms step 536 | loss: 2.921372 | lr 1.8000e-04 | norm: 0.2052 | dt: 14514.40ms | tok/sec: 36121.9 | mem: 108811MB | Time Left: 00 hrs : 51 mins [TIMING] data: 9.2ms | fwd: 3585.6ms | loss: 13.0ms | bwd: 10868.8ms | clip: 0.0ms | optim: 23.9ms step 537 | loss: 2.850994 | lr 1.8000e-04 | norm: 0.2559 | dt: 14520.25ms | tok/sec: 36107.4 | mem: 108811MB | Time Left: 00 hrs : 51 mins [TIMING] data: 9.1ms | fwd: 3583.7ms | loss: 13.0ms | bwd: 10876.6ms | clip: 0.0ms | optim: 24.0ms step 538 | loss: 2.820993 | lr 1.8000e-04 | norm: 0.2226 | dt: 14507.05ms | tok/sec: 36140.2 | mem: 108811MB | Time Left: 00 hrs : 51 mins [TIMING] data: 9.2ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10868.8ms | clip: 0.0ms | optim: 24.1ms step 539 | loss: 3.141716 | lr 1.8000e-04 | norm: 0.2170 | dt: 14516.24ms | tok/sec: 36117.3 | mem: 108811MB | Time Left: 00 hrs : 51 mins [TIMING] data: 9.1ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10872.8ms | clip: 0.0ms | optim: 24.0ms step 540 | loss: 3.170415 | lr 1.8000e-04 | norm: 0.2854 | dt: 14503.11ms | tok/sec: 36150.0 | mem: 108811MB | Time Left: 00 hrs : 50 mins [TIMING] data: 9.1ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10860.9ms | clip: 0.0ms | optim: 24.0ms step 541 | loss: 3.167759 | lr 1.8000e-04 | norm: 0.1950 | dt: 14510.26ms | tok/sec: 36132.2 | mem: 108811MB | Time Left: 00 hrs : 50 mins [TIMING] data: 9.1ms | fwd: 3579.7ms | loss: 13.0ms | bwd: 10870.6ms | clip: 0.0ms | optim: 24.0ms step 542 | loss: 2.977990 | lr 1.8000e-04 | norm: 0.2860 | dt: 14513.31ms | tok/sec: 36124.6 | mem: 108811MB | Time Left: 00 hrs : 50 mins [TIMING] data: 9.1ms | fwd: 3584.8ms | loss: 13.0ms | bwd: 10868.7ms | clip: 0.0ms | optim: 24.0ms step 543 | loss: 2.797268 | lr 1.8000e-04 | norm: 0.1892 | dt: 14518.58ms | tok/sec: 36111.5 | mem: 108811MB | Time Left: 00 hrs : 50 mins [TIMING] data: 9.1ms | fwd: 3583.2ms | loss: 13.0ms | bwd: 10875.4ms | clip: 0.0ms | optim: 24.0ms step 544 | loss: 2.727496 | lr 1.8000e-04 | norm: 0.7966 | dt: 14521.04ms | tok/sec: 36105.4 | mem: 108811MB | Time Left: 00 hrs : 49 mins [TIMING] data: 9.1ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10880.2ms | clip: 0.0ms | optim: 24.0ms step 545 | loss: 2.722150 | lr 1.8000e-04 | norm: 0.1920 | dt: 14521.77ms | tok/sec: 36103.6 | mem: 108811MB | Time Left: 00 hrs : 49 mins [TIMING] data: 9.2ms | fwd: 3583.8ms | loss: 13.1ms | bwd: 10877.8ms | clip: 0.0ms | optim: 24.0ms step 546 | loss: 2.997433 | lr 1.8000e-04 | norm: 0.1957 | dt: 14512.93ms | tok/sec: 36125.6 | mem: 108811MB | Time Left: 00 hrs : 49 mins [TIMING] data: 9.1ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10870.9ms | clip: 0.0ms | optim: 24.0ms step 547 | loss: 2.823830 | lr 1.8000e-04 | norm: 0.1720 | dt: 14522.07ms | tok/sec: 36102.8 | mem: 108811MB | Time Left: 00 hrs : 49 mins [TIMING] data: 9.1ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10879.2ms | clip: 0.0ms | optim: 23.9ms step 548 | loss: 3.253661 | lr 1.8000e-04 | norm: 0.2075 | dt: 14501.87ms | tok/sec: 36153.1 | mem: 108811MB | Time Left: 00 hrs : 48 mins [TIMING] data: 9.4ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10858.4ms | clip: 0.0ms | optim: 24.0ms step 549 | loss: 3.028228 | lr 1.8000e-04 | norm: 0.1641 | dt: 14510.76ms | tok/sec: 36131.0 | mem: 108811MB | Time Left: 00 hrs : 48 mins [TIMING] data: 9.0ms | fwd: 3581.0ms | loss: 13.0ms | bwd: 10869.9ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_550.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_500.ckpt step 550 | loss: 2.872953 | lr 1.8000e-04 | norm: 0.1871 | dt: 14379.94ms | tok/sec: 36459.7 | mem: 108811MB | Time Left: 00 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3570.2ms | loss: 13.1ms | bwd: 10749.5ms | clip: 0.0ms | optim: 23.9ms step 551 | loss: 2.461514 | lr 1.8000e-04 | norm: 0.1833 | dt: 14424.18ms | tok/sec: 36347.8 | mem: 108811MB | Time Left: 00 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3563.8ms | loss: 13.0ms | bwd: 10800.4ms | clip: 0.0ms | optim: 24.0ms step 552 | loss: 3.023923 | lr 1.8000e-04 | norm: 0.2349 | dt: 14463.24ms | tok/sec: 36249.7 | mem: 108811MB | Time Left: 00 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10827.0ms | clip: 0.0ms | optim: 23.9ms step 553 | loss: 3.341274 | lr 1.8000e-04 | norm: 0.2859 | dt: 14455.21ms | tok/sec: 36269.8 | mem: 108811MB | Time Left: 00 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3571.7ms | loss: 13.0ms | bwd: 10823.4ms | clip: 0.0ms | optim: 24.0ms step 554 | loss: 2.948566 | lr 1.8000e-04 | norm: 0.2474 | dt: 14484.40ms | tok/sec: 36196.7 | mem: 108811MB | Time Left: 00 hrs : 47 mins [TIMING] data: 9.1ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10844.2ms | clip: 0.0ms | optim: 24.0ms step 555 | loss: 2.703537 | lr 1.8000e-04 | norm: 0.6018 | dt: 14475.51ms | tok/sec: 36219.0 | mem: 108811MB | Time Left: 00 hrs : 47 mins [TIMING] data: 9.0ms | fwd: 3571.0ms | loss: 13.1ms | bwd: 10844.5ms | clip: 0.0ms | optim: 24.0ms step 556 | loss: 3.073534 | lr 1.8000e-04 | norm: 0.2525 | dt: 14481.56ms | tok/sec: 36203.8 | mem: 108811MB | Time Left: 00 hrs : 46 mins [TIMING] data: 9.1ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10841.7ms | clip: 0.0ms | optim: 23.8ms step 557 | loss: 3.057580 | lr 1.8000e-04 | norm: 0.2348 | dt: 14498.96ms | tok/sec: 36160.4 | mem: 108811MB | Time Left: 00 hrs : 46 mins [TIMING] data: 9.1ms | fwd: 3579.1ms | loss: 13.0ms | bwd: 10859.9ms | clip: 0.0ms | optim: 23.9ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00003.bin 100000000 step 558 | loss: 2.832289 | lr 1.8000e-04 | norm: 0.1965 | dt: 14490.95ms | tok/sec: 36180.4 | mem: 108811MB | Time Left: 00 hrs : 46 mins [TIMING] data: 14.0ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10848.8ms | clip: 0.0ms | optim: 24.0ms step 559 | loss: 2.894110 | lr 1.8000e-04 | norm: 0.2333 | dt: 14493.38ms | tok/sec: 36174.3 | mem: 108811MB | Time Left: 00 hrs : 46 mins [TIMING] data: 9.2ms | fwd: 3582.4ms | loss: 13.0ms | bwd: 10851.0ms | clip: 0.0ms | optim: 23.9ms step 560 | loss: 2.950360 | lr 1.8000e-04 | norm: 0.2116 | dt: 14492.35ms | tok/sec: 36176.9 | mem: 108811MB | Time Left: 00 hrs : 45 mins [TIMING] data: 9.1ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10855.5ms | clip: 0.0ms | optim: 24.0ms step 561 | loss: 3.064004 | lr 1.8000e-04 | norm: 0.2247 | dt: 14500.46ms | tok/sec: 36156.6 | mem: 108811MB | Time Left: 00 hrs : 45 mins [TIMING] data: 9.5ms | fwd: 3586.4ms | loss: 13.0ms | bwd: 10853.8ms | clip: 0.0ms | optim: 24.0ms step 562 | loss: 2.871527 | lr 1.8000e-04 | norm: 0.2058 | dt: 14484.23ms | tok/sec: 36197.2 | mem: 108811MB | Time Left: 00 hrs : 45 mins [TIMING] data: 9.2ms | fwd: 3580.0ms | loss: 13.0ms | bwd: 10844.1ms | clip: 0.0ms | optim: 24.0ms step 563 | loss: 2.860193 | lr 1.8000e-04 | norm: 0.2559 | dt: 14489.52ms | tok/sec: 36183.9 | mem: 108811MB | Time Left: 00 hrs : 45 mins [TIMING] data: 9.1ms | fwd: 3574.6ms | loss: 13.0ms | bwd: 10854.9ms | clip: 0.0ms | optim: 24.0ms step 564 | loss: 3.074220 | lr 1.8000e-04 | norm: 0.2614 | dt: 14496.66ms | tok/sec: 36166.1 | mem: 108811MB | Time Left: 00 hrs : 44 mins [TIMING] data: 9.1ms | fwd: 3574.5ms | loss: 13.0ms | bwd: 10862.2ms | clip: 0.0ms | optim: 24.0ms step 565 | loss: 2.652312 | lr 1.8000e-04 | norm: 0.2098 | dt: 14498.60ms | tok/sec: 36161.3 | mem: 108811MB | Time Left: 00 hrs : 44 mins [TIMING] data: 9.0ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10860.0ms | clip: 0.0ms | optim: 24.0ms step 566 | loss: 2.835320 | lr 1.8000e-04 | norm: 0.1878 | dt: 14487.14ms | tok/sec: 36189.9 | mem: 108811MB | Time Left: 00 hrs : 44 mins [TIMING] data: 9.1ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10847.0ms | clip: 0.0ms | optim: 23.9ms step 567 | loss: 2.887269 | lr 1.8000e-04 | norm: 0.2339 | dt: 14488.20ms | tok/sec: 36187.2 | mem: 108811MB | Time Left: 00 hrs : 44 mins [TIMING] data: 9.2ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10846.4ms | clip: 0.0ms | optim: 23.9ms step 568 | loss: 2.907742 | lr 1.8000e-04 | norm: 0.2114 | dt: 14489.50ms | tok/sec: 36184.0 | mem: 108811MB | Time Left: 00 hrs : 43 mins [TIMING] data: 9.1ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10852.8ms | clip: 0.0ms | optim: 23.9ms step 569 | loss: 3.148080 | lr 1.8000e-04 | norm: 0.2275 | dt: 14499.18ms | tok/sec: 36159.8 | mem: 108811MB | Time Left: 00 hrs : 43 mins [TIMING] data: 9.1ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10855.1ms | clip: 0.0ms | optim: 23.9ms step 570 | loss: 2.768334 | lr 1.8000e-04 | norm: 0.2341 | dt: 14497.96ms | tok/sec: 36162.9 | mem: 108811MB | Time Left: 00 hrs : 43 mins [TIMING] data: 9.1ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10855.1ms | clip: 0.0ms | optim: 24.0ms step 571 | loss: 2.744710 | lr 1.8000e-04 | norm: 0.2206 | dt: 14500.77ms | tok/sec: 36155.9 | mem: 108811MB | Time Left: 00 hrs : 43 mins [TIMING] data: 9.2ms | fwd: 3584.0ms | loss: 13.0ms | bwd: 10856.8ms | clip: 0.0ms | optim: 23.9ms step 572 | loss: 2.896245 | lr 1.8000e-04 | norm: 0.1984 | dt: 14497.58ms | tok/sec: 36163.8 | mem: 108811MB | Time Left: 00 hrs : 43 mins [TIMING] data: 9.1ms | fwd: 3583.9ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.9ms step 573 | loss: 3.117884 | lr 1.8000e-04 | norm: 0.2830 | dt: 14493.63ms | tok/sec: 36173.7 | mem: 108811MB | Time Left: 00 hrs : 42 mins [TIMING] data: 9.0ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10852.5ms | clip: 0.0ms | optim: 24.0ms step 574 | loss: 2.924914 | lr 1.8000e-04 | norm: 0.1763 | dt: 14501.39ms | tok/sec: 36154.3 | mem: 108811MB | Time Left: 00 hrs : 42 mins [TIMING] data: 9.2ms | fwd: 3586.9ms | loss: 13.1ms | bwd: 10854.2ms | clip: 0.0ms | optim: 24.0ms step 575 | loss: 2.782796 | lr 1.8000e-04 | norm: 0.2017 | dt: 14489.41ms | tok/sec: 36184.2 | mem: 108811MB | Time Left: 00 hrs : 42 mins [TIMING] data: 9.0ms | fwd: 3577.4ms | loss: 13.0ms | bwd: 10852.0ms | clip: 0.0ms | optim: 24.0ms step 576 | loss: 2.884538 | lr 1.8000e-04 | norm: 0.2106 | dt: 14494.27ms | tok/sec: 36172.1 | mem: 108811MB | Time Left: 00 hrs : 42 mins [TIMING] data: 9.1ms | fwd: 3581.5ms | loss: 13.0ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.9ms step 577 | loss: 3.032324 | lr 1.8000e-04 | norm: 0.2343 | dt: 14487.70ms | tok/sec: 36188.5 | mem: 108811MB | Time Left: 00 hrs : 41 mins [TIMING] data: 9.0ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10848.3ms | clip: 0.0ms | optim: 24.1ms step 578 | loss: 2.628248 | lr 1.8000e-04 | norm: 0.1897 | dt: 14499.09ms | tok/sec: 36160.1 | mem: 108811MB | Time Left: 00 hrs : 41 mins [TIMING] data: 9.1ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10856.9ms | clip: 0.0ms | optim: 23.9ms step 579 | loss: 3.223622 | lr 1.8000e-04 | norm: 0.2389 | dt: 14500.61ms | tok/sec: 36156.3 | mem: 108811MB | Time Left: 00 hrs : 41 mins [TIMING] data: 9.1ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10861.4ms | clip: 0.0ms | optim: 23.9ms step 580 | loss: 2.916237 | lr 1.8000e-04 | norm: 0.2041 | dt: 14495.47ms | tok/sec: 36169.1 | mem: 108811MB | Time Left: 00 hrs : 41 mins [TIMING] data: 9.3ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10855.6ms | clip: 0.0ms | optim: 23.9ms step 581 | loss: 3.016014 | lr 1.8000e-04 | norm: 0.2293 | dt: 14489.76ms | tok/sec: 36183.3 | mem: 108811MB | Time Left: 00 hrs : 40 mins [TIMING] data: 9.0ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 24.0ms step 582 | loss: 3.063994 | lr 1.8000e-04 | norm: 0.1910 | dt: 14497.05ms | tok/sec: 36165.2 | mem: 108811MB | Time Left: 00 hrs : 40 mins [TIMING] data: 9.1ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10852.3ms | clip: 0.0ms | optim: 23.9ms step 583 | loss: 2.968232 | lr 1.8000e-04 | norm: 0.1854 | dt: 14485.97ms | tok/sec: 36192.8 | mem: 108811MB | Time Left: 00 hrs : 40 mins [TIMING] data: 9.1ms | fwd: 3575.1ms | loss: 13.0ms | bwd: 10850.8ms | clip: 0.0ms | optim: 24.1ms step 584 | loss: 2.868133 | lr 1.8000e-04 | norm: 0.1758 | dt: 14479.53ms | tok/sec: 36208.9 | mem: 108811MB | Time Left: 00 hrs : 40 mins [TIMING] data: 9.1ms | fwd: 3578.2ms | loss: 13.0ms | bwd: 10841.4ms | clip: 0.0ms | optim: 23.9ms step 585 | loss: 2.870216 | lr 1.8000e-04 | norm: 0.2028 | dt: 14491.03ms | tok/sec: 36180.2 | mem: 108811MB | Time Left: 00 hrs : 39 mins [TIMING] data: 9.2ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10852.2ms | clip: 0.0ms | optim: 23.9ms step 586 | loss: 2.978692 | lr 1.8000e-04 | norm: 0.2168 | dt: 14492.06ms | tok/sec: 36177.6 | mem: 108811MB | Time Left: 00 hrs : 39 mins [TIMING] data: 9.1ms | fwd: 3585.7ms | loss: 13.0ms | bwd: 10846.4ms | clip: 0.0ms | optim: 24.0ms step 587 | loss: 2.964594 | lr 1.8000e-04 | norm: 0.3131 | dt: 14489.24ms | tok/sec: 36184.6 | mem: 108811MB | Time Left: 00 hrs : 39 mins [TIMING] data: 9.1ms | fwd: 3580.0ms | loss: 13.0ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.9ms step 588 | loss: 2.907275 | lr 1.8000e-04 | norm: 0.2672 | dt: 14488.42ms | tok/sec: 36186.7 | mem: 108811MB | Time Left: 00 hrs : 39 mins [TIMING] data: 9.1ms | fwd: 3583.5ms | loss: 13.0ms | bwd: 10845.1ms | clip: 0.0ms | optim: 23.9ms step 589 | loss: 2.952869 | lr 1.8000e-04 | norm: 0.1928 | dt: 14484.67ms | tok/sec: 36196.1 | mem: 108811MB | Time Left: 00 hrs : 38 mins [TIMING] data: 9.1ms | fwd: 3584.3ms | loss: 13.0ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.9ms step 590 | loss: 2.557090 | lr 1.8000e-04 | norm: 0.2017 | dt: 14496.34ms | tok/sec: 36166.9 | mem: 108811MB | Time Left: 00 hrs : 38 mins [TIMING] data: 9.1ms | fwd: 3583.4ms | loss: 13.0ms | bwd: 10853.0ms | clip: 0.0ms | optim: 23.9ms step 591 | loss: 2.710946 | lr 1.8000e-04 | norm: 0.1862 | dt: 14488.85ms | tok/sec: 36185.6 | mem: 108811MB | Time Left: 00 hrs : 38 mins [TIMING] data: 9.0ms | fwd: 3577.7ms | loss: 13.0ms | bwd: 10851.3ms | clip: 0.0ms | optim: 23.9ms step 592 | loss: 2.833590 | lr 1.8000e-04 | norm: 0.2101 | dt: 14495.84ms | tok/sec: 36168.2 | mem: 108811MB | Time Left: 00 hrs : 38 mins [TIMING] data: 9.1ms | fwd: 3581.1ms | loss: 13.0ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.9ms step 593 | loss: 2.904993 | lr 1.8000e-04 | norm: 0.2769 | dt: 14485.33ms | tok/sec: 36194.4 | mem: 108811MB | Time Left: 00 hrs : 37 mins [TIMING] data: 9.1ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10847.2ms | clip: 0.0ms | optim: 23.9ms step 594 | loss: 3.235111 | lr 1.8000e-04 | norm: 0.2294 | dt: 14493.95ms | tok/sec: 36172.9 | mem: 108811MB | Time Left: 00 hrs : 37 mins [TIMING] data: 9.0ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10854.0ms | clip: 0.0ms | optim: 23.9ms step 595 | loss: 2.973606 | lr 1.8000e-04 | norm: 0.2724 | dt: 14497.18ms | tok/sec: 36164.8 | mem: 108811MB | Time Left: 00 hrs : 37 mins [TIMING] data: 9.1ms | fwd: 3580.6ms | loss: 13.0ms | bwd: 10856.5ms | clip: 0.0ms | optim: 24.0ms step 596 | loss: 3.091542 | lr 1.8000e-04 | norm: 0.2419 | dt: 14489.90ms | tok/sec: 36183.0 | mem: 108811MB | Time Left: 00 hrs : 37 mins [TIMING] data: 9.2ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10851.4ms | clip: 0.0ms | optim: 24.0ms step 597 | loss: 2.787784 | lr 1.8000e-04 | norm: 0.2553 | dt: 14490.93ms | tok/sec: 36180.4 | mem: 108811MB | Time Left: 00 hrs : 36 mins [TIMING] data: 9.1ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10847.7ms | clip: 0.0ms | optim: 24.0ms step 598 | loss: 3.105371 | lr 1.8000e-04 | norm: 0.2243 | dt: 14502.57ms | tok/sec: 36151.4 | mem: 108811MB | Time Left: 00 hrs : 36 mins [TIMING] data: 9.1ms | fwd: 3583.0ms | loss: 13.0ms | bwd: 10859.8ms | clip: 0.0ms | optim: 23.9ms step 599 | loss: 2.788259 | lr 1.8000e-04 | norm: 0.2647 | dt: 14499.68ms | tok/sec: 36158.6 | mem: 108811MB | Time Left: 00 hrs : 36 mins [TIMING] data: 9.1ms | fwd: 3583.2ms | loss: 13.0ms | bwd: 10856.6ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_600.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_550.ckpt step 600 | loss: 2.847475 | lr 1.8000e-04 | norm: 0.2103 | dt: 14361.50ms | tok/sec: 36506.5 | mem: 108811MB | Time Left: 00 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3569.1ms | loss: 13.1ms | bwd: 10732.1ms | clip: 0.0ms | optim: 24.0ms step 601 | loss: 2.873457 | lr 1.8000e-04 | norm: 0.2287 | dt: 14409.69ms | tok/sec: 36384.4 | mem: 108811MB | Time Left: 00 hrs : 35 mins [TIMING] data: 9.2ms | fwd: 3568.5ms | loss: 13.1ms | bwd: 10781.0ms | clip: 0.0ms | optim: 23.8ms step 602 | loss: 2.802947 | lr 1.8000e-04 | norm: 0.2193 | dt: 14415.46ms | tok/sec: 36369.8 | mem: 108811MB | Time Left: 00 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3565.7ms | loss: 13.1ms | bwd: 10789.5ms | clip: 0.0ms | optim: 23.9ms step 603 | loss: 2.843135 | lr 1.8000e-04 | norm: 0.2007 | dt: 14473.70ms | tok/sec: 36223.5 | mem: 108811MB | Time Left: 00 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10833.8ms | clip: 0.0ms | optim: 23.9ms step 604 | loss: 2.679140 | lr 1.8000e-04 | norm: 0.2336 | dt: 14495.00ms | tok/sec: 36170.3 | mem: 108811MB | Time Left: 00 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3579.6ms | loss: 13.0ms | bwd: 10855.3ms | clip: 0.0ms | optim: 24.0ms step 605 | loss: 2.864019 | lr 1.8000e-04 | norm: 0.2516 | dt: 14511.21ms | tok/sec: 36129.9 | mem: 108811MB | Time Left: 00 hrs : 35 mins [TIMING] data: 9.1ms | fwd: 3583.5ms | loss: 13.0ms | bwd: 10867.9ms | clip: 0.0ms | optim: 23.9ms step 606 | loss: 2.713534 | lr 1.8000e-04 | norm: 0.2087 | dt: 14523.57ms | tok/sec: 36099.1 | mem: 108811MB | Time Left: 00 hrs : 34 mins [TIMING] data: 9.1ms | fwd: 3584.4ms | loss: 13.0ms | bwd: 10879.1ms | clip: 0.0ms | optim: 24.0ms step 607 | loss: 2.696956 | lr 1.8000e-04 | norm: 0.2134 | dt: 14532.66ms | tok/sec: 36076.5 | mem: 108811MB | Time Left: 00 hrs : 34 mins [TIMING] data: 9.4ms | fwd: 3588.5ms | loss: 13.1ms | bwd: 10883.6ms | clip: 0.0ms | optim: 23.9ms step 608 | loss: 3.110168 | lr 1.8000e-04 | norm: 0.2021 | dt: 14523.69ms | tok/sec: 36098.8 | mem: 108811MB | Time Left: 00 hrs : 34 mins [TIMING] data: 9.1ms | fwd: 3586.2ms | loss: 13.0ms | bwd: 10877.2ms | clip: 0.0ms | optim: 24.1ms step 609 | loss: 2.914902 | lr 1.8000e-04 | norm: 0.2329 | dt: 14515.98ms | tok/sec: 36118.0 | mem: 108811MB | Time Left: 00 hrs : 34 mins [TIMING] data: 9.0ms | fwd: 3583.9ms | loss: 13.0ms | bwd: 10872.1ms | clip: 0.0ms | optim: 24.0ms step 610 | loss: 2.985199 | lr 1.8000e-04 | norm: 0.4635 | dt: 14523.38ms | tok/sec: 36099.6 | mem: 108811MB | Time Left: 00 hrs : 33 mins [TIMING] data: 9.1ms | fwd: 3584.3ms | loss: 13.0ms | bwd: 10878.9ms | clip: 0.0ms | optim: 24.0ms step 611 | loss: 3.134099 | lr 1.8000e-04 | norm: 0.4043 | dt: 14520.26ms | tok/sec: 36107.3 | mem: 108811MB | Time Left: 00 hrs : 33 mins [TIMING] data: 9.0ms | fwd: 3581.5ms | loss: 13.0ms | bwd: 10878.8ms | clip: 0.0ms | optim: 24.0ms step 612 | loss: 2.860500 | lr 1.8000e-04 | norm: 0.1722 | dt: 14524.36ms | tok/sec: 36097.2 | mem: 108811MB | Time Left: 00 hrs : 33 mins [TIMING] data: 9.0ms | fwd: 3589.4ms | loss: 13.0ms | bwd: 10875.1ms | clip: 0.0ms | optim: 23.9ms step 613 | loss: 3.158753 | lr 1.8000e-04 | norm: 0.1768 | dt: 14532.07ms | tok/sec: 36078.0 | mem: 108811MB | Time Left: 00 hrs : 33 mins [TIMING] data: 9.0ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10889.9ms | clip: 0.0ms | optim: 25.0ms step 614 | loss: 3.091527 | lr 1.8000e-04 | norm: 0.1816 | dt: 14523.51ms | tok/sec: 36099.3 | mem: 108811MB | Time Left: 00 hrs : 32 mins [TIMING] data: 9.0ms | fwd: 3586.4ms | loss: 13.1ms | bwd: 10877.1ms | clip: 0.0ms | optim: 24.0ms step 615 | loss: 2.905709 | lr 1.8000e-04 | norm: 0.1852 | dt: 14519.40ms | tok/sec: 36109.5 | mem: 108811MB | Time Left: 00 hrs : 32 mins [TIMING] data: 9.1ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10878.6ms | clip: 0.0ms | optim: 23.8ms step 616 | loss: 2.845296 | lr 1.8000e-04 | norm: 0.1809 | dt: 14526.89ms | tok/sec: 36090.9 | mem: 108811MB | Time Left: 00 hrs : 32 mins [TIMING] data: 9.0ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10886.9ms | clip: 0.0ms | optim: 24.0ms step 617 | loss: 2.896917 | lr 1.8000e-04 | norm: 0.2003 | dt: 14528.53ms | tok/sec: 36086.8 | mem: 108811MB | Time Left: 00 hrs : 32 mins [TIMING] data: 9.0ms | fwd: 3586.3ms | loss: 13.0ms | bwd: 10882.3ms | clip: 0.0ms | optim: 24.0ms step 618 | loss: 2.502716 | lr 1.8000e-04 | norm: 0.1837 | dt: 14531.78ms | tok/sec: 36078.7 | mem: 108811MB | Time Left: 00 hrs : 31 mins [TIMING] data: 9.2ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10887.5ms | clip: 0.0ms | optim: 24.0ms step 619 | loss: 2.916397 | lr 1.8000e-04 | norm: 0.1785 | dt: 14525.28ms | tok/sec: 36094.9 | mem: 108811MB | Time Left: 00 hrs : 31 mins [TIMING] data: 9.1ms | fwd: 3589.3ms | loss: 13.0ms | bwd: 10876.0ms | clip: 0.0ms | optim: 23.9ms step 620 | loss: 2.738529 | lr 1.8000e-04 | norm: 0.1661 | dt: 14522.52ms | tok/sec: 36101.7 | mem: 108811MB | Time Left: 00 hrs : 31 mins [TIMING] data: 9.7ms | fwd: 3588.3ms | loss: 13.0ms | bwd: 10873.8ms | clip: 0.0ms | optim: 23.9ms step 621 | loss: 2.775776 | lr 1.8000e-04 | norm: 0.2211 | dt: 14522.62ms | tok/sec: 36101.5 | mem: 108811MB | Time Left: 00 hrs : 31 mins [TIMING] data: 9.1ms | fwd: 3587.0ms | loss: 13.1ms | bwd: 10875.5ms | clip: 0.0ms | optim: 24.0ms step 622 | loss: 2.354427 | lr 1.8000e-04 | norm: 0.2163 | dt: 14523.49ms | tok/sec: 36099.3 | mem: 108811MB | Time Left: 00 hrs : 30 mins [TIMING] data: 9.2ms | fwd: 3584.3ms | loss: 13.0ms | bwd: 10879.0ms | clip: 0.0ms | optim: 24.0ms step 623 | loss: 2.603827 | lr 1.8000e-04 | norm: 0.1908 | dt: 14527.43ms | tok/sec: 36089.5 | mem: 108811MB | Time Left: 00 hrs : 30 mins [TIMING] data: 9.1ms | fwd: 3587.6ms | loss: 13.0ms | bwd: 10879.7ms | clip: 0.0ms | optim: 24.0ms step 624 | loss: 2.951881 | lr 1.8000e-04 | norm: 0.2658 | dt: 14522.97ms | tok/sec: 36100.6 | mem: 108811MB | Time Left: 00 hrs : 30 mins [TIMING] data: 9.1ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10878.9ms | clip: 0.0ms | optim: 24.0ms step 625 | loss: 2.853805 | lr 1.8000e-04 | norm: 0.1907 | dt: 14524.14ms | tok/sec: 36097.7 | mem: 108811MB | Time Left: 00 hrs : 30 mins [TIMING] data: 9.2ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10881.2ms | clip: 0.0ms | optim: 24.0ms step 626 | loss: 2.694692 | lr 1.8000e-04 | norm: 0.4340 | dt: 14519.36ms | tok/sec: 36109.6 | mem: 108811MB | Time Left: 00 hrs : 30 mins [TIMING] data: 9.1ms | fwd: 3585.8ms | loss: 13.1ms | bwd: 10873.6ms | clip: 0.0ms | optim: 23.9ms step 627 | loss: 3.115720 | lr 1.8000e-04 | norm: 0.2407 | dt: 14517.20ms | tok/sec: 36115.0 | mem: 108811MB | Time Left: 00 hrs : 29 mins [TIMING] data: 9.1ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10877.2ms | clip: 0.0ms | optim: 24.0ms step 628 | loss: 2.942566 | lr 1.8000e-04 | norm: 0.1867 | dt: 14507.04ms | tok/sec: 36140.2 | mem: 108811MB | Time Left: 00 hrs : 29 mins [TIMING] data: 9.0ms | fwd: 3581.1ms | loss: 13.0ms | bwd: 10866.1ms | clip: 0.0ms | optim: 23.9ms step 629 | loss: 2.977096 | lr 1.8000e-04 | norm: 0.2337 | dt: 14493.79ms | tok/sec: 36173.3 | mem: 108811MB | Time Left: 00 hrs : 29 mins [TIMING] data: 9.0ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10854.5ms | clip: 0.0ms | optim: 23.9ms step 630 | loss: 2.742052 | lr 1.8000e-04 | norm: 0.1942 | dt: 14512.71ms | tok/sec: 36126.1 | mem: 108811MB | Time Left: 00 hrs : 29 mins [TIMING] data: 9.0ms | fwd: 3583.8ms | loss: 13.0ms | bwd: 10869.0ms | clip: 0.0ms | optim: 24.1ms step 631 | loss: 3.083669 | lr 1.8000e-04 | norm: 0.2014 | dt: 14513.84ms | tok/sec: 36123.3 | mem: 108811MB | Time Left: 00 hrs : 28 mins [TIMING] data: 9.0ms | fwd: 3584.4ms | loss: 13.0ms | bwd: 10869.6ms | clip: 0.0ms | optim: 23.9ms step 632 | loss: 2.783914 | lr 1.8000e-04 | norm: 0.2190 | dt: 14484.91ms | tok/sec: 36195.5 | mem: 108811MB | Time Left: 00 hrs : 28 mins [TIMING] data: 9.0ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10846.7ms | clip: 0.0ms | optim: 23.9ms step 633 | loss: 2.878674 | lr 1.8000e-04 | norm: 0.1912 | dt: 14498.16ms | tok/sec: 36162.4 | mem: 108811MB | Time Left: 00 hrs : 28 mins [TIMING] data: 9.1ms | fwd: 3580.2ms | loss: 13.0ms | bwd: 10857.8ms | clip: 0.0ms | optim: 24.1ms step 634 | loss: 2.870126 | lr 1.8000e-04 | norm: 0.1845 | dt: 14493.82ms | tok/sec: 36173.2 | mem: 108811MB | Time Left: 00 hrs : 28 mins [TIMING] data: 9.0ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10854.6ms | clip: 0.0ms | optim: 23.9ms step 635 | loss: 2.952508 | lr 1.8000e-04 | norm: 0.2217 | dt: 14500.85ms | tok/sec: 36155.7 | mem: 108811MB | Time Left: 00 hrs : 27 mins [TIMING] data: 9.2ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10862.8ms | clip: 0.0ms | optim: 23.9ms step 636 | loss: 2.915274 | lr 1.8000e-04 | norm: 0.2736 | dt: 14485.56ms | tok/sec: 36193.8 | mem: 108811MB | Time Left: 00 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10850.5ms | clip: 0.0ms | optim: 24.0ms step 637 | loss: 3.081783 | lr 1.8000e-04 | norm: 0.2311 | dt: 14499.67ms | tok/sec: 36158.6 | mem: 108811MB | Time Left: 00 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3578.0ms | loss: 13.0ms | bwd: 10861.8ms | clip: 0.0ms | optim: 23.9ms step 638 | loss: 2.772395 | lr 1.8000e-04 | norm: 0.2052 | dt: 14504.20ms | tok/sec: 36147.3 | mem: 108811MB | Time Left: 00 hrs : 27 mins [TIMING] data: 9.1ms | fwd: 3584.0ms | loss: 13.0ms | bwd: 10859.8ms | clip: 0.0ms | optim: 24.0ms step 639 | loss: 2.898967 | lr 1.8000e-04 | norm: 0.2059 | dt: 14494.23ms | tok/sec: 36172.2 | mem: 108811MB | Time Left: 00 hrs : 26 mins [TIMING] data: 9.0ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10856.4ms | clip: 0.0ms | optim: 24.1ms step 640 | loss: 2.938752 | lr 1.8000e-04 | norm: 0.2512 | dt: 14495.64ms | tok/sec: 36168.7 | mem: 108811MB | Time Left: 00 hrs : 26 mins [TIMING] data: 9.3ms | fwd: 3580.3ms | loss: 13.0ms | bwd: 10855.2ms | clip: 0.0ms | optim: 23.9ms step 641 | loss: 2.922678 | lr 1.8000e-04 | norm: 0.2774 | dt: 14497.74ms | tok/sec: 36163.4 | mem: 108811MB | Time Left: 00 hrs : 26 mins [TIMING] data: 9.1ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10859.1ms | clip: 0.0ms | optim: 24.0ms step 642 | loss: 3.060248 | lr 1.8000e-04 | norm: 0.2083 | dt: 14493.37ms | tok/sec: 36174.3 | mem: 108811MB | Time Left: 00 hrs : 26 mins [TIMING] data: 9.1ms | fwd: 3580.6ms | loss: 13.0ms | bwd: 10852.7ms | clip: 0.0ms | optim: 24.1ms step 643 | loss: 2.666942 | lr 1.8000e-04 | norm: 0.2197 | dt: 14489.32ms | tok/sec: 36184.4 | mem: 108811MB | Time Left: 00 hrs : 25 mins [TIMING] data: 9.0ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.9ms step 644 | loss: 2.953260 | lr 1.8000e-04 | norm: 0.2694 | dt: 14497.58ms | tok/sec: 36163.8 | mem: 108811MB | Time Left: 00 hrs : 25 mins [TIMING] data: 9.1ms | fwd: 3580.0ms | loss: 13.0ms | bwd: 10857.7ms | clip: 0.0ms | optim: 23.9ms step 645 | loss: 3.183621 | lr 1.8000e-04 | norm: 0.2412 | dt: 14492.53ms | tok/sec: 36176.4 | mem: 108811MB | Time Left: 00 hrs : 25 mins [TIMING] data: 9.2ms | fwd: 3577.9ms | loss: 13.0ms | bwd: 10854.4ms | clip: 0.0ms | optim: 23.9ms step 646 | loss: 3.165539 | lr 1.8000e-04 | norm: 0.2209 | dt: 14488.73ms | tok/sec: 36185.9 | mem: 108811MB | Time Left: 00 hrs : 25 mins [TIMING] data: 9.0ms | fwd: 3579.7ms | loss: 13.0ms | bwd: 10848.4ms | clip: 0.0ms | optim: 24.0ms step 647 | loss: 2.979330 | lr 1.8000e-04 | norm: 0.2340 | dt: 14487.50ms | tok/sec: 36189.0 | mem: 108811MB | Time Left: 00 hrs : 24 mins [TIMING] data: 9.0ms | fwd: 3578.9ms | loss: 13.0ms | bwd: 10848.7ms | clip: 0.0ms | optim: 24.0ms step 648 | loss: 3.054091 | lr 1.8000e-04 | norm: 0.2238 | dt: 14497.86ms | tok/sec: 36163.1 | mem: 108811MB | Time Left: 00 hrs : 24 mins [TIMING] data: 9.1ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10861.9ms | clip: 0.0ms | optim: 24.0ms step 649 | loss: 2.795266 | lr 1.8000e-04 | norm: 0.2031 | dt: 14487.45ms | tok/sec: 36189.1 | mem: 108811MB | Time Left: 00 hrs : 24 mins [TIMING] data: 9.1ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10849.0ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_650.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_600.ckpt step 650 | loss: 2.818265 | lr 1.8000e-04 | norm: 0.2137 | dt: 14357.36ms | tok/sec: 36517.0 | mem: 108811MB | Time Left: 00 hrs : 23 mins [TIMING] data: 9.2ms | fwd: 3569.1ms | loss: 13.1ms | bwd: 10727.8ms | clip: 0.0ms | optim: 24.0ms step 651 | loss: 2.739148 | lr 1.7997e-04 | norm: 0.1804 | dt: 14402.19ms | tok/sec: 36403.3 | mem: 108811MB | Time Left: 00 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3562.9ms | loss: 13.1ms | bwd: 10778.5ms | clip: 0.0ms | optim: 23.9ms step 652 | loss: 3.011411 | lr 1.7988e-04 | norm: 0.2215 | dt: 14427.68ms | tok/sec: 36339.0 | mem: 108811MB | Time Left: 00 hrs : 23 mins [TIMING] data: 9.1ms | fwd: 3569.4ms | loss: 13.0ms | bwd: 10798.4ms | clip: 0.0ms | optim: 23.9ms step 653 | loss: 2.800376 | lr 1.7973e-04 | norm: 0.2819 | dt: 14475.60ms | tok/sec: 36218.7 | mem: 108811MB | Time Left: 00 hrs : 23 mins [TIMING] data: 9.0ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10836.5ms | clip: 0.0ms | optim: 24.0ms step 654 | loss: 2.961151 | lr 1.7953e-04 | norm: 0.1998 | dt: 14500.23ms | tok/sec: 36157.2 | mem: 108811MB | Time Left: 00 hrs : 23 mins [TIMING] data: 9.0ms | fwd: 3580.4ms | loss: 13.0ms | bwd: 10859.1ms | clip: 0.0ms | optim: 23.9ms step 655 | loss: 2.765943 | lr 1.7926e-04 | norm: 0.1808 | dt: 14504.33ms | tok/sec: 36147.0 | mem: 108811MB | Time Left: 00 hrs : 22 mins [TIMING] data: 9.1ms | fwd: 3581.0ms | loss: 13.0ms | bwd: 10862.9ms | clip: 0.0ms | optim: 24.0ms step 656 | loss: 2.988137 | lr 1.7894e-04 | norm: 0.1883 | dt: 14525.30ms | tok/sec: 36094.8 | mem: 108811MB | Time Left: 00 hrs : 22 mins [TIMING] data: 9.2ms | fwd: 3584.4ms | loss: 13.1ms | bwd: 10880.6ms | clip: 0.0ms | optim: 23.9ms step 657 | loss: 2.927378 | lr 1.7856e-04 | norm: 0.3693 | dt: 14528.82ms | tok/sec: 36086.1 | mem: 108811MB | Time Left: 00 hrs : 22 mins [TIMING] data: 9.0ms | fwd: 3585.1ms | loss: 13.0ms | bwd: 10883.7ms | clip: 0.0ms | optim: 24.1ms step 658 | loss: 2.800832 | lr 1.7812e-04 | norm: 0.2001 | dt: 14520.69ms | tok/sec: 36106.3 | mem: 108811MB | Time Left: 00 hrs : 22 mins [TIMING] data: 9.1ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10878.6ms | clip: 0.0ms | optim: 24.1ms step 659 | loss: 3.087357 | lr 1.7762e-04 | norm: 0.7881 | dt: 14518.59ms | tok/sec: 36111.5 | mem: 108811MB | Time Left: 00 hrs : 22 mins [TIMING] data: 9.2ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10873.7ms | clip: 0.0ms | optim: 23.9ms step 660 | loss: 2.865639 | lr 1.7706e-04 | norm: 0.1962 | dt: 14519.75ms | tok/sec: 36108.6 | mem: 108811MB | Time Left: 00 hrs : 21 mins [TIMING] data: 9.2ms | fwd: 3584.3ms | loss: 13.0ms | bwd: 10875.4ms | clip: 0.0ms | optim: 23.9ms step 661 | loss: 2.945657 | lr 1.7645e-04 | norm: 0.2149 | dt: 14507.64ms | tok/sec: 36138.8 | mem: 108811MB | Time Left: 00 hrs : 21 mins [TIMING] data: 9.1ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10864.9ms | clip: 0.0ms | optim: 23.9ms step 662 | loss: 2.922560 | lr 1.7579e-04 | norm: 0.2078 | dt: 14499.73ms | tok/sec: 36158.5 | mem: 108811MB | Time Left: 00 hrs : 21 mins [TIMING] data: 9.1ms | fwd: 3578.9ms | loss: 13.0ms | bwd: 10860.9ms | clip: 0.0ms | optim: 23.9ms step 663 | loss: 3.053896 | lr 1.7507e-04 | norm: 0.2244 | dt: 14505.46ms | tok/sec: 36144.2 | mem: 108811MB | Time Left: 00 hrs : 21 mins [TIMING] data: 9.1ms | fwd: 3585.7ms | loss: 13.0ms | bwd: 10859.9ms | clip: 0.0ms | optim: 23.9ms step 664 | loss: 2.838351 | lr 1.7429e-04 | norm: 0.1825 | dt: 14498.93ms | tok/sec: 36160.5 | mem: 108811MB | Time Left: 00 hrs : 20 mins [TIMING] data: 9.2ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.9ms step 665 | loss: 2.805978 | lr 1.7346e-04 | norm: 0.2160 | dt: 14495.81ms | tok/sec: 36168.2 | mem: 108811MB | Time Left: 00 hrs : 20 mins [TIMING] data: 9.2ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10852.4ms | clip: 0.0ms | optim: 24.0ms step 666 | loss: 2.672565 | lr 1.7258e-04 | norm: 0.2310 | dt: 14490.62ms | tok/sec: 36181.2 | mem: 108811MB | Time Left: 00 hrs : 20 mins [TIMING] data: 9.1ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10852.1ms | clip: 0.0ms | optim: 23.9ms step 667 | loss: 2.870351 | lr 1.7164e-04 | norm: 0.2041 | dt: 14488.37ms | tok/sec: 36186.8 | mem: 108811MB | Time Left: 00 hrs : 20 mins [TIMING] data: 9.1ms | fwd: 3586.0ms | loss: 13.0ms | bwd: 10842.4ms | clip: 0.0ms | optim: 23.9ms step 668 | loss: 2.604917 | lr 1.7066e-04 | norm: 0.2572 | dt: 14482.98ms | tok/sec: 36200.3 | mem: 108811MB | Time Left: 00 hrs : 19 mins [TIMING] data: 9.0ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10844.9ms | clip: 0.0ms | optim: 24.0ms step 669 | loss: 2.862070 | lr 1.6962e-04 | norm: 0.2343 | dt: 14492.66ms | tok/sec: 36176.1 | mem: 108811MB | Time Left: 00 hrs : 19 mins [TIMING] data: 9.2ms | fwd: 3587.9ms | loss: 13.0ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.9ms step 670 | loss: 3.015907 | lr 1.6854e-04 | norm: 0.1955 | dt: 14487.40ms | tok/sec: 36189.2 | mem: 108811MB | Time Left: 00 hrs : 19 mins [TIMING] data: 9.0ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10851.0ms | clip: 0.0ms | optim: 24.0ms step 671 | loss: 2.698934 | lr 1.6741e-04 | norm: 0.1876 | dt: 14488.65ms | tok/sec: 36186.1 | mem: 108811MB | Time Left: 00 hrs : 19 mins [TIMING] data: 9.1ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 672 | loss: 2.846245 | lr 1.6623e-04 | norm: 0.2081 | dt: 14503.09ms | tok/sec: 36150.1 | mem: 108811MB | Time Left: 00 hrs : 18 mins [TIMING] data: 9.1ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10863.2ms | clip: 0.0ms | optim: 23.9ms step 673 | loss: 3.118910 | lr 1.6501e-04 | norm: 0.1884 | dt: 14492.18ms | tok/sec: 36177.3 | mem: 108811MB | Time Left: 00 hrs : 18 mins [TIMING] data: 9.0ms | fwd: 3582.5ms | loss: 13.0ms | bwd: 10849.6ms | clip: 0.0ms | optim: 24.1ms step 674 | loss: 2.670631 | lr 1.6374e-04 | norm: 0.2017 | dt: 14498.93ms | tok/sec: 36160.5 | mem: 108811MB | Time Left: 00 hrs : 18 mins [TIMING] data: 9.1ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10857.9ms | clip: 0.0ms | optim: 24.1ms step 675 | loss: 2.946723 | lr 1.6243e-04 | norm: 0.1700 | dt: 14486.89ms | tok/sec: 36190.5 | mem: 108811MB | Time Left: 00 hrs : 18 mins [TIMING] data: 9.0ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10848.4ms | clip: 0.0ms | optim: 24.0ms step 676 | loss: 2.917534 | lr 1.6107e-04 | norm: 0.1854 | dt: 14491.03ms | tok/sec: 36180.2 | mem: 108811MB | Time Left: 00 hrs : 17 mins [TIMING] data: 9.2ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10853.5ms | clip: 0.0ms | optim: 24.0ms step 677 | loss: 2.951978 | lr 1.5968e-04 | norm: 0.2843 | dt: 14493.31ms | tok/sec: 36174.5 | mem: 108811MB | Time Left: 00 hrs : 17 mins [TIMING] data: 9.1ms | fwd: 3578.3ms | loss: 13.2ms | bwd: 10854.7ms | clip: 0.0ms | optim: 24.0ms step 678 | loss: 2.842678 | lr 1.5825e-04 | norm: 0.3119 | dt: 14488.54ms | tok/sec: 36186.4 | mem: 108811MB | Time Left: 00 hrs : 17 mins [TIMING] data: 9.1ms | fwd: 3580.6ms | loss: 13.2ms | bwd: 10847.5ms | clip: 0.0ms | optim: 24.0ms step 679 | loss: 2.633100 | lr 1.5677e-04 | norm: 0.1752 | dt: 14482.31ms | tok/sec: 36202.0 | mem: 108811MB | Time Left: 00 hrs : 17 mins [TIMING] data: 9.1ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10843.6ms | clip: 0.0ms | optim: 24.0ms step 680 | loss: 2.860926 | lr 1.5527e-04 | norm: 0.2019 | dt: 14484.94ms | tok/sec: 36195.4 | mem: 108811MB | Time Left: 00 hrs : 16 mins [TIMING] data: 9.2ms | fwd: 3581.0ms | loss: 13.2ms | bwd: 10843.6ms | clip: 0.0ms | optim: 23.9ms step 681 | loss: 2.914783 | lr 1.5373e-04 | norm: 0.2059 | dt: 14496.14ms | tok/sec: 36167.4 | mem: 108811MB | Time Left: 00 hrs : 16 mins [TIMING] data: 9.2ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10852.8ms | clip: 0.0ms | optim: 23.9ms step 682 | loss: 3.036906 | lr 1.5215e-04 | norm: 0.1912 | dt: 14488.94ms | tok/sec: 36185.4 | mem: 108811MB | Time Left: 00 hrs : 16 mins [TIMING] data: 9.3ms | fwd: 3578.2ms | loss: 13.3ms | bwd: 10850.1ms | clip: 0.0ms | optim: 24.0ms step 683 | loss: 3.134027 | lr 1.5054e-04 | norm: 0.2844 | dt: 14481.45ms | tok/sec: 36204.1 | mem: 108811MB | Time Left: 00 hrs : 16 mins [TIMING] data: 10.2ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10840.8ms | clip: 0.0ms | optim: 24.1ms step 684 | loss: 3.058554 | lr 1.4891e-04 | norm: 0.1996 | dt: 14491.19ms | tok/sec: 36179.8 | mem: 108811MB | Time Left: 00 hrs : 15 mins [TIMING] data: 9.1ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10850.3ms | clip: 0.0ms | optim: 23.9ms step 685 | loss: 2.979847 | lr 1.4724e-04 | norm: 0.2450 | dt: 14495.08ms | tok/sec: 36170.1 | mem: 108811MB | Time Left: 00 hrs : 15 mins [TIMING] data: 9.1ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10858.0ms | clip: 0.0ms | optim: 23.9ms step 686 | loss: 3.218063 | lr 1.4555e-04 | norm: 0.1985 | dt: 14493.00ms | tok/sec: 36175.3 | mem: 108811MB | Time Left: 00 hrs : 15 mins [TIMING] data: 9.1ms | fwd: 3584.7ms | loss: 13.0ms | bwd: 10848.3ms | clip: 0.0ms | optim: 24.1ms step 687 | loss: 3.168598 | lr 1.4383e-04 | norm: 0.1743 | dt: 14487.61ms | tok/sec: 36188.7 | mem: 108811MB | Time Left: 00 hrs : 15 mins [TIMING] data: 9.1ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10847.8ms | clip: 0.0ms | optim: 23.9ms step 688 | loss: 3.175100 | lr 1.4209e-04 | norm: 0.2191 | dt: 14495.03ms | tok/sec: 36170.2 | mem: 108811MB | Time Left: 00 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10852.0ms | clip: 0.0ms | optim: 23.9ms step 689 | loss: 3.015139 | lr 1.4032e-04 | norm: 0.1869 | dt: 14490.93ms | tok/sec: 36180.4 | mem: 108811MB | Time Left: 00 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 23.9ms step 690 | loss: 3.015502 | lr 1.3854e-04 | norm: 0.2230 | dt: 14482.46ms | tok/sec: 36201.6 | mem: 108811MB | Time Left: 00 hrs : 14 mins [TIMING] data: 9.1ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10841.1ms | clip: 0.0ms | optim: 24.0ms step 691 | loss: 2.894999 | lr 1.3674e-04 | norm: 0.2745 | dt: 14480.07ms | tok/sec: 36207.6 | mem: 108811MB | Time Left: 00 hrs : 14 mins [TIMING] data: 9.3ms | fwd: 3580.7ms | loss: 13.0ms | bwd: 10839.1ms | clip: 0.0ms | optim: 24.0ms step 692 | loss: 2.537016 | lr 1.3492e-04 | norm: 0.1815 | dt: 14485.71ms | tok/sec: 36193.5 | mem: 108811MB | Time Left: 00 hrs : 14 mins [TIMING] data: 9.2ms | fwd: 3580.6ms | loss: 13.1ms | bwd: 10844.8ms | clip: 0.0ms | optim: 23.9ms step 693 | loss: 3.088858 | lr 1.3309e-04 | norm: 0.2186 | dt: 14481.15ms | tok/sec: 36204.9 | mem: 108811MB | Time Left: 00 hrs : 13 mins [TIMING] data: 9.2ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10841.6ms | clip: 0.0ms | optim: 24.0ms step 694 | loss: 2.682919 | lr 1.3124e-04 | norm: 0.2339 | dt: 14485.45ms | tok/sec: 36194.1 | mem: 108811MB | Time Left: 00 hrs : 13 mins [TIMING] data: 9.0ms | fwd: 3580.7ms | loss: 13.2ms | bwd: 10844.5ms | clip: 0.0ms | optim: 23.9ms step 695 | loss: 2.971765 | lr 1.2939e-04 | norm: 0.2308 | dt: 14494.21ms | tok/sec: 36172.2 | mem: 108811MB | Time Left: 00 hrs : 13 mins [TIMING] data: 9.1ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10852.1ms | clip: 0.0ms | optim: 24.1ms step 696 | loss: 2.630846 | lr 1.2752e-04 | norm: 0.2243 | dt: 14490.19ms | tok/sec: 36182.3 | mem: 108811MB | Time Left: 00 hrs : 13 mins [TIMING] data: 9.1ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10852.4ms | clip: 0.0ms | optim: 24.0ms step 697 | loss: 3.089769 | lr 1.2565e-04 | norm: 0.2004 | dt: 14480.31ms | tok/sec: 36207.0 | mem: 108811MB | Time Left: 00 hrs : 12 mins [TIMING] data: 9.2ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10839.7ms | clip: 0.0ms | optim: 24.0ms step 698 | loss: 3.028811 | lr 1.2377e-04 | norm: 0.2290 | dt: 14483.90ms | tok/sec: 36198.0 | mem: 108811MB | Time Left: 00 hrs : 12 mins [TIMING] data: 9.1ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10847.0ms | clip: 0.0ms | optim: 23.9ms step 699 | loss: 3.221430 | lr 1.2188e-04 | norm: 0.2076 | dt: 14482.69ms | tok/sec: 36201.0 | mem: 108811MB | Time Left: 00 hrs : 12 mins [TIMING] data: 9.2ms | fwd: 3578.6ms | loss: 13.2ms | bwd: 10843.7ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_700.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_650.ckpt step 700 | loss: 2.996058 | lr 1.2000e-04 | norm: 0.2206 | dt: 14397.45ms | tok/sec: 36415.3 | mem: 108811MB | Time Left: 00 hrs : 11 mins [TIMING] data: 9.2ms | fwd: 3577.9ms | loss: 13.2ms | bwd: 10758.8ms | clip: 0.0ms | optim: 24.0ms step 701 | loss: 3.046350 | lr 1.1812e-04 | norm: 0.1890 | dt: 14409.82ms | tok/sec: 36384.1 | mem: 108811MB | Time Left: 00 hrs : 11 mins [TIMING] data: 9.2ms | fwd: 3564.2ms | loss: 13.2ms | bwd: 10784.7ms | clip: 0.0ms | optim: 24.1ms step 702 | loss: 2.896552 | lr 1.1623e-04 | norm: 0.1980 | dt: 14451.31ms | tok/sec: 36279.6 | mem: 108811MB | Time Left: 00 hrs : 11 mins [TIMING] data: 9.3ms | fwd: 3583.5ms | loss: 13.1ms | bwd: 10807.0ms | clip: 0.0ms | optim: 23.9ms step 703 | loss: 3.225375 | lr 1.1435e-04 | norm: 0.2070 | dt: 14455.39ms | tok/sec: 36269.4 | mem: 108811MB | Time Left: 00 hrs : 11 mins [TIMING] data: 9.2ms | fwd: 3577.0ms | loss: 13.2ms | bwd: 10817.5ms | clip: 0.0ms | optim: 24.0ms step 704 | loss: 2.983716 | lr 1.1248e-04 | norm: 0.2102 | dt: 14478.30ms | tok/sec: 36212.0 | mem: 108811MB | Time Left: 00 hrs : 11 mins [TIMING] data: 9.1ms | fwd: 3577.6ms | loss: 13.2ms | bwd: 10840.3ms | clip: 0.0ms | optim: 24.0ms step 705 | loss: 2.697732 | lr 1.1061e-04 | norm: 0.1709 | dt: 14482.78ms | tok/sec: 36200.8 | mem: 108811MB | Time Left: 00 hrs : 10 mins [TIMING] data: 9.2ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10843.8ms | clip: 0.0ms | optim: 24.0ms step 706 | loss: 2.655070 | lr 1.0876e-04 | norm: 0.1999 | dt: 14504.91ms | tok/sec: 36145.6 | mem: 108811MB | Time Left: 00 hrs : 10 mins [TIMING] data: 9.3ms | fwd: 3580.3ms | loss: 13.2ms | bwd: 10864.0ms | clip: 0.0ms | optim: 24.0ms step 707 | loss: 2.835270 | lr 1.0691e-04 | norm: 0.2104 | dt: 14500.49ms | tok/sec: 36156.6 | mem: 108811MB | Time Left: 00 hrs : 10 mins [TIMING] data: 9.2ms | fwd: 3583.4ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.9ms step 708 | loss: 2.972885 | lr 1.0508e-04 | norm: 0.2147 | dt: 14505.79ms | tok/sec: 36143.4 | mem: 108811MB | Time Left: 00 hrs : 10 mins [TIMING] data: 9.2ms | fwd: 3579.3ms | loss: 13.2ms | bwd: 10865.8ms | clip: 0.0ms | optim: 24.0ms step 709 | loss: 3.083288 | lr 1.0326e-04 | norm: 0.1721 | dt: 14495.32ms | tok/sec: 36169.5 | mem: 108811MB | Time Left: 00 hrs : 09 mins [TIMING] data: 9.2ms | fwd: 3579.0ms | loss: 13.2ms | bwd: 10855.8ms | clip: 0.0ms | optim: 23.9ms step 710 | loss: 2.786848 | lr 1.0146e-04 | norm: 0.2010 | dt: 14498.91ms | tok/sec: 36160.5 | mem: 108811MB | Time Left: 00 hrs : 09 mins [TIMING] data: 9.2ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10855.7ms | clip: 0.0ms | optim: 24.0ms step 711 | loss: 2.945994 | lr 9.9676e-05 | norm: 0.1912 | dt: 14501.46ms | tok/sec: 36154.2 | mem: 108811MB | Time Left: 00 hrs : 09 mins [TIMING] data: 9.2ms | fwd: 3586.1ms | loss: 13.1ms | bwd: 10855.3ms | clip: 0.0ms | optim: 23.9ms step 712 | loss: 2.865387 | lr 9.7913e-05 | norm: 0.1661 | dt: 14505.38ms | tok/sec: 36144.4 | mem: 108811MB | Time Left: 00 hrs : 09 mins [TIMING] data: 9.2ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10863.7ms | clip: 0.0ms | optim: 23.9ms step 713 | loss: 3.055687 | lr 9.6171e-05 | norm: 0.1949 | dt: 14510.13ms | tok/sec: 36132.5 | mem: 108811MB | Time Left: 00 hrs : 08 mins [TIMING] data: 9.1ms | fwd: 3585.8ms | loss: 13.1ms | bwd: 10864.1ms | clip: 0.0ms | optim: 23.9ms step 714 | loss: 3.122402 | lr 9.4453e-05 | norm: 0.4405 | dt: 14503.40ms | tok/sec: 36149.3 | mem: 108811MB | Time Left: 00 hrs : 08 mins [TIMING] data: 9.1ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10861.4ms | clip: 0.0ms | optim: 23.9ms step 715 | loss: 2.948328 | lr 9.2761e-05 | norm: 0.2012 | dt: 14501.38ms | tok/sec: 36154.3 | mem: 108811MB | Time Left: 00 hrs : 08 mins [TIMING] data: 9.1ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10859.7ms | clip: 0.0ms | optim: 23.9ms step 716 | loss: 2.876810 | lr 9.1095e-05 | norm: 0.1868 | dt: 14510.40ms | tok/sec: 36131.9 | mem: 108811MB | Time Left: 00 hrs : 08 mins [TIMING] data: 9.1ms | fwd: 3585.9ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 23.9ms step 717 | loss: 2.829185 | lr 8.9458e-05 | norm: 0.1747 | dt: 14514.16ms | tok/sec: 36122.5 | mem: 108811MB | Time Left: 00 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3585.5ms | loss: 13.1ms | bwd: 10868.5ms | clip: 0.0ms | optim: 24.0ms step 718 | loss: 2.706986 | lr 8.7850e-05 | norm: 0.1531 | dt: 14518.62ms | tok/sec: 36111.4 | mem: 108811MB | Time Left: 00 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10880.0ms | clip: 0.0ms | optim: 23.9ms step 719 | loss: 2.940437 | lr 8.6275e-05 | norm: 0.1510 | dt: 14500.59ms | tok/sec: 36156.3 | mem: 108811MB | Time Left: 00 hrs : 07 mins [TIMING] data: 9.2ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10860.8ms | clip: 0.0ms | optim: 24.0ms step 720 | loss: 2.858772 | lr 8.4733e-05 | norm: 0.1756 | dt: 14516.58ms | tok/sec: 36116.5 | mem: 108811MB | Time Left: 00 hrs : 07 mins [TIMING] data: 9.0ms | fwd: 3586.3ms | loss: 13.1ms | bwd: 10870.0ms | clip: 0.0ms | optim: 23.9ms step 721 | loss: 2.748113 | lr 8.3226e-05 | norm: 0.1796 | dt: 14511.16ms | tok/sec: 36130.0 | mem: 108811MB | Time Left: 00 hrs : 07 mins [TIMING] data: 9.1ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10868.3ms | clip: 0.0ms | optim: 24.0ms step 722 | loss: 2.863867 | lr 8.1755e-05 | norm: 0.1748 | dt: 14507.52ms | tok/sec: 36139.0 | mem: 108811MB | Time Left: 00 hrs : 06 mins [TIMING] data: 9.1ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10865.8ms | clip: 0.0ms | optim: 23.9ms step 723 | loss: 2.450197 | lr 8.0321e-05 | norm: 0.1590 | dt: 14502.68ms | tok/sec: 36151.1 | mem: 108811MB | Time Left: 00 hrs : 06 mins [TIMING] data: 9.2ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10861.5ms | clip: 0.0ms | optim: 23.9ms step 724 | loss: 2.909838 | lr 7.8927e-05 | norm: 0.1810 | dt: 14518.01ms | tok/sec: 36112.9 | mem: 108811MB | Time Left: 00 hrs : 06 mins [TIMING] data: 9.2ms | fwd: 3586.9ms | loss: 13.2ms | bwd: 10870.6ms | clip: 0.0ms | optim: 23.9ms step 725 | loss: 2.953246 | lr 7.7574e-05 | norm: 0.1828 | dt: 14502.62ms | tok/sec: 36151.2 | mem: 108811MB | Time Left: 00 hrs : 06 mins [TIMING] data: 9.2ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10866.3ms | clip: 0.0ms | optim: 24.0ms step 726 | loss: 2.983500 | lr 7.6262e-05 | norm: 0.1534 | dt: 14518.86ms | tok/sec: 36110.8 | mem: 108811MB | Time Left: 00 hrs : 05 mins [TIMING] data: 9.1ms | fwd: 3582.8ms | loss: 13.2ms | bwd: 10875.3ms | clip: 0.0ms | optim: 24.0ms step 727 | loss: 2.860818 | lr 7.4993e-05 | norm: 0.1517 | dt: 14505.49ms | tok/sec: 36144.1 | mem: 108811MB | Time Left: 00 hrs : 05 mins [TIMING] data: 9.2ms | fwd: 3584.6ms | loss: 13.2ms | bwd: 10860.3ms | clip: 0.0ms | optim: 24.0ms step 728 | loss: 2.942635 | lr 7.3769e-05 | norm: 0.1425 | dt: 14516.28ms | tok/sec: 36117.2 | mem: 108811MB | Time Left: 00 hrs : 05 mins [TIMING] data: 9.3ms | fwd: 3581.2ms | loss: 13.2ms | bwd: 10874.3ms | clip: 0.0ms | optim: 24.0ms step 729 | loss: 3.035636 | lr 7.2591e-05 | norm: 0.1831 | dt: 14514.13ms | tok/sec: 36122.6 | mem: 108811MB | Time Left: 00 hrs : 05 mins [TIMING] data: 9.2ms | fwd: 3584.3ms | loss: 13.2ms | bwd: 10869.2ms | clip: 0.0ms | optim: 23.9ms step 730 | loss: 3.051334 | lr 7.1459e-05 | norm: 0.1600 | dt: 14509.20ms | tok/sec: 36134.9 | mem: 108811MB | Time Left: 00 hrs : 04 mins [TIMING] data: 9.1ms | fwd: 3581.3ms | loss: 13.2ms | bwd: 10866.5ms | clip: 0.0ms | optim: 24.1ms step 731 | loss: 2.689409 | lr 7.0375e-05 | norm: 0.1551 | dt: 14498.44ms | tok/sec: 36161.7 | mem: 108811MB | Time Left: 00 hrs : 04 mins [TIMING] data: 9.0ms | fwd: 3585.0ms | loss: 13.2ms | bwd: 10852.8ms | clip: 0.0ms | optim: 23.9ms step 732 | loss: 3.115417 | lr 6.9340e-05 | norm: 0.6182 | dt: 14503.61ms | tok/sec: 36148.8 | mem: 108811MB | Time Left: 00 hrs : 04 mins [TIMING] data: 9.3ms | fwd: 3585.1ms | loss: 13.2ms | bwd: 10857.8ms | clip: 0.0ms | optim: 23.9ms step 733 | loss: 2.733018 | lr 6.8355e-05 | norm: 0.1648 | dt: 14509.29ms | tok/sec: 36134.6 | mem: 108811MB | Time Left: 00 hrs : 04 mins [TIMING] data: 9.3ms | fwd: 3586.6ms | loss: 13.2ms | bwd: 10861.8ms | clip: 0.0ms | optim: 24.0ms step 734 | loss: 2.788075 | lr 6.7422e-05 | norm: 0.1548 | dt: 14505.64ms | tok/sec: 36143.7 | mem: 108811MB | Time Left: 00 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3585.8ms | loss: 13.1ms | bwd: 10859.5ms | clip: 0.0ms | optim: 24.0ms step 735 | loss: 2.759241 | lr 6.6540e-05 | norm: 0.1333 | dt: 14513.22ms | tok/sec: 36124.9 | mem: 108811MB | Time Left: 00 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10872.0ms | clip: 0.0ms | optim: 23.9ms step 736 | loss: 2.999213 | lr 6.5710e-05 | norm: 0.1597 | dt: 14520.92ms | tok/sec: 36105.7 | mem: 108811MB | Time Left: 00 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3587.6ms | loss: 13.1ms | bwd: 10872.6ms | clip: 0.0ms | optim: 24.2ms step 737 | loss: 2.899827 | lr 6.4935e-05 | norm: 0.2164 | dt: 14515.53ms | tok/sec: 36119.1 | mem: 108811MB | Time Left: 00 hrs : 03 mins [TIMING] data: 9.1ms | fwd: 3583.9ms | loss: 13.1ms | bwd: 10871.2ms | clip: 0.0ms | optim: 24.0ms step 738 | loss: 3.059820 | lr 6.4213e-05 | norm: 0.1513 | dt: 14508.24ms | tok/sec: 36137.3 | mem: 108811MB | Time Left: 00 hrs : 02 mins [TIMING] data: 9.0ms | fwd: 3585.1ms | loss: 13.1ms | bwd: 10862.8ms | clip: 0.0ms | optim: 24.0ms step 739 | loss: 3.132354 | lr 6.3547e-05 | norm: 0.1594 | dt: 14501.27ms | tok/sec: 36154.6 | mem: 108811MB | Time Left: 00 hrs : 02 mins [TIMING] data: 9.1ms | fwd: 3586.2ms | loss: 13.1ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.9ms step 740 | loss: 2.902499 | lr 6.2937e-05 | norm: 0.1552 | dt: 14501.48ms | tok/sec: 36154.1 | mem: 108811MB | Time Left: 00 hrs : 02 mins [TIMING] data: 9.1ms | fwd: 3584.9ms | loss: 13.1ms | bwd: 10856.2ms | clip: 0.0ms | optim: 23.9ms step 741 | loss: 2.870949 | lr 6.2382e-05 | norm: 0.1707 | dt: 14501.22ms | tok/sec: 36154.7 | mem: 108811MB | Time Left: 00 hrs : 02 mins [TIMING] data: 9.1ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10859.6ms | clip: 0.0ms | optim: 23.9ms step 742 | loss: 3.036855 | lr 6.1885e-05 | norm: 0.1433 | dt: 14502.34ms | tok/sec: 36152.0 | mem: 108811MB | Time Left: 00 hrs : 01 mins [TIMING] data: 9.1ms | fwd: 3584.0ms | loss: 13.1ms | bwd: 10858.0ms | clip: 0.0ms | optim: 23.9ms step 743 | loss: 2.798404 | lr 6.1445e-05 | norm: 0.2129 | dt: 14492.85ms | tok/sec: 36175.6 | mem: 108811MB | Time Left: 00 hrs : 01 mins [TIMING] data: 9.2ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10847.8ms | clip: 0.0ms | optim: 24.1ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00004.bin 100000000 step 744 | loss: 2.904940 | lr 6.1063e-05 | norm: 0.1540 | dt: 14505.53ms | tok/sec: 36144.0 | mem: 108811MB | Time Left: 00 hrs : 01 mins [TIMING] data: 14.3ms | fwd: 3584.0ms | loss: 13.1ms | bwd: 10856.1ms | clip: 0.0ms | optim: 23.9ms step 745 | loss: 2.885013 | lr 6.0739e-05 | norm: 0.1441 | dt: 14519.39ms | tok/sec: 36109.5 | mem: 108811MB | Time Left: 00 hrs : 01 mins [TIMING] data: 9.2ms | fwd: 3585.1ms | loss: 13.1ms | bwd: 10872.9ms | clip: 0.0ms | optim: 23.9ms step 746 | loss: 2.952884 | lr 6.0473e-05 | norm: 0.1544 | dt: 14501.85ms | tok/sec: 36153.2 | mem: 108811MB | Time Left: 00 hrs : 00 mins [TIMING] data: 9.3ms | fwd: 3583.4ms | loss: 13.1ms | bwd: 10858.2ms | clip: 0.0ms | optim: 23.9ms step 747 | loss: 3.121088 | lr 6.0266e-05 | norm: 0.1546 | dt: 14513.46ms | tok/sec: 36124.3 | mem: 108811MB | Time Left: 00 hrs : 00 mins [TIMING] data: 9.2ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10872.4ms | clip: 0.0ms | optim: 23.9ms step 748 | loss: 2.969872 | lr 6.0118e-05 | norm: 0.1911 | dt: 14512.38ms | tok/sec: 36127.0 | mem: 108811MB | Time Left: 00 hrs : 00 mins [TIMING] data: 9.2ms | fwd: 3584.6ms | loss: 13.1ms | bwd: 10867.5ms | clip: 0.0ms | optim: 24.1ms validation loss: 3.0714 step 749 | loss: 2.570224 | lr 6.0030e-05 | norm: 0.1536 | dt: 19093.83ms | tok/sec: 27458.5 | mem: 108811MB | Time Left: 00 hrs : 00 mins [TIMING] data: 9.1ms | fwd: 3586.4ms | loss: 13.1ms | bwd: 10867.3ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_700.ckpt === Training complete ===
=== Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
d_model/layers : 1536 / 36
q/kv heads : 12 / 4 (G=3, hd=128)
ffn_hidden : 4096 (SwiGLU)
context_length : 32768
vocab_size : 50304 tying=0
B x T : 1 x 32768
world_size : 2 (dp=1 x cp=2)
CP_ATTN_MODE : ring (load_balance=HeadTail)
global_batch : 524288 (grad_accum=16)
precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
max_lr/min_lr : 0.0006 / 6e-05
warmup/steps : 100 / 750 (WSD: stable=550, decay tail=100)
checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/lenbucket_l2_blumodelcp_run2_step_400.ckpt (base step 400, loss 2.79415) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 750
warmup_steps: 100
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 6 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (run 2)
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 1)
[re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
Starting training...
[CP ring P2P] OVERLAP: dedicated non-blocking stream
validation loss: 3.2605
[CP bwd] overlap=ON
step 0 | loss: 3.091515 | lr 1.8000e-06 | norm: 0.4821 | dt: 22251.54ms | tok/sec: 23561.9 | mem: 107275MB | Time Left: 04 hrs : 38 mins
[TIMING] data: 7.2ms | fwd: 4388.1ms | loss: 13.2ms | bwd: 11046.9ms | clip: 0.0ms | optim: 24.4ms
step 1 | loss: 2.851759 | lr 3.6000e-06 | norm: 0.4942 | dt: 15109.42ms | tok/sec: 34699.4 | mem: 108811MB | Time Left: 03 hrs : 08 mins
[TIMING] data: 8.5ms | fwd: 4285.8ms | loss: 13.2ms | bwd: 10764.1ms | clip: 0.0ms | optim: 23.8ms
step 2 | loss: 3.259181 | lr 5.4000e-06 | norm: 0.5146 | dt: 15152.84ms | tok/sec: 34600.0 | mem: 108811MB | Time Left: 03 hrs : 08 mins
[TIMING] data: 7.4ms | fwd: 4317.4ms | loss: 13.1ms | bwd: 10777.0ms | clip: 0.0ms | optim: 23.9ms
step 3 | loss: 3.104079 | lr 7.2000e-06 | norm: 0.4620 | dt: 14419.06ms | tok/sec: 36360.8 | mem: 108811MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 7.5ms | fwd: 3555.4ms | loss: 13.1ms | bwd: 10804.7ms | clip: 0.0ms | optim: 23.9ms
step 4 | loss: 2.992731 | lr 9.0000e-06 | norm: 0.3655 | dt: 14449.92ms | tok/sec: 36283.1 | mem: 108811MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 7.4ms | fwd: 3571.0ms | loss: 13.0ms | bwd: 10820.3ms | clip: 0.0ms | optim: 23.9ms
step 5 | loss: 2.949207 | lr 1.0800e-05 | norm: 0.3143 | dt: 14492.56ms | tok/sec: 36176.4 | mem: 108811MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 7.3ms | fwd: 3570.1ms | loss: 13.0ms | bwd: 10864.1ms | clip: 0.0ms | optim: 24.0ms
step 6 | loss: 2.879876 | lr 1.2600e-05 | norm: 0.2052 | dt: 14508.82ms | tok/sec: 36135.8 | mem: 108811MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 7.4ms | fwd: 3592.1ms | loss: 13.0ms | bwd: 10858.0ms | clip: 0.0ms | optim: 23.9ms
step 7 | loss: 3.127487 | lr 1.4400e-05 | norm: 0.1999 | dt: 14508.29ms | tok/sec: 36137.1 | mem: 108811MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 7.2ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10873.0ms | clip: 0.0ms | optim: 23.8ms
step 8 | loss: 3.158005 | lr 1.6200e-05 | norm: 0.2220 | dt: 14501.11ms | tok/sec: 36155.0 | mem: 108811MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 7.2ms | fwd: 3575.5ms | loss: 13.0ms | bwd: 10867.3ms | clip: 0.0ms | optim: 24.0ms
step 9 | loss: 2.938362 | lr 1.8000e-05 | norm: 0.3525 | dt: 14492.75ms | tok/sec: 36175.9 | mem: 108811MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 7.2ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10854.7ms | clip: 0.0ms | optim: 23.8ms
step 10 | loss: 2.920319 | lr 1.9800e-05 | norm: 0.3592 | dt: 14504.87ms | tok/sec: 36145.6 | mem: 108811MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 7.1ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10867.4ms | clip: 0.0ms | optim: 24.0ms
step 11 | loss: 2.894566 | lr 2.1600e-05 | norm: 0.2804 | dt: 14494.86ms | tok/sec: 36170.6 | mem: 108811MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 7.3ms | fwd: 3571.9ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 24.0ms
step 12 | loss: 3.158563 | lr 2.3400e-05 | norm: 0.2971 | dt: 14498.34ms | tok/sec: 36161.9 | mem: 108811MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 7.1ms | fwd: 3574.5ms | loss: 13.1ms | bwd: 10865.7ms | clip: 0.0ms | optim: 23.9ms
step 13 | loss: 2.949954 | lr 2.5200e-05 | norm: 0.2729 | dt: 14487.26ms | tok/sec: 36189.6 | mem: 108811MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 7.1ms | fwd: 3586.9ms | loss: 13.2ms | bwd: 10841.8ms | clip: 0.0ms | optim: 24.0ms
step 14 | loss: 3.281897 | lr 2.7000e-05 | norm: 0.2473 | dt: 14486.78ms | tok/sec: 36190.8 | mem: 108811MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 7.1ms | fwd: 3596.4ms | loss: 13.1ms | bwd: 10832.2ms | clip: 0.0ms | optim: 23.9ms
step 15 | loss: 2.717209 | lr 2.8800e-05 | norm: 0.2128 | dt: 14492.40ms | tok/sec: 36176.7 | mem: 108811MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 7.2ms | fwd: 3587.9ms | loss: 13.1ms | bwd: 10846.1ms | clip: 0.0ms | optim: 24.1ms
step 16 | loss: 2.953643 | lr 3.0600e-05 | norm: 1.7032 | dt: 14493.01ms | tok/sec: 36175.2 | mem: 108811MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 7.1ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10858.8ms | clip: 0.0ms | optim: 23.8ms
step 17 | loss: 2.865139 | lr 3.2400e-05 | norm: 0.1980 | dt: 14513.24ms | tok/sec: 36124.8 | mem: 108811MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 7.1ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10879.3ms | clip: 0.0ms | optim: 24.0ms
step 18 | loss: 2.843351 | lr 3.4200e-05 | norm: 0.2114 | dt: 14498.66ms | tok/sec: 36161.1 | mem: 108811MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 7.1ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10861.1ms | clip: 0.0ms | optim: 24.0ms
step 19 | loss: 3.087666 | lr 3.6000e-05 | norm: 0.2159 | dt: 14508.12ms | tok/sec: 36137.6 | mem: 108811MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 7.2ms | fwd: 3573.7ms | loss: 13.0ms | bwd: 10875.9ms | clip: 0.0ms | optim: 24.1ms
step 20 | loss: 2.901704 | lr 3.7800e-05 | norm: 0.1994 | dt: 14478.30ms | tok/sec: 36212.0 | mem: 108811MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 7.1ms | fwd: 3571.5ms | loss: 13.1ms | bwd: 10848.8ms | clip: 0.0ms | optim: 23.9ms
step 21 | loss: 2.904223 | lr 3.9600e-05 | norm: 0.2292 | dt: 14469.39ms | tok/sec: 36234.3 | mem: 108811MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 7.2ms | fwd: 3572.4ms | loss: 13.0ms | bwd: 10838.5ms | clip: 0.0ms | optim: 24.3ms
step 22 | loss: 2.614633 | lr 4.1400e-05 | norm: 0.1743 | dt: 14486.56ms | tok/sec: 36191.3 | mem: 108811MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 7.1ms | fwd: 3576.6ms | loss: 13.0ms | bwd: 10851.9ms | clip: 0.0ms | optim: 23.9ms
step 23 | loss: 2.814205 | lr 4.3200e-05 | norm: 0.1738 | dt: 14475.65ms | tok/sec: 36218.6 | mem: 108811MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 7.3ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10842.1ms | clip: 0.0ms | optim: 24.0ms
step 24 | loss: 2.763636 | lr 4.5000e-05 | norm: 0.1923 | dt: 14475.62ms | tok/sec: 36218.7 | mem: 108811MB | Time Left: 02 hrs : 55 mins