=== Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
d_model/layers : 1536 / 36
q/kv heads : 12 / 4 (G=3, hd=128)
ffn_hidden : 4096 (SwiGLU)
context_length : 65536
vocab_size : 50304 tying=0
B x T : 1 x 65536
world_size : 8 (dp=1 x cp=8)
CP_ATTN_MODE : ring (load_balance=HeadTail)
global_batch : 524288 (grad_accum=8)
precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
max_lr/min_lr : 0.0006 / 6e-05
warmup/steps : 100 / 1651 (WSD: stable=1451, decay tail=100)
checkpointing : true (ckpt_freq=100, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_695.ckpt (base step 695, loss 2.93407) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 1651
warmup_steps: 100
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=983236608 padded=983236608 W=8 shard=122904576 | 38 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 1406.53 MB (vs 11252.2 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=77388288 padded=77388288 W=8 shard=9673536 | 6 buckets of 50 MB (1638400 elems/rank/bucket) | optimizer state/GPU: 110.705 MB (vs 885.639 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 9 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log4.csv (run 4)
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run4_step_<N>.ckpt (every 100 steps, keep 2)
[re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
Starting training...
[CP ring P2P] OVERLAP: dedicated non-blocking stream
validation loss: 2.9599
[CP bwd] overlap=ON
step 0 | loss: 2.840227 | lr 1.8000e-06 | norm: 0.1970 | dt: 29719.04ms | tok/sec: 17641.5 | mem: 70435MB | Time Left: 13 hrs : 37 mins
[TIMING] data: 7.8ms | fwd: 9418.1ms | loss: 5.0ms | bwd: 7032.6ms | clip: 0.0ms | optim: 24.2ms
step 1 | loss: 3.123918 | lr 3.6000e-06 | norm: 0.3054 | dt: 15186.85ms | tok/sec: 34522.5 | mem: 72099MB | Time Left: 06 hrs : 57 mins
[TIMING] data: 8.7ms | fwd: 9279.4ms | loss: 4.8ms | bwd: 5860.0ms | clip: 0.0ms | optim: 23.5ms
step 2 | loss: 2.830065 | lr 5.4000e-06 | norm: 0.2178 | dt: 13818.87ms | tok/sec: 37940.0 | mem: 72099MB | Time Left: 06 hrs : 19 mins
[TIMING] data: 12.5ms | fwd: 7140.5ms | loss: 4.9ms | bwd: 6626.1ms | clip: 0.0ms | optim: 23.5ms
step 3 | loss: 3.057204 | lr 7.2000e-06 | norm: 0.2737 | dt: 7780.31ms | tok/sec: 67386.5 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.0ms | fwd: 2385.2ms | loss: 4.9ms | bwd: 5342.7ms | clip: 0.0ms | optim: 23.5ms
step 4 | loss: 2.997180 | lr 9.0000e-06 | norm: 0.2248 | dt: 7853.07ms | tok/sec: 66762.2 | mem: 72099MB | Time Left: 03 hrs : 35 mins
[TIMING] data: 12.4ms | fwd: 2455.4ms | loss: 4.9ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.4ms
step 5 | loss: 3.072381 | lr 1.0800e-05 | norm: 0.2459 | dt: 7792.71ms | tok/sec: 67279.3 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.8ms | fwd: 2393.4ms | loss: 4.9ms | bwd: 5347.1ms | clip: 0.0ms | optim: 23.4ms
step 6 | loss: 2.878336 | lr 1.2600e-05 | norm: 0.2714 | dt: 7853.19ms | tok/sec: 66761.1 | mem: 72099MB | Time Left: 03 hrs : 35 mins
[TIMING] data: 12.7ms | fwd: 2454.9ms | loss: 4.9ms | bwd: 5345.6ms | clip: 0.0ms | optim: 23.5ms
step 7 | loss: 3.410254 | lr 1.4400e-05 | norm: 0.4509 | dt: 7792.49ms | tok/sec: 67281.2 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.0ms | fwd: 2395.6ms | loss: 4.9ms | bwd: 5344.4ms | clip: 0.0ms | optim: 23.7ms
step 8 | loss: 2.799212 | lr 1.6200e-05 | norm: 0.2074 | dt: 7791.99ms | tok/sec: 67285.5 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.5ms | fwd: 2391.7ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.5ms
step 9 | loss: 2.779159 | lr 1.8000e-05 | norm: 0.2054 | dt: 7797.31ms | tok/sec: 67239.6 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.8ms | fwd: 2397.4ms | loss: 4.9ms | bwd: 5347.8ms | clip: 0.0ms | optim: 23.5ms
step 10 | loss: 3.073190 | lr 1.9800e-05 | norm: 0.1787 | dt: 7799.03ms | tok/sec: 67224.8 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.7ms | fwd: 2396.3ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 11 | loss: 2.813490 | lr 2.1600e-05 | norm: 0.2286 | dt: 7795.71ms | tok/sec: 67253.4 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.0ms | fwd: 2393.2ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 12 | loss: 2.975980 | lr 2.3400e-05 | norm: 0.2559 | dt: 7797.30ms | tok/sec: 67239.7 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 11.9ms | fwd: 2394.7ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.5ms
step 13 | loss: 2.579046 | lr 2.5200e-05 | norm: 0.2544 | dt: 7801.83ms | tok/sec: 67200.7 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 13.2ms | fwd: 2396.1ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.7ms
step 14 | loss: 3.115849 | lr 2.7000e-05 | norm: 0.2300 | dt: 7805.95ms | tok/sec: 67165.2 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 13.9ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 15 | loss: 2.858879 | lr 2.8800e-05 | norm: 0.2256 | dt: 7811.34ms | tok/sec: 67118.8 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 13.9ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5349.0ms | clip: 0.0ms | optim: 23.5ms
step 16 | loss: 2.949294 | lr 3.0600e-05 | norm: 0.2114 | dt: 7885.15ms | tok/sec: 66490.5 | mem: 72099MB | Time Left: 03 hrs : 34 mins
[TIMING] data: 17.1ms | fwd: 2477.2ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.6ms
step 17 | loss: 2.993725 | lr 3.2400e-05 | norm: 0.2044 | dt: 7808.61ms | tok/sec: 67142.3 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 14.5ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5340.4ms | clip: 0.0ms | optim: 23.6ms
step 18 | loss: 2.746970 | lr 3.4200e-05 | norm: 0.2289 | dt: 7806.60ms | tok/sec: 67159.6 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 14.9ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.5ms
step 19 | loss: 2.770875 | lr 3.6000e-05 | norm: 0.1799 | dt: 7809.73ms | tok/sec: 67132.6 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 12.6ms | fwd: 2406.8ms | loss: 4.9ms | bwd: 5349.4ms | clip: 0.0ms | optim: 23.7ms
step 20 | loss: 2.935153 | lr 3.7800e-05 | norm: 0.1628 | dt: 7880.46ms | tok/sec: 66530.1 | mem: 72099MB | Time Left: 03 hrs : 34 mins
[TIMING] data: 14.9ms | fwd: 2476.7ms | loss: 4.9ms | bwd: 5346.8ms | clip: 0.0ms | optim: 23.5ms
step 21 | loss: 2.582499 | lr 3.9600e-05 | norm: 0.2081 | dt: 7870.75ms | tok/sec: 66612.2 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.5ms | fwd: 2470.1ms | loss: 4.9ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.5ms
step 22 | loss: 3.091026 | lr 4.1400e-05 | norm: 0.2121 | dt: 7807.95ms | tok/sec: 67148.0 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 13.5ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5345.1ms | clip: 0.0ms | optim: 23.5ms
step 23 | loss: 2.649695 | lr 4.3200e-05 | norm: 0.1683 | dt: 7818.67ms | tok/sec: 67055.9 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 14.8ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 24 | loss: 2.540149 | lr 4.5000e-05 | norm: 0.1690 | dt: 7826.39ms | tok/sec: 66989.8 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 16.2ms | fwd: 2413.3ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.6ms
step 25 | loss: 3.112947 | lr 4.6800e-05 | norm: 0.1875 | dt: 7818.54ms | tok/sec: 67057.0 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 14.2ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.7ms
step 26 | loss: 2.841418 | lr 4.8600e-05 | norm: 0.1719 | dt: 7812.25ms | tok/sec: 67111.0 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 12.9ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.8ms
step 27 | loss: 2.858640 | lr 5.0400e-05 | norm: 0.1908 | dt: 7809.07ms | tok/sec: 67138.3 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 13.6ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.6ms
step 28 | loss: 2.994960 | lr 5.2200e-05 | norm: 0.2152 | dt: 7820.00ms | tok/sec: 67044.5 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 13.4ms | fwd: 2413.7ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 29 | loss: 2.507958 | lr 5.4000e-05 | norm: 0.2805 | dt: 7882.00ms | tok/sec: 66517.2 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.2ms | fwd: 2477.8ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.6ms
step 30 | loss: 3.152998 | lr 5.5800e-05 | norm: 0.1709 | dt: 7819.38ms | tok/sec: 67049.8 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 13.8ms | fwd: 2412.4ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 31 | loss: 2.837375 | lr 5.7600e-05 | norm: 0.1859 | dt: 7818.88ms | tok/sec: 67054.1 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 14.8ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
step 32 | loss: 2.792946 | lr 5.9400e-05 | norm: 0.1846 | dt: 7814.37ms | tok/sec: 67092.8 | mem: 72099MB | Time Left: 03 hrs : 30 mins
[TIMING] data: 14.1ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.7ms
step 33 | loss: 2.898873 | lr 6.1200e-05 | norm: 0.1719 | dt: 7823.14ms | tok/sec: 67017.6 | mem: 72099MB | Time Left: 03 hrs : 30 mins
[TIMING] data: 13.9ms | fwd: 2411.0ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 34 | loss: 2.548232 | lr 6.3000e-05 | norm: 0.1930 | dt: 7818.72ms | tok/sec: 67055.5 | mem: 72099MB | Time Left: 03 hrs : 30 mins
[TIMING] data: 14.8ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 35 | loss: 2.847215 | lr 6.4800e-05 | norm: 0.2671 | dt: 7818.94ms | tok/sec: 67053.6 | mem: 72099MB | Time Left: 03 hrs : 30 mins
[TIMING] data: 13.8ms | fwd: 2412.2ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 36 | loss: 2.788475 | lr 6.6600e-05 | norm: 0.1700 | dt: 7817.46ms | tok/sec: 67066.3 | mem: 72099MB | Time Left: 03 hrs : 30 mins
[TIMING] data: 11.4ms | fwd: 2414.1ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
step 37 | loss: 2.965258 | lr 6.8400e-05 | norm: 0.1712 | dt: 7817.38ms | tok/sec: 67066.9 | mem: 72099MB | Time Left: 03 hrs : 30 mins
[TIMING] data: 15.7ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.7ms
step 38 | loss: 2.561030 | lr 7.0200e-05 | norm: 0.1855 | dt: 7805.09ms | tok/sec: 67172.6 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 13.1ms | fwd: 2398.6ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 39 | loss: 3.023897 | lr 7.2000e-05 | norm: 0.1748 | dt: 7823.13ms | tok/sec: 67017.7 | mem: 72099MB | Time Left: 03 hrs : 30 mins
[TIMING] data: 13.8ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.5ms
step 40 | loss: 2.623007 | lr 7.3800e-05 | norm: 0.1812 | dt: 7812.93ms | tok/sec: 67105.2 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 12.4ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 41 | loss: 3.098504 | lr 7.5600e-05 | norm: 0.6506 | dt: 7817.91ms | tok/sec: 67062.4 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 12.6ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 42 | loss: 3.055842 | lr 7.7400e-05 | norm: 0.2278 | dt: 7821.00ms | tok/sec: 67035.9 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 13.5ms | fwd: 2414.7ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.5ms
step 43 | loss: 2.964234 | lr 7.9200e-05 | norm: 0.1784 | dt: 7825.53ms | tok/sec: 66997.1 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 14.9ms | fwd: 2412.8ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.5ms
step 44 | loss: 3.077931 | lr 8.1000e-05 | norm: 0.1777 | dt: 7817.66ms | tok/sec: 67064.6 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 15.0ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 45 | loss: 2.790999 | lr 8.2800e-05 | norm: 0.2054 | dt: 7823.02ms | tok/sec: 67018.6 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 13.1ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.6ms
step 46 | loss: 2.833812 | lr 8.4600e-05 | norm: 0.1975 | dt: 7820.73ms | tok/sec: 67038.3 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 13.4ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.5ms
step 47 | loss: 2.492061 | lr 8.6400e-05 | norm: 0.3045 | dt: 7818.62ms | tok/sec: 67056.3 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 17.3ms | fwd: 2412.1ms | loss: 4.9ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.6ms
step 48 | loss: 3.267377 | lr 8.8200e-05 | norm: 0.2414 | dt: 7810.46ms | tok/sec: 67126.4 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 14.1ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5349.4ms | clip: 0.0ms | optim: 23.5ms
step 49 | loss: 3.181830 | lr 9.0000e-05 | norm: 0.1800 | dt: 7815.71ms | tok/sec: 67081.3 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 11.3ms | fwd: 2412.9ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.6ms
step 50 | loss: 2.876213 | lr 9.1800e-05 | norm: 0.2717 | dt: 7820.28ms | tok/sec: 67042.1 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 14.7ms | fwd: 2411.1ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 51 | loss: 2.580332 | lr 9.3600e-05 | norm: 0.2319 | dt: 7830.44ms | tok/sec: 66955.1 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 16.6ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5363.5ms | clip: 0.0ms | optim: 23.7ms
step 52 | loss: 2.720790 | lr 9.5400e-05 | norm: 0.2169 | dt: 7824.15ms | tok/sec: 67008.9 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 14.9ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.5ms
step 53 | loss: 3.007932 | lr 9.7200e-05 | norm: 0.2149 | dt: 7818.77ms | tok/sec: 67055.0 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 14.3ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.6ms
step 54 | loss: 3.099779 | lr 9.9000e-05 | norm: 0.1934 | dt: 7820.83ms | tok/sec: 67037.4 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 15.9ms | fwd: 2414.1ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.6ms
step 55 | loss: 3.222561 | lr 1.0080e-04 | norm: 0.1911 | dt: 7887.16ms | tok/sec: 66473.6 | mem: 72099MB | Time Left: 03 hrs : 29 mins
[TIMING] data: 13.2ms | fwd: 2475.6ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.7ms
step 56 | loss: 2.943473 | lr 1.0260e-04 | norm: 0.2088 | dt: 7818.23ms | tok/sec: 67059.7 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 12.6ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.7ms
step 57 | loss: 2.897210 | lr 1.0440e-04 | norm: 0.1885 | dt: 7826.28ms | tok/sec: 66990.7 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 12.8ms | fwd: 2412.9ms | loss: 4.9ms | bwd: 5360.6ms | clip: 0.0ms | optim: 23.6ms
step 58 | loss: 2.776062 | lr 1.0620e-04 | norm: 0.1800 | dt: 7817.91ms | tok/sec: 67062.4 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 13.4ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.5ms
step 59 | loss: 2.712540 | lr 1.0800e-04 | norm: 0.1888 | dt: 7825.32ms | tok/sec: 66998.9 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 13.8ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5357.3ms | clip: 0.0ms | optim: 23.5ms
step 60 | loss: 2.773854 | lr 1.0980e-04 | norm: 0.2067 | dt: 7818.48ms | tok/sec: 67057.6 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 13.4ms | fwd: 2412.5ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 61 | loss: 2.449683 | lr 1.1160e-04 | norm: 0.2367 | dt: 7820.97ms | tok/sec: 67036.2 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 14.2ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5359.5ms | clip: 0.0ms | optim: 23.6ms
step 62 | loss: 2.860871 | lr 1.1340e-04 | norm: 0.2009 | dt: 7843.73ms | tok/sec: 66841.6 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 34.2ms | fwd: 2411.0ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 63 | loss: 2.819612 | lr 1.1520e-04 | norm: 0.1733 | dt: 7827.14ms | tok/sec: 66983.3 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 21.1ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 64 | loss: 3.001667 | lr 1.1700e-04 | norm: 0.2155 | dt: 7824.55ms | tok/sec: 67005.5 | mem: 72099MB | Time Left: 03 hrs : 26 mins
[TIMING] data: 13.7ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5364.8ms | clip: 0.0ms | optim: 23.5ms
step 65 | loss: 2.881340 | lr 1.1880e-04 | norm: 0.2913 | dt: 7816.46ms | tok/sec: 67074.9 | mem: 72099MB | Time Left: 03 hrs : 26 mins
[TIMING] data: 13.3ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.5ms
step 66 | loss: 2.744796 | lr 1.2060e-04 | norm: 0.2005 | dt: 7821.51ms | tok/sec: 67031.5 | mem: 72099MB | Time Left: 03 hrs : 26 mins
[TIMING] data: 16.4ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5360.1ms | clip: 0.0ms | optim: 23.5ms
step 67 | loss: 2.773693 | lr 1.2240e-04 | norm: 0.1725 | dt: 7880.62ms | tok/sec: 66528.8 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 14.5ms | fwd: 2472.5ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 68 | loss: 2.841912 | lr 1.2420e-04 | norm: 0.2263 | dt: 7814.86ms | tok/sec: 67088.6 | mem: 72099MB | Time Left: 03 hrs : 26 mins
[TIMING] data: 13.8ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 69 | loss: 2.833423 | lr 1.2600e-04 | norm: 0.1933 | dt: 7819.29ms | tok/sec: 67050.5 | mem: 72099MB | Time Left: 03 hrs : 26 mins
[TIMING] data: 12.3ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5359.7ms | clip: 0.0ms | optim: 23.6ms
step 70 | loss: 2.967788 | lr 1.2780e-04 | norm: 0.2560 | dt: 7820.82ms | tok/sec: 67037.5 | mem: 72099MB | Time Left: 03 hrs : 26 mins
[TIMING] data: 16.6ms | fwd: 2405.9ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.6ms
step 71 | loss: 2.651450 | lr 1.2960e-04 | norm: 0.1868 | dt: 7820.85ms | tok/sec: 67037.2 | mem: 72099MB | Time Left: 03 hrs : 25 mins
[TIMING] data: 13.5ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 72 | loss: 3.095555 | lr 1.3140e-04 | norm: 0.2196 | dt: 7817.54ms | tok/sec: 67065.6 | mem: 72099MB | Time Left: 03 hrs : 25 mins
[TIMING] data: 14.2ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 73 | loss: 2.941569 | lr 1.3320e-04 | norm: 0.2029 | dt: 7817.53ms | tok/sec: 67065.7 | mem: 72099MB | Time Left: 03 hrs : 25 mins
[TIMING] data: 13.7ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 74 | loss: 2.511052 | lr 1.3500e-04 | norm: 0.1923 | dt: 7814.92ms | tok/sec: 67088.1 | mem: 72099MB | Time Left: 03 hrs : 25 mins
[TIMING] data: 13.6ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 75 | loss: 2.996131 | lr 1.3680e-04 | norm: 0.1986 | dt: 7824.01ms | tok/sec: 67010.1 | mem: 72099MB | Time Left: 03 hrs : 25 mins
[TIMING] data: 13.1ms | fwd: 2413.5ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.5ms
step 76 | loss: 2.368078 | lr 1.3860e-04 | norm: 0.2207 | dt: 7828.80ms | tok/sec: 66969.1 | mem: 72099MB | Time Left: 03 hrs : 25 mins
[TIMING] data: 14.4ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5361.2ms | clip: 0.0ms | optim: 23.6ms
step 77 | loss: 2.768067 | lr 1.4040e-04 | norm: 0.2478 | dt: 7823.77ms | tok/sec: 67012.2 | mem: 72099MB | Time Left: 03 hrs : 25 mins
[TIMING] data: 13.5ms | fwd: 2414.9ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 78 | loss: 2.666159 | lr 1.4220e-04 | norm: 0.1915 | dt: 7817.22ms | tok/sec: 67068.4 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 13.4ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.5ms
step 79 | loss: 2.559243 | lr 1.4400e-04 | norm: 0.2988 | dt: 7821.51ms | tok/sec: 67031.6 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 14.6ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.6ms
step 80 | loss: 2.520707 | lr 1.4580e-04 | norm: 0.1785 | dt: 7814.43ms | tok/sec: 67092.3 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 15.0ms | fwd: 2399.3ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.6ms
step 81 | loss: 2.985871 | lr 1.4760e-04 | norm: 0.2261 | dt: 7824.43ms | tok/sec: 67006.5 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 15.0ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.7ms
step 82 | loss: 3.038070 | lr 1.4940e-04 | norm: 0.2222 | dt: 7824.87ms | tok/sec: 67002.8 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 12.4ms | fwd: 2417.2ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.5ms
step 83 | loss: 2.878777 | lr 1.5120e-04 | norm: 0.2142 | dt: 7824.18ms | tok/sec: 67008.7 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 14.2ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5364.2ms | clip: 0.0ms | optim: 23.6ms
step 84 | loss: 2.919199 | lr 1.5300e-04 | norm: 0.1993 | dt: 7818.89ms | tok/sec: 67054.0 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 12.8ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.6ms
step 85 | loss: 2.894819 | lr 1.5480e-04 | norm: 0.2277 | dt: 7818.06ms | tok/sec: 67061.2 | mem: 72099MB | Time Left: 03 hrs : 24 mins
[TIMING] data: 14.0ms | fwd: 2406.8ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.7ms
step 86 | loss: 3.042660 | lr 1.5660e-04 | norm: 0.2056 | dt: 7820.69ms | tok/sec: 67038.6 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 14.8ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5358.2ms | clip: 0.0ms | optim: 23.6ms
step 87 | loss: 2.656151 | lr 1.5840e-04 | norm: 0.2706 | dt: 7822.30ms | tok/sec: 67024.8 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 13.8ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.5ms
step 88 | loss: 2.926129 | lr 1.6020e-04 | norm: 0.2329 | dt: 7819.61ms | tok/sec: 67047.8 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 14.2ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 89 | loss: 2.996919 | lr 1.6200e-04 | norm: 0.2139 | dt: 7808.34ms | tok/sec: 67144.6 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 13.3ms | fwd: 2398.9ms | loss: 5.0ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.7ms
step 90 | loss: 2.829486 | lr 1.6380e-04 | norm: 0.2189 | dt: 7820.12ms | tok/sec: 67043.5 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 12.6ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.5ms
step 91 | loss: 2.747809 | lr 1.6560e-04 | norm: 0.3602 | dt: 7813.10ms | tok/sec: 67103.7 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 14.4ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.7ms
step 92 | loss: 2.904679 | lr 1.6740e-04 | norm: 0.2456 | dt: 7822.37ms | tok/sec: 67024.2 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 13.8ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 93 | loss: 2.760648 | lr 1.6920e-04 | norm: 0.4409 | dt: 7834.29ms | tok/sec: 66922.2 | mem: 72099MB | Time Left: 03 hrs : 23 mins
[TIMING] data: 15.4ms | fwd: 2413.2ms | loss: 4.9ms | bwd: 5365.0ms | clip: 0.0ms | optim: 23.6ms
step 94 | loss: 2.912911 | lr 1.7100e-04 | norm: 0.2818 | dt: 7820.82ms | tok/sec: 67037.5 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 15.2ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.6ms
step 95 | loss: 2.928957 | lr 1.7280e-04 | norm: 0.2237 | dt: 7821.58ms | tok/sec: 67030.9 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 13.5ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 96 | loss: 2.702905 | lr 1.7460e-04 | norm: 0.2173 | dt: 7820.23ms | tok/sec: 67042.5 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 12.3ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5359.6ms | clip: 0.0ms | optim: 23.6ms
step 97 | loss: 2.760113 | lr 1.7640e-04 | norm: 0.2141 | dt: 7818.07ms | tok/sec: 67061.0 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 14.7ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5360.0ms | clip: 0.0ms | optim: 23.6ms
step 98 | loss: 2.930555 | lr 1.7820e-04 | norm: 0.2340 | dt: 7819.73ms | tok/sec: 67046.8 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 14.5ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.6ms
step 99 | loss: 2.797276 | lr 1.8000e-04 | norm: 0.2102 | dt: 7824.59ms | tok/sec: 67005.2 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 13.1ms | fwd: 2416.3ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_100.ckpt (safely)
step 100 | loss: 2.580256 | lr 1.8000e-04 | norm: 0.2992 | dt: 8081.97ms | tok/sec: 64871.3 | mem: 72099MB | Time Left: 03 hrs : 28 mins
[TIMING] data: 10.9ms | fwd: 2686.6ms | loss: 4.9ms | bwd: 5344.6ms | clip: 0.0ms | optim: 23.4ms
step 101 | loss: 2.894757 | lr 1.8000e-04 | norm: 0.2233 | dt: 7794.10ms | tok/sec: 67267.3 | mem: 72099MB | Time Left: 03 hrs : 21 mins
[TIMING] data: 12.7ms | fwd: 2402.0ms | loss: 4.9ms | bwd: 5339.5ms | clip: 0.0ms | optim: 23.5ms
step 102 | loss: 3.132063 | lr 1.8000e-04 | norm: 0.2206 | dt: 7804.11ms | tok/sec: 67181.0 | mem: 72099MB | Time Left: 03 hrs : 21 mins
[TIMING] data: 14.6ms | fwd: 2399.5ms | loss: 4.9ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.4ms
step 103 | loss: 3.011836 | lr 1.8000e-04 | norm: 0.2088 | dt: 7821.06ms | tok/sec: 67035.4 | mem: 72099MB | Time Left: 03 hrs : 21 mins
[TIMING] data: 37.0ms | fwd: 2397.5ms | loss: 4.9ms | bwd: 5345.3ms | clip: 0.0ms | optim: 23.9ms
step 104 | loss: 2.945472 | lr 1.8000e-04 | norm: 0.2241 | dt: 7854.76ms | tok/sec: 66747.8 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 64.3ms | fwd: 2400.6ms | loss: 4.9ms | bwd: 5348.8ms | clip: 0.0ms | optim: 23.5ms
step 105 | loss: 3.009385 | lr 1.8000e-04 | norm: 0.2216 | dt: 7873.10ms | tok/sec: 66592.3 | mem: 72099MB | Time Left: 03 hrs : 22 mins
[TIMING] data: 54.3ms | fwd: 2429.8ms | loss: 4.9ms | bwd: 5348.9ms | clip: 0.0ms | optim: 23.5ms
step 106 | loss: 2.525123 | lr 1.8000e-04 | norm: 0.3052 | dt: 7811.14ms | tok/sec: 67120.5 | mem: 72099MB | Time Left: 03 hrs : 21 mins
[TIMING] data: 14.0ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.6ms
step 107 | loss: 2.686652 | lr 1.8000e-04 | norm: 0.2707 | dt: 7824.80ms | tok/sec: 67003.4 | mem: 72099MB | Time Left: 03 hrs : 21 mins
[TIMING] data: 13.7ms | fwd: 2413.1ms | loss: 4.9ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.5ms
step 108 | loss: 2.806834 | lr 1.8000e-04 | norm: 0.2358 | dt: 7811.96ms | tok/sec: 67113.5 | mem: 72099MB | Time Left: 03 hrs : 20 mins
[TIMING] data: 12.0ms | fwd: 2403.0ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.5ms
step 109 | loss: 2.859240 | lr 1.8000e-04 | norm: 0.1975 | dt: 7822.59ms | tok/sec: 67022.3 | mem: 72099MB | Time Left: 03 hrs : 21 mins
[TIMING] data: 12.6ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5363.1ms | clip: 0.0ms | optim: 23.5ms
step 110 | loss: 2.735763 | lr 1.8000e-04 | norm: 0.2022 | dt: 7814.22ms | tok/sec: 67094.1 | mem: 72099MB | Time Left: 03 hrs : 20 mins
[TIMING] data: 14.9ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.5ms
step 111 | loss: 2.884863 | lr 1.8000e-04 | norm: 0.2539 | dt: 7811.27ms | tok/sec: 67119.4 | mem: 72099MB | Time Left: 03 hrs : 20 mins
[TIMING] data: 14.1ms | fwd: 2403.4ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 112 | loss: 2.935062 | lr 1.8000e-04 | norm: 0.4899 | dt: 7820.87ms | tok/sec: 67037.1 | mem: 72099MB | Time Left: 03 hrs : 20 mins
[TIMING] data: 14.3ms | fwd: 2409.9ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.6ms
step 113 | loss: 2.908048 | lr 1.8000e-04 | norm: 0.2444 | dt: 7822.00ms | tok/sec: 67027.4 | mem: 72099MB | Time Left: 03 hrs : 20 mins
[TIMING] data: 13.6ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5361.7ms | clip: 0.0ms | optim: 23.6ms
step 114 | loss: 2.852918 | lr 1.8000e-04 | norm: 0.2624 | dt: 7809.35ms | tok/sec: 67135.9 | mem: 72099MB | Time Left: 03 hrs : 20 mins
[TIMING] data: 14.0ms | fwd: 2400.7ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 115 | loss: 2.941428 | lr 1.8000e-04 | norm: 0.3759 | dt: 7816.35ms | tok/sec: 67075.8 | mem: 72099MB | Time Left: 03 hrs : 20 mins
[TIMING] data: 14.6ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
step 116 | loss: 2.978065 | lr 1.8000e-04 | norm: 0.2886 | dt: 7816.96ms | tok/sec: 67070.5 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 13.6ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.5ms
step 117 | loss: 3.038124 | lr 1.8000e-04 | norm: 0.1880 | dt: 7821.54ms | tok/sec: 67031.3 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 13.1ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.6ms
step 118 | loss: 3.232168 | lr 1.8000e-04 | norm: 0.2377 | dt: 7813.23ms | tok/sec: 67102.6 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 15.2ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 119 | loss: 2.603676 | lr 1.8000e-04 | norm: 0.2613 | dt: 7817.02ms | tok/sec: 67070.1 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 13.5ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 120 | loss: 3.109175 | lr 1.8000e-04 | norm: 0.3523 | dt: 7815.69ms | tok/sec: 67081.5 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 11.5ms | fwd: 2402.7ms | loss: 4.9ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.4ms
step 121 | loss: 2.958578 | lr 1.8000e-04 | norm: 0.2539 | dt: 7819.06ms | tok/sec: 67052.5 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 13.5ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 122 | loss: 3.120374 | lr 1.8000e-04 | norm: 0.3991 | dt: 7820.70ms | tok/sec: 67038.5 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 15.6ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5360.4ms | clip: 0.0ms | optim: 23.6ms
step 123 | loss: 2.751444 | lr 1.8000e-04 | norm: 0.3458 | dt: 7831.75ms | tok/sec: 66943.9 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 16.5ms | fwd: 2413.1ms | loss: 4.9ms | bwd: 5362.7ms | clip: 0.0ms | optim: 23.5ms
step 124 | loss: 2.773659 | lr 1.8000e-04 | norm: 0.2471 | dt: 7816.78ms | tok/sec: 67072.1 | mem: 72099MB | Time Left: 03 hrs : 18 mins
[TIMING] data: 13.1ms | fwd: 2401.6ms | loss: 4.9ms | bwd: 5361.7ms | clip: 0.0ms | optim: 23.6ms
step 125 | loss: 2.736323 | lr 1.8000e-04 | norm: 0.2678 | dt: 7821.15ms | tok/sec: 67034.6 | mem: 72099MB | Time Left: 03 hrs : 18 mins
[TIMING] data: 16.9ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.5ms
step 126 | loss: 2.943342 | lr 1.8000e-04 | norm: 0.2510 | dt: 7816.05ms | tok/sec: 67078.3 | mem: 72099MB | Time Left: 03 hrs : 18 mins
[TIMING] data: 12.6ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.6ms
step 127 | loss: 2.950623 | lr 1.8000e-04 | norm: 0.2738 | dt: 7817.39ms | tok/sec: 67066.9 | mem: 72099MB | Time Left: 03 hrs : 18 mins
[TIMING] data: 14.7ms | fwd: 2410.0ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 128 | loss: 2.988625 | lr 1.8000e-04 | norm: 0.2203 | dt: 7818.98ms | tok/sec: 67053.3 | mem: 72099MB | Time Left: 03 hrs : 18 mins
[TIMING] data: 11.9ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.5ms
step 129 | loss: 2.723875 | lr 1.8000e-04 | norm: 0.2927 | dt: 7818.19ms | tok/sec: 67060.1 | mem: 72099MB | Time Left: 03 hrs : 18 mins
[TIMING] data: 14.5ms | fwd: 2410.8ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.6ms
step 130 | loss: 2.975360 | lr 1.8000e-04 | norm: 0.3926 | dt: 7885.35ms | tok/sec: 66488.9 | mem: 72099MB | Time Left: 03 hrs : 19 mins
[TIMING] data: 11.9ms | fwd: 2478.7ms | loss: 4.9ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.5ms
step 131 | loss: 2.819370 | lr 1.8000e-04 | norm: 0.2733 | dt: 7821.73ms | tok/sec: 67029.7 | mem: 72099MB | Time Left: 03 hrs : 18 mins
[TIMING] data: 11.9ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5361.0ms | clip: 0.0ms | optim: 23.5ms
step 132 | loss: 2.620174 | lr 1.8000e-04 | norm: 0.2238 | dt: 7817.78ms | tok/sec: 67063.6 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 13.2ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.5ms
step 133 | loss: 3.125543 | lr 1.8000e-04 | norm: 0.2223 | dt: 7817.42ms | tok/sec: 67066.6 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 12.1ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.5ms
step 134 | loss: 2.775470 | lr 1.8000e-04 | norm: 0.2758 | dt: 7820.40ms | tok/sec: 67041.1 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 14.1ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.6ms
step 135 | loss: 2.582886 | lr 1.8000e-04 | norm: 0.2508 | dt: 7819.38ms | tok/sec: 67049.8 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 14.8ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 136 | loss: 2.818479 | lr 1.8000e-04 | norm: 0.2568 | dt: 7827.89ms | tok/sec: 66976.9 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 15.2ms | fwd: 2414.1ms | loss: 4.9ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.6ms
step 137 | loss: 3.124125 | lr 1.8000e-04 | norm: 0.3269 | dt: 7824.55ms | tok/sec: 67005.5 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 13.6ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5359.3ms | clip: 0.0ms | optim: 23.6ms
step 138 | loss: 2.697395 | lr 1.8000e-04 | norm: 0.2788 | dt: 7816.30ms | tok/sec: 67076.2 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 15.6ms | fwd: 2404.5ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 139 | loss: 2.962272 | lr 1.8000e-04 | norm: 0.2584 | dt: 7817.70ms | tok/sec: 67064.2 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 15.2ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.5ms
step 140 | loss: 2.891029 | lr 1.8000e-04 | norm: 0.2816 | dt: 7821.22ms | tok/sec: 67034.1 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 15.1ms | fwd: 2411.3ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 141 | loss: 2.933413 | lr 1.8000e-04 | norm: 0.2671 | dt: 7824.68ms | tok/sec: 67004.4 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 13.0ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5363.0ms | clip: 0.0ms | optim: 23.4ms
step 142 | loss: 3.099108 | lr 1.8000e-04 | norm: 0.2454 | dt: 7817.01ms | tok/sec: 67070.1 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 14.7ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 143 | loss: 2.902442 | lr 1.8000e-04 | norm: 0.3406 | dt: 7819.64ms | tok/sec: 67047.6 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 13.5ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 144 | loss: 2.760140 | lr 1.8000e-04 | norm: 0.2358 | dt: 7818.96ms | tok/sec: 67053.4 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 17.3ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 145 | loss: 3.059258 | lr 1.8000e-04 | norm: 0.2532 | dt: 7820.30ms | tok/sec: 67041.9 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 13.5ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5360.7ms | clip: 0.0ms | optim: 23.6ms
step 146 | loss: 2.654944 | lr 1.8000e-04 | norm: 0.2150 | dt: 7824.54ms | tok/sec: 67005.6 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 16.0ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5357.2ms | clip: 0.0ms | optim: 23.6ms
step 147 | loss: 2.817395 | lr 1.8000e-04 | norm: 0.3083 | dt: 7813.45ms | tok/sec: 67100.7 | mem: 72099MB | Time Left: 03 hrs : 15 mins
[TIMING] data: 14.9ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.6ms
step 148 | loss: 3.133484 | lr 1.8000e-04 | norm: 0.3147 | dt: 7832.48ms | tok/sec: 66937.7 | mem: 72099MB | Time Left: 03 hrs : 16 mins
[TIMING] data: 19.3ms | fwd: 2411.0ms | loss: 4.9ms | bwd: 5361.4ms | clip: 0.0ms | optim: 23.7ms
step 149 | loss: 2.737945 | lr 1.8000e-04 | norm: 0.3279 | dt: 7886.18ms | tok/sec: 66481.9 | mem: 72099MB | Time Left: 03 hrs : 17 mins
[TIMING] data: 13.9ms | fwd: 2474.8ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.5ms
step 150 | loss: 2.438618 | lr 1.8000e-04 | norm: 0.2780 | dt: 7811.20ms | tok/sec: 67120.0 | mem: 72099MB | Time Left: 03 hrs : 15 mins
[TIMING] data: 12.5ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 151 | loss: 2.981455 | lr 1.8000e-04 | norm: 0.3314 | dt: 7811.60ms | tok/sec: 67116.6 | mem: 72099MB | Time Left: 03 hrs : 15 mins
[TIMING] data: 13.0ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 152 | loss: 2.747618 | lr 1.8000e-04 | norm: 0.3029 | dt: 7823.66ms | tok/sec: 67013.1 | mem: 72099MB | Time Left: 03 hrs : 15 mins
[TIMING] data: 13.7ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5363.5ms | clip: 0.0ms | optim: 23.6ms
step 153 | loss: 2.609881 | lr 1.8000e-04 | norm: 0.7249 | dt: 7818.51ms | tok/sec: 67057.2 | mem: 72099MB | Time Left: 03 hrs : 15 mins
[TIMING] data: 14.0ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 154 | loss: 2.985833 | lr 1.8000e-04 | norm: 0.4567 | dt: 7820.90ms | tok/sec: 67036.8 | mem: 72099MB | Time Left: 03 hrs : 15 mins
[TIMING] data: 12.1ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.5ms
step 155 | loss: 2.489286 | lr 1.8000e-04 | norm: 0.2393 | dt: 7813.54ms | tok/sec: 67100.0 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 12.9ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.7ms
step 156 | loss: 3.099371 | lr 1.8000e-04 | norm: 0.3601 | dt: 7814.21ms | tok/sec: 67094.2 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 12.9ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.4ms
step 157 | loss: 2.646076 | lr 1.8000e-04 | norm: 0.2987 | dt: 7821.69ms | tok/sec: 67030.0 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 13.9ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.6ms
step 158 | loss: 2.866046 | lr 1.8000e-04 | norm: 0.2360 | dt: 7824.61ms | tok/sec: 67005.0 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 12.5ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 159 | loss: 2.867364 | lr 1.8000e-04 | norm: 0.2487 | dt: 7818.60ms | tok/sec: 67056.5 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 14.9ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 160 | loss: 2.798890 | lr 1.8000e-04 | norm: 0.2647 | dt: 7833.04ms | tok/sec: 66932.9 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 14.3ms | fwd: 2416.9ms | loss: 4.9ms | bwd: 5361.2ms | clip: 0.0ms | optim: 23.8ms
step 161 | loss: 3.004354 | lr 1.8000e-04 | norm: 0.2586 | dt: 7822.19ms | tok/sec: 67025.7 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 14.1ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 162 | loss: 3.052295 | lr 1.8000e-04 | norm: 0.2444 | dt: 7828.30ms | tok/sec: 66973.4 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 14.8ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5360.4ms | clip: 0.0ms | optim: 23.5ms
step 163 | loss: 3.077438 | lr 1.8000e-04 | norm: 0.2926 | dt: 7826.22ms | tok/sec: 66991.2 | mem: 72099MB | Time Left: 03 hrs : 14 mins
[TIMING] data: 14.5ms | fwd: 2415.0ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 164 | loss: 2.944351 | lr 1.8000e-04 | norm: 0.2758 | dt: 7885.35ms | tok/sec: 66488.9 | mem: 72099MB | Time Left: 03 hrs : 15 mins
[TIMING] data: 15.6ms | fwd: 2472.3ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 165 | loss: 2.305563 | lr 1.8000e-04 | norm: 0.3156 | dt: 7818.83ms | tok/sec: 67054.5 | mem: 72099MB | Time Left: 03 hrs : 13 mins
[TIMING] data: 13.3ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 166 | loss: 2.745550 | lr 1.8000e-04 | norm: 0.6386 | dt: 7829.93ms | tok/sec: 66959.5 | mem: 72099MB | Time Left: 03 hrs : 13 mins
[TIMING] data: 13.6ms | fwd: 2414.9ms | loss: 4.9ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.5ms
step 167 | loss: 2.889800 | lr 1.8000e-04 | norm: 0.2562 | dt: 7820.36ms | tok/sec: 67041.5 | mem: 72099MB | Time Left: 03 hrs : 13 mins
[TIMING] data: 13.3ms | fwd: 2402.8ms | loss: 4.9ms | bwd: 5363.2ms | clip: 0.0ms | optim: 23.6ms
step 168 | loss: 3.081470 | lr 1.8000e-04 | norm: 0.4171 | dt: 7821.54ms | tok/sec: 67031.3 | mem: 72099MB | Time Left: 03 hrs : 13 mins
[TIMING] data: 13.2ms | fwd: 2412.9ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.7ms
step 169 | loss: 2.838651 | lr 1.8000e-04 | norm: 0.2418 | dt: 7815.42ms | tok/sec: 67083.8 | mem: 72099MB | Time Left: 03 hrs : 13 mins
[TIMING] data: 13.1ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.6ms
step 170 | loss: 2.991177 | lr 1.8000e-04 | norm: 0.2986 | dt: 7819.77ms | tok/sec: 67046.5 | mem: 72099MB | Time Left: 03 hrs : 13 mins
[TIMING] data: 13.4ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.7ms
step 171 | loss: 2.567647 | lr 1.8000e-04 | norm: 0.2443 | dt: 7815.48ms | tok/sec: 67083.3 | mem: 72099MB | Time Left: 03 hrs : 12 mins
[TIMING] data: 11.0ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.5ms
step 172 | loss: 2.813091 | lr 1.8000e-04 | norm: 0.2348 | dt: 7818.93ms | tok/sec: 67053.7 | mem: 72099MB | Time Left: 03 hrs : 12 mins
[TIMING] data: 15.3ms | fwd: 2409.9ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.7ms
step 173 | loss: 3.066455 | lr 1.8000e-04 | norm: 0.2353 | dt: 7821.26ms | tok/sec: 67033.7 | mem: 72099MB | Time Left: 03 hrs : 12 mins
[TIMING] data: 14.6ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.5ms
step 174 | loss: 2.991852 | lr 1.8000e-04 | norm: 0.2339 | dt: 7813.36ms | tok/sec: 67101.5 | mem: 72099MB | Time Left: 03 hrs : 12 mins
[TIMING] data: 12.4ms | fwd: 2401.1ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.7ms
step 175 | loss: 2.690243 | lr 1.8000e-04 | norm: 0.3766 | dt: 7822.13ms | tok/sec: 67026.2 | mem: 72099MB | Time Left: 03 hrs : 12 mins
[TIMING] data: 17.3ms | fwd: 2402.6ms | loss: 4.9ms | bwd: 5361.3ms | clip: 0.0ms | optim: 23.6ms
step 176 | loss: 2.856326 | lr 1.8000e-04 | norm: 0.2619 | dt: 7821.58ms | tok/sec: 67030.9 | mem: 72099MB | Time Left: 03 hrs : 12 mins
[TIMING] data: 12.3ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5359.0ms | clip: 0.0ms | optim: 23.6ms
step 177 | loss: 2.995249 | lr 1.8000e-04 | norm: 0.2671 | dt: 7816.34ms | tok/sec: 67075.9 | mem: 72099MB | Time Left: 03 hrs : 12 mins
[TIMING] data: 13.2ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
step 178 | loss: 2.787471 | lr 1.8000e-04 | norm: 0.2098 | dt: 7819.09ms | tok/sec: 67052.3 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 14.0ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5357.3ms | clip: 0.0ms | optim: 23.5ms
step 179 | loss: 3.002866 | lr 1.8000e-04 | norm: 0.2057 | dt: 7812.72ms | tok/sec: 67106.9 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 14.0ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 180 | loss: 2.974228 | lr 1.8000e-04 | norm: 0.2266 | dt: 7831.01ms | tok/sec: 66950.2 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 15.6ms | fwd: 2412.7ms | loss: 4.9ms | bwd: 5362.1ms | clip: 0.0ms | optim: 23.7ms
step 181 | loss: 2.840197 | lr 1.8000e-04 | norm: 0.2156 | dt: 7815.24ms | tok/sec: 67085.3 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 14.0ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 182 | loss: 2.868779 | lr 1.8000e-04 | norm: 0.9237 | dt: 7819.81ms | tok/sec: 67046.1 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 12.5ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5360.5ms | clip: 0.0ms | optim: 23.7ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00001.bin
100000000
step 183 | loss: 2.817125 | lr 1.8000e-04 | norm: 0.2574 | dt: 7828.03ms | tok/sec: 66975.8 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 20.8ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.4ms
step 184 | loss: 2.853192 | lr 1.8000e-04 | norm: 0.2363 | dt: 7821.65ms | tok/sec: 67030.4 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 9.1ms | fwd: 2413.1ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.5ms
step 185 | loss: 2.827386 | lr 1.8000e-04 | norm: 0.2116 | dt: 7820.26ms | tok/sec: 67042.2 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 15.0ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5358.4ms | clip: 0.0ms | optim: 23.5ms
step 186 | loss: 2.730311 | lr 1.8000e-04 | norm: 0.3214 | dt: 7813.23ms | tok/sec: 67102.6 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 14.5ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.6ms
step 187 | loss: 3.121159 | lr 1.8000e-04 | norm: 0.3449 | dt: 7821.07ms | tok/sec: 67035.3 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 13.6ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 188 | loss: 3.003454 | lr 1.8000e-04 | norm: 0.2580 | dt: 7819.54ms | tok/sec: 67048.4 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 16.5ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.6ms
step 189 | loss: 2.931279 | lr 1.8000e-04 | norm: 0.2091 | dt: 7824.28ms | tok/sec: 67007.8 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 13.5ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.7ms
step 190 | loss: 2.913467 | lr 1.8000e-04 | norm: 0.2138 | dt: 7817.66ms | tok/sec: 67064.5 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 13.9ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.4ms
step 191 | loss: 2.780973 | lr 1.8000e-04 | norm: 0.3813 | dt: 7823.68ms | tok/sec: 67013.0 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 12.2ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5363.1ms | clip: 0.0ms | optim: 23.6ms
step 192 | loss: 3.150949 | lr 1.8000e-04 | norm: 0.2155 | dt: 7822.21ms | tok/sec: 67025.6 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 12.2ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5359.6ms | clip: 0.0ms | optim: 23.4ms
step 193 | loss: 2.975601 | lr 1.8000e-04 | norm: 0.1956 | dt: 7823.36ms | tok/sec: 67015.7 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 15.4ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5357.3ms | clip: 0.0ms | optim: 23.6ms
step 194 | loss: 2.539850 | lr 1.8000e-04 | norm: 0.2407 | dt: 7818.02ms | tok/sec: 67061.5 | mem: 72099MB | Time Left: 03 hrs : 09 mins
[TIMING] data: 13.4ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 195 | loss: 2.359057 | lr 1.8000e-04 | norm: 0.1962 | dt: 7818.94ms | tok/sec: 67053.6 | mem: 72099MB | Time Left: 03 hrs : 09 mins
[TIMING] data: 14.3ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.6ms
step 196 | loss: 2.884645 | lr 1.8000e-04 | norm: 0.2570 | dt: 7889.00ms | tok/sec: 66458.1 | mem: 72099MB | Time Left: 03 hrs : 11 mins
[TIMING] data: 13.5ms | fwd: 2482.8ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 197 | loss: 3.040720 | lr 1.8000e-04 | norm: 0.2651 | dt: 7821.36ms | tok/sec: 67032.8 | mem: 72099MB | Time Left: 03 hrs : 09 mins
[TIMING] data: 13.2ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 198 | loss: 2.820660 | lr 1.8000e-04 | norm: 0.1922 | dt: 7820.50ms | tok/sec: 67040.2 | mem: 72099MB | Time Left: 03 hrs : 09 mins
[TIMING] data: 15.0ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5362.0ms | clip: 0.0ms | optim: 23.6ms
step 199 | loss: 3.057528 | lr 1.8000e-04 | norm: 0.2407 | dt: 7827.32ms | tok/sec: 66981.8 | mem: 72099MB | Time Left: 03 hrs : 09 mins
[TIMING] data: 13.0ms | fwd: 2416.8ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.6ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_200.ckpt (safely)
step 200 | loss: 2.852405 | lr 1.8000e-04 | norm: 0.2468 | dt: 7877.08ms | tok/sec: 66558.7 | mem: 72099MB | Time Left: 03 hrs : 10 mins
[TIMING] data: 12.1ms | fwd: 2480.6ms | loss: 4.9ms | bwd: 5344.4ms | clip: 0.0ms | optim: 23.4ms
step 201 | loss: 2.778087 | lr 1.8000e-04 | norm: 0.1974 | dt: 7805.00ms | tok/sec: 67173.3 | mem: 72099MB | Time Left: 03 hrs : 08 mins
[TIMING] data: 16.2ms | fwd: 2401.8ms | loss: 4.9ms | bwd: 5345.1ms | clip: 0.0ms | optim: 23.5ms
step 202 | loss: 2.653492 | lr 1.8000e-04 | norm: 0.2771 | dt: 7803.59ms | tok/sec: 67185.5 | mem: 72099MB | Time Left: 03 hrs : 08 mins
[TIMING] data: 13.6ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5345.4ms | clip: 0.0ms | optim: 23.4ms
step 203 | loss: 2.604328 | lr 1.8000e-04 | norm: 0.2688 | dt: 7822.20ms | tok/sec: 67025.6 | mem: 72099MB | Time Left: 03 hrs : 08 mins
[TIMING] data: 30.5ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5346.0ms | clip: 0.0ms | optim: 23.5ms
step 204 | loss: 2.818771 | lr 1.8000e-04 | norm: 0.2622 | dt: 7848.62ms | tok/sec: 66800.0 | mem: 72099MB | Time Left: 03 hrs : 09 mins
[TIMING] data: 54.7ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5347.2ms | clip: 0.0ms | optim: 23.6ms
step 205 | loss: 2.610917 | lr 1.8000e-04 | norm: 0.2799 | dt: 7860.43ms | tok/sec: 66699.7 | mem: 72099MB | Time Left: 03 hrs : 09 mins
[TIMING] data: 76.2ms | fwd: 2392.8ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.6ms
step 206 | loss: 2.716873 | lr 1.8000e-04 | norm: 0.5407 | dt: 7824.11ms | tok/sec: 67009.3 | mem: 72099MB | Time Left: 03 hrs : 08 mins
[TIMING] data: 27.1ms | fwd: 2401.9ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.5ms
step 207 | loss: 2.999454 | lr 1.8000e-04 | norm: 0.2181 | dt: 7813.36ms | tok/sec: 67101.5 | mem: 72099MB | Time Left: 03 hrs : 08 mins
[TIMING] data: 14.4ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
step 208 | loss: 2.464639 | lr 1.8000e-04 | norm: 0.3310 | dt: 7814.77ms | tok/sec: 67089.4 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 13.4ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 209 | loss: 3.111967 | lr 1.8000e-04 | norm: 0.2096 | dt: 7818.46ms | tok/sec: 67057.7 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 13.4ms | fwd: 2405.9ms | loss: 4.9ms | bwd: 5357.9ms | clip: 0.0ms | optim: 23.8ms
step 210 | loss: 3.113807 | lr 1.8000e-04 | norm: 0.2312 | dt: 7810.29ms | tok/sec: 67127.9 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 15.6ms | fwd: 2401.2ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.6ms
step 211 | loss: 2.849459 | lr 1.8000e-04 | norm: 0.2214 | dt: 7815.05ms | tok/sec: 67086.9 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 11.9ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.6ms
step 212 | loss: 2.653818 | lr 1.8000e-04 | norm: 0.2382 | dt: 7818.99ms | tok/sec: 67053.2 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 14.3ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 213 | loss: 2.830781 | lr 1.8000e-04 | norm: 0.2167 | dt: 7814.30ms | tok/sec: 67093.4 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 17.1ms | fwd: 2408.5ms | loss: 4.9ms | bwd: 5348.7ms | clip: 0.0ms | optim: 23.6ms
step 214 | loss: 2.770122 | lr 1.8000e-04 | norm: 0.2449 | dt: 7817.24ms | tok/sec: 67068.2 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 15.0ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.6ms
step 215 | loss: 2.941538 | lr 1.8000e-04 | norm: 0.2423 | dt: 7811.75ms | tok/sec: 67115.3 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 12.7ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 216 | loss: 2.770646 | lr 1.8000e-04 | norm: 0.2600 | dt: 7816.75ms | tok/sec: 67072.4 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 14.4ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5358.4ms | clip: 0.0ms | optim: 23.7ms
step 217 | loss: 2.669334 | lr 1.8000e-04 | norm: 0.2368 | dt: 7830.99ms | tok/sec: 66950.4 | mem: 72099MB | Time Left: 03 hrs : 07 mins
[TIMING] data: 14.6ms | fwd: 2415.4ms | loss: 4.9ms | bwd: 5360.4ms | clip: 0.0ms | optim: 23.5ms
step 218 | loss: 2.508659 | lr 1.8000e-04 | norm: 0.2245 | dt: 7820.03ms | tok/sec: 67044.3 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 14.7ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.5ms
step 219 | loss: 2.915088 | lr 1.8000e-04 | norm: 0.2618 | dt: 7819.30ms | tok/sec: 67050.5 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 14.6ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.6ms
step 220 | loss: 2.898007 | lr 1.8000e-04 | norm: 0.2331 | dt: 7816.21ms | tok/sec: 67077.0 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 14.8ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.7ms
step 221 | loss: 3.196023 | lr 1.8000e-04 | norm: 0.2936 | dt: 7819.54ms | tok/sec: 67048.5 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 15.2ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5357.2ms | clip: 0.0ms | optim: 23.5ms
step 222 | loss: 2.813270 | lr 1.8000e-04 | norm: 0.2399 | dt: 7817.21ms | tok/sec: 67068.4 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 14.7ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.6ms
step 223 | loss: 2.921408 | lr 1.8000e-04 | norm: 0.2176 | dt: 7825.18ms | tok/sec: 67000.1 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 13.9ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.5ms
step 224 | loss: 2.991516 | lr 1.8000e-04 | norm: 0.2272 | dt: 7821.79ms | tok/sec: 67029.1 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 13.6ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.5ms
step 225 | loss: 2.910104 | lr 1.8000e-04 | norm: 0.2059 | dt: 7829.98ms | tok/sec: 66959.1 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 15.7ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5360.3ms | clip: 0.0ms | optim: 23.5ms
step 226 | loss: 3.031386 | lr 1.8000e-04 | norm: 0.2219 | dt: 7820.15ms | tok/sec: 67043.2 | mem: 72099MB | Time Left: 03 hrs : 05 mins
[TIMING] data: 13.6ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.6ms
step 227 | loss: 3.233400 | lr 1.8000e-04 | norm: 0.2681 | dt: 7813.24ms | tok/sec: 67102.5 | mem: 72099MB | Time Left: 03 hrs : 05 mins
[TIMING] data: 13.1ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.6ms
step 228 | loss: 2.702644 | lr 1.8000e-04 | norm: 0.2030 | dt: 7815.75ms | tok/sec: 67081.0 | mem: 72099MB | Time Left: 03 hrs : 05 mins
[TIMING] data: 14.9ms | fwd: 2402.9ms | loss: 5.0ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 229 | loss: 2.727783 | lr 1.8000e-04 | norm: 0.2884 | dt: 7823.21ms | tok/sec: 67017.0 | mem: 72099MB | Time Left: 03 hrs : 05 mins
[TIMING] data: 14.7ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5357.9ms | clip: 0.0ms | optim: 23.6ms
step 230 | loss: 2.345054 | lr 1.8000e-04 | norm: 0.2484 | dt: 7821.74ms | tok/sec: 67029.6 | mem: 72099MB | Time Left: 03 hrs : 05 mins
[TIMING] data: 14.1ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 231 | loss: 2.810749 | lr 1.8000e-04 | norm: 0.2257 | dt: 7813.70ms | tok/sec: 67098.6 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 14.9ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.4ms
step 232 | loss: 2.859225 | lr 1.8000e-04 | norm: 0.4375 | dt: 7817.09ms | tok/sec: 67069.4 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 14.6ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5350.0ms | clip: 0.0ms | optim: 23.5ms
step 233 | loss: 2.472826 | lr 1.8000e-04 | norm: 0.2564 | dt: 7816.68ms | tok/sec: 67073.0 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 15.0ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.7ms
step 234 | loss: 2.662909 | lr 1.8000e-04 | norm: 0.3058 | dt: 7830.49ms | tok/sec: 66954.7 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 13.5ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5365.1ms | clip: 0.0ms | optim: 23.5ms
step 235 | loss: 2.960697 | lr 1.8000e-04 | norm: 0.2001 | dt: 7817.44ms | tok/sec: 67066.5 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 14.2ms | fwd: 2403.6ms | loss: 4.9ms | bwd: 5359.7ms | clip: 0.0ms | optim: 23.5ms
step 236 | loss: 2.456114 | lr 1.8000e-04 | norm: 0.2827 | dt: 7827.26ms | tok/sec: 66982.3 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 13.2ms | fwd: 2416.7ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.6ms
step 237 | loss: 3.109556 | lr 1.8000e-04 | norm: 0.2129 | dt: 7816.68ms | tok/sec: 67073.0 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 13.5ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5358.7ms | clip: 0.0ms | optim: 23.5ms
step 238 | loss: 2.793779 | lr 1.8000e-04 | norm: 0.2556 | dt: 7817.45ms | tok/sec: 67066.4 | mem: 72099MB | Time Left: 03 hrs : 04 mins
[TIMING] data: 13.2ms | fwd: 2404.9ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.5ms
step 239 | loss: 3.139490 | lr 1.8000e-04 | norm: 0.3799 | dt: 7885.14ms | tok/sec: 66490.6 | mem: 72099MB | Time Left: 03 hrs : 05 mins
[TIMING] data: 14.3ms | fwd: 2470.9ms | loss: 4.9ms | bwd: 5360.1ms | clip: 0.0ms | optim: 23.5ms
step 240 | loss: 2.985081 | lr 1.8000e-04 | norm: 0.2161 | dt: 7819.79ms | tok/sec: 67046.3 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 17.4ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 241 | loss: 2.744594 | lr 1.8000e-04 | norm: 0.2440 | dt: 7825.59ms | tok/sec: 66996.6 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 14.3ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.5ms
step 242 | loss: 2.932421 | lr 1.8000e-04 | norm: 0.2320 | dt: 7822.54ms | tok/sec: 67022.8 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 18.6ms | fwd: 2406.8ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 243 | loss: 2.807653 | lr 1.8000e-04 | norm: 0.5670 | dt: 7823.58ms | tok/sec: 67013.8 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 13.3ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5362.7ms | clip: 0.0ms | optim: 23.6ms
step 244 | loss: 2.779371 | lr 1.8000e-04 | norm: 0.2511 | dt: 7821.85ms | tok/sec: 67028.7 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 13.6ms | fwd: 2415.3ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.6ms
step 245 | loss: 2.744847 | lr 1.8000e-04 | norm: 4.7220 | dt: 7826.11ms | tok/sec: 66992.2 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 14.3ms | fwd: 2414.7ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 246 | loss: 3.013911 | lr 1.8000e-04 | norm: 0.2422 | dt: 7819.78ms | tok/sec: 67046.4 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 13.2ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.5ms
step 247 | loss: 2.736719 | lr 1.8000e-04 | norm: 0.2522 | dt: 7813.55ms | tok/sec: 67099.9 | mem: 72099MB | Time Left: 03 hrs : 02 mins
[TIMING] data: 13.9ms | fwd: 2402.0ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.5ms
step 248 | loss: 2.772788 | lr 1.8000e-04 | norm: 0.2340 | dt: 7816.96ms | tok/sec: 67070.5 | mem: 72099MB | Time Left: 03 hrs : 02 mins
[TIMING] data: 12.7ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 249 | loss: 2.852266 | lr 1.8000e-04 | norm: 0.3332 | dt: 7816.83ms | tok/sec: 67071.7 | mem: 72099MB | Time Left: 03 hrs : 02 mins
[TIMING] data: 13.8ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.7ms
validation loss: 2.9689
step 250 | loss: 2.977678 | lr 1.8000e-04 | norm: 0.2566 | dt: 13802.71ms | tok/sec: 37984.4 | mem: 72099MB | Time Left: 05 hrs : 22 mins
[TIMING] data: 14.5ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 251 | loss: 2.722430 | lr 1.8000e-04 | norm: 0.2655 | dt: 7821.76ms | tok/sec: 67029.4 | mem: 72099MB | Time Left: 03 hrs : 02 mins
[TIMING] data: 13.1ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.6ms
step 252 | loss: 3.111939 | lr 1.8000e-04 | norm: 0.2957 | dt: 7812.70ms | tok/sec: 67107.2 | mem: 72099MB | Time Left: 03 hrs : 02 mins
[TIMING] data: 13.5ms | fwd: 2404.4ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 253 | loss: 2.967920 | lr 1.8000e-04 | norm: 0.2893 | dt: 7885.74ms | tok/sec: 66485.6 | mem: 72099MB | Time Left: 03 hrs : 03 mins
[TIMING] data: 14.1ms | fwd: 2471.6ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.5ms
step 254 | loss: 2.602114 | lr 1.8000e-04 | norm: 0.2433 | dt: 7828.55ms | tok/sec: 66971.3 | mem: 72099MB | Time Left: 03 hrs : 02 mins
[TIMING] data: 14.1ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5360.6ms | clip: 0.0ms | optim: 23.5ms
step 255 | loss: 2.776347 | lr 1.8000e-04 | norm: 0.2183 | dt: 7819.13ms | tok/sec: 67052.0 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 14.4ms | fwd: 2409.9ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 256 | loss: 3.031132 | lr 1.8000e-04 | norm: 0.2557 | dt: 7820.70ms | tok/sec: 67038.5 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 15.1ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 257 | loss: 2.643022 | lr 1.8000e-04 | norm: 0.2964 | dt: 7825.81ms | tok/sec: 66994.7 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 19.5ms | fwd: 2413.0ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.5ms
step 258 | loss: 2.824308 | lr 1.8000e-04 | norm: 0.4105 | dt: 7820.33ms | tok/sec: 67041.6 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 14.6ms | fwd: 2411.8ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.5ms
step 259 | loss: 2.675786 | lr 1.8000e-04 | norm: 0.2495 | dt: 7817.74ms | tok/sec: 67063.9 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 13.5ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5359.3ms | clip: 0.0ms | optim: 23.5ms
step 260 | loss: 2.686760 | lr 1.8000e-04 | norm: 0.2496 | dt: 7817.37ms | tok/sec: 67067.0 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 11.9ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.6ms
step 261 | loss: 2.625328 | lr 1.8000e-04 | norm: 0.3077 | dt: 7813.47ms | tok/sec: 67100.5 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 13.0ms | fwd: 2400.5ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 262 | loss: 2.925518 | lr 1.8000e-04 | norm: 0.3116 | dt: 7823.82ms | tok/sec: 67011.8 | mem: 72099MB | Time Left: 03 hrs : 01 mins
[TIMING] data: 14.4ms | fwd: 2412.3ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.7ms
step 263 | loss: 2.618061 | lr 1.8000e-04 | norm: 0.2780 | dt: 7821.67ms | tok/sec: 67030.2 | mem: 72099MB | Time Left: 03 hrs : 00 mins
[TIMING] data: 14.8ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5358.2ms | clip: 0.0ms | optim: 23.5ms
step 264 | loss: 3.137913 | lr 1.8000e-04 | norm: 0.3551 | dt: 7824.91ms | tok/sec: 67002.4 | mem: 72099MB | Time Left: 03 hrs : 00 mins
[TIMING] data: 13.9ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.5ms
step 265 | loss: 2.886762 | lr 1.8000e-04 | norm: 0.2474 | dt: 7820.07ms | tok/sec: 67043.9 | mem: 72099MB | Time Left: 03 hrs : 00 mins
[TIMING] data: 14.2ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5359.5ms | clip: 0.0ms | optim: 23.5ms
step 266 | loss: 2.906691 | lr 1.8000e-04 | norm: 0.2188 | dt: 7831.91ms | tok/sec: 66942.5 | mem: 72099MB | Time Left: 03 hrs : 00 mins
[TIMING] data: 12.5ms | fwd: 2415.2ms | loss: 4.9ms | bwd: 5361.4ms | clip: 0.0ms | optim: 24.2ms
step 267 | loss: 2.841478 | lr 1.8000e-04 | norm: 0.2929 | dt: 7829.93ms | tok/sec: 66959.5 | mem: 72099MB | Time Left: 03 hrs : 00 mins
[TIMING] data: 12.0ms | fwd: 2412.1ms | loss: 4.9ms | bwd: 5364.7ms | clip: 0.0ms | optim: 23.5ms
step 268 | loss: 2.411623 | lr 1.8000e-04 | norm: 0.2903 | dt: 7816.06ms | tok/sec: 67078.3 | mem: 72099MB | Time Left: 03 hrs : 00 mins
[TIMING] data: 15.0ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 269 | loss: 2.715798 | lr 1.8000e-04 | norm: 0.2840 | dt: 7815.11ms | tok/sec: 67086.5 | mem: 72099MB | Time Left: 03 hrs : 00 mins
[TIMING] data: 14.1ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.7ms
step 270 | loss: 2.994917 | lr 1.8000e-04 | norm: 0.2661 | dt: 7819.85ms | tok/sec: 67045.8 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 14.7ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.5ms
step 271 | loss: 2.650205 | lr 1.8000e-04 | norm: 0.2546 | dt: 7812.71ms | tok/sec: 67107.1 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 13.8ms | fwd: 2400.5ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.6ms
step 272 | loss: 3.206560 | lr 1.8000e-04 | norm: 0.2639 | dt: 7819.42ms | tok/sec: 67049.5 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 13.8ms | fwd: 2410.8ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
step 273 | loss: 2.625773 | lr 1.8000e-04 | norm: 0.2446 | dt: 7814.48ms | tok/sec: 67091.8 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 16.5ms | fwd: 2400.9ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.8ms
step 274 | loss: 3.008951 | lr 1.8000e-04 | norm: 0.2538 | dt: 7816.69ms | tok/sec: 67072.9 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 13.6ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.5ms
step 275 | loss: 2.920411 | lr 1.8000e-04 | norm: 0.3358 | dt: 7812.25ms | tok/sec: 67111.0 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 10.5ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 276 | loss: 2.715621 | lr 1.8000e-04 | norm: 0.3054 | dt: 7812.75ms | tok/sec: 67106.7 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 16.7ms | fwd: 2402.7ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 277 | loss: 3.057517 | lr 1.8000e-04 | norm: 0.2655 | dt: 7817.68ms | tok/sec: 67064.4 | mem: 72099MB | Time Left: 02 hrs : 59 mins
[TIMING] data: 13.0ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5358.7ms | clip: 0.0ms | optim: 23.6ms
step 278 | loss: 3.016796 | lr 1.8000e-04 | norm: 0.2865 | dt: 7814.58ms | tok/sec: 67091.0 | mem: 72099MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 12.9ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.6ms
step 279 | loss: 3.056976 | lr 1.8000e-04 | norm: 0.2568 | dt: 7813.97ms | tok/sec: 67096.2 | mem: 72099MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 15.4ms | fwd: 2403.2ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.6ms
step 280 | loss: 2.739704 | lr 1.8000e-04 | norm: 0.3546 | dt: 7814.09ms | tok/sec: 67095.2 | mem: 72099MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 14.9ms | fwd: 2400.8ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.6ms
step 281 | loss: 2.724315 | lr 1.8000e-04 | norm: 0.2108 | dt: 7816.22ms | tok/sec: 67076.9 | mem: 72099MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 12.7ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 282 | loss: 2.968993 | lr 1.8000e-04 | norm: 0.1941 | dt: 7815.82ms | tok/sec: 67080.3 | mem: 72099MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 13.3ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.6ms
step 283 | loss: 2.893228 | lr 1.8000e-04 | norm: 0.2427 | dt: 7807.95ms | tok/sec: 67148.0 | mem: 72099MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 12.8ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.5ms
step 284 | loss: 2.530917 | lr 1.8000e-04 | norm: 0.2629 | dt: 7813.25ms | tok/sec: 67102.4 | mem: 72099MB | Time Left: 02 hrs : 58 mins
[TIMING] data: 14.2ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.7ms
step 285 | loss: 2.643353 | lr 1.8000e-04 | norm: 0.2028 | dt: 7817.39ms | tok/sec: 67066.9 | mem: 72099MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 15.4ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.6ms
step 286 | loss: 2.593655 | lr 1.8000e-04 | norm: 0.2403 | dt: 7809.61ms | tok/sec: 67133.7 | mem: 72099MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 14.8ms | fwd: 2398.6ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.4ms
step 287 | loss: 2.819513 | lr 1.8000e-04 | norm: 0.2337 | dt: 7823.04ms | tok/sec: 67018.4 | mem: 72099MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 14.3ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.6ms
step 288 | loss: 2.987386 | lr 1.8000e-04 | norm: 0.2604 | dt: 7820.18ms | tok/sec: 67043.0 | mem: 72099MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 13.4ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.6ms
step 289 | loss: 2.700021 | lr 1.8000e-04 | norm: 0.2300 | dt: 7820.70ms | tok/sec: 67038.5 | mem: 72099MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 15.6ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.6ms
step 290 | loss: 2.663064 | lr 1.8000e-04 | norm: 0.3370 | dt: 7817.84ms | tok/sec: 67063.1 | mem: 72099MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 15.7ms | fwd: 2411.1ms | loss: 4.9ms | bwd: 5350.3ms | clip: 0.0ms | optim: 23.5ms
step 291 | loss: 2.810455 | lr 1.8000e-04 | norm: 0.3415 | dt: 7812.79ms | tok/sec: 67106.3 | mem: 72099MB | Time Left: 02 hrs : 57 mins
[TIMING] data: 13.8ms | fwd: 2404.4ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.5ms
step 292 | loss: 3.034776 | lr 1.8000e-04 | norm: 0.2526 | dt: 7811.07ms | tok/sec: 67121.1 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 16.7ms | fwd: 2400.4ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 293 | loss: 3.021693 | lr 1.8000e-04 | norm: 0.2778 | dt: 7819.08ms | tok/sec: 67052.4 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 15.6ms | fwd: 2412.5ms | loss: 4.9ms | bwd: 5349.2ms | clip: 0.0ms | optim: 23.5ms
step 294 | loss: 2.702269 | lr 1.8000e-04 | norm: 0.5531 | dt: 7822.01ms | tok/sec: 67027.2 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 16.5ms | fwd: 2411.3ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.5ms
step 295 | loss: 2.741399 | lr 1.8000e-04 | norm: 0.3093 | dt: 7817.66ms | tok/sec: 67064.6 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 14.6ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.5ms
step 296 | loss: 2.461945 | lr 1.8000e-04 | norm: 0.3025 | dt: 7818.62ms | tok/sec: 67056.3 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 15.5ms | fwd: 2410.0ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 297 | loss: 2.787208 | lr 1.8000e-04 | norm: 0.3007 | dt: 7815.51ms | tok/sec: 67083.0 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 14.7ms | fwd: 2405.2ms | loss: 5.0ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.6ms
step 298 | loss: 2.859488 | lr 1.8000e-04 | norm: 0.3311 | dt: 7822.54ms | tok/sec: 67022.8 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 16.4ms | fwd: 2410.0ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.5ms
step 299 | loss: 2.911857 | lr 1.8000e-04 | norm: 0.2059 | dt: 7809.95ms | tok/sec: 67130.7 | mem: 72099MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 17.6ms | fwd: 2403.2ms | loss: 4.9ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.6ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_300.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_100.ckpt
step 300 | loss: 2.636260 | lr 1.8000e-04 | norm: 0.2577 | dt: 8268.00ms | tok/sec: 63411.7 | mem: 72099MB | Time Left: 03 hrs : 06 mins
[TIMING] data: 13.6ms | fwd: 2874.6ms | loss: 4.8ms | bwd: 5342.2ms | clip: 0.0ms | optim: 23.4ms
step 301 | loss: 2.818360 | lr 1.8000e-04 | norm: 0.4637 | dt: 7800.93ms | tok/sec: 67208.4 | mem: 72099MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 16.9ms | fwd: 2401.0ms | loss: 4.8ms | bwd: 5345.6ms | clip: 0.0ms | optim: 23.4ms
step 302 | loss: 2.831501 | lr 1.8000e-04 | norm: 0.2241 | dt: 7810.92ms | tok/sec: 67122.4 | mem: 72099MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 22.7ms | fwd: 2399.7ms | loss: 4.8ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.5ms
step 303 | loss: 3.020349 | lr 1.8000e-04 | norm: 0.2215 | dt: 7854.12ms | tok/sec: 66753.2 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 75.4ms | fwd: 2392.4ms | loss: 4.7ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.5ms
step 304 | loss: 2.472758 | lr 1.8000e-04 | norm: 0.2326 | dt: 7879.78ms | tok/sec: 66535.8 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 95.2ms | fwd: 2399.4ms | loss: 4.8ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.5ms
step 305 | loss: 2.794170 | lr 1.8000e-04 | norm: 0.3134 | dt: 7851.47ms | tok/sec: 66775.8 | mem: 72099MB | Time Left: 02 hrs : 56 mins
[TIMING] data: 77.1ms | fwd: 2392.5ms | loss: 4.7ms | bwd: 5342.8ms | clip: 0.0ms | optim: 23.5ms
step 306 | loss: 3.047400 | lr 1.8000e-04 | norm: 0.2161 | dt: 7811.40ms | tok/sec: 67118.4 | mem: 72099MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 13.0ms | fwd: 2407.5ms | loss: 4.8ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.5ms
step 307 | loss: 2.856745 | lr 1.8000e-04 | norm: 0.2648 | dt: 7814.40ms | tok/sec: 67092.6 | mem: 72099MB | Time Left: 02 hrs : 55 mins
[TIMING] data: 14.3ms | fwd: 2411.8ms | loss: 4.8ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
step 308 | loss: 2.869136 | lr 1.8000e-04 | norm: 0.3010 | dt: 7806.51ms | tok/sec: 67160.4 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 14.2ms | fwd: 2403.2ms | loss: 4.8ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 309 | loss: 3.001639 | lr 1.8000e-04 | norm: 0.2461 | dt: 7822.74ms | tok/sec: 67021.0 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 13.3ms | fwd: 2412.7ms | loss: 4.7ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.5ms
step 310 | loss: 2.960176 | lr 1.8000e-04 | norm: 0.2154 | dt: 7810.11ms | tok/sec: 67129.4 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 16.8ms | fwd: 2403.0ms | loss: 4.8ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.5ms
step 311 | loss: 3.051016 | lr 1.8000e-04 | norm: 0.2072 | dt: 7811.74ms | tok/sec: 67115.4 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 14.5ms | fwd: 2406.3ms | loss: 4.7ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 312 | loss: 2.743954 | lr 1.8000e-04 | norm: 0.2962 | dt: 7810.53ms | tok/sec: 67125.8 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 15.5ms | fwd: 2404.7ms | loss: 4.7ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.5ms
step 313 | loss: 3.030007 | lr 1.8000e-04 | norm: 0.2895 | dt: 7819.62ms | tok/sec: 67047.8 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 16.0ms | fwd: 2407.6ms | loss: 4.8ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.5ms
step 314 | loss: 3.010666 | lr 1.8000e-04 | norm: 0.2183 | dt: 7805.45ms | tok/sec: 67169.5 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 13.5ms | fwd: 2402.7ms | loss: 4.8ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.5ms
step 315 | loss: 2.859729 | lr 1.8000e-04 | norm: 0.2582 | dt: 7814.38ms | tok/sec: 67092.8 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 15.2ms | fwd: 2403.9ms | loss: 4.7ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.5ms
step 316 | loss: 2.771516 | lr 1.8000e-04 | norm: 0.2416 | dt: 7812.16ms | tok/sec: 67111.8 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 14.3ms | fwd: 2404.4ms | loss: 4.7ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.4ms
step 317 | loss: 3.170321 | lr 1.8000e-04 | norm: 0.3165 | dt: 7818.14ms | tok/sec: 67060.4 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 11.7ms | fwd: 2410.3ms | loss: 4.8ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.6ms
step 318 | loss: 2.828101 | lr 1.8000e-04 | norm: 0.3305 | dt: 7812.55ms | tok/sec: 67108.5 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 14.8ms | fwd: 2406.5ms | loss: 4.8ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.5ms
step 319 | loss: 3.067851 | lr 1.8000e-04 | norm: 0.2365 | dt: 7812.78ms | tok/sec: 67106.5 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 15.3ms | fwd: 2403.2ms | loss: 4.8ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 320 | loss: 2.879760 | lr 1.8000e-04 | norm: 0.2696 | dt: 7815.08ms | tok/sec: 67086.7 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 16.9ms | fwd: 2406.1ms | loss: 4.7ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 321 | loss: 2.794193 | lr 1.8000e-04 | norm: 0.2437 | dt: 7879.31ms | tok/sec: 66539.8 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 15.6ms | fwd: 2476.5ms | loss: 4.7ms | bwd: 5349.0ms | clip: 0.0ms | optim: 23.5ms
step 322 | loss: 3.065828 | lr 1.8000e-04 | norm: 0.2309 | dt: 7826.03ms | tok/sec: 66992.8 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 14.2ms | fwd: 2418.7ms | loss: 4.7ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 323 | loss: 3.021882 | lr 1.8000e-04 | norm: 0.2639 | dt: 7830.04ms | tok/sec: 66958.5 | mem: 72099MB | Time Left: 02 hrs : 53 mins
[TIMING] data: 12.8ms | fwd: 2421.9ms | loss: 4.7ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.5ms
step 324 | loss: 2.822289 | lr 1.8000e-04 | norm: 0.2216 | dt: 7817.10ms | tok/sec: 67069.4 | mem: 72099MB | Time Left: 02 hrs : 52 mins
[TIMING] data: 18.2ms | fwd: 2407.7ms | loss: 4.7ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.4ms
step 325 | loss: 2.931117 | lr 1.8000e-04 | norm: 0.2896 | dt: 7887.91ms | tok/sec: 66467.3 | mem: 72099MB | Time Left: 02 hrs : 54 mins
[TIMING] data: 15.7ms | fwd: 2481.5ms | loss: 4.7ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.7ms
step 326 | loss: 3.041464 | lr 1.8000e-04 | norm: 0.2454 | dt: 7812.08ms | tok/sec: 67112.5 | mem: 72099MB | Time Left: 02 hrs : 52 mins
[TIMING] data: 15.3ms | fwd: 2408.5ms | loss: 4.7ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.5ms
step 327 | loss: 2.549238 | lr 1.8000e-04 | norm: 0.2670 | dt: 7817.74ms | tok/sec: 67063.9 | mem: 72099MB | Time Left: 02 hrs : 52 mins
[TIMING] data: 14.4ms | fwd: 2407.6ms | loss: 4.8ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.5ms
step 328 | loss: 2.998165 | lr 1.8000e-04 | norm: 0.6243 | dt: 7817.52ms | tok/sec: 67065.7 | mem: 72099MB | Time Left: 02 hrs : 52 mins
[TIMING] data: 14.2ms | fwd: 2412.4ms | loss: 4.8ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 329 | loss: 2.808309 | lr 1.8000e-04 | norm: 0.2951 | dt: 7813.76ms | tok/sec: 67098.0 | mem: 72099MB | Time Left: 02 hrs : 52 mins
[TIMING] data: 14.6ms | fwd: 2403.8ms | loss: 4.7ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 330 | loss: 3.025751 | lr 1.8000e-04 | norm: 0.2562 | dt: 7822.48ms | tok/sec: 67023.3 | mem: 72099MB | Time Left: 02 hrs : 52 mins
[TIMING] data: 12.2ms | fwd: 2412.3ms | loss: 4.8ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.5ms
step 331 | loss: 2.765308 | lr 1.8000e-04 | norm: 0.2777 | dt: 7811.88ms | tok/sec: 67114.2 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 14.7ms | fwd: 2407.0ms | loss: 4.8ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.6ms
step 332 | loss: 2.862925 | lr 1.8000e-04 | norm: 0.2764 | dt: 7819.85ms | tok/sec: 67045.8 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 14.9ms | fwd: 2410.2ms | loss: 4.7ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 333 | loss: 2.995287 | lr 1.8000e-04 | norm: 0.2501 | dt: 7817.02ms | tok/sec: 67070.1 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 15.2ms | fwd: 2407.3ms | loss: 4.7ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 334 | loss: 2.931133 | lr 1.8000e-04 | norm: 0.2595 | dt: 7810.17ms | tok/sec: 67128.9 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 13.4ms | fwd: 2407.5ms | loss: 4.7ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.5ms
step 335 | loss: 2.809650 | lr 1.8000e-04 | norm: 0.2687 | dt: 7822.44ms | tok/sec: 67023.6 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 15.4ms | fwd: 2413.2ms | loss: 4.7ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 336 | loss: 2.842376 | lr 1.8000e-04 | norm: 0.2142 | dt: 7828.80ms | tok/sec: 66969.1 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 14.3ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.6ms
step 337 | loss: 2.916311 | lr 1.8000e-04 | norm: 0.2346 | dt: 7812.97ms | tok/sec: 67104.9 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 13.4ms | fwd: 2406.4ms | loss: 4.7ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.6ms
step 338 | loss: 2.667136 | lr 1.8000e-04 | norm: 0.3124 | dt: 7811.26ms | tok/sec: 67119.5 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 15.7ms | fwd: 2408.2ms | loss: 4.7ms | bwd: 5348.4ms | clip: 0.0ms | optim: 23.5ms
step 339 | loss: 2.502676 | lr 1.8000e-04 | norm: 0.2261 | dt: 7817.26ms | tok/sec: 67068.0 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 15.5ms | fwd: 2406.9ms | loss: 4.7ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 340 | loss: 2.873840 | lr 1.8000e-04 | norm: 0.2341 | dt: 7816.01ms | tok/sec: 67078.7 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 14.1ms | fwd: 2411.2ms | loss: 4.7ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.5ms
step 341 | loss: 2.970060 | lr 1.8000e-04 | norm: 0.2618 | dt: 7825.02ms | tok/sec: 67001.5 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 12.1ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5359.1ms | clip: 0.0ms | optim: 23.5ms
step 342 | loss: 3.010669 | lr 1.8000e-04 | norm: 0.2752 | dt: 7813.18ms | tok/sec: 67103.0 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 13.9ms | fwd: 2405.9ms | loss: 4.7ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 343 | loss: 2.867569 | lr 1.8000e-04 | norm: 0.2013 | dt: 7817.75ms | tok/sec: 67063.8 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 16.7ms | fwd: 2411.1ms | loss: 4.7ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.6ms
step 344 | loss: 3.039850 | lr 1.8000e-04 | norm: 0.2166 | dt: 7815.63ms | tok/sec: 67082.0 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 14.4ms | fwd: 2409.5ms | loss: 4.8ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.5ms
step 345 | loss: 2.879299 | lr 1.8000e-04 | norm: 0.2084 | dt: 7820.71ms | tok/sec: 67038.4 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 14.0ms | fwd: 2411.5ms | loss: 4.7ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.6ms
step 346 | loss: 3.164414 | lr 1.8000e-04 | norm: 0.2426 | dt: 7815.15ms | tok/sec: 67086.1 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 14.3ms | fwd: 2406.1ms | loss: 4.7ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.4ms
step 347 | loss: 3.042817 | lr 1.8000e-04 | norm: 0.2151 | dt: 7818.34ms | tok/sec: 67058.8 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 14.0ms | fwd: 2415.4ms | loss: 4.8ms | bwd: 5350.3ms | clip: 0.0ms | optim: 23.5ms
step 348 | loss: 2.988883 | lr 1.8000e-04 | norm: 0.3916 | dt: 7813.14ms | tok/sec: 67103.3 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 14.7ms | fwd: 2407.0ms | loss: 4.8ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 349 | loss: 3.068761 | lr 1.8000e-04 | norm: 0.2411 | dt: 7883.92ms | tok/sec: 66501.0 | mem: 72099MB | Time Left: 02 hrs : 51 mins
[TIMING] data: 14.3ms | fwd: 2473.6ms | loss: 4.8ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 350 | loss: 2.864789 | lr 1.8000e-04 | norm: 0.2502 | dt: 7820.63ms | tok/sec: 67039.1 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 11.8ms | fwd: 2413.9ms | loss: 4.8ms | bwd: 5357.2ms | clip: 0.0ms | optim: 23.6ms
step 351 | loss: 2.894976 | lr 1.8000e-04 | norm: 0.1974 | dt: 7809.77ms | tok/sec: 67132.3 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 11.4ms | fwd: 2403.0ms | loss: 4.7ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.5ms
step 352 | loss: 3.067198 | lr 1.8000e-04 | norm: 0.2066 | dt: 7812.85ms | tok/sec: 67105.8 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 14.0ms | fwd: 2404.5ms | loss: 4.8ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.6ms
step 353 | loss: 2.925445 | lr 1.8000e-04 | norm: 0.2000 | dt: 7875.61ms | tok/sec: 66571.1 | mem: 72099MB | Time Left: 02 hrs : 50 mins
[TIMING] data: 12.9ms | fwd: 2474.7ms | loss: 4.7ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.5ms
step 354 | loss: 2.710821 | lr 1.8000e-04 | norm: 0.2429 | dt: 7816.06ms | tok/sec: 67078.3 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 13.1ms | fwd: 2410.9ms | loss: 4.8ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.7ms
step 355 | loss: 2.970938 | lr 1.8000e-04 | norm: 0.2767 | dt: 7816.23ms | tok/sec: 67076.9 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 13.9ms | fwd: 2404.9ms | loss: 4.7ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.5ms
step 356 | loss: 2.845410 | lr 1.8000e-04 | norm: 0.2150 | dt: 7830.13ms | tok/sec: 66957.7 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 13.2ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5360.0ms | clip: 0.0ms | optim: 23.5ms
step 357 | loss: 2.515317 | lr 1.8000e-04 | norm: 0.2816 | dt: 7820.00ms | tok/sec: 67044.5 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 13.7ms | fwd: 2410.6ms | loss: 4.7ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.6ms
step 358 | loss: 2.428969 | lr 1.8000e-04 | norm: 0.2523 | dt: 7818.35ms | tok/sec: 67058.7 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 14.4ms | fwd: 2408.6ms | loss: 4.8ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.7ms
step 359 | loss: 2.820267 | lr 1.8000e-04 | norm: 0.2654 | dt: 7820.57ms | tok/sec: 67039.6 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 14.2ms | fwd: 2410.7ms | loss: 4.8ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.6ms
step 360 | loss: 2.867252 | lr 1.8000e-04 | norm: 0.2250 | dt: 7816.41ms | tok/sec: 67075.3 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 12.7ms | fwd: 2407.7ms | loss: 4.7ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 361 | loss: 2.998608 | lr 1.8000e-04 | norm: 0.2168 | dt: 7812.89ms | tok/sec: 67105.5 | mem: 72099MB | Time Left: 02 hrs : 47 mins
[TIMING] data: 12.0ms | fwd: 2406.4ms | loss: 4.7ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.5ms
step 362 | loss: 2.487839 | lr 1.8000e-04 | norm: 0.2499 | dt: 7815.46ms | tok/sec: 67083.5 | mem: 72099MB | Time Left: 02 hrs : 47 mins
[TIMING] data: 12.3ms | fwd: 2409.5ms | loss: 4.7ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 363 | loss: 3.103752 | lr 1.8000e-04 | norm: 0.2305 | dt: 7816.73ms | tok/sec: 67072.6 | mem: 72099MB | Time Left: 02 hrs : 47 mins
[TIMING] data: 12.5ms | fwd: 2410.8ms | loss: 4.8ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 364 | loss: 2.597869 | lr 1.8000e-04 | norm: 0.2458 | dt: 7883.05ms | tok/sec: 66508.3 | mem: 72099MB | Time Left: 02 hrs : 49 mins
[TIMING] data: 11.7ms | fwd: 2478.9ms | loss: 4.8ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 365 | loss: 2.792683 | lr 1.8000e-04 | norm: 0.1944 | dt: 7814.77ms | tok/sec: 67089.4 | mem: 72099MB | Time Left: 02 hrs : 47 mins
[TIMING] data: 14.3ms | fwd: 2407.1ms | loss: 4.8ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00002.bin
100000000
step 366 | loss: 2.881708 | lr 1.8000e-04 | norm: 0.2397 | dt: 7837.77ms | tok/sec: 66892.5 | mem: 72099MB | Time Left: 02 hrs : 47 mins
[TIMING] data: 32.1ms | fwd: 2412.6ms | loss: 4.7ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 367 | loss: 2.953289 | lr 1.8000e-04 | norm: 0.1923 | dt: 7808.78ms | tok/sec: 67140.8 | mem: 72099MB | Time Left: 02 hrs : 47 mins
[TIMING] data: 4.9ms | fwd: 2411.0ms | loss: 4.7ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.4ms
step 368 | loss: 3.200708 | lr 1.8000e-04 | norm: 0.2312 | dt: 7814.13ms | tok/sec: 67094.8 | mem: 72099MB | Time Left: 02 hrs : 47 mins
[TIMING] data: 8.9ms | fwd: 2411.7ms | loss: 4.8ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.5ms
step 369 | loss: 3.181402 | lr 1.8000e-04 | norm: 0.2476 | dt: 7882.04ms | tok/sec: 66516.8 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 13.7ms | fwd: 2477.6ms | loss: 4.7ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.5ms
step 370 | loss: 2.809419 | lr 1.8000e-04 | norm: 0.2778 | dt: 7812.64ms | tok/sec: 67107.6 | mem: 72099MB | Time Left: 02 hrs : 46 mins
[TIMING] data: 14.6ms | fwd: 2405.9ms | loss: 4.7ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 371 | loss: 2.524578 | lr 1.8000e-04 | norm: 0.2887 | dt: 7884.08ms | tok/sec: 66499.6 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 14.3ms | fwd: 2477.6ms | loss: 4.8ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 372 | loss: 2.497722 | lr 1.8000e-04 | norm: 0.4638 | dt: 7821.19ms | tok/sec: 67034.3 | mem: 72099MB | Time Left: 02 hrs : 46 mins
[TIMING] data: 15.0ms | fwd: 2411.6ms | loss: 4.8ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 373 | loss: 2.936451 | lr 1.8000e-04 | norm: 0.2892 | dt: 7808.56ms | tok/sec: 67142.7 | mem: 72099MB | Time Left: 02 hrs : 46 mins
[TIMING] data: 13.4ms | fwd: 2402.9ms | loss: 4.8ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 374 | loss: 2.682808 | lr 1.8000e-04 | norm: 0.2818 | dt: 7822.26ms | tok/sec: 67025.1 | mem: 72099MB | Time Left: 02 hrs : 46 mins
[TIMING] data: 15.1ms | fwd: 2412.0ms | loss: 4.7ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 375 | loss: 2.659974 | lr 1.8000e-04 | norm: 0.1824 | dt: 7819.22ms | tok/sec: 67051.2 | mem: 72099MB | Time Left: 02 hrs : 46 mins
[TIMING] data: 14.1ms | fwd: 2413.1ms | loss: 4.8ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 376 | loss: 2.891670 | lr 1.8000e-04 | norm: 0.1979 | dt: 7820.72ms | tok/sec: 67038.3 | mem: 72099MB | Time Left: 02 hrs : 46 mins
[TIMING] data: 14.6ms | fwd: 2411.9ms | loss: 4.8ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 377 | loss: 2.909538 | lr 1.8000e-04 | norm: 0.3094 | dt: 7814.47ms | tok/sec: 67092.0 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 16.7ms | fwd: 2404.8ms | loss: 4.7ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.6ms
step 378 | loss: 2.911615 | lr 1.8000e-04 | norm: 0.3883 | dt: 7815.88ms | tok/sec: 67079.8 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 14.7ms | fwd: 2408.8ms | loss: 4.8ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.6ms
step 379 | loss: 2.960093 | lr 1.8000e-04 | norm: 0.2562 | dt: 7823.36ms | tok/sec: 67015.7 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 13.9ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.6ms
step 380 | loss: 2.887461 | lr 1.8000e-04 | norm: 0.2062 | dt: 7875.73ms | tok/sec: 66570.1 | mem: 72099MB | Time Left: 02 hrs : 46 mins
[TIMING] data: 14.6ms | fwd: 2472.7ms | loss: 4.8ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.5ms
step 381 | loss: 2.776631 | lr 1.8000e-04 | norm: 0.8163 | dt: 7819.10ms | tok/sec: 67052.2 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 16.4ms | fwd: 2407.1ms | loss: 4.8ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 382 | loss: 2.976094 | lr 1.8000e-04 | norm: 0.2192 | dt: 7814.09ms | tok/sec: 67095.2 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 14.2ms | fwd: 2407.8ms | loss: 4.7ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 383 | loss: 2.962312 | lr 1.8000e-04 | norm: 0.2216 | dt: 7816.23ms | tok/sec: 67076.8 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 14.4ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.6ms
step 384 | loss: 2.882998 | lr 1.8000e-04 | norm: 0.2093 | dt: 7820.38ms | tok/sec: 67041.2 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 13.2ms | fwd: 2413.3ms | loss: 4.8ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.4ms
step 385 | loss: 2.674818 | lr 1.8000e-04 | norm: 0.2363 | dt: 7827.55ms | tok/sec: 66979.8 | mem: 72099MB | Time Left: 02 hrs : 45 mins
[TIMING] data: 21.4ms | fwd: 2411.7ms | loss: 4.8ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 386 | loss: 2.882162 | lr 1.8000e-04 | norm: 0.2452 | dt: 7815.11ms | tok/sec: 67086.5 | mem: 72099MB | Time Left: 02 hrs : 44 mins
[TIMING] data: 15.3ms | fwd: 2408.0ms | loss: 4.8ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 387 | loss: 3.007239 | lr 1.8000e-04 | norm: 0.2508 | dt: 7824.29ms | tok/sec: 67007.7 | mem: 72099MB | Time Left: 02 hrs : 44 mins
[TIMING] data: 12.9ms | fwd: 2415.6ms | loss: 4.8ms | bwd: 5357.3ms | clip: 0.0ms | optim: 23.5ms
step 388 | loss: 2.729478 | lr 1.8000e-04 | norm: 0.2138 | dt: 7814.65ms | tok/sec: 67090.4 | mem: 72099MB | Time Left: 02 hrs : 44 mins
[TIMING] data: 13.6ms | fwd: 2407.3ms | loss: 4.8ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.5ms
step 389 | loss: 3.018740 | lr 1.8000e-04 | norm: 0.2495 | dt: 7824.63ms | tok/sec: 67004.8 | mem: 72099MB | Time Left: 02 hrs : 44 mins
[TIMING] data: 17.8ms | fwd: 2418.9ms | loss: 4.7ms | bwd: 5349.4ms | clip: 0.0ms | optim: 23.6ms
step 390 | loss: 2.803033 | lr 1.8000e-04 | norm: 0.6550 | dt: 7816.79ms | tok/sec: 67072.0 | mem: 72099MB | Time Left: 02 hrs : 44 mins
[TIMING] data: 15.5ms | fwd: 2408.7ms | loss: 4.8ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 391 | loss: 2.856046 | lr 1.8000e-04 | norm: 0.2215 | dt: 7807.34ms | tok/sec: 67153.2 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 12.6ms | fwd: 2404.8ms | loss: 4.7ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.6ms
step 392 | loss: 2.632756 | lr 1.8000e-04 | norm: 0.2123 | dt: 7815.47ms | tok/sec: 67083.3 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 13.6ms | fwd: 2409.7ms | loss: 4.8ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 393 | loss: 2.922002 | lr 1.8000e-04 | norm: 0.2176 | dt: 7810.28ms | tok/sec: 67127.9 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 17.3ms | fwd: 2404.2ms | loss: 4.8ms | bwd: 5350.0ms | clip: 0.0ms | optim: 23.5ms
step 394 | loss: 3.132426 | lr 1.8000e-04 | norm: 0.2109 | dt: 7812.63ms | tok/sec: 67107.7 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 14.0ms | fwd: 2404.5ms | loss: 4.7ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.5ms
step 395 | loss: 2.639308 | lr 1.8000e-04 | norm: 0.2441 | dt: 7813.78ms | tok/sec: 67097.9 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 13.3ms | fwd: 2404.9ms | loss: 4.7ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.5ms
step 396 | loss: 2.927966 | lr 1.8000e-04 | norm: 0.2550 | dt: 7820.51ms | tok/sec: 67040.1 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 13.5ms | fwd: 2413.6ms | loss: 4.8ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.6ms
step 397 | loss: 3.191270 | lr 1.8000e-04 | norm: 0.2057 | dt: 7819.29ms | tok/sec: 67050.6 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 13.8ms | fwd: 2412.9ms | loss: 4.7ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 398 | loss: 2.667762 | lr 1.8000e-04 | norm: 0.2114 | dt: 7821.29ms | tok/sec: 67033.5 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 14.0ms | fwd: 2412.7ms | loss: 4.7ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.5ms
step 399 | loss: 3.007249 | lr 1.8000e-04 | norm: 0.2229 | dt: 7818.73ms | tok/sec: 67055.4 | mem: 72099MB | Time Left: 02 hrs : 43 mins
[TIMING] data: 11.7ms | fwd: 2412.8ms | loss: 4.8ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.5ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_400.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_200.ckpt
step 400 | loss: 2.806521 | lr 1.8000e-04 | norm: 0.3086 | dt: 8058.67ms | tok/sec: 65058.9 | mem: 72099MB | Time Left: 02 hrs : 48 mins
[TIMING] data: 11.5ms | fwd: 2660.5ms | loss: 4.8ms | bwd: 5345.9ms | clip: 0.0ms | optim: 23.5ms
step 401 | loss: 2.832193 | lr 1.8000e-04 | norm: 0.2208 | dt: 7793.90ms | tok/sec: 67269.0 | mem: 72099MB | Time Left: 02 hrs : 42 mins
[TIMING] data: 13.5ms | fwd: 2396.0ms | loss: 4.8ms | bwd: 5344.6ms | clip: 0.0ms | optim: 23.5ms
step 402 | loss: 3.192397 | lr 1.8000e-04 | norm: 0.2321 | dt: 7798.04ms | tok/sec: 67233.3 | mem: 72099MB | Time Left: 02 hrs : 42 mins
[TIMING] data: 15.0ms | fwd: 2396.8ms | loss: 4.9ms | bwd: 5345.7ms | clip: 0.0ms | optim: 23.5ms
step 403 | loss: 2.991236 | lr 1.8000e-04 | norm: 0.2248 | dt: 7815.19ms | tok/sec: 67085.8 | mem: 72099MB | Time Left: 02 hrs : 42 mins
[TIMING] data: 25.2ms | fwd: 2398.0ms | loss: 4.8ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 404 | loss: 2.724464 | lr 1.8000e-04 | norm: 0.1873 | dt: 7824.32ms | tok/sec: 67007.5 | mem: 72099MB | Time Left: 02 hrs : 42 mins
[TIMING] data: 41.0ms | fwd: 2397.7ms | loss: 4.9ms | bwd: 5344.5ms | clip: 0.0ms | optim: 23.5ms
step 405 | loss: 2.809619 | lr 1.8000e-04 | norm: 0.2266 | dt: 7814.43ms | tok/sec: 67092.3 | mem: 72099MB | Time Left: 02 hrs : 42 mins
[TIMING] data: 18.6ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5348.9ms | clip: 0.0ms | optim: 23.6ms
step 406 | loss: 2.926108 | lr 1.8000e-04 | norm: 0.3125 | dt: 7810.43ms | tok/sec: 67126.7 | mem: 72099MB | Time Left: 02 hrs : 42 mins
[TIMING] data: 18.6ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5346.3ms | clip: 0.0ms | optim: 23.6ms
step 407 | loss: 3.136992 | lr 1.8000e-04 | norm: 0.3673 | dt: 7811.60ms | tok/sec: 67116.6 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 15.9ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.6ms
step 408 | loss: 2.851133 | lr 1.8000e-04 | norm: 0.3465 | dt: 7814.33ms | tok/sec: 67093.2 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 15.8ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5347.7ms | clip: 0.0ms | optim: 23.6ms
step 409 | loss: 2.752090 | lr 1.8000e-04 | norm: 0.2033 | dt: 7816.76ms | tok/sec: 67072.3 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 14.3ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.6ms
step 410 | loss: 2.931116 | lr 1.8000e-04 | norm: 0.2699 | dt: 7809.67ms | tok/sec: 67133.2 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 15.0ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5346.7ms | clip: 0.0ms | optim: 23.5ms
step 411 | loss: 2.888863 | lr 1.8000e-04 | norm: 0.2498 | dt: 7813.75ms | tok/sec: 67098.2 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 15.6ms | fwd: 2402.0ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.5ms
step 412 | loss: 2.961412 | lr 1.8000e-04 | norm: 0.2273 | dt: 7820.10ms | tok/sec: 67043.6 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 17.4ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5350.3ms | clip: 0.0ms | optim: 23.7ms
step 413 | loss: 2.661156 | lr 1.8000e-04 | norm: 0.3017 | dt: 7816.23ms | tok/sec: 67076.9 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 14.9ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
step 414 | loss: 2.313619 | lr 1.8000e-04 | norm: 0.3414 | dt: 7821.23ms | tok/sec: 67033.9 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 15.8ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 415 | loss: 2.772603 | lr 1.8000e-04 | norm: 0.2534 | dt: 7819.15ms | tok/sec: 67051.8 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 15.5ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.7ms
step 416 | loss: 2.980647 | lr 1.8000e-04 | norm: 0.2557 | dt: 7821.19ms | tok/sec: 67034.3 | mem: 72099MB | Time Left: 02 hrs : 40 mins
[TIMING] data: 14.5ms | fwd: 2405.1ms | loss: 4.9ms | bwd: 5359.5ms | clip: 0.0ms | optim: 23.6ms
step 417 | loss: 2.425819 | lr 1.8000e-04 | norm: 0.2366 | dt: 7819.18ms | tok/sec: 67051.6 | mem: 72099MB | Time Left: 02 hrs : 40 mins
[TIMING] data: 14.8ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 418 | loss: 3.030983 | lr 1.8000e-04 | norm: 0.2590 | dt: 7821.07ms | tok/sec: 67035.3 | mem: 72099MB | Time Left: 02 hrs : 40 mins
[TIMING] data: 17.4ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.7ms
step 419 | loss: 2.907376 | lr 1.8000e-04 | norm: 0.2627 | dt: 7825.56ms | tok/sec: 66996.8 | mem: 72099MB | Time Left: 02 hrs : 40 mins
[TIMING] data: 14.2ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.5ms
step 420 | loss: 2.769475 | lr 1.8000e-04 | norm: 0.3065 | dt: 7816.61ms | tok/sec: 67073.6 | mem: 72099MB | Time Left: 02 hrs : 40 mins
[TIMING] data: 16.9ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.6ms
step 421 | loss: 2.924163 | lr 1.8000e-04 | norm: 0.2389 | dt: 7888.86ms | tok/sec: 66459.3 | mem: 72099MB | Time Left: 02 hrs : 41 mins
[TIMING] data: 15.5ms | fwd: 2475.4ms | loss: 4.9ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.6ms
step 422 | loss: 2.967349 | lr 1.8000e-04 | norm: 0.2710 | dt: 7827.83ms | tok/sec: 66977.5 | mem: 72099MB | Time Left: 02 hrs : 40 mins
[TIMING] data: 15.6ms | fwd: 2412.4ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.6ms
step 423 | loss: 2.608540 | lr 1.8000e-04 | norm: 0.2539 | dt: 7811.86ms | tok/sec: 67114.4 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 14.2ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.5ms
step 424 | loss: 2.767216 | lr 1.8000e-04 | norm: 0.2095 | dt: 7821.95ms | tok/sec: 67027.8 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 16.5ms | fwd: 2412.7ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 425 | loss: 3.096659 | lr 1.8000e-04 | norm: 0.2568 | dt: 7820.26ms | tok/sec: 67042.3 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 15.8ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 426 | loss: 3.157419 | lr 1.8000e-04 | norm: 0.2935 | dt: 7814.09ms | tok/sec: 67095.2 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 14.2ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 427 | loss: 2.812790 | lr 1.8000e-04 | norm: 0.2368 | dt: 7812.68ms | tok/sec: 67107.3 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 11.8ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 428 | loss: 2.796491 | lr 1.8000e-04 | norm: 0.2851 | dt: 7823.21ms | tok/sec: 67017.0 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 14.1ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.5ms
step 429 | loss: 2.919619 | lr 1.8000e-04 | norm: 0.3690 | dt: 7818.46ms | tok/sec: 67057.7 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 12.8ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.6ms
step 430 | loss: 2.599972 | lr 1.8000e-04 | norm: 0.3611 | dt: 7821.32ms | tok/sec: 67033.2 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 14.3ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.6ms
step 431 | loss: 3.019199 | lr 1.8000e-04 | norm: 0.4362 | dt: 7818.12ms | tok/sec: 67060.6 | mem: 72099MB | Time Left: 02 hrs : 38 mins
[TIMING] data: 13.7ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.6ms
step 432 | loss: 3.110521 | lr 1.8000e-04 | norm: 0.2747 | dt: 7824.51ms | tok/sec: 67005.8 | mem: 72099MB | Time Left: 02 hrs : 38 mins
[TIMING] data: 13.9ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 433 | loss: 2.877022 | lr 1.8000e-04 | norm: 0.2261 | dt: 7813.39ms | tok/sec: 67101.2 | mem: 72099MB | Time Left: 02 hrs : 38 mins
[TIMING] data: 15.0ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 434 | loss: 2.863173 | lr 1.8000e-04 | norm: 0.2316 | dt: 7893.93ms | tok/sec: 66416.6 | mem: 72099MB | Time Left: 02 hrs : 40 mins
[TIMING] data: 15.6ms | fwd: 2481.6ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.6ms
step 435 | loss: 2.951332 | lr 1.8000e-04 | norm: 0.4271 | dt: 7883.66ms | tok/sec: 66503.1 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 14.4ms | fwd: 2472.1ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.6ms
step 436 | loss: 2.908975 | lr 1.8000e-04 | norm: 0.2219 | dt: 7814.50ms | tok/sec: 67091.7 | mem: 72099MB | Time Left: 02 hrs : 38 mins
[TIMING] data: 15.4ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.5ms
step 437 | loss: 2.534609 | lr 1.8000e-04 | norm: 0.2999 | dt: 7818.61ms | tok/sec: 67056.4 | mem: 72099MB | Time Left: 02 hrs : 38 mins
[TIMING] data: 16.0ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5351.0ms | clip: 0.0ms | optim: 23.6ms
step 438 | loss: 2.686295 | lr 1.8000e-04 | norm: 0.2642 | dt: 7812.39ms | tok/sec: 67109.8 | mem: 72099MB | Time Left: 02 hrs : 37 mins
[TIMING] data: 16.2ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.6ms
step 439 | loss: 3.062030 | lr 1.8000e-04 | norm: 0.2550 | dt: 7812.82ms | tok/sec: 67106.1 | mem: 72099MB | Time Left: 02 hrs : 37 mins
[TIMING] data: 14.3ms | fwd: 2400.8ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.5ms
step 440 | loss: 2.938438 | lr 1.8000e-04 | norm: 0.2244 | dt: 7884.67ms | tok/sec: 66494.6 | mem: 72099MB | Time Left: 02 hrs : 39 mins
[TIMING] data: 15.2ms | fwd: 2474.6ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 441 | loss: 2.538013 | lr 1.8000e-04 | norm: 0.2508 | dt: 7820.94ms | tok/sec: 67036.5 | mem: 72099MB | Time Left: 02 hrs : 37 mins
[TIMING] data: 14.8ms | fwd: 2414.4ms | loss: 4.9ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.6ms
step 442 | loss: 2.731732 | lr 1.8000e-04 | norm: 0.1895 | dt: 7822.95ms | tok/sec: 67019.2 | mem: 72099MB | Time Left: 02 hrs : 37 mins
[TIMING] data: 16.9ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.6ms
step 443 | loss: 2.876798 | lr 1.8000e-04 | norm: 0.2567 | dt: 7820.47ms | tok/sec: 67040.5 | mem: 72099MB | Time Left: 02 hrs : 37 mins
[TIMING] data: 15.0ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 444 | loss: 2.982586 | lr 1.8000e-04 | norm: 0.2863 | dt: 7826.77ms | tok/sec: 66986.5 | mem: 72099MB | Time Left: 02 hrs : 37 mins
[TIMING] data: 15.7ms | fwd: 2411.3ms | loss: 4.9ms | bwd: 5359.7ms | clip: 0.0ms | optim: 23.6ms
step 445 | loss: 2.738212 | lr 1.8000e-04 | norm: 0.2834 | dt: 7810.02ms | tok/sec: 67130.2 | mem: 72099MB | Time Left: 02 hrs : 36 mins
[TIMING] data: 14.8ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.6ms
step 446 | loss: 2.882792 | lr 1.8000e-04 | norm: 0.2377 | dt: 7888.90ms | tok/sec: 66459.0 | mem: 72099MB | Time Left: 02 hrs : 38 mins
[TIMING] data: 17.9ms | fwd: 2475.0ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.7ms
step 447 | loss: 2.578511 | lr 1.8000e-04 | norm: 0.2454 | dt: 7825.92ms | tok/sec: 66993.8 | mem: 72099MB | Time Left: 02 hrs : 37 mins
[TIMING] data: 14.9ms | fwd: 2411.3ms | loss: 4.9ms | bwd: 5357.9ms | clip: 0.0ms | optim: 23.6ms
step 448 | loss: 2.677471 | lr 1.8000e-04 | norm: 0.2645 | dt: 7815.36ms | tok/sec: 67084.3 | mem: 72099MB | Time Left: 02 hrs : 36 mins
[TIMING] data: 19.9ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5347.8ms | clip: 0.0ms | optim: 23.5ms
step 449 | loss: 2.975238 | lr 1.8000e-04 | norm: 0.2323 | dt: 7826.21ms | tok/sec: 66991.3 | mem: 72099MB | Time Left: 02 hrs : 36 mins
[TIMING] data: 15.9ms | fwd: 2412.7ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 450 | loss: 2.910864 | lr 1.8000e-04 | norm: 0.2252 | dt: 7820.15ms | tok/sec: 67043.2 | mem: 72099MB | Time Left: 02 hrs : 36 mins
[TIMING] data: 16.1ms | fwd: 2410.2ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 451 | loss: 2.757216 | lr 1.8000e-04 | norm: 0.2724 | dt: 7827.70ms | tok/sec: 66978.5 | mem: 72099MB | Time Left: 02 hrs : 36 mins
[TIMING] data: 18.0ms | fwd: 2415.8ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 452 | loss: 2.830509 | lr 1.8000e-04 | norm: 0.2111 | dt: 7820.00ms | tok/sec: 67044.5 | mem: 72099MB | Time Left: 02 hrs : 36 mins
[TIMING] data: 15.1ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 453 | loss: 2.894886 | lr 1.8000e-04 | norm: 0.2313 | dt: 7812.37ms | tok/sec: 67109.9 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 15.3ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5347.4ms | clip: 0.0ms | optim: 23.6ms
step 454 | loss: 2.665647 | lr 1.8000e-04 | norm: 0.2481 | dt: 7819.90ms | tok/sec: 67045.4 | mem: 72099MB | Time Left: 02 hrs : 36 mins
[TIMING] data: 16.7ms | fwd: 2404.7ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
step 455 | loss: 2.875380 | lr 1.8000e-04 | norm: 0.2279 | dt: 7822.25ms | tok/sec: 67025.2 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 12.6ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5360.4ms | clip: 0.0ms | optim: 23.4ms
step 456 | loss: 3.078440 | lr 1.8000e-04 | norm: 0.2764 | dt: 7824.49ms | tok/sec: 67006.0 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 13.9ms | fwd: 2413.9ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 457 | loss: 2.765251 | lr 1.8000e-04 | norm: 0.3010 | dt: 7822.41ms | tok/sec: 67023.8 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 12.7ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5357.3ms | clip: 0.0ms | optim: 23.6ms
step 458 | loss: 3.064617 | lr 1.8000e-04 | norm: 0.2958 | dt: 7824.99ms | tok/sec: 67001.7 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 14.2ms | fwd: 2416.5ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.5ms
step 459 | loss: 2.706254 | lr 1.8000e-04 | norm: 0.2174 | dt: 7819.73ms | tok/sec: 67046.8 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 14.4ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.6ms
step 460 | loss: 2.768926 | lr 1.8000e-04 | norm: 0.6927 | dt: 7824.65ms | tok/sec: 67004.6 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 14.5ms | fwd: 2415.9ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 461 | loss: 2.926013 | lr 1.8000e-04 | norm: 0.2543 | dt: 7817.01ms | tok/sec: 67070.2 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 14.1ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.6ms
step 462 | loss: 2.819990 | lr 1.8000e-04 | norm: 0.2093 | dt: 7825.99ms | tok/sec: 66993.2 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 16.5ms | fwd: 2414.4ms | loss: 4.9ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 463 | loss: 2.759146 | lr 1.8000e-04 | norm: 0.2240 | dt: 7820.84ms | tok/sec: 67037.3 | mem: 72099MB | Time Left: 02 hrs : 34 mins
[TIMING] data: 15.3ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 464 | loss: 2.593467 | lr 1.8000e-04 | norm: 0.3446 | dt: 7819.94ms | tok/sec: 67045.0 | mem: 72099MB | Time Left: 02 hrs : 34 mins
[TIMING] data: 15.6ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.5ms
step 465 | loss: 2.738202 | lr 1.8000e-04 | norm: 0.4428 | dt: 7820.25ms | tok/sec: 67042.4 | mem: 72099MB | Time Left: 02 hrs : 34 mins
[TIMING] data: 15.4ms | fwd: 2414.3ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.5ms
step 466 | loss: 2.971794 | lr 1.8000e-04 | norm: 0.1894 | dt: 7855.38ms | tok/sec: 66742.5 | mem: 72099MB | Time Left: 02 hrs : 35 mins
[TIMING] data: 15.5ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5391.3ms | clip: 0.0ms | optim: 23.5ms
step 467 | loss: 2.730577 | lr 1.8000e-04 | norm: 0.1870 | dt: 7819.65ms | tok/sec: 67047.5 | mem: 72099MB | Time Left: 02 hrs : 34 mins
[TIMING] data: 16.8ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 468 | loss: 2.752822 | lr 1.8000e-04 | norm: 0.2236 | dt: 7826.35ms | tok/sec: 66990.1 | mem: 72099MB | Time Left: 02 hrs : 34 mins
[TIMING] data: 15.0ms | fwd: 2415.3ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 469 | loss: 2.880101 | lr 1.8000e-04 | norm: 0.2095 | dt: 7818.96ms | tok/sec: 67053.4 | mem: 72099MB | Time Left: 02 hrs : 34 mins
[TIMING] data: 13.7ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.5ms
step 470 | loss: 2.712054 | lr 1.8000e-04 | norm: 0.2505 | dt: 7814.38ms | tok/sec: 67092.7 | mem: 72099MB | Time Left: 02 hrs : 33 mins
[TIMING] data: 16.4ms | fwd: 2402.0ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 471 | loss: 2.883191 | lr 1.8000e-04 | norm: 0.2228 | dt: 7825.40ms | tok/sec: 66998.2 | mem: 72099MB | Time Left: 02 hrs : 33 mins
[TIMING] data: 13.2ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.6ms
step 472 | loss: 2.732019 | lr 1.8000e-04 | norm: 0.2799 | dt: 7819.06ms | tok/sec: 67052.6 | mem: 72099MB | Time Left: 02 hrs : 33 mins
[TIMING] data: 14.9ms | fwd: 2412.4ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.7ms
step 473 | loss: 2.858928 | lr 1.8000e-04 | norm: 0.2485 | dt: 7826.74ms | tok/sec: 66986.8 | mem: 72099MB | Time Left: 02 hrs : 33 mins
[TIMING] data: 14.6ms | fwd: 2413.7ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.5ms
step 474 | loss: 2.871937 | lr 1.8000e-04 | norm: 0.2702 | dt: 7832.76ms | tok/sec: 66935.3 | mem: 72099MB | Time Left: 02 hrs : 33 mins
[TIMING] data: 20.2ms | fwd: 2414.0ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.6ms
step 475 | loss: 2.593247 | lr 1.8000e-04 | norm: 0.2151 | dt: 7816.71ms | tok/sec: 67072.7 | mem: 72099MB | Time Left: 02 hrs : 33 mins
[TIMING] data: 15.0ms | fwd: 2404.5ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.6ms
step 476 | loss: 2.685888 | lr 1.8000e-04 | norm: 0.2154 | dt: 7812.40ms | tok/sec: 67109.7 | mem: 72099MB | Time Left: 02 hrs : 32 mins
[TIMING] data: 13.2ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 477 | loss: 2.938451 | lr 1.8000e-04 | norm: 0.2584 | dt: 7819.25ms | tok/sec: 67051.0 | mem: 72099MB | Time Left: 02 hrs : 32 mins
[TIMING] data: 14.3ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 478 | loss: 2.432622 | lr 1.8000e-04 | norm: 0.2237 | dt: 7829.75ms | tok/sec: 66961.0 | mem: 72099MB | Time Left: 02 hrs : 33 mins
[TIMING] data: 14.7ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5366.9ms | clip: 0.0ms | optim: 23.6ms
step 479 | loss: 2.855492 | lr 1.8000e-04 | norm: 0.2537 | dt: 7825.36ms | tok/sec: 66998.6 | mem: 72099MB | Time Left: 02 hrs : 32 mins
[TIMING] data: 17.9ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.8ms
step 480 | loss: 2.939346 | lr 1.8000e-04 | norm: 0.2053 | dt: 7816.07ms | tok/sec: 67078.3 | mem: 72099MB | Time Left: 02 hrs : 32 mins
[TIMING] data: 16.3ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 481 | loss: 2.691962 | lr 1.8000e-04 | norm: 0.2263 | dt: 7823.99ms | tok/sec: 67010.3 | mem: 72099MB | Time Left: 02 hrs : 32 mins
[TIMING] data: 14.5ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
step 482 | loss: 2.596405 | lr 1.8000e-04 | norm: 0.2527 | dt: 7817.43ms | tok/sec: 67066.5 | mem: 72099MB | Time Left: 02 hrs : 32 mins
[TIMING] data: 14.4ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.6ms
step 483 | loss: 2.904422 | lr 1.8000e-04 | norm: 0.3721 | dt: 7810.02ms | tok/sec: 67130.2 | mem: 72099MB | Time Left: 02 hrs : 32 mins
[TIMING] data: 13.6ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 484 | loss: 2.931988 | lr 1.8000e-04 | norm: 0.2625 | dt: 7813.09ms | tok/sec: 67103.8 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 14.2ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.5ms
step 485 | loss: 2.919129 | lr 1.8000e-04 | norm: 0.2281 | dt: 7818.32ms | tok/sec: 67058.9 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 16.1ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.5ms
step 486 | loss: 3.056974 | lr 1.8000e-04 | norm: 0.3013 | dt: 7816.90ms | tok/sec: 67071.1 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 15.8ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.6ms
step 487 | loss: 2.705192 | lr 1.8000e-04 | norm: 0.2836 | dt: 7822.46ms | tok/sec: 67023.4 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 14.5ms | fwd: 2411.8ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 488 | loss: 2.785932 | lr 1.8000e-04 | norm: 0.2675 | dt: 7816.34ms | tok/sec: 67075.9 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 15.7ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 489 | loss: 2.934935 | lr 1.8000e-04 | norm: 0.3309 | dt: 7821.77ms | tok/sec: 67029.3 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 14.5ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.6ms
step 490 | loss: 2.918564 | lr 1.8000e-04 | norm: 0.3586 | dt: 7820.66ms | tok/sec: 67038.9 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 14.9ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 491 | loss: 2.655756 | lr 1.8000e-04 | norm: 0.1885 | dt: 7814.64ms | tok/sec: 67090.5 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 14.1ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.6ms
step 492 | loss: 3.073330 | lr 1.8000e-04 | norm: 0.2274 | dt: 7815.42ms | tok/sec: 67083.8 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 14.7ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 493 | loss: 2.935989 | lr 1.8000e-04 | norm: 0.2214 | dt: 7826.96ms | tok/sec: 66984.9 | mem: 72099MB | Time Left: 02 hrs : 31 mins
[TIMING] data: 14.0ms | fwd: 2416.9ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 494 | loss: 2.819641 | lr 1.8000e-04 | norm: 0.2336 | dt: 7823.53ms | tok/sec: 67014.2 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 15.2ms | fwd: 2415.2ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.6ms
step 495 | loss: 2.967891 | lr 1.8000e-04 | norm: 0.2411 | dt: 7822.05ms | tok/sec: 67026.9 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 14.8ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 496 | loss: 3.033203 | lr 1.8000e-04 | norm: 0.2763 | dt: 7823.52ms | tok/sec: 67014.3 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 14.5ms | fwd: 2414.5ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.5ms
step 497 | loss: 2.983359 | lr 1.8000e-04 | norm: 0.2533 | dt: 7817.93ms | tok/sec: 67062.2 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 14.3ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 498 | loss: 2.985061 | lr 1.8000e-04 | norm: 0.5425 | dt: 7838.34ms | tok/sec: 66887.7 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 17.6ms | fwd: 2424.2ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 499 | loss: 2.973888 | lr 1.8000e-04 | norm: 0.3108 | dt: 7825.63ms | tok/sec: 66996.3 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 19.6ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.7ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_500.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_300.ckpt
validation loss: 2.9562
step 500 | loss: 2.792718 | lr 1.8000e-04 | norm: 0.2428 | dt: 13769.84ms | tok/sec: 38075.1 | mem: 72099MB | Time Left: 04 hrs : 24 mins
[TIMING] data: 14.1ms | fwd: 2395.3ms | loss: 4.9ms | bwd: 5339.7ms | clip: 0.0ms | optim: 23.6ms
step 501 | loss: 2.930902 | lr 1.8000e-04 | norm: 0.2697 | dt: 7801.61ms | tok/sec: 67202.5 | mem: 72099MB | Time Left: 02 hrs : 29 mins
[TIMING] data: 14.7ms | fwd: 2397.6ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.5ms
step 502 | loss: 2.572151 | lr 1.8000e-04 | norm: 0.2885 | dt: 7834.86ms | tok/sec: 66917.4 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 54.8ms | fwd: 2397.7ms | loss: 4.9ms | bwd: 5342.3ms | clip: 0.0ms | optim: 23.5ms
step 503 | loss: 2.934346 | lr 1.8000e-04 | norm: 0.2584 | dt: 7825.29ms | tok/sec: 66999.2 | mem: 72099MB | Time Left: 02 hrs : 29 mins
[TIMING] data: 43.1ms | fwd: 2397.0ms | loss: 4.9ms | bwd: 5344.3ms | clip: 0.0ms | optim: 23.5ms
step 504 | loss: 2.914587 | lr 1.8000e-04 | norm: 0.2668 | dt: 7856.81ms | tok/sec: 66730.4 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 68.2ms | fwd: 2398.8ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.6ms
step 505 | loss: 2.947196 | lr 1.8000e-04 | norm: 0.2489 | dt: 7813.99ms | tok/sec: 67096.0 | mem: 72099MB | Time Left: 02 hrs : 29 mins
[TIMING] data: 14.6ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5350.0ms | clip: 0.0ms | optim: 23.5ms
step 506 | loss: 2.633547 | lr 1.8000e-04 | norm: 0.3324 | dt: 7817.98ms | tok/sec: 67061.8 | mem: 72099MB | Time Left: 02 hrs : 29 mins
[TIMING] data: 15.1ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 507 | loss: 3.067468 | lr 1.8000e-04 | norm: 0.2565 | dt: 7817.47ms | tok/sec: 67066.2 | mem: 72099MB | Time Left: 02 hrs : 29 mins
[TIMING] data: 14.0ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.6ms
step 508 | loss: 2.847971 | lr 1.8000e-04 | norm: 0.2698 | dt: 7819.37ms | tok/sec: 67049.9 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 15.4ms | fwd: 2412.2ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
step 509 | loss: 2.936364 | lr 1.8000e-04 | norm: 0.2751 | dt: 7809.67ms | tok/sec: 67133.2 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 16.1ms | fwd: 2403.2ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.6ms
step 510 | loss: 2.973891 | lr 1.8000e-04 | norm: 0.5365 | dt: 7818.86ms | tok/sec: 67054.2 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 15.4ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 511 | loss: 2.763668 | lr 1.8000e-04 | norm: 0.3145 | dt: 7812.40ms | tok/sec: 67109.7 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 15.3ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.6ms
step 512 | loss: 2.902479 | lr 1.8000e-04 | norm: 0.2549 | dt: 7818.06ms | tok/sec: 67061.1 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 14.6ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.6ms
step 513 | loss: 2.616871 | lr 1.8000e-04 | norm: 0.3187 | dt: 7813.00ms | tok/sec: 67104.6 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 15.2ms | fwd: 2398.6ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 514 | loss: 2.983768 | lr 1.8000e-04 | norm: 0.3900 | dt: 7818.90ms | tok/sec: 67053.9 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 13.6ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.7ms
step 515 | loss: 2.962359 | lr 1.8000e-04 | norm: 0.2929 | dt: 7822.02ms | tok/sec: 67027.2 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 16.1ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.6ms
step 516 | loss: 2.912125 | lr 1.8000e-04 | norm: 0.3373 | dt: 7824.90ms | tok/sec: 67002.5 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 14.2ms | fwd: 2414.4ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.7ms
step 517 | loss: 3.461195 | lr 1.8000e-04 | norm: 0.6280 | dt: 7811.92ms | tok/sec: 67113.8 | mem: 72099MB | Time Left: 02 hrs : 27 mins
[TIMING] data: 12.8ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 518 | loss: 3.018747 | lr 1.8000e-04 | norm: 0.2864 | dt: 7831.08ms | tok/sec: 66949.6 | mem: 72099MB | Time Left: 02 hrs : 27 mins
[TIMING] data: 16.6ms | fwd: 2416.1ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.6ms
step 519 | loss: 2.823431 | lr 1.8000e-04 | norm: 0.2605 | dt: 7818.63ms | tok/sec: 67056.3 | mem: 72099MB | Time Left: 02 hrs : 27 mins
[TIMING] data: 14.1ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 520 | loss: 2.659997 | lr 1.8000e-04 | norm: 0.2880 | dt: 7828.81ms | tok/sec: 66969.1 | mem: 72099MB | Time Left: 02 hrs : 27 mins
[TIMING] data: 12.5ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5363.4ms | clip: 0.0ms | optim: 23.5ms
step 521 | loss: 2.971991 | lr 1.8000e-04 | norm: 0.3245 | dt: 7817.31ms | tok/sec: 67067.6 | mem: 72099MB | Time Left: 02 hrs : 27 mins
[TIMING] data: 14.4ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 522 | loss: 2.952991 | lr 1.8000e-04 | norm: 0.3451 | dt: 7879.23ms | tok/sec: 66540.5 | mem: 72099MB | Time Left: 02 hrs : 28 mins
[TIMING] data: 84.8ms | fwd: 2400.7ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.8ms
step 523 | loss: 3.104596 | lr 1.8000e-04 | norm: 0.2826 | dt: 7822.54ms | tok/sec: 67022.8 | mem: 72099MB | Time Left: 02 hrs : 27 mins
[TIMING] data: 16.5ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.5ms
step 524 | loss: 2.938687 | lr 1.8000e-04 | norm: 0.3494 | dt: 7818.76ms | tok/sec: 67055.1 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 14.9ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.5ms
step 525 | loss: 2.714121 | lr 1.8000e-04 | norm: 0.2651 | dt: 7823.04ms | tok/sec: 67018.4 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 15.7ms | fwd: 2413.3ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.6ms
step 526 | loss: 2.900012 | lr 1.8000e-04 | norm: 0.2802 | dt: 7817.11ms | tok/sec: 67069.3 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 17.0ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.6ms
step 527 | loss: 2.928252 | lr 1.8000e-04 | norm: 0.2500 | dt: 7824.47ms | tok/sec: 67006.2 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 15.1ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.7ms
step 528 | loss: 2.544303 | lr 1.8000e-04 | norm: 0.2570 | dt: 7828.03ms | tok/sec: 66975.7 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 15.4ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5362.2ms | clip: 0.0ms | optim: 23.6ms
step 529 | loss: 2.742530 | lr 1.8000e-04 | norm: 0.4939 | dt: 7819.30ms | tok/sec: 67050.5 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 14.9ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 530 | loss: 3.020648 | lr 1.8000e-04 | norm: 0.2974 | dt: 7820.54ms | tok/sec: 67039.9 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 12.8ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5359.1ms | clip: 0.0ms | optim: 23.6ms
step 531 | loss: 2.935235 | lr 1.8000e-04 | norm: 0.2194 | dt: 7827.42ms | tok/sec: 66981.0 | mem: 72099MB | Time Left: 02 hrs : 26 mins
[TIMING] data: 15.7ms | fwd: 2417.0ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.5ms
step 532 | loss: 3.126317 | lr 1.8000e-04 | norm: 0.2199 | dt: 7817.19ms | tok/sec: 67068.6 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 13.4ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 533 | loss: 2.925151 | lr 1.8000e-04 | norm: 0.2226 | dt: 7820.24ms | tok/sec: 67042.5 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 15.1ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 534 | loss: 3.129692 | lr 1.8000e-04 | norm: 0.2278 | dt: 7814.46ms | tok/sec: 67092.0 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 13.5ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.7ms
step 535 | loss: 2.786156 | lr 1.8000e-04 | norm: 0.2171 | dt: 7819.01ms | tok/sec: 67053.0 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 15.7ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 536 | loss: 2.704255 | lr 1.8000e-04 | norm: 0.1847 | dt: 7816.66ms | tok/sec: 67073.2 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 14.0ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 537 | loss: 2.883324 | lr 1.8000e-04 | norm: 0.1996 | dt: 7817.84ms | tok/sec: 67063.1 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 15.1ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.6ms
step 538 | loss: 3.061332 | lr 1.8000e-04 | norm: 0.4789 | dt: 7824.26ms | tok/sec: 67008.0 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 20.4ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 539 | loss: 2.750193 | lr 1.8000e-04 | norm: 0.2264 | dt: 7818.87ms | tok/sec: 67054.2 | mem: 72099MB | Time Left: 02 hrs : 24 mins
[TIMING] data: 14.7ms | fwd: 2410.8ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
step 540 | loss: 2.628300 | lr 1.8000e-04 | norm: 0.2909 | dt: 7834.19ms | tok/sec: 66923.1 | mem: 72099MB | Time Left: 02 hrs : 25 mins
[TIMING] data: 12.7ms | fwd: 2415.7ms | loss: 4.9ms | bwd: 5364.4ms | clip: 0.0ms | optim: 23.4ms
step 541 | loss: 2.989721 | lr 1.8000e-04 | norm: 0.5248 | dt: 7824.12ms | tok/sec: 67009.2 | mem: 72099MB | Time Left: 02 hrs : 24 mins
[TIMING] data: 18.8ms | fwd: 2410.8ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 542 | loss: 2.575011 | lr 1.8000e-04 | norm: 0.2451 | dt: 7819.71ms | tok/sec: 67047.0 | mem: 72099MB | Time Left: 02 hrs : 24 mins
[TIMING] data: 13.1ms | fwd: 2412.5ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.7ms
step 543 | loss: 2.814584 | lr 1.8000e-04 | norm: 0.5063 | dt: 7822.89ms | tok/sec: 67019.7 | mem: 72099MB | Time Left: 02 hrs : 24 mins
[TIMING] data: 13.0ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 544 | loss: 2.782661 | lr 1.8000e-04 | norm: 0.2345 | dt: 7819.22ms | tok/sec: 67051.2 | mem: 72099MB | Time Left: 02 hrs : 24 mins
[TIMING] data: 14.4ms | fwd: 2405.9ms | loss: 4.9ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.5ms
step 545 | loss: 3.250631 | lr 1.8000e-04 | norm: 0.6380 | dt: 7822.92ms | tok/sec: 67019.5 | mem: 72099MB | Time Left: 02 hrs : 24 mins
[TIMING] data: 17.7ms | fwd: 2411.1ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.7ms
step 546 | loss: 2.818037 | lr 1.8000e-04 | norm: 0.2197 | dt: 7823.70ms | tok/sec: 67012.8 | mem: 72099MB | Time Left: 02 hrs : 24 mins
[TIMING] data: 15.8ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.5ms
step 547 | loss: 2.765015 | lr 1.8000e-04 | norm: 0.2855 | dt: 7814.12ms | tok/sec: 67095.0 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 15.0ms | fwd: 2403.7ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 548 | loss: 2.812506 | lr 1.8000e-04 | norm: 0.3102 | dt: 7816.29ms | tok/sec: 67076.4 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 15.2ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 549 | loss: 2.886487 | lr 1.8000e-04 | norm: 0.2074 | dt: 7827.51ms | tok/sec: 66980.2 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 17.3ms | fwd: 2412.0ms | loss: 4.9ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.6ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00003.bin
100000000
step 550 | loss: 2.834620 | lr 1.8000e-04 | norm: 0.2883 | dt: 7829.46ms | tok/sec: 66963.5 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 23.1ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.7ms
step 551 | loss: 2.772402 | lr 1.8000e-04 | norm: 0.2248 | dt: 7817.55ms | tok/sec: 67065.5 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 5.2ms | fwd: 2413.8ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 552 | loss: 2.807906 | lr 1.8000e-04 | norm: 0.2095 | dt: 7820.97ms | tok/sec: 67036.2 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 10.9ms | fwd: 2411.3ms | loss: 4.9ms | bwd: 5359.1ms | clip: 0.0ms | optim: 23.6ms
step 553 | loss: 2.774983 | lr 1.8000e-04 | norm: 0.2520 | dt: 7816.33ms | tok/sec: 67076.0 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 15.7ms | fwd: 2406.8ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 554 | loss: 2.858669 | lr 1.8000e-04 | norm: 0.2240 | dt: 7828.58ms | tok/sec: 66971.1 | mem: 72099MB | Time Left: 02 hrs : 23 mins
[TIMING] data: 15.2ms | fwd: 2413.0ms | loss: 4.9ms | bwd: 5359.5ms | clip: 0.0ms | optim: 23.5ms
step 555 | loss: 2.685862 | lr 1.8000e-04 | norm: 0.2095 | dt: 7812.86ms | tok/sec: 67105.8 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 14.1ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 556 | loss: 2.924917 | lr 1.8000e-04 | norm: 0.2578 | dt: 7819.21ms | tok/sec: 67051.3 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 16.3ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 557 | loss: 2.693005 | lr 1.8000e-04 | norm: 0.2439 | dt: 7813.96ms | tok/sec: 67096.4 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 15.1ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 558 | loss: 2.941055 | lr 1.8000e-04 | norm: 0.2139 | dt: 7818.38ms | tok/sec: 67058.4 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 14.6ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.8ms
step 559 | loss: 3.037432 | lr 1.8000e-04 | norm: 0.2266 | dt: 7819.70ms | tok/sec: 67047.1 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 15.4ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 560 | loss: 2.857064 | lr 1.8000e-04 | norm: 0.2111 | dt: 7823.83ms | tok/sec: 67011.7 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 13.9ms | fwd: 2411.5ms | loss: 4.9ms | bwd: 5357.9ms | clip: 0.0ms | optim: 23.7ms
step 561 | loss: 3.106721 | lr 1.8000e-04 | norm: 0.2210 | dt: 7824.46ms | tok/sec: 67006.3 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 13.5ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.6ms
step 562 | loss: 2.915764 | lr 1.8000e-04 | norm: 0.2238 | dt: 7819.48ms | tok/sec: 67049.0 | mem: 72099MB | Time Left: 02 hrs : 21 mins
[TIMING] data: 13.4ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5359.7ms | clip: 0.0ms | optim: 23.6ms
step 563 | loss: 2.196445 | lr 1.8000e-04 | norm: 0.2333 | dt: 7829.63ms | tok/sec: 66962.0 | mem: 72099MB | Time Left: 02 hrs : 21 mins
[TIMING] data: 15.1ms | fwd: 2413.9ms | loss: 4.9ms | bwd: 5360.0ms | clip: 0.0ms | optim: 23.8ms
step 564 | loss: 2.850060 | lr 1.8000e-04 | norm: 0.2220 | dt: 7821.16ms | tok/sec: 67034.5 | mem: 72099MB | Time Left: 02 hrs : 21 mins
[TIMING] data: 15.2ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5358.2ms | clip: 0.0ms | optim: 23.5ms
step 565 | loss: 2.823945 | lr 1.8000e-04 | norm: 0.2284 | dt: 7821.15ms | tok/sec: 67034.6 | mem: 72099MB | Time Left: 02 hrs : 21 mins
[TIMING] data: 14.5ms | fwd: 2412.8ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 566 | loss: 2.751215 | lr 1.8000e-04 | norm: 0.2515 | dt: 7822.11ms | tok/sec: 67026.4 | mem: 72099MB | Time Left: 02 hrs : 21 mins
[TIMING] data: 15.0ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 567 | loss: 2.967774 | lr 1.8000e-04 | norm: 0.2226 | dt: 7815.96ms | tok/sec: 67079.2 | mem: 72099MB | Time Left: 02 hrs : 21 mins
[TIMING] data: 14.6ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.6ms
step 568 | loss: 2.802357 | lr 1.8000e-04 | norm: 0.2961 | dt: 7821.47ms | tok/sec: 67031.9 | mem: 72099MB | Time Left: 02 hrs : 21 mins
[TIMING] data: 15.1ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.6ms
step 569 | loss: 2.674650 | lr 1.8000e-04 | norm: 0.1948 | dt: 7881.67ms | tok/sec: 66519.9 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 14.5ms | fwd: 2470.7ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.6ms
step 570 | loss: 3.031774 | lr 1.8000e-04 | norm: 0.2787 | dt: 7824.23ms | tok/sec: 67008.2 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 15.0ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.5ms
step 571 | loss: 2.752745 | lr 1.8000e-04 | norm: 0.2518 | dt: 7814.59ms | tok/sec: 67090.9 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 16.2ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.7ms
step 572 | loss: 2.996163 | lr 1.8000e-04 | norm: 0.3111 | dt: 7824.52ms | tok/sec: 67005.8 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 13.6ms | fwd: 2411.5ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.7ms
step 573 | loss: 2.816242 | lr 1.8000e-04 | norm: 0.3656 | dt: 7818.40ms | tok/sec: 67058.2 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 13.7ms | fwd: 2402.1ms | loss: 4.9ms | bwd: 5362.0ms | clip: 0.0ms | optim: 23.6ms
step 574 | loss: 3.012659 | lr 1.8000e-04 | norm: 0.3513 | dt: 7822.18ms | tok/sec: 67025.9 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 15.7ms | fwd: 2410.2ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.6ms
step 575 | loss: 2.896825 | lr 1.8000e-04 | norm: 0.4097 | dt: 7814.00ms | tok/sec: 67096.0 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 14.9ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5349.2ms | clip: 0.0ms | optim: 23.6ms
step 576 | loss: 3.086307 | lr 1.8000e-04 | norm: 0.2458 | dt: 7822.17ms | tok/sec: 67025.9 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 20.0ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 24.9ms
step 577 | loss: 2.566905 | lr 1.8000e-04 | norm: 0.2116 | dt: 7813.45ms | tok/sec: 67100.7 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 15.1ms | fwd: 2401.6ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.7ms
step 578 | loss: 2.793840 | lr 1.8000e-04 | norm: 0.2434 | dt: 7882.89ms | tok/sec: 66509.6 | mem: 72099MB | Time Left: 02 hrs : 20 mins
[TIMING] data: 16.0ms | fwd: 2474.5ms | loss: 4.9ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.5ms
step 579 | loss: 2.859135 | lr 1.8000e-04 | norm: 0.2483 | dt: 7952.26ms | tok/sec: 65929.4 | mem: 72099MB | Time Left: 02 hrs : 22 mins
[TIMING] data: 14.5ms | fwd: 2538.6ms | loss: 4.9ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.5ms
step 580 | loss: 2.840033 | lr 1.8000e-04 | norm: 0.3213 | dt: 7808.99ms | tok/sec: 67139.0 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 15.7ms | fwd: 2398.6ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 581 | loss: 2.965203 | lr 1.8000e-04 | norm: 0.2290 | dt: 7820.88ms | tok/sec: 67037.0 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 14.9ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5360.4ms | clip: 0.0ms | optim: 23.5ms
step 582 | loss: 2.740716 | lr 1.8000e-04 | norm: 0.2142 | dt: 7832.45ms | tok/sec: 66938.0 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 12.6ms | fwd: 2415.0ms | loss: 4.9ms | bwd: 5363.3ms | clip: 0.0ms | optim: 23.7ms
step 583 | loss: 3.195622 | lr 1.8000e-04 | norm: 0.3031 | dt: 7819.35ms | tok/sec: 67050.1 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 14.9ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.6ms
step 584 | loss: 2.722642 | lr 1.8000e-04 | norm: 0.2525 | dt: 7821.59ms | tok/sec: 67030.9 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 15.5ms | fwd: 2412.8ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.6ms
step 585 | loss: 2.634157 | lr 1.8000e-04 | norm: 0.2704 | dt: 7828.87ms | tok/sec: 66968.6 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 15.1ms | fwd: 2416.2ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
step 586 | loss: 2.474833 | lr 1.8000e-04 | norm: 0.3005 | dt: 7814.62ms | tok/sec: 67090.6 | mem: 72099MB | Time Left: 02 hrs : 18 mins
[TIMING] data: 13.8ms | fwd: 2404.7ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.6ms
step 587 | loss: 3.084416 | lr 1.8000e-04 | norm: 0.2207 | dt: 7822.48ms | tok/sec: 67023.2 | mem: 72099MB | Time Left: 02 hrs : 18 mins
[TIMING] data: 13.0ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.6ms
step 588 | loss: 2.989316 | lr 1.8000e-04 | norm: 0.2323 | dt: 7823.47ms | tok/sec: 67014.7 | mem: 72099MB | Time Left: 02 hrs : 18 mins
[TIMING] data: 12.3ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.7ms
step 589 | loss: 2.688573 | lr 1.8000e-04 | norm: 0.3064 | dt: 7889.99ms | tok/sec: 66449.8 | mem: 72099MB | Time Left: 02 hrs : 19 mins
[TIMING] data: 15.1ms | fwd: 2481.3ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 590 | loss: 2.741427 | lr 1.8000e-04 | norm: 0.2526 | dt: 7826.49ms | tok/sec: 66988.9 | mem: 72099MB | Time Left: 02 hrs : 18 mins
[TIMING] data: 15.3ms | fwd: 2412.8ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.5ms
step 591 | loss: 2.852728 | lr 1.8000e-04 | norm: 0.2242 | dt: 7812.23ms | tok/sec: 67111.2 | mem: 72099MB | Time Left: 02 hrs : 18 mins
[TIMING] data: 16.0ms | fwd: 2401.1ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 592 | loss: 2.887499 | lr 1.8000e-04 | norm: 0.2505 | dt: 7813.24ms | tok/sec: 67102.5 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 14.9ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 593 | loss: 2.866674 | lr 1.8000e-04 | norm: 0.2344 | dt: 7815.15ms | tok/sec: 67086.1 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 13.3ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.6ms
step 594 | loss: 2.760437 | lr 1.8000e-04 | norm: 0.2599 | dt: 7823.27ms | tok/sec: 67016.5 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 12.4ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5360.9ms | clip: 0.0ms | optim: 23.5ms
step 595 | loss: 2.844915 | lr 1.8000e-04 | norm: 0.2880 | dt: 7820.96ms | tok/sec: 67036.2 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 14.7ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5360.0ms | clip: 0.0ms | optim: 23.5ms
step 596 | loss: 2.969873 | lr 1.8000e-04 | norm: 0.2384 | dt: 7827.33ms | tok/sec: 66981.7 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 14.7ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 597 | loss: 3.068050 | lr 1.8000e-04 | norm: 0.2988 | dt: 7812.75ms | tok/sec: 67106.8 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 14.4ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 598 | loss: 2.769346 | lr 1.8000e-04 | norm: 0.2529 | dt: 7820.13ms | tok/sec: 67043.4 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 13.2ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.7ms
step 599 | loss: 2.960597 | lr 1.8000e-04 | norm: 0.2023 | dt: 7826.48ms | tok/sec: 66989.0 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 14.3ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5363.8ms | clip: 0.0ms | optim: 23.5ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_600.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_400.ckpt
step 600 | loss: 3.163858 | lr 1.8000e-04 | norm: 0.7582 | dt: 7921.30ms | tok/sec: 66187.2 | mem: 72099MB | Time Left: 02 hrs : 18 mins
[TIMING] data: 14.2ms | fwd: 2525.6ms | loss: 4.8ms | bwd: 5342.7ms | clip: 0.0ms | optim: 23.5ms
step 601 | loss: 3.219712 | lr 1.8000e-04 | norm: 0.4506 | dt: 7801.25ms | tok/sec: 67205.7 | mem: 72099MB | Time Left: 02 hrs : 16 mins
[TIMING] data: 15.3ms | fwd: 2397.1ms | loss: 4.8ms | bwd: 5349.6ms | clip: 0.0ms | optim: 23.4ms
step 602 | loss: 3.125331 | lr 1.8000e-04 | norm: 0.3209 | dt: 7810.76ms | tok/sec: 67123.8 | mem: 72099MB | Time Left: 02 hrs : 16 mins
[TIMING] data: 18.6ms | fwd: 2400.2ms | loss: 4.8ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 603 | loss: 2.872280 | lr 1.8000e-04 | norm: 0.2383 | dt: 7844.61ms | tok/sec: 66834.2 | mem: 72099MB | Time Left: 02 hrs : 17 mins
[TIMING] data: 46.9ms | fwd: 2403.4ms | loss: 4.8ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 604 | loss: 3.426134 | lr 1.8000e-04 | norm: 0.2471 | dt: 7832.37ms | tok/sec: 66938.6 | mem: 72099MB | Time Left: 02 hrs : 16 mins
[TIMING] data: 37.1ms | fwd: 2402.1ms | loss: 4.8ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 605 | loss: 3.063794 | lr 1.8000e-04 | norm: 0.2127 | dt: 7822.49ms | tok/sec: 67023.2 | mem: 72099MB | Time Left: 02 hrs : 16 mins
[TIMING] data: 19.9ms | fwd: 2409.3ms | loss: 4.8ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 606 | loss: 2.765444 | lr 1.8000e-04 | norm: 0.2116 | dt: 7808.48ms | tok/sec: 67143.4 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 16.1ms | fwd: 2401.3ms | loss: 4.8ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
step 607 | loss: 2.809398 | lr 1.8000e-04 | norm: 0.2366 | dt: 7800.78ms | tok/sec: 67209.7 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 14.7ms | fwd: 2400.7ms | loss: 4.8ms | bwd: 5345.9ms | clip: 0.0ms | optim: 23.5ms
step 608 | loss: 2.885848 | lr 1.8000e-04 | norm: 0.2367 | dt: 7814.32ms | tok/sec: 67093.2 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 12.8ms | fwd: 2414.4ms | loss: 4.8ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.5ms
step 609 | loss: 2.907620 | lr 1.8000e-04 | norm: 0.2667 | dt: 7884.58ms | tok/sec: 66495.3 | mem: 72099MB | Time Left: 02 hrs : 16 mins
[TIMING] data: 14.5ms | fwd: 2478.3ms | loss: 4.8ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 610 | loss: 2.520245 | lr 1.8000e-04 | norm: 0.2536 | dt: 7807.37ms | tok/sec: 67152.9 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 15.8ms | fwd: 2404.7ms | loss: 4.8ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.5ms
step 611 | loss: 2.828954 | lr 1.8000e-04 | norm: 0.2904 | dt: 7891.71ms | tok/sec: 66435.3 | mem: 72099MB | Time Left: 02 hrs : 16 mins
[TIMING] data: 13.9ms | fwd: 2481.8ms | loss: 4.8ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.4ms
step 612 | loss: 2.799252 | lr 1.8000e-04 | norm: 0.2943 | dt: 7815.62ms | tok/sec: 67082.1 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 13.3ms | fwd: 2409.9ms | loss: 4.8ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.5ms
step 613 | loss: 2.819131 | lr 1.8000e-04 | norm: 0.2125 | dt: 7814.86ms | tok/sec: 67088.6 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 14.8ms | fwd: 2412.4ms | loss: 4.8ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.5ms
step 614 | loss: 3.303657 | lr 1.8000e-04 | norm: 1.0566 | dt: 7814.30ms | tok/sec: 67093.4 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 15.9ms | fwd: 2408.6ms | loss: 4.8ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.6ms
step 615 | loss: 2.927806 | lr 1.8000e-04 | norm: 0.3390 | dt: 7811.97ms | tok/sec: 67113.4 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 16.1ms | fwd: 2402.9ms | loss: 4.8ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 616 | loss: 2.785054 | lr 1.8000e-04 | norm: 0.4190 | dt: 7829.13ms | tok/sec: 66966.4 | mem: 72099MB | Time Left: 02 hrs : 15 mins
[TIMING] data: 18.3ms | fwd: 2414.5ms | loss: 4.8ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 617 | loss: 2.905208 | lr 1.8000e-04 | norm: 0.3193 | dt: 7820.16ms | tok/sec: 67043.2 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 12.2ms | fwd: 2411.4ms | loss: 4.8ms | bwd: 5358.7ms | clip: 0.0ms | optim: 23.6ms
step 618 | loss: 3.009234 | lr 1.8000e-04 | norm: 0.2754 | dt: 7823.51ms | tok/sec: 67014.4 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 15.3ms | fwd: 2410.0ms | loss: 4.8ms | bwd: 5359.6ms | clip: 0.0ms | optim: 23.6ms
step 619 | loss: 2.828272 | lr 1.8000e-04 | norm: 0.2321 | dt: 7815.05ms | tok/sec: 67087.0 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 15.6ms | fwd: 2410.3ms | loss: 4.8ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.6ms
step 620 | loss: 2.761901 | lr 1.8000e-04 | norm: 0.3018 | dt: 7808.08ms | tok/sec: 67146.8 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 14.7ms | fwd: 2400.4ms | loss: 4.8ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 621 | loss: 2.549311 | lr 1.8000e-04 | norm: 0.2984 | dt: 7821.04ms | tok/sec: 67035.6 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 14.5ms | fwd: 2410.4ms | loss: 4.8ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.6ms
step 622 | loss: 2.893700 | lr 1.8000e-04 | norm: 0.2523 | dt: 7814.83ms | tok/sec: 67088.9 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 15.7ms | fwd: 2409.6ms | loss: 4.8ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.6ms
step 623 | loss: 2.368857 | lr 1.8000e-04 | norm: 0.2830 | dt: 7821.58ms | tok/sec: 67031.0 | mem: 72099MB | Time Left: 02 hrs : 14 mins
[TIMING] data: 14.8ms | fwd: 2413.1ms | loss: 4.8ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.7ms
step 624 | loss: 2.807893 | lr 1.8000e-04 | norm: 0.2370 | dt: 7814.82ms | tok/sec: 67088.9 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 16.3ms | fwd: 2407.6ms | loss: 4.8ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 625 | loss: 2.864594 | lr 1.8000e-04 | norm: 0.2453 | dt: 7820.23ms | tok/sec: 67042.6 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 15.4ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5356.6ms | clip: 0.0ms | optim: 24.2ms
step 626 | loss: 2.917055 | lr 1.8000e-04 | norm: 0.3392 | dt: 7821.85ms | tok/sec: 67028.6 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 13.7ms | fwd: 2409.9ms | loss: 4.8ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.5ms
step 627 | loss: 2.997999 | lr 1.8000e-04 | norm: 0.2198 | dt: 7823.24ms | tok/sec: 67016.7 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 14.4ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.5ms
step 628 | loss: 2.946014 | lr 1.8000e-04 | norm: 0.2516 | dt: 7824.25ms | tok/sec: 67008.1 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 16.2ms | fwd: 2413.3ms | loss: 4.8ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.7ms
step 629 | loss: 2.807717 | lr 1.8000e-04 | norm: 0.2554 | dt: 7813.09ms | tok/sec: 67103.8 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 15.8ms | fwd: 2407.4ms | loss: 4.8ms | bwd: 5351.0ms | clip: 0.0ms | optim: 23.7ms
step 630 | loss: 2.619642 | lr 1.8000e-04 | norm: 0.2925 | dt: 7824.09ms | tok/sec: 67009.4 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 15.7ms | fwd: 2413.7ms | loss: 4.8ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.6ms
step 631 | loss: 2.520890 | lr 1.8000e-04 | norm: 0.2221 | dt: 7818.83ms | tok/sec: 67054.5 | mem: 72099MB | Time Left: 02 hrs : 12 mins
[TIMING] data: 15.3ms | fwd: 2407.9ms | loss: 4.8ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.5ms
step 632 | loss: 2.606897 | lr 1.8000e-04 | norm: 0.2207 | dt: 7818.66ms | tok/sec: 67056.0 | mem: 72099MB | Time Left: 02 hrs : 12 mins
[TIMING] data: 15.5ms | fwd: 2405.5ms | loss: 4.8ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 633 | loss: 2.776877 | lr 1.8000e-04 | norm: 0.2161 | dt: 7883.76ms | tok/sec: 66502.3 | mem: 72099MB | Time Left: 02 hrs : 13 mins
[TIMING] data: 14.7ms | fwd: 2476.3ms | loss: 4.8ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 634 | loss: 2.825418 | lr 1.8000e-04 | norm: 0.2667 | dt: 7814.16ms | tok/sec: 67094.6 | mem: 72099MB | Time Left: 02 hrs : 12 mins
[TIMING] data: 16.0ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.5ms
step 635 | loss: 2.718092 | lr 1.8000e-04 | norm: 0.2426 | dt: 7812.33ms | tok/sec: 67110.3 | mem: 72099MB | Time Left: 02 hrs : 12 mins
[TIMING] data: 14.4ms | fwd: 2409.1ms | loss: 4.8ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.6ms
step 636 | loss: 2.522713 | lr 1.8000e-04 | norm: 0.1927 | dt: 7817.86ms | tok/sec: 67062.9 | mem: 72099MB | Time Left: 02 hrs : 12 mins
[TIMING] data: 15.0ms | fwd: 2408.9ms | loss: 4.8ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.6ms
step 637 | loss: 2.667505 | lr 1.8000e-04 | norm: 0.2977 | dt: 7819.54ms | tok/sec: 67048.5 | mem: 72099MB | Time Left: 02 hrs : 12 mins
[TIMING] data: 14.6ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.7ms
step 638 | loss: 2.891452 | lr 1.8000e-04 | norm: 0.2283 | dt: 7816.24ms | tok/sec: 67076.7 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 14.4ms | fwd: 2410.4ms | loss: 4.8ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 639 | loss: 2.821642 | lr 1.8000e-04 | norm: 0.2695 | dt: 7820.24ms | tok/sec: 67042.4 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 17.1ms | fwd: 2409.2ms | loss: 4.8ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.9ms
step 640 | loss: 3.001255 | lr 1.8000e-04 | norm: 0.2277 | dt: 7814.31ms | tok/sec: 67093.4 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 14.7ms | fwd: 2411.5ms | loss: 4.8ms | bwd: 5349.1ms | clip: 0.0ms | optim: 23.6ms
step 641 | loss: 3.017979 | lr 1.8000e-04 | norm: 0.2310 | dt: 7819.05ms | tok/sec: 67052.7 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 15.0ms | fwd: 2407.9ms | loss: 4.8ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.6ms
step 642 | loss: 2.851364 | lr 1.8000e-04 | norm: 0.2411 | dt: 7807.31ms | tok/sec: 67153.5 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 16.5ms | fwd: 2403.0ms | loss: 4.8ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.6ms
step 643 | loss: 2.880984 | lr 1.8000e-04 | norm: 0.2822 | dt: 7806.05ms | tok/sec: 67164.3 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 14.9ms | fwd: 2403.3ms | loss: 4.8ms | bwd: 5349.3ms | clip: 0.0ms | optim: 23.5ms
step 644 | loss: 3.064970 | lr 1.8000e-04 | norm: 0.2433 | dt: 7809.70ms | tok/sec: 67132.9 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 15.5ms | fwd: 2405.3ms | loss: 4.8ms | bwd: 5349.3ms | clip: 0.0ms | optim: 23.5ms
step 645 | loss: 3.026419 | lr 1.8000e-04 | norm: 0.2241 | dt: 7821.88ms | tok/sec: 67028.4 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 15.2ms | fwd: 2414.8ms | loss: 4.8ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
step 646 | loss: 2.928844 | lr 1.8000e-04 | norm: 0.2823 | dt: 7815.24ms | tok/sec: 67085.3 | mem: 72099MB | Time Left: 02 hrs : 10 mins
[TIMING] data: 13.7ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 647 | loss: 2.881264 | lr 1.8000e-04 | norm: 0.3731 | dt: 7825.44ms | tok/sec: 66997.9 | mem: 72099MB | Time Left: 02 hrs : 10 mins
[TIMING] data: 20.7ms | fwd: 2410.7ms | loss: 4.8ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.6ms
step 648 | loss: 2.921669 | lr 1.8000e-04 | norm: 0.2163 | dt: 7824.45ms | tok/sec: 67006.4 | mem: 72099MB | Time Left: 02 hrs : 10 mins
[TIMING] data: 12.9ms | fwd: 2406.9ms | loss: 4.8ms | bwd: 5365.2ms | clip: 0.0ms | optim: 23.5ms
step 649 | loss: 2.566304 | lr 1.8000e-04 | norm: 0.2239 | dt: 7813.37ms | tok/sec: 67101.4 | mem: 72099MB | Time Left: 02 hrs : 10 mins
[TIMING] data: 17.4ms | fwd: 2407.5ms | loss: 4.8ms | bwd: 5348.9ms | clip: 0.0ms | optim: 23.7ms
step 650 | loss: 2.700313 | lr 1.8000e-04 | norm: 0.2064 | dt: 7829.83ms | tok/sec: 66960.3 | mem: 72099MB | Time Left: 02 hrs : 10 mins
[TIMING] data: 15.7ms | fwd: 2405.5ms | loss: 4.8ms | bwd: 5359.0ms | clip: 0.0ms | optim: 23.6ms
step 651 | loss: 2.897596 | lr 1.8000e-04 | norm: 0.2857 | dt: 7902.98ms | tok/sec: 66340.5 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 17.5ms | fwd: 2496.4ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.6ms
step 652 | loss: 2.831638 | lr 1.8000e-04 | norm: 0.2065 | dt: 7882.88ms | tok/sec: 66509.7 | mem: 72099MB | Time Left: 02 hrs : 11 mins
[TIMING] data: 14.2ms | fwd: 2479.2ms | loss: 4.8ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.7ms
step 653 | loss: 3.084590 | lr 1.8000e-04 | norm: 0.5307 | dt: 7817.50ms | tok/sec: 67065.9 | mem: 72099MB | Time Left: 02 hrs : 10 mins
[TIMING] data: 15.3ms | fwd: 2406.6ms | loss: 4.8ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.5ms
step 654 | loss: 2.956929 | lr 1.8000e-04 | norm: 0.3030 | dt: 7810.70ms | tok/sec: 67124.3 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 18.6ms | fwd: 2405.3ms | loss: 4.8ms | bwd: 5348.6ms | clip: 0.0ms | optim: 23.6ms
step 655 | loss: 2.810403 | lr 1.8000e-04 | norm: 0.3352 | dt: 7814.77ms | tok/sec: 67089.4 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 14.1ms | fwd: 2406.4ms | loss: 4.8ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 656 | loss: 2.764901 | lr 1.8000e-04 | norm: 0.3673 | dt: 7819.60ms | tok/sec: 67047.9 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 16.7ms | fwd: 2412.7ms | loss: 4.8ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.4ms
step 657 | loss: 3.045187 | lr 1.8000e-04 | norm: 0.2696 | dt: 7818.91ms | tok/sec: 67053.9 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 17.9ms | fwd: 2407.8ms | loss: 4.8ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 658 | loss: 2.889917 | lr 1.8000e-04 | norm: 0.2229 | dt: 7834.71ms | tok/sec: 66918.6 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 18.1ms | fwd: 2417.4ms | loss: 4.8ms | bwd: 5360.6ms | clip: 0.0ms | optim: 23.6ms
step 659 | loss: 2.853548 | lr 1.8000e-04 | norm: 0.2434 | dt: 7817.07ms | tok/sec: 67069.6 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 15.9ms | fwd: 2411.9ms | loss: 4.8ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.7ms
step 660 | loss: 2.615963 | lr 1.8000e-04 | norm: 0.2548 | dt: 7822.63ms | tok/sec: 67022.0 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 13.3ms | fwd: 2413.9ms | loss: 4.8ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 661 | loss: 2.623489 | lr 1.8000e-04 | norm: 0.1980 | dt: 7818.63ms | tok/sec: 67056.2 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 13.9ms | fwd: 2412.3ms | loss: 4.8ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.5ms
step 662 | loss: 3.052288 | lr 1.8000e-04 | norm: 0.3346 | dt: 7815.70ms | tok/sec: 67081.4 | mem: 72099MB | Time Left: 02 hrs : 08 mins
[TIMING] data: 18.6ms | fwd: 2404.1ms | loss: 4.8ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.8ms
step 663 | loss: 2.958857 | lr 1.8000e-04 | norm: 0.2787 | dt: 7875.67ms | tok/sec: 66570.6 | mem: 72099MB | Time Left: 02 hrs : 09 mins
[TIMING] data: 15.1ms | fwd: 2466.6ms | loss: 4.8ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.5ms
step 664 | loss: 2.701391 | lr 1.8000e-04 | norm: 0.2237 | dt: 7828.23ms | tok/sec: 66974.0 | mem: 72099MB | Time Left: 02 hrs : 08 mins
[TIMING] data: 14.9ms | fwd: 2422.5ms | loss: 4.8ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.7ms
step 665 | loss: 2.535341 | lr 1.8000e-04 | norm: 0.1952 | dt: 7821.25ms | tok/sec: 67033.8 | mem: 72099MB | Time Left: 02 hrs : 08 mins
[TIMING] data: 14.0ms | fwd: 2410.3ms | loss: 4.8ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.5ms
step 666 | loss: 2.943307 | lr 1.8000e-04 | norm: 0.7677 | dt: 7817.40ms | tok/sec: 67066.8 | mem: 72099MB | Time Left: 02 hrs : 08 mins
[TIMING] data: 16.4ms | fwd: 2411.9ms | loss: 4.8ms | bwd: 5350.0ms | clip: 0.0ms | optim: 24.0ms
step 667 | loss: 2.908436 | lr 1.8000e-04 | norm: 0.2137 | dt: 7813.36ms | tok/sec: 67101.5 | mem: 72099MB | Time Left: 02 hrs : 08 mins
[TIMING] data: 13.9ms | fwd: 2409.1ms | loss: 4.8ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.5ms
step 668 | loss: 2.856768 | lr 1.8000e-04 | norm: 0.2270 | dt: 7814.13ms | tok/sec: 67094.9 | mem: 72099MB | Time Left: 02 hrs : 08 mins
[TIMING] data: 14.8ms | fwd: 2406.8ms | loss: 4.8ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.6ms
step 669 | loss: 2.753603 | lr 1.8000e-04 | norm: 0.2445 | dt: 7812.84ms | tok/sec: 67106.0 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 17.4ms | fwd: 2401.0ms | loss: 4.8ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 670 | loss: 2.634752 | lr 1.8000e-04 | norm: 0.2097 | dt: 7816.86ms | tok/sec: 67071.4 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 14.3ms | fwd: 2409.1ms | loss: 4.8ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 671 | loss: 3.118633 | lr 1.8000e-04 | norm: 0.2331 | dt: 7815.70ms | tok/sec: 67081.4 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 13.8ms | fwd: 2405.1ms | loss: 4.8ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.6ms
step 672 | loss: 2.917928 | lr 1.8000e-04 | norm: 0.3257 | dt: 7821.04ms | tok/sec: 67035.6 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 13.6ms | fwd: 2414.5ms | loss: 4.8ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.5ms
step 673 | loss: 2.291845 | lr 1.8000e-04 | norm: 0.2883 | dt: 7823.86ms | tok/sec: 67011.4 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 19.9ms | fwd: 2410.6ms | loss: 4.8ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 674 | loss: 2.603871 | lr 1.8000e-04 | norm: 0.2349 | dt: 7817.51ms | tok/sec: 67065.9 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 14.6ms | fwd: 2409.1ms | loss: 4.8ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 675 | loss: 2.923642 | lr 1.8000e-04 | norm: 0.2725 | dt: 7827.22ms | tok/sec: 66982.7 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 25.5ms | fwd: 2410.6ms | loss: 4.8ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.6ms
step 676 | loss: 2.810585 | lr 1.8000e-04 | norm: 0.2585 | dt: 7818.83ms | tok/sec: 67054.5 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 15.5ms | fwd: 2411.9ms | loss: 4.8ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 677 | loss: 2.762619 | lr 1.8000e-04 | norm: 0.2591 | dt: 7814.08ms | tok/sec: 67095.3 | mem: 72099MB | Time Left: 02 hrs : 06 mins
[TIMING] data: 15.5ms | fwd: 2407.3ms | loss: 4.8ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.6ms
step 678 | loss: 3.116189 | lr 1.8000e-04 | norm: 0.2637 | dt: 7813.65ms | tok/sec: 67099.0 | mem: 72099MB | Time Left: 02 hrs : 06 mins
[TIMING] data: 15.1ms | fwd: 2408.1ms | loss: 4.8ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.6ms
step 679 | loss: 2.390558 | lr 1.8000e-04 | norm: 0.2328 | dt: 7855.90ms | tok/sec: 66738.1 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 13.1ms | fwd: 2450.8ms | loss: 4.8ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.6ms
step 680 | loss: 2.866865 | lr 1.8000e-04 | norm: 0.2330 | dt: 7821.16ms | tok/sec: 67034.6 | mem: 72099MB | Time Left: 02 hrs : 06 mins
[TIMING] data: 15.4ms | fwd: 2412.5ms | loss: 4.8ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.5ms
step 681 | loss: 2.998008 | lr 1.8000e-04 | norm: 0.2870 | dt: 7816.91ms | tok/sec: 67071.0 | mem: 72099MB | Time Left: 02 hrs : 06 mins
[TIMING] data: 13.7ms | fwd: 2406.2ms | loss: 4.8ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.7ms
step 682 | loss: 2.918557 | lr 1.8000e-04 | norm: 0.2094 | dt: 7823.30ms | tok/sec: 67016.3 | mem: 72099MB | Time Left: 02 hrs : 06 mins
[TIMING] data: 14.9ms | fwd: 2410.9ms | loss: 4.8ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.6ms
step 683 | loss: 2.984234 | lr 1.8000e-04 | norm: 0.2226 | dt: 7812.90ms | tok/sec: 67105.4 | mem: 72099MB | Time Left: 02 hrs : 06 mins
[TIMING] data: 13.9ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.5ms
step 684 | loss: 2.924987 | lr 1.8000e-04 | norm: 0.2233 | dt: 7817.85ms | tok/sec: 67062.9 | mem: 72099MB | Time Left: 02 hrs : 05 mins
[TIMING] data: 15.0ms | fwd: 2410.4ms | loss: 4.8ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.7ms
step 685 | loss: 2.591628 | lr 1.8000e-04 | norm: 0.3057 | dt: 7818.24ms | tok/sec: 67059.6 | mem: 72099MB | Time Left: 02 hrs : 05 mins
[TIMING] data: 13.6ms | fwd: 2411.1ms | loss: 4.8ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 686 | loss: 2.818563 | lr 1.8000e-04 | norm: 0.2211 | dt: 7930.31ms | tok/sec: 66111.9 | mem: 72099MB | Time Left: 02 hrs : 07 mins
[TIMING] data: 69.7ms | fwd: 2468.0ms | loss: 4.8ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 687 | loss: 2.734948 | lr 1.8000e-04 | norm: 0.3018 | dt: 7813.46ms | tok/sec: 67100.6 | mem: 72099MB | Time Left: 02 hrs : 05 mins
[TIMING] data: 14.6ms | fwd: 2406.0ms | loss: 4.8ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 688 | loss: 2.738410 | lr 1.8000e-04 | norm: 0.2030 | dt: 7810.49ms | tok/sec: 67126.1 | mem: 72099MB | Time Left: 02 hrs : 05 mins
[TIMING] data: 13.9ms | fwd: 2406.0ms | loss: 4.8ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.7ms
step 689 | loss: 2.971666 | lr 1.8000e-04 | norm: 0.2721 | dt: 7823.18ms | tok/sec: 67017.3 | mem: 72099MB | Time Left: 02 hrs : 05 mins
[TIMING] data: 18.0ms | fwd: 2414.4ms | loss: 4.8ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 690 | loss: 2.881315 | lr 1.8000e-04 | norm: 0.2529 | dt: 7813.91ms | tok/sec: 67096.8 | mem: 72099MB | Time Left: 02 hrs : 05 mins
[TIMING] data: 13.1ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.7ms
step 691 | loss: 2.760863 | lr 1.8000e-04 | norm: 0.2592 | dt: 7852.97ms | tok/sec: 66763.1 | mem: 72099MB | Time Left: 02 hrs : 05 mins
[TIMING] data: 14.9ms | fwd: 2448.3ms | loss: 4.8ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.5ms
step 692 | loss: 2.725524 | lr 1.8000e-04 | norm: 0.2168 | dt: 7810.97ms | tok/sec: 67122.0 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 14.1ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.4ms
step 693 | loss: 2.650761 | lr 1.8000e-04 | norm: 0.2167 | dt: 7811.58ms | tok/sec: 67116.7 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 13.4ms | fwd: 2409.0ms | loss: 4.8ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.6ms
step 694 | loss: 2.902795 | lr 1.8000e-04 | norm: 0.2291 | dt: 7814.72ms | tok/sec: 67089.8 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 13.2ms | fwd: 2409.3ms | loss: 4.8ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.6ms
step 695 | loss: 3.131786 | lr 1.8000e-04 | norm: 0.2409 | dt: 7811.83ms | tok/sec: 67114.6 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 14.6ms | fwd: 2407.8ms | loss: 4.8ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.6ms
step 696 | loss: 2.822197 | lr 1.8000e-04 | norm: 0.2453 | dt: 7824.62ms | tok/sec: 67004.9 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 14.5ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.7ms
step 697 | loss: 2.894160 | lr 1.8000e-04 | norm: 0.2300 | dt: 7817.73ms | tok/sec: 67064.0 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 14.3ms | fwd: 2403.1ms | loss: 4.8ms | bwd: 5360.9ms | clip: 0.0ms | optim: 23.6ms
step 698 | loss: 3.184533 | lr 1.8000e-04 | norm: 0.2133 | dt: 7816.47ms | tok/sec: 67074.8 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 15.6ms | fwd: 2410.7ms | loss: 4.8ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.6ms
step 699 | loss: 3.066521 | lr 1.8000e-04 | norm: 0.2260 | dt: 7805.59ms | tok/sec: 67168.3 | mem: 72099MB | Time Left: 02 hrs : 03 mins
[TIMING] data: 13.2ms | fwd: 2402.9ms | loss: 4.8ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.5ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_700.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_500.ckpt
step 700 | loss: 2.687369 | lr 1.8000e-04 | norm: 0.2558 | dt: 7833.35ms | tok/sec: 66930.3 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 11.6ms | fwd: 2441.2ms | loss: 4.9ms | bwd: 5340.7ms | clip: 0.0ms | optim: 23.4ms
step 701 | loss: 2.757375 | lr 1.8000e-04 | norm: 0.1971 | dt: 7802.09ms | tok/sec: 67198.4 | mem: 72099MB | Time Left: 02 hrs : 03 mins
[TIMING] data: 11.6ms | fwd: 2400.6ms | loss: 4.9ms | bwd: 5350.0ms | clip: 0.0ms | optim: 23.5ms
step 702 | loss: 2.828192 | lr 1.8000e-04 | norm: 0.1892 | dt: 7799.92ms | tok/sec: 67217.1 | mem: 72099MB | Time Left: 02 hrs : 03 mins
[TIMING] data: 14.9ms | fwd: 2401.7ms | loss: 4.9ms | bwd: 5342.5ms | clip: 0.0ms | optim: 23.6ms
step 703 | loss: 3.061823 | lr 1.8000e-04 | norm: 0.2357 | dt: 7808.24ms | tok/sec: 67145.5 | mem: 72099MB | Time Left: 02 hrs : 03 mins
[TIMING] data: 23.2ms | fwd: 2398.6ms | loss: 4.9ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.5ms
step 704 | loss: 2.884289 | lr 1.8000e-04 | norm: 0.2221 | dt: 7864.50ms | tok/sec: 66665.2 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 82.7ms | fwd: 2395.9ms | loss: 4.9ms | bwd: 5345.1ms | clip: 0.0ms | optim: 23.7ms
step 705 | loss: 2.881665 | lr 1.8000e-04 | norm: 0.2589 | dt: 7885.64ms | tok/sec: 66486.4 | mem: 72099MB | Time Left: 02 hrs : 04 mins
[TIMING] data: 25.7ms | fwd: 2469.4ms | loss: 4.9ms | bwd: 5350.3ms | clip: 0.0ms | optim: 23.5ms
step 706 | loss: 2.799549 | lr 1.8000e-04 | norm: 0.2107 | dt: 7812.73ms | tok/sec: 67106.9 | mem: 72099MB | Time Left: 02 hrs : 03 mins
[TIMING] data: 14.4ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5347.8ms | clip: 0.0ms | optim: 23.5ms
step 707 | loss: 3.001076 | lr 1.8000e-04 | norm: 0.2071 | dt: 7809.67ms | tok/sec: 67133.2 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 14.1ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5348.9ms | clip: 0.0ms | optim: 23.5ms
step 708 | loss: 2.843807 | lr 1.8000e-04 | norm: 0.2064 | dt: 7818.29ms | tok/sec: 67059.1 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 14.6ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.5ms
step 709 | loss: 2.747122 | lr 1.8000e-04 | norm: 0.2123 | dt: 7808.17ms | tok/sec: 67146.1 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 14.5ms | fwd: 2402.5ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.7ms
step 710 | loss: 3.212272 | lr 1.8000e-04 | norm: 0.2450 | dt: 7836.68ms | tok/sec: 66901.8 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 32.8ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.6ms
step 711 | loss: 3.064130 | lr 1.8000e-04 | norm: 0.2690 | dt: 7815.78ms | tok/sec: 67080.7 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 14.1ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.6ms
step 712 | loss: 2.958218 | lr 1.8000e-04 | norm: 0.2487 | dt: 7821.49ms | tok/sec: 67031.7 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 16.1ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5358.4ms | clip: 0.0ms | optim: 23.6ms
step 713 | loss: 3.013741 | lr 1.8000e-04 | norm: 0.2088 | dt: 7813.10ms | tok/sec: 67103.7 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 14.5ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 714 | loss: 2.783258 | lr 1.8000e-04 | norm: 0.2255 | dt: 7819.03ms | tok/sec: 67052.9 | mem: 72099MB | Time Left: 02 hrs : 02 mins
[TIMING] data: 20.0ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 715 | loss: 2.986906 | lr 1.8000e-04 | norm: 0.2062 | dt: 7814.14ms | tok/sec: 67094.8 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 14.7ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 716 | loss: 2.707292 | lr 1.8000e-04 | norm: 0.2040 | dt: 7828.17ms | tok/sec: 66974.5 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 14.8ms | fwd: 2414.5ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.6ms
step 717 | loss: 3.022250 | lr 1.8000e-04 | norm: 0.2521 | dt: 7812.46ms | tok/sec: 67109.2 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 15.7ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.6ms
step 718 | loss: 2.493308 | lr 1.8000e-04 | norm: 0.2288 | dt: 7819.98ms | tok/sec: 67044.7 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 16.6ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 719 | loss: 3.126472 | lr 1.8000e-04 | norm: 0.2926 | dt: 7810.59ms | tok/sec: 67125.3 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 16.7ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5347.2ms | clip: 0.0ms | optim: 23.6ms
step 720 | loss: 2.923380 | lr 1.8000e-04 | norm: 0.2089 | dt: 7823.86ms | tok/sec: 67011.4 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 18.0ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.6ms
step 721 | loss: 2.996290 | lr 1.8000e-04 | norm: 0.2590 | dt: 7823.35ms | tok/sec: 67015.8 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 14.7ms | fwd: 2408.5ms | loss: 4.9ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.6ms
step 722 | loss: 3.065418 | lr 1.8000e-04 | norm: 0.2042 | dt: 7818.57ms | tok/sec: 67056.8 | mem: 72099MB | Time Left: 02 hrs : 01 mins
[TIMING] data: 19.1ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 723 | loss: 3.039548 | lr 1.8000e-04 | norm: 0.2330 | dt: 7814.34ms | tok/sec: 67093.1 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 15.4ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.5ms
step 724 | loss: 3.183644 | lr 1.8000e-04 | norm: 0.2351 | dt: 7817.22ms | tok/sec: 67068.3 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 16.3ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 725 | loss: 2.869721 | lr 1.8000e-04 | norm: 0.2361 | dt: 7808.67ms | tok/sec: 67141.8 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 13.9ms | fwd: 2400.2ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 726 | loss: 2.895711 | lr 1.8000e-04 | norm: 0.2268 | dt: 7824.55ms | tok/sec: 67005.5 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 17.9ms | fwd: 2411.8ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 727 | loss: 2.921812 | lr 1.8000e-04 | norm: 0.2477 | dt: 7822.72ms | tok/sec: 67021.2 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 15.5ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 728 | loss: 3.027908 | lr 1.8000e-04 | norm: 0.2763 | dt: 7819.34ms | tok/sec: 67050.2 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 17.5ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.6ms
step 729 | loss: 3.142056 | lr 1.8000e-04 | norm: 0.2407 | dt: 7810.74ms | tok/sec: 67124.0 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 14.8ms | fwd: 2401.3ms | loss: 4.9ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.6ms
step 730 | loss: 2.597941 | lr 1.8000e-04 | norm: 0.2307 | dt: 7819.15ms | tok/sec: 67051.8 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 14.3ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 731 | loss: 2.859463 | lr 1.8000e-04 | norm: 0.2397 | dt: 7818.32ms | tok/sec: 67058.9 | mem: 72099MB | Time Left: 01 hrs : 59 mins
[TIMING] data: 14.6ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.4ms
step 732 | loss: 2.941045 | lr 1.8000e-04 | norm: 0.2298 | dt: 7823.88ms | tok/sec: 67011.2 | mem: 72099MB | Time Left: 01 hrs : 59 mins
[TIMING] data: 15.5ms | fwd: 2410.2ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.6ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00004.bin
100000000
step 733 | loss: 3.079350 | lr 1.8000e-04 | norm: 0.2156 | dt: 7835.29ms | tok/sec: 66913.7 | mem: 72099MB | Time Left: 01 hrs : 59 mins
[TIMING] data: 32.4ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.6ms
step 734 | loss: 2.933001 | lr 1.8000e-04 | norm: 0.3317 | dt: 7812.87ms | tok/sec: 67105.7 | mem: 72099MB | Time Left: 01 hrs : 59 mins
[TIMING] data: 15.3ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 735 | loss: 2.786656 | lr 1.8000e-04 | norm: 0.2488 | dt: 7822.55ms | tok/sec: 67022.6 | mem: 72099MB | Time Left: 01 hrs : 59 mins
[TIMING] data: 19.7ms | fwd: 2409.9ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.6ms
step 736 | loss: 3.062197 | lr 1.8000e-04 | norm: 0.2886 | dt: 7810.82ms | tok/sec: 67123.3 | mem: 72099MB | Time Left: 01 hrs : 59 mins
[TIMING] data: 15.1ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5346.0ms | clip: 0.0ms | optim: 23.6ms
step 737 | loss: 2.776660 | lr 1.8000e-04 | norm: 0.2946 | dt: 7888.69ms | tok/sec: 66460.7 | mem: 72099MB | Time Left: 02 hrs : 00 mins
[TIMING] data: 15.7ms | fwd: 2481.0ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.5ms
step 738 | loss: 3.125713 | lr 1.8000e-04 | norm: 0.2322 | dt: 7818.04ms | tok/sec: 67061.3 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 14.6ms | fwd: 2402.5ms | loss: 4.9ms | bwd: 5360.1ms | clip: 0.0ms | optim: 23.6ms
step 739 | loss: 2.955600 | lr 1.8000e-04 | norm: 0.2258 | dt: 7820.90ms | tok/sec: 67036.8 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 14.1ms | fwd: 2415.0ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.6ms
step 740 | loss: 2.959178 | lr 1.8000e-04 | norm: 0.2397 | dt: 7823.43ms | tok/sec: 67015.1 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 14.1ms | fwd: 2410.6ms | loss: 5.0ms | bwd: 5358.2ms | clip: 0.0ms | optim: 23.6ms
step 741 | loss: 2.957743 | lr 1.8000e-04 | norm: 0.2764 | dt: 7830.72ms | tok/sec: 66952.7 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 16.8ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5361.5ms | clip: 0.0ms | optim: 23.6ms
step 742 | loss: 2.918139 | lr 1.8000e-04 | norm: 0.2149 | dt: 7815.70ms | tok/sec: 67081.4 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 13.8ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 743 | loss: 2.819252 | lr 1.8000e-04 | norm: 0.1900 | dt: 7822.17ms | tok/sec: 67025.9 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 17.0ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.5ms
step 744 | loss: 3.097696 | lr 1.8000e-04 | norm: 0.2304 | dt: 7817.37ms | tok/sec: 67067.1 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 16.7ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.6ms
step 745 | loss: 2.814520 | lr 1.8000e-04 | norm: 0.2219 | dt: 7816.75ms | tok/sec: 67072.3 | mem: 72099MB | Time Left: 01 hrs : 58 mins
[TIMING] data: 16.9ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 746 | loss: 3.048714 | lr 1.8000e-04 | norm: 0.2862 | dt: 7818.29ms | tok/sec: 67059.1 | mem: 72099MB | Time Left: 01 hrs : 57 mins
[TIMING] data: 14.0ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 747 | loss: 3.012232 | lr 1.8000e-04 | norm: 0.3154 | dt: 7817.08ms | tok/sec: 67069.5 | mem: 72099MB | Time Left: 01 hrs : 57 mins
[TIMING] data: 16.0ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 748 | loss: 2.859267 | lr 1.8000e-04 | norm: 0.5008 | dt: 7814.86ms | tok/sec: 67088.6 | mem: 72099MB | Time Left: 01 hrs : 57 mins
[TIMING] data: 14.5ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.6ms
step 749 | loss: 2.806750 | lr 1.8000e-04 | norm: 0.5296 | dt: 7824.65ms | tok/sec: 67004.6 | mem: 72099MB | Time Left: 01 hrs : 57 mins
[TIMING] data: 15.1ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
validation loss: 2.9395
step 750 | loss: 2.715778 | lr 1.8000e-04 | norm: 0.3023 | dt: 13843.25ms | tok/sec: 37873.2 | mem: 72099MB | Time Left: 03 hrs : 27 mins
[TIMING] data: 14.5ms | fwd: 2401.8ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.6ms
step 751 | loss: 3.068203 | lr 1.8000e-04 | norm: 0.2887 | dt: 7816.92ms | tok/sec: 67070.9 | mem: 72099MB | Time Left: 01 hrs : 57 mins
[TIMING] data: 13.1ms | fwd: 2412.5ms | loss: 5.0ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.4ms
step 752 | loss: 2.938362 | lr 1.8000e-04 | norm: 0.2799 | dt: 7807.05ms | tok/sec: 67155.7 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 12.4ms | fwd: 2401.5ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
step 753 | loss: 3.048961 | lr 1.8000e-04 | norm: 0.2785 | dt: 7809.10ms | tok/sec: 67138.0 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 15.0ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.5ms
step 754 | loss: 2.863284 | lr 1.8000e-04 | norm: 0.2435 | dt: 7826.42ms | tok/sec: 66989.5 | mem: 72099MB | Time Left: 01 hrs : 57 mins
[TIMING] data: 15.4ms | fwd: 2414.3ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 24.1ms
step 755 | loss: 3.084652 | lr 1.8000e-04 | norm: 0.2754 | dt: 7819.38ms | tok/sec: 67049.8 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 15.6ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.5ms
step 756 | loss: 3.021242 | lr 1.8000e-04 | norm: 0.2409 | dt: 7826.89ms | tok/sec: 66985.5 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 15.6ms | fwd: 2416.6ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.5ms
step 757 | loss: 2.821719 | lr 1.8000e-04 | norm: 0.3444 | dt: 7809.39ms | tok/sec: 67135.6 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 15.0ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5347.6ms | clip: 0.0ms | optim: 23.6ms
step 758 | loss: 2.622197 | lr 1.8000e-04 | norm: 0.2363 | dt: 7815.25ms | tok/sec: 67085.3 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 14.7ms | fwd: 2402.5ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 759 | loss: 3.243839 | lr 1.8000e-04 | norm: 0.3559 | dt: 7814.07ms | tok/sec: 67095.4 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 17.0ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.5ms
step 760 | loss: 2.623304 | lr 1.8000e-04 | norm: 0.2111 | dt: 7815.65ms | tok/sec: 67081.8 | mem: 72099MB | Time Left: 01 hrs : 56 mins
[TIMING] data: 13.9ms | fwd: 2409.4ms | loss: 4.9ms | bwd: 5351.0ms | clip: 0.0ms | optim: 23.6ms
step 761 | loss: 2.883075 | lr 1.8000e-04 | norm: 0.4079 | dt: 7812.55ms | tok/sec: 67108.4 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 16.3ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.5ms
step 762 | loss: 2.835704 | lr 1.8000e-04 | norm: 0.5930 | dt: 7814.45ms | tok/sec: 67092.1 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 15.7ms | fwd: 2410.0ms | loss: 4.9ms | bwd: 5348.0ms | clip: 0.0ms | optim: 23.6ms
step 763 | loss: 3.024881 | lr 1.8000e-04 | norm: 0.2317 | dt: 7808.36ms | tok/sec: 67144.4 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 16.5ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5348.1ms | clip: 0.0ms | optim: 23.6ms
step 764 | loss: 2.820690 | lr 1.8000e-04 | norm: 0.2231 | dt: 7821.45ms | tok/sec: 67032.1 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 14.1ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.7ms
step 765 | loss: 3.055565 | lr 1.8000e-04 | norm: 0.2173 | dt: 7815.50ms | tok/sec: 67083.1 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 15.2ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.5ms
step 766 | loss: 2.872467 | lr 1.8000e-04 | norm: 0.2278 | dt: 7818.78ms | tok/sec: 67055.0 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 15.9ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 767 | loss: 3.130697 | lr 1.8000e-04 | norm: 0.2257 | dt: 7819.22ms | tok/sec: 67051.2 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 13.9ms | fwd: 2411.0ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.7ms
step 768 | loss: 3.071174 | lr 1.8000e-04 | norm: 0.2057 | dt: 7820.34ms | tok/sec: 67041.6 | mem: 72099MB | Time Left: 01 hrs : 55 mins
[TIMING] data: 13.9ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.6ms
step 769 | loss: 2.883981 | lr 1.8000e-04 | norm: 0.3096 | dt: 7809.75ms | tok/sec: 67132.5 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 15.9ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5347.7ms | clip: 0.0ms | optim: 23.6ms
step 770 | loss: 2.694420 | lr 1.8000e-04 | norm: 0.2088 | dt: 7822.44ms | tok/sec: 67023.6 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 14.9ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 771 | loss: 3.101111 | lr 1.8000e-04 | norm: 0.2927 | dt: 7817.52ms | tok/sec: 67065.8 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 14.2ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 772 | loss: 2.990016 | lr 1.8000e-04 | norm: 0.1981 | dt: 7817.68ms | tok/sec: 67064.4 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 14.9ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 773 | loss: 2.643853 | lr 1.8000e-04 | norm: 0.2562 | dt: 7811.44ms | tok/sec: 67117.9 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 14.1ms | fwd: 2405.1ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.6ms
step 774 | loss: 2.775532 | lr 1.8000e-04 | norm: 0.2744 | dt: 7824.51ms | tok/sec: 67005.9 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 16.2ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 775 | loss: 2.808096 | lr 1.8000e-04 | norm: 0.2336 | dt: 7816.64ms | tok/sec: 67073.4 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 15.7ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.5ms
step 776 | loss: 2.329773 | lr 1.8000e-04 | norm: 0.6673 | dt: 7814.54ms | tok/sec: 67091.4 | mem: 72099MB | Time Left: 01 hrs : 53 mins
[TIMING] data: 15.1ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.6ms
step 777 | loss: 2.869659 | lr 1.8000e-04 | norm: 0.2271 | dt: 7825.95ms | tok/sec: 66993.5 | mem: 72099MB | Time Left: 01 hrs : 53 mins
[TIMING] data: 14.8ms | fwd: 2413.9ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.7ms
step 778 | loss: 2.842283 | lr 1.8000e-04 | norm: 0.2747 | dt: 7823.63ms | tok/sec: 67013.4 | mem: 72099MB | Time Left: 01 hrs : 53 mins
[TIMING] data: 14.3ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5360.7ms | clip: 0.0ms | optim: 23.5ms
step 779 | loss: 2.894380 | lr 1.8000e-04 | norm: 0.2332 | dt: 7817.14ms | tok/sec: 67069.0 | mem: 72099MB | Time Left: 01 hrs : 53 mins
[TIMING] data: 19.9ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.5ms
step 780 | loss: 3.076057 | lr 1.8000e-04 | norm: 0.2076 | dt: 7814.19ms | tok/sec: 67094.4 | mem: 72099MB | Time Left: 01 hrs : 53 mins
[TIMING] data: 14.8ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5350.0ms | clip: 0.0ms | optim: 23.5ms
step 781 | loss: 3.045901 | lr 1.8000e-04 | norm: 0.4930 | dt: 7881.66ms | tok/sec: 66520.0 | mem: 72099MB | Time Left: 01 hrs : 54 mins
[TIMING] data: 14.7ms | fwd: 2471.8ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 782 | loss: 2.603698 | lr 1.8000e-04 | norm: 0.1942 | dt: 7813.64ms | tok/sec: 67099.1 | mem: 72099MB | Time Left: 01 hrs : 53 mins
[TIMING] data: 15.2ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.6ms
step 783 | loss: 2.960295 | lr 1.8000e-04 | norm: 0.2092 | dt: 7821.88ms | tok/sec: 67028.4 | mem: 72099MB | Time Left: 01 hrs : 53 mins
[TIMING] data: 15.0ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.5ms
step 784 | loss: 3.222509 | lr 1.8000e-04 | norm: 0.2307 | dt: 7818.18ms | tok/sec: 67060.1 | mem: 72099MB | Time Left: 01 hrs : 52 mins
[TIMING] data: 14.9ms | fwd: 2409.9ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.6ms
step 785 | loss: 2.907172 | lr 1.8000e-04 | norm: 0.1904 | dt: 7821.88ms | tok/sec: 67028.4 | mem: 72099MB | Time Left: 01 hrs : 52 mins
[TIMING] data: 14.5ms | fwd: 2412.1ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 786 | loss: 2.839201 | lr 1.8000e-04 | norm: 0.2350 | dt: 7820.69ms | tok/sec: 67038.6 | mem: 72099MB | Time Left: 01 hrs : 52 mins
[TIMING] data: 15.7ms | fwd: 2416.7ms | loss: 4.9ms | bwd: 5346.3ms | clip: 0.0ms | optim: 23.6ms
step 787 | loss: 2.763173 | lr 1.8000e-04 | norm: 0.2945 | dt: 7821.44ms | tok/sec: 67032.2 | mem: 72099MB | Time Left: 01 hrs : 52 mins
[TIMING] data: 14.6ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.7ms
step 788 | loss: 2.959941 | lr 1.8000e-04 | norm: 0.3591 | dt: 7813.58ms | tok/sec: 67099.6 | mem: 72099MB | Time Left: 01 hrs : 52 mins
[TIMING] data: 14.5ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
step 789 | loss: 3.010280 | lr 1.8000e-04 | norm: 0.2174 | dt: 7818.77ms | tok/sec: 67055.0 | mem: 72099MB | Time Left: 01 hrs : 52 mins
[TIMING] data: 15.2ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.5ms
step 790 | loss: 3.038149 | lr 1.8000e-04 | norm: 0.2450 | dt: 7823.11ms | tok/sec: 67017.9 | mem: 72099MB | Time Left: 01 hrs : 52 mins
[TIMING] data: 14.6ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.5ms
step 791 | loss: 3.316003 | lr 1.8000e-04 | norm: 0.2191 | dt: 7813.32ms | tok/sec: 67101.8 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 13.4ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5350.3ms | clip: 0.0ms | optim: 23.5ms
step 792 | loss: 3.283958 | lr 1.8000e-04 | norm: 0.2400 | dt: 7815.43ms | tok/sec: 67083.7 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 15.1ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.6ms
step 793 | loss: 3.287378 | lr 1.8000e-04 | norm: 0.2556 | dt: 7821.33ms | tok/sec: 67033.1 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 15.4ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 794 | loss: 3.047162 | lr 1.8000e-04 | norm: 0.3210 | dt: 7809.23ms | tok/sec: 67137.0 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 14.6ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5346.2ms | clip: 0.0ms | optim: 23.6ms
step 795 | loss: 2.932522 | lr 1.8000e-04 | norm: 1.1220 | dt: 7807.24ms | tok/sec: 67154.1 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 14.1ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.5ms
step 796 | loss: 2.618268 | lr 1.8000e-04 | norm: 0.5316 | dt: 7818.85ms | tok/sec: 67054.4 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 11.2ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5363.0ms | clip: 0.0ms | optim: 23.4ms
step 797 | loss: 2.398394 | lr 1.8000e-04 | norm: 0.2505 | dt: 7814.63ms | tok/sec: 67090.6 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 13.3ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 798 | loss: 2.587621 | lr 1.8000e-04 | norm: 0.2429 | dt: 7822.58ms | tok/sec: 67022.4 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 15.8ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.6ms
step 799 | loss: 2.768221 | lr 1.8000e-04 | norm: 0.2663 | dt: 7826.30ms | tok/sec: 66990.5 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 14.6ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_800.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_600.ckpt
step 800 | loss: 2.711841 | lr 1.8000e-04 | norm: 0.2509 | dt: 7808.35ms | tok/sec: 67144.5 | mem: 72099MB | Time Left: 01 hrs : 50 mins
[TIMING] data: 12.0ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5343.6ms | clip: 0.0ms | optim: 23.4ms
step 801 | loss: 3.142781 | lr 1.8000e-04 | norm: 0.2230 | dt: 7802.09ms | tok/sec: 67198.4 | mem: 72099MB | Time Left: 01 hrs : 50 mins
[TIMING] data: 16.1ms | fwd: 2405.7ms | loss: 5.0ms | bwd: 5340.7ms | clip: 0.0ms | optim: 23.4ms
step 802 | loss: 2.834548 | lr 1.8000e-04 | norm: 0.2501 | dt: 7797.88ms | tok/sec: 67234.7 | mem: 72099MB | Time Left: 01 hrs : 50 mins
[TIMING] data: 15.3ms | fwd: 2400.0ms | loss: 4.9ms | bwd: 5340.7ms | clip: 0.0ms | optim: 23.5ms
step 803 | loss: 3.002645 | lr 1.8000e-04 | norm: 0.2579 | dt: 7820.18ms | tok/sec: 67043.0 | mem: 72099MB | Time Left: 01 hrs : 50 mins
[TIMING] data: 30.0ms | fwd: 2399.8ms | loss: 4.9ms | bwd: 5348.9ms | clip: 0.0ms | optim: 23.5ms
step 804 | loss: 2.982668 | lr 1.8000e-04 | norm: 0.2674 | dt: 7866.20ms | tok/sec: 66650.8 | mem: 72099MB | Time Left: 01 hrs : 51 mins
[TIMING] data: 79.7ms | fwd: 2398.1ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.5ms
step 805 | loss: 2.795406 | lr 1.8000e-04 | norm: 0.2412 | dt: 7814.15ms | tok/sec: 67094.7 | mem: 72099MB | Time Left: 01 hrs : 50 mins
[TIMING] data: 21.1ms | fwd: 2400.5ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.5ms
step 806 | loss: 2.637773 | lr 1.8000e-04 | norm: 0.2554 | dt: 7858.48ms | tok/sec: 66716.2 | mem: 72099MB | Time Left: 01 hrs : 50 mins
[TIMING] data: 16.1ms | fwd: 2451.5ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.4ms
step 807 | loss: 2.945825 | lr 1.8000e-04 | norm: 0.2781 | dt: 7814.12ms | tok/sec: 67095.0 | mem: 72099MB | Time Left: 01 hrs : 49 mins
[TIMING] data: 14.4ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.4ms
step 808 | loss: 3.049894 | lr 1.8000e-04 | norm: 0.2166 | dt: 7817.37ms | tok/sec: 67067.0 | mem: 72099MB | Time Left: 01 hrs : 49 mins
[TIMING] data: 15.6ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
step 809 | loss: 2.892908 | lr 1.8000e-04 | norm: 0.2146 | dt: 7822.02ms | tok/sec: 67027.1 | mem: 72099MB | Time Left: 01 hrs : 49 mins
[TIMING] data: 15.4ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 24.5ms
step 810 | loss: 2.795979 | lr 1.8000e-04 | norm: 0.2306 | dt: 7814.80ms | tok/sec: 67089.2 | mem: 72099MB | Time Left: 01 hrs : 49 mins
[TIMING] data: 12.6ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 811 | loss: 2.754790 | lr 1.8000e-04 | norm: 0.2004 | dt: 7824.26ms | tok/sec: 67008.0 | mem: 72099MB | Time Left: 01 hrs : 49 mins
[TIMING] data: 14.5ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.5ms
step 812 | loss: 2.600101 | lr 1.8000e-04 | norm: 0.2253 | dt: 7880.02ms | tok/sec: 66533.8 | mem: 72099MB | Time Left: 01 hrs : 50 mins
[TIMING] data: 14.1ms | fwd: 2474.7ms | loss: 4.9ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.5ms
step 813 | loss: 2.812464 | lr 1.8000e-04 | norm: 0.3328 | dt: 7822.00ms | tok/sec: 67027.4 | mem: 72099MB | Time Left: 01 hrs : 49 mins
[TIMING] data: 14.5ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.5ms
step 814 | loss: 2.651801 | lr 1.8000e-04 | norm: 0.2981 | dt: 7826.72ms | tok/sec: 66986.9 | mem: 72099MB | Time Left: 01 hrs : 49 mins
[TIMING] data: 23.4ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.5ms
step 815 | loss: 3.245114 | lr 1.8000e-04 | norm: 0.2274 | dt: 7815.35ms | tok/sec: 67084.4 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 15.2ms | fwd: 2403.0ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.4ms
step 816 | loss: 2.836473 | lr 1.8000e-04 | norm: 0.2370 | dt: 7818.66ms | tok/sec: 67056.0 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 16.2ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 817 | loss: 2.878559 | lr 1.8000e-04 | norm: 0.2217 | dt: 7815.75ms | tok/sec: 67080.9 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 14.4ms | fwd: 2402.7ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.6ms
step 818 | loss: 2.970731 | lr 1.8000e-04 | norm: 0.2070 | dt: 7818.79ms | tok/sec: 67054.9 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 15.3ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.6ms
step 819 | loss: 2.591885 | lr 1.8000e-04 | norm: 0.2989 | dt: 7817.34ms | tok/sec: 67067.3 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 15.9ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.5ms
step 820 | loss: 2.806034 | lr 1.8000e-04 | norm: 0.2520 | dt: 7830.02ms | tok/sec: 66958.7 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 13.2ms | fwd: 2416.4ms | loss: 4.8ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.5ms
step 821 | loss: 3.030702 | lr 1.8000e-04 | norm: 0.1933 | dt: 7814.76ms | tok/sec: 67089.5 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 14.3ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 822 | loss: 2.406862 | lr 1.8000e-04 | norm: 0.2159 | dt: 7820.60ms | tok/sec: 67039.4 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 16.6ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.6ms
step 823 | loss: 3.183097 | lr 1.8000e-04 | norm: 0.2607 | dt: 7814.70ms | tok/sec: 67090.0 | mem: 72099MB | Time Left: 01 hrs : 47 mins
[TIMING] data: 15.7ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.6ms
step 824 | loss: 2.982183 | lr 1.8000e-04 | norm: 0.4595 | dt: 7816.33ms | tok/sec: 67076.0 | mem: 72099MB | Time Left: 01 hrs : 47 mins
[TIMING] data: 14.6ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.5ms
step 825 | loss: 3.024006 | lr 1.8000e-04 | norm: 0.1999 | dt: 7815.11ms | tok/sec: 67086.5 | mem: 72099MB | Time Left: 01 hrs : 47 mins
[TIMING] data: 16.4ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.7ms
step 826 | loss: 3.005327 | lr 1.8000e-04 | norm: 0.3622 | dt: 7823.66ms | tok/sec: 67013.1 | mem: 72099MB | Time Left: 01 hrs : 47 mins
[TIMING] data: 14.9ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.5ms
step 827 | loss: 2.858097 | lr 1.8000e-04 | norm: 0.2132 | dt: 7816.93ms | tok/sec: 67070.8 | mem: 72099MB | Time Left: 01 hrs : 47 mins
[TIMING] data: 15.1ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 828 | loss: 2.560955 | lr 1.8000e-04 | norm: 0.2546 | dt: 7880.48ms | tok/sec: 66530.0 | mem: 72099MB | Time Left: 01 hrs : 48 mins
[TIMING] data: 14.7ms | fwd: 2473.9ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.5ms
step 829 | loss: 2.807645 | lr 1.8000e-04 | norm: 0.3826 | dt: 7820.21ms | tok/sec: 67042.7 | mem: 72099MB | Time Left: 01 hrs : 47 mins
[TIMING] data: 15.3ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
step 830 | loss: 3.047016 | lr 1.8000e-04 | norm: 0.2149 | dt: 7813.80ms | tok/sec: 67097.7 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 15.7ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5349.6ms | clip: 0.0ms | optim: 23.4ms
step 831 | loss: 2.834594 | lr 1.8000e-04 | norm: 0.2669 | dt: 7821.64ms | tok/sec: 67030.5 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 14.1ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.5ms
step 832 | loss: 2.988050 | lr 1.8000e-04 | norm: 0.2811 | dt: 7814.20ms | tok/sec: 67094.2 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 14.6ms | fwd: 2404.2ms | loss: 5.0ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 833 | loss: 2.795952 | lr 1.8000e-04 | norm: 0.2487 | dt: 7825.08ms | tok/sec: 67000.9 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 16.5ms | fwd: 2411.1ms | loss: 4.8ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 834 | loss: 2.695703 | lr 1.8000e-04 | norm: 0.4842 | dt: 7813.93ms | tok/sec: 67096.6 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 19.2ms | fwd: 2402.6ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 835 | loss: 2.813097 | lr 1.8000e-04 | norm: 0.2145 | dt: 7819.87ms | tok/sec: 67045.6 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 15.2ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.5ms
step 836 | loss: 2.792169 | lr 1.8000e-04 | norm: 0.2606 | dt: 7823.20ms | tok/sec: 67017.1 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 15.6ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 837 | loss: 3.099056 | lr 1.8000e-04 | norm: 0.2059 | dt: 7815.54ms | tok/sec: 67082.7 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 13.4ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.5ms
step 838 | loss: 2.849178 | lr 1.8000e-04 | norm: 0.2565 | dt: 7840.44ms | tok/sec: 66869.7 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 19.1ms | fwd: 2427.5ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 839 | loss: 2.905386 | lr 1.8000e-04 | norm: 0.2155 | dt: 7813.76ms | tok/sec: 67098.0 | mem: 72099MB | Time Left: 01 hrs : 45 mins
[TIMING] data: 15.6ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 840 | loss: 3.002583 | lr 1.8000e-04 | norm: 0.3301 | dt: 7883.71ms | tok/sec: 66502.7 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 13.6ms | fwd: 2475.5ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 841 | loss: 3.066978 | lr 1.8000e-04 | norm: 0.2710 | dt: 7819.21ms | tok/sec: 67051.3 | mem: 72099MB | Time Left: 01 hrs : 45 mins
[TIMING] data: 14.0ms | fwd: 2412.4ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 842 | loss: 2.997909 | lr 1.8000e-04 | norm: 0.2601 | dt: 7896.56ms | tok/sec: 66394.5 | mem: 72099MB | Time Left: 01 hrs : 46 mins
[TIMING] data: 17.7ms | fwd: 2478.9ms | loss: 4.9ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.4ms
step 843 | loss: 2.845929 | lr 1.8000e-04 | norm: 0.2380 | dt: 7821.88ms | tok/sec: 67028.4 | mem: 72099MB | Time Left: 01 hrs : 45 mins
[TIMING] data: 15.0ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5358.4ms | clip: 0.0ms | optim: 23.5ms
step 844 | loss: 2.907814 | lr 1.8000e-04 | norm: 0.2699 | dt: 7822.35ms | tok/sec: 67024.4 | mem: 72099MB | Time Left: 01 hrs : 45 mins
[TIMING] data: 13.3ms | fwd: 2412.5ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 845 | loss: 3.006182 | lr 1.8000e-04 | norm: 0.2404 | dt: 7817.34ms | tok/sec: 67067.3 | mem: 72099MB | Time Left: 01 hrs : 45 mins
[TIMING] data: 15.3ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 846 | loss: 2.946121 | lr 1.8000e-04 | norm: 0.2595 | dt: 7825.47ms | tok/sec: 66997.6 | mem: 72099MB | Time Left: 01 hrs : 44 mins
[TIMING] data: 17.6ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.5ms
step 847 | loss: 2.870426 | lr 1.8000e-04 | norm: 0.2808 | dt: 7820.41ms | tok/sec: 67041.0 | mem: 72099MB | Time Left: 01 hrs : 44 mins
[TIMING] data: 15.3ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5359.3ms | clip: 0.0ms | optim: 23.9ms
step 848 | loss: 3.082407 | lr 1.8000e-04 | norm: 0.3617 | dt: 7812.41ms | tok/sec: 67109.7 | mem: 72099MB | Time Left: 01 hrs : 44 mins
[TIMING] data: 15.7ms | fwd: 2404.5ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.7ms
step 849 | loss: 2.861084 | lr 1.8000e-04 | norm: 0.2359 | dt: 7817.40ms | tok/sec: 67066.8 | mem: 72099MB | Time Left: 01 hrs : 44 mins
[TIMING] data: 14.7ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.5ms
step 850 | loss: 3.124802 | lr 1.8000e-04 | norm: 0.2120 | dt: 7817.26ms | tok/sec: 67068.0 | mem: 72099MB | Time Left: 01 hrs : 44 mins
[TIMING] data: 19.7ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.5ms
step 851 | loss: 2.882375 | lr 1.8000e-04 | norm: 0.2805 | dt: 7822.58ms | tok/sec: 67022.4 | mem: 72099MB | Time Left: 01 hrs : 44 mins
[TIMING] data: 16.3ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.6ms
step 852 | loss: 2.835512 | lr 1.8000e-04 | norm: 0.3080 | dt: 7822.23ms | tok/sec: 67025.4 | mem: 72099MB | Time Left: 01 hrs : 44 mins
[TIMING] data: 15.3ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 853 | loss: 2.536317 | lr 1.8000e-04 | norm: 0.2363 | dt: 7818.97ms | tok/sec: 67053.4 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 15.7ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 854 | loss: 2.891272 | lr 1.8000e-04 | norm: 0.2160 | dt: 7817.86ms | tok/sec: 67062.9 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 13.9ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 855 | loss: 2.503308 | lr 1.8000e-04 | norm: 0.2446 | dt: 7817.57ms | tok/sec: 67065.3 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 14.0ms | fwd: 2408.1ms | loss: 4.8ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.5ms
step 856 | loss: 3.207890 | lr 1.8000e-04 | norm: 0.2251 | dt: 7821.19ms | tok/sec: 67034.3 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 11.3ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5359.9ms | clip: 0.0ms | optim: 23.5ms
step 857 | loss: 3.066038 | lr 1.8000e-04 | norm: 0.2103 | dt: 7817.30ms | tok/sec: 67067.7 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 14.8ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.6ms
step 858 | loss: 2.876233 | lr 1.8000e-04 | norm: 0.3351 | dt: 7818.16ms | tok/sec: 67060.3 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 14.7ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.5ms
step 859 | loss: 2.825342 | lr 1.8000e-04 | norm: 0.2420 | dt: 7822.51ms | tok/sec: 67023.0 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 16.1ms | fwd: 2415.0ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 860 | loss: 2.845701 | lr 1.8000e-04 | norm: 0.2755 | dt: 7821.72ms | tok/sec: 67029.7 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 16.3ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.6ms
step 861 | loss: 2.901604 | lr 1.8000e-04 | norm: 0.2389 | dt: 7891.63ms | tok/sec: 66435.9 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 20.1ms | fwd: 2477.1ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 862 | loss: 3.036738 | lr 1.8000e-04 | norm: 0.2455 | dt: 7821.92ms | tok/sec: 67028.1 | mem: 72099MB | Time Left: 01 hrs : 42 mins
[TIMING] data: 17.0ms | fwd: 2411.0ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.5ms
step 863 | loss: 2.690045 | lr 1.8000e-04 | norm: 0.2258 | dt: 7813.04ms | tok/sec: 67104.3 | mem: 72099MB | Time Left: 01 hrs : 42 mins
[TIMING] data: 16.5ms | fwd: 2406.2ms | loss: 5.0ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.5ms
step 864 | loss: 2.931232 | lr 1.8000e-04 | norm: 0.2494 | dt: 7819.45ms | tok/sec: 67049.2 | mem: 72099MB | Time Left: 01 hrs : 42 mins
[TIMING] data: 15.3ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 865 | loss: 2.957009 | lr 1.8000e-04 | norm: 0.2778 | dt: 7820.37ms | tok/sec: 67041.4 | mem: 72099MB | Time Left: 01 hrs : 42 mins
[TIMING] data: 21.0ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.5ms
step 866 | loss: 3.263241 | lr 1.8000e-04 | norm: 0.2262 | dt: 7817.08ms | tok/sec: 67069.5 | mem: 72099MB | Time Left: 01 hrs : 42 mins
[TIMING] data: 16.5ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 867 | loss: 2.913542 | lr 1.8000e-04 | norm: 0.2103 | dt: 7823.34ms | tok/sec: 67015.9 | mem: 72099MB | Time Left: 01 hrs : 42 mins
[TIMING] data: 14.8ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.5ms
step 868 | loss: 2.875652 | lr 1.8000e-04 | norm: 0.2288 | dt: 7814.09ms | tok/sec: 67095.2 | mem: 72099MB | Time Left: 01 hrs : 41 mins
[TIMING] data: 14.8ms | fwd: 2406.8ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.5ms
step 869 | loss: 2.776435 | lr 1.8000e-04 | norm: 0.1893 | dt: 7830.03ms | tok/sec: 66958.6 | mem: 72099MB | Time Left: 01 hrs : 42 mins
[TIMING] data: 18.6ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5359.3ms | clip: 0.0ms | optim: 23.5ms
step 870 | loss: 2.895929 | lr 1.8000e-04 | norm: 0.2434 | dt: 7817.00ms | tok/sec: 67070.2 | mem: 72099MB | Time Left: 01 hrs : 41 mins
[TIMING] data: 13.9ms | fwd: 2400.5ms | loss: 4.9ms | bwd: 5362.0ms | clip: 0.0ms | optim: 23.6ms
step 871 | loss: 2.902155 | lr 1.8000e-04 | norm: 0.2297 | dt: 7826.53ms | tok/sec: 66988.6 | mem: 72099MB | Time Left: 01 hrs : 41 mins
[TIMING] data: 19.7ms | fwd: 2412.2ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 872 | loss: 2.967463 | lr 1.8000e-04 | norm: 0.1831 | dt: 7826.48ms | tok/sec: 66989.0 | mem: 72099MB | Time Left: 01 hrs : 41 mins
[TIMING] data: 13.4ms | fwd: 2415.4ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 873 | loss: 2.913329 | lr 1.8000e-04 | norm: 0.2543 | dt: 7820.75ms | tok/sec: 67038.1 | mem: 72099MB | Time Left: 01 hrs : 41 mins
[TIMING] data: 15.2ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 874 | loss: 2.951466 | lr 1.8000e-04 | norm: 0.2457 | dt: 7819.22ms | tok/sec: 67051.2 | mem: 72099MB | Time Left: 01 hrs : 41 mins
[TIMING] data: 13.4ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.6ms
step 875 | loss: 2.895791 | lr 1.8000e-04 | norm: 0.2648 | dt: 7832.27ms | tok/sec: 66939.5 | mem: 72099MB | Time Left: 01 hrs : 41 mins
[TIMING] data: 15.7ms | fwd: 2418.2ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.5ms
step 876 | loss: 3.061144 | lr 1.8000e-04 | norm: 0.2360 | dt: 7817.37ms | tok/sec: 67067.1 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 15.3ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.5ms
step 877 | loss: 2.930233 | lr 1.8000e-04 | norm: 0.2563 | dt: 7818.42ms | tok/sec: 67058.1 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 19.5ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.5ms
step 878 | loss: 2.791046 | lr 1.8000e-04 | norm: 0.1872 | dt: 7821.22ms | tok/sec: 67034.1 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 14.9ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 879 | loss: 2.552594 | lr 1.8000e-04 | norm: 0.1947 | dt: 7822.75ms | tok/sec: 67020.9 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 14.7ms | fwd: 2413.1ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 880 | loss: 2.874165 | lr 1.8000e-04 | norm: 0.2457 | dt: 7853.87ms | tok/sec: 66755.4 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 17.9ms | fwd: 2448.5ms | loss: 4.9ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.5ms
step 881 | loss: 2.894294 | lr 1.8000e-04 | norm: 0.1720 | dt: 7821.67ms | tok/sec: 67030.2 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 12.8ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5360.2ms | clip: 0.0ms | optim: 23.7ms
step 882 | loss: 2.902654 | lr 1.8000e-04 | norm: 0.2230 | dt: 7831.08ms | tok/sec: 66949.6 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 15.7ms | fwd: 2419.6ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 24.1ms
step 883 | loss: 2.805105 | lr 1.8000e-04 | norm: 0.2102 | dt: 7818.56ms | tok/sec: 67056.8 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 15.5ms | fwd: 2411.8ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.5ms
step 884 | loss: 2.721015 | lr 1.8000e-04 | norm: 0.2143 | dt: 7811.43ms | tok/sec: 67118.1 | mem: 72099MB | Time Left: 01 hrs : 39 mins
[TIMING] data: 17.4ms | fwd: 2400.7ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 885 | loss: 2.718965 | lr 1.8000e-04 | norm: 0.2436 | dt: 7888.25ms | tok/sec: 66464.5 | mem: 72099MB | Time Left: 01 hrs : 40 mins
[TIMING] data: 14.7ms | fwd: 2477.4ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 886 | loss: 2.631758 | lr 1.8000e-04 | norm: 0.2480 | dt: 7823.87ms | tok/sec: 67011.3 | mem: 72099MB | Time Left: 01 hrs : 39 mins
[TIMING] data: 14.0ms | fwd: 2412.1ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.5ms
step 887 | loss: 3.040531 | lr 1.8000e-04 | norm: 0.2514 | dt: 7816.72ms | tok/sec: 67072.6 | mem: 72099MB | Time Left: 01 hrs : 39 mins
[TIMING] data: 13.3ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.5ms
step 888 | loss: 2.639863 | lr 1.8000e-04 | norm: 0.2429 | dt: 7814.45ms | tok/sec: 67092.1 | mem: 72099MB | Time Left: 01 hrs : 39 mins
[TIMING] data: 17.5ms | fwd: 2403.2ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.5ms
step 889 | loss: 2.917217 | lr 1.8000e-04 | norm: 0.2525 | dt: 7809.13ms | tok/sec: 67137.8 | mem: 72099MB | Time Left: 01 hrs : 39 mins
[TIMING] data: 15.0ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 890 | loss: 2.832230 | lr 1.8000e-04 | norm: 0.2272 | dt: 7826.56ms | tok/sec: 66988.3 | mem: 72099MB | Time Left: 01 hrs : 39 mins
[TIMING] data: 12.3ms | fwd: 2415.6ms | loss: 4.9ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.6ms
step 891 | loss: 2.784011 | lr 1.8000e-04 | norm: 0.2110 | dt: 7819.24ms | tok/sec: 67051.0 | mem: 72099MB | Time Left: 01 hrs : 39 mins
[TIMING] data: 17.0ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.6ms
step 892 | loss: 3.135813 | lr 1.8000e-04 | norm: 0.1945 | dt: 7821.90ms | tok/sec: 67028.2 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 20.6ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 893 | loss: 2.725814 | lr 1.8000e-04 | norm: 0.2272 | dt: 7819.45ms | tok/sec: 67049.3 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 15.5ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.6ms
step 894 | loss: 2.802638 | lr 1.8000e-04 | norm: 0.2025 | dt: 7821.15ms | tok/sec: 67034.6 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 15.6ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.7ms
step 895 | loss: 2.858776 | lr 1.8000e-04 | norm: 0.2054 | dt: 7823.38ms | tok/sec: 67015.5 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 15.1ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 896 | loss: 2.925487 | lr 1.8000e-04 | norm: 0.2112 | dt: 7822.53ms | tok/sec: 67022.8 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 15.3ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5358.7ms | clip: 0.0ms | optim: 23.4ms
step 897 | loss: 2.751902 | lr 1.8000e-04 | norm: 0.4721 | dt: 7819.55ms | tok/sec: 67048.4 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 14.6ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.5ms
step 898 | loss: 2.996279 | lr 1.8000e-04 | norm: 0.2286 | dt: 7820.39ms | tok/sec: 67041.1 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 16.0ms | fwd: 2412.2ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
step 899 | loss: 2.842124 | lr 1.8000e-04 | norm: 0.5404 | dt: 7823.78ms | tok/sec: 67012.1 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 15.7ms | fwd: 2413.0ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_900.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_700.ckpt
step 900 | loss: 3.234769 | lr 1.8000e-04 | norm: 0.2800 | dt: 8246.78ms | tok/sec: 63574.8 | mem: 72099MB | Time Left: 01 hrs : 43 mins
[TIMING] data: 11.0ms | fwd: 2853.9ms | loss: 4.8ms | bwd: 5342.1ms | clip: 0.0ms | optim: 23.5ms
step 901 | loss: 2.953679 | lr 1.8000e-04 | norm: 0.3241 | dt: 7798.50ms | tok/sec: 67229.4 | mem: 72099MB | Time Left: 01 hrs : 37 mins
[TIMING] data: 12.9ms | fwd: 2400.9ms | loss: 4.9ms | bwd: 5344.4ms | clip: 0.0ms | optim: 23.4ms
step 902 | loss: 2.587395 | lr 1.8000e-04 | norm: 0.3030 | dt: 7791.86ms | tok/sec: 67286.6 | mem: 72099MB | Time Left: 01 hrs : 37 mins
[TIMING] data: 14.1ms | fwd: 2397.6ms | loss: 4.9ms | bwd: 5339.9ms | clip: 0.0ms | optim: 23.4ms
step 903 | loss: 2.820631 | lr 1.8000e-04 | norm: 0.2265 | dt: 7863.51ms | tok/sec: 66673.6 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 76.0ms | fwd: 2400.0ms | loss: 4.9ms | bwd: 5347.6ms | clip: 0.0ms | optim: 23.5ms
step 904 | loss: 2.867308 | lr 1.8000e-04 | norm: 0.6062 | dt: 7878.59ms | tok/sec: 66545.9 | mem: 72099MB | Time Left: 01 hrs : 38 mins
[TIMING] data: 96.4ms | fwd: 2393.2ms | loss: 4.9ms | bwd: 5347.2ms | clip: 0.0ms | optim: 23.4ms
step 905 | loss: 3.088950 | lr 1.8000e-04 | norm: 0.2200 | dt: 7860.38ms | tok/sec: 66700.1 | mem: 72099MB | Time Left: 01 hrs : 37 mins
[TIMING] data: 68.7ms | fwd: 2401.1ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.5ms
step 906 | loss: 3.182456 | lr 1.8000e-04 | norm: 0.2430 | dt: 7813.19ms | tok/sec: 67102.9 | mem: 72099MB | Time Left: 01 hrs : 37 mins
[TIMING] data: 14.9ms | fwd: 2401.2ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 907 | loss: 2.935171 | lr 1.8000e-04 | norm: 0.2217 | dt: 7821.16ms | tok/sec: 67034.6 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 15.5ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.7ms
step 908 | loss: 2.880445 | lr 1.8000e-04 | norm: 0.1956 | dt: 7818.28ms | tok/sec: 67059.2 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 14.6ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 909 | loss: 2.796860 | lr 1.8000e-04 | norm: 0.2989 | dt: 7824.65ms | tok/sec: 67004.6 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 13.8ms | fwd: 2415.3ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.6ms
step 910 | loss: 3.026286 | lr 1.8000e-04 | norm: 0.2231 | dt: 7821.83ms | tok/sec: 67028.8 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 18.9ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.7ms
step 911 | loss: 2.640134 | lr 1.8000e-04 | norm: 0.2115 | dt: 7824.22ms | tok/sec: 67008.3 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 15.9ms | fwd: 2415.2ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 912 | loss: 3.135612 | lr 1.8000e-04 | norm: 0.2020 | dt: 7816.09ms | tok/sec: 67078.1 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 15.6ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 913 | loss: 2.793744 | lr 1.8000e-04 | norm: 0.2059 | dt: 7880.00ms | tok/sec: 66534.0 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 14.1ms | fwd: 2472.5ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.6ms
step 914 | loss: 2.929471 | lr 1.8000e-04 | norm: 0.3022 | dt: 7820.80ms | tok/sec: 67037.6 | mem: 72099MB | Time Left: 01 hrs : 36 mins
[TIMING] data: 13.0ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.5ms
step 915 | loss: 2.870401 | lr 1.8000e-04 | norm: 0.2686 | dt: 7811.19ms | tok/sec: 67120.1 | mem: 72099MB | Time Left: 01 hrs : 35 mins
[TIMING] data: 13.2ms | fwd: 2402.9ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 916 | loss: 2.525707 | lr 1.8000e-04 | norm: 0.3280 | dt: 7812.00ms | tok/sec: 67113.1 | mem: 72099MB | Time Left: 01 hrs : 35 mins
[TIMING] data: 14.2ms | fwd: 2408.5ms | loss: 4.9ms | bwd: 5348.8ms | clip: 0.0ms | optim: 23.6ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00005.bin
100000000
step 917 | loss: 3.105353 | lr 1.8000e-04 | norm: 0.2390 | dt: 7833.01ms | tok/sec: 66933.2 | mem: 72099MB | Time Left: 01 hrs : 35 mins
[TIMING] data: 34.0ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.5ms
step 918 | loss: 2.622725 | lr 1.8000e-04 | norm: 0.2357 | dt: 7818.37ms | tok/sec: 67058.4 | mem: 72099MB | Time Left: 01 hrs : 35 mins
[TIMING] data: 12.4ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 919 | loss: 2.779804 | lr 1.8000e-04 | norm: 0.2495 | dt: 7815.83ms | tok/sec: 67080.3 | mem: 72099MB | Time Left: 01 hrs : 35 mins
[TIMING] data: 13.8ms | fwd: 2411.1ms | loss: 4.8ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.6ms
step 920 | loss: 2.804464 | lr 1.8000e-04 | norm: 0.2189 | dt: 7819.23ms | tok/sec: 67051.1 | mem: 72099MB | Time Left: 01 hrs : 35 mins
[TIMING] data: 14.2ms | fwd: 2404.9ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.5ms
step 921 | loss: 3.101538 | lr 1.8000e-04 | norm: 0.2463 | dt: 7814.50ms | tok/sec: 67091.7 | mem: 72099MB | Time Left: 01 hrs : 35 mins
[TIMING] data: 15.0ms | fwd: 2409.4ms | loss: 4.8ms | bwd: 5349.3ms | clip: 0.0ms | optim: 23.7ms
step 922 | loss: 2.920707 | lr 1.8000e-04 | norm: 0.2445 | dt: 7818.73ms | tok/sec: 67055.4 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 13.5ms | fwd: 2407.4ms | loss: 4.8ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.5ms
step 923 | loss: 2.627184 | lr 1.8000e-04 | norm: 0.2288 | dt: 7818.33ms | tok/sec: 67058.9 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 14.5ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.5ms
step 924 | loss: 2.783385 | lr 1.8000e-04 | norm: 0.2623 | dt: 7814.12ms | tok/sec: 67095.0 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 12.3ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 925 | loss: 2.769241 | lr 1.8000e-04 | norm: 0.3307 | dt: 7822.81ms | tok/sec: 67020.4 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 13.6ms | fwd: 2414.7ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 926 | loss: 2.667679 | lr 1.8000e-04 | norm: 0.2365 | dt: 7821.39ms | tok/sec: 67032.6 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 15.8ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5359.5ms | clip: 0.0ms | optim: 23.5ms
step 927 | loss: 2.860363 | lr 1.8000e-04 | norm: 0.2526 | dt: 7821.71ms | tok/sec: 67029.8 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 11.7ms | fwd: 2411.1ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.4ms
step 928 | loss: 3.007092 | lr 1.8000e-04 | norm: 0.2878 | dt: 7817.89ms | tok/sec: 67062.6 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 14.0ms | fwd: 2413.2ms | loss: 4.8ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.6ms
step 929 | loss: 2.860363 | lr 1.8000e-04 | norm: 0.2603 | dt: 7819.67ms | tok/sec: 67047.3 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 13.4ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.6ms
step 930 | loss: 3.025044 | lr 1.8000e-04 | norm: 0.2611 | dt: 7812.40ms | tok/sec: 67109.8 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 13.7ms | fwd: 2404.8ms | loss: 4.8ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 931 | loss: 3.155108 | lr 1.8000e-04 | norm: 0.2695 | dt: 7817.12ms | tok/sec: 67069.2 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 13.3ms | fwd: 2407.3ms | loss: 4.8ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 932 | loss: 2.709360 | lr 1.8000e-04 | norm: 0.1916 | dt: 7824.99ms | tok/sec: 67001.8 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 13.6ms | fwd: 2413.0ms | loss: 4.8ms | bwd: 5357.2ms | clip: 0.0ms | optim: 23.5ms
step 933 | loss: 3.008101 | lr 1.8000e-04 | norm: 0.2076 | dt: 7814.91ms | tok/sec: 67088.1 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 16.8ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.6ms
step 934 | loss: 3.037589 | lr 1.8000e-04 | norm: 0.2280 | dt: 7870.92ms | tok/sec: 66610.7 | mem: 72099MB | Time Left: 01 hrs : 34 mins
[TIMING] data: 14.5ms | fwd: 2465.2ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
step 935 | loss: 3.028785 | lr 1.8000e-04 | norm: 0.3441 | dt: 7816.33ms | tok/sec: 67076.0 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 15.9ms | fwd: 2405.1ms | loss: 4.9ms | bwd: 5355.8ms | clip: 0.0ms | optim: 23.5ms
step 936 | loss: 3.082989 | lr 1.8000e-04 | norm: 0.2124 | dt: 7815.82ms | tok/sec: 67080.4 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 14.5ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 937 | loss: 3.182222 | lr 1.8000e-04 | norm: 0.1962 | dt: 7884.54ms | tok/sec: 66495.7 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 15.4ms | fwd: 2476.1ms | loss: 4.8ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.4ms
step 938 | loss: 2.959017 | lr 1.8000e-04 | norm: 0.2778 | dt: 7883.26ms | tok/sec: 66506.5 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 14.3ms | fwd: 2472.8ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 939 | loss: 2.989564 | lr 1.8000e-04 | norm: 0.2231 | dt: 7878.20ms | tok/sec: 66549.2 | mem: 72099MB | Time Left: 01 hrs : 33 mins
[TIMING] data: 14.3ms | fwd: 2468.2ms | loss: 4.8ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 940 | loss: 2.954025 | lr 1.8000e-04 | norm: 0.2455 | dt: 7818.15ms | tok/sec: 67060.4 | mem: 72099MB | Time Left: 01 hrs : 32 mins
[TIMING] data: 15.2ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 941 | loss: 2.670152 | lr 1.8000e-04 | norm: 0.1908 | dt: 7820.97ms | tok/sec: 67036.2 | mem: 72099MB | Time Left: 01 hrs : 32 mins
[TIMING] data: 12.6ms | fwd: 2413.8ms | loss: 4.9ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.6ms
step 942 | loss: 3.146488 | lr 1.8000e-04 | norm: 0.2514 | dt: 7818.65ms | tok/sec: 67056.1 | mem: 72099MB | Time Left: 01 hrs : 32 mins
[TIMING] data: 13.7ms | fwd: 2408.8ms | loss: 4.8ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 943 | loss: 2.896751 | lr 1.8000e-04 | norm: 0.3103 | dt: 7814.27ms | tok/sec: 67093.6 | mem: 72099MB | Time Left: 01 hrs : 32 mins
[TIMING] data: 14.7ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 944 | loss: 3.031915 | lr 1.8000e-04 | norm: 0.2505 | dt: 7815.46ms | tok/sec: 67083.5 | mem: 72099MB | Time Left: 01 hrs : 32 mins
[TIMING] data: 13.7ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.5ms
step 945 | loss: 2.692589 | lr 1.8000e-04 | norm: 0.2755 | dt: 7813.45ms | tok/sec: 67100.7 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 14.4ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
step 946 | loss: 2.749050 | lr 1.8000e-04 | norm: 0.8196 | dt: 7807.57ms | tok/sec: 67151.2 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 14.1ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
step 947 | loss: 2.650692 | lr 1.8000e-04 | norm: 0.1977 | dt: 7824.53ms | tok/sec: 67005.7 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 14.2ms | fwd: 2411.4ms | loss: 4.8ms | bwd: 5358.7ms | clip: 0.0ms | optim: 23.6ms
step 948 | loss: 2.564419 | lr 1.8000e-04 | norm: 0.3849 | dt: 7820.76ms | tok/sec: 67038.0 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 15.4ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.7ms
step 949 | loss: 2.964263 | lr 1.8000e-04 | norm: 0.2425 | dt: 7809.83ms | tok/sec: 67131.8 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 12.9ms | fwd: 2402.2ms | loss: 4.8ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 950 | loss: 2.497884 | lr 1.8000e-04 | norm: 0.2494 | dt: 7829.48ms | tok/sec: 66963.3 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 15.2ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.4ms
step 951 | loss: 2.560052 | lr 1.8000e-04 | norm: 0.2361 | dt: 7815.91ms | tok/sec: 67079.6 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 14.8ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
step 952 | loss: 2.835895 | lr 1.8000e-04 | norm: 0.2491 | dt: 7815.75ms | tok/sec: 67081.0 | mem: 72099MB | Time Left: 01 hrs : 31 mins
[TIMING] data: 14.2ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.6ms
step 953 | loss: 2.885171 | lr 1.8000e-04 | norm: 0.2350 | dt: 7820.25ms | tok/sec: 67042.3 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 15.3ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 954 | loss: 2.967582 | lr 1.8000e-04 | norm: 0.2268 | dt: 7813.90ms | tok/sec: 67096.8 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 13.1ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 955 | loss: 2.930557 | lr 1.8000e-04 | norm: 0.3556 | dt: 7820.73ms | tok/sec: 67038.3 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 16.8ms | fwd: 2414.1ms | loss: 4.9ms | bwd: 5349.4ms | clip: 0.0ms | optim: 23.5ms
step 956 | loss: 2.943324 | lr 1.8000e-04 | norm: 0.2111 | dt: 7818.29ms | tok/sec: 67059.1 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 15.4ms | fwd: 2406.4ms | loss: 4.8ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 957 | loss: 2.870757 | lr 1.8000e-04 | norm: 0.2545 | dt: 7818.53ms | tok/sec: 67057.1 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 12.7ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5361.1ms | clip: 0.0ms | optim: 23.5ms
step 958 | loss: 2.886872 | lr 1.8000e-04 | norm: 0.2430 | dt: 7817.50ms | tok/sec: 67065.9 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 14.5ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5360.0ms | clip: 0.0ms | optim: 23.5ms
step 959 | loss: 3.032310 | lr 1.8000e-04 | norm: 0.2363 | dt: 7813.73ms | tok/sec: 67098.3 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 13.1ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.5ms
step 960 | loss: 2.880030 | lr 1.8000e-04 | norm: 0.2760 | dt: 7822.72ms | tok/sec: 67021.2 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 15.5ms | fwd: 2414.0ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.4ms
step 961 | loss: 2.953454 | lr 1.8000e-04 | norm: 0.2890 | dt: 7828.65ms | tok/sec: 66970.4 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 13.7ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 962 | loss: 2.441819 | lr 1.8000e-04 | norm: 0.1975 | dt: 7822.91ms | tok/sec: 67019.6 | mem: 72099MB | Time Left: 01 hrs : 29 mins
[TIMING] data: 14.3ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.5ms
step 963 | loss: 2.979933 | lr 1.8000e-04 | norm: 0.2142 | dt: 7820.89ms | tok/sec: 67036.9 | mem: 72099MB | Time Left: 01 hrs : 29 mins
[TIMING] data: 14.5ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5361.3ms | clip: 0.0ms | optim: 23.5ms
step 964 | loss: 2.773444 | lr 1.8000e-04 | norm: 0.2398 | dt: 7820.43ms | tok/sec: 67040.8 | mem: 72099MB | Time Left: 01 hrs : 29 mins
[TIMING] data: 13.1ms | fwd: 2415.4ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.5ms
step 965 | loss: 3.034783 | lr 1.8000e-04 | norm: 0.2740 | dt: 7887.21ms | tok/sec: 66473.2 | mem: 72099MB | Time Left: 01 hrs : 30 mins
[TIMING] data: 13.8ms | fwd: 2481.0ms | loss: 4.8ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.5ms
step 966 | loss: 2.761559 | lr 1.8000e-04 | norm: 0.2344 | dt: 7860.94ms | tok/sec: 66695.3 | mem: 72099MB | Time Left: 01 hrs : 29 mins
[TIMING] data: 13.7ms | fwd: 2451.2ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 967 | loss: 2.707366 | lr 1.8000e-04 | norm: 0.2398 | dt: 7882.62ms | tok/sec: 66511.9 | mem: 72099MB | Time Left: 01 hrs : 29 mins
[TIMING] data: 17.3ms | fwd: 2470.1ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 968 | loss: 2.877552 | lr 1.8000e-04 | norm: 0.2218 | dt: 7821.10ms | tok/sec: 67035.1 | mem: 72099MB | Time Left: 01 hrs : 29 mins
[TIMING] data: 14.7ms | fwd: 2411.5ms | loss: 4.9ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.5ms
step 969 | loss: 2.783687 | lr 1.8000e-04 | norm: 0.2141 | dt: 7818.46ms | tok/sec: 67057.7 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 14.6ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 970 | loss: 3.037994 | lr 1.8000e-04 | norm: 0.1984 | dt: 7826.40ms | tok/sec: 66989.7 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 15.8ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 971 | loss: 2.795472 | lr 1.8000e-04 | norm: 0.6682 | dt: 7821.20ms | tok/sec: 67034.2 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 15.3ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.5ms
step 972 | loss: 2.815830 | lr 1.8000e-04 | norm: 0.6970 | dt: 7816.17ms | tok/sec: 67077.4 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 15.4ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.5ms
step 973 | loss: 2.709714 | lr 1.8000e-04 | norm: 0.3157 | dt: 7816.44ms | tok/sec: 67075.1 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 12.3ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.6ms
step 974 | loss: 2.610018 | lr 1.8000e-04 | norm: 0.2260 | dt: 7820.07ms | tok/sec: 67043.9 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 12.8ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.7ms
step 975 | loss: 3.028664 | lr 1.8000e-04 | norm: 0.2218 | dt: 7821.77ms | tok/sec: 67029.4 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 15.2ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.6ms
step 976 | loss: 2.861120 | lr 1.8000e-04 | norm: 0.2842 | dt: 7827.52ms | tok/sec: 66980.1 | mem: 72099MB | Time Left: 01 hrs : 28 mins
[TIMING] data: 16.7ms | fwd: 2417.2ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.5ms
step 977 | loss: 3.035544 | lr 1.8000e-04 | norm: 0.3027 | dt: 7820.96ms | tok/sec: 67036.3 | mem: 72099MB | Time Left: 01 hrs : 27 mins
[TIMING] data: 12.2ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.5ms
step 978 | loss: 2.943259 | lr 1.8000e-04 | norm: 0.2328 | dt: 7820.80ms | tok/sec: 67037.6 | mem: 72099MB | Time Left: 01 hrs : 27 mins
[TIMING] data: 17.1ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.7ms
step 979 | loss: 2.860526 | lr 1.8000e-04 | norm: 0.2270 | dt: 7828.45ms | tok/sec: 66972.1 | mem: 72099MB | Time Left: 01 hrs : 27 mins
[TIMING] data: 12.7ms | fwd: 2421.1ms | loss: 4.8ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 980 | loss: 2.929049 | lr 1.8000e-04 | norm: 0.2671 | dt: 7818.42ms | tok/sec: 67058.1 | mem: 72099MB | Time Left: 01 hrs : 27 mins
[TIMING] data: 15.2ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.6ms
step 981 | loss: 2.742639 | lr 1.8000e-04 | norm: 0.2296 | dt: 7812.87ms | tok/sec: 67105.7 | mem: 72099MB | Time Left: 01 hrs : 27 mins
[TIMING] data: 15.2ms | fwd: 2403.8ms | loss: 4.8ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 982 | loss: 2.718542 | lr 1.8000e-04 | norm: 0.1848 | dt: 7815.80ms | tok/sec: 67080.6 | mem: 72099MB | Time Left: 01 hrs : 27 mins
[TIMING] data: 17.4ms | fwd: 2404.1ms | loss: 4.8ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 983 | loss: 2.693783 | lr 1.8000e-04 | norm: 0.2437 | dt: 7816.70ms | tok/sec: 67072.8 | mem: 72099MB | Time Left: 01 hrs : 27 mins
[TIMING] data: 13.3ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.6ms
step 984 | loss: 3.092325 | lr 1.8000e-04 | norm: 0.5869 | dt: 7812.64ms | tok/sec: 67107.7 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 12.4ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.6ms
step 985 | loss: 2.900625 | lr 1.8000e-04 | norm: 0.2497 | dt: 7814.13ms | tok/sec: 67094.9 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 13.1ms | fwd: 2404.8ms | loss: 4.8ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.7ms
step 986 | loss: 2.837266 | lr 1.8000e-04 | norm: 0.2493 | dt: 7830.16ms | tok/sec: 66957.5 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 19.0ms | fwd: 2413.0ms | loss: 4.9ms | bwd: 5358.7ms | clip: 0.0ms | optim: 23.4ms
step 987 | loss: 3.028414 | lr 1.8000e-04 | norm: 0.2210 | dt: 7821.81ms | tok/sec: 67029.0 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 14.4ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.6ms
step 988 | loss: 2.698897 | lr 1.8000e-04 | norm: 0.2276 | dt: 7827.55ms | tok/sec: 66979.8 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 15.7ms | fwd: 2409.3ms | loss: 4.8ms | bwd: 5362.4ms | clip: 0.0ms | optim: 23.9ms
step 989 | loss: 3.007383 | lr 1.8000e-04 | norm: 0.2070 | dt: 7877.03ms | tok/sec: 66559.1 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 14.0ms | fwd: 2469.4ms | loss: 4.8ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
step 990 | loss: 2.914299 | lr 1.8000e-04 | norm: 0.2080 | dt: 7887.44ms | tok/sec: 66471.2 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 23.2ms | fwd: 2472.9ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 991 | loss: 2.711135 | lr 1.8000e-04 | norm: 0.2086 | dt: 7820.63ms | tok/sec: 67039.1 | mem: 72099MB | Time Left: 01 hrs : 26 mins
[TIMING] data: 13.7ms | fwd: 2406.1ms | loss: 4.8ms | bwd: 5359.9ms | clip: 0.0ms | optim: 23.7ms
step 992 | loss: 2.932899 | lr 1.8000e-04 | norm: 0.1984 | dt: 7807.08ms | tok/sec: 67155.5 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 13.6ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5348.6ms | clip: 0.0ms | optim: 23.6ms
step 993 | loss: 3.178344 | lr 1.8000e-04 | norm: 0.2407 | dt: 7817.20ms | tok/sec: 67068.5 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 9.8ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.5ms
step 994 | loss: 2.898725 | lr 1.8000e-04 | norm: 0.3026 | dt: 7823.18ms | tok/sec: 67017.3 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 13.1ms | fwd: 2411.9ms | loss: 4.8ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.6ms
step 995 | loss: 2.726881 | lr 1.8000e-04 | norm: 0.3526 | dt: 7818.38ms | tok/sec: 67058.4 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 12.8ms | fwd: 2410.5ms | loss: 4.8ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 996 | loss: 2.736847 | lr 1.8000e-04 | norm: 0.2924 | dt: 7826.21ms | tok/sec: 66991.3 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 13.9ms | fwd: 2418.1ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.6ms
step 997 | loss: 2.939140 | lr 1.8000e-04 | norm: 0.2048 | dt: 7820.56ms | tok/sec: 67039.7 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 13.8ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.6ms
step 998 | loss: 3.021967 | lr 1.8000e-04 | norm: 0.2511 | dt: 7814.40ms | tok/sec: 67092.6 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 13.4ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.6ms
step 999 | loss: 2.662035 | lr 1.8000e-04 | norm: 0.2338 | dt: 7815.59ms | tok/sec: 67082.3 | mem: 72099MB | Time Left: 01 hrs : 24 mins
[TIMING] data: 15.3ms | fwd: 2403.0ms | loss: 4.8ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.6ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1000.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_800.ckpt
validation loss: 2.9133
step 1000 | loss: 2.951226 | lr 1.8000e-04 | norm: 0.2316 | dt: 13915.79ms | tok/sec: 37675.8 | mem: 72099MB | Time Left: 02 hrs : 30 mins
[TIMING] data: 13.0ms | fwd: 2397.1ms | loss: 4.9ms | bwd: 5343.3ms | clip: 0.0ms | optim: 23.5ms
step 1001 | loss: 2.919324 | lr 1.8000e-04 | norm: 0.2297 | dt: 7797.64ms | tok/sec: 67236.8 | mem: 72099MB | Time Left: 01 hrs : 24 mins
[TIMING] data: 14.8ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5340.8ms | clip: 0.0ms | optim: 23.6ms
step 1002 | loss: 2.958556 | lr 1.8000e-04 | norm: 0.2787 | dt: 7819.03ms | tok/sec: 67052.8 | mem: 72099MB | Time Left: 01 hrs : 24 mins
[TIMING] data: 20.0ms | fwd: 2413.7ms | loss: 4.9ms | bwd: 5345.1ms | clip: 0.0ms | optim: 23.5ms
step 1003 | loss: 3.010872 | lr 1.8000e-04 | norm: 0.2005 | dt: 7893.42ms | tok/sec: 66420.9 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 110.7ms | fwd: 2392.8ms | loss: 4.9ms | bwd: 5349.0ms | clip: 0.0ms | optim: 23.6ms
step 1004 | loss: 3.006602 | lr 1.8000e-04 | norm: 0.2537 | dt: 7917.45ms | tok/sec: 66219.3 | mem: 72099MB | Time Left: 01 hrs : 25 mins
[TIMING] data: 136.0ms | fwd: 2394.3ms | loss: 4.9ms | bwd: 5346.4ms | clip: 0.0ms | optim: 23.8ms
step 1005 | loss: 2.844437 | lr 1.8000e-04 | norm: 0.2715 | dt: 7823.02ms | tok/sec: 67018.6 | mem: 72099MB | Time Left: 01 hrs : 24 mins
[TIMING] data: 14.6ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5365.4ms | clip: 0.0ms | optim: 23.5ms
step 1006 | loss: 2.813604 | lr 1.8000e-04 | norm: 0.2043 | dt: 7816.67ms | tok/sec: 67073.1 | mem: 72099MB | Time Left: 01 hrs : 24 mins
[TIMING] data: 15.1ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.6ms
step 1007 | loss: 2.920053 | lr 1.8000e-04 | norm: 0.2593 | dt: 7819.13ms | tok/sec: 67051.9 | mem: 72099MB | Time Left: 01 hrs : 23 mins
[TIMING] data: 17.7ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5348.8ms | clip: 0.0ms | optim: 23.5ms
step 1008 | loss: 2.913403 | lr 1.8000e-04 | norm: 0.3129 | dt: 7812.88ms | tok/sec: 67105.6 | mem: 72099MB | Time Left: 01 hrs : 23 mins
[TIMING] data: 17.0ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5348.2ms | clip: 0.0ms | optim: 23.5ms
step 1009 | loss: 2.918028 | lr 1.8000e-04 | norm: 0.2373 | dt: 7819.61ms | tok/sec: 67047.8 | mem: 72099MB | Time Left: 01 hrs : 23 mins
[TIMING] data: 14.4ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 1010 | loss: 2.694700 | lr 1.8000e-04 | norm: 0.2271 | dt: 7819.21ms | tok/sec: 67051.3 | mem: 72099MB | Time Left: 01 hrs : 23 mins
[TIMING] data: 14.6ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 1011 | loss: 2.582282 | lr 1.8000e-04 | norm: 0.2125 | dt: 7826.81ms | tok/sec: 66986.2 | mem: 72099MB | Time Left: 01 hrs : 23 mins
[TIMING] data: 16.0ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5361.4ms | clip: 0.0ms | optim: 23.5ms
step 1012 | loss: 3.257880 | lr 1.8000e-04 | norm: 0.3843 | dt: 7809.62ms | tok/sec: 67133.6 | mem: 72099MB | Time Left: 01 hrs : 23 mins
[TIMING] data: 11.7ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.5ms
step 1013 | loss: 3.007172 | lr 1.8000e-04 | norm: 0.2085 | dt: 7812.26ms | tok/sec: 67110.9 | mem: 72099MB | Time Left: 01 hrs : 23 mins
[TIMING] data: 14.5ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.6ms
step 1014 | loss: 2.871137 | lr 1.8000e-04 | norm: 0.2327 | dt: 7813.00ms | tok/sec: 67104.6 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 13.4ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.6ms
step 1015 | loss: 2.913477 | lr 1.8000e-04 | norm: 0.1993 | dt: 7821.76ms | tok/sec: 67029.5 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 15.0ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.5ms
step 1016 | loss: 2.840600 | lr 1.8000e-04 | norm: 0.2243 | dt: 7821.44ms | tok/sec: 67032.1 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 16.4ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.6ms
step 1017 | loss: 2.852322 | lr 1.8000e-04 | norm: 0.2080 | dt: 7813.87ms | tok/sec: 67097.1 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 11.0ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5359.0ms | clip: 0.0ms | optim: 23.6ms
step 1018 | loss: 3.144833 | lr 1.8000e-04 | norm: 0.2186 | dt: 7816.63ms | tok/sec: 67073.4 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 14.3ms | fwd: 2404.9ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.5ms
step 1019 | loss: 2.846650 | lr 1.8000e-04 | norm: 0.1935 | dt: 7818.65ms | tok/sec: 67056.1 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 14.9ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 1020 | loss: 2.855958 | lr 1.8000e-04 | norm: 0.2299 | dt: 7818.97ms | tok/sec: 67053.3 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 11.9ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.5ms
step 1021 | loss: 3.010135 | lr 1.8000e-04 | norm: 0.2391 | dt: 7823.89ms | tok/sec: 67011.2 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 15.4ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.5ms
step 1022 | loss: 3.246263 | lr 1.8000e-04 | norm: 0.2374 | dt: 7871.47ms | tok/sec: 66606.1 | mem: 72099MB | Time Left: 01 hrs : 22 mins
[TIMING] data: 12.5ms | fwd: 2461.9ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 1023 | loss: 3.116976 | lr 1.8000e-04 | norm: 0.2190 | dt: 7816.62ms | tok/sec: 67073.5 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 13.8ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 1024 | loss: 2.585683 | lr 1.8000e-04 | norm: 0.2549 | dt: 7815.72ms | tok/sec: 67081.2 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 14.5ms | fwd: 2404.9ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.5ms
step 1025 | loss: 2.620874 | lr 1.8000e-04 | norm: 0.2259 | dt: 7807.10ms | tok/sec: 67155.2 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 13.5ms | fwd: 2401.5ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.6ms
step 1026 | loss: 3.001567 | lr 1.8000e-04 | norm: 0.3203 | dt: 7817.92ms | tok/sec: 67062.3 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 15.1ms | fwd: 2414.9ms | loss: 4.9ms | bwd: 5347.8ms | clip: 0.0ms | optim: 23.7ms
step 1027 | loss: 2.514844 | lr 1.8000e-04 | norm: 0.2508 | dt: 7815.47ms | tok/sec: 67083.4 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 13.3ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.6ms
step 1028 | loss: 2.520462 | lr 1.8000e-04 | norm: 0.2753 | dt: 7820.42ms | tok/sec: 67040.9 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 12.8ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.5ms
step 1029 | loss: 3.184329 | lr 1.8000e-04 | norm: 0.3676 | dt: 7815.40ms | tok/sec: 67083.9 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 14.3ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.5ms
step 1030 | loss: 2.725964 | lr 1.8000e-04 | norm: 0.1955 | dt: 7810.77ms | tok/sec: 67123.7 | mem: 72099MB | Time Left: 01 hrs : 20 mins
[TIMING] data: 18.3ms | fwd: 2401.5ms | loss: 4.9ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.5ms
step 1031 | loss: 3.038873 | lr 1.8000e-04 | norm: 0.2273 | dt: 7815.69ms | tok/sec: 67081.4 | mem: 72099MB | Time Left: 01 hrs : 20 mins
[TIMING] data: 12.8ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5356.3ms | clip: 0.0ms | optim: 23.6ms
step 1032 | loss: 3.059420 | lr 1.8000e-04 | norm: 0.2544 | dt: 7818.30ms | tok/sec: 67059.1 | mem: 72099MB | Time Left: 01 hrs : 20 mins
[TIMING] data: 15.8ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 1033 | loss: 2.769115 | lr 1.8000e-04 | norm: 0.2421 | dt: 7844.94ms | tok/sec: 66831.4 | mem: 72099MB | Time Left: 01 hrs : 20 mins
[TIMING] data: 39.7ms | fwd: 2412.0ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.6ms
step 1034 | loss: 2.804353 | lr 1.8000e-04 | norm: 0.3056 | dt: 7880.48ms | tok/sec: 66530.0 | mem: 72099MB | Time Left: 01 hrs : 21 mins
[TIMING] data: 14.3ms | fwd: 2471.4ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.7ms
step 1035 | loss: 2.778177 | lr 1.8000e-04 | norm: 0.3913 | dt: 7820.60ms | tok/sec: 67039.3 | mem: 72099MB | Time Left: 01 hrs : 20 mins
[TIMING] data: 14.0ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.6ms
step 1036 | loss: 2.616494 | lr 1.8000e-04 | norm: 0.2243 | dt: 7820.11ms | tok/sec: 67043.6 | mem: 72099MB | Time Left: 01 hrs : 20 mins
[TIMING] data: 13.4ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.5ms
step 1037 | loss: 2.589684 | lr 1.8000e-04 | norm: 0.2935 | dt: 7820.49ms | tok/sec: 67040.3 | mem: 72099MB | Time Left: 01 hrs : 20 mins
[TIMING] data: 16.9ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 1038 | loss: 2.853798 | lr 1.8000e-04 | norm: 0.2595 | dt: 7816.54ms | tok/sec: 67074.1 | mem: 72099MB | Time Left: 01 hrs : 19 mins
[TIMING] data: 13.6ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 1039 | loss: 2.545526 | lr 1.8000e-04 | norm: 0.2043 | dt: 7811.95ms | tok/sec: 67113.6 | mem: 72099MB | Time Left: 01 hrs : 19 mins
[TIMING] data: 14.8ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.5ms
step 1040 | loss: 2.698363 | lr 1.8000e-04 | norm: 0.2370 | dt: 7821.64ms | tok/sec: 67030.5 | mem: 72099MB | Time Left: 01 hrs : 19 mins
[TIMING] data: 13.4ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.6ms
step 1041 | loss: 2.706461 | lr 1.8000e-04 | norm: 0.2596 | dt: 7827.31ms | tok/sec: 66981.9 | mem: 72099MB | Time Left: 01 hrs : 19 mins
[TIMING] data: 16.7ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5360.2ms | clip: 0.0ms | optim: 23.5ms
step 1042 | loss: 2.754553 | lr 1.8000e-04 | norm: 0.2539 | dt: 7811.78ms | tok/sec: 67115.0 | mem: 72099MB | Time Left: 01 hrs : 19 mins
[TIMING] data: 14.4ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.6ms
step 1043 | loss: 2.664060 | lr 1.8000e-04 | norm: 0.2508 | dt: 7818.91ms | tok/sec: 67053.9 | mem: 72099MB | Time Left: 01 hrs : 19 mins
[TIMING] data: 12.7ms | fwd: 2403.4ms | loss: 4.9ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.6ms
step 1044 | loss: 2.478382 | lr 1.8000e-04 | norm: 0.1959 | dt: 7822.35ms | tok/sec: 67024.3 | mem: 72099MB | Time Left: 01 hrs : 19 mins
[TIMING] data: 16.2ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.5ms
step 1045 | loss: 2.784842 | lr 1.8000e-04 | norm: 0.4335 | dt: 7812.38ms | tok/sec: 67109.9 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 14.2ms | fwd: 2401.4ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.6ms
step 1046 | loss: 2.876858 | lr 1.8000e-04 | norm: 0.2861 | dt: 7810.24ms | tok/sec: 67128.3 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 14.1ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.6ms
step 1047 | loss: 3.178836 | lr 1.8000e-04 | norm: 0.2849 | dt: 7823.58ms | tok/sec: 67013.9 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 11.6ms | fwd: 2412.0ms | loss: 4.9ms | bwd: 5359.3ms | clip: 0.0ms | optim: 23.5ms
step 1048 | loss: 3.036989 | lr 1.8000e-04 | norm: 0.2271 | dt: 7825.63ms | tok/sec: 66996.2 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 13.1ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5362.8ms | clip: 0.0ms | optim: 23.5ms
step 1049 | loss: 2.660572 | lr 1.8000e-04 | norm: 0.2433 | dt: 7825.56ms | tok/sec: 66996.9 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 18.3ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.7ms
step 1050 | loss: 3.249118 | lr 1.8000e-04 | norm: 0.2768 | dt: 7815.10ms | tok/sec: 67086.6 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 13.6ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5358.6ms | clip: 0.0ms | optim: 23.5ms
step 1051 | loss: 2.926598 | lr 1.8000e-04 | norm: 0.2432 | dt: 7817.99ms | tok/sec: 67061.7 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 14.6ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1052 | loss: 2.780838 | lr 1.8000e-04 | norm: 0.2232 | dt: 7822.32ms | tok/sec: 67024.6 | mem: 72099MB | Time Left: 01 hrs : 18 mins
[TIMING] data: 15.0ms | fwd: 2411.1ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1053 | loss: 2.813151 | lr 1.8000e-04 | norm: 0.2315 | dt: 7817.25ms | tok/sec: 67068.1 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 17.6ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.6ms
step 1054 | loss: 2.942259 | lr 1.8000e-04 | norm: 0.2039 | dt: 7816.66ms | tok/sec: 67073.2 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 14.7ms | fwd: 2412.0ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 1055 | loss: 2.448238 | lr 1.8000e-04 | norm: 0.2670 | dt: 7815.78ms | tok/sec: 67080.7 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 15.3ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.7ms
step 1056 | loss: 3.101345 | lr 1.8000e-04 | norm: 0.2350 | dt: 7818.50ms | tok/sec: 67057.3 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 15.4ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 1057 | loss: 3.235331 | lr 1.8000e-04 | norm: 0.2653 | dt: 7815.27ms | tok/sec: 67085.1 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 12.5ms | fwd: 2405.9ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.7ms
step 1058 | loss: 2.996457 | lr 1.8000e-04 | norm: 0.2285 | dt: 7811.99ms | tok/sec: 67113.3 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 14.4ms | fwd: 2403.0ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 1059 | loss: 2.792976 | lr 1.8000e-04 | norm: 0.2149 | dt: 7825.68ms | tok/sec: 66995.8 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 13.5ms | fwd: 2418.9ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.8ms
step 1060 | loss: 2.866734 | lr 1.8000e-04 | norm: 0.3011 | dt: 7824.34ms | tok/sec: 67007.3 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 13.7ms | fwd: 2417.8ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.6ms
step 1061 | loss: 2.927672 | lr 1.8000e-04 | norm: 0.1920 | dt: 7875.67ms | tok/sec: 66570.6 | mem: 72099MB | Time Left: 01 hrs : 17 mins
[TIMING] data: 13.6ms | fwd: 2470.0ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.5ms
step 1062 | loss: 2.887524 | lr 1.8000e-04 | norm: 0.2558 | dt: 7821.92ms | tok/sec: 67028.0 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 14.0ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.5ms
step 1063 | loss: 2.929920 | lr 1.8000e-04 | norm: 0.2145 | dt: 7821.55ms | tok/sec: 67031.2 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 14.9ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 1064 | loss: 2.625232 | lr 1.8000e-04 | norm: 0.1831 | dt: 7825.66ms | tok/sec: 66996.0 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 13.2ms | fwd: 2412.5ms | loss: 4.9ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.5ms
step 1065 | loss: 2.936958 | lr 1.8000e-04 | norm: 0.2403 | dt: 7828.63ms | tok/sec: 66970.6 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 14.6ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5365.1ms | clip: 0.0ms | optim: 23.6ms
step 1066 | loss: 2.972353 | lr 1.8000e-04 | norm: 0.2260 | dt: 7819.33ms | tok/sec: 67050.2 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 15.0ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.6ms
step 1067 | loss: 2.877344 | lr 1.8000e-04 | norm: 0.2218 | dt: 7812.98ms | tok/sec: 67104.8 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 15.4ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.6ms
step 1068 | loss: 2.812823 | lr 1.8000e-04 | norm: 0.1816 | dt: 7885.24ms | tok/sec: 66489.8 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 15.5ms | fwd: 2471.5ms | loss: 4.9ms | bwd: 5358.2ms | clip: 0.0ms | optim: 23.6ms
step 1069 | loss: 2.700074 | lr 1.8000e-04 | norm: 0.4310 | dt: 7823.70ms | tok/sec: 67012.8 | mem: 72099MB | Time Left: 01 hrs : 15 mins
[TIMING] data: 15.3ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5361.8ms | clip: 0.0ms | optim: 23.6ms
step 1070 | loss: 2.809552 | lr 1.8000e-04 | norm: 0.3403 | dt: 7815.29ms | tok/sec: 67084.9 | mem: 72099MB | Time Left: 01 hrs : 15 mins
[TIMING] data: 13.7ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 1071 | loss: 2.723644 | lr 1.8000e-04 | norm: 0.4467 | dt: 7891.37ms | tok/sec: 66438.2 | mem: 72099MB | Time Left: 01 hrs : 16 mins
[TIMING] data: 16.6ms | fwd: 2480.1ms | loss: 4.9ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.5ms
step 1072 | loss: 2.916325 | lr 1.8000e-04 | norm: 0.2164 | dt: 7823.37ms | tok/sec: 67015.6 | mem: 72099MB | Time Left: 01 hrs : 15 mins
[TIMING] data: 16.7ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.6ms
step 1073 | loss: 3.274528 | lr 1.8000e-04 | norm: 0.2348 | dt: 7816.22ms | tok/sec: 67077.0 | mem: 72099MB | Time Left: 01 hrs : 15 mins
[TIMING] data: 14.2ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.6ms
step 1074 | loss: 2.785890 | lr 1.8000e-04 | norm: 0.2147 | dt: 7820.88ms | tok/sec: 67036.9 | mem: 72099MB | Time Left: 01 hrs : 15 mins
[TIMING] data: 14.5ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.6ms
step 1075 | loss: 3.026166 | lr 1.8000e-04 | norm: 0.2057 | dt: 7816.60ms | tok/sec: 67073.7 | mem: 72099MB | Time Left: 01 hrs : 15 mins
[TIMING] data: 13.3ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.5ms
step 1076 | loss: 2.864450 | lr 1.8000e-04 | norm: 0.2091 | dt: 7825.86ms | tok/sec: 66994.3 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 16.0ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.5ms
step 1077 | loss: 2.907479 | lr 1.8000e-04 | norm: 0.2350 | dt: 7818.69ms | tok/sec: 67055.8 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 12.3ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.7ms
step 1078 | loss: 2.793859 | lr 1.8000e-04 | norm: 0.2992 | dt: 7824.66ms | tok/sec: 67004.6 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 13.3ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.6ms
step 1079 | loss: 2.945787 | lr 1.8000e-04 | norm: 0.2948 | dt: 7824.21ms | tok/sec: 67008.4 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 13.0ms | fwd: 2412.1ms | loss: 4.9ms | bwd: 5358.2ms | clip: 0.0ms | optim: 23.5ms
step 1080 | loss: 2.633791 | lr 1.8000e-04 | norm: 0.2032 | dt: 7823.70ms | tok/sec: 67012.8 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 11.7ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5364.4ms | clip: 0.0ms | optim: 23.5ms
step 1081 | loss: 3.080378 | lr 1.8000e-04 | norm: 0.2432 | dt: 7817.43ms | tok/sec: 67066.6 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 15.2ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.5ms
step 1082 | loss: 2.706882 | lr 1.8000e-04 | norm: 0.2033 | dt: 7820.01ms | tok/sec: 67044.4 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 15.3ms | fwd: 2414.5ms | loss: 4.9ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.5ms
step 1083 | loss: 2.878821 | lr 1.8000e-04 | norm: 0.1985 | dt: 7815.91ms | tok/sec: 67079.6 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 18.2ms | fwd: 2398.1ms | loss: 4.9ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.5ms
step 1084 | loss: 2.994390 | lr 1.8000e-04 | norm: 0.2013 | dt: 7820.74ms | tok/sec: 67038.2 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 14.8ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 1085 | loss: 3.037691 | lr 1.8000e-04 | norm: 0.2299 | dt: 7888.62ms | tok/sec: 66461.3 | mem: 72099MB | Time Left: 01 hrs : 14 mins
[TIMING] data: 13.2ms | fwd: 2480.4ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 1086 | loss: 3.007230 | lr 1.8000e-04 | norm: 0.4027 | dt: 7813.70ms | tok/sec: 67098.5 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 12.3ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.6ms
step 1087 | loss: 2.240056 | lr 1.8000e-04 | norm: 0.2446 | dt: 7818.94ms | tok/sec: 67053.6 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 13.1ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.6ms
step 1088 | loss: 3.101279 | lr 1.8000e-04 | norm: 0.1951 | dt: 7817.87ms | tok/sec: 67062.8 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 12.8ms | fwd: 2414.3ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
step 1089 | loss: 2.928738 | lr 1.8000e-04 | norm: 0.2250 | dt: 7821.97ms | tok/sec: 67027.6 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 14.5ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.5ms
step 1090 | loss: 2.858465 | lr 1.8000e-04 | norm: 0.2086 | dt: 7813.14ms | tok/sec: 67103.4 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 13.4ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5348.7ms | clip: 0.0ms | optim: 23.6ms
step 1091 | loss: 3.001982 | lr 1.8000e-04 | norm: 0.2190 | dt: 7825.51ms | tok/sec: 66997.3 | mem: 72099MB | Time Left: 01 hrs : 13 mins
[TIMING] data: 17.1ms | fwd: 2413.1ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1092 | loss: 2.621534 | lr 1.8000e-04 | norm: 0.2500 | dt: 7818.14ms | tok/sec: 67060.5 | mem: 72099MB | Time Left: 01 hrs : 12 mins
[TIMING] data: 14.1ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.6ms
step 1093 | loss: 2.975782 | lr 1.8000e-04 | norm: 0.2542 | dt: 7825.04ms | tok/sec: 67001.3 | mem: 72099MB | Time Left: 01 hrs : 12 mins
[TIMING] data: 15.0ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5359.0ms | clip: 0.0ms | optim: 23.5ms
step 1094 | loss: 2.597873 | lr 1.8000e-04 | norm: 0.2059 | dt: 7823.73ms | tok/sec: 67012.6 | mem: 72099MB | Time Left: 01 hrs : 12 mins
[TIMING] data: 13.7ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5359.0ms | clip: 0.0ms | optim: 23.7ms
step 1095 | loss: 2.716653 | lr 1.8000e-04 | norm: 0.3346 | dt: 7820.04ms | tok/sec: 67044.2 | mem: 72099MB | Time Left: 01 hrs : 12 mins
[TIMING] data: 13.5ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 1096 | loss: 3.250242 | lr 1.8000e-04 | norm: 0.2633 | dt: 7816.88ms | tok/sec: 67071.2 | mem: 72099MB | Time Left: 01 hrs : 12 mins
[TIMING] data: 14.3ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.5ms
step 1097 | loss: 3.088303 | lr 1.8000e-04 | norm: 0.1766 | dt: 7821.10ms | tok/sec: 67035.0 | mem: 72099MB | Time Left: 01 hrs : 12 mins
[TIMING] data: 14.7ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.5ms
step 1098 | loss: 3.049232 | lr 1.8000e-04 | norm: 0.2541 | dt: 7818.04ms | tok/sec: 67061.3 | mem: 72099MB | Time Left: 01 hrs : 12 mins
[TIMING] data: 14.0ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 1099 | loss: 2.755927 | lr 1.8000e-04 | norm: 0.2454 | dt: 7815.97ms | tok/sec: 67079.1 | mem: 72099MB | Time Left: 01 hrs : 11 mins
[TIMING] data: 14.7ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.7ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1100.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_900.ckpt
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00006.bin
100000000
step 1100 | loss: 2.859061 | lr 1.8000e-04 | norm: 0.2125 | dt: 7810.40ms | tok/sec: 67126.9 | mem: 72099MB | Time Left: 01 hrs : 11 mins
[TIMING] data: 36.1ms | fwd: 2394.4ms | loss: 4.8ms | bwd: 5341.9ms | clip: 0.0ms | optim: 23.4ms
step 1101 | loss: 2.952051 | lr 1.8000e-04 | norm: 0.2079 | dt: 7800.25ms | tok/sec: 67214.3 | mem: 72099MB | Time Left: 01 hrs : 11 mins
[TIMING] data: 16.6ms | fwd: 2398.5ms | loss: 4.8ms | bwd: 5346.2ms | clip: 0.0ms | optim: 23.5ms
step 1102 | loss: 3.096370 | lr 1.8000e-04 | norm: 0.2413 | dt: 7801.61ms | tok/sec: 67202.5 | mem: 72099MB | Time Left: 01 hrs : 11 mins
[TIMING] data: 15.5ms | fwd: 2404.9ms | loss: 4.8ms | bwd: 5342.3ms | clip: 0.0ms | optim: 23.5ms
step 1103 | loss: 2.674606 | lr 1.8000e-04 | norm: 0.2067 | dt: 7811.61ms | tok/sec: 67116.5 | mem: 72099MB | Time Left: 01 hrs : 11 mins
[TIMING] data: 21.5ms | fwd: 2406.4ms | loss: 4.8ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.5ms
step 1104 | loss: 2.980090 | lr 1.8000e-04 | norm: 0.2026 | dt: 7830.66ms | tok/sec: 66953.3 | mem: 72099MB | Time Left: 01 hrs : 11 mins
[TIMING] data: 43.8ms | fwd: 2403.0ms | loss: 4.9ms | bwd: 5344.8ms | clip: 0.0ms | optim: 23.6ms
step 1105 | loss: 2.376422 | lr 1.8000e-04 | norm: 0.2288 | dt: 7805.20ms | tok/sec: 67171.7 | mem: 72099MB | Time Left: 01 hrs : 11 mins
[TIMING] data: 18.8ms | fwd: 2404.1ms | loss: 4.8ms | bwd: 5343.2ms | clip: 0.0ms | optim: 23.5ms
step 1106 | loss: 2.952861 | lr 1.8000e-04 | norm: 0.2570 | dt: 7815.47ms | tok/sec: 67083.3 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 14.9ms | fwd: 2413.6ms | loss: 4.8ms | bwd: 5347.5ms | clip: 0.0ms | optim: 23.5ms
step 1107 | loss: 2.804055 | lr 1.8000e-04 | norm: 0.2002 | dt: 7810.76ms | tok/sec: 67123.8 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 15.3ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5349.3ms | clip: 0.0ms | optim: 23.6ms
step 1108 | loss: 2.952178 | lr 1.8000e-04 | norm: 0.2222 | dt: 7810.45ms | tok/sec: 67126.5 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 13.8ms | fwd: 2407.6ms | loss: 4.8ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.5ms
step 1109 | loss: 2.955214 | lr 1.8000e-04 | norm: 0.2516 | dt: 7815.08ms | tok/sec: 67086.7 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 12.1ms | fwd: 2409.5ms | loss: 4.8ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.5ms
step 1110 | loss: 2.904880 | lr 1.8000e-04 | norm: 0.2069 | dt: 7810.45ms | tok/sec: 67126.5 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 14.4ms | fwd: 2408.6ms | loss: 4.8ms | bwd: 5348.6ms | clip: 0.0ms | optim: 23.5ms
step 1111 | loss: 3.037266 | lr 1.8000e-04 | norm: 0.2233 | dt: 7807.25ms | tok/sec: 67154.0 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 14.1ms | fwd: 2408.6ms | loss: 4.8ms | bwd: 5346.5ms | clip: 0.0ms | optim: 23.5ms
step 1112 | loss: 2.701343 | lr 1.8000e-04 | norm: 0.2524 | dt: 7811.42ms | tok/sec: 67118.1 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 14.7ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5349.2ms | clip: 0.0ms | optim: 23.6ms
step 1113 | loss: 3.014986 | lr 1.8000e-04 | norm: 0.2828 | dt: 7880.92ms | tok/sec: 66526.2 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 15.4ms | fwd: 2480.3ms | loss: 4.8ms | bwd: 5347.0ms | clip: 0.0ms | optim: 23.5ms
step 1114 | loss: 2.903775 | lr 1.8000e-04 | norm: 0.2945 | dt: 7815.37ms | tok/sec: 67084.3 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 14.7ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 1115 | loss: 3.035404 | lr 1.8000e-04 | norm: 0.3241 | dt: 7881.03ms | tok/sec: 66525.3 | mem: 72099MB | Time Left: 01 hrs : 10 mins
[TIMING] data: 15.0ms | fwd: 2473.7ms | loss: 4.8ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.4ms
step 1116 | loss: 2.648009 | lr 1.8000e-04 | norm: 0.2691 | dt: 7807.63ms | tok/sec: 67150.7 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 16.6ms | fwd: 2401.3ms | loss: 4.8ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
step 1117 | loss: 2.919686 | lr 1.8000e-04 | norm: 0.2186 | dt: 7819.65ms | tok/sec: 67047.5 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 13.0ms | fwd: 2418.7ms | loss: 4.8ms | bwd: 5349.0ms | clip: 0.0ms | optim: 23.5ms
step 1118 | loss: 2.846677 | lr 1.8000e-04 | norm: 0.2183 | dt: 7811.36ms | tok/sec: 67118.7 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 13.3ms | fwd: 2410.9ms | loss: 4.8ms | bwd: 5348.2ms | clip: 0.0ms | optim: 23.6ms
step 1119 | loss: 2.701707 | lr 1.8000e-04 | norm: 0.2830 | dt: 7819.65ms | tok/sec: 67047.5 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 14.9ms | fwd: 2409.7ms | loss: 4.8ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.6ms
step 1120 | loss: 2.943614 | lr 1.8000e-04 | norm: 0.2705 | dt: 7814.42ms | tok/sec: 67092.3 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 16.5ms | fwd: 2409.2ms | loss: 4.8ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.6ms
step 1121 | loss: 2.830604 | lr 1.8000e-04 | norm: 0.2305 | dt: 7817.17ms | tok/sec: 67068.8 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 14.7ms | fwd: 2408.5ms | loss: 4.8ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 1122 | loss: 2.930592 | lr 1.8000e-04 | norm: 0.4303 | dt: 7810.26ms | tok/sec: 67128.1 | mem: 72099MB | Time Left: 01 hrs : 08 mins
[TIMING] data: 13.6ms | fwd: 2401.6ms | loss: 4.8ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 1123 | loss: 3.179018 | lr 1.8000e-04 | norm: 0.2484 | dt: 7820.49ms | tok/sec: 67040.3 | mem: 72099MB | Time Left: 01 hrs : 08 mins
[TIMING] data: 13.1ms | fwd: 2415.0ms | loss: 4.8ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.5ms
step 1124 | loss: 2.780043 | lr 1.8000e-04 | norm: 0.4874 | dt: 7879.12ms | tok/sec: 66541.5 | mem: 72099MB | Time Left: 01 hrs : 09 mins
[TIMING] data: 18.1ms | fwd: 2475.3ms | loss: 4.8ms | bwd: 5346.8ms | clip: 0.0ms | optim: 23.5ms
step 1125 | loss: 3.030602 | lr 1.8000e-04 | norm: 0.2661 | dt: 7811.84ms | tok/sec: 67114.6 | mem: 72099MB | Time Left: 01 hrs : 08 mins
[TIMING] data: 14.7ms | fwd: 2410.9ms | loss: 4.8ms | bwd: 5347.1ms | clip: 0.0ms | optim: 23.6ms
step 1126 | loss: 3.102347 | lr 1.8000e-04 | norm: 0.3303 | dt: 7824.40ms | tok/sec: 67006.8 | mem: 72099MB | Time Left: 01 hrs : 08 mins
[TIMING] data: 14.2ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 1127 | loss: 3.228023 | lr 1.8000e-04 | norm: 0.4233 | dt: 7863.01ms | tok/sec: 66677.8 | mem: 72099MB | Time Left: 01 hrs : 08 mins
[TIMING] data: 14.7ms | fwd: 2456.1ms | loss: 4.8ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 1128 | loss: 2.505585 | lr 1.8000e-04 | norm: 0.3058 | dt: 7880.66ms | tok/sec: 66528.5 | mem: 72099MB | Time Left: 01 hrs : 08 mins
[TIMING] data: 17.4ms | fwd: 2473.3ms | loss: 4.8ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
step 1129 | loss: 2.746986 | lr 1.8000e-04 | norm: 0.2478 | dt: 7817.84ms | tok/sec: 67063.1 | mem: 72099MB | Time Left: 01 hrs : 08 mins
[TIMING] data: 13.3ms | fwd: 2412.6ms | loss: 4.8ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 1130 | loss: 2.858541 | lr 1.8000e-04 | norm: 0.2463 | dt: 7816.22ms | tok/sec: 67076.9 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 14.2ms | fwd: 2412.3ms | loss: 4.8ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.5ms
step 1131 | loss: 2.929829 | lr 1.8000e-04 | norm: 0.2350 | dt: 7820.22ms | tok/sec: 67042.6 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 14.7ms | fwd: 2408.1ms | loss: 4.8ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.7ms
step 1132 | loss: 2.817442 | lr 1.8000e-04 | norm: 0.2625 | dt: 7813.24ms | tok/sec: 67102.5 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 15.1ms | fwd: 2409.4ms | loss: 4.8ms | bwd: 5349.3ms | clip: 0.0ms | optim: 23.6ms
step 1133 | loss: 2.942637 | lr 1.8000e-04 | norm: 0.2553 | dt: 7815.46ms | tok/sec: 67083.5 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 15.5ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5351.8ms | clip: 0.0ms | optim: 23.7ms
step 1134 | loss: 2.683722 | lr 1.8000e-04 | norm: 0.2854 | dt: 7813.23ms | tok/sec: 67102.6 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 14.4ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.5ms
step 1135 | loss: 3.053473 | lr 1.8000e-04 | norm: 0.4293 | dt: 7820.86ms | tok/sec: 67037.1 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 18.0ms | fwd: 2412.9ms | loss: 4.8ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
step 1136 | loss: 2.729772 | lr 1.8000e-04 | norm: 0.2577 | dt: 7815.75ms | tok/sec: 67080.9 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 13.2ms | fwd: 2407.7ms | loss: 4.8ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.5ms
step 1137 | loss: 3.159492 | lr 1.8000e-04 | norm: 0.2637 | dt: 7877.86ms | tok/sec: 66552.1 | mem: 72099MB | Time Left: 01 hrs : 07 mins
[TIMING] data: 15.2ms | fwd: 2471.2ms | loss: 4.8ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1138 | loss: 2.994043 | lr 1.8000e-04 | norm: 0.2178 | dt: 7829.54ms | tok/sec: 66962.8 | mem: 72099MB | Time Left: 01 hrs : 06 mins
[TIMING] data: 14.9ms | fwd: 2422.6ms | loss: 4.8ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 1139 | loss: 2.852053 | lr 1.8000e-04 | norm: 0.2971 | dt: 7818.79ms | tok/sec: 67054.9 | mem: 72099MB | Time Left: 01 hrs : 06 mins
[TIMING] data: 13.3ms | fwd: 2407.4ms | loss: 4.8ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.5ms
step 1140 | loss: 2.914925 | lr 1.8000e-04 | norm: 0.3150 | dt: 7810.00ms | tok/sec: 67130.3 | mem: 72099MB | Time Left: 01 hrs : 06 mins
[TIMING] data: 15.0ms | fwd: 2402.6ms | loss: 4.8ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 1141 | loss: 2.775459 | lr 1.8000e-04 | norm: 0.2283 | dt: 7827.34ms | tok/sec: 66981.6 | mem: 72099MB | Time Left: 01 hrs : 06 mins
[TIMING] data: 12.8ms | fwd: 2418.1ms | loss: 4.8ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.4ms
step 1142 | loss: 2.970935 | lr 1.8000e-04 | norm: 0.2992 | dt: 7821.32ms | tok/sec: 67033.2 | mem: 72099MB | Time Left: 01 hrs : 06 mins
[TIMING] data: 14.9ms | fwd: 2412.4ms | loss: 4.8ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 1143 | loss: 2.800073 | lr 1.8000e-04 | norm: 0.2548 | dt: 7887.00ms | tok/sec: 66475.0 | mem: 72099MB | Time Left: 01 hrs : 06 mins
[TIMING] data: 15.0ms | fwd: 2478.3ms | loss: 4.8ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.5ms
step 1144 | loss: 3.003452 | lr 1.8000e-04 | norm: 0.2355 | dt: 7815.17ms | tok/sec: 67085.9 | mem: 72099MB | Time Left: 01 hrs : 06 mins
[TIMING] data: 14.2ms | fwd: 2407.6ms | loss: 4.8ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.5ms
step 1145 | loss: 2.763333 | lr 1.8000e-04 | norm: 0.4878 | dt: 7808.68ms | tok/sec: 67141.7 | mem: 72099MB | Time Left: 01 hrs : 05 mins
[TIMING] data: 13.2ms | fwd: 2404.2ms | loss: 4.8ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.7ms
step 1146 | loss: 2.901082 | lr 1.8000e-04 | norm: 0.3475 | dt: 7820.01ms | tok/sec: 67044.4 | mem: 72099MB | Time Left: 01 hrs : 05 mins
[TIMING] data: 14.1ms | fwd: 2412.6ms | loss: 4.8ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.5ms
step 1147 | loss: 2.738057 | lr 1.8000e-04 | norm: 0.2899 | dt: 7815.36ms | tok/sec: 67084.3 | mem: 72099MB | Time Left: 01 hrs : 05 mins
[TIMING] data: 16.7ms | fwd: 2404.7ms | loss: 4.8ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.5ms
step 1148 | loss: 2.874837 | lr 1.8000e-04 | norm: 0.2706 | dt: 7809.61ms | tok/sec: 67133.7 | mem: 72099MB | Time Left: 01 hrs : 05 mins
[TIMING] data: 16.0ms | fwd: 2403.8ms | loss: 4.8ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.6ms
step 1149 | loss: 2.929116 | lr 1.8000e-04 | norm: 0.2557 | dt: 7813.40ms | tok/sec: 67101.1 | mem: 72099MB | Time Left: 01 hrs : 05 mins
[TIMING] data: 14.8ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.4ms
step 1150 | loss: 2.874526 | lr 1.8000e-04 | norm: 0.2672 | dt: 7888.43ms | tok/sec: 66462.9 | mem: 72099MB | Time Left: 01 hrs : 05 mins
[TIMING] data: 13.9ms | fwd: 2481.7ms | loss: 4.8ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 1151 | loss: 2.676874 | lr 1.8000e-04 | norm: 0.3336 | dt: 7851.40ms | tok/sec: 66776.3 | mem: 72099MB | Time Left: 01 hrs : 05 mins
[TIMING] data: 13.4ms | fwd: 2444.8ms | loss: 4.8ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.5ms
step 1152 | loss: 2.876886 | lr 1.8000e-04 | norm: 0.2920 | dt: 7811.99ms | tok/sec: 67113.2 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 14.1ms | fwd: 2407.1ms | loss: 4.8ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.6ms
step 1153 | loss: 2.838279 | lr 1.8000e-04 | norm: 0.2736 | dt: 7818.89ms | tok/sec: 67054.0 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 14.5ms | fwd: 2418.2ms | loss: 4.8ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.5ms
step 1154 | loss: 2.573199 | lr 1.8000e-04 | norm: 0.2956 | dt: 7819.77ms | tok/sec: 67046.4 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 13.5ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 1155 | loss: 2.833646 | lr 1.8000e-04 | norm: 0.3758 | dt: 7816.48ms | tok/sec: 67074.7 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 14.6ms | fwd: 2409.5ms | loss: 4.8ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.5ms
step 1156 | loss: 2.654677 | lr 1.8000e-04 | norm: 0.2293 | dt: 7818.55ms | tok/sec: 67056.9 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 13.5ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.7ms
step 1157 | loss: 2.913909 | lr 1.8000e-04 | norm: 0.4253 | dt: 7816.40ms | tok/sec: 67075.4 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 12.0ms | fwd: 2413.3ms | loss: 4.8ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.7ms
step 1158 | loss: 2.985494 | lr 1.8000e-04 | norm: 0.3650 | dt: 7820.76ms | tok/sec: 67038.0 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 16.5ms | fwd: 2412.1ms | loss: 4.8ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.8ms
step 1159 | loss: 2.688713 | lr 1.8000e-04 | norm: 0.3007 | dt: 7880.10ms | tok/sec: 66533.2 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 14.9ms | fwd: 2475.5ms | loss: 4.8ms | bwd: 5351.0ms | clip: 0.0ms | optim: 23.6ms
step 1160 | loss: 2.991403 | lr 1.8000e-04 | norm: 0.4314 | dt: 7814.07ms | tok/sec: 67095.4 | mem: 72099MB | Time Left: 01 hrs : 03 mins
[TIMING] data: 13.5ms | fwd: 2410.1ms | loss: 4.8ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.6ms
step 1161 | loss: 2.703085 | lr 1.8000e-04 | norm: 0.3460 | dt: 7884.30ms | tok/sec: 66497.7 | mem: 72099MB | Time Left: 01 hrs : 04 mins
[TIMING] data: 15.9ms | fwd: 2473.4ms | loss: 4.8ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1162 | loss: 2.771257 | lr 1.8000e-04 | norm: 0.2790 | dt: 7812.48ms | tok/sec: 67109.0 | mem: 72099MB | Time Left: 01 hrs : 03 mins
[TIMING] data: 17.8ms | fwd: 2405.2ms | loss: 4.8ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.5ms
step 1163 | loss: 2.828936 | lr 1.8000e-04 | norm: 0.3306 | dt: 7820.97ms | tok/sec: 67036.2 | mem: 72099MB | Time Left: 01 hrs : 03 mins
[TIMING] data: 14.8ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.5ms
step 1164 | loss: 2.841625 | lr 1.8000e-04 | norm: 0.2850 | dt: 7820.93ms | tok/sec: 67036.5 | mem: 72099MB | Time Left: 01 hrs : 03 mins
[TIMING] data: 14.4ms | fwd: 2414.0ms | loss: 4.8ms | bwd: 5354.1ms | clip: 0.0ms | optim: 23.5ms
step 1165 | loss: 2.626571 | lr 1.8000e-04 | norm: 0.3058 | dt: 7879.59ms | tok/sec: 66537.5 | mem: 72099MB | Time Left: 01 hrs : 03 mins
[TIMING] data: 14.2ms | fwd: 2476.8ms | loss: 4.8ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.5ms
step 1166 | loss: 2.368817 | lr 1.8000e-04 | norm: 0.3112 | dt: 7817.91ms | tok/sec: 67062.4 | mem: 72099MB | Time Left: 01 hrs : 03 mins
[TIMING] data: 17.9ms | fwd: 2404.7ms | loss: 4.8ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 1167 | loss: 2.769201 | lr 1.8000e-04 | norm: 0.3000 | dt: 7813.41ms | tok/sec: 67101.0 | mem: 72099MB | Time Left: 01 hrs : 03 mins
[TIMING] data: 13.7ms | fwd: 2408.2ms | loss: 4.8ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.7ms
step 1168 | loss: 2.748942 | lr 1.8000e-04 | norm: 0.3168 | dt: 7816.34ms | tok/sec: 67075.9 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 15.6ms | fwd: 2406.5ms | loss: 4.8ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.7ms
step 1169 | loss: 2.733431 | lr 1.8000e-04 | norm: 0.5324 | dt: 7814.74ms | tok/sec: 67089.6 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 12.7ms | fwd: 2409.2ms | loss: 4.8ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.5ms
step 1170 | loss: 2.798992 | lr 1.8000e-04 | norm: 0.3563 | dt: 7810.67ms | tok/sec: 67124.6 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 17.5ms | fwd: 2402.5ms | loss: 4.8ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.6ms
step 1171 | loss: 2.703026 | lr 1.8000e-04 | norm: 0.2938 | dt: 7811.46ms | tok/sec: 67117.8 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 14.5ms | fwd: 2405.7ms | loss: 4.8ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.5ms
step 1172 | loss: 2.895680 | lr 1.8000e-04 | norm: 0.3700 | dt: 7822.78ms | tok/sec: 67020.7 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 14.2ms | fwd: 2408.5ms | loss: 4.8ms | bwd: 5362.4ms | clip: 0.0ms | optim: 23.5ms
step 1173 | loss: 2.787272 | lr 1.8000e-04 | norm: 0.3013 | dt: 7894.78ms | tok/sec: 66409.5 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 14.8ms | fwd: 2480.9ms | loss: 4.8ms | bwd: 5360.2ms | clip: 0.0ms | optim: 23.6ms
step 1174 | loss: 2.882004 | lr 1.8000e-04 | norm: 0.3846 | dt: 7889.24ms | tok/sec: 66456.1 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 17.1ms | fwd: 2475.5ms | loss: 4.8ms | bwd: 5357.9ms | clip: 0.0ms | optim: 23.5ms
step 1175 | loss: 2.607023 | lr 1.8000e-04 | norm: 0.3730 | dt: 7884.61ms | tok/sec: 66495.1 | mem: 72099MB | Time Left: 01 hrs : 02 mins
[TIMING] data: 13.1ms | fwd: 2477.8ms | loss: 4.8ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 1176 | loss: 2.276398 | lr 1.8000e-04 | norm: 0.3120 | dt: 7812.98ms | tok/sec: 67104.7 | mem: 72099MB | Time Left: 01 hrs : 01 mins
[TIMING] data: 13.5ms | fwd: 2408.5ms | loss: 4.8ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1177 | loss: 2.482641 | lr 1.8000e-04 | norm: 0.3207 | dt: 7825.64ms | tok/sec: 66996.2 | mem: 72099MB | Time Left: 01 hrs : 01 mins
[TIMING] data: 17.0ms | fwd: 2410.7ms | loss: 4.8ms | bwd: 5359.3ms | clip: 0.0ms | optim: 23.6ms
step 1178 | loss: 2.642052 | lr 1.8000e-04 | norm: 0.3237 | dt: 7819.66ms | tok/sec: 67047.4 | mem: 72099MB | Time Left: 01 hrs : 01 mins
[TIMING] data: 14.3ms | fwd: 2410.9ms | loss: 4.8ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 1179 | loss: 2.499576 | lr 1.8000e-04 | norm: 0.3050 | dt: 7818.43ms | tok/sec: 67058.0 | mem: 72099MB | Time Left: 01 hrs : 01 mins
[TIMING] data: 14.5ms | fwd: 2411.6ms | loss: 4.8ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.4ms
step 1180 | loss: 2.423213 | lr 1.8000e-04 | norm: 0.3181 | dt: 7820.45ms | tok/sec: 67040.7 | mem: 72099MB | Time Left: 01 hrs : 01 mins
[TIMING] data: 13.3ms | fwd: 2409.6ms | loss: 4.8ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 1181 | loss: 2.549380 | lr 1.8000e-04 | norm: 0.3556 | dt: 7816.38ms | tok/sec: 67075.6 | mem: 72099MB | Time Left: 01 hrs : 01 mins
[TIMING] data: 17.5ms | fwd: 2410.2ms | loss: 4.8ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.5ms
step 1182 | loss: 2.617488 | lr 1.8000e-04 | norm: 0.3176 | dt: 7813.89ms | tok/sec: 67096.9 | mem: 72099MB | Time Left: 01 hrs : 01 mins
[TIMING] data: 15.0ms | fwd: 2405.7ms | loss: 4.8ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 1183 | loss: 2.737719 | lr 1.8000e-04 | norm: 0.3677 | dt: 7817.21ms | tok/sec: 67068.4 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 14.3ms | fwd: 2404.6ms | loss: 4.8ms | bwd: 5360.2ms | clip: 0.0ms | optim: 23.6ms
step 1184 | loss: 2.422019 | lr 1.8000e-04 | norm: 0.3209 | dt: 7814.39ms | tok/sec: 67092.6 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 12.6ms | fwd: 2410.6ms | loss: 4.8ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.6ms
step 1185 | loss: 2.512711 | lr 1.8000e-04 | norm: 0.4010 | dt: 7812.86ms | tok/sec: 67105.8 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 11.6ms | fwd: 2414.9ms | loss: 4.8ms | bwd: 5347.4ms | clip: 0.0ms | optim: 23.7ms
step 1186 | loss: 2.601442 | lr 1.8000e-04 | norm: 0.3113 | dt: 7869.24ms | tok/sec: 66625.0 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 14.0ms | fwd: 2464.3ms | loss: 4.8ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 1187 | loss: 2.579043 | lr 1.8000e-04 | norm: 0.3580 | dt: 7807.09ms | tok/sec: 67155.4 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 12.5ms | fwd: 2400.9ms | loss: 4.8ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.5ms
step 1188 | loss: 2.571261 | lr 1.8000e-04 | norm: 0.3287 | dt: 7815.44ms | tok/sec: 67083.6 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 14.1ms | fwd: 2408.7ms | loss: 4.8ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 1189 | loss: 2.436512 | lr 1.8000e-04 | norm: 0.4132 | dt: 7806.65ms | tok/sec: 67159.2 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 14.1ms | fwd: 2400.3ms | loss: 4.8ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.6ms
step 1190 | loss: 2.455115 | lr 1.8000e-04 | norm: 0.3955 | dt: 7810.09ms | tok/sec: 67129.5 | mem: 72099MB | Time Left: 01 hrs : 00 mins
[TIMING] data: 14.6ms | fwd: 2407.1ms | loss: 4.8ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.5ms
step 1191 | loss: 2.546046 | lr 1.8000e-04 | norm: 0.4855 | dt: 7811.05ms | tok/sec: 67121.4 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 15.1ms | fwd: 2404.8ms | loss: 4.8ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.5ms
step 1192 | loss: 2.528792 | lr 1.8000e-04 | norm: 0.5169 | dt: 7816.84ms | tok/sec: 67071.6 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 15.2ms | fwd: 2411.1ms | loss: 4.8ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.4ms
step 1193 | loss: 2.502843 | lr 1.8000e-04 | norm: 0.4088 | dt: 7819.85ms | tok/sec: 67045.8 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 17.6ms | fwd: 2411.2ms | loss: 4.8ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 1194 | loss: 2.388406 | lr 1.8000e-04 | norm: 0.4771 | dt: 7810.53ms | tok/sec: 67125.8 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 14.0ms | fwd: 2404.7ms | loss: 4.8ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 1195 | loss: 2.337172 | lr 1.8000e-04 | norm: 0.4245 | dt: 7819.19ms | tok/sec: 67051.5 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 13.6ms | fwd: 2413.6ms | loss: 4.8ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 1196 | loss: 2.380273 | lr 1.8000e-04 | norm: 0.3886 | dt: 7814.10ms | tok/sec: 67095.1 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 13.6ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.5ms
step 1197 | loss: 2.532571 | lr 1.8000e-04 | norm: 0.4714 | dt: 7861.26ms | tok/sec: 66692.6 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 15.0ms | fwd: 2397.3ms | loss: 4.8ms | bwd: 5409.2ms | clip: 0.0ms | optim: 23.6ms
step 1198 | loss: 2.580435 | lr 1.8000e-04 | norm: 0.3853 | dt: 7820.09ms | tok/sec: 67043.7 | mem: 72099MB | Time Left: 00 hrs : 59 mins
[TIMING] data: 14.7ms | fwd: 2410.6ms | loss: 4.8ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.7ms
step 1199 | loss: 2.525365 | lr 1.8000e-04 | norm: 0.4509 | dt: 7820.09ms | tok/sec: 67043.7 | mem: 72099MB | Time Left: 00 hrs : 58 mins
[TIMING] data: 14.8ms | fwd: 2415.8ms | loss: 4.8ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1200.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1000.ckpt
step 1200 | loss: 2.469009 | lr 1.8000e-04 | norm: 0.4327 | dt: 7825.21ms | tok/sec: 66999.8 | mem: 72099MB | Time Left: 00 hrs : 58 mins
[TIMING] data: 13.4ms | fwd: 2433.8ms | loss: 4.9ms | bwd: 5338.6ms | clip: 0.0ms | optim: 23.4ms
step 1201 | loss: 2.074116 | lr 1.8000e-04 | norm: 0.3697 | dt: 7788.42ms | tok/sec: 67316.4 | mem: 72099MB | Time Left: 00 hrs : 58 mins
[TIMING] data: 11.4ms | fwd: 2392.5ms | loss: 4.8ms | bwd: 5345.2ms | clip: 0.0ms | optim: 23.5ms
step 1202 | loss: 2.295425 | lr 1.8000e-04 | norm: 0.4153 | dt: 7800.64ms | tok/sec: 67210.9 | mem: 72099MB | Time Left: 00 hrs : 58 mins
[TIMING] data: 12.3ms | fwd: 2402.8ms | loss: 4.9ms | bwd: 5345.5ms | clip: 0.0ms | optim: 23.6ms
step 1203 | loss: 2.505186 | lr 1.8000e-04 | norm: 0.5233 | dt: 7808.79ms | tok/sec: 67140.7 | mem: 72099MB | Time Left: 00 hrs : 58 mins
[TIMING] data: 18.0ms | fwd: 2402.7ms | loss: 4.9ms | bwd: 5347.2ms | clip: 0.0ms | optim: 23.5ms
step 1204 | loss: 2.464781 | lr 1.8000e-04 | norm: 0.3905 | dt: 7823.42ms | tok/sec: 67015.2 | mem: 72099MB | Time Left: 00 hrs : 58 mins
[TIMING] data: 32.8ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5345.3ms | clip: 0.0ms | optim: 23.5ms
step 1205 | loss: 2.586101 | lr 1.8000e-04 | norm: 0.4317 | dt: 7855.63ms | tok/sec: 66740.4 | mem: 72099MB | Time Left: 00 hrs : 58 mins
[TIMING] data: 64.3ms | fwd: 2399.1ms | loss: 4.9ms | bwd: 5351.0ms | clip: 0.0ms | optim: 23.5ms
step 1206 | loss: 2.314411 | lr 1.8000e-04 | norm: 0.4552 | dt: 7807.96ms | tok/sec: 67147.9 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 13.3ms | fwd: 2402.8ms | loss: 4.9ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.6ms
step 1207 | loss: 2.384947 | lr 1.8000e-04 | norm: 0.4203 | dt: 7816.80ms | tok/sec: 67071.9 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 14.7ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.5ms
step 1208 | loss: 2.394536 | lr 1.8000e-04 | norm: 0.4097 | dt: 7816.17ms | tok/sec: 67077.4 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 15.6ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 1209 | loss: 2.208231 | lr 1.8000e-04 | norm: 0.4087 | dt: 7805.28ms | tok/sec: 67170.9 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 14.2ms | fwd: 2402.0ms | loss: 4.9ms | bwd: 5348.4ms | clip: 0.0ms | optim: 23.6ms
step 1210 | loss: 2.056485 | lr 1.8000e-04 | norm: 0.3825 | dt: 7807.35ms | tok/sec: 67153.1 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 15.2ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5346.7ms | clip: 0.0ms | optim: 23.6ms
step 1211 | loss: 2.241603 | lr 1.8000e-04 | norm: 0.4342 | dt: 7814.64ms | tok/sec: 67090.5 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 14.3ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.6ms
step 1212 | loss: 2.319663 | lr 1.8000e-04 | norm: 0.5224 | dt: 7819.66ms | tok/sec: 67047.4 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 14.8ms | fwd: 2412.7ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 1213 | loss: 2.074671 | lr 1.8000e-04 | norm: 0.4544 | dt: 7811.68ms | tok/sec: 67115.9 | mem: 72099MB | Time Left: 00 hrs : 57 mins
[TIMING] data: 14.5ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 1214 | loss: 2.323332 | lr 1.8000e-04 | norm: 0.5186 | dt: 7808.51ms | tok/sec: 67143.1 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 15.7ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5347.6ms | clip: 0.0ms | optim: 23.5ms
step 1215 | loss: 2.260353 | lr 1.8000e-04 | norm: 0.5145 | dt: 7828.85ms | tok/sec: 66968.7 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 13.9ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5361.5ms | clip: 0.0ms | optim: 23.7ms
step 1216 | loss: 2.242217 | lr 1.8000e-04 | norm: 0.5210 | dt: 7812.44ms | tok/sec: 67109.4 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 14.0ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5349.6ms | clip: 0.0ms | optim: 23.6ms
step 1217 | loss: 2.250883 | lr 1.8000e-04 | norm: 0.5087 | dt: 7816.61ms | tok/sec: 67073.5 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 12.9ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5359.9ms | clip: 0.0ms | optim: 23.6ms
step 1218 | loss: 2.196702 | lr 1.8000e-04 | norm: 0.5129 | dt: 7820.96ms | tok/sec: 67036.3 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 13.3ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 1219 | loss: 2.387649 | lr 1.8000e-04 | norm: 0.5773 | dt: 7813.09ms | tok/sec: 67103.8 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 13.9ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1220 | loss: 2.233498 | lr 1.8000e-04 | norm: 0.4439 | dt: 7887.42ms | tok/sec: 66471.4 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 14.5ms | fwd: 2476.5ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.5ms
step 1221 | loss: 2.238431 | lr 1.8000e-04 | norm: 0.6144 | dt: 7820.55ms | tok/sec: 67039.8 | mem: 72099MB | Time Left: 00 hrs : 56 mins
[TIMING] data: 15.0ms | fwd: 2413.8ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.6ms
step 1222 | loss: 2.237464 | lr 1.8000e-04 | norm: 0.5184 | dt: 7812.18ms | tok/sec: 67111.6 | mem: 72099MB | Time Left: 00 hrs : 55 mins
[TIMING] data: 15.0ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.6ms
step 1223 | loss: 2.017622 | lr 1.8000e-04 | norm: 0.4541 | dt: 7818.57ms | tok/sec: 67056.8 | mem: 72099MB | Time Left: 00 hrs : 55 mins
[TIMING] data: 13.3ms | fwd: 2410.2ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.5ms
step 1224 | loss: 2.291095 | lr 1.8000e-04 | norm: 0.5180 | dt: 7815.74ms | tok/sec: 67081.1 | mem: 72099MB | Time Left: 00 hrs : 55 mins
[TIMING] data: 13.1ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 1225 | loss: 2.091800 | lr 1.8000e-04 | norm: 0.5302 | dt: 7811.81ms | tok/sec: 67114.8 | mem: 72099MB | Time Left: 00 hrs : 55 mins
[TIMING] data: 13.9ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 1226 | loss: 2.332454 | lr 1.8000e-04 | norm: 0.5581 | dt: 7845.85ms | tok/sec: 66823.6 | mem: 72099MB | Time Left: 00 hrs : 55 mins
[TIMING] data: 18.8ms | fwd: 2434.9ms | loss: 4.9ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.6ms
step 1227 | loss: 2.174400 | lr 1.8000e-04 | norm: 0.5173 | dt: 7814.92ms | tok/sec: 67088.1 | mem: 72099MB | Time Left: 00 hrs : 55 mins
[TIMING] data: 13.1ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.5ms
step 1228 | loss: 2.008437 | lr 1.8000e-04 | norm: 0.5736 | dt: 7817.45ms | tok/sec: 67066.4 | mem: 72099MB | Time Left: 00 hrs : 55 mins
[TIMING] data: 15.4ms | fwd: 2414.1ms | loss: 4.9ms | bwd: 5346.6ms | clip: 0.0ms | optim: 23.5ms
step 1229 | loss: 2.354146 | lr 1.8000e-04 | norm: 0.5903 | dt: 7812.98ms | tok/sec: 67104.8 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 13.9ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 1230 | loss: 2.120149 | lr 1.8000e-04 | norm: 0.5250 | dt: 7819.15ms | tok/sec: 67051.8 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 14.8ms | fwd: 2410.8ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.7ms
step 1231 | loss: 2.192447 | lr 1.8000e-04 | norm: 0.6028 | dt: 7817.14ms | tok/sec: 67069.0 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 14.9ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.5ms
step 1232 | loss: 2.294483 | lr 1.8000e-04 | norm: 0.5653 | dt: 7816.49ms | tok/sec: 67074.6 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 15.2ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.5ms
step 1233 | loss: 2.004633 | lr 1.8000e-04 | norm: 0.5293 | dt: 7810.07ms | tok/sec: 67129.8 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 15.3ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5344.8ms | clip: 0.0ms | optim: 23.6ms
step 1234 | loss: 2.205567 | lr 1.8000e-04 | norm: 0.5316 | dt: 7817.54ms | tok/sec: 67065.6 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 13.7ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.5ms
step 1235 | loss: 2.059828 | lr 1.8000e-04 | norm: 0.5529 | dt: 7804.42ms | tok/sec: 67178.3 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 13.6ms | fwd: 2399.5ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 1236 | loss: 2.217766 | lr 1.8000e-04 | norm: 0.5506 | dt: 7816.07ms | tok/sec: 67078.2 | mem: 72099MB | Time Left: 00 hrs : 54 mins
[TIMING] data: 16.2ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.4ms
step 1237 | loss: 2.135288 | lr 1.8000e-04 | norm: 0.5819 | dt: 7819.99ms | tok/sec: 67044.6 | mem: 72099MB | Time Left: 00 hrs : 53 mins
[TIMING] data: 19.8ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.5ms
step 1238 | loss: 2.072137 | lr 1.8000e-04 | norm: 0.5217 | dt: 7811.01ms | tok/sec: 67121.6 | mem: 72099MB | Time Left: 00 hrs : 53 mins
[TIMING] data: 15.7ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 1239 | loss: 1.926937 | lr 1.8000e-04 | norm: 0.5857 | dt: 7819.67ms | tok/sec: 67047.4 | mem: 72099MB | Time Left: 00 hrs : 53 mins
[TIMING] data: 15.4ms | fwd: 2414.5ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 1240 | loss: 1.909543 | lr 1.8000e-04 | norm: 0.6095 | dt: 7812.44ms | tok/sec: 67109.4 | mem: 72099MB | Time Left: 00 hrs : 53 mins
[TIMING] data: 15.6ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5347.7ms | clip: 0.0ms | optim: 23.6ms
step 1241 | loss: 2.023264 | lr 1.8000e-04 | norm: 0.7373 | dt: 7819.96ms | tok/sec: 67044.9 | mem: 72099MB | Time Left: 00 hrs : 53 mins
[TIMING] data: 18.7ms | fwd: 2413.0ms | loss: 4.9ms | bwd: 5346.8ms | clip: 0.0ms | optim: 23.6ms
step 1242 | loss: 2.030682 | lr 1.8000e-04 | norm: 0.6608 | dt: 7814.61ms | tok/sec: 67090.8 | mem: 72099MB | Time Left: 00 hrs : 53 mins
[TIMING] data: 13.9ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5348.0ms | clip: 0.0ms | optim: 23.6ms
step 1243 | loss: 2.184453 | lr 1.8000e-04 | norm: 0.6686 | dt: 7820.21ms | tok/sec: 67042.7 | mem: 72099MB | Time Left: 00 hrs : 53 mins
[TIMING] data: 15.9ms | fwd: 2411.9ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1244 | loss: 1.924561 | lr 1.8000e-04 | norm: 0.6715 | dt: 7809.14ms | tok/sec: 67137.7 | mem: 72099MB | Time Left: 00 hrs : 52 mins
[TIMING] data: 15.9ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.5ms
step 1245 | loss: 1.929209 | lr 1.8000e-04 | norm: 0.5946 | dt: 7821.00ms | tok/sec: 67035.9 | mem: 72099MB | Time Left: 00 hrs : 52 mins
[TIMING] data: 18.7ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 1246 | loss: 2.200744 | lr 1.8000e-04 | norm: 0.7295 | dt: 7816.61ms | tok/sec: 67073.6 | mem: 72099MB | Time Left: 00 hrs : 52 mins
[TIMING] data: 14.7ms | fwd: 2404.9ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.5ms
step 1247 | loss: 2.001815 | lr 1.8000e-04 | norm: 0.7115 | dt: 7820.73ms | tok/sec: 67038.3 | mem: 72099MB | Time Left: 00 hrs : 52 mins
[TIMING] data: 16.4ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 1248 | loss: 1.911315 | lr 1.8000e-04 | norm: 0.6055 | dt: 7809.23ms | tok/sec: 67136.9 | mem: 72099MB | Time Left: 00 hrs : 52 mins
[TIMING] data: 11.4ms | fwd: 2398.6ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.7ms
step 1249 | loss: 1.862983 | lr 1.8000e-04 | norm: 0.6602 | dt: 7826.40ms | tok/sec: 66989.7 | mem: 72099MB | Time Left: 00 hrs : 52 mins
[TIMING] data: 15.1ms | fwd: 2418.1ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
validation loss: 2.0000
step 1250 | loss: 2.092338 | lr 1.8000e-04 | norm: 0.6472 | dt: 13786.86ms | tok/sec: 38028.1 | mem: 72099MB | Time Left: 01 hrs : 32 mins
[TIMING] data: 14.1ms | fwd: 2402.8ms | loss: 4.9ms | bwd: 5348.3ms | clip: 0.0ms | optim: 23.5ms
step 1251 | loss: 2.192431 | lr 1.8000e-04 | norm: 0.8099 | dt: 7820.08ms | tok/sec: 67043.8 | mem: 72099MB | Time Left: 00 hrs : 52 mins
[TIMING] data: 14.3ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.5ms
step 1252 | loss: 2.146884 | lr 1.8000e-04 | norm: 0.6518 | dt: 7815.42ms | tok/sec: 67083.8 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 17.9ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.6ms
step 1253 | loss: 2.042071 | lr 1.8000e-04 | norm: 0.6746 | dt: 7815.88ms | tok/sec: 67079.8 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 14.9ms | fwd: 2408.5ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.6ms
step 1254 | loss: 1.935817 | lr 1.8000e-04 | norm: 0.7137 | dt: 7809.69ms | tok/sec: 67133.0 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 13.6ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.4ms
step 1255 | loss: 2.159811 | lr 1.8000e-04 | norm: 0.7761 | dt: 7810.06ms | tok/sec: 67129.8 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 14.2ms | fwd: 2399.3ms | loss: 4.9ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.6ms
step 1256 | loss: 1.785673 | lr 1.8000e-04 | norm: 0.6448 | dt: 7816.68ms | tok/sec: 67073.0 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 12.2ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.5ms
step 1257 | loss: 1.721801 | lr 1.8000e-04 | norm: 0.6567 | dt: 7818.12ms | tok/sec: 67060.7 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 14.9ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 1258 | loss: 2.058389 | lr 1.8000e-04 | norm: 0.6583 | dt: 7806.01ms | tok/sec: 67164.6 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 14.1ms | fwd: 2399.5ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
step 1259 | loss: 1.825542 | lr 1.8000e-04 | norm: 0.6835 | dt: 7822.69ms | tok/sec: 67021.4 | mem: 72099MB | Time Left: 00 hrs : 51 mins
[TIMING] data: 16.4ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5358.4ms | clip: 0.0ms | optim: 23.5ms
step 1260 | loss: 2.100982 | lr 1.8000e-04 | norm: 0.7431 | dt: 7824.83ms | tok/sec: 67003.1 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 15.2ms | fwd: 2419.8ms | loss: 4.9ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.5ms
step 1261 | loss: 1.656557 | lr 1.8000e-04 | norm: 0.6931 | dt: 7820.76ms | tok/sec: 67038.0 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 16.1ms | fwd: 2409.9ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
step 1262 | loss: 1.857690 | lr 1.8000e-04 | norm: 0.6952 | dt: 7818.18ms | tok/sec: 67060.1 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 13.2ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.4ms
step 1263 | loss: 1.903728 | lr 1.8000e-04 | norm: 0.7279 | dt: 7823.34ms | tok/sec: 67015.9 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 16.6ms | fwd: 2409.4ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.5ms
step 1264 | loss: 1.755751 | lr 1.8000e-04 | norm: 0.7485 | dt: 7810.54ms | tok/sec: 67125.7 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 15.8ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5346.5ms | clip: 0.0ms | optim: 23.6ms
step 1265 | loss: 1.730398 | lr 1.8000e-04 | norm: 0.7008 | dt: 7807.84ms | tok/sec: 67148.9 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 14.4ms | fwd: 2401.5ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.5ms
step 1266 | loss: 1.824821 | lr 1.8000e-04 | norm: 0.7483 | dt: 7814.42ms | tok/sec: 67092.4 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 15.1ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.6ms
step 1267 | loss: 1.769657 | lr 1.8000e-04 | norm: 0.7506 | dt: 7822.26ms | tok/sec: 67025.1 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 13.9ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 1268 | loss: 1.618674 | lr 1.8000e-04 | norm: 0.7188 | dt: 7853.73ms | tok/sec: 66756.5 | mem: 72099MB | Time Left: 00 hrs : 50 mins
[TIMING] data: 15.1ms | fwd: 2449.1ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.6ms
step 1269 | loss: 1.984002 | lr 1.8000e-04 | norm: 0.8579 | dt: 7816.98ms | tok/sec: 67070.4 | mem: 72099MB | Time Left: 00 hrs : 49 mins
[TIMING] data: 14.5ms | fwd: 2410.8ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.6ms
step 1270 | loss: 1.892820 | lr 1.8000e-04 | norm: 0.7320 | dt: 7819.39ms | tok/sec: 67049.8 | mem: 72099MB | Time Left: 00 hrs : 49 mins
[TIMING] data: 15.0ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
step 1271 | loss: 1.846342 | lr 1.8000e-04 | norm: 0.8179 | dt: 7814.25ms | tok/sec: 67093.8 | mem: 72099MB | Time Left: 00 hrs : 49 mins
[TIMING] data: 18.9ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5351.0ms | clip: 0.0ms | optim: 23.6ms
step 1272 | loss: 1.827705 | lr 1.8000e-04 | norm: 0.7927 | dt: 7811.00ms | tok/sec: 67121.7 | mem: 72099MB | Time Left: 00 hrs : 49 mins
[TIMING] data: 14.1ms | fwd: 2401.8ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 1273 | loss: 1.924919 | lr 1.8000e-04 | norm: 0.7786 | dt: 7813.62ms | tok/sec: 67099.2 | mem: 72099MB | Time Left: 00 hrs : 49 mins
[TIMING] data: 13.0ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.6ms
step 1274 | loss: 1.986733 | lr 1.8000e-04 | norm: 0.9272 | dt: 7818.53ms | tok/sec: 67057.1 | mem: 72099MB | Time Left: 00 hrs : 49 mins
[TIMING] data: 12.3ms | fwd: 2414.4ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.6ms
step 1275 | loss: 1.776429 | lr 1.8000e-04 | norm: 0.7304 | dt: 7825.13ms | tok/sec: 67000.5 | mem: 72099MB | Time Left: 00 hrs : 49 mins
[TIMING] data: 14.6ms | fwd: 2413.1ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.6ms
step 1276 | loss: 1.826254 | lr 1.8000e-04 | norm: 0.8645 | dt: 7817.68ms | tok/sec: 67064.4 | mem: 72099MB | Time Left: 00 hrs : 48 mins
[TIMING] data: 17.4ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.6ms
step 1277 | loss: 1.759482 | lr 1.8000e-04 | norm: 0.8942 | dt: 7815.69ms | tok/sec: 67081.5 | mem: 72099MB | Time Left: 00 hrs : 48 mins
[TIMING] data: 14.4ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.6ms
step 1278 | loss: 1.534371 | lr 1.8000e-04 | norm: 0.7384 | dt: 7819.26ms | tok/sec: 67050.8 | mem: 72099MB | Time Left: 00 hrs : 48 mins
[TIMING] data: 15.3ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1279 | loss: 1.665461 | lr 1.8000e-04 | norm: 0.8857 | dt: 7810.77ms | tok/sec: 67123.7 | mem: 72099MB | Time Left: 00 hrs : 48 mins
[TIMING] data: 19.4ms | fwd: 2405.1ms | loss: 4.9ms | bwd: 5346.0ms | clip: 0.0ms | optim: 23.4ms
step 1280 | loss: 1.694126 | lr 1.8000e-04 | norm: 0.8268 | dt: 7816.25ms | tok/sec: 67076.7 | mem: 72099MB | Time Left: 00 hrs : 48 mins
[TIMING] data: 16.1ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.5ms
step 1281 | loss: 1.645968 | lr 1.8000e-04 | norm: 0.8118 | dt: 7816.52ms | tok/sec: 67074.4 | mem: 72099MB | Time Left: 00 hrs : 48 mins
[TIMING] data: 15.0ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.7ms
step 1282 | loss: 1.735278 | lr 1.8000e-04 | norm: 0.8212 | dt: 7803.76ms | tok/sec: 67184.0 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 14.8ms | fwd: 2398.8ms | loss: 4.9ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.5ms
step 1283 | loss: 1.610270 | lr 1.8000e-04 | norm: 0.8025 | dt: 7806.10ms | tok/sec: 67163.9 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 12.6ms | fwd: 2400.6ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.5ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00007.bin
100000000
step 1284 | loss: 1.861776 | lr 1.8000e-04 | norm: 0.8815 | dt: 7833.36ms | tok/sec: 66930.1 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 29.7ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.5ms
step 1285 | loss: 1.597304 | lr 1.8000e-04 | norm: 0.8303 | dt: 7814.90ms | tok/sec: 67088.3 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 14.5ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 1286 | loss: 1.635956 | lr 1.8000e-04 | norm: 0.8319 | dt: 7815.96ms | tok/sec: 67079.1 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 17.3ms | fwd: 2403.7ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 1287 | loss: 1.701271 | lr 1.8000e-04 | norm: 0.8401 | dt: 7815.83ms | tok/sec: 67080.3 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 15.0ms | fwd: 2404.7ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 1288 | loss: 1.793898 | lr 1.8000e-04 | norm: 0.9380 | dt: 7815.04ms | tok/sec: 67087.0 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 15.1ms | fwd: 2402.6ms | loss: 4.9ms | bwd: 5357.0ms | clip: 0.0ms | optim: 23.5ms
step 1289 | loss: 1.568918 | lr 1.8000e-04 | norm: 0.8775 | dt: 7874.98ms | tok/sec: 66576.4 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 15.1ms | fwd: 2466.0ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.5ms
step 1290 | loss: 1.830605 | lr 1.8000e-04 | norm: 0.9042 | dt: 7817.55ms | tok/sec: 67065.5 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 16.8ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.5ms
step 1291 | loss: 1.585046 | lr 1.8000e-04 | norm: 0.9149 | dt: 7807.50ms | tok/sec: 67151.8 | mem: 72099MB | Time Left: 00 hrs : 46 mins
[TIMING] data: 14.5ms | fwd: 2399.6ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 1292 | loss: 1.762243 | lr 1.8000e-04 | norm: 0.9509 | dt: 7863.41ms | tok/sec: 66674.4 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 13.6ms | fwd: 2458.1ms | loss: 4.9ms | bwd: 5351.6ms | clip: 0.0ms | optim: 23.5ms
step 1293 | loss: 1.709228 | lr 1.8000e-04 | norm: 0.9833 | dt: 7810.07ms | tok/sec: 67129.8 | mem: 72099MB | Time Left: 00 hrs : 46 mins
[TIMING] data: 14.5ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5347.4ms | clip: 0.0ms | optim: 23.6ms
step 1294 | loss: 1.747816 | lr 1.8000e-04 | norm: 0.9250 | dt: 7813.04ms | tok/sec: 67104.2 | mem: 72099MB | Time Left: 00 hrs : 46 mins
[TIMING] data: 14.4ms | fwd: 2402.6ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.4ms
step 1295 | loss: 1.745513 | lr 1.8000e-04 | norm: 0.9317 | dt: 7810.44ms | tok/sec: 67126.5 | mem: 72099MB | Time Left: 00 hrs : 46 mins
[TIMING] data: 14.9ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.5ms
step 1296 | loss: 1.913510 | lr 1.8000e-04 | norm: 1.0129 | dt: 7813.13ms | tok/sec: 67103.5 | mem: 72099MB | Time Left: 00 hrs : 46 mins
[TIMING] data: 14.7ms | fwd: 2402.9ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 1297 | loss: 1.742359 | lr 1.8000e-04 | norm: 1.0049 | dt: 7820.01ms | tok/sec: 67044.4 | mem: 72099MB | Time Left: 00 hrs : 46 mins
[TIMING] data: 14.0ms | fwd: 2415.3ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.6ms
step 1298 | loss: 1.582077 | lr 1.8000e-04 | norm: 1.1399 | dt: 7810.67ms | tok/sec: 67124.6 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 13.7ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.5ms
step 1299 | loss: 1.424486 | lr 1.8000e-04 | norm: 0.8012 | dt: 7807.95ms | tok/sec: 67148.0 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 14.7ms | fwd: 2400.9ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.6ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1300.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1100.ckpt
step 1300 | loss: 1.561893 | lr 1.8000e-04 | norm: 0.8636 | dt: 8161.71ms | tok/sec: 64237.5 | mem: 72099MB | Time Left: 00 hrs : 47 mins
[TIMING] data: 13.7ms | fwd: 2765.3ms | loss: 4.9ms | bwd: 5342.7ms | clip: 0.0ms | optim: 23.5ms
step 1301 | loss: 1.483114 | lr 1.8000e-04 | norm: 0.8840 | dt: 7792.00ms | tok/sec: 67285.4 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 13.7ms | fwd: 2394.5ms | loss: 4.9ms | bwd: 5344.3ms | clip: 0.0ms | optim: 23.5ms
step 1302 | loss: 1.579002 | lr 1.8000e-04 | norm: 0.9689 | dt: 7798.66ms | tok/sec: 67227.9 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 15.1ms | fwd: 2397.2ms | loss: 4.9ms | bwd: 5346.2ms | clip: 0.0ms | optim: 23.5ms
step 1303 | loss: 1.597895 | lr 1.8000e-04 | norm: 1.0268 | dt: 7806.95ms | tok/sec: 67156.6 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 21.4ms | fwd: 2399.8ms | loss: 4.9ms | bwd: 5345.9ms | clip: 0.0ms | optim: 23.6ms
step 1304 | loss: 1.641950 | lr 1.8000e-04 | norm: 0.8910 | dt: 7905.10ms | tok/sec: 66322.7 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 47.0ms | fwd: 2462.7ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.5ms
step 1305 | loss: 1.668032 | lr 1.8000e-04 | norm: 1.0183 | dt: 7833.60ms | tok/sec: 66928.1 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 50.2ms | fwd: 2399.7ms | loss: 4.9ms | bwd: 5343.4ms | clip: 0.0ms | optim: 23.6ms
step 1306 | loss: 1.724787 | lr 1.8000e-04 | norm: 0.9647 | dt: 7837.90ms | tok/sec: 66891.4 | mem: 72099MB | Time Left: 00 hrs : 45 mins
[TIMING] data: 13.4ms | fwd: 2436.2ms | loss: 4.9ms | bwd: 5347.5ms | clip: 0.0ms | optim: 23.5ms
step 1307 | loss: 1.685119 | lr 1.8000e-04 | norm: 1.0233 | dt: 7802.64ms | tok/sec: 67193.6 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 12.6ms | fwd: 2400.9ms | loss: 4.9ms | bwd: 5349.2ms | clip: 0.0ms | optim: 23.6ms
step 1308 | loss: 1.566702 | lr 1.8000e-04 | norm: 0.9750 | dt: 7801.76ms | tok/sec: 67201.3 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 13.8ms | fwd: 2400.6ms | loss: 4.9ms | bwd: 5346.8ms | clip: 0.0ms | optim: 23.7ms
step 1309 | loss: 1.626250 | lr 1.8000e-04 | norm: 1.0407 | dt: 7814.23ms | tok/sec: 67094.0 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 13.8ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.6ms
step 1310 | loss: 1.563033 | lr 1.8000e-04 | norm: 1.0047 | dt: 7817.99ms | tok/sec: 67061.7 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 19.1ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.6ms
step 1311 | loss: 1.403434 | lr 1.8000e-04 | norm: 0.9372 | dt: 7881.23ms | tok/sec: 66523.6 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 14.3ms | fwd: 2475.8ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.7ms
step 1312 | loss: 1.590053 | lr 1.8000e-04 | norm: 0.9199 | dt: 7812.07ms | tok/sec: 67112.5 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 13.0ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 1313 | loss: 1.650931 | lr 1.8000e-04 | norm: 0.9726 | dt: 7811.55ms | tok/sec: 67117.0 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 13.0ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 1314 | loss: 1.511894 | lr 1.8000e-04 | norm: 1.0568 | dt: 7813.14ms | tok/sec: 67103.4 | mem: 72099MB | Time Left: 00 hrs : 43 mins
[TIMING] data: 16.3ms | fwd: 2402.1ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 1315 | loss: 1.519559 | lr 1.8000e-04 | norm: 0.9685 | dt: 7863.28ms | tok/sec: 66675.5 | mem: 72099MB | Time Left: 00 hrs : 44 mins
[TIMING] data: 15.5ms | fwd: 2451.0ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 1316 | loss: 1.623467 | lr 1.8000e-04 | norm: 0.9396 | dt: 7808.88ms | tok/sec: 67140.0 | mem: 72099MB | Time Left: 00 hrs : 43 mins
[TIMING] data: 14.2ms | fwd: 2402.0ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 1317 | loss: 1.509758 | lr 1.8000e-04 | norm: 0.9391 | dt: 7820.09ms | tok/sec: 67043.7 | mem: 72099MB | Time Left: 00 hrs : 43 mins
[TIMING] data: 12.8ms | fwd: 2413.0ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 1318 | loss: 1.378390 | lr 1.8000e-04 | norm: 0.9667 | dt: 7807.08ms | tok/sec: 67155.5 | mem: 72099MB | Time Left: 00 hrs : 43 mins
[TIMING] data: 14.7ms | fwd: 2399.5ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.6ms
step 1319 | loss: 1.647575 | lr 1.8000e-04 | norm: 1.0793 | dt: 7812.27ms | tok/sec: 67110.9 | mem: 72099MB | Time Left: 00 hrs : 43 mins
[TIMING] data: 14.2ms | fwd: 2401.4ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.6ms
step 1320 | loss: 1.429387 | lr 1.8000e-04 | norm: 1.1213 | dt: 7808.29ms | tok/sec: 67145.0 | mem: 72099MB | Time Left: 00 hrs : 43 mins
[TIMING] data: 15.5ms | fwd: 2397.1ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1321 | loss: 1.529141 | lr 1.8000e-04 | norm: 1.0062 | dt: 7807.48ms | tok/sec: 67152.0 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 15.2ms | fwd: 2400.4ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.5ms
step 1322 | loss: 1.528067 | lr 1.8000e-04 | norm: 1.0525 | dt: 7816.40ms | tok/sec: 67075.4 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 13.7ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5356.8ms | clip: 0.0ms | optim: 23.5ms
step 1323 | loss: 1.435905 | lr 1.8000e-04 | norm: 1.0151 | dt: 7812.82ms | tok/sec: 67106.1 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 18.4ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.6ms
step 1324 | loss: 1.480324 | lr 1.8000e-04 | norm: 0.9810 | dt: 7877.26ms | tok/sec: 66557.2 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 15.0ms | fwd: 2466.8ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.6ms
step 1325 | loss: 1.478367 | lr 1.8000e-04 | norm: 1.0225 | dt: 7810.86ms | tok/sec: 67122.9 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 15.2ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5346.1ms | clip: 0.0ms | optim: 23.6ms
step 1326 | loss: 1.678355 | lr 1.8000e-04 | norm: 1.1179 | dt: 7813.53ms | tok/sec: 67100.0 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 17.4ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.5ms
step 1327 | loss: 1.551474 | lr 1.8000e-04 | norm: 1.0409 | dt: 7812.40ms | tok/sec: 67109.8 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 15.6ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.5ms
step 1328 | loss: 1.305952 | lr 1.8000e-04 | norm: 0.9807 | dt: 7831.73ms | tok/sec: 66944.1 | mem: 72099MB | Time Left: 00 hrs : 42 mins
[TIMING] data: 15.3ms | fwd: 2420.7ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.7ms
step 1329 | loss: 1.689403 | lr 1.8000e-04 | norm: 1.1966 | dt: 7816.89ms | tok/sec: 67071.2 | mem: 72099MB | Time Left: 00 hrs : 41 mins
[TIMING] data: 17.6ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5351.2ms | clip: 0.0ms | optim: 24.4ms
step 1330 | loss: 1.512617 | lr 1.8000e-04 | norm: 1.1276 | dt: 7809.97ms | tok/sec: 67130.6 | mem: 72099MB | Time Left: 00 hrs : 41 mins
[TIMING] data: 13.9ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 1331 | loss: 1.583404 | lr 1.8000e-04 | norm: 1.1167 | dt: 7807.02ms | tok/sec: 67156.0 | mem: 72099MB | Time Left: 00 hrs : 41 mins
[TIMING] data: 16.3ms | fwd: 2399.2ms | loss: 4.9ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.5ms
step 1332 | loss: 1.598999 | lr 1.8000e-04 | norm: 1.1184 | dt: 7814.60ms | tok/sec: 67090.9 | mem: 72099MB | Time Left: 00 hrs : 41 mins
[TIMING] data: 15.7ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.6ms
step 1333 | loss: 1.421445 | lr 1.8000e-04 | norm: 0.9539 | dt: 7826.27ms | tok/sec: 66990.8 | mem: 72099MB | Time Left: 00 hrs : 41 mins
[TIMING] data: 14.0ms | fwd: 2418.1ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.5ms
step 1334 | loss: 1.581371 | lr 1.8000e-04 | norm: 1.0929 | dt: 7817.88ms | tok/sec: 67062.7 | mem: 72099MB | Time Left: 00 hrs : 41 mins
[TIMING] data: 14.9ms | fwd: 2405.8ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 1335 | loss: 1.648687 | lr 1.8000e-04 | norm: 1.3396 | dt: 7821.52ms | tok/sec: 67031.5 | mem: 72099MB | Time Left: 00 hrs : 41 mins
[TIMING] data: 18.5ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.5ms
step 1336 | loss: 1.498142 | lr 1.8000e-04 | norm: 1.2234 | dt: 7803.36ms | tok/sec: 67187.4 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 13.7ms | fwd: 2398.6ms | loss: 4.9ms | bwd: 5350.0ms | clip: 0.0ms | optim: 23.5ms
step 1337 | loss: 1.687821 | lr 1.8000e-04 | norm: 1.1673 | dt: 7815.30ms | tok/sec: 67084.8 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 14.7ms | fwd: 2410.2ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 1338 | loss: 1.305620 | lr 1.8000e-04 | norm: 0.9714 | dt: 7814.22ms | tok/sec: 67094.1 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 12.4ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5357.9ms | clip: 0.0ms | optim: 23.6ms
step 1339 | loss: 1.607921 | lr 1.8000e-04 | norm: 1.1829 | dt: 7827.71ms | tok/sec: 66978.4 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 14.8ms | fwd: 2412.8ms | loss: 4.9ms | bwd: 5359.4ms | clip: 0.0ms | optim: 23.6ms
step 1340 | loss: 1.386371 | lr 1.8000e-04 | norm: 1.0002 | dt: 7814.79ms | tok/sec: 67089.2 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 15.9ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.5ms
step 1341 | loss: 1.354437 | lr 1.8000e-04 | norm: 0.9935 | dt: 7808.40ms | tok/sec: 67144.1 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 12.9ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5347.9ms | clip: 0.0ms | optim: 23.7ms
step 1342 | loss: 1.343263 | lr 1.8000e-04 | norm: 1.0361 | dt: 7811.92ms | tok/sec: 67113.9 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 16.9ms | fwd: 2405.1ms | loss: 4.9ms | bwd: 5349.1ms | clip: 0.0ms | optim: 23.5ms
step 1343 | loss: 1.347670 | lr 1.8000e-04 | norm: 0.9625 | dt: 7818.59ms | tok/sec: 67056.6 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 14.8ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5358.5ms | clip: 0.0ms | optim: 23.6ms
step 1344 | loss: 1.489050 | lr 1.8000e-04 | norm: 1.0761 | dt: 7819.53ms | tok/sec: 67048.5 | mem: 72099MB | Time Left: 00 hrs : 40 mins
[TIMING] data: 13.9ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.4ms
step 1345 | loss: 1.381999 | lr 1.8000e-04 | norm: 1.1341 | dt: 7813.28ms | tok/sec: 67102.2 | mem: 72099MB | Time Left: 00 hrs : 39 mins
[TIMING] data: 15.3ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.5ms
step 1346 | loss: 1.605769 | lr 1.8000e-04 | norm: 1.1972 | dt: 7816.77ms | tok/sec: 67072.2 | mem: 72099MB | Time Left: 00 hrs : 39 mins
[TIMING] data: 14.8ms | fwd: 2408.5ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1347 | loss: 1.646022 | lr 1.8000e-04 | norm: 1.2383 | dt: 7816.64ms | tok/sec: 67073.3 | mem: 72099MB | Time Left: 00 hrs : 39 mins
[TIMING] data: 16.3ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.7ms
step 1348 | loss: 1.380131 | lr 1.8000e-04 | norm: 1.1117 | dt: 7820.62ms | tok/sec: 67039.1 | mem: 72099MB | Time Left: 00 hrs : 39 mins
[TIMING] data: 13.0ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.6ms
step 1349 | loss: 1.375865 | lr 1.8000e-04 | norm: 1.0544 | dt: 7812.69ms | tok/sec: 67107.2 | mem: 72099MB | Time Left: 00 hrs : 39 mins
[TIMING] data: 13.8ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.7ms
step 1350 | loss: 1.458602 | lr 1.8000e-04 | norm: 1.2067 | dt: 7807.47ms | tok/sec: 67152.1 | mem: 72099MB | Time Left: 00 hrs : 39 mins
[TIMING] data: 15.5ms | fwd: 2404.4ms | loss: 4.9ms | bwd: 5347.8ms | clip: 0.0ms | optim: 23.5ms
step 1351 | loss: 1.280829 | lr 1.8000e-04 | norm: 0.9646 | dt: 7804.65ms | tok/sec: 67176.4 | mem: 72099MB | Time Left: 00 hrs : 39 mins
[TIMING] data: 15.5ms | fwd: 2396.5ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.6ms
step 1352 | loss: 1.445033 | lr 1.8000e-04 | norm: 1.0192 | dt: 7817.65ms | tok/sec: 67064.7 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 15.9ms | fwd: 2410.1ms | loss: 4.9ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.6ms
step 1353 | loss: 1.403465 | lr 1.8000e-04 | norm: 1.0741 | dt: 7814.27ms | tok/sec: 67093.7 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 13.1ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 1354 | loss: 1.495527 | lr 1.8000e-04 | norm: 1.1144 | dt: 7815.77ms | tok/sec: 67080.8 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 17.4ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.6ms
step 1355 | loss: 1.452060 | lr 1.8000e-04 | norm: 1.1131 | dt: 7815.14ms | tok/sec: 67086.2 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 16.2ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.6ms
step 1356 | loss: 1.407404 | lr 1.8000e-04 | norm: 1.1224 | dt: 7811.18ms | tok/sec: 67120.2 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 14.1ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 1357 | loss: 1.396448 | lr 1.8000e-04 | norm: 1.0826 | dt: 7815.74ms | tok/sec: 67081.0 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 14.0ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 1358 | loss: 1.408879 | lr 1.8000e-04 | norm: 1.1333 | dt: 7807.24ms | tok/sec: 67154.1 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 14.5ms | fwd: 2399.1ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 1359 | loss: 1.427751 | lr 1.8000e-04 | norm: 1.1611 | dt: 7818.35ms | tok/sec: 67058.7 | mem: 72099MB | Time Left: 00 hrs : 38 mins
[TIMING] data: 14.9ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 1360 | loss: 1.303495 | lr 1.8000e-04 | norm: 1.1860 | dt: 7821.91ms | tok/sec: 67028.1 | mem: 72099MB | Time Left: 00 hrs : 37 mins
[TIMING] data: 13.5ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 1361 | loss: 1.317616 | lr 1.8000e-04 | norm: 0.9978 | dt: 7814.15ms | tok/sec: 67094.7 | mem: 72099MB | Time Left: 00 hrs : 37 mins
[TIMING] data: 14.6ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 23.6ms
step 1362 | loss: 1.448542 | lr 1.8000e-04 | norm: 1.1180 | dt: 7811.77ms | tok/sec: 67115.1 | mem: 72099MB | Time Left: 00 hrs : 37 mins
[TIMING] data: 12.3ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.5ms
step 1363 | loss: 1.387951 | lr 1.8000e-04 | norm: 1.2038 | dt: 7816.20ms | tok/sec: 67077.1 | mem: 72099MB | Time Left: 00 hrs : 37 mins
[TIMING] data: 14.7ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.5ms
step 1364 | loss: 1.475123 | lr 1.8000e-04 | norm: 1.1529 | dt: 7814.47ms | tok/sec: 67092.0 | mem: 72099MB | Time Left: 00 hrs : 37 mins
[TIMING] data: 15.8ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.7ms
step 1365 | loss: 1.406439 | lr 1.8000e-04 | norm: 1.1795 | dt: 7823.78ms | tok/sec: 67012.1 | mem: 72099MB | Time Left: 00 hrs : 37 mins
[TIMING] data: 16.8ms | fwd: 2411.7ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 1366 | loss: 1.441553 | lr 1.8000e-04 | norm: 1.1971 | dt: 7810.57ms | tok/sec: 67125.4 | mem: 72099MB | Time Left: 00 hrs : 37 mins
[TIMING] data: 13.5ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.7ms
step 1367 | loss: 1.499326 | lr 1.8000e-04 | norm: 1.1260 | dt: 7813.48ms | tok/sec: 67100.4 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 15.3ms | fwd: 2401.4ms | loss: 4.9ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.7ms
step 1368 | loss: 1.372954 | lr 1.8000e-04 | norm: 1.1674 | dt: 7822.74ms | tok/sec: 67021.0 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 13.8ms | fwd: 2414.5ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.5ms
step 1369 | loss: 1.347214 | lr 1.8000e-04 | norm: 1.1197 | dt: 7821.43ms | tok/sec: 67032.3 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 13.8ms | fwd: 2412.5ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 1370 | loss: 1.262198 | lr 1.8000e-04 | norm: 1.1032 | dt: 7873.95ms | tok/sec: 66585.2 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 13.5ms | fwd: 2469.5ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.5ms
step 1371 | loss: 1.356118 | lr 1.8000e-04 | norm: 1.1040 | dt: 7819.27ms | tok/sec: 67050.7 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 16.9ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.4ms
step 1372 | loss: 1.392194 | lr 1.8000e-04 | norm: 1.0296 | dt: 7822.06ms | tok/sec: 67026.8 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 13.2ms | fwd: 2414.5ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.6ms
step 1373 | loss: 1.332198 | lr 1.8000e-04 | norm: 1.1571 | dt: 7810.59ms | tok/sec: 67125.3 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 14.4ms | fwd: 2400.5ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 1374 | loss: 1.241001 | lr 1.8000e-04 | norm: 1.0225 | dt: 7812.56ms | tok/sec: 67108.3 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 15.8ms | fwd: 2401.5ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.7ms
step 1375 | loss: 1.403464 | lr 1.8000e-04 | norm: 1.0922 | dt: 7812.87ms | tok/sec: 67105.7 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 13.3ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1376 | loss: 1.407885 | lr 1.8000e-04 | norm: 1.1765 | dt: 7881.84ms | tok/sec: 66518.5 | mem: 72099MB | Time Left: 00 hrs : 36 mins
[TIMING] data: 13.4ms | fwd: 2477.6ms | loss: 4.9ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.4ms
step 1377 | loss: 1.301699 | lr 1.8000e-04 | norm: 1.1748 | dt: 7823.04ms | tok/sec: 67018.5 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 17.3ms | fwd: 2410.0ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.6ms
step 1378 | loss: 1.407647 | lr 1.8000e-04 | norm: 1.0588 | dt: 7868.22ms | tok/sec: 66633.6 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 12.9ms | fwd: 2460.5ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
step 1379 | loss: 1.459819 | lr 1.8000e-04 | norm: 1.1384 | dt: 7871.59ms | tok/sec: 66605.1 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 13.4ms | fwd: 2461.2ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.7ms
step 1380 | loss: 1.242826 | lr 1.8000e-04 | norm: 1.2515 | dt: 7819.28ms | tok/sec: 67050.7 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 14.3ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.6ms
step 1381 | loss: 1.226236 | lr 1.8000e-04 | norm: 1.1073 | dt: 7823.99ms | tok/sec: 67010.3 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 14.3ms | fwd: 2409.8ms | loss: 4.9ms | bwd: 5360.0ms | clip: 0.0ms | optim: 23.5ms
step 1382 | loss: 1.323732 | lr 1.8000e-04 | norm: 1.3920 | dt: 7849.19ms | tok/sec: 66795.2 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 14.6ms | fwd: 2442.1ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.6ms
step 1383 | loss: 1.072862 | lr 1.8000e-04 | norm: 0.9402 | dt: 7812.59ms | tok/sec: 67108.1 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 14.9ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.6ms
step 1384 | loss: 1.418396 | lr 1.8000e-04 | norm: 1.2656 | dt: 7825.58ms | tok/sec: 66996.7 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 16.1ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.5ms
step 1385 | loss: 1.410168 | lr 1.8000e-04 | norm: 1.1778 | dt: 7815.67ms | tok/sec: 67081.6 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 12.1ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.5ms
step 1386 | loss: 1.568725 | lr 1.8000e-04 | norm: 1.2384 | dt: 7810.55ms | tok/sec: 67125.6 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 15.7ms | fwd: 2400.8ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 1387 | loss: 1.301454 | lr 1.8000e-04 | norm: 1.1120 | dt: 7811.64ms | tok/sec: 67116.3 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 15.2ms | fwd: 2401.5ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.6ms
step 1388 | loss: 1.337742 | lr 1.8000e-04 | norm: 1.3304 | dt: 7816.30ms | tok/sec: 67076.2 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 17.7ms | fwd: 2409.9ms | loss: 4.9ms | bwd: 5348.2ms | clip: 0.0ms | optim: 23.7ms
step 1389 | loss: 1.247089 | lr 1.8000e-04 | norm: 0.9526 | dt: 7815.37ms | tok/sec: 67084.2 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 12.7ms | fwd: 2406.0ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.7ms
step 1390 | loss: 1.572382 | lr 1.8000e-04 | norm: 1.4013 | dt: 7813.81ms | tok/sec: 67097.6 | mem: 72099MB | Time Left: 00 hrs : 33 mins
[TIMING] data: 14.0ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.6ms
step 1391 | loss: 1.380150 | lr 1.8000e-04 | norm: 1.2991 | dt: 7805.55ms | tok/sec: 67168.6 | mem: 72099MB | Time Left: 00 hrs : 33 mins
[TIMING] data: 14.7ms | fwd: 2399.0ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.5ms
step 1392 | loss: 1.277830 | lr 1.8000e-04 | norm: 1.1840 | dt: 7881.41ms | tok/sec: 66522.1 | mem: 72099MB | Time Left: 00 hrs : 34 mins
[TIMING] data: 14.9ms | fwd: 2471.7ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 1393 | loss: 1.448891 | lr 1.8000e-04 | norm: 1.2375 | dt: 7819.03ms | tok/sec: 67052.8 | mem: 72099MB | Time Left: 00 hrs : 33 mins
[TIMING] data: 11.6ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.5ms
step 1394 | loss: 1.353256 | lr 1.8000e-04 | norm: 1.3081 | dt: 7821.01ms | tok/sec: 67035.9 | mem: 72099MB | Time Left: 00 hrs : 33 mins
[TIMING] data: 12.3ms | fwd: 2414.2ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 1395 | loss: 1.332615 | lr 1.8000e-04 | norm: 1.2099 | dt: 7816.41ms | tok/sec: 67075.3 | mem: 72099MB | Time Left: 00 hrs : 33 mins
[TIMING] data: 14.5ms | fwd: 2409.2ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.6ms
step 1396 | loss: 1.390188 | lr 1.8000e-04 | norm: 1.1731 | dt: 7826.69ms | tok/sec: 66987.2 | mem: 72099MB | Time Left: 00 hrs : 33 mins
[TIMING] data: 14.6ms | fwd: 2418.5ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 1397 | loss: 1.330562 | lr 1.8000e-04 | norm: 1.1472 | dt: 7879.89ms | tok/sec: 66534.9 | mem: 72099MB | Time Left: 00 hrs : 33 mins
[TIMING] data: 12.4ms | fwd: 2474.3ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 1398 | loss: 1.437953 | lr 1.8000e-04 | norm: 1.2846 | dt: 7811.88ms | tok/sec: 67114.2 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 14.3ms | fwd: 2401.6ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.7ms
step 1399 | loss: 1.407458 | lr 1.8000e-04 | norm: 1.2060 | dt: 7823.17ms | tok/sec: 67017.4 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 14.4ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5360.0ms | clip: 0.0ms | optim: 23.7ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1400.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1200.ckpt
step 1400 | loss: 1.352186 | lr 1.8000e-04 | norm: 1.0936 | dt: 7788.60ms | tok/sec: 67314.8 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 11.6ms | fwd: 2393.9ms | loss: 4.9ms | bwd: 5341.5ms | clip: 0.0ms | optim: 23.5ms
step 1401 | loss: 1.335609 | lr 1.8000e-04 | norm: 1.2217 | dt: 7794.72ms | tok/sec: 67261.9 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 12.5ms | fwd: 2396.8ms | loss: 4.9ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.4ms
step 1402 | loss: 1.063608 | lr 1.8000e-04 | norm: 1.1959 | dt: 7797.89ms | tok/sec: 67234.6 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 13.4ms | fwd: 2399.6ms | loss: 4.9ms | bwd: 5345.3ms | clip: 0.0ms | optim: 23.5ms
step 1403 | loss: 1.449983 | lr 1.8000e-04 | norm: 1.2784 | dt: 7823.15ms | tok/sec: 67017.5 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 46.3ms | fwd: 2393.0ms | loss: 4.9ms | bwd: 5343.1ms | clip: 0.0ms | optim: 23.5ms
step 1404 | loss: 1.251183 | lr 1.8000e-04 | norm: 1.2136 | dt: 7832.53ms | tok/sec: 66937.2 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 41.9ms | fwd: 2400.9ms | loss: 4.9ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.5ms
step 1405 | loss: 1.297120 | lr 1.8000e-04 | norm: 1.1710 | dt: 7809.14ms | tok/sec: 67137.7 | mem: 72099MB | Time Left: 00 hrs : 32 mins
[TIMING] data: 24.1ms | fwd: 2400.2ms | loss: 4.9ms | bwd: 5343.8ms | clip: 0.0ms | optim: 23.5ms
step 1406 | loss: 1.311900 | lr 1.8000e-04 | norm: 1.0553 | dt: 7815.81ms | tok/sec: 67080.5 | mem: 72099MB | Time Left: 00 hrs : 31 mins
[TIMING] data: 14.9ms | fwd: 2402.8ms | loss: 4.9ms | bwd: 5357.3ms | clip: 0.0ms | optim: 23.5ms
step 1407 | loss: 1.351509 | lr 1.8000e-04 | norm: 1.1932 | dt: 7815.94ms | tok/sec: 67079.4 | mem: 72099MB | Time Left: 00 hrs : 31 mins
[TIMING] data: 12.6ms | fwd: 2410.3ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.5ms
step 1408 | loss: 1.180624 | lr 1.8000e-04 | norm: 1.0419 | dt: 7815.32ms | tok/sec: 67084.6 | mem: 72099MB | Time Left: 00 hrs : 31 mins
[TIMING] data: 14.8ms | fwd: 2401.8ms | loss: 4.9ms | bwd: 5357.8ms | clip: 0.0ms | optim: 23.5ms
step 1409 | loss: 1.322965 | lr 1.8000e-04 | norm: 1.1878 | dt: 7881.61ms | tok/sec: 66520.4 | mem: 72099MB | Time Left: 00 hrs : 31 mins
[TIMING] data: 13.0ms | fwd: 2477.6ms | loss: 4.9ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.7ms
step 1410 | loss: 1.338719 | lr 1.8000e-04 | norm: 1.1452 | dt: 7811.09ms | tok/sec: 67121.0 | mem: 72099MB | Time Left: 00 hrs : 31 mins
[TIMING] data: 14.8ms | fwd: 2402.3ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.5ms
step 1411 | loss: 1.309943 | lr 1.8000e-04 | norm: 1.1843 | dt: 7816.38ms | tok/sec: 67075.6 | mem: 72099MB | Time Left: 00 hrs : 31 mins
[TIMING] data: 14.6ms | fwd: 2407.7ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.7ms
step 1412 | loss: 1.203938 | lr 1.8000e-04 | norm: 1.1135 | dt: 7807.04ms | tok/sec: 67155.8 | mem: 72099MB | Time Left: 00 hrs : 31 mins
[TIMING] data: 13.4ms | fwd: 2404.4ms | loss: 4.9ms | bwd: 5349.6ms | clip: 0.0ms | optim: 23.6ms
step 1413 | loss: 1.298626 | lr 1.8000e-04 | norm: 1.1497 | dt: 7812.01ms | tok/sec: 67113.1 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 13.3ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.6ms
step 1414 | loss: 1.346417 | lr 1.8000e-04 | norm: 1.1072 | dt: 7813.22ms | tok/sec: 67102.7 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 14.7ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.6ms
step 1415 | loss: 1.221861 | lr 1.8000e-04 | norm: 1.2847 | dt: 7802.97ms | tok/sec: 67190.8 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 14.5ms | fwd: 2397.6ms | loss: 4.9ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.5ms
step 1416 | loss: 1.372840 | lr 1.8000e-04 | norm: 1.1970 | dt: 7809.36ms | tok/sec: 67135.9 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 15.6ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5345.9ms | clip: 0.0ms | optim: 23.7ms
step 1417 | loss: 1.294260 | lr 1.8000e-04 | norm: 1.0983 | dt: 7813.31ms | tok/sec: 67101.9 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 15.8ms | fwd: 2406.9ms | loss: 4.9ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.5ms
step 1418 | loss: 1.276669 | lr 1.8000e-04 | norm: 1.0743 | dt: 7812.58ms | tok/sec: 67108.2 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 14.7ms | fwd: 2402.5ms | loss: 4.9ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.5ms
step 1419 | loss: 1.268096 | lr 1.8000e-04 | norm: 1.0278 | dt: 7876.07ms | tok/sec: 66567.2 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 14.0ms | fwd: 2472.9ms | loss: 4.9ms | bwd: 5348.7ms | clip: 0.0ms | optim: 23.6ms
step 1420 | loss: 1.440384 | lr 1.8000e-04 | norm: 1.0838 | dt: 7813.00ms | tok/sec: 67104.5 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 15.3ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 1421 | loss: 1.197074 | lr 1.8000e-04 | norm: 1.0212 | dt: 7877.76ms | tok/sec: 66553.0 | mem: 72099MB | Time Left: 00 hrs : 30 mins
[TIMING] data: 21.3ms | fwd: 2466.8ms | loss: 4.9ms | bwd: 5349.0ms | clip: 0.0ms | optim: 23.6ms
step 1422 | loss: 1.212184 | lr 1.8000e-04 | norm: 1.1539 | dt: 7849.27ms | tok/sec: 66794.5 | mem: 72099MB | Time Left: 00 hrs : 29 mins
[TIMING] data: 15.5ms | fwd: 2444.6ms | loss: 4.9ms | bwd: 5348.3ms | clip: 0.0ms | optim: 23.5ms
step 1423 | loss: 1.294138 | lr 1.8000e-04 | norm: 1.4436 | dt: 7817.35ms | tok/sec: 67067.3 | mem: 72099MB | Time Left: 00 hrs : 29 mins
[TIMING] data: 15.5ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.6ms
step 1424 | loss: 1.194331 | lr 1.8000e-04 | norm: 1.1101 | dt: 7820.43ms | tok/sec: 67040.8 | mem: 72099MB | Time Left: 00 hrs : 29 mins
[TIMING] data: 14.0ms | fwd: 2409.4ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 1425 | loss: 1.290797 | lr 1.8000e-04 | norm: 1.1759 | dt: 7817.75ms | tok/sec: 67063.8 | mem: 72099MB | Time Left: 00 hrs : 29 mins
[TIMING] data: 16.0ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 1426 | loss: 1.353280 | lr 1.8000e-04 | norm: 1.2367 | dt: 7808.85ms | tok/sec: 67140.2 | mem: 72099MB | Time Left: 00 hrs : 29 mins
[TIMING] data: 14.5ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5343.5ms | clip: 0.0ms | optim: 23.6ms
step 1427 | loss: 1.200993 | lr 1.8000e-04 | norm: 1.0258 | dt: 7817.71ms | tok/sec: 67064.1 | mem: 72099MB | Time Left: 00 hrs : 29 mins
[TIMING] data: 16.0ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 1428 | loss: 1.305550 | lr 1.8000e-04 | norm: 1.1604 | dt: 7815.53ms | tok/sec: 67082.9 | mem: 72099MB | Time Left: 00 hrs : 29 mins
[TIMING] data: 15.9ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.6ms
step 1429 | loss: 1.358730 | lr 1.8000e-04 | norm: 1.4110 | dt: 7813.84ms | tok/sec: 67097.3 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 17.6ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5352.2ms | clip: 0.0ms | optim: 23.6ms
step 1430 | loss: 1.452089 | lr 1.8000e-04 | norm: 1.1010 | dt: 7819.23ms | tok/sec: 67051.1 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 15.3ms | fwd: 2411.0ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.6ms
step 1431 | loss: 1.372197 | lr 1.8000e-04 | norm: 1.2160 | dt: 7805.55ms | tok/sec: 67168.6 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 15.2ms | fwd: 2401.8ms | loss: 4.9ms | bwd: 5348.4ms | clip: 0.0ms | optim: 23.5ms
step 1432 | loss: 1.380843 | lr 1.8000e-04 | norm: 1.1706 | dt: 7815.29ms | tok/sec: 67084.9 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 12.1ms | fwd: 2404.4ms | loss: 4.9ms | bwd: 5358.8ms | clip: 0.0ms | optim: 23.5ms
step 1433 | loss: 1.217088 | lr 1.8000e-04 | norm: 1.0741 | dt: 7808.16ms | tok/sec: 67146.1 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 14.8ms | fwd: 2399.7ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 1434 | loss: 1.088354 | lr 1.8000e-04 | norm: 1.0129 | dt: 7812.69ms | tok/sec: 67107.3 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 14.9ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5352.6ms | clip: 0.0ms | optim: 24.1ms
step 1435 | loss: 1.248707 | lr 1.8000e-04 | norm: 1.1565 | dt: 7875.70ms | tok/sec: 66570.3 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 14.4ms | fwd: 2470.9ms | loss: 4.9ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.5ms
step 1436 | loss: 1.380906 | lr 1.8000e-04 | norm: 1.0538 | dt: 7814.66ms | tok/sec: 67090.3 | mem: 72099MB | Time Left: 00 hrs : 28 mins
[TIMING] data: 16.6ms | fwd: 2401.3ms | loss: 4.9ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.6ms
step 1437 | loss: 1.515351 | lr 1.8000e-04 | norm: 1.4012 | dt: 7818.70ms | tok/sec: 67055.6 | mem: 72099MB | Time Left: 00 hrs : 27 mins
[TIMING] data: 13.1ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 1438 | loss: 1.390624 | lr 1.8000e-04 | norm: 1.2461 | dt: 7812.35ms | tok/sec: 67110.2 | mem: 72099MB | Time Left: 00 hrs : 27 mins
[TIMING] data: 12.4ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.7ms
step 1439 | loss: 1.286902 | lr 1.8000e-04 | norm: 1.1649 | dt: 7810.82ms | tok/sec: 67123.3 | mem: 72099MB | Time Left: 00 hrs : 27 mins
[TIMING] data: 13.6ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.6ms
step 1440 | loss: 1.210372 | lr 1.8000e-04 | norm: 1.1398 | dt: 7808.42ms | tok/sec: 67143.9 | mem: 72099MB | Time Left: 00 hrs : 27 mins
[TIMING] data: 11.8ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.6ms
step 1441 | loss: 1.100217 | lr 1.8000e-04 | norm: 1.0288 | dt: 7822.53ms | tok/sec: 67022.8 | mem: 72099MB | Time Left: 00 hrs : 27 mins
[TIMING] data: 16.4ms | fwd: 2408.5ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.5ms
step 1442 | loss: 1.359551 | lr 1.8000e-04 | norm: 1.0606 | dt: 7813.82ms | tok/sec: 67097.5 | mem: 72099MB | Time Left: 00 hrs : 27 mins
[TIMING] data: 17.5ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5347.6ms | clip: 0.0ms | optim: 23.5ms
step 1443 | loss: 1.136957 | lr 1.8000e-04 | norm: 1.0141 | dt: 7816.71ms | tok/sec: 67072.7 | mem: 72099MB | Time Left: 00 hrs : 27 mins
[TIMING] data: 16.3ms | fwd: 2402.9ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.6ms
step 1444 | loss: 1.171940 | lr 1.8000e-04 | norm: 0.9621 | dt: 7824.22ms | tok/sec: 67008.4 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 30.9ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.6ms
step 1445 | loss: 1.201710 | lr 1.8000e-04 | norm: 1.0672 | dt: 7809.88ms | tok/sec: 67131.4 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 13.1ms | fwd: 2404.9ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.5ms
step 1446 | loss: 1.192295 | lr 1.8000e-04 | norm: 1.0544 | dt: 7806.32ms | tok/sec: 67162.0 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 14.2ms | fwd: 2398.4ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.6ms
step 1447 | loss: 1.275432 | lr 1.8000e-04 | norm: 1.2499 | dt: 7810.12ms | tok/sec: 67129.4 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 13.4ms | fwd: 2400.8ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 1448 | loss: 1.353067 | lr 1.8000e-04 | norm: 1.5020 | dt: 7807.12ms | tok/sec: 67155.1 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 14.4ms | fwd: 2402.6ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 1449 | loss: 1.258310 | lr 1.8000e-04 | norm: 1.1863 | dt: 7858.62ms | tok/sec: 66715.0 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 14.1ms | fwd: 2451.7ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.6ms
step 1450 | loss: 1.359985 | lr 1.8000e-04 | norm: 1.3269 | dt: 7817.39ms | tok/sec: 67066.9 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 15.2ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.5ms
step 1451 | loss: 1.204250 | lr 1.8000e-04 | norm: 0.9855 | dt: 7811.78ms | tok/sec: 67115.0 | mem: 72099MB | Time Left: 00 hrs : 26 mins
[TIMING] data: 14.4ms | fwd: 2406.8ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 1452 | loss: 1.397237 | lr 1.8000e-04 | norm: 1.1943 | dt: 7822.26ms | tok/sec: 67025.1 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 15.4ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.5ms
step 1453 | loss: 1.287730 | lr 1.8000e-04 | norm: 1.0367 | dt: 7808.94ms | tok/sec: 67139.5 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 16.0ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5347.6ms | clip: 0.0ms | optim: 23.7ms
step 1454 | loss: 1.275872 | lr 1.8000e-04 | norm: 1.3870 | dt: 7817.45ms | tok/sec: 67066.4 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 14.9ms | fwd: 2408.9ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.7ms
step 1455 | loss: 1.121111 | lr 1.8000e-04 | norm: 0.9762 | dt: 7800.73ms | tok/sec: 67210.1 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 15.2ms | fwd: 2398.8ms | loss: 4.9ms | bwd: 5347.2ms | clip: 0.0ms | optim: 23.6ms
step 1456 | loss: 1.293050 | lr 1.8000e-04 | norm: 1.1363 | dt: 7813.97ms | tok/sec: 67096.2 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 13.2ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.6ms
step 1457 | loss: 1.298708 | lr 1.8000e-04 | norm: 1.2017 | dt: 7828.12ms | tok/sec: 66974.9 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 15.2ms | fwd: 2416.9ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.7ms
step 1458 | loss: 1.189975 | lr 1.8000e-04 | norm: 1.0782 | dt: 7810.33ms | tok/sec: 67127.5 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 15.2ms | fwd: 2402.8ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.5ms
step 1459 | loss: 1.258695 | lr 1.8000e-04 | norm: 0.9227 | dt: 7827.19ms | tok/sec: 66982.9 | mem: 72099MB | Time Left: 00 hrs : 25 mins
[TIMING] data: 18.2ms | fwd: 2413.0ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 1460 | loss: 1.307868 | lr 1.8000e-04 | norm: 0.9561 | dt: 7809.24ms | tok/sec: 67136.9 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 14.1ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5348.4ms | clip: 0.0ms | optim: 23.5ms
step 1461 | loss: 1.234876 | lr 1.8000e-04 | norm: 1.2109 | dt: 7818.37ms | tok/sec: 67058.4 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 15.2ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.4ms
step 1462 | loss: 1.042611 | lr 1.8000e-04 | norm: 0.8914 | dt: 7812.28ms | tok/sec: 67110.7 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 15.3ms | fwd: 2406.3ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 1463 | loss: 1.227962 | lr 1.8000e-04 | norm: 1.1497 | dt: 7812.51ms | tok/sec: 67108.7 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 15.0ms | fwd: 2401.6ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.5ms
step 1464 | loss: 1.384212 | lr 1.8000e-04 | norm: 1.5435 | dt: 7807.76ms | tok/sec: 67149.6 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 15.0ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5348.3ms | clip: 0.0ms | optim: 23.6ms
step 1465 | loss: 1.246336 | lr 1.8000e-04 | norm: 1.1458 | dt: 7816.10ms | tok/sec: 67078.0 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 17.5ms | fwd: 2405.9ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.5ms
step 1466 | loss: 1.435249 | lr 1.8000e-04 | norm: 1.3692 | dt: 7815.71ms | tok/sec: 67081.3 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 14.8ms | fwd: 2409.3ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.5ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00008.bin
100000000
step 1467 | loss: 1.369467 | lr 1.8000e-04 | norm: 1.0640 | dt: 7862.35ms | tok/sec: 66683.4 | mem: 72099MB | Time Left: 00 hrs : 24 mins
[TIMING] data: 69.0ms | fwd: 2401.7ms | loss: 4.9ms | bwd: 5351.9ms | clip: 0.0ms | optim: 23.6ms
step 1468 | loss: 1.236415 | lr 1.8000e-04 | norm: 1.2238 | dt: 7807.98ms | tok/sec: 67147.7 | mem: 72099MB | Time Left: 00 hrs : 23 mins
[TIMING] data: 14.2ms | fwd: 2397.6ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 24.2ms
step 1469 | loss: 1.176036 | lr 1.8000e-04 | norm: 1.1495 | dt: 7886.00ms | tok/sec: 66483.4 | mem: 72099MB | Time Left: 00 hrs : 23 mins
[TIMING] data: 14.1ms | fwd: 2479.5ms | loss: 4.9ms | bwd: 5351.2ms | clip: 0.0ms | optim: 23.6ms
step 1470 | loss: 1.091662 | lr 1.8000e-04 | norm: 1.1475 | dt: 7813.30ms | tok/sec: 67102.0 | mem: 72099MB | Time Left: 00 hrs : 23 mins
[TIMING] data: 13.2ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.6ms
step 1471 | loss: 1.345521 | lr 1.8000e-04 | norm: 1.1650 | dt: 7822.49ms | tok/sec: 67023.2 | mem: 72099MB | Time Left: 00 hrs : 23 mins
[TIMING] data: 18.2ms | fwd: 2411.2ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.6ms
step 1472 | loss: 1.129535 | lr 1.8000e-04 | norm: 0.9684 | dt: 7809.86ms | tok/sec: 67131.6 | mem: 72099MB | Time Left: 00 hrs : 23 mins
[TIMING] data: 13.5ms | fwd: 2401.7ms | loss: 4.9ms | bwd: 5355.0ms | clip: 0.0ms | optim: 23.6ms
step 1473 | loss: 1.320691 | lr 1.8000e-04 | norm: 1.3052 | dt: 7818.58ms | tok/sec: 67056.6 | mem: 72099MB | Time Left: 00 hrs : 23 mins
[TIMING] data: 15.7ms | fwd: 2413.9ms | loss: 4.9ms | bwd: 5348.4ms | clip: 0.0ms | optim: 23.6ms
step 1474 | loss: 1.368436 | lr 1.8000e-04 | norm: 1.0872 | dt: 7810.99ms | tok/sec: 67121.9 | mem: 72099MB | Time Left: 00 hrs : 23 mins
[TIMING] data: 13.2ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.5ms
step 1475 | loss: 1.251332 | lr 1.8000e-04 | norm: 1.0030 | dt: 7818.73ms | tok/sec: 67055.4 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 17.8ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5348.6ms | clip: 0.0ms | optim: 23.5ms
step 1476 | loss: 1.206822 | lr 1.8000e-04 | norm: 1.1478 | dt: 7815.19ms | tok/sec: 67085.8 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 13.7ms | fwd: 2407.4ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.6ms
step 1477 | loss: 1.367875 | lr 1.8000e-04 | norm: 1.0398 | dt: 7813.63ms | tok/sec: 67099.2 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 15.5ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5347.5ms | clip: 0.0ms | optim: 23.5ms
step 1478 | loss: 1.388129 | lr 1.8000e-04 | norm: 1.1069 | dt: 7818.02ms | tok/sec: 67061.5 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 13.9ms | fwd: 2411.6ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.6ms
step 1479 | loss: 1.253885 | lr 1.8000e-04 | norm: 1.1244 | dt: 7818.98ms | tok/sec: 67053.3 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 18.5ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.6ms
step 1480 | loss: 1.297770 | lr 1.8000e-04 | norm: 1.0445 | dt: 7816.52ms | tok/sec: 67074.4 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 12.1ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.6ms
step 1481 | loss: 1.269149 | lr 1.8000e-04 | norm: 1.0851 | dt: 7813.06ms | tok/sec: 67104.1 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 13.4ms | fwd: 2406.4ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.5ms
step 1482 | loss: 1.337508 | lr 1.8000e-04 | norm: 0.9920 | dt: 7813.90ms | tok/sec: 67096.8 | mem: 72099MB | Time Left: 00 hrs : 22 mins
[TIMING] data: 18.2ms | fwd: 2406.5ms | loss: 4.9ms | bwd: 5348.1ms | clip: 0.0ms | optim: 23.7ms
step 1483 | loss: 1.470704 | lr 1.8000e-04 | norm: 1.4187 | dt: 7813.91ms | tok/sec: 67096.7 | mem: 72099MB | Time Left: 00 hrs : 21 mins
[TIMING] data: 14.3ms | fwd: 2409.5ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.6ms
step 1484 | loss: 1.271786 | lr 1.8000e-04 | norm: 1.0423 | dt: 7811.60ms | tok/sec: 67116.6 | mem: 72099MB | Time Left: 00 hrs : 21 mins
[TIMING] data: 13.3ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.6ms
step 1485 | loss: 1.154954 | lr 1.8000e-04 | norm: 1.0861 | dt: 7818.25ms | tok/sec: 67059.5 | mem: 72099MB | Time Left: 00 hrs : 21 mins
[TIMING] data: 13.3ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1486 | loss: 1.133864 | lr 1.8000e-04 | norm: 1.2555 | dt: 7817.89ms | tok/sec: 67062.6 | mem: 72099MB | Time Left: 00 hrs : 21 mins
[TIMING] data: 17.4ms | fwd: 2401.7ms | loss: 4.9ms | bwd: 5358.3ms | clip: 0.0ms | optim: 23.5ms
step 1487 | loss: 1.263058 | lr 1.8000e-04 | norm: 1.2645 | dt: 7809.65ms | tok/sec: 67133.4 | mem: 72099MB | Time Left: 00 hrs : 21 mins
[TIMING] data: 14.8ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.5ms
step 1488 | loss: 1.259770 | lr 1.8000e-04 | norm: 1.2231 | dt: 7817.45ms | tok/sec: 67066.3 | mem: 72099MB | Time Left: 00 hrs : 21 mins
[TIMING] data: 15.2ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.5ms
step 1489 | loss: 1.374304 | lr 1.8000e-04 | norm: 1.1077 | dt: 7819.09ms | tok/sec: 67052.3 | mem: 72099MB | Time Left: 00 hrs : 21 mins
[TIMING] data: 16.0ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5349.3ms | clip: 0.0ms | optim: 23.5ms
step 1490 | loss: 1.200793 | lr 1.8000e-04 | norm: 1.0420 | dt: 7821.97ms | tok/sec: 67027.7 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 14.5ms | fwd: 2410.9ms | loss: 4.9ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.6ms
step 1491 | loss: 1.357155 | lr 1.8000e-04 | norm: 1.1630 | dt: 7813.22ms | tok/sec: 67102.7 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 13.6ms | fwd: 2403.0ms | loss: 4.9ms | bwd: 5357.2ms | clip: 0.0ms | optim: 23.4ms
step 1492 | loss: 1.219023 | lr 1.8000e-04 | norm: 1.0978 | dt: 7804.97ms | tok/sec: 67173.6 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 15.9ms | fwd: 2399.7ms | loss: 4.9ms | bwd: 5349.2ms | clip: 0.0ms | optim: 23.6ms
step 1493 | loss: 1.231203 | lr 1.8000e-04 | norm: 1.0906 | dt: 7806.29ms | tok/sec: 67162.3 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 14.6ms | fwd: 2401.9ms | loss: 4.9ms | bwd: 5348.7ms | clip: 0.0ms | optim: 23.6ms
step 1494 | loss: 1.207950 | lr 1.8000e-04 | norm: 0.9668 | dt: 7812.88ms | tok/sec: 67105.6 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 16.1ms | fwd: 2401.1ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.6ms
step 1495 | loss: 1.389056 | lr 1.8000e-04 | norm: 1.9731 | dt: 7880.63ms | tok/sec: 66528.7 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 15.2ms | fwd: 2478.0ms | loss: 4.9ms | bwd: 5346.5ms | clip: 0.0ms | optim: 23.5ms
step 1496 | loss: 1.169307 | lr 1.8000e-04 | norm: 1.1697 | dt: 7813.45ms | tok/sec: 67100.7 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 14.3ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.5ms
step 1497 | loss: 1.257688 | lr 1.8000e-04 | norm: 1.0799 | dt: 7816.48ms | tok/sec: 67074.7 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 13.2ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5355.7ms | clip: 0.0ms | optim: 23.6ms
step 1498 | loss: 1.238875 | lr 1.8000e-04 | norm: 1.1414 | dt: 7885.03ms | tok/sec: 66491.6 | mem: 72099MB | Time Left: 00 hrs : 20 mins
[TIMING] data: 18.6ms | fwd: 2473.4ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.5ms
step 1499 | loss: 1.349622 | lr 1.8000e-04 | norm: 1.4533 | dt: 7803.17ms | tok/sec: 67189.1 | mem: 72099MB | Time Left: 00 hrs : 19 mins
[TIMING] data: 14.2ms | fwd: 2399.6ms | loss: 4.9ms | bwd: 5348.3ms | clip: 0.0ms | optim: 23.5ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1500.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1300.ckpt
validation loss: 1.2647
step 1500 | loss: 1.233095 | lr 1.8000e-04 | norm: 1.0690 | dt: 13972.05ms | tok/sec: 37524.0 | mem: 72099MB | Time Left: 00 hrs : 35 mins
[TIMING] data: 15.6ms | fwd: 2391.2ms | loss: 4.9ms | bwd: 5347.8ms | clip: 0.0ms | optim: 23.5ms
step 1501 | loss: 1.285088 | lr 1.8000e-04 | norm: 1.3200 | dt: 7795.87ms | tok/sec: 67252.0 | mem: 72099MB | Time Left: 00 hrs : 19 mins
[TIMING] data: 16.7ms | fwd: 2396.8ms | loss: 4.9ms | bwd: 5341.6ms | clip: 0.0ms | optim: 23.6ms
step 1502 | loss: 1.160646 | lr 1.8000e-04 | norm: 1.0528 | dt: 7840.10ms | tok/sec: 66872.6 | mem: 72099MB | Time Left: 00 hrs : 19 mins
[TIMING] data: 60.0ms | fwd: 2394.4ms | loss: 4.9ms | bwd: 5345.2ms | clip: 0.0ms | optim: 23.5ms
step 1503 | loss: 1.239704 | lr 1.8000e-04 | norm: 0.9611 | dt: 7844.61ms | tok/sec: 66834.2 | mem: 72099MB | Time Left: 00 hrs : 19 mins
[TIMING] data: 74.9ms | fwd: 2386.7ms | loss: 4.9ms | bwd: 5342.0ms | clip: 0.0ms | optim: 23.6ms
step 1504 | loss: 1.263737 | lr 1.8000e-04 | norm: 1.0217 | dt: 7865.65ms | tok/sec: 66655.4 | mem: 72099MB | Time Left: 00 hrs : 19 mins
[TIMING] data: 88.8ms | fwd: 2387.5ms | loss: 4.9ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.5ms
step 1505 | loss: 1.225602 | lr 1.8000e-04 | norm: 1.4194 | dt: 7810.12ms | tok/sec: 67129.3 | mem: 72099MB | Time Left: 00 hrs : 19 mins
[TIMING] data: 19.3ms | fwd: 2400.6ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.6ms
step 1506 | loss: 1.355846 | lr 1.8000e-04 | norm: 1.0219 | dt: 7814.27ms | tok/sec: 67093.6 | mem: 72099MB | Time Left: 00 hrs : 18 mins
[TIMING] data: 15.3ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.6ms
step 1507 | loss: 1.247173 | lr 1.8000e-04 | norm: 0.9694 | dt: 7826.17ms | tok/sec: 66991.7 | mem: 72099MB | Time Left: 00 hrs : 18 mins
[TIMING] data: 17.5ms | fwd: 2410.7ms | loss: 4.9ms | bwd: 5357.7ms | clip: 0.0ms | optim: 23.5ms
step 1508 | loss: 1.307646 | lr 1.8000e-04 | norm: 1.0731 | dt: 7805.62ms | tok/sec: 67168.0 | mem: 72099MB | Time Left: 00 hrs : 18 mins
[TIMING] data: 17.0ms | fwd: 2399.4ms | loss: 4.9ms | bwd: 5348.4ms | clip: 0.0ms | optim: 23.5ms
step 1509 | loss: 1.330871 | lr 1.8000e-04 | norm: 1.2258 | dt: 7805.21ms | tok/sec: 67171.5 | mem: 72099MB | Time Left: 00 hrs : 18 mins
[TIMING] data: 18.3ms | fwd: 2396.8ms | loss: 4.9ms | bwd: 5349.2ms | clip: 0.0ms | optim: 23.6ms
step 1510 | loss: 1.366974 | lr 1.8000e-04 | norm: 1.3043 | dt: 7813.72ms | tok/sec: 67098.4 | mem: 72099MB | Time Left: 00 hrs : 18 mins
[TIMING] data: 15.6ms | fwd: 2404.2ms | loss: 5.0ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 1511 | loss: 1.308841 | lr 1.8000e-04 | norm: 1.2088 | dt: 7817.78ms | tok/sec: 67063.6 | mem: 72099MB | Time Left: 00 hrs : 18 mins
[TIMING] data: 16.3ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.7ms
step 1512 | loss: 1.304388 | lr 1.8000e-04 | norm: 1.2823 | dt: 7818.93ms | tok/sec: 67053.7 | mem: 72099MB | Time Left: 00 hrs : 18 mins
[TIMING] data: 17.4ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.7ms
step 1513 | loss: 1.388971 | lr 1.8000e-04 | norm: 1.1337 | dt: 7821.18ms | tok/sec: 67034.4 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 17.5ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5356.2ms | clip: 0.0ms | optim: 23.6ms
step 1514 | loss: 1.311193 | lr 1.8000e-04 | norm: 1.7844 | dt: 7811.04ms | tok/sec: 67121.4 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 18.3ms | fwd: 2401.0ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.5ms
step 1515 | loss: 1.269253 | lr 1.8000e-04 | norm: 0.9728 | dt: 7883.77ms | tok/sec: 66502.2 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 15.0ms | fwd: 2473.0ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.6ms
step 1516 | loss: 1.293246 | lr 1.8000e-04 | norm: 0.9978 | dt: 7815.64ms | tok/sec: 67081.9 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 17.5ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5350.6ms | clip: 0.0ms | optim: 23.6ms
step 1517 | loss: 1.247858 | lr 1.8000e-04 | norm: 1.1155 | dt: 7822.25ms | tok/sec: 67025.2 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 13.0ms | fwd: 2411.4ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.5ms
step 1518 | loss: 1.275864 | lr 1.8000e-04 | norm: 1.0354 | dt: 7822.75ms | tok/sec: 67020.9 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 30.2ms | fwd: 2401.7ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 1519 | loss: 1.233127 | lr 1.8000e-04 | norm: 1.1671 | dt: 7810.15ms | tok/sec: 67129.1 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 16.5ms | fwd: 2403.7ms | loss: 4.9ms | bwd: 5349.7ms | clip: 0.0ms | optim: 23.5ms
step 1520 | loss: 1.217086 | lr 1.8000e-04 | norm: 1.2196 | dt: 7811.92ms | tok/sec: 67113.9 | mem: 72099MB | Time Left: 00 hrs : 17 mins
[TIMING] data: 15.2ms | fwd: 2401.5ms | loss: 4.9ms | bwd: 5354.6ms | clip: 0.0ms | optim: 23.5ms
step 1521 | loss: 1.189503 | lr 1.8000e-04 | norm: 1.1647 | dt: 7815.34ms | tok/sec: 67084.5 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 15.1ms | fwd: 2400.5ms | loss: 4.9ms | bwd: 5359.8ms | clip: 0.0ms | optim: 23.5ms
step 1522 | loss: 1.286132 | lr 1.8000e-04 | norm: 1.3894 | dt: 7819.42ms | tok/sec: 67049.5 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 16.2ms | fwd: 2410.5ms | loss: 4.9ms | bwd: 5352.3ms | clip: 0.0ms | optim: 23.7ms
step 1523 | loss: 1.219196 | lr 1.8000e-04 | norm: 1.0519 | dt: 7880.92ms | tok/sec: 66526.3 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 17.1ms | fwd: 2472.2ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.6ms
step 1524 | loss: 1.274430 | lr 1.8000e-04 | norm: 1.1464 | dt: 7815.44ms | tok/sec: 67083.7 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 14.0ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.7ms
step 1525 | loss: 1.222335 | lr 1.8000e-04 | norm: 1.2321 | dt: 7821.68ms | tok/sec: 67030.1 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 16.3ms | fwd: 2399.6ms | loss: 4.9ms | bwd: 5365.9ms | clip: 0.0ms | optim: 23.6ms
step 1526 | loss: 1.190813 | lr 1.8000e-04 | norm: 1.2312 | dt: 7812.48ms | tok/sec: 67109.0 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 20.2ms | fwd: 2403.1ms | loss: 4.9ms | bwd: 5348.4ms | clip: 0.0ms | optim: 23.6ms
step 1527 | loss: 1.202623 | lr 1.8000e-04 | norm: 1.0343 | dt: 7816.61ms | tok/sec: 67073.5 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 14.8ms | fwd: 2408.3ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.6ms
step 1528 | loss: 1.231412 | lr 1.8000e-04 | norm: 0.9719 | dt: 7811.22ms | tok/sec: 67119.8 | mem: 72099MB | Time Left: 00 hrs : 16 mins
[TIMING] data: 15.3ms | fwd: 2402.2ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 1529 | loss: 1.278926 | lr 1.8000e-04 | norm: 1.1132 | dt: 7810.96ms | tok/sec: 67122.1 | mem: 72099MB | Time Left: 00 hrs : 15 mins
[TIMING] data: 16.1ms | fwd: 2398.7ms | loss: 4.9ms | bwd: 5356.4ms | clip: 0.0ms | optim: 23.6ms
step 1530 | loss: 1.202650 | lr 1.8000e-04 | norm: 1.0948 | dt: 7819.89ms | tok/sec: 67045.5 | mem: 72099MB | Time Left: 00 hrs : 15 mins
[TIMING] data: 15.8ms | fwd: 2403.2ms | loss: 4.9ms | bwd: 5360.2ms | clip: 0.0ms | optim: 23.5ms
step 1531 | loss: 1.227827 | lr 1.8000e-04 | norm: 1.1672 | dt: 7814.61ms | tok/sec: 67090.8 | mem: 72099MB | Time Left: 00 hrs : 15 mins
[TIMING] data: 15.9ms | fwd: 2404.6ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.7ms
step 1532 | loss: 1.343432 | lr 1.8000e-04 | norm: 1.1385 | dt: 7830.64ms | tok/sec: 66953.4 | mem: 72099MB | Time Left: 00 hrs : 15 mins
[TIMING] data: 18.0ms | fwd: 2423.3ms | loss: 4.9ms | bwd: 5349.4ms | clip: 0.0ms | optim: 23.7ms
step 1533 | loss: 1.171617 | lr 1.8000e-04 | norm: 0.9830 | dt: 7820.44ms | tok/sec: 67040.7 | mem: 72099MB | Time Left: 00 hrs : 15 mins
[TIMING] data: 18.0ms | fwd: 2409.0ms | loss: 4.9ms | bwd: 5352.7ms | clip: 0.0ms | optim: 23.6ms
step 1534 | loss: 1.162325 | lr 1.8000e-04 | norm: 1.2075 | dt: 7818.08ms | tok/sec: 67060.9 | mem: 72099MB | Time Left: 00 hrs : 15 mins
[TIMING] data: 16.7ms | fwd: 2403.5ms | loss: 4.9ms | bwd: 5357.9ms | clip: 0.0ms | optim: 23.6ms
step 1535 | loss: 1.207541 | lr 1.8000e-04 | norm: 1.1145 | dt: 7812.23ms | tok/sec: 67111.2 | mem: 72099MB | Time Left: 00 hrs : 15 mins
[TIMING] data: 16.7ms | fwd: 2403.6ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.7ms
step 1536 | loss: 1.240749 | lr 1.8000e-04 | norm: 1.2759 | dt: 7820.28ms | tok/sec: 67042.1 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 14.3ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5359.2ms | clip: 0.0ms | optim: 23.6ms
step 1537 | loss: 1.287317 | lr 1.8000e-04 | norm: 0.9721 | dt: 7808.08ms | tok/sec: 67146.8 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 14.2ms | fwd: 2399.7ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 1538 | loss: 1.232781 | lr 1.8000e-04 | norm: 0.9573 | dt: 7822.59ms | tok/sec: 67022.3 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 16.2ms | fwd: 2400.8ms | loss: 4.9ms | bwd: 5364.4ms | clip: 0.0ms | optim: 23.6ms
step 1539 | loss: 1.260195 | lr 1.8000e-04 | norm: 1.1319 | dt: 7814.13ms | tok/sec: 67094.9 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 17.2ms | fwd: 2404.7ms | loss: 4.9ms | bwd: 5351.7ms | clip: 0.0ms | optim: 23.5ms
step 1540 | loss: 1.269723 | lr 1.8000e-04 | norm: 1.0666 | dt: 7822.93ms | tok/sec: 67019.4 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 16.9ms | fwd: 2409.6ms | loss: 4.9ms | bwd: 5355.6ms | clip: 0.0ms | optim: 23.5ms
step 1541 | loss: 1.254019 | lr 1.8000e-04 | norm: 1.0196 | dt: 7824.42ms | tok/sec: 67006.6 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 16.4ms | fwd: 2409.7ms | loss: 4.9ms | bwd: 5357.3ms | clip: 0.0ms | optim: 23.6ms
step 1542 | loss: 1.291829 | lr 1.8000e-04 | norm: 1.2544 | dt: 7823.49ms | tok/sec: 67014.6 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 16.9ms | fwd: 2411.0ms | loss: 4.9ms | bwd: 5354.7ms | clip: 0.0ms | optim: 23.6ms
step 1543 | loss: 1.178192 | lr 1.8000e-04 | norm: 1.0689 | dt: 7816.28ms | tok/sec: 67076.4 | mem: 72099MB | Time Left: 00 hrs : 14 mins
[TIMING] data: 15.5ms | fwd: 2407.1ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.7ms
step 1544 | loss: 1.203143 | lr 1.8000e-04 | norm: 1.0055 | dt: 7813.60ms | tok/sec: 67099.4 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 17.2ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1545 | loss: 1.398370 | lr 1.8000e-04 | norm: 1.2919 | dt: 7810.84ms | tok/sec: 67123.1 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 20.6ms | fwd: 2399.7ms | loss: 4.9ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.6ms
step 1546 | loss: 1.221808 | lr 1.8000e-04 | norm: 1.0157 | dt: 7815.93ms | tok/sec: 67079.4 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 17.5ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.6ms
step 1547 | loss: 1.186722 | lr 1.8000e-04 | norm: 1.0955 | dt: 7823.69ms | tok/sec: 67012.8 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 16.2ms | fwd: 2406.6ms | loss: 4.9ms | bwd: 5360.5ms | clip: 0.0ms | optim: 23.6ms
step 1548 | loss: 1.224651 | lr 1.8000e-04 | norm: 1.0439 | dt: 7816.52ms | tok/sec: 67074.4 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 17.3ms | fwd: 2399.0ms | loss: 4.9ms | bwd: 5360.3ms | clip: 0.0ms | optim: 23.5ms
step 1549 | loss: 1.137469 | lr 1.8000e-04 | norm: 1.0506 | dt: 7820.50ms | tok/sec: 67040.3 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 20.6ms | fwd: 2401.2ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
step 1550 | loss: 1.235125 | lr 1.8000e-04 | norm: 1.0905 | dt: 7809.40ms | tok/sec: 67135.5 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 16.3ms | fwd: 2398.5ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 1551 | loss: 1.077533 | lr 1.8000e-04 | norm: 1.1658 | dt: 7816.04ms | tok/sec: 67078.5 | mem: 72099MB | Time Left: 00 hrs : 13 mins
[TIMING] data: 18.7ms | fwd: 2403.4ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.5ms
step 1552 | loss: 1.277788 | lr 1.7997e-04 | norm: 0.9571 | dt: 7821.65ms | tok/sec: 67030.4 | mem: 72099MB | Time Left: 00 hrs : 12 mins
[TIMING] data: 16.5ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5352.8ms | clip: 0.0ms | optim: 23.5ms
step 1553 | loss: 1.374324 | lr 1.7988e-04 | norm: 1.3010 | dt: 7816.35ms | tok/sec: 67075.8 | mem: 72099MB | Time Left: 00 hrs : 12 mins
[TIMING] data: 18.2ms | fwd: 2403.4ms | loss: 4.9ms | bwd: 5354.4ms | clip: 0.0ms | optim: 23.8ms
step 1554 | loss: 1.469083 | lr 1.7973e-04 | norm: 1.2918 | dt: 7819.32ms | tok/sec: 67050.3 | mem: 72099MB | Time Left: 00 hrs : 12 mins
[TIMING] data: 15.3ms | fwd: 2406.7ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.6ms
step 1555 | loss: 1.355126 | lr 1.7953e-04 | norm: 1.0991 | dt: 7812.27ms | tok/sec: 67110.8 | mem: 72099MB | Time Left: 00 hrs : 12 mins
[TIMING] data: 15.5ms | fwd: 2401.1ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.7ms
step 1556 | loss: 1.128554 | lr 1.7926e-04 | norm: 0.8756 | dt: 7816.70ms | tok/sec: 67072.8 | mem: 72099MB | Time Left: 00 hrs : 12 mins
[TIMING] data: 16.6ms | fwd: 2405.3ms | loss: 4.9ms | bwd: 5353.4ms | clip: 0.0ms | optim: 23.6ms
step 1557 | loss: 1.220714 | lr 1.7894e-04 | norm: 0.8881 | dt: 7820.79ms | tok/sec: 67037.7 | mem: 72099MB | Time Left: 00 hrs : 12 mins
[TIMING] data: 18.9ms | fwd: 2408.2ms | loss: 4.9ms | bwd: 5353.1ms | clip: 0.0ms | optim: 23.6ms
step 1558 | loss: 1.407812 | lr 1.7856e-04 | norm: 1.3105 | dt: 7814.50ms | tok/sec: 67091.7 | mem: 72099MB | Time Left: 00 hrs : 12 mins
[TIMING] data: 16.6ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.6ms
step 1559 | loss: 1.250552 | lr 1.7812e-04 | norm: 1.0571 | dt: 7820.91ms | tok/sec: 67036.7 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 14.3ms | fwd: 2405.6ms | loss: 4.9ms | bwd: 5360.2ms | clip: 0.0ms | optim: 23.5ms
step 1560 | loss: 1.206215 | lr 1.7762e-04 | norm: 1.0973 | dt: 7812.99ms | tok/sec: 67104.7 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 18.7ms | fwd: 2402.9ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.6ms
step 1561 | loss: 1.244336 | lr 1.7706e-04 | norm: 0.9781 | dt: 7807.16ms | tok/sec: 67154.8 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 16.5ms | fwd: 2401.9ms | loss: 4.9ms | bwd: 5349.1ms | clip: 0.0ms | optim: 23.6ms
step 1562 | loss: 1.314606 | lr 1.7645e-04 | norm: 1.1587 | dt: 7812.75ms | tok/sec: 67106.8 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 18.5ms | fwd: 2401.3ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1563 | loss: 1.281016 | lr 1.7579e-04 | norm: 1.0771 | dt: 7817.35ms | tok/sec: 67067.2 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 17.7ms | fwd: 2401.6ms | loss: 4.9ms | bwd: 5357.1ms | clip: 0.0ms | optim: 23.5ms
step 1564 | loss: 1.249706 | lr 1.7507e-04 | norm: 1.2325 | dt: 7817.54ms | tok/sec: 67065.6 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 15.1ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.5ms
step 1565 | loss: 1.243497 | lr 1.7429e-04 | norm: 1.0860 | dt: 7809.83ms | tok/sec: 67131.8 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 15.6ms | fwd: 2404.1ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.6ms
step 1566 | loss: 1.313329 | lr 1.7346e-04 | norm: 1.0017 | dt: 7882.47ms | tok/sec: 66513.2 | mem: 72099MB | Time Left: 00 hrs : 11 mins
[TIMING] data: 14.9ms | fwd: 2470.3ms | loss: 4.9ms | bwd: 5356.6ms | clip: 0.0ms | optim: 23.6ms
step 1567 | loss: 1.078716 | lr 1.7258e-04 | norm: 1.1819 | dt: 7811.58ms | tok/sec: 67116.7 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 18.1ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5349.9ms | clip: 0.0ms | optim: 23.5ms
step 1568 | loss: 1.220803 | lr 1.7164e-04 | norm: 1.2531 | dt: 7810.27ms | tok/sec: 67128.1 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 18.1ms | fwd: 2398.5ms | loss: 4.9ms | bwd: 5353.6ms | clip: 0.0ms | optim: 23.5ms
step 1569 | loss: 1.221589 | lr 1.7066e-04 | norm: 1.2318 | dt: 7829.43ms | tok/sec: 66963.8 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 33.2ms | fwd: 2402.5ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 1570 | loss: 1.190839 | lr 1.6962e-04 | norm: 1.1502 | dt: 7814.19ms | tok/sec: 67094.3 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 19.1ms | fwd: 2401.9ms | loss: 4.9ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.5ms
step 1571 | loss: 1.294721 | lr 1.6854e-04 | norm: 1.3328 | dt: 7816.20ms | tok/sec: 67077.1 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 18.0ms | fwd: 2404.8ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
step 1572 | loss: 1.193118 | lr 1.6741e-04 | norm: 1.1391 | dt: 7818.12ms | tok/sec: 67060.6 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 17.2ms | fwd: 2407.9ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.6ms
step 1573 | loss: 1.134392 | lr 1.6623e-04 | norm: 0.8613 | dt: 7816.98ms | tok/sec: 67070.4 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 17.4ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.6ms
step 1574 | loss: 1.254033 | lr 1.6501e-04 | norm: 1.0637 | dt: 7818.84ms | tok/sec: 67054.5 | mem: 72099MB | Time Left: 00 hrs : 10 mins
[TIMING] data: 15.7ms | fwd: 2405.2ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.6ms
step 1575 | loss: 1.152730 | lr 1.6374e-04 | norm: 1.0841 | dt: 7813.31ms | tok/sec: 67101.9 | mem: 72099MB | Time Left: 00 hrs : 09 mins
[TIMING] data: 16.5ms | fwd: 2401.6ms | loss: 4.9ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
step 1576 | loss: 1.260830 | lr 1.6243e-04 | norm: 0.9521 | dt: 7813.62ms | tok/sec: 67099.3 | mem: 72099MB | Time Left: 00 hrs : 09 mins
[TIMING] data: 14.9ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.5ms
step 1577 | loss: 1.226766 | lr 1.6107e-04 | norm: 0.9406 | dt: 7824.72ms | tok/sec: 67004.1 | mem: 72099MB | Time Left: 00 hrs : 09 mins
[TIMING] data: 16.3ms | fwd: 2410.0ms | loss: 4.9ms | bwd: 5357.6ms | clip: 0.0ms | optim: 23.6ms
step 1578 | loss: 1.243428 | lr 1.5968e-04 | norm: 0.9916 | dt: 7816.21ms | tok/sec: 67077.0 | mem: 72099MB | Time Left: 00 hrs : 09 mins
[TIMING] data: 17.6ms | fwd: 2404.7ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.6ms
step 1579 | loss: 1.301050 | lr 1.5825e-04 | norm: 1.1568 | dt: 7811.98ms | tok/sec: 67113.3 | mem: 72099MB | Time Left: 00 hrs : 09 mins
[TIMING] data: 16.7ms | fwd: 2402.1ms | loss: 4.9ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 1580 | loss: 1.298924 | lr 1.5677e-04 | norm: 2.0073 | dt: 7818.59ms | tok/sec: 67056.6 | mem: 72099MB | Time Left: 00 hrs : 09 mins
[TIMING] data: 18.4ms | fwd: 2403.4ms | loss: 4.9ms | bwd: 5356.5ms | clip: 0.0ms | optim: 23.6ms
step 1581 | loss: 1.177465 | lr 1.5527e-04 | norm: 0.8785 | dt: 7820.05ms | tok/sec: 67044.1 | mem: 72099MB | Time Left: 00 hrs : 09 mins
[TIMING] data: 16.6ms | fwd: 2403.4ms | loss: 4.9ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.5ms
step 1582 | loss: 1.265492 | lr 1.5373e-04 | norm: 0.9842 | dt: 7817.76ms | tok/sec: 67063.7 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 14.5ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.5ms
step 1583 | loss: 1.236817 | lr 1.5215e-04 | norm: 1.0193 | dt: 7813.30ms | tok/sec: 67102.0 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 16.5ms | fwd: 2399.9ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 23.6ms
step 1584 | loss: 1.238941 | lr 1.5054e-04 | norm: 1.0149 | dt: 7820.33ms | tok/sec: 67041.7 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 20.0ms | fwd: 2407.6ms | loss: 4.9ms | bwd: 5352.4ms | clip: 0.0ms | optim: 23.5ms
step 1585 | loss: 1.110817 | lr 1.4891e-04 | norm: 1.0643 | dt: 7817.64ms | tok/sec: 67064.7 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 17.3ms | fwd: 2405.5ms | loss: 4.9ms | bwd: 5354.0ms | clip: 0.0ms | optim: 23.6ms
step 1586 | loss: 1.293768 | lr 1.4724e-04 | norm: 1.0346 | dt: 7817.16ms | tok/sec: 67068.9 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 18.4ms | fwd: 2410.4ms | loss: 4.9ms | bwd: 5348.0ms | clip: 0.0ms | optim: 23.6ms
step 1587 | loss: 1.173843 | lr 1.4555e-04 | norm: 1.0146 | dt: 7821.32ms | tok/sec: 67033.2 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 20.9ms | fwd: 2405.7ms | loss: 4.9ms | bwd: 5353.8ms | clip: 0.0ms | optim: 23.5ms
step 1588 | loss: 1.255829 | lr 1.4383e-04 | norm: 1.0727 | dt: 7818.33ms | tok/sec: 67058.8 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 18.3ms | fwd: 2402.1ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.5ms
step 1589 | loss: 1.259566 | lr 1.4209e-04 | norm: 0.9189 | dt: 7817.11ms | tok/sec: 67069.3 | mem: 72099MB | Time Left: 00 hrs : 08 mins
[TIMING] data: 18.0ms | fwd: 2404.2ms | loss: 4.9ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.6ms
step 1590 | loss: 1.252019 | lr 1.4032e-04 | norm: 1.1211 | dt: 7818.14ms | tok/sec: 67060.5 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 18.3ms | fwd: 2403.6ms | loss: 4.9ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 1591 | loss: 1.250742 | lr 1.3854e-04 | norm: 1.0452 | dt: 7812.53ms | tok/sec: 67108.6 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 18.4ms | fwd: 2396.7ms | loss: 4.9ms | bwd: 5356.0ms | clip: 0.0ms | optim: 23.6ms
step 1592 | loss: 1.341874 | lr 1.3674e-04 | norm: 0.9774 | dt: 7824.35ms | tok/sec: 67007.3 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 19.0ms | fwd: 2406.2ms | loss: 4.9ms | bwd: 5358.9ms | clip: 0.0ms | optim: 23.5ms
step 1593 | loss: 1.248424 | lr 1.3492e-04 | norm: 0.8539 | dt: 7819.06ms | tok/sec: 67052.5 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 16.9ms | fwd: 2407.5ms | loss: 4.9ms | bwd: 5353.9ms | clip: 0.0ms | optim: 23.7ms
step 1594 | loss: 1.205069 | lr 1.3309e-04 | norm: 1.2216 | dt: 7812.49ms | tok/sec: 67108.9 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 19.3ms | fwd: 2398.9ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.7ms
step 1595 | loss: 1.193912 | lr 1.3124e-04 | norm: 1.0050 | dt: 7823.09ms | tok/sec: 67018.0 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 19.0ms | fwd: 2407.0ms | loss: 4.9ms | bwd: 5356.9ms | clip: 0.0ms | optim: 23.6ms
step 1596 | loss: 1.182748 | lr 1.2939e-04 | norm: 0.9644 | dt: 7809.77ms | tok/sec: 67132.3 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 17.2ms | fwd: 2396.4ms | loss: 4.9ms | bwd: 5355.5ms | clip: 0.0ms | optim: 23.5ms
step 1597 | loss: 1.113575 | lr 1.2752e-04 | norm: 1.1117 | dt: 7828.55ms | tok/sec: 66971.2 | mem: 72099MB | Time Left: 00 hrs : 07 mins
[TIMING] data: 34.8ms | fwd: 2401.1ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.4ms
step 1598 | loss: 1.216396 | lr 1.2565e-04 | norm: 1.0476 | dt: 7814.52ms | tok/sec: 67091.5 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 16.4ms | fwd: 2398.3ms | loss: 4.9ms | bwd: 5359.5ms | clip: 0.0ms | optim: 23.6ms
step 1599 | loss: 1.204482 | lr 1.2377e-04 | norm: 1.0541 | dt: 7818.82ms | tok/sec: 67054.6 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 19.8ms | fwd: 2403.3ms | loss: 4.9ms | bwd: 5354.3ms | clip: 0.0ms | optim: 23.6ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1600.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1400.ckpt
step 1600 | loss: 1.127827 | lr 1.2188e-04 | norm: 0.9726 | dt: 7800.60ms | tok/sec: 67211.3 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 13.1ms | fwd: 2402.8ms | loss: 4.9ms | bwd: 5343.6ms | clip: 0.0ms | optim: 23.5ms
step 1601 | loss: 1.108025 | lr 1.2000e-04 | norm: 1.0705 | dt: 7793.91ms | tok/sec: 67268.9 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 17.9ms | fwd: 2395.0ms | loss: 4.9ms | bwd: 5340.2ms | clip: 0.0ms | optim: 23.5ms
step 1602 | loss: 1.240375 | lr 1.1812e-04 | norm: 0.9580 | dt: 7796.27ms | tok/sec: 67248.6 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 16.6ms | fwd: 2391.3ms | loss: 4.8ms | bwd: 5346.4ms | clip: 0.0ms | optim: 23.5ms
step 1603 | loss: 1.084311 | lr 1.1623e-04 | norm: 0.9134 | dt: 7810.93ms | tok/sec: 67122.3 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 15.3ms | fwd: 2400.9ms | loss: 4.8ms | bwd: 5343.9ms | clip: 0.0ms | optim: 23.6ms
step 1604 | loss: 1.163715 | lr 1.1435e-04 | norm: 0.8906 | dt: 7925.37ms | tok/sec: 66153.2 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 138.9ms | fwd: 2385.8ms | loss: 4.9ms | bwd: 5360.3ms | clip: 0.0ms | optim: 23.5ms
step 1605 | loss: 1.134250 | lr 1.1248e-04 | norm: 0.8630 | dt: 7824.00ms | tok/sec: 67010.3 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 21.5ms | fwd: 2402.5ms | loss: 4.8ms | bwd: 5358.1ms | clip: 0.0ms | optim: 23.5ms
step 1606 | loss: 1.181737 | lr 1.1061e-04 | norm: 1.0775 | dt: 7815.66ms | tok/sec: 67081.8 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 14.0ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5353.2ms | clip: 0.0ms | optim: 23.6ms
step 1607 | loss: 1.073119 | lr 1.0876e-04 | norm: 0.9189 | dt: 7811.50ms | tok/sec: 67117.5 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 15.5ms | fwd: 2404.4ms | loss: 4.9ms | bwd: 5351.0ms | clip: 0.0ms | optim: 23.5ms
step 1608 | loss: 1.175481 | lr 1.0691e-04 | norm: 1.0449 | dt: 7812.35ms | tok/sec: 67110.1 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 18.1ms | fwd: 2407.2ms | loss: 4.9ms | bwd: 5346.2ms | clip: 0.0ms | optim: 23.5ms
step 1609 | loss: 1.170071 | lr 1.0508e-04 | norm: 0.8115 | dt: 7809.52ms | tok/sec: 67134.5 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 15.7ms | fwd: 2407.3ms | loss: 4.9ms | bwd: 5346.5ms | clip: 0.0ms | optim: 23.5ms
step 1610 | loss: 1.254619 | lr 1.0326e-04 | norm: 0.8679 | dt: 7872.31ms | tok/sec: 66599.0 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 17.3ms | fwd: 2464.2ms | loss: 4.9ms | bwd: 5351.1ms | clip: 0.0ms | optim: 23.5ms
step 1611 | loss: 1.209059 | lr 1.0146e-04 | norm: 0.8447 | dt: 7851.52ms | tok/sec: 66775.3 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 22.6ms | fwd: 2397.8ms | loss: 4.9ms | bwd: 5391.4ms | clip: 0.0ms | optim: 23.5ms
step 1612 | loss: 1.129874 | lr 9.9676e-05 | norm: 1.0798 | dt: 7876.18ms | tok/sec: 66566.3 | mem: 72099MB | Time Left: 00 hrs : 05 mins
[TIMING] data: 17.9ms | fwd: 2473.1ms | loss: 4.9ms | bwd: 5345.1ms | clip: 0.0ms | optim: 23.5ms
step 1613 | loss: 1.155663 | lr 9.7913e-05 | norm: 0.9037 | dt: 7881.76ms | tok/sec: 66519.1 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 17.6ms | fwd: 2471.6ms | loss: 4.9ms | bwd: 5352.9ms | clip: 0.0ms | optim: 23.5ms
step 1614 | loss: 1.210181 | lr 9.6171e-05 | norm: 1.2879 | dt: 7814.45ms | tok/sec: 67092.2 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 17.5ms | fwd: 2409.1ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.5ms
step 1615 | loss: 1.236677 | lr 9.4453e-05 | norm: 1.1388 | dt: 7812.18ms | tok/sec: 67111.6 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 16.8ms | fwd: 2408.4ms | loss: 4.9ms | bwd: 5346.8ms | clip: 0.0ms | optim: 23.6ms
step 1616 | loss: 1.303416 | lr 9.2761e-05 | norm: 0.9311 | dt: 7822.96ms | tok/sec: 67019.1 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 16.5ms | fwd: 2414.3ms | loss: 4.9ms | bwd: 5352.0ms | clip: 0.0ms | optim: 23.5ms
step 1617 | loss: 1.197015 | lr 9.1095e-05 | norm: 1.0400 | dt: 7818.48ms | tok/sec: 67057.5 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 16.6ms | fwd: 2405.4ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 1618 | loss: 1.155305 | lr 8.9458e-05 | norm: 0.9408 | dt: 7812.86ms | tok/sec: 67105.8 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 16.8ms | fwd: 2403.9ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.6ms
step 1619 | loss: 1.167640 | lr 8.7850e-05 | norm: 0.9665 | dt: 7810.99ms | tok/sec: 67121.9 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 15.9ms | fwd: 2406.1ms | loss: 4.9ms | bwd: 5348.7ms | clip: 0.0ms | optim: 23.5ms
step 1620 | loss: 1.205910 | lr 8.6275e-05 | norm: 1.1015 | dt: 7811.04ms | tok/sec: 67121.4 | mem: 72099MB | Time Left: 00 hrs : 04 mins
[TIMING] data: 15.9ms | fwd: 2395.9ms | loss: 4.8ms | bwd: 5359.1ms | clip: 0.0ms | optim: 23.6ms
step 1621 | loss: 1.225119 | lr 8.4733e-05 | norm: 1.5180 | dt: 7816.40ms | tok/sec: 67075.4 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 15.6ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5354.8ms | clip: 0.0ms | optim: 23.5ms
step 1622 | loss: 1.087918 | lr 8.3226e-05 | norm: 1.0009 | dt: 7808.76ms | tok/sec: 67141.0 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 17.6ms | fwd: 2400.7ms | loss: 4.8ms | bwd: 5350.4ms | clip: 0.0ms | optim: 23.5ms
step 1623 | loss: 1.242468 | lr 8.1755e-05 | norm: 0.9243 | dt: 7814.13ms | tok/sec: 67094.9 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 16.1ms | fwd: 2403.7ms | loss: 4.8ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
step 1624 | loss: 1.142807 | lr 8.0321e-05 | norm: 0.9412 | dt: 7816.88ms | tok/sec: 67071.2 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 18.9ms | fwd: 2402.6ms | loss: 4.8ms | bwd: 5355.4ms | clip: 0.0ms | optim: 23.6ms
step 1625 | loss: 1.177183 | lr 7.8927e-05 | norm: 1.0796 | dt: 7810.14ms | tok/sec: 67129.2 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 17.7ms | fwd: 2401.6ms | loss: 4.9ms | bwd: 5350.2ms | clip: 0.0ms | optim: 23.6ms
step 1626 | loss: 1.266763 | lr 7.7574e-05 | norm: 1.1686 | dt: 7815.07ms | tok/sec: 67086.8 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 16.7ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5349.8ms | clip: 0.0ms | optim: 23.4ms
step 1627 | loss: 1.172556 | lr 7.6262e-05 | norm: 1.0126 | dt: 7813.12ms | tok/sec: 67103.6 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 16.9ms | fwd: 2398.9ms | loss: 4.9ms | bwd: 5356.1ms | clip: 0.0ms | optim: 23.5ms
step 1628 | loss: 1.231929 | lr 7.4993e-05 | norm: 0.8638 | dt: 7871.36ms | tok/sec: 66607.1 | mem: 72099MB | Time Left: 00 hrs : 03 mins
[TIMING] data: 17.5ms | fwd: 2465.0ms | loss: 4.9ms | bwd: 5349.5ms | clip: 0.0ms | optim: 23.5ms
step 1629 | loss: 1.205837 | lr 7.3769e-05 | norm: 0.9145 | dt: 7818.85ms | tok/sec: 67054.4 | mem: 72099MB | Time Left: 00 hrs : 02 mins
[TIMING] data: 17.4ms | fwd: 2408.8ms | loss: 4.9ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 1630 | loss: 1.285491 | lr 7.2591e-05 | norm: 1.0306 | dt: 7810.13ms | tok/sec: 67129.3 | mem: 72099MB | Time Left: 00 hrs : 02 mins
[TIMING] data: 17.4ms | fwd: 2399.5ms | loss: 4.8ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.5ms
step 1631 | loss: 1.155377 | lr 7.1459e-05 | norm: 1.6325 | dt: 7829.37ms | tok/sec: 66964.3 | mem: 72099MB | Time Left: 00 hrs : 02 mins
[TIMING] data: 18.7ms | fwd: 2412.2ms | loss: 4.8ms | bwd: 5357.4ms | clip: 0.0ms | optim: 23.6ms
step 1632 | loss: 1.255268 | lr 7.0375e-05 | norm: 1.0579 | dt: 7816.59ms | tok/sec: 67073.8 | mem: 72099MB | Time Left: 00 hrs : 02 mins
[TIMING] data: 16.3ms | fwd: 2406.0ms | loss: 4.8ms | bwd: 5353.7ms | clip: 0.0ms | optim: 23.6ms
step 1633 | loss: 1.221732 | lr 6.9340e-05 | norm: 0.8338 | dt: 7817.87ms | tok/sec: 67062.7 | mem: 72099MB | Time Left: 00 hrs : 02 mins
[TIMING] data: 18.3ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5355.3ms | clip: 0.0ms | optim: 23.5ms
step 1634 | loss: 1.263771 | lr 6.8355e-05 | norm: 0.9406 | dt: 7814.53ms | tok/sec: 67091.4 | mem: 72099MB | Time Left: 00 hrs : 02 mins
[TIMING] data: 16.0ms | fwd: 2403.8ms | loss: 4.9ms | bwd: 5354.5ms | clip: 0.0ms | optim: 23.5ms
step 1635 | loss: 1.197012 | lr 6.7422e-05 | norm: 0.9620 | dt: 25738.10ms | tok/sec: 20370.1 | mem: 72099MB | Time Left: 00 hrs : 06 mins
[TIMING] data: 28.4ms | fwd: 2399.4ms | loss: 4.9ms | bwd: 5356.7ms | clip: 0.0ms | optim: 17937.3ms
step 1636 | loss: 1.072389 | lr 6.6540e-05 | norm: 0.8927 | dt: 7789.76ms | tok/sec: 67304.8 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 17.0ms | fwd: 2391.0ms | loss: 4.9ms | bwd: 5342.0ms | clip: 0.0ms | optim: 23.4ms
step 1637 | loss: 0.921950 | lr 6.5710e-05 | norm: 0.8799 | dt: 7861.68ms | tok/sec: 66689.1 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 15.8ms | fwd: 2456.3ms | loss: 4.9ms | bwd: 5348.8ms | clip: 0.0ms | optim: 23.5ms
step 1638 | loss: 1.229988 | lr 6.4935e-05 | norm: 0.9747 | dt: 7793.52ms | tok/sec: 67272.3 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 15.7ms | fwd: 2391.5ms | loss: 4.9ms | bwd: 5346.6ms | clip: 0.0ms | optim: 23.5ms
step 1639 | loss: 1.099949 | lr 6.4213e-05 | norm: 1.0113 | dt: 7792.42ms | tok/sec: 67281.8 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 15.9ms | fwd: 2395.0ms | loss: 4.9ms | bwd: 5341.8ms | clip: 0.0ms | optim: 23.5ms
step 1640 | loss: 1.260071 | lr 6.3547e-05 | norm: 1.2374 | dt: 7802.40ms | tok/sec: 67195.7 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 15.4ms | fwd: 2395.6ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 1641 | loss: 1.052412 | lr 6.2937e-05 | norm: 1.0500 | dt: 7806.16ms | tok/sec: 67163.3 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 19.3ms | fwd: 2393.3ms | loss: 4.9ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.5ms
step 1642 | loss: 1.238169 | lr 6.2382e-05 | norm: 1.1215 | dt: 7803.45ms | tok/sec: 67186.7 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 16.4ms | fwd: 2405.0ms | loss: 4.9ms | bwd: 5340.7ms | clip: 0.0ms | optim: 23.5ms
step 1643 | loss: 1.324833 | lr 6.1885e-05 | norm: 1.3045 | dt: 7810.06ms | tok/sec: 67129.8 | mem: 72099MB | Time Left: 00 hrs : 01 mins
[TIMING] data: 17.2ms | fwd: 2401.2ms | loss: 4.9ms | bwd: 5351.3ms | clip: 0.0ms | optim: 23.5ms
step 1644 | loss: 1.068650 | lr 6.1445e-05 | norm: 0.9132 | dt: 7813.73ms | tok/sec: 67098.3 | mem: 72099MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 21.4ms | fwd: 2397.4ms | loss: 4.9ms | bwd: 5354.9ms | clip: 0.0ms | optim: 23.6ms
step 1645 | loss: 1.206796 | lr 6.1063e-05 | norm: 1.0654 | dt: 7800.54ms | tok/sec: 67211.8 | mem: 72099MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 20.2ms | fwd: 2395.5ms | loss: 4.9ms | bwd: 5344.0ms | clip: 0.0ms | optim: 23.6ms
step 1646 | loss: 1.376950 | lr 6.0739e-05 | norm: 0.9855 | dt: 7808.44ms | tok/sec: 67143.8 | mem: 72099MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 15.0ms | fwd: 2401.3ms | loss: 4.9ms | bwd: 5351.5ms | clip: 0.0ms | optim: 23.5ms
step 1647 | loss: 1.161117 | lr 6.0473e-05 | norm: 1.1528 | dt: 7808.05ms | tok/sec: 67147.1 | mem: 72099MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 18.0ms | fwd: 2397.3ms | loss: 4.9ms | bwd: 5352.1ms | clip: 0.0ms | optim: 23.6ms
step 1648 | loss: 1.189162 | lr 6.0266e-05 | norm: 1.0246 | dt: 7807.30ms | tok/sec: 67153.5 | mem: 72099MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 18.4ms | fwd: 2399.3ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.6ms
step 1649 | loss: 1.167197 | lr 6.0118e-05 | norm: 1.1009 | dt: 7811.81ms | tok/sec: 67114.8 | mem: 72099MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 17.3ms | fwd: 2404.0ms | loss: 4.9ms | bwd: 5349.0ms | clip: 0.0ms | optim: 23.5ms
validation loss: 1.1942
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/b5_65536_131072/train_shard_00000.bin
100000000
step 1650 | loss: 1.143309 | lr 6.0030e-05 | norm: 1.3213 | dt: 13821.44ms | tok/sec: 37932.9 | mem: 72099MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 36.4ms | fwd: 2407.8ms | loss: 4.9ms | bwd: 5362.6ms | clip: 0.0ms | optim: 23.6ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1651.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run4_step_1500.ckpt
=== Training complete ===
=== Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
d_model/layers : 1536 / 36
q/kv heads : 12 / 4 (G=3, hd=128)
ffn_hidden : 4096 (SwiGLU)
context_length : 65536
vocab_size : 50304 tying=0
B x T : 1 x 65536
world_size : 8 (dp=1 x cp=8)
CP_ATTN_MODE : ring (load_balance=HeadTail)
global_batch : 524288 (grad_accum=8)
precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
max_lr/min_lr : 0.0006 / 6e-05
warmup/steps : 100 / 1651 (WSD: stable=1451, decay tail=100)
checkpointing : true (ckpt_freq=100, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_695.ckpt (base step 695, loss 2.93407) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 1651
warmup_steps: 100
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=983236608 padded=983236608 W=8 shard=122904576 | 38 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 1406.53 MB (vs 11252.2 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=77388288 padded=77388288 W=8 shard=9673536 | 6 buckets of 50 MB (1638400 elems/rank/bucket) | optimizer state/GPU: 110.705 MB (vs 885.639 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 9 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log4.csv (run 4)
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run4_step_<N>.ckpt (every 100 steps, keep 2)
[re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
Starting training...
[CP ring P2P] OVERLAP: dedicated non-blocking stream
validation loss: 2.9599
[CP bwd] overlap=ON
step 0 | loss: 2.840227 | lr 1.8000e-06 | norm: 0.1970 | dt: 29719.04ms | tok/sec: 17641.5 | mem: 70435MB | Time Left: 13 hrs : 37 mins
[TIMING] data: 7.8ms | fwd: 9418.1ms | loss: 5.0ms | bwd: 7032.6ms | clip: 0.0ms | optim: 24.2ms
step 1 | loss: 3.123918 | lr 3.6000e-06 | norm: 0.3054 | dt: 15186.85ms | tok/sec: 34522.5 | mem: 72099MB | Time Left: 06 hrs : 57 mins
[TIMING] data: 8.7ms | fwd: 9279.4ms | loss: 4.8ms | bwd: 5860.0ms | clip: 0.0ms | optim: 23.5ms
step 2 | loss: 2.830065 | lr 5.4000e-06 | norm: 0.2178 | dt: 13818.87ms | tok/sec: 37940.0 | mem: 72099MB | Time Left: 06 hrs : 19 mins
[TIMING] data: 12.5ms | fwd: 7140.5ms | loss: 4.9ms | bwd: 6626.1ms | clip: 0.0ms | optim: 23.5ms
step 3 | loss: 3.057204 | lr 7.2000e-06 | norm: 0.2737 | dt: 7780.31ms | tok/sec: 67386.5 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.0ms | fwd: 2385.2ms | loss: 4.9ms | bwd: 5342.7ms | clip: 0.0ms | optim: 23.5ms
step 4 | loss: 2.997180 | lr 9.0000e-06 | norm: 0.2248 | dt: 7853.07ms | tok/sec: 66762.2 | mem: 72099MB | Time Left: 03 hrs : 35 mins
[TIMING] data: 12.4ms | fwd: 2455.4ms | loss: 4.9ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.4ms
step 5 | loss: 3.072381 | lr 1.0800e-05 | norm: 0.2459 | dt: 7792.71ms | tok/sec: 67279.3 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.8ms | fwd: 2393.4ms | loss: 4.9ms | bwd: 5347.1ms | clip: 0.0ms | optim: 23.4ms
step 6 | loss: 2.878336 | lr 1.2600e-05 | norm: 0.2714 | dt: 7853.19ms | tok/sec: 66761.1 | mem: 72099MB | Time Left: 03 hrs : 35 mins
[TIMING] data: 12.7ms | fwd: 2454.9ms | loss: 4.9ms | bwd: 5345.6ms | clip: 0.0ms | optim: 23.5ms
step 7 | loss: 3.410254 | lr 1.4400e-05 | norm: 0.4509 | dt: 7792.49ms | tok/sec: 67281.2 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.0ms | fwd: 2395.6ms | loss: 4.9ms | bwd: 5344.4ms | clip: 0.0ms | optim: 23.7ms
step 8 | loss: 2.799212 | lr 1.6200e-05 | norm: 0.2074 | dt: 7791.99ms | tok/sec: 67285.5 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.5ms | fwd: 2391.7ms | loss: 4.9ms | bwd: 5348.5ms | clip: 0.0ms | optim: 23.5ms
step 9 | loss: 2.779159 | lr 1.8000e-05 | norm: 0.2054 | dt: 7797.31ms | tok/sec: 67239.6 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.8ms | fwd: 2397.4ms | loss: 4.9ms | bwd: 5347.8ms | clip: 0.0ms | optim: 23.5ms
step 10 | loss: 3.073190 | lr 1.9800e-05 | norm: 0.1787 | dt: 7799.03ms | tok/sec: 67224.8 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 12.7ms | fwd: 2396.3ms | loss: 4.9ms | bwd: 5350.8ms | clip: 0.0ms | optim: 23.5ms
step 11 | loss: 2.813490 | lr 2.1600e-05 | norm: 0.2286 | dt: 7795.71ms | tok/sec: 67253.4 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.0ms | fwd: 2393.2ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 12 | loss: 2.975980 | lr 2.3400e-05 | norm: 0.2559 | dt: 7797.30ms | tok/sec: 67239.7 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 11.9ms | fwd: 2394.7ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.5ms
step 13 | loss: 2.579046 | lr 2.5200e-05 | norm: 0.2544 | dt: 7801.83ms | tok/sec: 67200.7 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 13.2ms | fwd: 2396.1ms | loss: 4.9ms | bwd: 5351.4ms | clip: 0.0ms | optim: 23.7ms
step 14 | loss: 3.115849 | lr 2.7000e-05 | norm: 0.2300 | dt: 7805.95ms | tok/sec: 67165.2 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 13.9ms | fwd: 2402.4ms | loss: 4.9ms | bwd: 5350.1ms | clip: 0.0ms | optim: 23.5ms
step 15 | loss: 2.858879 | lr 2.8800e-05 | norm: 0.2256 | dt: 7811.34ms | tok/sec: 67118.8 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 13.9ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5349.0ms | clip: 0.0ms | optim: 23.5ms
step 16 | loss: 2.949294 | lr 3.0600e-05 | norm: 0.2114 | dt: 7885.15ms | tok/sec: 66490.5 | mem: 72099MB | Time Left: 03 hrs : 34 mins
[TIMING] data: 17.1ms | fwd: 2477.2ms | loss: 4.9ms | bwd: 5350.7ms | clip: 0.0ms | optim: 23.6ms
step 17 | loss: 2.993725 | lr 3.2400e-05 | norm: 0.2044 | dt: 7808.61ms | tok/sec: 67142.3 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 14.5ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5340.4ms | clip: 0.0ms | optim: 23.6ms
step 18 | loss: 2.746970 | lr 3.4200e-05 | norm: 0.2289 | dt: 7806.60ms | tok/sec: 67159.6 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 14.9ms | fwd: 2404.3ms | loss: 4.9ms | bwd: 5347.3ms | clip: 0.0ms | optim: 23.5ms
step 19 | loss: 2.770875 | lr 3.6000e-05 | norm: 0.1799 | dt: 7809.73ms | tok/sec: 67132.6 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 12.6ms | fwd: 2406.8ms | loss: 4.9ms | bwd: 5349.4ms | clip: 0.0ms | optim: 23.7ms
step 20 | loss: 2.935153 | lr 3.7800e-05 | norm: 0.1628 | dt: 7880.46ms | tok/sec: 66530.1 | mem: 72099MB | Time Left: 03 hrs : 34 mins
[TIMING] data: 14.9ms | fwd: 2476.7ms | loss: 4.9ms | bwd: 5346.8ms | clip: 0.0ms | optim: 23.5ms
step 21 | loss: 2.582499 | lr 3.9600e-05 | norm: 0.2081 | dt: 7870.75ms | tok/sec: 66612.2 | mem: 72099MB | Time Left: 03 hrs : 33 mins
[TIMING] data: 13.5ms | fwd: 2470.1ms | loss: 4.9ms | bwd: 5345.8ms | clip: 0.0ms | optim: 23.5ms
step 22 | loss: 3.091026 | lr 4.1400e-05 | norm: 0.2121 | dt: 7807.95ms | tok/sec: 67148.0 | mem: 72099MB | Time Left: 03 hrs : 31 mins
[TIMING] data: 13.5ms | fwd: 2408.0ms | loss: 4.9ms | bwd: 5345.1ms | clip: 0.0ms | optim: 23.5ms
step 23 | loss: 2.649695 | lr 4.3200e-05 | norm: 0.1683 | dt: 7818.67ms | tok/sec: 67055.9 | mem: 72099MB | Time Left: 03 hrs : 32 mins
[TIMING] data: 14.8ms | fwd: 2408.1ms | loss: 4.9ms | bwd: 5355.1ms | clip: 0.0ms | optim: 23.5ms
step 24 | loss: 2.540149 | lr 4.5000e-05 | norm: 0.1690 | dt: 7826.39ms | tok/sec: 66989.8 | mem: 72099MB | Time Left: 03 hrs : 32 mins