=== Llama Context Parallel Training Script (bluscriptCP) === [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) ================ Configuration (bluscriptCP) ================ d_model/layers : 1536 / 36 q/kv heads : 12 / 4 (G=3, hd=128) ffn_hidden : 4096 (SwiGLU) context_length : 16384 vocab_size : 50304 tying=0 B x T : 2 x 16384 world_size : 4 (dp=1 x cp=4) CP_ATTN_MODE : ring (load_balance=HeadTail) global_batch : 524288 (grad_accum=16) precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] max_lr/min_lr : 0.0006 / 6e-05 warmup/steps : 1 / 400 (cosine, no stable phase) checkpointing : true (ckpt_freq=200, dir=checkpoints_bluscriptcp) ============================================================= Initializing model on CUDA device 0... [YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386 [InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run6_step_750.ckpt (base step 750, loss 2.38385) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP Parameters: 1060624896 Parameters per GPU: 1060624896 max_steps: 400 warmup_steps: 1 [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=983236608 padded=983236608 W=4 shard=245809152 | 38 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 2813.06 MB (vs 11252.2 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.41 MB (vs 885.639 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] [ZeRO-1] DataParallel is DISABLED (both would double-reduce) [precision] bf16 params=217, all with fp32 master weights found 4 shards for split train found 1 shards for split val Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (run 2) [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base) Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_.ckpt (every 200 steps, keep 1) [re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp) Starting training... [CP ring P2P] OVERLAP: dedicated non-blocking stream validation loss: 3.0735 [CP bwd] overlap=ON step 0 | loss: 2.959782 | lr 1.8000e-06 | norm: 0.2170 | dt: 13217.28ms | tok/sec: 39666.9 | mem: 67649MB | Time Left: 01 hrs : 28 mins [TIMING] data: 5.6ms | fwd: 3460.7ms | loss: 9.3ms | bwd: 4971.1ms | clip: 0.0ms | optim: 25.3ms step 1 | loss: 2.841558 | lr 3.6000e-06 | norm: 0.2494 | dt: 7843.37ms | tok/sec: 66844.8 | mem: 68801MB | Time Left: 00 hrs : 52 mins [TIMING] data: 6.1ms | fwd: 3248.6ms | loss: 9.2ms | bwd: 4541.5ms | clip: 0.0ms | optim: 24.8ms step 2 | loss: 2.866521 | lr 5.4000e-06 | norm: 0.2064 | dt: 7894.67ms | tok/sec: 66410.4 | mem: 68769MB | Time Left: 00 hrs : 52 mins [TIMING] data: 6.8ms | fwd: 2880.5ms | loss: 9.2ms | bwd: 4960.1ms | clip: 0.0ms | optim: 24.8ms step 3 | loss: 2.896216 | lr 7.2000e-06 | norm: 0.1957 | dt: 6910.71ms | tok/sec: 75866.0 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 6.1ms | fwd: 2332.0ms | loss: 9.2ms | bwd: 4525.5ms | clip: 0.0ms | optim: 24.7ms step 4 | loss: 3.037557 | lr 9.0000e-06 | norm: 0.1734 | dt: 6907.60ms | tok/sec: 75900.2 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 5.9ms | fwd: 2332.0ms | loss: 9.2ms | bwd: 4522.5ms | clip: 0.0ms | optim: 24.8ms step 5 | loss: 2.777790 | lr 1.0800e-05 | norm: 0.1705 | dt: 6909.71ms | tok/sec: 75877.0 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 6.1ms | fwd: 2331.2ms | loss: 9.2ms | bwd: 4525.0ms | clip: 0.0ms | optim: 24.9ms step 6 | loss: 3.049557 | lr 1.2600e-05 | norm: 0.1654 | dt: 6949.15ms | tok/sec: 75446.3 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 6.2ms | fwd: 2370.9ms | loss: 9.2ms | bwd: 4524.8ms | clip: 0.0ms | optim: 24.8ms step 7 | loss: 2.887465 | lr 1.4400e-05 | norm: 0.1839 | dt: 6909.60ms | tok/sec: 75878.2 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 6.1ms | fwd: 2331.9ms | loss: 9.2ms | bwd: 4524.6ms | clip: 0.0ms | optim: 24.7ms step 8 | loss: 2.966998 | lr 1.6200e-05 | norm: 0.1933 | dt: 6912.11ms | tok/sec: 75850.6 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 6.1ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4524.2ms | clip: 0.0ms | optim: 24.8ms step 9 | loss: 2.981712 | lr 1.8000e-05 | norm: 0.2251 | dt: 6950.70ms | tok/sec: 75429.5 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 6.2ms | fwd: 2373.7ms | loss: 9.2ms | bwd: 4523.5ms | clip: 0.0ms | optim: 24.8ms step 10 | loss: 2.824136 | lr 1.9800e-05 | norm: 0.2115 | dt: 6911.00ms | tok/sec: 75862.8 | mem: 68769MB | Time Left: 00 hrs : 44 mins [TIMING] data: 6.1ms | fwd: 2332.9ms | loss: 9.2ms | bwd: 4524.8ms | clip: 0.0ms | optim: 24.8ms step 11 | loss: 2.804077 | lr 2.1600e-05 | norm: 0.1839 | dt: 6909.19ms | tok/sec: 75882.7 | mem: 68769MB | Time Left: 00 hrs : 44 mins [TIMING] data: 6.1ms | fwd: 2332.8ms | loss: 9.2ms | bwd: 4523.0ms | clip: 0.0ms | optim: 24.8ms step 12 | loss: 2.857808 | lr 2.3400e-05 | norm: 0.1713 | dt: 6974.93ms | tok/sec: 75167.4 | mem: 68769MB | Time Left: 00 hrs : 45 mins [TIMING] data: 6.2ms | fwd: 2398.8ms | loss: 9.2ms | bwd: 4522.7ms | clip: 0.0ms | optim: 24.8ms step 13 | loss: 2.941759 | lr 2.5200e-05 | norm: 0.1709 | dt: 6913.77ms | tok/sec: 75832.4 | mem: 68769MB | Time Left: 00 hrs : 44 mins [TIMING] data: 6.1ms | fwd: 2332.9ms | loss: 9.2ms | bwd: 4527.5ms | clip: 0.0ms | optim: 24.8ms step 14 | loss: 2.651574 | lr 2.7000e-05 | norm: 0.1597 | dt: 6911.47ms | tok/sec: 75857.6 | mem: 68769MB | Time Left: 00 hrs : 44 mins [TIMING] data: 6.1ms | fwd: 2333.3ms | loss: 9.2ms | bwd: 4524.8ms | clip: 0.0ms | optim: 24.8ms step 15 | loss: 2.688379 | lr 2.8800e-05 | norm: 0.1647 | dt: 6921.69ms | tok/sec: 75745.6 | mem: 68769MB | Time Left: 00 hrs : 44 mins [TIMING] data: 6.1ms | fwd: 2343.7ms | loss: 9.2ms | bwd: 4524.7ms | clip: 0.0ms | optim: 24.7ms step 16 | loss: 2.883613 | lr 3.0600e-05 | norm: 0.1662 | dt: 6912.50ms | tok/sec: 75846.3 | mem: 68769MB | Time Left: 00 hrs : 44 mins [TIMING] data: 5.9ms | fwd: 2333.5ms | loss: 9.2ms | bwd: 4525.7ms | clip: 0.0ms | optim: 24.9ms step 17 | loss: 2.803720 | lr 3.2400e-05 | norm: 0.1663 | dt: 6910.40ms | tok/sec: 75869.5 | mem: 68769MB | Time Left: 00 hrs : 44 mins [TIMING] data: 6.1ms | fwd: 2334.4ms | loss: 9.2ms | bwd: 4522.7ms | clip: 0.0ms | optim: 24.7ms step 18 | loss: 3.140740 | lr 3.4200e-05 | norm: 0.1634 | dt: 6909.18ms | tok/sec: 75882.8 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.2ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4521.0ms | clip: 0.0ms | optim: 24.8ms step 19 | loss: 2.735036 | lr 3.6000e-05 | norm: 0.1510 | dt: 6913.23ms | tok/sec: 75838.4 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.1ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4525.9ms | clip: 0.0ms | optim: 24.8ms step 20 | loss: 2.941762 | lr 3.7800e-05 | norm: 0.1707 | dt: 6914.10ms | tok/sec: 75828.8 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.1ms | fwd: 2333.4ms | loss: 9.2ms | bwd: 4527.3ms | clip: 0.0ms | optim: 24.7ms step 21 | loss: 2.826214 | lr 3.9600e-05 | norm: 0.1481 | dt: 6957.96ms | tok/sec: 75350.8 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.0ms | fwd: 2374.8ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.7ms step 22 | loss: 2.955554 | lr 4.1400e-05 | norm: 0.1493 | dt: 6911.95ms | tok/sec: 75852.4 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.1ms | fwd: 2333.8ms | loss: 9.2ms | bwd: 4524.8ms | clip: 0.0ms | optim: 24.8ms step 23 | loss: 2.986146 | lr 4.3200e-05 | norm: 0.1580 | dt: 6915.53ms | tok/sec: 75813.1 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.1ms | fwd: 2333.8ms | loss: 9.2ms | bwd: 4528.4ms | clip: 0.0ms | optim: 24.8ms step 24 | loss: 2.776773 | lr 4.5000e-05 | norm: 0.1452 | dt: 6914.73ms | tok/sec: 75822.0 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.2ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4526.8ms | clip: 0.0ms | optim: 24.8ms step 25 | loss: 2.817302 | lr 4.6800e-05 | norm: 0.1577 | dt: 6967.81ms | tok/sec: 75244.3 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 5.9ms | fwd: 2389.1ms | loss: 9.2ms | bwd: 4525.6ms | clip: 0.0ms | optim: 24.7ms step 26 | loss: 2.826208 | lr 4.8600e-05 | norm: 0.1446 | dt: 6911.59ms | tok/sec: 75856.3 | mem: 68769MB | Time Left: 00 hrs : 43 mins [TIMING] data: 6.2ms | fwd: 2333.7ms | loss: 9.2ms | bwd: 4524.5ms | clip: 0.0ms | optim: 24.8ms step 27 | loss: 2.871861 | lr 5.0400e-05 | norm: 0.1372 | dt: 6914.36ms | tok/sec: 75826.0 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 6.1ms | fwd: 2333.7ms | loss: 9.2ms | bwd: 4527.3ms | clip: 0.0ms | optim: 24.8ms step 28 | loss: 2.890775 | lr 5.2200e-05 | norm: 0.1414 | dt: 6912.42ms | tok/sec: 75847.3 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 6.0ms | fwd: 2334.3ms | loss: 9.2ms | bwd: 4524.9ms | clip: 0.0ms | optim: 24.7ms step 29 | loss: 2.860940 | lr 5.4000e-05 | norm: 0.1579 | dt: 6935.54ms | tok/sec: 75594.5 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 6.2ms | fwd: 2353.5ms | loss: 9.2ms | bwd: 4528.6ms | clip: 0.0ms | optim: 24.8ms step 30 | loss: 2.799390 | lr 5.5800e-05 | norm: 0.1417 | dt: 6914.75ms | tok/sec: 75821.7 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 5.9ms | fwd: 2334.4ms | loss: 9.2ms | bwd: 4527.1ms | clip: 0.0ms | optim: 24.8ms step 31 | loss: 2.900790 | lr 5.7600e-05 | norm: 0.1415 | dt: 6914.29ms | tok/sec: 75826.8 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 6.0ms | fwd: 2333.7ms | loss: 9.2ms | bwd: 4527.4ms | clip: 0.0ms | optim: 24.7ms step 32 | loss: 2.912354 | lr 5.9400e-05 | norm: 0.1412 | dt: 6939.43ms | tok/sec: 75552.0 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 6.1ms | fwd: 2360.8ms | loss: 9.2ms | bwd: 4525.5ms | clip: 0.0ms | optim: 24.8ms step 33 | loss: 2.901439 | lr 6.1200e-05 | norm: 0.1402 | dt: 6911.29ms | tok/sec: 75859.6 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 6.1ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4523.9ms | clip: 0.0ms | optim: 24.8ms step 34 | loss: 2.856484 | lr 6.3000e-05 | norm: 0.1370 | dt: 6917.73ms | tok/sec: 75789.0 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 6.2ms | fwd: 2333.8ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 35 | loss: 2.777911 | lr 6.4800e-05 | norm: 0.1533 | dt: 6914.66ms | tok/sec: 75822.6 | mem: 68769MB | Time Left: 00 hrs : 42 mins [TIMING] data: 5.9ms | fwd: 2335.8ms | loss: 9.2ms | bwd: 4525.5ms | clip: 0.0ms | optim: 24.7ms step 36 | loss: 2.943847 | lr 6.6600e-05 | norm: 0.1442 | dt: 6912.88ms | tok/sec: 75842.2 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.0ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4525.6ms | clip: 0.0ms | optim: 24.8ms step 37 | loss: 2.913118 | lr 6.8400e-05 | norm: 0.1476 | dt: 6914.30ms | tok/sec: 75826.7 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.0ms | fwd: 2333.8ms | loss: 9.2ms | bwd: 4527.2ms | clip: 0.0ms | optim: 24.8ms step 38 | loss: 2.845747 | lr 7.0200e-05 | norm: 0.1523 | dt: 6913.90ms | tok/sec: 75831.0 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.1ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4526.4ms | clip: 0.0ms | optim: 24.8ms step 39 | loss: 2.891100 | lr 7.2000e-05 | norm: 0.1435 | dt: 6952.14ms | tok/sec: 75413.9 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.2ms | fwd: 2375.0ms | loss: 9.2ms | bwd: 4523.7ms | clip: 0.0ms | optim: 24.8ms step 40 | loss: 2.876539 | lr 7.3800e-05 | norm: 0.1537 | dt: 6913.42ms | tok/sec: 75836.3 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.2ms | fwd: 2333.5ms | loss: 9.2ms | bwd: 4526.5ms | clip: 0.0ms | optim: 24.8ms step 41 | loss: 2.802080 | lr 7.5600e-05 | norm: 0.1606 | dt: 6915.43ms | tok/sec: 75814.2 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.0ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4527.2ms | clip: 0.0ms | optim: 24.8ms step 42 | loss: 2.842204 | lr 7.7400e-05 | norm: 0.1425 | dt: 6949.68ms | tok/sec: 75440.6 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.2ms | fwd: 2369.9ms | loss: 9.2ms | bwd: 4526.3ms | clip: 0.0ms | optim: 24.8ms step 43 | loss: 2.802278 | lr 7.9200e-05 | norm: 0.1488 | dt: 6909.41ms | tok/sec: 75880.3 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.2ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4521.9ms | clip: 0.0ms | optim: 24.8ms step 44 | loss: 2.933663 | lr 8.1000e-05 | norm: 0.1530 | dt: 6910.59ms | tok/sec: 75867.4 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.0ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4523.3ms | clip: 0.0ms | optim: 24.8ms step 45 | loss: 2.814606 | lr 8.2800e-05 | norm: 0.1617 | dt: 6964.60ms | tok/sec: 75278.9 | mem: 68769MB | Time Left: 00 hrs : 41 mins [TIMING] data: 6.0ms | fwd: 2387.0ms | loss: 9.2ms | bwd: 4524.4ms | clip: 0.0ms | optim: 24.8ms step 46 | loss: 2.768441 | lr 8.4600e-05 | norm: 0.1569 | dt: 6911.24ms | tok/sec: 75860.2 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 6.1ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4523.9ms | clip: 0.0ms | optim: 24.8ms step 47 | loss: 3.054946 | lr 8.6400e-05 | norm: 0.1709 | dt: 6913.30ms | tok/sec: 75837.6 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 6.1ms | fwd: 2334.3ms | loss: 9.2ms | bwd: 4525.7ms | clip: 0.0ms | optim: 24.8ms step 48 | loss: 2.960767 | lr 8.8200e-05 | norm: 0.1474 | dt: 6924.24ms | tok/sec: 75717.8 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 5.9ms | fwd: 2344.5ms | loss: 9.2ms | bwd: 4526.5ms | clip: 0.0ms | optim: 24.8ms step 49 | loss: 2.920072 | lr 9.0000e-05 | norm: 0.1500 | dt: 6913.84ms | tok/sec: 75831.6 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 5.9ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4526.6ms | clip: 0.0ms | optim: 24.8ms step 50 | loss: 2.828606 | lr 9.1800e-05 | norm: 0.1548 | dt: 6913.66ms | tok/sec: 75833.6 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 6.2ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4526.3ms | clip: 0.0ms | optim: 24.8ms step 51 | loss: 2.937525 | lr 9.3600e-05 | norm: 0.1563 | dt: 6910.88ms | tok/sec: 75864.2 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 6.0ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4523.1ms | clip: 0.0ms | optim: 24.8ms step 52 | loss: 3.043337 | lr 9.5400e-05 | norm: 0.1640 | dt: 6912.23ms | tok/sec: 75849.3 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 6.3ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4524.6ms | clip: 0.0ms | optim: 24.8ms step 53 | loss: 2.879141 | lr 9.7200e-05 | norm: 0.1790 | dt: 6943.01ms | tok/sec: 75513.1 | mem: 68769MB | Time Left: 00 hrs : 40 mins [TIMING] data: 6.2ms | fwd: 2363.4ms | loss: 9.2ms | bwd: 4526.1ms | clip: 0.0ms | optim: 24.8ms step 54 | loss: 2.972244 | lr 9.9000e-05 | norm: 0.1513 | dt: 6915.05ms | tok/sec: 75818.4 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 6.2ms | fwd: 2333.5ms | loss: 9.2ms | bwd: 4528.1ms | clip: 0.0ms | optim: 24.8ms step 55 | loss: 2.999154 | lr 1.0080e-04 | norm: 0.1672 | dt: 6912.32ms | tok/sec: 75848.3 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 6.2ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4524.8ms | clip: 0.0ms | optim: 24.8ms step 56 | loss: 2.949492 | lr 1.0260e-04 | norm: 0.1612 | dt: 6928.82ms | tok/sec: 75667.7 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 6.1ms | fwd: 2349.8ms | loss: 9.2ms | bwd: 4525.7ms | clip: 0.0ms | optim: 24.8ms step 57 | loss: 2.975551 | lr 1.0440e-04 | norm: 0.1886 | dt: 6915.52ms | tok/sec: 75813.3 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 6.0ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4527.4ms | clip: 0.0ms | optim: 24.8ms step 58 | loss: 2.758769 | lr 1.0620e-04 | norm: 0.1653 | dt: 6921.28ms | tok/sec: 75750.2 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 5.9ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4533.1ms | clip: 0.0ms | optim: 24.8ms step 59 | loss: 2.958816 | lr 1.0800e-04 | norm: 0.1690 | dt: 6954.38ms | tok/sec: 75389.6 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 5.9ms | fwd: 2372.9ms | loss: 9.2ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.8ms step 60 | loss: 2.909616 | lr 1.0980e-04 | norm: 0.1590 | dt: 6918.65ms | tok/sec: 75779.0 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 6.0ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4530.9ms | clip: 0.0ms | optim: 24.8ms step 61 | loss: 2.923694 | lr 1.1160e-04 | norm: 0.1605 | dt: 6918.57ms | tok/sec: 75779.9 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 6.0ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.8ms step 62 | loss: 2.899854 | lr 1.1340e-04 | norm: 0.1653 | dt: 6955.67ms | tok/sec: 75375.6 | mem: 68769MB | Time Left: 00 hrs : 39 mins [TIMING] data: 6.2ms | fwd: 2372.6ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 63 | loss: 2.940152 | lr 1.1520e-04 | norm: 0.1725 | dt: 6918.17ms | tok/sec: 75784.2 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 5.9ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 64 | loss: 2.705931 | lr 1.1700e-04 | norm: 0.1522 | dt: 6918.01ms | tok/sec: 75786.0 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 5.9ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.7ms step 65 | loss: 2.642195 | lr 1.1880e-04 | norm: 0.1600 | dt: 6978.75ms | tok/sec: 75126.4 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 5.9ms | fwd: 2397.2ms | loss: 9.2ms | bwd: 4528.5ms | clip: 0.0ms | optim: 24.8ms step 66 | loss: 2.823886 | lr 1.2060e-04 | norm: 0.1518 | dt: 6917.51ms | tok/sec: 75791.4 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 5.9ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.7ms step 67 | loss: 2.826914 | lr 1.2240e-04 | norm: 0.1538 | dt: 6945.06ms | tok/sec: 75490.8 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 5.9ms | fwd: 2361.8ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 68 | loss: 2.882534 | lr 1.2420e-04 | norm: 0.1493 | dt: 6929.94ms | tok/sec: 75655.5 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 5.8ms | fwd: 2347.7ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.8ms step 69 | loss: 2.752670 | lr 1.2600e-04 | norm: 0.1583 | dt: 6917.82ms | tok/sec: 75788.0 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 6.1ms | fwd: 2334.4ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 70 | loss: 3.066311 | lr 1.2780e-04 | norm: 0.1618 | dt: 6919.19ms | tok/sec: 75773.0 | mem: 68769MB | Time Left: 00 hrs : 38 mins [TIMING] data: 6.1ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4531.0ms | clip: 0.0ms | optim: 24.8ms step 71 | loss: 3.005011 | lr 1.2960e-04 | norm: 0.1683 | dt: 6918.20ms | tok/sec: 75783.8 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.2ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 72 | loss: 3.025673 | lr 1.3140e-04 | norm: 0.1707 | dt: 6916.95ms | tok/sec: 75797.5 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.4ms | fwd: 2333.6ms | loss: 9.2ms | bwd: 4529.7ms | clip: 0.0ms | optim: 24.8ms step 73 | loss: 2.833005 | lr 1.3320e-04 | norm: 0.1619 | dt: 6917.16ms | tok/sec: 75795.2 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.0ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4528.9ms | clip: 0.0ms | optim: 24.8ms step 74 | loss: 2.778100 | lr 1.3500e-04 | norm: 0.1847 | dt: 6923.49ms | tok/sec: 75726.0 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.1ms | fwd: 2340.7ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 75 | loss: 2.944230 | lr 1.3680e-04 | norm: 0.1657 | dt: 6918.59ms | tok/sec: 75779.6 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.0ms | fwd: 2335.4ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.8ms step 76 | loss: 2.974186 | lr 1.3860e-04 | norm: 0.1746 | dt: 6918.37ms | tok/sec: 75782.0 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.0ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.7ms step 77 | loss: 2.984059 | lr 1.4040e-04 | norm: 0.1683 | dt: 6945.70ms | tok/sec: 75483.8 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.0ms | fwd: 2362.1ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 78 | loss: 2.969082 | lr 1.4220e-04 | norm: 0.1680 | dt: 6920.86ms | tok/sec: 75754.7 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.0ms | fwd: 2335.8ms | loss: 9.2ms | bwd: 4531.9ms | clip: 0.0ms | optim: 24.8ms step 79 | loss: 3.023555 | lr 1.4400e-04 | norm: 0.1726 | dt: 6919.64ms | tok/sec: 75768.1 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 6.0ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4531.5ms | clip: 0.0ms | optim: 24.8ms step 80 | loss: 2.947484 | lr 1.4580e-04 | norm: 0.1611 | dt: 6952.26ms | tok/sec: 75412.6 | mem: 68769MB | Time Left: 00 hrs : 37 mins [TIMING] data: 5.9ms | fwd: 2370.5ms | loss: 9.2ms | bwd: 4528.6ms | clip: 0.0ms | optim: 24.8ms step 81 | loss: 2.886832 | lr 1.4760e-04 | norm: 0.1700 | dt: 6919.14ms | tok/sec: 75773.6 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 6.0ms | fwd: 2335.2ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.8ms step 82 | loss: 2.995768 | lr 1.4940e-04 | norm: 0.1762 | dt: 6917.09ms | tok/sec: 75796.0 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 5.9ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 83 | loss: 2.865708 | lr 1.5120e-04 | norm: 0.1665 | dt: 6985.96ms | tok/sec: 75048.8 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 6.1ms | fwd: 2398.7ms | loss: 9.2ms | bwd: 4534.0ms | clip: 0.0ms | optim: 24.8ms step 84 | loss: 2.857377 | lr 1.5300e-04 | norm: 0.1668 | dt: 6917.27ms | tok/sec: 75794.0 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 5.9ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.8ms step 85 | loss: 3.017304 | lr 1.5480e-04 | norm: 0.1694 | dt: 6919.10ms | tok/sec: 75774.0 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 5.9ms | fwd: 2335.3ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.8ms step 86 | loss: 2.914292 | lr 1.5660e-04 | norm: 0.1632 | dt: 6922.13ms | tok/sec: 75740.8 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 6.0ms | fwd: 2339.1ms | loss: 9.2ms | bwd: 4529.8ms | clip: 0.0ms | optim: 24.8ms step 87 | loss: 2.925972 | lr 1.5840e-04 | norm: 0.1765 | dt: 6919.53ms | tok/sec: 75769.3 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 5.9ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4531.0ms | clip: 0.0ms | optim: 25.3ms step 88 | loss: 2.859728 | lr 1.6020e-04 | norm: 0.1581 | dt: 6918.17ms | tok/sec: 75784.2 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 6.0ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4529.8ms | clip: 0.0ms | optim: 24.8ms step 89 | loss: 2.978377 | lr 1.6200e-04 | norm: 0.1840 | dt: 6984.27ms | tok/sec: 75067.0 | mem: 68769MB | Time Left: 00 hrs : 36 mins [TIMING] data: 5.9ms | fwd: 2400.6ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 90 | loss: 2.742138 | lr 1.6380e-04 | norm: 0.1787 | dt: 6916.41ms | tok/sec: 75803.5 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 5.9ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.8ms step 91 | loss: 2.756475 | lr 1.6560e-04 | norm: 0.1898 | dt: 6918.98ms | tok/sec: 75775.3 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 6.0ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.8ms step 92 | loss: 2.970927 | lr 1.6740e-04 | norm: 0.1762 | dt: 6983.03ms | tok/sec: 75080.3 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 6.0ms | fwd: 2399.2ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.7ms step 93 | loss: 3.033848 | lr 1.6920e-04 | norm: 0.2136 | dt: 6917.89ms | tok/sec: 75787.3 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 6.0ms | fwd: 2335.3ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 94 | loss: 2.781953 | lr 1.7100e-04 | norm: 0.1848 | dt: 6918.80ms | tok/sec: 75777.3 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 6.0ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.9ms | clip: 0.0ms | optim: 24.7ms step 95 | loss: 2.851102 | lr 1.7280e-04 | norm: 0.1948 | dt: 6961.22ms | tok/sec: 75315.5 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 6.0ms | fwd: 2379.1ms | loss: 9.2ms | bwd: 4529.0ms | clip: 0.0ms | optim: 24.8ms step 96 | loss: 3.002559 | lr 1.7460e-04 | norm: 0.1903 | dt: 6991.23ms | tok/sec: 74992.3 | mem: 68769MB | Time Left: 00 hrs : 35 mins [TIMING] data: 5.9ms | fwd: 2335.6ms | loss: 9.2ms | bwd: 4602.5ms | clip: 0.0ms | optim: 24.8ms step 97 | loss: 2.901280 | lr 1.7640e-04 | norm: 0.1878 | dt: 6919.65ms | tok/sec: 75768.0 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.2ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4531.4ms | clip: 0.0ms | optim: 24.7ms step 98 | loss: 3.076037 | lr 1.7820e-04 | norm: 0.1772 | dt: 6916.89ms | tok/sec: 75798.2 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.2ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.8ms step 99 | loss: 2.880864 | lr 1.8000e-04 | norm: 0.1730 | dt: 6943.64ms | tok/sec: 75506.2 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.2ms | fwd: 2357.9ms | loss: 9.2ms | bwd: 4532.2ms | clip: 0.0ms | optim: 24.8ms step 100 | loss: 2.880449 | lr 1.8000e-04 | norm: 0.1690 | dt: 6921.82ms | tok/sec: 75744.2 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.2ms | fwd: 2336.6ms | loss: 9.2ms | bwd: 4531.7ms | clip: 0.0ms | optim: 24.7ms step 101 | loss: 2.894127 | lr 1.8000e-04 | norm: 0.1600 | dt: 6951.32ms | tok/sec: 75422.8 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.2ms | fwd: 2369.8ms | loss: 9.2ms | bwd: 4528.0ms | clip: 0.0ms | optim: 24.8ms step 102 | loss: 2.868625 | lr 1.7999e-04 | norm: 0.1680 | dt: 6917.90ms | tok/sec: 75787.1 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.3ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.8ms step 103 | loss: 2.999022 | lr 1.7997e-04 | norm: 0.1841 | dt: 6920.82ms | tok/sec: 75755.2 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.4ms | fwd: 2337.1ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 104 | loss: 2.936193 | lr 1.7995e-04 | norm: 0.1592 | dt: 6916.43ms | tok/sec: 75803.2 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.2ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4528.9ms | clip: 0.0ms | optim: 24.7ms step 105 | loss: 3.028981 | lr 1.7992e-04 | norm: 0.2138 | dt: 6948.47ms | tok/sec: 75453.7 | mem: 68769MB | Time Left: 00 hrs : 34 mins [TIMING] data: 6.1ms | fwd: 2365.0ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 106 | loss: 2.918709 | lr 1.7988e-04 | norm: 0.1758 | dt: 6916.73ms | tok/sec: 75800.0 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.9ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4527.8ms | clip: 0.0ms | optim: 24.8ms step 107 | loss: 2.979311 | lr 1.7984e-04 | norm: 0.1712 | dt: 6918.51ms | tok/sec: 75780.5 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.2ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.8ms step 108 | loss: 2.794305 | lr 1.7979e-04 | norm: 0.1873 | dt: 6945.43ms | tok/sec: 75486.7 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.0ms | fwd: 2362.1ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.7ms step 109 | loss: 2.891953 | lr 1.7973e-04 | norm: 0.1729 | dt: 6919.04ms | tok/sec: 75774.7 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.1ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.8ms step 110 | loss: 3.058260 | lr 1.7967e-04 | norm: 0.1777 | dt: 6918.19ms | tok/sec: 75783.9 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.0ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4529.6ms | clip: 0.0ms | optim: 24.8ms step 111 | loss: 2.811223 | lr 1.7960e-04 | norm: 0.1792 | dt: 6980.32ms | tok/sec: 75109.5 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.0ms | fwd: 2395.3ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.8ms step 112 | loss: 3.014529 | lr 1.7953e-04 | norm: 0.1788 | dt: 6918.03ms | tok/sec: 75785.8 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.2ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4529.8ms | clip: 0.0ms | optim: 24.8ms step 113 | loss: 2.790852 | lr 1.7944e-04 | norm: 0.1547 | dt: 6920.55ms | tok/sec: 75758.2 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.2ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4531.9ms | clip: 0.0ms | optim: 24.8ms step 114 | loss: 2.999370 | lr 1.7936e-04 | norm: 0.1647 | dt: 6933.75ms | tok/sec: 75613.9 | mem: 68769MB | Time Left: 00 hrs : 33 mins [TIMING] data: 6.4ms | fwd: 2350.7ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.7ms step 115 | loss: 2.883359 | lr 1.7926e-04 | norm: 0.1574 | dt: 6918.34ms | tok/sec: 75782.4 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.2ms | fwd: 2335.3ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 116 | loss: 2.806687 | lr 1.7916e-04 | norm: 0.1618 | dt: 6918.92ms | tok/sec: 75776.0 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.1ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.7ms step 117 | loss: 2.787320 | lr 1.7905e-04 | norm: 0.1716 | dt: 6962.72ms | tok/sec: 75299.3 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.1ms | fwd: 2378.7ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 118 | loss: 2.820269 | lr 1.7894e-04 | norm: 0.1618 | dt: 6918.80ms | tok/sec: 75777.3 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.1ms | fwd: 2335.2ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 119 | loss: 2.913639 | lr 1.7882e-04 | norm: 0.1648 | dt: 6917.53ms | tok/sec: 75791.2 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.0ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 120 | loss: 2.803736 | lr 1.7869e-04 | norm: 0.1623 | dt: 6972.63ms | tok/sec: 75192.3 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.1ms | fwd: 2392.2ms | loss: 9.2ms | bwd: 4527.1ms | clip: 0.0ms | optim: 24.8ms step 121 | loss: 2.865543 | lr 1.7856e-04 | norm: 0.1677 | dt: 6920.24ms | tok/sec: 75761.5 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.1ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4531.9ms | clip: 0.0ms | optim: 24.8ms step 122 | loss: 2.974976 | lr 1.7841e-04 | norm: 0.1804 | dt: 6917.59ms | tok/sec: 75790.6 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.3ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.8ms step 123 | loss: 2.852792 | lr 1.7827e-04 | norm: 0.1625 | dt: 6983.44ms | tok/sec: 75075.9 | mem: 68769MB | Time Left: 00 hrs : 32 mins [TIMING] data: 6.4ms | fwd: 2400.5ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 124 | loss: 3.148803 | lr 1.7812e-04 | norm: 0.1677 | dt: 6919.14ms | tok/sec: 75773.5 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.2ms | fwd: 2335.5ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.7ms step 125 | loss: 3.007370 | lr 1.7796e-04 | norm: 0.1630 | dt: 6919.61ms | tok/sec: 75768.4 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.3ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4531.2ms | clip: 0.0ms | optim: 24.8ms step 126 | loss: 2.854838 | lr 1.7779e-04 | norm: 0.1634 | dt: 6945.11ms | tok/sec: 75490.3 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.0ms | fwd: 2361.8ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.7ms step 127 | loss: 2.807126 | lr 1.7762e-04 | norm: 0.1625 | dt: 6917.59ms | tok/sec: 75790.6 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.3ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 128 | loss: 3.031279 | lr 1.7744e-04 | norm: 0.1805 | dt: 6918.93ms | tok/sec: 75775.9 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.2ms | fwd: 2335.3ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.7ms step 129 | loss: 2.893268 | lr 1.7725e-04 | norm: 0.1895 | dt: 6942.51ms | tok/sec: 75518.5 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.2ms | fwd: 2361.1ms | loss: 9.2ms | bwd: 4527.9ms | clip: 0.0ms | optim: 24.8ms step 130 | loss: 2.996319 | lr 1.7706e-04 | norm: 0.1828 | dt: 6916.23ms | tok/sec: 75805.5 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.3ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4527.9ms | clip: 0.0ms | optim: 24.8ms step 131 | loss: 2.889647 | lr 1.7687e-04 | norm: 0.1690 | dt: 6916.84ms | tok/sec: 75798.8 | mem: 68769MB | Time Left: 00 hrs : 31 mins [TIMING] data: 6.0ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 132 | loss: 2.856715 | lr 1.7666e-04 | norm: 0.1708 | dt: 6938.47ms | tok/sec: 75562.5 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.4ms | fwd: 2352.9ms | loss: 9.2ms | bwd: 4531.9ms | clip: 0.0ms | optim: 24.7ms step 133 | loss: 2.898498 | lr 1.7645e-04 | norm: 0.1555 | dt: 6918.04ms | tok/sec: 75785.6 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.3ms | fwd: 2334.3ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 134 | loss: 2.887420 | lr 1.7624e-04 | norm: 0.1672 | dt: 6921.90ms | tok/sec: 75743.4 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.0ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4533.1ms | clip: 0.0ms | optim: 24.7ms step 135 | loss: 2.839339 | lr 1.7601e-04 | norm: 0.1764 | dt: 6966.02ms | tok/sec: 75263.6 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.1ms | fwd: 2383.5ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.7ms step 136 | loss: 2.888696 | lr 1.7579e-04 | norm: 0.2230 | dt: 6919.18ms | tok/sec: 75773.1 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.3ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.9ms | clip: 0.0ms | optim: 24.8ms step 137 | loss: 2.974666 | lr 1.7555e-04 | norm: 0.1774 | dt: 6918.75ms | tok/sec: 75777.8 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.4ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4531.0ms | clip: 0.0ms | optim: 24.7ms step 138 | loss: 2.944411 | lr 1.7531e-04 | norm: 0.1732 | dt: 6967.12ms | tok/sec: 75251.7 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.2ms | fwd: 2381.9ms | loss: 9.2ms | bwd: 4531.7ms | clip: 0.0ms | optim: 24.8ms step 139 | loss: 2.849583 | lr 1.7507e-04 | norm: 0.1854 | dt: 6918.97ms | tok/sec: 75775.5 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.0ms | fwd: 2335.2ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.8ms step 140 | loss: 2.902894 | lr 1.7481e-04 | norm: 0.1681 | dt: 6925.71ms | tok/sec: 75701.7 | mem: 68769MB | Time Left: 00 hrs : 30 mins [TIMING] data: 6.1ms | fwd: 2336.9ms | loss: 9.2ms | bwd: 4535.3ms | clip: 0.0ms | optim: 24.8ms step 141 | loss: 2.873610 | lr 1.7455e-04 | norm: 0.1570 | dt: 6926.42ms | tok/sec: 75694.0 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.1ms | fwd: 2343.0ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 142 | loss: 2.808603 | lr 1.7429e-04 | norm: 0.1660 | dt: 6924.95ms | tok/sec: 75710.1 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.4ms | fwd: 2342.8ms | loss: 9.2ms | bwd: 4528.5ms | clip: 0.0ms | optim: 24.8ms step 143 | loss: 2.958304 | lr 1.7402e-04 | norm: 0.1653 | dt: 6917.58ms | tok/sec: 75790.7 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.3ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4529.6ms | clip: 0.0ms | optim: 24.7ms step 144 | loss: 2.891419 | lr 1.7374e-04 | norm: 0.1798 | dt: 6974.44ms | tok/sec: 75172.8 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.1ms | fwd: 2391.3ms | loss: 9.2ms | bwd: 4529.8ms | clip: 0.0ms | optim: 24.7ms step 145 | loss: 2.729595 | lr 1.7346e-04 | norm: 0.1639 | dt: 6917.27ms | tok/sec: 75794.1 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.3ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 146 | loss: 3.058663 | lr 1.7317e-04 | norm: 0.1817 | dt: 6918.23ms | tok/sec: 75783.5 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.1ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 147 | loss: 2.867789 | lr 1.7288e-04 | norm: 0.1725 | dt: 6967.16ms | tok/sec: 75251.3 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.2ms | fwd: 2383.9ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.7ms step 148 | loss: 2.762562 | lr 1.7258e-04 | norm: 0.1566 | dt: 6916.67ms | tok/sec: 75800.6 | mem: 68769MB | Time Left: 00 hrs : 29 mins [TIMING] data: 6.1ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.8ms step 149 | loss: 2.906806 | lr 1.7227e-04 | norm: 0.1748 | dt: 6918.10ms | tok/sec: 75785.0 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.0ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.8ms step 150 | loss: 2.911504 | lr 1.7196e-04 | norm: 0.1778 | dt: 6954.78ms | tok/sec: 75385.2 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.2ms | fwd: 2372.4ms | loss: 9.2ms | bwd: 4529.0ms | clip: 0.0ms | optim: 24.7ms step 151 | loss: 2.854765 | lr 1.7164e-04 | norm: 0.1704 | dt: 6915.85ms | tok/sec: 75809.7 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.1ms | fwd: 2334.3ms | loss: 9.2ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.8ms step 152 | loss: 2.684196 | lr 1.7132e-04 | norm: 0.1635 | dt: 6917.76ms | tok/sec: 75788.7 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 5.9ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 153 | loss: 2.956713 | lr 1.7099e-04 | norm: 0.1760 | dt: 6945.78ms | tok/sec: 75483.0 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.3ms | fwd: 2362.3ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.7ms step 154 | loss: 2.904748 | lr 1.7066e-04 | norm: 0.1662 | dt: 6916.98ms | tok/sec: 75797.3 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.3ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4528.9ms | clip: 0.0ms | optim: 24.8ms step 155 | loss: 2.817366 | lr 1.7032e-04 | norm: 0.1590 | dt: 6917.15ms | tok/sec: 75795.3 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.4ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 156 | loss: 2.964866 | lr 1.6998e-04 | norm: 0.1811 | dt: 6918.69ms | tok/sec: 75778.5 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.2ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.8ms step 157 | loss: 2.951969 | lr 1.6962e-04 | norm: 0.1796 | dt: 6974.31ms | tok/sec: 75174.2 | mem: 68769MB | Time Left: 00 hrs : 28 mins [TIMING] data: 6.1ms | fwd: 2392.4ms | loss: 9.2ms | bwd: 4528.5ms | clip: 0.0ms | optim: 24.7ms step 158 | loss: 2.989079 | lr 1.6927e-04 | norm: 0.1750 | dt: 6914.89ms | tok/sec: 75820.2 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 6.3ms | fwd: 2333.8ms | loss: 9.2ms | bwd: 4527.5ms | clip: 0.0ms | optim: 24.7ms step 159 | loss: 2.867670 | lr 1.6891e-04 | norm: 0.1603 | dt: 6917.89ms | tok/sec: 75787.2 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 6.1ms | fwd: 2333.9ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 160 | loss: 3.032139 | lr 1.6854e-04 | norm: 0.1581 | dt: 6967.06ms | tok/sec: 75252.4 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 5.9ms | fwd: 2382.8ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.8ms step 161 | loss: 2.772293 | lr 1.6817e-04 | norm: 0.1780 | dt: 6916.57ms | tok/sec: 75801.7 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 6.0ms | fwd: 2334.4ms | loss: 9.2ms | bwd: 4528.9ms | clip: 0.0ms | optim: 24.8ms step 162 | loss: 2.943566 | lr 1.6779e-04 | norm: 0.2079 | dt: 6917.54ms | tok/sec: 75791.1 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 6.3ms | fwd: 2333.7ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 163 | loss: 2.707849 | lr 1.6741e-04 | norm: 0.1681 | dt: 6937.48ms | tok/sec: 75573.2 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 5.8ms | fwd: 2353.7ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 164 | loss: 2.860154 | lr 1.6702e-04 | norm: 0.1565 | dt: 6919.44ms | tok/sec: 75770.3 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 6.1ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 25.2ms step 165 | loss: 2.794744 | lr 1.6663e-04 | norm: 0.1679 | dt: 6917.10ms | tok/sec: 75795.9 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 6.2ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4529.6ms | clip: 0.0ms | optim: 24.8ms step 166 | loss: 2.940033 | lr 1.6623e-04 | norm: 0.1597 | dt: 6978.00ms | tok/sec: 75134.4 | mem: 68769MB | Time Left: 00 hrs : 27 mins [TIMING] data: 6.2ms | fwd: 2395.0ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 167 | loss: 2.892095 | lr 1.6583e-04 | norm: 0.1920 | dt: 6917.13ms | tok/sec: 75795.6 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 6.1ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4529.6ms | clip: 0.0ms | optim: 24.8ms step 168 | loss: 2.864672 | lr 1.6542e-04 | norm: 0.1543 | dt: 6916.55ms | tok/sec: 75801.9 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 6.2ms | fwd: 2334.3ms | loss: 9.2ms | bwd: 4528.9ms | clip: 0.0ms | optim: 24.7ms step 169 | loss: 2.898218 | lr 1.6501e-04 | norm: 0.1710 | dt: 6921.11ms | tok/sec: 75752.0 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 5.8ms | fwd: 2337.6ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 170 | loss: 2.837439 | lr 1.6459e-04 | norm: 0.1618 | dt: 6917.49ms | tok/sec: 75791.7 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 6.2ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.7ms step 171 | loss: 2.934631 | lr 1.6417e-04 | norm: 0.1688 | dt: 6916.09ms | tok/sec: 75807.0 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 5.9ms | fwd: 2335.2ms | loss: 9.2ms | bwd: 4527.7ms | clip: 0.0ms | optim: 24.8ms step 172 | loss: 2.909516 | lr 1.6374e-04 | norm: 0.1643 | dt: 6918.07ms | tok/sec: 75785.3 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 6.0ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.7ms step 173 | loss: 3.030828 | lr 1.6331e-04 | norm: 0.1564 | dt: 6915.34ms | tok/sec: 75815.2 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 6.2ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4527.7ms | clip: 0.0ms | optim: 24.8ms step 174 | loss: 2.820354 | lr 1.6287e-04 | norm: 0.1600 | dt: 6915.55ms | tok/sec: 75812.9 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 6.4ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4527.3ms | clip: 0.0ms | optim: 24.8ms step 175 | loss: 2.673743 | lr 1.6243e-04 | norm: 0.1579 | dt: 6945.84ms | tok/sec: 75482.3 | mem: 68769MB | Time Left: 00 hrs : 26 mins [TIMING] data: 6.4ms | fwd: 2360.9ms | loss: 9.2ms | bwd: 4531.2ms | clip: 0.0ms | optim: 24.7ms step 176 | loss: 2.878312 | lr 1.6198e-04 | norm: 0.1676 | dt: 6918.29ms | tok/sec: 75782.9 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.3ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.7ms step 177 | loss: 2.878418 | lr 1.6153e-04 | norm: 0.1618 | dt: 6918.83ms | tok/sec: 75776.9 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.2ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4531.2ms | clip: 0.0ms | optim: 24.8ms step 178 | loss: 2.748242 | lr 1.6107e-04 | norm: 0.1574 | dt: 6978.32ms | tok/sec: 75130.9 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.1ms | fwd: 2396.1ms | loss: 9.2ms | bwd: 4528.8ms | clip: 0.0ms | optim: 24.8ms step 179 | loss: 2.861601 | lr 1.6061e-04 | norm: 0.1529 | dt: 6920.28ms | tok/sec: 75761.1 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.1ms | fwd: 2335.2ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.8ms step 180 | loss: 3.058117 | lr 1.6015e-04 | norm: 0.1560 | dt: 6917.95ms | tok/sec: 75786.6 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.4ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 181 | loss: 3.073493 | lr 1.5968e-04 | norm: 0.1569 | dt: 6921.35ms | tok/sec: 75749.4 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.3ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4533.1ms | clip: 0.0ms | optim: 24.7ms step 182 | loss: 2.850883 | lr 1.5921e-04 | norm: 0.1635 | dt: 6964.54ms | tok/sec: 75279.6 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.2ms | fwd: 2381.7ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 183 | loss: 2.867672 | lr 1.5873e-04 | norm: 0.1628 | dt: 6918.81ms | tok/sec: 75777.1 | mem: 68769MB | Time Left: 00 hrs : 25 mins [TIMING] data: 6.0ms | fwd: 2335.4ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 184 | loss: 2.892456 | lr 1.5825e-04 | norm: 0.1756 | dt: 6916.53ms | tok/sec: 75802.1 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.0ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4528.6ms | clip: 0.0ms | optim: 24.8ms step 185 | loss: 2.945441 | lr 1.5776e-04 | norm: 0.1825 | dt: 6962.48ms | tok/sec: 75301.9 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.4ms | fwd: 2379.2ms | loss: 9.2ms | bwd: 4529.7ms | clip: 0.0ms | optim: 24.7ms step 186 | loss: 2.866949 | lr 1.5727e-04 | norm: 0.1493 | dt: 6919.25ms | tok/sec: 75772.4 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.3ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.7ms step 187 | loss: 2.995467 | lr 1.5677e-04 | norm: 0.1754 | dt: 6935.31ms | tok/sec: 75596.9 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.2ms | fwd: 2351.7ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 188 | loss: 3.095694 | lr 1.5628e-04 | norm: 0.1594 | dt: 6939.78ms | tok/sec: 75548.2 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.0ms | fwd: 2354.5ms | loss: 9.2ms | bwd: 4531.9ms | clip: 0.0ms | optim: 24.7ms step 189 | loss: 2.848757 | lr 1.5577e-04 | norm: 0.1602 | dt: 6918.59ms | tok/sec: 75779.7 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.1ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b2_8192_16384/train_shard_00001.bin 100000000 step 190 | loss: 2.781833 | lr 1.5527e-04 | norm: 0.1645 | dt: 6969.53ms | tok/sec: 75225.8 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 22.5ms | fwd: 2371.2ms | loss: 9.2ms | bwd: 4528.5ms | clip: 0.0ms | optim: 24.8ms step 191 | loss: 2.913383 | lr 1.5476e-04 | norm: 0.1601 | dt: 6920.15ms | tok/sec: 75762.5 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.5ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4531.6ms | clip: 0.0ms | optim: 24.8ms step 192 | loss: 2.855875 | lr 1.5424e-04 | norm: 0.1545 | dt: 6918.72ms | tok/sec: 75778.2 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.1ms | fwd: 2335.0ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.7ms step 193 | loss: 2.916164 | lr 1.5373e-04 | norm: 0.1577 | dt: 6973.66ms | tok/sec: 75181.1 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.1ms | fwd: 2391.7ms | loss: 9.2ms | bwd: 4528.6ms | clip: 0.0ms | optim: 24.7ms step 194 | loss: 2.927748 | lr 1.5320e-04 | norm: 0.1527 | dt: 6916.05ms | tok/sec: 75807.5 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 5.9ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4528.1ms | clip: 0.0ms | optim: 25.1ms step 195 | loss: 2.788964 | lr 1.5268e-04 | norm: 0.1501 | dt: 6983.83ms | tok/sec: 75071.7 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.3ms | fwd: 2397.8ms | loss: 9.2ms | bwd: 4532.4ms | clip: 0.0ms | optim: 24.8ms step 196 | loss: 2.876329 | lr 1.5215e-04 | norm: 0.1611 | dt: 6918.43ms | tok/sec: 75781.3 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.0ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.8ms step 197 | loss: 3.052807 | lr 1.5162e-04 | norm: 0.1549 | dt: 6919.24ms | tok/sec: 75772.5 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.2ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4531.0ms | clip: 0.0ms | optim: 24.8ms step 198 | loss: 2.824064 | lr 1.5108e-04 | norm: 0.1670 | dt: 6918.24ms | tok/sec: 75783.5 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.1ms | fwd: 2337.6ms | loss: 9.2ms | bwd: 4527.4ms | clip: 0.0ms | optim: 24.7ms step 199 | loss: 2.897192 | lr 1.5054e-04 | norm: 0.1582 | dt: 6920.71ms | tok/sec: 75756.4 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.1ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4532.5ms | clip: 0.0ms | optim: 24.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt (safely) step 200 | loss: 2.797292 | lr 1.5000e-04 | norm: 0.1622 | dt: 6919.41ms | tok/sec: 75770.7 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.5ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 201 | loss: 3.002931 | lr 1.4945e-04 | norm: 0.1765 | dt: 6939.60ms | tok/sec: 75550.2 | mem: 68769MB | Time Left: 00 hrs : 23 mins [TIMING] data: 6.4ms | fwd: 2361.5ms | loss: 9.2ms | bwd: 4524.1ms | clip: 0.0ms | optim: 24.8ms step 202 | loss: 2.983289 | lr 1.4891e-04 | norm: 0.1605 | dt: 6910.83ms | tok/sec: 75864.7 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.4ms | fwd: 2332.8ms | loss: 9.2ms | bwd: 4524.1ms | clip: 0.0ms | optim: 24.8ms step 203 | loss: 2.805693 | lr 1.4835e-04 | norm: 0.1534 | dt: 6911.34ms | tok/sec: 75859.1 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.7ms | fwd: 2333.0ms | loss: 9.2ms | bwd: 4524.1ms | clip: 0.0ms | optim: 24.7ms step 204 | loss: 2.940782 | lr 1.4780e-04 | norm: 0.1511 | dt: 6970.13ms | tok/sec: 75219.3 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.6ms | fwd: 2390.9ms | loss: 9.2ms | bwd: 4525.2ms | clip: 0.0ms | optim: 24.8ms step 205 | loss: 2.811560 | lr 1.4724e-04 | norm: 0.1723 | dt: 6917.80ms | tok/sec: 75788.3 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.7ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.8ms step 206 | loss: 2.773270 | lr 1.4668e-04 | norm: 0.1434 | dt: 6957.67ms | tok/sec: 75353.9 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.6ms | fwd: 2373.5ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 207 | loss: 2.806527 | lr 1.4611e-04 | norm: 0.1640 | dt: 6919.37ms | tok/sec: 75771.1 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.5ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.8ms step 208 | loss: 2.954681 | lr 1.4555e-04 | norm: 0.1527 | dt: 6921.03ms | tok/sec: 75752.9 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.6ms | fwd: 2337.3ms | loss: 9.2ms | bwd: 4529.7ms | clip: 0.0ms | optim: 24.8ms step 209 | loss: 2.877160 | lr 1.4498e-04 | norm: 0.1481 | dt: 6965.73ms | tok/sec: 75266.8 | mem: 68769MB | Time Left: 00 hrs : 22 mins [TIMING] data: 6.5ms | fwd: 2382.2ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 210 | loss: 2.917668 | lr 1.4440e-04 | norm: 0.1528 | dt: 6918.12ms | tok/sec: 75784.7 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.6ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 211 | loss: 2.913803 | lr 1.4383e-04 | norm: 0.1499 | dt: 6919.19ms | tok/sec: 75773.1 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.6ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 212 | loss: 3.037122 | lr 1.4325e-04 | norm: 0.1438 | dt: 6917.10ms | tok/sec: 75796.0 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.3ms | fwd: 2335.4ms | loss: 9.2ms | bwd: 4527.7ms | clip: 0.0ms | optim: 24.8ms step 213 | loss: 3.027609 | lr 1.4267e-04 | norm: 0.1547 | dt: 6916.92ms | tok/sec: 75797.9 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.6ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4528.2ms | clip: 0.0ms | optim: 24.8ms step 214 | loss: 2.960036 | lr 1.4209e-04 | norm: 0.1411 | dt: 6974.91ms | tok/sec: 75167.7 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.7ms | fwd: 2390.8ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 215 | loss: 2.872456 | lr 1.4150e-04 | norm: 0.1509 | dt: 6921.24ms | tok/sec: 75750.6 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.6ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4532.2ms | clip: 0.0ms | optim: 24.8ms step 216 | loss: 2.860536 | lr 1.4091e-04 | norm: 0.1417 | dt: 6921.50ms | tok/sec: 75747.7 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.6ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4532.4ms | clip: 0.0ms | optim: 24.8ms step 217 | loss: 2.932910 | lr 1.4032e-04 | norm: 0.1429 | dt: 6974.55ms | tok/sec: 75171.5 | mem: 68769MB | Time Left: 00 hrs : 21 mins [TIMING] data: 6.6ms | fwd: 2389.2ms | loss: 9.2ms | bwd: 4531.4ms | clip: 0.0ms | optim: 24.8ms step 218 | loss: 2.834899 | lr 1.3973e-04 | norm: 0.1498 | dt: 6919.99ms | tok/sec: 75764.3 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.4ms | fwd: 2335.6ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 219 | loss: 2.844939 | lr 1.3914e-04 | norm: 0.1556 | dt: 6921.47ms | tok/sec: 75748.1 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.2ms | fwd: 2335.7ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.8ms step 220 | loss: 2.871496 | lr 1.3854e-04 | norm: 0.1475 | dt: 6967.58ms | tok/sec: 75246.8 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.6ms | fwd: 2380.5ms | loss: 9.2ms | bwd: 4533.1ms | clip: 0.0ms | optim: 24.7ms step 221 | loss: 2.955955 | lr 1.3794e-04 | norm: 0.1448 | dt: 6920.98ms | tok/sec: 75753.5 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.6ms | fwd: 2336.2ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.7ms step 222 | loss: 2.825102 | lr 1.3734e-04 | norm: 0.1453 | dt: 6924.51ms | tok/sec: 75714.8 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.6ms | fwd: 2337.6ms | loss: 9.2ms | bwd: 4533.0ms | clip: 0.0ms | optim: 24.8ms step 223 | loss: 2.846527 | lr 1.3674e-04 | norm: 0.1491 | dt: 6968.66ms | tok/sec: 75235.1 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.6ms | fwd: 2384.3ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.7ms step 224 | loss: 2.854002 | lr 1.3614e-04 | norm: 0.1562 | dt: 6920.34ms | tok/sec: 75760.4 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.5ms | fwd: 2335.5ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.8ms step 225 | loss: 2.776747 | lr 1.3553e-04 | norm: 0.1539 | dt: 6923.08ms | tok/sec: 75730.5 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.6ms | fwd: 2337.8ms | loss: 9.2ms | bwd: 4531.3ms | clip: 0.0ms | optim: 24.8ms step 226 | loss: 2.694734 | lr 1.3492e-04 | norm: 0.1548 | dt: 6984.84ms | tok/sec: 75060.8 | mem: 68769MB | Time Left: 00 hrs : 20 mins [TIMING] data: 6.6ms | fwd: 2400.6ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.9ms step 227 | loss: 2.906092 | lr 1.3431e-04 | norm: 0.1559 | dt: 6918.38ms | tok/sec: 75781.9 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.6ms | fwd: 2335.2ms | loss: 9.2ms | bwd: 4529.0ms | clip: 0.0ms | optim: 24.8ms step 228 | loss: 2.948908 | lr 1.3370e-04 | norm: 0.1555 | dt: 6919.12ms | tok/sec: 75773.8 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.6ms | fwd: 2334.7ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 229 | loss: 2.988147 | lr 1.3309e-04 | norm: 0.2050 | dt: 6948.89ms | tok/sec: 75449.1 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.7ms | fwd: 2363.2ms | loss: 9.2ms | bwd: 4531.6ms | clip: 0.0ms | optim: 24.8ms step 230 | loss: 2.785255 | lr 1.3247e-04 | norm: 0.1486 | dt: 6920.02ms | tok/sec: 75763.9 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.6ms | fwd: 2334.8ms | loss: 9.2ms | bwd: 4531.1ms | clip: 0.0ms | optim: 24.8ms step 231 | loss: 2.910809 | lr 1.3186e-04 | norm: 0.1690 | dt: 6916.21ms | tok/sec: 75805.7 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.6ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4527.8ms | clip: 0.0ms | optim: 24.8ms step 232 | loss: 2.830968 | lr 1.3124e-04 | norm: 0.1602 | dt: 6929.97ms | tok/sec: 75655.2 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.6ms | fwd: 2345.9ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 233 | loss: 2.960292 | lr 1.3063e-04 | norm: 0.1822 | dt: 6918.39ms | tok/sec: 75781.8 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.6ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 234 | loss: 2.923452 | lr 1.3001e-04 | norm: 0.1618 | dt: 6917.03ms | tok/sec: 75796.7 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.4ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4528.6ms | clip: 0.0ms | optim: 24.7ms step 235 | loss: 2.860797 | lr 1.2939e-04 | norm: 0.1646 | dt: 6920.74ms | tok/sec: 75756.0 | mem: 68769MB | Time Left: 00 hrs : 19 mins [TIMING] data: 6.3ms | fwd: 2336.2ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.7ms step 236 | loss: 2.957754 | lr 1.2876e-04 | norm: 0.1545 | dt: 6923.17ms | tok/sec: 75729.5 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.5ms | fwd: 2335.6ms | loss: 9.2ms | bwd: 4533.6ms | clip: 0.0ms | optim: 24.7ms step 237 | loss: 2.853318 | lr 1.2814e-04 | norm: 0.1566 | dt: 6918.73ms | tok/sec: 75778.0 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.7ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 238 | loss: 2.834044 | lr 1.2752e-04 | norm: 0.1580 | dt: 6938.75ms | tok/sec: 75559.4 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.6ms | fwd: 2355.0ms | loss: 9.2ms | bwd: 4529.7ms | clip: 0.0ms | optim: 24.8ms step 239 | loss: 2.814684 | lr 1.2690e-04 | norm: 0.1651 | dt: 6916.41ms | tok/sec: 75803.5 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.5ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4528.0ms | clip: 0.0ms | optim: 24.8ms step 240 | loss: 2.948754 | lr 1.2627e-04 | norm: 0.1545 | dt: 6919.37ms | tok/sec: 75771.0 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.6ms | fwd: 2334.6ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.8ms step 241 | loss: 2.943558 | lr 1.2565e-04 | norm: 0.1651 | dt: 6970.01ms | tok/sec: 75220.6 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.6ms | fwd: 2387.3ms | loss: 9.2ms | bwd: 4528.7ms | clip: 0.0ms | optim: 24.7ms step 242 | loss: 2.898232 | lr 1.2502e-04 | norm: 0.1558 | dt: 6918.22ms | tok/sec: 75783.6 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.4ms | fwd: 2335.1ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.8ms step 243 | loss: 2.901834 | lr 1.2439e-04 | norm: 0.1483 | dt: 6980.09ms | tok/sec: 75111.9 | mem: 68769MB | Time Left: 00 hrs : 18 mins [TIMING] data: 6.6ms | fwd: 2399.3ms | loss: 9.2ms | bwd: 4526.7ms | clip: 0.0ms | optim: 24.8ms step 244 | loss: 2.861255 | lr 1.2377e-04 | norm: 0.1529 | dt: 6917.02ms | tok/sec: 75796.7 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.5ms | fwd: 2333.6ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.7ms step 245 | loss: 2.774570 | lr 1.2314e-04 | norm: 0.1403 | dt: 6915.04ms | tok/sec: 75818.5 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.6ms | fwd: 2332.0ms | loss: 9.2ms | bwd: 4528.9ms | clip: 0.0ms | optim: 24.8ms step 246 | loss: 2.894992 | lr 1.2251e-04 | norm: 0.1512 | dt: 6976.00ms | tok/sec: 75155.9 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.5ms | fwd: 2390.2ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.8ms step 247 | loss: 2.951046 | lr 1.2188e-04 | norm: 0.1982 | dt: 6914.19ms | tok/sec: 75827.8 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.6ms | fwd: 2332.6ms | loss: 9.2ms | bwd: 4527.6ms | clip: 0.0ms | optim: 24.8ms step 248 | loss: 2.856204 | lr 1.2126e-04 | norm: 0.1460 | dt: 6913.05ms | tok/sec: 75840.3 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.2ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4526.9ms | clip: 0.0ms | optim: 24.7ms step 249 | loss: 2.765717 | lr 1.2063e-04 | norm: 0.1462 | dt: 6975.97ms | tok/sec: 75156.2 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.5ms | fwd: 2391.1ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.8ms validation loss: 3.0436 step 250 | loss: 2.776503 | lr 1.2000e-04 | norm: 0.1423 | dt: 9843.65ms | tok/sec: 53261.5 | mem: 68769MB | Time Left: 00 hrs : 24 mins [TIMING] data: 6.3ms | fwd: 2332.7ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.8ms step 251 | loss: 2.958012 | lr 1.1937e-04 | norm: 0.1454 | dt: 6915.64ms | tok/sec: 75811.9 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.2ms | fwd: 2332.3ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 252 | loss: 2.975899 | lr 1.1874e-04 | norm: 0.1489 | dt: 6914.66ms | tok/sec: 75822.6 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.4ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.8ms step 253 | loss: 2.799511 | lr 1.1812e-04 | norm: 0.1480 | dt: 6955.76ms | tok/sec: 75374.6 | mem: 68769MB | Time Left: 00 hrs : 17 mins [TIMING] data: 6.6ms | fwd: 2373.0ms | loss: 9.1ms | bwd: 4528.8ms | clip: 0.0ms | optim: 24.8ms step 254 | loss: 2.871658 | lr 1.1749e-04 | norm: 0.1502 | dt: 6916.94ms | tok/sec: 75797.7 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.5ms | fwd: 2332.6ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 255 | loss: 2.819382 | lr 1.1686e-04 | norm: 0.1619 | dt: 6955.11ms | tok/sec: 75381.7 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.6ms | fwd: 2371.7ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.7ms step 256 | loss: 2.875739 | lr 1.1623e-04 | norm: 0.1524 | dt: 6914.77ms | tok/sec: 75821.5 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.6ms | fwd: 2332.6ms | loss: 9.2ms | bwd: 4528.1ms | clip: 0.0ms | optim: 24.8ms step 257 | loss: 2.831245 | lr 1.1561e-04 | norm: 0.1446 | dt: 6912.84ms | tok/sec: 75842.6 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.7ms | fwd: 2332.4ms | loss: 9.2ms | bwd: 4526.3ms | clip: 0.0ms | optim: 24.8ms step 258 | loss: 2.873638 | lr 1.1498e-04 | norm: 0.1459 | dt: 6962.82ms | tok/sec: 75298.2 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.4ms | fwd: 2380.0ms | loss: 9.2ms | bwd: 4528.9ms | clip: 0.0ms | optim: 24.8ms step 259 | loss: 3.002246 | lr 1.1435e-04 | norm: 0.1520 | dt: 6916.10ms | tok/sec: 75806.9 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.6ms | fwd: 2331.9ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.8ms step 260 | loss: 2.761470 | lr 1.1373e-04 | norm: 0.1459 | dt: 6914.96ms | tok/sec: 75819.4 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.7ms | fwd: 2332.6ms | loss: 9.2ms | bwd: 4528.4ms | clip: 0.0ms | optim: 24.7ms step 261 | loss: 2.859469 | lr 1.1310e-04 | norm: 0.1493 | dt: 6914.29ms | tok/sec: 75826.8 | mem: 68769MB | Time Left: 00 hrs : 16 mins [TIMING] data: 6.6ms | fwd: 2331.8ms | loss: 9.2ms | bwd: 4528.5ms | clip: 0.0ms | optim: 24.8ms step 262 | loss: 2.844394 | lr 1.1248e-04 | norm: 0.1417 | dt: 6956.17ms | tok/sec: 75370.2 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.6ms | fwd: 2372.2ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 263 | loss: 2.796999 | lr 1.1186e-04 | norm: 0.1524 | dt: 6913.48ms | tok/sec: 75835.6 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.6ms | fwd: 2331.8ms | loss: 9.2ms | bwd: 4527.8ms | clip: 0.0ms | optim: 24.8ms step 264 | loss: 3.104138 | lr 1.1124e-04 | norm: 0.1731 | dt: 6928.37ms | tok/sec: 75672.7 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.6ms | fwd: 2345.3ms | loss: 9.2ms | bwd: 4529.2ms | clip: 0.0ms | optim: 24.8ms step 265 | loss: 2.765642 | lr 1.1061e-04 | norm: 0.1497 | dt: 6916.47ms | tok/sec: 75802.8 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.5ms | fwd: 2332.4ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 266 | loss: 3.042243 | lr 1.0999e-04 | norm: 0.1595 | dt: 6915.43ms | tok/sec: 75814.3 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.2ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4529.0ms | clip: 0.0ms | optim: 24.8ms step 267 | loss: 2.970291 | lr 1.0937e-04 | norm: 0.1386 | dt: 6955.22ms | tok/sec: 75380.5 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.0ms | fwd: 2372.7ms | loss: 9.2ms | bwd: 4528.7ms | clip: 0.0ms | optim: 24.8ms step 268 | loss: 2.708671 | lr 1.0876e-04 | norm: 0.1647 | dt: 6916.15ms | tok/sec: 75806.3 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.7ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.7ms step 269 | loss: 2.806233 | lr 1.0814e-04 | norm: 0.2404 | dt: 6915.88ms | tok/sec: 75809.3 | mem: 68769MB | Time Left: 00 hrs : 15 mins [TIMING] data: 6.7ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 270 | loss: 2.847450 | lr 1.0753e-04 | norm: 0.1380 | dt: 6918.22ms | tok/sec: 75783.7 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.6ms | fwd: 2333.6ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.7ms step 271 | loss: 2.903801 | lr 1.0691e-04 | norm: 0.1678 | dt: 6969.08ms | tok/sec: 75230.6 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.7ms | fwd: 2383.2ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.8ms step 272 | loss: 2.814969 | lr 1.0630e-04 | norm: 0.1413 | dt: 6918.41ms | tok/sec: 75781.5 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.5ms | fwd: 2333.7ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.7ms step 273 | loss: 2.725717 | lr 1.0569e-04 | norm: 0.1362 | dt: 6938.49ms | tok/sec: 75562.3 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.5ms | fwd: 2354.3ms | loss: 9.2ms | bwd: 4529.6ms | clip: 0.0ms | optim: 24.8ms step 274 | loss: 2.816581 | lr 1.0508e-04 | norm: 0.1416 | dt: 6918.85ms | tok/sec: 75776.7 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.6ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.8ms step 275 | loss: 2.891945 | lr 1.0447e-04 | norm: 0.1374 | dt: 6917.81ms | tok/sec: 75788.2 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.6ms | fwd: 2334.5ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 276 | loss: 2.975525 | lr 1.0386e-04 | norm: 0.1488 | dt: 6939.52ms | tok/sec: 75551.0 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.7ms | fwd: 2354.4ms | loss: 9.2ms | bwd: 4531.0ms | clip: 0.0ms | optim: 24.8ms step 277 | loss: 2.832519 | lr 1.0326e-04 | norm: 0.1401 | dt: 6916.19ms | tok/sec: 75805.9 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.6ms | fwd: 2333.2ms | loss: 9.2ms | bwd: 4529.0ms | clip: 0.0ms | optim: 24.8ms step 278 | loss: 2.769354 | lr 1.0266e-04 | norm: 0.1418 | dt: 6916.62ms | tok/sec: 75801.2 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.5ms | fwd: 2333.4ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.8ms step 279 | loss: 2.848495 | lr 1.0206e-04 | norm: 0.1414 | dt: 6980.94ms | tok/sec: 75102.8 | mem: 68769MB | Time Left: 00 hrs : 14 mins [TIMING] data: 6.7ms | fwd: 2397.5ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.7ms step 280 | loss: 2.742930 | lr 1.0146e-04 | norm: 0.1464 | dt: 6917.27ms | tok/sec: 75794.1 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.6ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 281 | loss: 2.557336 | lr 1.0086e-04 | norm: 0.1540 | dt: 6918.66ms | tok/sec: 75778.9 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.6ms | fwd: 2332.9ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.7ms step 282 | loss: 2.846231 | lr 1.0027e-04 | norm: 0.1424 | dt: 6935.61ms | tok/sec: 75593.7 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.5ms | fwd: 2352.2ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 283 | loss: 2.720650 | lr 9.9676e-05 | norm: 0.1421 | dt: 6925.50ms | tok/sec: 75703.9 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.6ms | fwd: 2343.6ms | loss: 9.2ms | bwd: 4527.9ms | clip: 0.0ms | optim: 24.8ms step 284 | loss: 2.780203 | lr 9.9086e-05 | norm: 0.1340 | dt: 6918.24ms | tok/sec: 75783.4 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.4ms | fwd: 2333.9ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 285 | loss: 2.924182 | lr 9.8498e-05 | norm: 0.1543 | dt: 6918.30ms | tok/sec: 75782.8 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.5ms | fwd: 2334.2ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 286 | loss: 2.888474 | lr 9.7913e-05 | norm: 0.1520 | dt: 6975.35ms | tok/sec: 75163.0 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.4ms | fwd: 2390.7ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.7ms step 287 | loss: 2.770988 | lr 9.7330e-05 | norm: 0.1567 | dt: 6918.12ms | tok/sec: 75784.7 | mem: 68769MB | Time Left: 00 hrs : 13 mins [TIMING] data: 6.6ms | fwd: 2333.9ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.8ms step 288 | loss: 2.944279 | lr 9.6749e-05 | norm: 0.1397 | dt: 6917.97ms | tok/sec: 75786.4 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.6ms | fwd: 2333.3ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 289 | loss: 2.882476 | lr 9.6171e-05 | norm: 0.1410 | dt: 6987.48ms | tok/sec: 75032.4 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.6ms | fwd: 2402.2ms | loss: 9.2ms | bwd: 4531.4ms | clip: 0.0ms | optim: 24.7ms step 290 | loss: 2.613557 | lr 9.5596e-05 | norm: 0.1508 | dt: 6916.03ms | tok/sec: 75807.7 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.6ms | fwd: 2333.8ms | loss: 9.1ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.7ms step 291 | loss: 2.883360 | lr 9.5023e-05 | norm: 0.1424 | dt: 6924.55ms | tok/sec: 75714.3 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.6ms | fwd: 2339.8ms | loss: 9.2ms | bwd: 4530.8ms | clip: 0.0ms | optim: 24.8ms step 292 | loss: 2.833606 | lr 9.4453e-05 | norm: 0.1514 | dt: 6915.50ms | tok/sec: 75813.4 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.4ms | fwd: 2332.1ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 293 | loss: 2.978414 | lr 9.3886e-05 | norm: 0.1548 | dt: 6919.26ms | tok/sec: 75772.2 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.7ms | fwd: 2334.1ms | loss: 9.2ms | bwd: 4531.1ms | clip: 0.0ms | optim: 24.8ms step 294 | loss: 2.976890 | lr 9.3322e-05 | norm: 0.1358 | dt: 6983.70ms | tok/sec: 75073.1 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.5ms | fwd: 2400.2ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 295 | loss: 2.986186 | lr 9.2761e-05 | norm: 0.1475 | dt: 6916.44ms | tok/sec: 75803.2 | mem: 68769MB | Time Left: 00 hrs : 12 mins [TIMING] data: 6.6ms | fwd: 2333.0ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 296 | loss: 2.705819 | lr 9.2202e-05 | norm: 0.1454 | dt: 6920.83ms | tok/sec: 75755.1 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.4ms | fwd: 2333.5ms | loss: 9.2ms | bwd: 4533.4ms | clip: 0.0ms | optim: 24.8ms step 297 | loss: 2.596311 | lr 9.1647e-05 | norm: 0.1492 | dt: 6918.86ms | tok/sec: 75776.7 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.6ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 298 | loss: 2.941808 | lr 9.1095e-05 | norm: 0.1534 | dt: 6918.90ms | tok/sec: 75776.2 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.6ms | fwd: 2333.3ms | loss: 9.2ms | bwd: 4531.6ms | clip: 0.0ms | optim: 24.8ms step 299 | loss: 2.900815 | lr 9.0546e-05 | norm: 0.1487 | dt: 6916.10ms | tok/sec: 75806.9 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.6ms | fwd: 2332.8ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 300 | loss: 2.727150 | lr 9.0000e-05 | norm: 0.1415 | dt: 6916.45ms | tok/sec: 75803.0 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.6ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 301 | loss: 2.929566 | lr 8.9458e-05 | norm: 0.1361 | dt: 6973.75ms | tok/sec: 75180.2 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.7ms | fwd: 2390.7ms | loss: 9.2ms | bwd: 4529.0ms | clip: 0.0ms | optim: 24.8ms step 302 | loss: 3.086802 | lr 8.8918e-05 | norm: 0.1467 | dt: 6917.48ms | tok/sec: 75791.7 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.7ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4530.4ms | clip: 0.0ms | optim: 24.8ms step 303 | loss: 2.900136 | lr 8.8383e-05 | norm: 0.1389 | dt: 6975.08ms | tok/sec: 75165.9 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.6ms | fwd: 2389.8ms | loss: 9.2ms | bwd: 4531.3ms | clip: 0.0ms | optim: 24.8ms step 304 | loss: 2.820658 | lr 8.7850e-05 | norm: 0.1599 | dt: 6918.80ms | tok/sec: 75777.3 | mem: 68769MB | Time Left: 00 hrs : 11 mins [TIMING] data: 6.6ms | fwd: 2332.7ms | loss: 9.2ms | bwd: 4532.1ms | clip: 0.0ms | optim: 24.7ms step 305 | loss: 3.127460 | lr 8.7322e-05 | norm: 0.1500 | dt: 6919.23ms | tok/sec: 75772.6 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.4ms | fwd: 2332.9ms | loss: 9.2ms | bwd: 4532.4ms | clip: 0.0ms | optim: 24.7ms step 306 | loss: 2.711572 | lr 8.6797e-05 | norm: 0.1487 | dt: 6920.94ms | tok/sec: 75753.9 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.7ms | fwd: 2333.0ms | loss: 9.2ms | bwd: 4534.0ms | clip: 0.0ms | optim: 24.8ms step 307 | loss: 2.925205 | lr 8.6275e-05 | norm: 0.1442 | dt: 6946.22ms | tok/sec: 75478.2 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.6ms | fwd: 2359.7ms | loss: 9.2ms | bwd: 4532.6ms | clip: 0.0ms | optim: 24.7ms step 308 | loss: 2.850485 | lr 8.5757e-05 | norm: 0.1417 | dt: 6922.00ms | tok/sec: 75742.3 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.5ms | fwd: 2334.0ms | loss: 9.2ms | bwd: 4534.1ms | clip: 0.0ms | optim: 24.7ms step 309 | loss: 2.843923 | lr 8.5243e-05 | norm: 0.1441 | dt: 6921.31ms | tok/sec: 75749.8 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.4ms | fwd: 2335.2ms | loss: 9.2ms | bwd: 4532.2ms | clip: 0.0ms | optim: 24.7ms step 310 | loss: 2.927427 | lr 8.4733e-05 | norm: 0.1531 | dt: 6917.24ms | tok/sec: 75794.4 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.7ms | fwd: 2333.3ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.8ms step 311 | loss: 3.177454 | lr 8.4227e-05 | norm: 0.1462 | dt: 6919.56ms | tok/sec: 75768.9 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.6ms | fwd: 2333.2ms | loss: 9.2ms | bwd: 4532.4ms | clip: 0.0ms | optim: 24.8ms step 312 | loss: 2.845066 | lr 8.3724e-05 | norm: 0.1396 | dt: 6917.56ms | tok/sec: 75790.9 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.4ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 313 | loss: 2.722527 | lr 8.3226e-05 | norm: 0.1463 | dt: 6916.59ms | tok/sec: 75801.5 | mem: 68769MB | Time Left: 00 hrs : 10 mins [TIMING] data: 6.6ms | fwd: 2331.9ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.7ms step 314 | loss: 2.807845 | lr 8.2731e-05 | norm: 0.1574 | dt: 6921.46ms | tok/sec: 75748.1 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.5ms | fwd: 2337.0ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.7ms step 315 | loss: 2.787709 | lr 8.2241e-05 | norm: 0.1464 | dt: 6917.95ms | tok/sec: 75786.7 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.3ms | fwd: 2332.7ms | loss: 9.2ms | bwd: 4531.3ms | clip: 0.0ms | optim: 24.8ms step 316 | loss: 2.988481 | lr 8.1755e-05 | norm: 0.1402 | dt: 6919.71ms | tok/sec: 75767.3 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.6ms | fwd: 2333.6ms | loss: 9.2ms | bwd: 4532.0ms | clip: 0.0ms | optim: 24.8ms step 317 | loss: 2.726112 | lr 8.1273e-05 | norm: 0.1557 | dt: 6986.40ms | tok/sec: 75044.1 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.5ms | fwd: 2401.2ms | loss: 9.2ms | bwd: 4531.2ms | clip: 0.0ms | optim: 24.7ms step 318 | loss: 2.760153 | lr 8.0795e-05 | norm: 0.1411 | dt: 6915.37ms | tok/sec: 75814.9 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.7ms | fwd: 2332.4ms | loss: 9.2ms | bwd: 4528.8ms | clip: 0.0ms | optim: 24.8ms step 319 | loss: 2.920295 | lr 8.0321e-05 | norm: 0.1557 | dt: 6917.70ms | tok/sec: 75789.3 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.4ms | fwd: 2333.2ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.8ms step 320 | loss: 2.863719 | lr 7.9852e-05 | norm: 0.1423 | dt: 6927.92ms | tok/sec: 75677.5 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.6ms | fwd: 2344.2ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.8ms step 321 | loss: 2.844861 | lr 7.9387e-05 | norm: 0.1391 | dt: 6915.70ms | tok/sec: 75811.3 | mem: 68769MB | Time Left: 00 hrs : 09 mins [TIMING] data: 6.6ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.7ms step 322 | loss: 2.900485 | lr 7.8927e-05 | norm: 0.1433 | dt: 6914.94ms | tok/sec: 75819.5 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.7ms | fwd: 2333.0ms | loss: 9.2ms | bwd: 4527.9ms | clip: 0.0ms | optim: 24.8ms step 323 | loss: 2.773129 | lr 7.8471e-05 | norm: 0.1398 | dt: 6919.31ms | tok/sec: 75771.7 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.6ms | fwd: 2333.4ms | loss: 9.2ms | bwd: 4531.9ms | clip: 0.0ms | optim: 24.8ms step 324 | loss: 2.778138 | lr 7.8020e-05 | norm: 0.1315 | dt: 6916.33ms | tok/sec: 75804.3 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.5ms | fwd: 2332.6ms | loss: 9.2ms | bwd: 4529.8ms | clip: 0.0ms | optim: 24.8ms step 325 | loss: 3.006305 | lr 7.7574e-05 | norm: 0.1455 | dt: 6915.39ms | tok/sec: 75814.7 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.4ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.7ms step 326 | loss: 2.798800 | lr 7.7132e-05 | norm: 0.1466 | dt: 6933.03ms | tok/sec: 75621.8 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.5ms | fwd: 2349.2ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.8ms step 327 | loss: 2.920716 | lr 7.6694e-05 | norm: 0.1457 | dt: 6915.72ms | tok/sec: 75811.0 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.6ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 328 | loss: 2.795486 | lr 7.6262e-05 | norm: 0.1375 | dt: 6918.49ms | tok/sec: 75780.7 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.8ms | fwd: 2332.9ms | loss: 9.2ms | bwd: 4531.4ms | clip: 0.0ms | optim: 24.8ms step 329 | loss: 2.864070 | lr 7.5834e-05 | norm: 0.1400 | dt: 6964.57ms | tok/sec: 75279.3 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.6ms | fwd: 2381.3ms | loss: 9.2ms | bwd: 4529.3ms | clip: 0.0ms | optim: 24.8ms step 330 | loss: 2.946831 | lr 7.5411e-05 | norm: 0.1574 | dt: 6915.93ms | tok/sec: 75808.7 | mem: 68769MB | Time Left: 00 hrs : 08 mins [TIMING] data: 6.6ms | fwd: 2332.3ms | loss: 9.2ms | bwd: 4529.7ms | clip: 0.0ms | optim: 24.8ms step 331 | loss: 2.817824 | lr 7.4993e-05 | norm: 0.1297 | dt: 6917.13ms | tok/sec: 75795.6 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.6ms | fwd: 2333.2ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 332 | loss: 2.786872 | lr 7.4580e-05 | norm: 0.1410 | dt: 6938.78ms | tok/sec: 75559.1 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.5ms | fwd: 2356.5ms | loss: 9.2ms | bwd: 4528.4ms | clip: 0.0ms | optim: 24.8ms step 333 | loss: 2.598512 | lr 7.4172e-05 | norm: 0.1310 | dt: 6914.80ms | tok/sec: 75821.1 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.7ms | fwd: 2331.7ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.7ms step 334 | loss: 2.861376 | lr 7.3769e-05 | norm: 0.1405 | dt: 6919.33ms | tok/sec: 75771.5 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.6ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.7ms step 335 | loss: 2.857138 | lr 7.3371e-05 | norm: 0.1391 | dt: 6971.76ms | tok/sec: 75201.7 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.5ms | fwd: 2386.6ms | loss: 9.2ms | bwd: 4531.1ms | clip: 0.0ms | optim: 24.7ms step 336 | loss: 2.882327 | lr 7.2978e-05 | norm: 0.1426 | dt: 6914.83ms | tok/sec: 75820.8 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.6ms | fwd: 2332.6ms | loss: 9.2ms | bwd: 4528.1ms | clip: 0.0ms | optim: 24.8ms step 337 | loss: 2.966919 | lr 7.2591e-05 | norm: 0.1352 | dt: 6915.80ms | tok/sec: 75810.1 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.6ms | fwd: 2332.4ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 338 | loss: 2.958948 | lr 7.2208e-05 | norm: 0.1387 | dt: 6916.10ms | tok/sec: 75806.9 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.6ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.8ms step 339 | loss: 2.769508 | lr 7.1831e-05 | norm: 0.1374 | dt: 6916.30ms | tok/sec: 75804.6 | mem: 68769MB | Time Left: 00 hrs : 07 mins [TIMING] data: 6.5ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4528.7ms | clip: 0.0ms | optim: 24.8ms step 340 | loss: 2.946433 | lr 7.1459e-05 | norm: 0.1407 | dt: 6916.80ms | tok/sec: 75799.2 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.6ms | fwd: 2333.3ms | loss: 9.2ms | bwd: 4529.5ms | clip: 0.0ms | optim: 24.7ms step 341 | loss: 2.904063 | lr 7.1092e-05 | norm: 0.1380 | dt: 6928.90ms | tok/sec: 75666.8 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.6ms | fwd: 2345.1ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.7ms step 342 | loss: 2.913668 | lr 7.0731e-05 | norm: 0.1363 | dt: 6914.91ms | tok/sec: 75820.0 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.6ms | fwd: 2332.7ms | loss: 9.2ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.8ms step 343 | loss: 2.687752 | lr 7.0375e-05 | norm: 0.1330 | dt: 6917.31ms | tok/sec: 75793.6 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.6ms | fwd: 2333.5ms | loss: 9.2ms | bwd: 4529.7ms | clip: 0.0ms | optim: 24.7ms step 344 | loss: 2.833128 | lr 7.0025e-05 | norm: 0.1354 | dt: 6976.37ms | tok/sec: 75151.9 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.7ms | fwd: 2392.8ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 345 | loss: 2.739064 | lr 6.9680e-05 | norm: 0.1327 | dt: 6919.34ms | tok/sec: 75771.3 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.6ms | fwd: 2332.8ms | loss: 9.2ms | bwd: 4532.5ms | clip: 0.0ms | optim: 24.8ms step 346 | loss: 2.783349 | lr 6.9340e-05 | norm: 0.1426 | dt: 6915.83ms | tok/sec: 75809.8 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.5ms | fwd: 2331.1ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.8ms step 347 | loss: 2.897111 | lr 6.9006e-05 | norm: 0.1293 | dt: 6913.80ms | tok/sec: 75832.1 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.6ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4527.6ms | clip: 0.0ms | optim: 24.8ms step 348 | loss: 2.836001 | lr 6.8678e-05 | norm: 0.1366 | dt: 6978.60ms | tok/sec: 75127.9 | mem: 68769MB | Time Left: 00 hrs : 06 mins [TIMING] data: 6.4ms | fwd: 2395.9ms | loss: 9.2ms | bwd: 4528.6ms | clip: 0.0ms | optim: 24.8ms step 349 | loss: 3.025473 | lr 6.8355e-05 | norm: 0.1413 | dt: 6917.27ms | tok/sec: 75794.1 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.6ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 350 | loss: 2.767489 | lr 6.8038e-05 | norm: 0.1304 | dt: 6917.57ms | tok/sec: 75790.8 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.5ms | fwd: 2332.1ms | loss: 9.2ms | bwd: 4531.5ms | clip: 0.0ms | optim: 24.8ms step 351 | loss: 2.816988 | lr 6.7727e-05 | norm: 0.1260 | dt: 6965.80ms | tok/sec: 75266.0 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.4ms | fwd: 2381.4ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 352 | loss: 2.828808 | lr 6.7422e-05 | norm: 0.1394 | dt: 6917.02ms | tok/sec: 75796.8 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.6ms | fwd: 2332.3ms | loss: 9.2ms | bwd: 4530.7ms | clip: 0.0ms | optim: 24.8ms step 353 | loss: 2.817611 | lr 6.7122e-05 | norm: 0.1400 | dt: 6915.93ms | tok/sec: 75808.7 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.6ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4529.6ms | clip: 0.0ms | optim: 24.8ms step 354 | loss: 2.999857 | lr 6.6828e-05 | norm: 0.1277 | dt: 6915.88ms | tok/sec: 75809.2 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.4ms | fwd: 2332.4ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 355 | loss: 2.797555 | lr 6.6540e-05 | norm: 0.1362 | dt: 6985.58ms | tok/sec: 75052.9 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.7ms | fwd: 2399.7ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.8ms step 356 | loss: 2.858858 | lr 6.6257e-05 | norm: 0.1348 | dt: 6916.84ms | tok/sec: 75798.8 | mem: 68769MB | Time Left: 00 hrs : 05 mins [TIMING] data: 6.6ms | fwd: 2332.3ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.8ms step 357 | loss: 2.926979 | lr 6.5981e-05 | norm: 0.1296 | dt: 6914.65ms | tok/sec: 75822.7 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.6ms | fwd: 2331.9ms | loss: 9.2ms | bwd: 4528.8ms | clip: 0.0ms | optim: 24.7ms step 358 | loss: 2.838635 | lr 6.5710e-05 | norm: 0.1779 | dt: 6916.54ms | tok/sec: 75802.0 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.7ms | fwd: 2333.1ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 359 | loss: 2.679262 | lr 6.5446e-05 | norm: 0.1298 | dt: 6952.10ms | tok/sec: 75414.3 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.4ms | fwd: 2368.2ms | loss: 9.1ms | bwd: 4530.1ms | clip: 0.0ms | optim: 24.8ms step 360 | loss: 2.915030 | lr 6.5187e-05 | norm: 0.1462 | dt: 6919.58ms | tok/sec: 75768.8 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.6ms | fwd: 2333.2ms | loss: 9.2ms | bwd: 4532.3ms | clip: 0.0ms | optim: 24.8ms step 361 | loss: 2.928519 | lr 6.4935e-05 | norm: 0.1328 | dt: 6915.16ms | tok/sec: 75817.1 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.6ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4528.7ms | clip: 0.0ms | optim: 24.8ms step 362 | loss: 2.907701 | lr 6.4688e-05 | norm: 0.1331 | dt: 6935.40ms | tok/sec: 75595.9 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.3ms | fwd: 2350.9ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 363 | loss: 2.753704 | lr 6.4448e-05 | norm: 0.1556 | dt: 6914.35ms | tok/sec: 75826.1 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.4ms | fwd: 2332.1ms | loss: 9.2ms | bwd: 4528.2ms | clip: 0.0ms | optim: 24.7ms step 364 | loss: 2.749675 | lr 6.4213e-05 | norm: 0.1427 | dt: 6916.03ms | tok/sec: 75807.7 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.6ms | fwd: 2332.0ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms step 365 | loss: 2.927006 | lr 6.3985e-05 | norm: 0.1353 | dt: 6983.84ms | tok/sec: 75071.6 | mem: 68769MB | Time Left: 00 hrs : 04 mins [TIMING] data: 6.6ms | fwd: 2399.7ms | loss: 9.2ms | bwd: 4530.2ms | clip: 0.0ms | optim: 24.8ms step 366 | loss: 2.889222 | lr 6.3763e-05 | norm: 0.1390 | dt: 6917.81ms | tok/sec: 75788.2 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.6ms | fwd: 2332.8ms | loss: 9.2ms | bwd: 4531.0ms | clip: 0.0ms | optim: 24.7ms step 367 | loss: 2.799849 | lr 6.3547e-05 | norm: 0.1325 | dt: 6963.37ms | tok/sec: 75292.3 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.7ms | fwd: 2380.6ms | loss: 9.2ms | bwd: 4528.8ms | clip: 0.0ms | optim: 24.8ms step 368 | loss: 2.749058 | lr 6.3337e-05 | norm: 0.1295 | dt: 6929.08ms | tok/sec: 75664.9 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.6ms | fwd: 2344.5ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 369 | loss: 2.701562 | lr 6.3134e-05 | norm: 0.1314 | dt: 6913.23ms | tok/sec: 75838.3 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.6ms | fwd: 2331.7ms | loss: 9.2ms | bwd: 4527.5ms | clip: 0.0ms | optim: 24.8ms step 370 | loss: 2.916738 | lr 6.2937e-05 | norm: 0.1423 | dt: 6952.54ms | tok/sec: 75409.6 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.6ms | fwd: 2370.2ms | loss: 9.2ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.7ms step 371 | loss: 2.937566 | lr 6.2746e-05 | norm: 0.1417 | dt: 6914.99ms | tok/sec: 75819.1 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.6ms | fwd: 2331.2ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.8ms step 372 | loss: 2.599833 | lr 6.2561e-05 | norm: 0.1317 | dt: 6918.15ms | tok/sec: 75784.4 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.4ms | fwd: 2331.9ms | loss: 9.2ms | bwd: 4532.3ms | clip: 0.0ms | optim: 24.7ms step 373 | loss: 2.997310 | lr 6.2382e-05 | norm: 0.1286 | dt: 6970.76ms | tok/sec: 75212.4 | mem: 68769MB | Time Left: 00 hrs : 03 mins [TIMING] data: 6.7ms | fwd: 2384.3ms | loss: 9.2ms | bwd: 4532.4ms | clip: 0.0ms | optim: 24.7ms step 374 | loss: 2.795067 | lr 6.2210e-05 | norm: 0.1281 | dt: 6913.74ms | tok/sec: 75832.8 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.6ms | fwd: 2331.9ms | loss: 9.2ms | bwd: 4527.9ms | clip: 0.0ms | optim: 24.8ms step 375 | loss: 2.776080 | lr 6.2044e-05 | norm: 0.1323 | dt: 6914.43ms | tok/sec: 75825.2 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.7ms | fwd: 2332.1ms | loss: 9.2ms | bwd: 4528.4ms | clip: 0.0ms | optim: 24.7ms step 376 | loss: 2.932393 | lr 6.1885e-05 | norm: 0.1394 | dt: 6969.43ms | tok/sec: 75226.8 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.6ms | fwd: 2384.9ms | loss: 9.2ms | bwd: 4530.6ms | clip: 0.0ms | optim: 24.8ms step 377 | loss: 2.858429 | lr 6.1732e-05 | norm: 0.1366 | dt: 6917.38ms | tok/sec: 75792.8 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.6ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4531.2ms | clip: 0.0ms | optim: 24.8ms step 378 | loss: 2.824596 | lr 6.1585e-05 | norm: 0.1526 | dt: 6917.47ms | tok/sec: 75791.9 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.6ms | fwd: 2332.3ms | loss: 9.2ms | bwd: 4531.2ms | clip: 0.0ms | optim: 24.8ms step 379 | loss: 2.684332 | lr 6.1445e-05 | norm: 0.1308 | dt: 6914.11ms | tok/sec: 75828.7 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.7ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4527.9ms | clip: 0.0ms | optim: 24.8ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b2_8192_16384/train_shard_00002.bin 100000000 step 380 | loss: 2.771928 | lr 6.1311e-05 | norm: 0.1325 | dt: 6967.31ms | tok/sec: 75249.7 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 23.3ms | fwd: 2366.3ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 381 | loss: 2.755242 | lr 6.1184e-05 | norm: 0.1337 | dt: 6917.87ms | tok/sec: 75787.5 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.6ms | fwd: 2332.0ms | loss: 9.2ms | bwd: 4531.8ms | clip: 0.0ms | optim: 24.8ms step 382 | loss: 2.747681 | lr 6.1063e-05 | norm: 0.1443 | dt: 6914.25ms | tok/sec: 75827.2 | mem: 68769MB | Time Left: 00 hrs : 02 mins [TIMING] data: 6.7ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4528.0ms | clip: 0.0ms | optim: 24.8ms step 383 | loss: 2.969704 | lr 6.0948e-05 | norm: 0.1332 | dt: 6924.27ms | tok/sec: 75717.5 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.4ms | fwd: 2339.7ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 384 | loss: 2.787799 | lr 6.0840e-05 | norm: 0.1336 | dt: 6916.62ms | tok/sec: 75801.2 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.7ms | fwd: 2333.0ms | loss: 9.2ms | bwd: 4529.6ms | clip: 0.0ms | optim: 24.8ms step 385 | loss: 2.863563 | lr 6.0739e-05 | norm: 0.1341 | dt: 6917.67ms | tok/sec: 75789.7 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.5ms | fwd: 2332.8ms | loss: 9.2ms | bwd: 4530.9ms | clip: 0.0ms | optim: 24.8ms step 386 | loss: 2.775815 | lr 6.0644e-05 | norm: 0.1364 | dt: 6940.00ms | tok/sec: 75545.8 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.7ms | fwd: 2355.5ms | loss: 9.2ms | bwd: 4530.5ms | clip: 0.0ms | optim: 24.8ms step 387 | loss: 3.066909 | lr 6.0555e-05 | norm: 0.1322 | dt: 6914.53ms | tok/sec: 75824.1 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.6ms | fwd: 2332.0ms | loss: 9.2ms | bwd: 4528.4ms | clip: 0.0ms | optim: 24.7ms step 388 | loss: 2.814371 | lr 6.0473e-05 | norm: 0.1361 | dt: 6915.13ms | tok/sec: 75817.5 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.5ms | fwd: 2332.0ms | loss: 9.2ms | bwd: 4529.1ms | clip: 0.0ms | optim: 24.8ms step 389 | loss: 2.952535 | lr 6.0398e-05 | norm: 0.1406 | dt: 6976.73ms | tok/sec: 75148.1 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.5ms | fwd: 2394.3ms | loss: 9.2ms | bwd: 4528.4ms | clip: 0.0ms | optim: 24.7ms step 390 | loss: 2.908341 | lr 6.0329e-05 | norm: 0.1324 | dt: 6917.59ms | tok/sec: 75790.6 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.6ms | fwd: 2332.3ms | loss: 9.1ms | bwd: 4531.2ms | clip: 0.0ms | optim: 24.8ms step 391 | loss: 3.073706 | lr 6.0266e-05 | norm: 0.1628 | dt: 6930.49ms | tok/sec: 75649.5 | mem: 68769MB | Time Left: 00 hrs : 01 mins [TIMING] data: 6.6ms | fwd: 2346.6ms | loss: 9.2ms | bwd: 4529.9ms | clip: 0.0ms | optim: 24.8ms step 392 | loss: 2.594715 | lr 6.0210e-05 | norm: 0.1328 | dt: 6916.15ms | tok/sec: 75806.3 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.5ms | fwd: 2331.2ms | loss: 9.2ms | bwd: 4530.9ms | clip: 0.0ms | optim: 24.8ms step 393 | loss: 2.808028 | lr 6.0161e-05 | norm: 0.1292 | dt: 6915.90ms | tok/sec: 75809.0 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.6ms | fwd: 2332.5ms | loss: 9.2ms | bwd: 4529.4ms | clip: 0.0ms | optim: 24.8ms step 394 | loss: 2.744793 | lr 6.0118e-05 | norm: 0.1338 | dt: 6917.94ms | tok/sec: 75786.7 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.6ms | fwd: 2334.9ms | loss: 9.2ms | bwd: 4529.0ms | clip: 0.0ms | optim: 24.7ms step 395 | loss: 2.888179 | lr 6.0082e-05 | norm: 0.1339 | dt: 6915.68ms | tok/sec: 75811.5 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.6ms | fwd: 2331.9ms | loss: 9.2ms | bwd: 4529.7ms | clip: 0.0ms | optim: 24.8ms step 396 | loss: 2.836074 | lr 6.0053e-05 | norm: 0.1369 | dt: 6915.09ms | tok/sec: 75818.0 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.4ms | fwd: 2332.7ms | loss: 9.2ms | bwd: 4528.5ms | clip: 0.0ms | optim: 24.8ms step 397 | loss: 2.996349 | lr 6.0030e-05 | norm: 0.1377 | dt: 6948.00ms | tok/sec: 75458.8 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.6ms | fwd: 2363.8ms | loss: 9.2ms | bwd: 4530.3ms | clip: 0.0ms | optim: 24.8ms step 398 | loss: 2.796116 | lr 6.0013e-05 | norm: 0.1267 | dt: 6916.31ms | tok/sec: 75804.6 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.4ms | fwd: 2332.2ms | loss: 9.2ms | bwd: 4530.0ms | clip: 0.0ms | optim: 24.8ms validation loss: 3.0124 step 399 | loss: 2.794149 | lr 6.0003e-05 | norm: 0.1381 | dt: 9852.27ms | tok/sec: 53214.9 | mem: 68769MB | Time Left: 00 hrs : 00 mins [TIMING] data: 6.6ms | fwd: 2332.4ms | loss: 9.2ms | bwd: 4528.3ms | clip: 0.0ms | optim: 24.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_400.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt === Training complete ===