Hugging Face's logo

unparallelled
/
BluScriptCP

Safetensors
Model card Files
xet
 Community
BluScriptCP / logs /b4cont /train.log
unparallelled's picture
unparallelled
Upload folder using huggingface_hub
6e0358e verified
Raw History Blame Contribute Delete
375 kB
=== Llama Context Parallel Training Script (bluscriptCP) === [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) ================ Configuration (bluscriptCP) ================ d_model/layers : 1536 / 36 q/kv heads : 12 / 4 (G=3, hd=128) ffn_hidden : 4096 (SwiGLU) context_length : 32768 vocab_size : 50304 tying=0 B x T : 1 x 32768 world_size : 2 (dp=1 x cp=2) CP_ATTN_MODE : ring (load_balance=HeadTail) global_batch : 524288 (grad_accum=16) precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] max_lr/min_lr : 0.0006 / 6e-05 warmup/steps : 100 / 2227 (WSD: stable=1177, decay tail=950) checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp) ============================================================= Initializing model on CUDA device 0... [YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693 Parameters: 1060624896 Parameters per GPU: 1060624896 max_steps: 2227 warmup_steps: 100 [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] [ZeRO-1] DataParallel is DISABLED (both would double-reduce) [precision] bf16 params=217, all with fp32 master weights found 12 shards for split train found 1 shards for split val Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (resume run 2) [rope_theta] 500000 from checkpoints_bluscriptcp/blumodelcp_run2.meta (authoritative) Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 2) [CheckpointManager] Auto-loading latest: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.ckpt [Resume] run 2 from step 750 (loss 2.57022) [resume] fast-forwarded data loader 12000 batches (393216000 tokens) [re-warmup] anchored at step 750 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 750 (mid-ramp) Starting training... [CP ring P2P] OVERLAP: dedicated non-blocking stream validation loss: 3.0712 [CP bwd] overlap=ON step 750 | loss: 2.954028 | lr 1.8000e-06 | norm: 0.1530 | dt: 21471.66ms | tok/sec: 24417.7 | mem: 107435MB | Time Left: 08 hrs : 48 mins [TIMING] data: 7.2ms | fwd: 3943.4ms | loss: 13.2ms | bwd: 10719.7ms | clip: 0.0ms | optim: 24.2ms step 751 | loss: 3.161970 | lr 3.6000e-06 | norm: 0.1821 | dt: 14331.36ms | tok/sec: 36583.3 | mem: 107435MB | Time Left: 05 hrs : 52 mins [TIMING] data: 6.8ms | fwd: 3549.6ms | loss: 13.1ms | bwd: 10724.8ms | clip: 0.0ms | optim: 23.8ms step 752 | loss: 2.582914 | lr 5.4000e-06 | norm: 0.1560 | dt: 14459.85ms | tok/sec: 36258.2 | mem: 107435MB | Time Left: 05 hrs : 55 mins [TIMING] data: 6.7ms | fwd: 3643.5ms | loss: 13.1ms | bwd: 10758.6ms | clip: 0.0ms | optim: 23.8ms step 753 | loss: 2.722396 | lr 7.2000e-06 | norm: 0.1417 | dt: 14472.63ms | tok/sec: 36226.2 | mem: 107435MB | Time Left: 05 hrs : 55 mins [TIMING] data: 6.5ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.8ms step 754 | loss: 3.092211 | lr 9.0000e-06 | norm: 0.1586 | dt: 14571.99ms | tok/sec: 35979.2 | mem: 107435MB | Time Left: 05 hrs : 57 mins [TIMING] data: 6.7ms | fwd: 3631.5ms | loss: 13.1ms | bwd: 10883.7ms | clip: 0.0ms | optim: 23.8ms step 755 | loss: 3.240096 | lr 1.0800e-05 | norm: 0.1663 | dt: 14564.68ms | tok/sec: 35997.2 | mem: 107435MB | Time Left: 05 hrs : 57 mins [TIMING] data: 6.6ms | fwd: 3588.9ms | loss: 13.1ms | bwd: 10919.1ms | clip: 0.0ms | optim: 23.7ms step 756 | loss: 2.918514 | lr 1.2600e-05 | norm: 0.1260 | dt: 14550.79ms | tok/sec: 36031.6 | mem: 107435MB | Time Left: 05 hrs : 56 mins [TIMING] data: 6.5ms | fwd: 3588.9ms | loss: 13.1ms | bwd: 10905.2ms | clip: 0.0ms | optim: 23.8ms step 757 | loss: 2.768518 | lr 1.4400e-05 | norm: 0.1319 | dt: 14514.43ms | tok/sec: 36121.8 | mem: 107435MB | Time Left: 05 hrs : 55 mins [TIMING] data: 6.7ms | fwd: 3598.3ms | loss: 13.0ms | bwd: 10859.3ms | clip: 0.0ms | optim: 23.8ms step 758 | loss: 2.682922 | lr 1.6200e-05 | norm: 0.1442 | dt: 14521.89ms | tok/sec: 36103.3 | mem: 107435MB | Time Left: 05 hrs : 55 mins [TIMING] data: 6.7ms | fwd: 3594.0ms | loss: 13.1ms | bwd: 10870.9ms | clip: 0.0ms | optim: 23.9ms step 759 | loss: 3.084433 | lr 1.8000e-05 | norm: 0.2578 | dt: 14501.41ms | tok/sec: 36154.3 | mem: 107435MB | Time Left: 05 hrs : 54 mins [TIMING] data: 6.5ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10865.0ms | clip: 0.0ms | optim: 23.9ms step 760 | loss: 3.090617 | lr 1.9800e-05 | norm: 0.1488 | dt: 14482.85ms | tok/sec: 36200.6 | mem: 107435MB | Time Left: 05 hrs : 54 mins [TIMING] data: 6.5ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.8ms step 761 | loss: 2.709423 | lr 2.1600e-05 | norm: 0.1968 | dt: 14475.07ms | tok/sec: 36220.1 | mem: 107435MB | Time Left: 05 hrs : 53 mins [TIMING] data: 6.9ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.8ms step 762 | loss: 2.803009 | lr 2.3400e-05 | norm: 0.1644 | dt: 14530.08ms | tok/sec: 36082.9 | mem: 107435MB | Time Left: 05 hrs : 54 mins [TIMING] data: 6.6ms | fwd: 3642.3ms | loss: 13.1ms | bwd: 10831.1ms | clip: 0.0ms | optim: 23.8ms step 763 | loss: 2.581849 | lr 2.5200e-05 | norm: 0.1512 | dt: 14485.62ms | tok/sec: 36193.7 | mem: 107435MB | Time Left: 05 hrs : 53 mins [TIMING] data: 6.8ms | fwd: 3588.4ms | loss: 13.1ms | bwd: 10839.8ms | clip: 0.0ms | optim: 23.9ms step 764 | loss: 2.728063 | lr 2.7000e-05 | norm: 0.1384 | dt: 14487.22ms | tok/sec: 36189.7 | mem: 107435MB | Time Left: 05 hrs : 53 mins [TIMING] data: 6.7ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.8ms step 765 | loss: 2.939891 | lr 2.8800e-05 | norm: 0.1529 | dt: 14494.60ms | tok/sec: 36171.3 | mem: 107435MB | Time Left: 05 hrs : 53 mins [TIMING] data: 6.5ms | fwd: 3603.7ms | loss: 13.1ms | bwd: 10834.2ms | clip: 0.0ms | optim: 23.9ms step 766 | loss: 2.942754 | lr 3.0600e-05 | norm: 0.1718 | dt: 14479.83ms | tok/sec: 36208.2 | mem: 107435MB | Time Left: 05 hrs : 52 mins [TIMING] data: 6.7ms | fwd: 3572.8ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.9ms step 767 | loss: 3.021307 | lr 3.2400e-05 | norm: 0.1802 | dt: 14470.71ms | tok/sec: 36231.0 | mem: 107435MB | Time Left: 05 hrs : 52 mins [TIMING] data: 6.6ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10835.4ms | clip: 0.0ms | optim: 23.8ms step 768 | loss: 2.818981 | lr 3.4200e-05 | norm: 0.1527 | dt: 14474.57ms | tok/sec: 36221.3 | mem: 107435MB | Time Left: 05 hrs : 51 mins [TIMING] data: 6.6ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10838.2ms | clip: 0.0ms | optim: 23.9ms step 769 | loss: 2.896601 | lr 3.6000e-05 | norm: 0.1379 | dt: 14496.02ms | tok/sec: 36167.7 | mem: 107435MB | Time Left: 05 hrs : 52 mins [TIMING] data: 6.5ms | fwd: 3605.2ms | loss: 13.0ms | bwd: 10834.1ms | clip: 0.0ms | optim: 23.9ms step 770 | loss: 2.832743 | lr 3.7800e-05 | norm: 0.1875 | dt: 14474.14ms | tok/sec: 36222.4 | mem: 107435MB | Time Left: 05 hrs : 51 mins [TIMING] data: 6.5ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10833.3ms | clip: 0.0ms | optim: 23.8ms step 771 | loss: 3.000637 | lr 3.9600e-05 | norm: 0.1732 | dt: 14474.20ms | tok/sec: 36222.2 | mem: 107435MB | Time Left: 05 hrs : 51 mins [TIMING] data: 6.6ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10840.1ms | clip: 0.0ms | optim: 23.8ms step 772 | loss: 2.885429 | lr 4.1400e-05 | norm: 0.1665 | dt: 14471.80ms | tok/sec: 36228.2 | mem: 107435MB | Time Left: 05 hrs : 50 mins [TIMING] data: 6.6ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10835.6ms | clip: 0.0ms | optim: 23.8ms step 773 | loss: 2.760646 | lr 4.3200e-05 | norm: 0.1863 | dt: 14476.03ms | tok/sec: 36217.7 | mem: 107435MB | Time Left: 05 hrs : 50 mins [TIMING] data: 6.7ms | fwd: 3579.9ms | loss: 13.0ms | bwd: 10839.2ms | clip: 0.0ms | optim: 23.8ms step 774 | loss: 2.874835 | lr 4.5000e-05 | norm: 0.1408 | dt: 14477.02ms | tok/sec: 36215.2 | mem: 107435MB | Time Left: 05 hrs : 50 mins [TIMING] data: 6.6ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10842.2ms | clip: 0.0ms | optim: 23.9ms step 775 | loss: 2.980758 | lr 4.6800e-05 | norm: 0.1867 | dt: 14472.61ms | tok/sec: 36226.2 | mem: 107435MB | Time Left: 05 hrs : 50 mins [TIMING] data: 7.5ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10839.3ms | clip: 0.0ms | optim: 23.7ms step 776 | loss: 2.509439 | lr 4.8600e-05 | norm: 0.1306 | dt: 14477.92ms | tok/sec: 36212.9 | mem: 107435MB | Time Left: 05 hrs : 50 mins [TIMING] data: 6.6ms | fwd: 3595.2ms | loss: 13.1ms | bwd: 10826.1ms | clip: 0.0ms | optim: 23.8ms step 777 | loss: 3.016775 | lr 5.0400e-05 | norm: 0.1386 | dt: 14480.96ms | tok/sec: 36205.3 | mem: 107435MB | Time Left: 05 hrs : 49 mins [TIMING] data: 6.7ms | fwd: 3572.6ms | loss: 13.0ms | bwd: 10851.4ms | clip: 0.0ms | optim: 24.0ms step 778 | loss: 3.123814 | lr 5.2200e-05 | norm: 0.1525 | dt: 14485.88ms | tok/sec: 36193.0 | mem: 107435MB | Time Left: 05 hrs : 49 mins [TIMING] data: 6.5ms | fwd: 3583.8ms | loss: 13.0ms | bwd: 10845.5ms | clip: 0.0ms | optim: 23.7ms step 779 | loss: 2.933503 | lr 5.4000e-05 | norm: 0.1394 | dt: 14478.11ms | tok/sec: 36212.5 | mem: 107435MB | Time Left: 05 hrs : 49 mins [TIMING] data: 6.7ms | fwd: 3576.4ms | loss: 13.0ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.8ms step 780 | loss: 2.933123 | lr 5.5800e-05 | norm: 0.1481 | dt: 14473.96ms | tok/sec: 36222.9 | mem: 107435MB | Time Left: 05 hrs : 49 mins [TIMING] data: 6.5ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10840.4ms | clip: 0.0ms | optim: 23.9ms step 781 | loss: 2.675422 | lr 5.7600e-05 | norm: 0.1455 | dt: 14476.84ms | tok/sec: 36215.6 | mem: 107435MB | Time Left: 05 hrs : 48 mins [TIMING] data: 6.5ms | fwd: 3576.2ms | loss: 13.0ms | bwd: 10844.0ms | clip: 0.0ms | optim: 23.8ms step 782 | loss: 2.898966 | lr 5.9400e-05 | norm: 0.1347 | dt: 14480.57ms | tok/sec: 36206.3 | mem: 107435MB | Time Left: 05 hrs : 48 mins [TIMING] data: 6.6ms | fwd: 3573.2ms | loss: 13.0ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms step 783 | loss: 3.121679 | lr 6.1200e-05 | norm: 0.1536 | dt: 14474.35ms | tok/sec: 36221.9 | mem: 107435MB | Time Left: 05 hrs : 48 mins [TIMING] data: 6.6ms | fwd: 3577.9ms | loss: 13.0ms | bwd: 10839.9ms | clip: 0.0ms | optim: 23.8ms step 784 | loss: 2.803367 | lr 6.3000e-05 | norm: 0.1553 | dt: 14485.62ms | tok/sec: 36193.7 | mem: 107435MB | Time Left: 05 hrs : 48 mins [TIMING] data: 6.6ms | fwd: 3581.7ms | loss: 13.0ms | bwd: 10847.4ms | clip: 0.0ms | optim: 23.8ms step 785 | loss: 2.744741 | lr 6.4800e-05 | norm: 0.1317 | dt: 14477.35ms | tok/sec: 36214.4 | mem: 107435MB | Time Left: 05 hrs : 47 mins [TIMING] data: 6.6ms | fwd: 3573.7ms | loss: 13.0ms | bwd: 10847.0ms | clip: 0.0ms | optim: 23.8ms step 786 | loss: 2.934856 | lr 6.6600e-05 | norm: 0.1383 | dt: 14469.42ms | tok/sec: 36234.2 | mem: 107435MB | Time Left: 05 hrs : 47 mins [TIMING] data: 6.6ms | fwd: 3573.2ms | loss: 13.0ms | bwd: 10839.5ms | clip: 0.0ms | optim: 23.8ms step 787 | loss: 2.906707 | lr 6.8400e-05 | norm: 0.1335 | dt: 14480.09ms | tok/sec: 36207.5 | mem: 107435MB | Time Left: 05 hrs : 47 mins [TIMING] data: 6.7ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10844.8ms | clip: 0.0ms | optim: 23.9ms step 788 | loss: 3.518390 | lr 7.0200e-05 | norm: 0.1934 | dt: 14469.09ms | tok/sec: 36235.0 | mem: 107435MB | Time Left: 05 hrs : 47 mins [TIMING] data: 6.8ms | fwd: 3585.0ms | loss: 13.1ms | bwd: 10827.0ms | clip: 0.0ms | optim: 23.9ms step 789 | loss: 2.640352 | lr 7.2000e-05 | norm: 1.4532 | dt: 14467.13ms | tok/sec: 36240.0 | mem: 107435MB | Time Left: 05 hrs : 46 mins [TIMING] data: 6.5ms | fwd: 3572.8ms | loss: 13.1ms | bwd: 10837.6ms | clip: 0.0ms | optim: 23.8ms step 790 | loss: 2.776472 | lr 7.3800e-05 | norm: 0.2565 | dt: 14475.29ms | tok/sec: 36219.5 | mem: 107435MB | Time Left: 05 hrs : 46 mins [TIMING] data: 6.6ms | fwd: 3581.2ms | loss: 13.0ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.9ms step 791 | loss: 2.930469 | lr 7.5600e-05 | norm: 0.1839 | dt: 14477.24ms | tok/sec: 36214.6 | mem: 107435MB | Time Left: 05 hrs : 46 mins [TIMING] data: 6.6ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10841.2ms | clip: 0.0ms | optim: 23.9ms step 792 | loss: 3.028378 | lr 7.7400e-05 | norm: 0.1638 | dt: 14474.49ms | tok/sec: 36221.5 | mem: 107435MB | Time Left: 05 hrs : 46 mins [TIMING] data: 6.6ms | fwd: 3572.4ms | loss: 13.0ms | bwd: 10845.4ms | clip: 0.0ms | optim: 23.9ms step 793 | loss: 3.392564 | lr 7.9200e-05 | norm: 0.1799 | dt: 14485.80ms | tok/sec: 36193.2 | mem: 107435MB | Time Left: 05 hrs : 46 mins [TIMING] data: 6.4ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 23.9ms step 794 | loss: 2.590893 | lr 8.1000e-05 | norm: 0.1481 | dt: 14476.79ms | tok/sec: 36215.8 | mem: 107435MB | Time Left: 05 hrs : 45 mins [TIMING] data: 6.6ms | fwd: 3590.0ms | loss: 13.0ms | bwd: 10830.1ms | clip: 0.0ms | optim: 23.8ms step 795 | loss: 3.168222 | lr 8.2800e-05 | norm: 0.2106 | dt: 14480.05ms | tok/sec: 36207.6 | mem: 107435MB | Time Left: 05 hrs : 45 mins [TIMING] data: 6.7ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10846.7ms | clip: 0.0ms | optim: 24.1ms step 796 | loss: 2.970008 | lr 8.4600e-05 | norm: 0.2086 | dt: 14467.07ms | tok/sec: 36240.1 | mem: 107435MB | Time Left: 05 hrs : 45 mins [TIMING] data: 6.6ms | fwd: 3594.2ms | loss: 13.1ms | bwd: 10816.2ms | clip: 0.0ms | optim: 23.8ms step 797 | loss: 3.063489 | lr 8.6400e-05 | norm: 0.1840 | dt: 14462.85ms | tok/sec: 36250.7 | mem: 107435MB | Time Left: 05 hrs : 44 mins [TIMING] data: 6.6ms | fwd: 3585.2ms | loss: 13.1ms | bwd: 10820.9ms | clip: 0.0ms | optim: 23.9ms step 798 | loss: 2.673750 | lr 8.8200e-05 | norm: 0.1714 | dt: 14497.38ms | tok/sec: 36164.3 | mem: 107435MB | Time Left: 05 hrs : 45 mins [TIMING] data: 6.5ms | fwd: 3604.1ms | loss: 13.1ms | bwd: 10836.6ms | clip: 0.0ms | optim: 23.8ms step 799 | loss: 2.586432 | lr 9.0000e-05 | norm: 0.2715 | dt: 14476.17ms | tok/sec: 36217.3 | mem: 107435MB | Time Left: 05 hrs : 44 mins [TIMING] data: 6.6ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10838.1ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_800.ckpt (safely) step 800 | loss: 2.573454 | lr 9.1800e-05 | norm: 0.1645 | dt: 14379.99ms | tok/sec: 36459.6 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.4ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10743.7ms | clip: 0.0ms | optim: 23.9ms step 801 | loss: 2.925294 | lr 9.3600e-05 | norm: 0.1646 | dt: 14410.59ms | tok/sec: 36382.1 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.4ms | fwd: 3565.9ms | loss: 13.1ms | bwd: 10787.1ms | clip: 0.0ms | optim: 23.8ms step 802 | loss: 2.820365 | lr 9.5400e-05 | norm: 0.1633 | dt: 14418.33ms | tok/sec: 36362.6 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.4ms | fwd: 3575.0ms | loss: 13.1ms | bwd: 10785.7ms | clip: 0.0ms | optim: 23.8ms step 803 | loss: 2.848994 | lr 9.7200e-05 | norm: 0.1623 | dt: 14435.50ms | tok/sec: 36319.4 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.4ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10799.9ms | clip: 0.0ms | optim: 23.8ms step 804 | loss: 3.027385 | lr 9.9000e-05 | norm: 0.1636 | dt: 14440.72ms | tok/sec: 36306.2 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.3ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10799.9ms | clip: 0.0ms | optim: 23.8ms step 805 | loss: 2.917511 | lr 1.0080e-04 | norm: 0.1559 | dt: 14449.67ms | tok/sec: 36283.7 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.3ms | fwd: 3584.7ms | loss: 13.1ms | bwd: 10807.4ms | clip: 0.0ms | optim: 23.8ms step 806 | loss: 2.676723 | lr 1.0260e-04 | norm: 0.1672 | dt: 14450.31ms | tok/sec: 36282.1 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.4ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10816.3ms | clip: 0.0ms | optim: 24.0ms step 807 | loss: 2.677406 | lr 1.0440e-04 | norm: 0.1682 | dt: 14465.53ms | tok/sec: 36244.0 | mem: 107435MB | Time Left: 05 hrs : 42 mins [TIMING] data: 7.5ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10827.3ms | clip: 0.0ms | optim: 23.8ms step 808 | loss: 3.027586 | lr 1.0620e-04 | norm: 0.1685 | dt: 14447.01ms | tok/sec: 36290.4 | mem: 107435MB | Time Left: 05 hrs : 41 mins [TIMING] data: 7.5ms | fwd: 3570.0ms | loss: 13.1ms | bwd: 10818.9ms | clip: 0.0ms | optim: 23.8ms step 809 | loss: 3.183782 | lr 1.0800e-04 | norm: 0.1864 | dt: 14466.20ms | tok/sec: 36242.3 | mem: 107435MB | Time Left: 05 hrs : 41 mins [TIMING] data: 7.4ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10830.3ms | clip: 0.0ms | optim: 23.8ms step 810 | loss: 2.659116 | lr 1.0980e-04 | norm: 0.9178 | dt: 14465.24ms | tok/sec: 36244.7 | mem: 107435MB | Time Left: 05 hrs : 41 mins [TIMING] data: 7.3ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10830.2ms | clip: 0.0ms | optim: 23.8ms step 811 | loss: 2.959210 | lr 1.1160e-04 | norm: 0.1782 | dt: 14458.82ms | tok/sec: 36260.8 | mem: 107435MB | Time Left: 05 hrs : 41 mins [TIMING] data: 7.4ms | fwd: 3573.3ms | loss: 13.1ms | bwd: 10828.0ms | clip: 0.0ms | optim: 23.7ms step 812 | loss: 2.910862 | lr 1.1340e-04 | norm: 0.1714 | dt: 14462.65ms | tok/sec: 36251.2 | mem: 107435MB | Time Left: 05 hrs : 41 mins [TIMING] data: 7.3ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10827.6ms | clip: 0.0ms | optim: 23.8ms step 813 | loss: 3.297562 | lr 1.1520e-04 | norm: 0.2050 | dt: 14476.48ms | tok/sec: 36216.5 | mem: 107435MB | Time Left: 05 hrs : 41 mins [TIMING] data: 7.4ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10841.4ms | clip: 0.0ms | optim: 23.8ms step 814 | loss: 2.967859 | lr 1.1700e-04 | norm: 0.1702 | dt: 14468.08ms | tok/sec: 36237.6 | mem: 107435MB | Time Left: 05 hrs : 40 mins [TIMING] data: 7.3ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10834.6ms | clip: 0.0ms | optim: 23.9ms step 815 | loss: 2.812455 | lr 1.1880e-04 | norm: 0.1885 | dt: 14468.39ms | tok/sec: 36236.8 | mem: 107435MB | Time Left: 05 hrs : 40 mins [TIMING] data: 7.5ms | fwd: 3573.1ms | loss: 13.1ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.8ms step 816 | loss: 2.702981 | lr 1.2060e-04 | norm: 0.2083 | dt: 14473.97ms | tok/sec: 36222.8 | mem: 107435MB | Time Left: 05 hrs : 40 mins [TIMING] data: 7.3ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10838.8ms | clip: 0.0ms | optim: 23.8ms step 817 | loss: 2.911349 | lr 1.2240e-04 | norm: 0.1811 | dt: 14459.60ms | tok/sec: 36258.8 | mem: 107435MB | Time Left: 05 hrs : 39 mins [TIMING] data: 7.4ms | fwd: 3572.6ms | loss: 13.1ms | bwd: 10829.4ms | clip: 0.0ms | optim: 23.8ms step 818 | loss: 2.793933 | lr 1.2420e-04 | norm: 0.1816 | dt: 14460.59ms | tok/sec: 36256.3 | mem: 107435MB | Time Left: 05 hrs : 39 mins [TIMING] data: 7.6ms | fwd: 3574.5ms | loss: 13.1ms | bwd: 10828.0ms | clip: 0.0ms | optim: 23.8ms step 819 | loss: 3.002342 | lr 1.2600e-04 | norm: 0.1509 | dt: 14473.39ms | tok/sec: 36224.3 | mem: 107435MB | Time Left: 05 hrs : 39 mins [TIMING] data: 7.2ms | fwd: 3578.9ms | loss: 13.2ms | bwd: 10836.9ms | clip: 0.0ms | optim: 23.9ms step 820 | loss: 3.101271 | lr 1.2780e-04 | norm: 1.3645 | dt: 14469.58ms | tok/sec: 36233.8 | mem: 107435MB | Time Left: 05 hrs : 39 mins [TIMING] data: 7.7ms | fwd: 3570.8ms | loss: 13.1ms | bwd: 10840.8ms | clip: 0.0ms | optim: 23.8ms step 821 | loss: 2.341957 | lr 1.2960e-04 | norm: 0.1948 | dt: 14471.69ms | tok/sec: 36228.5 | mem: 107435MB | Time Left: 05 hrs : 39 mins [TIMING] data: 7.3ms | fwd: 3572.0ms | loss: 13.1ms | bwd: 10842.0ms | clip: 0.0ms | optim: 24.0ms step 822 | loss: 2.509327 | lr 1.3140e-04 | norm: 0.1663 | dt: 14474.43ms | tok/sec: 36221.7 | mem: 107435MB | Time Left: 05 hrs : 38 mins [TIMING] data: 7.4ms | fwd: 3592.8ms | loss: 13.2ms | bwd: 10823.9ms | clip: 0.0ms | optim: 23.7ms step 823 | loss: 2.859375 | lr 1.3320e-04 | norm: 0.1572 | dt: 14469.69ms | tok/sec: 36233.5 | mem: 107435MB | Time Left: 05 hrs : 38 mins [TIMING] data: 7.4ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10830.6ms | clip: 0.0ms | optim: 23.8ms step 824 | loss: 2.873102 | lr 1.3500e-04 | norm: 0.1956 | dt: 14493.98ms | tok/sec: 36172.8 | mem: 107435MB | Time Left: 05 hrs : 38 mins [TIMING] data: 7.2ms | fwd: 3593.9ms | loss: 13.0ms | bwd: 10842.9ms | clip: 0.0ms | optim: 23.9ms step 825 | loss: 2.881723 | lr 1.3680e-04 | norm: 0.1830 | dt: 14486.40ms | tok/sec: 36191.7 | mem: 107435MB | Time Left: 05 hrs : 38 mins [TIMING] data: 7.5ms | fwd: 3579.9ms | loss: 13.0ms | bwd: 10848.4ms | clip: 0.0ms | optim: 24.0ms step 826 | loss: 3.032596 | lr 1.3860e-04 | norm: 0.2075 | dt: 14491.66ms | tok/sec: 36178.6 | mem: 107435MB | Time Left: 05 hrs : 38 mins [TIMING] data: 7.5ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10849.1ms | clip: 0.0ms | optim: 24.5ms step 827 | loss: 3.085069 | lr 1.4040e-04 | norm: 0.2024 | dt: 14473.89ms | tok/sec: 36223.0 | mem: 107435MB | Time Left: 05 hrs : 37 mins [TIMING] data: 7.2ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10836.6ms | clip: 0.0ms | optim: 23.8ms step 828 | loss: 2.735523 | lr 1.4220e-04 | norm: 0.2192 | dt: 14536.52ms | tok/sec: 36066.9 | mem: 107435MB | Time Left: 05 hrs : 38 mins [TIMING] data: 7.6ms | fwd: 3659.2ms | loss: 13.1ms | bwd: 10819.7ms | clip: 0.0ms | optim: 23.8ms step 829 | loss: 2.712095 | lr 1.4400e-04 | norm: 0.2423 | dt: 14477.62ms | tok/sec: 36213.7 | mem: 107435MB | Time Left: 05 hrs : 37 mins [TIMING] data: 7.3ms | fwd: 3576.0ms | loss: 13.0ms | bwd: 10844.3ms | clip: 0.0ms | optim: 23.8ms step 830 | loss: 2.964975 | lr 1.4580e-04 | norm: 0.1866 | dt: 14481.92ms | tok/sec: 36202.9 | mem: 107435MB | Time Left: 05 hrs : 37 mins [TIMING] data: 7.3ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10843.9ms | clip: 0.0ms | optim: 23.9ms step 831 | loss: 2.624734 | lr 1.4760e-04 | norm: 0.1981 | dt: 14467.67ms | tok/sec: 36238.6 | mem: 107435MB | Time Left: 05 hrs : 36 mins [TIMING] data: 7.4ms | fwd: 3580.6ms | loss: 13.0ms | bwd: 10829.6ms | clip: 0.0ms | optim: 23.9ms step 832 | loss: 3.032712 | lr 1.4940e-04 | norm: 0.1931 | dt: 14470.27ms | tok/sec: 36232.1 | mem: 107435MB | Time Left: 05 hrs : 36 mins [TIMING] data: 7.3ms | fwd: 3575.2ms | loss: 13.0ms | bwd: 10837.8ms | clip: 0.0ms | optim: 23.8ms step 833 | loss: 2.995473 | lr 1.5120e-04 | norm: 0.3012 | dt: 14477.98ms | tok/sec: 36212.8 | mem: 107435MB | Time Left: 05 hrs : 36 mins [TIMING] data: 7.2ms | fwd: 3578.8ms | loss: 13.0ms | bwd: 10841.8ms | clip: 0.0ms | optim: 24.0ms step 834 | loss: 2.771485 | lr 1.5300e-04 | norm: 0.2168 | dt: 14481.82ms | tok/sec: 36203.2 | mem: 107435MB | Time Left: 05 hrs : 36 mins [TIMING] data: 7.2ms | fwd: 3577.4ms | loss: 13.0ms | bwd: 10847.2ms | clip: 0.0ms | optim: 23.7ms step 835 | loss: 2.866414 | lr 1.5480e-04 | norm: 0.1933 | dt: 14462.26ms | tok/sec: 36252.2 | mem: 107435MB | Time Left: 05 hrs : 35 mins [TIMING] data: 7.3ms | fwd: 3574.8ms | loss: 13.0ms | bwd: 10830.3ms | clip: 0.0ms | optim: 23.8ms step 836 | loss: 2.793266 | lr 1.5660e-04 | norm: 0.1711 | dt: 14474.44ms | tok/sec: 36221.6 | mem: 107435MB | Time Left: 05 hrs : 35 mins [TIMING] data: 7.3ms | fwd: 3578.2ms | loss: 13.0ms | bwd: 10838.9ms | clip: 0.0ms | optim: 23.8ms step 837 | loss: 3.148466 | lr 1.5840e-04 | norm: 0.2583 | dt: 14470.30ms | tok/sec: 36232.0 | mem: 107435MB | Time Left: 05 hrs : 35 mins [TIMING] data: 7.3ms | fwd: 3578.9ms | loss: 13.0ms | bwd: 10834.1ms | clip: 0.0ms | optim: 23.8ms step 838 | loss: 2.918412 | lr 1.6020e-04 | norm: 0.1685 | dt: 14461.04ms | tok/sec: 36255.2 | mem: 107435MB | Time Left: 05 hrs : 34 mins [TIMING] data: 7.5ms | fwd: 3573.4ms | loss: 13.1ms | bwd: 10830.1ms | clip: 0.0ms | optim: 23.8ms step 839 | loss: 2.916226 | lr 1.6200e-04 | norm: 0.1810 | dt: 14479.92ms | tok/sec: 36207.9 | mem: 107435MB | Time Left: 05 hrs : 34 mins [TIMING] data: 7.5ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10842.3ms | clip: 0.0ms | optim: 24.0ms step 840 | loss: 2.642492 | lr 1.6380e-04 | norm: 0.1710 | dt: 14470.41ms | tok/sec: 36231.7 | mem: 107435MB | Time Left: 05 hrs : 34 mins [TIMING] data: 7.6ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.7ms step 841 | loss: 2.784270 | lr 1.6560e-04 | norm: 0.1847 | dt: 14476.54ms | tok/sec: 36216.4 | mem: 107435MB | Time Left: 05 hrs : 34 mins [TIMING] data: 7.4ms | fwd: 3576.1ms | loss: 13.0ms | bwd: 10842.9ms | clip: 0.0ms | optim: 23.9ms step 842 | loss: 2.736636 | lr 1.6740e-04 | norm: 0.1866 | dt: 14476.90ms | tok/sec: 36215.5 | mem: 107435MB | Time Left: 05 hrs : 34 mins [TIMING] data: 7.2ms | fwd: 3574.3ms | loss: 13.1ms | bwd: 10845.1ms | clip: 0.0ms | optim: 23.8ms step 843 | loss: 2.617802 | lr 1.6920e-04 | norm: 0.1582 | dt: 14477.17ms | tok/sec: 36214.8 | mem: 107435MB | Time Left: 05 hrs : 33 mins [TIMING] data: 7.4ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10841.5ms | clip: 0.0ms | optim: 23.8ms step 844 | loss: 2.927630 | lr 1.7100e-04 | norm: 0.1684 | dt: 14483.59ms | tok/sec: 36198.8 | mem: 107435MB | Time Left: 05 hrs : 33 mins [TIMING] data: 7.3ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10844.2ms | clip: 0.0ms | optim: 23.8ms step 845 | loss: 2.698425 | lr 1.7280e-04 | norm: 0.4248 | dt: 14476.16ms | tok/sec: 36217.3 | mem: 107435MB | Time Left: 05 hrs : 33 mins [TIMING] data: 7.5ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10840.2ms | clip: 0.0ms | optim: 23.8ms step 846 | loss: 2.851482 | lr 1.7460e-04 | norm: 0.2002 | dt: 14480.20ms | tok/sec: 36207.2 | mem: 107435MB | Time Left: 05 hrs : 33 mins [TIMING] data: 7.3ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10843.0ms | clip: 0.0ms | optim: 23.8ms step 847 | loss: 3.047212 | lr 1.7640e-04 | norm: 0.3471 | dt: 14516.31ms | tok/sec: 36117.2 | mem: 107435MB | Time Left: 05 hrs : 33 mins [TIMING] data: 7.4ms | fwd: 3632.4ms | loss: 13.1ms | bwd: 10825.7ms | clip: 0.0ms | optim: 24.3ms step 848 | loss: 2.775957 | lr 1.7820e-04 | norm: 0.1814 | dt: 14478.23ms | tok/sec: 36212.2 | mem: 107435MB | Time Left: 05 hrs : 32 mins [TIMING] data: 7.5ms | fwd: 3572.4ms | loss: 13.1ms | bwd: 10847.9ms | clip: 0.0ms | optim: 23.9ms step 849 | loss: 2.965243 | lr 1.8000e-04 | norm: 0.2378 | dt: 14459.54ms | tok/sec: 36259.0 | mem: 107435MB | Time Left: 05 hrs : 32 mins [TIMING] data: 7.3ms | fwd: 3584.0ms | loss: 13.0ms | bwd: 10818.3ms | clip: 0.0ms | optim: 23.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_850.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.ckpt step 850 | loss: 2.945216 | lr 1.8000e-04 | norm: 0.1767 | dt: 14363.97ms | tok/sec: 36500.2 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 8.0ms | fwd: 3566.0ms | loss: 13.2ms | bwd: 10738.5ms | clip: 0.0ms | optim: 23.8ms step 851 | loss: 2.986545 | lr 1.8000e-04 | norm: 0.1833 | dt: 14414.72ms | tok/sec: 36371.7 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.2ms | fwd: 3561.0ms | loss: 13.0ms | bwd: 10796.2ms | clip: 0.0ms | optim: 23.8ms step 852 | loss: 3.035398 | lr 1.8000e-04 | norm: 0.1843 | dt: 14422.92ms | tok/sec: 36351.0 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.5ms | fwd: 3568.7ms | loss: 13.1ms | bwd: 10796.0ms | clip: 0.0ms | optim: 23.7ms step 853 | loss: 2.805237 | lr 1.8000e-04 | norm: 1.5532 | dt: 14422.66ms | tok/sec: 36351.7 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.2ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10786.8ms | clip: 0.0ms | optim: 23.9ms step 854 | loss: 2.974912 | lr 1.8000e-04 | norm: 0.2100 | dt: 14457.99ms | tok/sec: 36262.8 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.1ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10821.9ms | clip: 0.0ms | optim: 23.8ms step 855 | loss: 2.705589 | lr 1.8000e-04 | norm: 0.1794 | dt: 14442.63ms | tok/sec: 36301.4 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.3ms | fwd: 3583.2ms | loss: 13.0ms | bwd: 10801.7ms | clip: 0.0ms | optim: 23.8ms step 856 | loss: 3.066740 | lr 1.8000e-04 | norm: 0.1907 | dt: 14460.79ms | tok/sec: 36255.8 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.3ms | fwd: 3589.1ms | loss: 13.0ms | bwd: 10814.0ms | clip: 0.0ms | optim: 23.8ms step 857 | loss: 2.828382 | lr 1.8000e-04 | norm: 0.1737 | dt: 14466.70ms | tok/sec: 36241.0 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.1ms | fwd: 3584.3ms | loss: 13.0ms | bwd: 10825.2ms | clip: 0.0ms | optim: 23.8ms step 858 | loss: 2.707695 | lr 1.8000e-04 | norm: 0.1752 | dt: 14474.71ms | tok/sec: 36221.0 | mem: 107435MB | Time Left: 05 hrs : 30 mins [TIMING] data: 7.4ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10834.7ms | clip: 0.0ms | optim: 23.7ms step 859 | loss: 2.884325 | lr 1.8000e-04 | norm: 0.2174 | dt: 14464.92ms | tok/sec: 36245.5 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 7.1ms | fwd: 3571.6ms | loss: 13.1ms | bwd: 10835.8ms | clip: 0.0ms | optim: 24.0ms step 860 | loss: 2.846708 | lr 1.8000e-04 | norm: 0.2879 | dt: 14472.80ms | tok/sec: 36225.7 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 7.3ms | fwd: 3575.3ms | loss: 13.0ms | bwd: 10840.1ms | clip: 0.0ms | optim: 23.8ms step 861 | loss: 3.098118 | lr 1.8000e-04 | norm: 0.2033 | dt: 14470.11ms | tok/sec: 36232.5 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 7.3ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10831.4ms | clip: 0.0ms | optim: 23.8ms step 862 | loss: 2.783081 | lr 1.8000e-04 | norm: 0.1933 | dt: 14482.48ms | tok/sec: 36201.5 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 7.2ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10847.5ms | clip: 0.0ms | optim: 23.9ms step 863 | loss: 2.565199 | lr 1.8000e-04 | norm: 0.1921 | dt: 14500.39ms | tok/sec: 36156.8 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 7.1ms | fwd: 3619.5ms | loss: 13.1ms | bwd: 10823.5ms | clip: 0.0ms | optim: 23.8ms step 864 | loss: 3.193242 | lr 1.8000e-04 | norm: 0.2755 | dt: 14514.69ms | tok/sec: 36121.2 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 7.3ms | fwd: 3634.1ms | loss: 13.1ms | bwd: 10823.2ms | clip: 0.0ms | optim: 23.8ms step 865 | loss: 2.981857 | lr 1.8000e-04 | norm: 0.1920 | dt: 14474.00ms | tok/sec: 36222.7 | mem: 107435MB | Time Left: 05 hrs : 28 mins [TIMING] data: 7.3ms | fwd: 3583.7ms | loss: 13.0ms | bwd: 10832.7ms | clip: 0.0ms | optim: 24.0ms step 866 | loss: 2.954495 | lr 1.8000e-04 | norm: 0.2377 | dt: 14536.09ms | tok/sec: 36068.0 | mem: 107435MB | Time Left: 05 hrs : 29 mins [TIMING] data: 7.4ms | fwd: 3639.7ms | loss: 13.1ms | bwd: 10838.6ms | clip: 0.0ms | optim: 23.7ms step 867 | loss: 2.858272 | lr 1.8000e-04 | norm: 0.1912 | dt: 14476.99ms | tok/sec: 36215.3 | mem: 107435MB | Time Left: 05 hrs : 28 mins [TIMING] data: 7.2ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10843.4ms | clip: 0.0ms | optim: 23.7ms step 868 | loss: 3.014984 | lr 1.8000e-04 | norm: 0.2078 | dt: 14508.28ms | tok/sec: 36137.2 | mem: 107435MB | Time Left: 05 hrs : 28 mins [TIMING] data: 7.0ms | fwd: 3621.2ms | loss: 13.1ms | bwd: 10829.9ms | clip: 0.0ms | optim: 23.7ms step 869 | loss: 2.912760 | lr 1.8000e-04 | norm: 0.1879 | dt: 14512.93ms | tok/sec: 36125.6 | mem: 107435MB | Time Left: 05 hrs : 28 mins [TIMING] data: 7.3ms | fwd: 3633.6ms | loss: 13.1ms | bwd: 10821.7ms | clip: 0.0ms | optim: 23.8ms step 870 | loss: 2.864514 | lr 1.8000e-04 | norm: 0.2126 | dt: 14480.87ms | tok/sec: 36205.5 | mem: 107435MB | Time Left: 05 hrs : 27 mins [TIMING] data: 7.3ms | fwd: 3586.5ms | loss: 13.0ms | bwd: 10837.1ms | clip: 0.0ms | optim: 23.8ms step 871 | loss: 2.975680 | lr 1.8000e-04 | norm: 0.1706 | dt: 14520.32ms | tok/sec: 36107.2 | mem: 107435MB | Time Left: 05 hrs : 28 mins [TIMING] data: 7.0ms | fwd: 3634.3ms | loss: 13.1ms | bwd: 10828.6ms | clip: 0.0ms | optim: 23.8ms step 872 | loss: 2.776859 | lr 1.8000e-04 | norm: 0.2342 | dt: 14479.48ms | tok/sec: 36209.0 | mem: 107435MB | Time Left: 05 hrs : 26 mins [TIMING] data: 7.5ms | fwd: 3576.6ms | loss: 13.0ms | bwd: 10844.7ms | clip: 0.0ms | optim: 23.8ms step 873 | loss: 2.855322 | lr 1.8000e-04 | norm: 0.2098 | dt: 14481.05ms | tok/sec: 36205.1 | mem: 107435MB | Time Left: 05 hrs : 26 mins [TIMING] data: 7.1ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10844.5ms | clip: 0.0ms | optim: 23.8ms step 874 | loss: 2.760797 | lr 1.8000e-04 | norm: 0.1938 | dt: 14482.68ms | tok/sec: 36201.0 | mem: 107435MB | Time Left: 05 hrs : 26 mins [TIMING] data: 7.2ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10846.2ms | clip: 0.0ms | optim: 23.8ms step 875 | loss: 2.907828 | lr 1.8000e-04 | norm: 0.2197 | dt: 14466.70ms | tok/sec: 36241.0 | mem: 107435MB | Time Left: 05 hrs : 25 mins [TIMING] data: 7.1ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10832.0ms | clip: 0.0ms | optim: 23.8ms step 876 | loss: 2.965818 | lr 1.8000e-04 | norm: 0.2275 | dt: 14479.62ms | tok/sec: 36208.7 | mem: 107435MB | Time Left: 05 hrs : 26 mins [TIMING] data: 7.0ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10842.7ms | clip: 0.0ms | optim: 23.7ms step 877 | loss: 2.890493 | lr 1.8000e-04 | norm: 0.1885 | dt: 14469.14ms | tok/sec: 36234.9 | mem: 107435MB | Time Left: 05 hrs : 25 mins [TIMING] data: 7.1ms | fwd: 3579.6ms | loss: 13.0ms | bwd: 10832.3ms | clip: 0.0ms | optim: 23.9ms step 878 | loss: 2.979156 | lr 1.8000e-04 | norm: 0.2340 | dt: 14474.34ms | tok/sec: 36221.9 | mem: 107435MB | Time Left: 05 hrs : 25 mins [TIMING] data: 7.0ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10838.6ms | clip: 0.0ms | optim: 23.8ms step 879 | loss: 3.131982 | lr 1.8000e-04 | norm: 0.2746 | dt: 14482.21ms | tok/sec: 36202.2 | mem: 107435MB | Time Left: 05 hrs : 25 mins [TIMING] data: 7.0ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10844.5ms | clip: 0.0ms | optim: 23.8ms step 880 | loss: 2.846823 | lr 1.8000e-04 | norm: 0.2015 | dt: 14469.53ms | tok/sec: 36233.9 | mem: 107435MB | Time Left: 05 hrs : 24 mins [TIMING] data: 7.1ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10833.9ms | clip: 0.0ms | optim: 23.8ms step 881 | loss: 2.735988 | lr 1.8000e-04 | norm: 1.0590 | dt: 14479.74ms | tok/sec: 36208.4 | mem: 107435MB | Time Left: 05 hrs : 24 mins [TIMING] data: 7.1ms | fwd: 3575.5ms | loss: 13.0ms | bwd: 10846.8ms | clip: 0.0ms | optim: 24.1ms step 882 | loss: 2.824592 | lr 1.8000e-04 | norm: 0.2904 | dt: 14463.83ms | tok/sec: 36248.2 | mem: 107435MB | Time Left: 05 hrs : 24 mins [TIMING] data: 7.2ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10828.1ms | clip: 0.0ms | optim: 23.7ms step 883 | loss: 2.862134 | lr 1.8000e-04 | norm: 0.1966 | dt: 14476.05ms | tok/sec: 36217.6 | mem: 107435MB | Time Left: 05 hrs : 24 mins [TIMING] data: 7.0ms | fwd: 3580.4ms | loss: 13.0ms | bwd: 10838.4ms | clip: 0.0ms | optim: 23.8ms step 884 | loss: 2.945449 | lr 1.8000e-04 | norm: 0.2075 | dt: 14465.40ms | tok/sec: 36244.3 | mem: 107435MB | Time Left: 05 hrs : 23 mins [TIMING] data: 7.3ms | fwd: 3570.3ms | loss: 13.1ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.9ms step 885 | loss: 2.781929 | lr 1.8000e-04 | norm: 0.1959 | dt: 14465.21ms | tok/sec: 36244.8 | mem: 107435MB | Time Left: 05 hrs : 23 mins [TIMING] data: 7.0ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10830.4ms | clip: 0.0ms | optim: 23.9ms step 886 | loss: 2.893898 | lr 1.8000e-04 | norm: 0.1964 | dt: 14465.92ms | tok/sec: 36243.0 | mem: 107435MB | Time Left: 05 hrs : 23 mins [TIMING] data: 7.3ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10828.7ms | clip: 0.0ms | optim: 24.0ms step 887 | loss: 2.824120 | lr 1.8000e-04 | norm: 0.2462 | dt: 14466.98ms | tok/sec: 36240.3 | mem: 107435MB | Time Left: 05 hrs : 23 mins [TIMING] data: 7.3ms | fwd: 3575.7ms | loss: 13.0ms | bwd: 10833.5ms | clip: 0.0ms | optim: 23.7ms step 888 | loss: 2.795377 | lr 1.8000e-04 | norm: 0.2135 | dt: 14465.04ms | tok/sec: 36245.2 | mem: 107435MB | Time Left: 05 hrs : 22 mins [TIMING] data: 7.3ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10829.0ms | clip: 0.0ms | optim: 23.8ms step 889 | loss: 2.520073 | lr 1.8000e-04 | norm: 0.1850 | dt: 14475.30ms | tok/sec: 36219.5 | mem: 107435MB | Time Left: 05 hrs : 22 mins [TIMING] data: 7.6ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10842.3ms | clip: 0.0ms | optim: 23.9ms step 890 | loss: 3.061476 | lr 1.8000e-04 | norm: 0.1803 | dt: 14478.46ms | tok/sec: 36211.6 | mem: 107435MB | Time Left: 05 hrs : 22 mins [TIMING] data: 7.5ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10842.4ms | clip: 0.0ms | optim: 23.8ms step 891 | loss: 3.023923 | lr 1.8000e-04 | norm: 0.1821 | dt: 14473.65ms | tok/sec: 36223.6 | mem: 107435MB | Time Left: 05 hrs : 22 mins [TIMING] data: 7.8ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10839.4ms | clip: 0.0ms | optim: 23.8ms step 892 | loss: 3.023176 | lr 1.8000e-04 | norm: 0.3650 | dt: 14474.32ms | tok/sec: 36221.9 | mem: 107435MB | Time Left: 05 hrs : 22 mins [TIMING] data: 7.3ms | fwd: 3579.2ms | loss: 13.2ms | bwd: 10837.1ms | clip: 0.0ms | optim: 23.9ms step 893 | loss: 3.090324 | lr 1.8000e-04 | norm: 0.1709 | dt: 14470.37ms | tok/sec: 36231.8 | mem: 107435MB | Time Left: 05 hrs : 21 mins [TIMING] data: 7.6ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10837.3ms | clip: 0.0ms | optim: 23.8ms step 894 | loss: 2.482926 | lr 1.8000e-04 | norm: 0.1700 | dt: 14463.74ms | tok/sec: 36248.4 | mem: 107435MB | Time Left: 05 hrs : 21 mins [TIMING] data: 7.2ms | fwd: 3571.6ms | loss: 13.1ms | bwd: 10834.6ms | clip: 0.0ms | optim: 23.8ms step 895 | loss: 3.130907 | lr 1.8000e-04 | norm: 0.2190 | dt: 14466.09ms | tok/sec: 36242.5 | mem: 107435MB | Time Left: 05 hrs : 21 mins [TIMING] data: 7.5ms | fwd: 3578.9ms | loss: 13.0ms | bwd: 10829.5ms | clip: 0.0ms | optim: 24.0ms step 896 | loss: 3.174737 | lr 1.8000e-04 | norm: 0.2150 | dt: 14463.63ms | tok/sec: 36248.7 | mem: 107435MB | Time Left: 05 hrs : 20 mins [TIMING] data: 7.1ms | fwd: 3578.4ms | loss: 13.0ms | bwd: 10828.0ms | clip: 0.0ms | optim: 24.0ms step 897 | loss: 2.794674 | lr 1.8000e-04 | norm: 0.1929 | dt: 14473.99ms | tok/sec: 36222.8 | mem: 107435MB | Time Left: 05 hrs : 20 mins [TIMING] data: 7.4ms | fwd: 3574.4ms | loss: 13.0ms | bwd: 10842.1ms | clip: 0.0ms | optim: 23.8ms step 898 | loss: 2.681480 | lr 1.8000e-04 | norm: 0.1917 | dt: 14466.45ms | tok/sec: 36241.6 | mem: 107435MB | Time Left: 05 hrs : 20 mins [TIMING] data: 7.7ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10832.0ms | clip: 0.0ms | optim: 23.9ms step 899 | loss: 2.797155 | lr 1.8000e-04 | norm: 0.2367 | dt: 14462.34ms | tok/sec: 36251.9 | mem: 107435MB | Time Left: 05 hrs : 20 mins [TIMING] data: 7.4ms | fwd: 3572.6ms | loss: 13.1ms | bwd: 10832.1ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_900.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_800.ckpt step 900 | loss: 3.175222 | lr 1.8000e-04 | norm: 0.4209 | dt: 14367.49ms | tok/sec: 36491.3 | mem: 107435MB | Time Left: 05 hrs : 17 mins [TIMING] data: 10.7ms | fwd: 3566.6ms | loss: 13.2ms | bwd: 10737.0ms | clip: 0.0ms | optim: 23.9ms step 901 | loss: 2.978494 | lr 1.8000e-04 | norm: 0.2015 | dt: 14398.46ms | tok/sec: 36412.8 | mem: 107435MB | Time Left: 05 hrs : 18 mins [TIMING] data: 8.3ms | fwd: 3562.3ms | loss: 13.1ms | bwd: 10777.6ms | clip: 0.0ms | optim: 23.8ms step 902 | loss: 3.045586 | lr 1.8000e-04 | norm: 0.2964 | dt: 14421.14ms | tok/sec: 36355.5 | mem: 107435MB | Time Left: 05 hrs : 18 mins [TIMING] data: 8.9ms | fwd: 3563.5ms | loss: 13.1ms | bwd: 10797.9ms | clip: 0.0ms | optim: 23.8ms step 903 | loss: 2.777039 | lr 1.8000e-04 | norm: 0.2201 | dt: 14427.26ms | tok/sec: 36340.1 | mem: 107435MB | Time Left: 05 hrs : 18 mins [TIMING] data: 8.4ms | fwd: 3572.5ms | loss: 13.1ms | bwd: 10796.1ms | clip: 0.0ms | optim: 24.0ms step 904 | loss: 2.684497 | lr 1.8000e-04 | norm: 0.2184 | dt: 14455.11ms | tok/sec: 36270.1 | mem: 107435MB | Time Left: 05 hrs : 18 mins [TIMING] data: 8.5ms | fwd: 3569.8ms | loss: 13.1ms | bwd: 10826.3ms | clip: 0.0ms | optim: 23.8ms step 905 | loss: 2.999719 | lr 1.8000e-04 | norm: 0.2450 | dt: 14447.85ms | tok/sec: 36288.3 | mem: 107435MB | Time Left: 05 hrs : 18 mins [TIMING] data: 8.3ms | fwd: 3576.1ms | loss: 13.0ms | bwd: 10813.3ms | clip: 0.0ms | optim: 23.9ms step 906 | loss: 3.236767 | lr 1.8000e-04 | norm: 0.2278 | dt: 14455.83ms | tok/sec: 36268.3 | mem: 107435MB | Time Left: 05 hrs : 18 mins [TIMING] data: 8.4ms | fwd: 3576.3ms | loss: 13.0ms | bwd: 10821.2ms | clip: 0.0ms | optim: 23.8ms step 907 | loss: 2.891258 | lr 1.8000e-04 | norm: 0.2048 | dt: 14454.63ms | tok/sec: 36271.3 | mem: 107435MB | Time Left: 05 hrs : 18 mins [TIMING] data: 8.2ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10820.3ms | clip: 0.0ms | optim: 23.8ms step 908 | loss: 2.989557 | lr 1.8000e-04 | norm: 0.2344 | dt: 14444.75ms | tok/sec: 36296.1 | mem: 107435MB | Time Left: 05 hrs : 17 mins [TIMING] data: 8.3ms | fwd: 3571.4ms | loss: 13.0ms | bwd: 10815.3ms | clip: 0.0ms | optim: 23.8ms step 909 | loss: 2.749501 | lr 1.8000e-04 | norm: 0.2810 | dt: 14452.17ms | tok/sec: 36277.5 | mem: 107435MB | Time Left: 05 hrs : 17 mins [TIMING] data: 8.3ms | fwd: 3577.3ms | loss: 13.0ms | bwd: 10816.5ms | clip: 0.0ms | optim: 23.9ms step 910 | loss: 2.894303 | lr 1.8000e-04 | norm: 0.2440 | dt: 14447.82ms | tok/sec: 36288.4 | mem: 107435MB | Time Left: 05 hrs : 17 mins [TIMING] data: 8.3ms | fwd: 3576.0ms | loss: 13.0ms | bwd: 10813.4ms | clip: 0.0ms | optim: 24.0ms step 911 | loss: 2.852988 | lr 1.8000e-04 | norm: 0.2181 | dt: 14466.45ms | tok/sec: 36241.7 | mem: 107435MB | Time Left: 05 hrs : 17 mins [TIMING] data: 8.3ms | fwd: 3581.8ms | loss: 13.0ms | bwd: 10826.5ms | clip: 0.0ms | optim: 23.7ms step 912 | loss: 2.783571 | lr 1.8000e-04 | norm: 0.1918 | dt: 14458.46ms | tok/sec: 36261.7 | mem: 107435MB | Time Left: 05 hrs : 16 mins [TIMING] data: 8.2ms | fwd: 3572.8ms | loss: 13.1ms | bwd: 10827.5ms | clip: 0.0ms | optim: 23.7ms step 913 | loss: 2.519879 | lr 1.8000e-04 | norm: 0.1990 | dt: 14464.25ms | tok/sec: 36247.2 | mem: 107435MB | Time Left: 05 hrs : 16 mins [TIMING] data: 8.3ms | fwd: 3574.1ms | loss: 13.0ms | bwd: 10831.9ms | clip: 0.0ms | optim: 23.8ms step 914 | loss: 2.883387 | lr 1.8000e-04 | norm: 0.2383 | dt: 14460.05ms | tok/sec: 36257.7 | mem: 107435MB | Time Left: 05 hrs : 16 mins [TIMING] data: 8.6ms | fwd: 3575.9ms | loss: 13.0ms | bwd: 10825.4ms | clip: 0.0ms | optim: 23.9ms step 915 | loss: 3.010661 | lr 1.8000e-04 | norm: 0.2439 | dt: 14463.16ms | tok/sec: 36249.9 | mem: 107435MB | Time Left: 05 hrs : 16 mins [TIMING] data: 8.3ms | fwd: 3575.2ms | loss: 13.0ms | bwd: 10829.7ms | clip: 0.0ms | optim: 23.8ms step 916 | loss: 3.216219 | lr 1.8000e-04 | norm: 0.2229 | dt: 14458.47ms | tok/sec: 36261.7 | mem: 107435MB | Time Left: 05 hrs : 15 mins [TIMING] data: 8.5ms | fwd: 3576.1ms | loss: 13.0ms | bwd: 10823.6ms | clip: 0.0ms | optim: 23.9ms step 917 | loss: 3.025495 | lr 1.8000e-04 | norm: 0.2347 | dt: 14468.19ms | tok/sec: 36237.3 | mem: 107435MB | Time Left: 05 hrs : 15 mins [TIMING] data: 8.4ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10834.5ms | clip: 0.0ms | optim: 23.8ms step 918 | loss: 3.059360 | lr 1.8000e-04 | norm: 0.1993 | dt: 14466.12ms | tok/sec: 36242.5 | mem: 107435MB | Time Left: 05 hrs : 15 mins [TIMING] data: 8.3ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10829.4ms | clip: 0.0ms | optim: 23.8ms step 919 | loss: 3.118351 | lr 1.8000e-04 | norm: 0.2174 | dt: 14467.76ms | tok/sec: 36238.4 | mem: 107435MB | Time Left: 05 hrs : 15 mins [TIMING] data: 8.3ms | fwd: 3573.0ms | loss: 13.0ms | bwd: 10836.5ms | clip: 0.0ms | optim: 23.9ms step 920 | loss: 3.155517 | lr 1.8000e-04 | norm: 0.1912 | dt: 14447.38ms | tok/sec: 36289.5 | mem: 107435MB | Time Left: 05 hrs : 14 mins [TIMING] data: 8.4ms | fwd: 3574.0ms | loss: 13.0ms | bwd: 10814.8ms | clip: 0.0ms | optim: 23.9ms step 921 | loss: 2.810313 | lr 1.8000e-04 | norm: 0.2206 | dt: 14470.42ms | tok/sec: 36231.7 | mem: 107435MB | Time Left: 05 hrs : 14 mins [TIMING] data: 8.4ms | fwd: 3576.6ms | loss: 13.0ms | bwd: 10835.4ms | clip: 0.0ms | optim: 23.8ms step 922 | loss: 2.992997 | lr 1.8000e-04 | norm: 0.2456 | dt: 14469.25ms | tok/sec: 36234.6 | mem: 107435MB | Time Left: 05 hrs : 14 mins [TIMING] data: 8.3ms | fwd: 3574.5ms | loss: 13.1ms | bwd: 10836.5ms | clip: 0.0ms | optim: 23.7ms step 923 | loss: 2.772933 | lr 1.8000e-04 | norm: 0.2187 | dt: 14458.12ms | tok/sec: 36262.5 | mem: 107435MB | Time Left: 05 hrs : 14 mins [TIMING] data: 8.3ms | fwd: 3577.1ms | loss: 13.0ms | bwd: 10822.1ms | clip: 0.0ms | optim: 24.0ms step 924 | loss: 2.936476 | lr 1.8000e-04 | norm: 0.2020 | dt: 14483.31ms | tok/sec: 36199.5 | mem: 107435MB | Time Left: 05 hrs : 14 mins [TIMING] data: 8.3ms | fwd: 3579.9ms | loss: 13.0ms | bwd: 10845.2ms | clip: 0.0ms | optim: 23.8ms step 925 | loss: 2.699044 | lr 1.8000e-04 | norm: 0.2155 | dt: 14484.48ms | tok/sec: 36196.5 | mem: 107435MB | Time Left: 05 hrs : 14 mins [TIMING] data: 8.3ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10848.1ms | clip: 0.0ms | optim: 23.8ms step 926 | loss: 2.885452 | lr 1.8000e-04 | norm: 0.1921 | dt: 14469.18ms | tok/sec: 36234.8 | mem: 107435MB | Time Left: 05 hrs : 13 mins [TIMING] data: 8.3ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10835.1ms | clip: 0.0ms | optim: 23.8ms step 927 | loss: 2.961079 | lr 1.8000e-04 | norm: 0.2385 | dt: 14487.13ms | tok/sec: 36189.9 | mem: 107435MB | Time Left: 05 hrs : 13 mins [TIMING] data: 8.6ms | fwd: 3577.9ms | loss: 13.0ms | bwd: 10850.3ms | clip: 0.0ms | optim: 23.9ms step 928 | loss: 2.750947 | lr 1.8000e-04 | norm: 0.2030 | dt: 14486.94ms | tok/sec: 36190.4 | mem: 107435MB | Time Left: 05 hrs : 13 mins [TIMING] data: 8.5ms | fwd: 3577.3ms | loss: 13.0ms | bwd: 10851.2ms | clip: 0.0ms | optim: 23.9ms step 929 | loss: 3.111631 | lr 1.8000e-04 | norm: 0.2016 | dt: 14494.33ms | tok/sec: 36171.9 | mem: 107435MB | Time Left: 05 hrs : 13 mins [TIMING] data: 8.4ms | fwd: 3594.3ms | loss: 13.1ms | bwd: 10841.7ms | clip: 0.0ms | optim: 24.0ms step 930 | loss: 2.777698 | lr 1.8000e-04 | norm: 0.6509 | dt: 14468.72ms | tok/sec: 36236.0 | mem: 107435MB | Time Left: 05 hrs : 12 mins [TIMING] data: 8.4ms | fwd: 3571.2ms | loss: 13.1ms | bwd: 10839.3ms | clip: 0.0ms | optim: 23.7ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00005.bin 100000000 step 931 | loss: 2.712865 | lr 1.8000e-04 | norm: 0.2143 | dt: 14479.95ms | tok/sec: 36207.9 | mem: 107435MB | Time Left: 05 hrs : 12 mins [TIMING] data: 14.3ms | fwd: 3575.7ms | loss: 13.0ms | bwd: 10839.9ms | clip: 0.0ms | optim: 23.8ms step 932 | loss: 2.637484 | lr 1.8000e-04 | norm: 0.1949 | dt: 14475.22ms | tok/sec: 36219.7 | mem: 107435MB | Time Left: 05 hrs : 12 mins [TIMING] data: 8.2ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10835.6ms | clip: 0.0ms | optim: 23.9ms step 933 | loss: 2.672297 | lr 1.8000e-04 | norm: 0.1884 | dt: 14484.89ms | tok/sec: 36195.5 | mem: 107435MB | Time Left: 05 hrs : 12 mins [TIMING] data: 8.3ms | fwd: 3591.8ms | loss: 13.1ms | bwd: 10834.5ms | clip: 0.0ms | optim: 23.9ms step 934 | loss: 3.113895 | lr 1.8000e-04 | norm: 0.1874 | dt: 14482.44ms | tok/sec: 36201.6 | mem: 107435MB | Time Left: 05 hrs : 12 mins [TIMING] data: 8.3ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10846.2ms | clip: 0.0ms | optim: 23.7ms step 935 | loss: 2.877343 | lr 1.8000e-04 | norm: 0.2074 | dt: 14484.58ms | tok/sec: 36196.3 | mem: 107435MB | Time Left: 05 hrs : 11 mins [TIMING] data: 8.4ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.8ms step 936 | loss: 2.881541 | lr 1.8000e-04 | norm: 0.2881 | dt: 14488.12ms | tok/sec: 36187.4 | mem: 107435MB | Time Left: 05 hrs : 11 mins [TIMING] data: 8.3ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10853.0ms | clip: 0.0ms | optim: 23.8ms step 937 | loss: 2.675729 | lr 1.8000e-04 | norm: 0.1982 | dt: 14479.24ms | tok/sec: 36209.6 | mem: 107435MB | Time Left: 05 hrs : 11 mins [TIMING] data: 8.3ms | fwd: 3590.2ms | loss: 13.0ms | bwd: 10830.8ms | clip: 0.0ms | optim: 23.8ms step 938 | loss: 2.873478 | lr 1.8000e-04 | norm: 0.2525 | dt: 14478.03ms | tok/sec: 36212.7 | mem: 107435MB | Time Left: 05 hrs : 11 mins [TIMING] data: 8.4ms | fwd: 3573.4ms | loss: 13.1ms | bwd: 10846.1ms | clip: 0.0ms | optim: 23.8ms step 939 | loss: 2.791921 | lr 1.8000e-04 | norm: 0.2183 | dt: 14476.66ms | tok/sec: 36216.1 | mem: 107435MB | Time Left: 05 hrs : 10 mins [TIMING] data: 8.7ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10838.4ms | clip: 0.0ms | optim: 23.7ms step 940 | loss: 3.074709 | lr 1.8000e-04 | norm: 0.2074 | dt: 14530.82ms | tok/sec: 36081.1 | mem: 107435MB | Time Left: 05 hrs : 11 mins [TIMING] data: 8.6ms | fwd: 3652.4ms | loss: 13.0ms | bwd: 10819.4ms | clip: 0.0ms | optim: 24.1ms step 941 | loss: 2.806712 | lr 1.8000e-04 | norm: 0.2009 | dt: 14471.02ms | tok/sec: 36230.2 | mem: 107435MB | Time Left: 05 hrs : 10 mins [TIMING] data: 8.3ms | fwd: 3591.3ms | loss: 13.0ms | bwd: 10821.6ms | clip: 0.0ms | optim: 23.9ms step 942 | loss: 2.860209 | lr 1.8000e-04 | norm: 0.2182 | dt: 14536.88ms | tok/sec: 36066.1 | mem: 107435MB | Time Left: 05 hrs : 11 mins [TIMING] data: 8.2ms | fwd: 3661.1ms | loss: 13.1ms | bwd: 10817.4ms | clip: 0.0ms | optim: 23.8ms step 943 | loss: 2.858933 | lr 1.8000e-04 | norm: 0.2275 | dt: 14472.88ms | tok/sec: 36225.6 | mem: 107435MB | Time Left: 05 hrs : 09 mins [TIMING] data: 8.3ms | fwd: 3574.1ms | loss: 13.1ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.9ms step 944 | loss: 3.012909 | lr 1.8000e-04 | norm: 0.1973 | dt: 14478.39ms | tok/sec: 36211.8 | mem: 107435MB | Time Left: 05 hrs : 09 mins [TIMING] data: 8.3ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10843.1ms | clip: 0.0ms | optim: 24.0ms step 945 | loss: 2.679197 | lr 1.8000e-04 | norm: 0.2144 | dt: 14465.75ms | tok/sec: 36243.4 | mem: 107435MB | Time Left: 05 hrs : 09 mins [TIMING] data: 8.4ms | fwd: 3593.1ms | loss: 13.0ms | bwd: 10814.3ms | clip: 0.0ms | optim: 23.8ms step 946 | loss: 2.900413 | lr 1.8000e-04 | norm: 0.2122 | dt: 14471.87ms | tok/sec: 36228.1 | mem: 107435MB | Time Left: 05 hrs : 08 mins [TIMING] data: 8.5ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10837.6ms | clip: 0.0ms | optim: 23.9ms step 947 | loss: 2.837929 | lr 1.8000e-04 | norm: 0.1940 | dt: 14476.53ms | tok/sec: 36216.4 | mem: 107435MB | Time Left: 05 hrs : 08 mins [TIMING] data: 8.4ms | fwd: 3578.4ms | loss: 13.0ms | bwd: 10839.5ms | clip: 0.0ms | optim: 23.9ms step 948 | loss: 2.811319 | lr 1.8000e-04 | norm: 0.1952 | dt: 14475.32ms | tok/sec: 36219.4 | mem: 107435MB | Time Left: 05 hrs : 08 mins [TIMING] data: 8.6ms | fwd: 3580.4ms | loss: 13.0ms | bwd: 10836.1ms | clip: 0.0ms | optim: 23.8ms step 949 | loss: 2.886255 | lr 1.8000e-04 | norm: 0.1791 | dt: 14470.14ms | tok/sec: 36232.4 | mem: 107435MB | Time Left: 05 hrs : 08 mins [TIMING] data: 8.3ms | fwd: 3593.0ms | loss: 13.1ms | bwd: 10819.0ms | clip: 0.0ms | optim: 23.7ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_950.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_850.ckpt step 950 | loss: 2.728557 | lr 1.8000e-04 | norm: 0.1941 | dt: 14382.07ms | tok/sec: 36454.3 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 9.6ms | fwd: 3574.3ms | loss: 13.2ms | bwd: 10746.7ms | clip: 0.0ms | optim: 23.8ms step 951 | loss: 2.690464 | lr 1.8000e-04 | norm: 0.1804 | dt: 14407.52ms | tok/sec: 36389.9 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.3ms | fwd: 3560.3ms | loss: 13.1ms | bwd: 10788.7ms | clip: 0.0ms | optim: 23.9ms step 952 | loss: 2.884651 | lr 1.8000e-04 | norm: 0.1882 | dt: 14467.51ms | tok/sec: 36239.0 | mem: 107435MB | Time Left: 05 hrs : 07 mins [TIMING] data: 8.5ms | fwd: 3617.9ms | loss: 13.1ms | bwd: 10791.0ms | clip: 0.0ms | optim: 23.8ms step 953 | loss: 3.236392 | lr 1.8000e-04 | norm: 0.1914 | dt: 14448.29ms | tok/sec: 36287.2 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.9ms | fwd: 3572.5ms | loss: 13.1ms | bwd: 10816.0ms | clip: 0.0ms | optim: 23.9ms step 954 | loss: 2.930556 | lr 1.8000e-04 | norm: 0.1997 | dt: 14462.87ms | tok/sec: 36250.6 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.4ms | fwd: 3578.0ms | loss: 13.1ms | bwd: 10826.0ms | clip: 0.0ms | optim: 24.0ms step 955 | loss: 2.796322 | lr 1.8000e-04 | norm: 0.1802 | dt: 14463.84ms | tok/sec: 36248.2 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.3ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10827.1ms | clip: 0.0ms | optim: 23.8ms step 956 | loss: 2.818668 | lr 1.8000e-04 | norm: 0.1810 | dt: 14483.37ms | tok/sec: 36199.3 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.3ms | fwd: 3597.2ms | loss: 13.1ms | bwd: 10827.6ms | clip: 0.0ms | optim: 23.8ms step 957 | loss: 2.856150 | lr 1.8000e-04 | norm: 0.2067 | dt: 14476.87ms | tok/sec: 36215.6 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.3ms | fwd: 3583.9ms | loss: 13.0ms | bwd: 10834.8ms | clip: 0.0ms | optim: 23.8ms step 958 | loss: 3.046442 | lr 1.8000e-04 | norm: 0.2185 | dt: 14502.55ms | tok/sec: 36151.4 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.2ms | fwd: 3601.2ms | loss: 13.1ms | bwd: 10843.1ms | clip: 0.0ms | optim: 23.8ms step 959 | loss: 3.146666 | lr 1.8000e-04 | norm: 0.2364 | dt: 14507.69ms | tok/sec: 36138.6 | mem: 107435MB | Time Left: 05 hrs : 06 mins [TIMING] data: 8.4ms | fwd: 3614.0ms | loss: 13.0ms | bwd: 10835.2ms | clip: 0.0ms | optim: 23.8ms step 960 | loss: 2.812133 | lr 1.8000e-04 | norm: 0.2462 | dt: 14481.32ms | tok/sec: 36204.4 | mem: 107435MB | Time Left: 05 hrs : 05 mins [TIMING] data: 8.2ms | fwd: 3574.9ms | loss: 13.1ms | bwd: 10848.3ms | clip: 0.0ms | optim: 23.8ms step 961 | loss: 3.026418 | lr 1.8000e-04 | norm: 0.2074 | dt: 14498.42ms | tok/sec: 36161.7 | mem: 107435MB | Time Left: 05 hrs : 05 mins [TIMING] data: 8.2ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10858.5ms | clip: 0.0ms | optim: 23.8ms step 962 | loss: 3.037725 | lr 1.8000e-04 | norm: 0.4452 | dt: 14503.37ms | tok/sec: 36149.4 | mem: 107435MB | Time Left: 05 hrs : 05 mins [TIMING] data: 8.3ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10864.8ms | clip: 0.0ms | optim: 23.8ms step 963 | loss: 3.180679 | lr 1.8000e-04 | norm: 0.3094 | dt: 14501.56ms | tok/sec: 36153.9 | mem: 107435MB | Time Left: 05 hrs : 05 mins [TIMING] data: 8.2ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10865.3ms | clip: 0.0ms | optim: 23.9ms step 964 | loss: 2.718937 | lr 1.8000e-04 | norm: 0.1766 | dt: 14483.90ms | tok/sec: 36198.0 | mem: 107435MB | Time Left: 05 hrs : 04 mins [TIMING] data: 8.3ms | fwd: 3576.8ms | loss: 13.0ms | bwd: 10849.0ms | clip: 0.0ms | optim: 23.7ms step 965 | loss: 2.913689 | lr 1.8000e-04 | norm: 0.2566 | dt: 14476.73ms | tok/sec: 36215.9 | mem: 107435MB | Time Left: 05 hrs : 04 mins [TIMING] data: 8.2ms | fwd: 3573.7ms | loss: 13.0ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.8ms step 966 | loss: 2.900545 | lr 1.8000e-04 | norm: 0.1760 | dt: 14484.34ms | tok/sec: 36196.9 | mem: 107435MB | Time Left: 05 hrs : 04 mins [TIMING] data: 8.3ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10843.8ms | clip: 0.0ms | optim: 23.9ms step 967 | loss: 2.816532 | lr 1.8000e-04 | norm: 0.2079 | dt: 14498.64ms | tok/sec: 36161.2 | mem: 107435MB | Time Left: 05 hrs : 04 mins [TIMING] data: 8.4ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 23.8ms step 968 | loss: 2.711881 | lr 1.8000e-04 | norm: 0.2188 | dt: 14488.87ms | tok/sec: 36185.6 | mem: 107435MB | Time Left: 05 hrs : 04 mins [TIMING] data: 8.2ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10853.5ms | clip: 0.0ms | optim: 23.9ms step 969 | loss: 2.628616 | lr 1.8000e-04 | norm: 0.1824 | dt: 14486.37ms | tok/sec: 36191.8 | mem: 107435MB | Time Left: 05 hrs : 03 mins [TIMING] data: 8.4ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10847.8ms | clip: 0.0ms | optim: 23.9ms step 970 | loss: 2.852334 | lr 1.8000e-04 | norm: 0.2883 | dt: 14485.56ms | tok/sec: 36193.8 | mem: 107435MB | Time Left: 05 hrs : 03 mins [TIMING] data: 8.3ms | fwd: 3577.4ms | loss: 13.0ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.9ms step 971 | loss: 2.778922 | lr 1.8000e-04 | norm: 0.1911 | dt: 14482.17ms | tok/sec: 36202.3 | mem: 107435MB | Time Left: 05 hrs : 03 mins [TIMING] data: 8.3ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10848.8ms | clip: 0.0ms | optim: 23.9ms step 972 | loss: 2.645961 | lr 1.8000e-04 | norm: 0.2978 | dt: 14497.53ms | tok/sec: 36163.9 | mem: 107435MB | Time Left: 05 hrs : 03 mins [TIMING] data: 8.4ms | fwd: 3576.1ms | loss: 13.0ms | bwd: 10863.0ms | clip: 0.0ms | optim: 23.8ms step 973 | loss: 3.308422 | lr 1.8000e-04 | norm: 1.1486 | dt: 14485.36ms | tok/sec: 36194.3 | mem: 107435MB | Time Left: 05 hrs : 02 mins [TIMING] data: 8.3ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10845.9ms | clip: 0.0ms | optim: 23.8ms step 974 | loss: 2.716569 | lr 1.8000e-04 | norm: 0.2223 | dt: 14494.31ms | tok/sec: 36172.0 | mem: 107435MB | Time Left: 05 hrs : 02 mins [TIMING] data: 8.2ms | fwd: 3581.2ms | loss: 13.0ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.8ms step 975 | loss: 2.811617 | lr 1.8000e-04 | norm: 0.2083 | dt: 14486.16ms | tok/sec: 36192.3 | mem: 107435MB | Time Left: 05 hrs : 02 mins [TIMING] data: 8.2ms | fwd: 3589.1ms | loss: 13.1ms | bwd: 10838.8ms | clip: 0.0ms | optim: 23.9ms step 976 | loss: 3.011859 | lr 1.8000e-04 | norm: 0.1935 | dt: 14538.33ms | tok/sec: 36062.5 | mem: 107435MB | Time Left: 05 hrs : 03 mins [TIMING] data: 8.6ms | fwd: 3651.1ms | loss: 13.1ms | bwd: 10828.6ms | clip: 0.0ms | optim: 23.8ms step 977 | loss: 2.963858 | lr 1.8000e-04 | norm: 0.5844 | dt: 14494.91ms | tok/sec: 36170.5 | mem: 107435MB | Time Left: 05 hrs : 01 mins [TIMING] data: 8.2ms | fwd: 3578.0ms | loss: 13.1ms | bwd: 10858.5ms | clip: 0.0ms | optim: 23.8ms step 978 | loss: 2.773454 | lr 1.8000e-04 | norm: 0.2254 | dt: 14483.29ms | tok/sec: 36199.5 | mem: 107435MB | Time Left: 05 hrs : 01 mins [TIMING] data: 8.4ms | fwd: 3577.5ms | loss: 13.0ms | bwd: 10847.5ms | clip: 0.0ms | optim: 23.8ms step 979 | loss: 3.143406 | lr 1.8000e-04 | norm: 0.2492 | dt: 14484.89ms | tok/sec: 36195.5 | mem: 107435MB | Time Left: 05 hrs : 01 mins [TIMING] data: 8.3ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10846.7ms | clip: 0.0ms | optim: 23.8ms step 980 | loss: 2.980063 | lr 1.8000e-04 | norm: 0.2167 | dt: 14533.17ms | tok/sec: 36075.3 | mem: 107435MB | Time Left: 05 hrs : 02 mins [TIMING] data: 8.3ms | fwd: 3646.4ms | loss: 13.0ms | bwd: 10828.3ms | clip: 0.0ms | optim: 23.9ms step 981 | loss: 2.713692 | lr 1.8000e-04 | norm: 0.1803 | dt: 14481.18ms | tok/sec: 36204.8 | mem: 107435MB | Time Left: 05 hrs : 00 mins [TIMING] data: 8.4ms | fwd: 3589.0ms | loss: 13.1ms | bwd: 10833.5ms | clip: 0.0ms | optim: 23.8ms step 982 | loss: 2.972310 | lr 1.8000e-04 | norm: 0.2783 | dt: 14485.84ms | tok/sec: 36193.1 | mem: 107435MB | Time Left: 05 hrs : 00 mins [TIMING] data: 8.6ms | fwd: 3589.4ms | loss: 13.1ms | bwd: 10837.6ms | clip: 0.0ms | optim: 23.9ms step 983 | loss: 2.879886 | lr 1.8000e-04 | norm: 0.1949 | dt: 14476.98ms | tok/sec: 36215.3 | mem: 107435MB | Time Left: 05 hrs : 00 mins [TIMING] data: 8.3ms | fwd: 3573.2ms | loss: 13.0ms | bwd: 10845.4ms | clip: 0.0ms | optim: 23.9ms step 984 | loss: 2.913619 | lr 1.8000e-04 | norm: 0.2026 | dt: 14514.97ms | tok/sec: 36120.5 | mem: 107435MB | Time Left: 05 hrs : 00 mins [TIMING] data: 8.2ms | fwd: 3618.8ms | loss: 13.0ms | bwd: 10837.6ms | clip: 0.0ms | optim: 23.8ms step 985 | loss: 2.962351 | lr 1.8000e-04 | norm: 0.2574 | dt: 14478.96ms | tok/sec: 36210.3 | mem: 107435MB | Time Left: 04 hrs : 59 mins [TIMING] data: 8.2ms | fwd: 3585.0ms | loss: 13.0ms | bwd: 10835.8ms | clip: 0.0ms | optim: 23.8ms step 986 | loss: 2.999091 | lr 1.8000e-04 | norm: 0.1884 | dt: 14480.22ms | tok/sec: 36207.2 | mem: 107435MB | Time Left: 04 hrs : 59 mins [TIMING] data: 8.0ms | fwd: 3578.9ms | loss: 13.0ms | bwd: 10843.3ms | clip: 0.0ms | optim: 23.8ms step 987 | loss: 2.882305 | lr 1.8000e-04 | norm: 0.1899 | dt: 14482.85ms | tok/sec: 36200.6 | mem: 107435MB | Time Left: 04 hrs : 59 mins [TIMING] data: 8.3ms | fwd: 3576.4ms | loss: 13.1ms | bwd: 10847.9ms | clip: 0.0ms | optim: 23.8ms step 988 | loss: 2.812951 | lr 1.8000e-04 | norm: 0.2265 | dt: 14476.68ms | tok/sec: 36216.0 | mem: 107435MB | Time Left: 04 hrs : 58 mins [TIMING] data: 8.2ms | fwd: 3577.1ms | loss: 13.0ms | bwd: 10841.3ms | clip: 0.0ms | optim: 23.9ms step 989 | loss: 2.951220 | lr 1.8000e-04 | norm: 0.1904 | dt: 14471.18ms | tok/sec: 36229.8 | mem: 107435MB | Time Left: 04 hrs : 58 mins [TIMING] data: 8.1ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10832.2ms | clip: 0.0ms | optim: 23.9ms step 990 | loss: 2.889110 | lr 1.8000e-04 | norm: 0.1926 | dt: 14496.56ms | tok/sec: 36166.4 | mem: 107435MB | Time Left: 04 hrs : 58 mins [TIMING] data: 8.4ms | fwd: 3603.1ms | loss: 13.0ms | bwd: 10835.0ms | clip: 0.0ms | optim: 23.8ms step 991 | loss: 2.758888 | lr 1.8000e-04 | norm: 0.2075 | dt: 14481.87ms | tok/sec: 36203.1 | mem: 107435MB | Time Left: 04 hrs : 58 mins [TIMING] data: 8.4ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10846.3ms | clip: 0.0ms | optim: 23.9ms step 992 | loss: 3.127996 | lr 1.8000e-04 | norm: 0.2286 | dt: 14465.38ms | tok/sec: 36244.3 | mem: 107435MB | Time Left: 04 hrs : 57 mins [TIMING] data: 8.1ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10829.5ms | clip: 0.0ms | optim: 23.8ms step 993 | loss: 2.951625 | lr 1.8000e-04 | norm: 0.1885 | dt: 14553.69ms | tok/sec: 36024.4 | mem: 107435MB | Time Left: 04 hrs : 59 mins [TIMING] data: 8.2ms | fwd: 3662.2ms | loss: 13.1ms | bwd: 10833.2ms | clip: 0.0ms | optim: 23.8ms step 994 | loss: 3.106214 | lr 1.8000e-04 | norm: 0.1903 | dt: 14477.68ms | tok/sec: 36213.5 | mem: 107435MB | Time Left: 04 hrs : 57 mins [TIMING] data: 8.3ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10841.3ms | clip: 0.0ms | optim: 23.9ms step 995 | loss: 2.555558 | lr 1.8000e-04 | norm: 0.2090 | dt: 14489.69ms | tok/sec: 36183.5 | mem: 107435MB | Time Left: 04 hrs : 57 mins [TIMING] data: 8.3ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10852.7ms | clip: 0.0ms | optim: 23.9ms step 996 | loss: 2.927370 | lr 1.8000e-04 | norm: 0.2221 | dt: 14481.79ms | tok/sec: 36203.3 | mem: 107435MB | Time Left: 04 hrs : 57 mins [TIMING] data: 8.2ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10845.1ms | clip: 0.0ms | optim: 23.8ms step 997 | loss: 2.911766 | lr 1.8000e-04 | norm: 0.2364 | dt: 14486.36ms | tok/sec: 36191.8 | mem: 107435MB | Time Left: 04 hrs : 56 mins [TIMING] data: 8.1ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10845.9ms | clip: 0.0ms | optim: 24.4ms step 998 | loss: 2.641560 | lr 1.8000e-04 | norm: 0.1749 | dt: 14481.82ms | tok/sec: 36203.2 | mem: 107435MB | Time Left: 04 hrs : 56 mins [TIMING] data: 8.3ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10840.4ms | clip: 0.0ms | optim: 23.7ms step 999 | loss: 2.811823 | lr 1.8000e-04 | norm: 0.3630 | dt: 14488.76ms | tok/sec: 36185.8 | mem: 107435MB | Time Left: 04 hrs : 56 mins [TIMING] data: 8.7ms | fwd: 3575.9ms | loss: 13.0ms | bwd: 10854.3ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1000.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_900.ckpt validation loss: 3.0908 step 1000 | loss: 3.204825 | lr 1.8000e-04 | norm: 0.2289 | dt: 18986.01ms | tok/sec: 27614.4 | mem: 107435MB | Time Left: 06 hrs : 28 mins [TIMING] data: 8.6ms | fwd: 3554.3ms | loss: 13.1ms | bwd: 10782.7ms | clip: 0.0ms | optim: 23.8ms step 1001 | loss: 2.833997 | lr 1.8000e-04 | norm: 0.2169 | dt: 14465.81ms | tok/sec: 36243.2 | mem: 107435MB | Time Left: 04 hrs : 55 mins [TIMING] data: 8.5ms | fwd: 3639.0ms | loss: 13.0ms | bwd: 10768.2ms | clip: 0.0ms | optim: 23.8ms step 1002 | loss: 3.025404 | lr 1.8000e-04 | norm: 0.1814 | dt: 14439.23ms | tok/sec: 36310.0 | mem: 107435MB | Time Left: 04 hrs : 54 mins [TIMING] data: 8.3ms | fwd: 3584.4ms | loss: 13.1ms | bwd: 10796.5ms | clip: 0.0ms | optim: 23.8ms step 1003 | loss: 2.940546 | lr 1.8000e-04 | norm: 0.2049 | dt: 14440.71ms | tok/sec: 36306.2 | mem: 107435MB | Time Left: 04 hrs : 54 mins [TIMING] data: 8.2ms | fwd: 3571.0ms | loss: 13.0ms | bwd: 10811.4ms | clip: 0.0ms | optim: 24.0ms step 1004 | loss: 2.983417 | lr 1.8000e-04 | norm: 0.2327 | dt: 14465.89ms | tok/sec: 36243.0 | mem: 107435MB | Time Left: 04 hrs : 54 mins [TIMING] data: 8.3ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10830.8ms | clip: 0.0ms | optim: 23.7ms step 1005 | loss: 3.263031 | lr 1.8000e-04 | norm: 0.2452 | dt: 14464.17ms | tok/sec: 36247.4 | mem: 107435MB | Time Left: 04 hrs : 54 mins [TIMING] data: 8.3ms | fwd: 3577.5ms | loss: 13.0ms | bwd: 10828.4ms | clip: 0.0ms | optim: 23.7ms step 1006 | loss: 2.899553 | lr 1.8000e-04 | norm: 0.1717 | dt: 14461.76ms | tok/sec: 36253.4 | mem: 107435MB | Time Left: 04 hrs : 54 mins [TIMING] data: 8.4ms | fwd: 3572.3ms | loss: 13.1ms | bwd: 10831.1ms | clip: 0.0ms | optim: 23.8ms step 1007 | loss: 2.739835 | lr 1.8000e-04 | norm: 0.1985 | dt: 14470.40ms | tok/sec: 36231.7 | mem: 107435MB | Time Left: 04 hrs : 54 mins [TIMING] data: 8.4ms | fwd: 3591.9ms | loss: 13.0ms | bwd: 10820.2ms | clip: 0.0ms | optim: 23.8ms step 1008 | loss: 2.895662 | lr 1.8000e-04 | norm: 0.1855 | dt: 14512.18ms | tok/sec: 36127.4 | mem: 107435MB | Time Left: 04 hrs : 54 mins [TIMING] data: 8.4ms | fwd: 3637.8ms | loss: 13.1ms | bwd: 10816.0ms | clip: 0.0ms | optim: 23.8ms step 1009 | loss: 3.083142 | lr 1.8000e-04 | norm: 0.1895 | dt: 14480.97ms | tok/sec: 36205.3 | mem: 107435MB | Time Left: 04 hrs : 53 mins [TIMING] data: 8.7ms | fwd: 3575.0ms | loss: 13.1ms | bwd: 10847.0ms | clip: 0.0ms | optim: 23.7ms step 1010 | loss: 2.981208 | lr 1.8000e-04 | norm: 0.2406 | dt: 14471.71ms | tok/sec: 36228.5 | mem: 107435MB | Time Left: 04 hrs : 53 mins [TIMING] data: 8.3ms | fwd: 3585.0ms | loss: 13.0ms | bwd: 10828.6ms | clip: 0.0ms | optim: 23.7ms step 1011 | loss: 2.953230 | lr 1.8000e-04 | norm: 0.2002 | dt: 14488.79ms | tok/sec: 36185.8 | mem: 107435MB | Time Left: 04 hrs : 53 mins [TIMING] data: 8.3ms | fwd: 3596.5ms | loss: 13.0ms | bwd: 10834.2ms | clip: 0.0ms | optim: 23.8ms step 1012 | loss: 3.054540 | lr 1.8000e-04 | norm: 0.2122 | dt: 14479.58ms | tok/sec: 36208.8 | mem: 107435MB | Time Left: 04 hrs : 53 mins [TIMING] data: 8.3ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10837.2ms | clip: 0.0ms | optim: 23.8ms step 1013 | loss: 3.119441 | lr 1.8000e-04 | norm: 0.1879 | dt: 14468.90ms | tok/sec: 36235.5 | mem: 107435MB | Time Left: 04 hrs : 52 mins [TIMING] data: 8.3ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10833.8ms | clip: 0.0ms | optim: 23.8ms step 1014 | loss: 2.777749 | lr 1.8000e-04 | norm: 0.1955 | dt: 14470.97ms | tok/sec: 36230.3 | mem: 107435MB | Time Left: 04 hrs : 52 mins [TIMING] data: 8.3ms | fwd: 3575.1ms | loss: 13.0ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.9ms step 1015 | loss: 3.006164 | lr 1.8000e-04 | norm: 0.2355 | dt: 14492.03ms | tok/sec: 36177.7 | mem: 107435MB | Time Left: 04 hrs : 52 mins [TIMING] data: 8.4ms | fwd: 3609.9ms | loss: 13.0ms | bwd: 10823.7ms | clip: 0.0ms | optim: 23.8ms step 1016 | loss: 2.807524 | lr 1.8000e-04 | norm: 0.2666 | dt: 14462.53ms | tok/sec: 36251.5 | mem: 107435MB | Time Left: 04 hrs : 51 mins [TIMING] data: 8.4ms | fwd: 3576.2ms | loss: 13.0ms | bwd: 10828.0ms | clip: 0.0ms | optim: 23.8ms step 1017 | loss: 3.096107 | lr 1.8000e-04 | norm: 0.2052 | dt: 14475.17ms | tok/sec: 36219.8 | mem: 107435MB | Time Left: 04 hrs : 51 mins [TIMING] data: 8.3ms | fwd: 3579.8ms | loss: 13.0ms | bwd: 10837.1ms | clip: 0.0ms | optim: 23.8ms step 1018 | loss: 3.041782 | lr 1.8000e-04 | norm: 0.1884 | dt: 14470.94ms | tok/sec: 36230.4 | mem: 107435MB | Time Left: 04 hrs : 51 mins [TIMING] data: 8.3ms | fwd: 3572.9ms | loss: 13.1ms | bwd: 10839.6ms | clip: 0.0ms | optim: 23.8ms step 1019 | loss: 3.098658 | lr 1.8000e-04 | norm: 0.2405 | dt: 14520.31ms | tok/sec: 36107.2 | mem: 107435MB | Time Left: 04 hrs : 52 mins [TIMING] data: 8.3ms | fwd: 3631.1ms | loss: 13.1ms | bwd: 10830.9ms | clip: 0.0ms | optim: 23.9ms step 1020 | loss: 2.824921 | lr 1.8000e-04 | norm: 0.3402 | dt: 14484.32ms | tok/sec: 36196.9 | mem: 107435MB | Time Left: 04 hrs : 51 mins [TIMING] data: 8.4ms | fwd: 3574.7ms | loss: 13.0ms | bwd: 10851.3ms | clip: 0.0ms | optim: 23.9ms step 1021 | loss: 2.966714 | lr 1.8000e-04 | norm: 0.2243 | dt: 14481.81ms | tok/sec: 36203.2 | mem: 107435MB | Time Left: 04 hrs : 51 mins [TIMING] data: 8.4ms | fwd: 3576.2ms | loss: 13.0ms | bwd: 10847.2ms | clip: 0.0ms | optim: 23.8ms step 1022 | loss: 2.920844 | lr 1.8000e-04 | norm: 0.1975 | dt: 14468.58ms | tok/sec: 36236.3 | mem: 107435MB | Time Left: 04 hrs : 50 mins [TIMING] data: 8.4ms | fwd: 3590.6ms | loss: 13.1ms | bwd: 10819.3ms | clip: 0.0ms | optim: 23.8ms step 1023 | loss: 3.076255 | lr 1.8000e-04 | norm: 0.2180 | dt: 14476.96ms | tok/sec: 36215.3 | mem: 107435MB | Time Left: 04 hrs : 50 mins [TIMING] data: 8.2ms | fwd: 3592.8ms | loss: 13.1ms | bwd: 10825.9ms | clip: 0.0ms | optim: 23.8ms step 1024 | loss: 3.140490 | lr 1.8000e-04 | norm: 0.2087 | dt: 14522.38ms | tok/sec: 36102.1 | mem: 107435MB | Time Left: 04 hrs : 51 mins [TIMING] data: 8.3ms | fwd: 3627.6ms | loss: 13.1ms | bwd: 10836.3ms | clip: 0.0ms | optim: 23.8ms step 1025 | loss: 2.611504 | lr 1.8000e-04 | norm: 0.2101 | dt: 14470.66ms | tok/sec: 36231.1 | mem: 107435MB | Time Left: 04 hrs : 49 mins [TIMING] data: 8.4ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10834.5ms | clip: 0.0ms | optim: 23.8ms step 1026 | loss: 2.752965 | lr 1.8000e-04 | norm: 0.3228 | dt: 14464.74ms | tok/sec: 36245.9 | mem: 107435MB | Time Left: 04 hrs : 49 mins [TIMING] data: 8.3ms | fwd: 3591.7ms | loss: 13.1ms | bwd: 10814.6ms | clip: 0.0ms | optim: 23.9ms step 1027 | loss: 2.916321 | lr 1.8000e-04 | norm: 0.1936 | dt: 14472.39ms | tok/sec: 36226.8 | mem: 107435MB | Time Left: 04 hrs : 49 mins [TIMING] data: 8.3ms | fwd: 3582.5ms | loss: 13.0ms | bwd: 10831.6ms | clip: 0.0ms | optim: 23.8ms step 1028 | loss: 2.715501 | lr 1.8000e-04 | norm: 0.1840 | dt: 14469.00ms | tok/sec: 36235.3 | mem: 107435MB | Time Left: 04 hrs : 49 mins [TIMING] data: 8.3ms | fwd: 3590.8ms | loss: 13.1ms | bwd: 10819.9ms | clip: 0.0ms | optim: 23.8ms step 1029 | loss: 2.860794 | lr 1.8000e-04 | norm: 0.1682 | dt: 14478.63ms | tok/sec: 36211.2 | mem: 107435MB | Time Left: 04 hrs : 49 mins [TIMING] data: 8.3ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10839.7ms | clip: 0.0ms | optim: 23.9ms step 1030 | loss: 2.827435 | lr 1.8000e-04 | norm: 0.2029 | dt: 14477.29ms | tok/sec: 36214.5 | mem: 107435MB | Time Left: 04 hrs : 48 mins [TIMING] data: 8.3ms | fwd: 3575.4ms | loss: 13.1ms | bwd: 10843.5ms | clip: 0.0ms | optim: 23.9ms step 1031 | loss: 2.919817 | lr 1.8000e-04 | norm: 0.1891 | dt: 14468.81ms | tok/sec: 36235.7 | mem: 107435MB | Time Left: 04 hrs : 48 mins [TIMING] data: 8.4ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10834.4ms | clip: 0.0ms | optim: 23.9ms step 1032 | loss: 2.805831 | lr 1.8000e-04 | norm: 0.1939 | dt: 14490.06ms | tok/sec: 36182.6 | mem: 107435MB | Time Left: 04 hrs : 48 mins [TIMING] data: 8.3ms | fwd: 3581.3ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 23.8ms step 1033 | loss: 2.700475 | lr 1.8000e-04 | norm: 0.1630 | dt: 14475.67ms | tok/sec: 36218.6 | mem: 107435MB | Time Left: 04 hrs : 48 mins [TIMING] data: 8.3ms | fwd: 3575.7ms | loss: 13.0ms | bwd: 10841.8ms | clip: 0.0ms | optim: 23.7ms step 1034 | loss: 2.902546 | lr 1.8000e-04 | norm: 0.1914 | dt: 14487.58ms | tok/sec: 36188.8 | mem: 107435MB | Time Left: 04 hrs : 48 mins [TIMING] data: 8.4ms | fwd: 3577.7ms | loss: 13.0ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.8ms step 1035 | loss: 3.060051 | lr 1.8000e-04 | norm: 0.1987 | dt: 14482.02ms | tok/sec: 36202.7 | mem: 107435MB | Time Left: 04 hrs : 47 mins [TIMING] data: 8.7ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10844.3ms | clip: 0.0ms | optim: 23.8ms step 1036 | loss: 2.898318 | lr 1.8000e-04 | norm: 0.1827 | dt: 14478.70ms | tok/sec: 36211.0 | mem: 107435MB | Time Left: 04 hrs : 47 mins [TIMING] data: 8.3ms | fwd: 3591.0ms | loss: 13.1ms | bwd: 10829.2ms | clip: 0.0ms | optim: 23.8ms step 1037 | loss: 2.834133 | lr 1.8000e-04 | norm: 0.5997 | dt: 14469.67ms | tok/sec: 36233.6 | mem: 107435MB | Time Left: 04 hrs : 46 mins [TIMING] data: 8.3ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10832.7ms | clip: 0.0ms | optim: 23.8ms step 1038 | loss: 2.674307 | lr 1.8000e-04 | norm: 0.2371 | dt: 14471.32ms | tok/sec: 36229.5 | mem: 107435MB | Time Left: 04 hrs : 46 mins [TIMING] data: 8.6ms | fwd: 3579.5ms | loss: 13.0ms | bwd: 10833.0ms | clip: 0.0ms | optim: 23.8ms step 1039 | loss: 2.734721 | lr 1.8000e-04 | norm: 0.2068 | dt: 14487.00ms | tok/sec: 36190.2 | mem: 107435MB | Time Left: 04 hrs : 46 mins [TIMING] data: 8.3ms | fwd: 3583.7ms | loss: 13.0ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.9ms step 1040 | loss: 2.570559 | lr 1.8000e-04 | norm: 0.2116 | dt: 14520.08ms | tok/sec: 36107.8 | mem: 107435MB | Time Left: 04 hrs : 47 mins [TIMING] data: 8.3ms | fwd: 3622.4ms | loss: 13.1ms | bwd: 10839.3ms | clip: 0.0ms | optim: 23.9ms step 1041 | loss: 3.069093 | lr 1.8000e-04 | norm: 0.3075 | dt: 14498.88ms | tok/sec: 36160.6 | mem: 107435MB | Time Left: 04 hrs : 46 mins [TIMING] data: 8.2ms | fwd: 3612.9ms | loss: 13.0ms | bwd: 10827.6ms | clip: 0.0ms | optim: 23.9ms step 1042 | loss: 3.115431 | lr 1.8000e-04 | norm: 4.4178 | dt: 14473.93ms | tok/sec: 36222.9 | mem: 107435MB | Time Left: 04 hrs : 45 mins [TIMING] data: 8.3ms | fwd: 3577.1ms | loss: 13.1ms | bwd: 10838.5ms | clip: 0.0ms | optim: 23.8ms step 1043 | loss: 2.889689 | lr 1.8000e-04 | norm: 0.1911 | dt: 14498.01ms | tok/sec: 36162.8 | mem: 107435MB | Time Left: 04 hrs : 46 mins [TIMING] data: 8.3ms | fwd: 3598.9ms | loss: 13.1ms | bwd: 10840.8ms | clip: 0.0ms | optim: 23.7ms step 1044 | loss: 2.627629 | lr 1.8000e-04 | norm: 0.1833 | dt: 14476.89ms | tok/sec: 36215.5 | mem: 107435MB | Time Left: 04 hrs : 45 mins [TIMING] data: 8.8ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10838.7ms | clip: 0.0ms | optim: 23.8ms step 1045 | loss: 2.672990 | lr 1.8000e-04 | norm: 0.2044 | dt: 14473.81ms | tok/sec: 36223.2 | mem: 107435MB | Time Left: 04 hrs : 45 mins [TIMING] data: 8.3ms | fwd: 3573.9ms | loss: 13.0ms | bwd: 10841.4ms | clip: 0.0ms | optim: 24.0ms step 1046 | loss: 2.445233 | lr 1.8000e-04 | norm: 0.1887 | dt: 14484.02ms | tok/sec: 36197.7 | mem: 107435MB | Time Left: 04 hrs : 45 mins [TIMING] data: 8.3ms | fwd: 3574.6ms | loss: 13.0ms | bwd: 10851.2ms | clip: 0.0ms | optim: 23.8ms step 1047 | loss: 3.148376 | lr 1.8000e-04 | norm: 0.2344 | dt: 14484.50ms | tok/sec: 36196.5 | mem: 107435MB | Time Left: 04 hrs : 44 mins [TIMING] data: 8.3ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10844.7ms | clip: 0.0ms | optim: 23.8ms step 1048 | loss: 3.164443 | lr 1.8000e-04 | norm: 0.2219 | dt: 14494.94ms | tok/sec: 36170.4 | mem: 107435MB | Time Left: 04 hrs : 44 mins [TIMING] data: 8.3ms | fwd: 3585.0ms | loss: 13.0ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.9ms step 1049 | loss: 2.788805 | lr 1.8000e-04 | norm: 0.1926 | dt: 14486.39ms | tok/sec: 36191.8 | mem: 107435MB | Time Left: 04 hrs : 44 mins [TIMING] data: 8.4ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10847.5ms | clip: 0.0ms | optim: 24.1ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1050.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_950.ckpt step 1050 | loss: 2.963957 | lr 1.8000e-04 | norm: 0.1997 | dt: 14399.50ms | tok/sec: 36410.2 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3601.3ms | loss: 13.1ms | bwd: 10739.6ms | clip: 0.0ms | optim: 23.8ms step 1051 | loss: 3.156375 | lr 1.8000e-04 | norm: 0.1959 | dt: 14460.55ms | tok/sec: 36256.4 | mem: 107435MB | Time Left: 04 hrs : 43 mins [TIMING] data: 8.6ms | fwd: 3642.7ms | loss: 13.1ms | bwd: 10759.2ms | clip: 0.0ms | optim: 23.8ms step 1052 | loss: 2.901036 | lr 1.8000e-04 | norm: 0.1942 | dt: 14423.31ms | tok/sec: 36350.0 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3572.6ms | loss: 13.0ms | bwd: 10792.2ms | clip: 0.0ms | optim: 23.9ms step 1053 | loss: 2.847297 | lr 1.8000e-04 | norm: 0.2097 | dt: 14446.59ms | tok/sec: 36291.5 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3572.6ms | loss: 13.0ms | bwd: 10815.7ms | clip: 0.0ms | optim: 23.8ms step 1054 | loss: 3.130059 | lr 1.8000e-04 | norm: 0.2062 | dt: 14445.17ms | tok/sec: 36295.0 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3569.4ms | loss: 13.0ms | bwd: 10817.4ms | clip: 0.0ms | optim: 23.9ms step 1055 | loss: 2.995328 | lr 1.8000e-04 | norm: 0.2002 | dt: 14463.33ms | tok/sec: 36249.5 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3574.6ms | loss: 13.0ms | bwd: 10830.6ms | clip: 0.0ms | optim: 23.8ms step 1056 | loss: 3.027566 | lr 1.8000e-04 | norm: 0.2049 | dt: 14470.13ms | tok/sec: 36232.4 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3578.7ms | loss: 13.0ms | bwd: 10833.2ms | clip: 0.0ms | optim: 23.8ms step 1057 | loss: 2.690526 | lr 1.8000e-04 | norm: 0.1719 | dt: 14467.57ms | tok/sec: 36238.8 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3572.0ms | loss: 13.1ms | bwd: 10836.6ms | clip: 0.0ms | optim: 23.8ms step 1058 | loss: 3.125510 | lr 1.8000e-04 | norm: 0.2104 | dt: 14470.81ms | tok/sec: 36230.7 | mem: 107435MB | Time Left: 04 hrs : 41 mins [TIMING] data: 8.4ms | fwd: 3578.8ms | loss: 13.0ms | bwd: 10833.6ms | clip: 0.0ms | optim: 23.9ms step 1059 | loss: 3.316837 | lr 1.8000e-04 | norm: 0.2029 | dt: 14539.74ms | tok/sec: 36059.0 | mem: 107435MB | Time Left: 04 hrs : 43 mins [TIMING] data: 8.3ms | fwd: 3659.7ms | loss: 13.1ms | bwd: 10821.6ms | clip: 0.0ms | optim: 23.9ms step 1060 | loss: 2.791296 | lr 1.8000e-04 | norm: 0.2356 | dt: 14477.43ms | tok/sec: 36214.2 | mem: 107435MB | Time Left: 04 hrs : 41 mins [TIMING] data: 8.5ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10836.4ms | clip: 0.0ms | optim: 23.8ms step 1061 | loss: 2.802635 | lr 1.8000e-04 | norm: 0.1966 | dt: 14534.03ms | tok/sec: 36073.1 | mem: 107435MB | Time Left: 04 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3650.1ms | loss: 13.1ms | bwd: 10825.4ms | clip: 0.0ms | optim: 23.9ms step 1062 | loss: 2.887814 | lr 1.8000e-04 | norm: 0.1849 | dt: 14469.29ms | tok/sec: 36234.5 | mem: 107435MB | Time Left: 04 hrs : 40 mins [TIMING] data: 8.6ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10834.5ms | clip: 0.0ms | optim: 24.2ms step 1063 | loss: 3.074479 | lr 1.8000e-04 | norm: 0.1894 | dt: 14478.97ms | tok/sec: 36210.3 | mem: 107435MB | Time Left: 04 hrs : 40 mins [TIMING] data: 8.3ms | fwd: 3577.7ms | loss: 13.2ms | bwd: 10842.7ms | clip: 0.0ms | optim: 23.9ms step 1064 | loss: 2.931401 | lr 1.8000e-04 | norm: 0.1819 | dt: 14489.72ms | tok/sec: 36183.4 | mem: 107435MB | Time Left: 04 hrs : 40 mins [TIMING] data: 8.2ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10855.7ms | clip: 0.0ms | optim: 23.7ms step 1065 | loss: 3.125021 | lr 1.8000e-04 | norm: 0.1785 | dt: 14491.42ms | tok/sec: 36179.2 | mem: 107435MB | Time Left: 04 hrs : 40 mins [TIMING] data: 8.3ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.9ms step 1066 | loss: 3.128237 | lr 1.8000e-04 | norm: 0.2203 | dt: 14485.62ms | tok/sec: 36193.7 | mem: 107435MB | Time Left: 04 hrs : 40 mins [TIMING] data: 8.4ms | fwd: 3577.9ms | loss: 13.0ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.8ms step 1067 | loss: 2.873320 | lr 1.8000e-04 | norm: 0.1782 | dt: 14486.17ms | tok/sec: 36192.3 | mem: 107435MB | Time Left: 04 hrs : 40 mins [TIMING] data: 8.3ms | fwd: 3582.1ms | loss: 13.0ms | bwd: 10845.7ms | clip: 0.0ms | optim: 23.9ms step 1068 | loss: 2.697539 | lr 1.8000e-04 | norm: 0.1747 | dt: 14481.29ms | tok/sec: 36204.5 | mem: 107435MB | Time Left: 04 hrs : 39 mins [TIMING] data: 8.4ms | fwd: 3575.2ms | loss: 13.0ms | bwd: 10847.7ms | clip: 0.0ms | optim: 23.8ms step 1069 | loss: 3.091111 | lr 1.8000e-04 | norm: 0.2084 | dt: 14483.21ms | tok/sec: 36199.7 | mem: 107435MB | Time Left: 04 hrs : 39 mins [TIMING] data: 8.3ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10847.0ms | clip: 0.0ms | optim: 23.8ms step 1070 | loss: 2.719641 | lr 1.8000e-04 | norm: 0.1908 | dt: 14545.14ms | tok/sec: 36045.6 | mem: 107435MB | Time Left: 04 hrs : 40 mins [TIMING] data: 8.4ms | fwd: 3655.5ms | loss: 13.1ms | bwd: 10831.1ms | clip: 0.0ms | optim: 23.9ms step 1071 | loss: 3.172497 | lr 1.8000e-04 | norm: 0.1775 | dt: 14487.45ms | tok/sec: 36189.1 | mem: 107435MB | Time Left: 04 hrs : 39 mins [TIMING] data: 8.3ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 23.9ms step 1072 | loss: 2.970168 | lr 1.8000e-04 | norm: 0.2726 | dt: 14473.87ms | tok/sec: 36223.1 | mem: 107435MB | Time Left: 04 hrs : 38 mins [TIMING] data: 8.5ms | fwd: 3584.6ms | loss: 13.1ms | bwd: 10830.5ms | clip: 0.0ms | optim: 23.8ms step 1073 | loss: 2.753874 | lr 1.8000e-04 | norm: 0.1928 | dt: 14493.67ms | tok/sec: 36173.6 | mem: 107435MB | Time Left: 04 hrs : 38 mins [TIMING] data: 8.3ms | fwd: 3613.1ms | loss: 13.1ms | bwd: 10822.2ms | clip: 0.0ms | optim: 23.9ms step 1074 | loss: 3.162558 | lr 1.8000e-04 | norm: 0.3821 | dt: 14494.86ms | tok/sec: 36170.6 | mem: 107435MB | Time Left: 04 hrs : 38 mins [TIMING] data: 8.3ms | fwd: 3588.7ms | loss: 13.0ms | bwd: 10847.8ms | clip: 0.0ms | optim: 24.1ms step 1075 | loss: 2.877597 | lr 1.8000e-04 | norm: 0.1783 | dt: 14476.61ms | tok/sec: 36216.2 | mem: 107435MB | Time Left: 04 hrs : 37 mins [TIMING] data: 8.3ms | fwd: 3573.6ms | loss: 13.1ms | bwd: 10844.5ms | clip: 0.0ms | optim: 23.8ms step 1076 | loss: 3.214265 | lr 1.8000e-04 | norm: 0.2388 | dt: 14478.21ms | tok/sec: 36212.2 | mem: 107435MB | Time Left: 04 hrs : 37 mins [TIMING] data: 8.3ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10841.7ms | clip: 0.0ms | optim: 23.7ms step 1077 | loss: 2.806963 | lr 1.8000e-04 | norm: 0.1976 | dt: 14485.08ms | tok/sec: 36195.0 | mem: 107435MB | Time Left: 04 hrs : 37 mins [TIMING] data: 8.3ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 24.0ms step 1078 | loss: 2.882992 | lr 1.8000e-04 | norm: 0.1815 | dt: 14487.06ms | tok/sec: 36190.1 | mem: 107435MB | Time Left: 04 hrs : 37 mins [TIMING] data: 8.2ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10851.7ms | clip: 0.0ms | optim: 23.8ms step 1079 | loss: 3.172485 | lr 1.8000e-04 | norm: 0.2003 | dt: 14481.25ms | tok/sec: 36204.6 | mem: 107435MB | Time Left: 04 hrs : 37 mins [TIMING] data: 8.3ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10847.7ms | clip: 0.0ms | optim: 23.8ms step 1080 | loss: 2.732080 | lr 1.8000e-04 | norm: 0.1673 | dt: 14489.83ms | tok/sec: 36183.2 | mem: 107435MB | Time Left: 04 hrs : 36 mins [TIMING] data: 8.3ms | fwd: 3580.9ms | loss: 13.0ms | bwd: 10850.5ms | clip: 0.0ms | optim: 24.0ms step 1081 | loss: 2.748728 | lr 1.8000e-04 | norm: 0.2323 | dt: 14490.77ms | tok/sec: 36180.8 | mem: 107435MB | Time Left: 04 hrs : 36 mins [TIMING] data: 8.2ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10854.5ms | clip: 0.0ms | optim: 23.8ms step 1082 | loss: 3.052840 | lr 1.8000e-04 | norm: 0.1849 | dt: 14487.56ms | tok/sec: 36188.8 | mem: 107435MB | Time Left: 04 hrs : 36 mins [TIMING] data: 8.4ms | fwd: 3576.6ms | loss: 13.0ms | bwd: 10852.6ms | clip: 0.0ms | optim: 23.9ms step 1083 | loss: 3.153268 | lr 1.8000e-04 | norm: 0.2078 | dt: 14505.29ms | tok/sec: 36144.6 | mem: 107435MB | Time Left: 04 hrs : 36 mins [TIMING] data: 8.4ms | fwd: 3615.3ms | loss: 13.0ms | bwd: 10831.6ms | clip: 0.0ms | optim: 23.9ms step 1084 | loss: 2.540907 | lr 1.8000e-04 | norm: 0.1828 | dt: 14548.16ms | tok/sec: 36038.1 | mem: 107435MB | Time Left: 04 hrs : 37 mins [TIMING] data: 8.3ms | fwd: 3661.8ms | loss: 13.1ms | bwd: 10828.0ms | clip: 0.0ms | optim: 23.8ms step 1085 | loss: 3.164791 | lr 1.8000e-04 | norm: 0.1961 | dt: 14485.93ms | tok/sec: 36192.9 | mem: 107435MB | Time Left: 04 hrs : 35 mins [TIMING] data: 8.3ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10846.0ms | clip: 0.0ms | optim: 23.8ms step 1086 | loss: 2.865286 | lr 1.8000e-04 | norm: 0.2794 | dt: 14482.72ms | tok/sec: 36200.9 | mem: 107435MB | Time Left: 04 hrs : 35 mins [TIMING] data: 8.3ms | fwd: 3577.1ms | loss: 13.1ms | bwd: 10847.3ms | clip: 0.0ms | optim: 23.9ms step 1087 | loss: 2.932539 | lr 1.8000e-04 | norm: 0.2151 | dt: 14480.23ms | tok/sec: 36207.2 | mem: 107435MB | Time Left: 04 hrs : 35 mins [TIMING] data: 8.3ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10845.9ms | clip: 0.0ms | optim: 23.9ms step 1088 | loss: 3.100545 | lr 1.8000e-04 | norm: 0.1965 | dt: 14481.60ms | tok/sec: 36203.7 | mem: 107435MB | Time Left: 04 hrs : 34 mins [TIMING] data: 8.4ms | fwd: 3577.7ms | loss: 13.0ms | bwd: 10844.6ms | clip: 0.0ms | optim: 23.8ms step 1089 | loss: 2.901849 | lr 1.8000e-04 | norm: 0.1857 | dt: 14478.63ms | tok/sec: 36211.2 | mem: 107435MB | Time Left: 04 hrs : 34 mins [TIMING] data: 8.3ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10843.2ms | clip: 0.0ms | optim: 23.9ms step 1090 | loss: 2.683437 | lr 1.8000e-04 | norm: 0.2229 | dt: 14468.33ms | tok/sec: 36236.9 | mem: 107435MB | Time Left: 04 hrs : 34 mins [TIMING] data: 8.3ms | fwd: 3574.9ms | loss: 13.0ms | bwd: 10835.2ms | clip: 0.0ms | optim: 23.8ms step 1091 | loss: 2.780293 | lr 1.8000e-04 | norm: 0.2032 | dt: 14479.11ms | tok/sec: 36210.0 | mem: 107435MB | Time Left: 04 hrs : 34 mins [TIMING] data: 8.2ms | fwd: 3580.0ms | loss: 13.0ms | bwd: 10840.9ms | clip: 0.0ms | optim: 23.8ms step 1092 | loss: 2.963082 | lr 1.8000e-04 | norm: 0.1826 | dt: 14475.56ms | tok/sec: 36218.8 | mem: 107435MB | Time Left: 04 hrs : 33 mins [TIMING] data: 8.3ms | fwd: 3576.8ms | loss: 13.0ms | bwd: 10840.6ms | clip: 0.0ms | optim: 23.8ms step 1093 | loss: 2.846881 | lr 1.8000e-04 | norm: 0.1713 | dt: 14481.27ms | tok/sec: 36204.6 | mem: 107435MB | Time Left: 04 hrs : 33 mins [TIMING] data: 8.3ms | fwd: 3577.0ms | loss: 13.0ms | bwd: 10846.0ms | clip: 0.0ms | optim: 23.8ms step 1094 | loss: 2.947020 | lr 1.8000e-04 | norm: 0.2248 | dt: 14474.10ms | tok/sec: 36222.5 | mem: 107435MB | Time Left: 04 hrs : 33 mins [TIMING] data: 8.4ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10833.3ms | clip: 0.0ms | optim: 23.8ms step 1095 | loss: 2.820344 | lr 1.8000e-04 | norm: 0.2018 | dt: 14477.07ms | tok/sec: 36215.1 | mem: 107435MB | Time Left: 04 hrs : 33 mins [TIMING] data: 8.4ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10841.3ms | clip: 0.0ms | optim: 23.9ms step 1096 | loss: 2.906204 | lr 1.8000e-04 | norm: 0.1948 | dt: 14471.68ms | tok/sec: 36228.6 | mem: 107435MB | Time Left: 04 hrs : 32 mins [TIMING] data: 8.3ms | fwd: 3573.0ms | loss: 13.0ms | bwd: 10840.4ms | clip: 0.0ms | optim: 23.8ms step 1097 | loss: 2.991589 | lr 1.8000e-04 | norm: 0.2036 | dt: 14486.61ms | tok/sec: 36191.2 | mem: 107435MB | Time Left: 04 hrs : 32 mins [TIMING] data: 8.4ms | fwd: 3573.8ms | loss: 13.0ms | bwd: 10854.6ms | clip: 0.0ms | optim: 23.9ms step 1098 | loss: 2.679496 | lr 1.8000e-04 | norm: 0.1901 | dt: 14472.51ms | tok/sec: 36226.5 | mem: 107435MB | Time Left: 04 hrs : 32 mins [TIMING] data: 8.2ms | fwd: 3572.9ms | loss: 13.0ms | bwd: 10841.2ms | clip: 0.0ms | optim: 23.7ms step 1099 | loss: 2.558936 | lr 1.8000e-04 | norm: 0.2082 | dt: 14467.96ms | tok/sec: 36237.9 | mem: 107435MB | Time Left: 04 hrs : 31 mins [TIMING] data: 8.4ms | fwd: 3571.9ms | loss: 13.0ms | bwd: 10837.8ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1100.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1000.ckpt === Llama Context Parallel Training Script (bluscriptCP) === [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) ================ Configuration (bluscriptCP) ================ d_model/layers : 1536 / 36 q/kv heads : 12 / 4 (G=3, hd=128) ffn_hidden : 4096 (SwiGLU) context_length : 32768 vocab_size : 50304 tying=0 B x T : 1 x 32768 world_size : 2 (dp=1 x cp=2) CP_ATTN_MODE : ring (load_balance=HeadTail) global_batch : 524288 (grad_accum=16) precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] max_lr/min_lr : 0.0006 / 6e-05 warmup/steps : 100 / 2227 (WSD: stable=1177, decay tail=950) checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp) ============================================================= Initializing model on CUDA device 0... [YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693 Parameters: 1060624896 Parameters per GPU: 1060624896 max_steps: 2227 warmup_steps: 100 [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] [ZeRO-1] DataParallel is DISABLED (both would double-reduce) [precision] bf16 params=217, all with fp32 master weights found 12 shards for split train found 1 shards for split val Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (resume run 2) [rope_theta] 500000 from checkpoints_bluscriptcp/blumodelcp_run2.meta (authoritative) Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 2) [CheckpointManager] Auto-loading latest: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1100.ckpt [Resume] run 2 from step 1100 (loss 2.55894) [resume] fast-forwarded data loader 17600 batches (576716800 tokens) [re-warmup] anchored at step 750 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 1100 (past ramp, in cosine-decay) Starting training... [CP ring P2P] OVERLAP: dedicated non-blocking stream [CP bwd] overlap=ON step 1100 | loss: 2.747939 | lr 1.8000e-04 | norm: 0.1836 | dt: 16924.17ms | tok/sec: 30978.6 | mem: 107563MB | Time Left: 05 hrs : 17 mins [TIMING] data: 10.1ms | fwd: 6118.3ms | loss: 16.2ms | bwd: 10735.0ms | clip: 0.0ms | optim: 24.4ms step 1101 | loss: 2.929918 | lr 1.8000e-04 | norm: 0.1821 | dt: 14347.73ms | tok/sec: 36541.5 | mem: 107563MB | Time Left: 04 hrs : 29 mins [TIMING] data: 6.9ms | fwd: 3548.9ms | loss: 13.1ms | bwd: 10741.3ms | clip: 0.0ms | optim: 23.9ms step 1102 | loss: 3.300049 | lr 1.8000e-04 | norm: 0.2095 | dt: 14387.05ms | tok/sec: 36441.7 | mem: 107563MB | Time Left: 04 hrs : 29 mins [TIMING] data: 6.9ms | fwd: 3562.2ms | loss: 13.1ms | bwd: 10767.5ms | clip: 0.0ms | optim: 23.8ms step 1103 | loss: 2.758244 | lr 1.8000e-04 | norm: 0.1828 | dt: 14435.04ms | tok/sec: 36320.5 | mem: 107563MB | Time Left: 04 hrs : 30 mins [TIMING] data: 6.9ms | fwd: 3583.2ms | loss: 13.0ms | bwd: 10794.5ms | clip: 0.0ms | optim: 23.8ms step 1104 | loss: 2.924301 | lr 1.8000e-04 | norm: 0.1637 | dt: 14459.94ms | tok/sec: 36258.0 | mem: 107563MB | Time Left: 04 hrs : 30 mins [TIMING] data: 6.8ms | fwd: 3588.4ms | loss: 13.0ms | bwd: 10814.4ms | clip: 0.0ms | optim: 23.8ms step 1105 | loss: 2.837910 | lr 1.8000e-04 | norm: 0.2090 | dt: 14518.83ms | tok/sec: 36110.9 | mem: 107563MB | Time Left: 04 hrs : 31 mins [TIMING] data: 6.8ms | fwd: 3603.9ms | loss: 13.0ms | bwd: 10858.0ms | clip: 0.0ms | optim: 23.8ms step 1106 | loss: 2.868182 | lr 1.8000e-04 | norm: 0.1902 | dt: 14574.56ms | tok/sec: 35972.8 | mem: 107563MB | Time Left: 04 hrs : 32 mins [TIMING] data: 6.7ms | fwd: 3659.1ms | loss: 13.1ms | bwd: 10858.3ms | clip: 0.0ms | optim: 23.8ms step 1107 | loss: 2.946159 | lr 1.8000e-04 | norm: 0.1651 | dt: 14541.99ms | tok/sec: 36053.4 | mem: 107563MB | Time Left: 04 hrs : 31 mins [TIMING] data: 6.8ms | fwd: 3629.8ms | loss: 13.0ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.9ms step 1108 | loss: 2.709523 | lr 1.8000e-04 | norm: 0.1687 | dt: 14492.52ms | tok/sec: 36176.4 | mem: 107563MB | Time Left: 04 hrs : 30 mins [TIMING] data: 7.0ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.9ms step 1109 | loss: 2.895964 | lr 1.8000e-04 | norm: 0.1880 | dt: 14500.96ms | tok/sec: 36155.4 | mem: 107563MB | Time Left: 04 hrs : 30 mins [TIMING] data: 6.7ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10862.7ms | clip: 0.0ms | optim: 23.8ms step 1110 | loss: 3.011069 | lr 1.8000e-04 | norm: 0.2341 | dt: 14479.67ms | tok/sec: 36208.6 | mem: 107563MB | Time Left: 04 hrs : 29 mins [TIMING] data: 6.8ms | fwd: 3586.5ms | loss: 13.1ms | bwd: 10835.8ms | clip: 0.0ms | optim: 23.8ms step 1111 | loss: 2.654892 | lr 1.8000e-04 | norm: 0.1938 | dt: 14483.67ms | tok/sec: 36198.6 | mem: 107563MB | Time Left: 04 hrs : 29 mins [TIMING] data: 6.7ms | fwd: 3595.2ms | loss: 13.1ms | bwd: 10831.3ms | clip: 0.0ms | optim: 23.8ms step 1112 | loss: 3.121296 | lr 1.8000e-04 | norm: 0.2122 | dt: 14485.60ms | tok/sec: 36193.7 | mem: 107563MB | Time Left: 04 hrs : 29 mins [TIMING] data: 6.9ms | fwd: 3592.6ms | loss: 13.0ms | bwd: 10835.9ms | clip: 0.0ms | optim: 23.8ms step 1113 | loss: 2.992892 | lr 1.8000e-04 | norm: 0.2022 | dt: 14542.01ms | tok/sec: 36053.3 | mem: 107563MB | Time Left: 04 hrs : 29 mins [TIMING] data: 6.7ms | fwd: 3658.4ms | loss: 13.1ms | bwd: 10826.7ms | clip: 0.0ms | optim: 23.7ms step 1114 | loss: 3.030076 | lr 1.8000e-04 | norm: 0.2067 | dt: 14524.86ms | tok/sec: 36095.9 | mem: 107563MB | Time Left: 04 hrs : 29 mins [TIMING] data: 6.8ms | fwd: 3623.2ms | loss: 13.1ms | bwd: 10844.4ms | clip: 0.0ms | optim: 23.9ms step 1115 | loss: 2.901121 | lr 1.8000e-04 | norm: 0.2375 | dt: 14489.05ms | tok/sec: 36185.1 | mem: 107563MB | Time Left: 04 hrs : 28 mins [TIMING] data: 6.8ms | fwd: 3584.0ms | loss: 13.0ms | bwd: 10848.0ms | clip: 0.0ms | optim: 23.8ms step 1116 | loss: 3.099316 | lr 1.8000e-04 | norm: 0.2044 | dt: 14504.51ms | tok/sec: 36146.5 | mem: 107563MB | Time Left: 04 hrs : 28 mins [TIMING] data: 6.8ms | fwd: 3588.5ms | loss: 13.1ms | bwd: 10858.9ms | clip: 0.0ms | optim: 23.7ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00006.bin 100000000 step 1117 | loss: 2.828031 | lr 1.8000e-04 | norm: 0.2590 | dt: 14501.81ms | tok/sec: 36153.3 | mem: 107563MB | Time Left: 04 hrs : 28 mins [TIMING] data: 7.5ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10861.7ms | clip: 0.0ms | optim: 23.8ms step 1118 | loss: 2.853995 | lr 1.8000e-04 | norm: 0.2082 | dt: 14504.07ms | tok/sec: 36147.6 | mem: 107563MB | Time Left: 04 hrs : 28 mins [TIMING] data: 6.9ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10861.7ms | clip: 0.0ms | optim: 23.8ms step 1119 | loss: 2.981134 | lr 1.8000e-04 | norm: 0.2329 | dt: 14501.90ms | tok/sec: 36153.1 | mem: 107563MB | Time Left: 04 hrs : 27 mins [TIMING] data: 6.9ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10862.0ms | clip: 0.0ms | optim: 23.8ms step 1120 | loss: 2.929284 | lr 1.8000e-04 | norm: 0.2863 | dt: 14496.93ms | tok/sec: 36165.4 | mem: 107563MB | Time Left: 04 hrs : 27 mins [TIMING] data: 6.9ms | fwd: 3595.2ms | loss: 13.1ms | bwd: 10844.6ms | clip: 0.0ms | optim: 23.8ms step 1121 | loss: 2.775578 | lr 1.8000e-04 | norm: 0.2072 | dt: 14505.96ms | tok/sec: 36142.9 | mem: 107563MB | Time Left: 04 hrs : 27 mins [TIMING] data: 6.8ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10867.5ms | clip: 0.0ms | optim: 23.8ms step 1122 | loss: 2.889667 | lr 1.8000e-04 | norm: 0.3808 | dt: 14499.20ms | tok/sec: 36159.8 | mem: 107563MB | Time Left: 04 hrs : 27 mins [TIMING] data: 6.9ms | fwd: 3583.0ms | loss: 13.0ms | bwd: 10857.9ms | clip: 0.0ms | optim: 23.8ms step 1123 | loss: 2.935147 | lr 1.8000e-04 | norm: 0.1881 | dt: 14496.50ms | tok/sec: 36166.5 | mem: 107563MB | Time Left: 04 hrs : 26 mins [TIMING] data: 6.9ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10856.4ms | clip: 0.0ms | optim: 23.8ms step 1124 | loss: 2.707117 | lr 1.8000e-04 | norm: 0.1989 | dt: 14489.73ms | tok/sec: 36183.4 | mem: 107563MB | Time Left: 04 hrs : 26 mins [TIMING] data: 6.9ms | fwd: 3577.0ms | loss: 13.0ms | bwd: 10855.6ms | clip: 0.0ms | optim: 23.7ms step 1125 | loss: 2.750706 | lr 1.8000e-04 | norm: 0.2235 | dt: 14508.77ms | tok/sec: 36135.9 | mem: 107563MB | Time Left: 04 hrs : 26 mins [TIMING] data: 6.7ms | fwd: 3586.5ms | loss: 13.1ms | bwd: 10865.2ms | clip: 0.0ms | optim: 23.8ms step 1126 | loss: 3.087324 | lr 1.8000e-04 | norm: 0.2117 | dt: 14506.90ms | tok/sec: 36140.6 | mem: 107563MB | Time Left: 04 hrs : 26 mins [TIMING] data: 6.8ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10869.0ms | clip: 0.0ms | optim: 23.8ms step 1127 | loss: 2.984977 | lr 1.8000e-04 | norm: 0.2034 | dt: 14499.59ms | tok/sec: 36158.8 | mem: 107563MB | Time Left: 04 hrs : 25 mins [TIMING] data: 6.7ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10864.1ms | clip: 0.0ms | optim: 23.8ms step 1128 | loss: 2.727384 | lr 1.8000e-04 | norm: 0.5425 | dt: 14501.91ms | tok/sec: 36153.0 | mem: 107563MB | Time Left: 04 hrs : 25 mins [TIMING] data: 6.8ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10865.7ms | clip: 0.0ms | optim: 23.8ms step 1129 | loss: 2.771081 | lr 1.8000e-04 | norm: 0.2092 | dt: 14496.96ms | tok/sec: 36165.4 | mem: 107563MB | Time Left: 04 hrs : 25 mins [TIMING] data: 6.7ms | fwd: 3599.5ms | loss: 13.1ms | bwd: 10840.3ms | clip: 0.0ms | optim: 23.9ms step 1130 | loss: 3.064333 | lr 1.8000e-04 | norm: 0.2221 | dt: 14510.68ms | tok/sec: 36131.2 | mem: 107563MB | Time Left: 04 hrs : 25 mins [TIMING] data: 6.7ms | fwd: 3577.3ms | loss: 13.0ms | bwd: 10876.4ms | clip: 0.0ms | optim: 23.9ms step 1131 | loss: 2.822470 | lr 1.8000e-04 | norm: 0.2013 | dt: 14492.99ms | tok/sec: 36175.3 | mem: 107563MB | Time Left: 04 hrs : 24 mins [TIMING] data: 6.7ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10859.2ms | clip: 0.0ms | optim: 23.7ms step 1132 | loss: 3.003887 | lr 1.8000e-04 | norm: 0.1988 | dt: 14504.35ms | tok/sec: 36146.9 | mem: 107563MB | Time Left: 04 hrs : 24 mins [TIMING] data: 6.8ms | fwd: 3581.5ms | loss: 13.0ms | bwd: 10865.5ms | clip: 0.0ms | optim: 23.8ms step 1133 | loss: 2.836367 | lr 1.8000e-04 | norm: 0.1799 | dt: 14489.22ms | tok/sec: 36184.7 | mem: 107563MB | Time Left: 04 hrs : 24 mins [TIMING] data: 6.8ms | fwd: 3580.1ms | loss: 13.0ms | bwd: 10852.1ms | clip: 0.0ms | optim: 23.8ms step 1134 | loss: 2.863930 | lr 1.8000e-04 | norm: 0.1795 | dt: 14491.35ms | tok/sec: 36179.4 | mem: 107563MB | Time Left: 04 hrs : 23 mins [TIMING] data: 6.7ms | fwd: 3580.5ms | loss: 13.0ms | bwd: 10853.9ms | clip: 0.0ms | optim: 23.8ms step 1135 | loss: 2.892244 | lr 1.8000e-04 | norm: 0.1617 | dt: 14494.40ms | tok/sec: 36171.8 | mem: 107563MB | Time Left: 04 hrs : 23 mins [TIMING] data: 6.7ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10858.9ms | clip: 0.0ms | optim: 23.8ms step 1136 | loss: 2.788088 | lr 1.8000e-04 | norm: 0.2523 | dt: 14504.07ms | tok/sec: 36147.6 | mem: 107563MB | Time Left: 04 hrs : 23 mins [TIMING] data: 6.9ms | fwd: 3579.2ms | loss: 13.0ms | bwd: 10867.7ms | clip: 0.0ms | optim: 23.8ms step 1137 | loss: 2.966834 | lr 1.8000e-04 | norm: 0.2748 | dt: 14511.33ms | tok/sec: 36129.6 | mem: 107563MB | Time Left: 04 hrs : 23 mins [TIMING] data: 6.7ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10870.0ms | clip: 0.0ms | optim: 24.0ms step 1138 | loss: 2.879273 | lr 1.8000e-04 | norm: 0.1687 | dt: 14508.40ms | tok/sec: 36136.9 | mem: 107563MB | Time Left: 04 hrs : 23 mins [TIMING] data: 6.8ms | fwd: 3585.1ms | loss: 13.1ms | bwd: 10866.2ms | clip: 0.0ms | optim: 23.7ms step 1139 | loss: 2.885215 | lr 1.8000e-04 | norm: 0.1676 | dt: 14506.94ms | tok/sec: 36140.5 | mem: 107563MB | Time Left: 04 hrs : 23 mins [TIMING] data: 6.8ms | fwd: 3590.1ms | loss: 13.0ms | bwd: 10859.7ms | clip: 0.0ms | optim: 23.8ms step 1140 | loss: 2.700619 | lr 1.8000e-04 | norm: 0.2009 | dt: 14508.38ms | tok/sec: 36136.9 | mem: 107563MB | Time Left: 04 hrs : 22 mins [TIMING] data: 6.7ms | fwd: 3584.1ms | loss: 13.0ms | bwd: 10867.1ms | clip: 0.0ms | optim: 23.9ms step 1141 | loss: 2.686164 | lr 1.8000e-04 | norm: 0.1891 | dt: 14505.65ms | tok/sec: 36143.7 | mem: 107563MB | Time Left: 04 hrs : 22 mins [TIMING] data: 6.9ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10865.0ms | clip: 0.0ms | optim: 23.8ms step 1142 | loss: 2.842885 | lr 1.8000e-04 | norm: 0.1808 | dt: 14502.02ms | tok/sec: 36152.8 | mem: 107563MB | Time Left: 04 hrs : 22 mins [TIMING] data: 6.8ms | fwd: 3585.0ms | loss: 13.1ms | bwd: 10859.9ms | clip: 0.0ms | optim: 23.8ms step 1143 | loss: 2.792494 | lr 1.8000e-04 | norm: 0.1976 | dt: 14510.81ms | tok/sec: 36130.9 | mem: 107563MB | Time Left: 04 hrs : 22 mins [TIMING] data: 6.8ms | fwd: 3592.4ms | loss: 13.0ms | bwd: 10861.5ms | clip: 0.0ms | optim: 23.8ms step 1144 | loss: 2.848651 | lr 1.8000e-04 | norm: 0.1701 | dt: 14508.40ms | tok/sec: 36136.9 | mem: 107563MB | Time Left: 04 hrs : 21 mins [TIMING] data: 6.7ms | fwd: 3580.4ms | loss: 13.0ms | bwd: 10871.0ms | clip: 0.0ms | optim: 23.9ms step 1145 | loss: 2.959126 | lr 1.8000e-04 | norm: 0.2136 | dt: 14508.46ms | tok/sec: 36136.7 | mem: 107563MB | Time Left: 04 hrs : 21 mins [TIMING] data: 6.8ms | fwd: 3583.0ms | loss: 13.0ms | bwd: 10868.4ms | clip: 0.0ms | optim: 23.9ms step 1146 | loss: 2.656505 | lr 1.8000e-04 | norm: 0.1811 | dt: 14500.10ms | tok/sec: 36157.5 | mem: 107563MB | Time Left: 04 hrs : 21 mins [TIMING] data: 6.9ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10859.8ms | clip: 0.0ms | optim: 23.8ms step 1147 | loss: 2.742019 | lr 1.8000e-04 | norm: 0.1802 | dt: 14505.03ms | tok/sec: 36145.3 | mem: 107563MB | Time Left: 04 hrs : 21 mins [TIMING] data: 6.9ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10866.2ms | clip: 0.0ms | optim: 23.8ms step 1148 | loss: 2.830948 | lr 1.8000e-04 | norm: 0.1873 | dt: 14507.39ms | tok/sec: 36139.4 | mem: 107563MB | Time Left: 04 hrs : 20 mins [TIMING] data: 6.8ms | fwd: 3584.6ms | loss: 13.1ms | bwd: 10865.5ms | clip: 0.0ms | optim: 23.9ms step 1149 | loss: 2.773872 | lr 1.8000e-04 | norm: 0.1745 | dt: 14500.45ms | tok/sec: 36156.7 | mem: 107563MB | Time Left: 04 hrs : 20 mins [TIMING] data: 6.7ms | fwd: 3598.9ms | loss: 13.1ms | bwd: 10844.5ms | clip: 0.0ms | optim: 23.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1150.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1050.ckpt step 1150 | loss: 3.092772 | lr 1.8000e-04 | norm: 0.2016 | dt: 14377.00ms | tok/sec: 36467.1 | mem: 107563MB | Time Left: 04 hrs : 18 mins [TIMING] data: 8.2ms | fwd: 3573.1ms | loss: 13.1ms | bwd: 10744.9ms | clip: 0.0ms | optim: 23.8ms step 1151 | loss: 2.971756 | lr 1.8000e-04 | norm: 0.1693 | dt: 14411.63ms | tok/sec: 36379.5 | mem: 107563MB | Time Left: 04 hrs : 18 mins [TIMING] data: 7.4ms | fwd: 3569.6ms | loss: 13.1ms | bwd: 10784.0ms | clip: 0.0ms | optim: 23.8ms step 1152 | loss: 2.936946 | lr 1.8000e-04 | norm: 0.1662 | dt: 14454.31ms | tok/sec: 36272.1 | mem: 107563MB | Time Left: 04 hrs : 18 mins [TIMING] data: 7.5ms | fwd: 3607.8ms | loss: 13.1ms | bwd: 10788.0ms | clip: 0.0ms | optim: 23.8ms step 1153 | loss: 3.011013 | lr 1.8000e-04 | norm: 0.1949 | dt: 14421.38ms | tok/sec: 36354.9 | mem: 107563MB | Time Left: 04 hrs : 18 mins [TIMING] data: 7.4ms | fwd: 3568.2ms | loss: 13.1ms | bwd: 10795.3ms | clip: 0.0ms | optim: 23.8ms step 1154 | loss: 2.804275 | lr 1.8000e-04 | norm: 0.1725 | dt: 14454.55ms | tok/sec: 36271.5 | mem: 107563MB | Time Left: 04 hrs : 18 mins [TIMING] data: 7.5ms | fwd: 3585.7ms | loss: 13.0ms | bwd: 10811.1ms | clip: 0.0ms | optim: 23.8ms step 1155 | loss: 2.879763 | lr 1.8000e-04 | norm: 0.2169 | dt: 14448.79ms | tok/sec: 36285.9 | mem: 107563MB | Time Left: 04 hrs : 18 mins [TIMING] data: 7.5ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10807.8ms | clip: 0.0ms | optim: 24.0ms step 1156 | loss: 2.782462 | lr 1.8000e-04 | norm: 0.2055 | dt: 14465.19ms | tok/sec: 36244.8 | mem: 107563MB | Time Left: 04 hrs : 18 mins [TIMING] data: 7.5ms | fwd: 3577.9ms | loss: 13.0ms | bwd: 10829.1ms | clip: 0.0ms | optim: 23.8ms step 1157 | loss: 2.890258 | lr 1.8000e-04 | norm: 0.2263 | dt: 14464.20ms | tok/sec: 36247.3 | mem: 107563MB | Time Left: 04 hrs : 17 mins [TIMING] data: 7.5ms | fwd: 3580.6ms | loss: 13.0ms | bwd: 10825.7ms | clip: 0.0ms | optim: 23.8ms step 1158 | loss: 2.971067 | lr 1.8000e-04 | norm: 0.2390 | dt: 14461.56ms | tok/sec: 36253.9 | mem: 107563MB | Time Left: 04 hrs : 17 mins [TIMING] data: 7.5ms | fwd: 3576.3ms | loss: 13.0ms | bwd: 10827.5ms | clip: 0.0ms | optim: 23.8ms step 1159 | loss: 2.872422 | lr 1.8000e-04 | norm: 0.1918 | dt: 14466.16ms | tok/sec: 36242.4 | mem: 107563MB | Time Left: 04 hrs : 17 mins [TIMING] data: 7.5ms | fwd: 3580.9ms | loss: 13.0ms | bwd: 10827.5ms | clip: 0.0ms | optim: 23.7ms step 1160 | loss: 2.959932 | lr 1.8000e-04 | norm: 0.1950 | dt: 14470.93ms | tok/sec: 36230.4 | mem: 107563MB | Time Left: 04 hrs : 17 mins [TIMING] data: 7.5ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10832.3ms | clip: 0.0ms | optim: 23.9ms step 1161 | loss: 3.114330 | lr 1.8000e-04 | norm: 0.2432 | dt: 14459.91ms | tok/sec: 36258.0 | mem: 107563MB | Time Left: 04 hrs : 16 mins [TIMING] data: 7.5ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10822.9ms | clip: 0.0ms | optim: 25.0ms step 1162 | loss: 2.797482 | lr 1.8000e-04 | norm: 0.4156 | dt: 14463.31ms | tok/sec: 36249.5 | mem: 107563MB | Time Left: 04 hrs : 16 mins [TIMING] data: 7.5ms | fwd: 3591.4ms | loss: 13.1ms | bwd: 10813.8ms | clip: 0.0ms | optim: 23.9ms step 1163 | loss: 3.078445 | lr 1.8000e-04 | norm: 0.1933 | dt: 14472.20ms | tok/sec: 36227.2 | mem: 107563MB | Time Left: 04 hrs : 16 mins [TIMING] data: 7.6ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10835.3ms | clip: 0.0ms | optim: 23.9ms step 1164 | loss: 2.745986 | lr 1.8000e-04 | norm: 0.2306 | dt: 14466.62ms | tok/sec: 36241.2 | mem: 107563MB | Time Left: 04 hrs : 16 mins [TIMING] data: 7.5ms | fwd: 3574.1ms | loss: 13.1ms | bwd: 10834.7ms | clip: 0.0ms | optim: 23.8ms step 1165 | loss: 2.742216 | lr 1.8000e-04 | norm: 0.2708 | dt: 14463.36ms | tok/sec: 36249.4 | mem: 107563MB | Time Left: 04 hrs : 16 mins [TIMING] data: 7.5ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10827.0ms | clip: 0.0ms | optim: 24.6ms step 1166 | loss: 2.798336 | lr 1.8000e-04 | norm: 0.2084 | dt: 14465.62ms | tok/sec: 36243.7 | mem: 107563MB | Time Left: 04 hrs : 15 mins [TIMING] data: 7.5ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10828.1ms | clip: 0.0ms | optim: 23.9ms step 1167 | loss: 3.002430 | lr 1.8000e-04 | norm: 0.2033 | dt: 14470.22ms | tok/sec: 36232.2 | mem: 107563MB | Time Left: 04 hrs : 15 mins [TIMING] data: 7.4ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10832.8ms | clip: 0.0ms | optim: 23.9ms step 1168 | loss: 3.010418 | lr 1.8000e-04 | norm: 0.2004 | dt: 14462.48ms | tok/sec: 36251.6 | mem: 107563MB | Time Left: 04 hrs : 15 mins [TIMING] data: 7.6ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10824.2ms | clip: 0.0ms | optim: 23.7ms step 1169 | loss: 2.810498 | lr 1.8000e-04 | norm: 0.2018 | dt: 14478.49ms | tok/sec: 36211.5 | mem: 107563MB | Time Left: 04 hrs : 15 mins [TIMING] data: 7.4ms | fwd: 3580.7ms | loss: 13.0ms | bwd: 10840.0ms | clip: 0.0ms | optim: 23.8ms step 1170 | loss: 2.590692 | lr 1.8000e-04 | norm: 0.3648 | dt: 14471.13ms | tok/sec: 36229.9 | mem: 107563MB | Time Left: 04 hrs : 14 mins [TIMING] data: 7.5ms | fwd: 3574.4ms | loss: 13.0ms | bwd: 10838.7ms | clip: 0.0ms | optim: 23.9ms step 1171 | loss: 3.145187 | lr 1.8000e-04 | norm: 0.2028 | dt: 14473.15ms | tok/sec: 36224.9 | mem: 107563MB | Time Left: 04 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10835.7ms | clip: 0.0ms | optim: 23.8ms step 1172 | loss: 2.664511 | lr 1.8000e-04 | norm: 0.1851 | dt: 14466.18ms | tok/sec: 36242.3 | mem: 107563MB | Time Left: 04 hrs : 14 mins [TIMING] data: 7.5ms | fwd: 3579.4ms | loss: 13.0ms | bwd: 10828.9ms | clip: 0.0ms | optim: 23.9ms step 1173 | loss: 2.800880 | lr 1.8000e-04 | norm: 0.2585 | dt: 14476.08ms | tok/sec: 36217.5 | mem: 107563MB | Time Left: 04 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3577.6ms | loss: 13.0ms | bwd: 10840.6ms | clip: 0.0ms | optim: 23.9ms step 1174 | loss: 3.077401 | lr 1.8000e-04 | norm: 0.1913 | dt: 14473.65ms | tok/sec: 36223.6 | mem: 107563MB | Time Left: 04 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3573.6ms | loss: 13.1ms | bwd: 10840.8ms | clip: 0.0ms | optim: 24.9ms step 1175 | loss: 2.771671 | lr 1.8000e-04 | norm: 0.1946 | dt: 14462.88ms | tok/sec: 36250.6 | mem: 107563MB | Time Left: 04 hrs : 13 mins [TIMING] data: 7.5ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10828.0ms | clip: 0.0ms | optim: 23.8ms step 1176 | loss: 2.825554 | lr 1.8000e-04 | norm: 0.8627 | dt: 14474.28ms | tok/sec: 36222.0 | mem: 107563MB | Time Left: 04 hrs : 13 mins [TIMING] data: 7.5ms | fwd: 3597.9ms | loss: 13.1ms | bwd: 10818.5ms | clip: 0.0ms | optim: 23.8ms step 1177 | loss: 2.602387 | lr 1.8000e-04 | norm: 0.2568 | dt: 14474.76ms | tok/sec: 36220.9 | mem: 107563MB | Time Left: 04 hrs : 13 mins [TIMING] data: 7.5ms | fwd: 3569.7ms | loss: 13.0ms | bwd: 10847.1ms | clip: 0.0ms | optim: 23.8ms step 1178 | loss: 2.860441 | lr 1.8000e-04 | norm: 0.2586 | dt: 14479.15ms | tok/sec: 36209.9 | mem: 107563MB | Time Left: 04 hrs : 13 mins [TIMING] data: 7.6ms | fwd: 3581.9ms | loss: 13.3ms | bwd: 10839.1ms | clip: 0.0ms | optim: 23.8ms step 1179 | loss: 2.901540 | lr 1.8000e-04 | norm: 0.2217 | dt: 14467.52ms | tok/sec: 36239.0 | mem: 107563MB | Time Left: 04 hrs : 12 mins [TIMING] data: 7.5ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10826.7ms | clip: 0.0ms | optim: 23.8ms step 1180 | loss: 2.553148 | lr 1.8000e-04 | norm: 0.2003 | dt: 14469.48ms | tok/sec: 36234.1 | mem: 107563MB | Time Left: 04 hrs : 12 mins [TIMING] data: 7.5ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10833.4ms | clip: 0.0ms | optim: 23.8ms step 1181 | loss: 2.929569 | lr 1.8000e-04 | norm: 0.2511 | dt: 14469.48ms | tok/sec: 36234.1 | mem: 107563MB | Time Left: 04 hrs : 12 mins [TIMING] data: 7.5ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10829.9ms | clip: 0.0ms | optim: 23.9ms step 1182 | loss: 3.140515 | lr 1.8000e-04 | norm: 2.2012 | dt: 14467.27ms | tok/sec: 36239.6 | mem: 107563MB | Time Left: 04 hrs : 11 mins [TIMING] data: 7.6ms | fwd: 3572.6ms | loss: 13.1ms | bwd: 10836.7ms | clip: 0.0ms | optim: 23.8ms step 1183 | loss: 2.718797 | lr 1.8000e-04 | norm: 0.2228 | dt: 14469.02ms | tok/sec: 36235.2 | mem: 107563MB | Time Left: 04 hrs : 11 mins [TIMING] data: 7.5ms | fwd: 3584.4ms | loss: 13.1ms | bwd: 10826.8ms | clip: 0.0ms | optim: 23.8ms step 1184 | loss: 3.002739 | lr 1.8000e-04 | norm: 0.2311 | dt: 14464.39ms | tok/sec: 36246.8 | mem: 107563MB | Time Left: 04 hrs : 11 mins [TIMING] data: 7.7ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10826.8ms | clip: 0.0ms | optim: 23.7ms step 1185 | loss: 2.932802 | lr 1.8000e-04 | norm: 0.2210 | dt: 14474.65ms | tok/sec: 36221.1 | mem: 107563MB | Time Left: 04 hrs : 11 mins [TIMING] data: 7.5ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10838.0ms | clip: 0.0ms | optim: 24.0ms step 1186 | loss: 3.065282 | lr 1.8000e-04 | norm: 0.2268 | dt: 14475.10ms | tok/sec: 36220.0 | mem: 107563MB | Time Left: 04 hrs : 11 mins [TIMING] data: 7.5ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10837.1ms | clip: 0.0ms | optim: 23.8ms step 1187 | loss: 2.869110 | lr 1.8000e-04 | norm: 0.2218 | dt: 14471.36ms | tok/sec: 36229.4 | mem: 107563MB | Time Left: 04 hrs : 10 mins [TIMING] data: 7.5ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10834.4ms | clip: 0.0ms | optim: 23.8ms step 1188 | loss: 2.952128 | lr 1.8000e-04 | norm: 0.2377 | dt: 14483.96ms | tok/sec: 36197.8 | mem: 107563MB | Time Left: 04 hrs : 10 mins [TIMING] data: 7.5ms | fwd: 3595.2ms | loss: 13.0ms | bwd: 10830.5ms | clip: 0.0ms | optim: 23.8ms step 1189 | loss: 2.922984 | lr 1.8000e-04 | norm: 0.2678 | dt: 14467.05ms | tok/sec: 36240.1 | mem: 107563MB | Time Left: 04 hrs : 10 mins [TIMING] data: 7.6ms | fwd: 3577.5ms | loss: 13.0ms | bwd: 10831.8ms | clip: 0.0ms | optim: 23.7ms step 1190 | loss: 2.927126 | lr 1.8000e-04 | norm: 0.2734 | dt: 14486.71ms | tok/sec: 36191.0 | mem: 107563MB | Time Left: 04 hrs : 10 mins [TIMING] data: 7.5ms | fwd: 3586.8ms | loss: 13.1ms | bwd: 10841.9ms | clip: 0.0ms | optim: 23.8ms step 1191 | loss: 3.114862 | lr 1.8000e-04 | norm: 0.2120 | dt: 14471.39ms | tok/sec: 36229.3 | mem: 107563MB | Time Left: 04 hrs : 09 mins [TIMING] data: 7.4ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10836.5ms | clip: 0.0ms | optim: 24.5ms step 1192 | loss: 2.743488 | lr 1.8000e-04 | norm: 0.2367 | dt: 14477.68ms | tok/sec: 36213.5 | mem: 107563MB | Time Left: 04 hrs : 09 mins [TIMING] data: 7.5ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10841.3ms | clip: 0.0ms | optim: 23.8ms step 1193 | loss: 3.228868 | lr 1.8000e-04 | norm: 0.2382 | dt: 14477.29ms | tok/sec: 36214.5 | mem: 107563MB | Time Left: 04 hrs : 09 mins [TIMING] data: 7.6ms | fwd: 3579.6ms | loss: 13.0ms | bwd: 10839.7ms | clip: 0.0ms | optim: 23.7ms step 1194 | loss: 3.057916 | lr 1.8000e-04 | norm: 0.2494 | dt: 14476.15ms | tok/sec: 36217.4 | mem: 107563MB | Time Left: 04 hrs : 09 mins [TIMING] data: 7.4ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10840.3ms | clip: 0.0ms | optim: 23.8ms step 1195 | loss: 3.049322 | lr 1.8000e-04 | norm: 0.1926 | dt: 14473.37ms | tok/sec: 36224.3 | mem: 107563MB | Time Left: 04 hrs : 08 mins [TIMING] data: 7.6ms | fwd: 3590.8ms | loss: 13.1ms | bwd: 10824.5ms | clip: 0.0ms | optim: 23.9ms step 1196 | loss: 2.817470 | lr 1.8000e-04 | norm: 0.2024 | dt: 14463.57ms | tok/sec: 36248.9 | mem: 107563MB | Time Left: 04 hrs : 08 mins [TIMING] data: 7.5ms | fwd: 3575.8ms | loss: 13.0ms | bwd: 10830.0ms | clip: 0.0ms | optim: 23.8ms step 1197 | loss: 2.431797 | lr 1.8000e-04 | norm: 0.2409 | dt: 14468.10ms | tok/sec: 36237.5 | mem: 107563MB | Time Left: 04 hrs : 08 mins [TIMING] data: 7.5ms | fwd: 3576.4ms | loss: 13.0ms | bwd: 10833.6ms | clip: 0.0ms | optim: 23.9ms step 1198 | loss: 2.683433 | lr 1.8000e-04 | norm: 0.2107 | dt: 14469.61ms | tok/sec: 36233.7 | mem: 107563MB | Time Left: 04 hrs : 08 mins [TIMING] data: 7.5ms | fwd: 3579.2ms | loss: 13.0ms | bwd: 10832.5ms | clip: 0.0ms | optim: 23.8ms step 1199 | loss: 3.047005 | lr 1.8000e-04 | norm: 0.2024 | dt: 14470.84ms | tok/sec: 36230.6 | mem: 107563MB | Time Left: 04 hrs : 07 mins [TIMING] data: 7.5ms | fwd: 3584.3ms | loss: 13.1ms | bwd: 10828.0ms | clip: 0.0ms | optim: 24.3ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1200.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1100.ckpt step 1200 | loss: 2.930595 | lr 1.8000e-04 | norm: 0.2146 | dt: 14371.04ms | tok/sec: 36482.3 | mem: 107563MB | Time Left: 04 hrs : 05 mins [TIMING] data: 7.6ms | fwd: 3565.9ms | loss: 13.1ms | bwd: 10746.8ms | clip: 0.0ms | optim: 23.8ms step 1201 | loss: 2.858047 | lr 1.8000e-04 | norm: 0.2112 | dt: 14410.86ms | tok/sec: 36381.4 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 7.5ms | fwd: 3565.2ms | loss: 13.0ms | bwd: 10786.8ms | clip: 0.0ms | optim: 23.8ms step 1202 | loss: 3.027985 | lr 1.8000e-04 | norm: 0.2012 | dt: 14425.29ms | tok/sec: 36345.1 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 7.7ms | fwd: 3564.2ms | loss: 13.0ms | bwd: 10803.2ms | clip: 0.0ms | optim: 23.8ms step 1203 | loss: 2.916592 | lr 1.8000e-04 | norm: 0.2586 | dt: 14429.65ms | tok/sec: 36334.1 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 7.4ms | fwd: 3571.8ms | loss: 13.1ms | bwd: 10800.0ms | clip: 0.0ms | optim: 23.8ms step 1204 | loss: 2.811263 | lr 1.8000e-04 | norm: 0.1976 | dt: 14460.32ms | tok/sec: 36257.0 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 7.9ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10825.7ms | clip: 0.0ms | optim: 23.8ms step 1205 | loss: 2.682670 | lr 1.8000e-04 | norm: 0.2065 | dt: 14460.13ms | tok/sec: 36257.5 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 7.6ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10827.3ms | clip: 0.0ms | optim: 23.7ms step 1206 | loss: 3.013444 | lr 1.8000e-04 | norm: 0.2563 | dt: 14459.05ms | tok/sec: 36260.2 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 7.7ms | fwd: 3594.3ms | loss: 13.0ms | bwd: 10806.6ms | clip: 0.0ms | optim: 23.9ms step 1207 | loss: 2.851919 | lr 1.8000e-04 | norm: 0.1752 | dt: 14488.32ms | tok/sec: 36186.9 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 7.5ms | fwd: 3602.2ms | loss: 13.1ms | bwd: 10828.2ms | clip: 0.0ms | optim: 23.7ms step 1208 | loss: 3.033643 | lr 1.8000e-04 | norm: 0.1850 | dt: 14470.62ms | tok/sec: 36231.2 | mem: 107563MB | Time Left: 04 hrs : 05 mins [TIMING] data: 7.6ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10831.6ms | clip: 0.0ms | optim: 23.8ms step 1209 | loss: 2.851513 | lr 1.8000e-04 | norm: 0.1586 | dt: 14512.88ms | tok/sec: 36125.7 | mem: 107563MB | Time Left: 04 hrs : 06 mins [TIMING] data: 8.3ms | fwd: 3618.9ms | loss: 13.0ms | bwd: 10834.3ms | clip: 0.0ms | optim: 23.9ms step 1210 | loss: 2.935349 | lr 1.8000e-04 | norm: 0.2053 | dt: 14494.68ms | tok/sec: 36171.1 | mem: 107563MB | Time Left: 04 hrs : 05 mins [TIMING] data: 7.5ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10858.1ms | clip: 0.0ms | optim: 23.9ms step 1211 | loss: 2.781053 | lr 1.8000e-04 | norm: 0.1907 | dt: 14504.24ms | tok/sec: 36147.2 | mem: 107563MB | Time Left: 04 hrs : 05 mins [TIMING] data: 7.4ms | fwd: 3608.5ms | loss: 13.1ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.8ms step 1212 | loss: 3.230680 | lr 1.8000e-04 | norm: 0.1877 | dt: 14510.95ms | tok/sec: 36130.5 | mem: 107563MB | Time Left: 04 hrs : 05 mins [TIMING] data: 7.4ms | fwd: 3613.0ms | loss: 13.0ms | bwd: 10840.1ms | clip: 0.0ms | optim: 23.8ms step 1213 | loss: 3.145440 | lr 1.8000e-04 | norm: 0.1828 | dt: 14470.96ms | tok/sec: 36230.4 | mem: 107563MB | Time Left: 04 hrs : 04 mins [TIMING] data: 7.6ms | fwd: 3592.8ms | loss: 13.1ms | bwd: 10819.8ms | clip: 0.0ms | optim: 23.9ms step 1214 | loss: 3.025179 | lr 1.8000e-04 | norm: 0.2189 | dt: 14491.04ms | tok/sec: 36180.1 | mem: 107563MB | Time Left: 04 hrs : 04 mins [TIMING] data: 7.5ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.8ms step 1215 | loss: 2.710504 | lr 1.8000e-04 | norm: 0.1735 | dt: 14491.60ms | tok/sec: 36178.8 | mem: 107563MB | Time Left: 04 hrs : 04 mins [TIMING] data: 7.5ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10853.5ms | clip: 0.0ms | optim: 23.9ms step 1216 | loss: 2.940471 | lr 1.8000e-04 | norm: 0.2194 | dt: 14490.39ms | tok/sec: 36181.8 | mem: 107563MB | Time Left: 04 hrs : 04 mins [TIMING] data: 7.4ms | fwd: 3588.1ms | loss: 13.2ms | bwd: 10844.1ms | clip: 0.0ms | optim: 23.9ms step 1217 | loss: 3.036198 | lr 1.8000e-04 | norm: 0.2501 | dt: 14544.96ms | tok/sec: 36046.0 | mem: 107563MB | Time Left: 04 hrs : 04 mins [TIMING] data: 7.5ms | fwd: 3650.4ms | loss: 13.2ms | bwd: 10836.2ms | clip: 0.0ms | optim: 23.8ms step 1218 | loss: 2.900917 | lr 1.8000e-04 | norm: 0.1853 | dt: 14493.46ms | tok/sec: 36174.1 | mem: 107563MB | Time Left: 04 hrs : 03 mins [TIMING] data: 7.6ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10853.2ms | clip: 0.0ms | optim: 23.8ms step 1219 | loss: 2.936207 | lr 1.8000e-04 | norm: 0.1815 | dt: 14485.10ms | tok/sec: 36195.0 | mem: 107563MB | Time Left: 04 hrs : 03 mins [TIMING] data: 7.8ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10847.9ms | clip: 0.0ms | optim: 23.8ms step 1220 | loss: 2.586973 | lr 1.8000e-04 | norm: 0.1810 | dt: 14489.61ms | tok/sec: 36183.7 | mem: 107563MB | Time Left: 04 hrs : 03 mins [TIMING] data: 7.3ms | fwd: 3578.9ms | loss: 13.0ms | bwd: 10853.1ms | clip: 0.0ms | optim: 23.8ms step 1221 | loss: 2.903755 | lr 1.8000e-04 | norm: 0.1925 | dt: 14489.81ms | tok/sec: 36183.2 | mem: 107563MB | Time Left: 04 hrs : 02 mins [TIMING] data: 7.8ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 23.9ms step 1222 | loss: 2.782238 | lr 1.8000e-04 | norm: 0.1894 | dt: 14489.10ms | tok/sec: 36185.0 | mem: 107563MB | Time Left: 04 hrs : 02 mins [TIMING] data: 7.5ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10854.9ms | clip: 0.0ms | optim: 23.9ms step 1223 | loss: 2.639956 | lr 1.8000e-04 | norm: 0.1977 | dt: 14560.78ms | tok/sec: 36006.9 | mem: 107563MB | Time Left: 04 hrs : 03 mins [TIMING] data: 7.3ms | fwd: 3660.5ms | loss: 13.1ms | bwd: 10842.4ms | clip: 0.0ms | optim: 23.8ms step 1224 | loss: 2.933099 | lr 1.8000e-04 | norm: 0.1747 | dt: 14498.25ms | tok/sec: 36162.1 | mem: 107563MB | Time Left: 04 hrs : 02 mins [TIMING] data: 7.8ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10859.9ms | clip: 0.0ms | optim: 24.0ms step 1225 | loss: 2.858877 | lr 1.8000e-04 | norm: 0.1716 | dt: 14486.64ms | tok/sec: 36191.1 | mem: 107563MB | Time Left: 04 hrs : 01 mins [TIMING] data: 7.4ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10848.6ms | clip: 0.0ms | optim: 23.8ms step 1226 | loss: 2.697016 | lr 1.8000e-04 | norm: 0.1465 | dt: 14539.10ms | tok/sec: 36060.5 | mem: 107563MB | Time Left: 04 hrs : 02 mins [TIMING] data: 7.7ms | fwd: 3654.6ms | loss: 13.1ms | bwd: 10826.1ms | clip: 0.0ms | optim: 23.9ms step 1227 | loss: 3.035377 | lr 1.8000e-04 | norm: 0.2148 | dt: 14505.53ms | tok/sec: 36144.0 | mem: 107563MB | Time Left: 04 hrs : 01 mins [TIMING] data: 7.5ms | fwd: 3621.1ms | loss: 13.0ms | bwd: 10826.6ms | clip: 0.0ms | optim: 23.8ms step 1228 | loss: 3.205193 | lr 1.8000e-04 | norm: 0.2972 | dt: 14491.08ms | tok/sec: 36180.0 | mem: 107563MB | Time Left: 04 hrs : 01 mins [TIMING] data: 7.5ms | fwd: 3577.3ms | loss: 13.0ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.9ms step 1229 | loss: 2.978115 | lr 1.8000e-04 | norm: 0.2690 | dt: 14492.34ms | tok/sec: 36176.9 | mem: 107563MB | Time Left: 04 hrs : 01 mins [TIMING] data: 7.4ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10853.7ms | clip: 0.0ms | optim: 23.7ms step 1230 | loss: 2.849697 | lr 1.8000e-04 | norm: 0.1609 | dt: 14489.31ms | tok/sec: 36184.5 | mem: 107563MB | Time Left: 04 hrs : 00 mins [TIMING] data: 7.4ms | fwd: 3575.2ms | loss: 13.0ms | bwd: 10855.4ms | clip: 0.0ms | optim: 24.0ms step 1231 | loss: 3.082781 | lr 1.8000e-04 | norm: 0.1759 | dt: 14491.55ms | tok/sec: 36178.9 | mem: 107563MB | Time Left: 04 hrs : 00 mins [TIMING] data: 7.4ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10853.0ms | clip: 0.0ms | optim: 23.8ms step 1232 | loss: 2.416233 | lr 1.8000e-04 | norm: 0.2409 | dt: 14480.84ms | tok/sec: 36205.6 | mem: 107563MB | Time Left: 04 hrs : 00 mins [TIMING] data: 7.3ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10844.7ms | clip: 0.0ms | optim: 23.8ms step 1233 | loss: 2.878590 | lr 1.8000e-04 | norm: 0.1821 | dt: 14497.72ms | tok/sec: 36163.5 | mem: 107563MB | Time Left: 04 hrs : 00 mins [TIMING] data: 7.6ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10862.6ms | clip: 0.0ms | optim: 23.7ms step 1234 | loss: 2.958905 | lr 1.8000e-04 | norm: 0.1892 | dt: 14489.26ms | tok/sec: 36184.6 | mem: 107563MB | Time Left: 03 hrs : 59 mins [TIMING] data: 7.5ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10852.1ms | clip: 0.0ms | optim: 23.8ms step 1235 | loss: 2.515589 | lr 1.8000e-04 | norm: 0.1701 | dt: 14498.18ms | tok/sec: 36162.3 | mem: 107563MB | Time Left: 03 hrs : 59 mins [TIMING] data: 7.7ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10862.9ms | clip: 0.0ms | optim: 23.9ms step 1236 | loss: 3.026165 | lr 1.8000e-04 | norm: 0.2158 | dt: 14492.52ms | tok/sec: 36176.4 | mem: 107563MB | Time Left: 03 hrs : 59 mins [TIMING] data: 7.7ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10854.9ms | clip: 0.0ms | optim: 23.8ms step 1237 | loss: 3.073011 | lr 1.8000e-04 | norm: 0.2244 | dt: 14490.78ms | tok/sec: 36180.8 | mem: 107563MB | Time Left: 03 hrs : 59 mins [TIMING] data: 7.5ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10856.6ms | clip: 0.0ms | optim: 24.0ms step 1238 | loss: 2.947248 | lr 1.8000e-04 | norm: 0.2640 | dt: 14484.69ms | tok/sec: 36196.0 | mem: 107563MB | Time Left: 03 hrs : 58 mins [TIMING] data: 7.4ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.9ms step 1239 | loss: 2.769401 | lr 1.8000e-04 | norm: 0.2832 | dt: 14496.17ms | tok/sec: 36167.4 | mem: 107563MB | Time Left: 03 hrs : 58 mins [TIMING] data: 7.4ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10861.6ms | clip: 0.0ms | optim: 23.9ms step 1240 | loss: 3.157515 | lr 1.8000e-04 | norm: 0.2128 | dt: 14487.21ms | tok/sec: 36189.7 | mem: 107563MB | Time Left: 03 hrs : 58 mins [TIMING] data: 7.5ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10852.0ms | clip: 0.0ms | optim: 23.8ms step 1241 | loss: 3.005048 | lr 1.8000e-04 | norm: 0.1726 | dt: 14495.33ms | tok/sec: 36169.4 | mem: 107563MB | Time Left: 03 hrs : 58 mins [TIMING] data: 7.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.8ms step 1242 | loss: 2.704015 | lr 1.8000e-04 | norm: 0.1961 | dt: 14501.10ms | tok/sec: 36155.1 | mem: 107563MB | Time Left: 03 hrs : 58 mins [TIMING] data: 7.7ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10864.0ms | clip: 0.0ms | optim: 23.9ms step 1243 | loss: 2.805319 | lr 1.8000e-04 | norm: 0.1902 | dt: 14489.32ms | tok/sec: 36184.4 | mem: 107563MB | Time Left: 03 hrs : 57 mins [TIMING] data: 7.9ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.8ms step 1244 | loss: 2.501609 | lr 1.8000e-04 | norm: 0.2736 | dt: 14489.10ms | tok/sec: 36185.0 | mem: 107563MB | Time Left: 03 hrs : 57 mins [TIMING] data: 7.4ms | fwd: 3576.8ms | loss: 13.0ms | bwd: 10854.5ms | clip: 0.0ms | optim: 23.7ms step 1245 | loss: 3.106121 | lr 1.8000e-04 | norm: 0.2126 | dt: 14508.11ms | tok/sec: 36137.6 | mem: 107563MB | Time Left: 03 hrs : 57 mins [TIMING] data: 7.6ms | fwd: 3576.5ms | loss: 13.0ms | bwd: 10873.4ms | clip: 0.0ms | optim: 23.8ms step 1246 | loss: 2.962594 | lr 1.8000e-04 | norm: 0.2227 | dt: 14493.70ms | tok/sec: 36173.5 | mem: 107563MB | Time Left: 03 hrs : 56 mins [TIMING] data: 7.8ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10858.8ms | clip: 0.0ms | optim: 23.8ms step 1247 | loss: 2.919582 | lr 1.8000e-04 | norm: 0.2034 | dt: 14495.83ms | tok/sec: 36168.2 | mem: 107563MB | Time Left: 03 hrs : 56 mins [TIMING] data: 7.8ms | fwd: 3581.0ms | loss: 13.0ms | bwd: 10856.6ms | clip: 0.0ms | optim: 23.8ms step 1248 | loss: 3.079961 | lr 1.8000e-04 | norm: 0.5491 | dt: 14487.71ms | tok/sec: 36188.5 | mem: 107563MB | Time Left: 03 hrs : 56 mins [TIMING] data: 7.4ms | fwd: 3575.9ms | loss: 13.0ms | bwd: 10853.5ms | clip: 0.0ms | optim: 24.2ms step 1249 | loss: 2.807492 | lr 1.8000e-04 | norm: 0.2515 | dt: 14491.88ms | tok/sec: 36178.1 | mem: 107563MB | Time Left: 03 hrs : 56 mins [TIMING] data: 7.5ms | fwd: 3580.3ms | loss: 13.0ms | bwd: 10853.2ms | clip: 0.0ms | optim: 24.3ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1250.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1150.ckpt validation loss: 3.0702 step 1250 | loss: 2.916891 | lr 1.8000e-04 | norm: 0.2855 | dt: 18964.11ms | tok/sec: 27646.3 | mem: 107563MB | Time Left: 05 hrs : 08 mins [TIMING] data: 7.5ms | fwd: 3562.9ms | loss: 13.1ms | bwd: 10793.7ms | clip: 0.0ms | optim: 23.8ms step 1251 | loss: 2.971211 | lr 1.8000e-04 | norm: 0.1972 | dt: 14432.04ms | tok/sec: 36328.1 | mem: 107563MB | Time Left: 03 hrs : 54 mins [TIMING] data: 7.4ms | fwd: 3566.6ms | loss: 13.1ms | bwd: 10807.3ms | clip: 0.0ms | optim: 23.7ms step 1252 | loss: 2.943068 | lr 1.8000e-04 | norm: 0.1885 | dt: 14446.19ms | tok/sec: 36292.5 | mem: 107563MB | Time Left: 03 hrs : 54 mins [TIMING] data: 7.6ms | fwd: 3576.5ms | loss: 13.0ms | bwd: 10811.7ms | clip: 0.0ms | optim: 23.9ms step 1253 | loss: 2.745675 | lr 1.8000e-04 | norm: 0.1790 | dt: 14467.33ms | tok/sec: 36239.4 | mem: 107563MB | Time Left: 03 hrs : 54 mins [TIMING] data: 7.3ms | fwd: 3573.5ms | loss: 13.1ms | bwd: 10835.4ms | clip: 0.0ms | optim: 23.9ms step 1254 | loss: 3.021145 | lr 1.8000e-04 | norm: 0.2686 | dt: 14467.91ms | tok/sec: 36238.0 | mem: 107563MB | Time Left: 03 hrs : 54 mins [TIMING] data: 7.4ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10827.8ms | clip: 0.0ms | optim: 23.8ms step 1255 | loss: 2.818645 | lr 1.8000e-04 | norm: 0.1873 | dt: 14472.84ms | tok/sec: 36225.7 | mem: 107563MB | Time Left: 03 hrs : 54 mins [TIMING] data: 7.3ms | fwd: 3576.7ms | loss: 13.0ms | bwd: 10838.4ms | clip: 0.0ms | optim: 23.8ms step 1256 | loss: 2.758379 | lr 1.8000e-04 | norm: 0.1743 | dt: 14485.73ms | tok/sec: 36193.4 | mem: 107563MB | Time Left: 03 hrs : 54 mins [TIMING] data: 7.4ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10849.8ms | clip: 0.0ms | optim: 23.9ms step 1257 | loss: 3.009531 | lr 1.8000e-04 | norm: 0.2139 | dt: 14484.64ms | tok/sec: 36196.1 | mem: 107563MB | Time Left: 03 hrs : 54 mins [TIMING] data: 7.3ms | fwd: 3574.4ms | loss: 13.1ms | bwd: 10852.4ms | clip: 0.0ms | optim: 23.9ms step 1258 | loss: 2.940263 | lr 1.8000e-04 | norm: 0.1919 | dt: 14483.36ms | tok/sec: 36199.3 | mem: 107563MB | Time Left: 03 hrs : 53 mins [TIMING] data: 7.4ms | fwd: 3574.9ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 23.9ms step 1259 | loss: 2.971889 | lr 1.8000e-04 | norm: 0.2605 | dt: 14483.35ms | tok/sec: 36199.4 | mem: 107563MB | Time Left: 03 hrs : 53 mins [TIMING] data: 7.5ms | fwd: 3577.3ms | loss: 13.0ms | bwd: 10848.2ms | clip: 0.0ms | optim: 23.7ms step 1260 | loss: 2.642985 | lr 1.8000e-04 | norm: 0.1650 | dt: 14488.75ms | tok/sec: 36185.9 | mem: 107563MB | Time Left: 03 hrs : 53 mins [TIMING] data: 7.4ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 23.8ms step 1261 | loss: 2.557694 | lr 1.8000e-04 | norm: 0.2047 | dt: 14492.97ms | tok/sec: 36175.3 | mem: 107563MB | Time Left: 03 hrs : 53 mins [TIMING] data: 7.4ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10851.0ms | clip: 0.0ms | optim: 23.8ms step 1262 | loss: 2.705480 | lr 1.8000e-04 | norm: 0.2460 | dt: 14491.05ms | tok/sec: 36180.1 | mem: 107563MB | Time Left: 03 hrs : 53 mins [TIMING] data: 7.4ms | fwd: 3578.9ms | loss: 13.0ms | bwd: 10854.1ms | clip: 0.0ms | optim: 24.0ms step 1263 | loss: 3.231333 | lr 1.8000e-04 | norm: 0.2426 | dt: 14498.82ms | tok/sec: 36160.7 | mem: 107563MB | Time Left: 03 hrs : 52 mins [TIMING] data: 7.3ms | fwd: 3585.8ms | loss: 13.1ms | bwd: 10855.2ms | clip: 0.0ms | optim: 23.8ms step 1264 | loss: 2.903748 | lr 1.8000e-04 | norm: 0.2005 | dt: 14495.41ms | tok/sec: 36169.2 | mem: 107563MB | Time Left: 03 hrs : 52 mins [TIMING] data: 7.4ms | fwd: 3582.1ms | loss: 13.0ms | bwd: 10854.4ms | clip: 0.0ms | optim: 23.9ms step 1265 | loss: 3.073877 | lr 1.8000e-04 | norm: 0.2117 | dt: 14487.35ms | tok/sec: 36189.4 | mem: 107563MB | Time Left: 03 hrs : 52 mins [TIMING] data: 7.3ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.8ms step 1266 | loss: 2.827391 | lr 1.8000e-04 | norm: 0.1797 | dt: 14503.00ms | tok/sec: 36150.3 | mem: 107563MB | Time Left: 03 hrs : 52 mins [TIMING] data: 7.4ms | fwd: 3585.5ms | loss: 13.1ms | bwd: 10859.4ms | clip: 0.0ms | optim: 23.8ms step 1267 | loss: 3.149271 | lr 1.8000e-04 | norm: 0.1765 | dt: 14508.67ms | tok/sec: 36136.2 | mem: 107563MB | Time Left: 03 hrs : 52 mins [TIMING] data: 7.3ms | fwd: 3580.1ms | loss: 13.0ms | bwd: 10870.9ms | clip: 0.0ms | optim: 23.8ms step 1268 | loss: 3.083722 | lr 1.8000e-04 | norm: 0.1755 | dt: 14516.43ms | tok/sec: 36116.9 | mem: 107563MB | Time Left: 03 hrs : 52 mins [TIMING] data: 7.4ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10880.5ms | clip: 0.0ms | optim: 23.9ms step 1269 | loss: 3.284284 | lr 1.8000e-04 | norm: 0.2164 | dt: 14508.70ms | tok/sec: 36136.1 | mem: 107563MB | Time Left: 03 hrs : 51 mins [TIMING] data: 7.4ms | fwd: 3579.5ms | loss: 13.0ms | bwd: 10870.7ms | clip: 0.0ms | optim: 23.8ms step 1270 | loss: 3.036680 | lr 1.8000e-04 | norm: 0.1714 | dt: 14515.03ms | tok/sec: 36120.4 | mem: 107563MB | Time Left: 03 hrs : 51 mins [TIMING] data: 7.3ms | fwd: 3584.7ms | loss: 13.1ms | bwd: 10872.4ms | clip: 0.0ms | optim: 23.8ms step 1271 | loss: 2.820575 | lr 1.8000e-04 | norm: 0.1600 | dt: 14509.57ms | tok/sec: 36133.9 | mem: 107563MB | Time Left: 03 hrs : 51 mins [TIMING] data: 7.3ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10869.0ms | clip: 0.0ms | optim: 23.9ms step 1272 | loss: 2.772017 | lr 1.8000e-04 | norm: 0.2214 | dt: 14520.89ms | tok/sec: 36105.8 | mem: 107563MB | Time Left: 03 hrs : 51 mins [TIMING] data: 7.2ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10879.9ms | clip: 0.0ms | optim: 24.6ms step 1273 | loss: 3.090850 | lr 1.8000e-04 | norm: 0.2052 | dt: 14512.27ms | tok/sec: 36127.2 | mem: 107563MB | Time Left: 03 hrs : 50 mins [TIMING] data: 7.4ms | fwd: 3584.9ms | loss: 13.0ms | bwd: 10869.6ms | clip: 0.0ms | optim: 23.8ms step 1274 | loss: 2.971785 | lr 1.8000e-04 | norm: 0.1828 | dt: 14515.94ms | tok/sec: 36118.1 | mem: 107563MB | Time Left: 03 hrs : 50 mins [TIMING] data: 7.5ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10876.0ms | clip: 0.0ms | optim: 23.9ms step 1275 | loss: 2.918564 | lr 1.8000e-04 | norm: 0.2243 | dt: 14512.12ms | tok/sec: 36127.6 | mem: 107563MB | Time Left: 03 hrs : 50 mins [TIMING] data: 7.4ms | fwd: 3580.8ms | loss: 13.0ms | bwd: 10873.5ms | clip: 0.0ms | optim: 23.9ms step 1276 | loss: 2.927073 | lr 1.8000e-04 | norm: 0.2868 | dt: 14515.54ms | tok/sec: 36119.1 | mem: 107563MB | Time Left: 03 hrs : 50 mins [TIMING] data: 7.4ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10874.7ms | clip: 0.0ms | optim: 23.8ms step 1277 | loss: 2.866825 | lr 1.8000e-04 | norm: 0.1612 | dt: 14507.41ms | tok/sec: 36139.3 | mem: 107563MB | Time Left: 03 hrs : 49 mins [TIMING] data: 7.3ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10866.9ms | clip: 0.0ms | optim: 23.9ms step 1278 | loss: 2.502290 | lr 1.8000e-04 | norm: 0.1816 | dt: 14508.95ms | tok/sec: 36135.5 | mem: 107563MB | Time Left: 03 hrs : 49 mins [TIMING] data: 7.4ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10871.1ms | clip: 0.0ms | optim: 23.8ms step 1279 | loss: 2.779167 | lr 1.8000e-04 | norm: 0.1864 | dt: 14513.24ms | tok/sec: 36124.8 | mem: 107563MB | Time Left: 03 hrs : 49 mins [TIMING] data: 7.2ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10873.8ms | clip: 0.0ms | optim: 23.8ms step 1280 | loss: 2.823459 | lr 1.8000e-04 | norm: 0.1924 | dt: 14509.80ms | tok/sec: 36133.4 | mem: 107563MB | Time Left: 03 hrs : 49 mins [TIMING] data: 7.3ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10870.3ms | clip: 0.0ms | optim: 23.8ms step 1281 | loss: 3.209310 | lr 1.8000e-04 | norm: 0.1904 | dt: 14507.89ms | tok/sec: 36138.1 | mem: 107563MB | Time Left: 03 hrs : 48 mins [TIMING] data: 8.1ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10869.5ms | clip: 0.0ms | optim: 23.8ms step 1282 | loss: 2.831942 | lr 1.8000e-04 | norm: 0.1743 | dt: 14515.11ms | tok/sec: 36120.2 | mem: 107563MB | Time Left: 03 hrs : 48 mins [TIMING] data: 7.3ms | fwd: 3581.9ms | loss: 13.0ms | bwd: 10875.3ms | clip: 0.0ms | optim: 23.9ms step 1283 | loss: 2.642089 | lr 1.8000e-04 | norm: 0.2262 | dt: 14518.84ms | tok/sec: 36110.9 | mem: 107563MB | Time Left: 03 hrs : 48 mins [TIMING] data: 7.3ms | fwd: 3586.4ms | loss: 13.1ms | bwd: 10874.5ms | clip: 0.0ms | optim: 23.8ms step 1284 | loss: 2.712812 | lr 1.8000e-04 | norm: 0.1578 | dt: 14509.19ms | tok/sec: 36134.9 | mem: 107563MB | Time Left: 03 hrs : 48 mins [TIMING] data: 7.3ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10868.9ms | clip: 0.0ms | optim: 23.9ms step 1285 | loss: 2.963677 | lr 1.8000e-04 | norm: 0.1868 | dt: 14509.43ms | tok/sec: 36134.3 | mem: 107563MB | Time Left: 03 hrs : 47 mins [TIMING] data: 7.4ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10873.0ms | clip: 0.0ms | optim: 23.8ms step 1286 | loss: 3.079166 | lr 1.8000e-04 | norm: 0.2076 | dt: 14508.02ms | tok/sec: 36137.8 | mem: 107563MB | Time Left: 03 hrs : 47 mins [TIMING] data: 7.3ms | fwd: 3585.7ms | loss: 13.0ms | bwd: 10864.6ms | clip: 0.0ms | optim: 23.9ms step 1287 | loss: 2.727073 | lr 1.8000e-04 | norm: 0.1672 | dt: 14514.35ms | tok/sec: 36122.0 | mem: 107563MB | Time Left: 03 hrs : 47 mins [TIMING] data: 7.4ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10875.0ms | clip: 0.0ms | optim: 23.9ms step 1288 | loss: 2.939881 | lr 1.8000e-04 | norm: 0.2154 | dt: 14509.04ms | tok/sec: 36135.3 | mem: 107563MB | Time Left: 03 hrs : 47 mins [TIMING] data: 7.3ms | fwd: 3585.8ms | loss: 13.1ms | bwd: 10865.3ms | clip: 0.0ms | optim: 23.8ms step 1289 | loss: 2.918105 | lr 1.8000e-04 | norm: 0.2464 | dt: 14498.87ms | tok/sec: 36160.6 | mem: 107563MB | Time Left: 03 hrs : 46 mins [TIMING] data: 7.8ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10858.2ms | clip: 0.0ms | optim: 24.0ms step 1290 | loss: 2.941351 | lr 1.8000e-04 | norm: 0.1761 | dt: 14499.86ms | tok/sec: 36158.1 | mem: 107563MB | Time Left: 03 hrs : 46 mins [TIMING] data: 7.2ms | fwd: 3578.0ms | loss: 13.0ms | bwd: 10864.1ms | clip: 0.0ms | optim: 23.8ms step 1291 | loss: 3.017927 | lr 1.8000e-04 | norm: 0.1980 | dt: 14510.49ms | tok/sec: 36131.7 | mem: 107563MB | Time Left: 03 hrs : 46 mins [TIMING] data: 7.3ms | fwd: 3589.9ms | loss: 13.1ms | bwd: 10862.8ms | clip: 0.0ms | optim: 23.8ms step 1292 | loss: 2.968163 | lr 1.8000e-04 | norm: 0.1735 | dt: 14498.36ms | tok/sec: 36161.9 | mem: 107563MB | Time Left: 03 hrs : 45 mins [TIMING] data: 7.8ms | fwd: 3584.5ms | loss: 13.2ms | bwd: 10855.4ms | clip: 0.0ms | optim: 24.0ms step 1293 | loss: 2.687996 | lr 1.8000e-04 | norm: 0.2044 | dt: 14507.16ms | tok/sec: 36139.9 | mem: 107563MB | Time Left: 03 hrs : 45 mins [TIMING] data: 7.3ms | fwd: 3584.5ms | loss: 13.1ms | bwd: 10864.7ms | clip: 0.0ms | optim: 23.8ms step 1294 | loss: 2.950156 | lr 1.8000e-04 | norm: 0.1948 | dt: 14505.75ms | tok/sec: 36143.5 | mem: 107563MB | Time Left: 03 hrs : 45 mins [TIMING] data: 7.2ms | fwd: 3578.7ms | loss: 13.0ms | bwd: 10869.5ms | clip: 0.0ms | optim: 23.8ms step 1295 | loss: 3.019298 | lr 1.8000e-04 | norm: 0.2063 | dt: 14496.31ms | tok/sec: 36167.0 | mem: 107563MB | Time Left: 03 hrs : 45 mins [TIMING] data: 7.3ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10860.6ms | clip: 0.0ms | optim: 23.9ms step 1296 | loss: 2.995997 | lr 1.8000e-04 | norm: 0.1721 | dt: 14499.15ms | tok/sec: 36159.9 | mem: 107563MB | Time Left: 03 hrs : 44 mins [TIMING] data: 7.2ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10860.2ms | clip: 0.0ms | optim: 23.9ms step 1297 | loss: 2.687945 | lr 1.8000e-04 | norm: 0.2482 | dt: 14501.43ms | tok/sec: 36154.2 | mem: 107563MB | Time Left: 03 hrs : 44 mins [TIMING] data: 7.4ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10863.3ms | clip: 0.0ms | optim: 23.8ms step 1298 | loss: 2.712944 | lr 1.8000e-04 | norm: 0.2170 | dt: 14501.20ms | tok/sec: 36154.8 | mem: 107563MB | Time Left: 03 hrs : 44 mins [TIMING] data: 7.3ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10862.5ms | clip: 0.0ms | optim: 23.7ms step 1299 | loss: 2.820602 | lr 1.8000e-04 | norm: 0.1897 | dt: 14504.31ms | tok/sec: 36147.1 | mem: 107563MB | Time Left: 03 hrs : 44 mins [TIMING] data: 7.3ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10863.7ms | clip: 0.0ms | optim: 23.7ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1300.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1200.ckpt step 1300 | loss: 2.751933 | lr 1.8000e-04 | norm: 0.2107 | dt: 14390.78ms | tok/sec: 36432.2 | mem: 107563MB | Time Left: 03 hrs : 42 mins [TIMING] data: 8.5ms | fwd: 3569.8ms | loss: 13.1ms | bwd: 10761.8ms | clip: 0.0ms | optim: 23.8ms step 1301 | loss: 2.677571 | lr 1.8000e-04 | norm: 0.2023 | dt: 14410.77ms | tok/sec: 36381.7 | mem: 107563MB | Time Left: 03 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3561.7ms | loss: 13.1ms | bwd: 10790.3ms | clip: 0.0ms | optim: 23.8ms step 1302 | loss: 2.875799 | lr 1.8000e-04 | norm: 0.1939 | dt: 14424.00ms | tok/sec: 36348.3 | mem: 107563MB | Time Left: 03 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3569.1ms | loss: 13.1ms | bwd: 10796.1ms | clip: 0.0ms | optim: 23.8ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00007.bin 100000000 step 1303 | loss: 2.947572 | lr 1.8000e-04 | norm: 0.2000 | dt: 14451.52ms | tok/sec: 36279.1 | mem: 107563MB | Time Left: 03 hrs : 42 mins [TIMING] data: 13.3ms | fwd: 3573.0ms | loss: 13.1ms | bwd: 10814.8ms | clip: 0.0ms | optim: 23.8ms step 1304 | loss: 2.930834 | lr 1.8000e-04 | norm: 0.2999 | dt: 14459.76ms | tok/sec: 36258.4 | mem: 107563MB | Time Left: 03 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3568.7ms | loss: 13.1ms | bwd: 10832.1ms | clip: 0.0ms | optim: 23.9ms step 1305 | loss: 2.634796 | lr 1.8000e-04 | norm: 0.1840 | dt: 14465.64ms | tok/sec: 36243.7 | mem: 107563MB | Time Left: 03 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3571.2ms | loss: 13.1ms | bwd: 10835.7ms | clip: 0.0ms | optim: 23.9ms step 1306 | loss: 2.774796 | lr 1.8000e-04 | norm: 0.2879 | dt: 14478.05ms | tok/sec: 36212.6 | mem: 107563MB | Time Left: 03 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10843.1ms | clip: 0.0ms | optim: 23.8ms step 1307 | loss: 2.870085 | lr 1.8000e-04 | norm: 0.1821 | dt: 14477.10ms | tok/sec: 36215.0 | mem: 107563MB | Time Left: 03 hrs : 41 mins [TIMING] data: 8.5ms | fwd: 3578.4ms | loss: 13.2ms | bwd: 10839.8ms | clip: 0.0ms | optim: 23.8ms step 1308 | loss: 2.833636 | lr 1.8000e-04 | norm: 0.1872 | dt: 14483.89ms | tok/sec: 36198.0 | mem: 107563MB | Time Left: 03 hrs : 41 mins [TIMING] data: 8.2ms | fwd: 3571.9ms | loss: 13.1ms | bwd: 10853.2ms | clip: 0.0ms | optim: 23.8ms step 1309 | loss: 3.174387 | lr 1.8000e-04 | norm: 0.1884 | dt: 14480.99ms | tok/sec: 36205.3 | mem: 107563MB | Time Left: 03 hrs : 41 mins [TIMING] data: 8.4ms | fwd: 3573.5ms | loss: 13.1ms | bwd: 10848.6ms | clip: 0.0ms | optim: 23.8ms step 1310 | loss: 3.008313 | lr 1.8000e-04 | norm: 0.1861 | dt: 14491.49ms | tok/sec: 36179.0 | mem: 107563MB | Time Left: 03 hrs : 41 mins [TIMING] data: 8.3ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10854.1ms | clip: 0.0ms | optim: 23.9ms step 1311 | loss: 2.768798 | lr 1.8000e-04 | norm: 0.2147 | dt: 14483.93ms | tok/sec: 36197.9 | mem: 107563MB | Time Left: 03 hrs : 41 mins [TIMING] data: 8.5ms | fwd: 3574.2ms | loss: 13.0ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.8ms step 1312 | loss: 2.777717 | lr 1.8000e-04 | norm: 0.1881 | dt: 14505.27ms | tok/sec: 36144.7 | mem: 107563MB | Time Left: 03 hrs : 41 mins [TIMING] data: 8.2ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10869.3ms | clip: 0.0ms | optim: 23.8ms step 1313 | loss: 2.939313 | lr 1.8000e-04 | norm: 0.1819 | dt: 14491.55ms | tok/sec: 36178.9 | mem: 107563MB | Time Left: 03 hrs : 40 mins [TIMING] data: 8.4ms | fwd: 3575.6ms | loss: 13.1ms | bwd: 10856.9ms | clip: 0.0ms | optim: 24.1ms step 1314 | loss: 2.997847 | lr 1.8000e-04 | norm: 0.1893 | dt: 14496.41ms | tok/sec: 36166.8 | mem: 107563MB | Time Left: 03 hrs : 40 mins [TIMING] data: 8.4ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10854.4ms | clip: 0.0ms | optim: 23.8ms step 1315 | loss: 2.973136 | lr 1.8000e-04 | norm: 0.1915 | dt: 14485.03ms | tok/sec: 36195.2 | mem: 107563MB | Time Left: 03 hrs : 40 mins [TIMING] data: 8.3ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10848.0ms | clip: 0.0ms | optim: 23.9ms step 1316 | loss: 2.825548 | lr 1.8000e-04 | norm: 0.1923 | dt: 14486.67ms | tok/sec: 36191.1 | mem: 107563MB | Time Left: 03 hrs : 39 mins [TIMING] data: 8.2ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10850.5ms | clip: 0.0ms | optim: 23.8ms step 1317 | loss: 2.906786 | lr 1.8000e-04 | norm: 0.2147 | dt: 14498.12ms | tok/sec: 36162.5 | mem: 107563MB | Time Left: 03 hrs : 39 mins [TIMING] data: 8.4ms | fwd: 3577.0ms | loss: 13.0ms | bwd: 10862.5ms | clip: 0.0ms | optim: 23.8ms step 1318 | loss: 2.805311 | lr 1.8000e-04 | norm: 0.1651 | dt: 14486.37ms | tok/sec: 36191.8 | mem: 107563MB | Time Left: 03 hrs : 39 mins [TIMING] data: 8.3ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10851.5ms | clip: 0.0ms | optim: 23.8ms step 1319 | loss: 2.603653 | lr 1.8000e-04 | norm: 0.1893 | dt: 14490.20ms | tok/sec: 36182.3 | mem: 107563MB | Time Left: 03 hrs : 39 mins [TIMING] data: 8.5ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10853.2ms | clip: 0.0ms | optim: 23.9ms step 1320 | loss: 2.944095 | lr 1.8000e-04 | norm: 0.1643 | dt: 14484.29ms | tok/sec: 36197.0 | mem: 107563MB | Time Left: 03 hrs : 38 mins [TIMING] data: 8.0ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10844.8ms | clip: 0.0ms | optim: 23.8ms step 1321 | loss: 2.930614 | lr 1.8000e-04 | norm: 0.1975 | dt: 14492.02ms | tok/sec: 36177.7 | mem: 107563MB | Time Left: 03 hrs : 38 mins [TIMING] data: 8.2ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10854.3ms | clip: 0.0ms | optim: 23.8ms step 1322 | loss: 2.615974 | lr 1.8000e-04 | norm: 0.1631 | dt: 14502.58ms | tok/sec: 36151.4 | mem: 107563MB | Time Left: 03 hrs : 38 mins [TIMING] data: 8.3ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10863.4ms | clip: 0.0ms | optim: 23.8ms step 1323 | loss: 3.033681 | lr 1.8000e-04 | norm: 0.1883 | dt: 14496.12ms | tok/sec: 36167.5 | mem: 107563MB | Time Left: 03 hrs : 38 mins [TIMING] data: 8.4ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10857.2ms | clip: 0.0ms | optim: 23.8ms step 1324 | loss: 2.934077 | lr 1.8000e-04 | norm: 0.1791 | dt: 14482.94ms | tok/sec: 36200.4 | mem: 107563MB | Time Left: 03 hrs : 37 mins [TIMING] data: 8.3ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10847.3ms | clip: 0.0ms | optim: 23.8ms step 1325 | loss: 3.110812 | lr 1.8000e-04 | norm: 0.1839 | dt: 14498.79ms | tok/sec: 36160.8 | mem: 107563MB | Time Left: 03 hrs : 37 mins [TIMING] data: 9.1ms | fwd: 3580.7ms | loss: 13.0ms | bwd: 10858.6ms | clip: 0.0ms | optim: 23.9ms step 1326 | loss: 2.725284 | lr 1.8000e-04 | norm: 0.2078 | dt: 14481.84ms | tok/sec: 36203.1 | mem: 107563MB | Time Left: 03 hrs : 37 mins [TIMING] data: 8.5ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10842.7ms | clip: 0.0ms | optim: 23.8ms step 1327 | loss: 2.731702 | lr 1.8000e-04 | norm: 0.1821 | dt: 14472.08ms | tok/sec: 36227.6 | mem: 107563MB | Time Left: 03 hrs : 37 mins [TIMING] data: 8.4ms | fwd: 3575.5ms | loss: 13.1ms | bwd: 10837.8ms | clip: 0.0ms | optim: 23.8ms step 1328 | loss: 2.948140 | lr 1.8000e-04 | norm: 0.3609 | dt: 14480.34ms | tok/sec: 36206.9 | mem: 107563MB | Time Left: 03 hrs : 36 mins [TIMING] data: 8.5ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10843.1ms | clip: 0.0ms | optim: 23.9ms step 1329 | loss: 2.898211 | lr 1.8000e-04 | norm: 0.1922 | dt: 14478.64ms | tok/sec: 36211.1 | mem: 107563MB | Time Left: 03 hrs : 36 mins [TIMING] data: 8.5ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10839.4ms | clip: 0.0ms | optim: 23.8ms step 1330 | loss: 3.131566 | lr 1.8000e-04 | norm: 0.2398 | dt: 14490.63ms | tok/sec: 36181.2 | mem: 107563MB | Time Left: 03 hrs : 36 mins [TIMING] data: 8.5ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10851.0ms | clip: 0.0ms | optim: 23.8ms step 1331 | loss: 3.007541 | lr 1.8000e-04 | norm: 0.2097 | dt: 14490.55ms | tok/sec: 36181.4 | mem: 107563MB | Time Left: 03 hrs : 36 mins [TIMING] data: 8.4ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10851.5ms | clip: 0.0ms | optim: 23.8ms step 1332 | loss: 2.702677 | lr 1.8000e-04 | norm: 0.2068 | dt: 14477.37ms | tok/sec: 36214.3 | mem: 107563MB | Time Left: 03 hrs : 35 mins [TIMING] data: 8.1ms | fwd: 3574.2ms | loss: 13.1ms | bwd: 10844.5ms | clip: 0.0ms | optim: 23.9ms step 1333 | loss: 2.723425 | lr 1.8000e-04 | norm: 0.1974 | dt: 14483.99ms | tok/sec: 36197.8 | mem: 107563MB | Time Left: 03 hrs : 35 mins [TIMING] data: 8.6ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10845.8ms | clip: 0.0ms | optim: 23.8ms step 1334 | loss: 2.941449 | lr 1.8000e-04 | norm: 0.1886 | dt: 14485.37ms | tok/sec: 36194.3 | mem: 107563MB | Time Left: 03 hrs : 35 mins [TIMING] data: 8.5ms | fwd: 3577.4ms | loss: 13.2ms | bwd: 10848.0ms | clip: 0.0ms | optim: 23.9ms step 1335 | loss: 2.709846 | lr 1.8000e-04 | norm: 0.3291 | dt: 14480.74ms | tok/sec: 36205.9 | mem: 107563MB | Time Left: 03 hrs : 35 mins [TIMING] data: 8.5ms | fwd: 3576.2ms | loss: 13.0ms | bwd: 10845.7ms | clip: 0.0ms | optim: 23.8ms step 1336 | loss: 2.795305 | lr 1.8000e-04 | norm: 0.1866 | dt: 14490.78ms | tok/sec: 36180.8 | mem: 107563MB | Time Left: 03 hrs : 35 mins [TIMING] data: 8.4ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10854.1ms | clip: 0.0ms | optim: 23.8ms step 1337 | loss: 2.885769 | lr 1.8000e-04 | norm: 0.1842 | dt: 14491.22ms | tok/sec: 36179.7 | mem: 107563MB | Time Left: 03 hrs : 34 mins [TIMING] data: 8.6ms | fwd: 3576.4ms | loss: 13.1ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.8ms step 1338 | loss: 2.952849 | lr 1.8000e-04 | norm: 0.2056 | dt: 14498.59ms | tok/sec: 36161.3 | mem: 107563MB | Time Left: 03 hrs : 34 mins [TIMING] data: 8.4ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10864.0ms | clip: 0.0ms | optim: 23.7ms step 1339 | loss: 2.831620 | lr 1.8000e-04 | norm: 0.1577 | dt: 14495.10ms | tok/sec: 36170.0 | mem: 107563MB | Time Left: 03 hrs : 34 mins [TIMING] data: 8.5ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10858.9ms | clip: 0.0ms | optim: 23.8ms step 1340 | loss: 2.704482 | lr 1.8000e-04 | norm: 0.2012 | dt: 14487.49ms | tok/sec: 36189.0 | mem: 107563MB | Time Left: 03 hrs : 34 mins [TIMING] data: 8.3ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10851.4ms | clip: 0.0ms | optim: 23.8ms step 1341 | loss: 2.928706 | lr 1.8000e-04 | norm: 0.2305 | dt: 14490.17ms | tok/sec: 36182.3 | mem: 107563MB | Time Left: 03 hrs : 33 mins [TIMING] data: 8.3ms | fwd: 3578.7ms | loss: 13.0ms | bwd: 10852.6ms | clip: 0.0ms | optim: 23.7ms step 1342 | loss: 2.731091 | lr 1.8000e-04 | norm: 0.1969 | dt: 14487.16ms | tok/sec: 36189.8 | mem: 107563MB | Time Left: 03 hrs : 33 mins [TIMING] data: 8.5ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10849.7ms | clip: 0.0ms | optim: 23.8ms step 1343 | loss: 2.760019 | lr 1.8000e-04 | norm: 0.1672 | dt: 14479.15ms | tok/sec: 36209.9 | mem: 107563MB | Time Left: 03 hrs : 33 mins [TIMING] data: 8.5ms | fwd: 3573.6ms | loss: 13.1ms | bwd: 10846.6ms | clip: 0.0ms | optim: 23.9ms step 1344 | loss: 2.848040 | lr 1.8000e-04 | norm: 0.1870 | dt: 14489.29ms | tok/sec: 36184.5 | mem: 107563MB | Time Left: 03 hrs : 33 mins [TIMING] data: 8.2ms | fwd: 3581.2ms | loss: 13.0ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 1345 | loss: 2.778184 | lr 1.8000e-04 | norm: 0.1836 | dt: 14487.24ms | tok/sec: 36189.6 | mem: 107563MB | Time Left: 03 hrs : 32 mins [TIMING] data: 8.5ms | fwd: 3573.7ms | loss: 13.0ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.8ms step 1346 | loss: 2.899295 | lr 1.8000e-04 | norm: 0.1952 | dt: 14481.96ms | tok/sec: 36202.8 | mem: 107563MB | Time Left: 03 hrs : 32 mins [TIMING] data: 8.5ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10846.2ms | clip: 0.0ms | optim: 23.8ms step 1347 | loss: 2.583215 | lr 1.8000e-04 | norm: 0.3700 | dt: 14488.91ms | tok/sec: 36185.5 | mem: 107563MB | Time Left: 03 hrs : 32 mins [TIMING] data: 8.4ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.8ms step 1348 | loss: 2.902390 | lr 1.8000e-04 | norm: 0.1989 | dt: 14488.89ms | tok/sec: 36185.5 | mem: 107563MB | Time Left: 03 hrs : 32 mins [TIMING] data: 9.2ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10846.7ms | clip: 0.0ms | optim: 23.8ms step 1349 | loss: 2.595682 | lr 1.8000e-04 | norm: 0.1559 | dt: 14478.92ms | tok/sec: 36210.4 | mem: 107563MB | Time Left: 03 hrs : 31 mins [TIMING] data: 8.4ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1350.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1250.ckpt step 1350 | loss: 3.106918 | lr 1.8000e-04 | norm: 0.2577 | dt: 14392.83ms | tok/sec: 36427.0 | mem: 107563MB | Time Left: 03 hrs : 30 mins [TIMING] data: 9.5ms | fwd: 3578.9ms | loss: 13.2ms | bwd: 10753.2ms | clip: 0.0ms | optim: 23.8ms step 1351 | loss: 2.893814 | lr 1.8000e-04 | norm: 0.1706 | dt: 14408.31ms | tok/sec: 36387.9 | mem: 107563MB | Time Left: 03 hrs : 30 mins [TIMING] data: 8.5ms | fwd: 3566.9ms | loss: 13.1ms | bwd: 10782.6ms | clip: 0.0ms | optim: 23.8ms step 1352 | loss: 2.964212 | lr 1.8000e-04 | norm: 0.1991 | dt: 14430.14ms | tok/sec: 36332.8 | mem: 107563MB | Time Left: 03 hrs : 30 mins [TIMING] data: 8.6ms | fwd: 3568.0ms | loss: 13.1ms | bwd: 10802.9ms | clip: 0.0ms | optim: 23.8ms step 1353 | loss: 2.815975 | lr 1.8000e-04 | norm: 0.1689 | dt: 14452.15ms | tok/sec: 36277.5 | mem: 107563MB | Time Left: 03 hrs : 30 mins [TIMING] data: 8.5ms | fwd: 3571.7ms | loss: 13.1ms | bwd: 10821.4ms | clip: 0.0ms | optim: 23.9ms step 1354 | loss: 2.742588 | lr 1.8000e-04 | norm: 0.2169 | dt: 14452.84ms | tok/sec: 36275.8 | mem: 107563MB | Time Left: 03 hrs : 30 mins [TIMING] data: 8.5ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10818.4ms | clip: 0.0ms | optim: 23.8ms step 1355 | loss: 2.889076 | lr 1.8000e-04 | norm: 0.1682 | dt: 14469.66ms | tok/sec: 36233.6 | mem: 107563MB | Time Left: 03 hrs : 30 mins [TIMING] data: 8.4ms | fwd: 3575.4ms | loss: 13.1ms | bwd: 10835.4ms | clip: 0.0ms | optim: 23.8ms step 1356 | loss: 2.800293 | lr 1.8000e-04 | norm: 0.3305 | dt: 14460.09ms | tok/sec: 36257.6 | mem: 107563MB | Time Left: 03 hrs : 29 mins [TIMING] data: 8.5ms | fwd: 3572.6ms | loss: 13.1ms | bwd: 10828.5ms | clip: 0.0ms | optim: 23.8ms step 1357 | loss: 3.068258 | lr 1.8000e-04 | norm: 0.1923 | dt: 14476.15ms | tok/sec: 36217.4 | mem: 107563MB | Time Left: 03 hrs : 29 mins [TIMING] data: 8.5ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10838.3ms | clip: 0.0ms | optim: 23.9ms step 1358 | loss: 2.858443 | lr 1.8000e-04 | norm: 0.1796 | dt: 14477.22ms | tok/sec: 36214.7 | mem: 107563MB | Time Left: 03 hrs : 29 mins [TIMING] data: 8.4ms | fwd: 3575.3ms | loss: 13.2ms | bwd: 10842.9ms | clip: 0.0ms | optim: 23.8ms step 1359 | loss: 2.770777 | lr 1.8000e-04 | norm: 0.1860 | dt: 14479.10ms | tok/sec: 36210.0 | mem: 107563MB | Time Left: 03 hrs : 29 mins [TIMING] data: 8.4ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10839.0ms | clip: 0.0ms | optim: 23.9ms step 1360 | loss: 2.687751 | lr 1.8000e-04 | norm: 0.1705 | dt: 14479.70ms | tok/sec: 36208.5 | mem: 107563MB | Time Left: 03 hrs : 29 mins [TIMING] data: 8.6ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.8ms step 1361 | loss: 2.867905 | lr 1.8000e-04 | norm: 0.1800 | dt: 14458.74ms | tok/sec: 36261.0 | mem: 107563MB | Time Left: 03 hrs : 28 mins [TIMING] data: 8.3ms | fwd: 3574.7ms | loss: 13.1ms | bwd: 10825.1ms | clip: 0.0ms | optim: 23.8ms step 1362 | loss: 3.026402 | lr 1.8000e-04 | norm: 0.2617 | dt: 14480.01ms | tok/sec: 36207.7 | mem: 107563MB | Time Left: 03 hrs : 28 mins [TIMING] data: 8.5ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10844.2ms | clip: 0.0ms | optim: 23.8ms step 1363 | loss: 3.125556 | lr 1.8000e-04 | norm: 0.1825 | dt: 14482.06ms | tok/sec: 36202.6 | mem: 107563MB | Time Left: 03 hrs : 28 mins [TIMING] data: 8.0ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10844.4ms | clip: 0.0ms | optim: 23.9ms step 1364 | loss: 3.037429 | lr 1.8000e-04 | norm: 0.2176 | dt: 14474.61ms | tok/sec: 36221.2 | mem: 107563MB | Time Left: 03 hrs : 28 mins [TIMING] data: 8.5ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10833.4ms | clip: 0.0ms | optim: 23.8ms step 1365 | loss: 2.634530 | lr 1.8000e-04 | norm: 0.2170 | dt: 14479.86ms | tok/sec: 36208.1 | mem: 107563MB | Time Left: 03 hrs : 28 mins [TIMING] data: 8.3ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10840.0ms | clip: 0.0ms | optim: 23.8ms step 1366 | loss: 2.931042 | lr 1.8000e-04 | norm: 0.2810 | dt: 14487.11ms | tok/sec: 36190.0 | mem: 107563MB | Time Left: 03 hrs : 27 mins [TIMING] data: 8.5ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10850.5ms | clip: 0.0ms | optim: 23.8ms step 1367 | loss: 2.610635 | lr 1.8000e-04 | norm: 0.2233 | dt: 14482.34ms | tok/sec: 36201.9 | mem: 107563MB | Time Left: 03 hrs : 27 mins [TIMING] data: 8.4ms | fwd: 3576.4ms | loss: 13.1ms | bwd: 10846.9ms | clip: 0.0ms | optim: 23.9ms step 1368 | loss: 2.885077 | lr 1.8000e-04 | norm: 0.2235 | dt: 14483.08ms | tok/sec: 36200.0 | mem: 107563MB | Time Left: 03 hrs : 27 mins [TIMING] data: 9.2ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10845.1ms | clip: 0.0ms | optim: 23.9ms step 1369 | loss: 2.899537 | lr 1.8000e-04 | norm: 0.1778 | dt: 14483.56ms | tok/sec: 36198.8 | mem: 107563MB | Time Left: 03 hrs : 27 mins [TIMING] data: 8.4ms | fwd: 3585.0ms | loss: 13.1ms | bwd: 10839.4ms | clip: 0.0ms | optim: 23.9ms step 1370 | loss: 2.806419 | lr 1.8000e-04 | norm: 0.1719 | dt: 14480.13ms | tok/sec: 36207.4 | mem: 107563MB | Time Left: 03 hrs : 26 mins [TIMING] data: 8.4ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10841.1ms | clip: 0.0ms | optim: 23.7ms step 1371 | loss: 2.863074 | lr 1.8000e-04 | norm: 0.2116 | dt: 14480.41ms | tok/sec: 36206.7 | mem: 107563MB | Time Left: 03 hrs : 26 mins [TIMING] data: 8.4ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10841.9ms | clip: 0.0ms | optim: 23.8ms step 1372 | loss: 2.845156 | lr 1.8000e-04 | norm: 0.1999 | dt: 14490.84ms | tok/sec: 36180.7 | mem: 107563MB | Time Left: 03 hrs : 26 mins [TIMING] data: 8.1ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10850.6ms | clip: 0.0ms | optim: 23.8ms step 1373 | loss: 2.529311 | lr 1.8000e-04 | norm: 0.1928 | dt: 14483.54ms | tok/sec: 36198.9 | mem: 107563MB | Time Left: 03 hrs : 26 mins [TIMING] data: 8.6ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10847.2ms | clip: 0.0ms | optim: 23.9ms step 1374 | loss: 2.788997 | lr 1.8000e-04 | norm: 0.8608 | dt: 14482.97ms | tok/sec: 36200.3 | mem: 107563MB | Time Left: 03 hrs : 25 mins [TIMING] data: 8.3ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10844.7ms | clip: 0.0ms | optim: 24.4ms step 1375 | loss: 2.822968 | lr 1.8000e-04 | norm: 0.1901 | dt: 14473.46ms | tok/sec: 36224.1 | mem: 107563MB | Time Left: 03 hrs : 25 mins [TIMING] data: 8.4ms | fwd: 3579.2ms | loss: 13.0ms | bwd: 10835.5ms | clip: 0.0ms | optim: 23.8ms step 1376 | loss: 2.800792 | lr 1.8000e-04 | norm: 0.1700 | dt: 14476.26ms | tok/sec: 36217.1 | mem: 107563MB | Time Left: 03 hrs : 25 mins [TIMING] data: 8.3ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10837.0ms | clip: 0.0ms | optim: 23.8ms step 1377 | loss: 3.100550 | lr 1.8000e-04 | norm: 0.2400 | dt: 14476.49ms | tok/sec: 36216.5 | mem: 107563MB | Time Left: 03 hrs : 25 mins [TIMING] data: 8.4ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10841.0ms | clip: 0.0ms | optim: 23.9ms step 1378 | loss: 3.163127 | lr 1.8000e-04 | norm: 0.2027 | dt: 14486.08ms | tok/sec: 36192.5 | mem: 107563MB | Time Left: 03 hrs : 24 mins [TIMING] data: 8.1ms | fwd: 3574.6ms | loss: 13.1ms | bwd: 10852.8ms | clip: 0.0ms | optim: 23.8ms step 1379 | loss: 2.692501 | lr 1.8000e-04 | norm: 0.1689 | dt: 14490.57ms | tok/sec: 36181.3 | mem: 107563MB | Time Left: 03 hrs : 24 mins [TIMING] data: 8.4ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10851.4ms | clip: 0.0ms | optim: 23.9ms step 1380 | loss: 3.102488 | lr 1.8000e-04 | norm: 0.1957 | dt: 14488.49ms | tok/sec: 36186.5 | mem: 107563MB | Time Left: 03 hrs : 24 mins [TIMING] data: 8.6ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.8ms step 1381 | loss: 2.948800 | lr 1.8000e-04 | norm: 0.1908 | dt: 14480.07ms | tok/sec: 36207.6 | mem: 107563MB | Time Left: 03 hrs : 24 mins [TIMING] data: 8.5ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10841.6ms | clip: 0.0ms | optim: 23.9ms step 1382 | loss: 2.745876 | lr 1.8000e-04 | norm: 0.2441 | dt: 14487.03ms | tok/sec: 36190.2 | mem: 107563MB | Time Left: 03 hrs : 24 mins [TIMING] data: 8.6ms | fwd: 3583.9ms | loss: 13.1ms | bwd: 10844.2ms | clip: 0.0ms | optim: 23.7ms step 1383 | loss: 2.863665 | lr 1.8000e-04 | norm: 0.1966 | dt: 14487.81ms | tok/sec: 36188.2 | mem: 107563MB | Time Left: 03 hrs : 23 mins [TIMING] data: 8.5ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10846.7ms | clip: 0.0ms | optim: 23.8ms step 1384 | loss: 2.875562 | lr 1.8000e-04 | norm: 0.1958 | dt: 14495.63ms | tok/sec: 36168.7 | mem: 107563MB | Time Left: 03 hrs : 23 mins [TIMING] data: 8.3ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10857.2ms | clip: 0.0ms | optim: 23.8ms step 1385 | loss: 2.491483 | lr 1.8000e-04 | norm: 0.1947 | dt: 14483.68ms | tok/sec: 36198.5 | mem: 107563MB | Time Left: 03 hrs : 23 mins [TIMING] data: 8.4ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10844.8ms | clip: 0.0ms | optim: 23.8ms step 1386 | loss: 2.826955 | lr 1.8000e-04 | norm: 0.1985 | dt: 14493.53ms | tok/sec: 36173.9 | mem: 107563MB | Time Left: 03 hrs : 23 mins [TIMING] data: 8.5ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.7ms step 1387 | loss: 2.713579 | lr 1.8000e-04 | norm: 0.1629 | dt: 14487.93ms | tok/sec: 36187.9 | mem: 107563MB | Time Left: 03 hrs : 22 mins [TIMING] data: 8.4ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10851.5ms | clip: 0.0ms | optim: 23.9ms step 1388 | loss: 2.922580 | lr 1.8000e-04 | norm: 0.2341 | dt: 14500.96ms | tok/sec: 36155.4 | mem: 107563MB | Time Left: 03 hrs : 22 mins [TIMING] data: 8.4ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10864.4ms | clip: 0.0ms | optim: 24.0ms step 1389 | loss: 2.831674 | lr 1.8000e-04 | norm: 0.1621 | dt: 14487.91ms | tok/sec: 36188.0 | mem: 107563MB | Time Left: 03 hrs : 22 mins [TIMING] data: 8.7ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10848.6ms | clip: 0.0ms | optim: 23.8ms step 1390 | loss: 2.746745 | lr 1.8000e-04 | norm: 0.2477 | dt: 14487.20ms | tok/sec: 36189.7 | mem: 107563MB | Time Left: 03 hrs : 22 mins [TIMING] data: 8.7ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10843.9ms | clip: 0.0ms | optim: 23.9ms step 1391 | loss: 3.062326 | lr 1.8000e-04 | norm: 0.1743 | dt: 14485.26ms | tok/sec: 36194.6 | mem: 107563MB | Time Left: 03 hrs : 21 mins [TIMING] data: 8.4ms | fwd: 3579.8ms | loss: 13.2ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.8ms step 1392 | loss: 2.849412 | lr 1.8000e-04 | norm: 0.2766 | dt: 14489.55ms | tok/sec: 36183.9 | mem: 107563MB | Time Left: 03 hrs : 21 mins [TIMING] data: 8.4ms | fwd: 3579.0ms | loss: 13.2ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.8ms step 1393 | loss: 2.789699 | lr 1.8000e-04 | norm: 0.1960 | dt: 14495.89ms | tok/sec: 36168.1 | mem: 107563MB | Time Left: 03 hrs : 21 mins [TIMING] data: 8.5ms | fwd: 3580.6ms | loss: 13.1ms | bwd: 10856.2ms | clip: 0.0ms | optim: 23.8ms step 1394 | loss: 2.901407 | lr 1.8000e-04 | norm: 0.1985 | dt: 14486.43ms | tok/sec: 36191.7 | mem: 107563MB | Time Left: 03 hrs : 21 mins [TIMING] data: 8.4ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.9ms step 1395 | loss: 2.574965 | lr 1.8000e-04 | norm: 0.1681 | dt: 14496.39ms | tok/sec: 36166.8 | mem: 107563MB | Time Left: 03 hrs : 21 mins [TIMING] data: 8.5ms | fwd: 3584.3ms | loss: 13.1ms | bwd: 10853.2ms | clip: 0.0ms | optim: 23.7ms step 1396 | loss: 2.929664 | lr 1.8000e-04 | norm: 0.2038 | dt: 14492.68ms | tok/sec: 36176.0 | mem: 107563MB | Time Left: 03 hrs : 20 mins [TIMING] data: 8.3ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.9ms step 1397 | loss: 3.054278 | lr 1.8000e-04 | norm: 0.2462 | dt: 14490.62ms | tok/sec: 36181.2 | mem: 107563MB | Time Left: 03 hrs : 20 mins [TIMING] data: 8.7ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.9ms step 1398 | loss: 2.976108 | lr 1.8000e-04 | norm: 0.1804 | dt: 14486.10ms | tok/sec: 36192.5 | mem: 107563MB | Time Left: 03 hrs : 20 mins [TIMING] data: 8.3ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10847.5ms | clip: 0.0ms | optim: 23.8ms step 1399 | loss: 3.064082 | lr 1.8000e-04 | norm: 0.2139 | dt: 14487.83ms | tok/sec: 36188.2 | mem: 107563MB | Time Left: 03 hrs : 19 mins [TIMING] data: 8.6ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10849.5ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1400.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1300.ckpt step 1400 | loss: 2.986846 | lr 1.8000e-04 | norm: 0.1889 | dt: 14390.16ms | tok/sec: 36433.8 | mem: 107563MB | Time Left: 03 hrs : 18 mins [TIMING] data: 9.1ms | fwd: 3567.7ms | loss: 13.1ms | bwd: 10762.0ms | clip: 0.0ms | optim: 23.7ms step 1401 | loss: 2.687617 | lr 1.8000e-04 | norm: 0.1595 | dt: 14416.47ms | tok/sec: 36367.3 | mem: 107563MB | Time Left: 03 hrs : 18 mins [TIMING] data: 8.2ms | fwd: 3558.0ms | loss: 13.1ms | bwd: 10799.6ms | clip: 0.0ms | optim: 23.8ms step 1402 | loss: 2.552547 | lr 1.8000e-04 | norm: 0.1726 | dt: 14427.90ms | tok/sec: 36338.5 | mem: 107563MB | Time Left: 03 hrs : 18 mins [TIMING] data: 8.4ms | fwd: 3568.1ms | loss: 13.1ms | bwd: 10801.1ms | clip: 0.0ms | optim: 23.8ms step 1403 | loss: 2.941677 | lr 1.8000e-04 | norm: 0.2136 | dt: 14433.27ms | tok/sec: 36325.0 | mem: 107563MB | Time Left: 03 hrs : 18 mins [TIMING] data: 8.3ms | fwd: 3573.6ms | loss: 13.1ms | bwd: 10800.8ms | clip: 0.0ms | optim: 23.9ms step 1404 | loss: 2.713806 | lr 1.8000e-04 | norm: 0.1918 | dt: 14439.25ms | tok/sec: 36309.9 | mem: 107563MB | Time Left: 03 hrs : 18 mins [TIMING] data: 8.4ms | fwd: 3564.9ms | loss: 13.1ms | bwd: 10815.6ms | clip: 0.0ms | optim: 23.8ms step 1405 | loss: 2.867110 | lr 1.8000e-04 | norm: 0.2161 | dt: 14453.29ms | tok/sec: 36274.7 | mem: 107563MB | Time Left: 03 hrs : 18 mins [TIMING] data: 8.2ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10818.7ms | clip: 0.0ms | optim: 23.8ms step 1406 | loss: 3.077420 | lr 1.8000e-04 | norm: 0.1892 | dt: 14477.07ms | tok/sec: 36215.1 | mem: 107563MB | Time Left: 03 hrs : 18 mins [TIMING] data: 7.7ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10838.1ms | clip: 0.0ms | optim: 23.8ms step 1407 | loss: 2.817098 | lr 1.8000e-04 | norm: 0.2046 | dt: 14474.44ms | tok/sec: 36221.7 | mem: 107563MB | Time Left: 03 hrs : 17 mins [TIMING] data: 8.5ms | fwd: 3573.7ms | loss: 13.1ms | bwd: 10841.9ms | clip: 0.0ms | optim: 23.8ms step 1408 | loss: 2.915158 | lr 1.8000e-04 | norm: 0.2141 | dt: 14478.09ms | tok/sec: 36212.5 | mem: 107563MB | Time Left: 03 hrs : 17 mins [TIMING] data: 8.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10837.6ms | clip: 0.0ms | optim: 23.9ms step 1409 | loss: 2.854771 | lr 1.8000e-04 | norm: 0.1977 | dt: 14481.24ms | tok/sec: 36204.6 | mem: 107563MB | Time Left: 03 hrs : 17 mins [TIMING] data: 8.4ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10841.5ms | clip: 0.0ms | optim: 23.9ms step 1410 | loss: 2.914188 | lr 1.8000e-04 | norm: 0.2016 | dt: 14482.61ms | tok/sec: 36201.2 | mem: 107563MB | Time Left: 03 hrs : 17 mins [TIMING] data: 8.4ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10842.8ms | clip: 0.0ms | optim: 23.8ms step 1411 | loss: 2.873212 | lr 1.8000e-04 | norm: 0.1867 | dt: 14479.23ms | tok/sec: 36209.7 | mem: 107563MB | Time Left: 03 hrs : 16 mins [TIMING] data: 8.4ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10844.2ms | clip: 0.0ms | optim: 23.8ms step 1412 | loss: 2.789384 | lr 1.8000e-04 | norm: 0.2315 | dt: 14474.56ms | tok/sec: 36221.3 | mem: 107563MB | Time Left: 03 hrs : 16 mins [TIMING] data: 8.5ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.8ms step 1413 | loss: 2.808175 | lr 1.8000e-04 | norm: 0.2057 | dt: 14477.99ms | tok/sec: 36212.8 | mem: 107563MB | Time Left: 03 hrs : 16 mins [TIMING] data: 8.4ms | fwd: 3575.5ms | loss: 13.1ms | bwd: 10843.1ms | clip: 0.0ms | optim: 23.8ms step 1414 | loss: 2.755496 | lr 1.8000e-04 | norm: 0.4710 | dt: 14477.79ms | tok/sec: 36213.3 | mem: 107563MB | Time Left: 03 hrs : 16 mins [TIMING] data: 8.5ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10840.1ms | clip: 0.0ms | optim: 23.9ms step 1415 | loss: 2.917889 | lr 1.8000e-04 | norm: 0.3344 | dt: 14474.87ms | tok/sec: 36220.6 | mem: 107563MB | Time Left: 03 hrs : 15 mins [TIMING] data: 8.4ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10840.0ms | clip: 0.0ms | optim: 23.9ms step 1416 | loss: 2.981442 | lr 1.8000e-04 | norm: 0.1868 | dt: 14499.01ms | tok/sec: 36160.3 | mem: 107563MB | Time Left: 03 hrs : 15 mins [TIMING] data: 8.5ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10857.4ms | clip: 0.0ms | optim: 23.9ms step 1417 | loss: 2.782377 | lr 1.8000e-04 | norm: 0.3122 | dt: 14473.87ms | tok/sec: 36223.1 | mem: 107563MB | Time Left: 03 hrs : 15 mins [TIMING] data: 8.5ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10839.2ms | clip: 0.0ms | optim: 23.8ms step 1418 | loss: 3.021623 | lr 1.8000e-04 | norm: 0.2077 | dt: 14480.70ms | tok/sec: 36206.0 | mem: 107563MB | Time Left: 03 hrs : 15 mins [TIMING] data: 8.6ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10840.9ms | clip: 0.0ms | optim: 23.8ms step 1419 | loss: 3.025213 | lr 1.8000e-04 | norm: 0.1886 | dt: 14478.52ms | tok/sec: 36211.4 | mem: 107563MB | Time Left: 03 hrs : 14 mins [TIMING] data: 8.5ms | fwd: 3579.2ms | loss: 13.0ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.8ms step 1420 | loss: 2.862630 | lr 1.8000e-04 | norm: 0.2055 | dt: 14495.84ms | tok/sec: 36168.2 | mem: 107563MB | Time Left: 03 hrs : 14 mins [TIMING] data: 8.4ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10855.4ms | clip: 0.0ms | optim: 23.9ms step 1421 | loss: 3.101984 | lr 1.8000e-04 | norm: 0.2468 | dt: 14477.99ms | tok/sec: 36212.8 | mem: 107563MB | Time Left: 03 hrs : 14 mins [TIMING] data: 8.5ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10842.5ms | clip: 0.0ms | optim: 23.9ms step 1422 | loss: 2.750111 | lr 1.8000e-04 | norm: 0.1907 | dt: 14481.25ms | tok/sec: 36204.6 | mem: 107563MB | Time Left: 03 hrs : 14 mins [TIMING] data: 8.4ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10845.8ms | clip: 0.0ms | optim: 23.8ms step 1423 | loss: 2.946759 | lr 1.8000e-04 | norm: 0.1887 | dt: 14492.37ms | tok/sec: 36176.8 | mem: 107563MB | Time Left: 03 hrs : 14 mins [TIMING] data: 8.5ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10852.5ms | clip: 0.0ms | optim: 23.9ms step 1424 | loss: 2.991897 | lr 1.8000e-04 | norm: 0.2200 | dt: 14482.24ms | tok/sec: 36202.1 | mem: 107563MB | Time Left: 03 hrs : 13 mins [TIMING] data: 8.3ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10845.1ms | clip: 0.0ms | optim: 23.7ms step 1425 | loss: 3.247571 | lr 1.8000e-04 | norm: 0.2088 | dt: 14487.72ms | tok/sec: 36188.4 | mem: 107563MB | Time Left: 03 hrs : 13 mins [TIMING] data: 8.4ms | fwd: 3577.7ms | loss: 13.2ms | bwd: 10851.1ms | clip: 0.0ms | optim: 23.8ms step 1426 | loss: 2.987339 | lr 1.8000e-04 | norm: 0.1778 | dt: 14480.48ms | tok/sec: 36206.5 | mem: 107563MB | Time Left: 03 hrs : 13 mins [TIMING] data: 8.5ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10842.4ms | clip: 0.0ms | optim: 23.8ms step 1427 | loss: 2.881803 | lr 1.8000e-04 | norm: 0.2221 | dt: 14469.02ms | tok/sec: 36235.2 | mem: 107563MB | Time Left: 03 hrs : 12 mins [TIMING] data: 8.2ms | fwd: 3572.4ms | loss: 13.1ms | bwd: 10837.9ms | clip: 0.0ms | optim: 23.9ms step 1428 | loss: 3.008713 | lr 1.8000e-04 | norm: 0.1791 | dt: 14480.05ms | tok/sec: 36207.6 | mem: 107563MB | Time Left: 03 hrs : 12 mins [TIMING] data: 8.4ms | fwd: 3576.4ms | loss: 13.1ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.8ms step 1429 | loss: 2.741542 | lr 1.8000e-04 | norm: 0.2377 | dt: 14477.89ms | tok/sec: 36213.0 | mem: 107563MB | Time Left: 03 hrs : 12 mins [TIMING] data: 8.3ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10843.1ms | clip: 0.0ms | optim: 23.8ms step 1430 | loss: 2.611301 | lr 1.8000e-04 | norm: 0.1904 | dt: 14485.36ms | tok/sec: 36194.3 | mem: 107563MB | Time Left: 03 hrs : 12 mins [TIMING] data: 8.5ms | fwd: 3575.5ms | loss: 13.1ms | bwd: 10850.3ms | clip: 0.0ms | optim: 23.9ms step 1431 | loss: 2.807518 | lr 1.8000e-04 | norm: 0.2157 | dt: 14482.51ms | tok/sec: 36201.5 | mem: 107563MB | Time Left: 03 hrs : 12 mins [TIMING] data: 7.8ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10846.2ms | clip: 0.0ms | optim: 23.8ms step 1432 | loss: 3.039720 | lr 1.8000e-04 | norm: 0.2971 | dt: 14481.02ms | tok/sec: 36205.2 | mem: 107563MB | Time Left: 03 hrs : 11 mins [TIMING] data: 8.2ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10845.3ms | clip: 0.0ms | optim: 24.0ms step 1433 | loss: 2.712802 | lr 1.8000e-04 | norm: 0.2381 | dt: 14476.32ms | tok/sec: 36216.9 | mem: 107563MB | Time Left: 03 hrs : 11 mins [TIMING] data: 8.2ms | fwd: 3577.2ms | loss: 13.2ms | bwd: 10840.2ms | clip: 0.0ms | optim: 23.8ms step 1434 | loss: 2.837031 | lr 1.8000e-04 | norm: 0.2098 | dt: 14488.37ms | tok/sec: 36186.8 | mem: 107563MB | Time Left: 03 hrs : 11 mins [TIMING] data: 8.4ms | fwd: 3574.4ms | loss: 13.1ms | bwd: 10854.4ms | clip: 0.0ms | optim: 23.9ms step 1435 | loss: 3.059385 | lr 1.8000e-04 | norm: 0.1907 | dt: 14489.42ms | tok/sec: 36184.2 | mem: 107563MB | Time Left: 03 hrs : 11 mins [TIMING] data: 8.4ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10853.6ms | clip: 0.0ms | optim: 23.9ms step 1436 | loss: 2.744774 | lr 1.8000e-04 | norm: 0.1960 | dt: 14479.74ms | tok/sec: 36208.4 | mem: 107563MB | Time Left: 03 hrs : 10 mins [TIMING] data: 8.3ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10841.5ms | clip: 0.0ms | optim: 23.8ms step 1437 | loss: 2.749815 | lr 1.8000e-04 | norm: 0.2777 | dt: 14484.88ms | tok/sec: 36195.5 | mem: 107563MB | Time Left: 03 hrs : 10 mins [TIMING] data: 8.1ms | fwd: 3585.5ms | loss: 13.1ms | bwd: 10841.0ms | clip: 0.0ms | optim: 23.8ms step 1438 | loss: 2.748949 | lr 1.8000e-04 | norm: 0.2570 | dt: 14487.23ms | tok/sec: 36189.7 | mem: 107563MB | Time Left: 03 hrs : 10 mins [TIMING] data: 8.4ms | fwd: 3574.3ms | loss: 13.1ms | bwd: 10854.3ms | clip: 0.0ms | optim: 23.7ms step 1439 | loss: 2.814694 | lr 1.8000e-04 | norm: 0.2366 | dt: 14478.45ms | tok/sec: 36211.6 | mem: 107563MB | Time Left: 03 hrs : 10 mins [TIMING] data: 8.4ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10841.7ms | clip: 0.0ms | optim: 23.7ms step 1440 | loss: 2.923324 | lr 1.8000e-04 | norm: 0.2093 | dt: 14477.64ms | tok/sec: 36213.6 | mem: 107563MB | Time Left: 03 hrs : 09 mins [TIMING] data: 8.2ms | fwd: 3574.4ms | loss: 13.1ms | bwd: 10844.6ms | clip: 0.0ms | optim: 23.7ms step 1441 | loss: 2.616450 | lr 1.8000e-04 | norm: 0.1586 | dt: 14482.64ms | tok/sec: 36201.1 | mem: 107563MB | Time Left: 03 hrs : 09 mins [TIMING] data: 8.4ms | fwd: 3577.0ms | loss: 13.1ms | bwd: 10846.4ms | clip: 0.0ms | optim: 23.8ms step 1442 | loss: 2.592801 | lr 1.8000e-04 | norm: 0.2030 | dt: 14488.90ms | tok/sec: 36185.5 | mem: 107563MB | Time Left: 03 hrs : 09 mins [TIMING] data: 8.3ms | fwd: 3574.2ms | loss: 13.1ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.8ms step 1443 | loss: 2.819068 | lr 1.8000e-04 | norm: 0.2292 | dt: 14483.36ms | tok/sec: 36199.3 | mem: 107563MB | Time Left: 03 hrs : 09 mins [TIMING] data: 8.4ms | fwd: 3581.0ms | loss: 13.0ms | bwd: 10843.7ms | clip: 0.0ms | optim: 23.9ms step 1444 | loss: 2.677975 | lr 1.8000e-04 | norm: 0.1804 | dt: 14481.62ms | tok/sec: 36203.7 | mem: 107563MB | Time Left: 03 hrs : 08 mins [TIMING] data: 9.5ms | fwd: 3574.3ms | loss: 13.1ms | bwd: 10847.3ms | clip: 0.0ms | optim: 23.8ms step 1445 | loss: 2.632776 | lr 1.8000e-04 | norm: 0.1824 | dt: 14479.38ms | tok/sec: 36209.3 | mem: 107563MB | Time Left: 03 hrs : 08 mins [TIMING] data: 8.5ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10845.8ms | clip: 0.0ms | optim: 23.8ms step 1446 | loss: 2.829960 | lr 1.8000e-04 | norm: 0.2059 | dt: 14485.96ms | tok/sec: 36192.8 | mem: 107563MB | Time Left: 03 hrs : 08 mins [TIMING] data: 8.5ms | fwd: 3579.7ms | loss: 13.0ms | bwd: 10847.5ms | clip: 0.0ms | optim: 23.9ms step 1447 | loss: 2.888546 | lr 1.8000e-04 | norm: 0.1857 | dt: 14476.30ms | tok/sec: 36217.0 | mem: 107563MB | Time Left: 03 hrs : 08 mins [TIMING] data: 8.5ms | fwd: 3574.1ms | loss: 13.1ms | bwd: 10843.4ms | clip: 0.0ms | optim: 23.7ms step 1448 | loss: 2.872638 | lr 1.8000e-04 | norm: 0.1745 | dt: 14478.55ms | tok/sec: 36211.4 | mem: 107563MB | Time Left: 03 hrs : 07 mins [TIMING] data: 8.4ms | fwd: 3577.2ms | loss: 13.0ms | bwd: 10842.6ms | clip: 0.0ms | optim: 23.8ms step 1449 | loss: 2.893196 | lr 1.8000e-04 | norm: 0.1931 | dt: 14481.57ms | tok/sec: 36203.8 | mem: 107563MB | Time Left: 03 hrs : 07 mins [TIMING] data: 8.3ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10844.0ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1450.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1350.ckpt step 1450 | loss: 3.058969 | lr 1.8000e-04 | norm: 0.1791 | dt: 14382.42ms | tok/sec: 36453.4 | mem: 107563MB | Time Left: 03 hrs : 06 mins [TIMING] data: 9.2ms | fwd: 3570.6ms | loss: 13.2ms | bwd: 10751.5ms | clip: 0.0ms | optim: 23.8ms step 1451 | loss: 2.604241 | lr 1.8000e-04 | norm: 0.2128 | dt: 14413.71ms | tok/sec: 36374.3 | mem: 107563MB | Time Left: 03 hrs : 06 mins [TIMING] data: 8.4ms | fwd: 3570.6ms | loss: 13.1ms | bwd: 10783.3ms | clip: 0.0ms | optim: 23.9ms step 1452 | loss: 2.673146 | lr 1.8000e-04 | norm: 0.2192 | dt: 14429.27ms | tok/sec: 36335.0 | mem: 107563MB | Time Left: 03 hrs : 06 mins [TIMING] data: 8.5ms | fwd: 3569.9ms | loss: 13.1ms | bwd: 10800.6ms | clip: 0.0ms | optim: 23.7ms step 1453 | loss: 2.796067 | lr 1.8000e-04 | norm: 0.1922 | dt: 14457.08ms | tok/sec: 36265.1 | mem: 107563MB | Time Left: 03 hrs : 06 mins [TIMING] data: 8.4ms | fwd: 3574.9ms | loss: 13.1ms | bwd: 10823.6ms | clip: 0.0ms | optim: 23.7ms step 1454 | loss: 2.961383 | lr 1.8000e-04 | norm: 0.1999 | dt: 14466.14ms | tok/sec: 36242.4 | mem: 107563MB | Time Left: 03 hrs : 06 mins [TIMING] data: 8.5ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10828.6ms | clip: 0.0ms | optim: 23.8ms step 1455 | loss: 2.888211 | lr 1.8000e-04 | norm: 0.2070 | dt: 14468.29ms | tok/sec: 36237.0 | mem: 107563MB | Time Left: 03 hrs : 06 mins [TIMING] data: 8.4ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10829.6ms | clip: 0.0ms | optim: 23.9ms step 1456 | loss: 2.690465 | lr 1.8000e-04 | norm: 0.2044 | dt: 14478.41ms | tok/sec: 36211.7 | mem: 107563MB | Time Left: 03 hrs : 06 mins [TIMING] data: 8.2ms | fwd: 3570.8ms | loss: 13.1ms | bwd: 10849.0ms | clip: 0.0ms | optim: 23.9ms step 1457 | loss: 2.599324 | lr 1.8000e-04 | norm: 0.2180 | dt: 14477.27ms | tok/sec: 36214.6 | mem: 107563MB | Time Left: 03 hrs : 05 mins [TIMING] data: 8.5ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10841.3ms | clip: 0.0ms | optim: 23.8ms step 1458 | loss: 2.767655 | lr 1.8000e-04 | norm: 0.1665 | dt: 14482.66ms | tok/sec: 36201.1 | mem: 107563MB | Time Left: 03 hrs : 05 mins [TIMING] data: 8.4ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10845.5ms | clip: 0.0ms | optim: 23.8ms step 1459 | loss: 2.785476 | lr 1.8000e-04 | norm: 0.1965 | dt: 14477.91ms | tok/sec: 36213.0 | mem: 107563MB | Time Left: 03 hrs : 05 mins [TIMING] data: 8.3ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.9ms step 1460 | loss: 2.739569 | lr 1.8000e-04 | norm: 0.2135 | dt: 14475.47ms | tok/sec: 36219.1 | mem: 107563MB | Time Left: 03 hrs : 05 mins [TIMING] data: 8.5ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10838.9ms | clip: 0.0ms | optim: 23.8ms step 1461 | loss: 3.060820 | lr 1.8000e-04 | norm: 0.2445 | dt: 14483.46ms | tok/sec: 36199.1 | mem: 107563MB | Time Left: 03 hrs : 04 mins [TIMING] data: 8.4ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10845.0ms | clip: 0.0ms | optim: 23.8ms step 1462 | loss: 3.020797 | lr 1.8000e-04 | norm: 0.2463 | dt: 14493.85ms | tok/sec: 36173.1 | mem: 107563MB | Time Left: 03 hrs : 04 mins [TIMING] data: 8.4ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10852.1ms | clip: 0.0ms | optim: 23.8ms step 1463 | loss: 2.921416 | lr 1.8000e-04 | norm: 0.2131 | dt: 14495.47ms | tok/sec: 36169.1 | mem: 107563MB | Time Left: 03 hrs : 04 mins [TIMING] data: 8.0ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10855.6ms | clip: 0.0ms | optim: 23.8ms step 1464 | loss: 2.995376 | lr 1.8000e-04 | norm: 0.2497 | dt: 14485.56ms | tok/sec: 36193.8 | mem: 107563MB | Time Left: 03 hrs : 04 mins [TIMING] data: 8.5ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10848.3ms | clip: 0.0ms | optim: 23.8ms step 1465 | loss: 2.815321 | lr 1.8000e-04 | norm: 0.1750 | dt: 14494.30ms | tok/sec: 36172.0 | mem: 107563MB | Time Left: 03 hrs : 04 mins [TIMING] data: 8.3ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.7ms step 1466 | loss: 2.708029 | lr 1.8000e-04 | norm: 0.1964 | dt: 14495.55ms | tok/sec: 36168.9 | mem: 107563MB | Time Left: 03 hrs : 03 mins [TIMING] data: 8.4ms | fwd: 3576.9ms | loss: 13.0ms | bwd: 10860.0ms | clip: 0.0ms | optim: 23.8ms step 1467 | loss: 2.732739 | lr 1.8000e-04 | norm: 0.2107 | dt: 14488.23ms | tok/sec: 36187.2 | mem: 107563MB | Time Left: 03 hrs : 03 mins [TIMING] data: 8.3ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10851.1ms | clip: 0.0ms | optim: 23.8ms step 1468 | loss: 2.817130 | lr 1.8000e-04 | norm: 0.1732 | dt: 14483.00ms | tok/sec: 36200.2 | mem: 107563MB | Time Left: 03 hrs : 03 mins [TIMING] data: 8.4ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10841.6ms | clip: 0.0ms | optim: 24.0ms step 1469 | loss: 3.029609 | lr 1.8000e-04 | norm: 0.2542 | dt: 14490.24ms | tok/sec: 36182.2 | mem: 107563MB | Time Left: 03 hrs : 03 mins [TIMING] data: 8.2ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10849.2ms | clip: 0.0ms | optim: 23.7ms step 1470 | loss: 3.190994 | lr 1.8000e-04 | norm: 0.2025 | dt: 14473.79ms | tok/sec: 36223.3 | mem: 107563MB | Time Left: 03 hrs : 02 mins [TIMING] data: 8.4ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10840.2ms | clip: 0.0ms | optim: 23.9ms step 1471 | loss: 2.701936 | lr 1.8000e-04 | norm: 0.1900 | dt: 14491.69ms | tok/sec: 36178.5 | mem: 107563MB | Time Left: 03 hrs : 02 mins [TIMING] data: 8.5ms | fwd: 3573.1ms | loss: 13.0ms | bwd: 10859.7ms | clip: 0.0ms | optim: 23.9ms step 1472 | loss: 3.100653 | lr 1.8000e-04 | norm: 0.1993 | dt: 14480.42ms | tok/sec: 36206.7 | mem: 107563MB | Time Left: 03 hrs : 02 mins [TIMING] data: 8.4ms | fwd: 3583.5ms | loss: 13.0ms | bwd: 10838.3ms | clip: 0.0ms | optim: 23.9ms step 1473 | loss: 2.557518 | lr 1.8000e-04 | norm: 0.6320 | dt: 14487.87ms | tok/sec: 36188.1 | mem: 107563MB | Time Left: 03 hrs : 02 mins [TIMING] data: 8.3ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10853.2ms | clip: 0.0ms | optim: 23.9ms step 1474 | loss: 2.877797 | lr 1.8000e-04 | norm: 0.2210 | dt: 14484.11ms | tok/sec: 36197.5 | mem: 107563MB | Time Left: 03 hrs : 01 mins [TIMING] data: 8.3ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10842.5ms | clip: 0.0ms | optim: 23.8ms step 1475 | loss: 2.920361 | lr 1.8000e-04 | norm: 0.1968 | dt: 14474.90ms | tok/sec: 36220.5 | mem: 107563MB | Time Left: 03 hrs : 01 mins [TIMING] data: 8.3ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10838.7ms | clip: 0.0ms | optim: 23.8ms step 1476 | loss: 2.947962 | lr 1.8000e-04 | norm: 0.2145 | dt: 14484.62ms | tok/sec: 36196.2 | mem: 107563MB | Time Left: 03 hrs : 01 mins [TIMING] data: 8.3ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10844.0ms | clip: 0.0ms | optim: 23.8ms step 1477 | loss: 3.316959 | lr 1.8000e-04 | norm: 0.2288 | dt: 14496.42ms | tok/sec: 36166.7 | mem: 107563MB | Time Left: 03 hrs : 01 mins [TIMING] data: 8.4ms | fwd: 3582.7ms | loss: 13.0ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.9ms step 1478 | loss: 2.757006 | lr 1.8000e-04 | norm: 0.1772 | dt: 14492.78ms | tok/sec: 36175.8 | mem: 107563MB | Time Left: 03 hrs : 00 mins [TIMING] data: 8.2ms | fwd: 3575.1ms | loss: 13.0ms | bwd: 10859.1ms | clip: 0.0ms | optim: 23.9ms step 1479 | loss: 2.753499 | lr 1.8000e-04 | norm: 0.1523 | dt: 14498.60ms | tok/sec: 36161.3 | mem: 107563MB | Time Left: 03 hrs : 00 mins [TIMING] data: 8.4ms | fwd: 3579.1ms | loss: 13.0ms | bwd: 10860.6ms | clip: 0.0ms | optim: 23.9ms step 1480 | loss: 2.996409 | lr 1.8000e-04 | norm: 0.2170 | dt: 14488.65ms | tok/sec: 36186.1 | mem: 107563MB | Time Left: 03 hrs : 00 mins [TIMING] data: 8.5ms | fwd: 3582.8ms | loss: 13.0ms | bwd: 10847.2ms | clip: 0.0ms | optim: 23.9ms step 1481 | loss: 2.668821 | lr 1.8000e-04 | norm: 0.2016 | dt: 14491.35ms | tok/sec: 36179.4 | mem: 107563MB | Time Left: 03 hrs : 00 mins [TIMING] data: 8.3ms | fwd: 3581.8ms | loss: 13.0ms | bwd: 10851.1ms | clip: 0.0ms | optim: 23.8ms step 1482 | loss: 2.881230 | lr 1.8000e-04 | norm: 0.2679 | dt: 14482.16ms | tok/sec: 36202.3 | mem: 107563MB | Time Left: 02 hrs : 59 mins [TIMING] data: 8.3ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10841.1ms | clip: 0.0ms | optim: 23.8ms step 1483 | loss: 2.659241 | lr 1.8000e-04 | norm: 0.1656 | dt: 14477.61ms | tok/sec: 36213.7 | mem: 107563MB | Time Left: 02 hrs : 59 mins [TIMING] data: 8.4ms | fwd: 3577.6ms | loss: 13.0ms | bwd: 10840.6ms | clip: 0.0ms | optim: 23.8ms step 1484 | loss: 3.140846 | lr 1.8000e-04 | norm: 0.1979 | dt: 14480.38ms | tok/sec: 36206.8 | mem: 107563MB | Time Left: 02 hrs : 59 mins [TIMING] data: 8.3ms | fwd: 3576.5ms | loss: 13.1ms | bwd: 10845.1ms | clip: 0.0ms | optim: 23.8ms step 1485 | loss: 2.907403 | lr 1.8000e-04 | norm: 0.2042 | dt: 14483.03ms | tok/sec: 36200.2 | mem: 107563MB | Time Left: 02 hrs : 59 mins [TIMING] data: 8.3ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10846.1ms | clip: 0.0ms | optim: 23.8ms step 1486 | loss: 2.795409 | lr 1.8000e-04 | norm: 0.2533 | dt: 14495.73ms | tok/sec: 36168.5 | mem: 107563MB | Time Left: 02 hrs : 59 mins [TIMING] data: 8.5ms | fwd: 3578.1ms | loss: 13.0ms | bwd: 10858.9ms | clip: 0.0ms | optim: 23.9ms step 1487 | loss: 3.050099 | lr 1.8000e-04 | norm: 0.2429 | dt: 14489.72ms | tok/sec: 36183.4 | mem: 107563MB | Time Left: 02 hrs : 58 mins [TIMING] data: 8.2ms | fwd: 3578.5ms | loss: 13.0ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.8ms step 1488 | loss: 2.878339 | lr 1.8000e-04 | norm: 0.1857 | dt: 14487.43ms | tok/sec: 36189.2 | mem: 107563MB | Time Left: 02 hrs : 58 mins [TIMING] data: 8.4ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10851.2ms | clip: 0.0ms | optim: 23.9ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00008.bin 100000000 step 1489 | loss: 2.813638 | lr 1.8000e-04 | norm: 0.1713 | dt: 14492.62ms | tok/sec: 36176.2 | mem: 107563MB | Time Left: 02 hrs : 58 mins [TIMING] data: 13.3ms | fwd: 3581.5ms | loss: 13.0ms | bwd: 10847.6ms | clip: 0.0ms | optim: 23.8ms step 1490 | loss: 3.093160 | lr 1.8000e-04 | norm: 0.1964 | dt: 14489.90ms | tok/sec: 36183.0 | mem: 107563MB | Time Left: 02 hrs : 57 mins [TIMING] data: 8.5ms | fwd: 3581.1ms | loss: 13.0ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.8ms step 1491 | loss: 2.489742 | lr 1.8000e-04 | norm: 0.1801 | dt: 14489.55ms | tok/sec: 36183.9 | mem: 107563MB | Time Left: 02 hrs : 57 mins [TIMING] data: 8.3ms | fwd: 3582.7ms | loss: 13.0ms | bwd: 10848.4ms | clip: 0.0ms | optim: 23.8ms step 1492 | loss: 3.035953 | lr 1.8000e-04 | norm: 0.1995 | dt: 14482.69ms | tok/sec: 36201.0 | mem: 107563MB | Time Left: 02 hrs : 57 mins [TIMING] data: 8.2ms | fwd: 3579.0ms | loss: 13.0ms | bwd: 10845.5ms | clip: 0.0ms | optim: 23.8ms step 1493 | loss: 2.819530 | lr 1.8000e-04 | norm: 0.1802 | dt: 14484.18ms | tok/sec: 36197.3 | mem: 107563MB | Time Left: 02 hrs : 57 mins [TIMING] data: 8.1ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10844.0ms | clip: 0.0ms | optim: 23.8ms step 1494 | loss: 3.181839 | lr 1.8000e-04 | norm: 0.2279 | dt: 14485.94ms | tok/sec: 36192.9 | mem: 107563MB | Time Left: 02 hrs : 56 mins [TIMING] data: 8.4ms | fwd: 3582.4ms | loss: 13.0ms | bwd: 10844.6ms | clip: 0.0ms | optim: 24.0ms step 1495 | loss: 2.833385 | lr 1.8000e-04 | norm: 0.1851 | dt: 14479.71ms | tok/sec: 36208.5 | mem: 107563MB | Time Left: 02 hrs : 56 mins [TIMING] data: 8.4ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10837.7ms | clip: 0.0ms | optim: 23.8ms step 1496 | loss: 2.832898 | lr 1.8000e-04 | norm: 0.1768 | dt: 14492.04ms | tok/sec: 36177.7 | mem: 107563MB | Time Left: 02 hrs : 56 mins [TIMING] data: 8.3ms | fwd: 3579.7ms | loss: 13.0ms | bwd: 10853.5ms | clip: 0.0ms | optim: 23.9ms step 1497 | loss: 3.003542 | lr 1.8000e-04 | norm: 0.1929 | dt: 14476.89ms | tok/sec: 36215.5 | mem: 107563MB | Time Left: 02 hrs : 56 mins [TIMING] data: 8.4ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10836.1ms | clip: 0.0ms | optim: 23.8ms step 1498 | loss: 3.075266 | lr 1.8000e-04 | norm: 0.1828 | dt: 14484.01ms | tok/sec: 36197.7 | mem: 107563MB | Time Left: 02 hrs : 55 mins [TIMING] data: 8.4ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10842.1ms | clip: 0.0ms | optim: 23.8ms step 1499 | loss: 3.083647 | lr 1.8000e-04 | norm: 0.1849 | dt: 14488.91ms | tok/sec: 36185.5 | mem: 107563MB | Time Left: 02 hrs : 55 mins [TIMING] data: 8.3ms | fwd: 3576.4ms | loss: 13.0ms | bwd: 10854.0ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1500.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1400.ckpt validation loss: 3.0375 step 1500 | loss: 3.058187 | lr 1.8000e-04 | norm: 0.1794 | dt: 18970.06ms | tok/sec: 27637.6 | mem: 107563MB | Time Left: 03 hrs : 49 mins [TIMING] data: 8.8ms | fwd: 3559.1ms | loss: 13.1ms | bwd: 10797.2ms | clip: 0.0ms | optim: 23.8ms step 1501 | loss: 2.869549 | lr 1.8000e-04 | norm: 0.2277 | dt: 14419.33ms | tok/sec: 36360.1 | mem: 107563MB | Time Left: 02 hrs : 54 mins [TIMING] data: 8.4ms | fwd: 3567.4ms | loss: 13.1ms | bwd: 10793.1ms | clip: 0.0ms | optim: 23.8ms step 1502 | loss: 2.672094 | lr 1.8000e-04 | norm: 0.2406 | dt: 14449.77ms | tok/sec: 36283.5 | mem: 107563MB | Time Left: 02 hrs : 54 mins [TIMING] data: 8.5ms | fwd: 3575.0ms | loss: 13.1ms | bwd: 10815.6ms | clip: 0.0ms | optim: 23.8ms step 1503 | loss: 2.639249 | lr 1.8000e-04 | norm: 0.1813 | dt: 14451.90ms | tok/sec: 36278.1 | mem: 107563MB | Time Left: 02 hrs : 54 mins [TIMING] data: 8.2ms | fwd: 3570.2ms | loss: 13.1ms | bwd: 10823.0ms | clip: 0.0ms | optim: 23.8ms step 1504 | loss: 2.906139 | lr 1.8000e-04 | norm: 0.2449 | dt: 14469.68ms | tok/sec: 36233.5 | mem: 107563MB | Time Left: 02 hrs : 54 mins [TIMING] data: 8.5ms | fwd: 3575.6ms | loss: 13.0ms | bwd: 10835.4ms | clip: 0.0ms | optim: 23.8ms step 1505 | loss: 2.592303 | lr 1.8000e-04 | norm: 0.2577 | dt: 14471.58ms | tok/sec: 36228.8 | mem: 107563MB | Time Left: 02 hrs : 54 mins [TIMING] data: 8.3ms | fwd: 3574.6ms | loss: 13.1ms | bwd: 10838.3ms | clip: 0.0ms | optim: 23.9ms step 1506 | loss: 3.041688 | lr 1.8000e-04 | norm: 0.2309 | dt: 14493.38ms | tok/sec: 36174.3 | mem: 107563MB | Time Left: 02 hrs : 54 mins [TIMING] data: 8.6ms | fwd: 3575.3ms | loss: 13.0ms | bwd: 10858.3ms | clip: 0.0ms | optim: 24.6ms step 1507 | loss: 2.653637 | lr 1.8000e-04 | norm: 0.2193 | dt: 14498.30ms | tok/sec: 36162.0 | mem: 107563MB | Time Left: 02 hrs : 53 mins [TIMING] data: 8.2ms | fwd: 3577.4ms | loss: 13.0ms | bwd: 10862.4ms | clip: 0.0ms | optim: 23.9ms step 1508 | loss: 3.021899 | lr 1.8000e-04 | norm: 0.2164 | dt: 14498.68ms | tok/sec: 36161.1 | mem: 107563MB | Time Left: 02 hrs : 53 mins [TIMING] data: 8.2ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10860.0ms | clip: 0.0ms | optim: 23.9ms step 1509 | loss: 2.977929 | lr 1.8000e-04 | norm: 0.2138 | dt: 14502.66ms | tok/sec: 36151.2 | mem: 107563MB | Time Left: 02 hrs : 53 mins [TIMING] data: 8.2ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10861.5ms | clip: 0.0ms | optim: 23.9ms step 1510 | loss: 3.096949 | lr 1.8000e-04 | norm: 0.2260 | dt: 14504.69ms | tok/sec: 36146.1 | mem: 107563MB | Time Left: 02 hrs : 53 mins [TIMING] data: 8.1ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 23.9ms step 1511 | loss: 3.032793 | lr 1.8000e-04 | norm: 0.1687 | dt: 14513.15ms | tok/sec: 36125.0 | mem: 107563MB | Time Left: 02 hrs : 53 mins [TIMING] data: 8.2ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10872.1ms | clip: 0.0ms | optim: 23.8ms step 1512 | loss: 2.746041 | lr 1.8000e-04 | norm: 1.3966 | dt: 14506.92ms | tok/sec: 36140.6 | mem: 107563MB | Time Left: 02 hrs : 52 mins [TIMING] data: 8.4ms | fwd: 3585.2ms | loss: 13.0ms | bwd: 10863.0ms | clip: 0.0ms | optim: 23.8ms step 1513 | loss: 3.076527 | lr 1.8000e-04 | norm: 0.2012 | dt: 14530.80ms | tok/sec: 36081.2 | mem: 107563MB | Time Left: 02 hrs : 52 mins [TIMING] data: 8.3ms | fwd: 3583.0ms | loss: 13.0ms | bwd: 10889.3ms | clip: 0.0ms | optim: 23.8ms step 1514 | loss: 3.076962 | lr 1.8000e-04 | norm: 0.1947 | dt: 14529.89ms | tok/sec: 36083.4 | mem: 107563MB | Time Left: 02 hrs : 52 mins [TIMING] data: 8.4ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10889.1ms | clip: 0.0ms | optim: 23.8ms step 1515 | loss: 2.701344 | lr 1.8000e-04 | norm: 0.4235 | dt: 14516.58ms | tok/sec: 36116.5 | mem: 107563MB | Time Left: 02 hrs : 52 mins [TIMING] data: 8.8ms | fwd: 3580.3ms | loss: 13.0ms | bwd: 10877.2ms | clip: 0.0ms | optim: 23.9ms step 1516 | loss: 2.934653 | lr 1.8000e-04 | norm: 0.1849 | dt: 14523.34ms | tok/sec: 36099.7 | mem: 107563MB | Time Left: 02 hrs : 52 mins [TIMING] data: 8.1ms | fwd: 3587.6ms | loss: 13.1ms | bwd: 10877.3ms | clip: 0.0ms | optim: 23.8ms step 1517 | loss: 3.129428 | lr 1.8000e-04 | norm: 0.2152 | dt: 14516.28ms | tok/sec: 36117.2 | mem: 107563MB | Time Left: 02 hrs : 51 mins [TIMING] data: 8.5ms | fwd: 3583.5ms | loss: 13.1ms | bwd: 10873.8ms | clip: 0.0ms | optim: 23.8ms step 1518 | loss: 2.706079 | lr 1.8000e-04 | norm: 0.1901 | dt: 14529.54ms | tok/sec: 36084.3 | mem: 107563MB | Time Left: 02 hrs : 51 mins [TIMING] data: 8.4ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10888.6ms | clip: 0.0ms | optim: 23.8ms step 1519 | loss: 3.159923 | lr 1.8000e-04 | norm: 0.1804 | dt: 14527.27ms | tok/sec: 36089.9 | mem: 107563MB | Time Left: 02 hrs : 51 mins [TIMING] data: 8.4ms | fwd: 3586.9ms | loss: 13.1ms | bwd: 10881.7ms | clip: 0.0ms | optim: 23.8ms step 1520 | loss: 2.538540 | lr 1.8000e-04 | norm: 0.2315 | dt: 14545.76ms | tok/sec: 36044.1 | mem: 107563MB | Time Left: 02 hrs : 51 mins [TIMING] data: 8.4ms | fwd: 3588.4ms | loss: 13.1ms | bwd: 10898.7ms | clip: 0.0ms | optim: 23.7ms step 1521 | loss: 2.791801 | lr 1.8000e-04 | norm: 0.1822 | dt: 14529.59ms | tok/sec: 36084.2 | mem: 107563MB | Time Left: 02 hrs : 50 mins [TIMING] data: 8.3ms | fwd: 3585.1ms | loss: 13.0ms | bwd: 10886.0ms | clip: 0.0ms | optim: 23.8ms step 1522 | loss: 3.000250 | lr 1.8000e-04 | norm: 0.2126 | dt: 14525.41ms | tok/sec: 36094.5 | mem: 107563MB | Time Left: 02 hrs : 50 mins [TIMING] data: 8.3ms | fwd: 3586.9ms | loss: 13.1ms | bwd: 10879.6ms | clip: 0.0ms | optim: 23.9ms step 1523 | loss: 2.795613 | lr 1.8000e-04 | norm: 0.1730 | dt: 14536.19ms | tok/sec: 36067.8 | mem: 107563MB | Time Left: 02 hrs : 50 mins [TIMING] data: 8.3ms | fwd: 3590.5ms | loss: 13.0ms | bwd: 10887.2ms | clip: 0.0ms | optim: 23.8ms step 1524 | loss: 2.845311 | lr 1.8000e-04 | norm: 0.2161 | dt: 14529.05ms | tok/sec: 36085.5 | mem: 107563MB | Time Left: 02 hrs : 50 mins [TIMING] data: 8.4ms | fwd: 3582.6ms | loss: 13.0ms | bwd: 10887.8ms | clip: 0.0ms | optim: 23.8ms step 1525 | loss: 2.963738 | lr 1.8000e-04 | norm: 0.1970 | dt: 14533.61ms | tok/sec: 36074.2 | mem: 107563MB | Time Left: 02 hrs : 50 mins [TIMING] data: 8.3ms | fwd: 3590.1ms | loss: 13.1ms | bwd: 10884.7ms | clip: 0.0ms | optim: 23.9ms step 1526 | loss: 2.733229 | lr 1.8000e-04 | norm: 0.1724 | dt: 14532.72ms | tok/sec: 36076.4 | mem: 107563MB | Time Left: 02 hrs : 49 mins [TIMING] data: 8.5ms | fwd: 3584.7ms | loss: 13.0ms | bwd: 10889.4ms | clip: 0.0ms | optim: 23.7ms step 1527 | loss: 2.647151 | lr 1.8000e-04 | norm: 0.1865 | dt: 14529.50ms | tok/sec: 36084.4 | mem: 107563MB | Time Left: 02 hrs : 49 mins [TIMING] data: 8.3ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10888.1ms | clip: 0.0ms | optim: 23.8ms step 1528 | loss: 2.939662 | lr 1.8000e-04 | norm: 0.1963 | dt: 14521.99ms | tok/sec: 36103.0 | mem: 107563MB | Time Left: 02 hrs : 49 mins [TIMING] data: 8.4ms | fwd: 3581.5ms | loss: 13.0ms | bwd: 10881.8ms | clip: 0.0ms | optim: 23.9ms step 1529 | loss: 2.904649 | lr 1.8000e-04 | norm: 0.1784 | dt: 14528.34ms | tok/sec: 36087.2 | mem: 107563MB | Time Left: 02 hrs : 49 mins [TIMING] data: 8.2ms | fwd: 3588.2ms | loss: 13.0ms | bwd: 10880.9ms | clip: 0.0ms | optim: 24.7ms step 1530 | loss: 2.868646 | lr 1.8000e-04 | norm: 0.1806 | dt: 14523.87ms | tok/sec: 36098.4 | mem: 107563MB | Time Left: 02 hrs : 48 mins [TIMING] data: 8.2ms | fwd: 3587.3ms | loss: 13.0ms | bwd: 10878.1ms | clip: 0.0ms | optim: 23.8ms step 1531 | loss: 2.809867 | lr 1.8000e-04 | norm: 0.1926 | dt: 14525.78ms | tok/sec: 36093.6 | mem: 107563MB | Time Left: 02 hrs : 48 mins [TIMING] data: 8.5ms | fwd: 3585.8ms | loss: 13.0ms | bwd: 10881.3ms | clip: 0.0ms | optim: 23.8ms step 1532 | loss: 2.825644 | lr 1.8000e-04 | norm: 0.2057 | dt: 14538.91ms | tok/sec: 36061.0 | mem: 107563MB | Time Left: 02 hrs : 48 mins [TIMING] data: 8.4ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10897.3ms | clip: 0.0ms | optim: 23.9ms step 1533 | loss: 2.657138 | lr 1.8000e-04 | norm: 0.1693 | dt: 14533.86ms | tok/sec: 36073.6 | mem: 107563MB | Time Left: 02 hrs : 48 mins [TIMING] data: 8.3ms | fwd: 3588.0ms | loss: 13.0ms | bwd: 10887.3ms | clip: 0.0ms | optim: 23.9ms step 1534 | loss: 2.800470 | lr 1.8000e-04 | norm: 0.1796 | dt: 14534.79ms | tok/sec: 36071.2 | mem: 107563MB | Time Left: 02 hrs : 47 mins [TIMING] data: 8.3ms | fwd: 3585.0ms | loss: 13.0ms | bwd: 10890.8ms | clip: 0.0ms | optim: 23.9ms step 1535 | loss: 2.744492 | lr 1.8000e-04 | norm: 0.1872 | dt: 14527.49ms | tok/sec: 36089.4 | mem: 107563MB | Time Left: 02 hrs : 47 mins [TIMING] data: 8.3ms | fwd: 3581.5ms | loss: 13.0ms | bwd: 10887.2ms | clip: 0.0ms | optim: 23.8ms step 1536 | loss: 2.963080 | lr 1.8000e-04 | norm: 0.2135 | dt: 14520.08ms | tok/sec: 36107.8 | mem: 107563MB | Time Left: 02 hrs : 47 mins [TIMING] data: 8.3ms | fwd: 3588.7ms | loss: 13.0ms | bwd: 10872.9ms | clip: 0.0ms | optim: 23.8ms step 1537 | loss: 2.742280 | lr 1.8000e-04 | norm: 0.1704 | dt: 14531.31ms | tok/sec: 36079.9 | mem: 107563MB | Time Left: 02 hrs : 47 mins [TIMING] data: 8.4ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10889.9ms | clip: 0.0ms | optim: 23.9ms step 1538 | loss: 3.184229 | lr 1.8000e-04 | norm: 0.1753 | dt: 14525.15ms | tok/sec: 36095.2 | mem: 107563MB | Time Left: 02 hrs : 46 mins [TIMING] data: 8.4ms | fwd: 3583.5ms | loss: 13.1ms | bwd: 10882.9ms | clip: 0.0ms | optim: 23.9ms step 1539 | loss: 2.685774 | lr 1.8000e-04 | norm: 0.1755 | dt: 14531.39ms | tok/sec: 36079.7 | mem: 107563MB | Time Left: 02 hrs : 46 mins [TIMING] data: 8.4ms | fwd: 3584.6ms | loss: 13.1ms | bwd: 10888.1ms | clip: 0.0ms | optim: 23.9ms step 1540 | loss: 2.684020 | lr 1.8000e-04 | norm: 0.1679 | dt: 14526.58ms | tok/sec: 36091.6 | mem: 107563MB | Time Left: 02 hrs : 46 mins [TIMING] data: 8.2ms | fwd: 3584.9ms | loss: 13.0ms | bwd: 10883.2ms | clip: 0.0ms | optim: 23.8ms step 1541 | loss: 2.866511 | lr 1.8000e-04 | norm: 0.1853 | dt: 14530.91ms | tok/sec: 36080.9 | mem: 107563MB | Time Left: 02 hrs : 46 mins [TIMING] data: 8.3ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10889.3ms | clip: 0.0ms | optim: 23.9ms step 1542 | loss: 2.665902 | lr 1.8000e-04 | norm: 0.2005 | dt: 14530.53ms | tok/sec: 36081.8 | mem: 107563MB | Time Left: 02 hrs : 45 mins [TIMING] data: 8.5ms | fwd: 3586.6ms | loss: 13.1ms | bwd: 10884.9ms | clip: 0.0ms | optim: 23.8ms step 1543 | loss: 3.022244 | lr 1.8000e-04 | norm: 0.2022 | dt: 14532.03ms | tok/sec: 36078.1 | mem: 107563MB | Time Left: 02 hrs : 45 mins [TIMING] data: 8.3ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10889.4ms | clip: 0.0ms | optim: 23.9ms step 1544 | loss: 2.691399 | lr 1.8000e-04 | norm: 0.1687 | dt: 14537.73ms | tok/sec: 36063.9 | mem: 107563MB | Time Left: 02 hrs : 45 mins [TIMING] data: 8.3ms | fwd: 3586.3ms | loss: 13.1ms | bwd: 10892.7ms | clip: 0.0ms | optim: 23.8ms step 1545 | loss: 2.930848 | lr 1.8000e-04 | norm: 0.2142 | dt: 14536.58ms | tok/sec: 36066.8 | mem: 107563MB | Time Left: 02 hrs : 45 mins [TIMING] data: 8.4ms | fwd: 3588.2ms | loss: 13.0ms | bwd: 10889.8ms | clip: 0.0ms | optim: 23.9ms step 1546 | loss: 2.937660 | lr 1.8000e-04 | norm: 0.2369 | dt: 14525.15ms | tok/sec: 36095.2 | mem: 107563MB | Time Left: 02 hrs : 44 mins [TIMING] data: 8.1ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10883.5ms | clip: 0.0ms | optim: 23.8ms step 1547 | loss: 2.780614 | lr 1.8000e-04 | norm: 0.2081 | dt: 14520.13ms | tok/sec: 36107.7 | mem: 107563MB | Time Left: 02 hrs : 44 mins [TIMING] data: 8.2ms | fwd: 3583.0ms | loss: 13.0ms | bwd: 10878.5ms | clip: 0.0ms | optim: 23.9ms step 1548 | loss: 2.822125 | lr 1.8000e-04 | norm: 0.1840 | dt: 14516.08ms | tok/sec: 36117.7 | mem: 107563MB | Time Left: 02 hrs : 44 mins [TIMING] data: 8.4ms | fwd: 3583.5ms | loss: 13.1ms | bwd: 10873.8ms | clip: 0.0ms | optim: 23.9ms step 1549 | loss: 2.767465 | lr 1.8000e-04 | norm: 0.2176 | dt: 14525.78ms | tok/sec: 36093.6 | mem: 107563MB | Time Left: 02 hrs : 44 mins [TIMING] data: 8.5ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10886.0ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1550.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1450.ckpt step 1550 | loss: 2.619229 | lr 1.8000e-04 | norm: 0.1534 | dt: 14410.71ms | tok/sec: 36381.8 | mem: 107563MB | Time Left: 02 hrs : 42 mins [TIMING] data: 9.6ms | fwd: 3571.6ms | loss: 13.2ms | bwd: 10777.9ms | clip: 0.0ms | optim: 23.9ms step 1551 | loss: 2.904991 | lr 1.8000e-04 | norm: 0.2299 | dt: 14423.28ms | tok/sec: 36350.1 | mem: 107563MB | Time Left: 02 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3571.4ms | loss: 13.0ms | bwd: 10793.2ms | clip: 0.0ms | optim: 23.9ms step 1552 | loss: 2.802724 | lr 1.8000e-04 | norm: 0.2956 | dt: 14449.31ms | tok/sec: 36284.6 | mem: 107563MB | Time Left: 02 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3568.3ms | loss: 13.1ms | bwd: 10822.2ms | clip: 0.0ms | optim: 23.8ms step 1553 | loss: 2.998712 | lr 1.8000e-04 | norm: 0.1685 | dt: 14454.22ms | tok/sec: 36272.3 | mem: 107563MB | Time Left: 02 hrs : 42 mins [TIMING] data: 8.4ms | fwd: 3575.2ms | loss: 13.0ms | bwd: 10820.5ms | clip: 0.0ms | optim: 23.8ms step 1554 | loss: 2.452790 | lr 1.8000e-04 | norm: 0.3095 | dt: 14470.34ms | tok/sec: 36231.9 | mem: 107563MB | Time Left: 02 hrs : 42 mins [TIMING] data: 8.3ms | fwd: 3572.2ms | loss: 13.1ms | bwd: 10839.4ms | clip: 0.0ms | optim: 23.9ms step 1555 | loss: 2.797144 | lr 1.8000e-04 | norm: 0.1873 | dt: 14476.85ms | tok/sec: 36215.6 | mem: 107563MB | Time Left: 02 hrs : 42 mins [TIMING] data: 8.7ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10834.8ms | clip: 0.0ms | optim: 23.9ms step 1556 | loss: 2.925484 | lr 1.8000e-04 | norm: 0.2157 | dt: 14473.37ms | tok/sec: 36224.3 | mem: 107563MB | Time Left: 02 hrs : 41 mins [TIMING] data: 8.3ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10835.8ms | clip: 0.0ms | optim: 23.9ms step 1557 | loss: 2.931056 | lr 1.8000e-04 | norm: 0.2154 | dt: 14490.71ms | tok/sec: 36181.0 | mem: 107563MB | Time Left: 02 hrs : 41 mins [TIMING] data: 8.4ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10855.3ms | clip: 0.0ms | optim: 23.9ms step 1558 | loss: 2.768024 | lr 1.8000e-04 | norm: 0.2606 | dt: 14483.45ms | tok/sec: 36199.1 | mem: 107563MB | Time Left: 02 hrs : 41 mins [TIMING] data: 8.4ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10844.4ms | clip: 0.0ms | optim: 24.1ms step 1559 | loss: 2.983724 | lr 1.8000e-04 | norm: 0.1940 | dt: 14496.65ms | tok/sec: 36166.2 | mem: 107563MB | Time Left: 02 hrs : 41 mins [TIMING] data: 8.5ms | fwd: 3584.0ms | loss: 13.1ms | bwd: 10853.7ms | clip: 0.0ms | optim: 23.8ms step 1560 | loss: 2.753392 | lr 1.8000e-04 | norm: 0.1767 | dt: 14492.90ms | tok/sec: 36175.5 | mem: 107563MB | Time Left: 02 hrs : 41 mins [TIMING] data: 8.3ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10854.4ms | clip: 0.0ms | optim: 23.8ms step 1561 | loss: 2.690513 | lr 1.8000e-04 | norm: 0.1725 | dt: 14492.18ms | tok/sec: 36177.3 | mem: 107563MB | Time Left: 02 hrs : 40 mins [TIMING] data: 8.5ms | fwd: 3576.5ms | loss: 13.0ms | bwd: 10857.0ms | clip: 0.0ms | optim: 23.8ms step 1562 | loss: 3.046535 | lr 1.8000e-04 | norm: 0.2741 | dt: 14484.11ms | tok/sec: 36197.5 | mem: 107563MB | Time Left: 02 hrs : 40 mins [TIMING] data: 8.5ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.8ms step 1563 | loss: 3.033268 | lr 1.8000e-04 | norm: 0.1904 | dt: 14495.34ms | tok/sec: 36169.4 | mem: 107563MB | Time Left: 02 hrs : 40 mins [TIMING] data: 8.4ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10859.0ms | clip: 0.0ms | optim: 23.9ms step 1564 | loss: 2.950675 | lr 1.8000e-04 | norm: 0.2130 | dt: 14490.84ms | tok/sec: 36180.6 | mem: 107563MB | Time Left: 02 hrs : 40 mins [TIMING] data: 8.5ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10851.4ms | clip: 0.0ms | optim: 23.9ms step 1565 | loss: 2.894982 | lr 1.8000e-04 | norm: 0.2146 | dt: 14499.15ms | tok/sec: 36159.9 | mem: 107563MB | Time Left: 02 hrs : 39 mins [TIMING] data: 8.3ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10856.9ms | clip: 0.0ms | optim: 23.9ms step 1566 | loss: 2.722707 | lr 1.8000e-04 | norm: 0.1738 | dt: 14489.91ms | tok/sec: 36183.0 | mem: 107563MB | Time Left: 02 hrs : 39 mins [TIMING] data: 8.3ms | fwd: 3580.7ms | loss: 13.0ms | bwd: 10850.8ms | clip: 0.0ms | optim: 23.7ms step 1567 | loss: 2.646653 | lr 1.8000e-04 | norm: 0.1962 | dt: 14486.71ms | tok/sec: 36191.0 | mem: 107563MB | Time Left: 02 hrs : 39 mins [TIMING] data: 8.3ms | fwd: 3572.8ms | loss: 13.0ms | bwd: 10855.3ms | clip: 0.0ms | optim: 23.9ms step 1568 | loss: 2.668487 | lr 1.8000e-04 | norm: 0.2531 | dt: 14492.32ms | tok/sec: 36177.0 | mem: 107563MB | Time Left: 02 hrs : 39 mins [TIMING] data: 8.5ms | fwd: 3576.0ms | loss: 13.0ms | bwd: 10857.4ms | clip: 0.0ms | optim: 23.9ms step 1569 | loss: 2.941680 | lr 1.8000e-04 | norm: 0.2023 | dt: 14487.81ms | tok/sec: 36188.2 | mem: 107563MB | Time Left: 02 hrs : 38 mins [TIMING] data: 8.4ms | fwd: 3576.4ms | loss: 13.0ms | bwd: 10852.8ms | clip: 0.0ms | optim: 23.9ms step 1570 | loss: 2.808310 | lr 1.8000e-04 | norm: 0.2029 | dt: 14491.31ms | tok/sec: 36179.5 | mem: 107563MB | Time Left: 02 hrs : 38 mins [TIMING] data: 8.2ms | fwd: 3577.7ms | loss: 13.2ms | bwd: 10854.0ms | clip: 0.0ms | optim: 23.8ms step 1571 | loss: 2.778223 | lr 1.8000e-04 | norm: 0.2290 | dt: 14483.36ms | tok/sec: 36199.3 | mem: 107563MB | Time Left: 02 hrs : 38 mins [TIMING] data: 8.4ms | fwd: 3572.2ms | loss: 13.1ms | bwd: 10852.5ms | clip: 0.0ms | optim: 23.7ms step 1572 | loss: 2.958487 | lr 1.8000e-04 | norm: 0.2013 | dt: 14480.02ms | tok/sec: 36207.7 | mem: 107563MB | Time Left: 02 hrs : 38 mins [TIMING] data: 8.3ms | fwd: 3579.8ms | loss: 13.0ms | bwd: 10841.6ms | clip: 0.0ms | optim: 23.8ms step 1573 | loss: 3.088838 | lr 1.8000e-04 | norm: 0.5351 | dt: 14480.72ms | tok/sec: 36205.9 | mem: 107563MB | Time Left: 02 hrs : 37 mins [TIMING] data: 8.4ms | fwd: 3574.3ms | loss: 13.0ms | bwd: 10847.9ms | clip: 0.0ms | optim: 23.7ms step 1574 | loss: 2.900544 | lr 1.8000e-04 | norm: 0.1826 | dt: 14485.51ms | tok/sec: 36194.0 | mem: 107563MB | Time Left: 02 hrs : 37 mins [TIMING] data: 8.4ms | fwd: 3575.0ms | loss: 13.0ms | bwd: 10851.8ms | clip: 0.0ms | optim: 23.9ms step 1575 | loss: 2.820397 | lr 1.8000e-04 | norm: 0.2118 | dt: 14490.25ms | tok/sec: 36182.1 | mem: 107563MB | Time Left: 02 hrs : 37 mins [TIMING] data: 8.4ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10854.0ms | clip: 0.0ms | optim: 23.9ms step 1576 | loss: 2.876695 | lr 1.8000e-04 | norm: 0.2124 | dt: 14472.42ms | tok/sec: 36226.7 | mem: 107563MB | Time Left: 02 hrs : 37 mins [TIMING] data: 8.6ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10835.3ms | clip: 0.0ms | optim: 23.9ms step 1577 | loss: 2.920840 | lr 1.8000e-04 | norm: 0.2075 | dt: 14481.31ms | tok/sec: 36204.4 | mem: 107563MB | Time Left: 02 hrs : 36 mins [TIMING] data: 8.1ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10844.3ms | clip: 0.0ms | optim: 24.1ms step 1578 | loss: 2.695863 | lr 1.8000e-04 | norm: 0.2582 | dt: 14467.32ms | tok/sec: 36239.5 | mem: 107563MB | Time Left: 02 hrs : 36 mins [TIMING] data: 8.3ms | fwd: 3573.9ms | loss: 13.1ms | bwd: 10835.0ms | clip: 0.0ms | optim: 23.7ms step 1579 | loss: 2.998101 | lr 1.8000e-04 | norm: 0.1918 | dt: 14458.11ms | tok/sec: 36262.5 | mem: 107563MB | Time Left: 02 hrs : 36 mins [TIMING] data: 8.4ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10821.4ms | clip: 0.0ms | optim: 23.8ms step 1580 | loss: 2.743140 | lr 1.8000e-04 | norm: 0.1675 | dt: 14472.61ms | tok/sec: 36226.2 | mem: 107563MB | Time Left: 02 hrs : 36 mins [TIMING] data: 8.4ms | fwd: 3574.7ms | loss: 13.1ms | bwd: 10839.2ms | clip: 0.0ms | optim: 23.8ms step 1581 | loss: 2.622073 | lr 1.8000e-04 | norm: 0.1765 | dt: 14465.78ms | tok/sec: 36243.3 | mem: 107563MB | Time Left: 02 hrs : 35 mins [TIMING] data: 8.4ms | fwd: 3575.6ms | loss: 13.1ms | bwd: 10831.4ms | clip: 0.0ms | optim: 23.8ms step 1582 | loss: 2.758998 | lr 1.8000e-04 | norm: 0.2038 | dt: 14461.73ms | tok/sec: 36253.5 | mem: 107563MB | Time Left: 02 hrs : 35 mins [TIMING] data: 8.1ms | fwd: 3575.5ms | loss: 13.1ms | bwd: 10827.6ms | clip: 0.0ms | optim: 23.8ms step 1583 | loss: 2.846400 | lr 1.8000e-04 | norm: 0.1970 | dt: 14467.95ms | tok/sec: 36237.9 | mem: 107563MB | Time Left: 02 hrs : 35 mins [TIMING] data: 8.4ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10832.8ms | clip: 0.0ms | optim: 23.9ms step 1584 | loss: 2.977966 | lr 1.8000e-04 | norm: 0.1694 | dt: 14459.37ms | tok/sec: 36259.4 | mem: 107563MB | Time Left: 02 hrs : 34 mins [TIMING] data: 8.3ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10824.4ms | clip: 0.0ms | optim: 23.9ms step 1585 | loss: 2.873491 | lr 1.8000e-04 | norm: 0.1946 | dt: 14470.17ms | tok/sec: 36232.3 | mem: 107563MB | Time Left: 02 hrs : 34 mins [TIMING] data: 8.2ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10835.8ms | clip: 0.0ms | optim: 23.9ms step 1586 | loss: 2.860874 | lr 1.8000e-04 | norm: 0.2037 | dt: 14471.61ms | tok/sec: 36228.7 | mem: 107563MB | Time Left: 02 hrs : 34 mins [TIMING] data: 8.3ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10833.2ms | clip: 0.0ms | optim: 23.8ms step 1587 | loss: 2.786964 | lr 1.8000e-04 | norm: 0.1870 | dt: 14459.08ms | tok/sec: 36260.1 | mem: 107563MB | Time Left: 02 hrs : 34 mins [TIMING] data: 8.5ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10824.6ms | clip: 0.0ms | optim: 23.8ms step 1588 | loss: 2.721284 | lr 1.8000e-04 | norm: 0.1776 | dt: 14457.09ms | tok/sec: 36265.1 | mem: 107563MB | Time Left: 02 hrs : 33 mins [TIMING] data: 8.3ms | fwd: 3570.8ms | loss: 13.1ms | bwd: 10827.4ms | clip: 0.0ms | optim: 23.9ms step 1589 | loss: 2.776896 | lr 1.8000e-04 | norm: 0.1647 | dt: 14462.55ms | tok/sec: 36251.4 | mem: 107563MB | Time Left: 02 hrs : 33 mins [TIMING] data: 8.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10822.1ms | clip: 0.0ms | optim: 23.9ms step 1590 | loss: 3.158499 | lr 1.8000e-04 | norm: 0.1855 | dt: 14466.31ms | tok/sec: 36242.0 | mem: 107563MB | Time Left: 02 hrs : 33 mins [TIMING] data: 8.3ms | fwd: 3575.5ms | loss: 13.0ms | bwd: 10831.6ms | clip: 0.0ms | optim: 23.9ms step 1591 | loss: 2.837645 | lr 1.8000e-04 | norm: 0.2396 | dt: 14458.75ms | tok/sec: 36260.9 | mem: 107563MB | Time Left: 02 hrs : 33 mins [TIMING] data: 8.4ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10821.1ms | clip: 0.0ms | optim: 23.8ms step 1592 | loss: 2.665658 | lr 1.8000e-04 | norm: 0.2170 | dt: 14457.30ms | tok/sec: 36264.6 | mem: 107563MB | Time Left: 02 hrs : 33 mins [TIMING] data: 8.4ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10819.6ms | clip: 0.0ms | optim: 23.9ms step 1593 | loss: 3.059995 | lr 1.8000e-04 | norm: 0.2161 | dt: 14457.94ms | tok/sec: 36263.0 | mem: 107563MB | Time Left: 02 hrs : 32 mins [TIMING] data: 8.5ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10823.2ms | clip: 0.0ms | optim: 23.9ms step 1594 | loss: 2.605000 | lr 1.8000e-04 | norm: 0.1593 | dt: 14465.02ms | tok/sec: 36245.2 | mem: 107563MB | Time Left: 02 hrs : 32 mins [TIMING] data: 8.5ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10830.4ms | clip: 0.0ms | optim: 23.9ms step 1595 | loss: 3.271989 | lr 1.8000e-04 | norm: 0.2382 | dt: 14460.59ms | tok/sec: 36256.3 | mem: 107563MB | Time Left: 02 hrs : 32 mins [TIMING] data: 8.4ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10826.6ms | clip: 0.0ms | optim: 23.9ms step 1596 | loss: 2.941068 | lr 1.8000e-04 | norm: 0.1806 | dt: 14457.09ms | tok/sec: 36265.1 | mem: 107563MB | Time Left: 02 hrs : 32 mins [TIMING] data: 8.3ms | fwd: 3578.3ms | loss: 13.0ms | bwd: 10820.2ms | clip: 0.0ms | optim: 23.9ms step 1597 | loss: 2.820417 | lr 1.8000e-04 | norm: 0.1924 | dt: 14455.45ms | tok/sec: 36269.2 | mem: 107563MB | Time Left: 02 hrs : 31 mins [TIMING] data: 8.5ms | fwd: 3573.5ms | loss: 13.0ms | bwd: 10823.1ms | clip: 0.0ms | optim: 24.0ms step 1598 | loss: 2.818999 | lr 1.8000e-04 | norm: 0.1885 | dt: 14455.44ms | tok/sec: 36269.3 | mem: 107563MB | Time Left: 02 hrs : 31 mins [TIMING] data: 8.3ms | fwd: 3575.4ms | loss: 13.0ms | bwd: 10821.3ms | clip: 0.0ms | optim: 23.9ms step 1599 | loss: 2.818247 | lr 1.8000e-04 | norm: 0.5525 | dt: 14446.73ms | tok/sec: 36291.1 | mem: 107563MB | Time Left: 02 hrs : 31 mins [TIMING] data: 8.4ms | fwd: 3576.5ms | loss: 13.1ms | bwd: 10811.4ms | clip: 0.0ms | optim: 23.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1600.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1500.ckpt step 1600 | loss: 2.897590 | lr 1.8000e-04 | norm: 0.2644 | dt: 14369.68ms | tok/sec: 36485.7 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 9.0ms | fwd: 3570.0ms | loss: 13.2ms | bwd: 10739.9ms | clip: 0.0ms | optim: 23.7ms step 1601 | loss: 3.160919 | lr 1.8000e-04 | norm: 0.2270 | dt: 14399.78ms | tok/sec: 36409.5 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 8.5ms | fwd: 3561.4ms | loss: 13.1ms | bwd: 10779.5ms | clip: 0.0ms | optim: 23.8ms step 1602 | loss: 2.865726 | lr 1.8000e-04 | norm: 0.2053 | dt: 14432.39ms | tok/sec: 36327.2 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 8.3ms | fwd: 3572.8ms | loss: 13.1ms | bwd: 10801.0ms | clip: 0.0ms | optim: 23.8ms step 1603 | loss: 2.872779 | lr 1.8000e-04 | norm: 0.3357 | dt: 14435.57ms | tok/sec: 36319.2 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 8.5ms | fwd: 3569.8ms | loss: 13.1ms | bwd: 10806.9ms | clip: 0.0ms | optim: 23.8ms step 1604 | loss: 2.811362 | lr 1.8000e-04 | norm: 0.2606 | dt: 14455.02ms | tok/sec: 36270.3 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 8.4ms | fwd: 3580.6ms | loss: 13.1ms | bwd: 10815.2ms | clip: 0.0ms | optim: 23.9ms step 1605 | loss: 2.968288 | lr 1.8000e-04 | norm: 0.1713 | dt: 14485.80ms | tok/sec: 36193.2 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 8.3ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10846.7ms | clip: 0.0ms | optim: 23.8ms step 1606 | loss: 2.902665 | lr 1.8000e-04 | norm: 0.1684 | dt: 14499.50ms | tok/sec: 36159.0 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 8.1ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10860.2ms | clip: 0.0ms | optim: 23.9ms step 1607 | loss: 2.978294 | lr 1.8000e-04 | norm: 0.1787 | dt: 14510.22ms | tok/sec: 36132.3 | mem: 107563MB | Time Left: 02 hrs : 29 mins [TIMING] data: 8.3ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10869.2ms | clip: 0.0ms | optim: 23.9ms step 1608 | loss: 3.033115 | lr 1.8000e-04 | norm: 0.2072 | dt: 14507.75ms | tok/sec: 36138.5 | mem: 107563MB | Time Left: 02 hrs : 29 mins [TIMING] data: 8.3ms | fwd: 3581.3ms | loss: 13.0ms | bwd: 10867.8ms | clip: 0.0ms | optim: 23.9ms step 1609 | loss: 2.724007 | lr 1.8000e-04 | norm: 0.1734 | dt: 14503.11ms | tok/sec: 36150.0 | mem: 107563MB | Time Left: 02 hrs : 29 mins [TIMING] data: 8.3ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10862.1ms | clip: 0.0ms | optim: 23.8ms step 1610 | loss: 3.014103 | lr 1.8000e-04 | norm: 0.2351 | dt: 14503.02ms | tok/sec: 36150.3 | mem: 107563MB | Time Left: 02 hrs : 29 mins [TIMING] data: 8.3ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 23.8ms step 1611 | loss: 2.707425 | lr 1.8000e-04 | norm: 0.2089 | dt: 14504.39ms | tok/sec: 36146.9 | mem: 107563MB | Time Left: 02 hrs : 28 mins [TIMING] data: 8.5ms | fwd: 3583.6ms | loss: 13.0ms | bwd: 10861.9ms | clip: 0.0ms | optim: 23.7ms step 1612 | loss: 2.772035 | lr 1.8000e-04 | norm: 0.2157 | dt: 14503.40ms | tok/sec: 36149.3 | mem: 107563MB | Time Left: 02 hrs : 28 mins [TIMING] data: 8.5ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10864.8ms | clip: 0.0ms | optim: 24.0ms step 1613 | loss: 2.561872 | lr 1.8000e-04 | norm: 0.1723 | dt: 14517.42ms | tok/sec: 36114.4 | mem: 107563MB | Time Left: 02 hrs : 28 mins [TIMING] data: 8.4ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10874.3ms | clip: 0.0ms | optim: 23.8ms step 1614 | loss: 2.654269 | lr 1.8000e-04 | norm: 0.1915 | dt: 14518.12ms | tok/sec: 36112.7 | mem: 107563MB | Time Left: 02 hrs : 28 mins [TIMING] data: 8.2ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10877.1ms | clip: 0.0ms | optim: 23.7ms step 1615 | loss: 2.744591 | lr 1.8000e-04 | norm: 0.2168 | dt: 14523.08ms | tok/sec: 36100.3 | mem: 107563MB | Time Left: 02 hrs : 28 mins [TIMING] data: 8.4ms | fwd: 3585.1ms | loss: 13.1ms | bwd: 10879.3ms | clip: 0.0ms | optim: 23.9ms step 1616 | loss: 3.113577 | lr 1.8000e-04 | norm: 0.1796 | dt: 14529.23ms | tok/sec: 36085.1 | mem: 107563MB | Time Left: 02 hrs : 27 mins [TIMING] data: 8.3ms | fwd: 3587.1ms | loss: 13.0ms | bwd: 10883.5ms | clip: 0.0ms | optim: 23.9ms step 1617 | loss: 2.894597 | lr 1.8000e-04 | norm: 0.1918 | dt: 14538.39ms | tok/sec: 36062.3 | mem: 107563MB | Time Left: 02 hrs : 27 mins [TIMING] data: 8.3ms | fwd: 3585.9ms | loss: 13.1ms | bwd: 10894.0ms | clip: 0.0ms | optim: 23.8ms step 1618 | loss: 2.901367 | lr 1.8000e-04 | norm: 0.1794 | dt: 14524.30ms | tok/sec: 36097.3 | mem: 107563MB | Time Left: 02 hrs : 27 mins [TIMING] data: 8.4ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10882.6ms | clip: 0.0ms | optim: 23.7ms step 1619 | loss: 2.704388 | lr 1.8000e-04 | norm: 0.2306 | dt: 14522.66ms | tok/sec: 36101.4 | mem: 107563MB | Time Left: 02 hrs : 27 mins [TIMING] data: 8.4ms | fwd: 3584.7ms | loss: 13.0ms | bwd: 10879.4ms | clip: 0.0ms | optim: 23.8ms step 1620 | loss: 3.151340 | lr 1.8000e-04 | norm: 0.2280 | dt: 14514.94ms | tok/sec: 36120.6 | mem: 107563MB | Time Left: 02 hrs : 26 mins [TIMING] data: 8.4ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10874.5ms | clip: 0.0ms | optim: 23.8ms step 1621 | loss: 2.815567 | lr 1.8000e-04 | norm: 0.1937 | dt: 14507.78ms | tok/sec: 36138.4 | mem: 107563MB | Time Left: 02 hrs : 26 mins [TIMING] data: 8.2ms | fwd: 3584.9ms | loss: 13.0ms | bwd: 10864.5ms | clip: 0.0ms | optim: 23.8ms step 1622 | loss: 2.739475 | lr 1.8000e-04 | norm: 0.2407 | dt: 14498.57ms | tok/sec: 36161.4 | mem: 107563MB | Time Left: 02 hrs : 26 mins [TIMING] data: 8.4ms | fwd: 3579.6ms | loss: 13.0ms | bwd: 10859.9ms | clip: 0.0ms | optim: 23.7ms step 1623 | loss: 3.060492 | lr 1.8000e-04 | norm: 0.2323 | dt: 14502.25ms | tok/sec: 36152.2 | mem: 107563MB | Time Left: 02 hrs : 25 mins [TIMING] data: 8.3ms | fwd: 3583.0ms | loss: 13.0ms | bwd: 10860.5ms | clip: 0.0ms | optim: 23.9ms step 1624 | loss: 2.839009 | lr 1.8000e-04 | norm: 0.1943 | dt: 14513.30ms | tok/sec: 36124.7 | mem: 107563MB | Time Left: 02 hrs : 25 mins [TIMING] data: 8.4ms | fwd: 3585.9ms | loss: 13.1ms | bwd: 10868.8ms | clip: 0.0ms | optim: 23.7ms step 1625 | loss: 2.898768 | lr 1.8000e-04 | norm: 0.1855 | dt: 14514.04ms | tok/sec: 36122.8 | mem: 107563MB | Time Left: 02 hrs : 25 mins [TIMING] data: 8.4ms | fwd: 3583.9ms | loss: 13.0ms | bwd: 10871.6ms | clip: 0.0ms | optim: 23.8ms step 1626 | loss: 3.038573 | lr 1.8000e-04 | norm: 0.2236 | dt: 14503.18ms | tok/sec: 36149.9 | mem: 107563MB | Time Left: 02 hrs : 25 mins [TIMING] data: 8.4ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10866.9ms | clip: 0.0ms | optim: 23.8ms step 1627 | loss: 2.836464 | lr 1.8000e-04 | norm: 0.2147 | dt: 14515.60ms | tok/sec: 36118.9 | mem: 107563MB | Time Left: 02 hrs : 25 mins [TIMING] data: 8.4ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10877.0ms | clip: 0.0ms | optim: 23.9ms step 1628 | loss: 2.727036 | lr 1.8000e-04 | norm: 0.2125 | dt: 14518.81ms | tok/sec: 36110.9 | mem: 107563MB | Time Left: 02 hrs : 24 mins [TIMING] data: 8.4ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10876.7ms | clip: 0.0ms | optim: 23.9ms step 1629 | loss: 2.836672 | lr 1.8000e-04 | norm: 0.2114 | dt: 14514.97ms | tok/sec: 36120.5 | mem: 107563MB | Time Left: 02 hrs : 24 mins [TIMING] data: 8.4ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10872.0ms | clip: 0.0ms | optim: 23.9ms step 1630 | loss: 2.741575 | lr 1.8000e-04 | norm: 0.1739 | dt: 14502.52ms | tok/sec: 36151.5 | mem: 107563MB | Time Left: 02 hrs : 24 mins [TIMING] data: 8.5ms | fwd: 3580.0ms | loss: 13.0ms | bwd: 10863.8ms | clip: 0.0ms | optim: 23.8ms step 1631 | loss: 2.871850 | lr 1.8000e-04 | norm: 0.2047 | dt: 14513.82ms | tok/sec: 36123.4 | mem: 107563MB | Time Left: 02 hrs : 24 mins [TIMING] data: 8.5ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10872.1ms | clip: 0.0ms | optim: 23.8ms step 1632 | loss: 2.715661 | lr 1.8000e-04 | norm: 0.1905 | dt: 14512.73ms | tok/sec: 36126.1 | mem: 107563MB | Time Left: 02 hrs : 23 mins [TIMING] data: 8.4ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10871.0ms | clip: 0.0ms | optim: 23.8ms step 1633 | loss: 2.729824 | lr 1.8000e-04 | norm: 0.1784 | dt: 14518.36ms | tok/sec: 36112.1 | mem: 107563MB | Time Left: 02 hrs : 23 mins [TIMING] data: 8.1ms | fwd: 3583.6ms | loss: 13.0ms | bwd: 10876.4ms | clip: 0.0ms | optim: 23.8ms step 1634 | loss: 2.994509 | lr 1.8000e-04 | norm: 0.2030 | dt: 14497.03ms | tok/sec: 36165.2 | mem: 107563MB | Time Left: 02 hrs : 23 mins [TIMING] data: 8.4ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10856.7ms | clip: 0.0ms | optim: 23.7ms step 1635 | loss: 2.985981 | lr 1.8000e-04 | norm: 0.1867 | dt: 14493.00ms | tok/sec: 36175.3 | mem: 107563MB | Time Left: 02 hrs : 22 mins [TIMING] data: 8.4ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.9ms step 1636 | loss: 3.035271 | lr 1.8000e-04 | norm: 0.1775 | dt: 14500.07ms | tok/sec: 36157.6 | mem: 107563MB | Time Left: 02 hrs : 22 mins [TIMING] data: 8.2ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.9ms step 1637 | loss: 2.931316 | lr 1.8000e-04 | norm: 0.2026 | dt: 14504.39ms | tok/sec: 36146.9 | mem: 107563MB | Time Left: 02 hrs : 22 mins [TIMING] data: 8.5ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10867.0ms | clip: 0.0ms | optim: 23.9ms step 1638 | loss: 2.955841 | lr 1.8000e-04 | norm: 0.1789 | dt: 14506.74ms | tok/sec: 36141.0 | mem: 107563MB | Time Left: 02 hrs : 22 mins [TIMING] data: 8.5ms | fwd: 3579.8ms | loss: 13.0ms | bwd: 10868.2ms | clip: 0.0ms | optim: 23.8ms step 1639 | loss: 2.638928 | lr 1.8000e-04 | norm: 0.2275 | dt: 14507.49ms | tok/sec: 36139.1 | mem: 107563MB | Time Left: 02 hrs : 22 mins [TIMING] data: 8.4ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10869.0ms | clip: 0.0ms | optim: 23.9ms step 1640 | loss: 3.115137 | lr 1.8000e-04 | norm: 0.2629 | dt: 14504.61ms | tok/sec: 36146.3 | mem: 107563MB | Time Left: 02 hrs : 21 mins [TIMING] data: 8.0ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10862.4ms | clip: 0.0ms | optim: 23.9ms step 1641 | loss: 2.951007 | lr 1.8000e-04 | norm: 0.2418 | dt: 14502.33ms | tok/sec: 36152.0 | mem: 107563MB | Time Left: 02 hrs : 21 mins [TIMING] data: 8.2ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10861.4ms | clip: 0.0ms | optim: 23.9ms step 1642 | loss: 2.903644 | lr 1.8000e-04 | norm: 0.2082 | dt: 14506.33ms | tok/sec: 36142.0 | mem: 107563MB | Time Left: 02 hrs : 21 mins [TIMING] data: 8.3ms | fwd: 3583.8ms | loss: 13.1ms | bwd: 10864.0ms | clip: 0.0ms | optim: 23.7ms step 1643 | loss: 2.936669 | lr 1.8000e-04 | norm: 0.2165 | dt: 14509.05ms | tok/sec: 36135.2 | mem: 107563MB | Time Left: 02 hrs : 21 mins [TIMING] data: 8.3ms | fwd: 3586.8ms | loss: 13.1ms | bwd: 10863.6ms | clip: 0.0ms | optim: 23.9ms step 1644 | loss: 2.700344 | lr 1.8000e-04 | norm: 0.2358 | dt: 14504.43ms | tok/sec: 36146.8 | mem: 107563MB | Time Left: 02 hrs : 20 mins [TIMING] data: 8.4ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10869.6ms | clip: 0.0ms | optim: 23.9ms step 1645 | loss: 2.682117 | lr 1.8000e-04 | norm: 0.2937 | dt: 14514.15ms | tok/sec: 36122.5 | mem: 107563MB | Time Left: 02 hrs : 20 mins [TIMING] data: 8.5ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10877.3ms | clip: 0.0ms | optim: 23.9ms step 1646 | loss: 2.810194 | lr 1.8000e-04 | norm: 0.2262 | dt: 14512.81ms | tok/sec: 36125.9 | mem: 107563MB | Time Left: 02 hrs : 20 mins [TIMING] data: 8.5ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10872.0ms | clip: 0.0ms | optim: 23.7ms step 1647 | loss: 2.942319 | lr 1.8000e-04 | norm: 0.2950 | dt: 14510.85ms | tok/sec: 36130.8 | mem: 107563MB | Time Left: 02 hrs : 20 mins [TIMING] data: 8.1ms | fwd: 3586.1ms | loss: 13.1ms | bwd: 10866.4ms | clip: 0.0ms | optim: 23.8ms step 1648 | loss: 2.569692 | lr 1.8000e-04 | norm: 0.2055 | dt: 14505.84ms | tok/sec: 36143.2 | mem: 107563MB | Time Left: 02 hrs : 19 mins [TIMING] data: 8.5ms | fwd: 3585.9ms | loss: 13.1ms | bwd: 10861.1ms | clip: 0.0ms | optim: 23.8ms step 1649 | loss: 2.773862 | lr 1.8000e-04 | norm: 0.2149 | dt: 14517.74ms | tok/sec: 36113.6 | mem: 107563MB | Time Left: 02 hrs : 19 mins [TIMING] data: 8.3ms | fwd: 3582.9ms | loss: 13.0ms | bwd: 10876.4ms | clip: 0.0ms | optim: 23.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1650.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1550.ckpt step 1650 | loss: 2.813203 | lr 1.8000e-04 | norm: 0.2231 | dt: 14397.64ms | tok/sec: 36414.9 | mem: 107563MB | Time Left: 02 hrs : 18 mins [TIMING] data: 9.7ms | fwd: 3579.4ms | loss: 13.3ms | bwd: 10756.6ms | clip: 0.0ms | optim: 23.9ms === Llama Context Parallel Training Script (bluscriptCP) === [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) ================ Configuration (bluscriptCP) ================ d_model/layers : 1536 / 36 q/kv heads : 12 / 4 (G=3, hd=128) ffn_hidden : 4096 (SwiGLU) context_length : 32768 vocab_size : 50304 tying=0 B x T : 1 x 32768 world_size : 2 (dp=1 x cp=2) CP_ATTN_MODE : ring (load_balance=HeadTail) global_batch : 524288 (grad_accum=16) precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] max_lr/min_lr : 0.0006 / 6e-05 warmup/steps : 100 / 2227 (WSD: stable=1177, decay tail=950) checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp) ============================================================= Initializing model on CUDA device 0... [YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693 Parameters: 1060624896 Parameters per GPU: 1060624896 max_steps: 2227 warmup_steps: 100 [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) [ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded) [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] [ZeRO-1] DataParallel is DISABLED (both would double-reduce) [precision] bf16 params=217, all with fp32 master weights found 12 shards for split train found 1 shards for split val Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (resume run 2) [rope_theta] 500000 from checkpoints_bluscriptcp/blumodelcp_run2.meta (authoritative) Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 2) [CheckpointManager] Auto-loading latest: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1650.ckpt [Resume] run 2 from step 1650 (loss 2.77386) [resume] fast-forwarded data loader 26400 batches (865075200 tokens) [re-warmup] anchored at step 750 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 1650 (past ramp, in cosine-decay) Starting training... [CP ring P2P] OVERLAP: dedicated non-blocking stream [CP bwd] overlap=ON step 1650 | loss: 2.813203 | lr 1.8000e-04 | norm: 0.2231 | dt: 15712.70ms | tok/sec: 33367.1 | mem: 107563MB | Time Left: 02 hrs : 31 mins [TIMING] data: 10.1ms | fwd: 4909.4ms | loss: 16.7ms | bwd: 10731.8ms | clip: 0.0ms | optim: 24.3ms step 1651 | loss: 2.738159 | lr 1.8000e-04 | norm: 0.1830 | dt: 14351.01ms | tok/sec: 36533.2 | mem: 107563MB | Time Left: 02 hrs : 17 mins [TIMING] data: 6.8ms | fwd: 3555.9ms | loss: 13.2ms | bwd: 10737.6ms | clip: 0.0ms | optim: 23.9ms step 1652 | loss: 3.089616 | lr 1.8000e-04 | norm: 0.2194 | dt: 14425.27ms | tok/sec: 36345.1 | mem: 107563MB | Time Left: 02 hrs : 18 mins [TIMING] data: 6.9ms | fwd: 3568.9ms | loss: 13.1ms | bwd: 10798.8ms | clip: 0.0ms | optim: 23.9ms step 1653 | loss: 2.839664 | lr 1.8000e-04 | norm: 0.2134 | dt: 14466.08ms | tok/sec: 36242.6 | mem: 107563MB | Time Left: 02 hrs : 18 mins [TIMING] data: 6.9ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10829.9ms | clip: 0.0ms | optim: 23.8ms step 1654 | loss: 3.105408 | lr 1.8000e-04 | norm: 0.2034 | dt: 14503.73ms | tok/sec: 36148.5 | mem: 107563MB | Time Left: 02 hrs : 18 mins [TIMING] data: 6.9ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10866.4ms | clip: 0.0ms | optim: 23.8ms step 1655 | loss: 3.104078 | lr 1.8000e-04 | norm: 0.1744 | dt: 14523.29ms | tok/sec: 36099.8 | mem: 107563MB | Time Left: 02 hrs : 18 mins [TIMING] data: 6.9ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10882.5ms | clip: 0.0ms | optim: 23.9ms step 1656 | loss: 2.923594 | lr 1.8000e-04 | norm: 0.1646 | dt: 14517.28ms | tok/sec: 36114.8 | mem: 107563MB | Time Left: 02 hrs : 18 mins [TIMING] data: 6.8ms | fwd: 3585.7ms | loss: 13.1ms | bwd: 10874.1ms | clip: 0.0ms | optim: 23.8ms step 1657 | loss: 3.070223 | lr 1.8000e-04 | norm: 0.2334 | dt: 14502.63ms | tok/sec: 36151.2 | mem: 107563MB | Time Left: 02 hrs : 17 mins [TIMING] data: 6.9ms | fwd: 3583.2ms | loss: 13.2ms | bwd: 10861.6ms | clip: 0.0ms | optim: 24.0ms step 1658 | loss: 2.874304 | lr 1.8000e-04 | norm: 0.1729 | dt: 14500.91ms | tok/sec: 36155.5 | mem: 107563MB | Time Left: 02 hrs : 17 mins [TIMING] data: 6.8ms | fwd: 3579.9ms | loss: 13.2ms | bwd: 10863.2ms | clip: 0.0ms | optim: 23.9ms step 1659 | loss: 2.961006 | lr 1.8000e-04 | norm: 0.2442 | dt: 14495.94ms | tok/sec: 36167.9 | mem: 107563MB | Time Left: 02 hrs : 17 mins [TIMING] data: 6.9ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10857.0ms | clip: 0.0ms | optim: 23.8ms step 1660 | loss: 2.854506 | lr 1.8000e-04 | norm: 0.1779 | dt: 14484.70ms | tok/sec: 36196.0 | mem: 107563MB | Time Left: 02 hrs : 16 mins [TIMING] data: 6.8ms | fwd: 3578.4ms | loss: 13.0ms | bwd: 10848.9ms | clip: 0.0ms | optim: 23.9ms step 1661 | loss: 2.780960 | lr 1.8000e-04 | norm: 0.1661 | dt: 14481.77ms | tok/sec: 36203.3 | mem: 107563MB | Time Left: 02 hrs : 16 mins [TIMING] data: 6.9ms | fwd: 3576.9ms | loss: 13.2ms | bwd: 10847.1ms | clip: 0.0ms | optim: 23.9ms step 1662 | loss: 2.796217 | lr 1.8000e-04 | norm: 0.2333 | dt: 14495.82ms | tok/sec: 36168.2 | mem: 107563MB | Time Left: 02 hrs : 16 mins [TIMING] data: 7.2ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10858.5ms | clip: 0.0ms | optim: 23.8ms step 1663 | loss: 2.672337 | lr 1.8000e-04 | norm: 0.1739 | dt: 14501.00ms | tok/sec: 36155.3 | mem: 107563MB | Time Left: 02 hrs : 16 mins [TIMING] data: 6.9ms | fwd: 3583.1ms | loss: 13.2ms | bwd: 10860.0ms | clip: 0.0ms | optim: 23.9ms step 1664 | loss: 2.794945 | lr 1.8000e-04 | norm: 0.1932 | dt: 14500.20ms | tok/sec: 36157.3 | mem: 107563MB | Time Left: 02 hrs : 16 mins [TIMING] data: 7.2ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10860.4ms | clip: 0.0ms | optim: 23.9ms step 1665 | loss: 2.878074 | lr 1.8000e-04 | norm: 0.2103 | dt: 14495.15ms | tok/sec: 36169.9 | mem: 107563MB | Time Left: 02 hrs : 15 mins [TIMING] data: 6.8ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10856.7ms | clip: 0.0ms | optim: 23.9ms step 1666 | loss: 2.805625 | lr 1.8000e-04 | norm: 0.2345 | dt: 14501.59ms | tok/sec: 36153.8 | mem: 107563MB | Time Left: 02 hrs : 15 mins [TIMING] data: 6.9ms | fwd: 3578.8ms | loss: 13.4ms | bwd: 10864.9ms | clip: 0.0ms | optim: 23.9ms step 1667 | loss: 2.643424 | lr 1.8000e-04 | norm: 0.1907 | dt: 14499.86ms | tok/sec: 36158.1 | mem: 107563MB | Time Left: 02 hrs : 15 mins [TIMING] data: 6.9ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10864.6ms | clip: 0.0ms | optim: 23.9ms step 1668 | loss: 2.914708 | lr 1.8000e-04 | norm: 0.2286 | dt: 14498.50ms | tok/sec: 36161.5 | mem: 107563MB | Time Left: 02 hrs : 15 mins [TIMING] data: 6.9ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10861.4ms | clip: 0.0ms | optim: 23.9ms step 1669 | loss: 3.149620 | lr 1.8000e-04 | norm: 0.1708 | dt: 14491.39ms | tok/sec: 36179.3 | mem: 107563MB | Time Left: 02 hrs : 14 mins [TIMING] data: 6.9ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10850.1ms | clip: 0.0ms | optim: 23.9ms step 1670 | loss: 2.716568 | lr 1.8000e-04 | norm: 0.1891 | dt: 14485.87ms | tok/sec: 36193.1 | mem: 107563MB | Time Left: 02 hrs : 14 mins [TIMING] data: 7.0ms | fwd: 3580.6ms | loss: 13.2ms | bwd: 10846.8ms | clip: 0.0ms | optim: 23.9ms step 1671 | loss: 2.800768 | lr 1.8000e-04 | norm: 0.1827 | dt: 14498.38ms | tok/sec: 36161.8 | mem: 107563MB | Time Left: 02 hrs : 14 mins [TIMING] data: 6.9ms | fwd: 3580.2ms | loss: 13.2ms | bwd: 10860.0ms | clip: 0.0ms | optim: 24.3ms step 1672 | loss: 3.047858 | lr 1.8000e-04 | norm: 0.1585 | dt: 14496.89ms | tok/sec: 36165.6 | mem: 107563MB | Time Left: 02 hrs : 14 mins [TIMING] data: 6.9ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10859.5ms | clip: 0.0ms | optim: 23.8ms step 1673 | loss: 2.623895 | lr 1.8000e-04 | norm: 0.1637 | dt: 14486.36ms | tok/sec: 36191.8 | mem: 107563MB | Time Left: 02 hrs : 13 mins [TIMING] data: 6.9ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10852.5ms | clip: 0.0ms | optim: 23.9ms step 1674 | loss: 2.890058 | lr 1.8000e-04 | norm: 0.2033 | dt: 14488.23ms | tok/sec: 36187.2 | mem: 107563MB | Time Left: 02 hrs : 13 mins [TIMING] data: 7.0ms | fwd: 3581.2ms | loss: 13.2ms | bwd: 10849.1ms | clip: 0.0ms | optim: 23.9ms step 1675 | loss: 2.696971 | lr 1.8000e-04 | norm: 0.1744 | dt: 14483.74ms | tok/sec: 36198.4 | mem: 107563MB | Time Left: 02 hrs : 13 mins [TIMING] data: 6.9ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10847.5ms | clip: 0.0ms | optim: 23.8ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00009.bin 100000000 step 1676 | loss: 2.905978 | lr 1.8000e-04 | norm: 0.1978 | dt: 14487.36ms | tok/sec: 36189.3 | mem: 107563MB | Time Left: 02 hrs : 13 mins [TIMING] data: 7.9ms | fwd: 3575.6ms | loss: 13.1ms | bwd: 10853.1ms | clip: 0.0ms | optim: 23.9ms step 1677 | loss: 2.765737 | lr 1.8000e-04 | norm: 0.1707 | dt: 14487.37ms | tok/sec: 36189.3 | mem: 107563MB | Time Left: 02 hrs : 12 mins [TIMING] data: 7.0ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10851.0ms | clip: 0.0ms | optim: 23.9ms step 1678 | loss: 2.892933 | lr 1.8000e-04 | norm: 0.1660 | dt: 14491.60ms | tok/sec: 36178.7 | mem: 107563MB | Time Left: 02 hrs : 12 mins [TIMING] data: 7.1ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10858.1ms | clip: 0.0ms | optim: 24.0ms step 1679 | loss: 2.629102 | lr 1.8000e-04 | norm: 0.2018 | dt: 14493.46ms | tok/sec: 36174.1 | mem: 107563MB | Time Left: 02 hrs : 12 mins [TIMING] data: 7.0ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10857.6ms | clip: 0.0ms | optim: 23.9ms step 1680 | loss: 2.889678 | lr 1.8000e-04 | norm: 0.1699 | dt: 14490.25ms | tok/sec: 36182.1 | mem: 107563MB | Time Left: 02 hrs : 12 mins [TIMING] data: 6.9ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10854.0ms | clip: 0.0ms | optim: 23.9ms step 1681 | loss: 2.655156 | lr 1.8000e-04 | norm: 0.1716 | dt: 14500.41ms | tok/sec: 36156.8 | mem: 107563MB | Time Left: 02 hrs : 11 mins [TIMING] data: 7.0ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10860.0ms | clip: 0.0ms | optim: 23.9ms step 1682 | loss: 2.826968 | lr 1.8000e-04 | norm: 0.1737 | dt: 14501.65ms | tok/sec: 36153.7 | mem: 107563MB | Time Left: 02 hrs : 11 mins [TIMING] data: 7.0ms | fwd: 3580.5ms | loss: 13.2ms | bwd: 10863.5ms | clip: 0.0ms | optim: 23.9ms step 1683 | loss: 3.116706 | lr 1.8000e-04 | norm: 0.1807 | dt: 14502.80ms | tok/sec: 36150.8 | mem: 107563MB | Time Left: 02 hrs : 11 mins [TIMING] data: 6.9ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10863.4ms | clip: 0.0ms | optim: 23.8ms step 1684 | loss: 2.607711 | lr 1.8000e-04 | norm: 0.1684 | dt: 14502.08ms | tok/sec: 36152.6 | mem: 107563MB | Time Left: 02 hrs : 11 mins [TIMING] data: 6.9ms | fwd: 3582.5ms | loss: 13.2ms | bwd: 10861.8ms | clip: 0.0ms | optim: 23.8ms step 1685 | loss: 3.052029 | lr 1.8000e-04 | norm: 0.2035 | dt: 14501.58ms | tok/sec: 36153.8 | mem: 107563MB | Time Left: 02 hrs : 10 mins [TIMING] data: 6.9ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10863.6ms | clip: 0.0ms | optim: 23.9ms step 1686 | loss: 2.850089 | lr 1.8000e-04 | norm: 0.1822 | dt: 14489.71ms | tok/sec: 36183.5 | mem: 107563MB | Time Left: 02 hrs : 10 mins [TIMING] data: 7.0ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.9ms step 1687 | loss: 2.998075 | lr 1.8000e-04 | norm: 0.2727 | dt: 14503.17ms | tok/sec: 36149.9 | mem: 107563MB | Time Left: 02 hrs : 10 mins [TIMING] data: 6.9ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10864.7ms | clip: 0.0ms | optim: 23.8ms step 1688 | loss: 2.927525 | lr 1.8000e-04 | norm: 0.2109 | dt: 14495.27ms | tok/sec: 36169.6 | mem: 107563MB | Time Left: 02 hrs : 10 mins [TIMING] data: 6.9ms | fwd: 3585.9ms | loss: 13.1ms | bwd: 10851.6ms | clip: 0.0ms | optim: 24.0ms step 1689 | loss: 2.876577 | lr 1.8000e-04 | norm: 0.1947 | dt: 14501.57ms | tok/sec: 36153.9 | mem: 107563MB | Time Left: 02 hrs : 10 mins [TIMING] data: 6.9ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10863.3ms | clip: 0.0ms | optim: 23.8ms step 1690 | loss: 2.943650 | lr 1.8000e-04 | norm: 2.9948 | dt: 14491.52ms | tok/sec: 36178.9 | mem: 107563MB | Time Left: 02 hrs : 09 mins [TIMING] data: 7.0ms | fwd: 3578.7ms | loss: 13.2ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.9ms step 1691 | loss: 2.854730 | lr 1.8000e-04 | norm: 0.1961 | dt: 14489.18ms | tok/sec: 36184.8 | mem: 107563MB | Time Left: 02 hrs : 09 mins [TIMING] data: 6.9ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10851.9ms | clip: 0.0ms | optim: 23.8ms step 1692 | loss: 2.864092 | lr 1.8000e-04 | norm: 0.1809 | dt: 14494.01ms | tok/sec: 36172.7 | mem: 107563MB | Time Left: 02 hrs : 09 mins [TIMING] data: 6.8ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10861.6ms | clip: 0.0ms | optim: 24.0ms step 1693 | loss: 2.981838 | lr 1.8000e-04 | norm: 0.3622 | dt: 14503.31ms | tok/sec: 36149.6 | mem: 107563MB | Time Left: 02 hrs : 09 mins [TIMING] data: 6.8ms | fwd: 3585.1ms | loss: 13.1ms | bwd: 10860.4ms | clip: 0.0ms | optim: 23.8ms step 1694 | loss: 2.772254 | lr 1.8000e-04 | norm: 0.1900 | dt: 14498.89ms | tok/sec: 36160.6 | mem: 107563MB | Time Left: 02 hrs : 08 mins [TIMING] data: 7.0ms | fwd: 3575.8ms | loss: 13.1ms | bwd: 10865.3ms | clip: 0.0ms | optim: 23.9ms step 1695 | loss: 2.865608 | lr 1.8000e-04 | norm: 0.1820 | dt: 14498.34ms | tok/sec: 36161.9 | mem: 107563MB | Time Left: 02 hrs : 08 mins [TIMING] data: 7.0ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10859.4ms | clip: 0.0ms | optim: 23.9ms step 1696 | loss: 2.894587 | lr 1.8000e-04 | norm: 0.1671 | dt: 14489.46ms | tok/sec: 36184.1 | mem: 107563MB | Time Left: 02 hrs : 08 mins [TIMING] data: 6.9ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10855.7ms | clip: 0.0ms | optim: 23.8ms step 1697 | loss: 2.779546 | lr 1.8000e-04 | norm: 0.1681 | dt: 14488.09ms | tok/sec: 36187.5 | mem: 107563MB | Time Left: 02 hrs : 07 mins [TIMING] data: 6.8ms | fwd: 3580.4ms | loss: 13.2ms | bwd: 10850.0ms | clip: 0.0ms | optim: 24.0ms step 1698 | loss: 2.730263 | lr 1.8000e-04 | norm: 0.1657 | dt: 14490.77ms | tok/sec: 36180.8 | mem: 107563MB | Time Left: 02 hrs : 07 mins [TIMING] data: 7.0ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10854.7ms | clip: 0.0ms | optim: 23.8ms step 1699 | loss: 2.769960 | lr 1.8000e-04 | norm: 0.2067 | dt: 14488.27ms | tok/sec: 36187.1 | mem: 107563MB | Time Left: 02 hrs : 07 mins [TIMING] data: 7.0ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10850.1ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1700.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1600.ckpt step 1700 | loss: 2.698478 | lr 1.8000e-04 | norm: 0.2064 | dt: 14472.72ms | tok/sec: 36225.9 | mem: 107563MB | Time Left: 02 hrs : 07 mins [TIMING] data: 7.6ms | fwd: 3645.5ms | loss: 13.1ms | bwd: 10768.7ms | clip: 0.0ms | optim: 23.8ms step 1701 | loss: 2.698147 | lr 1.8000e-04 | norm: 0.2032 | dt: 14432.50ms | tok/sec: 36326.9 | mem: 107563MB | Time Left: 02 hrs : 06 mins [TIMING] data: 7.7ms | fwd: 3565.4ms | loss: 13.1ms | bwd: 10808.6ms | clip: 0.0ms | optim: 23.9ms step 1702 | loss: 2.800990 | lr 1.8000e-04 | norm: 0.1901 | dt: 14450.42ms | tok/sec: 36281.9 | mem: 107563MB | Time Left: 02 hrs : 06 mins [TIMING] data: 7.8ms | fwd: 3569.3ms | loss: 13.1ms | bwd: 10822.5ms | clip: 0.0ms | optim: 23.8ms step 1703 | loss: 2.979387 | lr 1.8000e-04 | norm: 0.2464 | dt: 14460.09ms | tok/sec: 36257.6 | mem: 107563MB | Time Left: 02 hrs : 06 mins [TIMING] data: 7.6ms | fwd: 3578.3ms | loss: 13.2ms | bwd: 10823.3ms | clip: 0.0ms | optim: 23.8ms step 1704 | loss: 2.707264 | lr 1.8000e-04 | norm: 0.4124 | dt: 14466.22ms | tok/sec: 36242.2 | mem: 107563MB | Time Left: 02 hrs : 06 mins [TIMING] data: 7.8ms | fwd: 3575.4ms | loss: 13.1ms | bwd: 10832.3ms | clip: 0.0ms | optim: 23.8ms step 1705 | loss: 2.764707 | lr 1.8000e-04 | norm: 0.1998 | dt: 14480.37ms | tok/sec: 36206.8 | mem: 107563MB | Time Left: 02 hrs : 05 mins [TIMING] data: 7.7ms | fwd: 3574.7ms | loss: 13.1ms | bwd: 10847.1ms | clip: 0.0ms | optim: 24.0ms step 1706 | loss: 2.861862 | lr 1.8000e-04 | norm: 0.1813 | dt: 14484.78ms | tok/sec: 36195.8 | mem: 107563MB | Time Left: 02 hrs : 05 mins [TIMING] data: 7.7ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10849.2ms | clip: 0.0ms | optim: 23.8ms step 1707 | loss: 2.806221 | lr 1.8000e-04 | norm: 0.3112 | dt: 14480.47ms | tok/sec: 36206.6 | mem: 107563MB | Time Left: 02 hrs : 05 mins [TIMING] data: 7.7ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10844.4ms | clip: 0.0ms | optim: 23.8ms step 1708 | loss: 3.040443 | lr 1.8000e-04 | norm: 0.2517 | dt: 14488.29ms | tok/sec: 36187.0 | mem: 107563MB | Time Left: 02 hrs : 05 mins [TIMING] data: 7.6ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.8ms step 1709 | loss: 3.131993 | lr 1.8000e-04 | norm: 0.1674 | dt: 14489.75ms | tok/sec: 36183.4 | mem: 107563MB | Time Left: 02 hrs : 05 mins [TIMING] data: 7.6ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10849.0ms | clip: 0.0ms | optim: 23.8ms step 1710 | loss: 2.948288 | lr 1.8000e-04 | norm: 0.2115 | dt: 14483.66ms | tok/sec: 36198.6 | mem: 107563MB | Time Left: 02 hrs : 04 mins [TIMING] data: 7.6ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10847.6ms | clip: 0.0ms | optim: 23.8ms step 1711 | loss: 2.683624 | lr 1.8000e-04 | norm: 0.1962 | dt: 14482.17ms | tok/sec: 36202.3 | mem: 107563MB | Time Left: 02 hrs : 04 mins [TIMING] data: 7.7ms | fwd: 3574.6ms | loss: 13.1ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.8ms step 1712 | loss: 2.773330 | lr 1.8000e-04 | norm: 0.3883 | dt: 14482.69ms | tok/sec: 36201.0 | mem: 107563MB | Time Left: 02 hrs : 04 mins [TIMING] data: 7.7ms | fwd: 3574.7ms | loss: 13.2ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 1713 | loss: 2.715285 | lr 1.8000e-04 | norm: 0.1954 | dt: 14494.60ms | tok/sec: 36171.3 | mem: 107563MB | Time Left: 02 hrs : 04 mins [TIMING] data: 8.1ms | fwd: 3577.9ms | loss: 13.2ms | bwd: 10857.9ms | clip: 0.0ms | optim: 23.8ms step 1714 | loss: 2.636481 | lr 1.8000e-04 | norm: 0.1888 | dt: 14475.98ms | tok/sec: 36217.8 | mem: 107563MB | Time Left: 02 hrs : 03 mins [TIMING] data: 7.7ms | fwd: 3575.4ms | loss: 13.1ms | bwd: 10842.1ms | clip: 0.0ms | optim: 23.9ms step 1715 | loss: 2.518733 | lr 1.8000e-04 | norm: 0.1647 | dt: 14491.16ms | tok/sec: 36179.8 | mem: 107563MB | Time Left: 02 hrs : 03 mins [TIMING] data: 7.5ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10855.1ms | clip: 0.0ms | optim: 23.8ms step 1716 | loss: 2.726676 | lr 1.8000e-04 | norm: 0.1711 | dt: 14491.15ms | tok/sec: 36179.9 | mem: 107563MB | Time Left: 02 hrs : 03 mins [TIMING] data: 7.7ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.8ms step 1717 | loss: 2.823031 | lr 1.8000e-04 | norm: 0.2012 | dt: 14492.62ms | tok/sec: 36176.2 | mem: 107563MB | Time Left: 02 hrs : 03 mins [TIMING] data: 8.4ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10854.4ms | clip: 0.0ms | optim: 23.8ms step 1718 | loss: 2.738247 | lr 1.8000e-04 | norm: 0.2080 | dt: 14481.09ms | tok/sec: 36205.0 | mem: 107563MB | Time Left: 02 hrs : 02 mins [TIMING] data: 7.6ms | fwd: 3583.5ms | loss: 13.1ms | bwd: 10839.5ms | clip: 0.0ms | optim: 23.8ms step 1719 | loss: 3.052269 | lr 1.8000e-04 | norm: 0.2379 | dt: 14482.44ms | tok/sec: 36201.6 | mem: 107563MB | Time Left: 02 hrs : 02 mins [TIMING] data: 8.1ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10842.6ms | clip: 0.0ms | optim: 23.9ms step 1720 | loss: 2.691569 | lr 1.8000e-04 | norm: 0.1682 | dt: 14493.02ms | tok/sec: 36175.2 | mem: 107563MB | Time Left: 02 hrs : 02 mins [TIMING] data: 7.6ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10851.7ms | clip: 0.0ms | optim: 23.9ms step 1721 | loss: 2.808504 | lr 1.8000e-04 | norm: 0.1768 | dt: 14490.11ms | tok/sec: 36182.5 | mem: 107563MB | Time Left: 02 hrs : 02 mins [TIMING] data: 7.7ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.8ms step 1722 | loss: 2.946760 | lr 1.8000e-04 | norm: 0.1977 | dt: 14499.66ms | tok/sec: 36158.6 | mem: 107563MB | Time Left: 02 hrs : 02 mins [TIMING] data: 7.7ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10861.0ms | clip: 0.0ms | optim: 24.0ms step 1723 | loss: 3.055587 | lr 1.8000e-04 | norm: 0.2767 | dt: 14490.06ms | tok/sec: 36182.6 | mem: 107563MB | Time Left: 02 hrs : 01 mins [TIMING] data: 7.6ms | fwd: 3580.6ms | loss: 13.1ms | bwd: 10851.1ms | clip: 0.0ms | optim: 23.9ms step 1724 | loss: 2.875625 | lr 1.8000e-04 | norm: 0.1752 | dt: 14489.53ms | tok/sec: 36183.9 | mem: 107563MB | Time Left: 02 hrs : 01 mins [TIMING] data: 7.7ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.8ms step 1725 | loss: 2.712570 | lr 1.8000e-04 | norm: 0.2820 | dt: 14481.39ms | tok/sec: 36204.3 | mem: 107563MB | Time Left: 02 hrs : 01 mins [TIMING] data: 7.6ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10844.4ms | clip: 0.0ms | optim: 23.8ms step 1726 | loss: 2.706581 | lr 1.8000e-04 | norm: 0.1962 | dt: 14491.92ms | tok/sec: 36177.9 | mem: 107563MB | Time Left: 02 hrs : 01 mins [TIMING] data: 7.6ms | fwd: 3584.3ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 1727 | loss: 2.760709 | lr 1.8000e-04 | norm: 0.1862 | dt: 14492.15ms | tok/sec: 36177.4 | mem: 107563MB | Time Left: 02 hrs : 00 mins [TIMING] data: 7.6ms | fwd: 3576.6ms | loss: 13.2ms | bwd: 10857.1ms | clip: 0.0ms | optim: 23.9ms step 1728 | loss: 2.886271 | lr 1.8000e-04 | norm: 0.1672 | dt: 14480.44ms | tok/sec: 36206.6 | mem: 107563MB | Time Left: 02 hrs : 00 mins [TIMING] data: 7.6ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10845.3ms | clip: 0.0ms | optim: 23.9ms step 1729 | loss: 2.932682 | lr 1.8000e-04 | norm: 0.1761 | dt: 14480.20ms | tok/sec: 36207.2 | mem: 107563MB | Time Left: 02 hrs : 00 mins [TIMING] data: 7.6ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.9ms step 1730 | loss: 2.759471 | lr 1.8000e-04 | norm: 0.2798 | dt: 14487.27ms | tok/sec: 36189.6 | mem: 107563MB | Time Left: 02 hrs : 00 mins [TIMING] data: 7.7ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.8ms step 1731 | loss: 2.943433 | lr 1.8000e-04 | norm: 0.1964 | dt: 14488.91ms | tok/sec: 36185.5 | mem: 107563MB | Time Left: 01 hrs : 59 mins [TIMING] data: 7.6ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10849.8ms | clip: 0.0ms | optim: 23.9ms step 1732 | loss: 2.954053 | lr 1.8000e-04 | norm: 0.1954 | dt: 14487.66ms | tok/sec: 36188.6 | mem: 107563MB | Time Left: 01 hrs : 59 mins [TIMING] data: 7.7ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10850.5ms | clip: 0.0ms | optim: 23.8ms step 1733 | loss: 2.819556 | lr 1.8000e-04 | norm: 0.1791 | dt: 14474.71ms | tok/sec: 36221.0 | mem: 107563MB | Time Left: 01 hrs : 59 mins [TIMING] data: 7.7ms | fwd: 3583.0ms | loss: 13.2ms | bwd: 10833.2ms | clip: 0.0ms | optim: 23.9ms step 1734 | loss: 2.804323 | lr 1.8000e-04 | norm: 0.1963 | dt: 14492.17ms | tok/sec: 36177.3 | mem: 107563MB | Time Left: 01 hrs : 59 mins [TIMING] data: 7.6ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.8ms step 1735 | loss: 2.670355 | lr 1.8000e-04 | norm: 0.1849 | dt: 14488.54ms | tok/sec: 36186.4 | mem: 107563MB | Time Left: 01 hrs : 58 mins [TIMING] data: 7.6ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10850.8ms | clip: 0.0ms | optim: 23.9ms step 1736 | loss: 2.684348 | lr 1.8000e-04 | norm: 0.1971 | dt: 14503.14ms | tok/sec: 36150.0 | mem: 107563MB | Time Left: 01 hrs : 58 mins [TIMING] data: 7.8ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10867.0ms | clip: 0.0ms | optim: 23.8ms step 1737 | loss: 2.931789 | lr 1.8000e-04 | norm: 0.4158 | dt: 14507.93ms | tok/sec: 36138.0 | mem: 107563MB | Time Left: 01 hrs : 58 mins [TIMING] data: 7.7ms | fwd: 3585.8ms | loss: 13.1ms | bwd: 10863.9ms | clip: 0.0ms | optim: 23.8ms step 1738 | loss: 2.897421 | lr 1.8000e-04 | norm: 0.1598 | dt: 14503.97ms | tok/sec: 36147.9 | mem: 107563MB | Time Left: 01 hrs : 58 mins [TIMING] data: 7.7ms | fwd: 3584.2ms | loss: 13.1ms | bwd: 10861.3ms | clip: 0.0ms | optim: 23.9ms step 1739 | loss: 2.933740 | lr 1.8000e-04 | norm: 0.1639 | dt: 14505.68ms | tok/sec: 36143.6 | mem: 107563MB | Time Left: 01 hrs : 57 mins [TIMING] data: 7.7ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10866.4ms | clip: 0.0ms | optim: 23.8ms step 1740 | loss: 2.936384 | lr 1.8000e-04 | norm: 0.1638 | dt: 14502.99ms | tok/sec: 36150.3 | mem: 107563MB | Time Left: 01 hrs : 57 mins [TIMING] data: 7.7ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10864.2ms | clip: 0.0ms | optim: 23.8ms step 1741 | loss: 3.012782 | lr 1.8000e-04 | norm: 0.1868 | dt: 14496.82ms | tok/sec: 36165.7 | mem: 107563MB | Time Left: 01 hrs : 57 mins [TIMING] data: 7.5ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10860.9ms | clip: 0.0ms | optim: 23.8ms step 1742 | loss: 2.992661 | lr 1.8000e-04 | norm: 0.1710 | dt: 14514.31ms | tok/sec: 36122.1 | mem: 107563MB | Time Left: 01 hrs : 57 mins [TIMING] data: 7.6ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10875.7ms | clip: 0.0ms | optim: 23.9ms step 1743 | loss: 2.854587 | lr 1.8000e-04 | norm: 0.2259 | dt: 14499.40ms | tok/sec: 36159.3 | mem: 107563MB | Time Left: 01 hrs : 56 mins [TIMING] data: 7.3ms | fwd: 3578.0ms | loss: 13.1ms | bwd: 10863.4ms | clip: 0.0ms | optim: 23.9ms step 1744 | loss: 2.855500 | lr 1.8000e-04 | norm: 0.1920 | dt: 14508.60ms | tok/sec: 36136.4 | mem: 107563MB | Time Left: 01 hrs : 56 mins [TIMING] data: 7.3ms | fwd: 3586.9ms | loss: 13.2ms | bwd: 10863.7ms | clip: 0.0ms | optim: 23.9ms step 1745 | loss: 3.021560 | lr 1.8000e-04 | norm: 0.2033 | dt: 14494.97ms | tok/sec: 36170.3 | mem: 107563MB | Time Left: 01 hrs : 56 mins [TIMING] data: 7.3ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10857.3ms | clip: 0.0ms | optim: 23.8ms step 1746 | loss: 3.098256 | lr 1.8000e-04 | norm: 0.1605 | dt: 14512.52ms | tok/sec: 36126.6 | mem: 107563MB | Time Left: 01 hrs : 56 mins [TIMING] data: 7.6ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10870.1ms | clip: 0.0ms | optim: 23.8ms step 1747 | loss: 2.986973 | lr 1.8000e-04 | norm: 0.2097 | dt: 14505.41ms | tok/sec: 36144.3 | mem: 107563MB | Time Left: 01 hrs : 56 mins [TIMING] data: 7.6ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10865.3ms | clip: 0.0ms | optim: 23.8ms step 1748 | loss: 2.859750 | lr 1.8000e-04 | norm: 0.2263 | dt: 14508.80ms | tok/sec: 36135.9 | mem: 107563MB | Time Left: 01 hrs : 55 mins [TIMING] data: 7.7ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10870.1ms | clip: 0.0ms | optim: 24.1ms step 1749 | loss: 2.766629 | lr 1.8000e-04 | norm: 0.1837 | dt: 14514.39ms | tok/sec: 36121.9 | mem: 107563MB | Time Left: 01 hrs : 55 mins [TIMING] data: 7.6ms | fwd: 3585.7ms | loss: 13.1ms | bwd: 10870.3ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1750.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1650.ckpt validation loss: 3.0133 step 1750 | loss: 2.732269 | lr 1.8000e-04 | norm: 0.1908 | dt: 18976.87ms | tok/sec: 27627.7 | mem: 107563MB | Time Left: 02 hrs : 30 mins [TIMING] data: 7.8ms | fwd: 3571.5ms | loss: 13.1ms | bwd: 10792.3ms | clip: 0.0ms | optim: 23.9ms step 1751 | loss: 2.896583 | lr 1.8000e-04 | norm: 0.2125 | dt: 14447.96ms | tok/sec: 36288.0 | mem: 107563MB | Time Left: 01 hrs : 54 mins [TIMING] data: 8.4ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10811.6ms | clip: 0.0ms | optim: 23.9ms step 1752 | loss: 2.801067 | lr 1.8000e-04 | norm: 0.2040 | dt: 14462.53ms | tok/sec: 36251.5 | mem: 107563MB | Time Left: 01 hrs : 54 mins [TIMING] data: 7.6ms | fwd: 3572.0ms | loss: 13.1ms | bwd: 10832.2ms | clip: 0.0ms | optim: 23.9ms step 1753 | loss: 2.482529 | lr 1.8000e-04 | norm: 0.2147 | dt: 14480.75ms | tok/sec: 36205.9 | mem: 107563MB | Time Left: 01 hrs : 54 mins [TIMING] data: 7.7ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10843.2ms | clip: 0.0ms | optim: 23.9ms step 1754 | loss: 2.836965 | lr 1.8000e-04 | norm: 0.2010 | dt: 14473.15ms | tok/sec: 36224.9 | mem: 107563MB | Time Left: 01 hrs : 54 mins [TIMING] data: 7.5ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10833.3ms | clip: 0.0ms | optim: 23.8ms step 1755 | loss: 2.694579 | lr 1.8000e-04 | norm: 0.1682 | dt: 14487.88ms | tok/sec: 36188.1 | mem: 107563MB | Time Left: 01 hrs : 53 mins [TIMING] data: 7.6ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10846.7ms | clip: 0.0ms | optim: 23.7ms step 1756 | loss: 2.960959 | lr 1.8000e-04 | norm: 0.1902 | dt: 14494.80ms | tok/sec: 36170.8 | mem: 107563MB | Time Left: 01 hrs : 53 mins [TIMING] data: 7.7ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10854.5ms | clip: 0.0ms | optim: 23.9ms step 1757 | loss: 2.759419 | lr 1.8000e-04 | norm: 0.1993 | dt: 14484.87ms | tok/sec: 36195.6 | mem: 107563MB | Time Left: 01 hrs : 53 mins [TIMING] data: 7.7ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10846.4ms | clip: 0.0ms | optim: 23.9ms step 1758 | loss: 2.856746 | lr 1.8000e-04 | norm: 0.2284 | dt: 14491.78ms | tok/sec: 36178.3 | mem: 107563MB | Time Left: 01 hrs : 53 mins [TIMING] data: 7.6ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10853.1ms | clip: 0.0ms | optim: 23.9ms step 1759 | loss: 2.878489 | lr 1.8000e-04 | norm: 0.2126 | dt: 14511.80ms | tok/sec: 36128.4 | mem: 107563MB | Time Left: 01 hrs : 53 mins [TIMING] data: 7.7ms | fwd: 3586.2ms | loss: 13.1ms | bwd: 10866.6ms | clip: 0.0ms | optim: 23.8ms step 1760 | loss: 2.818038 | lr 1.8000e-04 | norm: 0.2321 | dt: 14505.77ms | tok/sec: 36143.4 | mem: 107563MB | Time Left: 01 hrs : 52 mins [TIMING] data: 7.7ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10864.7ms | clip: 0.0ms | optim: 24.0ms step 1761 | loss: 3.022625 | lr 1.8000e-04 | norm: 0.1733 | dt: 14499.55ms | tok/sec: 36158.9 | mem: 107563MB | Time Left: 01 hrs : 52 mins [TIMING] data: 7.7ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10857.0ms | clip: 0.0ms | optim: 24.0ms step 1762 | loss: 3.000825 | lr 1.8000e-04 | norm: 0.2547 | dt: 14500.44ms | tok/sec: 36156.7 | mem: 107563MB | Time Left: 01 hrs : 52 mins [TIMING] data: 7.6ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10859.2ms | clip: 0.0ms | optim: 23.8ms step 1763 | loss: 2.883297 | lr 1.8000e-04 | norm: 0.2145 | dt: 14508.03ms | tok/sec: 36137.8 | mem: 107563MB | Time Left: 01 hrs : 52 mins [TIMING] data: 7.7ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10866.3ms | clip: 0.0ms | optim: 23.8ms step 1764 | loss: 3.143959 | lr 1.8000e-04 | norm: 0.2896 | dt: 14498.05ms | tok/sec: 36162.7 | mem: 107563MB | Time Left: 01 hrs : 51 mins [TIMING] data: 7.6ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10858.6ms | clip: 0.0ms | optim: 23.8ms step 1765 | loss: 2.612845 | lr 1.8000e-04 | norm: 0.3482 | dt: 14504.55ms | tok/sec: 36146.4 | mem: 107563MB | Time Left: 01 hrs : 51 mins [TIMING] data: 7.6ms | fwd: 3585.7ms | loss: 13.1ms | bwd: 10860.7ms | clip: 0.0ms | optim: 23.9ms step 1766 | loss: 3.038323 | lr 1.8000e-04 | norm: 0.2661 | dt: 14509.24ms | tok/sec: 36134.8 | mem: 107563MB | Time Left: 01 hrs : 51 mins [TIMING] data: 7.6ms | fwd: 3585.2ms | loss: 13.1ms | bwd: 10865.7ms | clip: 0.0ms | optim: 23.9ms step 1767 | loss: 2.756742 | lr 1.8000e-04 | norm: 0.1971 | dt: 14506.17ms | tok/sec: 36142.4 | mem: 107563MB | Time Left: 01 hrs : 51 mins [TIMING] data: 7.6ms | fwd: 3583.4ms | loss: 13.1ms | bwd: 10864.5ms | clip: 0.0ms | optim: 23.9ms step 1768 | loss: 2.894229 | lr 1.8000e-04 | norm: 0.1999 | dt: 14503.12ms | tok/sec: 36150.0 | mem: 107563MB | Time Left: 01 hrs : 50 mins [TIMING] data: 7.7ms | fwd: 3583.9ms | loss: 13.1ms | bwd: 10860.4ms | clip: 0.0ms | optim: 23.8ms step 1769 | loss: 2.712419 | lr 1.8000e-04 | norm: 0.2310 | dt: 14502.67ms | tok/sec: 36151.1 | mem: 107563MB | Time Left: 01 hrs : 50 mins [TIMING] data: 7.7ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10861.8ms | clip: 0.0ms | optim: 23.9ms step 1770 | loss: 2.667759 | lr 1.8000e-04 | norm: 0.2279 | dt: 14493.72ms | tok/sec: 36173.5 | mem: 107563MB | Time Left: 01 hrs : 50 mins [TIMING] data: 7.8ms | fwd: 3579.9ms | loss: 13.2ms | bwd: 10855.2ms | clip: 0.0ms | optim: 23.9ms step 1771 | loss: 3.084964 | lr 1.8000e-04 | norm: 0.1936 | dt: 14502.39ms | tok/sec: 36151.8 | mem: 107563MB | Time Left: 01 hrs : 50 mins [TIMING] data: 7.5ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10861.2ms | clip: 0.0ms | optim: 23.8ms step 1772 | loss: 2.997502 | lr 1.8000e-04 | norm: 0.2589 | dt: 14485.08ms | tok/sec: 36195.0 | mem: 107563MB | Time Left: 01 hrs : 49 mins [TIMING] data: 7.6ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10848.9ms | clip: 0.0ms | optim: 23.9ms step 1773 | loss: 2.721931 | lr 1.8000e-04 | norm: 0.1869 | dt: 14494.02ms | tok/sec: 36172.7 | mem: 107563MB | Time Left: 01 hrs : 49 mins [TIMING] data: 7.6ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.8ms step 1774 | loss: 2.464392 | lr 1.8000e-04 | norm: 0.1770 | dt: 14511.66ms | tok/sec: 36128.7 | mem: 107563MB | Time Left: 01 hrs : 49 mins [TIMING] data: 7.8ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10871.6ms | clip: 0.0ms | optim: 23.9ms step 1775 | loss: 2.862517 | lr 1.8000e-04 | norm: 0.2575 | dt: 14513.62ms | tok/sec: 36123.9 | mem: 107563MB | Time Left: 01 hrs : 49 mins [TIMING] data: 7.6ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10872.1ms | clip: 0.0ms | optim: 23.9ms step 1776 | loss: 2.716624 | lr 1.8000e-04 | norm: 0.1895 | dt: 14512.59ms | tok/sec: 36126.4 | mem: 107563MB | Time Left: 01 hrs : 49 mins [TIMING] data: 7.6ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10871.8ms | clip: 0.0ms | optim: 23.8ms step 1777 | loss: 3.038326 | lr 1.8000e-04 | norm: 0.1801 | dt: 14497.45ms | tok/sec: 36164.2 | mem: 107563MB | Time Left: 01 hrs : 48 mins [TIMING] data: 7.6ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10857.3ms | clip: 0.0ms | optim: 23.9ms step 1778 | loss: 3.084159 | lr 1.8000e-04 | norm: 0.1938 | dt: 14516.94ms | tok/sec: 36115.6 | mem: 107563MB | Time Left: 01 hrs : 48 mins [TIMING] data: 7.7ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10879.4ms | clip: 0.0ms | optim: 23.9ms step 1779 | loss: 3.027243 | lr 1.8000e-04 | norm: 0.3110 | dt: 14510.84ms | tok/sec: 36130.8 | mem: 107563MB | Time Left: 01 hrs : 48 mins [TIMING] data: 7.7ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10868.3ms | clip: 0.0ms | optim: 23.9ms step 1780 | loss: 2.531160 | lr 1.8000e-04 | norm: 0.1880 | dt: 14505.98ms | tok/sec: 36142.9 | mem: 107563MB | Time Left: 01 hrs : 48 mins [TIMING] data: 8.3ms | fwd: 3582.4ms | loss: 13.0ms | bwd: 10864.7ms | clip: 0.0ms | optim: 23.9ms step 1781 | loss: 2.888412 | lr 1.8000e-04 | norm: 0.2132 | dt: 14510.98ms | tok/sec: 36130.4 | mem: 107563MB | Time Left: 01 hrs : 47 mins [TIMING] data: 7.7ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10873.3ms | clip: 0.0ms | optim: 23.9ms step 1782 | loss: 2.797989 | lr 1.8000e-04 | norm: 0.2002 | dt: 14507.75ms | tok/sec: 36138.5 | mem: 107563MB | Time Left: 01 hrs : 47 mins [TIMING] data: 7.6ms | fwd: 3585.6ms | loss: 13.1ms | bwd: 10863.8ms | clip: 0.0ms | optim: 23.9ms step 1783 | loss: 2.938617 | lr 1.8000e-04 | norm: 0.1980 | dt: 14504.62ms | tok/sec: 36146.3 | mem: 107563MB | Time Left: 01 hrs : 47 mins [TIMING] data: 7.6ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10865.0ms | clip: 0.0ms | optim: 23.9ms step 1784 | loss: 2.573659 | lr 1.8000e-04 | norm: 0.1816 | dt: 14495.33ms | tok/sec: 36169.4 | mem: 107563MB | Time Left: 01 hrs : 47 mins [TIMING] data: 7.6ms | fwd: 3581.3ms | loss: 13.1ms | bwd: 10855.7ms | clip: 0.0ms | optim: 23.9ms step 1785 | loss: 2.867684 | lr 1.8000e-04 | norm: 0.3582 | dt: 14509.53ms | tok/sec: 36134.0 | mem: 107563MB | Time Left: 01 hrs : 46 mins [TIMING] data: 7.7ms | fwd: 3587.3ms | loss: 13.1ms | bwd: 10863.7ms | clip: 0.0ms | optim: 23.8ms step 1786 | loss: 2.701043 | lr 1.8000e-04 | norm: 0.1711 | dt: 14501.84ms | tok/sec: 36153.2 | mem: 107563MB | Time Left: 01 hrs : 46 mins [TIMING] data: 7.6ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10862.8ms | clip: 0.0ms | optim: 23.8ms step 1787 | loss: 3.007004 | lr 1.8000e-04 | norm: 0.3772 | dt: 14489.41ms | tok/sec: 36184.2 | mem: 107563MB | Time Left: 01 hrs : 46 mins [TIMING] data: 7.7ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms step 1788 | loss: 2.897203 | lr 1.8000e-04 | norm: 0.1671 | dt: 14499.03ms | tok/sec: 36160.2 | mem: 107563MB | Time Left: 01 hrs : 46 mins [TIMING] data: 7.6ms | fwd: 3582.6ms | loss: 13.0ms | bwd: 10857.2ms | clip: 0.0ms | optim: 23.9ms step 1789 | loss: 2.731340 | lr 1.8000e-04 | norm: 0.2013 | dt: 14494.32ms | tok/sec: 36172.0 | mem: 107563MB | Time Left: 01 hrs : 45 mins [TIMING] data: 7.6ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10858.3ms | clip: 0.0ms | optim: 23.8ms step 1790 | loss: 3.156133 | lr 1.8000e-04 | norm: 0.2369 | dt: 14498.28ms | tok/sec: 36162.1 | mem: 107563MB | Time Left: 01 hrs : 45 mins [TIMING] data: 7.8ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10858.6ms | clip: 0.0ms | optim: 23.8ms step 1791 | loss: 2.673363 | lr 1.8000e-04 | norm: 0.2050 | dt: 14491.14ms | tok/sec: 36179.9 | mem: 107563MB | Time Left: 01 hrs : 45 mins [TIMING] data: 7.5ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10855.1ms | clip: 0.0ms | optim: 23.8ms step 1792 | loss: 2.870837 | lr 1.8000e-04 | norm: 0.1973 | dt: 14494.32ms | tok/sec: 36172.0 | mem: 107563MB | Time Left: 01 hrs : 45 mins [TIMING] data: 7.7ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10856.6ms | clip: 0.0ms | optim: 24.0ms step 1793 | loss: 2.906965 | lr 1.8000e-04 | norm: 0.2081 | dt: 14485.99ms | tok/sec: 36192.8 | mem: 107563MB | Time Left: 01 hrs : 44 mins [TIMING] data: 7.6ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10850.5ms | clip: 0.0ms | optim: 23.9ms step 1794 | loss: 2.890540 | lr 1.8000e-04 | norm: 0.1878 | dt: 14498.24ms | tok/sec: 36162.2 | mem: 107563MB | Time Left: 01 hrs : 44 mins [TIMING] data: 7.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10858.5ms | clip: 0.0ms | optim: 23.8ms step 1795 | loss: 2.896785 | lr 1.8000e-04 | norm: 0.2185 | dt: 14483.24ms | tok/sec: 36199.6 | mem: 107563MB | Time Left: 01 hrs : 44 mins [TIMING] data: 7.6ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10848.0ms | clip: 0.0ms | optim: 24.1ms step 1796 | loss: 2.720608 | lr 1.8000e-04 | norm: 0.1821 | dt: 14488.66ms | tok/sec: 36186.1 | mem: 107563MB | Time Left: 01 hrs : 44 mins [TIMING] data: 7.6ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10847.7ms | clip: 0.0ms | optim: 23.9ms step 1797 | loss: 3.114815 | lr 1.8000e-04 | norm: 0.2268 | dt: 14484.79ms | tok/sec: 36195.8 | mem: 107563MB | Time Left: 01 hrs : 43 mins [TIMING] data: 8.6ms | fwd: 3583.5ms | loss: 13.1ms | bwd: 10841.6ms | clip: 0.0ms | optim: 23.8ms step 1798 | loss: 2.783428 | lr 1.8000e-04 | norm: 0.1787 | dt: 14487.06ms | tok/sec: 36190.1 | mem: 107563MB | Time Left: 01 hrs : 43 mins [TIMING] data: 7.6ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10845.0ms | clip: 0.0ms | optim: 23.9ms step 1799 | loss: 2.591512 | lr 1.8000e-04 | norm: 0.2082 | dt: 14488.32ms | tok/sec: 36186.9 | mem: 107563MB | Time Left: 01 hrs : 43 mins [TIMING] data: 7.5ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1800.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1700.ckpt step 1800 | loss: 2.950590 | lr 1.8000e-04 | norm: 0.2385 | dt: 14395.35ms | tok/sec: 36420.7 | mem: 107563MB | Time Left: 01 hrs : 42 mins [TIMING] data: 7.7ms | fwd: 3575.6ms | loss: 13.2ms | bwd: 10760.4ms | clip: 0.0ms | optim: 24.4ms step 1801 | loss: 2.925609 | lr 1.8000e-04 | norm: 0.2263 | dt: 14417.62ms | tok/sec: 36364.4 | mem: 107563MB | Time Left: 01 hrs : 42 mins [TIMING] data: 8.2ms | fwd: 3572.5ms | loss: 13.2ms | bwd: 10783.6ms | clip: 0.0ms | optim: 25.2ms step 1802 | loss: 2.850598 | lr 1.8000e-04 | norm: 0.1865 | dt: 14429.82ms | tok/sec: 36333.7 | mem: 107563MB | Time Left: 01 hrs : 42 mins [TIMING] data: 7.5ms | fwd: 3572.3ms | loss: 13.2ms | bwd: 10798.8ms | clip: 0.0ms | optim: 23.8ms step 1803 | loss: 3.032801 | lr 1.8000e-04 | norm: 0.1795 | dt: 14450.22ms | tok/sec: 36282.4 | mem: 107563MB | Time Left: 01 hrs : 42 mins [TIMING] data: 7.5ms | fwd: 3571.5ms | loss: 13.1ms | bwd: 10820.5ms | clip: 0.0ms | optim: 23.9ms step 1804 | loss: 2.887400 | lr 1.8000e-04 | norm: 0.1759 | dt: 14468.37ms | tok/sec: 36236.9 | mem: 107563MB | Time Left: 01 hrs : 42 mins [TIMING] data: 7.5ms | fwd: 3576.5ms | loss: 13.1ms | bwd: 10833.9ms | clip: 0.0ms | optim: 23.8ms step 1805 | loss: 3.035709 | lr 1.8000e-04 | norm: 0.1897 | dt: 14481.76ms | tok/sec: 36203.3 | mem: 107563MB | Time Left: 01 hrs : 41 mins [TIMING] data: 7.5ms | fwd: 3582.6ms | loss: 13.1ms | bwd: 10841.0ms | clip: 0.0ms | optim: 23.8ms step 1806 | loss: 2.847904 | lr 1.8000e-04 | norm: 0.2253 | dt: 14469.51ms | tok/sec: 36234.0 | mem: 107563MB | Time Left: 01 hrs : 41 mins [TIMING] data: 7.5ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10830.7ms | clip: 0.0ms | optim: 23.9ms step 1807 | loss: 2.879552 | lr 1.8000e-04 | norm: 0.1992 | dt: 14475.40ms | tok/sec: 36219.2 | mem: 107563MB | Time Left: 01 hrs : 41 mins [TIMING] data: 7.6ms | fwd: 3575.6ms | loss: 13.1ms | bwd: 10840.6ms | clip: 0.0ms | optim: 24.9ms step 1808 | loss: 2.777458 | lr 1.8000e-04 | norm: 0.2756 | dt: 14479.65ms | tok/sec: 36208.6 | mem: 107563MB | Time Left: 01 hrs : 41 mins [TIMING] data: 7.4ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10841.3ms | clip: 0.0ms | optim: 23.9ms step 1809 | loss: 2.765084 | lr 1.8000e-04 | norm: 0.1984 | dt: 14476.46ms | tok/sec: 36216.6 | mem: 107563MB | Time Left: 01 hrs : 40 mins [TIMING] data: 7.5ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10838.2ms | clip: 0.0ms | optim: 23.8ms step 1810 | loss: 2.574621 | lr 1.8000e-04 | norm: 0.2082 | dt: 14485.34ms | tok/sec: 36194.4 | mem: 107563MB | Time Left: 01 hrs : 40 mins [TIMING] data: 7.5ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10848.8ms | clip: 0.0ms | optim: 24.0ms step 1811 | loss: 2.672822 | lr 1.8000e-04 | norm: 0.1571 | dt: 14486.90ms | tok/sec: 36190.5 | mem: 107563MB | Time Left: 01 hrs : 40 mins [TIMING] data: 7.5ms | fwd: 3578.8ms | loss: 13.2ms | bwd: 10849.7ms | clip: 0.0ms | optim: 23.9ms step 1812 | loss: 2.890911 | lr 1.8000e-04 | norm: 0.2124 | dt: 14479.58ms | tok/sec: 36208.8 | mem: 107563MB | Time Left: 01 hrs : 40 mins [TIMING] data: 7.5ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10843.0ms | clip: 0.0ms | optim: 23.8ms step 1813 | loss: 2.613259 | lr 1.8000e-04 | norm: 0.2100 | dt: 14502.10ms | tok/sec: 36152.6 | mem: 107563MB | Time Left: 01 hrs : 40 mins [TIMING] data: 7.6ms | fwd: 3581.7ms | loss: 13.2ms | bwd: 10861.7ms | clip: 0.0ms | optim: 24.1ms step 1814 | loss: 2.946828 | lr 1.8000e-04 | norm: 0.2135 | dt: 14492.83ms | tok/sec: 36175.7 | mem: 107563MB | Time Left: 01 hrs : 39 mins [TIMING] data: 7.6ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10858.8ms | clip: 0.0ms | optim: 24.0ms step 1815 | loss: 2.788577 | lr 1.8000e-04 | norm: 0.1852 | dt: 14479.54ms | tok/sec: 36208.9 | mem: 107563MB | Time Left: 01 hrs : 39 mins [TIMING] data: 7.7ms | fwd: 3580.3ms | loss: 13.2ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.9ms step 1816 | loss: 2.849288 | lr 1.8000e-04 | norm: 0.1679 | dt: 14488.38ms | tok/sec: 36186.8 | mem: 107563MB | Time Left: 01 hrs : 39 mins [TIMING] data: 7.5ms | fwd: 3577.0ms | loss: 13.1ms | bwd: 10853.1ms | clip: 0.0ms | optim: 23.8ms step 1817 | loss: 2.936029 | lr 1.8000e-04 | norm: 0.2276 | dt: 14487.63ms | tok/sec: 36188.7 | mem: 107563MB | Time Left: 01 hrs : 38 mins [TIMING] data: 7.5ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10849.7ms | clip: 0.0ms | optim: 23.9ms step 1818 | loss: 2.696410 | lr 1.8000e-04 | norm: 0.1814 | dt: 14491.23ms | tok/sec: 36179.7 | mem: 107563MB | Time Left: 01 hrs : 38 mins [TIMING] data: 7.6ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10855.3ms | clip: 0.0ms | optim: 23.9ms step 1819 | loss: 2.732376 | lr 1.8000e-04 | norm: 0.1815 | dt: 14486.72ms | tok/sec: 36190.9 | mem: 107563MB | Time Left: 01 hrs : 38 mins [TIMING] data: 7.5ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 1820 | loss: 2.764585 | lr 1.8000e-04 | norm: 0.2157 | dt: 14488.29ms | tok/sec: 36187.0 | mem: 107563MB | Time Left: 01 hrs : 38 mins [TIMING] data: 7.6ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10848.1ms | clip: 0.0ms | optim: 23.9ms step 1821 | loss: 2.617878 | lr 1.8000e-04 | norm: 0.2102 | dt: 14481.88ms | tok/sec: 36203.0 | mem: 107563MB | Time Left: 01 hrs : 37 mins [TIMING] data: 7.6ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10848.8ms | clip: 0.0ms | optim: 23.8ms step 1822 | loss: 2.425917 | lr 1.8000e-04 | norm: 0.1507 | dt: 14491.61ms | tok/sec: 36178.7 | mem: 107563MB | Time Left: 01 hrs : 37 mins [TIMING] data: 7.5ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.8ms step 1823 | loss: 3.028234 | lr 1.8000e-04 | norm: 0.1695 | dt: 14487.07ms | tok/sec: 36190.1 | mem: 107563MB | Time Left: 01 hrs : 37 mins [TIMING] data: 7.5ms | fwd: 3581.2ms | loss: 13.2ms | bwd: 10847.5ms | clip: 0.0ms | optim: 23.8ms step 1824 | loss: 3.124362 | lr 1.8000e-04 | norm: 0.2119 | dt: 14496.55ms | tok/sec: 36166.4 | mem: 107563MB | Time Left: 01 hrs : 37 mins [TIMING] data: 7.5ms | fwd: 3583.8ms | loss: 13.1ms | bwd: 10854.6ms | clip: 0.0ms | optim: 23.8ms step 1825 | loss: 2.690174 | lr 1.8000e-04 | norm: 0.3880 | dt: 14479.69ms | tok/sec: 36208.5 | mem: 107563MB | Time Left: 01 hrs : 37 mins [TIMING] data: 7.5ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10839.7ms | clip: 0.0ms | optim: 23.8ms step 1826 | loss: 2.824103 | lr 1.8000e-04 | norm: 0.1802 | dt: 14478.49ms | tok/sec: 36211.5 | mem: 107563MB | Time Left: 01 hrs : 36 mins [TIMING] data: 7.5ms | fwd: 3575.4ms | loss: 13.1ms | bwd: 10844.7ms | clip: 0.0ms | optim: 23.9ms step 1827 | loss: 2.686905 | lr 1.8000e-04 | norm: 0.1770 | dt: 14476.55ms | tok/sec: 36216.4 | mem: 107563MB | Time Left: 01 hrs : 36 mins [TIMING] data: 7.7ms | fwd: 3575.6ms | loss: 13.1ms | bwd: 10842.6ms | clip: 0.0ms | optim: 23.9ms step 1828 | loss: 2.959534 | lr 1.8000e-04 | norm: 0.1826 | dt: 14484.33ms | tok/sec: 36196.9 | mem: 107563MB | Time Left: 01 hrs : 36 mins [TIMING] data: 7.5ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10843.1ms | clip: 0.0ms | optim: 23.9ms step 1829 | loss: 2.891970 | lr 1.8000e-04 | norm: 0.2145 | dt: 14481.84ms | tok/sec: 36203.1 | mem: 107563MB | Time Left: 01 hrs : 36 mins [TIMING] data: 7.7ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10843.3ms | clip: 0.0ms | optim: 24.0ms step 1830 | loss: 2.885509 | lr 1.8000e-04 | norm: 0.2010 | dt: 14482.65ms | tok/sec: 36201.1 | mem: 107563MB | Time Left: 01 hrs : 35 mins [TIMING] data: 7.5ms | fwd: 3577.0ms | loss: 13.1ms | bwd: 10847.4ms | clip: 0.0ms | optim: 23.9ms step 1831 | loss: 3.141218 | lr 1.8000e-04 | norm: 0.1915 | dt: 14487.68ms | tok/sec: 36188.5 | mem: 107563MB | Time Left: 01 hrs : 35 mins [TIMING] data: 7.5ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10846.9ms | clip: 0.0ms | optim: 24.0ms step 1832 | loss: 2.721959 | lr 1.8000e-04 | norm: 0.1655 | dt: 14490.94ms | tok/sec: 36180.4 | mem: 107563MB | Time Left: 01 hrs : 35 mins [TIMING] data: 7.5ms | fwd: 3582.2ms | loss: 13.0ms | bwd: 10850.6ms | clip: 0.0ms | optim: 23.8ms step 1833 | loss: 2.705161 | lr 1.8000e-04 | norm: 0.1779 | dt: 14488.41ms | tok/sec: 36186.7 | mem: 107563MB | Time Left: 01 hrs : 35 mins [TIMING] data: 7.5ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10852.5ms | clip: 0.0ms | optim: 23.9ms step 1834 | loss: 3.035002 | lr 1.8000e-04 | norm: 0.2117 | dt: 14490.16ms | tok/sec: 36182.3 | mem: 107563MB | Time Left: 01 hrs : 34 mins [TIMING] data: 8.1ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms step 1835 | loss: 2.777864 | lr 1.8000e-04 | norm: 0.1798 | dt: 14494.71ms | tok/sec: 36171.0 | mem: 107563MB | Time Left: 01 hrs : 34 mins [TIMING] data: 7.6ms | fwd: 3574.3ms | loss: 13.2ms | bwd: 10862.0ms | clip: 0.0ms | optim: 24.0ms step 1836 | loss: 2.792088 | lr 1.8000e-04 | norm: 0.1796 | dt: 14489.70ms | tok/sec: 36183.5 | mem: 107563MB | Time Left: 01 hrs : 34 mins [TIMING] data: 7.5ms | fwd: 3582.1ms | loss: 13.0ms | bwd: 10848.5ms | clip: 0.0ms | optim: 23.8ms step 1837 | loss: 2.833462 | lr 1.8000e-04 | norm: 0.1708 | dt: 14479.01ms | tok/sec: 36210.2 | mem: 107563MB | Time Left: 01 hrs : 34 mins [TIMING] data: 7.6ms | fwd: 3577.1ms | loss: 13.1ms | bwd: 10843.5ms | clip: 0.0ms | optim: 23.8ms step 1838 | loss: 2.898079 | lr 1.8000e-04 | norm: 0.1957 | dt: 14485.30ms | tok/sec: 36194.5 | mem: 107563MB | Time Left: 01 hrs : 33 mins [TIMING] data: 7.6ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10844.8ms | clip: 0.0ms | optim: 23.9ms step 1839 | loss: 2.738356 | lr 1.8000e-04 | norm: 0.1913 | dt: 14488.10ms | tok/sec: 36187.5 | mem: 107563MB | Time Left: 01 hrs : 33 mins [TIMING] data: 8.0ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10850.9ms | clip: 0.0ms | optim: 23.8ms step 1840 | loss: 2.783014 | lr 1.8000e-04 | norm: 0.1737 | dt: 14491.16ms | tok/sec: 36179.8 | mem: 107563MB | Time Left: 01 hrs : 33 mins [TIMING] data: 7.5ms | fwd: 3582.4ms | loss: 13.2ms | bwd: 10850.5ms | clip: 0.0ms | optim: 23.9ms step 1841 | loss: 3.005477 | lr 1.8000e-04 | norm: 0.2063 | dt: 14489.93ms | tok/sec: 36182.9 | mem: 107563MB | Time Left: 01 hrs : 33 mins [TIMING] data: 7.7ms | fwd: 3582.2ms | loss: 13.2ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.8ms step 1842 | loss: 2.738539 | lr 1.8000e-04 | norm: 0.1958 | dt: 14488.59ms | tok/sec: 36186.3 | mem: 107563MB | Time Left: 01 hrs : 32 mins [TIMING] data: 7.6ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10851.5ms | clip: 0.0ms | optim: 23.9ms step 1843 | loss: 2.999081 | lr 1.8000e-04 | norm: 0.1768 | dt: 14486.16ms | tok/sec: 36192.3 | mem: 107563MB | Time Left: 01 hrs : 32 mins [TIMING] data: 7.5ms | fwd: 3574.6ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 24.0ms step 1844 | loss: 3.007643 | lr 1.8000e-04 | norm: 0.1667 | dt: 14490.48ms | tok/sec: 36181.5 | mem: 107563MB | Time Left: 01 hrs : 32 mins [TIMING] data: 7.5ms | fwd: 3574.7ms | loss: 13.1ms | bwd: 10857.8ms | clip: 0.0ms | optim: 23.8ms step 1845 | loss: 2.811353 | lr 1.8000e-04 | norm: 0.2544 | dt: 14492.48ms | tok/sec: 36176.6 | mem: 107563MB | Time Left: 01 hrs : 32 mins [TIMING] data: 7.4ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10858.7ms | clip: 0.0ms | optim: 23.9ms step 1846 | loss: 2.988854 | lr 1.8000e-04 | norm: 0.1950 | dt: 14487.38ms | tok/sec: 36189.3 | mem: 107563MB | Time Left: 01 hrs : 31 mins [TIMING] data: 7.6ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10849.1ms | clip: 0.0ms | optim: 23.9ms step 1847 | loss: 2.677415 | lr 1.8000e-04 | norm: 0.1740 | dt: 14487.12ms | tok/sec: 36189.9 | mem: 107563MB | Time Left: 01 hrs : 31 mins [TIMING] data: 7.5ms | fwd: 3581.6ms | loss: 13.2ms | bwd: 10847.3ms | clip: 0.0ms | optim: 23.8ms step 1848 | loss: 3.072555 | lr 1.8000e-04 | norm: 0.1965 | dt: 14502.08ms | tok/sec: 36152.6 | mem: 107563MB | Time Left: 01 hrs : 31 mins [TIMING] data: 7.5ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10864.1ms | clip: 0.0ms | optim: 23.8ms step 1849 | loss: 2.654505 | lr 1.8000e-04 | norm: 0.1828 | dt: 14483.88ms | tok/sec: 36198.0 | mem: 107563MB | Time Left: 01 hrs : 31 mins [TIMING] data: 7.5ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10850.9ms | clip: 0.0ms | optim: 23.8ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1850.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1750.ckpt step 1850 | loss: 2.715991 | lr 1.8000e-04 | norm: 0.1725 | dt: 14503.38ms | tok/sec: 36149.4 | mem: 107563MB | Time Left: 01 hrs : 31 mins [TIMING] data: 7.6ms | fwd: 3678.4ms | loss: 13.1ms | bwd: 10766.4ms | clip: 0.0ms | optim: 23.8ms step 1851 | loss: 2.964001 | lr 1.8000e-04 | norm: 0.3033 | dt: 14415.13ms | tok/sec: 36370.7 | mem: 107563MB | Time Left: 01 hrs : 30 mins [TIMING] data: 7.9ms | fwd: 3564.4ms | loss: 13.1ms | bwd: 10791.7ms | clip: 0.0ms | optim: 23.9ms step 1852 | loss: 3.029999 | lr 1.8000e-04 | norm: 0.2486 | dt: 14441.02ms | tok/sec: 36305.5 | mem: 107563MB | Time Left: 01 hrs : 30 mins [TIMING] data: 7.7ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10806.8ms | clip: 0.0ms | optim: 23.8ms step 1853 | loss: 2.939839 | lr 1.8000e-04 | norm: 0.2297 | dt: 14457.11ms | tok/sec: 36265.1 | mem: 107563MB | Time Left: 01 hrs : 30 mins [TIMING] data: 7.6ms | fwd: 3572.4ms | loss: 13.1ms | bwd: 10826.4ms | clip: 0.0ms | optim: 23.8ms step 1854 | loss: 2.768924 | lr 1.8000e-04 | norm: 0.1970 | dt: 14456.80ms | tok/sec: 36265.8 | mem: 107563MB | Time Left: 01 hrs : 29 mins [TIMING] data: 7.4ms | fwd: 3575.5ms | loss: 13.2ms | bwd: 10822.9ms | clip: 0.0ms | optim: 23.9ms step 1855 | loss: 3.095932 | lr 1.8000e-04 | norm: 0.1879 | dt: 14465.22ms | tok/sec: 36244.7 | mem: 107563MB | Time Left: 01 hrs : 29 mins [TIMING] data: 7.6ms | fwd: 3570.5ms | loss: 13.1ms | bwd: 10836.3ms | clip: 0.0ms | optim: 23.8ms step 1856 | loss: 2.643928 | lr 1.8000e-04 | norm: 0.1755 | dt: 14477.02ms | tok/sec: 36215.2 | mem: 107563MB | Time Left: 01 hrs : 29 mins [TIMING] data: 7.5ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10842.6ms | clip: 0.0ms | optim: 23.9ms step 1857 | loss: 2.684528 | lr 1.8000e-04 | norm: 0.1932 | dt: 14473.03ms | tok/sec: 36225.2 | mem: 107563MB | Time Left: 01 hrs : 29 mins [TIMING] data: 7.5ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10838.2ms | clip: 0.0ms | optim: 23.9ms step 1858 | loss: 2.615297 | lr 1.8000e-04 | norm: 0.2201 | dt: 14491.57ms | tok/sec: 36178.8 | mem: 107563MB | Time Left: 01 hrs : 29 mins [TIMING] data: 7.6ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.9ms step 1859 | loss: 2.856015 | lr 1.8000e-04 | norm: 0.2550 | dt: 14484.51ms | tok/sec: 36196.5 | mem: 107563MB | Time Left: 01 hrs : 28 mins [TIMING] data: 7.5ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10848.7ms | clip: 0.0ms | optim: 23.8ms step 1860 | loss: 2.861140 | lr 1.8000e-04 | norm: 0.1959 | dt: 14488.52ms | tok/sec: 36186.5 | mem: 107563MB | Time Left: 01 hrs : 28 mins [TIMING] data: 7.6ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10852.4ms | clip: 0.0ms | optim: 23.8ms step 1861 | loss: 2.830213 | lr 1.8000e-04 | norm: 0.1860 | dt: 14493.06ms | tok/sec: 36175.1 | mem: 107563MB | Time Left: 01 hrs : 28 mins [TIMING] data: 7.5ms | fwd: 3582.5ms | loss: 13.2ms | bwd: 10852.2ms | clip: 0.0ms | optim: 23.9ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00010.bin 100000000 step 1862 | loss: 3.095912 | lr 1.8000e-04 | norm: 0.2079 | dt: 14501.59ms | tok/sec: 36153.8 | mem: 107563MB | Time Left: 01 hrs : 28 mins [TIMING] data: 13.1ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10855.6ms | clip: 0.0ms | optim: 24.0ms step 1863 | loss: 2.770103 | lr 1.8000e-04 | norm: 0.1780 | dt: 14479.39ms | tok/sec: 36209.3 | mem: 107563MB | Time Left: 01 hrs : 27 mins [TIMING] data: 7.6ms | fwd: 3584.2ms | loss: 13.1ms | bwd: 10836.8ms | clip: 0.0ms | optim: 23.8ms step 1864 | loss: 2.836421 | lr 1.8000e-04 | norm: 0.1920 | dt: 14497.98ms | tok/sec: 36162.8 | mem: 107563MB | Time Left: 01 hrs : 27 mins [TIMING] data: 7.6ms | fwd: 3584.0ms | loss: 13.1ms | bwd: 10855.6ms | clip: 0.0ms | optim: 23.9ms step 1865 | loss: 2.562476 | lr 1.8000e-04 | norm: 0.2123 | dt: 14489.01ms | tok/sec: 36185.2 | mem: 107563MB | Time Left: 01 hrs : 27 mins [TIMING] data: 7.5ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10847.1ms | clip: 0.0ms | optim: 23.8ms step 1866 | loss: 2.686961 | lr 1.8000e-04 | norm: 0.2220 | dt: 14487.59ms | tok/sec: 36188.8 | mem: 107563MB | Time Left: 01 hrs : 27 mins [TIMING] data: 7.5ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 1867 | loss: 2.914601 | lr 1.8000e-04 | norm: 0.1686 | dt: 14481.87ms | tok/sec: 36203.1 | mem: 107563MB | Time Left: 01 hrs : 26 mins [TIMING] data: 7.6ms | fwd: 3575.6ms | loss: 13.1ms | bwd: 10847.6ms | clip: 0.0ms | optim: 23.9ms step 1868 | loss: 2.745337 | lr 1.8000e-04 | norm: 0.1577 | dt: 14493.31ms | tok/sec: 36174.5 | mem: 107563MB | Time Left: 01 hrs : 26 mins [TIMING] data: 7.6ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.8ms step 1869 | loss: 3.064931 | lr 1.8000e-04 | norm: 0.1721 | dt: 14490.75ms | tok/sec: 36180.9 | mem: 107563MB | Time Left: 01 hrs : 26 mins [TIMING] data: 7.6ms | fwd: 3584.0ms | loss: 13.1ms | bwd: 10848.3ms | clip: 0.0ms | optim: 23.9ms step 1870 | loss: 2.849917 | lr 1.8000e-04 | norm: 0.1792 | dt: 14491.06ms | tok/sec: 36180.1 | mem: 107563MB | Time Left: 01 hrs : 26 mins [TIMING] data: 7.6ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10853.3ms | clip: 0.0ms | optim: 23.9ms step 1871 | loss: 2.914339 | lr 1.8000e-04 | norm: 0.1818 | dt: 14490.77ms | tok/sec: 36180.8 | mem: 107563MB | Time Left: 01 hrs : 25 mins [TIMING] data: 7.5ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10857.3ms | clip: 0.0ms | optim: 23.8ms step 1872 | loss: 2.773583 | lr 1.8000e-04 | norm: 0.1703 | dt: 14481.75ms | tok/sec: 36203.4 | mem: 107563MB | Time Left: 01 hrs : 25 mins [TIMING] data: 7.5ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10845.0ms | clip: 0.0ms | optim: 23.9ms step 1873 | loss: 2.729106 | lr 1.8000e-04 | norm: 0.2468 | dt: 14474.06ms | tok/sec: 36222.6 | mem: 107563MB | Time Left: 01 hrs : 25 mins [TIMING] data: 8.1ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10837.9ms | clip: 0.0ms | optim: 23.9ms step 1874 | loss: 2.896062 | lr 1.8000e-04 | norm: 0.1907 | dt: 14482.44ms | tok/sec: 36201.6 | mem: 107563MB | Time Left: 01 hrs : 25 mins [TIMING] data: 7.6ms | fwd: 3575.0ms | loss: 13.1ms | bwd: 10849.1ms | clip: 0.0ms | optim: 23.9ms step 1875 | loss: 2.912103 | lr 1.8000e-04 | norm: 0.1893 | dt: 14484.94ms | tok/sec: 36195.4 | mem: 107563MB | Time Left: 01 hrs : 24 mins [TIMING] data: 7.5ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10846.1ms | clip: 0.0ms | optim: 23.8ms step 1876 | loss: 3.093455 | lr 1.8000e-04 | norm: 0.1796 | dt: 14476.61ms | tok/sec: 36216.2 | mem: 107563MB | Time Left: 01 hrs : 24 mins [TIMING] data: 7.6ms | fwd: 3580.8ms | loss: 13.2ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.8ms step 1877 | loss: 2.557891 | lr 1.8000e-04 | norm: 0.1853 | dt: 14480.51ms | tok/sec: 36206.5 | mem: 107563MB | Time Left: 01 hrs : 24 mins [TIMING] data: 7.5ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10846.0ms | clip: 0.0ms | optim: 23.9ms step 1878 | loss: 2.957745 | lr 1.8000e-04 | norm: 0.1858 | dt: 14503.68ms | tok/sec: 36148.6 | mem: 107563MB | Time Left: 01 hrs : 24 mins [TIMING] data: 7.5ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10865.7ms | clip: 0.0ms | optim: 23.8ms step 1879 | loss: 2.666543 | lr 1.8000e-04 | norm: 0.1725 | dt: 14492.66ms | tok/sec: 36176.1 | mem: 107563MB | Time Left: 01 hrs : 24 mins [TIMING] data: 7.4ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 23.9ms step 1880 | loss: 2.915489 | lr 1.8000e-04 | norm: 0.1979 | dt: 14489.39ms | tok/sec: 36184.3 | mem: 107563MB | Time Left: 01 hrs : 23 mins [TIMING] data: 7.6ms | fwd: 3575.4ms | loss: 13.2ms | bwd: 10855.6ms | clip: 0.0ms | optim: 23.8ms step 1881 | loss: 3.012342 | lr 1.8000e-04 | norm: 0.2016 | dt: 14481.03ms | tok/sec: 36205.2 | mem: 107563MB | Time Left: 01 hrs : 23 mins [TIMING] data: 7.6ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10842.5ms | clip: 0.0ms | optim: 23.8ms step 1882 | loss: 2.743228 | lr 1.8000e-04 | norm: 0.1774 | dt: 14476.46ms | tok/sec: 36216.6 | mem: 107563MB | Time Left: 01 hrs : 23 mins [TIMING] data: 7.5ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10835.6ms | clip: 0.0ms | optim: 24.0ms step 1883 | loss: 2.826159 | lr 1.8000e-04 | norm: 0.1900 | dt: 14493.08ms | tok/sec: 36175.0 | mem: 107563MB | Time Left: 01 hrs : 23 mins [TIMING] data: 7.5ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10856.9ms | clip: 0.0ms | optim: 23.9ms step 1884 | loss: 3.021414 | lr 1.8000e-04 | norm: 0.1742 | dt: 14495.69ms | tok/sec: 36168.6 | mem: 107563MB | Time Left: 01 hrs : 22 mins [TIMING] data: 8.3ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10859.9ms | clip: 0.0ms | optim: 23.9ms step 1885 | loss: 2.813432 | lr 1.8000e-04 | norm: 0.1611 | dt: 14486.32ms | tok/sec: 36191.9 | mem: 107563MB | Time Left: 01 hrs : 22 mins [TIMING] data: 7.6ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10851.5ms | clip: 0.0ms | optim: 23.9ms step 1886 | loss: 2.846250 | lr 1.8000e-04 | norm: 0.1917 | dt: 14489.69ms | tok/sec: 36183.5 | mem: 107563MB | Time Left: 01 hrs : 22 mins [TIMING] data: 7.5ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10851.0ms | clip: 0.0ms | optim: 24.0ms step 1887 | loss: 2.849790 | lr 1.8000e-04 | norm: 0.1650 | dt: 14476.51ms | tok/sec: 36216.5 | mem: 107563MB | Time Left: 01 hrs : 22 mins [TIMING] data: 7.7ms | fwd: 3577.2ms | loss: 13.2ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.8ms step 1888 | loss: 2.926909 | lr 1.8000e-04 | norm: 0.2050 | dt: 14494.17ms | tok/sec: 36172.3 | mem: 107563MB | Time Left: 01 hrs : 21 mins [TIMING] data: 7.6ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10854.9ms | clip: 0.0ms | optim: 23.8ms step 1889 | loss: 2.962643 | lr 1.8000e-04 | norm: 0.1692 | dt: 14496.14ms | tok/sec: 36167.4 | mem: 107563MB | Time Left: 01 hrs : 21 mins [TIMING] data: 7.5ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10855.2ms | clip: 0.0ms | optim: 23.8ms step 1890 | loss: 2.696980 | lr 1.8000e-04 | norm: 0.2122 | dt: 14471.84ms | tok/sec: 36228.1 | mem: 107563MB | Time Left: 01 hrs : 21 mins [TIMING] data: 7.6ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10837.4ms | clip: 0.0ms | optim: 23.9ms step 1891 | loss: 2.922843 | lr 1.8000e-04 | norm: 0.1711 | dt: 14494.30ms | tok/sec: 36172.0 | mem: 107563MB | Time Left: 01 hrs : 21 mins [TIMING] data: 7.4ms | fwd: 3581.4ms | loss: 13.2ms | bwd: 10853.7ms | clip: 0.0ms | optim: 23.8ms step 1892 | loss: 2.806426 | lr 1.8000e-04 | norm: 0.1542 | dt: 14483.27ms | tok/sec: 36199.6 | mem: 107563MB | Time Left: 01 hrs : 20 mins [TIMING] data: 7.5ms | fwd: 3575.2ms | loss: 13.1ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.8ms step 1893 | loss: 3.033097 | lr 1.8000e-04 | norm: 0.1662 | dt: 14476.54ms | tok/sec: 36216.4 | mem: 107563MB | Time Left: 01 hrs : 20 mins [TIMING] data: 7.4ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10840.5ms | clip: 0.0ms | optim: 23.8ms step 1894 | loss: 2.685775 | lr 1.8000e-04 | norm: 0.1940 | dt: 14484.75ms | tok/sec: 36195.9 | mem: 107563MB | Time Left: 01 hrs : 20 mins [TIMING] data: 7.5ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10846.4ms | clip: 0.0ms | optim: 23.9ms step 1895 | loss: 2.727896 | lr 1.8000e-04 | norm: 0.1648 | dt: 14474.20ms | tok/sec: 36222.2 | mem: 107563MB | Time Left: 01 hrs : 20 mins [TIMING] data: 7.8ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10832.5ms | clip: 0.0ms | optim: 23.9ms step 1896 | loss: 3.077616 | lr 1.8000e-04 | norm: 0.2465 | dt: 14478.43ms | tok/sec: 36211.7 | mem: 107563MB | Time Left: 01 hrs : 19 mins [TIMING] data: 7.5ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10841.8ms | clip: 0.0ms | optim: 23.9ms step 1897 | loss: 2.702003 | lr 1.8000e-04 | norm: 0.1671 | dt: 14484.92ms | tok/sec: 36195.4 | mem: 107563MB | Time Left: 01 hrs : 19 mins [TIMING] data: 7.5ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10850.0ms | clip: 0.0ms | optim: 23.9ms step 1898 | loss: 2.584465 | lr 1.8000e-04 | norm: 0.2064 | dt: 14508.14ms | tok/sec: 36137.5 | mem: 107563MB | Time Left: 01 hrs : 19 mins [TIMING] data: 7.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10868.4ms | clip: 0.0ms | optim: 23.9ms step 1899 | loss: 2.731491 | lr 1.8000e-04 | norm: 0.1900 | dt: 14493.10ms | tok/sec: 36175.0 | mem: 107563MB | Time Left: 01 hrs : 19 mins [TIMING] data: 7.5ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10857.4ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1900.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1800.ckpt step 1900 | loss: 2.879381 | lr 1.8000e-04 | norm: 0.1864 | dt: 14412.46ms | tok/sec: 36377.4 | mem: 107563MB | Time Left: 01 hrs : 18 mins [TIMING] data: 7.5ms | fwd: 3572.7ms | loss: 13.2ms | bwd: 10781.1ms | clip: 0.0ms | optim: 23.9ms step 1901 | loss: 2.707532 | lr 1.8000e-04 | norm: 0.1608 | dt: 14418.45ms | tok/sec: 36362.3 | mem: 107563MB | Time Left: 01 hrs : 18 mins [TIMING] data: 8.0ms | fwd: 3572.2ms | loss: 13.1ms | bwd: 10787.2ms | clip: 0.0ms | optim: 23.9ms step 1902 | loss: 3.122897 | lr 1.8000e-04 | norm: 0.1975 | dt: 14441.30ms | tok/sec: 36304.8 | mem: 107563MB | Time Left: 01 hrs : 18 mins [TIMING] data: 8.0ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10805.8ms | clip: 0.0ms | optim: 23.8ms step 1903 | loss: 2.800523 | lr 1.8000e-04 | norm: 0.1998 | dt: 14465.43ms | tok/sec: 36244.2 | mem: 107563MB | Time Left: 01 hrs : 18 mins [TIMING] data: 7.5ms | fwd: 3572.2ms | loss: 13.1ms | bwd: 10834.9ms | clip: 0.0ms | optim: 23.8ms step 1904 | loss: 2.755340 | lr 1.8000e-04 | norm: 0.1966 | dt: 14480.75ms | tok/sec: 36205.9 | mem: 107563MB | Time Left: 01 hrs : 17 mins [TIMING] data: 7.6ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10842.8ms | clip: 0.0ms | optim: 23.8ms step 1905 | loss: 2.631546 | lr 1.8000e-04 | norm: 0.1585 | dt: 14489.52ms | tok/sec: 36183.9 | mem: 107563MB | Time Left: 01 hrs : 17 mins [TIMING] data: 7.6ms | fwd: 3577.0ms | loss: 13.1ms | bwd: 10854.3ms | clip: 0.0ms | optim: 23.9ms step 1906 | loss: 2.930890 | lr 1.8000e-04 | norm: 0.1753 | dt: 14491.33ms | tok/sec: 36179.4 | mem: 107563MB | Time Left: 01 hrs : 17 mins [TIMING] data: 7.6ms | fwd: 3581.2ms | loss: 13.1ms | bwd: 10852.0ms | clip: 0.0ms | optim: 23.9ms step 1907 | loss: 2.965419 | lr 1.8000e-04 | norm: 0.2287 | dt: 14490.99ms | tok/sec: 36180.3 | mem: 107563MB | Time Left: 01 hrs : 17 mins [TIMING] data: 7.5ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10850.1ms | clip: 0.0ms | optim: 23.9ms step 1908 | loss: 3.074932 | lr 1.8000e-04 | norm: 0.2117 | dt: 14499.85ms | tok/sec: 36158.2 | mem: 107563MB | Time Left: 01 hrs : 17 mins [TIMING] data: 7.6ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10862.7ms | clip: 0.0ms | optim: 23.8ms step 1909 | loss: 3.064322 | lr 1.8000e-04 | norm: 0.1819 | dt: 14499.60ms | tok/sec: 36158.8 | mem: 107563MB | Time Left: 01 hrs : 16 mins [TIMING] data: 7.6ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10862.2ms | clip: 0.0ms | optim: 23.9ms step 1910 | loss: 2.739492 | lr 1.8000e-04 | norm: 0.2147 | dt: 14504.49ms | tok/sec: 36146.6 | mem: 107563MB | Time Left: 01 hrs : 16 mins [TIMING] data: 7.7ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10866.3ms | clip: 0.0ms | optim: 23.9ms step 1911 | loss: 2.864943 | lr 1.8000e-04 | norm: 0.1815 | dt: 14501.05ms | tok/sec: 36155.2 | mem: 107563MB | Time Left: 01 hrs : 16 mins [TIMING] data: 7.8ms | fwd: 3582.2ms | loss: 13.2ms | bwd: 10860.0ms | clip: 0.0ms | optim: 23.9ms step 1912 | loss: 2.935359 | lr 1.8000e-04 | norm: 0.1842 | dt: 14496.17ms | tok/sec: 36167.3 | mem: 107563MB | Time Left: 01 hrs : 16 mins [TIMING] data: 7.5ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10857.8ms | clip: 0.0ms | optim: 23.8ms step 1913 | loss: 3.050508 | lr 1.8000e-04 | norm: 0.2477 | dt: 14506.50ms | tok/sec: 36141.6 | mem: 107563MB | Time Left: 01 hrs : 15 mins [TIMING] data: 7.4ms | fwd: 3581.6ms | loss: 13.2ms | bwd: 10866.5ms | clip: 0.0ms | optim: 23.9ms step 1914 | loss: 2.972285 | lr 1.8000e-04 | norm: 0.1783 | dt: 14491.79ms | tok/sec: 36178.3 | mem: 107563MB | Time Left: 01 hrs : 15 mins [TIMING] data: 7.5ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10853.2ms | clip: 0.0ms | optim: 23.8ms step 1915 | loss: 2.794188 | lr 1.8000e-04 | norm: 0.1877 | dt: 14503.48ms | tok/sec: 36149.1 | mem: 107563MB | Time Left: 01 hrs : 15 mins [TIMING] data: 7.5ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10863.2ms | clip: 0.0ms | optim: 23.8ms step 1916 | loss: 2.581598 | lr 1.8000e-04 | norm: 0.1858 | dt: 14490.91ms | tok/sec: 36180.5 | mem: 107563MB | Time Left: 01 hrs : 15 mins [TIMING] data: 7.6ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.9ms step 1917 | loss: 2.887881 | lr 1.8000e-04 | norm: 0.2054 | dt: 14494.75ms | tok/sec: 36170.9 | mem: 107563MB | Time Left: 01 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10853.2ms | clip: 0.0ms | optim: 23.9ms step 1918 | loss: 2.913347 | lr 1.8000e-04 | norm: 0.1800 | dt: 14505.71ms | tok/sec: 36143.5 | mem: 107563MB | Time Left: 01 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3583.6ms | loss: 13.2ms | bwd: 10863.9ms | clip: 0.0ms | optim: 23.8ms step 1919 | loss: 2.811838 | lr 1.8000e-04 | norm: 0.1906 | dt: 14495.64ms | tok/sec: 36168.7 | mem: 107563MB | Time Left: 01 hrs : 14 mins [TIMING] data: 8.0ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10857.6ms | clip: 0.0ms | optim: 24.0ms step 1920 | loss: 2.645662 | lr 1.8000e-04 | norm: 0.1685 | dt: 14491.38ms | tok/sec: 36179.3 | mem: 107563MB | Time Left: 01 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10852.7ms | clip: 0.0ms | optim: 23.9ms step 1921 | loss: 2.857827 | lr 1.8000e-04 | norm: 0.2081 | dt: 14503.77ms | tok/sec: 36148.4 | mem: 107563MB | Time Left: 01 hrs : 13 mins [TIMING] data: 7.5ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10863.7ms | clip: 0.0ms | optim: 23.8ms step 1922 | loss: 2.772007 | lr 1.8000e-04 | norm: 0.1757 | dt: 14494.17ms | tok/sec: 36172.3 | mem: 107563MB | Time Left: 01 hrs : 13 mins [TIMING] data: 7.6ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10854.1ms | clip: 0.0ms | optim: 23.8ms step 1923 | loss: 2.905583 | lr 1.8000e-04 | norm: 0.1979 | dt: 14493.14ms | tok/sec: 36174.9 | mem: 107563MB | Time Left: 01 hrs : 13 mins [TIMING] data: 7.5ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10855.2ms | clip: 0.0ms | optim: 23.8ms step 1924 | loss: 2.887681 | lr 1.8000e-04 | norm: 0.1961 | dt: 14496.77ms | tok/sec: 36165.8 | mem: 107563MB | Time Left: 01 hrs : 13 mins [TIMING] data: 7.5ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.8ms step 1925 | loss: 2.940135 | lr 1.8000e-04 | norm: 0.1850 | dt: 14498.09ms | tok/sec: 36162.6 | mem: 107563MB | Time Left: 01 hrs : 12 mins [TIMING] data: 7.5ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10858.2ms | clip: 0.0ms | optim: 23.9ms step 1926 | loss: 2.952106 | lr 1.8000e-04 | norm: 0.1719 | dt: 14499.34ms | tok/sec: 36159.4 | mem: 107563MB | Time Left: 01 hrs : 12 mins [TIMING] data: 7.4ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10862.9ms | clip: 0.0ms | optim: 23.9ms step 1927 | loss: 2.921578 | lr 1.8000e-04 | norm: 0.1664 | dt: 14506.28ms | tok/sec: 36142.1 | mem: 107563MB | Time Left: 01 hrs : 12 mins [TIMING] data: 7.3ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10866.2ms | clip: 0.0ms | optim: 23.9ms step 1928 | loss: 2.902414 | lr 1.8000e-04 | norm: 0.1804 | dt: 14499.46ms | tok/sec: 36159.1 | mem: 107563MB | Time Left: 01 hrs : 12 mins [TIMING] data: 7.5ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10858.7ms | clip: 0.0ms | optim: 23.8ms step 1929 | loss: 2.768215 | lr 1.8000e-04 | norm: 0.1827 | dt: 14508.31ms | tok/sec: 36137.1 | mem: 107563MB | Time Left: 01 hrs : 12 mins [TIMING] data: 7.6ms | fwd: 3583.4ms | loss: 13.1ms | bwd: 10866.8ms | clip: 0.0ms | optim: 23.8ms step 1930 | loss: 2.728789 | lr 1.8000e-04 | norm: 0.1605 | dt: 14507.13ms | tok/sec: 36140.0 | mem: 107563MB | Time Left: 01 hrs : 11 mins [TIMING] data: 7.6ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10866.2ms | clip: 0.0ms | optim: 23.8ms step 1931 | loss: 3.036488 | lr 1.8000e-04 | norm: 0.1637 | dt: 14503.94ms | tok/sec: 36148.0 | mem: 107563MB | Time Left: 01 hrs : 11 mins [TIMING] data: 7.5ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10861.6ms | clip: 0.0ms | optim: 23.8ms step 1932 | loss: 2.864531 | lr 1.8000e-04 | norm: 0.1966 | dt: 14507.96ms | tok/sec: 36138.0 | mem: 107563MB | Time Left: 01 hrs : 11 mins [TIMING] data: 7.6ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10866.8ms | clip: 0.0ms | optim: 23.9ms step 1933 | loss: 2.892521 | lr 1.8000e-04 | norm: 0.1599 | dt: 14507.03ms | tok/sec: 36140.3 | mem: 107563MB | Time Left: 01 hrs : 11 mins [TIMING] data: 7.5ms | fwd: 3577.7ms | loss: 13.1ms | bwd: 10871.3ms | clip: 0.0ms | optim: 23.8ms step 1934 | loss: 2.736579 | lr 1.8000e-04 | norm: 0.1906 | dt: 14496.75ms | tok/sec: 36165.9 | mem: 107563MB | Time Left: 01 hrs : 10 mins [TIMING] data: 7.6ms | fwd: 3585.9ms | loss: 13.1ms | bwd: 10852.5ms | clip: 0.0ms | optim: 23.9ms step 1935 | loss: 2.731857 | lr 1.8000e-04 | norm: 0.2188 | dt: 14507.55ms | tok/sec: 36139.0 | mem: 107563MB | Time Left: 01 hrs : 10 mins [TIMING] data: 7.5ms | fwd: 3586.6ms | loss: 13.1ms | bwd: 10862.8ms | clip: 0.0ms | optim: 23.8ms step 1936 | loss: 2.772570 | lr 1.8000e-04 | norm: 0.2186 | dt: 14504.82ms | tok/sec: 36145.8 | mem: 107563MB | Time Left: 01 hrs : 10 mins [TIMING] data: 7.6ms | fwd: 3583.8ms | loss: 13.1ms | bwd: 10862.9ms | clip: 0.0ms | optim: 23.8ms step 1937 | loss: 2.751190 | lr 1.8000e-04 | norm: 0.1824 | dt: 14500.02ms | tok/sec: 36157.7 | mem: 107563MB | Time Left: 01 hrs : 10 mins [TIMING] data: 7.6ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10864.0ms | clip: 0.0ms | optim: 23.8ms step 1938 | loss: 2.483899 | lr 1.8000e-04 | norm: 0.1986 | dt: 14514.78ms | tok/sec: 36121.0 | mem: 107563MB | Time Left: 01 hrs : 09 mins [TIMING] data: 7.5ms | fwd: 3583.5ms | loss: 13.1ms | bwd: 10873.2ms | clip: 0.0ms | optim: 23.8ms step 1939 | loss: 3.021863 | lr 1.8000e-04 | norm: 0.2481 | dt: 14502.73ms | tok/sec: 36151.0 | mem: 107563MB | Time Left: 01 hrs : 09 mins [TIMING] data: 7.6ms | fwd: 3582.3ms | loss: 13.0ms | bwd: 10862.3ms | clip: 0.0ms | optim: 23.8ms step 1940 | loss: 2.814443 | lr 1.8000e-04 | norm: 0.1948 | dt: 14499.62ms | tok/sec: 36158.7 | mem: 107563MB | Time Left: 01 hrs : 09 mins [TIMING] data: 7.6ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10859.8ms | clip: 0.0ms | optim: 23.8ms step 1941 | loss: 2.807404 | lr 1.8000e-04 | norm: 0.1828 | dt: 14504.81ms | tok/sec: 36145.8 | mem: 107563MB | Time Left: 01 hrs : 09 mins [TIMING] data: 7.5ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10867.8ms | clip: 0.0ms | optim: 23.9ms step 1942 | loss: 3.071317 | lr 1.8000e-04 | norm: 0.2115 | dt: 14494.13ms | tok/sec: 36172.4 | mem: 107563MB | Time Left: 01 hrs : 08 mins [TIMING] data: 7.4ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10855.1ms | clip: 0.0ms | optim: 23.8ms step 1943 | loss: 2.886891 | lr 1.8000e-04 | norm: 0.1801 | dt: 14503.66ms | tok/sec: 36148.7 | mem: 107563MB | Time Left: 01 hrs : 08 mins [TIMING] data: 7.5ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10865.5ms | clip: 0.0ms | optim: 23.8ms step 1944 | loss: 2.722629 | lr 1.8000e-04 | norm: 0.2215 | dt: 14495.20ms | tok/sec: 36169.8 | mem: 107563MB | Time Left: 01 hrs : 08 mins [TIMING] data: 7.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10855.3ms | clip: 0.0ms | optim: 23.9ms step 1945 | loss: 2.769473 | lr 1.8000e-04 | norm: 0.2255 | dt: 14494.05ms | tok/sec: 36172.6 | mem: 107563MB | Time Left: 01 hrs : 08 mins [TIMING] data: 7.6ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10856.0ms | clip: 0.0ms | optim: 23.9ms step 1946 | loss: 3.105406 | lr 1.8000e-04 | norm: 0.1962 | dt: 14501.53ms | tok/sec: 36154.0 | mem: 107563MB | Time Left: 01 hrs : 07 mins [TIMING] data: 7.5ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10865.1ms | clip: 0.0ms | optim: 23.8ms step 1947 | loss: 2.723789 | lr 1.8000e-04 | norm: 0.1842 | dt: 14496.62ms | tok/sec: 36166.2 | mem: 107563MB | Time Left: 01 hrs : 07 mins [TIMING] data: 7.4ms | fwd: 3577.0ms | loss: 13.1ms | bwd: 10861.5ms | clip: 0.0ms | optim: 23.9ms step 1948 | loss: 2.634367 | lr 1.8000e-04 | norm: 0.1808 | dt: 14498.95ms | tok/sec: 36160.4 | mem: 107563MB | Time Left: 01 hrs : 07 mins [TIMING] data: 7.5ms | fwd: 3581.7ms | loss: 13.1ms | bwd: 10858.9ms | clip: 0.0ms | optim: 23.9ms step 1949 | loss: 2.928766 | lr 1.8000e-04 | norm: 0.1748 | dt: 14500.03ms | tok/sec: 36157.7 | mem: 107563MB | Time Left: 01 hrs : 07 mins [TIMING] data: 7.5ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10859.2ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1950.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1850.ckpt step 1950 | loss: 2.978698 | lr 1.8000e-04 | norm: 0.1856 | dt: 14412.45ms | tok/sec: 36377.4 | mem: 107563MB | Time Left: 01 hrs : 06 mins [TIMING] data: 7.8ms | fwd: 3575.7ms | loss: 13.2ms | bwd: 10777.8ms | clip: 0.0ms | optim: 23.8ms step 1951 | loss: 2.690041 | lr 1.8000e-04 | norm: 0.1787 | dt: 14442.13ms | tok/sec: 36302.7 | mem: 107563MB | Time Left: 01 hrs : 06 mins [TIMING] data: 8.9ms | fwd: 3578.7ms | loss: 13.2ms | bwd: 10802.7ms | clip: 0.0ms | optim: 23.9ms step 1952 | loss: 2.813731 | lr 1.8000e-04 | norm: 0.2112 | dt: 14462.26ms | tok/sec: 36252.2 | mem: 107563MB | Time Left: 01 hrs : 06 mins [TIMING] data: 7.6ms | fwd: 3573.3ms | loss: 13.2ms | bwd: 10830.5ms | clip: 0.0ms | optim: 23.9ms step 1953 | loss: 2.965660 | lr 1.8000e-04 | norm: 0.1791 | dt: 14464.68ms | tok/sec: 36246.1 | mem: 107563MB | Time Left: 01 hrs : 06 mins [TIMING] data: 7.5ms | fwd: 3577.3ms | loss: 13.2ms | bwd: 10829.1ms | clip: 0.0ms | optim: 23.8ms step 1954 | loss: 2.794764 | lr 1.8000e-04 | norm: 0.2011 | dt: 14478.67ms | tok/sec: 36211.1 | mem: 107563MB | Time Left: 01 hrs : 05 mins [TIMING] data: 7.9ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10840.3ms | clip: 0.0ms | optim: 23.8ms step 1955 | loss: 2.655722 | lr 1.8000e-04 | norm: 0.1758 | dt: 14485.97ms | tok/sec: 36192.8 | mem: 107563MB | Time Left: 01 hrs : 05 mins [TIMING] data: 7.9ms | fwd: 3576.4ms | loss: 13.1ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.8ms step 1956 | loss: 2.809147 | lr 1.8000e-04 | norm: 0.2038 | dt: 14486.41ms | tok/sec: 36191.7 | mem: 107563MB | Time Left: 01 hrs : 05 mins [TIMING] data: 8.5ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10850.6ms | clip: 0.0ms | optim: 23.8ms step 1957 | loss: 2.605677 | lr 1.8000e-04 | norm: 0.1605 | dt: 14485.78ms | tok/sec: 36193.3 | mem: 107563MB | Time Left: 01 hrs : 05 mins [TIMING] data: 7.5ms | fwd: 3583.1ms | loss: 13.1ms | bwd: 10844.4ms | clip: 0.0ms | optim: 23.8ms step 1958 | loss: 2.842559 | lr 1.8000e-04 | norm: 0.2130 | dt: 14498.25ms | tok/sec: 36162.2 | mem: 107563MB | Time Left: 01 hrs : 05 mins [TIMING] data: 7.7ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10858.4ms | clip: 0.0ms | optim: 23.9ms step 1959 | loss: 2.703764 | lr 1.8000e-04 | norm: 0.1855 | dt: 14498.82ms | tok/sec: 36160.7 | mem: 107563MB | Time Left: 01 hrs : 04 mins [TIMING] data: 7.9ms | fwd: 3576.1ms | loss: 13.1ms | bwd: 10864.1ms | clip: 0.0ms | optim: 23.8ms step 1960 | loss: 2.880764 | lr 1.8000e-04 | norm: 0.1740 | dt: 14503.69ms | tok/sec: 36148.6 | mem: 107563MB | Time Left: 01 hrs : 04 mins [TIMING] data: 7.7ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10863.4ms | clip: 0.0ms | optim: 23.9ms step 1961 | loss: 3.041982 | lr 1.8000e-04 | norm: 0.1796 | dt: 14506.75ms | tok/sec: 36141.0 | mem: 107563MB | Time Left: 01 hrs : 04 mins [TIMING] data: 7.6ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10866.4ms | clip: 0.0ms | optim: 23.9ms step 1962 | loss: 2.820318 | lr 1.8000e-04 | norm: 0.2019 | dt: 14488.36ms | tok/sec: 36186.8 | mem: 107563MB | Time Left: 01 hrs : 03 mins [TIMING] data: 7.5ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10848.0ms | clip: 0.0ms | optim: 24.0ms step 1963 | loss: 2.682416 | lr 1.8000e-04 | norm: 0.1913 | dt: 14512.40ms | tok/sec: 36126.9 | mem: 107563MB | Time Left: 01 hrs : 03 mins [TIMING] data: 7.7ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10873.7ms | clip: 0.0ms | optim: 23.9ms step 1964 | loss: 2.944715 | lr 1.8000e-04 | norm: 0.1788 | dt: 14505.65ms | tok/sec: 36143.7 | mem: 107563MB | Time Left: 01 hrs : 03 mins [TIMING] data: 7.6ms | fwd: 3582.4ms | loss: 13.1ms | bwd: 10865.0ms | clip: 0.0ms | optim: 23.9ms step 1965 | loss: 2.979285 | lr 1.8000e-04 | norm: 0.1829 | dt: 14511.19ms | tok/sec: 36129.9 | mem: 107563MB | Time Left: 01 hrs : 03 mins [TIMING] data: 7.8ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10869.0ms | clip: 0.0ms | optim: 23.8ms step 1966 | loss: 2.892990 | lr 1.8000e-04 | norm: 0.1730 | dt: 14516.40ms | tok/sec: 36116.9 | mem: 107563MB | Time Left: 01 hrs : 03 mins [TIMING] data: 7.8ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10876.5ms | clip: 0.0ms | optim: 23.9ms step 1967 | loss: 3.056666 | lr 1.8000e-04 | norm: 0.1695 | dt: 14511.99ms | tok/sec: 36127.9 | mem: 107563MB | Time Left: 01 hrs : 02 mins [TIMING] data: 7.7ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10869.9ms | clip: 0.0ms | optim: 23.9ms step 1968 | loss: 2.915977 | lr 1.8000e-04 | norm: 0.1957 | dt: 14511.11ms | tok/sec: 36130.1 | mem: 107563MB | Time Left: 01 hrs : 02 mins [TIMING] data: 7.7ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10873.6ms | clip: 0.0ms | optim: 23.8ms step 1969 | loss: 2.664688 | lr 1.8000e-04 | norm: 0.2849 | dt: 14508.92ms | tok/sec: 36135.6 | mem: 107563MB | Time Left: 01 hrs : 02 mins [TIMING] data: 7.5ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10867.9ms | clip: 0.0ms | optim: 23.9ms step 1970 | loss: 2.798992 | lr 1.8000e-04 | norm: 0.2690 | dt: 14512.56ms | tok/sec: 36126.5 | mem: 107563MB | Time Left: 01 hrs : 02 mins [TIMING] data: 7.7ms | fwd: 3582.0ms | loss: 13.1ms | bwd: 10872.2ms | clip: 0.0ms | optim: 23.9ms step 1971 | loss: 3.343174 | lr 1.8000e-04 | norm: 0.1951 | dt: 14506.70ms | tok/sec: 36141.1 | mem: 107563MB | Time Left: 01 hrs : 01 mins [TIMING] data: 7.5ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10864.4ms | clip: 0.0ms | optim: 23.9ms step 1972 | loss: 2.887892 | lr 1.8000e-04 | norm: 0.1921 | dt: 14505.42ms | tok/sec: 36144.3 | mem: 107563MB | Time Left: 01 hrs : 01 mins [TIMING] data: 7.5ms | fwd: 3580.1ms | loss: 13.2ms | bwd: 10866.8ms | clip: 0.0ms | optim: 24.0ms step 1973 | loss: 2.792786 | lr 1.8000e-04 | norm: 0.1906 | dt: 14508.82ms | tok/sec: 36135.8 | mem: 107563MB | Time Left: 01 hrs : 01 mins [TIMING] data: 7.6ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10871.0ms | clip: 0.0ms | optim: 24.0ms step 1974 | loss: 3.065810 | lr 1.8000e-04 | norm: 0.2123 | dt: 14505.96ms | tok/sec: 36142.9 | mem: 107563MB | Time Left: 01 hrs : 01 mins [TIMING] data: 7.6ms | fwd: 3585.5ms | loss: 13.1ms | bwd: 10862.2ms | clip: 0.0ms | optim: 23.9ms step 1975 | loss: 2.898015 | lr 1.8000e-04 | norm: 0.1992 | dt: 14502.89ms | tok/sec: 36150.6 | mem: 107563MB | Time Left: 01 hrs : 00 mins [TIMING] data: 7.5ms | fwd: 3583.0ms | loss: 13.1ms | bwd: 10861.7ms | clip: 0.0ms | optim: 23.8ms step 1976 | loss: 2.631146 | lr 1.8000e-04 | norm: 0.1925 | dt: 14513.98ms | tok/sec: 36123.0 | mem: 107563MB | Time Left: 01 hrs : 00 mins [TIMING] data: 7.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10874.3ms | clip: 0.0ms | optim: 23.9ms step 1977 | loss: 2.928052 | lr 1.8000e-04 | norm: 0.2033 | dt: 14507.31ms | tok/sec: 36139.6 | mem: 107563MB | Time Left: 01 hrs : 00 mins [TIMING] data: 7.5ms | fwd: 3582.3ms | loss: 13.2ms | bwd: 10866.4ms | clip: 0.0ms | optim: 23.9ms step 1978 | loss: 2.885261 | lr 1.8000e-04 | norm: 0.1721 | dt: 14509.38ms | tok/sec: 36134.4 | mem: 107563MB | Time Left: 01 hrs : 00 mins [TIMING] data: 7.7ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10871.0ms | clip: 0.0ms | optim: 23.9ms step 1979 | loss: 2.783150 | lr 1.8000e-04 | norm: 0.2614 | dt: 14505.03ms | tok/sec: 36145.2 | mem: 107563MB | Time Left: 00 hrs : 59 mins [TIMING] data: 7.4ms | fwd: 3584.8ms | loss: 13.2ms | bwd: 10862.0ms | clip: 0.0ms | optim: 23.8ms step 1980 | loss: 2.830530 | lr 1.8000e-04 | norm: 0.2247 | dt: 14510.15ms | tok/sec: 36132.5 | mem: 107563MB | Time Left: 00 hrs : 59 mins [TIMING] data: 7.8ms | fwd: 3585.7ms | loss: 13.1ms | bwd: 10866.0ms | clip: 0.0ms | optim: 23.8ms step 1981 | loss: 2.497614 | lr 1.8000e-04 | norm: 0.2309 | dt: 14516.32ms | tok/sec: 36117.1 | mem: 107563MB | Time Left: 00 hrs : 59 mins [TIMING] data: 8.0ms | fwd: 3583.2ms | loss: 13.1ms | bwd: 10874.0ms | clip: 0.0ms | optim: 23.8ms step 1982 | loss: 2.803470 | lr 1.8000e-04 | norm: 0.2245 | dt: 14514.85ms | tok/sec: 36120.8 | mem: 107563MB | Time Left: 00 hrs : 59 mins [TIMING] data: 7.6ms | fwd: 3584.6ms | loss: 13.1ms | bwd: 10871.5ms | clip: 0.0ms | optim: 24.0ms step 1983 | loss: 2.854444 | lr 1.8000e-04 | norm: 0.2536 | dt: 14506.22ms | tok/sec: 36142.3 | mem: 107563MB | Time Left: 00 hrs : 58 mins [TIMING] data: 7.5ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10868.8ms | clip: 0.0ms | optim: 23.8ms step 1984 | loss: 2.815143 | lr 1.8000e-04 | norm: 0.1973 | dt: 14502.11ms | tok/sec: 36152.5 | mem: 107563MB | Time Left: 00 hrs : 58 mins [TIMING] data: 7.5ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10863.5ms | clip: 0.0ms | optim: 24.0ms step 1985 | loss: 2.560176 | lr 1.8000e-04 | norm: 0.1863 | dt: 14501.43ms | tok/sec: 36154.2 | mem: 107563MB | Time Left: 00 hrs : 58 mins [TIMING] data: 7.7ms | fwd: 3578.4ms | loss: 13.2ms | bwd: 10864.4ms | clip: 0.0ms | optim: 23.9ms step 1986 | loss: 2.450070 | lr 1.8000e-04 | norm: 0.1627 | dt: 14505.80ms | tok/sec: 36143.3 | mem: 107563MB | Time Left: 00 hrs : 58 mins [TIMING] data: 7.5ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10867.7ms | clip: 0.0ms | optim: 23.9ms step 1987 | loss: 2.738282 | lr 1.8000e-04 | norm: 0.2036 | dt: 14498.56ms | tok/sec: 36161.4 | mem: 107563MB | Time Left: 00 hrs : 57 mins [TIMING] data: 7.6ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10860.2ms | clip: 0.0ms | optim: 23.8ms step 1988 | loss: 2.963662 | lr 1.8000e-04 | norm: 0.1664 | dt: 14500.99ms | tok/sec: 36155.3 | mem: 107563MB | Time Left: 00 hrs : 57 mins [TIMING] data: 7.5ms | fwd: 3589.1ms | loss: 13.1ms | bwd: 10853.6ms | clip: 0.0ms | optim: 23.8ms step 1989 | loss: 2.995144 | lr 1.8000e-04 | norm: 0.1778 | dt: 14510.21ms | tok/sec: 36132.4 | mem: 107563MB | Time Left: 00 hrs : 57 mins [TIMING] data: 7.5ms | fwd: 3583.9ms | loss: 13.1ms | bwd: 10867.9ms | clip: 0.0ms | optim: 23.9ms step 1990 | loss: 3.112815 | lr 1.8000e-04 | norm: 0.1998 | dt: 14501.50ms | tok/sec: 36154.1 | mem: 107563MB | Time Left: 00 hrs : 57 mins [TIMING] data: 7.7ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10862.9ms | clip: 0.0ms | optim: 23.8ms step 1991 | loss: 2.693871 | lr 1.8000e-04 | norm: 0.1816 | dt: 14488.87ms | tok/sec: 36185.6 | mem: 107563MB | Time Left: 00 hrs : 56 mins [TIMING] data: 7.6ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10848.0ms | clip: 0.0ms | optim: 23.8ms step 1992 | loss: 2.908854 | lr 1.8000e-04 | norm: 0.1733 | dt: 14496.09ms | tok/sec: 36167.5 | mem: 107563MB | Time Left: 00 hrs : 56 mins [TIMING] data: 7.6ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10858.2ms | clip: 0.0ms | optim: 23.9ms step 1993 | loss: 2.918655 | lr 1.8000e-04 | norm: 0.1928 | dt: 14498.24ms | tok/sec: 36162.2 | mem: 107563MB | Time Left: 00 hrs : 56 mins [TIMING] data: 7.8ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10856.4ms | clip: 0.0ms | optim: 23.8ms step 1994 | loss: 3.024800 | lr 1.8000e-04 | norm: 0.2127 | dt: 14495.61ms | tok/sec: 36168.7 | mem: 107563MB | Time Left: 00 hrs : 56 mins [TIMING] data: 7.5ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.9ms step 1995 | loss: 2.958577 | lr 1.8000e-04 | norm: 0.2543 | dt: 14485.90ms | tok/sec: 36193.0 | mem: 107563MB | Time Left: 00 hrs : 56 mins [TIMING] data: 7.5ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10845.2ms | clip: 0.0ms | optim: 24.0ms step 1996 | loss: 2.787618 | lr 1.8000e-04 | norm: 0.1863 | dt: 14504.76ms | tok/sec: 36145.9 | mem: 107563MB | Time Left: 00 hrs : 55 mins [TIMING] data: 7.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10865.0ms | clip: 0.0ms | optim: 23.8ms step 1997 | loss: 2.833596 | lr 1.8000e-04 | norm: 0.1947 | dt: 14494.25ms | tok/sec: 36172.1 | mem: 107563MB | Time Left: 00 hrs : 55 mins [TIMING] data: 7.7ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10857.4ms | clip: 0.0ms | optim: 23.8ms step 1998 | loss: 3.034306 | lr 1.8000e-04 | norm: 0.1796 | dt: 14497.16ms | tok/sec: 36164.9 | mem: 107563MB | Time Left: 00 hrs : 55 mins [TIMING] data: 7.5ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10856.4ms | clip: 0.0ms | optim: 24.0ms step 1999 | loss: 2.726956 | lr 1.8000e-04 | norm: 0.2311 | dt: 14497.07ms | tok/sec: 36165.1 | mem: 107563MB | Time Left: 00 hrs : 55 mins [TIMING] data: 7.6ms | fwd: 3581.2ms | loss: 13.2ms | bwd: 10857.3ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2000.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1900.ckpt validation loss: 2.9937 step 2000 | loss: 2.857254 | lr 1.8000e-04 | norm: 0.1868 | dt: 19006.69ms | tok/sec: 27584.4 | mem: 107563MB | Time Left: 01 hrs : 11 mins [TIMING] data: 7.5ms | fwd: 3565.2ms | loss: 13.1ms | bwd: 10796.7ms | clip: 0.0ms | optim: 24.3ms step 2001 | loss: 2.801605 | lr 1.8000e-04 | norm: 0.1974 | dt: 14419.00ms | tok/sec: 36360.9 | mem: 107563MB | Time Left: 00 hrs : 54 mins [TIMING] data: 7.6ms | fwd: 3570.7ms | loss: 13.1ms | bwd: 10789.3ms | clip: 0.0ms | optim: 24.2ms step 2002 | loss: 2.872388 | lr 1.8000e-04 | norm: 0.2106 | dt: 14449.52ms | tok/sec: 36284.1 | mem: 107563MB | Time Left: 00 hrs : 54 mins [TIMING] data: 7.5ms | fwd: 3569.2ms | loss: 13.2ms | bwd: 10822.1ms | clip: 0.0ms | optim: 23.8ms step 2003 | loss: 2.900469 | lr 1.8000e-04 | norm: 0.3394 | dt: 14452.90ms | tok/sec: 36275.6 | mem: 107563MB | Time Left: 00 hrs : 53 mins [TIMING] data: 7.8ms | fwd: 3574.6ms | loss: 13.1ms | bwd: 10819.8ms | clip: 0.0ms | optim: 23.8ms step 2004 | loss: 2.811032 | lr 1.8000e-04 | norm: 0.1727 | dt: 14476.96ms | tok/sec: 36215.3 | mem: 107563MB | Time Left: 00 hrs : 53 mins [TIMING] data: 7.6ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10839.3ms | clip: 0.0ms | optim: 23.9ms step 2005 | loss: 2.699111 | lr 1.8000e-04 | norm: 0.1716 | dt: 14486.82ms | tok/sec: 36190.7 | mem: 107563MB | Time Left: 00 hrs : 53 mins [TIMING] data: 8.3ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10851.9ms | clip: 0.0ms | optim: 23.8ms step 2006 | loss: 2.919079 | lr 1.8000e-04 | norm: 0.1813 | dt: 14483.43ms | tok/sec: 36199.2 | mem: 107563MB | Time Left: 00 hrs : 53 mins [TIMING] data: 7.7ms | fwd: 3581.3ms | loss: 13.1ms | bwd: 10843.6ms | clip: 0.0ms | optim: 23.9ms step 2007 | loss: 2.931618 | lr 1.8000e-04 | norm: 0.1653 | dt: 14484.40ms | tok/sec: 36196.7 | mem: 107563MB | Time Left: 00 hrs : 53 mins [TIMING] data: 7.5ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10847.3ms | clip: 0.0ms | optim: 23.9ms step 2008 | loss: 2.719528 | lr 1.8000e-04 | norm: 0.1902 | dt: 14480.60ms | tok/sec: 36206.2 | mem: 107563MB | Time Left: 00 hrs : 52 mins [TIMING] data: 7.6ms | fwd: 3586.5ms | loss: 13.1ms | bwd: 10835.6ms | clip: 0.0ms | optim: 23.9ms step 2009 | loss: 3.070099 | lr 1.8000e-04 | norm: 0.1949 | dt: 14487.81ms | tok/sec: 36188.2 | mem: 107563MB | Time Left: 00 hrs : 52 mins [TIMING] data: 7.6ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10847.6ms | clip: 0.0ms | optim: 23.9ms step 2010 | loss: 2.759990 | lr 1.8000e-04 | norm: 0.1922 | dt: 14494.78ms | tok/sec: 36170.8 | mem: 107563MB | Time Left: 00 hrs : 52 mins [TIMING] data: 7.6ms | fwd: 3586.9ms | loss: 13.1ms | bwd: 10849.5ms | clip: 0.0ms | optim: 23.8ms step 2011 | loss: 3.029814 | lr 1.8000e-04 | norm: 0.2202 | dt: 14512.39ms | tok/sec: 36126.9 | mem: 107563MB | Time Left: 00 hrs : 52 mins [TIMING] data: 7.7ms | fwd: 3583.0ms | loss: 13.0ms | bwd: 10871.0ms | clip: 0.0ms | optim: 23.9ms step 2012 | loss: 2.783234 | lr 1.8000e-04 | norm: 0.1839 | dt: 14489.19ms | tok/sec: 36184.8 | mem: 107563MB | Time Left: 00 hrs : 51 mins [TIMING] data: 7.5ms | fwd: 3578.0ms | loss: 13.1ms | bwd: 10853.0ms | clip: 0.0ms | optim: 23.8ms step 2013 | loss: 2.770378 | lr 1.8000e-04 | norm: 0.1933 | dt: 14490.00ms | tok/sec: 36182.7 | mem: 107563MB | Time Left: 00 hrs : 51 mins [TIMING] data: 7.6ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.9ms step 2014 | loss: 2.474566 | lr 1.8000e-04 | norm: 0.2043 | dt: 14501.81ms | tok/sec: 36153.3 | mem: 107563MB | Time Left: 00 hrs : 51 mins [TIMING] data: 7.5ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10865.3ms | clip: 0.0ms | optim: 23.9ms step 2015 | loss: 2.847054 | lr 1.8000e-04 | norm: 0.1741 | dt: 14492.41ms | tok/sec: 36176.7 | mem: 107563MB | Time Left: 00 hrs : 51 mins [TIMING] data: 7.5ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10858.4ms | clip: 0.0ms | optim: 24.0ms step 2016 | loss: 2.738836 | lr 1.8000e-04 | norm: 0.2251 | dt: 14508.83ms | tok/sec: 36135.8 | mem: 107563MB | Time Left: 00 hrs : 51 mins [TIMING] data: 7.6ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10871.9ms | clip: 0.0ms | optim: 23.9ms step 2017 | loss: 2.979005 | lr 1.8000e-04 | norm: 2.8168 | dt: 14491.29ms | tok/sec: 36179.5 | mem: 107563MB | Time Left: 00 hrs : 50 mins [TIMING] data: 7.7ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10850.7ms | clip: 0.0ms | optim: 24.4ms step 2018 | loss: 2.654549 | lr 1.8000e-04 | norm: 0.1902 | dt: 14491.27ms | tok/sec: 36179.6 | mem: 107563MB | Time Left: 00 hrs : 50 mins [TIMING] data: 7.6ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.9ms step 2019 | loss: 2.591452 | lr 1.8000e-04 | norm: 0.1899 | dt: 14497.45ms | tok/sec: 36164.2 | mem: 107563MB | Time Left: 00 hrs : 50 mins [TIMING] data: 7.6ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10855.8ms | clip: 0.0ms | optim: 23.9ms step 2020 | loss: 2.929610 | lr 1.8000e-04 | norm: 0.1828 | dt: 14498.65ms | tok/sec: 36161.2 | mem: 107563MB | Time Left: 00 hrs : 50 mins [TIMING] data: 7.5ms | fwd: 3582.2ms | loss: 13.1ms | bwd: 10858.2ms | clip: 0.0ms | optim: 23.9ms step 2021 | loss: 2.759315 | lr 1.8000e-04 | norm: 0.1718 | dt: 14502.53ms | tok/sec: 36151.5 | mem: 107563MB | Time Left: 00 hrs : 49 mins [TIMING] data: 7.4ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10861.1ms | clip: 0.0ms | optim: 24.0ms step 2022 | loss: 2.839329 | lr 1.8000e-04 | norm: 0.1795 | dt: 14500.30ms | tok/sec: 36157.0 | mem: 107563MB | Time Left: 00 hrs : 49 mins [TIMING] data: 7.7ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10859.8ms | clip: 0.0ms | optim: 23.9ms step 2023 | loss: 2.984996 | lr 1.8000e-04 | norm: 0.1558 | dt: 14498.88ms | tok/sec: 36160.6 | mem: 107563MB | Time Left: 00 hrs : 49 mins [TIMING] data: 7.5ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10863.3ms | clip: 0.0ms | optim: 24.0ms step 2024 | loss: 2.823367 | lr 1.8000e-04 | norm: 0.1638 | dt: 14496.54ms | tok/sec: 36166.4 | mem: 107563MB | Time Left: 00 hrs : 49 mins [TIMING] data: 7.8ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10859.9ms | clip: 0.0ms | optim: 23.9ms step 2025 | loss: 2.779012 | lr 1.8000e-04 | norm: 0.1673 | dt: 14498.30ms | tok/sec: 36162.0 | mem: 107563MB | Time Left: 00 hrs : 48 mins [TIMING] data: 7.6ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10860.2ms | clip: 0.0ms | optim: 23.9ms step 2026 | loss: 3.002947 | lr 1.8000e-04 | norm: 0.2225 | dt: 14480.69ms | tok/sec: 36206.0 | mem: 107563MB | Time Left: 00 hrs : 48 mins [TIMING] data: 7.8ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10846.0ms | clip: 0.0ms | optim: 24.0ms step 2027 | loss: 2.851352 | lr 1.8000e-04 | norm: 0.2512 | dt: 14499.53ms | tok/sec: 36159.0 | mem: 107563MB | Time Left: 00 hrs : 48 mins [TIMING] data: 7.5ms | fwd: 3582.7ms | loss: 13.1ms | bwd: 10858.6ms | clip: 0.0ms | optim: 23.9ms step 2028 | loss: 2.760778 | lr 1.7999e-04 | norm: 0.1735 | dt: 14493.38ms | tok/sec: 36174.3 | mem: 107563MB | Time Left: 00 hrs : 48 mins [TIMING] data: 7.5ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10856.7ms | clip: 0.0ms | optim: 23.9ms step 2029 | loss: 3.194459 | lr 1.7997e-04 | norm: 0.2556 | dt: 14503.24ms | tok/sec: 36149.7 | mem: 107563MB | Time Left: 00 hrs : 47 mins [TIMING] data: 7.6ms | fwd: 3586.4ms | loss: 13.1ms | bwd: 10858.3ms | clip: 0.0ms | optim: 23.9ms step 2030 | loss: 2.738278 | lr 1.7993e-04 | norm: 0.1711 | dt: 14488.66ms | tok/sec: 36186.1 | mem: 107563MB | Time Left: 00 hrs : 47 mins [TIMING] data: 7.6ms | fwd: 3579.8ms | loss: 13.2ms | bwd: 10850.5ms | clip: 0.0ms | optim: 23.8ms step 2031 | loss: 3.003148 | lr 1.7988e-04 | norm: 0.1977 | dt: 14491.53ms | tok/sec: 36178.9 | mem: 107563MB | Time Left: 00 hrs : 47 mins [TIMING] data: 7.7ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10853.5ms | clip: 0.0ms | optim: 23.8ms step 2032 | loss: 2.775582 | lr 1.7982e-04 | norm: 0.1778 | dt: 14480.59ms | tok/sec: 36206.3 | mem: 107563MB | Time Left: 00 hrs : 47 mins [TIMING] data: 7.7ms | fwd: 3581.7ms | loss: 13.2ms | bwd: 10840.1ms | clip: 0.0ms | optim: 24.0ms step 2033 | loss: 2.715568 | lr 1.7973e-04 | norm: 0.5611 | dt: 14488.04ms | tok/sec: 36187.7 | mem: 107563MB | Time Left: 00 hrs : 46 mins [TIMING] data: 7.5ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.8ms step 2034 | loss: 3.036869 | lr 1.7964e-04 | norm: 0.1669 | dt: 14493.39ms | tok/sec: 36174.3 | mem: 107563MB | Time Left: 00 hrs : 46 mins [TIMING] data: 7.6ms | fwd: 3577.6ms | loss: 13.1ms | bwd: 10857.5ms | clip: 0.0ms | optim: 23.9ms step 2035 | loss: 2.698480 | lr 1.7953e-04 | norm: 0.1826 | dt: 14484.18ms | tok/sec: 36197.3 | mem: 107563MB | Time Left: 00 hrs : 46 mins [TIMING] data: 7.5ms | fwd: 3580.2ms | loss: 13.1ms | bwd: 10845.5ms | clip: 0.0ms | optim: 24.0ms step 2036 | loss: 3.136530 | lr 1.7940e-04 | norm: 0.1892 | dt: 14501.59ms | tok/sec: 36153.8 | mem: 107563MB | Time Left: 00 hrs : 46 mins [TIMING] data: 7.7ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10860.5ms | clip: 0.0ms | optim: 24.0ms step 2037 | loss: 2.795974 | lr 1.7926e-04 | norm: 0.1799 | dt: 14514.25ms | tok/sec: 36122.3 | mem: 107563MB | Time Left: 00 hrs : 45 mins [TIMING] data: 7.6ms | fwd: 3583.9ms | loss: 13.1ms | bwd: 10872.0ms | clip: 0.0ms | optim: 23.8ms step 2038 | loss: 2.590867 | lr 1.7911e-04 | norm: 0.1669 | dt: 14495.51ms | tok/sec: 36169.0 | mem: 107563MB | Time Left: 00 hrs : 45 mins [TIMING] data: 7.6ms | fwd: 3577.8ms | loss: 13.1ms | bwd: 10859.2ms | clip: 0.0ms | optim: 23.9ms step 2039 | loss: 3.113446 | lr 1.7894e-04 | norm: 0.1929 | dt: 14500.43ms | tok/sec: 36156.7 | mem: 107563MB | Time Left: 00 hrs : 45 mins [TIMING] data: 7.8ms | fwd: 3586.6ms | loss: 13.1ms | bwd: 10855.1ms | clip: 0.0ms | optim: 24.0ms step 2040 | loss: 2.958542 | lr 1.7875e-04 | norm: 0.2244 | dt: 14494.53ms | tok/sec: 36171.4 | mem: 107563MB | Time Left: 00 hrs : 45 mins [TIMING] data: 7.5ms | fwd: 3580.1ms | loss: 13.1ms | bwd: 10856.0ms | clip: 0.0ms | optim: 23.9ms step 2041 | loss: 2.485846 | lr 1.7856e-04 | norm: 0.1774 | dt: 14500.90ms | tok/sec: 36155.5 | mem: 107563MB | Time Left: 00 hrs : 44 mins [TIMING] data: 7.6ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10863.3ms | clip: 0.0ms | optim: 23.8ms step 2042 | loss: 2.462488 | lr 1.7834e-04 | norm: 0.1734 | dt: 14494.11ms | tok/sec: 36172.5 | mem: 107563MB | Time Left: 00 hrs : 44 mins [TIMING] data: 7.5ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10858.4ms | clip: 0.0ms | optim: 23.9ms step 2043 | loss: 2.812481 | lr 1.7812e-04 | norm: 0.2049 | dt: 14501.84ms | tok/sec: 36153.2 | mem: 107563MB | Time Left: 00 hrs : 44 mins [TIMING] data: 7.6ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10858.8ms | clip: 0.0ms | optim: 24.0ms step 2044 | loss: 2.700121 | lr 1.7787e-04 | norm: 0.1557 | dt: 14498.17ms | tok/sec: 36162.4 | mem: 107563MB | Time Left: 00 hrs : 44 mins [TIMING] data: 7.8ms | fwd: 3577.3ms | loss: 13.2ms | bwd: 10861.8ms | clip: 0.0ms | optim: 23.9ms step 2045 | loss: 2.479372 | lr 1.7762e-04 | norm: 0.1995 | dt: 14495.83ms | tok/sec: 36168.2 | mem: 107563MB | Time Left: 00 hrs : 43 mins [TIMING] data: 7.9ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10858.5ms | clip: 0.0ms | optim: 24.0ms step 2046 | loss: 2.810911 | lr 1.7735e-04 | norm: 0.1884 | dt: 14504.64ms | tok/sec: 36146.2 | mem: 107563MB | Time Left: 00 hrs : 43 mins [TIMING] data: 7.6ms | fwd: 3586.5ms | loss: 13.1ms | bwd: 10859.8ms | clip: 0.0ms | optim: 23.9ms step 2047 | loss: 2.730116 | lr 1.7706e-04 | norm: 0.1762 | dt: 14502.12ms | tok/sec: 36152.5 | mem: 107563MB | Time Left: 00 hrs : 43 mins [TIMING] data: 7.6ms | fwd: 3578.5ms | loss: 13.1ms | bwd: 10865.2ms | clip: 0.0ms | optim: 23.9ms Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/b4_32768_65536/train_shard_00011.bin 100000000 step 2048 | loss: 2.494205 | lr 1.7677e-04 | norm: 0.1997 | dt: 14506.35ms | tok/sec: 36142.0 | mem: 107563MB | Time Left: 00 hrs : 43 mins [TIMING] data: 12.5ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10867.4ms | clip: 0.0ms | optim: 23.9ms step 2049 | loss: 2.868975 | lr 1.7645e-04 | norm: 0.2459 | dt: 14498.92ms | tok/sec: 36160.5 | mem: 107563MB | Time Left: 00 hrs : 43 mins [TIMING] data: 7.6ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10863.3ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2050.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_1950.ckpt step 2050 | loss: 2.855710 | lr 1.7613e-04 | norm: 0.2119 | dt: 14389.66ms | tok/sec: 36435.0 | mem: 107563MB | Time Left: 00 hrs : 42 mins [TIMING] data: 7.8ms | fwd: 3573.2ms | loss: 13.1ms | bwd: 10757.8ms | clip: 0.0ms | optim: 23.9ms step 2051 | loss: 2.647858 | lr 1.7579e-04 | norm: 0.1803 | dt: 14422.74ms | tok/sec: 36351.5 | mem: 107563MB | Time Left: 00 hrs : 42 mins [TIMING] data: 7.7ms | fwd: 3565.9ms | loss: 13.2ms | bwd: 10798.0ms | clip: 0.0ms | optim: 23.9ms step 2052 | loss: 2.761024 | lr 1.7543e-04 | norm: 0.1671 | dt: 14435.34ms | tok/sec: 36319.8 | mem: 107563MB | Time Left: 00 hrs : 42 mins [TIMING] data: 7.6ms | fwd: 3567.5ms | loss: 13.1ms | bwd: 10809.5ms | clip: 0.0ms | optim: 23.9ms step 2053 | loss: 2.938551 | lr 1.7507e-04 | norm: 0.1714 | dt: 14462.98ms | tok/sec: 36250.4 | mem: 107563MB | Time Left: 00 hrs : 41 mins [TIMING] data: 7.7ms | fwd: 3572.8ms | loss: 13.2ms | bwd: 10831.3ms | clip: 0.0ms | optim: 23.9ms step 2054 | loss: 2.845596 | lr 1.7468e-04 | norm: 0.4020 | dt: 14467.73ms | tok/sec: 36238.4 | mem: 107563MB | Time Left: 00 hrs : 41 mins [TIMING] data: 7.3ms | fwd: 3579.2ms | loss: 13.2ms | bwd: 10830.4ms | clip: 0.0ms | optim: 23.9ms step 2055 | loss: 3.049790 | lr 1.7429e-04 | norm: 0.1750 | dt: 14481.81ms | tok/sec: 36203.2 | mem: 107563MB | Time Left: 00 hrs : 41 mins [TIMING] data: 7.7ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10842.9ms | clip: 0.0ms | optim: 23.8ms step 2056 | loss: 2.654090 | lr 1.7388e-04 | norm: 0.1578 | dt: 14482.26ms | tok/sec: 36202.1 | mem: 107563MB | Time Left: 00 hrs : 41 mins [TIMING] data: 7.5ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10844.7ms | clip: 0.0ms | optim: 23.8ms step 2057 | loss: 2.474095 | lr 1.7346e-04 | norm: 0.2270 | dt: 14480.39ms | tok/sec: 36206.8 | mem: 107563MB | Time Left: 00 hrs : 41 mins [TIMING] data: 7.6ms | fwd: 3575.1ms | loss: 13.1ms | bwd: 10847.2ms | clip: 0.0ms | optim: 23.8ms step 2058 | loss: 2.872446 | lr 1.7303e-04 | norm: 0.1873 | dt: 14485.90ms | tok/sec: 36193.0 | mem: 107563MB | Time Left: 00 hrs : 40 mins [TIMING] data: 7.6ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10850.1ms | clip: 0.0ms | optim: 23.9ms step 2059 | loss: 2.864751 | lr 1.7258e-04 | norm: 0.9320 | dt: 14501.39ms | tok/sec: 36154.3 | mem: 107563MB | Time Left: 00 hrs : 40 mins [TIMING] data: 7.7ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10864.5ms | clip: 0.0ms | optim: 23.9ms step 2060 | loss: 2.969848 | lr 1.7212e-04 | norm: 0.1850 | dt: 14515.59ms | tok/sec: 36119.0 | mem: 107563MB | Time Left: 00 hrs : 40 mins [TIMING] data: 7.5ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10876.4ms | clip: 0.0ms | optim: 23.9ms step 2061 | loss: 2.838238 | lr 1.7164e-04 | norm: 0.1929 | dt: 14486.95ms | tok/sec: 36190.4 | mem: 107563MB | Time Left: 00 hrs : 40 mins [TIMING] data: 7.6ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10849.6ms | clip: 0.0ms | optim: 23.9ms step 2062 | loss: 3.224294 | lr 1.7116e-04 | norm: 0.1889 | dt: 14485.28ms | tok/sec: 36194.5 | mem: 107563MB | Time Left: 00 hrs : 39 mins [TIMING] data: 7.5ms | fwd: 3581.5ms | loss: 13.1ms | bwd: 10845.3ms | clip: 0.0ms | optim: 23.9ms step 2063 | loss: 2.461443 | lr 1.7066e-04 | norm: 0.1875 | dt: 14484.28ms | tok/sec: 36197.0 | mem: 107563MB | Time Left: 00 hrs : 39 mins [TIMING] data: 7.5ms | fwd: 3574.4ms | loss: 13.1ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.8ms step 2064 | loss: 2.971443 | lr 1.7015e-04 | norm: 0.3547 | dt: 14486.72ms | tok/sec: 36190.9 | mem: 107563MB | Time Left: 00 hrs : 39 mins [TIMING] data: 7.6ms | fwd: 3578.9ms | loss: 13.2ms | bwd: 10849.5ms | clip: 0.0ms | optim: 23.8ms step 2065 | loss: 2.873290 | lr 1.6962e-04 | norm: 0.1880 | dt: 14500.72ms | tok/sec: 36156.0 | mem: 107563MB | Time Left: 00 hrs : 39 mins [TIMING] data: 7.5ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10863.6ms | clip: 0.0ms | optim: 23.9ms step 2066 | loss: 2.678759 | lr 1.6909e-04 | norm: 0.2006 | dt: 14489.83ms | tok/sec: 36183.2 | mem: 107563MB | Time Left: 00 hrs : 38 mins [TIMING] data: 7.6ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 24.0ms step 2067 | loss: 3.077285 | lr 1.6854e-04 | norm: 0.3389 | dt: 14486.85ms | tok/sec: 36190.6 | mem: 107563MB | Time Left: 00 hrs : 38 mins [TIMING] data: 7.7ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10848.2ms | clip: 0.0ms | optim: 23.9ms step 2068 | loss: 2.459507 | lr 1.6798e-04 | norm: 0.1533 | dt: 14494.06ms | tok/sec: 36172.6 | mem: 107563MB | Time Left: 00 hrs : 38 mins [TIMING] data: 7.6ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10855.8ms | clip: 0.0ms | optim: 23.8ms step 2069 | loss: 2.749750 | lr 1.6741e-04 | norm: 0.1853 | dt: 14490.13ms | tok/sec: 36182.4 | mem: 107563MB | Time Left: 00 hrs : 38 mins [TIMING] data: 7.6ms | fwd: 3580.8ms | loss: 13.1ms | bwd: 10851.1ms | clip: 0.0ms | optim: 23.9ms step 2070 | loss: 2.767143 | lr 1.6683e-04 | norm: 0.1732 | dt: 14491.82ms | tok/sec: 36178.2 | mem: 107563MB | Time Left: 00 hrs : 37 mins [TIMING] data: 7.6ms | fwd: 3575.4ms | loss: 13.1ms | bwd: 10858.0ms | clip: 0.0ms | optim: 23.9ms step 2071 | loss: 3.011604 | lr 1.6623e-04 | norm: 0.1695 | dt: 14483.39ms | tok/sec: 36199.3 | mem: 107563MB | Time Left: 00 hrs : 37 mins [TIMING] data: 7.9ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10847.1ms | clip: 0.0ms | optim: 23.9ms step 2072 | loss: 2.859683 | lr 1.6562e-04 | norm: 0.1744 | dt: 14472.23ms | tok/sec: 36227.2 | mem: 107563MB | Time Left: 00 hrs : 37 mins [TIMING] data: 7.6ms | fwd: 3579.8ms | loss: 13.2ms | bwd: 10833.9ms | clip: 0.0ms | optim: 23.9ms step 2073 | loss: 2.802882 | lr 1.6501e-04 | norm: 0.1761 | dt: 14495.17ms | tok/sec: 36169.8 | mem: 107563MB | Time Left: 00 hrs : 37 mins [TIMING] data: 7.5ms | fwd: 3585.2ms | loss: 13.1ms | bwd: 10851.6ms | clip: 0.0ms | optim: 23.8ms step 2074 | loss: 2.697772 | lr 1.6438e-04 | norm: 0.1772 | dt: 14485.99ms | tok/sec: 36192.8 | mem: 107563MB | Time Left: 00 hrs : 36 mins [TIMING] data: 7.6ms | fwd: 3575.0ms | loss: 13.2ms | bwd: 10851.8ms | clip: 0.0ms | optim: 24.6ms step 2075 | loss: 2.861625 | lr 1.6374e-04 | norm: 0.1891 | dt: 14496.33ms | tok/sec: 36167.0 | mem: 107563MB | Time Left: 00 hrs : 36 mins [TIMING] data: 7.6ms | fwd: 3581.0ms | loss: 13.1ms | bwd: 10856.4ms | clip: 0.0ms | optim: 24.0ms step 2076 | loss: 2.927309 | lr 1.6309e-04 | norm: 0.1878 | dt: 14496.58ms | tok/sec: 36166.3 | mem: 107563MB | Time Left: 00 hrs : 36 mins [TIMING] data: 7.8ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10857.8ms | clip: 0.0ms | optim: 23.9ms step 2077 | loss: 2.834754 | lr 1.6243e-04 | norm: 0.1496 | dt: 14499.29ms | tok/sec: 36159.6 | mem: 107563MB | Time Left: 00 hrs : 36 mins [TIMING] data: 7.6ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10861.2ms | clip: 0.0ms | optim: 23.9ms step 2078 | loss: 2.949535 | lr 1.6175e-04 | norm: 0.1718 | dt: 14486.32ms | tok/sec: 36191.9 | mem: 107563MB | Time Left: 00 hrs : 35 mins [TIMING] data: 7.6ms | fwd: 3576.4ms | loss: 13.0ms | bwd: 10851.7ms | clip: 0.0ms | optim: 23.8ms step 2079 | loss: 2.705465 | lr 1.6107e-04 | norm: 0.1692 | dt: 14497.62ms | tok/sec: 36163.7 | mem: 107563MB | Time Left: 00 hrs : 35 mins [TIMING] data: 7.4ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10860.4ms | clip: 0.0ms | optim: 23.9ms step 2080 | loss: 2.751604 | lr 1.6038e-04 | norm: 0.2015 | dt: 14495.71ms | tok/sec: 36168.5 | mem: 107563MB | Time Left: 00 hrs : 35 mins [TIMING] data: 7.5ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10857.1ms | clip: 0.0ms | optim: 23.8ms step 2081 | loss: 2.756145 | lr 1.5968e-04 | norm: 0.3198 | dt: 14504.25ms | tok/sec: 36147.2 | mem: 107563MB | Time Left: 00 hrs : 35 mins [TIMING] data: 7.5ms | fwd: 3581.6ms | loss: 13.1ms | bwd: 10864.4ms | clip: 0.0ms | optim: 23.9ms step 2082 | loss: 2.634172 | lr 1.5897e-04 | norm: 0.2318 | dt: 14496.57ms | tok/sec: 36166.3 | mem: 107563MB | Time Left: 00 hrs : 35 mins [TIMING] data: 7.7ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10858.7ms | clip: 0.0ms | optim: 23.8ms step 2083 | loss: 3.114430 | lr 1.5825e-04 | norm: 0.2145 | dt: 14488.85ms | tok/sec: 36185.6 | mem: 107563MB | Time Left: 00 hrs : 34 mins [TIMING] data: 7.7ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10850.1ms | clip: 0.0ms | optim: 23.9ms step 2084 | loss: 2.831591 | lr 1.5751e-04 | norm: 0.1730 | dt: 14490.24ms | tok/sec: 36182.1 | mem: 107563MB | Time Left: 00 hrs : 34 mins [TIMING] data: 7.6ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10852.4ms | clip: 0.0ms | optim: 23.9ms step 2085 | loss: 2.814776 | lr 1.5677e-04 | norm: 0.2071 | dt: 14491.57ms | tok/sec: 36178.8 | mem: 107563MB | Time Left: 00 hrs : 34 mins [TIMING] data: 7.8ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 2086 | loss: 2.920465 | lr 1.5603e-04 | norm: 0.1952 | dt: 14500.43ms | tok/sec: 36156.7 | mem: 107563MB | Time Left: 00 hrs : 34 mins [TIMING] data: 7.6ms | fwd: 3587.7ms | loss: 13.1ms | bwd: 10854.3ms | clip: 0.0ms | optim: 23.9ms step 2087 | loss: 2.916613 | lr 1.5527e-04 | norm: 0.1942 | dt: 14495.01ms | tok/sec: 36170.2 | mem: 107563MB | Time Left: 00 hrs : 33 mins [TIMING] data: 7.4ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.9ms step 2088 | loss: 2.879078 | lr 1.5450e-04 | norm: 0.1953 | dt: 14493.38ms | tok/sec: 36174.3 | mem: 107563MB | Time Left: 00 hrs : 33 mins [TIMING] data: 7.6ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10853.8ms | clip: 0.0ms | optim: 23.9ms step 2089 | loss: 3.031913 | lr 1.5373e-04 | norm: 0.2076 | dt: 14492.42ms | tok/sec: 36176.7 | mem: 107563MB | Time Left: 00 hrs : 33 mins [TIMING] data: 7.9ms | fwd: 3580.9ms | loss: 13.2ms | bwd: 10852.8ms | clip: 0.0ms | optim: 23.8ms step 2090 | loss: 2.879623 | lr 1.5294e-04 | norm: 0.1714 | dt: 14486.94ms | tok/sec: 36190.4 | mem: 107563MB | Time Left: 00 hrs : 33 mins [TIMING] data: 7.5ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10848.0ms | clip: 0.0ms | optim: 23.9ms step 2091 | loss: 2.883024 | lr 1.5215e-04 | norm: 0.1756 | dt: 14491.76ms | tok/sec: 36178.4 | mem: 107563MB | Time Left: 00 hrs : 32 mins [TIMING] data: 7.6ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms step 2092 | loss: 2.707993 | lr 1.5135e-04 | norm: 0.1758 | dt: 14489.37ms | tok/sec: 36184.3 | mem: 107563MB | Time Left: 00 hrs : 32 mins [TIMING] data: 7.7ms | fwd: 3577.4ms | loss: 13.1ms | bwd: 10853.6ms | clip: 0.0ms | optim: 23.8ms step 2093 | loss: 2.746714 | lr 1.5054e-04 | norm: 0.1585 | dt: 14482.19ms | tok/sec: 36202.2 | mem: 107563MB | Time Left: 00 hrs : 32 mins [TIMING] data: 7.8ms | fwd: 3575.5ms | loss: 13.1ms | bwd: 10847.9ms | clip: 0.0ms | optim: 23.9ms step 2094 | loss: 2.771768 | lr 1.4973e-04 | norm: 0.1650 | dt: 14495.05ms | tok/sec: 36170.1 | mem: 107563MB | Time Left: 00 hrs : 32 mins [TIMING] data: 7.9ms | fwd: 3582.8ms | loss: 13.2ms | bwd: 10852.9ms | clip: 0.0ms | optim: 24.1ms step 2095 | loss: 2.931654 | lr 1.4891e-04 | norm: 0.1857 | dt: 14502.91ms | tok/sec: 36150.5 | mem: 107563MB | Time Left: 00 hrs : 31 mins [TIMING] data: 7.6ms | fwd: 3585.0ms | loss: 13.1ms | bwd: 10859.6ms | clip: 0.0ms | optim: 23.8ms step 2096 | loss: 2.711101 | lr 1.4808e-04 | norm: 0.2065 | dt: 14488.27ms | tok/sec: 36187.1 | mem: 107563MB | Time Left: 00 hrs : 31 mins [TIMING] data: 8.1ms | fwd: 3579.7ms | loss: 13.2ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.9ms step 2097 | loss: 2.992352 | lr 1.4724e-04 | norm: 0.1928 | dt: 14494.29ms | tok/sec: 36172.0 | mem: 107563MB | Time Left: 00 hrs : 31 mins [TIMING] data: 7.7ms | fwd: 3583.7ms | loss: 13.2ms | bwd: 10851.9ms | clip: 0.0ms | optim: 23.9ms step 2098 | loss: 3.012081 | lr 1.4640e-04 | norm: 0.1906 | dt: 14481.80ms | tok/sec: 36203.2 | mem: 107563MB | Time Left: 00 hrs : 31 mins [TIMING] data: 7.7ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10845.0ms | clip: 0.0ms | optim: 23.9ms step 2099 | loss: 2.579428 | lr 1.4555e-04 | norm: 0.1573 | dt: 14486.78ms | tok/sec: 36190.8 | mem: 107563MB | Time Left: 00 hrs : 30 mins [TIMING] data: 7.6ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2100.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2000.ckpt step 2100 | loss: 2.837517 | lr 1.4469e-04 | norm: 0.1754 | dt: 14390.75ms | tok/sec: 36432.3 | mem: 107563MB | Time Left: 00 hrs : 30 mins [TIMING] data: 7.5ms | fwd: 3593.6ms | loss: 13.1ms | bwd: 10738.7ms | clip: 0.0ms | optim: 24.0ms step 2101 | loss: 2.825599 | lr 1.4383e-04 | norm: 0.1714 | dt: 14425.36ms | tok/sec: 36344.9 | mem: 107563MB | Time Left: 00 hrs : 30 mins [TIMING] data: 7.7ms | fwd: 3564.6ms | loss: 13.1ms | bwd: 10802.2ms | clip: 0.0ms | optim: 23.9ms step 2102 | loss: 2.658654 | lr 1.4296e-04 | norm: 0.1853 | dt: 14448.18ms | tok/sec: 36287.5 | mem: 107563MB | Time Left: 00 hrs : 30 mins [TIMING] data: 7.6ms | fwd: 3574.1ms | loss: 13.1ms | bwd: 10815.7ms | clip: 0.0ms | optim: 23.8ms step 2103 | loss: 2.872358 | lr 1.4209e-04 | norm: 0.1735 | dt: 14467.43ms | tok/sec: 36239.2 | mem: 107563MB | Time Left: 00 hrs : 29 mins [TIMING] data: 7.7ms | fwd: 3573.8ms | loss: 13.2ms | bwd: 10835.0ms | clip: 0.0ms | optim: 23.9ms step 2104 | loss: 2.791243 | lr 1.4121e-04 | norm: 0.1591 | dt: 14474.74ms | tok/sec: 36220.9 | mem: 107563MB | Time Left: 00 hrs : 29 mins [TIMING] data: 7.6ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10835.8ms | clip: 0.0ms | optim: 23.9ms step 2105 | loss: 2.920568 | lr 1.4032e-04 | norm: 0.1720 | dt: 14469.13ms | tok/sec: 36234.9 | mem: 107563MB | Time Left: 00 hrs : 29 mins [TIMING] data: 7.6ms | fwd: 3578.2ms | loss: 13.2ms | bwd: 10832.2ms | clip: 0.0ms | optim: 23.9ms step 2106 | loss: 2.807056 | lr 1.3944e-04 | norm: 0.1613 | dt: 14483.61ms | tok/sec: 36198.7 | mem: 107563MB | Time Left: 00 hrs : 29 mins [TIMING] data: 7.7ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10845.8ms | clip: 0.0ms | optim: 23.9ms step 2107 | loss: 2.764795 | lr 1.3854e-04 | norm: 0.1585 | dt: 14482.50ms | tok/sec: 36201.5 | mem: 107563MB | Time Left: 00 hrs : 28 mins [TIMING] data: 7.6ms | fwd: 3575.3ms | loss: 13.1ms | bwd: 10848.9ms | clip: 0.0ms | optim: 23.8ms step 2108 | loss: 2.752253 | lr 1.3764e-04 | norm: 0.1465 | dt: 14493.65ms | tok/sec: 36173.6 | mem: 107563MB | Time Left: 00 hrs : 28 mins [TIMING] data: 7.5ms | fwd: 3582.1ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.9ms step 2109 | loss: 2.784595 | lr 1.3674e-04 | norm: 0.1607 | dt: 14485.43ms | tok/sec: 36194.2 | mem: 107563MB | Time Left: 00 hrs : 28 mins [TIMING] data: 7.6ms | fwd: 3574.9ms | loss: 13.1ms | bwd: 10852.3ms | clip: 0.0ms | optim: 23.9ms step 2110 | loss: 2.709687 | lr 1.3583e-04 | norm: 0.4053 | dt: 14478.76ms | tok/sec: 36210.8 | mem: 107563MB | Time Left: 00 hrs : 28 mins [TIMING] data: 7.7ms | fwd: 3579.3ms | loss: 13.1ms | bwd: 10841.0ms | clip: 0.0ms | optim: 23.9ms step 2111 | loss: 2.865763 | lr 1.3492e-04 | norm: 0.1544 | dt: 14495.43ms | tok/sec: 36169.2 | mem: 107563MB | Time Left: 00 hrs : 28 mins [TIMING] data: 7.6ms | fwd: 3583.4ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 23.8ms step 2112 | loss: 2.864967 | lr 1.3401e-04 | norm: 0.1770 | dt: 14488.10ms | tok/sec: 36187.5 | mem: 107563MB | Time Left: 00 hrs : 27 mins [TIMING] data: 7.5ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.8ms step 2113 | loss: 2.727388 | lr 1.3309e-04 | norm: 0.1685 | dt: 14488.52ms | tok/sec: 36186.4 | mem: 107563MB | Time Left: 00 hrs : 27 mins [TIMING] data: 7.6ms | fwd: 3583.4ms | loss: 13.2ms | bwd: 10846.7ms | clip: 0.0ms | optim: 23.8ms step 2114 | loss: 2.899249 | lr 1.3217e-04 | norm: 0.2665 | dt: 14483.81ms | tok/sec: 36198.2 | mem: 107563MB | Time Left: 00 hrs : 27 mins [TIMING] data: 7.6ms | fwd: 3575.9ms | loss: 13.1ms | bwd: 10849.5ms | clip: 0.0ms | optim: 23.9ms step 2115 | loss: 2.885885 | lr 1.3124e-04 | norm: 0.2024 | dt: 14495.74ms | tok/sec: 36168.4 | mem: 107563MB | Time Left: 00 hrs : 27 mins [TIMING] data: 7.6ms | fwd: 3581.9ms | loss: 13.1ms | bwd: 10855.5ms | clip: 0.0ms | optim: 23.9ms step 2116 | loss: 2.525328 | lr 1.3032e-04 | norm: 0.2716 | dt: 14479.96ms | tok/sec: 36207.8 | mem: 107563MB | Time Left: 00 hrs : 26 mins [TIMING] data: 7.6ms | fwd: 3574.0ms | loss: 13.1ms | bwd: 10847.7ms | clip: 0.0ms | optim: 23.9ms step 2117 | loss: 2.836936 | lr 1.2939e-04 | norm: 0.1642 | dt: 14492.00ms | tok/sec: 36177.8 | mem: 107563MB | Time Left: 00 hrs : 26 mins [TIMING] data: 7.6ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10856.6ms | clip: 0.0ms | optim: 23.9ms step 2118 | loss: 2.662047 | lr 1.2845e-04 | norm: 0.1486 | dt: 14482.50ms | tok/sec: 36201.5 | mem: 107563MB | Time Left: 00 hrs : 26 mins [TIMING] data: 7.6ms | fwd: 3577.1ms | loss: 13.1ms | bwd: 10847.2ms | clip: 0.0ms | optim: 23.8ms step 2119 | loss: 2.888086 | lr 1.2752e-04 | norm: 0.1512 | dt: 14492.15ms | tok/sec: 36177.4 | mem: 107563MB | Time Left: 00 hrs : 26 mins [TIMING] data: 7.7ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.8ms step 2120 | loss: 3.002871 | lr 1.2658e-04 | norm: 0.1532 | dt: 14477.00ms | tok/sec: 36215.2 | mem: 107563MB | Time Left: 00 hrs : 25 mins [TIMING] data: 7.6ms | fwd: 3577.1ms | loss: 13.1ms | bwd: 10841.3ms | clip: 0.0ms | optim: 23.9ms step 2121 | loss: 2.896552 | lr 1.2565e-04 | norm: 0.1540 | dt: 14484.27ms | tok/sec: 36197.1 | mem: 107563MB | Time Left: 00 hrs : 25 mins [TIMING] data: 7.6ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10845.4ms | clip: 0.0ms | optim: 23.9ms step 2122 | loss: 2.789839 | lr 1.2471e-04 | norm: 0.1642 | dt: 14493.69ms | tok/sec: 36173.5 | mem: 107563MB | Time Left: 00 hrs : 25 mins [TIMING] data: 7.6ms | fwd: 3578.3ms | loss: 13.1ms | bwd: 10857.1ms | clip: 0.0ms | optim: 23.9ms step 2123 | loss: 3.009579 | lr 1.2377e-04 | norm: 0.1917 | dt: 14481.51ms | tok/sec: 36204.0 | mem: 107563MB | Time Left: 00 hrs : 25 mins [TIMING] data: 7.6ms | fwd: 3584.0ms | loss: 13.2ms | bwd: 10838.8ms | clip: 0.0ms | optim: 23.9ms step 2124 | loss: 3.040935 | lr 1.2283e-04 | norm: 0.1671 | dt: 14495.47ms | tok/sec: 36169.1 | mem: 107563MB | Time Left: 00 hrs : 24 mins [TIMING] data: 7.5ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10857.9ms | clip: 0.0ms | optim: 23.9ms step 2125 | loss: 2.833434 | lr 1.2188e-04 | norm: 0.1854 | dt: 14500.38ms | tok/sec: 36156.8 | mem: 107563MB | Time Left: 00 hrs : 24 mins [TIMING] data: 7.6ms | fwd: 3580.7ms | loss: 13.1ms | bwd: 10861.5ms | clip: 0.0ms | optim: 23.8ms step 2126 | loss: 2.337834 | lr 1.2094e-04 | norm: 0.1673 | dt: 14490.31ms | tok/sec: 36182.0 | mem: 107563MB | Time Left: 00 hrs : 24 mins [TIMING] data: 7.8ms | fwd: 3573.5ms | loss: 13.1ms | bwd: 10858.4ms | clip: 0.0ms | optim: 23.9ms step 2127 | loss: 2.642808 | lr 1.2000e-04 | norm: 0.2004 | dt: 14477.82ms | tok/sec: 36213.2 | mem: 107563MB | Time Left: 00 hrs : 24 mins [TIMING] data: 7.5ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10841.0ms | clip: 0.0ms | optim: 23.9ms step 2128 | loss: 3.075935 | lr 1.1906e-04 | norm: 0.1916 | dt: 14490.18ms | tok/sec: 36182.3 | mem: 107563MB | Time Left: 00 hrs : 23 mins [TIMING] data: 7.5ms | fwd: 3579.9ms | loss: 13.1ms | bwd: 10852.0ms | clip: 0.0ms | optim: 23.8ms step 2129 | loss: 3.159524 | lr 1.1812e-04 | norm: 0.1954 | dt: 14475.55ms | tok/sec: 36218.9 | mem: 107563MB | Time Left: 00 hrs : 23 mins [TIMING] data: 7.7ms | fwd: 3583.6ms | loss: 13.1ms | bwd: 10833.6ms | clip: 0.0ms | optim: 23.8ms step 2130 | loss: 2.872507 | lr 1.1717e-04 | norm: 0.1782 | dt: 14482.77ms | tok/sec: 36200.8 | mem: 107563MB | Time Left: 00 hrs : 23 mins [TIMING] data: 7.5ms | fwd: 3581.2ms | loss: 13.0ms | bwd: 10843.4ms | clip: 0.0ms | optim: 23.9ms step 2131 | loss: 2.609238 | lr 1.1623e-04 | norm: 0.2258 | dt: 14472.29ms | tok/sec: 36227.0 | mem: 107563MB | Time Left: 00 hrs : 23 mins [TIMING] data: 7.9ms | fwd: 3574.1ms | loss: 13.1ms | bwd: 10839.7ms | clip: 0.0ms | optim: 23.9ms step 2132 | loss: 2.977309 | lr 1.1529e-04 | norm: 0.1958 | dt: 14487.31ms | tok/sec: 36189.5 | mem: 107563MB | Time Left: 00 hrs : 22 mins [TIMING] data: 7.7ms | fwd: 3578.8ms | loss: 13.1ms | bwd: 10849.9ms | clip: 0.0ms | optim: 23.9ms step 2133 | loss: 2.807916 | lr 1.1435e-04 | norm: 0.1577 | dt: 14491.74ms | tok/sec: 36178.4 | mem: 107563MB | Time Left: 00 hrs : 22 mins [TIMING] data: 7.7ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10848.6ms | clip: 0.0ms | optim: 23.8ms step 2134 | loss: 3.061259 | lr 1.1342e-04 | norm: 0.2506 | dt: 14487.52ms | tok/sec: 36188.9 | mem: 107563MB | Time Left: 00 hrs : 22 mins [TIMING] data: 7.5ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10847.9ms | clip: 0.0ms | optim: 23.8ms step 2135 | loss: 2.668936 | lr 1.1248e-04 | norm: 0.1640 | dt: 14492.17ms | tok/sec: 36177.3 | mem: 107563MB | Time Left: 00 hrs : 22 mins [TIMING] data: 7.6ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10855.4ms | clip: 0.0ms | optim: 23.9ms step 2136 | loss: 3.066122 | lr 1.1155e-04 | norm: 0.2402 | dt: 14478.89ms | tok/sec: 36210.5 | mem: 107563MB | Time Left: 00 hrs : 21 mins [TIMING] data: 7.6ms | fwd: 3582.5ms | loss: 13.2ms | bwd: 10837.7ms | clip: 0.0ms | optim: 23.9ms step 2137 | loss: 2.815085 | lr 1.1061e-04 | norm: 0.1781 | dt: 14489.37ms | tok/sec: 36184.3 | mem: 107563MB | Time Left: 00 hrs : 21 mins [TIMING] data: 7.7ms | fwd: 3576.6ms | loss: 13.1ms | bwd: 10854.2ms | clip: 0.0ms | optim: 24.0ms step 2138 | loss: 2.705946 | lr 1.0968e-04 | norm: 0.1641 | dt: 14489.30ms | tok/sec: 36184.5 | mem: 107563MB | Time Left: 00 hrs : 21 mins [TIMING] data: 7.7ms | fwd: 3586.0ms | loss: 13.1ms | bwd: 10844.9ms | clip: 0.0ms | optim: 23.9ms step 2139 | loss: 2.670748 | lr 1.0876e-04 | norm: 0.1527 | dt: 14488.49ms | tok/sec: 36186.5 | mem: 107563MB | Time Left: 00 hrs : 21 mins [TIMING] data: 7.6ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10851.9ms | clip: 0.0ms | optim: 23.9ms step 2140 | loss: 2.991256 | lr 1.0783e-04 | norm: 0.1643 | dt: 14489.99ms | tok/sec: 36182.8 | mem: 107563MB | Time Left: 00 hrs : 21 mins [TIMING] data: 7.9ms | fwd: 3580.1ms | loss: 13.2ms | bwd: 10851.2ms | clip: 0.0ms | optim: 23.9ms step 2141 | loss: 2.759750 | lr 1.0691e-04 | norm: 0.1611 | dt: 14501.81ms | tok/sec: 36153.3 | mem: 107563MB | Time Left: 00 hrs : 20 mins [TIMING] data: 7.6ms | fwd: 3580.4ms | loss: 13.2ms | bwd: 10862.9ms | clip: 0.0ms | optim: 23.9ms step 2142 | loss: 3.052539 | lr 1.0599e-04 | norm: 0.1720 | dt: 14486.69ms | tok/sec: 36191.0 | mem: 107563MB | Time Left: 00 hrs : 20 mins [TIMING] data: 7.7ms | fwd: 3584.4ms | loss: 13.1ms | bwd: 10843.9ms | clip: 0.0ms | optim: 23.8ms step 2143 | loss: 2.720039 | lr 1.0508e-04 | norm: 0.1704 | dt: 14486.89ms | tok/sec: 36190.5 | mem: 107563MB | Time Left: 00 hrs : 20 mins [TIMING] data: 8.0ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10849.2ms | clip: 0.0ms | optim: 23.9ms step 2144 | loss: 2.787747 | lr 1.0417e-04 | norm: 0.1589 | dt: 14495.10ms | tok/sec: 36170.0 | mem: 107563MB | Time Left: 00 hrs : 20 mins [TIMING] data: 7.8ms | fwd: 3581.4ms | loss: 13.1ms | bwd: 10854.8ms | clip: 0.0ms | optim: 23.9ms step 2145 | loss: 2.965613 | lr 1.0326e-04 | norm: 0.1912 | dt: 14492.11ms | tok/sec: 36177.5 | mem: 107563MB | Time Left: 00 hrs : 19 mins [TIMING] data: 7.7ms | fwd: 3584.3ms | loss: 13.1ms | bwd: 10849.4ms | clip: 0.0ms | optim: 23.9ms step 2146 | loss: 3.025221 | lr 1.0236e-04 | norm: 0.1658 | dt: 14492.20ms | tok/sec: 36177.3 | mem: 107563MB | Time Left: 00 hrs : 19 mins [TIMING] data: 7.6ms | fwd: 3580.5ms | loss: 13.2ms | bwd: 10852.9ms | clip: 0.0ms | optim: 23.9ms step 2147 | loss: 2.832484 | lr 1.0146e-04 | norm: 0.1935 | dt: 14495.51ms | tok/sec: 36169.0 | mem: 107563MB | Time Left: 00 hrs : 19 mins [TIMING] data: 7.6ms | fwd: 3579.4ms | loss: 13.1ms | bwd: 10857.6ms | clip: 0.0ms | optim: 23.9ms step 2148 | loss: 2.671650 | lr 1.0056e-04 | norm: 0.2105 | dt: 14495.06ms | tok/sec: 36170.1 | mem: 107563MB | Time Left: 00 hrs : 19 mins [TIMING] data: 8.2ms | fwd: 3580.9ms | loss: 13.1ms | bwd: 10855.3ms | clip: 0.0ms | optim: 23.8ms step 2149 | loss: 2.802674 | lr 9.9676e-05 | norm: 0.1617 | dt: 14488.39ms | tok/sec: 36186.8 | mem: 107563MB | Time Left: 00 hrs : 18 mins [TIMING] data: 7.7ms | fwd: 3579.4ms | loss: 13.2ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2150.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2050.ckpt step 2150 | loss: 2.845668 | lr 9.8792e-05 | norm: 0.1766 | dt: 14388.90ms | tok/sec: 36437.0 | mem: 107563MB | Time Left: 00 hrs : 18 mins [TIMING] data: 7.8ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10751.2ms | clip: 0.0ms | optim: 23.8ms step 2151 | loss: 2.681317 | lr 9.7913e-05 | norm: 0.1550 | dt: 14426.95ms | tok/sec: 36340.9 | mem: 107563MB | Time Left: 00 hrs : 18 mins [TIMING] data: 7.7ms | fwd: 3573.4ms | loss: 13.1ms | bwd: 10795.1ms | clip: 0.0ms | optim: 23.8ms step 2152 | loss: 2.713573 | lr 9.7039e-05 | norm: 0.1485 | dt: 14434.69ms | tok/sec: 36321.4 | mem: 107563MB | Time Left: 00 hrs : 18 mins [TIMING] data: 7.7ms | fwd: 3566.9ms | loss: 13.1ms | bwd: 10809.2ms | clip: 0.0ms | optim: 23.9ms step 2153 | loss: 2.783019 | lr 9.6171e-05 | norm: 0.1541 | dt: 14457.50ms | tok/sec: 36264.1 | mem: 107563MB | Time Left: 00 hrs : 17 mins [TIMING] data: 7.6ms | fwd: 3573.9ms | loss: 13.1ms | bwd: 10825.2ms | clip: 0.0ms | optim: 23.9ms step 2154 | loss: 2.926813 | lr 9.5309e-05 | norm: 0.2021 | dt: 14460.69ms | tok/sec: 36256.1 | mem: 107563MB | Time Left: 00 hrs : 17 mins [TIMING] data: 7.8ms | fwd: 3573.7ms | loss: 13.1ms | bwd: 10828.5ms | clip: 0.0ms | optim: 23.9ms step 2155 | loss: 2.848436 | lr 9.4453e-05 | norm: 0.1579 | dt: 14476.74ms | tok/sec: 36215.9 | mem: 107563MB | Time Left: 00 hrs : 17 mins [TIMING] data: 7.7ms | fwd: 3581.0ms | loss: 13.2ms | bwd: 10836.8ms | clip: 0.0ms | optim: 23.9ms step 2156 | loss: 2.810791 | lr 9.3604e-05 | norm: 0.1603 | dt: 14483.03ms | tok/sec: 36200.2 | mem: 107563MB | Time Left: 00 hrs : 17 mins [TIMING] data: 7.9ms | fwd: 3582.3ms | loss: 13.2ms | bwd: 10841.4ms | clip: 0.0ms | optim: 23.9ms step 2157 | loss: 2.849669 | lr 9.2761e-05 | norm: 0.1750 | dt: 14481.02ms | tok/sec: 36205.2 | mem: 107563MB | Time Left: 00 hrs : 16 mins [TIMING] data: 7.6ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10843.3ms | clip: 0.0ms | optim: 23.9ms step 2158 | loss: 2.746318 | lr 9.1924e-05 | norm: 0.2146 | dt: 14477.04ms | tok/sec: 36215.1 | mem: 107563MB | Time Left: 00 hrs : 16 mins [TIMING] data: 7.8ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10840.1ms | clip: 0.0ms | optim: 23.9ms step 2159 | loss: 2.821349 | lr 9.1095e-05 | norm: 0.1589 | dt: 14478.86ms | tok/sec: 36210.6 | mem: 107563MB | Time Left: 00 hrs : 16 mins [TIMING] data: 7.7ms | fwd: 3577.2ms | loss: 13.1ms | bwd: 10843.1ms | clip: 0.0ms | optim: 23.9ms step 2160 | loss: 2.699562 | lr 9.0272e-05 | norm: 1.6691 | dt: 14483.75ms | tok/sec: 36198.4 | mem: 107563MB | Time Left: 00 hrs : 16 mins [TIMING] data: 7.7ms | fwd: 3578.2ms | loss: 13.2ms | bwd: 10847.1ms | clip: 0.0ms | optim: 23.8ms step 2161 | loss: 2.830760 | lr 8.9458e-05 | norm: 0.3139 | dt: 14475.42ms | tok/sec: 36219.2 | mem: 107563MB | Time Left: 00 hrs : 15 mins [TIMING] data: 7.6ms | fwd: 3581.3ms | loss: 13.1ms | bwd: 10835.7ms | clip: 0.0ms | optim: 24.0ms step 2162 | loss: 2.912696 | lr 8.8650e-05 | norm: 0.1627 | dt: 14481.52ms | tok/sec: 36203.9 | mem: 107563MB | Time Left: 00 hrs : 15 mins [TIMING] data: 7.6ms | fwd: 3574.8ms | loss: 13.2ms | bwd: 10848.2ms | clip: 0.0ms | optim: 23.9ms step 2163 | loss: 2.474532 | lr 8.7850e-05 | norm: 0.1479 | dt: 14492.51ms | tok/sec: 36176.5 | mem: 107563MB | Time Left: 00 hrs : 15 mins [TIMING] data: 7.6ms | fwd: 3580.5ms | loss: 13.1ms | bwd: 10853.6ms | clip: 0.0ms | optim: 23.9ms step 2164 | loss: 3.101844 | lr 8.7059e-05 | norm: 0.1752 | dt: 14486.37ms | tok/sec: 36191.8 | mem: 107563MB | Time Left: 00 hrs : 15 mins [TIMING] data: 7.5ms | fwd: 3582.4ms | loss: 13.2ms | bwd: 10845.6ms | clip: 0.0ms | optim: 23.9ms step 2165 | loss: 2.881520 | lr 8.6275e-05 | norm: 0.1476 | dt: 14485.35ms | tok/sec: 36194.4 | mem: 107563MB | Time Left: 00 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3576.3ms | loss: 13.1ms | bwd: 10850.6ms | clip: 0.0ms | optim: 23.8ms step 2166 | loss: 3.076067 | lr 8.5500e-05 | norm: 0.1965 | dt: 14492.17ms | tok/sec: 36177.3 | mem: 107563MB | Time Left: 00 hrs : 14 mins [TIMING] data: 7.9ms | fwd: 3583.8ms | loss: 13.1ms | bwd: 10849.7ms | clip: 0.0ms | optim: 23.8ms step 2167 | loss: 2.985198 | lr 8.4733e-05 | norm: 0.1598 | dt: 14503.04ms | tok/sec: 36150.2 | mem: 107563MB | Time Left: 00 hrs : 14 mins [TIMING] data: 7.4ms | fwd: 3576.4ms | loss: 13.2ms | bwd: 10868.5ms | clip: 0.0ms | optim: 23.9ms step 2168 | loss: 2.763865 | lr 8.3975e-05 | norm: 0.2168 | dt: 14501.67ms | tok/sec: 36153.6 | mem: 107563MB | Time Left: 00 hrs : 14 mins [TIMING] data: 7.4ms | fwd: 3584.3ms | loss: 13.2ms | bwd: 10859.2ms | clip: 0.0ms | optim: 23.9ms step 2169 | loss: 2.616447 | lr 8.3226e-05 | norm: 0.1401 | dt: 14496.13ms | tok/sec: 36167.5 | mem: 107563MB | Time Left: 00 hrs : 14 mins [TIMING] data: 7.6ms | fwd: 3575.7ms | loss: 13.1ms | bwd: 10861.7ms | clip: 0.0ms | optim: 23.9ms step 2170 | loss: 2.581238 | lr 8.2485e-05 | norm: 0.1582 | dt: 14488.54ms | tok/sec: 36186.4 | mem: 107563MB | Time Left: 00 hrs : 13 mins [TIMING] data: 7.6ms | fwd: 3576.8ms | loss: 13.1ms | bwd: 10853.3ms | clip: 0.0ms | optim: 23.8ms step 2171 | loss: 2.858919 | lr 8.1755e-05 | norm: 0.1363 | dt: 14487.46ms | tok/sec: 36189.1 | mem: 107563MB | Time Left: 00 hrs : 13 mins [TIMING] data: 7.7ms | fwd: 3577.5ms | loss: 13.2ms | bwd: 10851.3ms | clip: 0.0ms | optim: 23.9ms step 2172 | loss: 3.005199 | lr 8.1033e-05 | norm: 0.1408 | dt: 14495.83ms | tok/sec: 36168.2 | mem: 107563MB | Time Left: 00 hrs : 13 mins [TIMING] data: 7.6ms | fwd: 3586.8ms | loss: 13.1ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms step 2173 | loss: 2.521548 | lr 8.0321e-05 | norm: 0.1463 | dt: 14487.57ms | tok/sec: 36188.8 | mem: 107563MB | Time Left: 00 hrs : 13 mins [TIMING] data: 7.4ms | fwd: 3577.9ms | loss: 13.1ms | bwd: 10851.5ms | clip: 0.0ms | optim: 23.9ms step 2174 | loss: 2.857008 | lr 7.9619e-05 | norm: 0.1471 | dt: 14478.31ms | tok/sec: 36212.0 | mem: 107563MB | Time Left: 00 hrs : 12 mins [TIMING] data: 7.6ms | fwd: 3577.8ms | loss: 13.2ms | bwd: 10842.0ms | clip: 0.0ms | optim: 24.0ms step 2175 | loss: 2.839947 | lr 7.8927e-05 | norm: 0.1517 | dt: 14485.64ms | tok/sec: 36193.6 | mem: 107563MB | Time Left: 00 hrs : 12 mins [TIMING] data: 7.4ms | fwd: 3573.5ms | loss: 13.1ms | bwd: 10853.8ms | clip: 0.0ms | optim: 23.9ms step 2176 | loss: 2.818293 | lr 7.8245e-05 | norm: 0.1518 | dt: 14481.42ms | tok/sec: 36204.2 | mem: 107563MB | Time Left: 00 hrs : 12 mins [TIMING] data: 8.4ms | fwd: 3578.1ms | loss: 13.1ms | bwd: 10844.1ms | clip: 0.0ms | optim: 23.9ms step 2177 | loss: 2.714397 | lr 7.7574e-05 | norm: 0.2638 | dt: 14489.58ms | tok/sec: 36183.8 | mem: 107563MB | Time Left: 00 hrs : 12 mins [TIMING] data: 7.7ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.8ms step 2178 | loss: 2.793543 | lr 7.6912e-05 | norm: 0.1521 | dt: 14494.67ms | tok/sec: 36171.1 | mem: 107563MB | Time Left: 00 hrs : 11 mins [TIMING] data: 7.6ms | fwd: 3581.4ms | loss: 13.0ms | bwd: 10855.0ms | clip: 0.0ms | optim: 23.9ms step 2179 | loss: 2.886053 | lr 7.6262e-05 | norm: 0.1404 | dt: 14496.20ms | tok/sec: 36167.3 | mem: 107563MB | Time Left: 00 hrs : 11 mins [TIMING] data: 7.6ms | fwd: 3581.8ms | loss: 13.1ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.8ms step 2180 | loss: 2.888594 | lr 7.5622e-05 | norm: 0.1581 | dt: 14499.56ms | tok/sec: 36158.9 | mem: 107563MB | Time Left: 00 hrs : 11 mins [TIMING] data: 7.5ms | fwd: 3579.7ms | loss: 13.1ms | bwd: 10861.5ms | clip: 0.0ms | optim: 23.9ms step 2181 | loss: 2.685252 | lr 7.4993e-05 | norm: 0.1402 | dt: 14490.06ms | tok/sec: 36182.6 | mem: 107563MB | Time Left: 00 hrs : 11 mins [TIMING] data: 7.6ms | fwd: 3576.0ms | loss: 13.1ms | bwd: 10855.8ms | clip: 0.0ms | optim: 23.8ms step 2182 | loss: 2.334655 | lr 7.4376e-05 | norm: 0.1249 | dt: 14505.34ms | tok/sec: 36144.5 | mem: 107563MB | Time Left: 00 hrs : 10 mins [TIMING] data: 7.7ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10865.9ms | clip: 0.0ms | optim: 23.9ms step 2183 | loss: 2.869099 | lr 7.3769e-05 | norm: 0.1394 | dt: 14504.08ms | tok/sec: 36147.6 | mem: 107563MB | Time Left: 00 hrs : 10 mins [TIMING] data: 7.7ms | fwd: 3582.5ms | loss: 13.1ms | bwd: 10863.0ms | clip: 0.0ms | optim: 23.9ms step 2184 | loss: 2.828929 | lr 7.3174e-05 | norm: 0.1587 | dt: 14496.95ms | tok/sec: 36165.4 | mem: 107563MB | Time Left: 00 hrs : 10 mins [TIMING] data: 7.7ms | fwd: 3582.3ms | loss: 13.1ms | bwd: 10856.1ms | clip: 0.0ms | optim: 23.9ms step 2185 | loss: 2.809223 | lr 7.2591e-05 | norm: 0.1373 | dt: 14496.01ms | tok/sec: 36167.7 | mem: 107563MB | Time Left: 00 hrs : 10 mins [TIMING] data: 7.6ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10854.3ms | clip: 0.0ms | optim: 23.9ms step 2186 | loss: 2.996271 | lr 7.2019e-05 | norm: 0.1576 | dt: 14508.92ms | tok/sec: 36135.6 | mem: 107563MB | Time Left: 00 hrs : 09 mins [TIMING] data: 7.6ms | fwd: 3583.3ms | loss: 13.2ms | bwd: 10866.2ms | clip: 0.0ms | optim: 24.1ms step 2187 | loss: 2.729771 | lr 7.1459e-05 | norm: 0.1718 | dt: 14522.21ms | tok/sec: 36102.5 | mem: 107563MB | Time Left: 00 hrs : 09 mins [TIMING] data: 7.5ms | fwd: 3587.8ms | loss: 13.1ms | bwd: 10876.1ms | clip: 0.0ms | optim: 23.9ms step 2188 | loss: 2.856746 | lr 7.0911e-05 | norm: 0.1616 | dt: 14499.22ms | tok/sec: 36159.7 | mem: 107563MB | Time Left: 00 hrs : 09 mins [TIMING] data: 7.6ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10858.1ms | clip: 0.0ms | optim: 23.9ms step 2189 | loss: 2.729537 | lr 7.0375e-05 | norm: 0.1622 | dt: 14499.07ms | tok/sec: 36160.1 | mem: 107563MB | Time Left: 00 hrs : 09 mins [TIMING] data: 7.5ms | fwd: 3584.2ms | loss: 13.1ms | bwd: 10856.5ms | clip: 0.0ms | optim: 23.8ms step 2190 | loss: 2.720333 | lr 6.9852e-05 | norm: 0.1534 | dt: 14507.89ms | tok/sec: 36138.1 | mem: 107563MB | Time Left: 00 hrs : 08 mins [TIMING] data: 7.8ms | fwd: 3586.3ms | loss: 13.1ms | bwd: 10863.0ms | clip: 0.0ms | optim: 23.9ms step 2191 | loss: 3.101346 | lr 6.9340e-05 | norm: 0.1436 | dt: 14500.19ms | tok/sec: 36157.3 | mem: 107563MB | Time Left: 00 hrs : 08 mins [TIMING] data: 7.5ms | fwd: 3580.0ms | loss: 13.1ms | bwd: 10861.7ms | clip: 0.0ms | optim: 23.9ms step 2192 | loss: 2.798658 | lr 6.8842e-05 | norm: 0.1783 | dt: 14511.23ms | tok/sec: 36129.8 | mem: 107563MB | Time Left: 00 hrs : 08 mins [TIMING] data: 7.7ms | fwd: 3585.8ms | loss: 13.1ms | bwd: 10866.8ms | clip: 0.0ms | optim: 24.2ms step 2193 | loss: 2.838547 | lr 6.8355e-05 | norm: 0.1493 | dt: 14505.82ms | tok/sec: 36143.3 | mem: 107563MB | Time Left: 00 hrs : 08 mins [TIMING] data: 7.7ms | fwd: 3584.1ms | loss: 13.1ms | bwd: 10862.7ms | clip: 0.0ms | optim: 24.5ms step 2194 | loss: 2.643431 | lr 6.7882e-05 | norm: 0.1280 | dt: 14495.40ms | tok/sec: 36169.3 | mem: 107563MB | Time Left: 00 hrs : 07 mins [TIMING] data: 7.5ms | fwd: 3579.8ms | loss: 13.1ms | bwd: 10857.4ms | clip: 0.0ms | optim: 23.8ms step 2195 | loss: 3.026601 | lr 6.7422e-05 | norm: 0.1547 | dt: 14501.09ms | tok/sec: 36155.1 | mem: 107563MB | Time Left: 00 hrs : 07 mins [TIMING] data: 7.6ms | fwd: 3584.0ms | loss: 13.1ms | bwd: 10858.8ms | clip: 0.0ms | optim: 23.9ms step 2196 | loss: 2.784484 | lr 6.6974e-05 | norm: 0.1545 | dt: 14496.30ms | tok/sec: 36167.0 | mem: 107563MB | Time Left: 00 hrs : 07 mins [TIMING] data: 7.8ms | fwd: 3575.6ms | loss: 13.2ms | bwd: 10861.9ms | clip: 0.0ms | optim: 23.8ms step 2197 | loss: 2.626242 | lr 6.6540e-05 | norm: 0.1545 | dt: 14509.99ms | tok/sec: 36132.9 | mem: 107563MB | Time Left: 00 hrs : 07 mins [TIMING] data: 7.5ms | fwd: 3583.3ms | loss: 13.1ms | bwd: 10868.1ms | clip: 0.0ms | optim: 23.9ms step 2198 | loss: 2.901745 | lr 6.6118e-05 | norm: 0.1545 | dt: 14495.50ms | tok/sec: 36169.0 | mem: 107563MB | Time Left: 00 hrs : 07 mins [TIMING] data: 7.7ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10856.8ms | clip: 0.0ms | optim: 23.8ms step 2199 | loss: 2.876444 | lr 6.5710e-05 | norm: 0.2111 | dt: 14490.18ms | tok/sec: 36182.3 | mem: 107563MB | Time Left: 00 hrs : 06 mins [TIMING] data: 7.7ms | fwd: 3581.1ms | loss: 13.1ms | bwd: 10850.7ms | clip: 0.0ms | optim: 23.9ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2200.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2100.ckpt step 2200 | loss: 2.719528 | lr 6.5316e-05 | norm: 0.1438 | dt: 14404.43ms | tok/sec: 36397.7 | mem: 107563MB | Time Left: 00 hrs : 06 mins [TIMING] data: 7.6ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10761.5ms | clip: 0.0ms | optim: 23.8ms step 2201 | loss: 2.826389 | lr 6.4935e-05 | norm: 0.1593 | dt: 14426.00ms | tok/sec: 36343.3 | mem: 107563MB | Time Left: 00 hrs : 06 mins [TIMING] data: 7.6ms | fwd: 3570.1ms | loss: 13.1ms | bwd: 10796.5ms | clip: 0.0ms | optim: 23.9ms step 2202 | loss: 2.833322 | lr 6.4567e-05 | norm: 0.2008 | dt: 14443.67ms | tok/sec: 36298.8 | mem: 107563MB | Time Left: 00 hrs : 06 mins [TIMING] data: 7.6ms | fwd: 3573.9ms | loss: 13.1ms | bwd: 10811.5ms | clip: 0.0ms | optim: 23.9ms step 2203 | loss: 2.760280 | lr 6.4213e-05 | norm: 0.1481 | dt: 14472.07ms | tok/sec: 36227.6 | mem: 107563MB | Time Left: 00 hrs : 05 mins [TIMING] data: 7.5ms | fwd: 3578.2ms | loss: 13.1ms | bwd: 10835.5ms | clip: 0.0ms | optim: 23.9ms step 2204 | loss: 2.819014 | lr 6.3873e-05 | norm: 0.1657 | dt: 14476.77ms | tok/sec: 36215.8 | mem: 107563MB | Time Left: 00 hrs : 05 mins [TIMING] data: 7.8ms | fwd: 3576.9ms | loss: 13.2ms | bwd: 10841.2ms | clip: 0.0ms | optim: 23.9ms step 2205 | loss: 2.654979 | lr 6.3547e-05 | norm: 0.1409 | dt: 14488.07ms | tok/sec: 36187.6 | mem: 107563MB | Time Left: 00 hrs : 05 mins [TIMING] data: 7.5ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10850.4ms | clip: 0.0ms | optim: 23.9ms step 2206 | loss: 2.892439 | lr 6.3235e-05 | norm: 0.1695 | dt: 14474.11ms | tok/sec: 36222.5 | mem: 107563MB | Time Left: 00 hrs : 05 mins [TIMING] data: 7.4ms | fwd: 3575.5ms | loss: 13.1ms | bwd: 10840.1ms | clip: 0.0ms | optim: 23.9ms step 2207 | loss: 2.577019 | lr 6.2937e-05 | norm: 0.1310 | dt: 14484.98ms | tok/sec: 36195.3 | mem: 107563MB | Time Left: 00 hrs : 04 mins [TIMING] data: 7.6ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10848.2ms | clip: 0.0ms | optim: 23.8ms step 2208 | loss: 2.509301 | lr 6.2652e-05 | norm: 0.1679 | dt: 14485.26ms | tok/sec: 36194.6 | mem: 107563MB | Time Left: 00 hrs : 04 mins [TIMING] data: 7.6ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10848.4ms | clip: 0.0ms | optim: 23.8ms step 2209 | loss: 3.136025 | lr 6.2382e-05 | norm: 0.1824 | dt: 14491.84ms | tok/sec: 36178.2 | mem: 107563MB | Time Left: 00 hrs : 04 mins [TIMING] data: 7.5ms | fwd: 3582.8ms | loss: 13.1ms | bwd: 10850.0ms | clip: 0.0ms | optim: 24.0ms step 2210 | loss: 2.802453 | lr 6.2127e-05 | norm: 0.1415 | dt: 14484.42ms | tok/sec: 36196.7 | mem: 107563MB | Time Left: 00 hrs : 04 mins [TIMING] data: 7.6ms | fwd: 3576.7ms | loss: 13.1ms | bwd: 10849.3ms | clip: 0.0ms | optim: 23.8ms step 2211 | loss: 2.846933 | lr 6.1885e-05 | norm: 0.1883 | dt: 14496.68ms | tok/sec: 36166.1 | mem: 107563MB | Time Left: 00 hrs : 03 mins [TIMING] data: 7.6ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10858.1ms | clip: 0.0ms | optim: 23.9ms step 2212 | loss: 2.613862 | lr 6.1658e-05 | norm: 0.1328 | dt: 14489.96ms | tok/sec: 36182.8 | mem: 107563MB | Time Left: 00 hrs : 03 mins [TIMING] data: 7.7ms | fwd: 3579.2ms | loss: 13.1ms | bwd: 10852.4ms | clip: 0.0ms | optim: 23.9ms step 2213 | loss: 2.896386 | lr 6.1445e-05 | norm: 0.1598 | dt: 14482.73ms | tok/sec: 36200.9 | mem: 107563MB | Time Left: 00 hrs : 03 mins [TIMING] data: 7.7ms | fwd: 3578.9ms | loss: 13.1ms | bwd: 10845.4ms | clip: 0.0ms | optim: 23.8ms step 2214 | loss: 2.863654 | lr 6.1247e-05 | norm: 0.1434 | dt: 14495.79ms | tok/sec: 36168.3 | mem: 107563MB | Time Left: 00 hrs : 03 mins [TIMING] data: 7.5ms | fwd: 3578.4ms | loss: 13.1ms | bwd: 10859.1ms | clip: 0.0ms | optim: 23.9ms step 2215 | loss: 2.981959 | lr 6.1063e-05 | norm: 0.1403 | dt: 14482.50ms | tok/sec: 36201.5 | mem: 107563MB | Time Left: 00 hrs : 02 mins [TIMING] data: 7.6ms | fwd: 3581.6ms | loss: 13.2ms | bwd: 10842.3ms | clip: 0.0ms | optim: 23.8ms step 2216 | loss: 3.075194 | lr 6.0893e-05 | norm: 0.1826 | dt: 14488.67ms | tok/sec: 36186.1 | mem: 107563MB | Time Left: 00 hrs : 02 mins [TIMING] data: 7.4ms | fwd: 3578.7ms | loss: 13.1ms | bwd: 10851.8ms | clip: 0.0ms | optim: 23.8ms step 2217 | loss: 2.728479 | lr 6.0739e-05 | norm: 0.1446 | dt: 14492.69ms | tok/sec: 36176.0 | mem: 107563MB | Time Left: 00 hrs : 02 mins [TIMING] data: 7.5ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10856.7ms | clip: 0.0ms | optim: 23.9ms step 2218 | loss: 2.572904 | lr 6.0599e-05 | norm: 0.1892 | dt: 14489.32ms | tok/sec: 36184.4 | mem: 107563MB | Time Left: 00 hrs : 02 mins [TIMING] data: 7.8ms | fwd: 3577.5ms | loss: 13.1ms | bwd: 10853.4ms | clip: 0.0ms | optim: 23.8ms step 2219 | loss: 2.664694 | lr 6.0473e-05 | norm: 0.1318 | dt: 14494.34ms | tok/sec: 36171.9 | mem: 107563MB | Time Left: 00 hrs : 01 mins [TIMING] data: 7.6ms | fwd: 3576.9ms | loss: 13.1ms | bwd: 10859.1ms | clip: 0.0ms | optim: 23.8ms step 2220 | loss: 2.781899 | lr 6.0362e-05 | norm: 0.1375 | dt: 14498.35ms | tok/sec: 36161.9 | mem: 107563MB | Time Left: 00 hrs : 01 mins [TIMING] data: 7.6ms | fwd: 3579.1ms | loss: 13.1ms | bwd: 10860.8ms | clip: 0.0ms | optim: 23.9ms step 2221 | loss: 2.773037 | lr 6.0266e-05 | norm: 0.1397 | dt: 14494.99ms | tok/sec: 36170.3 | mem: 107563MB | Time Left: 00 hrs : 01 mins [TIMING] data: 7.7ms | fwd: 3580.3ms | loss: 13.2ms | bwd: 10855.9ms | clip: 0.0ms | optim: 23.9ms step 2222 | loss: 2.826401 | lr 6.0185e-05 | norm: 0.1713 | dt: 14500.11ms | tok/sec: 36157.5 | mem: 107563MB | Time Left: 00 hrs : 01 mins [TIMING] data: 7.5ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10861.2ms | clip: 0.0ms | optim: 24.0ms step 2223 | loss: 2.919726 | lr 6.0118e-05 | norm: 0.1330 | dt: 14504.31ms | tok/sec: 36147.1 | mem: 107563MB | Time Left: 00 hrs : 00 mins [TIMING] data: 7.5ms | fwd: 3583.7ms | loss: 13.1ms | bwd: 10862.1ms | clip: 0.0ms | optim: 24.0ms step 2224 | loss: 2.647196 | lr 6.0067e-05 | norm: 0.1803 | dt: 14495.96ms | tok/sec: 36167.9 | mem: 107563MB | Time Left: 00 hrs : 00 mins [TIMING] data: 7.7ms | fwd: 3579.0ms | loss: 13.1ms | bwd: 10857.2ms | clip: 0.0ms | optim: 23.9ms step 2225 | loss: 2.729517 | lr 6.0030e-05 | norm: 0.1352 | dt: 14495.64ms | tok/sec: 36168.7 | mem: 107563MB | Time Left: 00 hrs : 00 mins [TIMING] data: 7.9ms | fwd: 3582.9ms | loss: 13.1ms | bwd: 10853.9ms | clip: 0.0ms | optim: 24.0ms validation loss: 2.9357 step 2226 | loss: 2.757876 | lr 6.0007e-05 | norm: 0.1443 | dt: 19080.13ms | tok/sec: 27478.2 | mem: 107563MB | Time Left: 00 hrs : 00 mins [TIMING] data: 7.6ms | fwd: 3584.8ms | loss: 13.1ms | bwd: 10857.8ms | clip: 0.0ms | optim: 24.0ms [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2227.ckpt (safely) [CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_2150.ckpt === Training complete ===
=== Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
d_model/layers : 1536 / 36
q/kv heads : 12 / 4 (G=3, hd=128)
ffn_hidden : 4096 (SwiGLU)
context_length : 32768
vocab_size : 50304 tying=0
B x T : 1 x 32768
world_size : 2 (dp=1 x cp=2)
CP_ATTN_MODE : ring (load_balance=HeadTail)
global_batch : 524288 (grad_accum=16)
precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
max_lr/min_lr : 0.0006 / 6e-05
warmup/steps : 100 / 2227 (WSD: stable=1177, decay tail=950)
checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 2227
warmup_steps: 100
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 12 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (resume run 2)
[rope_theta] 500000 from checkpoints_bluscriptcp/blumodelcp_run2.meta (authoritative)
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 2)
[CheckpointManager] Auto-loading latest: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.ckpt
[Resume] run 2 from step 750 (loss 2.57022)
[resume] fast-forwarded data loader 12000 batches (393216000 tokens)
[re-warmup] anchored at step 750 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 750 (mid-ramp)
Starting training...
[CP ring P2P] OVERLAP: dedicated non-blocking stream
validation loss: 3.0712
[CP bwd] overlap=ON
step 750 | loss: 2.954028 | lr 1.8000e-06 | norm: 0.1530 | dt: 21471.66ms | tok/sec: 24417.7 | mem: 107435MB | Time Left: 08 hrs : 48 mins
[TIMING] data: 7.2ms | fwd: 3943.4ms | loss: 13.2ms | bwd: 10719.7ms | clip: 0.0ms | optim: 24.2ms
step 751 | loss: 3.161970 | lr 3.6000e-06 | norm: 0.1821 | dt: 14331.36ms | tok/sec: 36583.3 | mem: 107435MB | Time Left: 05 hrs : 52 mins
[TIMING] data: 6.8ms | fwd: 3549.6ms | loss: 13.1ms | bwd: 10724.8ms | clip: 0.0ms | optim: 23.8ms
step 752 | loss: 2.582914 | lr 5.4000e-06 | norm: 0.1560 | dt: 14459.85ms | tok/sec: 36258.2 | mem: 107435MB | Time Left: 05 hrs : 55 mins
[TIMING] data: 6.7ms | fwd: 3643.5ms | loss: 13.1ms | bwd: 10758.6ms | clip: 0.0ms | optim: 23.8ms
step 753 | loss: 2.722396 | lr 7.2000e-06 | norm: 0.1417 | dt: 14472.63ms | tok/sec: 36226.2 | mem: 107435MB | Time Left: 05 hrs : 55 mins
[TIMING] data: 6.5ms | fwd: 3578.6ms | loss: 13.1ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.8ms
step 754 | loss: 3.092211 | lr 9.0000e-06 | norm: 0.1586 | dt: 14571.99ms | tok/sec: 35979.2 | mem: 107435MB | Time Left: 05 hrs : 57 mins
[TIMING] data: 6.7ms | fwd: 3631.5ms | loss: 13.1ms | bwd: 10883.7ms | clip: 0.0ms | optim: 23.8ms
step 755 | loss: 3.240096 | lr 1.0800e-05 | norm: 0.1663 | dt: 14564.68ms | tok/sec: 35997.2 | mem: 107435MB | Time Left: 05 hrs : 57 mins
[TIMING] data: 6.6ms | fwd: 3588.9ms | loss: 13.1ms | bwd: 10919.1ms | clip: 0.0ms | optim: 23.7ms
step 756 | loss: 2.918514 | lr 1.2600e-05 | norm: 0.1260 | dt: 14550.79ms | tok/sec: 36031.6 | mem: 107435MB | Time Left: 05 hrs : 56 mins
[TIMING] data: 6.5ms | fwd: 3588.9ms | loss: 13.1ms | bwd: 10905.2ms | clip: 0.0ms | optim: 23.8ms
step 757 | loss: 2.768518 | lr 1.4400e-05 | norm: 0.1319 | dt: 14514.43ms | tok/sec: 36121.8 | mem: 107435MB | Time Left: 05 hrs : 55 mins
[TIMING] data: 6.7ms | fwd: 3598.3ms | loss: 13.0ms | bwd: 10859.3ms | clip: 0.0ms | optim: 23.8ms
step 758 | loss: 2.682922 | lr 1.6200e-05 | norm: 0.1442 | dt: 14521.89ms | tok/sec: 36103.3 | mem: 107435MB | Time Left: 05 hrs : 55 mins
[TIMING] data: 6.7ms | fwd: 3594.0ms | loss: 13.1ms | bwd: 10870.9ms | clip: 0.0ms | optim: 23.9ms
step 759 | loss: 3.084433 | lr 1.8000e-05 | norm: 0.2578 | dt: 14501.41ms | tok/sec: 36154.3 | mem: 107435MB | Time Left: 05 hrs : 54 mins
[TIMING] data: 6.5ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10865.0ms | clip: 0.0ms | optim: 23.9ms
step 760 | loss: 3.090617 | lr 1.9800e-05 | norm: 0.1488 | dt: 14482.85ms | tok/sec: 36200.6 | mem: 107435MB | Time Left: 05 hrs : 54 mins
[TIMING] data: 6.5ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10846.5ms | clip: 0.0ms | optim: 23.8ms
step 761 | loss: 2.709423 | lr 2.1600e-05 | norm: 0.1968 | dt: 14475.07ms | tok/sec: 36220.1 | mem: 107435MB | Time Left: 05 hrs : 53 mins
[TIMING] data: 6.9ms | fwd: 3580.4ms | loss: 13.1ms | bwd: 10837.5ms | clip: 0.0ms | optim: 23.8ms
step 762 | loss: 2.803009 | lr 2.3400e-05 | norm: 0.1644 | dt: 14530.08ms | tok/sec: 36082.9 | mem: 107435MB | Time Left: 05 hrs : 54 mins
[TIMING] data: 6.6ms | fwd: 3642.3ms | loss: 13.1ms | bwd: 10831.1ms | clip: 0.0ms | optim: 23.8ms
step 763 | loss: 2.581849 | lr 2.5200e-05 | norm: 0.1512 | dt: 14485.62ms | tok/sec: 36193.7 | mem: 107435MB | Time Left: 05 hrs : 53 mins
[TIMING] data: 6.8ms | fwd: 3588.4ms | loss: 13.1ms | bwd: 10839.8ms | clip: 0.0ms | optim: 23.9ms
step 764 | loss: 2.728063 | lr 2.7000e-05 | norm: 0.1384 | dt: 14487.22ms | tok/sec: 36189.7 | mem: 107435MB | Time Left: 05 hrs : 53 mins
[TIMING] data: 6.7ms | fwd: 3580.3ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.8ms
step 765 | loss: 2.939891 | lr 2.8800e-05 | norm: 0.1529 | dt: 14494.60ms | tok/sec: 36171.3 | mem: 107435MB | Time Left: 05 hrs : 53 mins
[TIMING] data: 6.5ms | fwd: 3603.7ms | loss: 13.1ms | bwd: 10834.2ms | clip: 0.0ms | optim: 23.9ms
step 766 | loss: 2.942754 | lr 3.0600e-05 | norm: 0.1718 | dt: 14479.83ms | tok/sec: 36208.2 | mem: 107435MB | Time Left: 05 hrs : 52 mins
[TIMING] data: 6.7ms | fwd: 3572.8ms | loss: 13.1ms | bwd: 10850.2ms | clip: 0.0ms | optim: 23.9ms
step 767 | loss: 3.021307 | lr 3.2400e-05 | norm: 0.1802 | dt: 14470.71ms | tok/sec: 36231.0 | mem: 107435MB | Time Left: 05 hrs : 52 mins
[TIMING] data: 6.6ms | fwd: 3578.6ms | loss: 13.0ms | bwd: 10835.4ms | clip: 0.0ms | optim: 23.8ms
step 768 | loss: 2.818981 | lr 3.4200e-05 | norm: 0.1527 | dt: 14474.57ms | tok/sec: 36221.3 | mem: 107435MB | Time Left: 05 hrs : 51 mins
[TIMING] data: 6.6ms | fwd: 3579.6ms | loss: 13.1ms | bwd: 10838.2ms | clip: 0.0ms | optim: 23.9ms
step 769 | loss: 2.896601 | lr 3.6000e-05 | norm: 0.1379 | dt: 14496.02ms | tok/sec: 36167.7 | mem: 107435MB | Time Left: 05 hrs : 52 mins
[TIMING] data: 6.5ms | fwd: 3605.2ms | loss: 13.0ms | bwd: 10834.1ms | clip: 0.0ms | optim: 23.9ms
step 770 | loss: 2.832743 | lr 3.7800e-05 | norm: 0.1875 | dt: 14474.14ms | tok/sec: 36222.4 | mem: 107435MB | Time Left: 05 hrs : 51 mins
[TIMING] data: 6.5ms | fwd: 3584.2ms | loss: 13.0ms | bwd: 10833.3ms | clip: 0.0ms | optim: 23.8ms
step 771 | loss: 3.000637 | lr 3.9600e-05 | norm: 0.1732 | dt: 14474.20ms | tok/sec: 36222.2 | mem: 107435MB | Time Left: 05 hrs : 51 mins
[TIMING] data: 6.6ms | fwd: 3577.3ms | loss: 13.1ms | bwd: 10840.1ms | clip: 0.0ms | optim: 23.8ms
step 772 | loss: 2.885429 | lr 4.1400e-05 | norm: 0.1665 | dt: 14471.80ms | tok/sec: 36228.2 | mem: 107435MB | Time Left: 05 hrs : 50 mins
[TIMING] data: 6.6ms | fwd: 3579.5ms | loss: 13.1ms | bwd: 10835.6ms | clip: 0.0ms | optim: 23.8ms
step 773 | loss: 2.760646 | lr 4.3200e-05 | norm: 0.1863 | dt: 14476.03ms | tok/sec: 36217.7 | mem: 107435MB | Time Left: 05 hrs : 50 mins