BluScriptCP / logs /in2 /arm_a_train.log
Upload folder using huggingface_hub
1db5d09 verified 3 months ago
101 kB
=== Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[ZeRO-1] CP_ZERO_SYMMETRIC=1: NCCL symmetric memory mode (NVLS)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
  d_model/layers  : 1536 / 36
  q/kv heads      : 12 / 4  (G=3, hd=128)
  ffn_hidden      : 4096 (SwiGLU)
  context_length  : 16384
  vocab_size      : 50304  tying=0
  B x T           : 4 x 16384
  world_size      : 4  (dp=1 x cp=4)
  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
  global_batch    : 524288  (grad_accum=8)
  optimizer       : Muon (2-D body) + AdamW (wte + 1-D)  [ZeRO-1]  muon_lr=0.001000
  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
  max_lr/min_lr   : 0.001 / 0.0001
  warmup/steps    : 1 / 400  (cosine, no stable phase)
  checkpointing   : true  (ckpt_freq=100, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt  (base step 750, loss 1.28324)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 400
warmup_steps: 1
[Muon] adjust_lr=match_rms_adamw (default)  ns_steps=5  momentum=0.95
[ZeRO-1/Muon] reduce-scatter/all-gather layout: 6 bucket(s), 12 collectives/step, shard 226492416 elems/rank, padding 0.000000% (flat buffer 1728.000000 MB)
[ZeRO-1/Muon] adjust_lr=match_rms_adamw | params=905969664 (216 orthogonalized / 0 AdamW) muon_elems=905969664 | W=4 owned=226492416 (42 tensors) | master+momentum: 1728.000000 MB/GPU
[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype)  symmetric=ON  overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
[ZeRO-1 v2] params=77266944 padded=77266944 W=4 shard=19316736 | 3 buckets of 50.000000 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 221.062500 MB (vs 884.250000 MB unsharded)
[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype)  symmetric=ON  overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
[ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50.000000 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.409668 MB (vs 885.638672 MB unsharded)
[ZeRO-1] variant=v2 reduce-scatter/all-gather  groups=3  [muon:bfloat16/905M]  [adamw:bfloat16/77266944]  [adamw:float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 3 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log12.csv (run 12)
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run12_step_<N>.ckpt (every 100 steps, keep 2)
[re-warmup] anchored at step 0 for 100 steps -> peak 0.000300 (0.300000 x max_lr), then cosine-decay to 0.000100 | resuming at step 0 (mid-ramp)

Starting training...

[CP ring P2P] OVERLAP: dedicated non-blocking stream
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_a/in2a_val_00000.bin
100000000
validation loss: 1.2949
[CP bwd] overlap=ON
step     0 | loss: 1.339281 | lr 3.0000e-06 | norm: 0.4609 | dt: 15903.50ms | tok/sec: 32966.8 | mem: 107213MB | Time Left: 01 hrs : 46 mins
  [TIMING] data: 3.3ms | fwd: 4207.3ms | loss: 6.5ms | bwd: 4591.9ms | clip: 0.0ms | optim: 62.6ms
step     1 | loss: 1.293756 | lr 6.0000e-06 | norm: 0.4212 | dt: 8306.62ms | tok/sec: 63116.9 | mem: 108909MB | Time Left: 00 hrs : 55 mins
  [TIMING] data: 3.6ms | fwd: 4053.1ms | loss: 6.5ms | bwd: 4185.8ms | clip: 0.0ms | optim: 49.6ms
step     2 | loss: 1.312149 | lr 9.0000e-06 | norm: 0.5693 | dt: 8234.27ms | tok/sec: 63671.4 | mem: 108909MB | Time Left: 00 hrs : 54 mins
  [TIMING] data: 5.0ms | fwd: 4017.7ms | loss: 6.4ms | bwd: 4147.2ms | clip: 0.0ms | optim: 49.9ms
step     3 | loss: 1.350716 | lr 1.2000e-05 | norm: 0.5305 | dt: 6207.30ms | tok/sec: 84463.1 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 1993.9ms | loss: 6.5ms | bwd: 4146.1ms | clip: 0.0ms | optim: 49.4ms
step     4 | loss: 1.333467 | lr 1.5000e-05 | norm: 0.5074 | dt: 6218.86ms | tok/sec: 84306.2 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.4ms | fwd: 1994.6ms | loss: 6.5ms | bwd: 4157.1ms | clip: 0.0ms | optim: 49.3ms
step     5 | loss: 1.408316 | lr 1.8000e-05 | norm: 0.4886 | dt: 6243.31ms | tok/sec: 83975.9 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2002.3ms | loss: 6.5ms | bwd: 4172.8ms | clip: 0.0ms | optim: 50.1ms
step     6 | loss: 1.340124 | lr 2.1000e-05 | norm: 0.5491 | dt: 6298.95ms | tok/sec: 83234.1 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.6ms | fwd: 2017.9ms | loss: 6.5ms | bwd: 4212.9ms | clip: 0.0ms | optim: 50.0ms
step     7 | loss: 1.304078 | lr 2.4000e-05 | norm: 0.5952 | dt: 6327.50ms | tok/sec: 82858.7 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2018.3ms | loss: 6.5ms | bwd: 4241.7ms | clip: 0.0ms | optim: 49.4ms
step     8 | loss: 1.294508 | lr 2.7000e-05 | norm: 0.5823 | dt: 6396.76ms | tok/sec: 81961.5 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2054.9ms | loss: 6.5ms | bwd: 4274.2ms | clip: 0.0ms | optim: 49.6ms
step     9 | loss: 1.339643 | lr 3.0000e-05 | norm: 0.5572 | dt: 6400.50ms | tok/sec: 81913.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.3ms | fwd: 2085.0ms | loss: 6.5ms | bwd: 4248.3ms | clip: 0.0ms | optim: 49.5ms
step    10 | loss: 1.296743 | lr 3.3000e-05 | norm: 0.4024 | dt: 6345.26ms | tok/sec: 82626.7 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.4ms | fwd: 2020.3ms | loss: 6.5ms | bwd: 4257.3ms | clip: 0.0ms | optim: 49.7ms
step    11 | loss: 1.312237 | lr 3.6000e-05 | norm: 0.5541 | dt: 6353.10ms | tok/sec: 82524.8 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2023.8ms | loss: 6.5ms | bwd: 4260.8ms | clip: 0.0ms | optim: 50.5ms
step    12 | loss: 1.311145 | lr 3.9000e-05 | norm: 0.4285 | dt: 6334.82ms | tok/sec: 82762.9 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.4ms | fwd: 2020.0ms | loss: 6.6ms | bwd: 4247.1ms | clip: 0.0ms | optim: 49.7ms
step    13 | loss: 1.326215 | lr 4.2000e-05 | norm: 0.4568 | dt: 6309.56ms | tok/sec: 83094.2 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2018.3ms | loss: 6.5ms | bwd: 4223.2ms | clip: 0.0ms | optim: 50.1ms
step    14 | loss: 1.368519 | lr 4.5000e-05 | norm: 0.8183 | dt: 6299.72ms | tok/sec: 83224.0 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4218.0ms | clip: 0.0ms | optim: 50.0ms
step    15 | loss: 1.356700 | lr 4.8000e-05 | norm: 0.4792 | dt: 6285.38ms | tok/sec: 83413.9 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.4ms | fwd: 2007.2ms | loss: 6.5ms | bwd: 4210.5ms | clip: 0.0ms | optim: 49.8ms
step    16 | loss: 1.342912 | lr 5.1000e-05 | norm: 0.5263 | dt: 6281.11ms | tok/sec: 83470.5 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2006.4ms | loss: 6.5ms | bwd: 4206.8ms | clip: 0.0ms | optim: 49.9ms
step    17 | loss: 1.281001 | lr 5.4000e-05 | norm: 0.4786 | dt: 6287.56ms | tok/sec: 83385.0 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.4ms | fwd: 2008.0ms | loss: 6.5ms | bwd: 4212.1ms | clip: 0.0ms | optim: 49.6ms
step    18 | loss: 1.320383 | lr 5.7000e-05 | norm: 0.4215 | dt: 6258.54ms | tok/sec: 83771.6 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.4ms | fwd: 2004.9ms | loss: 6.5ms | bwd: 4185.9ms | clip: 0.0ms | optim: 49.9ms
step    19 | loss: 1.339903 | lr 6.0000e-05 | norm: 0.5262 | dt: 6269.64ms | tok/sec: 83623.3 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2006.0ms | loss: 6.5ms | bwd: 4196.1ms | clip: 0.0ms | optim: 49.6ms
step    20 | loss: 1.317470 | lr 6.3000e-05 | norm: 0.5163 | dt: 6280.50ms | tok/sec: 83478.7 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.6ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4204.7ms | clip: 0.0ms | optim: 50.4ms
step    21 | loss: 1.395066 | lr 6.6000e-05 | norm: 0.4968 | dt: 6266.54ms | tok/sec: 83664.6 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2007.1ms | loss: 6.5ms | bwd: 4191.8ms | clip: 0.0ms | optim: 49.7ms
step    22 | loss: 1.330080 | lr 6.9000e-05 | norm: 0.4109 | dt: 6268.90ms | tok/sec: 83633.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2002.9ms | loss: 6.5ms | bwd: 4198.5ms | clip: 0.0ms | optim: 49.5ms
step    23 | loss: 1.333572 | lr 7.2000e-05 | norm: 0.5038 | dt: 6262.54ms | tok/sec: 83718.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.4ms | fwd: 2012.9ms | loss: 6.5ms | bwd: 4182.1ms | clip: 0.0ms | optim: 49.5ms
step    24 | loss: 1.304763 | lr 7.5000e-05 | norm: 0.5002 | dt: 6271.37ms | tok/sec: 83600.2 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.4ms | fwd: 2017.1ms | loss: 6.5ms | bwd: 4187.0ms | clip: 0.0ms | optim: 49.3ms
step    25 | loss: 1.386602 | lr 7.8000e-05 | norm: 0.4604 | dt: 6270.59ms | tok/sec: 83610.6 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.4ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4192.1ms | clip: 0.0ms | optim: 49.5ms
step    26 | loss: 1.291438 | lr 8.1000e-05 | norm: 0.5215 | dt: 6254.86ms | tok/sec: 83820.9 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2008.5ms | loss: 6.5ms | bwd: 4179.2ms | clip: 0.0ms | optim: 49.3ms
step    27 | loss: 1.305346 | lr 8.4000e-05 | norm: 0.5344 | dt: 6256.12ms | tok/sec: 83804.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2008.6ms | loss: 6.5ms | bwd: 4179.3ms | clip: 0.0ms | optim: 50.2ms
step    28 | loss: 1.341260 | lr 8.7000e-05 | norm: 0.4362 | dt: 6269.88ms | tok/sec: 83620.1 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.6ms | fwd: 2002.8ms | loss: 6.5ms | bwd: 4199.5ms | clip: 0.0ms | optim: 49.6ms
step    29 | loss: 1.325810 | lr 9.0000e-05 | norm: 0.4675 | dt: 6267.44ms | tok/sec: 83652.7 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2010.5ms | loss: 6.5ms | bwd: 4189.6ms | clip: 0.0ms | optim: 49.4ms
step    30 | loss: 1.296540 | lr 9.3000e-05 | norm: 0.4909 | dt: 6261.86ms | tok/sec: 83727.2 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2001.8ms | loss: 6.5ms | bwd: 4192.5ms | clip: 0.0ms | optim: 49.5ms
step    31 | loss: 1.373392 | lr 9.6000e-05 | norm: 0.5038 | dt: 6286.11ms | tok/sec: 83404.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2006.9ms | loss: 6.4ms | bwd: 4212.2ms | clip: 0.0ms | optim: 49.3ms
step    32 | loss: 1.278024 | lr 9.9000e-05 | norm: 0.5274 | dt: 6255.45ms | tok/sec: 83813.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2001.5ms | loss: 6.4ms | bwd: 4186.9ms | clip: 0.0ms | optim: 49.3ms
step    33 | loss: 1.349225 | lr 1.0200e-04 | norm: 0.5121 | dt: 6301.24ms | tok/sec: 83203.9 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2015.1ms | loss: 6.4ms | bwd: 4218.2ms | clip: 0.0ms | optim: 50.3ms
step    34 | loss: 1.328486 | lr 1.0500e-04 | norm: 0.5185 | dt: 6263.76ms | tok/sec: 83701.8 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2003.4ms | loss: 6.4ms | bwd: 4192.9ms | clip: 0.0ms | optim: 49.8ms
step    35 | loss: 1.337970 | lr 1.0800e-04 | norm: 0.4261 | dt: 6271.29ms | tok/sec: 83601.4 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2005.8ms | loss: 6.4ms | bwd: 4198.0ms | clip: 0.0ms | optim: 49.8ms
step    36 | loss: 1.324051 | lr 1.1100e-04 | norm: 0.5229 | dt: 6279.23ms | tok/sec: 83495.6 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.3ms | fwd: 2012.1ms | loss: 6.4ms | bwd: 4199.8ms | clip: 0.0ms | optim: 49.6ms
step    37 | loss: 1.324726 | lr 1.1400e-04 | norm: 0.5633 | dt: 6299.07ms | tok/sec: 83232.6 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.3ms | fwd: 2013.4ms | loss: 6.4ms | bwd: 4217.8ms | clip: 0.0ms | optim: 50.3ms
step    38 | loss: 1.316116 | lr 1.1700e-04 | norm: 0.5158 | dt: 6260.18ms | tok/sec: 83749.7 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.3ms | fwd: 2005.0ms | loss: 6.4ms | bwd: 4187.9ms | clip: 0.0ms | optim: 49.6ms
step    39 | loss: 1.348061 | lr 1.2000e-04 | norm: 0.5382 | dt: 6284.03ms | tok/sec: 83431.8 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.4ms | fwd: 2012.3ms | loss: 6.4ms | bwd: 4204.6ms | clip: 0.0ms | optim: 49.4ms
step    40 | loss: 1.304712 | lr 1.2300e-04 | norm: 0.5223 | dt: 6297.13ms | tok/sec: 83258.3 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.3ms | fwd: 2007.4ms | loss: 6.4ms | bwd: 4222.3ms | clip: 0.0ms | optim: 49.8ms
step    41 | loss: 1.286568 | lr 1.2600e-04 | norm: 0.5137 | dt: 6278.64ms | tok/sec: 83503.4 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.2ms | fwd: 2008.5ms | loss: 6.4ms | bwd: 4202.7ms | clip: 0.0ms | optim: 49.9ms
step    42 | loss: 1.342654 | lr 1.2900e-04 | norm: 0.5053 | dt: 6289.85ms | tok/sec: 83354.6 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.3ms | fwd: 2008.3ms | loss: 6.4ms | bwd: 4214.4ms | clip: 0.0ms | optim: 49.5ms
step    43 | loss: 1.350590 | lr 1.3200e-04 | norm: 0.5024 | dt: 6273.01ms | tok/sec: 83578.4 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.3ms | fwd: 2006.4ms | loss: 6.4ms | bwd: 4198.8ms | clip: 0.0ms | optim: 50.3ms
step    44 | loss: 1.265114 | lr 1.3500e-04 | norm: 0.5557 | dt: 6265.24ms | tok/sec: 83682.1 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.4ms | fwd: 2007.8ms | loss: 6.4ms | bwd: 4190.0ms | clip: 0.0ms | optim: 49.8ms
step    45 | loss: 1.346877 | lr 1.3800e-04 | norm: 0.6119 | dt: 6295.65ms | tok/sec: 83277.9 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.4ms | fwd: 2007.5ms | loss: 6.4ms | bwd: 4220.9ms | clip: 0.0ms | optim: 49.6ms
step    46 | loss: 1.335825 | lr 1.4100e-04 | norm: 0.5600 | dt: 6310.76ms | tok/sec: 83078.4 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.4ms | fwd: 2014.0ms | loss: 6.4ms | bwd: 4228.9ms | clip: 0.0ms | optim: 50.0ms
step    47 | loss: 1.367033 | lr 1.4400e-04 | norm: 0.5652 | dt: 6282.05ms | tok/sec: 83458.0 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.3ms | fwd: 2008.1ms | loss: 6.4ms | bwd: 4206.7ms | clip: 0.0ms | optim: 49.8ms
step    48 | loss: 1.362973 | lr 1.4700e-04 | norm: 0.6819 | dt: 6286.66ms | tok/sec: 83396.9 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.3ms | fwd: 2008.4ms | loss: 6.4ms | bwd: 4210.9ms | clip: 0.0ms | optim: 49.9ms
step    49 | loss: 1.340159 | lr 1.5000e-04 | norm: 0.5457 | dt: 6296.07ms | tok/sec: 83272.2 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.3ms | fwd: 2010.1ms | loss: 6.4ms | bwd: 4218.1ms | clip: 0.0ms | optim: 50.3ms
step    50 | loss: 1.380490 | lr 1.5300e-04 | norm: 0.5828 | dt: 6279.76ms | tok/sec: 83488.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.3ms | fwd: 2006.9ms | loss: 6.4ms | bwd: 4205.8ms | clip: 0.0ms | optim: 49.5ms
step    51 | loss: 1.399287 | lr 1.5600e-04 | norm: 0.6035 | dt: 6277.29ms | tok/sec: 83521.4 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.3ms | fwd: 2012.6ms | loss: 6.4ms | bwd: 4197.3ms | clip: 0.0ms | optim: 49.8ms
step    52 | loss: 1.412886 | lr 1.5900e-04 | norm: 0.5649 | dt: 6288.65ms | tok/sec: 83370.5 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.3ms | fwd: 2004.4ms | loss: 6.4ms | bwd: 4217.2ms | clip: 0.0ms | optim: 49.5ms
step    53 | loss: 1.322852 | lr 1.6200e-04 | norm: 0.6452 | dt: 6281.73ms | tok/sec: 83462.4 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.4ms | fwd: 2009.0ms | loss: 6.4ms | bwd: 4205.2ms | clip: 0.0ms | optim: 49.9ms
step    54 | loss: 1.332037 | lr 1.6500e-04 | norm: 0.7218 | dt: 6320.49ms | tok/sec: 82950.5 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.3ms | fwd: 2011.2ms | loss: 6.4ms | bwd: 4242.0ms | clip: 0.0ms | optim: 49.7ms
step    55 | loss: 1.426419 | lr 1.6800e-04 | norm: 0.5305 | dt: 6291.80ms | tok/sec: 83328.8 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.4ms | fwd: 2012.6ms | loss: 6.4ms | bwd: 4211.9ms | clip: 0.0ms | optim: 49.6ms
step    56 | loss: 1.385721 | lr 1.7100e-04 | norm: 0.5574 | dt: 6275.59ms | tok/sec: 83544.0 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.3ms | fwd: 2009.9ms | loss: 6.4ms | bwd: 4198.7ms | clip: 0.0ms | optim: 49.5ms
step    57 | loss: 1.316328 | lr 1.7400e-04 | norm: 0.5315 | dt: 6300.96ms | tok/sec: 83207.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.4ms | fwd: 2016.2ms | loss: 6.4ms | bwd: 4217.6ms | clip: 0.0ms | optim: 49.5ms
step    58 | loss: 1.336606 | lr 1.7700e-04 | norm: 0.5579 | dt: 6267.41ms | tok/sec: 83653.1 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.2ms | fwd: 2007.6ms | loss: 6.4ms | bwd: 4192.8ms | clip: 0.0ms | optim: 49.6ms
step    59 | loss: 1.338307 | lr 1.8000e-04 | norm: 0.5564 | dt: 6268.46ms | tok/sec: 83639.0 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.4ms | fwd: 2005.6ms | loss: 6.4ms | bwd: 4194.8ms | clip: 0.0ms | optim: 50.4ms
step    60 | loss: 1.386156 | lr 1.8300e-04 | norm: 0.5604 | dt: 6274.82ms | tok/sec: 83554.2 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.3ms | fwd: 2008.9ms | loss: 6.4ms | bwd: 4199.1ms | clip: 0.0ms | optim: 49.3ms
step    61 | loss: 1.312847 | lr 1.8600e-04 | norm: 0.7834 | dt: 6273.96ms | tok/sec: 83565.7 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.2ms | fwd: 2006.3ms | loss: 6.4ms | bwd: 4200.9ms | clip: 0.0ms | optim: 49.3ms
step    62 | loss: 1.348668 | lr 1.8900e-04 | norm: 0.7028 | dt: 6283.37ms | tok/sec: 83440.6 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.2ms | fwd: 2008.2ms | loss: 6.4ms | bwd: 4208.2ms | clip: 0.0ms | optim: 49.5ms
step    63 | loss: 1.348439 | lr 1.9200e-04 | norm: 0.6487 | dt: 6282.39ms | tok/sec: 83453.7 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.2ms | fwd: 2008.8ms | loss: 6.4ms | bwd: 4205.9ms | clip: 0.0ms | optim: 50.2ms
step    64 | loss: 1.311772 | lr 1.9500e-04 | norm: 0.5226 | dt: 6278.65ms | tok/sec: 83503.3 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.2ms | fwd: 2010.9ms | loss: 6.4ms | bwd: 4200.6ms | clip: 0.0ms | optim: 49.7ms
step    65 | loss: 1.364191 | lr 1.9800e-04 | norm: 0.7063 | dt: 6271.51ms | tok/sec: 83598.3 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.2ms | fwd: 2007.1ms | loss: 6.4ms | bwd: 4197.8ms | clip: 0.0ms | optim: 49.3ms
step    66 | loss: 1.342329 | lr 2.0100e-04 | norm: 0.5430 | dt: 6286.75ms | tok/sec: 83395.7 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.2ms | fwd: 2010.0ms | loss: 6.4ms | bwd: 4209.4ms | clip: 0.0ms | optim: 49.9ms
step    67 | loss: 1.387063 | lr 2.0400e-04 | norm: 0.5621 | dt: 6297.75ms | tok/sec: 83250.0 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.3ms | fwd: 2013.8ms | loss: 6.4ms | bwd: 4217.0ms | clip: 0.0ms | optim: 49.3ms
step    68 | loss: 1.359468 | lr 2.0700e-04 | norm: 0.6248 | dt: 6290.34ms | tok/sec: 83348.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.3ms | fwd: 2010.7ms | loss: 6.4ms | bwd: 4211.8ms | clip: 0.0ms | optim: 50.3ms
step    69 | loss: 1.407157 | lr 2.1000e-04 | norm: 0.5663 | dt: 6284.27ms | tok/sec: 83428.6 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.3ms | fwd: 2002.5ms | loss: 6.4ms | bwd: 4214.3ms | clip: 0.0ms | optim: 49.9ms
step    70 | loss: 1.360724 | lr 2.1300e-04 | norm: 0.6477 | dt: 6262.20ms | tok/sec: 83722.7 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.3ms | fwd: 2003.5ms | loss: 6.4ms | bwd: 4191.9ms | clip: 0.0ms | optim: 49.3ms
step    71 | loss: 1.374638 | lr 2.1600e-04 | norm: 0.5484 | dt: 6298.64ms | tok/sec: 83238.3 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.2ms | fwd: 2010.8ms | loss: 6.4ms | bwd: 4220.9ms | clip: 0.0ms | optim: 49.5ms
step    72 | loss: 1.298399 | lr 2.1900e-04 | norm: 0.5469 | dt: 6283.73ms | tok/sec: 83435.8 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.2ms | fwd: 2010.6ms | loss: 6.4ms | bwd: 4206.0ms | clip: 0.0ms | optim: 49.8ms
step    73 | loss: 1.381727 | lr 2.2200e-04 | norm: 0.6936 | dt: 6269.74ms | tok/sec: 83621.9 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.2ms | fwd: 2004.4ms | loss: 6.4ms | bwd: 4198.6ms | clip: 0.0ms | optim: 49.4ms
step    74 | loss: 1.360476 | lr 2.2500e-04 | norm: 0.6300 | dt: 6288.64ms | tok/sec: 83370.6 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.3ms | fwd: 2004.7ms | loss: 6.4ms | bwd: 4217.0ms | clip: 0.0ms | optim: 49.5ms
step    75 | loss: 1.374706 | lr 2.2800e-04 | norm: 0.6576 | dt: 6284.98ms | tok/sec: 83419.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.3ms | fwd: 2009.5ms | loss: 6.4ms | bwd: 4208.5ms | clip: 0.0ms | optim: 49.5ms
step    76 | loss: 1.324301 | lr 2.3100e-04 | norm: 0.9681 | dt: 6271.95ms | tok/sec: 83592.5 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.3ms | fwd: 2004.3ms | loss: 6.4ms | bwd: 4200.7ms | clip: 0.0ms | optim: 49.4ms
step    77 | loss: 1.386806 | lr 2.3400e-04 | norm: 0.6393 | dt: 6306.63ms | tok/sec: 83132.9 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.3ms | fwd: 2007.5ms | loss: 6.4ms | bwd: 4232.2ms | clip: 0.0ms | optim: 49.4ms
step    78 | loss: 1.358803 | lr 2.3700e-04 | norm: 0.7661 | dt: 6298.03ms | tok/sec: 83246.3 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.2ms | fwd: 2004.6ms | loss: 6.4ms | bwd: 4226.4ms | clip: 0.0ms | optim: 49.5ms
step    79 | loss: 1.369025 | lr 2.4000e-04 | norm: 0.7244 | dt: 6282.02ms | tok/sec: 83458.5 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.4ms | fwd: 2013.5ms | loss: 6.4ms | bwd: 4201.2ms | clip: 0.0ms | optim: 49.6ms
step    80 | loss: 1.366161 | lr 2.4300e-04 | norm: 0.7897 | dt: 6300.06ms | tok/sec: 83219.5 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.4ms | fwd: 2009.2ms | loss: 6.4ms | bwd: 4223.4ms | clip: 0.0ms | optim: 49.8ms
step    81 | loss: 1.420741 | lr 2.4600e-04 | norm: 0.6751 | dt: 6276.88ms | tok/sec: 83526.8 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.4ms | fwd: 2009.6ms | loss: 6.4ms | bwd: 4200.2ms | clip: 0.0ms | optim: 49.4ms
step    82 | loss: 1.449490 | lr 2.4900e-04 | norm: 0.6504 | dt: 6302.70ms | tok/sec: 83184.7 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.3ms | fwd: 2013.4ms | loss: 6.4ms | bwd: 4222.1ms | clip: 0.0ms | optim: 49.6ms
step    83 | loss: 1.364364 | lr 2.5200e-04 | norm: 0.7181 | dt: 6287.53ms | tok/sec: 83385.3 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.2ms | fwd: 2011.4ms | loss: 6.4ms | bwd: 4209.3ms | clip: 0.0ms | optim: 49.3ms
step    84 | loss: 1.382751 | lr 2.5500e-04 | norm: 0.6228 | dt: 6275.37ms | tok/sec: 83547.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.3ms | fwd: 2010.4ms | loss: 6.4ms | bwd: 4197.8ms | clip: 0.0ms | optim: 49.7ms
step    85 | loss: 1.382727 | lr 2.5800e-04 | norm: 0.6895 | dt: 6278.23ms | tok/sec: 83508.9 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.1ms | fwd: 2008.4ms | loss: 6.4ms | bwd: 4203.1ms | clip: 0.0ms | optim: 49.5ms
step    86 | loss: 1.377730 | lr 2.6100e-04 | norm: 0.6594 | dt: 6281.10ms | tok/sec: 83470.7 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.2ms | fwd: 2014.9ms | loss: 6.4ms | bwd: 4199.2ms | clip: 0.0ms | optim: 49.7ms
step    87 | loss: 1.459780 | lr 2.6400e-04 | norm: 0.9010 | dt: 6281.16ms | tok/sec: 83470.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.1ms | fwd: 2012.2ms | loss: 6.4ms | bwd: 4202.1ms | clip: 0.0ms | optim: 49.5ms
step    88 | loss: 1.464082 | lr 2.6700e-04 | norm: 0.8791 | dt: 6301.08ms | tok/sec: 83206.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.3ms | fwd: 2013.5ms | loss: 6.4ms | bwd: 4220.5ms | clip: 0.0ms | optim: 49.6ms
step    89 | loss: 1.402862 | lr 2.7000e-04 | norm: 0.9981 | dt: 6294.04ms | tok/sec: 83299.2 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.4ms | fwd: 2009.1ms | loss: 6.4ms | bwd: 4217.9ms | clip: 0.0ms | optim: 49.4ms
step    90 | loss: 1.480830 | lr 2.7300e-04 | norm: 0.6865 | dt: 6269.21ms | tok/sec: 83629.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.2ms | fwd: 2005.5ms | loss: 6.4ms | bwd: 4196.5ms | clip: 0.0ms | optim: 49.7ms
step    91 | loss: 1.418258 | lr 2.7600e-04 | norm: 0.6247 | dt: 6297.33ms | tok/sec: 83255.7 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.2ms | fwd: 2010.0ms | loss: 6.4ms | bwd: 4220.2ms | clip: 0.0ms | optim: 49.7ms
step    92 | loss: 1.428064 | lr 2.7900e-04 | norm: 0.7224 | dt: 6277.13ms | tok/sec: 83523.5 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.3ms | fwd: 2006.9ms | loss: 6.4ms | bwd: 4203.2ms | clip: 0.0ms | optim: 49.5ms
step    93 | loss: 1.457621 | lr 2.8200e-04 | norm: 0.7129 | dt: 6277.64ms | tok/sec: 83516.7 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.1ms | fwd: 2006.5ms | loss: 6.4ms | bwd: 4204.1ms | clip: 0.0ms | optim: 49.7ms
step    94 | loss: 1.319050 | lr 2.8500e-04 | norm: 0.5655 | dt: 6289.24ms | tok/sec: 83362.7 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.2ms | fwd: 2010.2ms | loss: 6.4ms | bwd: 4212.2ms | clip: 0.0ms | optim: 49.4ms
step    95 | loss: 1.407644 | lr 2.8800e-04 | norm: 0.6727 | dt: 6316.61ms | tok/sec: 83001.4 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.1ms | fwd: 2012.2ms | loss: 6.4ms | bwd: 4237.3ms | clip: 0.0ms | optim: 49.8ms
step    96 | loss: 1.406497 | lr 2.9100e-04 | norm: 0.7071 | dt: 6289.44ms | tok/sec: 83360.1 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.4ms | fwd: 2017.0ms | loss: 6.4ms | bwd: 4205.3ms | clip: 0.0ms | optim: 49.6ms
step    97 | loss: 1.403561 | lr 2.9400e-04 | norm: 0.6632 | dt: 6267.93ms | tok/sec: 83646.2 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.3ms | fwd: 2003.6ms | loss: 6.4ms | bwd: 4197.1ms | clip: 0.0ms | optim: 49.6ms
step    98 | loss: 1.393360 | lr 2.9700e-04 | norm: 0.7353 | dt: 6296.07ms | tok/sec: 83272.2 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.4ms | fwd: 2014.0ms | loss: 6.4ms | bwd: 4214.7ms | clip: 0.0ms | optim: 49.7ms
step    99 | loss: 1.421664 | lr 3.0000e-04 | norm: 0.6789 | dt: 6272.13ms | tok/sec: 83590.1 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.3ms | fwd: 2012.2ms | loss: 6.4ms | bwd: 4192.6ms | clip: 0.0ms | optim: 49.8ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_100.ckpt (safely)
step   100 | loss: 1.442932 | lr 3.0000e-04 | norm: 0.6712 | dt: 6347.87ms | tok/sec: 82592.7 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.3ms | fwd: 2136.7ms | loss: 6.4ms | bwd: 4143.3ms | clip: 0.0ms | optim: 50.1ms
step   101 | loss: 1.480651 | lr 2.9999e-04 | norm: 0.7536 | dt: 6287.57ms | tok/sec: 83384.8 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.3ms | fwd: 2058.6ms | loss: 6.4ms | bwd: 4162.0ms | clip: 0.0ms | optim: 49.3ms
step   102 | loss: 1.375014 | lr 2.9998e-04 | norm: 0.6878 | dt: 6285.33ms | tok/sec: 83414.6 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.4ms | fwd: 2047.2ms | loss: 6.4ms | bwd: 4170.5ms | clip: 0.0ms | optim: 49.9ms
step   103 | loss: 1.441529 | lr 2.9995e-04 | norm: 0.7461 | dt: 6254.87ms | tok/sec: 83820.8 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2004.6ms | loss: 6.4ms | bwd: 4182.7ms | clip: 0.0ms | optim: 50.0ms
step   104 | loss: 1.510992 | lr 2.9991e-04 | norm: 0.7636 | dt: 6265.73ms | tok/sec: 83675.5 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2008.1ms | loss: 6.4ms | bwd: 4190.4ms | clip: 0.0ms | optim: 49.5ms
step   105 | loss: 1.419842 | lr 2.9986e-04 | norm: 0.7057 | dt: 6281.94ms | tok/sec: 83459.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.2ms | fwd: 2011.7ms | loss: 6.4ms | bwd: 4203.0ms | clip: 0.0ms | optim: 49.7ms
step   106 | loss: 1.435070 | lr 2.9980e-04 | norm: 0.8750 | dt: 6303.39ms | tok/sec: 83175.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2018.7ms | loss: 6.4ms | bwd: 4217.1ms | clip: 0.0ms | optim: 49.9ms
step   107 | loss: 1.502292 | lr 2.9973e-04 | norm: 0.8918 | dt: 6299.85ms | tok/sec: 83222.3 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2010.1ms | loss: 6.4ms | bwd: 4222.5ms | clip: 0.0ms | optim: 49.7ms
step   108 | loss: 1.378002 | lr 2.9965e-04 | norm: 0.7495 | dt: 6299.17ms | tok/sec: 83231.3 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2010.0ms | loss: 6.4ms | bwd: 4222.3ms | clip: 0.0ms | optim: 49.3ms
step   109 | loss: 1.419788 | lr 2.9956e-04 | norm: 0.6733 | dt: 6320.15ms | tok/sec: 82955.0 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.2ms | fwd: 2057.9ms | loss: 6.4ms | bwd: 4195.2ms | clip: 0.0ms | optim: 49.6ms
step   110 | loss: 1.450531 | lr 2.9945e-04 | norm: 0.6484 | dt: 6326.89ms | tok/sec: 82866.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2031.1ms | loss: 6.4ms | bwd: 4228.8ms | clip: 0.0ms | optim: 49.4ms
step   111 | loss: 1.426576 | lr 2.9934e-04 | norm: 0.7399 | dt: 6314.30ms | tok/sec: 83031.8 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2014.5ms | loss: 6.4ms | bwd: 4232.7ms | clip: 0.0ms | optim: 49.5ms
step   112 | loss: 1.403461 | lr 2.9921e-04 | norm: 0.6809 | dt: 6303.80ms | tok/sec: 83170.1 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2020.1ms | loss: 6.4ms | bwd: 4216.6ms | clip: 0.0ms | optim: 49.5ms
step   113 | loss: 1.387195 | lr 2.9907e-04 | norm: 0.6385 | dt: 6324.34ms | tok/sec: 82900.0 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2019.0ms | loss: 6.4ms | bwd: 4238.1ms | clip: 0.0ms | optim: 49.6ms
step   114 | loss: 1.429039 | lr 2.9893e-04 | norm: 0.6179 | dt: 6282.53ms | tok/sec: 83451.7 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.4ms | fwd: 2014.9ms | loss: 6.4ms | bwd: 4200.4ms | clip: 0.0ms | optim: 49.6ms
step   115 | loss: 1.412321 | lr 2.9877e-04 | norm: 0.8271 | dt: 6318.00ms | tok/sec: 82983.3 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.4ms | fwd: 2017.4ms | loss: 6.4ms | bwd: 4233.2ms | clip: 0.0ms | optim: 49.6ms
step   116 | loss: 1.351416 | lr 2.9860e-04 | norm: 0.6914 | dt: 6355.82ms | tok/sec: 82489.4 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.3ms | fwd: 2085.6ms | loss: 6.4ms | bwd: 4202.9ms | clip: 0.0ms | optim: 49.6ms
step   117 | loss: 1.361993 | lr 2.9842e-04 | norm: 0.6219 | dt: 6347.74ms | tok/sec: 82594.5 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.3ms | fwd: 2048.7ms | loss: 6.4ms | bwd: 4231.9ms | clip: 0.0ms | optim: 49.5ms
step   118 | loss: 1.438746 | lr 2.9823e-04 | norm: 0.6071 | dt: 6325.87ms | tok/sec: 82880.0 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.3ms | fwd: 2038.4ms | loss: 6.4ms | bwd: 4219.8ms | clip: 0.0ms | optim: 50.1ms
step   119 | loss: 1.453658 | lr 2.9803e-04 | norm: 0.6363 | dt: 6312.79ms | tok/sec: 83051.8 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.2ms | fwd: 2016.7ms | loss: 6.4ms | bwd: 4229.2ms | clip: 0.0ms | optim: 49.3ms
step   120 | loss: 1.387286 | lr 2.9781e-04 | norm: 0.7181 | dt: 6318.73ms | tok/sec: 82973.6 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.3ms | fwd: 2020.5ms | loss: 6.4ms | bwd: 4231.1ms | clip: 0.0ms | optim: 49.6ms
step   121 | loss: 1.415852 | lr 2.9759e-04 | norm: 0.6190 | dt: 6324.87ms | tok/sec: 82893.1 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.3ms | fwd: 2022.9ms | loss: 6.4ms | bwd: 4233.8ms | clip: 0.0ms | optim: 50.5ms
step   122 | loss: 1.479933 | lr 2.9736e-04 | norm: 0.5859 | dt: 6328.26ms | tok/sec: 82848.7 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.2ms | fwd: 2017.5ms | loss: 6.4ms | bwd: 4243.4ms | clip: 0.0ms | optim: 49.8ms
step   123 | loss: 1.440239 | lr 2.9711e-04 | norm: 0.7633 | dt: 6311.22ms | tok/sec: 83072.4 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.3ms | fwd: 2011.9ms | loss: 6.4ms | bwd: 4232.3ms | clip: 0.0ms | optim: 49.5ms
step   124 | loss: 1.466681 | lr 2.9686e-04 | norm: 0.6771 | dt: 6301.90ms | tok/sec: 83195.2 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.2ms | fwd: 2011.5ms | loss: 6.4ms | bwd: 4223.5ms | clip: 0.0ms | optim: 49.4ms
step   125 | loss: 1.422263 | lr 2.9659e-04 | norm: 0.6660 | dt: 6324.04ms | tok/sec: 82904.0 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.2ms | fwd: 2026.6ms | loss: 6.4ms | bwd: 4230.3ms | clip: 0.0ms | optim: 49.6ms
step   126 | loss: 1.435130 | lr 2.9632e-04 | norm: 0.6036 | dt: 6311.33ms | tok/sec: 83071.0 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.1ms | fwd: 2017.2ms | loss: 6.4ms | bwd: 4227.5ms | clip: 0.0ms | optim: 49.3ms
step   127 | loss: 1.478923 | lr 2.9603e-04 | norm: 0.6003 | dt: 6354.75ms | tok/sec: 82503.4 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.1ms | fwd: 2072.2ms | loss: 6.4ms | bwd: 4215.5ms | clip: 0.0ms | optim: 49.7ms
step   128 | loss: 1.507035 | lr 2.9573e-04 | norm: 0.5689 | dt: 6348.34ms | tok/sec: 82586.7 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.2ms | fwd: 2056.7ms | loss: 6.4ms | bwd: 4224.7ms | clip: 0.0ms | optim: 49.4ms
step   129 | loss: 1.452367 | lr 2.9542e-04 | norm: 0.8532 | dt: 6329.68ms | tok/sec: 82830.1 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.2ms | fwd: 2044.4ms | loss: 6.4ms | bwd: 4218.0ms | clip: 0.0ms | optim: 49.9ms
step   130 | loss: 1.569316 | lr 2.9511e-04 | norm: 0.7261 | dt: 6296.60ms | tok/sec: 83265.3 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.1ms | fwd: 2016.6ms | loss: 6.4ms | bwd: 4213.0ms | clip: 0.0ms | optim: 49.6ms
step   131 | loss: 1.450306 | lr 2.9478e-04 | norm: 0.7580 | dt: 6320.00ms | tok/sec: 82957.0 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.2ms | fwd: 2016.9ms | loss: 6.5ms | bwd: 4235.2ms | clip: 0.0ms | optim: 50.2ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_a/in2a_train_00001.bin
100000000
step   132 | loss: 1.454688 | lr 2.9444e-04 | norm: 0.6053 | dt: 6312.21ms | tok/sec: 83059.4 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 6.2ms | fwd: 2021.6ms | loss: 6.4ms | bwd: 4220.5ms | clip: 0.0ms | optim: 49.5ms
step   133 | loss: 1.431383 | lr 2.9409e-04 | norm: 0.5934 | dt: 6329.63ms | tok/sec: 82830.7 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.2ms | fwd: 2023.8ms | loss: 6.4ms | bwd: 4238.5ms | clip: 0.0ms | optim: 49.8ms
step   134 | loss: 1.439821 | lr 2.9373e-04 | norm: 0.5904 | dt: 6300.04ms | tok/sec: 83219.8 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.2ms | fwd: 2011.7ms | loss: 6.4ms | bwd: 4221.3ms | clip: 0.0ms | optim: 49.6ms
step   135 | loss: 1.477309 | lr 2.9336e-04 | norm: 0.5975 | dt: 6370.56ms | tok/sec: 82298.6 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.2ms | fwd: 2075.3ms | loss: 6.4ms | bwd: 4228.2ms | clip: 0.0ms | optim: 49.6ms
step   136 | loss: 1.419291 | lr 2.9298e-04 | norm: 0.6597 | dt: 6316.18ms | tok/sec: 83007.2 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.3ms | fwd: 2016.1ms | loss: 6.4ms | bwd: 4233.0ms | clip: 0.0ms | optim: 49.4ms
step   137 | loss: 1.411285 | lr 2.9259e-04 | norm: 0.6480 | dt: 6292.22ms | tok/sec: 83323.2 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.2ms | fwd: 2013.1ms | loss: 6.4ms | bwd: 4212.2ms | clip: 0.0ms | optim: 49.3ms
step   138 | loss: 1.478121 | lr 2.9219e-04 | norm: 0.5304 | dt: 6332.40ms | tok/sec: 82794.6 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.3ms | fwd: 2026.0ms | loss: 6.4ms | bwd: 4239.6ms | clip: 0.0ms | optim: 49.2ms
step   139 | loss: 1.480008 | lr 2.9178e-04 | norm: 0.5479 | dt: 6302.46ms | tok/sec: 83187.9 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.3ms | fwd: 2008.8ms | loss: 6.4ms | bwd: 4226.4ms | clip: 0.0ms | optim: 49.7ms
step   140 | loss: 1.493623 | lr 2.9135e-04 | norm: 0.5506 | dt: 6304.77ms | tok/sec: 83157.4 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.3ms | fwd: 2020.9ms | loss: 6.4ms | bwd: 4216.8ms | clip: 0.0ms | optim: 49.4ms
step   141 | loss: 1.482856 | lr 2.9092e-04 | norm: 0.6082 | dt: 6341.75ms | tok/sec: 82672.4 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.1ms | fwd: 2045.8ms | loss: 6.4ms | bwd: 4229.2ms | clip: 0.0ms | optim: 49.3ms
step   142 | loss: 1.466891 | lr 2.9048e-04 | norm: 0.5793 | dt: 6307.55ms | tok/sec: 83120.7 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.2ms | fwd: 2018.3ms | loss: 6.4ms | bwd: 4222.4ms | clip: 0.0ms | optim: 49.3ms
step   143 | loss: 1.405515 | lr 2.9003e-04 | norm: 0.5934 | dt: 6327.35ms | tok/sec: 82860.6 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.3ms | fwd: 2022.6ms | loss: 6.4ms | bwd: 4237.4ms | clip: 0.0ms | optim: 49.7ms
step   144 | loss: 1.492733 | lr 2.8957e-04 | norm: 0.6180 | dt: 6321.73ms | tok/sec: 82934.3 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.3ms | fwd: 2019.4ms | loss: 6.4ms | bwd: 4235.3ms | clip: 0.0ms | optim: 49.4ms
step   145 | loss: 1.393695 | lr 2.8910e-04 | norm: 0.6128 | dt: 6305.77ms | tok/sec: 83144.2 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.3ms | fwd: 2014.1ms | loss: 6.4ms | bwd: 4224.5ms | clip: 0.0ms | optim: 49.4ms
step   146 | loss: 1.406142 | lr 2.8862e-04 | norm: 0.4758 | dt: 6302.26ms | tok/sec: 83190.4 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.4ms | fwd: 2014.2ms | loss: 6.4ms | bwd: 4220.8ms | clip: 0.0ms | optim: 49.5ms
step   147 | loss: 1.404284 | lr 2.8813e-04 | norm: 0.4947 | dt: 6333.33ms | tok/sec: 82782.4 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.3ms | fwd: 2017.5ms | loss: 6.5ms | bwd: 4248.6ms | clip: 0.0ms | optim: 49.5ms
step   148 | loss: 1.456305 | lr 2.8763e-04 | norm: 0.5212 | dt: 6307.48ms | tok/sec: 83121.6 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.2ms | fwd: 2013.4ms | loss: 6.4ms | bwd: 4226.9ms | clip: 0.0ms | optim: 49.5ms
step   149 | loss: 1.452610 | lr 2.8712e-04 | norm: 0.4964 | dt: 6310.16ms | tok/sec: 83086.3 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.4ms | fwd: 2011.7ms | loss: 6.4ms | bwd: 4231.2ms | clip: 0.0ms | optim: 49.5ms
step   150 | loss: 1.402426 | lr 2.8660e-04 | norm: 0.5503 | dt: 6293.22ms | tok/sec: 83310.0 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.4ms | fwd: 2014.7ms | loss: 6.4ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.5ms
step   151 | loss: 1.426098 | lr 2.8607e-04 | norm: 0.5175 | dt: 6304.62ms | tok/sec: 83159.3 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.3ms | fwd: 2012.6ms | loss: 6.4ms | bwd: 4224.9ms | clip: 0.0ms | optim: 49.5ms
step   152 | loss: 1.476027 | lr 2.8554e-04 | norm: 0.6003 | dt: 6305.91ms | tok/sec: 83142.3 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.3ms | fwd: 2013.4ms | loss: 6.4ms | bwd: 4225.5ms | clip: 0.0ms | optim: 49.4ms
step   153 | loss: 1.428901 | lr 2.8499e-04 | norm: 0.5044 | dt: 6333.25ms | tok/sec: 82783.5 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.2ms | fwd: 2022.7ms | loss: 6.4ms | bwd: 4242.9ms | clip: 0.0ms | optim: 49.7ms
step   154 | loss: 1.461169 | lr 2.8443e-04 | norm: 0.5722 | dt: 6284.95ms | tok/sec: 83419.6 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.2ms | fwd: 2011.7ms | loss: 6.4ms | bwd: 4206.2ms | clip: 0.0ms | optim: 49.6ms
step   155 | loss: 1.442151 | lr 2.8387e-04 | norm: 0.5616 | dt: 6307.08ms | tok/sec: 83126.8 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.3ms | fwd: 2016.7ms | loss: 6.4ms | bwd: 4223.2ms | clip: 0.0ms | optim: 49.5ms
step   156 | loss: 1.428072 | lr 2.8329e-04 | norm: 0.4856 | dt: 6349.77ms | tok/sec: 82568.1 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.2ms | fwd: 2049.7ms | loss: 6.4ms | bwd: 4233.1ms | clip: 0.0ms | optim: 49.4ms
step   157 | loss: 1.488164 | lr 2.8271e-04 | norm: 0.4940 | dt: 6319.10ms | tok/sec: 82968.8 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.2ms | fwd: 2033.8ms | loss: 6.4ms | bwd: 4218.6ms | clip: 0.0ms | optim: 49.3ms
step   158 | loss: 1.497399 | lr 2.8211e-04 | norm: 0.4874 | dt: 6351.24ms | tok/sec: 82548.9 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.2ms | fwd: 2063.4ms | loss: 6.4ms | bwd: 4220.4ms | clip: 0.0ms | optim: 49.9ms
step   159 | loss: 1.444991 | lr 2.8151e-04 | norm: 0.4872 | dt: 6305.08ms | tok/sec: 83153.3 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.2ms | fwd: 2012.4ms | loss: 6.4ms | bwd: 4225.4ms | clip: 0.0ms | optim: 49.8ms
step   160 | loss: 1.488986 | lr 2.8090e-04 | norm: 0.5209 | dt: 6317.53ms | tok/sec: 82989.4 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.3ms | fwd: 2030.5ms | loss: 6.4ms | bwd: 4219.3ms | clip: 0.0ms | optim: 50.2ms
step   161 | loss: 1.451132 | lr 2.8028e-04 | norm: 0.4768 | dt: 6315.29ms | tok/sec: 83018.9 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.2ms | fwd: 2013.6ms | loss: 6.4ms | bwd: 4234.4ms | clip: 0.0ms | optim: 49.7ms
step   162 | loss: 1.445982 | lr 2.7965e-04 | norm: 0.5467 | dt: 6320.67ms | tok/sec: 82948.1 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.2ms | fwd: 2010.1ms | loss: 6.6ms | bwd: 4242.9ms | clip: 0.0ms | optim: 50.0ms
step   163 | loss: 1.454782 | lr 2.7902e-04 | norm: 0.4539 | dt: 6302.45ms | tok/sec: 83188.0 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.2ms | fwd: 2015.9ms | loss: 6.4ms | bwd: 4219.4ms | clip: 0.0ms | optim: 49.6ms
step   164 | loss: 1.377520 | lr 2.7837e-04 | norm: 0.6424 | dt: 6300.22ms | tok/sec: 83217.4 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.2ms | fwd: 2012.6ms | loss: 6.4ms | bwd: 4220.6ms | clip: 0.0ms | optim: 49.6ms
step   165 | loss: 1.443390 | lr 2.7771e-04 | norm: 0.5046 | dt: 6339.56ms | tok/sec: 82701.0 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.3ms | fwd: 2038.0ms | loss: 6.4ms | bwd: 4234.2ms | clip: 0.0ms | optim: 49.6ms
step   166 | loss: 1.443651 | lr 2.7705e-04 | norm: 0.4548 | dt: 6303.10ms | tok/sec: 83179.4 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.2ms | fwd: 2021.0ms | loss: 6.4ms | bwd: 4215.2ms | clip: 0.0ms | optim: 49.3ms
step   167 | loss: 1.459447 | lr 2.7638e-04 | norm: 0.4931 | dt: 6321.60ms | tok/sec: 82935.9 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.4ms | fwd: 2039.9ms | loss: 6.4ms | bwd: 4214.1ms | clip: 0.0ms | optim: 49.9ms
step   168 | loss: 1.499442 | lr 2.7570e-04 | norm: 0.4749 | dt: 6321.23ms | tok/sec: 82940.9 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.2ms | fwd: 2017.6ms | loss: 6.4ms | bwd: 4236.5ms | clip: 0.0ms | optim: 49.6ms
step   169 | loss: 1.408868 | lr 2.7501e-04 | norm: 0.5360 | dt: 6332.74ms | tok/sec: 82790.0 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.1ms | fwd: 2042.8ms | loss: 6.4ms | bwd: 4222.7ms | clip: 0.0ms | optim: 49.8ms
step   170 | loss: 1.415292 | lr 2.7431e-04 | norm: 0.4623 | dt: 6357.74ms | tok/sec: 82464.5 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.2ms | fwd: 2075.4ms | loss: 6.4ms | bwd: 4214.9ms | clip: 0.0ms | optim: 49.7ms
step   171 | loss: 1.486619 | lr 2.7361e-04 | norm: 0.4210 | dt: 6342.42ms | tok/sec: 82663.8 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.2ms | fwd: 2061.9ms | loss: 6.4ms | bwd: 4213.7ms | clip: 0.0ms | optim: 49.3ms
step   172 | loss: 1.444242 | lr 2.7290e-04 | norm: 0.4695 | dt: 6311.66ms | tok/sec: 83066.6 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.3ms | fwd: 2045.1ms | loss: 6.4ms | bwd: 4199.4ms | clip: 0.0ms | optim: 49.4ms
step   173 | loss: 1.427189 | lr 2.7218e-04 | norm: 0.4260 | dt: 6352.71ms | tok/sec: 82529.8 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.3ms | fwd: 2071.8ms | loss: 6.4ms | bwd: 4213.3ms | clip: 0.0ms | optim: 49.8ms
step   174 | loss: 1.450822 | lr 2.7145e-04 | norm: 0.4081 | dt: 6326.21ms | tok/sec: 82875.6 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.2ms | fwd: 2027.3ms | loss: 6.4ms | bwd: 4232.1ms | clip: 0.0ms | optim: 49.2ms
step   175 | loss: 1.438023 | lr 2.7071e-04 | norm: 0.5651 | dt: 6350.00ms | tok/sec: 82565.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.3ms | fwd: 2054.1ms | loss: 6.4ms | bwd: 4228.6ms | clip: 0.0ms | optim: 49.6ms
step   176 | loss: 1.403051 | lr 2.6997e-04 | norm: 0.5746 | dt: 6317.02ms | tok/sec: 82996.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.2ms | fwd: 2012.8ms | loss: 6.4ms | bwd: 4237.2ms | clip: 0.0ms | optim: 49.5ms
step   177 | loss: 1.494423 | lr 2.6921e-04 | norm: 0.4481 | dt: 6300.30ms | tok/sec: 83216.3 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.3ms | fwd: 2014.6ms | loss: 6.4ms | bwd: 4218.5ms | clip: 0.0ms | optim: 49.6ms
step   178 | loss: 1.461557 | lr 2.6845e-04 | norm: 0.4456 | dt: 6308.25ms | tok/sec: 83111.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.2ms | fwd: 2009.7ms | loss: 6.4ms | bwd: 4231.2ms | clip: 0.0ms | optim: 49.8ms
step   179 | loss: 1.462278 | lr 2.6769e-04 | norm: 0.5709 | dt: 6316.37ms | tok/sec: 83004.6 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.3ms | fwd: 2016.2ms | loss: 6.4ms | bwd: 4233.1ms | clip: 0.0ms | optim: 49.4ms
step   180 | loss: 1.506178 | lr 2.6691e-04 | norm: 0.4055 | dt: 6298.74ms | tok/sec: 83237.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.2ms | fwd: 2012.0ms | loss: 6.4ms | bwd: 4220.0ms | clip: 0.0ms | optim: 49.3ms
step   181 | loss: 1.408684 | lr 2.6613e-04 | norm: 0.3928 | dt: 6313.12ms | tok/sec: 83047.3 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.3ms | fwd: 2016.4ms | loss: 6.4ms | bwd: 4229.7ms | clip: 0.0ms | optim: 49.5ms
step   182 | loss: 1.409247 | lr 2.6534e-04 | norm: 0.5111 | dt: 6349.64ms | tok/sec: 82569.7 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.2ms | fwd: 2067.1ms | loss: 6.4ms | bwd: 4215.7ms | clip: 0.0ms | optim: 49.4ms
step   183 | loss: 1.442007 | lr 2.6455e-04 | norm: 0.4133 | dt: 6357.69ms | tok/sec: 82465.2 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.3ms | fwd: 2078.2ms | loss: 6.4ms | bwd: 4212.3ms | clip: 0.0ms | optim: 49.4ms
step   184 | loss: 1.444275 | lr 2.6374e-04 | norm: 0.4994 | dt: 6335.67ms | tok/sec: 82751.8 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2030.9ms | loss: 6.4ms | bwd: 4237.5ms | clip: 0.0ms | optim: 49.5ms
step   185 | loss: 1.408447 | lr 2.6293e-04 | norm: 0.4702 | dt: 6362.89ms | tok/sec: 82397.7 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.2ms | fwd: 2067.5ms | loss: 6.4ms | bwd: 4227.8ms | clip: 0.0ms | optim: 50.1ms
step   186 | loss: 1.479656 | lr 2.6211e-04 | norm: 0.4238 | dt: 6302.05ms | tok/sec: 83193.3 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.3ms | fwd: 2018.4ms | loss: 6.4ms | bwd: 4215.7ms | clip: 0.0ms | optim: 50.2ms
step   187 | loss: 1.460944 | lr 2.6129e-04 | norm: 0.4312 | dt: 6305.99ms | tok/sec: 83141.3 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.3ms | fwd: 2014.6ms | loss: 6.4ms | bwd: 4224.0ms | clip: 0.0ms | optim: 49.7ms
step   188 | loss: 1.431488 | lr 2.6046e-04 | norm: 0.4536 | dt: 6294.17ms | tok/sec: 83297.3 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.2ms | fwd: 2018.7ms | loss: 6.4ms | bwd: 4208.4ms | clip: 0.0ms | optim: 49.7ms
step   189 | loss: 1.444559 | lr 2.5962e-04 | norm: 0.4123 | dt: 6299.07ms | tok/sec: 83232.6 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.2ms | fwd: 2015.5ms | loss: 6.4ms | bwd: 4216.4ms | clip: 0.0ms | optim: 49.7ms
step   190 | loss: 1.415559 | lr 2.5878e-04 | norm: 0.4003 | dt: 6317.00ms | tok/sec: 82996.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.3ms | fwd: 2018.9ms | loss: 6.4ms | bwd: 4230.6ms | clip: 0.0ms | optim: 49.9ms
step   191 | loss: 1.441320 | lr 2.5793e-04 | norm: 0.3870 | dt: 6297.48ms | tok/sec: 83253.6 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.2ms | fwd: 2018.3ms | loss: 6.4ms | bwd: 4212.3ms | clip: 0.0ms | optim: 49.4ms
step   192 | loss: 1.424655 | lr 2.5707e-04 | norm: 0.4936 | dt: 6303.41ms | tok/sec: 83175.3 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.1ms | fwd: 2012.8ms | loss: 6.4ms | bwd: 4223.9ms | clip: 0.0ms | optim: 49.3ms
step   193 | loss: 1.420979 | lr 2.5621e-04 | norm: 0.3760 | dt: 6322.46ms | tok/sec: 82924.6 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.1ms | fwd: 2021.6ms | loss: 6.4ms | bwd: 4234.1ms | clip: 0.0ms | optim: 49.4ms
step   194 | loss: 1.427058 | lr 2.5534e-04 | norm: 0.4177 | dt: 6296.95ms | tok/sec: 83260.6 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.3ms | fwd: 2010.0ms | loss: 6.4ms | bwd: 4219.8ms | clip: 0.0ms | optim: 49.5ms
step   195 | loss: 1.410106 | lr 2.5446e-04 | norm: 0.3665 | dt: 6322.18ms | tok/sec: 82928.3 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.1ms | fwd: 2020.2ms | loss: 6.4ms | bwd: 4235.1ms | clip: 0.0ms | optim: 49.4ms
step   196 | loss: 1.435013 | lr 2.5358e-04 | norm: 0.6285 | dt: 6344.44ms | tok/sec: 82637.4 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.3ms | fwd: 2077.8ms | loss: 6.4ms | bwd: 4199.0ms | clip: 0.0ms | optim: 49.9ms
step   197 | loss: 1.451669 | lr 2.5270e-04 | norm: 0.4527 | dt: 6354.95ms | tok/sec: 82500.7 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.3ms | fwd: 2063.8ms | loss: 6.4ms | bwd: 4224.0ms | clip: 0.0ms | optim: 49.4ms
step   198 | loss: 1.398883 | lr 2.5180e-04 | norm: 0.4079 | dt: 6320.27ms | tok/sec: 82953.4 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.2ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4239.0ms | clip: 0.0ms | optim: 49.8ms
step   199 | loss: 1.404828 | lr 2.5090e-04 | norm: 0.3845 | dt: 6330.38ms | tok/sec: 82821.0 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.3ms | fwd: 2018.9ms | loss: 6.4ms | bwd: 4244.1ms | clip: 0.0ms | optim: 49.4ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_200.ckpt (safely)
step   200 | loss: 1.403817 | lr 2.5000e-04 | norm: 0.3828 | dt: 6724.16ms | tok/sec: 77970.8 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.0ms | fwd: 2514.9ms | loss: 6.4ms | bwd: 4141.8ms | clip: 0.0ms | optim: 49.8ms
step   201 | loss: 1.438127 | lr 2.4909e-04 | norm: 0.3192 | dt: 6242.05ms | tok/sec: 83992.9 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2001.7ms | loss: 6.4ms | bwd: 4173.1ms | clip: 0.0ms | optim: 49.5ms
step   202 | loss: 1.406299 | lr 2.4818e-04 | norm: 0.3947 | dt: 6249.00ms | tok/sec: 83899.5 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.2ms | fwd: 2001.9ms | loss: 6.3ms | bwd: 4179.7ms | clip: 0.0ms | optim: 49.9ms
step   203 | loss: 1.497176 | lr 2.4726e-04 | norm: 0.4321 | dt: 6287.29ms | tok/sec: 83388.5 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.2ms | fwd: 2010.6ms | loss: 6.4ms | bwd: 4210.0ms | clip: 0.0ms | optim: 49.3ms
step   204 | loss: 1.398200 | lr 2.4633e-04 | norm: 0.3365 | dt: 6285.03ms | tok/sec: 83418.6 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2014.4ms | loss: 6.5ms | bwd: 4203.2ms | clip: 0.0ms | optim: 49.4ms
step   205 | loss: 1.364092 | lr 2.4540e-04 | norm: 0.3681 | dt: 6301.30ms | tok/sec: 83203.1 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.3ms | fwd: 2019.0ms | loss: 6.5ms | bwd: 4214.8ms | clip: 0.0ms | optim: 49.5ms
step   206 | loss: 1.459170 | lr 2.4446e-04 | norm: 0.3920 | dt: 6362.35ms | tok/sec: 82404.7 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2075.1ms | loss: 6.5ms | bwd: 4219.6ms | clip: 0.0ms | optim: 49.5ms
step   207 | loss: 1.484398 | lr 2.4352e-04 | norm: 0.3502 | dt: 6342.23ms | tok/sec: 82666.2 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2059.9ms | loss: 6.5ms | bwd: 4214.5ms | clip: 0.0ms | optim: 49.8ms
step   208 | loss: 1.348089 | lr 2.4258e-04 | norm: 0.3915 | dt: 6355.72ms | tok/sec: 82490.7 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2069.2ms | loss: 6.5ms | bwd: 4218.9ms | clip: 0.0ms | optim: 49.5ms
step   209 | loss: 1.449291 | lr 2.4163e-04 | norm: 0.3918 | dt: 6319.40ms | tok/sec: 82964.9 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2021.7ms | loss: 6.5ms | bwd: 4229.9ms | clip: 0.0ms | optim: 49.8ms
step   210 | loss: 1.460217 | lr 2.4067e-04 | norm: 0.5043 | dt: 6311.21ms | tok/sec: 83072.5 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.3ms | fwd: 2015.5ms | loss: 6.5ms | bwd: 4228.5ms | clip: 0.0ms | optim: 49.4ms
step   211 | loss: 1.480269 | lr 2.3971e-04 | norm: 0.4325 | dt: 6329.08ms | tok/sec: 82837.9 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.4ms | fwd: 2019.7ms | loss: 6.5ms | bwd: 4241.8ms | clip: 0.0ms | optim: 49.6ms
step   212 | loss: 1.416566 | lr 2.3875e-04 | norm: 0.3779 | dt: 6321.43ms | tok/sec: 82938.2 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2022.4ms | loss: 6.5ms | bwd: 4231.3ms | clip: 0.0ms | optim: 49.8ms
step   213 | loss: 1.433720 | lr 2.3778e-04 | norm: 0.6583 | dt: 6307.52ms | tok/sec: 83121.1 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.3ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4226.5ms | clip: 0.0ms | optim: 49.5ms
step   214 | loss: 1.418691 | lr 2.3681e-04 | norm: 0.3317 | dt: 6316.35ms | tok/sec: 83004.9 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2020.9ms | loss: 6.5ms | bwd: 4227.7ms | clip: 0.0ms | optim: 49.5ms
step   215 | loss: 1.428402 | lr 2.3584e-04 | norm: 0.3616 | dt: 6316.67ms | tok/sec: 83000.8 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2023.9ms | loss: 6.5ms | bwd: 4224.8ms | clip: 0.0ms | optim: 49.8ms
step   216 | loss: 1.396070 | lr 2.3486e-04 | norm: 0.3461 | dt: 6352.21ms | tok/sec: 82536.3 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.3ms | fwd: 2065.0ms | loss: 6.5ms | bwd: 4219.9ms | clip: 0.0ms | optim: 49.3ms
step   217 | loss: 1.417023 | lr 2.3387e-04 | norm: 0.3914 | dt: 6346.34ms | tok/sec: 82612.7 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.4ms | fwd: 2049.4ms | loss: 6.5ms | bwd: 4229.4ms | clip: 0.0ms | optim: 49.6ms
step   218 | loss: 1.450046 | lr 2.3289e-04 | norm: 0.2994 | dt: 6354.80ms | tok/sec: 82502.7 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.3ms | fwd: 2080.7ms | loss: 6.5ms | bwd: 4206.5ms | clip: 0.0ms | optim: 49.5ms
step   219 | loss: 1.475388 | lr 2.3190e-04 | norm: 0.3489 | dt: 6335.40ms | tok/sec: 82755.4 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.4ms | fwd: 2026.1ms | loss: 6.5ms | bwd: 4241.9ms | clip: 0.0ms | optim: 49.4ms
step   220 | loss: 1.406461 | lr 2.3090e-04 | norm: 0.4261 | dt: 6305.40ms | tok/sec: 83149.0 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.4ms | fwd: 2017.7ms | loss: 6.5ms | bwd: 4220.2ms | clip: 0.0ms | optim: 49.6ms
step   221 | loss: 1.394355 | lr 2.2990e-04 | norm: 0.4302 | dt: 6305.01ms | tok/sec: 83154.2 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.4ms | fwd: 2018.8ms | loss: 6.5ms | bwd: 4218.5ms | clip: 0.0ms | optim: 49.7ms
step   222 | loss: 1.381861 | lr 2.2890e-04 | norm: 0.3070 | dt: 6324.84ms | tok/sec: 82893.5 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2018.5ms | loss: 6.5ms | bwd: 4238.2ms | clip: 0.0ms | optim: 50.1ms
step   223 | loss: 1.416641 | lr 2.2790e-04 | norm: 0.4308 | dt: 6309.97ms | tok/sec: 83088.8 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2014.6ms | loss: 6.5ms | bwd: 4227.4ms | clip: 0.0ms | optim: 49.7ms
step   224 | loss: 1.357221 | lr 2.2689e-04 | norm: 0.4500 | dt: 6306.86ms | tok/sec: 83129.8 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.4ms | fwd: 2012.9ms | loss: 6.5ms | bwd: 4226.1ms | clip: 0.0ms | optim: 50.0ms
step   225 | loss: 1.454751 | lr 2.2588e-04 | norm: 0.3562 | dt: 6348.48ms | tok/sec: 82584.8 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2066.7ms | loss: 6.5ms | bwd: 4214.2ms | clip: 0.0ms | optim: 49.5ms
step   226 | loss: 1.376981 | lr 2.2487e-04 | norm: 0.3224 | dt: 6345.98ms | tok/sec: 82617.3 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.6ms | fwd: 2055.0ms | loss: 6.5ms | bwd: 4223.2ms | clip: 0.0ms | optim: 49.5ms
step   227 | loss: 1.365093 | lr 2.2385e-04 | norm: 0.3519 | dt: 6324.66ms | tok/sec: 82895.9 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.7ms | fwd: 2031.6ms | loss: 6.5ms | bwd: 4225.4ms | clip: 0.0ms | optim: 49.4ms
step   228 | loss: 1.433433 | lr 2.2284e-04 | norm: 0.3206 | dt: 6306.06ms | tok/sec: 83140.4 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.7ms | fwd: 2023.1ms | loss: 6.5ms | bwd: 4215.2ms | clip: 0.0ms | optim: 49.3ms
step   229 | loss: 1.400847 | lr 2.2181e-04 | norm: 0.3215 | dt: 6314.38ms | tok/sec: 83030.8 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.6ms | fwd: 2030.2ms | loss: 6.5ms | bwd: 4215.9ms | clip: 0.0ms | optim: 50.0ms
step   230 | loss: 1.393643 | lr 2.2079e-04 | norm: 0.3351 | dt: 6307.75ms | tok/sec: 83118.0 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2019.5ms | loss: 6.5ms | bwd: 4220.9ms | clip: 0.0ms | optim: 49.4ms
step   231 | loss: 1.379871 | lr 2.1977e-04 | norm: 0.3441 | dt: 6332.35ms | tok/sec: 82795.2 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2033.7ms | loss: 6.5ms | bwd: 4230.9ms | clip: 0.0ms | optim: 49.8ms
step   232 | loss: 1.477580 | lr 2.1874e-04 | norm: 0.3887 | dt: 6331.77ms | tok/sec: 82802.7 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2017.5ms | loss: 6.5ms | bwd: 4246.7ms | clip: 0.0ms | optim: 49.6ms
step   233 | loss: 1.401012 | lr 2.1771e-04 | norm: 0.3217 | dt: 6325.29ms | tok/sec: 82887.6 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.7ms | bwd: 4244.0ms | clip: 0.0ms | optim: 49.6ms
step   234 | loss: 1.425210 | lr 2.1668e-04 | norm: 0.4616 | dt: 6327.96ms | tok/sec: 82852.6 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2015.3ms | loss: 6.6ms | bwd: 4245.1ms | clip: 0.0ms | optim: 49.5ms
step   235 | loss: 1.462096 | lr 2.1564e-04 | norm: 0.3184 | dt: 6304.93ms | tok/sec: 83155.3 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2013.8ms | loss: 6.5ms | bwd: 4223.1ms | clip: 0.0ms | optim: 50.1ms
step   236 | loss: 1.431871 | lr 2.1461e-04 | norm: 0.3267 | dt: 6313.00ms | tok/sec: 83049.0 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.6ms | fwd: 2010.4ms | loss: 6.5ms | bwd: 4234.9ms | clip: 0.0ms | optim: 49.4ms
step   237 | loss: 1.403202 | lr 2.1357e-04 | norm: 0.3199 | dt: 6306.60ms | tok/sec: 83133.3 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2015.4ms | loss: 6.5ms | bwd: 4223.6ms | clip: 0.0ms | optim: 49.5ms
step   238 | loss: 1.356215 | lr 2.1253e-04 | norm: 0.2856 | dt: 6316.46ms | tok/sec: 83003.5 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2017.4ms | loss: 6.5ms | bwd: 4231.4ms | clip: 0.0ms | optim: 49.5ms
step   239 | loss: 1.371449 | lr 2.1149e-04 | norm: 0.2823 | dt: 6302.89ms | tok/sec: 83182.2 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2015.2ms | loss: 6.5ms | bwd: 4220.1ms | clip: 0.0ms | optim: 49.6ms
step   240 | loss: 1.400371 | lr 2.1045e-04 | norm: 0.3787 | dt: 6313.62ms | tok/sec: 83040.8 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2016.4ms | loss: 6.5ms | bwd: 4229.8ms | clip: 0.0ms | optim: 49.4ms
step   241 | loss: 1.377639 | lr 2.0941e-04 | norm: 0.3114 | dt: 6324.98ms | tok/sec: 82891.7 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2015.3ms | loss: 6.5ms | bwd: 4242.1ms | clip: 0.0ms | optim: 49.6ms
step   242 | loss: 1.425816 | lr 2.0837e-04 | norm: 0.3465 | dt: 6319.45ms | tok/sec: 82964.1 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2022.6ms | loss: 6.5ms | bwd: 4228.9ms | clip: 0.0ms | optim: 50.0ms
step   243 | loss: 1.382616 | lr 2.0732e-04 | norm: 0.3088 | dt: 6315.89ms | tok/sec: 83010.9 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2030.3ms | loss: 6.5ms | bwd: 4218.2ms | clip: 0.0ms | optim: 49.5ms
step   244 | loss: 1.416425 | lr 2.0628e-04 | norm: 0.3194 | dt: 6317.64ms | tok/sec: 82987.9 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2013.9ms | loss: 6.5ms | bwd: 4236.2ms | clip: 0.0ms | optim: 49.5ms
step   245 | loss: 1.412601 | lr 2.0523e-04 | norm: 0.2997 | dt: 6314.21ms | tok/sec: 83033.0 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.6ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4234.5ms | clip: 0.0ms | optim: 49.5ms
step   246 | loss: 1.393735 | lr 2.0419e-04 | norm: 0.2756 | dt: 6340.52ms | tok/sec: 82688.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2029.0ms | loss: 6.6ms | bwd: 4244.0ms | clip: 0.0ms | optim: 49.3ms
step   247 | loss: 1.422358 | lr 2.0314e-04 | norm: 0.2888 | dt: 6321.20ms | tok/sec: 82941.2 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2022.1ms | loss: 6.5ms | bwd: 4231.4ms | clip: 0.0ms | optim: 49.6ms
step   248 | loss: 1.395848 | lr 2.0209e-04 | norm: 0.3428 | dt: 6310.27ms | tok/sec: 83084.9 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2019.8ms | loss: 6.5ms | bwd: 4222.9ms | clip: 0.0ms | optim: 49.6ms
step   249 | loss: 1.390024 | lr 2.0105e-04 | norm: 0.3033 | dt: 6342.11ms | tok/sec: 82667.8 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2042.3ms | loss: 6.5ms | bwd: 4232.1ms | clip: 0.0ms | optim: 49.6ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_a/in2a_val_00000.bin
100000000
validation loss: 1.4018
step   250 | loss: 1.370249 | lr 2.0000e-04 | norm: 0.2901 | dt: 11439.26ms | tok/sec: 45832.3 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.3ms | fwd: 2021.0ms | loss: 6.5ms | bwd: 4232.4ms | clip: 0.0ms | optim: 49.4ms
step   251 | loss: 1.419143 | lr 1.9895e-04 | norm: 0.3483 | dt: 6309.68ms | tok/sec: 83092.7 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2015.3ms | loss: 6.5ms | bwd: 4226.4ms | clip: 0.0ms | optim: 49.9ms
step   252 | loss: 1.425415 | lr 1.9791e-04 | norm: 0.2409 | dt: 6340.98ms | tok/sec: 82682.4 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2065.7ms | loss: 6.5ms | bwd: 4207.7ms | clip: 0.0ms | optim: 49.7ms
step   253 | loss: 1.396258 | lr 1.9686e-04 | norm: 0.3855 | dt: 6333.11ms | tok/sec: 82785.2 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2017.2ms | loss: 6.5ms | bwd: 4247.9ms | clip: 0.0ms | optim: 50.0ms
step   254 | loss: 1.413303 | lr 1.9581e-04 | norm: 0.2751 | dt: 6317.28ms | tok/sec: 82992.7 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2010.4ms | loss: 6.7ms | bwd: 4238.8ms | clip: 0.0ms | optim: 50.0ms
step   255 | loss: 1.425304 | lr 1.9477e-04 | norm: 0.2499 | dt: 6325.38ms | tok/sec: 82886.4 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2018.6ms | loss: 6.5ms | bwd: 4239.5ms | clip: 0.0ms | optim: 49.4ms
step   256 | loss: 1.367393 | lr 1.9372e-04 | norm: 0.2482 | dt: 6322.18ms | tok/sec: 82928.4 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2021.3ms | loss: 6.5ms | bwd: 4233.4ms | clip: 0.0ms | optim: 49.5ms
step   257 | loss: 1.370168 | lr 1.9268e-04 | norm: 0.3622 | dt: 6324.87ms | tok/sec: 82893.1 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.7ms | fwd: 2024.8ms | loss: 6.5ms | bwd: 4232.1ms | clip: 0.0ms | optim: 49.6ms
step   258 | loss: 1.373518 | lr 1.9163e-04 | norm: 0.2472 | dt: 6309.00ms | tok/sec: 83101.6 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2023.0ms | loss: 6.5ms | bwd: 4218.2ms | clip: 0.0ms | optim: 49.7ms
step   259 | loss: 1.361786 | lr 1.9059e-04 | norm: 0.2318 | dt: 6315.23ms | tok/sec: 83019.6 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2016.1ms | loss: 6.5ms | bwd: 4231.1ms | clip: 0.0ms | optim: 50.0ms
step   260 | loss: 1.375107 | lr 1.8955e-04 | norm: 0.2272 | dt: 6370.12ms | tok/sec: 82304.2 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.4ms | fwd: 2077.8ms | loss: 6.5ms | bwd: 4224.9ms | clip: 0.0ms | optim: 49.5ms
step   261 | loss: 1.364820 | lr 1.8851e-04 | norm: 0.3600 | dt: 6340.72ms | tok/sec: 82685.9 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.4ms | fwd: 2064.0ms | loss: 6.5ms | bwd: 4209.6ms | clip: 0.0ms | optim: 49.2ms
step   262 | loss: 1.402372 | lr 1.8747e-04 | norm: 0.2449 | dt: 6367.98ms | tok/sec: 82331.9 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.4ms | fwd: 2084.2ms | loss: 6.5ms | bwd: 4216.2ms | clip: 0.0ms | optim: 49.7ms
step   263 | loss: 1.414258 | lr 1.8643e-04 | norm: 0.2076 | dt: 6363.57ms | tok/sec: 82388.9 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.4ms | fwd: 2073.9ms | loss: 6.5ms | bwd: 4222.1ms | clip: 0.0ms | optim: 49.6ms
step   264 | loss: 1.417034 | lr 1.8539e-04 | norm: 0.2528 | dt: 6306.01ms | tok/sec: 83141.0 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.4ms | fwd: 2014.8ms | loss: 6.5ms | bwd: 4223.5ms | clip: 0.0ms | optim: 49.7ms
step   265 | loss: 1.355837 | lr 1.8436e-04 | norm: 0.2565 | dt: 6316.67ms | tok/sec: 83000.7 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.4ms | fwd: 2026.0ms | loss: 6.5ms | bwd: 4222.7ms | clip: 0.0ms | optim: 49.9ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_a/in2a_train_00002.bin
100000000
step   266 | loss: 1.322376 | lr 1.8332e-04 | norm: 0.2186 | dt: 6337.70ms | tok/sec: 82725.3 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 14.1ms | fwd: 2024.1ms | loss: 6.5ms | bwd: 4235.2ms | clip: 0.0ms | optim: 49.7ms
step   267 | loss: 1.440838 | lr 1.8229e-04 | norm: 0.3168 | dt: 6305.52ms | tok/sec: 83147.5 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2014.6ms | loss: 6.5ms | bwd: 4223.2ms | clip: 0.0ms | optim: 49.6ms
step   268 | loss: 1.420192 | lr 1.8126e-04 | norm: 0.2564 | dt: 6306.09ms | tok/sec: 83140.0 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.4ms | fwd: 2014.0ms | loss: 6.5ms | bwd: 4224.5ms | clip: 0.0ms | optim: 49.7ms
step   269 | loss: 1.392399 | lr 1.8023e-04 | norm: 0.2095 | dt: 6304.86ms | tok/sec: 83156.2 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2013.6ms | loss: 6.5ms | bwd: 4223.7ms | clip: 0.0ms | optim: 49.4ms
step   270 | loss: 1.346640 | lr 1.7921e-04 | norm: 0.2733 | dt: 6351.62ms | tok/sec: 82544.0 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.4ms | fwd: 2061.0ms | loss: 6.5ms | bwd: 4222.5ms | clip: 0.0ms | optim: 50.1ms
step   271 | loss: 1.433407 | lr 1.7819e-04 | norm: 0.2332 | dt: 6347.45ms | tok/sec: 82598.2 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.4ms | fwd: 2047.4ms | loss: 6.5ms | bwd: 4232.2ms | clip: 0.0ms | optim: 49.9ms
step   272 | loss: 1.397035 | lr 1.7716e-04 | norm: 0.2501 | dt: 6298.57ms | tok/sec: 83239.1 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.4ms | fwd: 2018.0ms | loss: 6.5ms | bwd: 4212.9ms | clip: 0.0ms | optim: 49.6ms
step   273 | loss: 1.466986 | lr 1.7615e-04 | norm: 0.2236 | dt: 6322.12ms | tok/sec: 82929.1 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2022.1ms | loss: 6.5ms | bwd: 4232.2ms | clip: 0.0ms | optim: 49.7ms
step   274 | loss: 1.389561 | lr 1.7513e-04 | norm: 0.2747 | dt: 6304.07ms | tok/sec: 83166.5 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.4ms | fwd: 2020.0ms | loss: 6.5ms | bwd: 4216.5ms | clip: 0.0ms | optim: 49.6ms
step   275 | loss: 1.431594 | lr 1.7412e-04 | norm: 0.2942 | dt: 6325.80ms | tok/sec: 82880.9 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2018.1ms | loss: 6.5ms | bwd: 4240.0ms | clip: 0.0ms | optim: 49.7ms
step   276 | loss: 1.361612 | lr 1.7311e-04 | norm: 0.2318 | dt: 6334.20ms | tok/sec: 82770.9 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2021.9ms | loss: 6.5ms | bwd: 4244.7ms | clip: 0.0ms | optim: 49.5ms
step   277 | loss: 1.433590 | lr 1.7210e-04 | norm: 0.2515 | dt: 6312.01ms | tok/sec: 83062.0 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.3ms | fwd: 2015.8ms | loss: 6.5ms | bwd: 4228.8ms | clip: 0.0ms | optim: 49.5ms
step   278 | loss: 1.356968 | lr 1.7110e-04 | norm: 0.2141 | dt: 6316.09ms | tok/sec: 83008.3 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.4ms | fwd: 2009.6ms | loss: 6.5ms | bwd: 4238.8ms | clip: 0.0ms | optim: 49.7ms
step   279 | loss: 1.454445 | lr 1.7010e-04 | norm: 0.2266 | dt: 6315.14ms | tok/sec: 83020.8 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2017.5ms | loss: 6.5ms | bwd: 4229.9ms | clip: 0.0ms | optim: 49.6ms
step   280 | loss: 1.379414 | lr 1.6910e-04 | norm: 0.3033 | dt: 6292.59ms | tok/sec: 83318.4 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.4ms | fwd: 2013.5ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.3ms
step   281 | loss: 1.386713 | lr 1.6810e-04 | norm: 0.3237 | dt: 6309.57ms | tok/sec: 83094.1 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2013.9ms | loss: 6.5ms | bwd: 4227.9ms | clip: 0.0ms | optim: 49.6ms
step   282 | loss: 1.383668 | lr 1.6711e-04 | norm: 0.2803 | dt: 6297.62ms | tok/sec: 83251.8 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4217.1ms | clip: 0.0ms | optim: 49.6ms
step   283 | loss: 1.299441 | lr 1.6613e-04 | norm: 0.2430 | dt: 6325.68ms | tok/sec: 82882.5 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2024.2ms | loss: 6.5ms | bwd: 4233.9ms | clip: 0.0ms | optim: 49.5ms
step   284 | loss: 1.425351 | lr 1.6514e-04 | norm: 0.3806 | dt: 6380.71ms | tok/sec: 82167.6 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.4ms | fwd: 2078.1ms | loss: 6.5ms | bwd: 4234.6ms | clip: 0.0ms | optim: 50.1ms
step   285 | loss: 1.390065 | lr 1.6416e-04 | norm: 0.2147 | dt: 6321.00ms | tok/sec: 82943.8 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.4ms | fwd: 2026.6ms | loss: 6.5ms | bwd: 4226.6ms | clip: 0.0ms | optim: 49.8ms
step   286 | loss: 1.374380 | lr 1.6319e-04 | norm: 0.2448 | dt: 6319.97ms | tok/sec: 82957.3 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.4ms | fwd: 2017.8ms | loss: 6.5ms | bwd: 4234.0ms | clip: 0.0ms | optim: 50.1ms
step   287 | loss: 1.316681 | lr 1.6222e-04 | norm: 0.2188 | dt: 6310.71ms | tok/sec: 83079.1 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2016.7ms | loss: 6.5ms | bwd: 4226.6ms | clip: 0.0ms | optim: 49.3ms
step   288 | loss: 1.351033 | lr 1.6125e-04 | norm: 0.2382 | dt: 6331.87ms | tok/sec: 82801.4 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.4ms | fwd: 2014.8ms | loss: 6.7ms | bwd: 4249.3ms | clip: 0.0ms | optim: 49.6ms
step   289 | loss: 1.321794 | lr 1.6029e-04 | norm: 0.1796 | dt: 6327.12ms | tok/sec: 82863.6 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.6ms | fwd: 2017.1ms | loss: 6.5ms | bwd: 4242.4ms | clip: 0.0ms | optim: 49.4ms
step   290 | loss: 1.298964 | lr 1.5933e-04 | norm: 0.2966 | dt: 6300.59ms | tok/sec: 83212.5 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2018.2ms | loss: 6.5ms | bwd: 4215.0ms | clip: 0.0ms | optim: 49.3ms
step   291 | loss: 1.376741 | lr 1.5837e-04 | norm: 0.2576 | dt: 6312.19ms | tok/sec: 83059.7 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.4ms | fwd: 2012.9ms | loss: 6.5ms | bwd: 4231.8ms | clip: 0.0ms | optim: 49.6ms
step   292 | loss: 1.383591 | lr 1.5742e-04 | norm: 0.2024 | dt: 6329.68ms | tok/sec: 82830.1 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2018.1ms | loss: 6.5ms | bwd: 4243.9ms | clip: 0.0ms | optim: 49.5ms
step   293 | loss: 1.401507 | lr 1.5648e-04 | norm: 0.2136 | dt: 6295.29ms | tok/sec: 83282.6 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.3ms | fwd: 2017.1ms | loss: 6.5ms | bwd: 4210.9ms | clip: 0.0ms | optim: 49.4ms
step   294 | loss: 1.331201 | lr 1.5554e-04 | norm: 0.2242 | dt: 6322.26ms | tok/sec: 82927.3 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2018.5ms | loss: 6.5ms | bwd: 4236.2ms | clip: 0.0ms | optim: 49.5ms
step   295 | loss: 1.387356 | lr 1.5460e-04 | norm: 0.2160 | dt: 6293.03ms | tok/sec: 83312.5 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.4ms | fwd: 2016.1ms | loss: 6.5ms | bwd: 4209.2ms | clip: 0.0ms | optim: 49.7ms
step   296 | loss: 1.405216 | lr 1.5367e-04 | norm: 0.2442 | dt: 6307.82ms | tok/sec: 83117.1 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4231.5ms | clip: 0.0ms | optim: 49.3ms
step   297 | loss: 1.327925 | lr 1.5274e-04 | norm: 0.2268 | dt: 6299.63ms | tok/sec: 83225.2 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4223.0ms | clip: 0.0ms | optim: 49.5ms
step   298 | loss: 1.343456 | lr 1.5182e-04 | norm: 0.2181 | dt: 6361.32ms | tok/sec: 82418.1 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 2081.8ms | loss: 6.5ms | bwd: 4212.0ms | clip: 0.0ms | optim: 49.6ms
step   299 | loss: 1.341276 | lr 1.5091e-04 | norm: 0.3151 | dt: 6364.26ms | tok/sec: 82380.0 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 2080.0ms | loss: 6.5ms | bwd: 4216.3ms | clip: 0.0ms | optim: 50.0ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_300.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_100.ckpt
step   300 | loss: 1.391191 | lr 1.5000e-04 | norm: 0.2160 | dt: 6192.44ms | tok/sec: 84665.8 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 1989.1ms | loss: 6.5ms | bwd: 4135.5ms | clip: 0.0ms | optim: 49.7ms
step   301 | loss: 1.362000 | lr 1.4910e-04 | norm: 0.2124 | dt: 6243.52ms | tok/sec: 83973.2 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 1998.7ms | loss: 6.5ms | bwd: 4177.5ms | clip: 0.0ms | optim: 49.3ms
step   302 | loss: 1.361197 | lr 1.4820e-04 | norm: 0.1901 | dt: 6241.76ms | tok/sec: 83996.9 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.3ms | fwd: 2001.5ms | loss: 6.5ms | bwd: 4173.0ms | clip: 0.0ms | optim: 49.3ms
step   303 | loss: 1.277367 | lr 1.4730e-04 | norm: 0.3982 | dt: 6303.34ms | tok/sec: 83176.3 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2032.5ms | loss: 6.5ms | bwd: 4203.3ms | clip: 0.0ms | optim: 49.5ms
step   304 | loss: 1.388580 | lr 1.4642e-04 | norm: 0.2468 | dt: 6310.98ms | tok/sec: 83075.6 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2019.8ms | loss: 6.5ms | bwd: 4223.5ms | clip: 0.0ms | optim: 49.7ms
step   305 | loss: 1.406972 | lr 1.4554e-04 | norm: 0.2275 | dt: 6287.48ms | tok/sec: 83386.1 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2011.8ms | loss: 6.5ms | bwd: 4208.3ms | clip: 0.0ms | optim: 49.4ms
step   306 | loss: 1.317721 | lr 1.4466e-04 | norm: 0.2053 | dt: 6337.80ms | tok/sec: 82723.9 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2043.4ms | loss: 6.5ms | bwd: 4226.8ms | clip: 0.0ms | optim: 49.6ms
step   307 | loss: 1.430418 | lr 1.4379e-04 | norm: 0.2306 | dt: 6327.15ms | tok/sec: 82863.2 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2024.9ms | loss: 6.5ms | bwd: 4234.8ms | clip: 0.0ms | optim: 49.4ms
step   308 | loss: 1.414106 | lr 1.4293e-04 | norm: 0.1776 | dt: 6371.33ms | tok/sec: 82288.6 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2080.7ms | loss: 6.5ms | bwd: 4222.9ms | clip: 0.0ms | optim: 49.7ms
step   309 | loss: 1.356122 | lr 1.4207e-04 | norm: 0.1986 | dt: 6321.46ms | tok/sec: 82937.8 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2023.9ms | loss: 6.5ms | bwd: 4230.0ms | clip: 0.0ms | optim: 49.5ms
step   310 | loss: 1.299470 | lr 1.4122e-04 | norm: 0.1979 | dt: 6334.44ms | tok/sec: 82767.8 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.3ms | fwd: 2015.9ms | loss: 6.5ms | bwd: 4251.4ms | clip: 0.0ms | optim: 49.4ms
step   311 | loss: 1.377680 | lr 1.4038e-04 | norm: 0.2274 | dt: 6345.94ms | tok/sec: 82617.9 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2025.6ms | loss: 6.5ms | bwd: 4252.8ms | clip: 0.0ms | optim: 49.5ms
step   312 | loss: 1.333104 | lr 1.3954e-04 | norm: 0.2316 | dt: 6324.44ms | tok/sec: 82898.7 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2020.1ms | loss: 6.5ms | bwd: 4236.8ms | clip: 0.0ms | optim: 49.6ms
step   313 | loss: 1.316221 | lr 1.3871e-04 | norm: 0.2666 | dt: 6333.14ms | tok/sec: 82784.8 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2022.7ms | loss: 6.5ms | bwd: 4243.0ms | clip: 0.0ms | optim: 49.4ms
step   314 | loss: 1.370312 | lr 1.3789e-04 | norm: 0.2199 | dt: 6329.23ms | tok/sec: 82835.9 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2023.2ms | loss: 6.5ms | bwd: 4238.2ms | clip: 0.0ms | optim: 49.8ms
step   315 | loss: 1.381422 | lr 1.3707e-04 | norm: 0.2065 | dt: 6326.27ms | tok/sec: 82874.8 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2019.2ms | loss: 6.5ms | bwd: 4239.7ms | clip: 0.0ms | optim: 49.4ms
step   316 | loss: 1.354267 | lr 1.3626e-04 | norm: 0.2090 | dt: 6325.81ms | tok/sec: 82880.7 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2020.0ms | loss: 6.5ms | bwd: 4238.1ms | clip: 0.0ms | optim: 49.8ms
step   317 | loss: 1.348798 | lr 1.3545e-04 | norm: 0.1857 | dt: 6362.95ms | tok/sec: 82396.9 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2044.8ms | loss: 6.5ms | bwd: 4250.8ms | clip: 0.0ms | optim: 49.4ms
step   318 | loss: 1.314881 | lr 1.3466e-04 | norm: 0.1938 | dt: 6333.48ms | tok/sec: 82780.5 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2020.9ms | loss: 6.5ms | bwd: 4244.9ms | clip: 0.0ms | optim: 49.5ms
step   319 | loss: 1.435550 | lr 1.3387e-04 | norm: 0.1807 | dt: 6356.31ms | tok/sec: 82483.1 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2025.9ms | loss: 6.5ms | bwd: 4262.7ms | clip: 0.0ms | optim: 49.4ms
step   320 | loss: 1.369530 | lr 1.3309e-04 | norm: 0.2233 | dt: 6344.01ms | tok/sec: 82643.0 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2030.1ms | loss: 6.5ms | bwd: 4246.5ms | clip: 0.0ms | optim: 49.3ms
step   321 | loss: 1.434570 | lr 1.3231e-04 | norm: 0.2046 | dt: 6321.87ms | tok/sec: 82932.5 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2019.8ms | loss: 6.5ms | bwd: 4234.5ms | clip: 0.0ms | optim: 49.7ms
step   322 | loss: 1.364758 | lr 1.3155e-04 | norm: 0.2017 | dt: 6318.53ms | tok/sec: 82976.2 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2017.2ms | loss: 6.5ms | bwd: 4233.7ms | clip: 0.0ms | optim: 49.5ms
step   323 | loss: 1.356215 | lr 1.3079e-04 | norm: 0.1822 | dt: 6366.75ms | tok/sec: 82347.8 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2031.0ms | loss: 6.5ms | bwd: 4268.2ms | clip: 0.0ms | optim: 49.5ms
step   324 | loss: 1.334546 | lr 1.3003e-04 | norm: 0.2161 | dt: 6325.44ms | tok/sec: 82885.6 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2018.6ms | loss: 6.5ms | bwd: 4239.2ms | clip: 0.0ms | optim: 49.5ms
step   325 | loss: 1.355955 | lr 1.2929e-04 | norm: 0.2677 | dt: 6320.63ms | tok/sec: 82948.7 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2022.6ms | loss: 6.5ms | bwd: 4230.6ms | clip: 0.0ms | optim: 49.3ms
step   326 | loss: 1.415459 | lr 1.2855e-04 | norm: 0.4096 | dt: 6338.03ms | tok/sec: 82721.0 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2021.3ms | loss: 6.5ms | bwd: 4249.2ms | clip: 0.0ms | optim: 49.3ms
step   327 | loss: 1.334630 | lr 1.2782e-04 | norm: 0.2380 | dt: 6323.20ms | tok/sec: 82914.9 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2018.1ms | loss: 6.5ms | bwd: 4237.4ms | clip: 0.0ms | optim: 49.7ms
step   328 | loss: 1.298984 | lr 1.2710e-04 | norm: 0.7395 | dt: 6326.01ms | tok/sec: 82878.2 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2018.4ms | loss: 6.5ms | bwd: 4240.1ms | clip: 0.0ms | optim: 49.6ms
step   329 | loss: 1.425144 | lr 1.2639e-04 | norm: 0.2025 | dt: 6331.02ms | tok/sec: 82812.5 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2021.1ms | loss: 6.5ms | bwd: 4242.7ms | clip: 0.0ms | optim: 49.3ms
step   330 | loss: 1.361766 | lr 1.2569e-04 | norm: 0.2503 | dt: 6312.86ms | tok/sec: 83050.8 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2015.5ms | loss: 6.5ms | bwd: 4230.1ms | clip: 0.0ms | optim: 49.4ms
step   331 | loss: 1.340282 | lr 1.2499e-04 | norm: 0.1725 | dt: 6353.84ms | tok/sec: 82515.1 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2023.4ms | loss: 6.5ms | bwd: 4262.6ms | clip: 0.0ms | optim: 49.9ms
step   332 | loss: 1.311442 | lr 1.2430e-04 | norm: 0.1716 | dt: 6309.97ms | tok/sec: 83088.9 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2018.3ms | loss: 6.5ms | bwd: 4224.0ms | clip: 0.0ms | optim: 49.7ms
step   333 | loss: 1.344754 | lr 1.2362e-04 | norm: 0.1880 | dt: 6345.22ms | tok/sec: 82627.3 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2041.0ms | loss: 6.5ms | bwd: 4236.8ms | clip: 0.0ms | optim: 49.4ms
step   334 | loss: 1.337159 | lr 1.2295e-04 | norm: 0.1570 | dt: 6350.10ms | tok/sec: 82563.7 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2056.3ms | loss: 6.5ms | bwd: 4226.1ms | clip: 0.0ms | optim: 49.5ms
step   335 | loss: 1.393037 | lr 1.2229e-04 | norm: 0.2108 | dt: 6334.41ms | tok/sec: 82768.2 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 4.1ms | fwd: 2034.9ms | loss: 6.5ms | bwd: 4231.1ms | clip: 0.0ms | optim: 49.7ms
step   336 | loss: 1.353839 | lr 1.2163e-04 | norm: 0.1790 | dt: 6376.78ms | tok/sec: 82218.3 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.3ms | fwd: 2088.8ms | loss: 6.5ms | bwd: 4220.7ms | clip: 0.0ms | optim: 49.4ms
step   337 | loss: 1.370545 | lr 1.2098e-04 | norm: 0.1646 | dt: 6317.62ms | tok/sec: 82988.2 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.4ms | fwd: 2019.6ms | loss: 6.5ms | bwd: 4230.5ms | clip: 0.0ms | optim: 49.6ms
step   338 | loss: 1.392171 | lr 1.2035e-04 | norm: 0.2048 | dt: 6331.41ms | tok/sec: 82807.5 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.6ms | fwd: 2018.7ms | loss: 6.5ms | bwd: 4244.9ms | clip: 0.0ms | optim: 49.6ms
step   339 | loss: 1.374082 | lr 1.1972e-04 | norm: 0.1662 | dt: 6327.89ms | tok/sec: 82853.5 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.4ms | fwd: 2020.4ms | loss: 6.5ms | bwd: 4239.7ms | clip: 0.0ms | optim: 49.8ms
step   340 | loss: 1.359741 | lr 1.1910e-04 | norm: 0.1590 | dt: 6308.21ms | tok/sec: 83112.1 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.4ms | fwd: 2018.1ms | loss: 6.5ms | bwd: 4222.7ms | clip: 0.0ms | optim: 49.4ms
step   341 | loss: 1.341903 | lr 1.1849e-04 | norm: 0.1668 | dt: 6317.02ms | tok/sec: 82996.1 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2018.3ms | loss: 6.5ms | bwd: 4231.2ms | clip: 0.0ms | optim: 49.4ms
step   342 | loss: 1.395497 | lr 1.1789e-04 | norm: 0.1887 | dt: 6321.74ms | tok/sec: 82934.1 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2018.5ms | loss: 6.5ms | bwd: 4235.5ms | clip: 0.0ms | optim: 49.8ms
step   343 | loss: 1.352996 | lr 1.1729e-04 | norm: 0.1867 | dt: 6379.22ms | tok/sec: 82186.9 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2087.5ms | loss: 6.5ms | bwd: 4224.4ms | clip: 0.0ms | optim: 49.3ms
step   344 | loss: 1.316176 | lr 1.1671e-04 | norm: 0.1691 | dt: 6314.94ms | tok/sec: 83023.4 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.4ms | fwd: 2028.4ms | loss: 6.5ms | bwd: 4219.1ms | clip: 0.0ms | optim: 49.4ms
step   345 | loss: 1.303779 | lr 1.1613e-04 | norm: 0.1597 | dt: 6357.66ms | tok/sec: 82465.6 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2081.0ms | loss: 6.5ms | bwd: 4209.0ms | clip: 0.0ms | optim: 49.6ms
step   346 | loss: 1.338077 | lr 1.1557e-04 | norm: 0.1600 | dt: 6359.48ms | tok/sec: 82442.0 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.4ms | fwd: 2036.5ms | loss: 6.5ms | bwd: 4255.4ms | clip: 0.0ms | optim: 49.6ms
step   347 | loss: 1.341638 | lr 1.1501e-04 | norm: 0.1692 | dt: 6309.57ms | tok/sec: 83094.1 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.4ms | fwd: 2022.5ms | loss: 6.5ms | bwd: 4219.2ms | clip: 0.0ms | optim: 49.8ms
step   348 | loss: 1.359211 | lr 1.1446e-04 | norm: 0.2055 | dt: 6296.76ms | tok/sec: 83263.2 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.4ms | fwd: 2014.7ms | loss: 6.5ms | bwd: 4214.6ms | clip: 0.0ms | optim: 49.6ms
step   349 | loss: 1.292385 | lr 1.1393e-04 | norm: 0.1510 | dt: 6318.84ms | tok/sec: 82972.2 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2021.3ms | loss: 6.5ms | bwd: 4229.8ms | clip: 0.0ms | optim: 49.7ms
step   350 | loss: 1.336500 | lr 1.1340e-04 | norm: 0.1895 | dt: 6320.99ms | tok/sec: 82944.0 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2017.5ms | loss: 6.8ms | bwd: 4235.5ms | clip: 0.0ms | optim: 49.6ms
step   351 | loss: 1.360193 | lr 1.1288e-04 | norm: 0.2685 | dt: 6343.21ms | tok/sec: 82653.4 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.3ms | fwd: 2021.2ms | loss: 6.5ms | bwd: 4254.5ms | clip: 0.0ms | optim: 49.6ms
step   352 | loss: 1.404982 | lr 1.1237e-04 | norm: 0.1703 | dt: 6317.10ms | tok/sec: 82995.0 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4236.2ms | clip: 0.0ms | optim: 49.4ms
step   353 | loss: 1.359454 | lr 1.1187e-04 | norm: 0.1571 | dt: 6331.24ms | tok/sec: 82809.7 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.3ms | fwd: 2018.6ms | loss: 6.5ms | bwd: 4245.0ms | clip: 0.0ms | optim: 49.7ms
step   354 | loss: 1.312837 | lr 1.1138e-04 | norm: 0.1797 | dt: 6320.55ms | tok/sec: 82949.7 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.4ms | fwd: 2023.0ms | loss: 6.5ms | bwd: 4229.6ms | clip: 0.0ms | optim: 49.8ms
step   355 | loss: 1.388402 | lr 1.1090e-04 | norm: 0.1953 | dt: 6316.64ms | tok/sec: 83001.1 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2025.8ms | loss: 6.5ms | bwd: 4222.5ms | clip: 0.0ms | optim: 50.1ms
step   356 | loss: 1.352636 | lr 1.1043e-04 | norm: 0.1466 | dt: 6346.95ms | tok/sec: 82604.7 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.4ms | fwd: 2046.5ms | loss: 6.5ms | bwd: 4233.1ms | clip: 0.0ms | optim: 49.4ms
step   357 | loss: 1.382410 | lr 1.0997e-04 | norm: 0.2104 | dt: 6311.31ms | tok/sec: 83071.2 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.4ms | fwd: 2021.8ms | loss: 6.5ms | bwd: 4221.6ms | clip: 0.0ms | optim: 49.9ms
step   358 | loss: 1.387089 | lr 1.0952e-04 | norm: 0.2091 | dt: 6370.65ms | tok/sec: 82297.4 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2062.6ms | loss: 6.5ms | bwd: 4240.5ms | clip: 0.0ms | optim: 49.3ms
step   359 | loss: 1.257791 | lr 1.0908e-04 | norm: 0.1538 | dt: 6305.10ms | tok/sec: 83153.0 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2016.5ms | loss: 6.5ms | bwd: 4221.0ms | clip: 0.0ms | optim: 49.6ms
step   360 | loss: 1.345636 | lr 1.0865e-04 | norm: 0.1572 | dt: 6314.99ms | tok/sec: 83022.7 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.4ms | fwd: 2018.7ms | loss: 6.5ms | bwd: 4228.8ms | clip: 0.0ms | optim: 49.6ms
step   361 | loss: 1.377840 | lr 1.0822e-04 | norm: 0.1865 | dt: 6319.93ms | tok/sec: 82957.8 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.4ms | fwd: 2012.4ms | loss: 6.5ms | bwd: 4240.1ms | clip: 0.0ms | optim: 49.4ms
step   362 | loss: 1.333666 | lr 1.0781e-04 | norm: 0.1799 | dt: 6323.63ms | tok/sec: 82909.4 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2026.0ms | loss: 6.5ms | bwd: 4229.7ms | clip: 0.0ms | optim: 49.9ms
step   363 | loss: 1.356241 | lr 1.0741e-04 | norm: 0.1625 | dt: 6311.40ms | tok/sec: 83070.0 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2020.7ms | loss: 6.5ms | bwd: 4223.2ms | clip: 0.0ms | optim: 49.6ms
step   364 | loss: 1.331657 | lr 1.0702e-04 | norm: 0.1664 | dt: 6383.35ms | tok/sec: 82133.7 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2086.8ms | loss: 6.5ms | bwd: 4229.0ms | clip: 0.0ms | optim: 49.5ms
step   365 | loss: 1.368689 | lr 1.0664e-04 | norm: 0.1673 | dt: 6375.09ms | tok/sec: 82240.1 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.9ms | fwd: 2081.2ms | loss: 6.7ms | bwd: 4225.8ms | clip: 0.0ms | optim: 49.4ms
step   366 | loss: 1.353878 | lr 1.0627e-04 | norm: 0.1713 | dt: 6331.72ms | tok/sec: 82803.5 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2020.4ms | loss: 6.5ms | bwd: 4243.8ms | clip: 0.0ms | optim: 49.6ms
step   367 | loss: 1.345177 | lr 1.0591e-04 | norm: 0.1488 | dt: 6323.10ms | tok/sec: 82916.3 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2020.7ms | loss: 6.5ms | bwd: 4234.7ms | clip: 0.0ms | optim: 49.7ms
step   368 | loss: 1.354756 | lr 1.0556e-04 | norm: 0.1610 | dt: 6328.72ms | tok/sec: 82842.7 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2018.2ms | loss: 6.5ms | bwd: 4242.3ms | clip: 0.0ms | optim: 50.3ms
step   369 | loss: 1.360016 | lr 1.0522e-04 | norm: 0.1686 | dt: 6310.30ms | tok/sec: 83084.4 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2014.6ms | loss: 6.5ms | bwd: 4227.2ms | clip: 0.0ms | optim: 49.5ms
step   370 | loss: 1.289404 | lr 1.0489e-04 | norm: 0.1345 | dt: 6318.59ms | tok/sec: 82975.5 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2020.8ms | loss: 6.5ms | bwd: 4230.1ms | clip: 0.0ms | optim: 49.7ms
step   371 | loss: 1.349306 | lr 1.0458e-04 | norm: 0.1813 | dt: 6302.31ms | tok/sec: 83189.8 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2016.4ms | loss: 6.5ms | bwd: 4218.3ms | clip: 0.0ms | optim: 49.6ms
step   372 | loss: 1.373928 | lr 1.0427e-04 | norm: 0.1441 | dt: 6328.48ms | tok/sec: 82845.8 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2023.3ms | loss: 6.5ms | bwd: 4237.3ms | clip: 0.0ms | optim: 49.9ms
step   373 | loss: 1.312651 | lr 1.0397e-04 | norm: 0.1611 | dt: 6318.58ms | tok/sec: 82975.6 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2014.7ms | loss: 6.5ms | bwd: 4236.5ms | clip: 0.0ms | optim: 49.4ms
step   374 | loss: 1.321823 | lr 1.0368e-04 | norm: 0.1500 | dt: 6298.86ms | tok/sec: 83235.4 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2020.6ms | loss: 6.5ms | bwd: 4210.5ms | clip: 0.0ms | optim: 49.7ms
step   375 | loss: 1.378791 | lr 1.0341e-04 | norm: 0.1801 | dt: 6300.06ms | tok/sec: 83219.5 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2019.3ms | loss: 6.5ms | bwd: 4213.3ms | clip: 0.0ms | optim: 49.6ms
step   376 | loss: 1.366055 | lr 1.0314e-04 | norm: 0.1419 | dt: 6313.94ms | tok/sec: 83036.6 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.3ms | fwd: 2024.4ms | loss: 6.5ms | bwd: 4222.1ms | clip: 0.0ms | optim: 49.6ms
step   377 | loss: 1.307925 | lr 1.0289e-04 | norm: 0.1381 | dt: 6312.74ms | tok/sec: 83052.4 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2013.8ms | loss: 6.5ms | bwd: 4231.6ms | clip: 0.0ms | optim: 49.3ms
step   378 | loss: 1.324249 | lr 1.0264e-04 | norm: 0.1505 | dt: 6303.69ms | tok/sec: 83171.6 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4222.6ms | clip: 0.0ms | optim: 49.6ms
step   379 | loss: 1.305159 | lr 1.0241e-04 | norm: 0.1344 | dt: 6298.56ms | tok/sec: 83239.3 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4219.7ms | clip: 0.0ms | optim: 49.6ms
step   380 | loss: 1.379794 | lr 1.0219e-04 | norm: 0.1864 | dt: 6368.75ms | tok/sec: 82322.0 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2057.6ms | loss: 6.5ms | bwd: 4242.7ms | clip: 0.0ms | optim: 50.4ms
step   381 | loss: 1.369458 | lr 1.0197e-04 | norm: 0.1669 | dt: 6319.32ms | tok/sec: 82965.9 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.3ms | fwd: 2029.4ms | loss: 6.5ms | bwd: 4222.4ms | clip: 0.0ms | optim: 49.6ms
step   382 | loss: 1.343205 | lr 1.0177e-04 | norm: 0.1390 | dt: 6361.76ms | tok/sec: 82412.4 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2065.8ms | loss: 6.5ms | bwd: 4228.4ms | clip: 0.0ms | optim: 49.5ms
step   383 | loss: 1.285721 | lr 1.0158e-04 | norm: 0.1478 | dt: 6328.33ms | tok/sec: 82847.8 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2024.0ms | loss: 6.5ms | bwd: 4236.8ms | clip: 0.0ms | optim: 49.4ms
step   384 | loss: 1.380865 | lr 1.0140e-04 | norm: 0.1620 | dt: 6339.78ms | tok/sec: 82698.1 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2026.4ms | loss: 6.5ms | bwd: 4245.8ms | clip: 0.0ms | optim: 49.6ms
step   385 | loss: 1.351010 | lr 1.0123e-04 | norm: 0.1834 | dt: 6330.00ms | tok/sec: 82825.9 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2022.6ms | loss: 6.5ms | bwd: 4239.8ms | clip: 0.0ms | optim: 49.5ms
step   386 | loss: 1.366956 | lr 1.0107e-04 | norm: 0.1519 | dt: 6317.08ms | tok/sec: 82995.3 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2015.6ms | loss: 6.8ms | bwd: 4233.9ms | clip: 0.0ms | optim: 49.3ms
step   387 | loss: 1.382017 | lr 1.0093e-04 | norm: 0.1785 | dt: 6327.28ms | tok/sec: 82861.5 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2015.1ms | loss: 6.5ms | bwd: 4244.5ms | clip: 0.0ms | optim: 49.6ms
step   388 | loss: 1.369122 | lr 1.0079e-04 | norm: 0.1681 | dt: 6335.35ms | tok/sec: 82755.9 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2022.5ms | loss: 6.5ms | bwd: 4245.2ms | clip: 0.0ms | optim: 49.6ms
step   389 | loss: 1.404818 | lr 1.0066e-04 | norm: 0.1697 | dt: 6303.50ms | tok/sec: 83174.2 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4226.2ms | clip: 0.0ms | optim: 49.5ms
step   390 | loss: 1.235723 | lr 1.0055e-04 | norm: 0.1330 | dt: 6315.98ms | tok/sec: 83009.7 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2037.4ms | loss: 6.5ms | bwd: 4210.4ms | clip: 0.0ms | optim: 50.1ms
step   391 | loss: 1.311336 | lr 1.0044e-04 | norm: 0.2306 | dt: 6323.52ms | tok/sec: 82910.8 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.4ms | fwd: 2018.0ms | loss: 6.5ms | bwd: 4238.1ms | clip: 0.0ms | optim: 49.5ms
step   392 | loss: 1.318072 | lr 1.0035e-04 | norm: 0.1570 | dt: 6337.11ms | tok/sec: 82733.0 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.4ms | fwd: 2022.6ms | loss: 6.5ms | bwd: 4247.1ms | clip: 0.0ms | optim: 49.4ms
step   393 | loss: 1.372988 | lr 1.0027e-04 | norm: 0.1533 | dt: 6303.34ms | tok/sec: 83176.3 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.4ms | fwd: 2014.7ms | loss: 6.5ms | bwd: 4220.9ms | clip: 0.0ms | optim: 49.9ms
step   394 | loss: 1.328925 | lr 1.0020e-04 | norm: 0.1807 | dt: 6306.25ms | tok/sec: 83137.8 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.4ms | fwd: 2016.2ms | loss: 6.5ms | bwd: 4222.6ms | clip: 0.0ms | optim: 49.6ms
step   395 | loss: 1.373340 | lr 1.0014e-04 | norm: 0.1956 | dt: 6340.86ms | tok/sec: 82684.0 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.4ms | fwd: 2024.5ms | loss: 6.5ms | bwd: 4248.8ms | clip: 0.0ms | optim: 49.7ms
step   396 | loss: 1.351218 | lr 1.0009e-04 | norm: 0.1567 | dt: 6294.05ms | tok/sec: 83298.9 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2015.8ms | loss: 6.5ms | bwd: 4210.3ms | clip: 0.0ms | optim: 49.8ms
step   397 | loss: 1.324154 | lr 1.0005e-04 | norm: 0.1398 | dt: 6327.40ms | tok/sec: 82860.0 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2017.6ms | loss: 6.5ms | bwd: 4242.3ms | clip: 0.0ms | optim: 49.5ms
step   398 | loss: 1.333024 | lr 1.0002e-04 | norm: 0.1600 | dt: 6354.76ms | tok/sec: 82503.2 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.4ms | fwd: 2051.3ms | loss: 6.5ms | bwd: 4235.4ms | clip: 0.0ms | optim: 50.1ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_a/in2a_val_00000.bin
100000000
validation loss: 1.3379
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_a/in2a_train_00000.bin
100000000
step   399 | loss: 1.349215 | lr 1.0001e-04 | norm: 0.1506 | dt: 11404.87ms | tok/sec: 45970.6 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 13.2ms | fwd: 2016.9ms | loss: 6.5ms | bwd: 4219.9ms | clip: 0.0ms | optim: 49.4ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_400.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run12_step_200.ckpt

=== Training complete ===