BluScriptCP / logs /in2 /arm_b_train.log
Upload folder using huggingface_hub
f127a07 verified 3 months ago
101 kB
=== Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[ZeRO-1] CP_ZERO_SYMMETRIC=1: NCCL symmetric memory mode (NVLS)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
  d_model/layers  : 1536 / 36
  q/kv heads      : 12 / 4  (G=3, hd=128)
  ffn_hidden      : 4096 (SwiGLU)
  context_length  : 16384
  vocab_size      : 50304  tying=0
  B x T           : 4 x 16384
  world_size      : 4  (dp=1 x cp=4)
  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
  global_batch    : 524288  (grad_accum=8)
  optimizer       : Muon (2-D body) + AdamW (wte + 1-D)  [ZeRO-1]  muon_lr=0.001000
  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
  max_lr/min_lr   : 0.001 / 0.0001
  warmup/steps    : 1 / 400  (cosine, no stable phase)
  checkpointing   : true  (ckpt_freq=200, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt  (base step 750, loss 1.28324)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 400
warmup_steps: 1
[Muon] adjust_lr=match_rms_adamw (default)  ns_steps=5  momentum=0.95
[ZeRO-1/Muon] reduce-scatter/all-gather layout: 6 bucket(s), 12 collectives/step, shard 226492416 elems/rank, padding 0.000000% (flat buffer 1728.000000 MB)
[ZeRO-1/Muon] adjust_lr=match_rms_adamw | params=905969664 (216 orthogonalized / 0 AdamW) muon_elems=905969664 | W=4 owned=226492416 (42 tensors) | master+momentum: 1728.000000 MB/GPU
[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype)  symmetric=ON  overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
[ZeRO-1 v2] params=77266944 padded=77266944 W=4 shard=19316736 | 3 buckets of 50.000000 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 221.062500 MB (vs 884.250000 MB unsharded)
[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype)  symmetric=ON  overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
[ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50.000000 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.409668 MB (vs 885.638672 MB unsharded)
[ZeRO-1] variant=v2 reduce-scatter/all-gather  groups=3  [muon:bfloat16/905M]  [adamw:bfloat16/77266944]  [adamw:float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 3 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log15.csv (run 15)
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run15_step_<N>.ckpt (every 200 steps, keep 1)
[re-warmup] anchored at step 0 for 100 steps -> peak 0.000300 (0.300000 x max_lr), then cosine-decay to 0.000100 | resuming at step 0 (mid-ramp)

Starting training...

[CP ring P2P] OVERLAP: dedicated non-blocking stream
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_val_00000.bin
100000000
validation loss: 1.6834
[CP bwd] overlap=ON
step     0 | loss: 1.719764 | lr 3.0000e-06 | norm: 0.9173 | dt: 12521.47ms | tok/sec: 41871.1 | mem: 107213MB | Time Left: 01 hrs : 23 mins
  [TIMING] data: 3.1ms | fwd: 2215.8ms | loss: 8.7ms | bwd: 4188.4ms | clip: 0.0ms | optim: 67.4ms
step     1 | loss: 1.689247 | lr 6.0000e-06 | norm: 0.9828 | dt: 6394.49ms | tok/sec: 81990.6 | mem: 108909MB | Time Left: 00 hrs : 42 mins
  [TIMING] data: 3.4ms | fwd: 2198.1ms | loss: 6.4ms | bwd: 4128.8ms | clip: 0.0ms | optim: 49.9ms
step     2 | loss: 1.710385 | lr 9.0000e-06 | norm: 1.1134 | dt: 6413.58ms | tok/sec: 81746.6 | mem: 108909MB | Time Left: 00 hrs : 42 mins
  [TIMING] data: 2.9ms | fwd: 2200.6ms | loss: 6.4ms | bwd: 4145.5ms | clip: 0.0ms | optim: 49.9ms
step     3 | loss: 1.731750 | lr 1.2000e-05 | norm: 0.9261 | dt: 6203.30ms | tok/sec: 84517.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.4ms | fwd: 1985.5ms | loss: 6.5ms | bwd: 4150.7ms | clip: 0.0ms | optim: 49.4ms
step     4 | loss: 1.702127 | lr 1.5000e-05 | norm: 0.9661 | dt: 6256.03ms | tok/sec: 83805.2 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.6ms | fwd: 1997.1ms | loss: 6.5ms | bwd: 4191.6ms | clip: 0.0ms | optim: 49.3ms
step     5 | loss: 1.766132 | lr 1.8000e-05 | norm: 0.9505 | dt: 6338.90ms | tok/sec: 82709.7 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.4ms | fwd: 2019.0ms | loss: 6.5ms | bwd: 4252.6ms | clip: 0.0ms | optim: 49.4ms
step     6 | loss: 1.704157 | lr 2.1000e-05 | norm: 0.9635 | dt: 6389.08ms | tok/sec: 82060.0 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.4ms | fwd: 2030.8ms | loss: 6.5ms | bwd: 4291.0ms | clip: 0.0ms | optim: 49.4ms
step     7 | loss: 1.661039 | lr 2.4000e-05 | norm: 0.8039 | dt: 6425.50ms | tok/sec: 81594.9 | mem: 108909MB | Time Left: 00 hrs : 42 mins
  [TIMING] data: 3.4ms | fwd: 2037.1ms | loss: 6.5ms | bwd: 4321.1ms | clip: 0.0ms | optim: 49.3ms
step     8 | loss: 1.628232 | lr 2.7000e-05 | norm: 0.6773 | dt: 6463.38ms | tok/sec: 81116.7 | mem: 108909MB | Time Left: 00 hrs : 42 mins
  [TIMING] data: 3.4ms | fwd: 2038.4ms | loss: 6.9ms | bwd: 4357.3ms | clip: 0.0ms | optim: 49.4ms
step     9 | loss: 1.665926 | lr 3.0000e-05 | norm: 0.6631 | dt: 6454.53ms | tok/sec: 81228.0 | mem: 108909MB | Time Left: 00 hrs : 42 mins
  [TIMING] data: 3.4ms | fwd: 2034.7ms | loss: 6.8ms | bwd: 4352.1ms | clip: 0.0ms | optim: 49.5ms
step    10 | loss: 1.609743 | lr 3.3000e-05 | norm: 0.7424 | dt: 6411.87ms | tok/sec: 81768.4 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2021.4ms | loss: 6.7ms | bwd: 4323.0ms | clip: 0.0ms | optim: 49.3ms
step    11 | loss: 1.618221 | lr 3.6000e-05 | norm: 0.8395 | dt: 6397.12ms | tok/sec: 81956.9 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2034.2ms | loss: 6.5ms | bwd: 4295.7ms | clip: 0.0ms | optim: 49.3ms
step    12 | loss: 1.606266 | lr 3.9000e-05 | norm: 0.9928 | dt: 6349.88ms | tok/sec: 82566.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2020.4ms | loss: 6.5ms | bwd: 4262.1ms | clip: 0.0ms | optim: 49.5ms
step    13 | loss: 1.612944 | lr 4.2000e-05 | norm: 0.9347 | dt: 6321.89ms | tok/sec: 82932.1 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2017.0ms | loss: 6.5ms | bwd: 4236.4ms | clip: 0.0ms | optim: 49.8ms
step    14 | loss: 1.638992 | lr 4.5000e-05 | norm: 0.8749 | dt: 6345.43ms | tok/sec: 82624.5 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.6ms | fwd: 2022.4ms | loss: 6.5ms | bwd: 4255.6ms | clip: 0.0ms | optim: 49.4ms
step    15 | loss: 1.627380 | lr 4.8000e-05 | norm: 0.7985 | dt: 6306.52ms | tok/sec: 83134.2 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.6ms | fwd: 2015.6ms | loss: 6.5ms | bwd: 4223.4ms | clip: 0.0ms | optim: 49.5ms
step    16 | loss: 1.601821 | lr 5.1000e-05 | norm: 1.2730 | dt: 6309.95ms | tok/sec: 83089.1 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2007.0ms | loss: 6.5ms | bwd: 4235.7ms | clip: 0.0ms | optim: 49.4ms
step    17 | loss: 1.545084 | lr 5.4000e-05 | norm: 0.7838 | dt: 6294.81ms | tok/sec: 83288.9 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2003.3ms | loss: 6.8ms | bwd: 4223.5ms | clip: 0.0ms | optim: 49.7ms
step    18 | loss: 1.575317 | lr 5.7000e-05 | norm: 0.8253 | dt: 6303.52ms | tok/sec: 83173.9 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2013.1ms | loss: 6.5ms | bwd: 4223.0ms | clip: 0.0ms | optim: 49.4ms
step    19 | loss: 1.588133 | lr 6.0000e-05 | norm: 1.0654 | dt: 6294.13ms | tok/sec: 83297.9 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.6ms | fwd: 2003.5ms | loss: 6.5ms | bwd: 4223.1ms | clip: 0.0ms | optim: 49.6ms
step    20 | loss: 1.558809 | lr 6.3000e-05 | norm: 0.6626 | dt: 6290.90ms | tok/sec: 83340.7 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2008.9ms | loss: 6.5ms | bwd: 4214.7ms | clip: 0.0ms | optim: 49.3ms
step    21 | loss: 1.628220 | lr 6.6000e-05 | norm: 0.8028 | dt: 6291.07ms | tok/sec: 83338.5 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.6ms | fwd: 2009.3ms | loss: 6.5ms | bwd: 4214.1ms | clip: 0.0ms | optim: 49.6ms
step    22 | loss: 1.556918 | lr 6.9000e-05 | norm: 0.8891 | dt: 6333.35ms | tok/sec: 82782.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.6ms | fwd: 2018.0ms | loss: 6.5ms | bwd: 4248.1ms | clip: 0.0ms | optim: 49.3ms
step    23 | loss: 1.555755 | lr 7.2000e-05 | norm: 0.7872 | dt: 6288.59ms | tok/sec: 83371.2 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.6ms | fwd: 2009.1ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.7ms
step    24 | loss: 1.524415 | lr 7.5000e-05 | norm: 1.0502 | dt: 6313.15ms | tok/sec: 83047.0 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4234.2ms | clip: 0.0ms | optim: 49.8ms
step    25 | loss: 1.595368 | lr 7.8000e-05 | norm: 0.8108 | dt: 6304.31ms | tok/sec: 83163.5 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4228.2ms | clip: 0.0ms | optim: 49.5ms
step    26 | loss: 1.504158 | lr 8.1000e-05 | norm: 0.6642 | dt: 6257.16ms | tok/sec: 83790.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.6ms | fwd: 1999.6ms | loss: 6.5ms | bwd: 4190.3ms | clip: 0.0ms | optim: 49.3ms
step    27 | loss: 1.509907 | lr 8.4000e-05 | norm: 0.6278 | dt: 6284.33ms | tok/sec: 83427.9 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2002.1ms | loss: 6.5ms | bwd: 4214.7ms | clip: 0.0ms | optim: 49.5ms
step    28 | loss: 1.541133 | lr 8.7000e-05 | norm: 0.9977 | dt: 6279.35ms | tok/sec: 83493.9 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4207.2ms | clip: 0.0ms | optim: 49.6ms
step    29 | loss: 1.519224 | lr 9.0000e-05 | norm: 0.7567 | dt: 6253.51ms | tok/sec: 83839.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2003.0ms | loss: 6.5ms | bwd: 4183.2ms | clip: 0.0ms | optim: 49.5ms
step    30 | loss: 1.496711 | lr 9.3000e-05 | norm: 0.8490 | dt: 6252.31ms | tok/sec: 83855.1 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 1999.9ms | loss: 6.5ms | bwd: 4184.9ms | clip: 0.0ms | optim: 49.6ms
step    31 | loss: 1.558817 | lr 9.6000e-05 | norm: 0.6784 | dt: 6291.09ms | tok/sec: 83338.2 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2008.3ms | loss: 6.5ms | bwd: 4215.6ms | clip: 0.0ms | optim: 49.3ms
step    32 | loss: 1.469642 | lr 9.9000e-05 | norm: 0.7759 | dt: 6278.42ms | tok/sec: 83506.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.6ms | fwd: 2006.0ms | loss: 6.5ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.4ms
step    33 | loss: 1.532576 | lr 1.0200e-04 | norm: 0.7361 | dt: 6263.29ms | tok/sec: 83708.1 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2009.5ms | loss: 6.5ms | bwd: 4186.4ms | clip: 0.0ms | optim: 49.5ms
step    34 | loss: 1.510420 | lr 1.0500e-04 | norm: 0.7062 | dt: 6269.12ms | tok/sec: 83630.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2005.8ms | loss: 6.5ms | bwd: 4195.9ms | clip: 0.0ms | optim: 49.5ms
step    35 | loss: 1.522740 | lr 1.0800e-04 | norm: 0.8165 | dt: 6266.22ms | tok/sec: 83669.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2001.9ms | loss: 6.5ms | bwd: 4197.1ms | clip: 0.0ms | optim: 49.4ms
step    36 | loss: 1.501231 | lr 1.1100e-04 | norm: 0.8856 | dt: 6285.58ms | tok/sec: 83411.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.6ms | fwd: 2013.1ms | loss: 6.5ms | bwd: 4204.9ms | clip: 0.0ms | optim: 49.5ms
step    37 | loss: 1.500940 | lr 1.1400e-04 | norm: 0.7240 | dt: 6279.76ms | tok/sec: 83488.5 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2004.2ms | loss: 6.5ms | bwd: 4207.9ms | clip: 0.0ms | optim: 49.7ms
step    38 | loss: 1.493368 | lr 1.1700e-04 | norm: 0.6580 | dt: 6269.52ms | tok/sec: 83624.9 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2003.8ms | loss: 6.5ms | bwd: 4198.5ms | clip: 0.0ms | optim: 49.4ms
step    39 | loss: 1.517024 | lr 1.2000e-04 | norm: 0.8705 | dt: 6260.41ms | tok/sec: 83746.6 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.6ms | fwd: 2000.3ms | loss: 6.5ms | bwd: 4192.4ms | clip: 0.0ms | optim: 49.6ms
step    40 | loss: 1.477205 | lr 1.2300e-04 | norm: 0.7975 | dt: 6279.08ms | tok/sec: 83497.6 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2005.8ms | loss: 6.5ms | bwd: 4206.1ms | clip: 0.0ms | optim: 49.3ms
step    41 | loss: 1.453860 | lr 1.2600e-04 | norm: 0.6779 | dt: 6253.72ms | tok/sec: 83836.2 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.6ms | fwd: 2001.7ms | loss: 6.5ms | bwd: 4184.4ms | clip: 0.0ms | optim: 49.6ms
step    42 | loss: 1.503194 | lr 1.2900e-04 | norm: 0.7840 | dt: 6270.52ms | tok/sec: 83611.6 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 1997.0ms | loss: 6.5ms | bwd: 4206.2ms | clip: 0.0ms | optim: 49.5ms
step    43 | loss: 1.505770 | lr 1.3200e-04 | norm: 0.7554 | dt: 6273.43ms | tok/sec: 83572.8 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.4ms | fwd: 2010.1ms | loss: 6.5ms | bwd: 4196.2ms | clip: 0.0ms | optim: 49.3ms
step    44 | loss: 1.424579 | lr 1.3500e-04 | norm: 0.7888 | dt: 6275.02ms | tok/sec: 83551.6 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.6ms | fwd: 2008.4ms | loss: 6.5ms | bwd: 4199.2ms | clip: 0.0ms | optim: 49.5ms
step    45 | loss: 1.505173 | lr 1.3800e-04 | norm: 0.7303 | dt: 6279.95ms | tok/sec: 83486.0 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2003.7ms | loss: 6.5ms | bwd: 4208.6ms | clip: 0.0ms | optim: 49.8ms
step    46 | loss: 1.492393 | lr 1.4100e-04 | norm: 0.6834 | dt: 6280.71ms | tok/sec: 83475.9 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.6ms | fwd: 2002.3ms | loss: 6.5ms | bwd: 4211.1ms | clip: 0.0ms | optim: 49.2ms
step    47 | loss: 1.516353 | lr 1.4400e-04 | norm: 0.6402 | dt: 6274.28ms | tok/sec: 83561.5 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.4ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4200.9ms | clip: 0.0ms | optim: 49.5ms
step    48 | loss: 1.510875 | lr 1.4700e-04 | norm: 0.8292 | dt: 6276.30ms | tok/sec: 83534.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2010.4ms | loss: 6.5ms | bwd: 4198.7ms | clip: 0.0ms | optim: 49.3ms
step    49 | loss: 1.487243 | lr 1.5000e-04 | norm: 0.6854 | dt: 6280.43ms | tok/sec: 83479.7 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2000.6ms | loss: 6.5ms | bwd: 4212.5ms | clip: 0.0ms | optim: 49.5ms
step    50 | loss: 1.520638 | lr 1.5300e-04 | norm: 0.6852 | dt: 6283.28ms | tok/sec: 83441.8 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 1999.8ms | loss: 6.5ms | bwd: 4216.0ms | clip: 0.0ms | optim: 49.6ms
step    51 | loss: 1.539354 | lr 1.5600e-04 | norm: 0.7460 | dt: 6263.29ms | tok/sec: 83708.1 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.5ms | bwd: 4185.9ms | clip: 0.0ms | optim: 49.5ms
step    52 | loss: 1.548643 | lr 1.5900e-04 | norm: 0.6508 | dt: 6267.94ms | tok/sec: 83646.0 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2001.8ms | loss: 6.5ms | bwd: 4198.8ms | clip: 0.0ms | optim: 49.4ms
step    53 | loss: 1.472013 | lr 1.6200e-04 | norm: 0.6998 | dt: 6289.78ms | tok/sec: 83355.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.6ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4213.5ms | clip: 0.0ms | optim: 49.3ms
step    54 | loss: 1.481758 | lr 1.6500e-04 | norm: 0.6636 | dt: 6274.13ms | tok/sec: 83563.5 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2006.2ms | loss: 6.5ms | bwd: 4200.7ms | clip: 0.0ms | optim: 49.3ms
step    55 | loss: 1.560293 | lr 1.6800e-04 | norm: 0.6843 | dt: 6262.88ms | tok/sec: 83713.5 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.6ms | fwd: 2002.4ms | loss: 6.5ms | bwd: 4193.2ms | clip: 0.0ms | optim: 49.4ms
step    56 | loss: 1.525007 | lr 1.7100e-04 | norm: 0.6459 | dt: 6276.39ms | tok/sec: 83533.3 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.4ms | fwd: 2005.5ms | loss: 6.5ms | bwd: 4202.2ms | clip: 0.0ms | optim: 50.8ms
step    57 | loss: 1.458180 | lr 1.7400e-04 | norm: 0.6643 | dt: 6265.95ms | tok/sec: 83672.6 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2004.3ms | loss: 6.5ms | bwd: 4194.4ms | clip: 0.0ms | optim: 49.4ms
step    58 | loss: 1.471069 | lr 1.7700e-04 | norm: 0.6362 | dt: 6263.44ms | tok/sec: 83706.1 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.4ms | fwd: 1999.5ms | loss: 6.5ms | bwd: 4196.7ms | clip: 0.0ms | optim: 49.5ms
step    59 | loss: 1.478795 | lr 1.8000e-04 | norm: 0.8623 | dt: 6280.62ms | tok/sec: 83477.2 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.6ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4207.1ms | clip: 0.0ms | optim: 49.7ms
step    60 | loss: 1.518571 | lr 1.8300e-04 | norm: 0.7173 | dt: 6277.75ms | tok/sec: 83515.3 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2009.1ms | loss: 6.5ms | bwd: 4201.2ms | clip: 0.0ms | optim: 49.5ms
step    61 | loss: 1.454732 | lr 1.8600e-04 | norm: 0.6899 | dt: 6256.81ms | tok/sec: 83794.8 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 1997.2ms | loss: 6.5ms | bwd: 4191.2ms | clip: 0.0ms | optim: 50.4ms
step    62 | loss: 1.485695 | lr 1.8900e-04 | norm: 0.8743 | dt: 6291.04ms | tok/sec: 83338.8 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2002.9ms | loss: 6.5ms | bwd: 4220.7ms | clip: 0.0ms | optim: 49.6ms
step    63 | loss: 1.483249 | lr 1.9200e-04 | norm: 0.9320 | dt: 6277.18ms | tok/sec: 83522.9 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.7ms | fwd: 2002.2ms | loss: 6.5ms | bwd: 4207.2ms | clip: 0.0ms | optim: 49.6ms
step    64 | loss: 1.450130 | lr 1.9500e-04 | norm: 0.7944 | dt: 6272.31ms | tok/sec: 83587.7 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2002.8ms | loss: 6.5ms | bwd: 4202.0ms | clip: 0.0ms | optim: 49.5ms
step    65 | loss: 1.497326 | lr 1.9800e-04 | norm: 0.7033 | dt: 6268.71ms | tok/sec: 83635.8 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2004.3ms | loss: 6.5ms | bwd: 4197.2ms | clip: 0.0ms | optim: 49.3ms
step    66 | loss: 1.479846 | lr 2.0100e-04 | norm: 0.8269 | dt: 6266.54ms | tok/sec: 83664.6 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.4ms | fwd: 2005.4ms | loss: 6.5ms | bwd: 4193.7ms | clip: 0.0ms | optim: 49.6ms
step    67 | loss: 1.520247 | lr 2.0400e-04 | norm: 0.8505 | dt: 6260.38ms | tok/sec: 83747.0 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4188.0ms | clip: 0.0ms | optim: 49.8ms
step    68 | loss: 1.492321 | lr 2.0700e-04 | norm: 0.6921 | dt: 6280.83ms | tok/sec: 83474.4 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2003.5ms | loss: 6.6ms | bwd: 4209.9ms | clip: 0.0ms | optim: 49.4ms
step    69 | loss: 1.530074 | lr 2.1000e-04 | norm: 0.9078 | dt: 6279.46ms | tok/sec: 83492.6 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.6ms | fwd: 2005.1ms | loss: 6.5ms | bwd: 4207.1ms | clip: 0.0ms | optim: 49.3ms
step    70 | loss: 1.496907 | lr 2.1300e-04 | norm: 0.9080 | dt: 6288.30ms | tok/sec: 83375.1 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.4ms | fwd: 2015.9ms | loss: 6.5ms | bwd: 4205.2ms | clip: 0.0ms | optim: 49.3ms
step    71 | loss: 1.501970 | lr 2.1600e-04 | norm: 0.8776 | dt: 6263.60ms | tok/sec: 83704.0 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2001.9ms | loss: 6.5ms | bwd: 4194.3ms | clip: 0.0ms | optim: 49.5ms
step    72 | loss: 1.431277 | lr 2.1900e-04 | norm: 0.6606 | dt: 6260.91ms | tok/sec: 83740.0 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.6ms | fwd: 2002.4ms | loss: 6.5ms | bwd: 4191.4ms | clip: 0.0ms | optim: 49.3ms
step    73 | loss: 1.506157 | lr 2.2200e-04 | norm: 0.7917 | dt: 6293.23ms | tok/sec: 83309.8 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.6ms | fwd: 2006.9ms | loss: 6.5ms | bwd: 4219.0ms | clip: 0.0ms | optim: 49.4ms
step    74 | loss: 1.487407 | lr 2.2500e-04 | norm: 0.9247 | dt: 6273.62ms | tok/sec: 83570.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2004.4ms | loss: 6.5ms | bwd: 4201.8ms | clip: 0.0ms | optim: 49.5ms
step    75 | loss: 1.501011 | lr 2.2800e-04 | norm: 0.6996 | dt: 6272.74ms | tok/sec: 83581.9 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2005.8ms | loss: 6.5ms | bwd: 4199.2ms | clip: 0.0ms | optim: 49.7ms
step    76 | loss: 1.456014 | lr 2.3100e-04 | norm: 0.7896 | dt: 6262.88ms | tok/sec: 83713.6 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2006.0ms | loss: 6.5ms | bwd: 4189.3ms | clip: 0.0ms | optim: 49.7ms
step    77 | loss: 1.513480 | lr 2.3400e-04 | norm: 0.6681 | dt: 6261.25ms | tok/sec: 83735.3 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2000.9ms | loss: 6.5ms | bwd: 4192.9ms | clip: 0.0ms | optim: 49.5ms
step    78 | loss: 1.486683 | lr 2.3700e-04 | norm: 0.7710 | dt: 6297.81ms | tok/sec: 83249.3 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2016.3ms | loss: 6.5ms | bwd: 4214.3ms | clip: 0.0ms | optim: 49.3ms
step    79 | loss: 1.497332 | lr 2.4000e-04 | norm: 0.8699 | dt: 6303.45ms | tok/sec: 83174.7 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2006.0ms | loss: 6.5ms | bwd: 4230.3ms | clip: 0.0ms | optim: 49.3ms
step    80 | loss: 1.499554 | lr 2.4300e-04 | norm: 0.7233 | dt: 6270.78ms | tok/sec: 83608.1 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 1997.2ms | loss: 6.5ms | bwd: 4206.2ms | clip: 0.0ms | optim: 49.4ms
step    81 | loss: 1.546454 | lr 2.4600e-04 | norm: 1.0824 | dt: 6287.99ms | tok/sec: 83379.3 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2008.6ms | loss: 6.5ms | bwd: 4211.9ms | clip: 0.0ms | optim: 49.6ms
step    82 | loss: 1.566263 | lr 2.4900e-04 | norm: 0.8050 | dt: 6270.27ms | tok/sec: 83614.9 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2005.6ms | loss: 6.5ms | bwd: 4197.3ms | clip: 0.0ms | optim: 49.3ms
step    83 | loss: 1.486411 | lr 2.5200e-04 | norm: 0.8218 | dt: 6279.80ms | tok/sec: 83488.1 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2007.2ms | loss: 6.5ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.5ms
step    84 | loss: 1.505806 | lr 2.5500e-04 | norm: 0.7429 | dt: 6279.81ms | tok/sec: 83487.8 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4203.7ms | clip: 0.0ms | optim: 49.9ms
step    85 | loss: 1.507277 | lr 2.5800e-04 | norm: 0.7817 | dt: 6275.33ms | tok/sec: 83547.5 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.6ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4200.0ms | clip: 0.0ms | optim: 49.9ms
step    86 | loss: 1.505058 | lr 2.6100e-04 | norm: 0.8321 | dt: 6282.79ms | tok/sec: 83448.3 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2007.0ms | loss: 6.5ms | bwd: 4208.5ms | clip: 0.0ms | optim: 49.2ms
step    87 | loss: 1.568943 | lr 2.6400e-04 | norm: 0.8474 | dt: 6269.21ms | tok/sec: 83629.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.7ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4193.5ms | clip: 0.0ms | optim: 49.4ms
step    88 | loss: 1.577075 | lr 2.6700e-04 | norm: 0.7998 | dt: 6286.96ms | tok/sec: 83393.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.6ms | fwd: 2006.5ms | loss: 6.5ms | bwd: 4212.9ms | clip: 0.0ms | optim: 49.5ms
step    89 | loss: 1.517823 | lr 2.7000e-04 | norm: 0.8074 | dt: 6287.60ms | tok/sec: 83384.4 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2004.8ms | loss: 6.5ms | bwd: 4215.6ms | clip: 0.0ms | optim: 49.3ms
step    90 | loss: 1.587696 | lr 2.7300e-04 | norm: 0.8428 | dt: 6289.39ms | tok/sec: 83360.7 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.4ms | fwd: 2010.2ms | loss: 6.5ms | bwd: 4211.3ms | clip: 0.0ms | optim: 50.1ms
step    91 | loss: 1.536380 | lr 2.7600e-04 | norm: 0.9942 | dt: 6262.55ms | tok/sec: 83718.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2002.6ms | loss: 6.5ms | bwd: 4192.9ms | clip: 0.0ms | optim: 49.2ms
step    92 | loss: 1.542903 | lr 2.7900e-04 | norm: 0.8280 | dt: 6268.91ms | tok/sec: 83633.1 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4196.8ms | clip: 0.0ms | optim: 49.6ms
step    93 | loss: 1.563778 | lr 2.8200e-04 | norm: 0.6616 | dt: 6278.16ms | tok/sec: 83509.8 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4203.4ms | clip: 0.0ms | optim: 49.4ms
step    94 | loss: 1.446115 | lr 2.8500e-04 | norm: 0.7590 | dt: 6269.28ms | tok/sec: 83628.1 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4191.6ms | clip: 0.0ms | optim: 49.8ms
step    95 | loss: 1.526014 | lr 2.8800e-04 | norm: 0.8079 | dt: 6302.90ms | tok/sec: 83182.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2008.5ms | loss: 6.5ms | bwd: 4226.6ms | clip: 0.0ms | optim: 49.9ms
step    96 | loss: 1.525283 | lr 2.9100e-04 | norm: 1.1439 | dt: 6269.66ms | tok/sec: 83623.1 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.4ms | fwd: 2008.1ms | loss: 6.5ms | bwd: 4194.1ms | clip: 0.0ms | optim: 49.6ms
step    97 | loss: 1.521146 | lr 2.9400e-04 | norm: 0.7440 | dt: 6303.01ms | tok/sec: 83180.6 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4221.3ms | clip: 0.0ms | optim: 49.4ms
step    98 | loss: 1.513996 | lr 2.9700e-04 | norm: 0.6855 | dt: 6258.14ms | tok/sec: 83777.0 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.6ms | fwd: 2010.6ms | loss: 6.5ms | bwd: 4179.8ms | clip: 0.0ms | optim: 49.7ms
step    99 | loss: 1.539086 | lr 3.0000e-04 | norm: 0.7669 | dt: 6277.71ms | tok/sec: 83515.8 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.6ms | fwd: 2012.4ms | loss: 6.5ms | bwd: 4197.9ms | clip: 0.0ms | optim: 49.4ms
step   100 | loss: 1.558138 | lr 3.0000e-04 | norm: 0.8741 | dt: 6293.76ms | tok/sec: 83302.8 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2006.0ms | loss: 6.5ms | bwd: 4220.6ms | clip: 0.0ms | optim: 49.3ms
step   101 | loss: 1.588794 | lr 2.9999e-04 | norm: 0.6750 | dt: 6284.74ms | tok/sec: 83422.4 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2004.8ms | loss: 6.5ms | bwd: 4212.7ms | clip: 0.0ms | optim: 49.4ms
step   102 | loss: 1.497377 | lr 2.9998e-04 | norm: 0.7302 | dt: 6278.94ms | tok/sec: 83499.5 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2004.2ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.3ms
step   103 | loss: 1.553991 | lr 2.9995e-04 | norm: 0.8335 | dt: 6280.93ms | tok/sec: 83473.0 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2002.0ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.2ms
step   104 | loss: 1.620800 | lr 2.9991e-04 | norm: 0.7650 | dt: 6292.99ms | tok/sec: 83313.0 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.6ms | fwd: 2008.6ms | loss: 6.5ms | bwd: 4217.0ms | clip: 0.0ms | optim: 49.4ms
step   105 | loss: 1.534936 | lr 2.9986e-04 | norm: 0.8890 | dt: 6283.20ms | tok/sec: 83442.8 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.6ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4209.0ms | clip: 0.0ms | optim: 50.1ms
step   106 | loss: 1.539826 | lr 2.9980e-04 | norm: 0.8278 | dt: 6287.00ms | tok/sec: 83392.4 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.5ms | fwd: 2013.0ms | loss: 6.5ms | bwd: 4206.5ms | clip: 0.0ms | optim: 49.5ms
step   107 | loss: 1.604628 | lr 2.9973e-04 | norm: 0.8909 | dt: 6298.66ms | tok/sec: 83238.0 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.6ms | fwd: 2012.8ms | loss: 6.5ms | bwd: 4217.6ms | clip: 0.0ms | optim: 50.3ms
step   108 | loss: 1.497698 | lr 2.9965e-04 | norm: 0.8882 | dt: 6303.84ms | tok/sec: 83169.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.5ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4223.7ms | clip: 0.0ms | optim: 49.6ms
step   109 | loss: 1.539937 | lr 2.9956e-04 | norm: 0.7859 | dt: 6280.29ms | tok/sec: 83481.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.4ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4199.3ms | clip: 0.0ms | optim: 49.5ms
step   110 | loss: 1.561216 | lr 2.9945e-04 | norm: 0.7698 | dt: 6266.68ms | tok/sec: 83662.7 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.6ms | fwd: 2004.1ms | loss: 6.5ms | bwd: 4195.0ms | clip: 0.0ms | optim: 49.6ms
step   111 | loss: 1.544489 | lr 2.9934e-04 | norm: 0.7564 | dt: 6294.91ms | tok/sec: 83287.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.5ms | fwd: 2003.9ms | loss: 6.5ms | bwd: 4223.1ms | clip: 0.0ms | optim: 49.9ms
step   112 | loss: 1.517063 | lr 2.9921e-04 | norm: 0.8039 | dt: 6274.91ms | tok/sec: 83553.1 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.4ms | fwd: 2011.5ms | loss: 6.5ms | bwd: 4195.9ms | clip: 0.0ms | optim: 49.6ms
step   113 | loss: 1.503844 | lr 2.9907e-04 | norm: 0.8222 | dt: 6261.25ms | tok/sec: 83735.3 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2004.3ms | loss: 6.5ms | bwd: 4189.6ms | clip: 0.0ms | optim: 49.4ms
step   114 | loss: 1.537449 | lr 2.9893e-04 | norm: 0.6629 | dt: 6273.41ms | tok/sec: 83573.1 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.7ms | fwd: 2010.3ms | loss: 6.5ms | bwd: 4194.8ms | clip: 0.0ms | optim: 50.3ms
step   115 | loss: 1.525515 | lr 2.9877e-04 | norm: 0.9001 | dt: 6288.55ms | tok/sec: 83371.8 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.6ms | fwd: 2008.3ms | loss: 6.5ms | bwd: 4212.6ms | clip: 0.0ms | optim: 49.7ms
step   116 | loss: 1.475275 | lr 2.9860e-04 | norm: 0.7550 | dt: 6270.72ms | tok/sec: 83608.9 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.6ms | fwd: 2003.2ms | loss: 6.5ms | bwd: 4200.1ms | clip: 0.0ms | optim: 49.4ms
step   117 | loss: 1.488417 | lr 2.9842e-04 | norm: 0.6370 | dt: 6278.87ms | tok/sec: 83500.4 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4205.5ms | clip: 0.0ms | optim: 49.3ms
step   118 | loss: 1.550351 | lr 2.9823e-04 | norm: 0.6556 | dt: 6278.16ms | tok/sec: 83509.8 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.6ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4201.8ms | clip: 0.0ms | optim: 49.7ms
step   119 | loss: 1.562920 | lr 2.9803e-04 | norm: 0.7118 | dt: 6283.08ms | tok/sec: 83444.4 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 1998.6ms | loss: 6.5ms | bwd: 4217.2ms | clip: 0.0ms | optim: 49.4ms
step   120 | loss: 1.506888 | lr 2.9781e-04 | norm: 0.8285 | dt: 6268.99ms | tok/sec: 83631.9 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2009.7ms | loss: 6.5ms | bwd: 4191.9ms | clip: 0.0ms | optim: 49.4ms
step   121 | loss: 1.530550 | lr 2.9759e-04 | norm: 0.6873 | dt: 6280.70ms | tok/sec: 83476.0 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4204.9ms | clip: 0.0ms | optim: 50.2ms
step   122 | loss: 1.582837 | lr 2.9736e-04 | norm: 0.7315 | dt: 6277.76ms | tok/sec: 83515.2 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4202.7ms | clip: 0.0ms | optim: 49.3ms
step   123 | loss: 1.547503 | lr 2.9711e-04 | norm: 0.6378 | dt: 6275.89ms | tok/sec: 83540.1 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.6ms | fwd: 2010.2ms | loss: 6.5ms | bwd: 4198.4ms | clip: 0.0ms | optim: 49.3ms
step   124 | loss: 1.572326 | lr 2.9686e-04 | norm: 0.6200 | dt: 6284.64ms | tok/sec: 83423.8 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.6ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4207.5ms | clip: 0.0ms | optim: 49.4ms
step   125 | loss: 1.528828 | lr 2.9659e-04 | norm: 0.6833 | dt: 6272.18ms | tok/sec: 83589.5 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4196.7ms | clip: 0.0ms | optim: 49.4ms
step   126 | loss: 1.546839 | lr 2.9632e-04 | norm: 0.6375 | dt: 6298.71ms | tok/sec: 83237.3 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4219.1ms | clip: 0.0ms | optim: 49.6ms
step   127 | loss: 1.585903 | lr 2.9603e-04 | norm: 0.6423 | dt: 6293.30ms | tok/sec: 83309.0 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.5ms | bwd: 4215.5ms | clip: 0.0ms | optim: 49.9ms
step   128 | loss: 1.603162 | lr 2.9573e-04 | norm: 0.6889 | dt: 6291.44ms | tok/sec: 83333.5 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4215.3ms | clip: 0.0ms | optim: 49.5ms
step   129 | loss: 1.555265 | lr 2.9542e-04 | norm: 0.5495 | dt: 6286.65ms | tok/sec: 83397.1 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.6ms | fwd: 2001.6ms | loss: 6.5ms | bwd: 4217.5ms | clip: 0.0ms | optim: 49.6ms
step   130 | loss: 1.673475 | lr 2.9511e-04 | norm: 0.7040 | dt: 6286.41ms | tok/sec: 83400.2 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2008.5ms | loss: 6.5ms | bwd: 4210.7ms | clip: 0.0ms | optim: 49.4ms
step   131 | loss: 1.557077 | lr 2.9478e-04 | norm: 0.6764 | dt: 6271.73ms | tok/sec: 83595.5 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2001.4ms | loss: 6.5ms | bwd: 4203.0ms | clip: 0.0ms | optim: 49.4ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_train_00001.bin
100000000
step   132 | loss: 1.558308 | lr 2.9444e-04 | norm: 0.6910 | dt: 6299.83ms | tok/sec: 83222.6 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 14.3ms | fwd: 2002.9ms | loss: 6.5ms | bwd: 4218.7ms | clip: 0.0ms | optim: 49.5ms
step   133 | loss: 1.542757 | lr 2.9409e-04 | norm: 0.5868 | dt: 6280.89ms | tok/sec: 83473.5 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2002.4ms | loss: 6.5ms | bwd: 4211.3ms | clip: 0.0ms | optim: 49.3ms
step   134 | loss: 1.552294 | lr 2.9373e-04 | norm: 0.6477 | dt: 6293.43ms | tok/sec: 83307.2 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.6ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4211.5ms | clip: 0.0ms | optim: 49.4ms
step   135 | loss: 1.574899 | lr 2.9336e-04 | norm: 0.6674 | dt: 6290.55ms | tok/sec: 83345.4 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2002.8ms | loss: 6.5ms | bwd: 4220.3ms | clip: 0.0ms | optim: 49.6ms
step   136 | loss: 1.528312 | lr 2.9298e-04 | norm: 0.5462 | dt: 6290.05ms | tok/sec: 83352.0 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2015.6ms | loss: 6.5ms | bwd: 4207.3ms | clip: 0.0ms | optim: 49.2ms
step   137 | loss: 1.520423 | lr 2.9259e-04 | norm: 0.5560 | dt: 6288.70ms | tok/sec: 83369.9 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.6ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4209.0ms | clip: 0.0ms | optim: 49.7ms
step   138 | loss: 1.582985 | lr 2.9219e-04 | norm: 0.6027 | dt: 6266.17ms | tok/sec: 83669.6 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.4ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4187.9ms | clip: 0.0ms | optim: 49.4ms
step   139 | loss: 1.579060 | lr 2.9178e-04 | norm: 0.5679 | dt: 6311.05ms | tok/sec: 83074.6 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2010.7ms | loss: 6.5ms | bwd: 4233.1ms | clip: 0.0ms | optim: 49.3ms
step   140 | loss: 1.593377 | lr 2.9135e-04 | norm: 0.5493 | dt: 6276.05ms | tok/sec: 83537.9 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4201.1ms | clip: 0.0ms | optim: 49.5ms
step   141 | loss: 1.584874 | lr 2.9092e-04 | norm: 0.6367 | dt: 6276.47ms | tok/sec: 83532.3 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4201.3ms | clip: 0.0ms | optim: 49.4ms
step   142 | loss: 1.570228 | lr 2.9048e-04 | norm: 0.6398 | dt: 6287.73ms | tok/sec: 83382.8 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.6ms | fwd: 2014.1ms | loss: 6.5ms | bwd: 4205.5ms | clip: 0.0ms | optim: 50.2ms
step   143 | loss: 1.515446 | lr 2.9003e-04 | norm: 0.5378 | dt: 6284.41ms | tok/sec: 83426.7 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2001.7ms | loss: 6.5ms | bwd: 4215.2ms | clip: 0.0ms | optim: 49.6ms
step   144 | loss: 1.588587 | lr 2.8957e-04 | norm: 0.6180 | dt: 6278.99ms | tok/sec: 83498.7 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2006.9ms | loss: 6.5ms | bwd: 4204.5ms | clip: 0.0ms | optim: 49.7ms
step   145 | loss: 1.501689 | lr 2.8910e-04 | norm: 0.5400 | dt: 6285.86ms | tok/sec: 83407.6 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.4ms | fwd: 2015.6ms | loss: 6.5ms | bwd: 4203.1ms | clip: 0.0ms | optim: 49.3ms
step   146 | loss: 1.516128 | lr 2.8862e-04 | norm: 0.5324 | dt: 6285.37ms | tok/sec: 83414.0 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2008.0ms | loss: 6.5ms | bwd: 4210.0ms | clip: 0.0ms | optim: 49.6ms
step   147 | loss: 1.516567 | lr 2.8813e-04 | norm: 0.6004 | dt: 6287.88ms | tok/sec: 83380.8 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.6ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4211.5ms | clip: 0.0ms | optim: 49.5ms
step   148 | loss: 1.558017 | lr 2.8763e-04 | norm: 0.5077 | dt: 6282.81ms | tok/sec: 83448.1 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.6ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.7ms
step   149 | loss: 1.559496 | lr 2.8712e-04 | norm: 0.5494 | dt: 6283.98ms | tok/sec: 83432.5 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2007.0ms | loss: 6.5ms | bwd: 4209.7ms | clip: 0.0ms | optim: 49.4ms
step   150 | loss: 1.511979 | lr 2.8660e-04 | norm: 0.5596 | dt: 6309.52ms | tok/sec: 83094.7 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2013.8ms | loss: 6.5ms | bwd: 4228.2ms | clip: 0.0ms | optim: 49.6ms
step   151 | loss: 1.533700 | lr 2.8607e-04 | norm: 0.4955 | dt: 6262.72ms | tok/sec: 83715.8 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.5ms | fwd: 2002.1ms | loss: 6.5ms | bwd: 4193.0ms | clip: 0.0ms | optim: 49.7ms
step   152 | loss: 1.578448 | lr 2.8554e-04 | norm: 0.5673 | dt: 6309.20ms | tok/sec: 83099.0 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.6ms | fwd: 2014.9ms | loss: 6.5ms | bwd: 4227.0ms | clip: 0.0ms | optim: 49.3ms
step   153 | loss: 1.534691 | lr 2.8499e-04 | norm: 0.5458 | dt: 6277.60ms | tok/sec: 83517.3 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.5ms | fwd: 2003.5ms | loss: 6.5ms | bwd: 4206.7ms | clip: 0.0ms | optim: 49.4ms
step   154 | loss: 1.563362 | lr 2.8443e-04 | norm: 0.5577 | dt: 6266.56ms | tok/sec: 83664.4 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.4ms | fwd: 2005.2ms | loss: 6.5ms | bwd: 4194.3ms | clip: 0.0ms | optim: 49.3ms
step   155 | loss: 1.539736 | lr 2.8387e-04 | norm: 0.5183 | dt: 6302.21ms | tok/sec: 83191.1 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.4ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4227.5ms | clip: 0.0ms | optim: 49.4ms
step   156 | loss: 1.534730 | lr 2.8329e-04 | norm: 0.4992 | dt: 6296.68ms | tok/sec: 83264.2 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.6ms | fwd: 2006.4ms | loss: 6.5ms | bwd: 4222.7ms | clip: 0.0ms | optim: 49.5ms
step   157 | loss: 1.590882 | lr 2.8271e-04 | norm: 0.4973 | dt: 6264.75ms | tok/sec: 83688.5 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.5ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4189.4ms | clip: 0.0ms | optim: 50.0ms
step   158 | loss: 1.599446 | lr 2.8211e-04 | norm: 0.5939 | dt: 6294.39ms | tok/sec: 83294.5 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.5ms | fwd: 2022.2ms | loss: 6.6ms | bwd: 4204.3ms | clip: 0.0ms | optim: 49.8ms
step   159 | loss: 1.549838 | lr 2.8151e-04 | norm: 0.4632 | dt: 6280.63ms | tok/sec: 83477.0 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.5ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4208.6ms | clip: 0.0ms | optim: 49.4ms
step   160 | loss: 1.585282 | lr 2.8090e-04 | norm: 0.4257 | dt: 6297.83ms | tok/sec: 83249.0 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.4ms | fwd: 2014.3ms | loss: 6.5ms | bwd: 4216.3ms | clip: 0.0ms | optim: 49.4ms
step   161 | loss: 1.553990 | lr 2.8028e-04 | norm: 0.4394 | dt: 6294.94ms | tok/sec: 83287.2 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.6ms | fwd: 2020.5ms | loss: 6.5ms | bwd: 4206.9ms | clip: 0.0ms | optim: 49.6ms
step   162 | loss: 1.547580 | lr 2.7965e-04 | norm: 0.5545 | dt: 6290.15ms | tok/sec: 83350.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2008.0ms | loss: 6.5ms | bwd: 4214.8ms | clip: 0.0ms | optim: 49.4ms
step   163 | loss: 1.559647 | lr 2.7902e-04 | norm: 0.4986 | dt: 6285.78ms | tok/sec: 83408.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2006.3ms | loss: 6.5ms | bwd: 4211.9ms | clip: 0.0ms | optim: 49.7ms
step   164 | loss: 1.491688 | lr 2.7837e-04 | norm: 0.4308 | dt: 6298.78ms | tok/sec: 83236.4 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2009.7ms | loss: 6.5ms | bwd: 4221.3ms | clip: 0.0ms | optim: 49.8ms
step   165 | loss: 1.546488 | lr 2.7771e-04 | norm: 0.5418 | dt: 6276.68ms | tok/sec: 83529.5 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2008.0ms | loss: 6.5ms | bwd: 4201.1ms | clip: 0.0ms | optim: 49.6ms
step   166 | loss: 1.547639 | lr 2.7705e-04 | norm: 0.5029 | dt: 6287.77ms | tok/sec: 83382.2 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2008.6ms | loss: 6.5ms | bwd: 4212.0ms | clip: 0.0ms | optim: 49.3ms
step   167 | loss: 1.563413 | lr 2.7638e-04 | norm: 0.4673 | dt: 6274.27ms | tok/sec: 83561.7 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.4ms | fwd: 2005.7ms | loss: 6.6ms | bwd: 4201.1ms | clip: 0.0ms | optim: 49.5ms
step   168 | loss: 1.593522 | lr 2.7570e-04 | norm: 0.6057 | dt: 6311.52ms | tok/sec: 83068.4 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4232.3ms | clip: 0.0ms | optim: 49.4ms
step   169 | loss: 1.512869 | lr 2.7501e-04 | norm: 0.5627 | dt: 6275.07ms | tok/sec: 83551.0 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2011.4ms | loss: 6.5ms | bwd: 4196.3ms | clip: 0.0ms | optim: 49.5ms
step   170 | loss: 1.522474 | lr 2.7431e-04 | norm: 0.4098 | dt: 6283.23ms | tok/sec: 83442.4 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.6ms | fwd: 2007.8ms | loss: 6.5ms | bwd: 4207.8ms | clip: 0.0ms | optim: 49.6ms
step   171 | loss: 1.585019 | lr 2.7361e-04 | norm: 0.4453 | dt: 6285.86ms | tok/sec: 83407.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.6ms | fwd: 2006.4ms | loss: 6.7ms | bwd: 4211.6ms | clip: 0.0ms | optim: 49.6ms
step   172 | loss: 1.547580 | lr 2.7290e-04 | norm: 0.4785 | dt: 6288.72ms | tok/sec: 83369.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2014.4ms | loss: 6.5ms | bwd: 4207.1ms | clip: 0.0ms | optim: 49.3ms
step   173 | loss: 1.535500 | lr 2.7218e-04 | norm: 0.5120 | dt: 6286.74ms | tok/sec: 83395.8 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2006.3ms | loss: 6.5ms | bwd: 4213.1ms | clip: 0.0ms | optim: 49.3ms
step   174 | loss: 1.562565 | lr 2.7145e-04 | norm: 0.8501 | dt: 6281.23ms | tok/sec: 83469.1 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.4ms | fwd: 2002.4ms | loss: 6.5ms | bwd: 4211.6ms | clip: 0.0ms | optim: 49.4ms
step   175 | loss: 1.544128 | lr 2.7071e-04 | norm: 0.4173 | dt: 6273.13ms | tok/sec: 83576.8 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2001.1ms | loss: 6.5ms | bwd: 4204.6ms | clip: 0.0ms | optim: 49.5ms
step   176 | loss: 1.509556 | lr 2.6997e-04 | norm: 0.4160 | dt: 6291.41ms | tok/sec: 83334.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2018.1ms | loss: 6.5ms | bwd: 4206.1ms | clip: 0.0ms | optim: 49.4ms
step   177 | loss: 1.593228 | lr 2.6921e-04 | norm: 0.4496 | dt: 6286.91ms | tok/sec: 83393.6 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4213.3ms | clip: 0.0ms | optim: 49.5ms
step   178 | loss: 1.562044 | lr 2.6845e-04 | norm: 0.4398 | dt: 6285.34ms | tok/sec: 83414.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.6ms | fwd: 2006.0ms | loss: 6.7ms | bwd: 4211.6ms | clip: 0.0ms | optim: 49.5ms
step   179 | loss: 1.564880 | lr 2.6769e-04 | norm: 0.4048 | dt: 6275.06ms | tok/sec: 83551.1 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2008.5ms | loss: 6.5ms | bwd: 4199.2ms | clip: 0.0ms | optim: 49.4ms
step   180 | loss: 1.606080 | lr 2.6691e-04 | norm: 0.4190 | dt: 6287.55ms | tok/sec: 83385.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2011.5ms | loss: 6.5ms | bwd: 4208.6ms | clip: 0.0ms | optim: 49.5ms
step   181 | loss: 1.515862 | lr 2.6613e-04 | norm: 0.3759 | dt: 6294.38ms | tok/sec: 83294.6 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2003.7ms | loss: 6.5ms | bwd: 4223.1ms | clip: 0.0ms | optim: 49.6ms
step   182 | loss: 1.515317 | lr 2.6534e-04 | norm: 0.4211 | dt: 6274.37ms | tok/sec: 83560.3 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.4ms | fwd: 2004.5ms | loss: 6.5ms | bwd: 4202.6ms | clip: 0.0ms | optim: 49.5ms
step   183 | loss: 1.542364 | lr 2.6455e-04 | norm: 0.4087 | dt: 6270.43ms | tok/sec: 83612.7 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2005.3ms | loss: 6.5ms | bwd: 4197.8ms | clip: 0.0ms | optim: 49.3ms
step   184 | loss: 1.544828 | lr 2.6374e-04 | norm: 0.4139 | dt: 6301.13ms | tok/sec: 83205.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2012.8ms | loss: 6.5ms | bwd: 4221.2ms | clip: 0.0ms | optim: 49.3ms
step   185 | loss: 1.513478 | lr 2.6293e-04 | norm: 0.3655 | dt: 6294.86ms | tok/sec: 83288.3 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2013.5ms | loss: 6.5ms | bwd: 4214.1ms | clip: 0.0ms | optim: 49.3ms
step   186 | loss: 1.582303 | lr 2.6211e-04 | norm: 0.3964 | dt: 6265.31ms | tok/sec: 83681.0 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4191.8ms | clip: 0.0ms | optim: 49.6ms
step   187 | loss: 1.554201 | lr 2.6129e-04 | norm: 0.4071 | dt: 6281.91ms | tok/sec: 83460.0 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.6ms | fwd: 2003.6ms | loss: 6.5ms | bwd: 4210.7ms | clip: 0.0ms | optim: 49.6ms
step   188 | loss: 1.534279 | lr 2.6046e-04 | norm: 0.4204 | dt: 6283.37ms | tok/sec: 83440.6 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.4ms | fwd: 2006.4ms | loss: 6.5ms | bwd: 4209.5ms | clip: 0.0ms | optim: 49.6ms
step   189 | loss: 1.547315 | lr 2.5962e-04 | norm: 0.3168 | dt: 6286.55ms | tok/sec: 83398.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.6ms | fwd: 2009.6ms | loss: 6.5ms | bwd: 4209.4ms | clip: 0.0ms | optim: 49.5ms
step   190 | loss: 1.518696 | lr 2.5878e-04 | norm: 0.3580 | dt: 6272.73ms | tok/sec: 83582.1 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 1997.7ms | loss: 6.5ms | bwd: 4207.5ms | clip: 0.0ms | optim: 49.5ms
step   191 | loss: 1.540766 | lr 2.5793e-04 | norm: 0.3142 | dt: 6292.74ms | tok/sec: 83316.3 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 2012.1ms | loss: 6.5ms | bwd: 4213.1ms | clip: 0.0ms | optim: 49.4ms
step   192 | loss: 1.529256 | lr 2.5707e-04 | norm: 0.4114 | dt: 6291.19ms | tok/sec: 83336.8 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 2009.6ms | loss: 6.5ms | bwd: 4214.3ms | clip: 0.0ms | optim: 49.3ms
step   193 | loss: 1.523573 | lr 2.5621e-04 | norm: 0.3459 | dt: 6300.37ms | tok/sec: 83215.5 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4225.3ms | clip: 0.0ms | optim: 49.4ms
step   194 | loss: 1.532913 | lr 2.5534e-04 | norm: 0.3599 | dt: 6295.65ms | tok/sec: 83277.8 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 2007.6ms | loss: 6.5ms | bwd: 4220.4ms | clip: 0.0ms | optim: 49.7ms
step   195 | loss: 1.514253 | lr 2.5446e-04 | norm: 0.4047 | dt: 6280.30ms | tok/sec: 83481.4 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4204.8ms | clip: 0.0ms | optim: 49.6ms
step   196 | loss: 1.528194 | lr 2.5358e-04 | norm: 0.4186 | dt: 6279.58ms | tok/sec: 83490.9 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4206.2ms | clip: 0.0ms | optim: 49.5ms
step   197 | loss: 1.554445 | lr 2.5270e-04 | norm: 0.4366 | dt: 6263.81ms | tok/sec: 83701.2 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.4ms | fwd: 2000.4ms | loss: 6.5ms | bwd: 4196.1ms | clip: 0.0ms | optim: 49.5ms
step   198 | loss: 1.502137 | lr 2.5180e-04 | norm: 0.3287 | dt: 6281.03ms | tok/sec: 83471.7 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 2002.9ms | loss: 6.5ms | bwd: 4210.4ms | clip: 0.0ms | optim: 49.7ms
step   199 | loss: 1.514722 | lr 2.5090e-04 | norm: 0.3312 | dt: 6269.77ms | tok/sec: 83621.6 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 1998.9ms | loss: 6.5ms | bwd: 4203.3ms | clip: 0.0ms | optim: 49.7ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run15_step_200.ckpt (safely)
step   200 | loss: 1.510058 | lr 2.5000e-04 | norm: 0.3472 | dt: 6196.03ms | tok/sec: 84616.7 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 1998.4ms | loss: 6.5ms | bwd: 4130.2ms | clip: 0.0ms | optim: 49.5ms
step   201 | loss: 1.537911 | lr 2.4909e-04 | norm: 0.3338 | dt: 6221.11ms | tok/sec: 84275.6 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.7ms | fwd: 1991.5ms | loss: 6.5ms | bwd: 4162.2ms | clip: 0.0ms | optim: 49.3ms
step   202 | loss: 1.513783 | lr 2.4818e-04 | norm: 0.3184 | dt: 6228.83ms | tok/sec: 84171.2 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 1996.4ms | loss: 6.5ms | bwd: 4164.7ms | clip: 0.0ms | optim: 49.8ms
step   203 | loss: 1.590754 | lr 2.4726e-04 | norm: 0.3981 | dt: 6251.24ms | tok/sec: 83869.5 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2001.4ms | loss: 6.5ms | bwd: 4182.5ms | clip: 0.0ms | optim: 49.5ms
step   204 | loss: 1.506860 | lr 2.4633e-04 | norm: 0.3159 | dt: 6252.73ms | tok/sec: 83849.4 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.8ms | fwd: 2004.7ms | loss: 6.5ms | bwd: 4180.2ms | clip: 0.0ms | optim: 49.6ms
step   205 | loss: 1.478778 | lr 2.4540e-04 | norm: 0.3400 | dt: 6265.75ms | tok/sec: 83675.2 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.7ms | fwd: 1997.6ms | loss: 6.5ms | bwd: 4200.4ms | clip: 0.0ms | optim: 49.5ms
step   206 | loss: 1.559009 | lr 2.4446e-04 | norm: 0.3214 | dt: 6272.03ms | tok/sec: 83591.5 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.6ms | fwd: 2004.7ms | loss: 6.5ms | bwd: 4199.5ms | clip: 0.0ms | optim: 49.4ms
step   207 | loss: 1.579555 | lr 2.4352e-04 | norm: 0.2891 | dt: 6270.97ms | tok/sec: 83605.5 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2004.7ms | loss: 6.5ms | bwd: 4199.0ms | clip: 0.0ms | optim: 49.4ms
step   208 | loss: 1.460985 | lr 2.4258e-04 | norm: 0.3072 | dt: 6290.45ms | tok/sec: 83346.7 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.6ms | fwd: 2013.3ms | loss: 6.5ms | bwd: 4209.4ms | clip: 0.0ms | optim: 49.7ms
step   209 | loss: 1.549868 | lr 2.4163e-04 | norm: 0.3220 | dt: 6294.35ms | tok/sec: 83295.0 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4222.3ms | clip: 0.0ms | optim: 49.6ms
step   210 | loss: 1.557330 | lr 2.4067e-04 | norm: 0.3002 | dt: 6278.52ms | tok/sec: 83505.1 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2004.2ms | loss: 6.5ms | bwd: 4206.8ms | clip: 0.0ms | optim: 49.4ms
step   211 | loss: 1.574781 | lr 2.3971e-04 | norm: 0.3479 | dt: 6290.74ms | tok/sec: 83342.9 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2004.3ms | loss: 6.5ms | bwd: 4219.1ms | clip: 0.0ms | optim: 49.3ms
step   212 | loss: 1.517553 | lr 2.3875e-04 | norm: 0.2982 | dt: 6265.61ms | tok/sec: 83677.1 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2010.2ms | loss: 6.5ms | bwd: 4187.8ms | clip: 0.0ms | optim: 49.7ms
step   213 | loss: 1.528307 | lr 2.3778e-04 | norm: 0.3382 | dt: 6304.68ms | tok/sec: 83158.5 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.4ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4223.2ms | clip: 0.0ms | optim: 50.1ms
step   214 | loss: 1.520213 | lr 2.3681e-04 | norm: 0.3582 | dt: 6271.28ms | tok/sec: 83601.5 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4195.3ms | clip: 0.0ms | optim: 49.3ms
step   215 | loss: 1.530610 | lr 2.3584e-04 | norm: 0.2744 | dt: 6298.63ms | tok/sec: 83238.4 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4217.7ms | clip: 0.0ms | optim: 49.3ms
step   216 | loss: 1.502046 | lr 2.3486e-04 | norm: 0.2919 | dt: 6279.63ms | tok/sec: 83490.3 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4203.8ms | clip: 0.0ms | optim: 49.7ms
step   217 | loss: 1.517489 | lr 2.3387e-04 | norm: 0.3077 | dt: 6263.69ms | tok/sec: 83702.8 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2003.6ms | loss: 6.5ms | bwd: 4192.3ms | clip: 0.0ms | optim: 49.8ms
step   218 | loss: 1.542618 | lr 2.3289e-04 | norm: 0.3658 | dt: 6296.95ms | tok/sec: 83260.6 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2012.9ms | loss: 6.5ms | bwd: 4216.6ms | clip: 0.0ms | optim: 49.4ms
step   219 | loss: 1.569870 | lr 2.3190e-04 | norm: 0.3860 | dt: 6278.64ms | tok/sec: 83503.4 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.6ms | fwd: 2015.5ms | loss: 6.5ms | bwd: 4195.5ms | clip: 0.0ms | optim: 49.7ms
step   220 | loss: 1.506267 | lr 2.3090e-04 | norm: 0.2647 | dt: 6259.60ms | tok/sec: 83757.5 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.6ms | fwd: 2003.3ms | loss: 6.5ms | bwd: 4188.5ms | clip: 0.0ms | optim: 49.8ms
step   221 | loss: 1.493292 | lr 2.2990e-04 | norm: 0.2586 | dt: 6276.88ms | tok/sec: 83526.9 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4197.1ms | clip: 0.0ms | optim: 49.7ms
step   222 | loss: 1.488327 | lr 2.2890e-04 | norm: 0.2369 | dt: 6277.85ms | tok/sec: 83513.9 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2006.6ms | loss: 6.5ms | bwd: 4204.0ms | clip: 0.0ms | optim: 49.4ms
step   223 | loss: 1.515566 | lr 2.2790e-04 | norm: 0.2846 | dt: 6291.82ms | tok/sec: 83328.6 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2007.6ms | loss: 6.5ms | bwd: 4217.0ms | clip: 0.0ms | optim: 49.3ms
step   224 | loss: 1.462511 | lr 2.2689e-04 | norm: 0.2349 | dt: 6286.47ms | tok/sec: 83399.4 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4211.4ms | clip: 0.0ms | optim: 49.6ms
step   225 | loss: 1.551289 | lr 2.2588e-04 | norm: 0.2854 | dt: 6258.79ms | tok/sec: 83768.3 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2003.8ms | loss: 6.5ms | bwd: 4187.5ms | clip: 0.0ms | optim: 49.6ms
step   226 | loss: 1.485188 | lr 2.2487e-04 | norm: 0.2624 | dt: 6302.80ms | tok/sec: 83183.4 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.6ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4224.0ms | clip: 0.0ms | optim: 49.5ms
step   227 | loss: 1.477064 | lr 2.2385e-04 | norm: 0.2761 | dt: 6289.84ms | tok/sec: 83354.7 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.6ms | fwd: 2006.4ms | loss: 6.5ms | bwd: 4216.1ms | clip: 0.0ms | optim: 49.3ms
step   228 | loss: 1.531475 | lr 2.2284e-04 | norm: 0.3208 | dt: 6272.91ms | tok/sec: 83579.6 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2003.0ms | loss: 6.5ms | bwd: 4202.8ms | clip: 0.0ms | optim: 49.3ms
step   229 | loss: 1.504093 | lr 2.2181e-04 | norm: 0.5493 | dt: 6274.92ms | tok/sec: 83553.0 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4198.8ms | clip: 0.0ms | optim: 49.5ms
step   230 | loss: 1.497003 | lr 2.2079e-04 | norm: 0.4434 | dt: 6292.81ms | tok/sec: 83315.4 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.5ms | bwd: 4215.3ms | clip: 0.0ms | optim: 49.7ms
step   231 | loss: 1.484642 | lr 2.1977e-04 | norm: 0.2852 | dt: 6287.99ms | tok/sec: 83379.3 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.6ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4212.7ms | clip: 0.0ms | optim: 49.8ms
step   232 | loss: 1.571955 | lr 2.1874e-04 | norm: 0.3154 | dt: 6281.22ms | tok/sec: 83469.1 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.6ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4206.5ms | clip: 0.0ms | optim: 49.3ms
step   233 | loss: 1.505156 | lr 2.1771e-04 | norm: 0.3633 | dt: 6283.78ms | tok/sec: 83435.2 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.5ms
step   234 | loss: 1.523280 | lr 2.1668e-04 | norm: 0.2757 | dt: 6283.81ms | tok/sec: 83434.7 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2005.4ms | loss: 6.5ms | bwd: 4210.9ms | clip: 0.0ms | optim: 49.6ms
step   235 | loss: 1.559093 | lr 2.1564e-04 | norm: 0.2750 | dt: 6300.74ms | tok/sec: 83210.5 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2006.2ms | loss: 6.5ms | bwd: 4227.4ms | clip: 0.0ms | optim: 49.3ms
step   236 | loss: 1.531076 | lr 2.1461e-04 | norm: 0.3403 | dt: 6266.95ms | tok/sec: 83659.2 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2013.9ms | loss: 6.5ms | bwd: 4185.5ms | clip: 0.0ms | optim: 49.5ms
step   237 | loss: 1.507729 | lr 2.1357e-04 | norm: 0.2822 | dt: 6305.01ms | tok/sec: 83154.2 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.6ms | fwd: 2011.2ms | loss: 6.5ms | bwd: 4226.3ms | clip: 0.0ms | optim: 49.5ms
step   238 | loss: 1.458905 | lr 2.1253e-04 | norm: 0.2618 | dt: 6278.25ms | tok/sec: 83508.7 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2005.5ms | loss: 6.5ms | bwd: 4205.4ms | clip: 0.0ms | optim: 49.3ms
step   239 | loss: 1.479126 | lr 2.1149e-04 | norm: 0.3102 | dt: 6284.24ms | tok/sec: 83429.1 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2010.1ms | loss: 6.5ms | bwd: 4206.7ms | clip: 0.0ms | optim: 49.5ms
step   240 | loss: 1.501266 | lr 2.1045e-04 | norm: 0.3532 | dt: 6296.37ms | tok/sec: 83268.3 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.6ms | fwd: 2010.1ms | loss: 6.5ms | bwd: 4219.0ms | clip: 0.0ms | optim: 49.3ms
step   241 | loss: 1.483563 | lr 2.0941e-04 | norm: 0.3302 | dt: 6281.04ms | tok/sec: 83471.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.6ms | fwd: 2006.4ms | loss: 6.5ms | bwd: 4207.4ms | clip: 0.0ms | optim: 49.3ms
step   242 | loss: 1.523439 | lr 2.0837e-04 | norm: 0.2864 | dt: 6295.08ms | tok/sec: 83285.3 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4217.8ms | clip: 0.0ms | optim: 49.4ms
step   243 | loss: 1.491824 | lr 2.0732e-04 | norm: 0.2516 | dt: 6295.91ms | tok/sec: 83274.4 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4214.5ms | clip: 0.0ms | optim: 50.0ms
step   244 | loss: 1.517366 | lr 2.0628e-04 | norm: 0.3017 | dt: 6282.83ms | tok/sec: 83447.7 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2004.9ms | loss: 6.5ms | bwd: 4210.7ms | clip: 0.0ms | optim: 49.3ms
step   245 | loss: 1.518789 | lr 2.0523e-04 | norm: 0.2438 | dt: 6291.41ms | tok/sec: 83333.9 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2010.9ms | loss: 6.5ms | bwd: 4213.2ms | clip: 0.0ms | optim: 49.4ms
step   246 | loss: 1.499409 | lr 2.0419e-04 | norm: 0.2604 | dt: 6283.45ms | tok/sec: 83439.6 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2005.8ms | loss: 6.5ms | bwd: 4210.2ms | clip: 0.0ms | optim: 49.6ms
step   247 | loss: 1.520633 | lr 2.0314e-04 | norm: 0.2421 | dt: 6279.82ms | tok/sec: 83487.7 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2016.9ms | loss: 6.5ms | bwd: 4195.4ms | clip: 0.0ms | optim: 49.5ms
step   248 | loss: 1.498210 | lr 2.0209e-04 | norm: 0.2382 | dt: 6284.31ms | tok/sec: 83428.1 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.6ms | fwd: 2007.2ms | loss: 6.5ms | bwd: 4209.6ms | clip: 0.0ms | optim: 49.6ms
step   249 | loss: 1.495320 | lr 2.0105e-04 | norm: 0.2269 | dt: 6297.83ms | tok/sec: 83249.0 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.6ms | fwd: 2007.0ms | loss: 6.5ms | bwd: 4223.0ms | clip: 0.0ms | optim: 49.8ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_val_00000.bin
100000000
validation loss: 1.5036
step   250 | loss: 1.477384 | lr 2.0000e-04 | norm: 0.2585 | dt: 11346.46ms | tok/sec: 46207.2 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.6ms | fwd: 2018.2ms | loss: 6.5ms | bwd: 4215.5ms | clip: 0.0ms | optim: 49.7ms
step   251 | loss: 1.522004 | lr 1.9895e-04 | norm: 0.2487 | dt: 6283.90ms | tok/sec: 83433.5 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4209.1ms | clip: 0.0ms | optim: 49.6ms
step   252 | loss: 1.528714 | lr 1.9791e-04 | norm: 0.2227 | dt: 6294.67ms | tok/sec: 83290.8 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2011.8ms | loss: 6.5ms | bwd: 4215.7ms | clip: 0.0ms | optim: 49.3ms
step   253 | loss: 1.498192 | lr 1.9686e-04 | norm: 0.2764 | dt: 6310.56ms | tok/sec: 83081.1 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.6ms | fwd: 2012.2ms | loss: 6.5ms | bwd: 4230.6ms | clip: 0.0ms | optim: 49.6ms
step   254 | loss: 1.512607 | lr 1.9581e-04 | norm: 0.2730 | dt: 6264.22ms | tok/sec: 83695.6 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2004.9ms | loss: 6.5ms | bwd: 4191.1ms | clip: 0.0ms | optim: 50.3ms
step   255 | loss: 1.532828 | lr 1.9477e-04 | norm: 0.2500 | dt: 6305.75ms | tok/sec: 83144.4 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2004.2ms | loss: 6.5ms | bwd: 4234.2ms | clip: 0.0ms | optim: 49.5ms
step   256 | loss: 1.476528 | lr 1.9372e-04 | norm: 0.3718 | dt: 6278.55ms | tok/sec: 83504.6 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2006.9ms | loss: 6.5ms | bwd: 4203.9ms | clip: 0.0ms | optim: 49.9ms
step   257 | loss: 1.473291 | lr 1.9268e-04 | norm: 0.3378 | dt: 6279.45ms | tok/sec: 83492.7 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2009.4ms | loss: 6.5ms | bwd: 4202.4ms | clip: 0.0ms | optim: 49.7ms
step   258 | loss: 1.478200 | lr 1.9163e-04 | norm: 0.3498 | dt: 6273.62ms | tok/sec: 83570.3 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2009.6ms | loss: 6.5ms | bwd: 4196.5ms | clip: 0.0ms | optim: 49.5ms
step   259 | loss: 1.468098 | lr 1.9059e-04 | norm: 0.2409 | dt: 6284.78ms | tok/sec: 83421.8 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2008.4ms | loss: 6.5ms | bwd: 4208.7ms | clip: 0.0ms | optim: 49.8ms
step   260 | loss: 1.481337 | lr 1.8955e-04 | norm: 0.2138 | dt: 6280.51ms | tok/sec: 83478.5 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2012.5ms | loss: 6.5ms | bwd: 4200.6ms | clip: 0.0ms | optim: 49.4ms
step   261 | loss: 1.470711 | lr 1.8851e-04 | norm: 0.2196 | dt: 6280.88ms | tok/sec: 83473.6 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2005.6ms | loss: 6.5ms | bwd: 4207.8ms | clip: 0.0ms | optim: 49.5ms
step   262 | loss: 1.509797 | lr 1.8747e-04 | norm: 0.2158 | dt: 6286.61ms | tok/sec: 83397.6 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2009.3ms | loss: 6.5ms | bwd: 4209.9ms | clip: 0.0ms | optim: 49.4ms
step   263 | loss: 1.521286 | lr 1.8643e-04 | norm: 0.2522 | dt: 6297.65ms | tok/sec: 83251.3 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.6ms | fwd: 2009.6ms | loss: 6.5ms | bwd: 4220.4ms | clip: 0.0ms | optim: 49.6ms
step   264 | loss: 1.516985 | lr 1.8539e-04 | norm: 0.2522 | dt: 6283.51ms | tok/sec: 83438.8 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.6ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4208.9ms | clip: 0.0ms | optim: 49.3ms
step   265 | loss: 1.465040 | lr 1.8436e-04 | norm: 0.2012 | dt: 6276.92ms | tok/sec: 83526.3 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2010.9ms | loss: 6.5ms | bwd: 4198.7ms | clip: 0.0ms | optim: 49.4ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_train_00002.bin
100000000
step   266 | loss: 1.437117 | lr 1.8332e-04 | norm: 0.2658 | dt: 6289.26ms | tok/sec: 83362.4 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 14.2ms | fwd: 2004.9ms | loss: 6.5ms | bwd: 4205.9ms | clip: 0.0ms | optim: 49.8ms
step   267 | loss: 1.541539 | lr 1.8229e-04 | norm: 0.2220 | dt: 6294.03ms | tok/sec: 83299.3 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.5ms | bwd: 4216.7ms | clip: 0.0ms | optim: 49.4ms
step   268 | loss: 1.522246 | lr 1.8126e-04 | norm: 0.2198 | dt: 6290.17ms | tok/sec: 83350.4 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2010.1ms | loss: 6.5ms | bwd: 4212.4ms | clip: 0.0ms | optim: 49.6ms
step   269 | loss: 1.495068 | lr 1.8023e-04 | norm: 0.2245 | dt: 6267.15ms | tok/sec: 83656.5 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2004.1ms | loss: 6.5ms | bwd: 4195.8ms | clip: 0.0ms | optim: 49.4ms
step   270 | loss: 1.455160 | lr 1.7921e-04 | norm: 0.1917 | dt: 6303.02ms | tok/sec: 83180.5 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2016.9ms | loss: 6.5ms | bwd: 4218.3ms | clip: 0.0ms | optim: 49.8ms
step   271 | loss: 1.534132 | lr 1.7819e-04 | norm: 0.1853 | dt: 6303.42ms | tok/sec: 83175.2 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2004.0ms | loss: 6.8ms | bwd: 4231.7ms | clip: 0.0ms | optim: 49.5ms
step   272 | loss: 1.498029 | lr 1.7716e-04 | norm: 0.1752 | dt: 6287.24ms | tok/sec: 83389.2 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2006.4ms | loss: 6.5ms | bwd: 4213.6ms | clip: 0.0ms | optim: 49.3ms
step   273 | loss: 1.563150 | lr 1.7615e-04 | norm: 0.1926 | dt: 6293.05ms | tok/sec: 83312.3 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4215.5ms | clip: 0.0ms | optim: 49.8ms
step   274 | loss: 1.494102 | lr 1.7513e-04 | norm: 0.2292 | dt: 6281.86ms | tok/sec: 83460.6 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2013.1ms | loss: 6.5ms | bwd: 4201.0ms | clip: 0.0ms | optim: 49.7ms
step   275 | loss: 1.527769 | lr 1.7412e-04 | norm: 0.1968 | dt: 6306.39ms | tok/sec: 83136.0 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2006.7ms | loss: 6.5ms | bwd: 4232.3ms | clip: 0.0ms | optim: 49.5ms
step   276 | loss: 1.464794 | lr 1.7311e-04 | norm: 0.2164 | dt: 6271.86ms | tok/sec: 83593.8 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.6ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4197.0ms | clip: 0.0ms | optim: 49.3ms
step   277 | loss: 1.532739 | lr 1.7210e-04 | norm: 0.2806 | dt: 6293.67ms | tok/sec: 83304.0 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2011.5ms | loss: 6.5ms | bwd: 4215.0ms | clip: 0.0ms | optim: 49.3ms
step   278 | loss: 1.466705 | lr 1.7110e-04 | norm: 0.1909 | dt: 6293.65ms | tok/sec: 83304.2 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.6ms | fwd: 2004.6ms | loss: 6.7ms | bwd: 4221.4ms | clip: 0.0ms | optim: 49.3ms
step   279 | loss: 1.549416 | lr 1.7010e-04 | norm: 0.2056 | dt: 6286.02ms | tok/sec: 83405.4 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.6ms | fwd: 2007.1ms | loss: 6.5ms | bwd: 4211.6ms | clip: 0.0ms | optim: 49.3ms
step   280 | loss: 1.483340 | lr 1.6910e-04 | norm: 0.2163 | dt: 6288.81ms | tok/sec: 83368.4 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.7ms | fwd: 2004.8ms | loss: 6.5ms | bwd: 4216.1ms | clip: 0.0ms | optim: 49.8ms
step   281 | loss: 1.489990 | lr 1.6810e-04 | norm: 0.2223 | dt: 6307.18ms | tok/sec: 83125.6 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2020.7ms | loss: 6.5ms | bwd: 4219.2ms | clip: 0.0ms | optim: 49.4ms
step   282 | loss: 1.488303 | lr 1.6711e-04 | norm: 0.2564 | dt: 6276.49ms | tok/sec: 83532.1 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2002.8ms | loss: 6.5ms | bwd: 4205.7ms | clip: 0.0ms | optim: 50.1ms
step   283 | loss: 1.415160 | lr 1.6613e-04 | norm: 0.2040 | dt: 6289.03ms | tok/sec: 83365.4 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.4ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4214.0ms | clip: 0.0ms | optim: 49.3ms
step   284 | loss: 1.523561 | lr 1.6514e-04 | norm: 0.2585 | dt: 6280.62ms | tok/sec: 83477.1 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2010.9ms | loss: 6.5ms | bwd: 4202.2ms | clip: 0.0ms | optim: 49.5ms
step   285 | loss: 1.495964 | lr 1.6416e-04 | norm: 0.2133 | dt: 6274.16ms | tok/sec: 83563.1 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2003.8ms | loss: 6.5ms | bwd: 4203.2ms | clip: 0.0ms | optim: 49.2ms
step   286 | loss: 1.474825 | lr 1.6319e-04 | norm: 0.2681 | dt: 6289.36ms | tok/sec: 83361.1 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4214.4ms | clip: 0.0ms | optim: 49.4ms
step   287 | loss: 1.428286 | lr 1.6222e-04 | norm: 0.1984 | dt: 6269.78ms | tok/sec: 83621.5 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.6ms | fwd: 2001.5ms | loss: 6.5ms | bwd: 4200.6ms | clip: 0.0ms | optim: 49.8ms
step   288 | loss: 1.455253 | lr 1.6125e-04 | norm: 0.2228 | dt: 6273.36ms | tok/sec: 83573.7 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4201.5ms | clip: 0.0ms | optim: 49.3ms
step   289 | loss: 1.431326 | lr 1.6029e-04 | norm: 0.2744 | dt: 6293.13ms | tok/sec: 83311.2 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.6ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4218.4ms | clip: 0.0ms | optim: 49.3ms
step   290 | loss: 1.412302 | lr 1.5933e-04 | norm: 0.1970 | dt: 6273.10ms | tok/sec: 83577.2 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2007.0ms | loss: 6.5ms | bwd: 4198.5ms | clip: 0.0ms | optim: 49.7ms
step   291 | loss: 1.481361 | lr 1.5837e-04 | norm: 0.1619 | dt: 6286.85ms | tok/sec: 83394.4 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.6ms | fwd: 2005.5ms | loss: 6.5ms | bwd: 4213.9ms | clip: 0.0ms | optim: 49.5ms
step   292 | loss: 1.490216 | lr 1.5742e-04 | norm: 0.1745 | dt: 6287.05ms | tok/sec: 83391.8 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.6ms | fwd: 2006.8ms | loss: 6.5ms | bwd: 4212.3ms | clip: 0.0ms | optim: 49.9ms
step   293 | loss: 1.500644 | lr 1.5648e-04 | norm: 0.1627 | dt: 6273.80ms | tok/sec: 83567.8 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2002.1ms | loss: 6.5ms | bwd: 4204.5ms | clip: 0.0ms | optim: 49.3ms
step   294 | loss: 1.434377 | lr 1.5554e-04 | norm: 0.2026 | dt: 6285.06ms | tok/sec: 83418.1 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2006.8ms | loss: 6.5ms | bwd: 4210.7ms | clip: 0.0ms | optim: 49.6ms
step   295 | loss: 1.490050 | lr 1.5460e-04 | norm: 0.1738 | dt: 6283.70ms | tok/sec: 83436.2 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4211.7ms | clip: 0.0ms | optim: 49.6ms
step   296 | loss: 1.503023 | lr 1.5367e-04 | norm: 0.1578 | dt: 6289.12ms | tok/sec: 83364.3 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.6ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4213.8ms | clip: 0.0ms | optim: 49.5ms
step   297 | loss: 1.435766 | lr 1.5274e-04 | norm: 0.2016 | dt: 6286.65ms | tok/sec: 83397.0 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.7ms | fwd: 2011.4ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.5ms
step   298 | loss: 1.449123 | lr 1.5182e-04 | norm: 0.3584 | dt: 6292.92ms | tok/sec: 83313.9 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2010.1ms | loss: 6.5ms | bwd: 4215.6ms | clip: 0.0ms | optim: 49.3ms
step   299 | loss: 1.446947 | lr 1.5091e-04 | norm: 0.1390 | dt: 6297.66ms | tok/sec: 83251.2 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2013.8ms | loss: 6.5ms | bwd: 4216.7ms | clip: 0.0ms | optim: 49.3ms
step   300 | loss: 1.497401 | lr 1.5000e-04 | norm: 0.1814 | dt: 6293.48ms | tok/sec: 83306.6 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4216.1ms | clip: 0.0ms | optim: 49.6ms
step   301 | loss: 1.469442 | lr 1.4910e-04 | norm: 0.1639 | dt: 6285.04ms | tok/sec: 83418.5 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.6ms | fwd: 2005.7ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.5ms
step   302 | loss: 1.471132 | lr 1.4820e-04 | norm: 0.1950 | dt: 6277.69ms | tok/sec: 83516.1 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4198.3ms | clip: 0.0ms | optim: 49.5ms
step   303 | loss: 1.396044 | lr 1.4730e-04 | norm: 0.1861 | dt: 6295.61ms | tok/sec: 83278.4 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.6ms | fwd: 2012.5ms | loss: 6.5ms | bwd: 4215.8ms | clip: 0.0ms | optim: 49.3ms
step   304 | loss: 1.490811 | lr 1.4642e-04 | norm: 0.2078 | dt: 6281.90ms | tok/sec: 83460.0 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.4ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4210.0ms | clip: 0.0ms | optim: 49.5ms
step   305 | loss: 1.506205 | lr 1.4554e-04 | norm: 0.1633 | dt: 6257.76ms | tok/sec: 83782.1 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.6ms | fwd: 2005.0ms | loss: 6.5ms | bwd: 4185.0ms | clip: 0.0ms | optim: 49.7ms
step   306 | loss: 1.431792 | lr 1.4466e-04 | norm: 0.1843 | dt: 6274.03ms | tok/sec: 83564.8 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.7ms | fwd: 2004.4ms | loss: 6.5ms | bwd: 4202.0ms | clip: 0.0ms | optim: 49.5ms
step   307 | loss: 1.528789 | lr 1.4379e-04 | norm: 0.1731 | dt: 6291.28ms | tok/sec: 83335.6 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2009.4ms | loss: 6.5ms | bwd: 4214.6ms | clip: 0.0ms | optim: 49.4ms
step   308 | loss: 1.513059 | lr 1.4293e-04 | norm: 0.2089 | dt: 6265.25ms | tok/sec: 83682.0 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4190.2ms | clip: 0.0ms | optim: 49.6ms
step   309 | loss: 1.457680 | lr 1.4207e-04 | norm: 0.2012 | dt: 6281.42ms | tok/sec: 83466.5 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2012.2ms | loss: 6.5ms | bwd: 4202.1ms | clip: 0.0ms | optim: 49.3ms
step   310 | loss: 1.409603 | lr 1.4122e-04 | norm: 0.1526 | dt: 6291.32ms | tok/sec: 83335.1 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.6ms | fwd: 2006.4ms | loss: 6.5ms | bwd: 4217.6ms | clip: 0.0ms | optim: 49.3ms
step   311 | loss: 1.481197 | lr 1.4038e-04 | norm: 0.1974 | dt: 6289.04ms | tok/sec: 83365.3 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.7ms | fwd: 2009.7ms | loss: 6.5ms | bwd: 4211.5ms | clip: 0.0ms | optim: 49.7ms
step   312 | loss: 1.442244 | lr 1.3954e-04 | norm: 0.1517 | dt: 6272.76ms | tok/sec: 83581.8 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.7ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4200.7ms | clip: 0.0ms | optim: 49.4ms
step   313 | loss: 1.426918 | lr 1.3871e-04 | norm: 0.1826 | dt: 6283.01ms | tok/sec: 83445.3 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.6ms | fwd: 2005.4ms | loss: 6.5ms | bwd: 4210.1ms | clip: 0.0ms | optim: 49.5ms
step   314 | loss: 1.475576 | lr 1.3789e-04 | norm: 0.1735 | dt: 6287.21ms | tok/sec: 83389.6 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.6ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4212.0ms | clip: 0.0ms | optim: 49.4ms
step   315 | loss: 1.485416 | lr 1.3707e-04 | norm: 0.1951 | dt: 6285.47ms | tok/sec: 83412.7 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2008.5ms | loss: 6.5ms | bwd: 4209.7ms | clip: 0.0ms | optim: 49.2ms
step   316 | loss: 1.462493 | lr 1.3626e-04 | norm: 0.1440 | dt: 6299.12ms | tok/sec: 83232.0 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4225.4ms | clip: 0.0ms | optim: 49.7ms
step   317 | loss: 1.452401 | lr 1.3545e-04 | norm: 0.1463 | dt: 6295.64ms | tok/sec: 83277.9 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2010.8ms | loss: 6.5ms | bwd: 4217.3ms | clip: 0.0ms | optim: 49.6ms
step   318 | loss: 1.423061 | lr 1.3466e-04 | norm: 0.1426 | dt: 6280.87ms | tok/sec: 83473.8 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4198.1ms | clip: 0.0ms | optim: 50.3ms
step   319 | loss: 1.529647 | lr 1.3387e-04 | norm: 0.1433 | dt: 6274.19ms | tok/sec: 83562.6 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4199.5ms | clip: 0.0ms | optim: 49.4ms
step   320 | loss: 1.470993 | lr 1.3309e-04 | norm: 0.1550 | dt: 6272.88ms | tok/sec: 83580.1 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2001.3ms | loss: 6.5ms | bwd: 4204.4ms | clip: 0.0ms | optim: 49.3ms
step   321 | loss: 1.534515 | lr 1.3231e-04 | norm: 0.1568 | dt: 6286.15ms | tok/sec: 83403.7 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.6ms
step   322 | loss: 1.471309 | lr 1.3155e-04 | norm: 0.1767 | dt: 6280.60ms | tok/sec: 83477.3 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2002.0ms | loss: 6.5ms | bwd: 4210.9ms | clip: 0.0ms | optim: 49.8ms
step   323 | loss: 1.463022 | lr 1.3079e-04 | norm: 0.1762 | dt: 6308.42ms | tok/sec: 83109.3 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.6ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4233.5ms | clip: 0.0ms | optim: 49.5ms
step   324 | loss: 1.445003 | lr 1.3003e-04 | norm: 0.2671 | dt: 6281.51ms | tok/sec: 83465.3 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.6ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4201.3ms | clip: 0.0ms | optim: 49.4ms
step   325 | loss: 1.467386 | lr 1.2929e-04 | norm: 0.3822 | dt: 6281.10ms | tok/sec: 83470.8 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.6ms | fwd: 2002.7ms | loss: 6.5ms | bwd: 4208.5ms | clip: 0.0ms | optim: 51.9ms
step   326 | loss: 1.513854 | lr 1.2855e-04 | norm: 0.1901 | dt: 6289.05ms | tok/sec: 83365.2 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4211.9ms | clip: 0.0ms | optim: 49.3ms
step   327 | loss: 1.445046 | lr 1.2782e-04 | norm: 0.1596 | dt: 6284.36ms | tok/sec: 83427.5 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2002.2ms | loss: 6.5ms | bwd: 4214.4ms | clip: 0.0ms | optim: 49.8ms
step   328 | loss: 1.400366 | lr 1.2710e-04 | norm: 0.2739 | dt: 6282.83ms | tok/sec: 83447.8 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2015.4ms | loss: 6.5ms | bwd: 4199.8ms | clip: 0.0ms | optim: 49.6ms
step   329 | loss: 1.525141 | lr 1.2639e-04 | norm: 0.2485 | dt: 6279.42ms | tok/sec: 83493.0 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.6ms | fwd: 2000.0ms | loss: 6.5ms | bwd: 4212.1ms | clip: 0.0ms | optim: 49.3ms
step   330 | loss: 1.467020 | lr 1.2569e-04 | norm: 0.1829 | dt: 6292.00ms | tok/sec: 83326.2 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4210.1ms | clip: 0.0ms | optim: 49.5ms
step   331 | loss: 1.444497 | lr 1.2499e-04 | norm: 0.1592 | dt: 6285.96ms | tok/sec: 83406.2 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2010.8ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.6ms
step   332 | loss: 1.423348 | lr 1.2430e-04 | norm: 0.1609 | dt: 6290.71ms | tok/sec: 83343.2 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4209.9ms | clip: 0.0ms | optim: 50.2ms
step   333 | loss: 1.448303 | lr 1.2362e-04 | norm: 0.2448 | dt: 6290.97ms | tok/sec: 83339.8 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.6ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4214.6ms | clip: 0.0ms | optim: 49.4ms
step   334 | loss: 1.447588 | lr 1.2295e-04 | norm: 0.1536 | dt: 6274.89ms | tok/sec: 83553.3 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2001.1ms | loss: 6.5ms | bwd: 4205.8ms | clip: 0.0ms | optim: 50.1ms
step   335 | loss: 1.496983 | lr 1.2229e-04 | norm: 0.1492 | dt: 6286.09ms | tok/sec: 83404.5 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4210.5ms | clip: 0.0ms | optim: 49.7ms
step   336 | loss: 1.454040 | lr 1.2163e-04 | norm: 0.1650 | dt: 6274.65ms | tok/sec: 83556.5 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 1999.8ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.3ms
step   337 | loss: 1.475746 | lr 1.2098e-04 | norm: 0.1506 | dt: 6299.30ms | tok/sec: 83229.6 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2011.4ms | loss: 6.5ms | bwd: 4220.3ms | clip: 0.0ms | optim: 49.6ms
step   338 | loss: 1.495171 | lr 1.2035e-04 | norm: 0.1437 | dt: 6248.92ms | tok/sec: 83900.5 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.8ms | fwd: 2000.3ms | loss: 6.5ms | bwd: 4180.7ms | clip: 0.0ms | optim: 49.6ms
step   339 | loss: 1.479284 | lr 1.1972e-04 | norm: 0.1300 | dt: 6281.52ms | tok/sec: 83465.1 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2011.8ms | loss: 6.5ms | bwd: 4202.2ms | clip: 0.0ms | optim: 49.6ms
step   340 | loss: 1.465289 | lr 1.1910e-04 | norm: 0.1817 | dt: 6275.22ms | tok/sec: 83548.9 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.6ms | fwd: 2011.6ms | loss: 6.5ms | bwd: 4196.2ms | clip: 0.0ms | optim: 49.5ms
step   341 | loss: 1.448179 | lr 1.1849e-04 | norm: 0.1543 | dt: 6304.35ms | tok/sec: 83162.9 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4231.0ms | clip: 0.0ms | optim: 49.5ms
step   342 | loss: 1.498588 | lr 1.1789e-04 | norm: 0.1311 | dt: 6273.83ms | tok/sec: 83567.4 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.4ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4198.8ms | clip: 0.0ms | optim: 49.8ms
step   343 | loss: 1.458866 | lr 1.1729e-04 | norm: 0.1933 | dt: 6288.28ms | tok/sec: 83375.4 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 1998.1ms | loss: 6.5ms | bwd: 4222.6ms | clip: 0.0ms | optim: 49.6ms
step   344 | loss: 1.422083 | lr 1.1671e-04 | norm: 0.1648 | dt: 6282.56ms | tok/sec: 83451.3 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2005.7ms | loss: 6.5ms | bwd: 4209.5ms | clip: 0.0ms | optim: 49.5ms
step   345 | loss: 1.418629 | lr 1.1613e-04 | norm: 0.1755 | dt: 6277.73ms | tok/sec: 83515.5 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.6ms | fwd: 2010.9ms | loss: 6.5ms | bwd: 4199.6ms | clip: 0.0ms | optim: 49.3ms
step   346 | loss: 1.442413 | lr 1.1557e-04 | norm: 0.1801 | dt: 6297.58ms | tok/sec: 83252.3 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4221.4ms | clip: 0.0ms | optim: 49.4ms
step   347 | loss: 1.446483 | lr 1.1501e-04 | norm: 0.1446 | dt: 6276.61ms | tok/sec: 83530.4 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2005.1ms | loss: 6.5ms | bwd: 4203.5ms | clip: 0.0ms | optim: 50.0ms
step   348 | loss: 1.464539 | lr 1.1446e-04 | norm: 0.1645 | dt: 6292.60ms | tok/sec: 83318.2 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.6ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4216.1ms | clip: 0.0ms | optim: 49.5ms
step   349 | loss: 1.405365 | lr 1.1393e-04 | norm: 0.1409 | dt: 6265.20ms | tok/sec: 83682.6 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.6ms | fwd: 2005.4ms | loss: 6.5ms | bwd: 4191.9ms | clip: 0.0ms | optim: 49.9ms
step   350 | loss: 1.445062 | lr 1.1340e-04 | norm: 0.1686 | dt: 6281.24ms | tok/sec: 83468.9 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2008.9ms | loss: 6.5ms | bwd: 4205.0ms | clip: 0.0ms | optim: 49.5ms
step   351 | loss: 1.464890 | lr 1.1288e-04 | norm: 0.2256 | dt: 6280.59ms | tok/sec: 83477.5 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.4ms | fwd: 2006.8ms | loss: 6.5ms | bwd: 4206.6ms | clip: 0.0ms | optim: 49.3ms
step   352 | loss: 1.509487 | lr 1.1237e-04 | norm: 0.1661 | dt: 6263.33ms | tok/sec: 83707.5 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.6ms | fwd: 2006.3ms | loss: 6.5ms | bwd: 4189.4ms | clip: 0.0ms | optim: 49.6ms
step   353 | loss: 1.465810 | lr 1.1187e-04 | norm: 0.1424 | dt: 6281.64ms | tok/sec: 83463.5 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2012.5ms | loss: 6.6ms | bwd: 4201.7ms | clip: 0.0ms | optim: 49.4ms
step   354 | loss: 1.421557 | lr 1.1138e-04 | norm: 0.1381 | dt: 6280.96ms | tok/sec: 83472.5 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2000.4ms | loss: 6.5ms | bwd: 4213.2ms | clip: 0.0ms | optim: 49.5ms
step   355 | loss: 1.494921 | lr 1.1090e-04 | norm: 0.1500 | dt: 6273.57ms | tok/sec: 83570.9 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2011.2ms | loss: 6.5ms | bwd: 4195.0ms | clip: 0.0ms | optim: 49.5ms
step   356 | loss: 1.458668 | lr 1.1043e-04 | norm: 0.1319 | dt: 6278.49ms | tok/sec: 83505.4 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.7ms | fwd: 2003.9ms | loss: 6.5ms | bwd: 4207.0ms | clip: 0.0ms | optim: 49.6ms
step   357 | loss: 1.483556 | lr 1.0997e-04 | norm: 0.1838 | dt: 6291.05ms | tok/sec: 83338.7 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4216.0ms | clip: 0.0ms | optim: 49.4ms
step   358 | loss: 1.490927 | lr 1.0952e-04 | norm: 0.1452 | dt: 6288.23ms | tok/sec: 83376.1 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2013.3ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.4ms
step   359 | loss: 1.372430 | lr 1.0908e-04 | norm: 0.1780 | dt: 6287.38ms | tok/sec: 83387.4 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2006.5ms | loss: 6.5ms | bwd: 4213.1ms | clip: 0.0ms | optim: 49.7ms
step   360 | loss: 1.452322 | lr 1.0865e-04 | norm: 0.1626 | dt: 6301.52ms | tok/sec: 83200.2 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2006.2ms | loss: 6.5ms | bwd: 4227.9ms | clip: 0.0ms | optim: 49.5ms
step   361 | loss: 1.485359 | lr 1.0822e-04 | norm: 0.1564 | dt: 6269.39ms | tok/sec: 83626.6 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2002.5ms | loss: 6.5ms | bwd: 4199.5ms | clip: 0.0ms | optim: 49.4ms
step   362 | loss: 1.438203 | lr 1.0781e-04 | norm: 0.1686 | dt: 6288.41ms | tok/sec: 83373.7 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2008.9ms | loss: 6.5ms | bwd: 4212.1ms | clip: 0.0ms | optim: 49.5ms
step   363 | loss: 1.459694 | lr 1.0741e-04 | norm: 0.2497 | dt: 6277.73ms | tok/sec: 83515.5 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2006.7ms | loss: 6.5ms | bwd: 4203.8ms | clip: 0.0ms | optim: 49.3ms
step   364 | loss: 1.439780 | lr 1.0702e-04 | norm: 0.1696 | dt: 6292.42ms | tok/sec: 83320.6 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.6ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4215.0ms | clip: 0.0ms | optim: 49.5ms
step   365 | loss: 1.472931 | lr 1.0664e-04 | norm: 0.1578 | dt: 6263.47ms | tok/sec: 83705.6 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2002.0ms | loss: 6.5ms | bwd: 4194.3ms | clip: 0.0ms | optim: 49.3ms
step   366 | loss: 1.466796 | lr 1.0627e-04 | norm: 0.1409 | dt: 6278.59ms | tok/sec: 83504.2 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2005.2ms | loss: 6.5ms | bwd: 4205.5ms | clip: 0.0ms | optim: 50.0ms
step   367 | loss: 1.451478 | lr 1.0591e-04 | norm: 0.1518 | dt: 6283.20ms | tok/sec: 83442.8 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.6ms | fwd: 2005.7ms | loss: 6.5ms | bwd: 4210.3ms | clip: 0.0ms | optim: 49.2ms
step   368 | loss: 1.457484 | lr 1.0556e-04 | norm: 0.1517 | dt: 6282.16ms | tok/sec: 83456.6 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.6ms | fwd: 2013.3ms | loss: 6.5ms | bwd: 4201.3ms | clip: 0.0ms | optim: 49.5ms
step   369 | loss: 1.464970 | lr 1.0522e-04 | norm: 0.1445 | dt: 6280.34ms | tok/sec: 83480.9 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.6ms | fwd: 2004.9ms | loss: 6.6ms | bwd: 4208.0ms | clip: 0.0ms | optim: 49.3ms
step   370 | loss: 1.408668 | lr 1.0489e-04 | norm: 0.1435 | dt: 6279.06ms | tok/sec: 83497.9 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2009.4ms | loss: 6.5ms | bwd: 4202.5ms | clip: 0.0ms | optim: 49.2ms
step   371 | loss: 1.454649 | lr 1.0458e-04 | norm: 0.1981 | dt: 6285.70ms | tok/sec: 83409.6 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.9ms
step   372 | loss: 1.476396 | lr 1.0427e-04 | norm: 0.1396 | dt: 6266.04ms | tok/sec: 83671.4 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2005.6ms | loss: 6.5ms | bwd: 4193.0ms | clip: 0.0ms | optim: 49.6ms
step   373 | loss: 1.428265 | lr 1.0397e-04 | norm: 0.1244 | dt: 6284.95ms | tok/sec: 83419.6 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2010.2ms | loss: 6.5ms | bwd: 4207.0ms | clip: 0.0ms | optim: 49.8ms
step   374 | loss: 1.433702 | lr 1.0368e-04 | norm: 0.1419 | dt: 6280.00ms | tok/sec: 83485.3 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.7ms | fwd: 2004.0ms | loss: 6.5ms | bwd: 4208.3ms | clip: 0.0ms | optim: 49.5ms
step   375 | loss: 1.475788 | lr 1.0341e-04 | norm: 0.1380 | dt: 6279.39ms | tok/sec: 83493.5 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2005.6ms | loss: 6.5ms | bwd: 4206.3ms | clip: 0.0ms | optim: 49.5ms
step   376 | loss: 1.469541 | lr 1.0314e-04 | norm: 0.1433 | dt: 6279.51ms | tok/sec: 83491.8 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2004.1ms | loss: 6.5ms | bwd: 4208.1ms | clip: 0.0ms | optim: 49.3ms
step   377 | loss: 1.421060 | lr 1.0289e-04 | norm: 0.1599 | dt: 6272.92ms | tok/sec: 83579.6 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2002.4ms | loss: 6.5ms | bwd: 4203.2ms | clip: 0.0ms | optim: 49.5ms
step   378 | loss: 1.431809 | lr 1.0264e-04 | norm: 0.1563 | dt: 6290.88ms | tok/sec: 83340.9 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.8ms | fwd: 2005.4ms | loss: 6.5ms | bwd: 4217.7ms | clip: 0.0ms | optim: 49.4ms
step   379 | loss: 1.421825 | lr 1.0241e-04 | norm: 0.1395 | dt: 6269.88ms | tok/sec: 83620.1 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2001.9ms | loss: 6.5ms | bwd: 4200.5ms | clip: 0.0ms | optim: 49.6ms
step   380 | loss: 1.482245 | lr 1.0219e-04 | norm: 0.1572 | dt: 6283.25ms | tok/sec: 83442.2 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2000.3ms | loss: 6.5ms | bwd: 4215.6ms | clip: 0.0ms | optim: 49.4ms
step   381 | loss: 1.472692 | lr 1.0197e-04 | norm: 0.1837 | dt: 6288.44ms | tok/sec: 83373.2 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.6ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4212.2ms | clip: 0.0ms | optim: 49.5ms
step   382 | loss: 1.447915 | lr 1.0177e-04 | norm: 0.1698 | dt: 6286.20ms | tok/sec: 83403.0 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2011.6ms | loss: 6.5ms | bwd: 4206.6ms | clip: 0.0ms | optim: 50.2ms
step   383 | loss: 1.398059 | lr 1.0158e-04 | norm: 0.1171 | dt: 6271.93ms | tok/sec: 83592.7 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2002.5ms | loss: 6.5ms | bwd: 4202.2ms | clip: 0.0ms | optim: 49.4ms
step   384 | loss: 1.480612 | lr 1.0140e-04 | norm: 0.1184 | dt: 6263.40ms | tok/sec: 83706.7 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2005.5ms | loss: 6.7ms | bwd: 4190.2ms | clip: 0.0ms | optim: 49.5ms
step   385 | loss: 1.458910 | lr 1.0123e-04 | norm: 0.1926 | dt: 6275.22ms | tok/sec: 83548.9 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2016.2ms | loss: 6.5ms | bwd: 4191.8ms | clip: 0.0ms | optim: 49.3ms
step   386 | loss: 1.472458 | lr 1.0107e-04 | norm: 0.1241 | dt: 6287.60ms | tok/sec: 83384.5 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.6ms | fwd: 2010.2ms | loss: 6.5ms | bwd: 4210.0ms | clip: 0.0ms | optim: 49.3ms
step   387 | loss: 1.483831 | lr 1.0093e-04 | norm: 0.1328 | dt: 6272.51ms | tok/sec: 83585.1 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 1999.7ms | loss: 6.5ms | bwd: 4205.5ms | clip: 0.0ms | optim: 49.4ms
step   388 | loss: 1.470569 | lr 1.0079e-04 | norm: 0.1420 | dt: 6275.48ms | tok/sec: 83545.4 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.6ms | fwd: 2011.2ms | loss: 6.7ms | bwd: 4196.7ms | clip: 0.0ms | optim: 49.4ms
step   389 | loss: 1.504077 | lr 1.0066e-04 | norm: 0.1462 | dt: 6274.49ms | tok/sec: 83558.7 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2006.9ms | loss: 6.5ms | bwd: 4199.6ms | clip: 0.0ms | optim: 50.1ms
step   390 | loss: 1.353893 | lr 1.0055e-04 | norm: 0.1480 | dt: 6272.14ms | tok/sec: 83590.0 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2005.1ms | loss: 6.5ms | bwd: 4199.6ms | clip: 0.0ms | optim: 49.4ms
step   391 | loss: 1.422655 | lr 1.0044e-04 | norm: 0.1491 | dt: 6292.81ms | tok/sec: 83315.4 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.6ms | fwd: 2011.4ms | loss: 6.5ms | bwd: 4213.5ms | clip: 0.0ms | optim: 49.8ms
step   392 | loss: 1.420415 | lr 1.0035e-04 | norm: 0.1385 | dt: 6272.03ms | tok/sec: 83591.4 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2001.1ms | loss: 6.5ms | bwd: 4203.0ms | clip: 0.0ms | optim: 50.0ms
step   393 | loss: 1.476672 | lr 1.0027e-04 | norm: 0.1823 | dt: 6267.99ms | tok/sec: 83645.3 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.6ms | fwd: 2007.6ms | loss: 6.5ms | bwd: 4192.6ms | clip: 0.0ms | optim: 49.8ms
step   394 | loss: 1.438653 | lr 1.0020e-04 | norm: 0.1393 | dt: 6301.47ms | tok/sec: 83200.9 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.7ms | fwd: 2015.1ms | loss: 6.5ms | bwd: 4218.5ms | clip: 0.0ms | optim: 49.7ms
step   395 | loss: 1.474114 | lr 1.0014e-04 | norm: 0.1348 | dt: 6281.16ms | tok/sec: 83470.0 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.5ms
step   396 | loss: 1.463211 | lr 1.0009e-04 | norm: 0.1282 | dt: 6287.38ms | tok/sec: 83387.3 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2010.8ms | loss: 6.5ms | bwd: 4209.2ms | clip: 0.0ms | optim: 49.5ms
step   397 | loss: 1.430611 | lr 1.0005e-04 | norm: 0.1387 | dt: 6290.84ms | tok/sec: 83341.5 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.6ms | fwd: 2013.3ms | loss: 6.5ms | bwd: 4209.1ms | clip: 0.0ms | optim: 50.4ms
step   398 | loss: 1.443197 | lr 1.0002e-04 | norm: 0.1841 | dt: 6274.67ms | tok/sec: 83556.3 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.6ms | fwd: 2002.1ms | loss: 6.7ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.3ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_val_00000.bin
100000000
validation loss: 1.4430
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_train_00000.bin
100000000
step   399 | loss: 1.450279 | lr 1.0001e-04 | norm: 0.1535 | dt: 11349.40ms | tok/sec: 46195.2 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 9.7ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4210.8ms | clip: 0.0ms | optim: 49.7ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run15_step_400.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run15_step_200.ckpt

=== Training complete ===