BluScriptCP / logs /in2 /arm_c_train.log
Upload folder using huggingface_hub
f127a07 verified 3 months ago
101 kB
=== Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[ZeRO-1] CP_ZERO_SYMMETRIC=1: NCCL symmetric memory mode (NVLS)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
  d_model/layers  : 1536 / 36
  q/kv heads      : 12 / 4  (G=3, hd=128)
  ffn_hidden      : 4096 (SwiGLU)
  context_length  : 16384
  vocab_size      : 50304  tying=0
  B x T           : 4 x 16384
  world_size      : 4  (dp=1 x cp=4)
  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
  global_batch    : 524288  (grad_accum=8)
  optimizer       : Muon (2-D body) + AdamW (wte + 1-D)  [ZeRO-1]  muon_lr=0.001000
  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
  max_lr/min_lr   : 0.001 / 0.0001
  warmup/steps    : 1 / 400  (cosine, no stable phase)
  checkpointing   : true  (ckpt_freq=200, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt  (base step 750, loss 1.28324)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 400
warmup_steps: 1
[Muon] adjust_lr=match_rms_adamw (default)  ns_steps=5  momentum=0.95
[ZeRO-1/Muon] reduce-scatter/all-gather layout: 6 bucket(s), 12 collectives/step, shard 226492416 elems/rank, padding 0.000000% (flat buffer 1728.000000 MB)
[ZeRO-1/Muon] adjust_lr=match_rms_adamw | params=905969664 (216 orthogonalized / 0 AdamW) muon_elems=905969664 | W=4 owned=226492416 (42 tensors) | master+momentum: 1728.000000 MB/GPU
[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype)  symmetric=ON  overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
[ZeRO-1 v2] params=77266944 padded=77266944 W=4 shard=19316736 | 3 buckets of 50.000000 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 221.062500 MB (vs 884.250000 MB unsharded)
[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype)  symmetric=ON  overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
[ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50.000000 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.409668 MB (vs 885.638672 MB unsharded)
[ZeRO-1] variant=v2 reduce-scatter/all-gather  groups=3  [muon:bfloat16/905M]  [adamw:bfloat16/77266944]  [adamw:float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 3 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log16.csv (run 16)
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run16_step_<N>.ckpt (every 200 steps, keep 1)
[re-warmup] anchored at step 0 for 100 steps -> peak 0.000300 (0.300000 x max_lr), then cosine-decay to 0.000100 | resuming at step 0 (mid-ramp)

Starting training...

[CP ring P2P] OVERLAP: dedicated non-blocking stream
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_c/in2c_val_00000.bin
100000000
validation loss: 1.8887
[CP bwd] overlap=ON
step     0 | loss: 1.920389 | lr 3.0000e-06 | norm: 1.1395 | dt: 12087.19ms | tok/sec: 43375.5 | mem: 107213MB | Time Left: 01 hrs : 20 mins
  [TIMING] data: 3.0ms | fwd: 2214.7ms | loss: 6.5ms | bwd: 4170.4ms | clip: 0.0ms | optim: 62.3ms
step     1 | loss: 1.898924 | lr 6.0000e-06 | norm: 1.2230 | dt: 6430.02ms | tok/sec: 81537.5 | mem: 108909MB | Time Left: 00 hrs : 42 mins
  [TIMING] data: 3.3ms | fwd: 2200.8ms | loss: 6.4ms | bwd: 4162.1ms | clip: 0.0ms | optim: 49.5ms
step     2 | loss: 1.910729 | lr 9.0000e-06 | norm: 1.2476 | dt: 6422.09ms | tok/sec: 81638.3 | mem: 108909MB | Time Left: 00 hrs : 42 mins
  [TIMING] data: 3.7ms | fwd: 2199.9ms | loss: 6.5ms | bwd: 4154.5ms | clip: 0.0ms | optim: 49.6ms
step     3 | loss: 1.936255 | lr 1.2000e-05 | norm: 1.1875 | dt: 6199.39ms | tok/sec: 84570.9 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.6ms | fwd: 1987.7ms | loss: 6.5ms | bwd: 4144.4ms | clip: 0.0ms | optim: 49.3ms
step     4 | loss: 1.905165 | lr 1.5000e-05 | norm: 1.1123 | dt: 6218.49ms | tok/sec: 84311.1 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 1991.7ms | loss: 6.5ms | bwd: 4159.6ms | clip: 0.0ms | optim: 49.4ms
step     5 | loss: 1.966639 | lr 1.8000e-05 | norm: 1.1160 | dt: 6246.72ms | tok/sec: 83930.1 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.5ms | fwd: 2002.0ms | loss: 6.5ms | bwd: 4177.7ms | clip: 0.0ms | optim: 49.2ms
step     6 | loss: 1.903948 | lr 2.1000e-05 | norm: 1.0296 | dt: 6286.89ms | tok/sec: 83393.8 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.4ms | fwd: 2011.5ms | loss: 6.5ms | bwd: 4208.1ms | clip: 0.0ms | optim: 49.6ms
step     7 | loss: 1.854514 | lr 2.4000e-05 | norm: 0.9822 | dt: 6349.48ms | tok/sec: 82571.9 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.6ms | fwd: 2015.7ms | loss: 6.6ms | bwd: 4265.9ms | clip: 0.0ms | optim: 49.8ms
step     8 | loss: 1.830381 | lr 2.7000e-05 | norm: 0.9207 | dt: 6336.55ms | tok/sec: 82740.3 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.8ms | fwd: 2021.5ms | loss: 6.6ms | bwd: 4247.6ms | clip: 0.0ms | optim: 49.3ms
step     9 | loss: 1.852779 | lr 3.0000e-05 | norm: 0.9291 | dt: 6340.36ms | tok/sec: 82690.5 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.6ms | fwd: 2021.6ms | loss: 6.5ms | bwd: 4251.2ms | clip: 0.0ms | optim: 49.5ms
step    10 | loss: 1.805994 | lr 3.3000e-05 | norm: 0.7259 | dt: 6366.81ms | tok/sec: 82347.0 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.6ms | fwd: 2021.5ms | loss: 6.6ms | bwd: 4277.9ms | clip: 0.0ms | optim: 49.2ms
step    11 | loss: 1.792487 | lr 3.6000e-05 | norm: 1.2026 | dt: 6340.82ms | tok/sec: 82684.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
  [TIMING] data: 3.4ms | fwd: 2021.2ms | loss: 6.5ms | bwd: 4252.4ms | clip: 0.0ms | optim: 49.5ms
step    12 | loss: 1.784026 | lr 3.9000e-05 | norm: 1.0447 | dt: 6326.62ms | tok/sec: 82870.1 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.6ms | fwd: 2018.6ms | loss: 6.5ms | bwd: 4240.7ms | clip: 0.0ms | optim: 49.3ms
step    13 | loss: 1.786181 | lr 4.2000e-05 | norm: 1.3345 | dt: 6334.02ms | tok/sec: 82773.3 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2015.8ms | loss: 6.5ms | bwd: 4251.0ms | clip: 0.0ms | optim: 49.4ms
step    14 | loss: 1.810337 | lr 4.5000e-05 | norm: 1.2511 | dt: 6310.79ms | tok/sec: 83078.1 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2009.2ms | loss: 6.5ms | bwd: 4234.3ms | clip: 0.0ms | optim: 49.3ms
step    15 | loss: 1.794225 | lr 4.8000e-05 | norm: 1.1307 | dt: 6311.36ms | tok/sec: 83070.5 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2017.4ms | loss: 6.5ms | bwd: 4226.5ms | clip: 0.0ms | optim: 49.4ms
step    16 | loss: 1.766993 | lr 5.1000e-05 | norm: 1.0209 | dt: 6294.85ms | tok/sec: 83288.3 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.4ms | fwd: 2008.0ms | loss: 6.5ms | bwd: 4219.6ms | clip: 0.0ms | optim: 49.5ms
step    17 | loss: 1.712914 | lr 5.4000e-05 | norm: 0.9423 | dt: 6280.86ms | tok/sec: 83473.9 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2006.9ms | loss: 6.5ms | bwd: 4206.8ms | clip: 0.0ms | optim: 49.4ms
step    18 | loss: 1.730556 | lr 5.7000e-05 | norm: 0.8981 | dt: 6293.78ms | tok/sec: 83302.6 | mem: 108909MB | Time Left: 00 hrs : 40 mins
  [TIMING] data: 3.5ms | fwd: 2011.5ms | loss: 6.5ms | bwd: 4214.6ms | clip: 0.0ms | optim: 49.8ms
step    19 | loss: 1.743095 | lr 6.0000e-05 | norm: 0.8300 | dt: 6297.61ms | tok/sec: 83251.9 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4216.6ms | clip: 0.0ms | optim: 49.4ms
step    20 | loss: 1.717104 | lr 6.3000e-05 | norm: 0.7612 | dt: 6303.12ms | tok/sec: 83179.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2014.7ms | loss: 6.5ms | bwd: 4220.8ms | clip: 0.0ms | optim: 49.8ms
step    21 | loss: 1.766137 | lr 6.6000e-05 | norm: 0.6331 | dt: 6287.98ms | tok/sec: 83379.4 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.4ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4208.8ms | clip: 0.0ms | optim: 49.4ms
step    22 | loss: 1.696655 | lr 6.9000e-05 | norm: 0.8694 | dt: 6279.02ms | tok/sec: 83498.4 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2016.3ms | loss: 6.5ms | bwd: 4195.6ms | clip: 0.0ms | optim: 49.2ms
step    23 | loss: 1.706770 | lr 7.2000e-05 | norm: 0.8840 | dt: 6317.23ms | tok/sec: 82993.4 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2020.2ms | loss: 6.5ms | bwd: 4229.5ms | clip: 0.0ms | optim: 49.6ms
step    24 | loss: 1.671199 | lr 7.5000e-05 | norm: 1.0508 | dt: 6281.40ms | tok/sec: 83466.7 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4198.6ms | clip: 0.0ms | optim: 51.6ms
step    25 | loss: 1.738687 | lr 7.8000e-05 | norm: 0.6771 | dt: 6297.03ms | tok/sec: 83259.6 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4221.0ms | clip: 0.0ms | optim: 49.5ms
step    26 | loss: 1.650625 | lr 8.1000e-05 | norm: 0.8301 | dt: 6303.68ms | tok/sec: 83171.7 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.4ms | fwd: 2012.4ms | loss: 6.5ms | bwd: 4224.2ms | clip: 0.0ms | optim: 49.3ms
step    27 | loss: 1.653422 | lr 8.4000e-05 | norm: 0.6288 | dt: 6307.32ms | tok/sec: 83123.8 | mem: 108909MB | Time Left: 00 hrs : 39 mins
  [TIMING] data: 3.5ms | fwd: 2011.8ms | loss: 6.5ms | bwd: 4228.3ms | clip: 0.0ms | optim: 49.4ms
step    28 | loss: 1.690652 | lr 8.7000e-05 | norm: 0.6114 | dt: 6268.65ms | tok/sec: 83636.6 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4190.2ms | clip: 0.0ms | optim: 49.3ms
step    29 | loss: 1.663217 | lr 9.0000e-05 | norm: 0.5733 | dt: 6294.98ms | tok/sec: 83286.7 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2006.3ms | loss: 6.5ms | bwd: 4221.5ms | clip: 0.0ms | optim: 49.3ms
step    30 | loss: 1.632603 | lr 9.3000e-05 | norm: 0.8687 | dt: 6297.43ms | tok/sec: 83254.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.6ms | fwd: 2015.2ms | loss: 6.5ms | bwd: 4214.9ms | clip: 0.0ms | optim: 49.4ms
step    31 | loss: 1.700583 | lr 9.6000e-05 | norm: 0.5644 | dt: 6303.32ms | tok/sec: 83176.5 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.4ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4226.9ms | clip: 0.0ms | optim: 49.8ms
step    32 | loss: 1.607806 | lr 9.9000e-05 | norm: 0.6609 | dt: 6296.45ms | tok/sec: 83267.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2017.6ms | loss: 6.5ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.7ms
step    33 | loss: 1.657655 | lr 1.0200e-04 | norm: 0.6748 | dt: 6273.92ms | tok/sec: 83566.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2003.9ms | loss: 6.5ms | bwd: 4201.4ms | clip: 0.0ms | optim: 50.6ms
step    34 | loss: 1.647877 | lr 1.0500e-04 | norm: 0.6801 | dt: 6278.95ms | tok/sec: 83499.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2007.0ms | loss: 6.5ms | bwd: 4204.8ms | clip: 0.0ms | optim: 49.2ms
step    35 | loss: 1.658700 | lr 1.0800e-04 | norm: 0.6870 | dt: 6296.71ms | tok/sec: 83263.8 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.6ms | fwd: 2007.8ms | loss: 6.5ms | bwd: 4221.7ms | clip: 0.0ms | optim: 49.2ms
step    36 | loss: 1.630579 | lr 1.1100e-04 | norm: 0.6784 | dt: 6275.14ms | tok/sec: 83550.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2006.6ms | loss: 6.5ms | bwd: 4201.3ms | clip: 0.0ms | optim: 49.3ms
step    37 | loss: 1.625483 | lr 1.1400e-04 | norm: 0.5692 | dt: 6291.14ms | tok/sec: 83337.5 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.6ms
step    38 | loss: 1.620134 | lr 1.1700e-04 | norm: 0.6733 | dt: 6307.12ms | tok/sec: 83126.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
  [TIMING] data: 3.5ms | fwd: 2012.4ms | loss: 6.5ms | bwd: 4227.5ms | clip: 0.0ms | optim: 49.4ms
step    39 | loss: 1.643586 | lr 1.2000e-04 | norm: 0.9255 | dt: 6291.83ms | tok/sec: 83328.4 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.6ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4215.6ms | clip: 0.0ms | optim: 49.4ms
step    40 | loss: 1.600184 | lr 1.2300e-04 | norm: 0.5724 | dt: 6285.14ms | tok/sec: 83417.1 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2012.9ms | loss: 6.5ms | bwd: 4205.0ms | clip: 0.0ms | optim: 49.5ms
step    41 | loss: 1.581100 | lr 1.2600e-04 | norm: 0.7044 | dt: 6286.83ms | tok/sec: 83394.7 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.6ms | fwd: 2008.6ms | loss: 6.5ms | bwd: 4211.0ms | clip: 0.0ms | optim: 49.3ms
step    42 | loss: 1.625179 | lr 1.2900e-04 | norm: 0.6146 | dt: 6282.81ms | tok/sec: 83448.0 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.4ms | fwd: 2014.3ms | loss: 6.5ms | bwd: 4201.5ms | clip: 0.0ms | optim: 49.3ms
step    43 | loss: 1.624955 | lr 1.3200e-04 | norm: 0.6602 | dt: 6305.45ms | tok/sec: 83148.4 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2005.0ms | loss: 6.5ms | bwd: 4233.1ms | clip: 0.0ms | optim: 49.5ms
step    44 | loss: 1.554927 | lr 1.3500e-04 | norm: 0.7843 | dt: 6285.44ms | tok/sec: 83413.1 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2014.3ms | loss: 6.5ms | bwd: 4203.8ms | clip: 0.0ms | optim: 49.5ms
step    45 | loss: 1.633413 | lr 1.3800e-04 | norm: 0.7982 | dt: 6297.85ms | tok/sec: 83248.7 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.6ms | fwd: 2016.2ms | loss: 6.5ms | bwd: 4214.6ms | clip: 0.0ms | optim: 49.1ms
step    46 | loss: 1.608072 | lr 1.4100e-04 | norm: 0.7168 | dt: 6302.35ms | tok/sec: 83189.2 | mem: 108909MB | Time Left: 00 hrs : 37 mins
  [TIMING] data: 3.5ms | fwd: 2017.3ms | loss: 6.5ms | bwd: 4217.8ms | clip: 0.0ms | optim: 49.3ms
step    47 | loss: 1.637407 | lr 1.4400e-04 | norm: 0.7034 | dt: 6278.67ms | tok/sec: 83503.0 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2009.2ms | loss: 6.5ms | bwd: 4201.8ms | clip: 0.0ms | optim: 49.9ms
step    48 | loss: 1.633284 | lr 1.4700e-04 | norm: 0.7146 | dt: 6269.33ms | tok/sec: 83627.4 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2009.4ms | loss: 6.5ms | bwd: 4192.9ms | clip: 0.0ms | optim: 49.2ms
step    49 | loss: 1.607582 | lr 1.5000e-04 | norm: 0.6922 | dt: 6286.33ms | tok/sec: 83401.3 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2016.8ms | loss: 6.5ms | bwd: 4202.4ms | clip: 0.0ms | optim: 49.2ms
step    50 | loss: 1.638627 | lr 1.5300e-04 | norm: 0.6730 | dt: 6292.76ms | tok/sec: 83316.0 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4212.6ms | clip: 0.0ms | optim: 49.5ms
step    51 | loss: 1.655446 | lr 1.5600e-04 | norm: 0.7812 | dt: 6284.55ms | tok/sec: 83424.9 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2008.1ms | loss: 6.5ms | bwd: 4209.1ms | clip: 0.0ms | optim: 49.5ms
step    52 | loss: 1.661532 | lr 1.5900e-04 | norm: 0.7279 | dt: 6270.71ms | tok/sec: 83609.1 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.6ms | fwd: 2003.0ms | loss: 6.5ms | bwd: 4200.2ms | clip: 0.0ms | optim: 49.6ms
step    53 | loss: 1.579630 | lr 1.6200e-04 | norm: 0.6092 | dt: 6299.81ms | tok/sec: 83222.8 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.5ms | bwd: 4222.5ms | clip: 0.0ms | optim: 49.2ms
step    54 | loss: 1.591753 | lr 1.6500e-04 | norm: 0.7526 | dt: 6278.86ms | tok/sec: 83500.5 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.4ms | fwd: 2010.2ms | loss: 6.5ms | bwd: 4201.1ms | clip: 0.0ms | optim: 49.8ms
step    55 | loss: 1.668421 | lr 1.6800e-04 | norm: 0.6737 | dt: 6291.54ms | tok/sec: 83332.2 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.4ms | fwd: 2016.2ms | loss: 6.5ms | bwd: 4208.1ms | clip: 0.0ms | optim: 49.4ms
step    56 | loss: 1.633100 | lr 1.7100e-04 | norm: 0.7937 | dt: 6288.66ms | tok/sec: 83370.4 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2013.9ms | loss: 6.5ms | bwd: 4207.4ms | clip: 0.0ms | optim: 49.6ms
step    57 | loss: 1.570646 | lr 1.7400e-04 | norm: 0.6318 | dt: 6303.68ms | tok/sec: 83171.7 | mem: 108909MB | Time Left: 00 hrs : 36 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4228.1ms | clip: 0.0ms | optim: 49.6ms
step    58 | loss: 1.588581 | lr 1.7700e-04 | norm: 0.9173 | dt: 6294.35ms | tok/sec: 83295.0 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.4ms | fwd: 2022.6ms | loss: 6.5ms | bwd: 4204.2ms | clip: 0.0ms | optim: 49.7ms
step    59 | loss: 1.584494 | lr 1.8000e-04 | norm: 0.6724 | dt: 6281.98ms | tok/sec: 83459.0 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.5ms | bwd: 4204.7ms | clip: 0.0ms | optim: 49.3ms
step    60 | loss: 1.626362 | lr 1.8300e-04 | norm: 0.7723 | dt: 6285.15ms | tok/sec: 83417.0 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2010.6ms | loss: 6.5ms | bwd: 4207.0ms | clip: 0.0ms | optim: 49.6ms
step    61 | loss: 1.553325 | lr 1.8600e-04 | norm: 0.7159 | dt: 6289.06ms | tok/sec: 83365.1 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2010.8ms | loss: 6.5ms | bwd: 4211.0ms | clip: 0.0ms | optim: 49.3ms
step    62 | loss: 1.592063 | lr 1.8900e-04 | norm: 0.8843 | dt: 6283.57ms | tok/sec: 83437.9 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2010.4ms | loss: 6.5ms | bwd: 4206.1ms | clip: 0.0ms | optim: 49.2ms
step    63 | loss: 1.590768 | lr 1.9200e-04 | norm: 0.9069 | dt: 6277.87ms | tok/sec: 83513.7 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.6ms | fwd: 2011.1ms | loss: 6.5ms | bwd: 4199.2ms | clip: 0.0ms | optim: 49.6ms
step    64 | loss: 1.556218 | lr 1.9500e-04 | norm: 0.7437 | dt: 6274.04ms | tok/sec: 83564.6 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.4ms | fwd: 2004.7ms | loss: 6.5ms | bwd: 4202.1ms | clip: 0.0ms | optim: 49.4ms
step    65 | loss: 1.603228 | lr 1.9800e-04 | norm: 0.7229 | dt: 6288.73ms | tok/sec: 83369.4 | mem: 108909MB | Time Left: 00 hrs : 35 mins
  [TIMING] data: 3.5ms | fwd: 2014.3ms | loss: 6.5ms | bwd: 4206.9ms | clip: 0.0ms | optim: 49.7ms
step    66 | loss: 1.590720 | lr 2.0100e-04 | norm: 0.8309 | dt: 6287.19ms | tok/sec: 83389.9 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4210.2ms | clip: 0.0ms | optim: 49.2ms
step    67 | loss: 1.621471 | lr 2.0400e-04 | norm: 0.8639 | dt: 6284.71ms | tok/sec: 83422.8 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.6ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4209.5ms | clip: 0.0ms | optim: 49.5ms
step    68 | loss: 1.598285 | lr 2.0700e-04 | norm: 0.7567 | dt: 6292.59ms | tok/sec: 83318.3 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.4ms | fwd: 2010.7ms | loss: 6.5ms | bwd: 4214.9ms | clip: 0.0ms | optim: 49.2ms
step    69 | loss: 1.633740 | lr 2.1000e-04 | norm: 0.7597 | dt: 6282.51ms | tok/sec: 83452.0 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2006.3ms | loss: 6.5ms | bwd: 4209.0ms | clip: 0.0ms | optim: 49.3ms
step    70 | loss: 1.607149 | lr 2.1300e-04 | norm: 0.7877 | dt: 6275.58ms | tok/sec: 83544.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4202.1ms | clip: 0.0ms | optim: 49.6ms
step    71 | loss: 1.608349 | lr 2.1600e-04 | norm: 0.8037 | dt: 6276.17ms | tok/sec: 83536.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4200.3ms | clip: 0.0ms | optim: 49.7ms
step    72 | loss: 1.543556 | lr 2.1900e-04 | norm: 0.7161 | dt: 6293.55ms | tok/sec: 83305.6 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.5ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4219.1ms | clip: 0.0ms | optim: 49.2ms
step    73 | loss: 1.615947 | lr 2.2200e-04 | norm: 1.6646 | dt: 6290.37ms | tok/sec: 83347.7 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.6ms | fwd: 2011.1ms | loss: 6.5ms | bwd: 4212.0ms | clip: 0.0ms | optim: 49.3ms
step    74 | loss: 1.593432 | lr 2.2500e-04 | norm: 0.8500 | dt: 6296.09ms | tok/sec: 83272.0 | mem: 108909MB | Time Left: 00 hrs : 34 mins
  [TIMING] data: 3.4ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4216.5ms | clip: 0.0ms | optim: 49.9ms
step    75 | loss: 1.608452 | lr 2.2800e-04 | norm: 0.8987 | dt: 6265.80ms | tok/sec: 83674.5 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2008.6ms | loss: 6.5ms | bwd: 4189.6ms | clip: 0.0ms | optim: 49.6ms
step    76 | loss: 1.554039 | lr 2.3100e-04 | norm: 0.8327 | dt: 6286.78ms | tok/sec: 83395.4 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.4ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4213.1ms | clip: 0.0ms | optim: 50.0ms
step    77 | loss: 1.623400 | lr 2.3400e-04 | norm: 0.7053 | dt: 6280.66ms | tok/sec: 83476.5 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4204.0ms | clip: 0.0ms | optim: 50.0ms
step    78 | loss: 1.591175 | lr 2.3700e-04 | norm: 0.8684 | dt: 6319.61ms | tok/sec: 82962.1 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4240.4ms | clip: 0.0ms | optim: 49.6ms
step    79 | loss: 1.597531 | lr 2.4000e-04 | norm: 0.9475 | dt: 6289.85ms | tok/sec: 83354.6 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2010.1ms | loss: 6.5ms | bwd: 4212.7ms | clip: 0.0ms | optim: 49.3ms
step    80 | loss: 1.595922 | lr 2.4300e-04 | norm: 0.7522 | dt: 6287.96ms | tok/sec: 83379.7 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4211.6ms | clip: 0.0ms | optim: 49.5ms
step    81 | loss: 1.639384 | lr 2.4600e-04 | norm: 0.7476 | dt: 6281.07ms | tok/sec: 83471.1 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4204.6ms | clip: 0.0ms | optim: 49.6ms
step    82 | loss: 1.666987 | lr 2.4900e-04 | norm: 0.7452 | dt: 6306.55ms | tok/sec: 83133.9 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2014.3ms | loss: 6.5ms | bwd: 4224.6ms | clip: 0.0ms | optim: 49.8ms
step    83 | loss: 1.594446 | lr 2.5200e-04 | norm: 0.9150 | dt: 6286.68ms | tok/sec: 83396.6 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2008.9ms | loss: 6.5ms | bwd: 4210.5ms | clip: 0.0ms | optim: 49.4ms
step    84 | loss: 1.602896 | lr 2.5500e-04 | norm: 0.9495 | dt: 6289.85ms | tok/sec: 83354.6 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.5ms | fwd: 2008.0ms | loss: 6.5ms | bwd: 4214.5ms | clip: 0.0ms | optim: 49.5ms
step    85 | loss: 1.603970 | lr 2.5800e-04 | norm: 0.7217 | dt: 6293.36ms | tok/sec: 83308.1 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.6ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4218.4ms | clip: 0.0ms | optim: 49.3ms
step    86 | loss: 1.606759 | lr 2.6100e-04 | norm: 0.7786 | dt: 6306.46ms | tok/sec: 83135.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
  [TIMING] data: 3.4ms | fwd: 2014.9ms | loss: 6.5ms | bwd: 4224.1ms | clip: 0.0ms | optim: 49.8ms
step    87 | loss: 1.669444 | lr 2.6400e-04 | norm: 0.8626 | dt: 6292.05ms | tok/sec: 83325.4 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2016.0ms | loss: 6.5ms | bwd: 4208.8ms | clip: 0.0ms | optim: 49.4ms
step    88 | loss: 1.671355 | lr 2.6700e-04 | norm: 0.7404 | dt: 6303.24ms | tok/sec: 83177.5 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4223.9ms | clip: 0.0ms | optim: 49.5ms
step    89 | loss: 1.628623 | lr 2.7000e-04 | norm: 0.7503 | dt: 6272.73ms | tok/sec: 83582.2 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2009.7ms | loss: 6.5ms | bwd: 4195.5ms | clip: 0.0ms | optim: 49.6ms
step    90 | loss: 1.686958 | lr 2.7300e-04 | norm: 0.7960 | dt: 6276.40ms | tok/sec: 83533.2 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2008.1ms | loss: 6.5ms | bwd: 4200.9ms | clip: 0.0ms | optim: 49.5ms
step    91 | loss: 1.633542 | lr 2.7600e-04 | norm: 0.6971 | dt: 6290.41ms | tok/sec: 83347.1 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4213.2ms | clip: 0.0ms | optim: 49.5ms
step    92 | loss: 1.640320 | lr 2.7900e-04 | norm: 0.7456 | dt: 6294.31ms | tok/sec: 83295.6 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.4ms | fwd: 2014.1ms | loss: 6.5ms | bwd: 4213.0ms | clip: 0.0ms | optim: 49.3ms
step    93 | loss: 1.669694 | lr 2.8200e-04 | norm: 0.6870 | dt: 6287.89ms | tok/sec: 83380.5 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4209.2ms | clip: 0.0ms | optim: 49.5ms
step    94 | loss: 1.549692 | lr 2.8500e-04 | norm: 0.8955 | dt: 6314.26ms | tok/sec: 83032.4 | mem: 108909MB | Time Left: 00 hrs : 32 mins
  [TIMING] data: 3.5ms | fwd: 2016.5ms | loss: 6.5ms | bwd: 4230.5ms | clip: 0.0ms | optim: 49.5ms
step    95 | loss: 1.625757 | lr 2.8800e-04 | norm: 0.7616 | dt: 6294.81ms | tok/sec: 83288.9 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2002.7ms | loss: 6.5ms | bwd: 4224.7ms | clip: 0.0ms | optim: 49.5ms
step    96 | loss: 1.619722 | lr 2.9100e-04 | norm: 0.8142 | dt: 6276.13ms | tok/sec: 83536.9 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2011.5ms | loss: 6.5ms | bwd: 4197.4ms | clip: 0.0ms | optim: 49.3ms
step    97 | loss: 1.619199 | lr 2.9400e-04 | norm: 0.7496 | dt: 6281.04ms | tok/sec: 83471.5 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4202.3ms | clip: 0.0ms | optim: 49.4ms
step    98 | loss: 1.612973 | lr 2.9700e-04 | norm: 0.8555 | dt: 6308.29ms | tok/sec: 83111.0 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2011.6ms | loss: 6.5ms | bwd: 4229.5ms | clip: 0.0ms | optim: 49.4ms
step    99 | loss: 1.639371 | lr 3.0000e-04 | norm: 0.8418 | dt: 6279.99ms | tok/sec: 83485.4 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.5ms | bwd: 4202.6ms | clip: 0.0ms | optim: 49.2ms
step   100 | loss: 1.650524 | lr 3.0000e-04 | norm: 0.7211 | dt: 6272.09ms | tok/sec: 83590.7 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2009.5ms | loss: 6.5ms | bwd: 4195.1ms | clip: 0.0ms | optim: 49.6ms
step   101 | loss: 1.682680 | lr 2.9999e-04 | norm: 0.6914 | dt: 6289.92ms | tok/sec: 83353.7 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.6ms | fwd: 2012.4ms | loss: 6.5ms | bwd: 4210.2ms | clip: 0.0ms | optim: 49.4ms
step   102 | loss: 1.590985 | lr 2.9998e-04 | norm: 0.6628 | dt: 6286.01ms | tok/sec: 83405.6 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.5ms | fwd: 2013.0ms | loss: 6.5ms | bwd: 4205.7ms | clip: 0.0ms | optim: 49.4ms
step   103 | loss: 1.650105 | lr 2.9995e-04 | norm: 0.7122 | dt: 6275.74ms | tok/sec: 83542.0 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.4ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4198.9ms | clip: 0.0ms | optim: 49.2ms
step   104 | loss: 1.711926 | lr 2.9991e-04 | norm: 0.7817 | dt: 6288.87ms | tok/sec: 83367.6 | mem: 108909MB | Time Left: 00 hrs : 31 mins
  [TIMING] data: 3.6ms | fwd: 2009.1ms | loss: 6.5ms | bwd: 4212.2ms | clip: 0.0ms | optim: 49.6ms
step   105 | loss: 1.633166 | lr 2.9986e-04 | norm: 0.8719 | dt: 6285.73ms | tok/sec: 83409.3 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.6ms | fwd: 2017.3ms | loss: 6.5ms | bwd: 4201.1ms | clip: 0.0ms | optim: 49.4ms
step   106 | loss: 1.636873 | lr 2.9980e-04 | norm: 0.8642 | dt: 6286.19ms | tok/sec: 83403.1 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.5ms | fwd: 2009.5ms | loss: 6.5ms | bwd: 4208.9ms | clip: 0.0ms | optim: 49.9ms
step   107 | loss: 1.695892 | lr 2.9973e-04 | norm: 0.7646 | dt: 6297.42ms | tok/sec: 83254.4 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.5ms | fwd: 2018.9ms | loss: 6.5ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.4ms
step   108 | loss: 1.586758 | lr 2.9965e-04 | norm: 0.7477 | dt: 6267.76ms | tok/sec: 83648.3 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.4ms | fwd: 2006.3ms | loss: 6.6ms | bwd: 4193.7ms | clip: 0.0ms | optim: 49.8ms
step   109 | loss: 1.628801 | lr 2.9956e-04 | norm: 1.4375 | dt: 6294.00ms | tok/sec: 83299.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.6ms | fwd: 2016.5ms | loss: 6.5ms | bwd: 4210.4ms | clip: 0.0ms | optim: 49.2ms
step   110 | loss: 1.648318 | lr 2.9945e-04 | norm: 0.6633 | dt: 6269.97ms | tok/sec: 83619.0 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.6ms | fwd: 2009.1ms | loss: 6.5ms | bwd: 4193.5ms | clip: 0.0ms | optim: 49.5ms
step   111 | loss: 1.639518 | lr 2.9934e-04 | norm: 0.6966 | dt: 6292.72ms | tok/sec: 83316.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.4ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4220.5ms | clip: 0.0ms | optim: 49.8ms
step   112 | loss: 1.612578 | lr 2.9921e-04 | norm: 0.7118 | dt: 6279.30ms | tok/sec: 83494.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
  [TIMING] data: 3.5ms | fwd: 2005.6ms | loss: 6.5ms | bwd: 4206.5ms | clip: 0.0ms | optim: 49.4ms
step   113 | loss: 1.601687 | lr 2.9907e-04 | norm: 0.7423 | dt: 6271.61ms | tok/sec: 83597.1 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.4ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4198.3ms | clip: 0.0ms | optim: 49.5ms
step   114 | loss: 1.636327 | lr 2.9893e-04 | norm: 0.8447 | dt: 6291.10ms | tok/sec: 83338.0 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.5ms | bwd: 4213.3ms | clip: 0.0ms | optim: 49.7ms
step   115 | loss: 1.621055 | lr 2.9877e-04 | norm: 0.7664 | dt: 6301.70ms | tok/sec: 83197.9 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2016.8ms | loss: 6.5ms | bwd: 4217.3ms | clip: 0.0ms | optim: 49.8ms
step   116 | loss: 1.569871 | lr 2.9860e-04 | norm: 0.7906 | dt: 6292.31ms | tok/sec: 83322.0 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4211.3ms | clip: 0.0ms | optim: 49.7ms
step   117 | loss: 1.585612 | lr 2.9842e-04 | norm: 0.6726 | dt: 6304.46ms | tok/sec: 83161.4 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4228.9ms | clip: 0.0ms | optim: 49.4ms
step   118 | loss: 1.648591 | lr 2.9823e-04 | norm: 0.6768 | dt: 6282.30ms | tok/sec: 83454.7 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.6ms | fwd: 2003.7ms | loss: 6.5ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.4ms
step   119 | loss: 1.658161 | lr 2.9803e-04 | norm: 0.7386 | dt: 6289.52ms | tok/sec: 83359.0 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2012.5ms | loss: 6.5ms | bwd: 4209.8ms | clip: 0.0ms | optim: 49.3ms
step   120 | loss: 1.605183 | lr 2.9781e-04 | norm: 0.8041 | dt: 6311.63ms | tok/sec: 83067.0 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.4ms | fwd: 2017.7ms | loss: 6.5ms | bwd: 4226.4ms | clip: 0.0ms | optim: 49.7ms
step   121 | loss: 1.623778 | lr 2.9759e-04 | norm: 0.7783 | dt: 6273.78ms | tok/sec: 83568.1 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.7ms | fwd: 2011.8ms | loss: 6.5ms | bwd: 4194.5ms | clip: 0.0ms | optim: 49.5ms
step   122 | loss: 1.673925 | lr 2.9736e-04 | norm: 0.7526 | dt: 6305.68ms | tok/sec: 83145.4 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2016.9ms | loss: 6.5ms | bwd: 4221.6ms | clip: 0.0ms | optim: 49.3ms
step   123 | loss: 1.639915 | lr 2.9711e-04 | norm: 0.7211 | dt: 6296.62ms | tok/sec: 83264.9 | mem: 108909MB | Time Left: 00 hrs : 29 mins
  [TIMING] data: 3.5ms | fwd: 2008.9ms | loss: 6.5ms | bwd: 4220.7ms | clip: 0.0ms | optim: 49.2ms
step   124 | loss: 1.671284 | lr 2.9686e-04 | norm: 0.6787 | dt: 6302.27ms | tok/sec: 83190.4 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2017.2ms | loss: 6.5ms | bwd: 4217.5ms | clip: 0.0ms | optim: 49.7ms
step   125 | loss: 1.624825 | lr 2.9659e-04 | norm: 0.7628 | dt: 6299.13ms | tok/sec: 83231.8 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2008.9ms | loss: 6.5ms | bwd: 4222.7ms | clip: 0.0ms | optim: 49.6ms
step   126 | loss: 1.638936 | lr 2.9632e-04 | norm: 0.6814 | dt: 6300.01ms | tok/sec: 83220.2 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.6ms | fwd: 2008.9ms | loss: 6.5ms | bwd: 4223.6ms | clip: 0.0ms | optim: 49.6ms
step   127 | loss: 1.680216 | lr 2.9603e-04 | norm: 0.7290 | dt: 6294.61ms | tok/sec: 83291.6 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4215.0ms | clip: 0.0ms | optim: 49.7ms
step   128 | loss: 1.691657 | lr 2.9573e-04 | norm: 0.7967 | dt: 6298.73ms | tok/sec: 83237.1 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4219.8ms | clip: 0.0ms | optim: 49.3ms
step   129 | loss: 1.649173 | lr 2.9542e-04 | norm: 0.6502 | dt: 6290.96ms | tok/sec: 83339.9 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4212.3ms | clip: 0.0ms | optim: 49.8ms
step   130 | loss: 1.763545 | lr 2.9511e-04 | norm: 0.6932 | dt: 6280.22ms | tok/sec: 83482.5 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4203.2ms | clip: 0.0ms | optim: 49.4ms
step   131 | loss: 1.656262 | lr 2.9478e-04 | norm: 0.6988 | dt: 6290.48ms | tok/sec: 83346.2 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2013.6ms | loss: 6.5ms | bwd: 4208.1ms | clip: 0.0ms | optim: 50.8ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_c/in2c_train_00001.bin
100000000
step   132 | loss: 1.647103 | lr 2.9444e-04 | norm: 0.6697 | dt: 6289.81ms | tok/sec: 83355.1 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 7.2ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4207.0ms | clip: 0.0ms | optim: 49.2ms
step   133 | loss: 1.634230 | lr 2.9409e-04 | norm: 0.5707 | dt: 6294.52ms | tok/sec: 83292.7 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.4ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4218.9ms | clip: 0.0ms | optim: 49.9ms
step   134 | loss: 1.643845 | lr 2.9373e-04 | norm: 0.6491 | dt: 6295.35ms | tok/sec: 83281.8 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2013.1ms | loss: 6.5ms | bwd: 4215.0ms | clip: 0.0ms | optim: 49.3ms
step   135 | loss: 1.668388 | lr 2.9336e-04 | norm: 0.5978 | dt: 6299.16ms | tok/sec: 83231.4 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.4ms | fwd: 2013.2ms | loss: 6.5ms | bwd: 4218.9ms | clip: 0.0ms | optim: 49.3ms
step   136 | loss: 1.626073 | lr 2.9298e-04 | norm: 0.6476 | dt: 6290.17ms | tok/sec: 83350.4 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.4ms | fwd: 2010.5ms | loss: 6.5ms | bwd: 4212.4ms | clip: 0.0ms | optim: 49.5ms
step   137 | loss: 1.610279 | lr 2.9259e-04 | norm: 0.6083 | dt: 6281.90ms | tok/sec: 83460.1 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.6ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4206.9ms | clip: 0.0ms | optim: 49.4ms
step   138 | loss: 1.675240 | lr 2.9219e-04 | norm: 0.6421 | dt: 6303.41ms | tok/sec: 83175.2 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2015.5ms | loss: 6.5ms | bwd: 4220.8ms | clip: 0.0ms | optim: 49.2ms
step   139 | loss: 1.674961 | lr 2.9178e-04 | norm: 0.5379 | dt: 6296.47ms | tok/sec: 83267.0 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2006.7ms | loss: 6.5ms | bwd: 4222.5ms | clip: 0.0ms | optim: 49.3ms
step   140 | loss: 1.685587 | lr 2.9135e-04 | norm: 0.6332 | dt: 6295.32ms | tok/sec: 83282.2 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 1998.5ms | loss: 6.5ms | bwd: 4229.6ms | clip: 0.0ms | optim: 49.4ms
step   141 | loss: 1.675855 | lr 2.9092e-04 | norm: 0.5615 | dt: 6312.64ms | tok/sec: 83053.7 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2016.6ms | loss: 6.5ms | bwd: 4228.6ms | clip: 0.0ms | optim: 49.7ms
step   142 | loss: 1.658513 | lr 2.9048e-04 | norm: 0.5219 | dt: 6300.94ms | tok/sec: 83207.9 | mem: 108909MB | Time Left: 00 hrs : 27 mins
  [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4220.1ms | clip: 0.0ms | optim: 49.5ms
step   143 | loss: 1.619050 | lr 2.9003e-04 | norm: 0.6539 | dt: 6292.49ms | tok/sec: 83319.6 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4212.3ms | clip: 0.0ms | optim: 49.7ms
step   144 | loss: 1.681830 | lr 2.8957e-04 | norm: 0.5827 | dt: 6295.91ms | tok/sec: 83274.4 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.4ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4215.5ms | clip: 0.0ms | optim: 49.2ms
step   145 | loss: 1.602408 | lr 2.8910e-04 | norm: 0.6689 | dt: 6275.10ms | tok/sec: 83550.5 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.6ms | fwd: 2009.1ms | loss: 6.5ms | bwd: 4198.3ms | clip: 0.0ms | optim: 49.9ms
step   146 | loss: 1.608207 | lr 2.8862e-04 | norm: 0.6836 | dt: 6317.82ms | tok/sec: 82985.6 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.4ms | fwd: 2017.3ms | loss: 6.5ms | bwd: 4233.2ms | clip: 0.0ms | optim: 49.6ms
step   147 | loss: 1.609954 | lr 2.8813e-04 | norm: 0.5251 | dt: 6276.65ms | tok/sec: 83529.9 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2005.8ms | loss: 6.5ms | bwd: 4203.6ms | clip: 0.0ms | optim: 49.4ms
step   148 | loss: 1.655501 | lr 2.8763e-04 | norm: 0.7405 | dt: 6284.95ms | tok/sec: 83419.5 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.6ms | fwd: 2015.3ms | loss: 6.5ms | bwd: 4202.2ms | clip: 0.0ms | optim: 49.6ms
step   149 | loss: 1.648694 | lr 2.8712e-04 | norm: 0.5504 | dt: 6303.07ms | tok/sec: 83179.8 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2012.2ms | loss: 6.5ms | bwd: 4223.4ms | clip: 0.0ms | optim: 49.7ms
step   150 | loss: 1.601120 | lr 2.8660e-04 | norm: 0.6674 | dt: 6299.37ms | tok/sec: 83228.6 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.6ms | fwd: 2016.3ms | loss: 6.5ms | bwd: 4215.7ms | clip: 0.0ms | optim: 49.3ms
step   151 | loss: 1.626890 | lr 2.8607e-04 | norm: 0.5222 | dt: 6275.48ms | tok/sec: 83545.5 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.4ms | fwd: 2014.2ms | loss: 6.5ms | bwd: 4193.5ms | clip: 0.0ms | optim: 50.0ms
step   152 | loss: 1.671792 | lr 2.8554e-04 | norm: 0.6090 | dt: 6314.39ms | tok/sec: 83030.6 | mem: 108909MB | Time Left: 00 hrs : 26 mins
  [TIMING] data: 3.5ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4233.6ms | clip: 0.0ms | optim: 49.2ms
step   153 | loss: 1.635783 | lr 2.8499e-04 | norm: 0.5830 | dt: 6283.12ms | tok/sec: 83443.9 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.5ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4210.1ms | clip: 0.0ms | optim: 49.3ms
step   154 | loss: 1.654668 | lr 2.8443e-04 | norm: 0.5301 | dt: 6304.87ms | tok/sec: 83156.1 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.6ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4222.9ms | clip: 0.0ms | optim: 49.5ms
step   155 | loss: 1.641853 | lr 2.8387e-04 | norm: 0.5492 | dt: 6284.93ms | tok/sec: 83419.9 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.6ms | fwd: 2005.5ms | loss: 6.5ms | bwd: 4212.2ms | clip: 0.0ms | optim: 49.3ms
step   156 | loss: 1.629475 | lr 2.8329e-04 | norm: 0.5482 | dt: 6290.65ms | tok/sec: 83344.0 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.4ms | fwd: 2018.0ms | loss: 6.5ms | bwd: 4205.3ms | clip: 0.0ms | optim: 49.5ms
step   157 | loss: 1.680739 | lr 2.8271e-04 | norm: 0.5257 | dt: 6298.73ms | tok/sec: 83237.1 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.6ms | fwd: 2012.0ms | loss: 6.5ms | bwd: 4219.3ms | clip: 0.0ms | optim: 49.4ms
step   158 | loss: 1.682127 | lr 2.8211e-04 | norm: 0.5934 | dt: 6280.42ms | tok/sec: 83479.8 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.6ms | fwd: 2011.1ms | loss: 6.5ms | bwd: 4201.9ms | clip: 0.0ms | optim: 49.4ms
step   159 | loss: 1.644408 | lr 2.8151e-04 | norm: 0.5308 | dt: 6280.87ms | tok/sec: 83473.7 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4205.3ms | clip: 0.0ms | optim: 49.6ms
step   160 | loss: 1.677973 | lr 2.8090e-04 | norm: 0.6353 | dt: 6291.53ms | tok/sec: 83332.3 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.4ms | fwd: 2021.0ms | loss: 6.5ms | bwd: 4203.5ms | clip: 0.0ms | optim: 49.2ms
step   161 | loss: 1.645268 | lr 2.8028e-04 | norm: 0.6444 | dt: 6297.61ms | tok/sec: 83251.9 | mem: 108909MB | Time Left: 00 hrs : 25 mins
  [TIMING] data: 3.6ms | fwd: 2011.5ms | loss: 6.5ms | bwd: 4218.9ms | clip: 0.0ms | optim: 49.2ms
step   162 | loss: 1.640342 | lr 2.7965e-04 | norm: 0.4984 | dt: 6299.42ms | tok/sec: 83227.9 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.7ms | fwd: 2007.4ms | loss: 6.5ms | bwd: 4224.6ms | clip: 0.0ms | optim: 49.4ms
step   163 | loss: 1.651720 | lr 2.7902e-04 | norm: 0.4775 | dt: 6273.27ms | tok/sec: 83574.8 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4198.2ms | clip: 0.0ms | optim: 49.3ms
step   164 | loss: 1.578010 | lr 2.7837e-04 | norm: 0.5391 | dt: 6297.37ms | tok/sec: 83255.1 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.6ms | fwd: 2017.6ms | loss: 6.5ms | bwd: 4212.1ms | clip: 0.0ms | optim: 49.8ms
step   165 | loss: 1.647337 | lr 2.7771e-04 | norm: 0.5263 | dt: 6306.13ms | tok/sec: 83139.4 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2014.4ms | loss: 6.5ms | bwd: 4224.2ms | clip: 0.0ms | optim: 49.6ms
step   166 | loss: 1.637582 | lr 2.7705e-04 | norm: 0.4101 | dt: 6288.94ms | tok/sec: 83366.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.5ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4207.2ms | clip: 0.0ms | optim: 49.4ms
step   167 | loss: 1.646919 | lr 2.7638e-04 | norm: 0.7910 | dt: 6291.36ms | tok/sec: 83334.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.4ms | fwd: 2009.7ms | loss: 6.5ms | bwd: 4214.5ms | clip: 0.0ms | optim: 49.4ms
step   168 | loss: 1.686418 | lr 2.7570e-04 | norm: 0.5273 | dt: 6302.71ms | tok/sec: 83184.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.4ms | fwd: 2008.5ms | loss: 6.7ms | bwd: 4226.9ms | clip: 0.0ms | optim: 49.2ms
step   169 | loss: 1.611556 | lr 2.7501e-04 | norm: 0.5136 | dt: 6268.76ms | tok/sec: 83635.1 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.6ms | fwd: 2010.4ms | loss: 6.5ms | bwd: 4191.0ms | clip: 0.0ms | optim: 49.4ms
step   170 | loss: 1.608316 | lr 2.7431e-04 | norm: 0.5426 | dt: 6287.29ms | tok/sec: 83388.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
  [TIMING] data: 3.4ms | fwd: 2012.6ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.4ms
step   171 | loss: 1.670778 | lr 2.7361e-04 | norm: 0.4461 | dt: 6286.97ms | tok/sec: 83392.8 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2012.6ms | loss: 6.5ms | bwd: 4207.0ms | clip: 0.0ms | optim: 49.6ms
step   172 | loss: 1.640925 | lr 2.7290e-04 | norm: 0.5423 | dt: 6301.35ms | tok/sec: 83202.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2015.5ms | loss: 6.5ms | bwd: 4218.6ms | clip: 0.0ms | optim: 49.5ms
step   173 | loss: 1.632449 | lr 2.7218e-04 | norm: 0.5062 | dt: 6293.33ms | tok/sec: 83308.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2004.6ms | loss: 6.5ms | bwd: 4221.5ms | clip: 0.0ms | optim: 49.4ms
step   174 | loss: 1.653476 | lr 2.7145e-04 | norm: 0.4979 | dt: 6304.89ms | tok/sec: 83155.8 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.6ms | fwd: 2017.4ms | loss: 6.5ms | bwd: 4220.3ms | clip: 0.0ms | optim: 49.4ms
step   175 | loss: 1.635847 | lr 2.7071e-04 | norm: 0.4910 | dt: 6286.43ms | tok/sec: 83400.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.6ms | fwd: 2015.7ms | loss: 6.5ms | bwd: 4203.3ms | clip: 0.0ms | optim: 49.4ms
step   176 | loss: 1.597414 | lr 2.6997e-04 | norm: 0.4610 | dt: 6299.98ms | tok/sec: 83220.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.4ms | fwd: 2012.2ms | loss: 6.5ms | bwd: 4220.4ms | clip: 0.0ms | optim: 49.6ms
step   177 | loss: 1.691110 | lr 2.6921e-04 | norm: 0.5593 | dt: 6303.07ms | tok/sec: 83179.8 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.6ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4225.8ms | clip: 0.0ms | optim: 49.4ms
step   178 | loss: 1.655160 | lr 2.6845e-04 | norm: 0.4862 | dt: 6288.76ms | tok/sec: 83369.1 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2010.7ms | loss: 6.5ms | bwd: 4211.1ms | clip: 0.0ms | optim: 49.2ms
step   179 | loss: 1.656607 | lr 2.6769e-04 | norm: 0.4968 | dt: 6279.24ms | tok/sec: 83495.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.5ms | fwd: 2008.3ms | loss: 6.5ms | bwd: 4203.6ms | clip: 0.0ms | optim: 49.5ms
step   180 | loss: 1.700045 | lr 2.6691e-04 | norm: 0.4740 | dt: 6300.90ms | tok/sec: 83208.5 | mem: 108909MB | Time Left: 00 hrs : 23 mins
  [TIMING] data: 3.6ms | fwd: 2005.9ms | loss: 6.5ms | bwd: 4227.6ms | clip: 0.0ms | optim: 49.5ms
step   181 | loss: 1.610477 | lr 2.6613e-04 | norm: 0.4708 | dt: 6293.84ms | tok/sec: 83301.8 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4219.2ms | clip: 0.0ms | optim: 49.5ms
step   182 | loss: 1.598330 | lr 2.6534e-04 | norm: 0.4877 | dt: 6283.17ms | tok/sec: 83443.2 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.6ms | fwd: 2014.3ms | loss: 6.5ms | bwd: 4201.6ms | clip: 0.0ms | optim: 49.3ms
step   183 | loss: 1.637440 | lr 2.6455e-04 | norm: 0.3927 | dt: 6290.67ms | tok/sec: 83343.7 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.7ms
step   184 | loss: 1.634746 | lr 2.6374e-04 | norm: 0.4604 | dt: 6286.35ms | tok/sec: 83401.1 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.6ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4207.3ms | clip: 0.0ms | optim: 49.4ms
step   185 | loss: 1.607180 | lr 2.6293e-04 | norm: 0.4000 | dt: 6276.77ms | tok/sec: 83528.3 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4200.9ms | clip: 0.0ms | optim: 49.2ms
step   186 | loss: 1.672035 | lr 2.6211e-04 | norm: 0.4745 | dt: 6299.51ms | tok/sec: 83226.7 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.4ms | fwd: 2009.4ms | loss: 6.5ms | bwd: 4223.0ms | clip: 0.0ms | optim: 49.4ms
step   187 | loss: 1.643560 | lr 2.6129e-04 | norm: 0.4298 | dt: 6289.53ms | tok/sec: 83358.8 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2010.4ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.5ms
step   188 | loss: 1.627212 | lr 2.6046e-04 | norm: 0.5720 | dt: 6297.96ms | tok/sec: 83247.2 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.7ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4217.3ms | clip: 0.0ms | optim: 49.3ms
step   189 | loss: 1.635044 | lr 2.5962e-04 | norm: 0.3458 | dt: 6289.86ms | tok/sec: 83354.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2009.1ms | loss: 6.5ms | bwd: 4213.4ms | clip: 0.0ms | optim: 49.5ms
step   190 | loss: 1.612992 | lr 2.5878e-04 | norm: 0.3639 | dt: 6289.24ms | tok/sec: 83362.7 | mem: 108909MB | Time Left: 00 hrs : 22 mins
  [TIMING] data: 3.5ms | fwd: 2017.0ms | loss: 6.5ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.2ms
step   191 | loss: 1.634282 | lr 2.5793e-04 | norm: 0.3844 | dt: 6297.32ms | tok/sec: 83255.7 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.5ms | bwd: 4219.9ms | clip: 0.0ms | optim: 49.6ms
step   192 | loss: 1.615877 | lr 2.5707e-04 | norm: 0.4302 | dt: 6292.19ms | tok/sec: 83323.6 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 2013.0ms | loss: 6.5ms | bwd: 4211.7ms | clip: 0.0ms | optim: 49.6ms
step   193 | loss: 1.611861 | lr 2.5621e-04 | norm: 0.4217 | dt: 6287.79ms | tok/sec: 83381.9 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 2013.1ms | loss: 6.5ms | bwd: 4207.5ms | clip: 0.0ms | optim: 49.3ms
step   194 | loss: 1.623422 | lr 2.5534e-04 | norm: 0.4475 | dt: 6273.63ms | tok/sec: 83570.2 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4198.5ms | clip: 0.0ms | optim: 49.9ms
step   195 | loss: 1.610478 | lr 2.5446e-04 | norm: 0.3793 | dt: 6288.61ms | tok/sec: 83371.0 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.4ms | fwd: 2008.1ms | loss: 6.5ms | bwd: 4213.5ms | clip: 0.0ms | optim: 49.3ms
step   196 | loss: 1.632913 | lr 2.5358e-04 | norm: 0.7702 | dt: 6291.10ms | tok/sec: 83338.0 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4216.1ms | clip: 0.0ms | optim: 49.4ms
step   197 | loss: 1.642853 | lr 2.5270e-04 | norm: 0.4710 | dt: 6282.43ms | tok/sec: 83453.1 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.5ms | fwd: 2006.1ms | loss: 6.5ms | bwd: 4209.0ms | clip: 0.0ms | optim: 49.5ms
step   198 | loss: 1.595821 | lr 2.5180e-04 | norm: 0.3562 | dt: 6286.72ms | tok/sec: 83396.1 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.6ms | fwd: 2006.6ms | loss: 6.5ms | bwd: 4212.6ms | clip: 0.0ms | optim: 49.5ms
step   199 | loss: 1.596427 | lr 2.5090e-04 | norm: 0.4123 | dt: 6294.60ms | tok/sec: 83291.7 | mem: 108909MB | Time Left: 00 hrs : 21 mins
  [TIMING] data: 3.7ms | fwd: 2011.4ms | loss: 6.5ms | bwd: 4215.5ms | clip: 0.0ms | optim: 49.7ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run16_step_200.ckpt (safely)
step   200 | loss: 1.604256 | lr 2.5000e-04 | norm: 0.3507 | dt: 6228.46ms | tok/sec: 84176.2 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2026.7ms | loss: 6.5ms | bwd: 4134.3ms | clip: 0.0ms | optim: 49.5ms
step   201 | loss: 1.627132 | lr 2.4909e-04 | norm: 0.4327 | dt: 6225.88ms | tok/sec: 84211.0 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 1995.2ms | loss: 6.5ms | bwd: 4163.2ms | clip: 0.0ms | optim: 49.6ms
step   202 | loss: 1.604902 | lr 2.4818e-04 | norm: 0.4299 | dt: 6253.35ms | tok/sec: 83841.1 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2002.8ms | loss: 6.5ms | bwd: 4183.4ms | clip: 0.0ms | optim: 49.4ms
step   203 | loss: 1.678243 | lr 2.4726e-04 | norm: 0.3760 | dt: 6269.48ms | tok/sec: 83625.4 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.7ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4194.6ms | clip: 0.0ms | optim: 49.5ms
step   204 | loss: 1.596172 | lr 2.4633e-04 | norm: 0.3228 | dt: 6294.74ms | tok/sec: 83289.8 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2011.6ms | loss: 6.5ms | bwd: 4215.7ms | clip: 0.0ms | optim: 49.5ms
step   205 | loss: 1.570853 | lr 2.4540e-04 | norm: 0.4125 | dt: 6277.10ms | tok/sec: 83523.9 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2004.8ms | loss: 6.5ms | bwd: 4204.9ms | clip: 0.0ms | optim: 49.6ms
step   206 | loss: 1.653432 | lr 2.4446e-04 | norm: 0.4723 | dt: 6299.08ms | tok/sec: 83232.5 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4224.1ms | clip: 0.0ms | optim: 49.3ms
step   207 | loss: 1.673861 | lr 2.4352e-04 | norm: 0.5171 | dt: 6300.51ms | tok/sec: 83213.6 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.6ms | fwd: 2014.7ms | loss: 6.5ms | bwd: 4218.1ms | clip: 0.0ms | optim: 49.9ms
step   208 | loss: 1.548682 | lr 2.4258e-04 | norm: 0.4068 | dt: 6320.76ms | tok/sec: 82947.0 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.5ms | fwd: 2012.9ms | loss: 6.5ms | bwd: 4240.7ms | clip: 0.0ms | optim: 49.3ms
step   209 | loss: 1.638034 | lr 2.4163e-04 | norm: 0.4800 | dt: 6301.25ms | tok/sec: 83203.8 | mem: 108909MB | Time Left: 00 hrs : 20 mins
  [TIMING] data: 3.4ms | fwd: 2015.8ms | loss: 6.5ms | bwd: 4218.1ms | clip: 0.0ms | optim: 49.5ms
step   210 | loss: 1.654943 | lr 2.4067e-04 | norm: 0.3409 | dt: 6291.57ms | tok/sec: 83331.8 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2015.3ms | loss: 6.5ms | bwd: 4209.0ms | clip: 0.0ms | optim: 49.4ms
step   211 | loss: 1.664395 | lr 2.3971e-04 | norm: 0.3565 | dt: 6303.78ms | tok/sec: 83170.5 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.4ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4224.8ms | clip: 0.0ms | optim: 49.3ms
step   212 | loss: 1.613967 | lr 2.3875e-04 | norm: 0.3546 | dt: 6324.90ms | tok/sec: 82892.6 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2021.6ms | loss: 6.5ms | bwd: 4234.3ms | clip: 0.0ms | optim: 49.7ms
step   213 | loss: 1.626905 | lr 2.3778e-04 | norm: 0.5268 | dt: 6304.78ms | tok/sec: 83157.2 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2016.6ms | loss: 6.5ms | bwd: 4220.8ms | clip: 0.0ms | optim: 49.4ms
step   214 | loss: 1.613850 | lr 2.3681e-04 | norm: 0.3999 | dt: 6312.25ms | tok/sec: 83058.8 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4233.1ms | clip: 0.0ms | optim: 49.5ms
step   215 | loss: 1.627297 | lr 2.3584e-04 | norm: 0.3359 | dt: 6302.55ms | tok/sec: 83186.6 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2015.9ms | loss: 6.6ms | bwd: 4219.2ms | clip: 0.0ms | optim: 49.4ms
step   216 | loss: 1.591792 | lr 2.3486e-04 | norm: 0.4661 | dt: 6313.61ms | tok/sec: 83040.9 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2020.9ms | loss: 6.5ms | bwd: 4225.4ms | clip: 0.0ms | optim: 49.5ms
step   217 | loss: 1.608279 | lr 2.3387e-04 | norm: 0.3837 | dt: 6334.04ms | tok/sec: 82773.1 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.5ms | fwd: 2023.5ms | loss: 6.5ms | bwd: 4243.3ms | clip: 0.0ms | optim: 49.3ms
step   218 | loss: 1.634645 | lr 2.3289e-04 | norm: 0.3663 | dt: 6298.87ms | tok/sec: 83235.2 | mem: 108909MB | Time Left: 00 hrs : 19 mins
  [TIMING] data: 3.6ms | fwd: 2013.9ms | loss: 6.5ms | bwd: 4217.4ms | clip: 0.0ms | optim: 49.6ms
step   219 | loss: 1.658390 | lr 2.3190e-04 | norm: 0.3608 | dt: 6296.75ms | tok/sec: 83263.3 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.4ms | fwd: 2011.3ms | loss: 6.5ms | bwd: 4217.9ms | clip: 0.0ms | optim: 49.8ms
step   220 | loss: 1.602371 | lr 2.3090e-04 | norm: 0.3350 | dt: 6291.13ms | tok/sec: 83337.6 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.4ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4216.5ms | clip: 0.0ms | optim: 49.5ms
step   221 | loss: 1.583729 | lr 2.2990e-04 | norm: 0.3478 | dt: 6341.20ms | tok/sec: 82679.7 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2015.3ms | loss: 6.5ms | bwd: 4258.7ms | clip: 0.0ms | optim: 49.4ms
step   222 | loss: 1.576446 | lr 2.2890e-04 | norm: 0.3141 | dt: 6312.12ms | tok/sec: 83060.5 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2013.6ms | loss: 6.5ms | bwd: 4231.5ms | clip: 0.0ms | optim: 49.3ms
step   223 | loss: 1.605052 | lr 2.2790e-04 | norm: 0.3993 | dt: 6314.24ms | tok/sec: 83032.7 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4235.9ms | clip: 0.0ms | optim: 49.5ms
step   224 | loss: 1.558170 | lr 2.2689e-04 | norm: 0.4157 | dt: 6284.37ms | tok/sec: 83427.3 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2008.4ms | loss: 6.5ms | bwd: 4208.8ms | clip: 0.0ms | optim: 49.3ms
step   225 | loss: 1.650233 | lr 2.2588e-04 | norm: 0.3469 | dt: 6301.99ms | tok/sec: 83194.0 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.5ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4220.1ms | clip: 0.0ms | optim: 49.5ms
step   226 | loss: 1.575653 | lr 2.2487e-04 | norm: 0.3168 | dt: 6295.17ms | tok/sec: 83284.1 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.4ms | fwd: 2016.5ms | loss: 6.5ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.7ms
step   227 | loss: 1.572176 | lr 2.2385e-04 | norm: 0.3213 | dt: 6292.86ms | tok/sec: 83314.8 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.6ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4216.7ms | clip: 0.0ms | optim: 49.4ms
step   228 | loss: 1.624457 | lr 2.2284e-04 | norm: 0.3605 | dt: 6291.91ms | tok/sec: 83327.4 | mem: 108909MB | Time Left: 00 hrs : 18 mins
  [TIMING] data: 3.4ms | fwd: 2002.4ms | loss: 6.8ms | bwd: 4222.0ms | clip: 0.0ms | optim: 49.3ms
step   229 | loss: 1.591865 | lr 2.2181e-04 | norm: 0.3340 | dt: 6290.43ms | tok/sec: 83346.9 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2005.4ms | loss: 6.5ms | bwd: 4217.7ms | clip: 0.0ms | optim: 49.5ms
step   230 | loss: 1.587979 | lr 2.2079e-04 | norm: 0.3759 | dt: 6311.56ms | tok/sec: 83067.9 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2020.1ms | loss: 6.5ms | bwd: 4224.3ms | clip: 0.0ms | optim: 49.3ms
step   231 | loss: 1.584930 | lr 2.1977e-04 | norm: 0.3982 | dt: 6296.73ms | tok/sec: 83263.5 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2015.0ms | loss: 6.5ms | bwd: 4214.5ms | clip: 0.0ms | optim: 49.5ms
step   232 | loss: 1.661761 | lr 2.1874e-04 | norm: 0.3736 | dt: 6305.59ms | tok/sec: 83146.5 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2014.7ms | loss: 6.5ms | bwd: 4223.5ms | clip: 0.0ms | optim: 49.5ms
step   233 | loss: 1.594975 | lr 2.1771e-04 | norm: 0.3320 | dt: 6295.89ms | tok/sec: 83274.6 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2015.8ms | loss: 6.5ms | bwd: 4212.7ms | clip: 0.0ms | optim: 49.5ms
step   234 | loss: 1.613444 | lr 2.1668e-04 | norm: 0.3161 | dt: 6292.53ms | tok/sec: 83319.0 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2009.0ms | loss: 6.6ms | bwd: 4215.5ms | clip: 0.0ms | optim: 50.1ms
step   235 | loss: 1.649118 | lr 2.1564e-04 | norm: 0.3147 | dt: 6307.15ms | tok/sec: 83125.9 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2014.5ms | loss: 6.5ms | bwd: 4225.4ms | clip: 0.0ms | optim: 49.5ms
step   236 | loss: 1.618161 | lr 2.1461e-04 | norm: 0.3329 | dt: 6309.40ms | tok/sec: 83096.3 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2015.1ms | loss: 6.5ms | bwd: 4226.1ms | clip: 0.0ms | optim: 50.2ms
step   237 | loss: 1.611800 | lr 2.1357e-04 | norm: 0.3966 | dt: 6305.59ms | tok/sec: 83146.5 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.5ms | fwd: 2012.4ms | loss: 6.5ms | bwd: 4225.9ms | clip: 0.0ms | optim: 49.5ms
step   238 | loss: 1.546178 | lr 2.1253e-04 | norm: 0.3293 | dt: 6300.35ms | tok/sec: 83215.7 | mem: 108909MB | Time Left: 00 hrs : 17 mins
  [TIMING] data: 3.4ms | fwd: 2012.8ms | loss: 6.5ms | bwd: 4220.2ms | clip: 0.0ms | optim: 49.5ms
step   239 | loss: 1.572473 | lr 2.1149e-04 | norm: 0.3221 | dt: 6297.79ms | tok/sec: 83249.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2013.5ms | loss: 6.5ms | bwd: 4216.7ms | clip: 0.0ms | optim: 49.6ms
step   240 | loss: 1.594225 | lr 2.1045e-04 | norm: 0.8519 | dt: 6282.54ms | tok/sec: 83451.6 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2010.5ms | loss: 6.5ms | bwd: 4204.6ms | clip: 0.0ms | optim: 49.6ms
step   241 | loss: 1.572953 | lr 2.0941e-04 | norm: 0.3326 | dt: 6303.96ms | tok/sec: 83168.1 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2014.4ms | loss: 6.5ms | bwd: 4222.1ms | clip: 0.0ms | optim: 49.5ms
step   242 | loss: 1.610751 | lr 2.0837e-04 | norm: 0.3091 | dt: 6288.02ms | tok/sec: 83378.9 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2014.4ms | loss: 6.5ms | bwd: 4206.3ms | clip: 0.0ms | optim: 49.6ms
step   243 | loss: 1.580441 | lr 2.0732e-04 | norm: 0.2968 | dt: 6280.97ms | tok/sec: 83472.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2009.4ms | loss: 6.5ms | bwd: 4204.2ms | clip: 0.0ms | optim: 49.6ms
step   244 | loss: 1.607623 | lr 2.0628e-04 | norm: 0.3155 | dt: 6315.11ms | tok/sec: 83021.2 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2015.7ms | loss: 6.5ms | bwd: 4232.3ms | clip: 0.0ms | optim: 49.3ms
step   245 | loss: 1.612033 | lr 2.0523e-04 | norm: 0.2642 | dt: 6305.67ms | tok/sec: 83145.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.5ms | bwd: 4228.4ms | clip: 0.0ms | optim: 49.4ms
step   246 | loss: 1.588122 | lr 2.0419e-04 | norm: 0.2887 | dt: 6279.16ms | tok/sec: 83496.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.4ms | fwd: 2004.9ms | loss: 6.5ms | bwd: 4207.0ms | clip: 0.0ms | optim: 49.4ms
step   247 | loss: 1.612135 | lr 2.0314e-04 | norm: 0.2700 | dt: 6291.87ms | tok/sec: 83327.8 | mem: 108909MB | Time Left: 00 hrs : 16 mins
  [TIMING] data: 3.5ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4214.9ms | clip: 0.0ms | optim: 49.3ms
step   248 | loss: 1.579265 | lr 2.0209e-04 | norm: 0.2767 | dt: 6295.76ms | tok/sec: 83276.4 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.6ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4215.9ms | clip: 0.0ms | optim: 49.3ms
step   249 | loss: 1.588055 | lr 2.0105e-04 | norm: 0.2836 | dt: 6307.21ms | tok/sec: 83125.2 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2011.1ms | loss: 6.5ms | bwd: 4228.9ms | clip: 0.0ms | optim: 49.4ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_c/in2c_val_00000.bin
100000000
validation loss: 1.5887
step   250 | loss: 1.567841 | lr 2.0000e-04 | norm: 0.3970 | dt: 11336.52ms | tok/sec: 46247.7 | mem: 108909MB | Time Left: 00 hrs : 28 mins
  [TIMING] data: 3.5ms | fwd: 2020.6ms | loss: 6.5ms | bwd: 4206.3ms | clip: 0.0ms | optim: 49.3ms
step   251 | loss: 1.610270 | lr 1.9895e-04 | norm: 0.2627 | dt: 6299.63ms | tok/sec: 83225.2 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.6ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4221.4ms | clip: 0.0ms | optim: 49.3ms
step   252 | loss: 1.624688 | lr 1.9791e-04 | norm: 0.2330 | dt: 6285.08ms | tok/sec: 83417.9 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2004.9ms | loss: 6.5ms | bwd: 4213.0ms | clip: 0.0ms | optim: 49.4ms
step   253 | loss: 1.595510 | lr 1.9686e-04 | norm: 0.3151 | dt: 6282.56ms | tok/sec: 83451.3 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2010.2ms | loss: 6.5ms | bwd: 4203.8ms | clip: 0.0ms | optim: 50.8ms
step   254 | loss: 1.599921 | lr 1.9581e-04 | norm: 0.2855 | dt: 6324.40ms | tok/sec: 82899.2 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2020.0ms | loss: 6.5ms | bwd: 4237.2ms | clip: 0.0ms | optim: 49.3ms
step   255 | loss: 1.621604 | lr 1.9477e-04 | norm: 0.2698 | dt: 6303.64ms | tok/sec: 83172.3 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.5ms | fwd: 2003.6ms | loss: 6.5ms | bwd: 4232.6ms | clip: 0.0ms | optim: 49.6ms
step   256 | loss: 1.561271 | lr 1.9372e-04 | norm: 0.2415 | dt: 6287.98ms | tok/sec: 83379.5 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.4ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4208.8ms | clip: 0.0ms | optim: 49.6ms
step   257 | loss: 1.567798 | lr 1.9268e-04 | norm: 0.3085 | dt: 6308.16ms | tok/sec: 83112.7 | mem: 108909MB | Time Left: 00 hrs : 15 mins
  [TIMING] data: 3.6ms | fwd: 2016.4ms | loss: 6.5ms | bwd: 4224.1ms | clip: 0.0ms | optim: 49.7ms
step   258 | loss: 1.569631 | lr 1.9163e-04 | norm: 0.2791 | dt: 6284.12ms | tok/sec: 83430.6 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.6ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4205.5ms | clip: 0.0ms | optim: 49.6ms
step   259 | loss: 1.561539 | lr 1.9059e-04 | norm: 0.2430 | dt: 6304.53ms | tok/sec: 83160.5 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2009.2ms | loss: 6.7ms | bwd: 4227.6ms | clip: 0.0ms | optim: 49.7ms
step   260 | loss: 1.573225 | lr 1.8955e-04 | norm: 0.2891 | dt: 6284.97ms | tok/sec: 83419.3 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2011.9ms | loss: 6.5ms | bwd: 4205.6ms | clip: 0.0ms | optim: 49.5ms
step   261 | loss: 1.561672 | lr 1.8851e-04 | norm: 0.3397 | dt: 6293.78ms | tok/sec: 83302.5 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2016.1ms | loss: 6.5ms | bwd: 4210.3ms | clip: 0.0ms | optim: 49.4ms
step   262 | loss: 1.597780 | lr 1.8747e-04 | norm: 0.2360 | dt: 6290.50ms | tok/sec: 83346.0 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2011.6ms | loss: 6.5ms | bwd: 4211.7ms | clip: 0.0ms | optim: 49.4ms
step   263 | loss: 1.605152 | lr 1.8643e-04 | norm: 0.2341 | dt: 6301.92ms | tok/sec: 83194.9 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.4ms | fwd: 2021.4ms | loss: 6.5ms | bwd: 4213.5ms | clip: 0.0ms | optim: 49.2ms
step   264 | loss: 1.595408 | lr 1.8539e-04 | norm: 0.2030 | dt: 6289.41ms | tok/sec: 83360.5 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.5ms | fwd: 2013.6ms | loss: 6.5ms | bwd: 4208.7ms | clip: 0.0ms | optim: 49.3ms
step   265 | loss: 1.558509 | lr 1.8436e-04 | norm: 0.2662 | dt: 6290.20ms | tok/sec: 83349.9 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 3.6ms | fwd: 2005.7ms | loss: 6.5ms | bwd: 4216.0ms | clip: 0.0ms | optim: 50.4ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_c/in2c_train_00002.bin
100000000
step   266 | loss: 1.523877 | lr 1.8332e-04 | norm: 0.3287 | dt: 6293.25ms | tok/sec: 83309.6 | mem: 108909MB | Time Left: 00 hrs : 14 mins
  [TIMING] data: 7.4ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4213.1ms | clip: 0.0ms | optim: 49.4ms
step   267 | loss: 1.629341 | lr 1.8229e-04 | norm: 0.3155 | dt: 6301.62ms | tok/sec: 83198.9 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2009.3ms | loss: 6.5ms | bwd: 4224.8ms | clip: 0.0ms | optim: 49.7ms
step   268 | loss: 1.614017 | lr 1.8126e-04 | norm: 0.2437 | dt: 6310.47ms | tok/sec: 83082.2 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2015.1ms | loss: 6.5ms | bwd: 4228.2ms | clip: 0.0ms | optim: 49.3ms
step   269 | loss: 1.584949 | lr 1.8023e-04 | norm: 0.2511 | dt: 6303.79ms | tok/sec: 83170.3 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.4ms | fwd: 2016.1ms | loss: 6.5ms | bwd: 4220.1ms | clip: 0.0ms | optim: 49.8ms
step   270 | loss: 1.548059 | lr 1.7921e-04 | norm: 0.3066 | dt: 6298.63ms | tok/sec: 83238.5 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2008.3ms | loss: 6.5ms | bwd: 4223.0ms | clip: 0.0ms | optim: 49.3ms
step   271 | loss: 1.622351 | lr 1.7819e-04 | norm: 0.2835 | dt: 6280.80ms | tok/sec: 83474.8 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2008.5ms | loss: 6.5ms | bwd: 4204.8ms | clip: 0.0ms | optim: 49.5ms
step   272 | loss: 1.587949 | lr 1.7716e-04 | norm: 0.2922 | dt: 6300.64ms | tok/sec: 83211.9 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2013.4ms | loss: 6.5ms | bwd: 4220.1ms | clip: 0.0ms | optim: 49.2ms
step   273 | loss: 1.647515 | lr 1.7615e-04 | norm: 0.2937 | dt: 6317.03ms | tok/sec: 82995.9 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.6ms | fwd: 2016.3ms | loss: 6.5ms | bwd: 4233.4ms | clip: 0.0ms | optim: 49.4ms
step   274 | loss: 1.577578 | lr 1.7513e-04 | norm: 0.2304 | dt: 6283.10ms | tok/sec: 83444.2 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2009.9ms | loss: 6.5ms | bwd: 4205.9ms | clip: 0.0ms | optim: 49.4ms
step   275 | loss: 1.610921 | lr 1.7412e-04 | norm: 0.2311 | dt: 6299.24ms | tok/sec: 83230.3 | mem: 108909MB | Time Left: 00 hrs : 13 mins
  [TIMING] data: 3.5ms | fwd: 2013.5ms | loss: 6.5ms | bwd: 4218.3ms | clip: 0.0ms | optim: 49.6ms
step   276 | loss: 1.561477 | lr 1.7311e-04 | norm: 0.2333 | dt: 6285.56ms | tok/sec: 83411.5 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.6ms | fwd: 2013.1ms | loss: 6.5ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.4ms
step   277 | loss: 1.618157 | lr 1.7210e-04 | norm: 0.2187 | dt: 6294.89ms | tok/sec: 83287.9 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.6ms | fwd: 2017.0ms | loss: 6.5ms | bwd: 4210.7ms | clip: 0.0ms | optim: 49.3ms
step   278 | loss: 1.557510 | lr 1.7110e-04 | norm: 0.2638 | dt: 6294.71ms | tok/sec: 83290.3 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.4ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4215.0ms | clip: 0.0ms | optim: 49.2ms
step   279 | loss: 1.638976 | lr 1.7010e-04 | norm: 0.2125 | dt: 6278.97ms | tok/sec: 83499.0 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2007.8ms | loss: 6.5ms | bwd: 4203.9ms | clip: 0.0ms | optim: 49.4ms
step   280 | loss: 1.575585 | lr 1.6910e-04 | norm: 0.2255 | dt: 6317.37ms | tok/sec: 82991.6 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2025.5ms | loss: 6.5ms | bwd: 4224.4ms | clip: 0.0ms | optim: 49.6ms
step   281 | loss: 1.579373 | lr 1.6810e-04 | norm: 0.2550 | dt: 6292.18ms | tok/sec: 83323.7 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.6ms | fwd: 2015.0ms | loss: 6.5ms | bwd: 4209.4ms | clip: 0.0ms | optim: 49.9ms
step   282 | loss: 1.579899 | lr 1.6711e-04 | norm: 0.2489 | dt: 6287.42ms | tok/sec: 83386.9 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.6ms | fwd: 2011.4ms | loss: 6.5ms | bwd: 4208.9ms | clip: 0.0ms | optim: 49.3ms
step   283 | loss: 1.505046 | lr 1.6613e-04 | norm: 0.2072 | dt: 6301.12ms | tok/sec: 83205.5 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2011.1ms | loss: 6.5ms | bwd: 4222.7ms | clip: 0.0ms | optim: 49.4ms
step   284 | loss: 1.607189 | lr 1.6514e-04 | norm: 0.2857 | dt: 6302.74ms | tok/sec: 83184.2 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2015.6ms | loss: 6.5ms | bwd: 4219.9ms | clip: 0.0ms | optim: 49.3ms
step   285 | loss: 1.586106 | lr 1.6416e-04 | norm: 0.2449 | dt: 6301.02ms | tok/sec: 83206.9 | mem: 108909MB | Time Left: 00 hrs : 12 mins
  [TIMING] data: 3.5ms | fwd: 2015.7ms | loss: 6.5ms | bwd: 4218.3ms | clip: 0.0ms | optim: 49.3ms
step   286 | loss: 1.565539 | lr 1.6319e-04 | norm: 0.2377 | dt: 6309.58ms | tok/sec: 83093.9 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2017.2ms | loss: 6.5ms | bwd: 4224.9ms | clip: 0.0ms | optim: 49.7ms
step   287 | loss: 1.518639 | lr 1.6222e-04 | norm: 0.1958 | dt: 6299.38ms | tok/sec: 83228.6 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2014.4ms | loss: 6.5ms | bwd: 4217.8ms | clip: 0.0ms | optim: 49.4ms
step   288 | loss: 1.548906 | lr 1.6125e-04 | norm: 0.2190 | dt: 6303.82ms | tok/sec: 83169.8 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2014.4ms | loss: 6.5ms | bwd: 4222.0ms | clip: 0.0ms | optim: 49.4ms
step   289 | loss: 1.519977 | lr 1.6029e-04 | norm: 0.2426 | dt: 6293.09ms | tok/sec: 83311.7 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2014.3ms | loss: 6.5ms | bwd: 4211.5ms | clip: 0.0ms | optim: 49.4ms
step   290 | loss: 1.501722 | lr 1.5933e-04 | norm: 0.1933 | dt: 6299.46ms | tok/sec: 83227.5 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2018.4ms | loss: 6.5ms | bwd: 4213.7ms | clip: 0.0ms | optim: 49.4ms
step   291 | loss: 1.564830 | lr 1.5837e-04 | norm: 0.2057 | dt: 6286.89ms | tok/sec: 83393.8 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.4ms | fwd: 2017.7ms | loss: 6.5ms | bwd: 4201.8ms | clip: 0.0ms | optim: 49.5ms
step   292 | loss: 1.574274 | lr 1.5742e-04 | norm: 0.2429 | dt: 6296.77ms | tok/sec: 83263.0 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2014.1ms | loss: 6.5ms | bwd: 4215.3ms | clip: 0.0ms | optim: 49.5ms
step   293 | loss: 1.594174 | lr 1.5648e-04 | norm: 0.2388 | dt: 6300.42ms | tok/sec: 83214.8 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2024.7ms | loss: 6.5ms | bwd: 4207.9ms | clip: 0.0ms | optim: 49.9ms
step   294 | loss: 1.526099 | lr 1.5554e-04 | norm: 0.2306 | dt: 6303.95ms | tok/sec: 83168.2 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.4ms | fwd: 2017.7ms | loss: 6.5ms | bwd: 4219.0ms | clip: 0.0ms | optim: 49.4ms
step   295 | loss: 1.578834 | lr 1.5460e-04 | norm: 0.1973 | dt: 6288.51ms | tok/sec: 83372.3 | mem: 108909MB | Time Left: 00 hrs : 11 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4212.6ms | clip: 0.0ms | optim: 49.3ms
step   296 | loss: 1.594393 | lr 1.5367e-04 | norm: 0.2697 | dt: 6290.61ms | tok/sec: 83344.6 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2013.5ms | loss: 6.5ms | bwd: 4209.6ms | clip: 0.0ms | optim: 49.7ms
step   297 | loss: 1.525728 | lr 1.5274e-04 | norm: 0.2133 | dt: 6291.06ms | tok/sec: 83338.6 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.6ms
step   298 | loss: 1.544921 | lr 1.5182e-04 | norm: 0.2033 | dt: 6286.29ms | tok/sec: 83401.8 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.6ms | fwd: 2011.8ms | loss: 6.5ms | bwd: 4206.5ms | clip: 0.0ms | optim: 50.0ms
step   299 | loss: 1.537824 | lr 1.5091e-04 | norm: 0.2802 | dt: 6278.32ms | tok/sec: 83507.7 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4200.3ms | clip: 0.0ms | optim: 49.2ms
step   300 | loss: 1.584719 | lr 1.5000e-04 | norm: 0.2154 | dt: 6306.10ms | tok/sec: 83139.9 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2013.2ms | loss: 6.5ms | bwd: 4225.7ms | clip: 0.0ms | optim: 49.3ms
step   301 | loss: 1.560579 | lr 1.4910e-04 | norm: 0.3132 | dt: 6297.89ms | tok/sec: 83248.1 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2019.4ms | loss: 6.5ms | bwd: 4211.3ms | clip: 0.0ms | optim: 49.3ms
step   302 | loss: 1.558529 | lr 1.4820e-04 | norm: 0.1698 | dt: 6280.02ms | tok/sec: 83485.1 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2011.2ms | loss: 6.5ms | bwd: 4201.7ms | clip: 0.0ms | optim: 49.2ms
step   303 | loss: 1.482039 | lr 1.4730e-04 | norm: 0.1907 | dt: 6301.98ms | tok/sec: 83194.1 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.6ms | fwd: 2013.6ms | loss: 6.5ms | bwd: 4221.3ms | clip: 0.0ms | optim: 49.2ms
step   304 | loss: 1.582335 | lr 1.4642e-04 | norm: 0.2075 | dt: 6299.99ms | tok/sec: 83220.4 | mem: 108909MB | Time Left: 00 hrs : 10 mins
  [TIMING] data: 3.5ms | fwd: 2017.4ms | loss: 6.5ms | bwd: 4215.3ms | clip: 0.0ms | optim: 49.4ms
step   305 | loss: 1.589868 | lr 1.4554e-04 | norm: 0.2076 | dt: 6280.01ms | tok/sec: 83485.2 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2009.7ms | loss: 6.5ms | bwd: 4203.1ms | clip: 0.0ms | optim: 49.2ms
step   306 | loss: 1.524248 | lr 1.4466e-04 | norm: 0.3127 | dt: 6290.24ms | tok/sec: 83349.5 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2014.6ms | loss: 6.5ms | bwd: 4208.6ms | clip: 0.0ms | optim: 49.3ms
step   307 | loss: 1.611667 | lr 1.4379e-04 | norm: 0.2218 | dt: 6288.14ms | tok/sec: 83377.3 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2015.9ms | loss: 6.5ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.3ms
step   308 | loss: 1.597169 | lr 1.4293e-04 | norm: 0.2033 | dt: 6298.87ms | tok/sec: 83235.2 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2017.5ms | loss: 6.5ms | bwd: 4213.9ms | clip: 0.0ms | optim: 49.6ms
step   309 | loss: 1.547810 | lr 1.4207e-04 | norm: 0.2271 | dt: 6285.48ms | tok/sec: 83412.6 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.7ms | fwd: 2007.8ms | loss: 6.5ms | bwd: 4210.1ms | clip: 0.0ms | optim: 49.6ms
step   310 | loss: 1.506653 | lr 1.4122e-04 | norm: 0.2330 | dt: 6292.00ms | tok/sec: 83326.1 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.6ms | fwd: 2012.8ms | loss: 6.5ms | bwd: 4212.0ms | clip: 0.0ms | optim: 49.2ms
step   311 | loss: 1.573502 | lr 1.4038e-04 | norm: 0.1860 | dt: 6274.64ms | tok/sec: 83556.7 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4199.3ms | clip: 0.0ms | optim: 49.5ms
step   312 | loss: 1.536382 | lr 1.3954e-04 | norm: 0.1891 | dt: 6276.49ms | tok/sec: 83532.0 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.6ms | fwd: 2011.1ms | loss: 6.5ms | bwd: 4198.2ms | clip: 0.0ms | optim: 49.2ms
step   313 | loss: 1.515550 | lr 1.3871e-04 | norm: 0.2137 | dt: 6281.68ms | tok/sec: 83463.0 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.4ms | fwd: 2009.2ms | loss: 6.5ms | bwd: 4205.4ms | clip: 0.0ms | optim: 49.3ms
step   314 | loss: 1.564761 | lr 1.3789e-04 | norm: 0.2385 | dt: 6295.92ms | tok/sec: 83274.2 | mem: 108909MB | Time Left: 00 hrs : 09 mins
  [TIMING] data: 3.5ms | fwd: 2010.4ms | loss: 6.6ms | bwd: 4218.2ms | clip: 0.0ms | optim: 49.3ms
step   315 | loss: 1.567764 | lr 1.3707e-04 | norm: 0.2367 | dt: 6303.67ms | tok/sec: 83171.8 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2015.5ms | loss: 6.5ms | bwd: 4221.0ms | clip: 0.0ms | optim: 49.3ms
step   316 | loss: 1.555395 | lr 1.3626e-04 | norm: 0.2026 | dt: 6284.46ms | tok/sec: 83426.2 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.5ms | bwd: 4207.1ms | clip: 0.0ms | optim: 49.4ms
step   317 | loss: 1.545570 | lr 1.3545e-04 | norm: 0.2098 | dt: 6279.36ms | tok/sec: 83493.9 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2008.6ms | loss: 6.5ms | bwd: 4203.6ms | clip: 0.0ms | optim: 49.3ms
step   318 | loss: 1.520685 | lr 1.3466e-04 | norm: 0.2886 | dt: 6302.42ms | tok/sec: 83188.4 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4221.0ms | clip: 0.0ms | optim: 49.9ms
step   319 | loss: 1.619971 | lr 1.3387e-04 | norm: 0.2264 | dt: 6283.57ms | tok/sec: 83437.9 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.6ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4204.5ms | clip: 0.0ms | optim: 49.5ms
step   320 | loss: 1.561258 | lr 1.3309e-04 | norm: 0.2114 | dt: 6282.32ms | tok/sec: 83454.6 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2013.0ms | loss: 6.5ms | bwd: 4201.9ms | clip: 0.0ms | optim: 49.6ms
step   321 | loss: 1.610805 | lr 1.3231e-04 | norm: 0.2235 | dt: 6301.10ms | tok/sec: 83205.8 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.4ms | fwd: 2017.7ms | loss: 6.5ms | bwd: 4216.3ms | clip: 0.0ms | optim: 49.3ms
step   322 | loss: 1.557204 | lr 1.3155e-04 | norm: 0.2123 | dt: 6277.52ms | tok/sec: 83518.4 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4201.7ms | clip: 0.0ms | optim: 49.2ms
step   323 | loss: 1.555442 | lr 1.3079e-04 | norm: 0.1591 | dt: 6284.14ms | tok/sec: 83430.3 | mem: 108909MB | Time Left: 00 hrs : 08 mins
  [TIMING] data: 3.6ms | fwd: 2009.2ms | loss: 6.5ms | bwd: 4207.6ms | clip: 0.0ms | optim: 49.4ms
step   324 | loss: 1.532157 | lr 1.3003e-04 | norm: 0.2678 | dt: 6303.74ms | tok/sec: 83170.9 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.6ms | fwd: 2015.1ms | loss: 6.5ms | bwd: 4220.8ms | clip: 0.0ms | optim: 49.8ms
step   325 | loss: 1.550836 | lr 1.2929e-04 | norm: 0.1983 | dt: 6284.35ms | tok/sec: 83427.5 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2008.5ms | loss: 6.5ms | bwd: 4208.6ms | clip: 0.0ms | optim: 49.3ms
step   326 | loss: 1.600215 | lr 1.2855e-04 | norm: 0.2309 | dt: 6280.02ms | tok/sec: 83485.1 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.6ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4204.5ms | clip: 0.0ms | optim: 49.7ms
step   327 | loss: 1.532846 | lr 1.2782e-04 | norm: 0.1892 | dt: 6305.44ms | tok/sec: 83148.5 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2015.6ms | loss: 6.5ms | bwd: 4222.6ms | clip: 0.0ms | optim: 49.3ms
step   328 | loss: 1.492785 | lr 1.2710e-04 | norm: 0.7632 | dt: 6297.88ms | tok/sec: 83248.3 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.6ms | fwd: 2010.5ms | loss: 6.5ms | bwd: 4220.0ms | clip: 0.0ms | optim: 49.3ms
step   329 | loss: 1.611322 | lr 1.2639e-04 | norm: 0.2145 | dt: 6281.53ms | tok/sec: 83465.0 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2007.0ms | loss: 6.5ms | bwd: 4207.2ms | clip: 0.0ms | optim: 49.4ms
step   330 | loss: 1.556568 | lr 1.2569e-04 | norm: 0.2502 | dt: 6282.32ms | tok/sec: 83454.6 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4204.1ms | clip: 0.0ms | optim: 49.3ms
step   331 | loss: 1.540055 | lr 1.2499e-04 | norm: 0.1702 | dt: 6293.56ms | tok/sec: 83305.5 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.4ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4214.4ms | clip: 0.0ms | optim: 49.7ms
step   332 | loss: 1.508213 | lr 1.2430e-04 | norm: 0.1727 | dt: 6289.45ms | tok/sec: 83359.9 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4212.3ms | clip: 0.0ms | optim: 49.6ms
step   333 | loss: 1.536343 | lr 1.2362e-04 | norm: 0.1959 | dt: 6287.96ms | tok/sec: 83379.6 | mem: 108909MB | Time Left: 00 hrs : 07 mins
  [TIMING] data: 3.5ms | fwd: 2009.0ms | loss: 6.7ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.6ms
step   334 | loss: 1.534929 | lr 1.2295e-04 | norm: 0.1732 | dt: 6293.58ms | tok/sec: 83305.2 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.6ms | fwd: 2013.8ms | loss: 6.5ms | bwd: 4212.0ms | clip: 0.0ms | optim: 49.9ms
step   335 | loss: 1.583400 | lr 1.2229e-04 | norm: 0.2082 | dt: 6276.22ms | tok/sec: 83535.6 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.4ms | fwd: 2012.6ms | loss: 6.5ms | bwd: 4196.5ms | clip: 0.0ms | optim: 49.3ms
step   336 | loss: 1.545350 | lr 1.2163e-04 | norm: 0.1817 | dt: 6288.65ms | tok/sec: 83370.5 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2007.5ms | loss: 6.5ms | bwd: 4213.5ms | clip: 0.0ms | optim: 49.8ms
step   337 | loss: 1.551884 | lr 1.2098e-04 | norm: 0.2457 | dt: 6280.66ms | tok/sec: 83476.6 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2009.5ms | loss: 6.5ms | bwd: 4203.7ms | clip: 0.0ms | optim: 49.5ms
step   338 | loss: 1.577290 | lr 1.2035e-04 | norm: 0.1947 | dt: 6283.50ms | tok/sec: 83438.8 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2014.9ms | loss: 6.5ms | bwd: 4201.3ms | clip: 0.0ms | optim: 49.5ms
step   339 | loss: 1.569820 | lr 1.1972e-04 | norm: 0.1440 | dt: 6272.34ms | tok/sec: 83587.3 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2006.7ms | loss: 6.5ms | bwd: 4198.6ms | clip: 0.0ms | optim: 49.3ms
step   340 | loss: 1.549749 | lr 1.1910e-04 | norm: 0.1923 | dt: 6301.85ms | tok/sec: 83195.9 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.5ms | fwd: 2014.6ms | loss: 6.5ms | bwd: 4219.8ms | clip: 0.0ms | optim: 49.6ms
step   341 | loss: 1.530541 | lr 1.1849e-04 | norm: 0.2002 | dt: 6275.11ms | tok/sec: 83550.4 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.6ms | fwd: 2007.3ms | loss: 6.5ms | bwd: 4200.4ms | clip: 0.0ms | optim: 49.5ms
step   342 | loss: 1.586044 | lr 1.1789e-04 | norm: 0.1985 | dt: 6318.18ms | tok/sec: 82980.8 | mem: 108909MB | Time Left: 00 hrs : 06 mins
  [TIMING] data: 3.4ms | fwd: 2016.3ms | loss: 6.5ms | bwd: 4234.8ms | clip: 0.0ms | optim: 49.2ms
step   343 | loss: 1.547667 | lr 1.1729e-04 | norm: 0.2001 | dt: 6284.02ms | tok/sec: 83431.9 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2006.9ms | loss: 6.5ms | bwd: 4209.5ms | clip: 0.0ms | optim: 49.8ms
step   344 | loss: 1.514682 | lr 1.1671e-04 | norm: 0.1661 | dt: 6293.33ms | tok/sec: 83308.5 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.6ms | fwd: 2015.3ms | loss: 6.5ms | bwd: 4210.7ms | clip: 0.0ms | optim: 49.4ms
step   345 | loss: 1.507596 | lr 1.1613e-04 | norm: 0.2175 | dt: 6291.19ms | tok/sec: 83336.9 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4214.7ms | clip: 0.0ms | optim: 50.0ms
step   346 | loss: 1.529621 | lr 1.1557e-04 | norm: 0.2108 | dt: 6287.05ms | tok/sec: 83391.7 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2011.0ms | loss: 6.5ms | bwd: 4208.6ms | clip: 0.0ms | optim: 49.5ms
step   347 | loss: 1.542377 | lr 1.1501e-04 | norm: 0.3202 | dt: 6292.64ms | tok/sec: 83317.6 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2014.8ms | loss: 6.5ms | bwd: 4210.7ms | clip: 0.0ms | optim: 49.2ms
step   348 | loss: 1.550273 | lr 1.1446e-04 | norm: 0.1697 | dt: 6291.26ms | tok/sec: 83335.9 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.6ms | fwd: 2011.8ms | loss: 6.5ms | bwd: 4212.0ms | clip: 0.0ms | optim: 49.4ms
step   349 | loss: 1.492292 | lr 1.1393e-04 | norm: 0.1707 | dt: 6269.45ms | tok/sec: 83625.8 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.4ms | fwd: 2007.7ms | loss: 6.5ms | bwd: 4194.4ms | clip: 0.0ms | optim: 49.6ms
step   350 | loss: 1.534280 | lr 1.1340e-04 | norm: 0.1915 | dt: 6286.22ms | tok/sec: 83402.8 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2013.9ms | loss: 6.5ms | bwd: 4205.1ms | clip: 0.0ms | optim: 49.4ms
step   351 | loss: 1.562456 | lr 1.1288e-04 | norm: 0.2033 | dt: 6295.45ms | tok/sec: 83280.5 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.5ms | fwd: 2004.4ms | loss: 6.5ms | bwd: 4223.6ms | clip: 0.0ms | optim: 49.7ms
step   352 | loss: 1.590386 | lr 1.1237e-04 | norm: 0.1697 | dt: 6294.32ms | tok/sec: 83295.4 | mem: 108909MB | Time Left: 00 hrs : 05 mins
  [TIMING] data: 3.7ms | fwd: 2018.4ms | loss: 6.5ms | bwd: 4208.6ms | clip: 0.0ms | optim: 49.3ms
step   353 | loss: 1.548521 | lr 1.1187e-04 | norm: 0.1544 | dt: 6287.79ms | tok/sec: 83381.9 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2005.5ms | loss: 6.5ms | bwd: 4215.0ms | clip: 0.0ms | optim: 49.5ms
step   354 | loss: 1.509603 | lr 1.1138e-04 | norm: 0.1741 | dt: 6303.05ms | tok/sec: 83180.1 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.6ms | fwd: 2016.9ms | loss: 6.5ms | bwd: 4218.6ms | clip: 0.0ms | optim: 49.5ms
step   355 | loss: 1.579461 | lr 1.1090e-04 | norm: 0.1738 | dt: 6275.02ms | tok/sec: 83551.6 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2009.0ms | loss: 6.5ms | bwd: 4198.8ms | clip: 0.0ms | optim: 49.4ms
step   356 | loss: 1.545916 | lr 1.1043e-04 | norm: 0.1518 | dt: 6307.83ms | tok/sec: 83117.0 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2017.7ms | loss: 6.5ms | bwd: 4222.7ms | clip: 0.0ms | optim: 49.6ms
step   357 | loss: 1.568479 | lr 1.0997e-04 | norm: 0.1748 | dt: 6286.44ms | tok/sec: 83399.8 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.4ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4206.5ms | clip: 0.0ms | optim: 49.4ms
step   358 | loss: 1.583796 | lr 1.0952e-04 | norm: 0.1970 | dt: 6280.87ms | tok/sec: 83473.8 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2008.3ms | loss: 6.5ms | bwd: 4205.6ms | clip: 0.0ms | optim: 49.2ms
step   359 | loss: 1.463891 | lr 1.0908e-04 | norm: 0.1989 | dt: 6288.35ms | tok/sec: 83374.5 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2007.9ms | loss: 6.5ms | bwd: 4213.2ms | clip: 0.0ms | optim: 49.5ms
step   360 | loss: 1.545825 | lr 1.0865e-04 | norm: 0.2100 | dt: 6282.77ms | tok/sec: 83448.6 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.5ms | fwd: 2013.8ms | loss: 6.5ms | bwd: 4201.8ms | clip: 0.0ms | optim: 49.3ms
step   361 | loss: 1.569464 | lr 1.0822e-04 | norm: 0.2132 | dt: 6298.64ms | tok/sec: 83238.2 | mem: 108909MB | Time Left: 00 hrs : 04 mins
  [TIMING] data: 3.4ms | fwd: 2019.8ms | loss: 6.5ms | bwd: 4211.9ms | clip: 0.0ms | optim: 49.2ms
step   362 | loss: 1.528486 | lr 1.0781e-04 | norm: 0.1843 | dt: 6281.22ms | tok/sec: 83469.1 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2008.7ms | loss: 6.5ms | bwd: 4205.5ms | clip: 0.0ms | optim: 49.2ms
step   363 | loss: 1.549927 | lr 1.0741e-04 | norm: 0.1540 | dt: 6274.09ms | tok/sec: 83564.0 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4198.2ms | clip: 0.0ms | optim: 49.3ms
step   364 | loss: 1.524004 | lr 1.0702e-04 | norm: 0.1762 | dt: 6289.56ms | tok/sec: 83358.4 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.6ms | fwd: 2011.0ms | loss: 6.7ms | bwd: 4211.1ms | clip: 0.0ms | optim: 49.3ms
step   365 | loss: 1.557932 | lr 1.0664e-04 | norm: 0.1779 | dt: 6283.17ms | tok/sec: 83443.3 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2015.0ms | loss: 6.5ms | bwd: 4200.7ms | clip: 0.0ms | optim: 49.8ms
step   366 | loss: 1.549898 | lr 1.0627e-04 | norm: 0.2104 | dt: 6297.41ms | tok/sec: 83254.6 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2012.2ms | loss: 6.5ms | bwd: 4217.9ms | clip: 0.0ms | optim: 49.5ms
step   367 | loss: 1.530879 | lr 1.0591e-04 | norm: 0.1583 | dt: 6280.83ms | tok/sec: 83474.3 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2012.4ms | loss: 6.5ms | bwd: 4201.3ms | clip: 0.0ms | optim: 49.4ms
step   368 | loss: 1.539944 | lr 1.0556e-04 | norm: 0.2079 | dt: 6283.41ms | tok/sec: 83440.0 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.6ms | fwd: 2006.0ms | loss: 6.5ms | bwd: 4210.0ms | clip: 0.0ms | optim: 49.5ms
step   369 | loss: 1.547123 | lr 1.0522e-04 | norm: 0.1644 | dt: 6305.90ms | tok/sec: 83142.5 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.4ms | fwd: 2014.7ms | loss: 6.5ms | bwd: 4224.1ms | clip: 0.0ms | optim: 49.3ms
step   370 | loss: 1.502090 | lr 1.0489e-04 | norm: 0.1687 | dt: 6285.89ms | tok/sec: 83407.2 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.6ms | fwd: 2008.8ms | loss: 6.5ms | bwd: 4209.5ms | clip: 0.0ms | optim: 49.7ms
step   371 | loss: 1.539692 | lr 1.0458e-04 | norm: 0.1850 | dt: 6313.53ms | tok/sec: 83042.0 | mem: 108909MB | Time Left: 00 hrs : 03 mins
  [TIMING] data: 3.5ms | fwd: 2016.4ms | loss: 6.5ms | bwd: 4229.9ms | clip: 0.0ms | optim: 49.3ms
step   372 | loss: 1.566259 | lr 1.0427e-04 | norm: 0.3402 | dt: 6295.70ms | tok/sec: 83277.1 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.6ms | fwd: 2012.6ms | loss: 6.5ms | bwd: 4215.5ms | clip: 0.0ms | optim: 49.7ms
step   373 | loss: 1.518983 | lr 1.0397e-04 | norm: 0.1726 | dt: 6276.02ms | tok/sec: 83538.2 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4199.2ms | clip: 0.0ms | optim: 49.3ms
step   374 | loss: 1.523308 | lr 1.0368e-04 | norm: 0.1868 | dt: 6308.82ms | tok/sec: 83103.9 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2020.7ms | loss: 6.5ms | bwd: 4220.6ms | clip: 0.0ms | optim: 49.7ms
step   375 | loss: 1.567796 | lr 1.0341e-04 | norm: 0.2081 | dt: 6288.10ms | tok/sec: 83377.9 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.5ms | fwd: 2006.8ms | loss: 6.5ms | bwd: 4214.0ms | clip: 0.0ms | optim: 49.4ms
step   376 | loss: 1.552955 | lr 1.0314e-04 | norm: 0.1678 | dt: 6305.90ms | tok/sec: 83142.4 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.6ms | fwd: 2017.6ms | loss: 6.5ms | bwd: 4220.9ms | clip: 0.0ms | optim: 49.4ms
step   377 | loss: 1.510528 | lr 1.0289e-04 | norm: 0.1548 | dt: 6299.49ms | tok/sec: 83227.1 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2010.4ms | loss: 6.5ms | bwd: 4221.9ms | clip: 0.0ms | optim: 49.5ms
step   378 | loss: 1.512734 | lr 1.0264e-04 | norm: 0.1612 | dt: 6292.82ms | tok/sec: 83315.3 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.6ms | fwd: 2009.8ms | loss: 6.5ms | bwd: 4215.8ms | clip: 0.0ms | optim: 49.3ms
step   379 | loss: 1.502775 | lr 1.0241e-04 | norm: 0.1464 | dt: 6286.39ms | tok/sec: 83400.6 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.4ms | fwd: 2009.4ms | loss: 6.5ms | bwd: 4209.5ms | clip: 0.0ms | optim: 49.7ms
step   380 | loss: 1.569133 | lr 1.0219e-04 | norm: 0.1828 | dt: 6304.55ms | tok/sec: 83160.2 | mem: 108909MB | Time Left: 00 hrs : 02 mins
  [TIMING] data: 3.6ms | fwd: 2013.6ms | loss: 6.5ms | bwd: 4223.3ms | clip: 0.0ms | optim: 49.7ms
step   381 | loss: 1.562892 | lr 1.0197e-04 | norm: 0.1915 | dt: 6293.61ms | tok/sec: 83304.9 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.6ms | fwd: 2013.7ms | loss: 6.5ms | bwd: 4212.8ms | clip: 0.0ms | optim: 49.2ms
step   382 | loss: 1.539532 | lr 1.0177e-04 | norm: 0.1914 | dt: 6269.80ms | tok/sec: 83621.1 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2004.8ms | loss: 6.5ms | bwd: 4197.9ms | clip: 0.0ms | optim: 49.2ms
step   383 | loss: 1.488328 | lr 1.0158e-04 | norm: 0.2315 | dt: 6308.77ms | tok/sec: 83104.7 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2017.2ms | loss: 6.5ms | bwd: 4224.3ms | clip: 0.0ms | optim: 49.4ms
step   384 | loss: 1.556500 | lr 1.0140e-04 | norm: 0.1809 | dt: 6286.33ms | tok/sec: 83401.3 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2011.7ms | loss: 6.5ms | bwd: 4207.2ms | clip: 0.0ms | optim: 49.6ms
step   385 | loss: 1.547946 | lr 1.0123e-04 | norm: 0.2027 | dt: 6288.08ms | tok/sec: 83378.1 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2015.9ms | loss: 6.5ms | bwd: 4204.8ms | clip: 0.0ms | optim: 49.6ms
step   386 | loss: 1.557911 | lr 1.0107e-04 | norm: 0.1581 | dt: 6300.51ms | tok/sec: 83213.6 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2013.5ms | loss: 6.5ms | bwd: 4219.0ms | clip: 0.0ms | optim: 50.3ms
step   387 | loss: 1.565535 | lr 1.0093e-04 | norm: 0.1891 | dt: 6277.07ms | tok/sec: 83524.4 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.5ms | fwd: 2006.6ms | loss: 6.5ms | bwd: 4203.3ms | clip: 0.0ms | optim: 49.3ms
step   388 | loss: 1.556757 | lr 1.0079e-04 | norm: 0.1916 | dt: 6288.45ms | tok/sec: 83373.2 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.3ms | fwd: 2012.7ms | loss: 6.5ms | bwd: 4208.5ms | clip: 0.0ms | optim: 49.6ms
step   389 | loss: 1.592374 | lr 1.0066e-04 | norm: 0.1882 | dt: 6313.66ms | tok/sec: 83040.3 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.4ms | fwd: 2012.6ms | loss: 6.5ms | bwd: 4234.1ms | clip: 0.0ms | optim: 49.3ms
step   390 | loss: 1.435425 | lr 1.0055e-04 | norm: 0.1936 | dt: 6298.11ms | tok/sec: 83245.2 | mem: 108909MB | Time Left: 00 hrs : 01 mins
  [TIMING] data: 3.6ms | fwd: 2010.7ms | loss: 6.5ms | bwd: 4219.9ms | clip: 0.0ms | optim: 49.6ms
step   391 | loss: 1.515368 | lr 1.0044e-04 | norm: 0.1479 | dt: 6294.25ms | tok/sec: 83296.3 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2012.5ms | loss: 6.5ms | bwd: 4213.9ms | clip: 0.0ms | optim: 50.0ms
step   392 | loss: 1.508965 | lr 1.0035e-04 | norm: 0.1423 | dt: 6290.34ms | tok/sec: 83348.1 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.6ms | fwd: 2013.0ms | loss: 6.5ms | bwd: 4210.1ms | clip: 0.0ms | optim: 49.2ms
step   393 | loss: 1.558363 | lr 1.0027e-04 | norm: 0.2960 | dt: 6299.96ms | tok/sec: 83220.8 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2011.3ms | loss: 6.6ms | bwd: 4221.4ms | clip: 0.0ms | optim: 49.3ms
step   394 | loss: 1.525680 | lr 1.0020e-04 | norm: 0.2125 | dt: 6312.61ms | tok/sec: 83054.0 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2013.6ms | loss: 6.5ms | bwd: 4231.9ms | clip: 0.0ms | optim: 49.3ms
step   395 | loss: 1.560365 | lr 1.0014e-04 | norm: 0.1522 | dt: 6287.88ms | tok/sec: 83380.7 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.6ms | fwd: 2016.8ms | loss: 6.5ms | bwd: 4203.0ms | clip: 0.0ms | optim: 50.1ms
step   396 | loss: 1.546882 | lr 1.0009e-04 | norm: 0.1554 | dt: 6289.54ms | tok/sec: 83358.8 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.6ms | fwd: 2003.6ms | loss: 6.5ms | bwd: 4218.4ms | clip: 0.0ms | optim: 49.6ms
step   397 | loss: 1.521094 | lr 1.0005e-04 | norm: 0.1445 | dt: 6297.95ms | tok/sec: 83247.4 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.4ms | fwd: 2015.2ms | loss: 6.5ms | bwd: 4215.7ms | clip: 0.0ms | optim: 49.3ms
step   398 | loss: 1.529026 | lr 1.0002e-04 | norm: 0.1885 | dt: 6282.62ms | tok/sec: 83450.5 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 3.5ms | fwd: 2008.2ms | loss: 6.5ms | bwd: 4207.2ms | clip: 0.0ms | optim: 49.3ms
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_c/in2c_val_00000.bin
100000000
validation loss: 1.5270
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_c/in2c_train_00000.bin
100000000
step   399 | loss: 1.536691 | lr 1.0001e-04 | norm: 0.2335 | dt: 11351.70ms | tok/sec: 46185.9 | mem: 108909MB | Time Left: 00 hrs : 00 mins
  [TIMING] data: 7.3ms | fwd: 2010.6ms | loss: 6.5ms | bwd: 4215.1ms | clip: 0.0ms | optim: 49.8ms
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run16_step_400.ckpt (safely)
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run16_step_200.ckpt

=== Training complete ===