Hugging Face's logo

unparallelled
/
BluScriptCP

Safetensors
Model card Files
xet
 Community
unparallelled commited on
Commit
1db5d09
·
verified ·
1 Parent(s): dcfebcc

Upload folder using huggingface_hub

Browse files
Files changed (10) hide show
  1. logs/in2/arm_a.trained +1 -0
  2. logs/in2/arm_a_train.log +0 -0
  3. logs/in2/arm_b.trained +1 -0
  4. logs/in2/arm_b_train.log +693 -0
  5. logs/in2/arm_c.trained +1 -0
  6. logs/in2/arm_c_train.log +297 -0
  7. logs/in2/driver_20260829_1535.log +114 -0
  8. logs/in2/driver_20260829_1542.log +92 -0
  9. logs/in2/driver_20260829_1545.log +94 -0
  10. logs/in2/driver_20260829_1548.log +100 -0
logs/in2/arm_a.trained ADDED
@@ -0,0 +1 @@
 
 
1
+ 12
logs/in2/arm_a_train.log ADDED
The diff for this file is too large to render. See raw diff
 
logs/in2/arm_b.trained ADDED
@@ -0,0 +1 @@
 
 
1
+ 13
logs/in2/arm_b_train.log ADDED
@@ -0,0 +1,693 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === Llama Context Parallel Training Script (bluscriptCP) ===
2
+ [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
3
+ [ZeRO-1] CP_ZERO_SYMMETRIC=1: NCCL symmetric memory mode (NVLS)
4
+ [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
5
+ [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
6
+ ================ Configuration (bluscriptCP) ================
7
+ d_model/layers : 1536 / 36
8
+ q/kv heads : 12 / 4 (G=3, hd=128)
9
+ ffn_hidden : 4096 (SwiGLU)
10
+ context_length : 16384
11
+ vocab_size : 50304 tying=0
12
+ B x T : 4 x 16384
13
+ world_size : 4 (dp=1 x cp=4)
14
+ CP_ATTN_MODE : ring (load_balance=HeadTail)
15
+ global_batch : 524288 (grad_accum=8)
16
+ optimizer : Muon (2-D body) + AdamW (wte + 1-D) [ZeRO-1] muon_lr=0.001000
17
+ precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
18
+ max_lr/min_lr : 0.001 / 0.0001
19
+ warmup/steps : 1 / 400 (cosine, no stable phase)
20
+ checkpointing : true (ckpt_freq=100, dir=checkpoints_bluscriptcp)
21
+ =============================================================
22
+ Initializing model on CUDA device 0...
23
+ [YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
24
+ [YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
25
+ [InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt (base step 750, loss 1.28324) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
26
+ Parameters: 1060624896
27
+ Parameters per GPU: 1060624896
28
+ max_steps: 400
29
+ warmup_steps: 1
30
+ [Muon] adjust_lr=match_rms_adamw (default) ns_steps=5 momentum=0.95
31
+ [ZeRO-1/Muon] reduce-scatter/all-gather layout: 6 bucket(s), 12 collectives/step, shard 226492416 elems/rank, padding 0.000000% (flat buffer 1728.000000 MB)
32
+ [ZeRO-1/Muon] adjust_lr=match_rms_adamw | params=905969664 (216 orthogonalized / 0 AdamW) muon_elems=905969664 | W=4 owned=226492416 (42 tensors) | master+momentum: 1728.000000 MB/GPU
33
+ [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
34
+ [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=ON overlap=-/-
35
+ [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
36
+ [ZeRO-1 v2] params=77266944 padded=77266944 W=4 shard=19316736 | 3 buckets of 50.000000 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 221.062500 MB (vs 884.250000 MB unsharded)
37
+ [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
38
+ [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=ON overlap=-/-
39
+ [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
40
+ [ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50.000000 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.409668 MB (vs 885.638672 MB unsharded)
41
+ [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=3 [muon:bfloat16/905M] [adamw:bfloat16/77266944] [adamw:float32/77388288]
42
+ [ZeRO-1] DataParallel is DISABLED (both would double-reduce)
43
+ [precision] bf16 params=217, all with fp32 master weights
44
+ found 3 shards for split train
45
+ found 1 shards for split val
46
+ Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log13.csv (run 13)
47
+ [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
48
+ Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run13_step_<N>.ckpt (every 100 steps, keep 2)
49
+ [re-warmup] anchored at step 0 for 100 steps -> peak 0.000300 (0.300000 x max_lr), then cosine-decay to 0.000100 | resuming at step 0 (mid-ramp)
50
+
51
+ Starting training...
52
+
53
+ [CP ring P2P] OVERLAP: dedicated non-blocking stream
54
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_val_00000.bin
55
+ 100000000
56
+ validation loss: 1.6834
57
+ [CP bwd] overlap=ON
58
+ step 0 | loss: 1.719764 | lr 3.0000e-06 | norm: 0.9177 | dt: 15904.57ms | tok/sec: 32964.6 | mem: 107213MB | Time Left: 01 hrs : 46 mins
59
+ [TIMING] data: 3.0ms | fwd: 4237.0ms | loss: 6.5ms | bwd: 4571.0ms | clip: 0.0ms | optim: 62.6ms
60
+ step 1 | loss: 1.689209 | lr 6.0000e-06 | norm: 0.9809 | dt: 8602.98ms | tok/sec: 60942.6 | mem: 108909MB | Time Left: 00 hrs : 57 mins
61
+ [TIMING] data: 3.5ms | fwd: 4004.7ms | loss: 6.4ms | bwd: 4530.9ms | clip: 0.0ms | optim: 49.6ms
62
+ step 2 | loss: 1.710424 | lr 9.0000e-06 | norm: 1.2316 | dt: 8313.79ms | tok/sec: 63062.5 | mem: 108909MB | Time Left: 00 hrs : 55 mins
63
+ [TIMING] data: 4.5ms | fwd: 4090.5ms | loss: 6.3ms | bwd: 4154.7ms | clip: 0.0ms | optim: 49.5ms
64
+ step 3 | loss: 1.731818 | lr 1.2000e-05 | norm: 0.9002 | dt: 6208.15ms | tok/sec: 84451.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
65
+ [TIMING] data: 3.5ms | fwd: 1990.5ms | loss: 6.4ms | bwd: 4150.5ms | clip: 0.0ms | optim: 49.4ms
66
+ step 4 | loss: 1.702109 | lr 1.5000e-05 | norm: 0.9435 | dt: 6212.17ms | tok/sec: 84396.9 | mem: 108909MB | Time Left: 00 hrs : 41 mins
67
+ [TIMING] data: 3.4ms | fwd: 1991.1ms | loss: 6.4ms | bwd: 4153.8ms | clip: 0.0ms | optim: 49.5ms
68
+ step 5 | loss: 1.766153 | lr 1.8000e-05 | norm: 0.8298 | dt: 6250.52ms | tok/sec: 83879.1 | mem: 108909MB | Time Left: 00 hrs : 41 mins
69
+ [TIMING] data: 3.4ms | fwd: 2008.1ms | loss: 6.4ms | bwd: 4175.3ms | clip: 0.0ms | optim: 49.4ms
70
+ step 6 | loss: 1.703925 | lr 2.1000e-05 | norm: 0.9083 | dt: 6372.13ms | tok/sec: 82278.3 | mem: 108909MB | Time Left: 00 hrs : 41 mins
71
+ [TIMING] data: 3.3ms | fwd: 2084.4ms | loss: 6.4ms | bwd: 4220.8ms | clip: 0.0ms | optim: 49.3ms
72
+ step 7 | loss: 1.660973 | lr 2.4000e-05 | norm: 0.7858 | dt: 6383.07ms | tok/sec: 82137.3 | mem: 108909MB | Time Left: 00 hrs : 41 mins
73
+ [TIMING] data: 3.3ms | fwd: 2029.9ms | loss: 6.4ms | bwd: 4286.0ms | clip: 0.0ms | optim: 49.6ms
74
+ step 8 | loss: 1.628360 | lr 2.7000e-05 | norm: 0.7381 | dt: 6420.10ms | tok/sec: 81663.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
75
+ [TIMING] data: 3.4ms | fwd: 2036.6ms | loss: 6.4ms | bwd: 4316.3ms | clip: 0.0ms | optim: 49.5ms
76
+ step 9 | loss: 1.665800 | lr 3.0000e-05 | norm: 0.7373 | dt: 6401.36ms | tok/sec: 81902.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
77
+ [TIMING] data: 3.5ms | fwd: 2035.0ms | loss: 6.4ms | bwd: 4299.1ms | clip: 0.0ms | optim: 49.5ms
78
+ step 10 | loss: 1.610076 | lr 3.3000e-05 | norm: 0.8668 | dt: 6412.79ms | tok/sec: 81756.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
79
+ [TIMING] data: 3.4ms | fwd: 2034.6ms | loss: 6.4ms | bwd: 4310.9ms | clip: 0.0ms | optim: 49.5ms
80
+ step 11 | loss: 1.618258 | lr 3.6000e-05 | norm: 0.7161 | dt: 6374.74ms | tok/sec: 82244.7 | mem: 108909MB | Time Left: 00 hrs : 41 mins
81
+ [TIMING] data: 3.6ms | fwd: 2030.3ms | loss: 6.4ms | bwd: 4277.1ms | clip: 0.0ms | optim: 49.4ms
82
+ step 12 | loss: 1.605687 | lr 3.9000e-05 | norm: 0.7537 | dt: 6461.94ms | tok/sec: 81134.8 | mem: 108909MB | Time Left: 00 hrs : 41 mins
83
+ [TIMING] data: 3.5ms | fwd: 2101.0ms | loss: 6.4ms | bwd: 4293.6ms | clip: 0.0ms | optim: 49.5ms
84
+ step 13 | loss: 1.612419 | lr 4.2000e-05 | norm: 0.8240 | dt: 6356.31ms | tok/sec: 82483.1 | mem: 108909MB | Time Left: 00 hrs : 40 mins
85
+ [TIMING] data: 3.5ms | fwd: 2021.3ms | loss: 6.4ms | bwd: 4267.4ms | clip: 0.0ms | optim: 49.7ms
86
+ step 14 | loss: 1.638093 | lr 4.5000e-05 | norm: 0.8258 | dt: 6346.77ms | tok/sec: 82607.0 | mem: 108909MB | Time Left: 00 hrs : 40 mins
87
+ [TIMING] data: 3.5ms | fwd: 2020.4ms | loss: 6.4ms | bwd: 4258.5ms | clip: 0.0ms | optim: 50.0ms
88
+ step 15 | loss: 1.627281 | lr 4.8000e-05 | norm: 0.8539 | dt: 6335.85ms | tok/sec: 82749.5 | mem: 108909MB | Time Left: 00 hrs : 40 mins
89
+ [TIMING] data: 3.5ms | fwd: 2022.1ms | loss: 6.4ms | bwd: 4246.4ms | clip: 0.0ms | optim: 49.6ms
90
+ step 16 | loss: 1.602286 | lr 5.1000e-05 | norm: 0.8507 | dt: 6329.67ms | tok/sec: 82830.3 | mem: 108909MB | Time Left: 00 hrs : 40 mins
91
+ [TIMING] data: 3.5ms | fwd: 2024.3ms | loss: 6.4ms | bwd: 4238.2ms | clip: 0.0ms | optim: 49.3ms
92
+ step 17 | loss: 1.543830 | lr 5.4000e-05 | norm: 0.7251 | dt: 6331.24ms | tok/sec: 82809.7 | mem: 108909MB | Time Left: 00 hrs : 40 mins
93
+ [TIMING] data: 3.5ms | fwd: 2029.2ms | loss: 6.4ms | bwd: 4234.7ms | clip: 0.0ms | optim: 49.4ms
94
+ step 18 | loss: 1.574815 | lr 5.7000e-05 | norm: 0.6813 | dt: 6321.41ms | tok/sec: 82938.5 | mem: 108909MB | Time Left: 00 hrs : 40 mins
95
+ [TIMING] data: 3.5ms | fwd: 2008.9ms | loss: 6.4ms | bwd: 4245.3ms | clip: 0.0ms | optim: 49.4ms
96
+ step 19 | loss: 1.587047 | lr 6.0000e-05 | norm: 0.7060 | dt: 6342.47ms | tok/sec: 82663.1 | mem: 108909MB | Time Left: 00 hrs : 40 mins
97
+ [TIMING] data: 3.4ms | fwd: 2014.7ms | loss: 6.4ms | bwd: 4260.5ms | clip: 0.0ms | optim: 49.4ms
98
+ step 20 | loss: 1.558500 | lr 6.3000e-05 | norm: 0.8647 | dt: 6334.12ms | tok/sec: 82772.0 | mem: 108909MB | Time Left: 00 hrs : 40 mins
99
+ [TIMING] data: 3.4ms | fwd: 2016.2ms | loss: 6.4ms | bwd: 4250.7ms | clip: 0.0ms | optim: 49.4ms
100
+ step 21 | loss: 1.628016 | lr 6.6000e-05 | norm: 0.8759 | dt: 6310.75ms | tok/sec: 83078.6 | mem: 108909MB | Time Left: 00 hrs : 39 mins
101
+ [TIMING] data: 3.4ms | fwd: 2009.1ms | loss: 6.4ms | bwd: 4234.4ms | clip: 0.0ms | optim: 49.4ms
102
+ step 22 | loss: 1.556988 | lr 6.9000e-05 | norm: 0.7970 | dt: 6326.02ms | tok/sec: 82878.0 | mem: 108909MB | Time Left: 00 hrs : 39 mins
103
+ [TIMING] data: 3.5ms | fwd: 2009.3ms | loss: 6.4ms | bwd: 4249.5ms | clip: 0.0ms | optim: 49.5ms
104
+ step 23 | loss: 1.555099 | lr 7.2000e-05 | norm: 0.8540 | dt: 6324.02ms | tok/sec: 82904.2 | mem: 108909MB | Time Left: 00 hrs : 39 mins
105
+ [TIMING] data: 3.5ms | fwd: 2013.5ms | loss: 6.4ms | bwd: 4243.2ms | clip: 0.0ms | optim: 49.4ms
106
+ step 24 | loss: 1.524715 | lr 7.5000e-05 | norm: 0.6101 | dt: 6328.00ms | tok/sec: 82852.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
107
+ [TIMING] data: 3.4ms | fwd: 2036.3ms | loss: 6.4ms | bwd: 4224.5ms | clip: 0.0ms | optim: 49.4ms
108
+ step 25 | loss: 1.595337 | lr 7.8000e-05 | norm: 0.6755 | dt: 6358.19ms | tok/sec: 82458.7 | mem: 108909MB | Time Left: 00 hrs : 39 mins
109
+ [TIMING] data: 3.5ms | fwd: 2036.2ms | loss: 6.4ms | bwd: 4254.8ms | clip: 0.0ms | optim: 49.4ms
110
+ step 26 | loss: 1.504139 | lr 8.1000e-05 | norm: 0.9090 | dt: 6344.17ms | tok/sec: 82641.0 | mem: 108909MB | Time Left: 00 hrs : 39 mins
111
+ [TIMING] data: 3.5ms | fwd: 2056.7ms | loss: 6.4ms | bwd: 4220.2ms | clip: 0.0ms | optim: 49.4ms
112
+ step 27 | loss: 1.509489 | lr 8.4000e-05 | norm: 0.6161 | dt: 6334.27ms | tok/sec: 82770.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
113
+ [TIMING] data: 3.5ms | fwd: 2021.7ms | loss: 6.4ms | bwd: 4244.9ms | clip: 0.0ms | optim: 49.8ms
114
+ step 28 | loss: 1.541055 | lr 8.7000e-05 | norm: 0.7380 | dt: 6335.22ms | tok/sec: 82757.6 | mem: 108909MB | Time Left: 00 hrs : 39 mins
115
+ [TIMING] data: 3.4ms | fwd: 2021.2ms | loss: 6.4ms | bwd: 4246.9ms | clip: 0.0ms | optim: 49.4ms
116
+ step 29 | loss: 1.520158 | lr 9.0000e-05 | norm: 0.6387 | dt: 6310.05ms | tok/sec: 83087.7 | mem: 108909MB | Time Left: 00 hrs : 39 mins
117
+ [TIMING] data: 3.5ms | fwd: 2014.5ms | loss: 6.4ms | bwd: 4228.3ms | clip: 0.0ms | optim: 49.5ms
118
+ step 30 | loss: 1.497158 | lr 9.3000e-05 | norm: 0.6212 | dt: 6331.83ms | tok/sec: 82801.9 | mem: 108909MB | Time Left: 00 hrs : 39 mins
119
+ [TIMING] data: 3.5ms | fwd: 2019.8ms | loss: 6.4ms | bwd: 4244.8ms | clip: 0.0ms | optim: 49.4ms
120
+ step 31 | loss: 1.558612 | lr 9.6000e-05 | norm: 0.6279 | dt: 6387.33ms | tok/sec: 82082.4 | mem: 108909MB | Time Left: 00 hrs : 39 mins
121
+ [TIMING] data: 3.5ms | fwd: 2064.0ms | loss: 6.4ms | bwd: 4256.1ms | clip: 0.0ms | optim: 49.4ms
122
+ step 32 | loss: 1.469230 | lr 9.9000e-05 | norm: 0.6770 | dt: 6318.88ms | tok/sec: 82971.6 | mem: 108909MB | Time Left: 00 hrs : 38 mins
123
+ [TIMING] data: 3.5ms | fwd: 2014.8ms | loss: 6.4ms | bwd: 4236.7ms | clip: 0.0ms | optim: 49.6ms
124
+ step 33 | loss: 1.531537 | lr 1.0200e-04 | norm: 0.6956 | dt: 6318.93ms | tok/sec: 82971.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
125
+ [TIMING] data: 3.4ms | fwd: 2017.4ms | loss: 6.4ms | bwd: 4234.6ms | clip: 0.0ms | optim: 49.3ms
126
+ step 34 | loss: 1.509960 | lr 1.0500e-04 | norm: 1.2464 | dt: 6330.87ms | tok/sec: 82814.5 | mem: 108909MB | Time Left: 00 hrs : 38 mins
127
+ [TIMING] data: 3.4ms | fwd: 2013.2ms | loss: 6.4ms | bwd: 4250.5ms | clip: 0.0ms | optim: 49.4ms
128
+ step 35 | loss: 1.521332 | lr 1.0800e-04 | norm: 0.6600 | dt: 6320.12ms | tok/sec: 82955.4 | mem: 108909MB | Time Left: 00 hrs : 38 mins
129
+ [TIMING] data: 3.4ms | fwd: 2014.6ms | loss: 6.4ms | bwd: 4238.4ms | clip: 0.0ms | optim: 49.4ms
130
+ step 36 | loss: 1.499588 | lr 1.1100e-04 | norm: 0.7906 | dt: 6328.81ms | tok/sec: 82841.5 | mem: 108909MB | Time Left: 00 hrs : 38 mins
131
+ [TIMING] data: 3.4ms | fwd: 2018.6ms | loss: 6.4ms | bwd: 4243.0ms | clip: 0.0ms | optim: 49.4ms
132
+ step 37 | loss: 1.499973 | lr 1.1400e-04 | norm: 0.6590 | dt: 6330.89ms | tok/sec: 82814.3 | mem: 108909MB | Time Left: 00 hrs : 38 mins
133
+ [TIMING] data: 3.4ms | fwd: 2018.6ms | loss: 6.4ms | bwd: 4245.1ms | clip: 0.0ms | optim: 49.4ms
134
+ step 38 | loss: 1.491481 | lr 1.1700e-04 | norm: 0.8726 | dt: 6324.11ms | tok/sec: 82903.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
135
+ [TIMING] data: 3.5ms | fwd: 2010.4ms | loss: 6.4ms | bwd: 4246.4ms | clip: 0.0ms | optim: 49.5ms
136
+ step 39 | loss: 1.515249 | lr 1.2000e-04 | norm: 0.6338 | dt: 6316.15ms | tok/sec: 83007.5 | mem: 108909MB | Time Left: 00 hrs : 38 mins
137
+ [TIMING] data: 3.5ms | fwd: 2014.1ms | loss: 6.4ms | bwd: 4234.7ms | clip: 0.0ms | optim: 49.4ms
138
+ step 40 | loss: 1.475689 | lr 1.2300e-04 | norm: 0.7981 | dt: 6356.20ms | tok/sec: 82484.5 | mem: 108909MB | Time Left: 00 hrs : 38 mins
139
+ [TIMING] data: 3.5ms | fwd: 2020.7ms | loss: 6.7ms | bwd: 4268.0ms | clip: 0.0ms | optim: 49.4ms
140
+ step 41 | loss: 1.453508 | lr 1.2600e-04 | norm: 0.6444 | dt: 6302.20ms | tok/sec: 83191.3 | mem: 108909MB | Time Left: 00 hrs : 37 mins
141
+ [TIMING] data: 3.4ms | fwd: 2008.4ms | loss: 6.4ms | bwd: 4226.6ms | clip: 0.0ms | optim: 49.4ms
142
+ step 42 | loss: 1.502223 | lr 1.2900e-04 | norm: 0.6451 | dt: 6360.25ms | tok/sec: 82431.9 | mem: 108909MB | Time Left: 00 hrs : 37 mins
143
+ [TIMING] data: 3.4ms | fwd: 2054.1ms | loss: 6.4ms | bwd: 4239.1ms | clip: 0.0ms | optim: 49.3ms
144
+ step 43 | loss: 1.504963 | lr 1.3200e-04 | norm: 0.6668 | dt: 6317.30ms | tok/sec: 82992.4 | mem: 108909MB | Time Left: 00 hrs : 37 mins
145
+ [TIMING] data: 3.5ms | fwd: 2015.3ms | loss: 6.4ms | bwd: 4234.8ms | clip: 0.0ms | optim: 49.4ms
146
+ step 44 | loss: 1.425290 | lr 1.3500e-04 | norm: 1.2208 | dt: 6359.73ms | tok/sec: 82438.7 | mem: 108909MB | Time Left: 00 hrs : 37 mins
147
+ [TIMING] data: 3.5ms | fwd: 2050.7ms | loss: 6.4ms | bwd: 4241.5ms | clip: 0.0ms | optim: 49.6ms
148
+ step 45 | loss: 1.504238 | lr 1.3800e-04 | norm: 0.6194 | dt: 6322.99ms | tok/sec: 82917.7 | mem: 108909MB | Time Left: 00 hrs : 37 mins
149
+ [TIMING] data: 3.5ms | fwd: 2016.5ms | loss: 6.4ms | bwd: 4238.9ms | clip: 0.0ms | optim: 49.8ms
150
+ step 46 | loss: 1.491822 | lr 1.4100e-04 | norm: 0.6576 | dt: 6328.94ms | tok/sec: 82839.7 | mem: 108909MB | Time Left: 00 hrs : 37 mins
151
+ [TIMING] data: 3.4ms | fwd: 2018.7ms | loss: 6.4ms | bwd: 4243.1ms | clip: 0.0ms | optim: 49.4ms
152
+ step 47 | loss: 1.515073 | lr 1.4400e-04 | norm: 0.7951 | dt: 6324.23ms | tok/sec: 82901.5 | mem: 108909MB | Time Left: 00 hrs : 37 mins
153
+ [TIMING] data: 3.5ms | fwd: 2010.3ms | loss: 6.4ms | bwd: 4246.7ms | clip: 0.0ms | optim: 49.3ms
154
+ step 48 | loss: 1.509569 | lr 1.4700e-04 | norm: 0.9722 | dt: 6319.16ms | tok/sec: 82968.0 | mem: 108909MB | Time Left: 00 hrs : 37 mins
155
+ [TIMING] data: 3.5ms | fwd: 2021.6ms | loss: 6.4ms | bwd: 4230.4ms | clip: 0.0ms | optim: 49.4ms
156
+ step 49 | loss: 1.487431 | lr 1.5000e-04 | norm: 0.6924 | dt: 6332.06ms | tok/sec: 82798.9 | mem: 108909MB | Time Left: 00 hrs : 37 mins
157
+ [TIMING] data: 3.5ms | fwd: 2020.5ms | loss: 6.7ms | bwd: 4243.8ms | clip: 0.0ms | optim: 49.6ms
158
+ step 50 | loss: 1.519441 | lr 1.5300e-04 | norm: 0.6099 | dt: 6325.66ms | tok/sec: 82882.8 | mem: 108909MB | Time Left: 00 hrs : 36 mins
159
+ [TIMING] data: 3.6ms | fwd: 2024.9ms | loss: 6.4ms | bwd: 4233.1ms | clip: 0.0ms | optim: 49.7ms
160
+ step 51 | loss: 1.539274 | lr 1.5600e-04 | norm: 0.6793 | dt: 6339.00ms | tok/sec: 82708.4 | mem: 108909MB | Time Left: 00 hrs : 36 mins
161
+ [TIMING] data: 3.6ms | fwd: 2011.3ms | loss: 6.4ms | bwd: 4260.0ms | clip: 0.0ms | optim: 49.9ms
162
+ step 52 | loss: 1.548381 | lr 1.5900e-04 | norm: 0.5928 | dt: 6296.16ms | tok/sec: 83271.0 | mem: 108909MB | Time Left: 00 hrs : 36 mins
163
+ [TIMING] data: 3.5ms | fwd: 2011.4ms | loss: 6.4ms | bwd: 4217.7ms | clip: 0.0ms | optim: 49.3ms
164
+ step 53 | loss: 1.472854 | lr 1.6200e-04 | norm: 0.8747 | dt: 6324.35ms | tok/sec: 82899.9 | mem: 108909MB | Time Left: 00 hrs : 36 mins
165
+ [TIMING] data: 3.5ms | fwd: 2009.7ms | loss: 6.6ms | bwd: 4247.1ms | clip: 0.0ms | optim: 49.4ms
166
+ step 54 | loss: 1.479821 | lr 1.6500e-04 | norm: 0.7724 | dt: 6332.94ms | tok/sec: 82787.5 | mem: 108909MB | Time Left: 00 hrs : 36 mins
167
+ [TIMING] data: 3.5ms | fwd: 2018.3ms | loss: 6.4ms | bwd: 4247.4ms | clip: 0.0ms | optim: 49.5ms
168
+ step 55 | loss: 1.559876 | lr 1.6800e-04 | norm: 0.6723 | dt: 6319.43ms | tok/sec: 82964.4 | mem: 108909MB | Time Left: 00 hrs : 36 mins
169
+ [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.4ms | bwd: 4242.0ms | clip: 0.0ms | optim: 49.4ms
170
+ step 56 | loss: 1.525480 | lr 1.7100e-04 | norm: 0.6874 | dt: 6329.41ms | tok/sec: 82833.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
171
+ [TIMING] data: 3.6ms | fwd: 2019.1ms | loss: 6.4ms | bwd: 4241.9ms | clip: 0.0ms | optim: 50.6ms
172
+ step 57 | loss: 1.459871 | lr 1.7400e-04 | norm: 0.6680 | dt: 6321.78ms | tok/sec: 82933.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
173
+ [TIMING] data: 3.5ms | fwd: 2011.0ms | loss: 6.4ms | bwd: 4243.3ms | clip: 0.0ms | optim: 49.6ms
174
+ step 58 | loss: 1.471684 | lr 1.7700e-04 | norm: 0.7933 | dt: 6347.73ms | tok/sec: 82594.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
175
+ [TIMING] data: 3.4ms | fwd: 2032.0ms | loss: 6.7ms | bwd: 4248.2ms | clip: 0.0ms | optim: 49.4ms
176
+ step 59 | loss: 1.480517 | lr 1.8000e-04 | norm: 0.7333 | dt: 6323.14ms | tok/sec: 82915.8 | mem: 108909MB | Time Left: 00 hrs : 35 mins
177
+ [TIMING] data: 3.5ms | fwd: 2015.6ms | loss: 6.4ms | bwd: 4240.3ms | clip: 0.0ms | optim: 49.4ms
178
+ step 60 | loss: 1.519410 | lr 1.8300e-04 | norm: 0.7315 | dt: 6331.64ms | tok/sec: 82804.5 | mem: 108909MB | Time Left: 00 hrs : 35 mins
179
+ [TIMING] data: 3.5ms | fwd: 2021.2ms | loss: 6.4ms | bwd: 4243.3ms | clip: 0.0ms | optim: 49.3ms
180
+ step 61 | loss: 1.455516 | lr 1.8600e-04 | norm: 0.6936 | dt: 6323.94ms | tok/sec: 82905.3 | mem: 108909MB | Time Left: 00 hrs : 35 mins
181
+ [TIMING] data: 3.5ms | fwd: 2014.4ms | loss: 6.4ms | bwd: 4242.4ms | clip: 0.0ms | optim: 49.3ms
182
+ step 62 | loss: 1.486469 | lr 1.8900e-04 | norm: 0.7827 | dt: 6324.21ms | tok/sec: 82901.7 | mem: 108909MB | Time Left: 00 hrs : 35 mins
183
+ [TIMING] data: 3.5ms | fwd: 2013.1ms | loss: 6.4ms | bwd: 4244.0ms | clip: 0.0ms | optim: 49.4ms
184
+ step 63 | loss: 1.483447 | lr 1.9200e-04 | norm: 0.8920 | dt: 6335.92ms | tok/sec: 82748.5 | mem: 108909MB | Time Left: 00 hrs : 35 mins
185
+ [TIMING] data: 3.5ms | fwd: 2021.0ms | loss: 6.4ms | bwd: 4247.6ms | clip: 0.0ms | optim: 49.6ms
186
+ step 64 | loss: 1.452058 | lr 1.9500e-04 | norm: 0.8363 | dt: 6364.76ms | tok/sec: 82373.6 | mem: 108909MB | Time Left: 00 hrs : 35 mins
187
+ [TIMING] data: 3.5ms | fwd: 2062.0ms | loss: 6.4ms | bwd: 4235.4ms | clip: 0.0ms | optim: 49.4ms
188
+ step 65 | loss: 1.500889 | lr 1.9800e-04 | norm: 0.8404 | dt: 6336.29ms | tok/sec: 82743.6 | mem: 108909MB | Time Left: 00 hrs : 35 mins
189
+ [TIMING] data: 3.4ms | fwd: 2023.0ms | loss: 6.5ms | bwd: 4245.9ms | clip: 0.0ms | optim: 49.4ms
190
+ step 66 | loss: 1.483080 | lr 2.0100e-04 | norm: 0.7789 | dt: 6334.96ms | tok/sec: 82761.1 | mem: 108909MB | Time Left: 00 hrs : 35 mins
191
+ [TIMING] data: 3.5ms | fwd: 2017.6ms | loss: 6.4ms | bwd: 4250.1ms | clip: 0.0ms | optim: 49.4ms
192
+ step 67 | loss: 1.524449 | lr 2.0400e-04 | norm: 0.8876 | dt: 6310.82ms | tok/sec: 83077.7 | mem: 108909MB | Time Left: 00 hrs : 35 mins
193
+ [TIMING] data: 3.5ms | fwd: 2011.6ms | loss: 6.4ms | bwd: 4232.0ms | clip: 0.0ms | optim: 49.4ms
194
+ step 68 | loss: 1.493887 | lr 2.0700e-04 | norm: 0.6793 | dt: 6330.85ms | tok/sec: 82814.8 | mem: 108909MB | Time Left: 00 hrs : 35 mins
195
+ [TIMING] data: 3.5ms | fwd: 2018.1ms | loss: 6.4ms | bwd: 4244.6ms | clip: 0.0ms | optim: 50.3ms
196
+ step 69 | loss: 1.532737 | lr 2.1000e-04 | norm: 0.7137 | dt: 6315.90ms | tok/sec: 83010.9 | mem: 108909MB | Time Left: 00 hrs : 34 mins
197
+ [TIMING] data: 3.5ms | fwd: 2019.8ms | loss: 6.4ms | bwd: 4228.6ms | clip: 0.0ms | optim: 49.7ms
198
+ step 70 | loss: 1.498729 | lr 2.1300e-04 | norm: 0.8502 | dt: 6321.53ms | tok/sec: 82936.9 | mem: 108909MB | Time Left: 00 hrs : 34 mins
199
+ [TIMING] data: 3.5ms | fwd: 2015.6ms | loss: 6.4ms | bwd: 4238.6ms | clip: 0.0ms | optim: 49.5ms
200
+ step 71 | loss: 1.501005 | lr 2.1600e-04 | norm: 0.6561 | dt: 6370.35ms | tok/sec: 82301.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
201
+ [TIMING] data: 3.5ms | fwd: 2075.7ms | loss: 6.4ms | bwd: 4227.3ms | clip: 0.0ms | optim: 49.5ms
202
+ step 72 | loss: 1.435198 | lr 2.1900e-04 | norm: 0.8811 | dt: 6322.63ms | tok/sec: 82922.5 | mem: 108909MB | Time Left: 00 hrs : 34 mins
203
+ [TIMING] data: 3.5ms | fwd: 2016.4ms | loss: 6.4ms | bwd: 4239.0ms | clip: 0.0ms | optim: 49.5ms
204
+ step 73 | loss: 1.507871 | lr 2.2200e-04 | norm: 0.8746 | dt: 6320.86ms | tok/sec: 82945.7 | mem: 108909MB | Time Left: 00 hrs : 34 mins
205
+ [TIMING] data: 3.5ms | fwd: 2010.0ms | loss: 6.4ms | bwd: 4243.6ms | clip: 0.0ms | optim: 49.4ms
206
+ step 74 | loss: 1.489377 | lr 2.2500e-04 | norm: 0.8673 | dt: 6323.05ms | tok/sec: 82916.9 | mem: 108909MB | Time Left: 00 hrs : 34 mins
207
+ [TIMING] data: 3.5ms | fwd: 2014.3ms | loss: 6.4ms | bwd: 4241.4ms | clip: 0.0ms | optim: 49.5ms
208
+ step 75 | loss: 1.502547 | lr 2.2800e-04 | norm: 0.7430 | dt: 6332.58ms | tok/sec: 82792.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
209
+ [TIMING] data: 3.5ms | fwd: 2015.2ms | loss: 6.4ms | bwd: 4250.1ms | clip: 0.0ms | optim: 49.4ms
210
+ step 76 | loss: 1.455447 | lr 2.3100e-04 | norm: 0.7772 | dt: 6310.79ms | tok/sec: 83078.1 | mem: 108909MB | Time Left: 00 hrs : 34 mins
211
+ [TIMING] data: 3.5ms | fwd: 2011.1ms | loss: 6.4ms | bwd: 4232.4ms | clip: 0.0ms | optim: 49.4ms
212
+ step 77 | loss: 1.513057 | lr 2.3400e-04 | norm: 0.6982 | dt: 6349.46ms | tok/sec: 82572.1 | mem: 108909MB | Time Left: 00 hrs : 34 mins
213
+ [TIMING] data: 3.6ms | fwd: 2030.2ms | loss: 6.4ms | bwd: 4252.0ms | clip: 0.0ms | optim: 49.3ms
214
+ step 78 | loss: 1.487326 | lr 2.3700e-04 | norm: 0.7463 | dt: 6344.28ms | tok/sec: 82639.5 | mem: 108909MB | Time Left: 00 hrs : 34 mins
215
+ [TIMING] data: 3.6ms | fwd: 2046.4ms | loss: 6.4ms | bwd: 4230.1ms | clip: 0.0ms | optim: 49.8ms
216
+ step 79 | loss: 1.497366 | lr 2.4000e-04 | norm: 0.7971 | dt: 6336.67ms | tok/sec: 82738.7 | mem: 108909MB | Time Left: 00 hrs : 33 mins
217
+ [TIMING] data: 3.5ms | fwd: 2010.9ms | loss: 6.6ms | bwd: 4258.2ms | clip: 0.0ms | optim: 49.5ms
218
+ step 80 | loss: 1.496414 | lr 2.4300e-04 | norm: 0.6843 | dt: 6305.02ms | tok/sec: 83154.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
219
+ [TIMING] data: 3.4ms | fwd: 2012.4ms | loss: 6.4ms | bwd: 4225.0ms | clip: 0.0ms | optim: 49.9ms
220
+ step 81 | loss: 1.544732 | lr 2.4600e-04 | norm: 0.7336 | dt: 6309.07ms | tok/sec: 83100.7 | mem: 108909MB | Time Left: 00 hrs : 33 mins
221
+ [TIMING] data: 3.4ms | fwd: 2023.1ms | loss: 6.4ms | bwd: 4218.9ms | clip: 0.0ms | optim: 49.3ms
222
+ step 82 | loss: 1.565590 | lr 2.4900e-04 | norm: 0.7821 | dt: 6314.54ms | tok/sec: 83028.7 | mem: 108909MB | Time Left: 00 hrs : 33 mins
223
+ [TIMING] data: 3.4ms | fwd: 2020.2ms | loss: 6.4ms | bwd: 4227.3ms | clip: 0.0ms | optim: 49.3ms
224
+ step 83 | loss: 1.488052 | lr 2.5200e-04 | norm: 0.7503 | dt: 6320.02ms | tok/sec: 82956.7 | mem: 108909MB | Time Left: 00 hrs : 33 mins
225
+ [TIMING] data: 3.6ms | fwd: 2012.4ms | loss: 6.4ms | bwd: 4240.3ms | clip: 0.0ms | optim: 49.4ms
226
+ step 84 | loss: 1.506120 | lr 2.5500e-04 | norm: 0.8259 | dt: 6357.75ms | tok/sec: 82464.4 | mem: 108909MB | Time Left: 00 hrs : 33 mins
227
+ [TIMING] data: 3.5ms | fwd: 2063.1ms | loss: 6.4ms | bwd: 4227.5ms | clip: 0.0ms | optim: 49.3ms
228
+ step 85 | loss: 1.508562 | lr 2.5800e-04 | norm: 0.8080 | dt: 6323.59ms | tok/sec: 82909.9 | mem: 108909MB | Time Left: 00 hrs : 33 mins
229
+ [TIMING] data: 3.6ms | fwd: 2016.4ms | loss: 6.4ms | bwd: 4240.0ms | clip: 0.0ms | optim: 49.3ms
230
+ step 86 | loss: 1.506531 | lr 2.6100e-04 | norm: 0.8455 | dt: 6321.49ms | tok/sec: 82937.4 | mem: 108909MB | Time Left: 00 hrs : 33 mins
231
+ [TIMING] data: 3.4ms | fwd: 2012.8ms | loss: 6.4ms | bwd: 4241.1ms | clip: 0.0ms | optim: 49.9ms
232
+ step 87 | loss: 1.568609 | lr 2.6400e-04 | norm: 0.8385 | dt: 6346.16ms | tok/sec: 82615.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
233
+ [TIMING] data: 3.4ms | fwd: 2055.7ms | loss: 6.4ms | bwd: 4223.2ms | clip: 0.0ms | optim: 49.6ms
234
+ step 88 | loss: 1.578616 | lr 2.6700e-04 | norm: 0.8432 | dt: 6316.88ms | tok/sec: 82997.9 | mem: 108909MB | Time Left: 00 hrs : 32 mins
235
+ [TIMING] data: 3.4ms | fwd: 2016.8ms | loss: 6.4ms | bwd: 4232.9ms | clip: 0.0ms | optim: 49.4ms
236
+ step 89 | loss: 1.520792 | lr 2.7000e-04 | norm: 0.8000 | dt: 6328.62ms | tok/sec: 82844.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
237
+ [TIMING] data: 3.7ms | fwd: 1999.6ms | loss: 6.4ms | bwd: 4261.6ms | clip: 0.0ms | optim: 49.4ms
238
+ step 90 | loss: 1.589205 | lr 2.7300e-04 | norm: 0.7820 | dt: 6301.03ms | tok/sec: 83206.7 | mem: 108909MB | Time Left: 00 hrs : 32 mins
239
+ [TIMING] data: 3.5ms | fwd: 2015.3ms | loss: 6.4ms | bwd: 4218.2ms | clip: 0.0ms | optim: 49.7ms
240
+ step 91 | loss: 1.535868 | lr 2.7600e-04 | norm: 0.8531 | dt: 6363.63ms | tok/sec: 82388.1 | mem: 108909MB | Time Left: 00 hrs : 32 mins
241
+ [TIMING] data: 3.5ms | fwd: 2066.5ms | loss: 6.4ms | bwd: 4229.5ms | clip: 0.0ms | optim: 49.7ms
242
+ step 92 | loss: 1.544926 | lr 2.7900e-04 | norm: 0.7781 | dt: 6364.68ms | tok/sec: 82374.6 | mem: 108909MB | Time Left: 00 hrs : 32 mins
243
+ [TIMING] data: 3.5ms | fwd: 2043.3ms | loss: 6.4ms | bwd: 4254.1ms | clip: 0.0ms | optim: 49.5ms
244
+ step 93 | loss: 1.568713 | lr 2.8200e-04 | norm: 0.7326 | dt: 6315.09ms | tok/sec: 83021.5 | mem: 108909MB | Time Left: 00 hrs : 32 mins
245
+ [TIMING] data: 3.4ms | fwd: 2012.7ms | loss: 6.4ms | bwd: 4235.1ms | clip: 0.0ms | optim: 49.5ms
246
+ step 94 | loss: 1.447728 | lr 2.8500e-04 | norm: 0.8374 | dt: 6328.15ms | tok/sec: 82850.1 | mem: 108909MB | Time Left: 00 hrs : 32 mins
247
+ [TIMING] data: 3.5ms | fwd: 2018.8ms | loss: 6.4ms | bwd: 4241.1ms | clip: 0.0ms | optim: 50.0ms
248
+ step 95 | loss: 1.528693 | lr 2.8800e-04 | norm: 0.7675 | dt: 6318.46ms | tok/sec: 82977.2 | mem: 108909MB | Time Left: 00 hrs : 32 mins
249
+ [TIMING] data: 3.4ms | fwd: 2017.1ms | loss: 6.4ms | bwd: 4234.2ms | clip: 0.0ms | optim: 49.4ms
250
+ step 96 | loss: 1.526013 | lr 2.9100e-04 | norm: 0.7406 | dt: 6334.85ms | tok/sec: 82762.5 | mem: 108909MB | Time Left: 00 hrs : 32 mins
251
+ [TIMING] data: 3.4ms | fwd: 2018.0ms | loss: 6.4ms | bwd: 4249.8ms | clip: 0.0ms | optim: 49.3ms
252
+ step 97 | loss: 1.522310 | lr 2.9400e-04 | norm: 0.7262 | dt: 6366.42ms | tok/sec: 82352.1 | mem: 108909MB | Time Left: 00 hrs : 32 mins
253
+ [TIMING] data: 3.5ms | fwd: 2054.2ms | loss: 6.4ms | bwd: 4244.9ms | clip: 0.0ms | optim: 49.4ms
254
+ step 98 | loss: 1.517379 | lr 2.9700e-04 | norm: 0.7505 | dt: 6335.56ms | tok/sec: 82753.2 | mem: 108909MB | Time Left: 00 hrs : 31 mins
255
+ [TIMING] data: 3.5ms | fwd: 2027.7ms | loss: 6.4ms | bwd: 4240.7ms | clip: 0.0ms | optim: 49.3ms
256
+ step 99 | loss: 1.542074 | lr 3.0000e-04 | norm: 0.9468 | dt: 6316.63ms | tok/sec: 83001.2 | mem: 108909MB | Time Left: 00 hrs : 31 mins
257
+ [TIMING] data: 3.4ms | fwd: 2015.2ms | loss: 6.4ms | bwd: 4234.2ms | clip: 0.0ms | optim: 49.4ms
258
+ [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run13_step_100.ckpt (safely)
259
+ step 100 | loss: 1.560971 | lr 3.0000e-04 | norm: 0.8281 | dt: 6193.54ms | tok/sec: 84650.8 | mem: 108909MB | Time Left: 00 hrs : 30 mins
260
+ [TIMING] data: 3.4ms | fwd: 1987.0ms | loss: 6.4ms | bwd: 4139.4ms | clip: 0.0ms | optim: 49.4ms
261
+ step 101 | loss: 1.590868 | lr 2.9999e-04 | norm: 0.7858 | dt: 6223.60ms | tok/sec: 84241.9 | mem: 108909MB | Time Left: 00 hrs : 31 mins
262
+ [TIMING] data: 3.3ms | fwd: 1992.2ms | loss: 6.4ms | bwd: 4164.1ms | clip: 0.0ms | optim: 49.6ms
263
+ step 102 | loss: 1.500099 | lr 2.9998e-04 | norm: 0.7740 | dt: 6261.98ms | tok/sec: 83725.6 | mem: 108909MB | Time Left: 00 hrs : 31 mins
264
+ [TIMING] data: 3.4ms | fwd: 1995.6ms | loss: 6.4ms | bwd: 4199.4ms | clip: 0.0ms | optim: 49.3ms
265
+ step 103 | loss: 1.553445 | lr 2.9995e-04 | norm: 0.7512 | dt: 6281.82ms | tok/sec: 83461.2 | mem: 108909MB | Time Left: 00 hrs : 31 mins
266
+ [TIMING] data: 3.3ms | fwd: 2017.0ms | loss: 6.4ms | bwd: 4197.8ms | clip: 0.0ms | optim: 49.4ms
267
+ step 104 | loss: 1.622695 | lr 2.9991e-04 | norm: 0.9523 | dt: 6280.21ms | tok/sec: 83482.6 | mem: 108909MB | Time Left: 00 hrs : 30 mins
268
+ [TIMING] data: 3.4ms | fwd: 2006.3ms | loss: 6.4ms | bwd: 4206.8ms | clip: 0.0ms | optim: 49.3ms
269
+ step 105 | loss: 1.534858 | lr 2.9986e-04 | norm: 0.8496 | dt: 6321.54ms | tok/sec: 82936.8 | mem: 108909MB | Time Left: 00 hrs : 31 mins
270
+ [TIMING] data: 3.4ms | fwd: 2015.3ms | loss: 6.4ms | bwd: 4239.0ms | clip: 0.0ms | optim: 49.4ms
271
+ step 106 | loss: 1.540255 | lr 2.9980e-04 | norm: 0.9536 | dt: 6364.30ms | tok/sec: 82379.5 | mem: 108909MB | Time Left: 00 hrs : 31 mins
272
+ [TIMING] data: 3.3ms | fwd: 2052.6ms | loss: 6.4ms | bwd: 4244.7ms | clip: 0.0ms | optim: 49.4ms
273
+ step 107 | loss: 1.606415 | lr 2.9973e-04 | norm: 0.7491 | dt: 6333.17ms | tok/sec: 82784.5 | mem: 108909MB | Time Left: 00 hrs : 30 mins
274
+ [TIMING] data: 3.5ms | fwd: 2016.5ms | loss: 6.4ms | bwd: 4249.4ms | clip: 0.0ms | optim: 49.5ms
275
+ step 108 | loss: 1.496417 | lr 2.9965e-04 | norm: 0.9238 | dt: 6313.81ms | tok/sec: 83038.3 | mem: 108909MB | Time Left: 00 hrs : 30 mins
276
+ [TIMING] data: 3.3ms | fwd: 2012.1ms | loss: 6.4ms | bwd: 4234.7ms | clip: 0.0ms | optim: 49.3ms
277
+ step 109 | loss: 1.536228 | lr 2.9956e-04 | norm: 0.7126 | dt: 6318.94ms | tok/sec: 82970.9 | mem: 108909MB | Time Left: 00 hrs : 30 mins
278
+ [TIMING] data: 3.4ms | fwd: 2017.6ms | loss: 6.4ms | bwd: 4233.7ms | clip: 0.0ms | optim: 49.9ms
279
+ step 110 | loss: 1.559572 | lr 2.9945e-04 | norm: 0.8443 | dt: 6326.96ms | tok/sec: 82865.7 | mem: 108909MB | Time Left: 00 hrs : 30 mins
280
+ [TIMING] data: 3.3ms | fwd: 2019.8ms | loss: 6.4ms | bwd: 4239.7ms | clip: 0.0ms | optim: 49.8ms
281
+ step 111 | loss: 1.541899 | lr 2.9934e-04 | norm: 0.7483 | dt: 6321.80ms | tok/sec: 82933.4 | mem: 108909MB | Time Left: 00 hrs : 30 mins
282
+ [TIMING] data: 3.4ms | fwd: 2009.3ms | loss: 6.6ms | bwd: 4245.2ms | clip: 0.0ms | optim: 49.2ms
283
+ step 112 | loss: 1.518180 | lr 2.9921e-04 | norm: 0.9077 | dt: 6326.17ms | tok/sec: 82876.0 | mem: 108909MB | Time Left: 00 hrs : 30 mins
284
+ [TIMING] data: 3.4ms | fwd: 2020.6ms | loss: 6.4ms | bwd: 4238.0ms | clip: 0.0ms | optim: 49.7ms
285
+ step 113 | loss: 1.504054 | lr 2.9907e-04 | norm: 0.8560 | dt: 6305.27ms | tok/sec: 83150.7 | mem: 108909MB | Time Left: 00 hrs : 30 mins
286
+ [TIMING] data: 3.4ms | fwd: 2015.3ms | loss: 6.4ms | bwd: 4223.0ms | clip: 0.0ms | optim: 49.3ms
287
+ step 114 | loss: 1.536195 | lr 2.9893e-04 | norm: 0.6512 | dt: 6374.01ms | tok/sec: 82254.0 | mem: 108909MB | Time Left: 00 hrs : 30 mins
288
+ [TIMING] data: 3.5ms | fwd: 2062.1ms | loss: 6.4ms | bwd: 4244.8ms | clip: 0.0ms | optim: 49.3ms
289
+ step 115 | loss: 1.523702 | lr 2.9877e-04 | norm: 0.7223 | dt: 6380.53ms | tok/sec: 82170.0 | mem: 108909MB | Time Left: 00 hrs : 30 mins
290
+ [TIMING] data: 3.3ms | fwd: 2062.4ms | loss: 6.4ms | bwd: 4250.8ms | clip: 0.0ms | optim: 49.6ms
291
+ step 116 | loss: 1.471415 | lr 2.9860e-04 | norm: 0.9105 | dt: 6321.89ms | tok/sec: 82932.1 | mem: 108909MB | Time Left: 00 hrs : 29 mins
292
+ [TIMING] data: 3.4ms | fwd: 2014.8ms | loss: 6.4ms | bwd: 4239.0ms | clip: 0.0ms | optim: 50.4ms
293
+ step 117 | loss: 1.486171 | lr 2.9842e-04 | norm: 0.8120 | dt: 6331.32ms | tok/sec: 82808.6 | mem: 108909MB | Time Left: 00 hrs : 29 mins
294
+ [TIMING] data: 3.4ms | fwd: 2022.3ms | loss: 6.4ms | bwd: 4241.8ms | clip: 0.0ms | optim: 49.5ms
295
+ step 118 | loss: 1.551453 | lr 2.9823e-04 | norm: 0.7651 | dt: 6317.55ms | tok/sec: 82989.2 | mem: 108909MB | Time Left: 00 hrs : 29 mins
296
+ [TIMING] data: 3.4ms | fwd: 2016.7ms | loss: 6.5ms | bwd: 4233.6ms | clip: 0.0ms | optim: 49.4ms
297
+ step 119 | loss: 1.563141 | lr 2.9803e-04 | norm: 0.7465 | dt: 6339.21ms | tok/sec: 82705.6 | mem: 108909MB | Time Left: 00 hrs : 29 mins
298
+ [TIMING] data: 3.3ms | fwd: 2017.5ms | loss: 6.4ms | bwd: 4254.6ms | clip: 0.0ms | optim: 49.4ms
299
+ step 120 | loss: 1.506313 | lr 2.9781e-04 | norm: 0.6294 | dt: 6324.94ms | tok/sec: 82892.2 | mem: 108909MB | Time Left: 00 hrs : 29 mins
300
+ [TIMING] data: 3.3ms | fwd: 2018.4ms | loss: 6.4ms | bwd: 4239.5ms | clip: 0.0ms | optim: 49.4ms
301
+ step 121 | loss: 1.531366 | lr 2.9759e-04 | norm: 0.7520 | dt: 6304.79ms | tok/sec: 83157.2 | mem: 108909MB | Time Left: 00 hrs : 29 mins
302
+ [TIMING] data: 3.4ms | fwd: 2019.4ms | loss: 6.4ms | bwd: 4218.3ms | clip: 0.0ms | optim: 49.4ms
303
+ step 122 | loss: 1.583121 | lr 2.9736e-04 | norm: 0.7222 | dt: 6373.51ms | tok/sec: 82260.5 | mem: 108909MB | Time Left: 00 hrs : 29 mins
304
+ [TIMING] data: 3.3ms | fwd: 2064.9ms | loss: 6.4ms | bwd: 4241.5ms | clip: 0.0ms | optim: 49.5ms
305
+ step 123 | loss: 1.549728 | lr 2.9711e-04 | norm: 0.6932 | dt: 6385.67ms | tok/sec: 82103.8 | mem: 108909MB | Time Left: 00 hrs : 29 mins
306
+ [TIMING] data: 3.4ms | fwd: 2075.5ms | loss: 6.4ms | bwd: 4243.0ms | clip: 0.0ms | optim: 49.4ms
307
+ step 124 | loss: 1.571770 | lr 2.9686e-04 | norm: 0.7142 | dt: 6307.75ms | tok/sec: 83118.0 | mem: 108909MB | Time Left: 00 hrs : 29 mins
308
+ [TIMING] data: 3.4ms | fwd: 2008.9ms | loss: 6.4ms | bwd: 4231.8ms | clip: 0.0ms | optim: 49.3ms
309
+ step 125 | loss: 1.530097 | lr 2.9659e-04 | norm: 0.5959 | dt: 6302.30ms | tok/sec: 83190.0 | mem: 108909MB | Time Left: 00 hrs : 28 mins
310
+ [TIMING] data: 3.4ms | fwd: 2014.8ms | loss: 6.4ms | bwd: 4220.4ms | clip: 0.0ms | optim: 49.4ms
311
+ step 126 | loss: 1.548526 | lr 2.9632e-04 | norm: 0.6799 | dt: 6342.53ms | tok/sec: 82662.3 | mem: 108909MB | Time Left: 00 hrs : 28 mins
312
+ [TIMING] data: 3.4ms | fwd: 2019.1ms | loss: 6.4ms | bwd: 4256.5ms | clip: 0.0ms | optim: 49.3ms
313
+ step 127 | loss: 1.587535 | lr 2.9603e-04 | norm: 0.5902 | dt: 6309.10ms | tok/sec: 83100.2 | mem: 108909MB | Time Left: 00 hrs : 28 mins
314
+ [TIMING] data: 3.4ms | fwd: 2015.9ms | loss: 6.4ms | bwd: 4225.9ms | clip: 0.0ms | optim: 49.6ms
315
+ step 128 | loss: 1.606323 | lr 2.9573e-04 | norm: 0.6112 | dt: 6359.27ms | tok/sec: 82444.7 | mem: 108909MB | Time Left: 00 hrs : 28 mins
316
+ [TIMING] data: 3.4ms | fwd: 2067.1ms | loss: 6.4ms | bwd: 4225.1ms | clip: 0.0ms | optim: 49.3ms
317
+ step 129 | loss: 1.556684 | lr 2.9542e-04 | norm: 0.6113 | dt: 6341.04ms | tok/sec: 82681.8 | mem: 108909MB | Time Left: 00 hrs : 28 mins
318
+ [TIMING] data: 3.4ms | fwd: 2057.6ms | loss: 6.4ms | bwd: 4216.4ms | clip: 0.0ms | optim: 49.3ms
319
+ step 130 | loss: 1.676277 | lr 2.9511e-04 | norm: 0.7016 | dt: 6325.64ms | tok/sec: 82883.0 | mem: 108909MB | Time Left: 00 hrs : 28 mins
320
+ [TIMING] data: 3.4ms | fwd: 2015.0ms | loss: 6.4ms | bwd: 4242.7ms | clip: 0.0ms | optim: 49.4ms
321
+ step 131 | loss: 1.559720 | lr 2.9478e-04 | norm: 0.6344 | dt: 6334.89ms | tok/sec: 82762.0 | mem: 108909MB | Time Left: 00 hrs : 28 mins
322
+ [TIMING] data: 3.3ms | fwd: 2019.9ms | loss: 6.4ms | bwd: 4247.9ms | clip: 0.0ms | optim: 49.4ms
323
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_train_00001.bin
324
+ 100000000
325
+ step 132 | loss: 1.561504 | lr 2.9444e-04 | norm: 0.6851 | dt: 6315.48ms | tok/sec: 83016.3 | mem: 108909MB | Time Left: 00 hrs : 28 mins
326
+ [TIMING] data: 15.4ms | fwd: 2015.8ms | loss: 6.4ms | bwd: 4220.3ms | clip: 0.0ms | optim: 49.7ms
327
+ step 133 | loss: 1.544661 | lr 2.9409e-04 | norm: 0.5797 | dt: 6326.40ms | tok/sec: 82873.1 | mem: 108909MB | Time Left: 00 hrs : 28 mins
328
+ [TIMING] data: 3.3ms | fwd: 2020.0ms | loss: 6.4ms | bwd: 4239.4ms | clip: 0.0ms | optim: 49.4ms
329
+ step 134 | loss: 1.552186 | lr 2.9373e-04 | norm: 0.6372 | dt: 6314.07ms | tok/sec: 83034.9 | mem: 108909MB | Time Left: 00 hrs : 27 mins
330
+ [TIMING] data: 3.4ms | fwd: 2019.4ms | loss: 6.4ms | bwd: 4227.6ms | clip: 0.0ms | optim: 49.3ms
331
+ step 135 | loss: 1.576610 | lr 2.9336e-04 | norm: 0.5871 | dt: 6309.44ms | tok/sec: 83095.8 | mem: 108909MB | Time Left: 00 hrs : 27 mins
332
+ [TIMING] data: 3.4ms | fwd: 2016.4ms | loss: 6.4ms | bwd: 4225.9ms | clip: 0.0ms | optim: 49.4ms
333
+ step 136 | loss: 1.529440 | lr 2.9298e-04 | norm: 0.6040 | dt: 6360.19ms | tok/sec: 82432.7 | mem: 108909MB | Time Left: 00 hrs : 27 mins
334
+ [TIMING] data: 3.3ms | fwd: 2091.1ms | loss: 6.4ms | bwd: 4202.1ms | clip: 0.0ms | optim: 49.4ms
335
+ step 137 | loss: 1.524042 | lr 2.9259e-04 | norm: 0.5933 | dt: 6319.37ms | tok/sec: 82965.3 | mem: 108909MB | Time Left: 00 hrs : 27 mins
336
+ [TIMING] data: 3.3ms | fwd: 2019.4ms | loss: 6.4ms | bwd: 4231.8ms | clip: 0.0ms | optim: 49.4ms
337
+ step 138 | loss: 1.582924 | lr 2.9219e-04 | norm: 0.5469 | dt: 6309.43ms | tok/sec: 83095.9 | mem: 108909MB | Time Left: 00 hrs : 27 mins
338
+ [TIMING] data: 3.4ms | fwd: 2018.0ms | loss: 6.4ms | bwd: 4224.4ms | clip: 0.0ms | optim: 49.4ms
339
+ step 139 | loss: 1.578228 | lr 2.9178e-04 | norm: 0.7709 | dt: 6318.26ms | tok/sec: 82979.8 | mem: 108909MB | Time Left: 00 hrs : 27 mins
340
+ [TIMING] data: 3.4ms | fwd: 2007.4ms | loss: 6.6ms | bwd: 4243.6ms | clip: 0.0ms | optim: 49.4ms
341
+ step 140 | loss: 1.595851 | lr 2.9135e-04 | norm: 0.5805 | dt: 6296.67ms | tok/sec: 83264.3 | mem: 108909MB | Time Left: 00 hrs : 27 mins
342
+ [TIMING] data: 3.5ms | fwd: 2014.1ms | loss: 6.4ms | bwd: 4214.2ms | clip: 0.0ms | optim: 50.4ms
343
+ step 141 | loss: 1.587405 | lr 2.9092e-04 | norm: 0.6079 | dt: 6355.14ms | tok/sec: 82498.3 | mem: 108909MB | Time Left: 00 hrs : 27 mins
344
+ [TIMING] data: 3.4ms | fwd: 2052.6ms | loss: 6.4ms | bwd: 4235.1ms | clip: 0.0ms | optim: 49.7ms
345
+ step 142 | loss: 1.570241 | lr 2.9048e-04 | norm: 0.5607 | dt: 6316.69ms | tok/sec: 83000.5 | mem: 108909MB | Time Left: 00 hrs : 27 mins
346
+ [TIMING] data: 3.3ms | fwd: 2013.2ms | loss: 6.4ms | bwd: 4236.3ms | clip: 0.0ms | optim: 49.5ms
347
+ step 143 | loss: 1.514309 | lr 2.9003e-04 | norm: 1.8534 | dt: 6304.19ms | tok/sec: 83165.0 | mem: 108909MB | Time Left: 00 hrs : 27 mins
348
+ [TIMING] data: 3.3ms | fwd: 2014.8ms | loss: 6.4ms | bwd: 4222.2ms | clip: 0.0ms | optim: 49.5ms
349
+ step 144 | loss: 1.589352 | lr 2.8957e-04 | norm: 0.5719 | dt: 6310.08ms | tok/sec: 83087.4 | mem: 108909MB | Time Left: 00 hrs : 26 mins
350
+ [TIMING] data: 3.4ms | fwd: 2014.5ms | loss: 6.6ms | bwd: 4227.9ms | clip: 0.0ms | optim: 49.7ms
351
+ step 145 | loss: 1.502340 | lr 2.8910e-04 | norm: 0.5646 | dt: 6302.00ms | tok/sec: 83194.0 | mem: 108909MB | Time Left: 00 hrs : 26 mins
352
+ [TIMING] data: 3.3ms | fwd: 2008.6ms | loss: 6.4ms | bwd: 4226.4ms | clip: 0.0ms | optim: 49.3ms
353
+ step 146 | loss: 1.517694 | lr 2.8862e-04 | norm: 0.5770 | dt: 6306.29ms | tok/sec: 83137.3 | mem: 108909MB | Time Left: 00 hrs : 26 mins
354
+ [TIMING] data: 3.4ms | fwd: 2009.6ms | loss: 6.4ms | bwd: 4229.5ms | clip: 0.0ms | optim: 49.5ms
355
+ step 147 | loss: 1.520943 | lr 2.8813e-04 | norm: 0.5190 | dt: 6292.18ms | tok/sec: 83323.8 | mem: 108909MB | Time Left: 00 hrs : 26 mins
356
+ [TIMING] data: 3.3ms | fwd: 2014.0ms | loss: 6.4ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.3ms
357
+ step 148 | loss: 1.562016 | lr 2.8763e-04 | norm: 0.6044 | dt: 6279.47ms | tok/sec: 83492.4 | mem: 108909MB | Time Left: 00 hrs : 26 mins
358
+ [TIMING] data: 3.4ms | fwd: 2011.2ms | loss: 6.4ms | bwd: 4201.2ms | clip: 0.0ms | optim: 49.4ms
359
+ step 149 | loss: 1.563453 | lr 2.8712e-04 | norm: 0.6146 | dt: 6333.33ms | tok/sec: 82782.4 | mem: 108909MB | Time Left: 00 hrs : 26 mins
360
+ [TIMING] data: 3.4ms | fwd: 2044.6ms | loss: 6.4ms | bwd: 4221.5ms | clip: 0.0ms | optim: 49.5ms
361
+ step 150 | loss: 1.517942 | lr 2.8660e-04 | norm: 0.6800 | dt: 6318.64ms | tok/sec: 82974.8 | mem: 108909MB | Time Left: 00 hrs : 26 mins
362
+ [TIMING] data: 3.4ms | fwd: 2032.5ms | loss: 6.4ms | bwd: 4219.1ms | clip: 0.0ms | optim: 49.3ms
363
+ step 151 | loss: 1.538941 | lr 2.8607e-04 | norm: 0.5141 | dt: 6293.21ms | tok/sec: 83310.2 | mem: 108909MB | Time Left: 00 hrs : 26 mins
364
+ [TIMING] data: 3.4ms | fwd: 2011.1ms | loss: 6.4ms | bwd: 4214.7ms | clip: 0.0ms | optim: 49.6ms
365
+ step 152 | loss: 1.583701 | lr 2.8554e-04 | norm: 0.4730 | dt: 6299.01ms | tok/sec: 83233.4 | mem: 108909MB | Time Left: 00 hrs : 26 mins
366
+ [TIMING] data: 3.4ms | fwd: 2007.5ms | loss: 6.4ms | bwd: 4224.3ms | clip: 0.0ms | optim: 49.5ms
367
+ step 153 | loss: 1.539420 | lr 2.8499e-04 | norm: 0.6077 | dt: 6288.10ms | tok/sec: 83377.8 | mem: 108909MB | Time Left: 00 hrs : 25 mins
368
+ [TIMING] data: 3.4ms | fwd: 2009.9ms | loss: 6.4ms | bwd: 4211.2ms | clip: 0.0ms | optim: 49.4ms
369
+ step 154 | loss: 1.568313 | lr 2.8443e-04 | norm: 0.7170 | dt: 6303.74ms | tok/sec: 83170.9 | mem: 108909MB | Time Left: 00 hrs : 25 mins
370
+ [TIMING] data: 3.3ms | fwd: 2015.0ms | loss: 6.4ms | bwd: 4221.6ms | clip: 0.0ms | optim: 49.4ms
371
+ step 155 | loss: 1.547035 | lr 2.8387e-04 | norm: 0.4806 | dt: 6288.74ms | tok/sec: 83369.3 | mem: 108909MB | Time Left: 00 hrs : 25 mins
372
+ [TIMING] data: 3.3ms | fwd: 2009.2ms | loss: 6.4ms | bwd: 4212.5ms | clip: 0.0ms | optim: 49.4ms
373
+ step 156 | loss: 1.542251 | lr 2.8329e-04 | norm: 0.6079 | dt: 6310.97ms | tok/sec: 83075.7 | mem: 108909MB | Time Left: 00 hrs : 25 mins
374
+ [TIMING] data: 3.4ms | fwd: 2015.4ms | loss: 6.4ms | bwd: 4228.5ms | clip: 0.0ms | optim: 49.4ms
375
+ step 157 | loss: 1.593042 | lr 2.8271e-04 | norm: 0.5018 | dt: 6307.18ms | tok/sec: 83125.6 | mem: 108909MB | Time Left: 00 hrs : 25 mins
376
+ [TIMING] data: 3.5ms | fwd: 2012.0ms | loss: 6.4ms | bwd: 4227.8ms | clip: 0.0ms | optim: 49.7ms
377
+ step 158 | loss: 1.604503 | lr 2.8211e-04 | norm: 0.6080 | dt: 6336.21ms | tok/sec: 82744.7 | mem: 108909MB | Time Left: 00 hrs : 25 mins
378
+ [TIMING] data: 3.4ms | fwd: 2065.8ms | loss: 6.4ms | bwd: 4203.2ms | clip: 0.0ms | optim: 49.5ms
379
+ step 159 | loss: 1.554489 | lr 2.8151e-04 | norm: 0.5733 | dt: 6309.68ms | tok/sec: 83092.6 | mem: 108909MB | Time Left: 00 hrs : 25 mins
380
+ [TIMING] data: 3.4ms | fwd: 2021.8ms | loss: 6.4ms | bwd: 4220.8ms | clip: 0.0ms | optim: 49.4ms
381
+ step 160 | loss: 1.589306 | lr 2.8090e-04 | norm: 0.4747 | dt: 6356.37ms | tok/sec: 82482.4 | mem: 108909MB | Time Left: 00 hrs : 25 mins
382
+ [TIMING] data: 3.5ms | fwd: 2075.3ms | loss: 6.4ms | bwd: 4213.9ms | clip: 0.0ms | optim: 49.5ms
383
+ step 161 | loss: 1.558533 | lr 2.8028e-04 | norm: 0.4875 | dt: 6343.32ms | tok/sec: 82651.9 | mem: 108909MB | Time Left: 00 hrs : 25 mins
384
+ [TIMING] data: 3.3ms | fwd: 2045.2ms | loss: 6.4ms | bwd: 4230.9ms | clip: 0.0ms | optim: 49.5ms
385
+ step 162 | loss: 1.552307 | lr 2.7965e-04 | norm: 0.4857 | dt: 6305.58ms | tok/sec: 83146.6 | mem: 108909MB | Time Left: 00 hrs : 25 mins
386
+ [TIMING] data: 3.4ms | fwd: 2016.4ms | loss: 6.4ms | bwd: 4222.1ms | clip: 0.0ms | optim: 49.4ms
387
+ step 163 | loss: 1.561074 | lr 2.7902e-04 | norm: 0.4315 | dt: 6313.94ms | tok/sec: 83036.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
388
+ [TIMING] data: 3.2ms | fwd: 2015.4ms | loss: 6.4ms | bwd: 4231.3ms | clip: 0.0ms | optim: 49.4ms
389
+ step 164 | loss: 1.494718 | lr 2.7837e-04 | norm: 0.4518 | dt: 6313.21ms | tok/sec: 83046.2 | mem: 108909MB | Time Left: 00 hrs : 24 mins
390
+ [TIMING] data: 3.4ms | fwd: 2010.0ms | loss: 6.4ms | bwd: 4236.0ms | clip: 0.0ms | optim: 49.5ms
391
+ step 165 | loss: 1.550173 | lr 2.7771e-04 | norm: 0.5249 | dt: 6329.39ms | tok/sec: 82833.9 | mem: 108909MB | Time Left: 00 hrs : 24 mins
392
+ [TIMING] data: 3.3ms | fwd: 2015.0ms | loss: 6.4ms | bwd: 4247.3ms | clip: 0.0ms | optim: 49.5ms
393
+ step 166 | loss: 1.549638 | lr 2.7705e-04 | norm: 0.5183 | dt: 6318.25ms | tok/sec: 82979.9 | mem: 108909MB | Time Left: 00 hrs : 24 mins
394
+ [TIMING] data: 3.4ms | fwd: 2019.4ms | loss: 6.4ms | bwd: 4231.6ms | clip: 0.0ms | optim: 49.5ms
395
+ step 167 | loss: 1.567677 | lr 2.7638e-04 | norm: 0.5612 | dt: 6319.49ms | tok/sec: 82963.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
396
+ [TIMING] data: 3.3ms | fwd: 2016.6ms | loss: 6.4ms | bwd: 4235.6ms | clip: 0.0ms | optim: 49.7ms
397
+ step 168 | loss: 1.598396 | lr 2.7570e-04 | norm: 0.4648 | dt: 6299.20ms | tok/sec: 83230.9 | mem: 108909MB | Time Left: 00 hrs : 24 mins
398
+ [TIMING] data: 3.4ms | fwd: 2015.9ms | loss: 6.4ms | bwd: 4216.2ms | clip: 0.0ms | optim: 49.4ms
399
+ step 169 | loss: 1.517734 | lr 2.7501e-04 | norm: 0.5128 | dt: 6302.75ms | tok/sec: 83184.0 | mem: 108909MB | Time Left: 00 hrs : 24 mins
400
+ [TIMING] data: 3.4ms | fwd: 2011.6ms | loss: 6.4ms | bwd: 4223.9ms | clip: 0.0ms | optim: 49.4ms
401
+ step 170 | loss: 1.526498 | lr 2.7431e-04 | norm: 0.4985 | dt: 6296.95ms | tok/sec: 83260.6 | mem: 108909MB | Time Left: 00 hrs : 24 mins
402
+ [TIMING] data: 3.4ms | fwd: 2008.0ms | loss: 6.4ms | bwd: 4221.9ms | clip: 0.0ms | optim: 49.3ms
403
+ step 171 | loss: 1.589458 | lr 2.7361e-04 | norm: 0.5621 | dt: 6331.36ms | tok/sec: 82808.1 | mem: 108909MB | Time Left: 00 hrs : 24 mins
404
+ [TIMING] data: 3.3ms | fwd: 2049.7ms | loss: 6.4ms | bwd: 4214.5ms | clip: 0.0ms | optim: 49.5ms
405
+ step 172 | loss: 1.551229 | lr 2.7290e-04 | norm: 0.4868 | dt: 6348.52ms | tok/sec: 82584.2 | mem: 108909MB | Time Left: 00 hrs : 24 mins
406
+ [TIMING] data: 3.4ms | fwd: 2061.1ms | loss: 6.4ms | bwd: 4219.9ms | clip: 0.0ms | optim: 49.8ms
407
+ step 173 | loss: 1.536721 | lr 2.7218e-04 | norm: 0.4725 | dt: 6330.28ms | tok/sec: 82822.2 | mem: 108909MB | Time Left: 00 hrs : 23 mins
408
+ [TIMING] data: 3.4ms | fwd: 2028.6ms | loss: 6.4ms | bwd: 4234.3ms | clip: 0.0ms | optim: 49.5ms
409
+ step 174 | loss: 1.561139 | lr 2.7145e-04 | norm: 0.4796 | dt: 6308.29ms | tok/sec: 83111.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
410
+ [TIMING] data: 3.4ms | fwd: 2020.0ms | loss: 6.4ms | bwd: 4221.1ms | clip: 0.0ms | optim: 49.4ms
411
+ step 175 | loss: 1.546170 | lr 2.7071e-04 | norm: 0.5087 | dt: 6284.76ms | tok/sec: 83422.1 | mem: 108909MB | Time Left: 00 hrs : 23 mins
412
+ [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.4ms | bwd: 4207.3ms | clip: 0.0ms | optim: 49.4ms
413
+ step 176 | loss: 1.510554 | lr 2.6997e-04 | norm: 0.4955 | dt: 6326.66ms | tok/sec: 82869.6 | mem: 108909MB | Time Left: 00 hrs : 23 mins
414
+ [TIMING] data: 3.4ms | fwd: 2015.5ms | loss: 6.4ms | bwd: 4244.2ms | clip: 0.0ms | optim: 49.3ms
415
+ step 177 | loss: 1.594345 | lr 2.6921e-04 | norm: 0.4702 | dt: 6303.45ms | tok/sec: 83174.7 | mem: 108909MB | Time Left: 00 hrs : 23 mins
416
+ [TIMING] data: 3.4ms | fwd: 2019.3ms | loss: 6.4ms | bwd: 4217.1ms | clip: 0.0ms | optim: 49.4ms
417
+ step 178 | loss: 1.565544 | lr 2.6845e-04 | norm: 0.6543 | dt: 6356.01ms | tok/sec: 82487.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
418
+ [TIMING] data: 3.5ms | fwd: 2067.6ms | loss: 6.4ms | bwd: 4220.6ms | clip: 0.0ms | optim: 49.8ms
419
+ step 179 | loss: 1.570838 | lr 2.6769e-04 | norm: 0.4854 | dt: 6338.31ms | tok/sec: 82717.4 | mem: 108909MB | Time Left: 00 hrs : 23 mins
420
+ [TIMING] data: 3.4ms | fwd: 2060.6ms | loss: 6.4ms | bwd: 4210.6ms | clip: 0.0ms | optim: 49.4ms
421
+ step 180 | loss: 1.608467 | lr 2.6691e-04 | norm: 0.4543 | dt: 6348.96ms | tok/sec: 82578.6 | mem: 108909MB | Time Left: 00 hrs : 23 mins
422
+ [TIMING] data: 3.3ms | fwd: 2078.2ms | loss: 6.4ms | bwd: 4203.8ms | clip: 0.0ms | optim: 49.4ms
423
+ step 181 | loss: 1.518504 | lr 2.6613e-04 | norm: 0.4037 | dt: 6300.05ms | tok/sec: 83219.7 | mem: 108909MB | Time Left: 00 hrs : 22 mins
424
+ [TIMING] data: 3.3ms | fwd: 2012.5ms | loss: 6.4ms | bwd: 4220.4ms | clip: 0.0ms | optim: 49.4ms
425
+ step 182 | loss: 1.519081 | lr 2.6534e-04 | norm: 0.3706 | dt: 6300.37ms | tok/sec: 83215.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
426
+ [TIMING] data: 3.5ms | fwd: 2013.4ms | loss: 6.4ms | bwd: 4219.8ms | clip: 0.0ms | optim: 49.5ms
427
+ step 183 | loss: 1.545064 | lr 2.6455e-04 | norm: 0.4007 | dt: 6298.80ms | tok/sec: 83236.2 | mem: 108909MB | Time Left: 00 hrs : 22 mins
428
+ [TIMING] data: 3.3ms | fwd: 2010.6ms | loss: 6.4ms | bwd: 4221.2ms | clip: 0.0ms | optim: 49.4ms
429
+ step 184 | loss: 1.546265 | lr 2.6374e-04 | norm: 0.4499 | dt: 6286.92ms | tok/sec: 83393.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
430
+ [TIMING] data: 3.3ms | fwd: 2014.6ms | loss: 6.4ms | bwd: 4205.4ms | clip: 0.0ms | optim: 49.2ms
431
+ step 185 | loss: 1.517903 | lr 2.6293e-04 | norm: 0.3830 | dt: 6304.95ms | tok/sec: 83155.0 | mem: 108909MB | Time Left: 00 hrs : 22 mins
432
+ [TIMING] data: 3.3ms | fwd: 2015.7ms | loss: 6.4ms | bwd: 4221.2ms | clip: 0.0ms | optim: 50.4ms
433
+ step 186 | loss: 1.585939 | lr 2.6211e-04 | norm: 0.4245 | dt: 6312.37ms | tok/sec: 83057.2 | mem: 108909MB | Time Left: 00 hrs : 22 mins
434
+ [TIMING] data: 3.4ms | fwd: 2015.8ms | loss: 6.4ms | bwd: 4229.2ms | clip: 0.0ms | optim: 49.6ms
435
+ step 187 | loss: 1.557112 | lr 2.6129e-04 | norm: 0.4315 | dt: 6293.00ms | tok/sec: 83312.9 | mem: 108909MB | Time Left: 00 hrs : 22 mins
436
+ [TIMING] data: 3.3ms | fwd: 2002.2ms | loss: 6.4ms | bwd: 4223.7ms | clip: 0.0ms | optim: 49.4ms
437
+ step 188 | loss: 1.538441 | lr 2.6046e-04 | norm: 0.4569 | dt: 6300.70ms | tok/sec: 83211.0 | mem: 108909MB | Time Left: 00 hrs : 22 mins
438
+ [TIMING] data: 3.4ms | fwd: 2009.8ms | loss: 6.4ms | bwd: 4223.4ms | clip: 0.0ms | optim: 49.6ms
439
+ step 189 | loss: 1.548894 | lr 2.5962e-04 | norm: 0.3525 | dt: 6327.37ms | tok/sec: 82860.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
440
+ [TIMING] data: 3.2ms | fwd: 2030.5ms | loss: 6.4ms | bwd: 4229.9ms | clip: 0.0ms | optim: 49.4ms
441
+ step 190 | loss: 1.522655 | lr 2.5878e-04 | norm: 0.4214 | dt: 6354.27ms | tok/sec: 82509.6 | mem: 108909MB | Time Left: 00 hrs : 22 mins
442
+ [TIMING] data: 3.3ms | fwd: 2063.5ms | loss: 6.4ms | bwd: 4223.4ms | clip: 0.0ms | optim: 49.4ms
443
+ step 191 | loss: 1.544763 | lr 2.5793e-04 | norm: 0.3714 | dt: 6323.34ms | tok/sec: 82913.1 | mem: 108909MB | Time Left: 00 hrs : 22 mins
444
+ [TIMING] data: 3.4ms | fwd: 2012.5ms | loss: 6.4ms | bwd: 4243.6ms | clip: 0.0ms | optim: 49.5ms
445
+ step 192 | loss: 1.531644 | lr 2.5707e-04 | norm: 0.5406 | dt: 6287.51ms | tok/sec: 83385.6 | mem: 108909MB | Time Left: 00 hrs : 21 mins
446
+ [TIMING] data: 3.5ms | fwd: 2009.8ms | loss: 6.4ms | bwd: 4210.6ms | clip: 0.0ms | optim: 49.4ms
447
+ step 193 | loss: 1.527502 | lr 2.5621e-04 | norm: 0.4157 | dt: 6304.97ms | tok/sec: 83154.7 | mem: 108909MB | Time Left: 00 hrs : 21 mins
448
+ [TIMING] data: 3.3ms | fwd: 2018.3ms | loss: 6.4ms | bwd: 4218.9ms | clip: 0.0ms | optim: 50.0ms
449
+ step 194 | loss: 1.537738 | lr 2.5534e-04 | norm: 0.4412 | dt: 6309.68ms | tok/sec: 83092.6 | mem: 108909MB | Time Left: 00 hrs : 21 mins
450
+ [TIMING] data: 3.5ms | fwd: 2010.9ms | loss: 6.4ms | bwd: 4231.6ms | clip: 0.0ms | optim: 49.4ms
451
+ step 195 | loss: 1.519402 | lr 2.5446e-04 | norm: 0.4267 | dt: 6281.65ms | tok/sec: 83463.5 | mem: 108909MB | Time Left: 00 hrs : 21 mins
452
+ [TIMING] data: 3.3ms | fwd: 2014.2ms | loss: 6.4ms | bwd: 4200.4ms | clip: 0.0ms | optim: 49.5ms
453
+ step 196 | loss: 1.552346 | lr 2.5358e-04 | norm: 0.9164 | dt: 6314.61ms | tok/sec: 83027.8 | mem: 108909MB | Time Left: 00 hrs : 21 mins
454
+ [TIMING] data: 3.3ms | fwd: 2010.0ms | loss: 6.4ms | bwd: 4237.6ms | clip: 0.0ms | optim: 49.3ms
455
+ step 197 | loss: 1.558954 | lr 2.5270e-04 | norm: 0.4911 | dt: 6335.28ms | tok/sec: 82756.9 | mem: 108909MB | Time Left: 00 hrs : 21 mins
456
+ [TIMING] data: 3.4ms | fwd: 2077.9ms | loss: 6.4ms | bwd: 4190.0ms | clip: 0.0ms | optim: 49.7ms
457
+ step 198 | loss: 1.508947 | lr 2.5180e-04 | norm: 0.3714 | dt: 6359.11ms | tok/sec: 82446.8 | mem: 108909MB | Time Left: 00 hrs : 21 mins
458
+ [TIMING] data: 3.5ms | fwd: 2068.8ms | loss: 6.4ms | bwd: 4222.9ms | clip: 0.0ms | optim: 49.6ms
459
+ step 199 | loss: 1.520034 | lr 2.5090e-04 | norm: 0.3761 | dt: 6309.29ms | tok/sec: 83097.8 | mem: 108909MB | Time Left: 00 hrs : 21 mins
460
+ [TIMING] data: 3.3ms | fwd: 2007.9ms | loss: 6.4ms | bwd: 4234.2ms | clip: 0.0ms | optim: 49.5ms
461
+ [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run13_step_200.ckpt (safely)
462
+ step 200 | loss: 1.515468 | lr 2.5000e-04 | norm: 0.3363 | dt: 6274.53ms | tok/sec: 83558.1 | mem: 108909MB | Time Left: 00 hrs : 20 mins
463
+ [TIMING] data: 3.3ms | fwd: 2091.9ms | loss: 6.4ms | bwd: 4115.5ms | clip: 0.0ms | optim: 49.4ms
464
+ step 201 | loss: 1.543592 | lr 2.4909e-04 | norm: 0.3853 | dt: 6197.18ms | tok/sec: 84601.1 | mem: 108909MB | Time Left: 00 hrs : 20 mins
465
+ [TIMING] data: 3.3ms | fwd: 1988.0ms | loss: 6.4ms | bwd: 4142.1ms | clip: 0.0ms | optim: 49.5ms
466
+ step 202 | loss: 1.518495 | lr 2.4818e-04 | norm: 0.3703 | dt: 6216.37ms | tok/sec: 84339.9 | mem: 108909MB | Time Left: 00 hrs : 20 mins
467
+ [TIMING] data: 3.5ms | fwd: 1992.1ms | loss: 6.3ms | bwd: 4157.0ms | clip: 0.0ms | optim: 49.5ms
468
+ step 203 | loss: 1.594687 | lr 2.4726e-04 | norm: 0.3279 | dt: 6224.03ms | tok/sec: 84236.1 | mem: 108909MB | Time Left: 00 hrs : 20 mins
469
+ [TIMING] data: 3.4ms | fwd: 1991.0ms | loss: 6.4ms | bwd: 4165.4ms | clip: 0.0ms | optim: 49.8ms
470
+ step 204 | loss: 1.508122 | lr 2.4633e-04 | norm: 0.3350 | dt: 6244.08ms | tok/sec: 83965.6 | mem: 108909MB | Time Left: 00 hrs : 20 mins
471
+ [TIMING] data: 3.2ms | fwd: 1998.7ms | loss: 6.3ms | bwd: 4178.5ms | clip: 0.0ms | optim: 49.3ms
472
+ step 205 | loss: 1.481425 | lr 2.4540e-04 | norm: 0.4867 | dt: 6300.96ms | tok/sec: 83207.6 | mem: 108909MB | Time Left: 00 hrs : 20 mins
473
+ [TIMING] data: 3.4ms | fwd: 2004.3ms | loss: 6.4ms | bwd: 4229.4ms | clip: 0.0ms | optim: 49.5ms
474
+ step 206 | loss: 1.561807 | lr 2.4446e-04 | norm: 0.4345 | dt: 6308.09ms | tok/sec: 83113.5 | mem: 108909MB | Time Left: 00 hrs : 20 mins
475
+ [TIMING] data: 3.5ms | fwd: 2014.5ms | loss: 6.4ms | bwd: 4226.3ms | clip: 0.0ms | optim: 49.3ms
476
+ step 207 | loss: 1.582822 | lr 2.4352e-04 | norm: 0.3482 | dt: 6330.54ms | tok/sec: 82818.9 | mem: 108909MB | Time Left: 00 hrs : 20 mins
477
+ [TIMING] data: 3.4ms | fwd: 2021.2ms | loss: 6.4ms | bwd: 4241.9ms | clip: 0.0ms | optim: 49.6ms
478
+ step 208 | loss: 1.465932 | lr 2.4258e-04 | norm: 0.3645 | dt: 6351.70ms | tok/sec: 82543.0 | mem: 108909MB | Time Left: 00 hrs : 20 mins
479
+ [TIMING] data: 3.4ms | fwd: 2027.3ms | loss: 6.4ms | bwd: 4257.3ms | clip: 0.0ms | optim: 49.3ms
480
+ step 209 | loss: 1.552704 | lr 2.4163e-04 | norm: 0.5003 | dt: 6382.09ms | tok/sec: 82149.9 | mem: 108909MB | Time Left: 00 hrs : 20 mins
481
+ [TIMING] data: 3.5ms | fwd: 2067.9ms | loss: 6.4ms | bwd: 4246.9ms | clip: 0.0ms | optim: 49.5ms
482
+ step 210 | loss: 1.560924 | lr 2.4067e-04 | norm: 0.4204 | dt: 6343.38ms | tok/sec: 82651.2 | mem: 108909MB | Time Left: 00 hrs : 20 mins
483
+ [TIMING] data: 3.4ms | fwd: 2021.8ms | loss: 6.4ms | bwd: 4254.5ms | clip: 0.0ms | optim: 49.3ms
484
+ step 211 | loss: 1.577441 | lr 2.3971e-04 | norm: 0.2780 | dt: 6369.29ms | tok/sec: 82315.0 | mem: 108909MB | Time Left: 00 hrs : 20 mins
485
+ [TIMING] data: 3.3ms | fwd: 2023.9ms | loss: 6.4ms | bwd: 4278.3ms | clip: 0.0ms | optim: 49.4ms
486
+ step 212 | loss: 1.523178 | lr 2.3875e-04 | norm: 0.3415 | dt: 6376.56ms | tok/sec: 82221.1 | mem: 108909MB | Time Left: 00 hrs : 19 mins
487
+ [TIMING] data: 3.4ms | fwd: 2029.4ms | loss: 6.5ms | bwd: 4279.9ms | clip: 0.0ms | optim: 49.4ms
488
+ step 213 | loss: 1.532332 | lr 2.3778e-04 | norm: 0.3428 | dt: 6336.42ms | tok/sec: 82742.0 | mem: 108909MB | Time Left: 00 hrs : 19 mins
489
+ [TIMING] data: 3.4ms | fwd: 2022.6ms | loss: 6.4ms | bwd: 4246.6ms | clip: 0.0ms | optim: 49.3ms
490
+ step 214 | loss: 1.526189 | lr 2.3681e-04 | norm: 0.4383 | dt: 6379.65ms | tok/sec: 82181.3 | mem: 108909MB | Time Left: 00 hrs : 19 mins
491
+ [TIMING] data: 3.3ms | fwd: 2044.2ms | loss: 6.4ms | bwd: 4268.4ms | clip: 0.0ms | optim: 49.4ms
492
+ step 215 | loss: 1.536538 | lr 2.3584e-04 | norm: 0.4139 | dt: 6378.23ms | tok/sec: 82199.7 | mem: 108909MB | Time Left: 00 hrs : 19 mins
493
+ [TIMING] data: 3.4ms | fwd: 2030.4ms | loss: 6.4ms | bwd: 4280.6ms | clip: 0.0ms | optim: 49.5ms
494
+ step 216 | loss: 1.505207 | lr 2.3486e-04 | norm: 0.3868 | dt: 6372.58ms | tok/sec: 82272.5 | mem: 108909MB | Time Left: 00 hrs : 19 mins
495
+ [TIMING] data: 3.4ms | fwd: 2027.4ms | loss: 6.4ms | bwd: 4278.1ms | clip: 0.0ms | optim: 49.4ms
496
+ step 217 | loss: 1.519254 | lr 2.3387e-04 | norm: 0.2924 | dt: 6370.76ms | tok/sec: 82296.0 | mem: 108909MB | Time Left: 00 hrs : 19 mins
497
+ [TIMING] data: 3.3ms | fwd: 2027.3ms | loss: 6.4ms | bwd: 4276.3ms | clip: 0.0ms | optim: 49.5ms
498
+ step 218 | loss: 1.547643 | lr 2.3289e-04 | norm: 0.3269 | dt: 6415.39ms | tok/sec: 81723.5 | mem: 108909MB | Time Left: 00 hrs : 19 mins
499
+ [TIMING] data: 3.4ms | fwd: 2060.3ms | loss: 6.4ms | bwd: 4287.9ms | clip: 0.0ms | optim: 49.4ms
500
+ step 219 | loss: 1.571966 | lr 2.3190e-04 | norm: 0.3471 | dt: 6380.81ms | tok/sec: 82166.4 | mem: 108909MB | Time Left: 00 hrs : 19 mins
501
+ [TIMING] data: 3.3ms | fwd: 2030.7ms | loss: 6.4ms | bwd: 4283.0ms | clip: 0.0ms | optim: 49.3ms
502
+ step 220 | loss: 1.509900 | lr 2.3090e-04 | norm: 0.2972 | dt: 6356.96ms | tok/sec: 82474.6 | mem: 108909MB | Time Left: 00 hrs : 19 mins
503
+ [TIMING] data: 3.3ms | fwd: 2023.5ms | loss: 6.4ms | bwd: 4266.3ms | clip: 0.0ms | optim: 49.4ms
504
+ step 221 | loss: 1.497495 | lr 2.2990e-04 | norm: 0.3361 | dt: 6365.62ms | tok/sec: 82362.4 | mem: 108909MB | Time Left: 00 hrs : 18 mins
505
+ [TIMING] data: 3.4ms | fwd: 2052.2ms | loss: 6.4ms | bwd: 4246.2ms | clip: 0.0ms | optim: 49.5ms
506
+ step 222 | loss: 1.492427 | lr 2.2890e-04 | norm: 0.2918 | dt: 6375.24ms | tok/sec: 82238.2 | mem: 108909MB | Time Left: 00 hrs : 18 mins
507
+ [TIMING] data: 3.4ms | fwd: 2051.9ms | loss: 6.7ms | bwd: 4255.9ms | clip: 0.0ms | optim: 49.4ms
508
+ step 223 | loss: 1.519729 | lr 2.2790e-04 | norm: 0.3266 | dt: 6357.23ms | tok/sec: 82471.1 | mem: 108909MB | Time Left: 00 hrs : 18 mins
509
+ [TIMING] data: 3.4ms | fwd: 2018.6ms | loss: 6.4ms | bwd: 4271.3ms | clip: 0.0ms | optim: 49.5ms
510
+ step 224 | loss: 1.466115 | lr 2.2689e-04 | norm: 0.3746 | dt: 6343.84ms | tok/sec: 82645.3 | mem: 108909MB | Time Left: 00 hrs : 18 mins
511
+ [TIMING] data: 3.4ms | fwd: 2023.9ms | loss: 6.4ms | bwd: 4252.8ms | clip: 0.0ms | optim: 49.4ms
512
+ step 225 | loss: 1.554846 | lr 2.2588e-04 | norm: 0.3662 | dt: 6337.59ms | tok/sec: 82726.7 | mem: 108909MB | Time Left: 00 hrs : 18 mins
513
+ [TIMING] data: 3.3ms | fwd: 2021.0ms | loss: 6.4ms | bwd: 4249.4ms | clip: 0.0ms | optim: 49.4ms
514
+ step 226 | loss: 1.487506 | lr 2.2487e-04 | norm: 0.4012 | dt: 6335.98ms | tok/sec: 82747.8 | mem: 108909MB | Time Left: 00 hrs : 18 mins
515
+ [TIMING] data: 3.4ms | fwd: 2021.6ms | loss: 6.4ms | bwd: 4247.2ms | clip: 0.0ms | optim: 49.4ms
516
+ step 227 | loss: 1.479402 | lr 2.2385e-04 | norm: 0.2841 | dt: 6320.69ms | tok/sec: 82947.9 | mem: 108909MB | Time Left: 00 hrs : 18 mins
517
+ [TIMING] data: 3.4ms | fwd: 2020.3ms | loss: 6.4ms | bwd: 4233.3ms | clip: 0.0ms | optim: 49.4ms
518
+ step 228 | loss: 1.533440 | lr 2.2284e-04 | norm: 0.5841 | dt: 6387.96ms | tok/sec: 82074.4 | mem: 108909MB | Time Left: 00 hrs : 18 mins
519
+ [TIMING] data: 3.4ms | fwd: 2070.9ms | loss: 6.4ms | bwd: 4250.1ms | clip: 0.0ms | optim: 49.3ms
520
+ step 229 | loss: 1.506774 | lr 2.2181e-04 | norm: 0.2893 | dt: 6334.24ms | tok/sec: 82770.5 | mem: 108909MB | Time Left: 00 hrs : 18 mins
521
+ [TIMING] data: 3.4ms | fwd: 2020.0ms | loss: 6.4ms | bwd: 4247.2ms | clip: 0.0ms | optim: 49.4ms
522
+ step 230 | loss: 1.499669 | lr 2.2079e-04 | norm: 0.3589 | dt: 6331.55ms | tok/sec: 82805.6 | mem: 108909MB | Time Left: 00 hrs : 17 mins
523
+ [TIMING] data: 3.5ms | fwd: 2021.4ms | loss: 6.4ms | bwd: 4243.0ms | clip: 0.0ms | optim: 49.4ms
524
+ step 231 | loss: 1.486993 | lr 2.1977e-04 | norm: 0.3249 | dt: 6306.00ms | tok/sec: 83141.2 | mem: 108909MB | Time Left: 00 hrs : 17 mins
525
+ [TIMING] data: 3.5ms | fwd: 2017.3ms | loss: 6.4ms | bwd: 4221.5ms | clip: 0.0ms | optim: 49.5ms
526
+ step 232 | loss: 1.574881 | lr 2.1874e-04 | norm: 0.3193 | dt: 6340.37ms | tok/sec: 82690.4 | mem: 108909MB | Time Left: 00 hrs : 17 mins
527
+ [TIMING] data: 3.5ms | fwd: 2021.6ms | loss: 6.4ms | bwd: 4251.7ms | clip: 0.0ms | optim: 49.4ms
528
+ step 233 | loss: 1.508013 | lr 2.1771e-04 | norm: 0.2468 | dt: 6327.01ms | tok/sec: 82865.0 | mem: 108909MB | Time Left: 00 hrs : 17 mins
529
+ [TIMING] data: 3.3ms | fwd: 2025.7ms | loss: 6.4ms | bwd: 4234.3ms | clip: 0.0ms | optim: 49.3ms
530
+ step 234 | loss: 1.524792 | lr 2.1668e-04 | norm: 0.2990 | dt: 6327.45ms | tok/sec: 82859.3 | mem: 108909MB | Time Left: 00 hrs : 17 mins
531
+ [TIMING] data: 3.3ms | fwd: 2011.9ms | loss: 6.4ms | bwd: 4248.5ms | clip: 0.0ms | optim: 49.4ms
532
+ step 235 | loss: 1.561522 | lr 2.1564e-04 | norm: 0.2423 | dt: 6412.86ms | tok/sec: 81755.8 | mem: 108909MB | Time Left: 00 hrs : 17 mins
533
+ [TIMING] data: 3.5ms | fwd: 2092.1ms | loss: 6.4ms | bwd: 4253.4ms | clip: 0.0ms | optim: 49.5ms
534
+ step 236 | loss: 1.532568 | lr 2.1461e-04 | norm: 0.2922 | dt: 6317.35ms | tok/sec: 82991.7 | mem: 108909MB | Time Left: 00 hrs : 17 mins
535
+ [TIMING] data: 3.3ms | fwd: 2024.8ms | loss: 6.4ms | bwd: 4225.2ms | clip: 0.0ms | optim: 49.7ms
536
+ step 237 | loss: 1.511153 | lr 2.1357e-04 | norm: 0.3742 | dt: 6312.04ms | tok/sec: 83061.6 | mem: 108909MB | Time Left: 00 hrs : 17 mins
537
+ [TIMING] data: 3.4ms | fwd: 2012.8ms | loss: 6.4ms | bwd: 4232.1ms | clip: 0.0ms | optim: 49.4ms
538
+ step 238 | loss: 1.461659 | lr 2.1253e-04 | norm: 0.3239 | dt: 6330.51ms | tok/sec: 82819.3 | mem: 108909MB | Time Left: 00 hrs : 17 mins
539
+ [TIMING] data: 3.4ms | fwd: 2024.6ms | loss: 6.4ms | bwd: 4238.8ms | clip: 0.0ms | optim: 49.3ms
540
+ step 239 | loss: 1.480994 | lr 2.1149e-04 | norm: 0.2434 | dt: 6309.65ms | tok/sec: 83093.0 | mem: 108909MB | Time Left: 00 hrs : 16 mins
541
+ [TIMING] data: 3.4ms | fwd: 2017.9ms | loss: 6.4ms | bwd: 4224.6ms | clip: 0.0ms | optim: 49.4ms
542
+ step 240 | loss: 1.504617 | lr 2.1045e-04 | norm: 0.3472 | dt: 6399.22ms | tok/sec: 81930.0 | mem: 108909MB | Time Left: 00 hrs : 17 mins
543
+ [TIMING] data: 3.4ms | fwd: 2084.7ms | loss: 6.4ms | bwd: 4247.0ms | clip: 0.0ms | optim: 49.7ms
544
+ step 241 | loss: 1.484900 | lr 2.0941e-04 | norm: 0.2350 | dt: 6329.71ms | tok/sec: 82829.8 | mem: 108909MB | Time Left: 00 hrs : 16 mins
545
+ [TIMING] data: 3.3ms | fwd: 2021.4ms | loss: 6.4ms | bwd: 4241.2ms | clip: 0.0ms | optim: 49.4ms
546
+ step 242 | loss: 1.526330 | lr 2.0837e-04 | norm: 0.3010 | dt: 6328.15ms | tok/sec: 82850.1 | mem: 108909MB | Time Left: 00 hrs : 16 mins
547
+ [TIMING] data: 3.4ms | fwd: 2017.8ms | loss: 6.4ms | bwd: 4243.3ms | clip: 0.0ms | optim: 49.3ms
548
+ step 243 | loss: 1.494656 | lr 2.0732e-04 | norm: 0.2456 | dt: 6314.70ms | tok/sec: 83026.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
549
+ [TIMING] data: 3.4ms | fwd: 2018.8ms | loss: 6.4ms | bwd: 4228.7ms | clip: 0.0ms | optim: 49.4ms
550
+ step 244 | loss: 1.518198 | lr 2.0628e-04 | norm: 0.2675 | dt: 6348.66ms | tok/sec: 82582.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
551
+ [TIMING] data: 3.4ms | fwd: 2023.7ms | loss: 6.4ms | bwd: 4257.0ms | clip: 0.0ms | optim: 50.2ms
552
+ step 245 | loss: 1.521052 | lr 2.0523e-04 | norm: 0.2581 | dt: 6308.14ms | tok/sec: 83112.9 | mem: 108909MB | Time Left: 00 hrs : 16 mins
553
+ [TIMING] data: 3.3ms | fwd: 2015.5ms | loss: 6.4ms | bwd: 4225.3ms | clip: 0.0ms | optim: 49.7ms
554
+ step 246 | loss: 1.502181 | lr 2.0419e-04 | norm: 0.1972 | dt: 6346.20ms | tok/sec: 82614.5 | mem: 108909MB | Time Left: 00 hrs : 16 mins
555
+ [TIMING] data: 3.4ms | fwd: 2057.8ms | loss: 6.4ms | bwd: 4221.3ms | clip: 0.0ms | optim: 49.3ms
556
+ step 247 | loss: 1.523221 | lr 2.0314e-04 | norm: 0.2856 | dt: 6349.26ms | tok/sec: 82574.7 | mem: 108909MB | Time Left: 00 hrs : 16 mins
557
+ [TIMING] data: 3.3ms | fwd: 2065.1ms | loss: 6.5ms | bwd: 4216.8ms | clip: 0.0ms | optim: 49.6ms
558
+ step 248 | loss: 1.499708 | lr 2.0209e-04 | norm: 0.2951 | dt: 6318.24ms | tok/sec: 82980.1 | mem: 108909MB | Time Left: 00 hrs : 16 mins
559
+ [TIMING] data: 3.3ms | fwd: 2012.7ms | loss: 6.4ms | bwd: 4238.3ms | clip: 0.0ms | optim: 49.5ms
560
+ step 249 | loss: 1.498004 | lr 2.0105e-04 | norm: 0.2532 | dt: 6318.23ms | tok/sec: 82980.2 | mem: 108909MB | Time Left: 00 hrs : 15 mins
561
+ [TIMING] data: 3.3ms | fwd: 2011.2ms | loss: 6.4ms | bwd: 4238.5ms | clip: 0.0ms | optim: 50.7ms
562
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_val_00000.bin
563
+ 100000000
564
+ validation loss: 1.5051
565
+ step 250 | loss: 1.479036 | lr 2.0000e-04 | norm: 0.2610 | dt: 11387.78ms | tok/sec: 46039.5 | mem: 108909MB | Time Left: 00 hrs : 28 mins
566
+ [TIMING] data: 3.3ms | fwd: 2013.6ms | loss: 6.4ms | bwd: 4229.9ms | clip: 0.0ms | optim: 49.6ms
567
+ step 251 | loss: 1.524658 | lr 1.9895e-04 | norm: 0.2094 | dt: 6312.76ms | tok/sec: 83052.2 | mem: 108909MB | Time Left: 00 hrs : 15 mins
568
+ [TIMING] data: 3.3ms | fwd: 2007.3ms | loss: 6.4ms | bwd: 4238.3ms | clip: 0.0ms | optim: 49.5ms
569
+ step 252 | loss: 1.530877 | lr 1.9791e-04 | norm: 0.3166 | dt: 6324.16ms | tok/sec: 82902.5 | mem: 108909MB | Time Left: 00 hrs : 15 mins
570
+ [TIMING] data: 3.3ms | fwd: 2023.0ms | loss: 6.4ms | bwd: 4234.2ms | clip: 0.0ms | optim: 49.3ms
571
+ step 253 | loss: 1.498913 | lr 1.9686e-04 | norm: 0.2670 | dt: 6348.04ms | tok/sec: 82590.5 | mem: 108909MB | Time Left: 00 hrs : 15 mins
572
+ [TIMING] data: 3.3ms | fwd: 2048.2ms | loss: 6.4ms | bwd: 4232.8ms | clip: 0.0ms | optim: 49.5ms
573
+ step 254 | loss: 1.514113 | lr 1.9581e-04 | norm: 0.2580 | dt: 6325.70ms | tok/sec: 82882.2 | mem: 108909MB | Time Left: 00 hrs : 15 mins
574
+ [TIMING] data: 3.3ms | fwd: 2033.2ms | loss: 6.4ms | bwd: 4225.4ms | clip: 0.0ms | optim: 49.4ms
575
+ step 255 | loss: 1.534029 | lr 1.9477e-04 | norm: 0.1956 | dt: 6346.03ms | tok/sec: 82616.8 | mem: 108909MB | Time Left: 00 hrs : 15 mins
576
+ [TIMING] data: 3.3ms | fwd: 2024.9ms | loss: 6.4ms | bwd: 4254.1ms | clip: 0.0ms | optim: 49.4ms
577
+ step 256 | loss: 1.479002 | lr 1.9372e-04 | norm: 0.2145 | dt: 6320.09ms | tok/sec: 82955.8 | mem: 108909MB | Time Left: 00 hrs : 15 mins
578
+ [TIMING] data: 3.3ms | fwd: 2017.1ms | loss: 6.4ms | bwd: 4236.2ms | clip: 0.0ms | optim: 49.2ms
579
+ step 257 | loss: 1.474892 | lr 1.9268e-04 | norm: 0.3064 | dt: 6323.17ms | tok/sec: 82915.4 | mem: 108909MB | Time Left: 00 hrs : 15 mins
580
+ [TIMING] data: 3.3ms | fwd: 2013.7ms | loss: 6.4ms | bwd: 4242.2ms | clip: 0.0ms | optim: 49.6ms
581
+ step 258 | loss: 1.479650 | lr 1.9163e-04 | norm: 0.2714 | dt: 6328.00ms | tok/sec: 82852.1 | mem: 108909MB | Time Left: 00 hrs : 14 mins
582
+ [TIMING] data: 3.4ms | fwd: 2018.5ms | loss: 6.4ms | bwd: 4242.2ms | clip: 0.0ms | optim: 49.5ms
583
+ step 259 | loss: 1.469126 | lr 1.9059e-04 | norm: 0.2147 | dt: 6316.69ms | tok/sec: 83000.4 | mem: 108909MB | Time Left: 00 hrs : 14 mins
584
+ [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.4ms | bwd: 4239.3ms | clip: 0.0ms | optim: 49.4ms
585
+ step 260 | loss: 1.482932 | lr 1.8955e-04 | norm: 0.3710 | dt: 6349.33ms | tok/sec: 82573.7 | mem: 108909MB | Time Left: 00 hrs : 14 mins
586
+ [TIMING] data: 3.5ms | fwd: 2025.4ms | loss: 6.4ms | bwd: 4256.7ms | clip: 0.0ms | optim: 49.3ms
587
+ step 261 | loss: 1.472604 | lr 1.8851e-04 | norm: 0.3087 | dt: 6299.28ms | tok/sec: 83229.9 | mem: 108909MB | Time Left: 00 hrs : 14 mins
588
+ [TIMING] data: 3.4ms | fwd: 2014.0ms | loss: 6.4ms | bwd: 4218.1ms | clip: 0.0ms | optim: 49.4ms
589
+ step 262 | loss: 1.510692 | lr 1.8747e-04 | norm: 0.2209 | dt: 6306.56ms | tok/sec: 83133.7 | mem: 108909MB | Time Left: 00 hrs : 14 mins
590
+ [TIMING] data: 3.5ms | fwd: 2010.7ms | loss: 6.4ms | bwd: 4228.6ms | clip: 0.0ms | optim: 49.4ms
591
+ step 263 | loss: 1.521401 | lr 1.8643e-04 | norm: 0.2542 | dt: 6335.75ms | tok/sec: 82750.8 | mem: 108909MB | Time Left: 00 hrs : 14 mins
592
+ [TIMING] data: 3.5ms | fwd: 2042.3ms | loss: 6.4ms | bwd: 4226.2ms | clip: 0.0ms | optim: 49.4ms
593
+ step 264 | loss: 1.518334 | lr 1.8539e-04 | norm: 0.2676 | dt: 6346.58ms | tok/sec: 82609.5 | mem: 108909MB | Time Left: 00 hrs : 14 mins
594
+ [TIMING] data: 3.3ms | fwd: 2058.0ms | loss: 6.4ms | bwd: 4221.4ms | clip: 0.0ms | optim: 49.4ms
595
+ step 265 | loss: 1.467476 | lr 1.8436e-04 | norm: 0.3925 | dt: 6373.01ms | tok/sec: 82267.0 | mem: 108909MB | Time Left: 00 hrs : 14 mins
596
+ [TIMING] data: 3.3ms | fwd: 2068.0ms | loss: 6.4ms | bwd: 4237.8ms | clip: 0.0ms | optim: 49.5ms
597
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_train_00002.bin
598
+ 100000000
599
+ step 266 | loss: 1.439376 | lr 1.8332e-04 | norm: 0.2033 | dt: 6320.79ms | tok/sec: 82946.6 | mem: 108909MB | Time Left: 00 hrs : 14 mins
600
+ [TIMING] data: 14.4ms | fwd: 2033.8ms | loss: 6.4ms | bwd: 4209.0ms | clip: 0.0ms | optim: 49.3ms
601
+ step 267 | loss: 1.543246 | lr 1.8229e-04 | norm: 0.2407 | dt: 6323.34ms | tok/sec: 82913.1 | mem: 108909MB | Time Left: 00 hrs : 14 mins
602
+ [TIMING] data: 3.3ms | fwd: 2019.1ms | loss: 6.4ms | bwd: 4237.3ms | clip: 0.0ms | optim: 49.3ms
603
+ step 268 | loss: 1.523543 | lr 1.8126e-04 | norm: 0.2025 | dt: 6298.32ms | tok/sec: 83242.5 | mem: 108909MB | Time Left: 00 hrs : 13 mins
604
+ [TIMING] data: 3.5ms | fwd: 2010.2ms | loss: 6.4ms | bwd: 4221.0ms | clip: 0.0ms | optim: 49.3ms
605
+ step 269 | loss: 1.496513 | lr 1.8023e-04 | norm: 0.1905 | dt: 6305.69ms | tok/sec: 83145.3 | mem: 108909MB | Time Left: 00 hrs : 13 mins
606
+ [TIMING] data: 3.4ms | fwd: 2014.6ms | loss: 6.4ms | bwd: 4223.9ms | clip: 0.0ms | optim: 49.4ms
607
+ step 270 | loss: 1.457562 | lr 1.7921e-04 | norm: 0.2139 | dt: 6299.40ms | tok/sec: 83228.3 | mem: 108909MB | Time Left: 00 hrs : 13 mins
608
+ [TIMING] data: 3.3ms | fwd: 2011.4ms | loss: 6.4ms | bwd: 4221.0ms | clip: 0.0ms | optim: 49.3ms
609
+ step 271 | loss: 1.535891 | lr 1.7819e-04 | norm: 0.2134 | dt: 6311.65ms | tok/sec: 83066.8 | mem: 108909MB | Time Left: 00 hrs : 13 mins
610
+ [TIMING] data: 3.4ms | fwd: 2015.1ms | loss: 6.4ms | bwd: 4229.5ms | clip: 0.0ms | optim: 49.4ms
611
+ step 272 | loss: 1.501040 | lr 1.7716e-04 | norm: 0.2187 | dt: 6319.93ms | tok/sec: 82957.9 | mem: 108909MB | Time Left: 00 hrs : 13 mins
612
+ [TIMING] data: 3.3ms | fwd: 2014.7ms | loss: 6.4ms | bwd: 4238.3ms | clip: 0.0ms | optim: 49.3ms
613
+ step 273 | loss: 1.566223 | lr 1.7615e-04 | norm: 0.2625 | dt: 6308.45ms | tok/sec: 83108.8 | mem: 108909MB | Time Left: 00 hrs : 13 mins
614
+ [TIMING] data: 3.3ms | fwd: 2019.4ms | loss: 6.4ms | bwd: 4222.0ms | clip: 0.0ms | optim: 49.4ms
615
+ step 274 | loss: 1.495419 | lr 1.7513e-04 | norm: 0.2264 | dt: 6317.88ms | tok/sec: 82984.8 | mem: 108909MB | Time Left: 00 hrs : 13 mins
616
+ [TIMING] data: 3.4ms | fwd: 2022.2ms | loss: 6.4ms | bwd: 4228.5ms | clip: 0.0ms | optim: 49.5ms
617
+ step 275 | loss: 1.529794 | lr 1.7412e-04 | norm: 0.2556 | dt: 6314.58ms | tok/sec: 83028.1 | mem: 108909MB | Time Left: 00 hrs : 13 mins
618
+ [TIMING] data: 3.4ms | fwd: 2021.7ms | loss: 6.4ms | bwd: 4225.6ms | clip: 0.0ms | optim: 49.5ms
619
+ step 276 | loss: 1.466739 | lr 1.7311e-04 | norm: 0.1947 | dt: 6323.10ms | tok/sec: 82916.3 | mem: 108909MB | Time Left: 00 hrs : 13 mins
620
+ [TIMING] data: 3.4ms | fwd: 2010.3ms | loss: 6.4ms | bwd: 4245.6ms | clip: 0.0ms | optim: 49.5ms
621
+ step 277 | loss: 1.534216 | lr 1.7210e-04 | norm: 0.2265 | dt: 6318.67ms | tok/sec: 82974.4 | mem: 108909MB | Time Left: 00 hrs : 12 mins
622
+ [TIMING] data: 3.3ms | fwd: 2007.6ms | loss: 6.6ms | bwd: 4243.7ms | clip: 0.0ms | optim: 49.4ms
623
+ step 278 | loss: 1.468664 | lr 1.7110e-04 | norm: 0.2138 | dt: 6335.29ms | tok/sec: 82756.8 | mem: 108909MB | Time Left: 00 hrs : 12 mins
624
+ [TIMING] data: 3.4ms | fwd: 2018.6ms | loss: 6.4ms | bwd: 4249.3ms | clip: 0.0ms | optim: 49.7ms
625
+ step 279 | loss: 1.551666 | lr 1.7010e-04 | norm: 0.2405 | dt: 6311.24ms | tok/sec: 83072.1 | mem: 108909MB | Time Left: 00 hrs : 12 mins
626
+ [TIMING] data: 3.4ms | fwd: 2018.4ms | loss: 6.4ms | bwd: 4225.7ms | clip: 0.0ms | optim: 49.4ms
627
+ step 280 | loss: 1.486062 | lr 1.6910e-04 | norm: 0.3126 | dt: 6307.03ms | tok/sec: 83127.6 | mem: 108909MB | Time Left: 00 hrs : 12 mins
628
+ [TIMING] data: 3.4ms | fwd: 2014.8ms | loss: 6.4ms | bwd: 4225.2ms | clip: 0.0ms | optim: 49.3ms
629
+ step 281 | loss: 1.491243 | lr 1.6810e-04 | norm: 0.2130 | dt: 6330.03ms | tok/sec: 82825.5 | mem: 108909MB | Time Left: 00 hrs : 12 mins
630
+ [TIMING] data: 3.4ms | fwd: 2013.2ms | loss: 6.4ms | bwd: 4249.7ms | clip: 0.0ms | optim: 49.4ms
631
+ step 282 | loss: 1.489311 | lr 1.6711e-04 | norm: 0.2014 | dt: 6292.83ms | tok/sec: 83315.1 | mem: 108909MB | Time Left: 00 hrs : 12 mins
632
+ [TIMING] data: 3.4ms | fwd: 2013.9ms | loss: 6.4ms | bwd: 4211.8ms | clip: 0.0ms | optim: 49.4ms
633
+ step 283 | loss: 1.416811 | lr 1.6613e-04 | norm: 0.1893 | dt: 6325.16ms | tok/sec: 82889.3 | mem: 108909MB | Time Left: 00 hrs : 12 mins
634
+ [TIMING] data: 3.4ms | fwd: 2031.2ms | loss: 6.4ms | bwd: 4226.7ms | clip: 0.0ms | optim: 49.4ms
635
+ step 284 | loss: 1.525040 | lr 1.6514e-04 | norm: 0.1914 | dt: 6333.96ms | tok/sec: 82774.1 | mem: 108909MB | Time Left: 00 hrs : 12 mins
636
+ [TIMING] data: 3.4ms | fwd: 2041.1ms | loss: 6.4ms | bwd: 4225.4ms | clip: 0.0ms | optim: 49.5ms
637
+ step 285 | loss: 1.497472 | lr 1.6416e-04 | norm: 0.2121 | dt: 6326.71ms | tok/sec: 82869.0 | mem: 108909MB | Time Left: 00 hrs : 12 mins
638
+ [TIMING] data: 3.4ms | fwd: 2041.4ms | loss: 6.4ms | bwd: 4218.3ms | clip: 0.0ms | optim: 49.3ms
639
+ step 286 | loss: 1.476106 | lr 1.6319e-04 | norm: 0.2120 | dt: 6310.10ms | tok/sec: 83087.1 | mem: 108909MB | Time Left: 00 hrs : 11 mins
640
+ [TIMING] data: 3.3ms | fwd: 2012.5ms | loss: 6.4ms | bwd: 4230.5ms | clip: 0.0ms | optim: 49.4ms
641
+ step 287 | loss: 1.430440 | lr 1.6222e-04 | norm: 0.1705 | dt: 6310.17ms | tok/sec: 83086.2 | mem: 108909MB | Time Left: 00 hrs : 11 mins
642
+ [TIMING] data: 3.3ms | fwd: 2015.5ms | loss: 6.4ms | bwd: 4227.6ms | clip: 0.0ms | optim: 49.4ms
643
+ step 288 | loss: 1.457449 | lr 1.6125e-04 | norm: 0.2106 | dt: 6313.65ms | tok/sec: 83040.4 | mem: 108909MB | Time Left: 00 hrs : 11 mins
644
+ [TIMING] data: 3.4ms | fwd: 2015.3ms | loss: 6.4ms | bwd: 4231.1ms | clip: 0.0ms | optim: 49.5ms
645
+ step 289 | loss: 1.432535 | lr 1.6029e-04 | norm: 0.1601 | dt: 6317.72ms | tok/sec: 82986.9 | mem: 108909MB | Time Left: 00 hrs : 11 mins
646
+ [TIMING] data: 3.3ms | fwd: 2011.7ms | loss: 6.4ms | bwd: 4238.6ms | clip: 0.0ms | optim: 49.7ms
647
+ step 290 | loss: 1.415617 | lr 1.5933e-04 | norm: 0.3453 | dt: 6311.40ms | tok/sec: 83070.0 | mem: 108909MB | Time Left: 00 hrs : 11 mins
648
+ [TIMING] data: 3.5ms | fwd: 2016.9ms | loss: 6.4ms | bwd: 4227.4ms | clip: 0.0ms | optim: 49.3ms
649
+ step 291 | loss: 1.482946 | lr 1.5837e-04 | norm: 0.1834 | dt: 6309.17ms | tok/sec: 83099.4 | mem: 108909MB | Time Left: 00 hrs : 11 mins
650
+ [TIMING] data: 3.3ms | fwd: 2009.8ms | loss: 6.4ms | bwd: 4232.3ms | clip: 0.0ms | optim: 49.4ms
651
+ step 292 | loss: 1.491100 | lr 1.5742e-04 | norm: 0.2317 | dt: 6353.94ms | tok/sec: 82513.8 | mem: 108909MB | Time Left: 00 hrs : 11 mins
652
+ [TIMING] data: 3.4ms | fwd: 2082.3ms | loss: 6.4ms | bwd: 4204.5ms | clip: 0.0ms | optim: 49.5ms
653
+ step 293 | loss: 1.501492 | lr 1.5648e-04 | norm: 0.1793 | dt: 6313.35ms | tok/sec: 83044.3 | mem: 108909MB | Time Left: 00 hrs : 11 mins
654
+ [TIMING] data: 3.4ms | fwd: 2013.2ms | loss: 6.4ms | bwd: 4233.0ms | clip: 0.0ms | optim: 49.4ms
655
+ step 294 | loss: 1.435696 | lr 1.5554e-04 | norm: 0.1851 | dt: 6319.12ms | tok/sec: 82968.6 | mem: 108909MB | Time Left: 00 hrs : 11 mins
656
+ [TIMING] data: 3.4ms | fwd: 2018.0ms | loss: 6.4ms | bwd: 4233.6ms | clip: 0.0ms | optim: 49.8ms
657
+ step 295 | loss: 1.491356 | lr 1.5460e-04 | norm: 0.1827 | dt: 6321.44ms | tok/sec: 82938.0 | mem: 108909MB | Time Left: 00 hrs : 11 mins
658
+ [TIMING] data: 3.4ms | fwd: 2015.2ms | loss: 6.4ms | bwd: 4239.2ms | clip: 0.0ms | optim: 49.2ms
659
+ step 296 | loss: 1.504897 | lr 1.5367e-04 | norm: 0.2337 | dt: 6323.81ms | tok/sec: 82907.0 | mem: 108909MB | Time Left: 00 hrs : 10 mins
660
+ [TIMING] data: 3.3ms | fwd: 2018.9ms | loss: 6.4ms | bwd: 4236.5ms | clip: 0.0ms | optim: 50.7ms
661
+ step 297 | loss: 1.437167 | lr 1.5274e-04 | norm: 0.1649 | dt: 6315.66ms | tok/sec: 83014.0 | mem: 108909MB | Time Left: 00 hrs : 10 mins
662
+ [TIMING] data: 3.5ms | fwd: 2015.1ms | loss: 6.4ms | bwd: 4230.9ms | clip: 0.0ms | optim: 51.8ms
663
+ step 298 | loss: 1.450542 | lr 1.5182e-04 | norm: 0.2649 | dt: 6318.76ms | tok/sec: 82973.3 | mem: 108909MB | Time Left: 00 hrs : 10 mins
664
+ [TIMING] data: 3.4ms | fwd: 2012.5ms | loss: 6.4ms | bwd: 4237.0ms | clip: 0.0ms | optim: 51.4ms
665
+ step 299 | loss: 1.449647 | lr 1.5091e-04 | norm: 0.1781 | dt: 6340.15ms | tok/sec: 82693.3 | mem: 108909MB | Time Left: 00 hrs : 10 mins
666
+ [TIMING] data: 3.4ms | fwd: 2048.9ms | loss: 6.4ms | bwd: 4222.6ms | clip: 0.0ms | optim: 50.9ms
667
+ [WARN] checkpoint save at step 300 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_3/blumodelcp_run13_step_300.tmp
668
+ [WARN] checkpoint save at step 300 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_2/blumodelcp_run13_step_300.tmp
669
+ [WARN] checkpoint save at step 300 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_1/blumodelcp_run13_step_300.tmp
670
+ [WARN] checkpoint save at step 300 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_0/blumodelcp_run13_step_300.tmp
671
+ step 300 | loss: 1.499225 | lr 1.5000e-04 | norm: 0.2531 | dt: 6180.78ms | tok/sec: 84825.5 | mem: 108909MB | Time Left: 00 hrs : 10 mins
672
+ [TIMING] data: 3.3ms | fwd: 1987.5ms | loss: 6.4ms | bwd: 4126.2ms | clip: 0.0ms | optim: 49.4ms
673
+ step 301 | loss: 1.472106 | lr 1.4910e-04 | norm: 0.1649 | dt: 6223.69ms | tok/sec: 84240.7 | mem: 108909MB | Time Left: 00 hrs : 10 mins
674
+ [TIMING] data: 3.3ms | fwd: 1990.1ms | loss: 6.4ms | bwd: 4166.3ms | clip: 0.0ms | optim: 49.8ms
675
+ step 302 | loss: 1.473293 | lr 1.4820e-04 | norm: 0.1976 | dt: 6230.85ms | tok/sec: 84144.0 | mem: 108909MB | Time Left: 00 hrs : 10 mins
676
+ [TIMING] data: 3.2ms | fwd: 1996.4ms | loss: 6.4ms | bwd: 4167.6ms | clip: 0.0ms | optim: 49.4ms
677
+ step 303 | loss: 1.397302 | lr 1.4730e-04 | norm: 0.2037 | dt: 6252.91ms | tok/sec: 83847.1 | mem: 108909MB | Time Left: 00 hrs : 10 mins
678
+ [TIMING] data: 3.2ms | fwd: 2001.3ms | loss: 6.4ms | bwd: 4184.7ms | clip: 0.0ms | optim: 49.4ms
679
+ step 304 | loss: 1.4step 396 | loss: 1.463978 | lr 1.0009e-04 | norm: 0.1220 | dt: 6371.53ms | tok/sec: 82286.0 | mem: 108909MB | Time Left: 00 hrs : 00 mins
680
+ [TIMING] data: 3.3ms | fwd: 2075.1ms | loss: 6.4ms | bwd: 4229.4ms | clip: 0.0ms | optim: 49.4ms
681
+ step 397 | loss: 1.432412 | lr 1.0005e-04 | norm: 0.1516 | dt: 6305.59ms | tok/sec: 83146.6 | mem: 108909MB | Time Left: 00 hrs : 00 mins
682
+ [TIMING] data: 3.3ms | fwd: 2008.0ms | loss: 6.4ms | bwd: 4230.5ms | clip: 0.0ms | optim: 49.5ms
683
+ step 398 | loss: 1.444381 | lr 1.0002e-04 | norm: 0.1591 | dt: 6290.80ms | tok/sec: 83342.1 | mem: 108909MB | Time Left: 00 hrs : 00 mins
684
+ [TIMING] data: 3.3ms | fwd: 2007.2ms | loss: 6.4ms | bwd: 4216.6ms | clip: 0.0ms | optim: 49.5ms
685
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_val_00000.bin
686
+ 100000000
687
+ validation loss: 1.4434
688
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_b/in2b_train_00000.bin
689
+ 100000000
690
+ step 399 | loss: 1.451133 | lr 1.0001e-04 | norm: 0.1280 | dt: 11381.27ms | tok/sec: 46065.9 | mem: 108909MB | Time Left: 00 hrs : 00 mins
691
+ [TIMING] data: 15.9ms | fwd: 2017.6ms | loss: 6.4ms | bwd: 4236.8ms | clip: 0.0ms | optim: 49.4ms
692
+
693
+ === Training complete ===
logs/in2/arm_c.trained ADDED
@@ -0,0 +1 @@
 
 
1
+ 14
logs/in2/arm_c_train.log ADDED
@@ -0,0 +1,297 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === Llama Context Parallel Training Script (bluscriptCP) ===
2
+ [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
3
+ [ZeRO-1] CP_ZERO_SYMMETRIC=1: NCCL symmetric memory mode (NVLS)
4
+ [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
5
+ [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
6
+ ================ Configuration (bluscriptCP) ================
7
+ d_model/layers : 1536 / 36
8
+ q/kv heads : 12 / 4 (G=3, hd=128)
9
+ ffn_hidden : 4096 (SwiGLU)
10
+ context_length : 16384
11
+ vocab_size : 50304 tying=0
12
+ B x T : 4 x 16384
13
+ world_size : 4 (dp=1 x cp=4)
14
+ CP_ATTN_MODE : ring (load_balance=HeadTail)
15
+ global_batch : 524288 (grad_accum=8)
16
+ optimizer : Muon (2-D body) + AdamW (wte + 1-D) [ZeRO-1] muon_lr=0.001000
17
+ precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
18
+ max_lr/min_lr : 0.001 / 0.0001
19
+ warmup/steps : 1 / 400 (cosine, no stable phase)
20
+ checkpointing : true (ckpt_freq=100, dir=checkpoints_bluscriptcp)
21
+ =============================================================
22
+ Initializing model on CUDA device 0...
23
+ [YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0 -- cache scaled for a different context than it is sized for
24
+ [YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=7 high=25 m=1.1386
25
+ [InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt (base step 750, loss 1.28324) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
26
+ Parameters: 1060624896
27
+ Parameters per GPU: 1060624896
28
+ max_steps: 400
29
+ warmup_steps: 1
30
+ [Muon] adjust_lr=match_rms_adamw (default) ns_steps=5 momentum=0.95
31
+ [ZeRO-1/Muon] reduce-scatter/all-gather layout: 6 bucket(s), 12 collectives/step, shard 226492416 elems/rank, padding 0.000000% (flat buffer 1728.000000 MB)
32
+ [ZeRO-1/Muon] adjust_lr=match_rms_adamw | params=905969664 (216 orthogonalized / 0 AdamW) muon_elems=905969664 | W=4 owned=226492416 (42 tensors) | master+momentum: 1728.000000 MB/GPU
33
+ [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
34
+ [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=ON overlap=-/-
35
+ [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
36
+ [ZeRO-1 v2] params=77266944 padded=77266944 W=4 shard=19316736 | 3 buckets of 50.000000 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 221.062500 MB (vs 884.250000 MB unsharded)
37
+ [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
38
+ [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=ON overlap=-/-
39
+ [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (symmetric/NVLS)
40
+ [ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50.000000 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.409668 MB (vs 885.638672 MB unsharded)
41
+ [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=3 [muon:bfloat16/905M] [adamw:bfloat16/77266944] [adamw:float32/77388288]
42
+ [ZeRO-1] DataParallel is DISABLED (both would double-reduce)
43
+ [precision] bf16 params=217, all with fp32 master weights
44
+ found 3 shards for split train
45
+ found 1 shards for split val
46
+ Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log14.csv (run 14)
47
+ [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
48
+ Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run14_step_<N>.ckpt (every 100 steps, keep 2)
49
+ [re-warmup] anchored at step 0 for 100 steps -> peak 0.000300 (0.300000 x max_lr), then cosine-decay to 0.000100 | resuming at step 0 (mid-ramp)
50
+
51
+ Starting training...
52
+
53
+ [CP ring P2P] OVERLAP: dedicated non-blocking stream
54
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/in2_data/arm_c/in2c_val_00000.bin
55
+ 100000000
56
+ validation loss: 1.8887
57
+ [CP bwd] overlap=ON
58
+ step 0 | loss: 1.920389 | lr 3.0000e-06 | norm: 1.1394 | dt: 16001.44ms | tok/sec: 32765.1 | mem: 107213MB | Time Left: 01 hrs : 46 mins
59
+ [TIMING] data: 2.9ms | fwd: 4222.9ms | loss: 6.4ms | bwd: 4663.3ms | clip: 0.0ms | optim: 62.6ms
60
+ step 1 | loss: 1.898834 | lr 6.0000e-06 | norm: 1.1567 | dt: 8280.12ms | tok/sec: 63318.9 | mem: 108909MB | Time Left: 00 hrs : 55 mins
61
+ [TIMING] data: 4.0ms | fwd: 3667.5ms | loss: 6.4ms | bwd: 4544.2ms | clip: 0.0ms | optim: 49.8ms
62
+ step 2 | loss: 1.910765 | lr 9.0000e-06 | norm: 1.2803 | dt: 9915.14ms | tok/sec: 52877.5 | mem: 108909MB | Time Left: 01 hrs : 05 mins
63
+ [TIMING] data: 2.6ms | fwd: 5223.5ms | loss: 6.4ms | bwd: 4625.0ms | clip: 0.0ms | optim: 49.6ms
64
+ step 3 | loss: 1.936233 | lr 1.2000e-05 | norm: 1.2666 | dt: 6216.73ms | tok/sec: 84335.0 | mem: 108909MB | Time Left: 00 hrs : 41 mins
65
+ [TIMING] data: 2.6ms | fwd: 1991.8ms | loss: 6.4ms | bwd: 4158.1ms | clip: 0.0ms | optim: 49.9ms
66
+ step 4 | loss: 1.905092 | lr 1.5000e-05 | norm: 1.1602 | dt: 6238.63ms | tok/sec: 84039.0 | mem: 108909MB | Time Left: 00 hrs : 41 mins
67
+ [TIMING] data: 3.1ms | fwd: 1995.3ms | loss: 6.4ms | bwd: 4176.5ms | clip: 0.0ms | optim: 49.3ms
68
+ step 5 | loss: 1.966748 | lr 1.8000e-05 | norm: 1.1494 | dt: 6276.42ms | tok/sec: 83532.9 | mem: 108909MB | Time Left: 00 hrs : 41 mins
69
+ [TIMING] data: 3.1ms | fwd: 2005.5ms | loss: 6.4ms | bwd: 4204.3ms | clip: 0.0ms | optim: 49.3ms
70
+ step 6 | loss: 1.904285 | lr 2.1000e-05 | norm: 1.0437 | dt: 6345.53ms | tok/sec: 82623.2 | mem: 108909MB | Time Left: 00 hrs : 41 mins
71
+ [TIMING] data: 3.2ms | fwd: 2013.2ms | loss: 6.4ms | bwd: 4265.3ms | clip: 0.0ms | optim: 49.4ms
72
+ step 7 | loss: 1.854531 | lr 2.4000e-05 | norm: 1.0216 | dt: 6402.65ms | tok/sec: 81886.1 | mem: 108909MB | Time Left: 00 hrs : 41 mins
73
+ [TIMING] data: 3.1ms | fwd: 2033.1ms | loss: 6.4ms | bwd: 4302.6ms | clip: 0.0ms | optim: 49.6ms
74
+ step 8 | loss: 1.830776 | lr 2.7000e-05 | norm: 0.9876 | dt: 6419.80ms | tok/sec: 81667.3 | mem: 108909MB | Time Left: 00 hrs : 41 mins
75
+ [TIMING] data: 3.0ms | fwd: 2037.5ms | loss: 6.4ms | bwd: 4315.3ms | clip: 0.0ms | optim: 49.6ms
76
+ step 9 | loss: 1.852669 | lr 3.0000e-05 | norm: 0.8843 | dt: 6426.62ms | tok/sec: 81580.7 | mem: 108909MB | Time Left: 00 hrs : 41 mins
77
+ [TIMING] data: 3.1ms | fwd: 2048.3ms | loss: 6.4ms | bwd: 4311.6ms | clip: 0.0ms | optim: 49.4ms
78
+ step 10 | loss: 1.806110 | lr 3.3000e-05 | norm: 0.8296 | dt: 6450.76ms | tok/sec: 81275.3 | mem: 108909MB | Time Left: 00 hrs : 41 mins
79
+ [TIMING] data: 3.1ms | fwd: 2058.2ms | loss: 6.7ms | bwd: 4325.5ms | clip: 0.0ms | optim: 49.4ms
80
+ step 11 | loss: 1.792623 | lr 3.6000e-05 | norm: 1.0501 | dt: 6435.64ms | tok/sec: 81466.3 | mem: 108909MB | Time Left: 00 hrs : 41 mins
81
+ [TIMING] data: 3.1ms | fwd: 2058.2ms | loss: 6.6ms | bwd: 4310.1ms | clip: 0.0ms | optim: 49.8ms
82
+ step 12 | loss: 1.784030 | lr 3.9000e-05 | norm: 1.0424 | dt: 6422.32ms | tok/sec: 81635.3 | mem: 108909MB | Time Left: 00 hrs : 41 mins
83
+ [TIMING] data: 3.1ms | fwd: 2025.7ms | loss: 6.4ms | bwd: 4329.7ms | clip: 0.0ms | optim: 49.5ms
84
+ step 13 | loss: 1.786141 | lr 4.2000e-05 | norm: 1.3447 | dt: 6375.98ms | tok/sec: 82228.6 | mem: 108909MB | Time Left: 00 hrs : 41 mins
85
+ [TIMING] data: 3.2ms | fwd: 2025.6ms | loss: 6.4ms | bwd: 4283.1ms | clip: 0.0ms | optim: 49.7ms
86
+ step 14 | loss: 1.809938 | lr 4.5000e-05 | norm: 1.2942 | dt: 6358.03ms | tok/sec: 82460.8 | mem: 108909MB | Time Left: 00 hrs : 40 mins
87
+ [TIMING] data: 3.1ms | fwd: 2027.8ms | loss: 6.4ms | bwd: 4263.5ms | clip: 0.0ms | optim: 49.4ms
88
+ step 15 | loss: 1.793668 | lr 4.8000e-05 | norm: 1.2557 | dt: 6368.77ms | tok/sec: 82321.7 | mem: 108909MB | Time Left: 00 hrs : 40 mins
89
+ [TIMING] data: 3.2ms | fwd: 2042.4ms | loss: 6.4ms | bwd: 4259.3ms | clip: 0.0ms | optim: 49.7ms
90
+ step 16 | loss: 1.765961 | lr 5.1000e-05 | norm: 1.0888 | dt: 6343.39ms | tok/sec: 82651.1 | mem: 108909MB | Time Left: 00 hrs : 40 mins
91
+ [TIMING] data: 3.2ms | fwd: 2021.3ms | loss: 6.4ms | bwd: 4254.8ms | clip: 0.0ms | optim: 49.9ms
92
+ step 17 | loss: 1.712484 | lr 5.4000e-05 | norm: 0.7903 | dt: 6323.92ms | tok/sec: 82905.5 | mem: 108909MB | Time Left: 00 hrs : 40 mins
93
+ [TIMING] data: 3.2ms | fwd: 2022.9ms | loss: 6.4ms | bwd: 4234.1ms | clip: 0.0ms | optim: 49.5ms
94
+ step 18 | loss: 1.731035 | lr 5.7000e-05 | norm: 0.8584 | dt: 6329.27ms | tok/sec: 82835.4 | mem: 108909MB | Time Left: 00 hrs : 40 mins
95
+ [TIMING] data: 3.0ms | fwd: 2017.8ms | loss: 6.6ms | bwd: 4244.6ms | clip: 0.0ms | optim: 49.3ms
96
+ step 19 | loss: 1.743202 | lr 6.0000e-05 | norm: 0.7385 | dt: 6334.25ms | tok/sec: 82770.3 | mem: 108909MB | Time Left: 00 hrs : 40 mins
97
+ [TIMING] data: 3.2ms | fwd: 2031.9ms | loss: 6.4ms | bwd: 4235.3ms | clip: 0.0ms | optim: 49.7ms
98
+ step 20 | loss: 1.718590 | lr 6.3000e-05 | norm: 1.1694 | dt: 6330.50ms | tok/sec: 82819.3 | mem: 108909MB | Time Left: 00 hrs : 40 mins
99
+ [TIMING] data: 3.1ms | fwd: 2022.3ms | loss: 6.4ms | bwd: 4241.4ms | clip: 0.0ms | optim: 49.6ms
100
+ step 21 | loss: 1.767774 | lr 6.6000e-05 | norm: 0.9925 | dt: 6338.12ms | tok/sec: 82719.8 | mem: 108909MB | Time Left: 00 hrs : 40 mins
101
+ [TIMING] data: 3.1ms | fwd: 2018.0ms | loss: 6.4ms | bwd: 4253.2ms | clip: 0.0ms | optim: 49.6ms
102
+ step 22 | loss: 1.697396 | lr 6.9000e-05 | norm: 0.6909 | dt: 6326.69ms | tok/sec: 82869.2 | mem: 108909MB | Time Left: 00 hrs : 39 mins
103
+ [TIMING] data: 3.2ms | fwd: 2019.1ms | loss: 6.4ms | bwd: 4240.6ms | clip: 0.0ms | optim: 49.6ms
104
+ step 23 | loss: 1.707077 | lr 7.2000e-05 | norm: 0.7638 | dt: 6371.41ms | tok/sec: 82287.6 | mem: 108909MB | Time Left: 00 hrs : 40 mins
105
+ [TIMING] data: 3.2ms | fwd: 2050.0ms | loss: 6.4ms | bwd: 4254.6ms | clip: 0.0ms | optim: 49.3ms
106
+ step 24 | loss: 1.671977 | lr 7.5000e-05 | norm: 0.7776 | dt: 6375.66ms | tok/sec: 82232.8 | mem: 108909MB | Time Left: 00 hrs : 39 mins
107
+ [TIMING] data: 3.3ms | fwd: 2069.2ms | loss: 6.4ms | bwd: 4239.3ms | clip: 0.0ms | optim: 49.6ms
108
+ step 25 | loss: 1.739844 | lr 7.8000e-05 | norm: 0.8230 | dt: 6332.83ms | tok/sec: 82788.9 | mem: 108909MB | Time Left: 00 hrs : 39 mins
109
+ [TIMING] data: 3.2ms | fwd: 2011.0ms | loss: 6.4ms | bwd: 4254.9ms | clip: 0.0ms | optim: 49.6ms
110
+ step 26 | loss: 1.652054 | lr 8.1000e-05 | norm: 0.9077 | dt: 6340.05ms | tok/sec: 82694.6 | mem: 108909MB | Time Left: 00 hrs : 39 mins
111
+ [TIMING] data: 3.1ms | fwd: 2025.9ms | loss: 6.4ms | bwd: 4247.1ms | clip: 0.0ms | optim: 49.7ms
112
+ step 27 | loss: 1.654510 | lr 8.4000e-05 | norm: 0.8483 | dt: 6341.17ms | tok/sec: 82680.0 | mem: 108909MB | Time Left: 00 hrs : 39 mins
113
+ [TIMING] data: 3.1ms | fwd: 2023.2ms | loss: 6.4ms | bwd: 4250.9ms | clip: 0.0ms | optim: 49.7ms
114
+ step 28 | loss: 1.691144 | lr 8.7000e-05 | norm: 0.7625 | dt: 6357.39ms | tok/sec: 82469.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
115
+ [TIMING] data: 3.2ms | fwd: 2045.9ms | loss: 6.4ms | bwd: 4244.7ms | clip: 0.0ms | optim: 49.4ms
116
+ step 29 | loss: 1.663364 | lr 9.0000e-05 | norm: 0.9021 | dt: 6359.41ms | tok/sec: 82442.9 | mem: 108909MB | Time Left: 00 hrs : 39 mins
117
+ [TIMING] data: 3.1ms | fwd: 2024.0ms | loss: 6.4ms | bwd: 4268.5ms | clip: 0.0ms | optim: 49.4ms
118
+ step 30 | loss: 1.633048 | lr 9.3000e-05 | norm: 0.7084 | dt: 6332.02ms | tok/sec: 82799.4 | mem: 108909MB | Time Left: 00 hrs : 39 mins
119
+ [TIMING] data: 3.1ms | fwd: 2012.0ms | loss: 6.4ms | bwd: 4253.2ms | clip: 0.0ms | optim: 49.5ms
120
+ step 31 | loss: 1.700897 | lr 9.6000e-05 | norm: 0.5941 | dt: 6339.41ms | tok/sec: 82702.9 | mem: 108909MB | Time Left: 00 hrs : 38 mins
121
+ [TIMING] data: 3.1ms | fwd: 2024.3ms | loss: 6.4ms | bwd: 4248.3ms | clip: 0.0ms | optim: 49.5ms
122
+ step 32 | loss: 1.608716 | lr 9.9000e-05 | norm: 0.6328 | dt: 6395.62ms | tok/sec: 81976.1 | mem: 108909MB | Time Left: 00 hrs : 39 mins
123
+ [TIMING] data: 3.2ms | fwd: 2066.5ms | loss: 6.4ms | bwd: 4262.0ms | clip: 0.0ms | optim: 49.6ms
124
+ step 33 | loss: 1.657396 | lr 1.0200e-04 | norm: 0.6388 | dt: 6355.23ms | tok/sec: 82497.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
125
+ [TIMING] data: 3.1ms | fwd: 2024.5ms | loss: 6.4ms | bwd: 4263.7ms | clip: 0.0ms | optim: 49.7ms
126
+ step 34 | loss: 1.648264 | lr 1.0500e-04 | norm: 0.6340 | dt: 6350.56ms | tok/sec: 82557.8 | mem: 108909MB | Time Left: 00 hrs : 38 mins
127
+ [TIMING] data: 3.1ms | fwd: 2026.4ms | loss: 6.4ms | bwd: 4257.2ms | clip: 0.0ms | optim: 49.6ms
128
+ step 35 | loss: 1.660285 | lr 1.0800e-04 | norm: 0.7759 | dt: 6348.23ms | tok/sec: 82588.0 | mem: 108909MB | Time Left: 00 hrs : 38 mins
129
+ [TIMING] data: 3.1ms | fwd: 2080.1ms | loss: 6.4ms | bwd: 4201.3ms | clip: 0.0ms | optim: 49.5ms
130
+ step 36 | loss: 1.631099 | lr 1.1100e-04 | norm: 0.5843 | dt: 6381.27ms | tok/sec: 82160.5 | mem: 108909MB | Time Left: 00 hrs : 38 mins
131
+ [TIMING] data: 3.1ms | fwd: 2063.8ms | loss: 6.4ms | bwd: 4250.2ms | clip: 0.0ms | optim: 50.0ms
132
+ step 37 | loss: 1.626856 | lr 1.1400e-04 | norm: 0.6801 | dt: 6330.01ms | tok/sec: 82825.7 | mem: 108909MB | Time Left: 00 hrs : 38 mins
133
+ [TIMING] data: 3.2ms | fwd: 2013.3ms | loss: 6.4ms | bwd: 4249.9ms | clip: 0.0ms | optim: 49.4ms
134
+ step 38 | loss: 1.619444 | lr 1.1700e-04 | norm: 0.6525 | dt: 6337.69ms | tok/sec: 82725.4 | mem: 108909MB | Time Left: 00 hrs : 38 mins
135
+ [TIMING] data: 3.2ms | fwd: 2020.1ms | loss: 6.4ms | bwd: 4250.6ms | clip: 0.0ms | optim: 49.7ms
136
+ step 39 | loss: 1.642475 | lr 1.2000e-04 | norm: 0.7724 | dt: 6377.82ms | tok/sec: 82204.8 | mem: 108909MB | Time Left: 00 hrs : 38 mins
137
+ [TIMING] data: 3.2ms | fwd: 2075.6ms | loss: 6.4ms | bwd: 4235.5ms | clip: 0.0ms | optim: 49.3ms
138
+ step 40 | loss: 1.600698 | lr 1.2300e-04 | norm: 0.8544 | dt: 6376.74ms | tok/sec: 82218.8 | mem: 108909MB | Time Left: 00 hrs : 38 mins
139
+ [TIMING] data: 3.2ms | fwd: 2071.6ms | loss: 6.4ms | bwd: 4238.0ms | clip: 0.0ms | optim: 49.7ms
140
+ step 41 | loss: 1.580887 | lr 1.2600e-04 | norm: 0.9816 | dt: 6324.28ms | tok/sec: 82900.8 | mem: 108909MB | Time Left: 00 hrs : 37 mins
141
+ [TIMING] data: 3.1ms | fwd: 2022.0ms | loss: 6.4ms | bwd: 4235.4ms | clip: 0.0ms | optim: 49.6ms
142
+ step 42 | loss: 1.624956 | lr 1.2900e-04 | norm: 0.8724 | dt: 6332.33ms | tok/sec: 82795.5 | mem: 108909MB | Time Left: 00 hrs : 37 mins
143
+ [TIMING] data: 3.1ms | fwd: 2018.1ms | loss: 6.7ms | bwd: 4246.9ms | clip: 0.0ms | optim: 49.7ms
144
+ step 43 | loss: 1.623077 | lr 1.3200e-04 | norm: 0.6374 | dt: 6346.73ms | tok/sec: 82607.6 | mem: 108909MB | Time Left: 00 hrs : 37 mins
145
+ [TIMING] data: 3.3ms | fwd: 2024.3ms | loss: 6.4ms | bwd: 4255.4ms | clip: 0.0ms | optim: 49.5ms
146
+ step 44 | loss: 1.554411 | lr 1.3500e-04 | norm: 0.8312 | dt: 6327.95ms | tok/sec: 82852.7 | mem: 108909MB | Time Left: 00 hrs : 37 mins
147
+ [TIMING] data: 3.3ms | fwd: 2012.0ms | loss: 6.4ms | bwd: 4248.7ms | clip: 0.0ms | optim: 49.7ms
148
+ step 45 | loss: 1.633371 | lr 1.3800e-04 | norm: 0.7739 | dt: 6346.30ms | tok/sec: 82613.2 | mem: 108909MB | Time Left: 00 hrs : 37 mins
149
+ [TIMING] data: 3.1ms | fwd: 2037.5ms | loss: 6.4ms | bwd: 4241.6ms | clip: 0.0ms | optim: 49.8ms
150
+ step 46 | loss: 1.607605 | lr 1.4100e-04 | norm: 0.7427 | dt: 6313.64ms | tok/sec: 83040.5 | mem: 108909MB | Time Left: 00 hrs : 37 mins
151
+ [TIMING] data: 3.1ms | fwd: 2019.0ms | loss: 6.4ms | bwd: 4227.8ms | clip: 0.0ms | optim: 49.5ms
152
+ step 47 | loss: 1.635939 | lr 1.4400e-04 | norm: 0.6916 | dt: 6347.58ms | tok/sec: 82596.6 | mem: 108909MB | Time Left: 00 hrs : 37 mins
153
+ [TIMING] data: 3.1ms | fwd: 2024.1ms | loss: 6.4ms | bwd: 4256.4ms | clip: 0.0ms | optim: 49.7ms
154
+ step 48 | loss: 1.633764 | lr 1.4700e-04 | norm: 0.8044 | dt: 6336.69ms | tok/sec: 82738.5 | mem: 108909MB | Time Left: 00 hrs : 37 mins
155
+ [TIMING] data: 3.3ms | fwd: 2021.2ms | loss: 6.4ms | bwd: 4248.1ms | clip: 0.0ms | optim: 49.6ms
156
+ step 49 | loss: 1.607873 | lr 1.5000e-04 | norm: 0.6211 | dt: 6305.45ms | tok/sec: 83148.4 | mem: 108909MB | Time Left: 00 hrs : 36 mins
157
+ [TIMING] data: 3.2ms | fwd: 2019.0ms | loss: 6.4ms | bwd: 4219.5ms | clip: 0.0ms | optim: 49.6ms
158
+ step 50 | loss: 1.639636 | lr 1.5300e-04 | norm: 0.7115 | dt: 6370.11ms | tok/sec: 82304.3 | mem: 108909MB | Time Left: 00 hrs : 37 mins
159
+ [TIMING] data: 3.2ms | fwd: 2078.6ms | loss: 6.4ms | bwd: 4224.5ms | clip: 0.0ms | optim: 49.5ms
160
+ step 51 | loss: 1.656035 | lr 1.5600e-04 | norm: 0.7963 | dt: 6345.47ms | tok/sec: 82624.0 | mem: 108909MB | Time Left: 00 hrs : 36 mins
161
+ [TIMING] data: 3.2ms | fwd: 2031.6ms | loss: 6.6ms | bwd: 4246.8ms | clip: 0.0ms | optim: 49.5ms
162
+ step 52 | loss: 1.662412 | lr 1.5900e-04 | norm: 0.7057 | dt: 6316.86ms | tok/sec: 82998.2 | mem: 108909MB | Time Left: 00 hrs : 36 mins
163
+ [TIMING] data: 3.1ms | fwd: 2016.9ms | loss: 6.4ms | bwd: 4232.1ms | clip: 0.0ms | optim: 49.5ms
164
+ step 53 | loss: 1.581693 | lr 1.6200e-04 | norm: 0.6668 | dt: 6342.80ms | tok/sec: 82658.8 | mem: 108909MB | Time Left: 00 hrs : 36 mins
165
+ [TIMING] data: 3.2ms | fwd: 2028.6ms | loss: 6.4ms | bwd: 4247.1ms | clip: 0.0ms | optim: 49.7ms
166
+ step 54 | loss: 1.594439 | lr 1.6500e-04 | norm: 0.8471 | dt: 6319.42ms | tok/sec: 82964.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
167
+ [TIMING] data: 3.2ms | fwd: 2024.6ms | loss: 6.4ms | bwd: 4227.7ms | clip: 0.0ms | optim: 49.6ms
168
+ step 55 | loss: 1.670181 | lr 1.6800e-04 | norm: 0.8091 | dt: 6371.77ms | tok/sec: 82282.9 | mem: 108909MB | Time Left: 00 hrs : 36 mins
169
+ [TIMING] data: 3.2ms | fwd: 2057.9ms | loss: 6.4ms | bwd: 4247.0ms | clip: 0.0ms | optim: 49.4ms
170
+ step 56 | loss: 1.632861 | lr 1.7100e-04 | norm: 0.7737 | dt: 6320.74ms | tok/sec: 82947.3 | mem: 108909MB | Time Left: 00 hrs : 36 mins
171
+ [TIMING] data: 3.2ms | fwd: 2018.6ms | loss: 6.4ms | bwd: 4235.3ms | clip: 0.0ms | optim: 49.5ms
172
+ step 57 | loss: 1.572145 | lr 1.7400e-04 | norm: 0.8922 | dt: 6340.90ms | tok/sec: 82683.6 | mem: 108909MB | Time Left: 00 hrs : 36 mins
173
+ [TIMING] data: 3.2ms | fwd: 2020.7ms | loss: 6.4ms | bwd: 4253.2ms | clip: 0.0ms | optim: 49.5ms
174
+ step 58 | loss: 1.589554 | lr 1.7700e-04 | norm: 0.6774 | dt: 6324.35ms | tok/sec: 82899.9 | mem: 108909MB | Time Left: 00 hrs : 36 mins
175
+ [TIMING] data: 3.1ms | fwd: 2026.5ms | loss: 6.4ms | bwd: 4230.7ms | clip: 0.0ms | optim: 49.8ms
176
+ step 59 | loss: 1.586414 | lr 1.8000e-04 | norm: 0.7578 | dt: 6376.86ms | tok/sec: 82217.2 | mem: 108909MB | Time Left: 00 hrs : 36 mins
177
+ [TIMING] data: 3.2ms | fwd: 2054.2ms | loss: 6.5ms | bwd: 4255.8ms | clip: 0.0ms | optim: 49.3ms
178
+ step 60 | loss: 1.630998 | lr 1.8300e-04 | norm: 0.7535 | dt: 6323.77ms | tok/sec: 82907.5 | mem: 108909MB | Time Left: 00 hrs : 35 mins
179
+ [TIMING] data: 3.2ms | fwd: 2018.2ms | loss: 6.4ms | bwd: 4238.6ms | clip: 0.0ms | optim: 49.5ms
180
+ step 61 | loss: 1.556511 | lr 1.8600e-04 | norm: 0.8945 | dt: 6345.84ms | tok/sec: 82619.2 | mem: 108909MB | Time Left: 00 hrs : 35 mins
181
+ [TIMING] data: 3.2ms | fwd: 2025.4ms | loss: 6.4ms | bwd: 4252.0ms | clip: 0.0ms | optim: 50.9ms
182
+ step 62 | loss: 1.594290 | lr 1.8900e-04 | norm: 0.8499 | dt: 6348.36ms | tok/sec: 82586.3 | mem: 108909MB | Time Left: 00 hrs : 35 mins
183
+ [TIMING] data: 3.2ms | fwd: 2026.3ms | loss: 6.4ms | bwd: 4255.1ms | clip: 0.0ms | optim: 49.6ms
184
+ step 63 | loss: 1.591010 | lr 1.9200e-04 | norm: 0.7212 | dt: 6382.64ms | tok/sec: 82142.8 | mem: 108909MB | Time Left: 00 hrs : 35 mins
185
+ [TIMING] data: 3.1ms | fwd: 2080.1ms | loss: 6.4ms | bwd: 4235.5ms | clip: 0.0ms | optim: 49.6ms
186
+ step 64 | loss: 1.558378 | lr 1.9500e-04 | norm: 0.8342 | dt: 6370.17ms | tok/sec: 82303.6 | mem: 108909MB | Time Left: 00 hrs : 35 mins
187
+ [TIMING] data: 3.3ms | fwd: 2050.7ms | loss: 6.6ms | bwd: 4252.4ms | clip: 0.0ms | optim: 49.4ms
188
+ step 65 | loss: 1.605214 | lr 1.9800e-04 | norm: 0.6799 | dt: 6333.11ms | tok/sec: 82785.2 | mem: 108909MB | Time Left: 00 hrs : 35 mins
189
+ [TIMING] data: 3.1ms | fwd: 2021.7ms | loss: 6.4ms | bwd: 4244.5ms | clip: 0.0ms | optim: 49.5ms
190
+ step 66 | loss: 1.590822 | lr 2.0100e-04 | norm: 0.9325 | dt: 6346.55ms | tok/sec: 82609.9 | mem: 108909MB | Time Left: 00 hrs : 35 mins
191
+ [TIMING] data: 3.2ms | fwd: 2013.1ms | loss: 6.4ms | bwd: 4266.6ms | clip: 0.0ms | optim: 49.6ms
192
+ step 67 | loss: 1.622504 | lr 2.0400e-04 | norm: 0.8867 | dt: 6339.60ms | tok/sec: 82700.5 | mem: 108909MB | Time Left: 00 hrs : 35 mins
193
+ [TIMING] data: 3.2ms | fwd: 2021.6ms | loss: 6.4ms | bwd: 4250.8ms | clip: 0.0ms | optim: 49.8ms
194
+ step 68 | loss: 1.598213 | lr 2.0700e-04 | norm: 0.9454 | dt: 6362.03ms | tok/sec: 82408.9 | mem: 108909MB | Time Left: 00 hrs : 35 mins
195
+ [TIMING] data: 3.0ms | fwd: 2052.2ms | loss: 6.4ms | bwd: 4242.9ms | clip: 0.0ms | optim: 49.7ms
196
+ step 69 | loss: 1.635720 | lr 2.1000e-04 | norm: 0.8285 | dt: 6375.84ms | tok/sec: 82230.5 | mem: 108909MB | Time Left: 00 hrs : 35 mins
197
+ [TIMING] data: 3.1ms | fwd: 2071.8ms | loss: 6.4ms | bwd: 4237.3ms | clip: 0.0ms | optim: 49.5ms
198
+ step 70 | loss: 1.608484 | lr 2.1300e-04 | norm: 0.7056 | dt: 6333.45ms | tok/sec: 82780.8 | mem: 108909MB | Time Left: 00 hrs : 34 mins
199
+ [TIMING] data: 3.1ms | fwd: 2030.7ms | loss: 6.4ms | bwd: 4235.7ms | clip: 0.0ms | optim: 49.8ms
200
+ step 71 | loss: 1.609575 | lr 2.1600e-04 | norm: 0.7693 | dt: 6333.45ms | tok/sec: 82780.8 | mem: 108909MB | Time Left: 00 hrs : 34 mins
201
+ [TIMING] data: 3.2ms | fwd: 2022.2ms | loss: 6.4ms | bwd: 4244.5ms | clip: 0.0ms | optim: 49.4ms
202
+ step 72 | loss: 1.544902 | lr 2.1900e-04 | norm: 0.7656 | dt: 6330.44ms | tok/sec: 82820.1 | mem: 108909MB | Time Left: 00 hrs : 34 mins
203
+ [TIMING] data: 3.0ms | fwd: 2021.9ms | loss: 6.4ms | bwd: 4241.3ms | clip: 0.0ms | optim: 50.0ms
204
+ step 73 | loss: 1.616233 | lr 2.2200e-04 | norm: 0.9089 | dt: 6347.53ms | tok/sec: 82597.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
205
+ [TIMING] data: 3.1ms | fwd: 2046.7ms | loss: 6.4ms | bwd: 4234.0ms | clip: 0.0ms | optim: 49.5ms
206
+ step 74 | loss: 1.594353 | lr 2.2500e-04 | norm: 0.8706 | dt: 6332.66ms | tok/sec: 82791.1 | mem: 108909MB | Time Left: 00 hrs : 34 mins
207
+ [TIMING] data: 3.0ms | fwd: 2023.6ms | loss: 6.4ms | bwd: 4242.1ms | clip: 0.0ms | optim: 49.7ms
208
+ step 75 | loss: 1.609479 | lr 2.2800e-04 | norm: 0.9014 | dt: 6339.55ms | tok/sec: 82701.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
209
+ [TIMING] data: 3.1ms | fwd: 2023.0ms | loss: 6.6ms | bwd: 4249.3ms | clip: 0.0ms | optim: 49.7ms
210
+ step 76 | loss: 1.552467 | lr 2.3100e-04 | norm: 0.7200 | dt: 6335.94ms | tok/sec: 82748.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
211
+ [TIMING] data: 3.1ms | fwd: 2018.6ms | loss: 6.4ms | bwd: 4250.5ms | clip: 0.0ms | optim: 49.5ms
212
+ step 77 | loss: 1.623230 | lr 2.3400e-04 | norm: 0.8687 | dt: 6327.53ms | tok/sec: 82858.2 | mem: 108909MB | Time Left: 00 hrs : 34 mins
213
+ [TIMING] data: 3.2ms | fwd: 2033.3ms | loss: 6.4ms | bwd: 4227.3ms | clip: 0.0ms | optim: 49.6ms
214
+ step 78 | loss: 1.589706 | lr 2.3700e-04 | norm: 0.7531 | dt: 6334.82ms | tok/sec: 82762.9 | mem: 108909MB | Time Left: 00 hrs : 33 mins
215
+ [TIMING] data: 3.1ms | fwd: 2022.4ms | loss: 6.4ms | bwd: 4245.0ms | clip: 0.0ms | optim: 50.0ms
216
+ step 79 | loss: 1.593231 | lr 2.4000e-04 | norm: 0.9309 | dt: 6320.51ms | tok/sec: 82950.2 | mem: 108909MB | Time Left: 00 hrs : 33 mins
217
+ [TIMING] data: 3.2ms | fwd: 2013.5ms | loss: 6.4ms | bwd: 4239.9ms | clip: 0.0ms | optim: 49.6ms
218
+ step 80 | loss: 1.596107 | lr 2.4300e-04 | norm: 1.6061 | dt: 6329.07ms | tok/sec: 82838.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
219
+ [TIMING] data: 3.1ms | fwd: 2014.9ms | loss: 6.4ms | bwd: 4247.6ms | clip: 0.0ms | optim: 49.2ms
220
+ step 81 | loss: 1.639003 | lr 2.4600e-04 | norm: 0.8177 | dt: 6320.26ms | tok/sec: 82953.6 | mem: 108909MB | Time Left: 00 hrs : 33 mins
221
+ [TIMING] data: 3.1ms | fwd: 2019.2ms | loss: 6.4ms | bwd: 4234.4ms | clip: 0.0ms | optim: 49.4ms
222
+ step 82 | loss: 1.667713 | lr 2.4900e-04 | norm: 0.8361 | dt: 6397.35ms | tok/sec: 81954.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
223
+ [TIMING] data: 3.1ms | fwd: 2098.5ms | loss: 6.4ms | bwd: 4232.2ms | clip: 0.0ms | optim: 49.3ms
224
+ step 83 | loss: 1.597706 | lr 2.5200e-04 | norm: 0.8508 | dt: 6376.57ms | tok/sec: 82221.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
225
+ [TIMING] data: 3.1ms | fwd: 2045.3ms | loss: 6.4ms | bwd: 4264.5ms | clip: 0.0ms | optim: 49.5ms
226
+ step 84 | loss: 1.606992 | lr 2.5500e-04 | norm: 0.8824 | dt: 6307.10ms | tok/sec: 83126.6 | mem: 108909MB | Time Left: 00 hrs : 33 mins
227
+ [TIMING] data: 3.2ms | fwd: 2014.7ms | loss: 6.4ms | bwd: 4225.3ms | clip: 0.0ms | optim: 49.5ms
228
+ step 85 | loss: 1.606081 | lr 2.5800e-04 | norm: 0.8665 | dt: 6330.49ms | tok/sec: 82819.5 | mem: 108909MB | Time Left: 00 hrs : 33 mins
229
+ [TIMING] data: 3.2ms | fwd: 2021.7ms | loss: 6.4ms | bwd: 4241.2ms | clip: 0.0ms | optim: 50.0ms
230
+ step 86 | loss: 1.608906 | lr 2.6100e-04 | norm: 0.9115 | dt: 6355.76ms | tok/sec: 82490.2 | mem: 108909MB | Time Left: 00 hrs : 33 mins
231
+ [TIMING] data: 3.1ms | fwd: 2027.0ms | loss: 6.4ms | bwd: 4261.7ms | clip: 0.0ms | optim: 49.7ms
232
+ step 87 | loss: 1.671125 | lr 2.6400e-04 | norm: 0.8144 | dt: 6334.82ms | tok/sec: 82762.8 | mem: 108909MB | Time Left: 00 hrs : 33 mins
233
+ [TIMING] data: 3.1ms | fwd: 2028.3ms | loss: 6.4ms | bwd: 4239.7ms | clip: 0.0ms | optim: 49.4ms
234
+ step 88 | loss: 1.673897 | lr 2.6700e-04 | norm: 0.8308 | dt: 6355.62ms | tok/sec: 82492.0 | mem: 108909MB | Time Left: 00 hrs : 33 mins
235
+ [TIMING] data: 3.1ms | fwd: 2057.7ms | loss: 6.4ms | bwd: 4230.9ms | clip: 0.0ms | optim: 49.7ms
236
+ step 89 | loss: 1.632511 | lr 2.7000e-04 | norm: 0.7274 | dt: 6347.37ms | tok/sec: 82599.2 | mem: 108909MB | Time Left: 00 hrs : 32 mins
237
+ [TIMING] data: 3.3ms | fwd: 2019.5ms | loss: 6.4ms | bwd: 4260.6ms | clip: 0.0ms | optim: 49.8ms
238
+ step 90 | loss: 1.688837 | lr 2.7300e-04 | norm: 0.8296 | dt: 6319.86ms | tok/sec: 82958.8 | mem: 108909MB | Time Left: 00 hrs : 32 mins
239
+ [TIMING] data: 3.2ms | fwd: 2016.4ms | loss: 6.4ms | bwd: 4236.4ms | clip: 0.0ms | optim: 49.5ms
240
+ step 91 | loss: 1.636074 | lr 2.7600e-04 | norm: 0.8177 | dt: 6328.08ms | tok/sec: 82851.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
241
+ [TIMING] data: 3.4ms | fwd: 2024.8ms | loss: 6.5ms | bwd: 4236.1ms | clip: 0.0ms | optim: 49.4ms
242
+ step 92 | loss: 1.643463 | lr 2.7900e-04 | norm: 0.8352 | dt: 6385.58ms | tok/sec: 82105.0 | mem: 108909MB | Time Left: 00 hrs : 32 mins
243
+ [TIMING] data: 3.1ms | fwd: 2085.3ms | loss: 6.4ms | bwd: 4233.4ms | clip: 0.0ms | optim: 49.5ms
244
+ step 93 | loss: 1.674679 | lr 2.8200e-04 | norm: 0.8445 | dt: 6345.84ms | tok/sec: 82619.2 | mem: 108909MB | Time Left: 00 hrs : 32 mins
245
+ [TIMING] data: 3.2ms | fwd: 2050.6ms | loss: 6.4ms | bwd: 4228.3ms | clip: 0.0ms | optim: 49.5ms
246
+ step 94 | loss: 1.552707 | lr 2.8500e-04 | norm: 0.8634 | dt: 6333.64ms | tok/sec: 82778.4 | mem: 108909MB | Time Left: 00 hrs : 32 mins
247
+ [TIMING] data: 3.1ms | fwd: 2024.6ms | loss: 6.4ms | bwd: 4242.3ms | clip: 0.0ms | optim: 49.4ms
248
+ step 95 | loss: 1.630905 | lr 2.8800e-04 | norm: 0.8320 | dt: 6353.83ms | tok/sec: 82515.3 | mem: 108909MB | Time Left: 00 hrs : 32 mins
249
+ [TIMING] data: 3.3ms | fwd: 2028.2ms | loss: 6.4ms | bwd: 4258.5ms | clip: 0.0ms | optim: 49.6ms
250
+ step 96 | loss: 1.621827 | lr 2.9100e-04 | norm: 0.9184 | dt: 6323.02ms | tok/sec: 82917.4 | mem: 108909MB | Time Left: 00 hrs : 32 mins
251
+ [TIMING] data: 3.2ms | fwd: 2012.9ms | loss: 6.5ms | bwd: 4242.1ms | clip: 0.0ms | optim: 50.3ms
252
+ step 97 | loss: 1.617399 | lr 2.9400e-04 | norm: 0.7699 | dt: 6362.22ms | tok/sec: 82406.5 | mem: 108909MB | Time Left: 00 hrs : 32 mins
253
+ [TIMING] data: 3.1ms | fwd: 2055.8ms | loss: 6.4ms | bwd: 4239.4ms | clip: 0.0ms | optim: 49.7ms
254
+ step 98 | loss: 1.612508 | lr 2.9700e-04 | norm: 0.7359 | dt: 6338.88ms | tok/sec: 82709.9 | mem: 108909MB | Time Left: 00 hrs : 31 mins
255
+ [TIMING] data: 3.3ms | fwd: 2024.5ms | loss: 6.4ms | bwd: 4246.3ms | clip: 0.0ms | optim: 50.0ms
256
+ step 99 | loss: 1.637773 | lr 3.0000e-04 | norm: 0.8468 | dt: 6349.17ms | tok/sec: 82575.8 | mem: 108909MB | Time Left: 00 hrs : 31 mins
257
+ [TIMING] data: 3.3ms | fwd: 2017.7ms | loss: 6.5ms | bwd: 4264.2ms | clip: 0.0ms | optim: 49.6ms
258
+ [WARN] checkpoint save at step 100 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_2/blumodelcp_run14_step_100.tmp
259
+ [WARN] checkpoint save at step 100 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_3/blumodelcp_run14_step_100.tmp
260
+ [WARN] checkpoint save at step 100 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_1/blumodelcp_run14_step_100.tmp
261
+ [WARN] checkpoint save at step 100 failed: Failed to flush checkpoint to: checkpoints_bluscriptcp/rank_0/blumodelcp_run14_step_100.tmp
262
+ step 100 | loss: 1.653108 | lr 3.0000e-04 | norm: 0.7348 | dt: 6202.84ms | tok/sec: 84523.8 | mem: 108909MB | Time Left: 00 hrs : 31 mins
263
+ [TIMING] data: 3.2ms | step 173 | loss: 1.634869 | lr 2.7218e-04 | norm: 0.4948 | dt: 6352.32ms | tok/sec: 82534.8 | mem: 108909MB | Time Left: 00 hrs : 24 mins
264
+ [TIMING] data: 3.4ms | fwd: 2025.4ms | loss: 6.5ms | bwd: 4259.7ms | clip: 0.0ms | optim: 49.4ms
265
+ step 174 | loss: 1.655519 | lr 2.7145e-04 | norm: 0.5231 | dt: 6317.55ms | tok/sec: 82989.1 | mem: 108909MB | Time Left: 00 hrs : 23 mins
266
+ [TIMING] data: 3.5ms | fwd: 2024.2ms | loss: 6.5ms | bwd: 4225.9ms | clip: 0.0ms | optim: 49.5ms
267
+ step 175 | loss: 1.636195 | lr 2.7071e-04 | norm: 0.4414 | dt: 6314.49ms | tok/sec: 83029.3 | mem: 108909MB | Time Left: 00 hrs : 23 mins
268
+ [TIMING] data: 3.5ms | fwd: 2019.0ms | loss: 6.5ms | bwd: 4228.3ms | clip: 0.0ms | optim: 49.3ms
269
+ step 176 | loss: 1.596313 | lr 2.6997e-04 | norm: 0.4120 | dt: 6341.42ms | tok/sec: 82676.8 | mem: 108909MB | Time Left: 00 hrs : 23 mins
270
+ [TIMING] data: 3.6ms | fwd: 2056.1ms | loss: 6.5ms | bwd: 4217.3ms | clip: 0.0ms | optim: 50.0ms
271
+ step 177 | loss: 1.690520 | lr 2.6921e-04 | norm: 0.5356 | dt: 6335.65ms | tok/sec: 82752.1 | mem: 108909MB | Time Left: 00 hrs : 23 mins
272
+ [TIMING] data: 3.6ms | fwd: 2028.3ms | loss: 6.5ms | bwd: 4239.8ms | clip: 0.0ms | optim: 49.5ms
273
+ step 178 | loss: 1.654294 | lr 2.6845e-04 | norm: 0.5014 | dt: 6333.05ms | tok/sec: 82786.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
274
+ [TIMING] data: 3.7ms | fwd: 2020.2ms | loss: 6.5ms | bwd: 4245.2ms | clip: 0.0ms | optim: 49.5ms
275
+ step 179 | loss: 1.655236 | lr 2.6769e-04 | norm: 0.5110 | dt: 6326.79ms | tok/sec: 82868.0 | mem: 108909MB | Time Left: 00 hrs : 23 mins
276
+ [TIMING] data: 3.4ms | fwd: 2023.6ms | loss: 6.5ms | bwd: 4235.8ms | clip: 0.0ms | optim: 49.5ms
277
+ step 180 | loss: 1.699248 | lr 2.6691e-04 | norm: 0.4708 | dt: 6315.29ms | tok/sec: 83018.9 | mem: 108909MB | Time Left: 00 hrs : 23 mins
278
+ [TIMING] data: 3.4ms | fwd: 2018.1ms | loss: 6.5ms | bwd: 4229.6ms | clip: 0.0ms | optim: 49.7ms
279
+ step 181 | loss: 1.610737 | lr 2.6613e-04 | norm: 0.5130 | dt: 6332.34ms | tok/sec: 82795.2 | mem: 108909MB | Time Left: 00 hrs : 23 mins
280
+ [TIMING] data: 3.4ms | fwd: 2021.4ms | loss: 6.5ms | bwd: 4243.0ms | clip: 0.0ms | optim: 50.1ms
281
+ step 182 | loss: 1.598042 | lr 2.6534e-04 | norm: 0.3652 | dt: 6330.79ms | tok/sec: 82815.6 | mem: 108909MB | Time Left: 00 hrs : 23 mins
282
+ [TIMING] data: 3.6ms | fwd: 2023.0ms | loss: 6.5ms | bwd: 4240.2ms | clip: 0.0ms | optim: 49.5ms
283
+ step 183 | loss: 1.636767 | lr 2.6455e-04 | norm: 0.3973 | dt: 6319.70ms | tok/sec: 82961.0 | mem: 108909MB | Time Left: 00 hrs : 22 mins
284
+ [TIMING] data: 3.5ms | fwd: 2023.7ms | loss: 6.5ms | bwd: 4228.4ms | clip: 0.0ms | optim: 49.6ms
285
+ step 184 | loss: 1.634775 | lr 2.6374e-04 | norm: 0.4649 | dt: 6357.98ms | tok/sec: 82461.5 | mem: 108909MB | Time Left: 00 hrs : 22 mins
286
+ [TIMING] data: 3.6ms | fwd: 2051.7ms | loss: 6.5ms | bwd: 4238.5ms | clip: 0.0ms | optim: 49.7ms
287
+ step 185 | loss: 1.609302 | lr 2.6293e-04 | norm: 0.4189 | dt: 6341.67ms | tok/sec: 82673.4 | mem: 108909MB | Time Left: 00 hrs : 22 mins
288
+ [TIMING] data: 3.5ms | fwd: 2027.3ms | loss: 6.5ms | bwd: 4246.8ms | clip: 0.0ms | optim: 49.5ms
289
+ step 186 | loss: 1.671908 | lr 2.6211e-04 | norm: 0.5121 | dt: 6334.59ms | tok/sec: 82765.8 | mem: 108909MB | Time Left: 00 hrs : 22 mins
290
+ [TIMING] data: 3.6ms | fwd: 2050.0ms | loss: 6.5ms | bwd: 4216.8ms | clip: 0.0ms | optim: 49.6ms
291
+ step 187 | loss: 1.645242 | lr 2.6129e-04 | norm: 0.4458 | dt: 6326.63ms | tok/sec: 82870.1 | mem: 108909MB | Time Left: 00 hrs : 22 mins
292
+ [TIMING] data: 3.6ms | fwd: 2029.0ms | loss: 6.8ms | bwd: 4229.5ms | clip: 0.0ms | optim: 49.8ms
293
+ step 188 | loss: 1.626171 | lr 2.6046e-04 | norm: 0.4034 | dt: 6339.96ms | tok/sec: 82695.8 | mem: 108909MB | Time Left: 00 hrs : 22 mins
294
+ [TIMING] data: 3.5ms | fwd: 2025.2ms | loss: 6.5ms | bwd: 4247.0ms | clip: 0.0ms | optim: 49.6ms
295
+ step 189 | loss: 1.634669 | lr 2.5962e-04 | norm: 0.3758 | dt: 6325.95ms | tok/sec: 82878.9 | mem: 108909MB | Time Left: 00 hrs : 22 mins
296
+ [TIMING] data: 3.5ms | fwd: 2019.9ms | loss: 6.5ms | bwd: 4238.5ms | clip: 0.0ms | optim: 49.5ms
297
+ step 190 | loss: 1.610499 | lr
logs/in2/driver_20260829_1535.log ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [15:35:38] IN2 driver starting. base=/root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt d_min=9835 R=9835
2
+ [15:35:38] generating arm C (train) at --d-min 9835
3
+ [15:35:45] IN2 driver starting. base=/root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt d_min=9835 R=9835
4
+ [15:35:45] generating arm C (train) at --d-min 9835
5
+ wrote 3 shard(s) in in2_data/arm_c 209977344 tokens total
6
+ answer tokens 6561792 (3.12%) synthetic 22146048 (10.55%)
7
+ distance: min 9847 median 10879 max 16348 n=820224
8
+ [15:36:06] generating arm C (val)
9
+ wrote 1 shard(s) in in2_data/arm_c 1048576 tokens total
10
+ answer tokens 32768 (3.12%) synthetic 110592 (10.55%)
11
+ distance: min 9847 median 10857 max 16157 n=4096
12
+ [15:36:06] arm C distance: min 9847 median 10879 max 16348 n=820224
13
+ [15:36:06] arm C distance: min 9847 median 10857 max 16157 n=4096
14
+ [15:36:06] gate (train) on all three arms
15
+ wrote 3 shard(s) in in2_data/arm_c 209977344 tokens total
16
+ answer tokens 6561792 (3.12%) synthetic 22146048 (10.55%)
17
+ distance: min 9847 median 10879 max 16348 n=820224
18
+ [15:36:15] generating arm C (val)
19
+ wrote 1 shard(s) in in2_data/arm_c 1048576 tokens total
20
+ answer tokens 32768 (3.12%) synthetic 110592 (10.55%)
21
+ distance: min 9847 median 10857 max 16157 n=4096
22
+ [15:36:15] arm C distance: min 9847 median 10879 max 16348 n=820224
23
+ [15:36:15] arm C distance: min 9847 median 10857 max 16157 n=4096
24
+ [15:36:15] gate (train) on all three arms
25
+
26
+ === arm a T=16384 examples=12816 K=64 ===
27
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
28
+ ok 2 tokens in [0,50304) (min 0, max 50256)
29
+ ok 3 no answer within the last 4 offsets (0 bad)
30
+ ok 4 planted spans never overlap (0 bad)
31
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
32
+ ok 7 arm-a distance invariant (0 bad)
33
+ ok 6a batch stride 65536 is a multiple of T=16384
34
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
35
+ ok 6c row r starts at token r*T for all B=4 rows
36
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
37
+
38
+ === arm b T=16384 examples=12816 K=64 ===
39
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
40
+ ok 2 tokens in [0,50304) (min 0, max 50256)
41
+ ok 3 no answer within the last 4 offsets (0 bad)
42
+ ok 4 planted spans never overlap (0 bad)
43
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
44
+ ok 7 arm-b distance invariant (0 bad)
45
+ ok 6a batch stride 65536 is a multiple of T=16384
46
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
47
+ ok 6c row r starts at token r*T for all B=4 rows
48
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
49
+
50
+ === arm c T=16384 examples=12816 K=64 ===
51
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
52
+ ok 2 tokens in [0,50304) (min 0, max 50256)
53
+ ok 3 no answer within the last 4 offsets (0 bad)
54
+ ok 4 planted spans never overlap (0 bad)
55
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
56
+ ok 7 arm-c distance invariant (0 bad)
57
+ ok 6a batch stride 65536 is a multiple of T=16384
58
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
59
+ ok 6c row r starts at token r*T for all B=4 rows
60
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
61
+
62
+ === cross-arm ===
63
+ FAIL 8 arms B and C have BIT-IDENTICAL answer_start histograms (16293 vs 6522 distinct offsets)
64
+ ok 8b same seed
65
+ ok 8c same answer-token count
66
+
67
+ 1 FAILURE(S)
68
+ [15:36:32] FATAL: GATE FAILED (train)
69
+ [15:36:32] aborting -- see in2_logs/driver_20260829_1535.log
70
+
71
+ === arm a T=16384 examples=12816 K=64 ===
72
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
73
+ ok 2 tokens in [0,50304) (min 0, max 50256)
74
+ ok 3 no answer within the last 4 offsets (0 bad)
75
+ ok 4 planted spans never overlap (0 bad)
76
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
77
+ ok 7 arm-a distance invariant (0 bad)
78
+ ok 6a batch stride 65536 is a multiple of T=16384
79
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
80
+ ok 6c row r starts at token r*T for all B=4 rows
81
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
82
+
83
+ === arm b T=16384 examples=12816 K=64 ===
84
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
85
+ ok 2 tokens in [0,50304) (min 0, max 50256)
86
+ ok 3 no answer within the last 4 offsets (0 bad)
87
+ ok 4 planted spans never overlap (0 bad)
88
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
89
+ ok 7 arm-b distance invariant (0 bad)
90
+ ok 6a batch stride 65536 is a multiple of T=16384
91
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
92
+ ok 6c row r starts at token r*T for all B=4 rows
93
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
94
+
95
+ === arm c T=16384 examples=12816 K=64 ===
96
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
97
+ ok 2 tokens in [0,50304) (min 0, max 50256)
98
+ ok 3 no answer within the last 4 offsets (0 bad)
99
+ ok 4 planted spans never overlap (0 bad)
100
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
101
+ ok 7 arm-c distance invariant (0 bad)
102
+ ok 6a batch stride 65536 is a multiple of T=16384
103
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
104
+ ok 6c row r starts at token r*T for all B=4 rows
105
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
106
+
107
+ === cross-arm ===
108
+ FAIL 8 arms B and C have BIT-IDENTICAL answer_start histograms (16293 vs 6522 distinct offsets)
109
+ ok 8b same seed
110
+ ok 8c same answer-token count
111
+
112
+ 1 FAILURE(S)
113
+ [15:36:41] FATAL: GATE FAILED (train)
114
+ [15:36:41] aborting -- see in2_logs/driver_20260829_1535.log
logs/in2/driver_20260829_1542.log ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [15:42:31] IN2 driver starting. base=/root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt d_min=64 R=9835
2
+ [15:42:31] gate (train) on all three arms
3
+
4
+ === arm a T=16384 examples=12816 K=64 ===
5
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
6
+ ok 2 tokens in [0,50304) (min 0, max 50256)
7
+ ok 3 no answer within the last 4 offsets (0 bad)
8
+ ok 4 planted spans never overlap (0 bad)
9
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
10
+ ok 7 arm-a distance invariant (0 bad)
11
+ ok 6a batch stride 65536 is a multiple of T=16384
12
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
13
+ ok 6c row r starts at token r*T for all B=4 rows
14
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
15
+
16
+ === arm b T=16384 examples=12816 K=64 ===
17
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
18
+ ok 2 tokens in [0,50304) (min 0, max 50256)
19
+ ok 3 no answer within the last 4 offsets (0 bad)
20
+ ok 4 planted spans never overlap (0 bad)
21
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
22
+ ok 7 arm-b distance invariant (0 bad)
23
+ ok 6a batch stride 65536 is a multiple of T=16384
24
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
25
+ ok 6c row r starts at token r*T for all B=4 rows
26
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
27
+
28
+ === arm c T=16384 examples=12816 K=64 ===
29
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
30
+ ok 2 tokens in [0,50304) (min 0, max 50256)
31
+ ok 3 no answer within the last 4 offsets (0 bad)
32
+ ok 4 planted spans never overlap (0 bad)
33
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
34
+ ok 7 arm-c distance invariant (0 bad)
35
+ ok 6a batch stride 65536 is a multiple of T=16384
36
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
37
+ ok 6c row r starts at token r*T for all B=4 rows
38
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
39
+
40
+ === cross-arm ===
41
+ ok 8 arms B and C have BIT-IDENTICAL answer_start histograms (16293 vs 16293 distinct offsets)
42
+ ok 8b same seed
43
+ ok 8c same answer-token count
44
+
45
+ ALL PASS
46
+ [15:42:57] gate train: ALL PASS
47
+ [15:42:57] gate (val) on all three arms
48
+
49
+ === arm a T=16384 examples=64 K=64 ===
50
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
51
+ ok 2 tokens in [0,50304) (min 0, max 50256)
52
+ ok 3 no answer within the last 4 offsets (0 bad)
53
+ ok 4 planted spans never overlap (0 bad)
54
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
55
+ ok 7 arm-a distance invariant (0 bad)
56
+ ok 6a batch stride 65536 is a multiple of T=16384
57
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
58
+ ok 6c row r starts at token r*T for all B=4 rows
59
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
60
+
61
+ === arm b T=16384 examples=64 K=64 ===
62
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
63
+ ok 2 tokens in [0,50304) (min 0, max 50256)
64
+ ok 3 no answer within the last 4 offsets (0 bad)
65
+ ok 4 planted spans never overlap (0 bad)
66
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
67
+ ok 7 arm-b distance invariant (0 bad)
68
+ ok 6a batch stride 65536 is a multiple of T=16384
69
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
70
+ ok 6c row r starts at token r*T for all B=4 rows
71
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
72
+
73
+ === arm c T=16384 examples=64 K=64 ===
74
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
75
+ ok 2 tokens in [0,50304) (min 0, max 50256)
76
+ ok 3 no answer within the last 4 offsets (0 bad)
77
+ ok 4 planted spans never overlap (0 bad)
78
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
79
+ ok 7 arm-c distance invariant (0 bad)
80
+ ok 6a batch stride 65536 is a multiple of T=16384
81
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
82
+ ok 6c row r starts at token r*T for all B=4 rows
83
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
84
+
85
+ === cross-arm ===
86
+ ok 8 arms B and C have BIT-IDENTICAL answer_start histograms (3593 vs 3593 distinct offsets)
87
+ ok 8b same seed
88
+ ok 8c same answer-token count
89
+
90
+ ALL PASS
91
+ [15:42:57] gate val: ALL PASS
92
+ [15:42:57] training arm a (400 steps x 524288 tokens, T=16384, cp=4, YaRN=4)
logs/in2/driver_20260829_1545.log ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [15:45:12] IN2 driver starting. base=/root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt d_min=64 R=9835
2
+ [15:45:12] gate (train) on all three arms
3
+
4
+ === arm a T=16384 examples=12816 K=64 ===
5
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
6
+ ok 2 tokens in [0,50304) (min 0, max 50256)
7
+ ok 3 no answer within the last 4 offsets (0 bad)
8
+ ok 4 planted spans never overlap (0 bad)
9
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
10
+ ok 7 arm-a distance invariant (0 bad)
11
+ ok 6a batch stride 65536 is a multiple of T=16384
12
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
13
+ ok 6c row r starts at token r*T for all B=4 rows
14
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
15
+
16
+ === arm b T=16384 examples=12816 K=64 ===
17
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
18
+ ok 2 tokens in [0,50304) (min 0, max 50256)
19
+ ok 3 no answer within the last 4 offsets (0 bad)
20
+ ok 4 planted spans never overlap (0 bad)
21
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
22
+ ok 7 arm-b distance invariant (0 bad)
23
+ ok 6a batch stride 65536 is a multiple of T=16384
24
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
25
+ ok 6c row r starts at token r*T for all B=4 rows
26
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
27
+
28
+ === arm c T=16384 examples=12816 K=64 ===
29
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
30
+ ok 2 tokens in [0,50304) (min 0, max 50256)
31
+ ok 3 no answer within the last 4 offsets (0 bad)
32
+ ok 4 planted spans never overlap (0 bad)
33
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
34
+ ok 7 arm-c distance invariant (0 bad)
35
+ ok 6a batch stride 65536 is a multiple of T=16384
36
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
37
+ ok 6c row r starts at token r*T for all B=4 rows
38
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
39
+
40
+ === cross-arm ===
41
+ ok 8 arms B and C have BIT-IDENTICAL answer_start histograms (16293 vs 16293 distinct offsets)
42
+ ok 8b same seed
43
+ ok 8c same answer-token count
44
+
45
+ ALL PASS
46
+ [15:45:38] gate train: ALL PASS
47
+ [15:45:38] gate (val) on all three arms
48
+
49
+ === arm a T=16384 examples=64 K=64 ===
50
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
51
+ ok 2 tokens in [0,50304) (min 0, max 50256)
52
+ ok 3 no answer within the last 4 offsets (0 bad)
53
+ ok 4 planted spans never overlap (0 bad)
54
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
55
+ ok 7 arm-a distance invariant (0 bad)
56
+ ok 6a batch stride 65536 is a multiple of T=16384
57
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
58
+ ok 6c row r starts at token r*T for all B=4 rows
59
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
60
+
61
+ === arm b T=16384 examples=64 K=64 ===
62
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
63
+ ok 2 tokens in [0,50304) (min 0, max 50256)
64
+ ok 3 no answer within the last 4 offsets (0 bad)
65
+ ok 4 planted spans never overlap (0 bad)
66
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
67
+ ok 7 arm-b distance invariant (0 bad)
68
+ ok 6a batch stride 65536 is a multiple of T=16384
69
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
70
+ ok 6c row r starts at token r*T for all B=4 rows
71
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
72
+
73
+ === arm c T=16384 examples=64 K=64 ===
74
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
75
+ ok 2 tokens in [0,50304) (min 0, max 50256)
76
+ ok 3 no answer within the last 4 offsets (0 bad)
77
+ ok 4 planted spans never overlap (0 bad)
78
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
79
+ ok 7 arm-c distance invariant (0 bad)
80
+ ok 6a batch stride 65536 is a multiple of T=16384
81
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
82
+ ok 6c row r starts at token r*T for all B=4 rows
83
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
84
+
85
+ === cross-arm ===
86
+ ok 8 arms B and C have BIT-IDENTICAL answer_start histograms (3593 vs 3593 distinct offsets)
87
+ ok 8b same seed
88
+ ok 8c same answer-token count
89
+
90
+ ALL PASS
91
+ [15:45:38] gate val: ALL PASS
92
+ [15:45:38] training arm a (400 steps x 524288 tokens, T=16384, cp=4, YaRN=4)
93
+ [15:46:34] FATAL: arm a training failed -- see in2_logs/arm_a_train.log
94
+ [15:46:34] aborting -- see in2_logs/driver_20260829_1545.log
logs/in2/driver_20260829_1548.log ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [15:48:58] IN2 driver starting. base=/root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run7_step_750.ckpt d_min=64 R=9835
2
+ [15:48:58] gate (train) on all three arms
3
+
4
+ === arm a T=16384 examples=12816 K=64 ===
5
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
6
+ ok 2 tokens in [0,50304) (min 0, max 50256)
7
+ ok 3 no answer within the last 4 offsets (0 bad)
8
+ ok 4 planted spans never overlap (0 bad)
9
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
10
+ ok 7 arm-a distance invariant (0 bad)
11
+ ok 6a batch stride 65536 is a multiple of T=16384
12
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
13
+ ok 6c row r starts at token r*T for all B=4 rows
14
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
15
+
16
+ === arm b T=16384 examples=12816 K=64 ===
17
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
18
+ ok 2 tokens in [0,50304) (min 0, max 50256)
19
+ ok 3 no answer within the last 4 offsets (0 bad)
20
+ ok 4 planted spans never overlap (0 bad)
21
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
22
+ ok 7 arm-b distance invariant (0 bad)
23
+ ok 6a batch stride 65536 is a multiple of T=16384
24
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
25
+ ok 6c row r starts at token r*T for all B=4 rows
26
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
27
+
28
+ === arm c T=16384 examples=12816 K=64 ===
29
+ ok 1 total size across 3 shard(s) == examples*T*2 (419954688 bytes, got 419954688)
30
+ ok 2 tokens in [0,50304) (min 0, max 50256)
31
+ ok 3 no answer within the last 4 offsets (0 bad)
32
+ ok 4 planted spans never overlap (0 bad)
33
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
34
+ ok 7 arm-c distance invariant (0 bad)
35
+ ok 6a batch stride 65536 is a multiple of T=16384
36
+ ok 6b total 209977344 tokens is a whole number of batches (stride 65536)
37
+ ok 6c row r starts at token r*T for all B=4 rows
38
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
39
+
40
+ === cross-arm ===
41
+ ok 8 arms B and C have BIT-IDENTICAL answer_start histograms (16293 vs 16293 distinct offsets)
42
+ ok 8b same seed
43
+ ok 8c same answer-token count
44
+
45
+ ALL PASS
46
+ [15:49:23] gate train: ALL PASS
47
+ [15:49:23] gate (val) on all three arms
48
+
49
+ === arm a T=16384 examples=64 K=64 ===
50
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
51
+ ok 2 tokens in [0,50304) (min 0, max 50256)
52
+ ok 3 no answer within the last 4 offsets (0 bad)
53
+ ok 4 planted spans never overlap (0 bad)
54
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
55
+ ok 7 arm-a distance invariant (0 bad)
56
+ ok 6a batch stride 65536 is a multiple of T=16384
57
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
58
+ ok 6c row r starts at token r*T for all B=4 rows
59
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
60
+
61
+ === arm b T=16384 examples=64 K=64 ===
62
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
63
+ ok 2 tokens in [0,50304) (min 0, max 50256)
64
+ ok 3 no answer within the last 4 offsets (0 bad)
65
+ ok 4 planted spans never overlap (0 bad)
66
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
67
+ ok 7 arm-b distance invariant (0 bad)
68
+ ok 6a batch stride 65536 is a multiple of T=16384
69
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
70
+ ok 6c row r starts at token r*T for all B=4 rows
71
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
72
+
73
+ === arm c T=16384 examples=64 K=64 ===
74
+ ok 1 total size across 1 shard(s) == examples*T*2 (2097152 bytes, got 2097152)
75
+ ok 2 tokens in [0,50304) (min 0, max 50256)
76
+ ok 3 no answer within the last 4 offsets (0 bad)
77
+ ok 4 planted spans never overlap (0 bad)
78
+ ok 5 RETRIEVAL POSSIBLE: probe==key and answer==value (0 bad)
79
+ ok 7 arm-c distance invariant (0 bad)
80
+ ok 6a batch stride 65536 is a multiple of T=16384
81
+ ok 6b total 1048576 tokens is a whole number of batches (stride 65536)
82
+ ok 6c row r starts at token r*T for all B=4 rows
83
+ ok 6d y[T-1] targets the next example's first token (why PAD_TAIL exists)
84
+
85
+ === cross-arm ===
86
+ ok 8 arms B and C have BIT-IDENTICAL answer_start histograms (3593 vs 3593 distinct offsets)
87
+ ok 8b same seed
88
+ ok 8c same answer-token count
89
+
90
+ ALL PASS
91
+ [15:49:23] gate val: ALL PASS
92
+ [15:49:23] training arm a (400 steps x 524288 tokens, T=16384, cp=4, YaRN=4)
93
+ [16:33:29] arm a done -> run 12, final loss loss: 1.349215
94
+ [16:33:29] training arm b (400 steps x 524288 tokens, T=16384, cp=4, YaRN=4)
95
+ [17:17:52] arm b done -> run 13, final loss loss: 1.451133
96
+ [17:17:52] training arm c (400 steps x 524288 tokens, T=16384, cp=4, YaRN=4)
97
+ [18:01:55] arm c done -> run 14, final loss loss: 1.610499
98
+ [18:01:55] arm C val_loss: first=1.888691 last=1.888691
99
+ [18:01:55] FATAL: LEARNABILITY GATE FAILED: arm C val_loss did not fall (1.888691 -> 1.888691). The run says nothing about IN2.
100
+ [18:01:55] aborting -- see in2_logs/driver_20260829_1548.log