unparallelled commited on
Commit
a1723c4
·
verified ·
1 Parent(s): 3a89964

Upload logs/mixcont/train.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. logs/mixcont/train.log +733 -0
logs/mixcont/train.log ADDED
@@ -0,0 +1,733 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ === Llama Context Parallel Training Script (bluscriptCP) ===
2
+ [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
3
+ [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
4
+ [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
5
+ ================ Configuration (bluscriptCP) ================
6
+ d_model/layers : 1536 / 36
7
+ q/kv heads : 12 / 4 (G=3, hd=128)
8
+ ffn_hidden : 4096 (SwiGLU)
9
+ context_length : 65536
10
+ vocab_size : 50304 tying=0
11
+ B x T : 1 x 65536
12
+ world_size : 8 (dp=1 x cp=8)
13
+ CP_ATTN_MODE : ring (load_balance=HeadTail)
14
+ global_batch : 524288 (grad_accum=8)
15
+ precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
16
+ max_lr/min_lr : 0.0006 / 6e-05
17
+ warmup/steps : 100 / 2100 (WSD: stable=1998, decay tail=2)
18
+ checkpointing : true (ckpt_freq=200, dir=checkpoints_bluscriptcp)
19
+ =============================================================
20
+ Initializing model on CUDA device 0...
21
+ [YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693
22
+ [InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/mix64k_step1200.ckpt (base step 1200, loss 2.81631) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
23
+ Parameters: 1060624896
24
+ Parameters per GPU: 1060624896
25
+ max_steps: 2100
26
+ warmup_steps: 100
27
+ [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
28
+ [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/-
29
+ [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
30
+ [ZeRO-1 v2] params=983236608 padded=983236608 W=8 shard=122904576 | 38 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 1406.53 MB (vs 11252.2 MB unsharded)
31
+ [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
32
+ [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
33
+ [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/-
34
+ [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
35
+ [ZeRO-1 v2] params=77388288 padded=77388288 W=8 shard=9673536 | 6 buckets of 50 MB (1638400 elems/rank/bucket) | optimizer state/GPU: 110.705 MB (vs 885.639 MB unsharded)
36
+ [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
37
+ [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288]
38
+ [ZeRO-1] DataParallel is DISABLED (both would double-reduce)
39
+ [precision] bf16 params=217, all with fp32 master weights
40
+ found 12 shards for split train
41
+ found 1 shards for split val
42
+ Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (run 2)
43
+ [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
44
+ Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 200 steps, keep 2)
45
+ [re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
46
+
47
+ Starting training...
48
+
49
+ [CP ring P2P] OVERLAP: dedicated non-blocking stream
50
+ validation loss: 2.9006
51
+ [CP bwd] overlap=ON
52
+ step 0 | loss: 2.849251 | lr 1.8000e-06 | norm: 0.4355 | dt: 16909.51ms | tok/sec: 31005.5 | mem: 70439MB | Time Left: 09 hrs : 51 mins
53
+ [TIMING] data: 3.6ms | fwd: 3730.6ms | loss: 4.9ms | bwd: 5624.6ms | clip: 0.0ms | optim: 24.3ms
54
+ step 1 | loss: 2.771659 | lr 3.6000e-06 | norm: 0.2168 | dt: 8944.07ms | tok/sec: 58618.5 | mem: 72103MB | Time Left: 05 hrs : 12 mins
55
+ [TIMING] data: 4.2ms | fwd: 3546.0ms | loss: 4.8ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.5ms
56
+ step 2 | loss: 3.129549 | lr 5.4000e-06 | norm: 0.3180 | dt: 8979.37ms | tok/sec: 58388.0 | mem: 72103MB | Time Left: 05 hrs : 13 mins
57
+ [TIMING] data: 3.8ms | fwd: 3529.5ms | loss: 4.8ms | bwd: 5408.4ms | clip: 0.0ms | optim: 23.7ms
58
+ step 3 | loss: 2.757302 | lr 7.2000e-06 | norm: 0.1858 | dt: 7794.04ms | tok/sec: 67267.8 | mem: 72103MB | Time Left: 04 hrs : 32 mins
59
+ [TIMING] data: 4.6ms | fwd: 2399.5ms | loss: 4.8ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.5ms
60
+ step 4 | loss: 3.064898 | lr 9.0000e-06 | norm: 0.2993 | dt: 7782.92ms | tok/sec: 67364.0 | mem: 72103MB | Time Left: 04 hrs : 31 mins
61
+ [TIMING] data: 4.3ms | fwd: 2392.4ms | loss: 4.8ms | bwd: 5348.6ms | clip: 0.0ms | optim: 23.5ms
62
+ step 5 | loss: 2.849279 | lr 1.0800e-05 | norm: 0.1736 | dt: 7793.06ms | tok/sec: 67276.3 | mem: 72103MB | Time Left: 04 hrs : 32 mins
63
+ [TIMING] data: 4.5ms | fwd: 2402.2ms | loss: 4.8ms | bwd: 5348.8ms | clip: 0.0ms | optim: 23.5ms
64
+ step 6 | loss: 2.705969 | lr 1.2600e-05 | norm: 0.1518 | dt: 7795.74ms | tok/sec: 67253.1 | mem: 72103MB | Time Left: 04 hrs : 32 mins
65
+ [TIMING] data: 4.3ms | fwd: 2402.7ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
66
+ step 7 | loss: 2.657195 | lr 1.4400e-05 | norm: 0.1932 | dt: 7804.29ms | tok/sec: 67179.4 | mem: 72103MB | Time Left: 04 hrs : 32 mins
67
+ [TIMING] data: 4.5ms | fwd: 2404.7ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.5ms
68
+ step 8 | loss: 2.967716 | lr 1.6200e-05 | norm: 0.1973 | dt: 7796.25ms | tok/sec: 67248.7 | mem: 72103MB | Time Left: 04 hrs : 31 mins
69
+ [TIMING] data: 4.4ms | fwd: 2401.1ms | loss: 4.8ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
70
+ step 9 | loss: 3.006548 | lr 1.8000e-05 | norm: 0.1757 | dt: 7807.98ms | tok/sec: 67147.7 | mem: 72103MB | Time Left: 04 hrs : 32 mins
71
+ [TIMING] data: 4.5ms | fwd: 2411.8ms | loss: 4.8ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
72
+ step 10 | loss: 2.919900 | lr 1.9800e-05 | norm: 0.1724 | dt: 7800.65ms | tok/sec: 67210.8 | mem: 72103MB | Time Left: 04 hrs : 31 mins
73
+ [TIMING] data: 4.3ms | fwd: 2406.5ms | loss: 4.8ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.6ms
74
+ step 11 | loss: 2.718448 | lr 2.1600e-05 | norm: 0.2161 | dt: 7813.40ms | tok/sec: 67101.1 | mem: 72103MB | Time Left: 04 hrs : 32 mins
75
+ [TIMING] data: 4.5ms | fwd: 2412.1ms | loss: 4.8ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.5ms
76
+ step 12 | loss: 2.775069 | lr 2.3400e-05 | norm: 0.1755 | dt: 7816.31ms | tok/sec: 67076.2 | mem: 72103MB | Time Left: 04 hrs : 32 mins
77
+ [TIMING] data: 4.4ms | fwd: 2419.5ms | loss: 4.8ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.8ms
78
+ step 13 | loss: 2.722095 | lr 2.5200e-05 | norm: 0.1907 | dt: 7816.41ms | tok/sec: 67075.3 | mem: 72103MB | Time Left: 04 hrs : 31 mins
79
+ [TIMING] data: 4.3ms | fwd: 2417.4ms | loss: 4.8ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
80
+ step 14 | loss: 2.614105 | lr 2.7000e-05 | norm: 0.1767 | dt: 7812.97ms | tok/sec: 67104.8 | mem: 72103MB | Time Left: 04 hrs : 31 mins
81
+ [TIMING] data: 4.3ms | fwd: 2410.9ms | loss: 4.8ms | bwd: 5358.4ms | clip: 0.0ms | optim: 23.6ms
82
+ step 15 | loss: 2.822798 | lr 2.8800e-05 | norm: 0.1889 | dt: 7814.98ms | tok/sec: 67087.6 | mem: 72103MB | Time Left: 04 hrs : 31 mins
83
+ [TIMING] data: 4.4ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
84
+ step 16 | loss: 2.910059 | lr 3.0600e-05 | norm: 0.2202 | dt: 7827.24ms | tok/sec: 66982.5 | mem: 72103MB | Time Left: 04 hrs : 31 mins
85
+ [TIMING] data: 4.6ms | fwd: 2422.1ms | loss: 4.8ms | bwd: 5362.1ms | clip: 0.0ms | optim: 23.7ms
86
+ step 17 | loss: 2.705841 | lr 3.2400e-05 | norm: 0.1678 | dt: 7827.25ms | tok/sec: 66982.4 | mem: 72103MB | Time Left: 04 hrs : 31 mins
87
+ [TIMING] data: 4.1ms | fwd: 2420.7ms | loss: 4.8ms | bwd: 5363.1ms | clip: 0.0ms | optim: 23.5ms
88
+ step 18 | loss: 2.806966 | lr 3.4200e-05 | norm: 0.2113 | dt: 7815.94ms | tok/sec: 67079.3 | mem: 72103MB | Time Left: 04 hrs : 31 mins
89
+ [TIMING] data: 4.0ms | fwd: 2412.7ms | loss: 4.8ms | bwd: 5360.7ms | clip: 0.0ms | optim: 23.6ms
90
+ step 19 | loss: 2.786724 | lr 3.6000e-05 | norm: 0.1685 | dt: 7820.28ms | tok/sec: 67042.1 | mem: 72103MB | Time Left: 04 hrs : 31 mins
91
+ [TIMING] data: 4.5ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5360.5ms | clip: 0.0ms | optim: 23.8ms
92
+ step 20 | loss: 2.494313 | lr 3.7800e-05 | norm: 0.1415 | dt: 7827.51ms | tok/sec: 66980.2 | mem: 72103MB | Time Left: 04 hrs : 31 mins
93
+ [TIMING] data: 4.5ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5364.1ms | clip: 0.0ms | optim: 23.6ms
94
+ step 21 | loss: 2.708891 | lr 3.9600e-05 | norm: 0.1796 | dt: 7828.57ms | tok/sec: 66971.1 | mem: 72103MB | Time Left: 04 hrs : 31 mins
95
+ [TIMING] data: 4.5ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5367.2ms | clip: 0.0ms | optim: 23.6ms
96
+ step 22 | loss: 2.863696 | lr 4.1400e-05 | norm: 0.2196 | dt: 7829.81ms | tok/sec: 66960.5 | mem: 72103MB | Time Left: 04 hrs : 31 mins
97
+ [TIMING] data: 4.5ms | fwd: 2421.7ms | loss: 4.9ms | bwd: 5363.9ms | clip: 0.0ms | optim: 23.6ms
98
+ step 23 | loss: 2.526138 | lr 4.3200e-05 | norm: 0.1477 | dt: 7822.01ms | tok/sec: 67027.2 | mem: 72103MB | Time Left: 04 hrs : 30 mins
99
+ [TIMING] data: 3.9ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
100
+ step 24 | loss: 2.931958 | lr 4.5000e-05 | norm: 0.1709 | dt: 7826.38ms | tok/sec: 66989.9 | mem: 72103MB | Time Left: 04 hrs : 30 mins
101
+ [TIMING] data: 4.2ms | fwd: 2416.1ms | loss: 4.9ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
102
+ step 25 | loss: 2.684256 | lr 4.6800e-05 | norm: 0.1464 | dt: 7827.49ms | tok/sec: 66980.4 | mem: 72103MB | Time Left: 04 hrs : 30 mins
103
+ [TIMING] data: 4.2ms | fwd: 2418.6ms | loss: 4.8ms | bwd: 5366.6ms | clip: 0.0ms | optim: 23.5ms
104
+ step 26 | loss: 2.781745 | lr 4.8600e-05 | norm: 0.1492 | dt: 7825.23ms | tok/sec: 66999.7 | mem: 72103MB | Time Left: 04 hrs : 30 mins
105
+ [TIMING] data: 4.4ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5361.1ms | clip: 0.0ms | optim: 23.6ms
106
+ step 27 | loss: 2.540646 | lr 5.0400e-05 | norm: 0.1553 | dt: 7818.08ms | tok/sec: 67060.9 | mem: 72103MB | Time Left: 04 hrs : 30 mins
107
+ [TIMING] data: 4.4ms | fwd: 2412.4ms | loss: 4.8ms | bwd: 5362.4ms | clip: 0.0ms | optim: 23.6ms
108
+ step 28 | loss: 2.775252 | lr 5.2200e-05 | norm: 0.2817 | dt: 7846.10ms | tok/sec: 66821.4 | mem: 72103MB | Time Left: 04 hrs : 30 mins
109
+ [TIMING] data: 4.5ms | fwd: 2427.4ms | loss: 4.9ms | bwd: 5373.9ms | clip: 0.0ms | optim: 23.5ms
110
+ step 29 | loss: 2.888937 | lr 5.4000e-05 | norm: 0.1731 | dt: 7821.65ms | tok/sec: 67030.3 | mem: 72103MB | Time Left: 04 hrs : 29 mins
111
+ [TIMING] data: 4.4ms | fwd: 2412.0ms | loss: 4.9ms | bwd: 5367.7ms | clip: 0.0ms | optim: 23.6ms
112
+ step 30 | loss: 2.655073 | lr 5.5800e-05 | norm: 0.1691 | dt: 7820.50ms | tok/sec: 67040.2 | mem: 72103MB | Time Left: 04 hrs : 29 mins
113
+ [TIMING] data: 4.4ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5365.3ms | clip: 0.0ms | optim: 23.6ms
114
+ step 31 | loss: 2.852468 | lr 5.7600e-05 | norm: 0.1892 | dt: 7830.83ms | tok/sec: 66951.8 | mem: 72103MB | Time Left: 04 hrs : 30 mins
115
+ [TIMING] data: 4.5ms | fwd: 2418.9ms | loss: 4.9ms | bwd: 5368.1ms | clip: 0.0ms | optim: 23.7ms
116
+ step 32 | loss: 2.525090 | lr 5.9400e-05 | norm: 0.1397 | dt: 7827.47ms | tok/sec: 66980.5 | mem: 72103MB | Time Left: 04 hrs : 29 mins
117
+ [TIMING] data: 4.3ms | fwd: 2420.0ms | loss: 4.8ms | bwd: 5365.3ms | clip: 0.0ms | optim: 23.6ms
118
+ step 33 | loss: 2.819749 | lr 6.1200e-05 | norm: 0.1496 | dt: 7833.34ms | tok/sec: 66930.4 | mem: 72103MB | Time Left: 04 hrs : 29 mins
119
+ [TIMING] data: 4.5ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.6ms
120
+ step 34 | loss: 2.692123 | lr 6.3000e-05 | norm: 0.1606 | dt: 7827.90ms | tok/sec: 66976.8 | mem: 72103MB | Time Left: 04 hrs : 29 mins
121
+ [TIMING] data: 4.5ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.6ms
122
+ step 35 | loss: 2.642615 | lr 6.4800e-05 | norm: 0.1714 | dt: 7828.29ms | tok/sec: 66973.5 | mem: 72103MB | Time Left: 04 hrs : 29 mins
123
+ [TIMING] data: 4.6ms | fwd: 2412.6ms | loss: 4.8ms | bwd: 5372.8ms | clip: 0.0ms | optim: 23.6ms
124
+ step 36 | loss: 2.612672 | lr 6.6600e-05 | norm: 0.1762 | dt: 7832.23ms | tok/sec: 66939.8 | mem: 72103MB | Time Left: 04 hrs : 29 mins
125
+ [TIMING] data: 4.2ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
126
+ step 37 | loss: 2.965949 | lr 6.8400e-05 | norm: 0.1934 | dt: 7827.61ms | tok/sec: 66979.3 | mem: 72103MB | Time Left: 04 hrs : 29 mins
127
+ [TIMING] data: 4.6ms | fwd: 2419.2ms | loss: 4.8ms | bwd: 5365.4ms | clip: 0.0ms | optim: 23.7ms
128
+ step 38 | loss: 2.560090 | lr 7.0200e-05 | norm: 0.1511 | dt: 7830.21ms | tok/sec: 66957.1 | mem: 72103MB | Time Left: 04 hrs : 29 mins
129
+ [TIMING] data: 4.4ms | fwd: 2418.1ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
130
+ step 39 | loss: 2.770146 | lr 7.2000e-05 | norm: 0.1514 | dt: 7834.61ms | tok/sec: 66919.4 | mem: 72103MB | Time Left: 04 hrs : 29 mins
131
+ [TIMING] data: 4.3ms | fwd: 2420.4ms | loss: 4.9ms | bwd: 5371.7ms | clip: 0.0ms | optim: 23.6ms
132
+ step 40 | loss: 2.460374 | lr 7.3800e-05 | norm: 0.3389 | dt: 7830.25ms | tok/sec: 66956.8 | mem: 72103MB | Time Left: 04 hrs : 28 mins
133
+ [TIMING] data: 4.5ms | fwd: 2421.1ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.6ms
134
+ step 41 | loss: 2.800878 | lr 7.5600e-05 | norm: 0.1722 | dt: 7834.25ms | tok/sec: 66922.6 | mem: 72103MB | Time Left: 04 hrs : 28 mins
135
+ [TIMING] data: 4.1ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5375.6ms | clip: 0.0ms | optim: 23.8ms
136
+ step 42 | loss: 2.536062 | lr 7.7400e-05 | norm: 0.1610 | dt: 7829.64ms | tok/sec: 66961.9 | mem: 72103MB | Time Left: 04 hrs : 28 mins
137
+ [TIMING] data: 4.3ms | fwd: 2420.6ms | loss: 4.8ms | bwd: 5366.5ms | clip: 0.0ms | optim: 23.7ms
138
+ step 43 | loss: 2.705255 | lr 7.9200e-05 | norm: 0.2146 | dt: 7833.39ms | tok/sec: 66929.9 | mem: 72103MB | Time Left: 04 hrs : 28 mins
139
+ [TIMING] data: 4.4ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5375.6ms | clip: 0.0ms | optim: 23.6ms
140
+ step 44 | loss: 2.720131 | lr 8.1000e-05 | norm: 0.1717 | dt: 7827.08ms | tok/sec: 66983.9 | mem: 72103MB | Time Left: 04 hrs : 28 mins
141
+ [TIMING] data: 4.2ms | fwd: 2413.7ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
142
+ step 45 | loss: 2.595790 | lr 8.2800e-05 | norm: 0.1594 | dt: 7826.71ms | tok/sec: 66987.0 | mem: 72103MB | Time Left: 04 hrs : 28 mins
143
+ [TIMING] data: 4.1ms | fwd: 2414.8ms | loss: 4.9ms | bwd: 5369.0ms | clip: 0.0ms | optim: 24.2ms
144
+ step 46 | loss: 2.483078 | lr 8.4600e-05 | norm: 0.2118 | dt: 7827.47ms | tok/sec: 66980.6 | mem: 72103MB | Time Left: 04 hrs : 27 mins
145
+ [TIMING] data: 4.2ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5376.8ms | clip: 0.0ms | optim: 23.7ms
146
+ step 47 | loss: 2.459733 | lr 8.6400e-05 | norm: 0.1622 | dt: 7825.51ms | tok/sec: 66997.3 | mem: 72103MB | Time Left: 04 hrs : 27 mins
147
+ [TIMING] data: 4.4ms | fwd: 2409.4ms | loss: 4.8ms | bwd: 5373.7ms | clip: 0.0ms | optim: 23.7ms
148
+ step 48 | loss: 2.847660 | lr 8.8200e-05 | norm: 0.1493 | dt: 7827.34ms | tok/sec: 66981.7 | mem: 72103MB | Time Left: 04 hrs : 27 mins
149
+ [TIMING] data: 4.3ms | fwd: 2413.1ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.6ms
150
+ step 49 | loss: 2.732789 | lr 9.0000e-05 | norm: 0.1771 | dt: 7830.92ms | tok/sec: 66951.0 | mem: 72103MB | Time Left: 04 hrs : 27 mins
151
+ [TIMING] data: 4.2ms | fwd: 2418.7ms | loss: 4.8ms | bwd: 5369.2ms | clip: 0.0ms | optim: 23.8ms
152
+ step 50 | loss: 2.706898 | lr 9.1800e-05 | norm: 0.1449 | dt: 7836.75ms | tok/sec: 66901.2 | mem: 72103MB | Time Left: 04 hrs : 27 mins
153
+ [TIMING] data: 4.3ms | fwd: 2419.0ms | loss: 4.8ms | bwd: 5374.4ms | clip: 0.0ms | optim: 23.6ms
154
+ step 51 | loss: 2.795459 | lr 9.3600e-05 | norm: 0.1691 | dt: 7828.69ms | tok/sec: 66970.1 | mem: 72103MB | Time Left: 04 hrs : 27 mins
155
+ [TIMING] data: 4.5ms | fwd: 2419.7ms | loss: 4.8ms | bwd: 5366.6ms | clip: 0.0ms | optim: 23.6ms
156
+ step 52 | loss: 2.742548 | lr 9.5400e-05 | norm: 0.1969 | dt: 7839.53ms | tok/sec: 66877.5 | mem: 72103MB | Time Left: 04 hrs : 27 mins
157
+ [TIMING] data: 3.9ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5376.6ms | clip: 0.0ms | optim: 23.6ms
158
+ step 53 | loss: 2.690454 | lr 9.7200e-05 | norm: 0.1784 | dt: 7834.65ms | tok/sec: 66919.1 | mem: 72103MB | Time Left: 04 hrs : 27 mins
159
+ [TIMING] data: 4.1ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5372.4ms | clip: 0.0ms | optim: 23.6ms
160
+ step 54 | loss: 2.766409 | lr 9.9000e-05 | norm: 0.1395 | dt: 7827.36ms | tok/sec: 66981.4 | mem: 72103MB | Time Left: 04 hrs : 26 mins
161
+ [TIMING] data: 4.4ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.8ms
162
+ step 55 | loss: 2.815801 | lr 1.0080e-04 | norm: 0.1430 | dt: 7830.62ms | tok/sec: 66953.6 | mem: 72103MB | Time Left: 04 hrs : 26 mins
163
+ [TIMING] data: 4.4ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.6ms
164
+ step 56 | loss: 2.834039 | lr 1.0260e-04 | norm: 0.1686 | dt: 7831.58ms | tok/sec: 66945.3 | mem: 72103MB | Time Left: 04 hrs : 26 mins
165
+ [TIMING] data: 4.5ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.7ms
166
+ step 57 | loss: 2.557956 | lr 1.0440e-04 | norm: 0.2808 | dt: 7836.55ms | tok/sec: 66902.9 | mem: 72103MB | Time Left: 04 hrs : 26 mins
167
+ [TIMING] data: 4.4ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5374.3ms | clip: 0.0ms | optim: 23.6ms
168
+ step 58 | loss: 2.871943 | lr 1.0620e-04 | norm: 0.1714 | dt: 7833.31ms | tok/sec: 66930.6 | mem: 72103MB | Time Left: 04 hrs : 26 mins
169
+ [TIMING] data: 4.3ms | fwd: 2419.4ms | loss: 4.8ms | bwd: 5371.1ms | clip: 0.0ms | optim: 23.6ms
170
+ step 59 | loss: 2.658476 | lr 1.0800e-04 | norm: 0.1638 | dt: 7839.37ms | tok/sec: 66878.9 | mem: 72103MB | Time Left: 04 hrs : 26 mins
171
+ [TIMING] data: 4.5ms | fwd: 2421.6ms | loss: 4.8ms | bwd: 5373.9ms | clip: 0.0ms | optim: 23.8ms
172
+ step 60 | loss: 2.835381 | lr 1.0980e-04 | norm: 0.1557 | dt: 7827.91ms | tok/sec: 66976.7 | mem: 72103MB | Time Left: 04 hrs : 26 mins
173
+ [TIMING] data: 4.3ms | fwd: 2414.9ms | loss: 4.9ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.6ms
174
+ step 61 | loss: 2.534974 | lr 1.1160e-04 | norm: 0.1439 | dt: 7830.63ms | tok/sec: 66953.5 | mem: 72103MB | Time Left: 04 hrs : 26 mins
175
+ [TIMING] data: 4.2ms | fwd: 2417.4ms | loss: 4.8ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
176
+ step 62 | loss: 2.676892 | lr 1.1340e-04 | norm: 0.1422 | dt: 7838.12ms | tok/sec: 66889.5 | mem: 72103MB | Time Left: 04 hrs : 26 mins
177
+ [TIMING] data: 4.3ms | fwd: 2419.8ms | loss: 4.8ms | bwd: 5375.8ms | clip: 0.0ms | optim: 23.7ms
178
+ step 63 | loss: 2.652859 | lr 1.1520e-04 | norm: 0.1431 | dt: 7831.59ms | tok/sec: 66945.3 | mem: 72103MB | Time Left: 04 hrs : 25 mins
179
+ [TIMING] data: 4.3ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
180
+ step 64 | loss: 2.316111 | lr 1.1700e-04 | norm: 0.1357 | dt: 7846.61ms | tok/sec: 66817.1 | mem: 72103MB | Time Left: 04 hrs : 26 mins
181
+ [TIMING] data: 4.6ms | fwd: 2433.2ms | loss: 4.9ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.7ms
182
+ step 65 | loss: 2.710837 | lr 1.1880e-04 | norm: 0.1608 | dt: 7826.37ms | tok/sec: 66990.0 | mem: 72103MB | Time Left: 04 hrs : 25 mins
183
+ [TIMING] data: 4.4ms | fwd: 2410.7ms | loss: 4.8ms | bwd: 5373.9ms | clip: 0.0ms | optim: 23.6ms
184
+ step 66 | loss: 2.764815 | lr 1.2060e-04 | norm: 0.1603 | dt: 7831.60ms | tok/sec: 66945.2 | mem: 72103MB | Time Left: 04 hrs : 25 mins
185
+ [TIMING] data: 4.4ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5373.3ms | clip: 0.0ms | optim: 23.7ms
186
+ step 67 | loss: 2.645531 | lr 1.2240e-04 | norm: 0.1757 | dt: 7837.03ms | tok/sec: 66898.8 | mem: 72103MB | Time Left: 04 hrs : 25 mins
187
+ [TIMING] data: 4.1ms | fwd: 2422.6ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.5ms
188
+ step 68 | loss: 2.767838 | lr 1.2420e-04 | norm: 0.1765 | dt: 7837.10ms | tok/sec: 66898.2 | mem: 72103MB | Time Left: 04 hrs : 25 mins
189
+ [TIMING] data: 4.2ms | fwd: 2425.2ms | loss: 4.9ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.8ms
190
+ step 69 | loss: 2.503338 | lr 1.2600e-04 | norm: 0.1771 | dt: 7834.52ms | tok/sec: 66920.3 | mem: 72103MB | Time Left: 04 hrs : 25 mins
191
+ [TIMING] data: 4.4ms | fwd: 2420.1ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.6ms
192
+ step 70 | loss: 2.709606 | lr 1.2780e-04 | norm: 0.1686 | dt: 7832.23ms | tok/sec: 66939.8 | mem: 72103MB | Time Left: 04 hrs : 24 mins
193
+ [TIMING] data: 4.3ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.6ms
194
+ step 71 | loss: 2.525243 | lr 1.2960e-04 | norm: 0.1739 | dt: 7831.46ms | tok/sec: 66946.4 | mem: 72103MB | Time Left: 04 hrs : 24 mins
195
+ [TIMING] data: 4.5ms | fwd: 2414.2ms | loss: 4.8ms | bwd: 5374.9ms | clip: 0.0ms | optim: 23.7ms
196
+ step 72 | loss: 2.312968 | lr 1.3140e-04 | norm: 0.1493 | dt: 7832.00ms | tok/sec: 66941.8 | mem: 72103MB | Time Left: 04 hrs : 24 mins
197
+ [TIMING] data: 4.0ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5372.3ms | clip: 0.0ms | optim: 23.6ms
198
+ step 73 | loss: 2.664239 | lr 1.3320e-04 | norm: 0.1723 | dt: 7835.30ms | tok/sec: 66913.6 | mem: 72103MB | Time Left: 04 hrs : 24 mins
199
+ [TIMING] data: 4.3ms | fwd: 2419.5ms | loss: 4.8ms | bwd: 5372.0ms | clip: 0.0ms | optim: 23.6ms
200
+ step 74 | loss: 2.733030 | lr 1.3500e-04 | norm: 0.2010 | dt: 7818.45ms | tok/sec: 67057.8 | mem: 72103MB | Time Left: 04 hrs : 24 mins
201
+ [TIMING] data: 4.4ms | fwd: 2409.2ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.6ms
202
+ step 75 | loss: 2.704966 | lr 1.3680e-04 | norm: 0.1679 | dt: 7835.19ms | tok/sec: 66914.5 | mem: 72103MB | Time Left: 04 hrs : 24 mins
203
+ [TIMING] data: 4.2ms | fwd: 2421.1ms | loss: 4.9ms | bwd: 5371.0ms | clip: 0.0ms | optim: 23.7ms
204
+ step 76 | loss: 2.793722 | lr 1.3860e-04 | norm: 0.1678 | dt: 7830.99ms | tok/sec: 66950.4 | mem: 72103MB | Time Left: 04 hrs : 24 mins
205
+ [TIMING] data: 4.0ms | fwd: 2418.2ms | loss: 4.8ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.7ms
206
+ step 77 | loss: 2.929346 | lr 1.4040e-04 | norm: 0.1572 | dt: 7838.49ms | tok/sec: 66886.4 | mem: 72103MB | Time Left: 04 hrs : 24 mins
207
+ [TIMING] data: 4.2ms | fwd: 2422.8ms | loss: 4.8ms | bwd: 5373.0ms | clip: 0.0ms | optim: 23.6ms
208
+ step 78 | loss: 2.679340 | lr 1.4220e-04 | norm: 0.1998 | dt: 7842.84ms | tok/sec: 66849.3 | mem: 72103MB | Time Left: 04 hrs : 24 mins
209
+ [TIMING] data: 4.3ms | fwd: 2429.8ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
210
+ step 79 | loss: 2.682140 | lr 1.4400e-04 | norm: 0.1545 | dt: 7835.05ms | tok/sec: 66915.7 | mem: 72103MB | Time Left: 04 hrs : 23 mins
211
+ [TIMING] data: 4.3ms | fwd: 2415.7ms | loss: 4.9ms | bwd: 5377.3ms | clip: 0.0ms | optim: 23.6ms
212
+ step 80 | loss: 2.735668 | lr 1.4580e-04 | norm: 0.1551 | dt: 7844.24ms | tok/sec: 66837.3 | mem: 72103MB | Time Left: 04 hrs : 24 mins
213
+ [TIMING] data: 4.2ms | fwd: 2429.1ms | loss: 4.8ms | bwd: 5371.7ms | clip: 0.0ms | optim: 23.5ms
214
+ step 81 | loss: 2.668484 | lr 1.4760e-04 | norm: 0.1800 | dt: 7842.69ms | tok/sec: 66850.6 | mem: 72103MB | Time Left: 04 hrs : 23 mins
215
+ [TIMING] data: 4.5ms | fwd: 2425.2ms | loss: 4.9ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
216
+ step 82 | loss: 2.951952 | lr 1.4940e-04 | norm: 0.1710 | dt: 7830.36ms | tok/sec: 66955.8 | mem: 72103MB | Time Left: 04 hrs : 23 mins
217
+ [TIMING] data: 4.4ms | fwd: 2415.9ms | loss: 4.8ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.7ms
218
+ step 83 | loss: 2.750396 | lr 1.5120e-04 | norm: 0.1845 | dt: 7829.16ms | tok/sec: 66966.1 | mem: 72103MB | Time Left: 04 hrs : 23 mins
219
+ [TIMING] data: 4.3ms | fwd: 2415.9ms | loss: 4.9ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
220
+ step 84 | loss: 2.647009 | lr 1.5300e-04 | norm: 0.1744 | dt: 7831.67ms | tok/sec: 66944.6 | mem: 72103MB | Time Left: 04 hrs : 23 mins
221
+ [TIMING] data: 4.2ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5371.2ms | clip: 0.0ms | optim: 23.7ms
222
+ step 85 | loss: 2.710377 | lr 1.5480e-04 | norm: 0.1629 | dt: 7829.49ms | tok/sec: 66963.3 | mem: 72103MB | Time Left: 04 hrs : 22 mins
223
+ [TIMING] data: 4.1ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.6ms
224
+ step 86 | loss: 2.821142 | lr 1.5660e-04 | norm: 0.2439 | dt: 7835.38ms | tok/sec: 66912.9 | mem: 72103MB | Time Left: 04 hrs : 23 mins
225
+ [TIMING] data: 4.3ms | fwd: 2418.1ms | loss: 4.8ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.6ms
226
+ step 87 | loss: 2.683125 | lr 1.5840e-04 | norm: 0.1564 | dt: 7835.41ms | tok/sec: 66912.6 | mem: 72103MB | Time Left: 04 hrs : 22 mins
227
+ [TIMING] data: 4.4ms | fwd: 2420.5ms | loss: 4.8ms | bwd: 5372.4ms | clip: 0.0ms | optim: 23.6ms
228
+ step 88 | loss: 2.778752 | lr 1.6020e-04 | norm: 0.2029 | dt: 7840.16ms | tok/sec: 66872.1 | mem: 72103MB | Time Left: 04 hrs : 22 mins
229
+ [TIMING] data: 4.3ms | fwd: 2420.6ms | loss: 4.8ms | bwd: 5376.5ms | clip: 0.0ms | optim: 23.7ms
230
+ step 89 | loss: 2.816476 | lr 1.6200e-04 | norm: 0.1741 | dt: 7832.22ms | tok/sec: 66939.9 | mem: 72103MB | Time Left: 04 hrs : 22 mins
231
+ [TIMING] data: 4.4ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5375.1ms | clip: 0.0ms | optim: 23.7ms
232
+ step 90 | loss: 2.832433 | lr 1.6380e-04 | norm: 0.1604 | dt: 7842.76ms | tok/sec: 66849.9 | mem: 72103MB | Time Left: 04 hrs : 22 mins
233
+ [TIMING] data: 3.6ms | fwd: 2428.0ms | loss: 4.8ms | bwd: 5369.9ms | clip: 0.0ms | optim: 23.8ms
234
+ step 91 | loss: 2.820065 | lr 1.6560e-04 | norm: 0.1708 | dt: 7836.67ms | tok/sec: 66901.9 | mem: 72103MB | Time Left: 04 hrs : 22 mins
235
+ [TIMING] data: 4.0ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.7ms
236
+ step 92 | loss: 2.862764 | lr 1.6740e-04 | norm: 0.2114 | dt: 7835.34ms | tok/sec: 66913.3 | mem: 72103MB | Time Left: 04 hrs : 22 mins
237
+ [TIMING] data: 4.5ms | fwd: 2423.0ms | loss: 4.9ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.6ms
238
+ step 93 | loss: 2.749069 | lr 1.6920e-04 | norm: 0.1888 | dt: 7831.75ms | tok/sec: 66943.9 | mem: 72103MB | Time Left: 04 hrs : 21 mins
239
+ [TIMING] data: 4.4ms | fwd: 2421.2ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.7ms
240
+ step 94 | loss: 2.762149 | lr 1.7100e-04 | norm: 0.2443 | dt: 7834.51ms | tok/sec: 66920.3 | mem: 72103MB | Time Left: 04 hrs : 21 mins
241
+ [TIMING] data: 4.4ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5374.4ms | clip: 0.0ms | optim: 23.7ms
242
+ step 95 | loss: 2.682379 | lr 1.7280e-04 | norm: 0.2071 | dt: 7827.43ms | tok/sec: 66980.9 | mem: 72103MB | Time Left: 04 hrs : 21 mins
243
+ [TIMING] data: 4.1ms | fwd: 2415.4ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.5ms
244
+ step 96 | loss: 2.707476 | lr 1.7460e-04 | norm: 0.1956 | dt: 7830.37ms | tok/sec: 66955.7 | mem: 72103MB | Time Left: 04 hrs : 21 mins
245
+ [TIMING] data: 4.2ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5375.9ms | clip: 0.0ms | optim: 23.6ms
246
+ step 97 | loss: 2.821827 | lr 1.7640e-04 | norm: 0.2093 | dt: 7839.63ms | tok/sec: 66876.6 | mem: 72103MB | Time Left: 04 hrs : 21 mins
247
+ [TIMING] data: 4.2ms | fwd: 2420.5ms | loss: 4.8ms | bwd: 5376.0ms | clip: 0.0ms | optim: 23.6ms
248
+ step 98 | loss: 2.357723 | lr 1.7820e-04 | norm: 0.1789 | dt: 7833.07ms | tok/sec: 66932.6 | mem: 72103MB | Time Left: 04 hrs : 21 mins
249
+ [TIMING] data: 4.1ms | fwd: 2416.1ms | loss: 4.9ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
250
+ step 99 | loss: 2.664635 | lr 1.8000e-04 | norm: 0.1818 | dt: 7827.31ms | tok/sec: 66981.9 | mem: 72103MB | Time Left: 04 hrs : 21 mins
251
+ [TIMING] data: 4.1ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.8ms
252
+ step 100 | loss: 2.698034 | lr 1.8000e-04 | norm: 0.1752 | dt: 7831.88ms | tok/sec: 66942.8 | mem: 72103MB | Time Left: 04 hrs : 21 mins
253
+ [TIMING] data: 4.0ms | fwd: 2415.3ms | loss: 4.9ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.7ms
254
+ step 101 | loss: 2.768258 | lr 1.8000e-04 | norm: 0.1847 | dt: 7838.35ms | tok/sec: 66887.6 | mem: 72103MB | Time Left: 04 hrs : 21 mins
255
+ [TIMING] data: 4.2ms | fwd: 2421.0ms | loss: 4.8ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.6ms
256
+ step 102 | loss: 2.793727 | lr 1.8000e-04 | norm: 0.1797 | dt: 7826.91ms | tok/sec: 66985.3 | mem: 72103MB | Time Left: 04 hrs : 20 mins
257
+ [TIMING] data: 4.3ms | fwd: 2417.6ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.6ms
258
+ step 103 | loss: 2.776408 | lr 1.8000e-04 | norm: 0.1732 | dt: 7834.20ms | tok/sec: 66923.0 | mem: 72103MB | Time Left: 04 hrs : 20 mins
259
+ [TIMING] data: 4.4ms | fwd: 2417.4ms | loss: 4.9ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.8ms
260
+ step 104 | loss: 2.497193 | lr 1.8000e-04 | norm: 0.1782 | dt: 7836.05ms | tok/sec: 66907.2 | mem: 72103MB | Time Left: 04 hrs : 20 mins
261
+ [TIMING] data: 4.4ms | fwd: 2419.8ms | loss: 4.8ms | bwd: 5373.6ms | clip: 0.0ms | optim: 23.6ms
262
+ step 105 | loss: 3.192918 | lr 1.8000e-04 | norm: 0.1940 | dt: 7835.79ms | tok/sec: 66909.4 | mem: 72103MB | Time Left: 04 hrs : 20 mins
263
+ [TIMING] data: 4.5ms | fwd: 2409.9ms | loss: 4.8ms | bwd: 5383.0ms | clip: 0.0ms | optim: 23.7ms
264
+ step 106 | loss: 2.820851 | lr 1.8000e-04 | norm: 0.3302 | dt: 7831.24ms | tok/sec: 66948.3 | mem: 72103MB | Time Left: 04 hrs : 20 mins
265
+ [TIMING] data: 4.5ms | fwd: 2414.7ms | loss: 4.8ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.6ms
266
+ step 107 | loss: 2.732643 | lr 1.8000e-04 | norm: 0.2064 | dt: 7836.58ms | tok/sec: 66902.7 | mem: 72103MB | Time Left: 04 hrs : 20 mins
267
+ [TIMING] data: 4.4ms | fwd: 2418.6ms | loss: 4.8ms | bwd: 5375.2ms | clip: 0.0ms | optim: 23.6ms
268
+ step 108 | loss: 2.585369 | lr 1.8000e-04 | norm: 0.1786 | dt: 7836.93ms | tok/sec: 66899.7 | mem: 72103MB | Time Left: 04 hrs : 20 mins
269
+ [TIMING] data: 4.0ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5376.4ms | clip: 0.0ms | optim: 23.6ms
270
+ step 109 | loss: 2.962152 | lr 1.8000e-04 | norm: 0.1899 | dt: 7836.19ms | tok/sec: 66906.0 | mem: 72103MB | Time Left: 04 hrs : 20 mins
271
+ [TIMING] data: 4.1ms | fwd: 2420.2ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
272
+ step 110 | loss: 2.441406 | lr 1.8000e-04 | norm: 0.2773 | dt: 7832.62ms | tok/sec: 66936.5 | mem: 72103MB | Time Left: 04 hrs : 19 mins
273
+ [TIMING] data: 4.3ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5374.4ms | clip: 0.0ms | optim: 23.6ms
274
+ step 111 | loss: 2.954189 | lr 1.8000e-04 | norm: 0.1995 | dt: 7831.38ms | tok/sec: 66947.1 | mem: 72103MB | Time Left: 04 hrs : 19 mins
275
+ [TIMING] data: 4.1ms | fwd: 2419.0ms | loss: 4.9ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
276
+ step 112 | loss: 2.697222 | lr 1.8000e-04 | norm: 0.3689 | dt: 7833.22ms | tok/sec: 66931.3 | mem: 72103MB | Time Left: 04 hrs : 19 mins
277
+ [TIMING] data: 4.2ms | fwd: 2423.4ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.7ms
278
+ step 113 | loss: 2.739350 | lr 1.8000e-04 | norm: 0.2062 | dt: 7838.86ms | tok/sec: 66883.2 | mem: 72103MB | Time Left: 04 hrs : 19 mins
279
+ [TIMING] data: 4.0ms | fwd: 2425.8ms | loss: 4.9ms | bwd: 5370.1ms | clip: 0.0ms | optim: 23.7ms
280
+ step 114 | loss: 2.749949 | lr 1.8000e-04 | norm: 0.1898 | dt: 7826.93ms | tok/sec: 66985.1 | mem: 72103MB | Time Left: 04 hrs : 19 mins
281
+ [TIMING] data: 4.0ms | fwd: 2411.7ms | loss: 4.8ms | bwd: 5372.6ms | clip: 0.0ms | optim: 23.6ms
282
+ step 115 | loss: 2.836510 | lr 1.8000e-04 | norm: 0.2183 | dt: 7828.37ms | tok/sec: 66972.9 | mem: 72103MB | Time Left: 04 hrs : 18 mins
283
+ [TIMING] data: 4.4ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.5ms
284
+ step 116 | loss: 2.801373 | lr 1.8000e-04 | norm: 0.1716 | dt: 7837.44ms | tok/sec: 66895.3 | mem: 72103MB | Time Left: 04 hrs : 19 mins
285
+ [TIMING] data: 4.4ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5378.9ms | clip: 0.0ms | optim: 23.6ms
286
+ step 117 | loss: 2.844561 | lr 1.8000e-04 | norm: 0.2244 | dt: 7835.18ms | tok/sec: 66914.6 | mem: 72103MB | Time Left: 04 hrs : 18 mins
287
+ [TIMING] data: 4.2ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5372.7ms | clip: 0.0ms | optim: 23.8ms
288
+ step 118 | loss: 2.863887 | lr 1.8000e-04 | norm: 0.1997 | dt: 7837.02ms | tok/sec: 66898.9 | mem: 72103MB | Time Left: 04 hrs : 18 mins
289
+ [TIMING] data: 4.2ms | fwd: 2419.7ms | loss: 4.9ms | bwd: 5373.4ms | clip: 0.0ms | optim: 23.6ms
290
+ step 119 | loss: 2.686055 | lr 1.8000e-04 | norm: 0.1676 | dt: 7827.74ms | tok/sec: 66978.2 | mem: 72103MB | Time Left: 04 hrs : 18 mins
291
+ [TIMING] data: 4.3ms | fwd: 2408.9ms | loss: 4.8ms | bwd: 5376.2ms | clip: 0.0ms | optim: 23.6ms
292
+ step 120 | loss: 2.752067 | lr 1.8000e-04 | norm: 0.1851 | dt: 7831.69ms | tok/sec: 66944.4 | mem: 72103MB | Time Left: 04 hrs : 18 mins
293
+ [TIMING] data: 4.2ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.7ms
294
+ step 121 | loss: 2.450900 | lr 1.8000e-04 | norm: 0.2143 | dt: 7831.44ms | tok/sec: 66946.6 | mem: 72103MB | Time Left: 04 hrs : 18 mins
295
+ [TIMING] data: 4.1ms | fwd: 2418.8ms | loss: 4.8ms | bwd: 5369.9ms | clip: 0.0ms | optim: 23.6ms
296
+ step 122 | loss: 2.662471 | lr 1.8000e-04 | norm: 0.1952 | dt: 7834.58ms | tok/sec: 66919.7 | mem: 72103MB | Time Left: 04 hrs : 18 mins
297
+ [TIMING] data: 4.2ms | fwd: 2421.8ms | loss: 4.8ms | bwd: 5370.1ms | clip: 0.0ms | optim: 23.6ms
298
+ step 123 | loss: 2.536113 | lr 1.8000e-04 | norm: 0.1946 | dt: 7836.54ms | tok/sec: 66903.0 | mem: 72103MB | Time Left: 04 hrs : 18 mins
299
+ [TIMING] data: 4.6ms | fwd: 2421.1ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.7ms
300
+ step 124 | loss: 2.916808 | lr 1.8000e-04 | norm: 0.3150 | dt: 7833.85ms | tok/sec: 66925.9 | mem: 72103MB | Time Left: 04 hrs : 17 mins
301
+ [TIMING] data: 4.5ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5368.9ms | clip: 0.0ms | optim: 23.6ms
302
+ step 125 | loss: 2.729327 | lr 1.8000e-04 | norm: 0.2281 | dt: 7836.30ms | tok/sec: 66905.0 | mem: 72103MB | Time Left: 04 hrs : 17 mins
303
+ [TIMING] data: 4.1ms | fwd: 2421.0ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
304
+ step 126 | loss: 2.845534 | lr 1.8000e-04 | norm: 0.2342 | dt: 7837.84ms | tok/sec: 66891.9 | mem: 72103MB | Time Left: 04 hrs : 17 mins
305
+ [TIMING] data: 4.3ms | fwd: 2419.8ms | loss: 4.9ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.7ms
306
+ step 127 | loss: 2.868937 | lr 1.8000e-04 | norm: 0.2717 | dt: 7827.31ms | tok/sec: 66981.9 | mem: 72103MB | Time Left: 04 hrs : 17 mins
307
+ [TIMING] data: 4.1ms | fwd: 2410.3ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.5ms
308
+ step 128 | loss: 2.541056 | lr 1.8000e-04 | norm: 0.1954 | dt: 7835.68ms | tok/sec: 66910.4 | mem: 72103MB | Time Left: 04 hrs : 17 mins
309
+ [TIMING] data: 4.3ms | fwd: 2419.3ms | loss: 4.9ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.6ms
310
+ step 129 | loss: 2.694195 | lr 1.8000e-04 | norm: 0.2187 | dt: 7830.91ms | tok/sec: 66951.1 | mem: 72103MB | Time Left: 04 hrs : 17 mins
311
+ [TIMING] data: 4.3ms | fwd: 2419.3ms | loss: 4.9ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
312
+ step 130 | loss: 2.644610 | lr 1.8000e-04 | norm: 0.2339 | dt: 7834.23ms | tok/sec: 66922.8 | mem: 72103MB | Time Left: 04 hrs : 17 mins
313
+ [TIMING] data: 4.4ms | fwd: 2419.4ms | loss: 4.9ms | bwd: 5372.5ms | clip: 0.0ms | optim: 23.6ms
314
+ step 131 | loss: 2.481298 | lr 1.8000e-04 | norm: 0.1903 | dt: 7822.42ms | tok/sec: 67023.8 | mem: 72103MB | Time Left: 04 hrs : 16 mins
315
+ [TIMING] data: 4.5ms | fwd: 2415.2ms | loss: 4.9ms | bwd: 5364.1ms | clip: 0.0ms | optim: 23.6ms
316
+ step 132 | loss: 2.863067 | lr 1.8000e-04 | norm: 0.2102 | dt: 7836.92ms | tok/sec: 66899.7 | mem: 72103MB | Time Left: 04 hrs : 17 mins
317
+ [TIMING] data: 4.1ms | fwd: 2424.1ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
318
+ step 133 | loss: 2.797191 | lr 1.8000e-04 | norm: 0.2385 | dt: 7834.17ms | tok/sec: 66923.3 | mem: 72103MB | Time Left: 04 hrs : 16 mins
319
+ [TIMING] data: 4.1ms | fwd: 2418.9ms | loss: 4.9ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
320
+ step 134 | loss: 2.811607 | lr 1.8000e-04 | norm: 0.2025 | dt: 7838.16ms | tok/sec: 66889.2 | mem: 72103MB | Time Left: 04 hrs : 16 mins
321
+ [TIMING] data: 4.3ms | fwd: 2422.1ms | loss: 4.8ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.6ms
322
+ step 135 | loss: 2.744267 | lr 1.8000e-04 | norm: 0.2590 | dt: 7842.79ms | tok/sec: 66849.7 | mem: 72103MB | Time Left: 04 hrs : 16 mins
323
+ [TIMING] data: 4.4ms | fwd: 2425.0ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.7ms
324
+ step 136 | loss: 2.712079 | lr 1.8000e-04 | norm: 0.1970 | dt: 7831.78ms | tok/sec: 66943.7 | mem: 72103MB | Time Left: 04 hrs : 16 mins
325
+ [TIMING] data: 4.4ms | fwd: 2419.4ms | loss: 4.9ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.7ms
326
+ step 137 | loss: 2.759872 | lr 1.8000e-04 | norm: 0.2726 | dt: 7839.30ms | tok/sec: 66879.5 | mem: 72103MB | Time Left: 04 hrs : 16 mins
327
+ [TIMING] data: 4.0ms | fwd: 2416.2ms | loss: 4.8ms | bwd: 5379.1ms | clip: 0.0ms | optim: 24.6ms
328
+ step 138 | loss: 2.621341 | lr 1.8000e-04 | norm: 0.2549 | dt: 7832.08ms | tok/sec: 66941.1 | mem: 72103MB | Time Left: 04 hrs : 16 mins
329
+ [TIMING] data: 3.5ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5371.1ms | clip: 0.0ms | optim: 23.7ms
330
+ step 139 | loss: 2.746702 | lr 1.8000e-04 | norm: 0.2566 | dt: 7830.79ms | tok/sec: 66952.2 | mem: 72103MB | Time Left: 04 hrs : 15 mins
331
+ [TIMING] data: 4.4ms | fwd: 2415.6ms | loss: 4.8ms | bwd: 5372.6ms | clip: 0.0ms | optim: 23.6ms
332
+ step 140 | loss: 2.800270 | lr 1.8000e-04 | norm: 0.2116 | dt: 7835.08ms | tok/sec: 66915.4 | mem: 72103MB | Time Left: 04 hrs : 15 mins
333
+ [TIMING] data: 4.2ms | fwd: 2418.4ms | loss: 4.8ms | bwd: 5374.6ms | clip: 0.0ms | optim: 23.7ms
334
+ step 141 | loss: 2.612864 | lr 1.8000e-04 | norm: 0.1952 | dt: 7838.81ms | tok/sec: 66883.6 | mem: 72103MB | Time Left: 04 hrs : 15 mins
335
+ [TIMING] data: 4.4ms | fwd: 2423.0ms | loss: 4.8ms | bwd: 5371.8ms | clip: 0.0ms | optim: 23.7ms
336
+ step 142 | loss: 2.863516 | lr 1.8000e-04 | norm: 0.2237 | dt: 7831.88ms | tok/sec: 66942.8 | mem: 72103MB | Time Left: 04 hrs : 15 mins
337
+ [TIMING] data: 3.9ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.7ms
338
+ step 143 | loss: 2.853393 | lr 1.8000e-04 | norm: 0.1985 | dt: 7831.02ms | tok/sec: 66950.2 | mem: 72103MB | Time Left: 04 hrs : 15 mins
339
+ [TIMING] data: 4.3ms | fwd: 2415.2ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.7ms
340
+ step 144 | loss: 2.810710 | lr 1.8000e-04 | norm: 0.2362 | dt: 7827.02ms | tok/sec: 66984.4 | mem: 72103MB | Time Left: 04 hrs : 15 mins
341
+ [TIMING] data: 4.4ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5365.0ms | clip: 0.0ms | optim: 23.8ms
342
+ step 145 | loss: 2.727190 | lr 1.8000e-04 | norm: 0.2459 | dt: 7834.58ms | tok/sec: 66919.8 | mem: 72103MB | Time Left: 04 hrs : 15 mins
343
+ [TIMING] data: 4.6ms | fwd: 2420.9ms | loss: 4.8ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.7ms
344
+ step 146 | loss: 2.490621 | lr 1.8000e-04 | norm: 0.2361 | dt: 7835.06ms | tok/sec: 66915.6 | mem: 72103MB | Time Left: 04 hrs : 15 mins
345
+ [TIMING] data: 4.5ms | fwd: 2423.8ms | loss: 4.8ms | bwd: 5368.0ms | clip: 0.0ms | optim: 23.6ms
346
+ step 147 | loss: 2.440445 | lr 1.8000e-04 | norm: 0.2350 | dt: 7829.40ms | tok/sec: 66964.0 | mem: 72103MB | Time Left: 04 hrs : 14 mins
347
+ [TIMING] data: 4.3ms | fwd: 2417.0ms | loss: 4.8ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.7ms
348
+ step 148 | loss: 2.722139 | lr 1.8000e-04 | norm: 0.2419 | dt: 7822.70ms | tok/sec: 67021.4 | mem: 72103MB | Time Left: 04 hrs : 14 mins
349
+ [TIMING] data: 4.4ms | fwd: 2413.3ms | loss: 4.8ms | bwd: 5366.4ms | clip: 0.0ms | optim: 23.6ms
350
+ step 149 | loss: 2.527200 | lr 1.8000e-04 | norm: 0.2305 | dt: 7825.10ms | tok/sec: 67000.8 | mem: 72103MB | Time Left: 04 hrs : 14 mins
351
+ [TIMING] data: 4.5ms | fwd: 2413.2ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.7ms
352
+ step 150 | loss: 2.664558 | lr 1.8000e-04 | norm: 0.2369 | dt: 7827.63ms | tok/sec: 66979.2 | mem: 72103MB | Time Left: 04 hrs : 14 mins
353
+ [TIMING] data: 4.4ms | fwd: 2417.6ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.8ms
354
+ step 151 | loss: 2.904971 | lr 1.8000e-04 | norm: 0.2537 | dt: 7830.84ms | tok/sec: 66951.7 | mem: 72103MB | Time Left: 04 hrs : 14 mins
355
+ [TIMING] data: 4.4ms | fwd: 2417.4ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.7ms
356
+ step 152 | loss: 2.746209 | lr 1.8000e-04 | norm: 0.3610 | dt: 7834.25ms | tok/sec: 66922.6 | mem: 72103MB | Time Left: 04 hrs : 14 mins
357
+ [TIMING] data: 4.5ms | fwd: 2416.9ms | loss: 4.8ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.7ms
358
+ step 153 | loss: 2.717691 | lr 1.8000e-04 | norm: 0.2418 | dt: 7837.54ms | tok/sec: 66894.4 | mem: 72103MB | Time Left: 04 hrs : 14 mins
359
+ [TIMING] data: 4.2ms | fwd: 2422.1ms | loss: 4.8ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.7ms
360
+ step 154 | loss: 2.696790 | lr 1.8000e-04 | norm: 0.2192 | dt: 7835.73ms | tok/sec: 66909.9 | mem: 72103MB | Time Left: 04 hrs : 14 mins
361
+ [TIMING] data: 4.3ms | fwd: 2421.4ms | loss: 4.9ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
362
+ step 155 | loss: 2.743514 | lr 1.8000e-04 | norm: 0.2044 | dt: 7839.40ms | tok/sec: 66878.6 | mem: 72103MB | Time Left: 04 hrs : 14 mins
363
+ [TIMING] data: 4.5ms | fwd: 2420.9ms | loss: 4.9ms | bwd: 5375.1ms | clip: 0.0ms | optim: 23.7ms
364
+ step 156 | loss: 2.695686 | lr 1.8000e-04 | norm: 0.3298 | dt: 7829.98ms | tok/sec: 66959.1 | mem: 72103MB | Time Left: 04 hrs : 13 mins
365
+ [TIMING] data: 4.5ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.6ms
366
+ step 157 | loss: 2.803331 | lr 1.8000e-04 | norm: 0.2812 | dt: 7824.18ms | tok/sec: 67008.7 | mem: 72103MB | Time Left: 04 hrs : 13 mins
367
+ [TIMING] data: 4.7ms | fwd: 2411.8ms | loss: 4.8ms | bwd: 5369.3ms | clip: 0.0ms | optim: 23.6ms
368
+ step 158 | loss: 2.566633 | lr 1.8000e-04 | norm: 0.2226 | dt: 7833.46ms | tok/sec: 66929.3 | mem: 72103MB | Time Left: 04 hrs : 13 mins
369
+ [TIMING] data: 4.6ms | fwd: 2420.9ms | loss: 4.9ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.8ms
370
+ step 159 | loss: 2.414896 | lr 1.8000e-04 | norm: 0.2340 | dt: 7844.29ms | tok/sec: 66836.9 | mem: 72103MB | Time Left: 04 hrs : 13 mins
371
+ [TIMING] data: 4.4ms | fwd: 2423.2ms | loss: 4.9ms | bwd: 5378.3ms | clip: 0.0ms | optim: 23.6ms
372
+ step 160 | loss: 2.776480 | lr 1.8000e-04 | norm: 0.3275 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 04 hrs : 13 mins
373
+ [TIMING] data: 4.3ms | fwd: 2413.9ms | loss: 4.9ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.7ms
374
+ step 161 | loss: 2.666731 | lr 1.8000e-04 | norm: 0.2277 | dt: 7830.16ms | tok/sec: 66957.5 | mem: 72103MB | Time Left: 04 hrs : 13 mins
375
+ [TIMING] data: 4.3ms | fwd: 2416.3ms | loss: 4.8ms | bwd: 5372.0ms | clip: 0.0ms | optim: 23.6ms
376
+ step 162 | loss: 2.890054 | lr 1.8000e-04 | norm: 0.2539 | dt: 7825.41ms | tok/sec: 66998.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
377
+ [TIMING] data: 4.5ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.7ms
378
+ step 163 | loss: 2.698914 | lr 1.8000e-04 | norm: 0.2679 | dt: 7831.07ms | tok/sec: 66949.7 | mem: 72103MB | Time Left: 04 hrs : 12 mins
379
+ [TIMING] data: 4.2ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.6ms
380
+ step 164 | loss: 2.809147 | lr 1.8000e-04 | norm: 0.2333 | dt: 7835.59ms | tok/sec: 66911.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
381
+ [TIMING] data: 4.7ms | fwd: 2421.8ms | loss: 4.9ms | bwd: 5370.7ms | clip: 0.0ms | optim: 23.6ms
382
+ step 165 | loss: 2.738762 | lr 1.8000e-04 | norm: 0.1988 | dt: 7841.92ms | tok/sec: 66857.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
383
+ [TIMING] data: 4.2ms | fwd: 2426.0ms | loss: 4.8ms | bwd: 5372.5ms | clip: 0.0ms | optim: 23.6ms
384
+ step 166 | loss: 2.692415 | lr 1.8000e-04 | norm: 0.2143 | dt: 7843.56ms | tok/sec: 66843.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
385
+ [TIMING] data: 4.4ms | fwd: 2426.5ms | loss: 4.9ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
386
+ step 167 | loss: 2.719501 | lr 1.8000e-04 | norm: 0.2727 | dt: 7833.75ms | tok/sec: 66926.8 | mem: 72103MB | Time Left: 04 hrs : 12 mins
387
+ [TIMING] data: 4.0ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
388
+ step 168 | loss: 2.901875 | lr 1.8000e-04 | norm: 0.2566 | dt: 7834.07ms | tok/sec: 66924.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
389
+ [TIMING] data: 4.5ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5375.6ms | clip: 0.0ms | optim: 23.6ms
390
+ step 169 | loss: 2.657280 | lr 1.8000e-04 | norm: 0.2217 | dt: 7823.22ms | tok/sec: 67016.9 | mem: 72103MB | Time Left: 04 hrs : 11 mins
391
+ [TIMING] data: 4.5ms | fwd: 2416.2ms | loss: 4.9ms | bwd: 5363.4ms | clip: 0.0ms | optim: 23.7ms
392
+ step 170 | loss: 2.508571 | lr 1.8000e-04 | norm: 0.2954 | dt: 7845.90ms | tok/sec: 66823.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
393
+ [TIMING] data: 4.1ms | fwd: 2423.9ms | loss: 4.8ms | bwd: 5378.8ms | clip: 0.0ms | optim: 23.6ms
394
+ step 171 | loss: 2.703474 | lr 1.8000e-04 | norm: 0.2781 | dt: 7833.89ms | tok/sec: 66925.6 | mem: 72103MB | Time Left: 04 hrs : 11 mins
395
+ [TIMING] data: 4.3ms | fwd: 2414.5ms | loss: 4.8ms | bwd: 5376.4ms | clip: 0.0ms | optim: 23.7ms
396
+ step 172 | loss: 2.485096 | lr 1.8000e-04 | norm: 0.2705 | dt: 7836.91ms | tok/sec: 66899.8 | mem: 72103MB | Time Left: 04 hrs : 11 mins
397
+ [TIMING] data: 4.4ms | fwd: 2418.4ms | loss: 4.8ms | bwd: 5375.8ms | clip: 0.0ms | optim: 23.6ms
398
+ step 173 | loss: 2.787684 | lr 1.8000e-04 | norm: 0.2739 | dt: 7831.90ms | tok/sec: 66942.7 | mem: 72103MB | Time Left: 04 hrs : 11 mins
399
+ [TIMING] data: 4.4ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5374.8ms | clip: 0.0ms | optim: 23.7ms
400
+ step 174 | loss: 2.589551 | lr 1.8000e-04 | norm: 0.2410 | dt: 7842.79ms | tok/sec: 66849.7 | mem: 72103MB | Time Left: 04 hrs : 11 mins
401
+ [TIMING] data: 4.6ms | fwd: 2424.5ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.7ms
402
+ step 175 | loss: 2.584219 | lr 1.8000e-04 | norm: 0.3058 | dt: 7835.70ms | tok/sec: 66910.2 | mem: 72103MB | Time Left: 04 hrs : 11 mins
403
+ [TIMING] data: 4.5ms | fwd: 2417.1ms | loss: 4.8ms | bwd: 5375.5ms | clip: 0.0ms | optim: 23.6ms
404
+ step 176 | loss: 2.491823 | lr 1.8000e-04 | norm: 0.2384 | dt: 7837.64ms | tok/sec: 66893.6 | mem: 72103MB | Time Left: 04 hrs : 11 mins
405
+ [TIMING] data: 4.5ms | fwd: 2416.0ms | loss: 4.8ms | bwd: 5378.6ms | clip: 0.0ms | optim: 23.6ms
406
+ step 177 | loss: 2.525588 | lr 1.8000e-04 | norm: 0.2601 | dt: 7830.52ms | tok/sec: 66954.5 | mem: 72103MB | Time Left: 04 hrs : 10 mins
407
+ [TIMING] data: 4.7ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5369.4ms | clip: 0.0ms | optim: 23.7ms
408
+ step 178 | loss: 2.731776 | lr 1.8000e-04 | norm: 0.2761 | dt: 7836.31ms | tok/sec: 66905.0 | mem: 72103MB | Time Left: 04 hrs : 11 mins
409
+ [TIMING] data: 4.6ms | fwd: 2424.9ms | loss: 4.9ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.7ms
410
+ step 179 | loss: 2.635677 | lr 1.8000e-04 | norm: 0.3731 | dt: 7828.18ms | tok/sec: 66974.5 | mem: 72103MB | Time Left: 04 hrs : 10 mins
411
+ [TIMING] data: 4.4ms | fwd: 2415.0ms | loss: 4.8ms | bwd: 5370.9ms | clip: 0.0ms | optim: 23.6ms
412
+ step 180 | loss: 2.596631 | lr 1.8000e-04 | norm: 0.3398 | dt: 7836.89ms | tok/sec: 66900.0 | mem: 72103MB | Time Left: 04 hrs : 10 mins
413
+ [TIMING] data: 4.5ms | fwd: 2424.8ms | loss: 4.8ms | bwd: 5368.7ms | clip: 0.0ms | optim: 23.7ms
414
+ step 181 | loss: 2.456206 | lr 1.8000e-04 | norm: 0.2772 | dt: 7830.11ms | tok/sec: 66958.0 | mem: 72103MB | Time Left: 04 hrs : 10 mins
415
+ [TIMING] data: 4.1ms | fwd: 2415.4ms | loss: 4.8ms | bwd: 5371.2ms | clip: 0.0ms | optim: 23.6ms
416
+ Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/mixcont/train_shard_00001.bin
417
+ 100000000
418
+ step 182 | loss: 2.585284 | lr 1.8000e-04 | norm: 0.2954 | dt: 7845.94ms | tok/sec: 66822.8 | mem: 72103MB | Time Left: 04 hrs : 10 mins
419
+ [TIMING] data: 11.7ms | fwd: 2424.5ms | loss: 4.8ms | bwd: 5372.0ms | clip: 0.0ms | optim: 23.6ms
420
+ step 183 | loss: 2.548141 | lr 1.8000e-04 | norm: 0.3066 | dt: 7834.46ms | tok/sec: 66920.8 | mem: 72103MB | Time Left: 04 hrs : 10 mins
421
+ [TIMING] data: 4.4ms | fwd: 2422.2ms | loss: 4.8ms | bwd: 5369.2ms | clip: 0.0ms | optim: 23.6ms
422
+ step 184 | loss: 2.478449 | lr 1.8000e-04 | norm: 0.2895 | dt: 7840.94ms | tok/sec: 66865.5 | mem: 72103MB | Time Left: 04 hrs : 10 mins
423
+ [TIMING] data: 4.4ms | fwd: 2427.2ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.7ms
424
+ step 185 | loss: 2.528314 | lr 1.8000e-04 | norm: 0.3043 | dt: 7831.82ms | tok/sec: 66943.3 | mem: 72103MB | Time Left: 04 hrs : 09 mins
425
+ [TIMING] data: 4.3ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5370.8ms | clip: 0.0ms | optim: 23.6ms
426
+ step 186 | loss: 2.352158 | lr 1.8000e-04 | norm: 0.2586 | dt: 7827.45ms | tok/sec: 66980.7 | mem: 72103MB | Time Left: 04 hrs : 09 mins
427
+ [TIMING] data: 4.4ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5366.7ms | clip: 0.0ms | optim: 23.7ms
428
+ step 187 | loss: 2.563450 | lr 1.8000e-04 | norm: 0.3322 | dt: 7834.83ms | tok/sec: 66917.6 | mem: 72103MB | Time Left: 04 hrs : 09 mins
429
+ [TIMING] data: 4.3ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.7ms
430
+ step 188 | loss: 2.539817 | lr 1.8000e-04 | norm: 0.2921 | dt: 7829.95ms | tok/sec: 66959.3 | mem: 72103MB | Time Left: 04 hrs : 09 mins
431
+ [TIMING] data: 4.1ms | fwd: 2416.3ms | loss: 4.8ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.7ms
432
+ step 189 | loss: 2.632222 | lr 1.8000e-04 | norm: 0.3063 | dt: 7840.75ms | tok/sec: 66867.1 | mem: 72103MB | Time Left: 04 hrs : 09 mins
433
+ [TIMING] data: 4.0ms | fwd: 2421.2ms | loss: 4.9ms | bwd: 5376.5ms | clip: 0.0ms | optim: 23.6ms
434
+ step 190 | loss: 2.607751 | lr 1.8000e-04 | norm: 0.3119 | dt: 7827.96ms | tok/sec: 66976.3 | mem: 72103MB | Time Left: 04 hrs : 09 mins
435
+ [TIMING] data: 4.3ms | fwd: 2416.4ms | loss: 4.8ms | bwd: 5367.8ms | clip: 0.0ms | optim: 23.7ms
436
+ step 191 | loss: 2.665597 | lr 1.8000e-04 | norm: 0.3547 | dt: 7840.90ms | tok/sec: 66865.8 | mem: 72103MB | Time Left: 04 hrs : 09 mins
437
+ [TIMING] data: 4.3ms | fwd: 2424.8ms | loss: 4.9ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.8ms
438
+ step 192 | loss: 2.514341 | lr 1.8000e-04 | norm: 0.3333 | dt: 7835.85ms | tok/sec: 66908.9 | mem: 72103MB | Time Left: 04 hrs : 09 mins
439
+ [TIMING] data: 4.1ms | fwd: 2416.4ms | loss: 4.8ms | bwd: 5376.6ms | clip: 0.0ms | optim: 23.7ms
440
+ step 193 | loss: 2.530896 | lr 1.8000e-04 | norm: 0.3155 | dt: 7832.06ms | tok/sec: 66941.2 | mem: 72103MB | Time Left: 04 hrs : 08 mins
441
+ [TIMING] data: 4.2ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.6ms
442
+ step 194 | loss: 2.378180 | lr 1.8000e-04 | norm: 0.3543 | dt: 7828.80ms | tok/sec: 66969.1 | mem: 72103MB | Time Left: 04 hrs : 08 mins
443
+ [TIMING] data: 4.4ms | fwd: 2410.8ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.8ms
444
+ step 195 | loss: 2.700363 | lr 1.8000e-04 | norm: 0.4001 | dt: 7833.91ms | tok/sec: 66925.4 | mem: 72103MB | Time Left: 04 hrs : 08 mins
445
+ [TIMING] data: 4.4ms | fwd: 2421.6ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.6ms
446
+ step 196 | loss: 2.455045 | lr 1.8000e-04 | norm: 0.3854 | dt: 7824.79ms | tok/sec: 67003.5 | mem: 72103MB | Time Left: 04 hrs : 08 mins
447
+ [TIMING] data: 4.5ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5363.8ms | clip: 0.0ms | optim: 23.7ms
448
+ step 197 | loss: 2.718957 | lr 1.8000e-04 | norm: 0.4156 | dt: 7826.31ms | tok/sec: 66990.4 | mem: 72103MB | Time Left: 04 hrs : 08 mins
449
+ [TIMING] data: 4.5ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5366.9ms | clip: 0.0ms | optim: 23.7ms
450
+ step 198 | loss: 2.476290 | lr 1.8000e-04 | norm: 0.4051 | dt: 7827.59ms | tok/sec: 66979.5 | mem: 72103MB | Time Left: 04 hrs : 08 mins
451
+ [TIMING] data: 4.1ms | fwd: 2413.4ms | loss: 4.8ms | bwd: 5370.7ms | clip: 0.0ms | optim: 23.6ms
452
+ step 199 | loss: 2.422088 | lr 1.8000e-04 | norm: 0.3190 | dt: 7836.25ms | tok/sec: 66905.5 | mem: 72103MB | Time Left: 04 hrs : 08 mins
453
+ [TIMING] data: 4.1ms | fwd: 2421.3ms | loss: 4.9ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.6ms
454
+ [CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt (safely)
455
+ step 200 | loss: 2.402291 | lr 1.8000e-04 | norm: 0.4287 | dt: 8298.17ms | tok/sec: 63181.2 | mem: 72103MB | Time Left: 04 hrs : 22 mins
456
+ [TIMING] data: 4.8ms | fwd: 2909.3ms | loss: 4.8ms | bwd: 5344.9ms | clip: 0.0ms | optim: 23.6ms
457
+ step 201 | loss: 2.387901 | lr 1.8000e-04 | norm: 0.3077 | dt: 7815.83ms | tok/sec: 67080.3 | mem: 72103MB | Time Left: 04 hrs : 07 mins
458
+ [TIMING] data: 4.9ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.7ms
459
+ step 202 | loss: 2.360209 | lr 1.8000e-04 | norm: 0.3530 | dt: 7813.41ms | tok/sec: 67101.0 | mem: 72103MB | Time Left: 04 hrs : 07 mins
460
+ [TIMING] data: 4.8ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5362.2ms | clip: 0.0ms | optim: 23.6ms
461
+ step 203 | loss: 2.409445 | lr 1.8000e-04 | norm: 0.3761 | dt: 7821.09ms | tok/sec: 67035.2 | mem: 72103MB | Time Left: 04 hrs : 07 mins
462
+ [TIMING] data: 4.9ms | fwd: 2416.8ms | loss: 4.8ms | bwd: 5360.5ms | clip: 0.0ms | optim: 23.6ms
463
+ step 204 | loss: 2.408840 | lr 1.8000e-04 | norm: 0.6411 | dt: 7817.69ms | tok/sec: 67064.3 | mem: 72103MB | Time Left: 04 hrs : 07 mins
464
+ [TIMING] data: 4.8ms | fwd: 2407.8ms | loss: 4.8ms | bwd: 5367.6ms | clip: 0.0ms | optim: 23.6ms
465
+ step 205 | loss: 2.459621 | lr 1.8000e-04 | norm: 0.4455 | dt: 7827.36ms | tok/sec: 66981.5 | mem: 72103MB | Time Left: 04 hrs : 07 mins
466
+ [TIMING] data: 4.9ms | fwd: 2417.5ms | loss: 4.8ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
467
+ step 206 | loss: 2.402254 | lr 1.8000e-04 | norm: 0.4152 | dt: 7831.62ms | tok/sec: 66945.0 | mem: 72103MB | Time Left: 04 hrs : 07 mins
468
+ [TIMING] data: 4.8ms | fwd: 2421.7ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.5ms
469
+ step 207 | loss: 2.387150 | lr 1.8000e-04 | norm: 0.4003 | dt: 7827.34ms | tok/sec: 66981.7 | mem: 72103MB | Time Left: 04 hrs : 06 mins
470
+ [TIMING] data: 4.9ms | fwd: 2414.8ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
471
+ step 208 | loss: 2.443284 | lr 1.8000e-04 | norm: 0.3607 | dt: 7830.10ms | tok/sec: 66958.0 | mem: 72103MB | Time Left: 04 hrs : 06 mins
472
+ [TIMING] data: 4.8ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.6ms
473
+ step 209 | loss: 2.525800 | lr 1.8000e-04 | norm: 0.4068 | dt: 7830.26ms | tok/sec: 66956.6 | mem: 72103MB | Time Left: 04 hrs : 06 mins
474
+ [TIMING] data: 4.8ms | fwd: 2422.8ms | loss: 4.8ms | bwd: 5364.6ms | clip: 0.0ms | optim: 23.5ms
475
+ step 210 | loss: 2.456215 | lr 1.8000e-04 | norm: 0.3717 | dt: 7825.41ms | tok/sec: 66998.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
476
+ [TIMING] data: 4.9ms | fwd: 2414.9ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.6ms
477
+ step 211 | loss: 2.357838 | lr 1.8000e-04 | norm: 0.4258 | dt: 7833.70ms | tok/sec: 66927.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
478
+ [TIMING] data: 4.8ms | fwd: 2420.7ms | loss: 4.9ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
479
+ step 212 | loss: 2.446153 | lr 1.8000e-04 | norm: 0.3971 | dt: 7830.20ms | tok/sec: 66957.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
480
+ [TIMING] data: 4.9ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5366.4ms | clip: 0.0ms | optim: 23.5ms
481
+ step 213 | loss: 2.249928 | lr 1.8000e-04 | norm: 0.4567 | dt: 7823.83ms | tok/sec: 67011.7 | mem: 72103MB | Time Left: 04 hrs : 06 mins
482
+ [TIMING] data: 4.9ms | fwd: 2416.2ms | loss: 4.8ms | bwd: 5364.0ms | clip: 0.0ms | optim: 23.6ms
483
+ step 214 | loss: 2.369561 | lr 1.8000e-04 | norm: 0.4493 | dt: 7832.30ms | tok/sec: 66939.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
484
+ [TIMING] data: 4.9ms | fwd: 2420.2ms | loss: 4.8ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.6ms
485
+ step 215 | loss: 2.343032 | lr 1.8000e-04 | norm: 0.4376 | dt: 7826.26ms | tok/sec: 66990.9 | mem: 72103MB | Time Left: 04 hrs : 05 mins
486
+ [TIMING] data: 4.8ms | fwd: 2417.1ms | loss: 4.8ms | bwd: 5364.8ms | clip: 0.0ms | optim: 23.7ms
487
+ step 216 | loss: 2.240227 | lr 1.8000e-04 | norm: 0.4092 | dt: 7821.41ms | tok/sec: 67032.4 | mem: 72103MB | Time Left: 04 hrs : 05 mins
488
+ [TIMING] data: 4.8ms | fwd: 2410.4ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.6ms
489
+ step 217 | loss: 2.362448 | lr 1.8000e-04 | norm: 0.4497 | dt: 7830.47ms | tok/sec: 66954.9 | mem: 72103MB | Time Left: 04 hrs : 05 mins
490
+ [TIMING] data: 5.0ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.5ms
491
+ step 218 | loss: 2.288676 | lr 1.8000e-04 | norm: 0.4341 | dt: 7829.31ms | tok/sec: 66964.8 | mem: 72103MB | Time Left: 04 hrs : 05 mins
492
+ [TIMING] data: 4.9ms | fwd: 2417.2ms | loss: 4.8ms | bwd: 5368.5ms | clip: 0.0ms | optim: 23.7ms
493
+ step 219 | loss: 2.323463 | lr 1.8000e-04 | norm: 0.4153 | dt: 7838.80ms | tok/sec: 66883.7 | mem: 72103MB | Time Left: 04 hrs : 05 mins
494
+ [TIMING] data: 4.9ms | fwd: 2419.0ms | loss: 4.9ms | bwd: 5377.4ms | clip: 0.0ms | optim: 23.6ms
495
+ step 220 | loss: 2.250193 | lr 1.8000e-04 | norm: 0.4058 | dt: 7833.43ms | tok/sec: 66929.6 | mem: 72103MB | Time Left: 04 hrs : 05 mins
496
+ [TIMING] data: 4.8ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.5ms
497
+ step 221 | loss: 2.382532 | lr 1.8000e-04 | norm: 0.4433 | dt: 7835.52ms | tok/sec: 66911.7 | mem: 72103MB | Time Left: 04 hrs : 05 mins
498
+ [TIMING] data: 4.8ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.7ms
499
+ step 222 | loss: 2.410047 | lr 1.8000e-04 | norm: 0.5359 | dt: 7823.76ms | tok/sec: 67012.3 | mem: 72103MB | Time Left: 04 hrs : 04 mins
500
+ [TIMING] data: 4.9ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5366.7ms | clip: 0.0ms | optim: 23.6ms
501
+ step 223 | loss: 2.355986 | lr 1.8000e-04 | norm: 0.4549 | dt: 7831.10ms | tok/sec: 66949.5 | mem: 72103MB | Time Left: 04 hrs : 04 mins
502
+ [TIMING] data: 4.8ms | fwd: 2418.7ms | loss: 4.9ms | bwd: 5369.2ms | clip: 0.0ms | optim: 23.6ms
503
+ step 224 | loss: 2.260771 | lr 1.8000e-04 | norm: 0.4940 | dt: 7835.26ms | tok/sec: 66913.9 | mem: 72103MB | Time Left: 04 hrs : 04 mins
504
+ [TIMING] data: 4.8ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
505
+ step 225 | loss: 2.336183 | lr 1.8000e-04 | norm: 0.4559 | dt: 7836.82ms | tok/sec: 66900.6 | mem: 72103MB | Time Left: 04 hrs : 04 mins
506
+ [TIMING] data: 4.9ms | fwd: 2416.9ms | loss: 4.8ms | bwd: 5376.5ms | clip: 0.0ms | optim: 23.7ms
507
+ step 226 | loss: 2.206506 | lr 1.8000e-04 | norm: 0.5058 | dt: 7836.13ms | tok/sec: 66906.5 | mem: 72103MB | Time Left: 04 hrs : 04 mins
508
+ [TIMING] data: 4.9ms | fwd: 2425.9ms | loss: 4.8ms | bwd: 5366.5ms | clip: 0.0ms | optim: 23.6ms
509
+ step 227 | loss: 2.221648 | lr 1.8000e-04 | norm: 0.4108 | dt: 7834.61ms | tok/sec: 66919.4 | mem: 72103MB | Time Left: 04 hrs : 04 mins
510
+ [TIMING] data: 4.8ms | fwd: 2419.4ms | loss: 4.8ms | bwd: 5372.5ms | clip: 0.0ms | optim: 23.5ms
511
+ step 228 | loss: 2.372312 | lr 1.8000e-04 | norm: 0.4778 | dt: 7838.01ms | tok/sec: 66890.5 | mem: 72103MB | Time Left: 04 hrs : 04 mins
512
+ [TIMING] data: 4.8ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5373.5ms | clip: 0.0ms | optim: 23.6ms
513
+ step 229 | loss: 2.021067 | lr 1.8000e-04 | norm: 0.4256 | dt: 7835.34ms | tok/sec: 66913.3 | mem: 72103MB | Time Left: 04 hrs : 04 mins
514
+ [TIMING] data: 4.9ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
515
+ step 230 | loss: 2.196075 | lr 1.8000e-04 | norm: 0.5318 | dt: 7825.84ms | tok/sec: 66994.5 | mem: 72103MB | Time Left: 04 hrs : 03 mins
516
+ [TIMING] data: 4.9ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.6ms
517
+ step 231 | loss: 2.229375 | lr 1.8000e-04 | norm: 0.4619 | dt: 7832.63ms | tok/sec: 66936.4 | mem: 72103MB | Time Left: 04 hrs : 03 mins
518
+ [TIMING] data: 4.8ms | fwd: 2417.5ms | loss: 4.8ms | bwd: 5372.8ms | clip: 0.0ms | optim: 23.6ms
519
+ step 232 | loss: 2.005525 | lr 1.8000e-04 | norm: 0.4127 | dt: 7829.49ms | tok/sec: 66963.2 | mem: 72103MB | Time Left: 04 hrs : 03 mins
520
+ [TIMING] data: 4.8ms | fwd: 2412.1ms | loss: 4.8ms | bwd: 5373.0ms | clip: 0.0ms | optim: 23.6ms
521
+ step 233 | loss: 2.188138 | lr 1.8000e-04 | norm: 0.5179 | dt: 7836.54ms | tok/sec: 66903.0 | mem: 72103MB | Time Left: 04 hrs : 03 mins
522
+ [TIMING] data: 4.8ms | fwd: 2419.1ms | loss: 4.9ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.7ms
523
+ step 234 | loss: 2.369011 | lr 1.8000e-04 | norm: 0.5784 | dt: 7826.64ms | tok/sec: 66987.6 | mem: 72103MB | Time Left: 04 hrs : 03 mins
524
+ [TIMING] data: 4.9ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.5ms
525
+ step 235 | loss: 2.315351 | lr 1.8000e-04 | norm: 0.5331 | dt: 7823.82ms | tok/sec: 67011.7 | mem: 72103MB | Time Left: 04 hrs : 03 mins
526
+ [TIMING] data: 4.8ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.5ms
527
+ step 236 | loss: 1.939878 | lr 1.8000e-04 | norm: 0.4362 | dt: 7831.17ms | tok/sec: 66948.8 | mem: 72103MB | Time Left: 04 hrs : 03 mins
528
+ [TIMING] data: 4.9ms | fwd: 2415.0ms | loss: 4.9ms | bwd: 5372.4ms | clip: 0.0ms | optim: 23.6ms
529
+ step 237 | loss: 2.137450 | lr 1.8000e-04 | norm: 0.5205 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 04 hrs : 03 mins
530
+ [TIMING] data: 4.8ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.6ms
531
+ step 238 | loss: 2.292795 | lr 1.8000e-04 | norm: 0.5377 | dt: 7825.25ms | tok/sec: 66999.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
532
+ [TIMING] data: 4.9ms | fwd: 2413.1ms | loss: 4.8ms | bwd: 5368.9ms | clip: 0.0ms | optim: 23.7ms
533
+ step 239 | loss: 2.211943 | lr 1.8000e-04 | norm: 0.5593 | dt: 7832.50ms | tok/sec: 66937.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
534
+ [TIMING] data: 4.9ms | fwd: 2419.1ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
535
+ step 240 | loss: 2.144189 | lr 1.8000e-04 | norm: 0.5692 | dt: 7830.98ms | tok/sec: 66950.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
536
+ [TIMING] data: 4.7ms | fwd: 2416.9ms | loss: 4.8ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.6ms
537
+ step 241 | loss: 2.031067 | lr 1.8000e-04 | norm: 0.5292 | dt: 7835.10ms | tok/sec: 66915.3 | mem: 72103MB | Time Left: 04 hrs : 02 mins
538
+ [TIMING] data: 4.8ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
539
+ step 242 | loss: 2.202729 | lr 1.8000e-04 | norm: 0.5180 | dt: 7831.67ms | tok/sec: 66944.6 | mem: 72103MB | Time Left: 04 hrs : 02 mins
540
+ [TIMING] data: 4.9ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.6ms
541
+ step 243 | loss: 2.130872 | lr 1.8000e-04 | norm: 0.6109 | dt: 7834.72ms | tok/sec: 66918.6 | mem: 72103MB | Time Left: 04 hrs : 02 mins
542
+ [TIMING] data: 4.8ms | fwd: 2419.4ms | loss: 4.9ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.7ms
543
+ step 244 | loss: 2.166822 | lr 1.8000e-04 | norm: 0.5125 | dt: 7821.30ms | tok/sec: 67033.4 | mem: 72103MB | Time Left: 04 hrs : 01 mins
544
+ [TIMING] data: 4.8ms | fwd: 2409.2ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.5ms
545
+ step 245 | loss: 2.147327 | lr 1.8000e-04 | norm: 0.6140 | dt: 7827.70ms | tok/sec: 66978.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
546
+ [TIMING] data: 4.7ms | fwd: 2415.7ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
547
+ step 246 | loss: 2.215064 | lr 1.8000e-04 | norm: 0.5953 | dt: 7838.44ms | tok/sec: 66886.8 | mem: 72103MB | Time Left: 04 hrs : 02 mins
548
+ [TIMING] data: 4.7ms | fwd: 2422.7ms | loss: 4.8ms | bwd: 5373.4ms | clip: 0.0ms | optim: 23.7ms
549
+ step 247 | loss: 2.186167 | lr 1.8000e-04 | norm: 0.5049 | dt: 7820.30ms | tok/sec: 67041.9 | mem: 72103MB | Time Left: 04 hrs : 01 mins
550
+ [TIMING] data: 4.9ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.7ms
551
+ step 248 | loss: 2.352702 | lr 1.8000e-04 | norm: 0.6359 | dt: 7825.95ms | tok/sec: 66993.5 | mem: 72103MB | Time Left: 04 hrs : 01 mins
552
+ [TIMING] data: 4.9ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5364.8ms | clip: 0.0ms | optim: 23.6ms
553
+ step 249 | loss: 2.165310 | lr 1.8000e-04 | norm: 0.5937 | dt: 7838.78ms | tok/sec: 66883.9 | mem: 72103MB | Time Left: 04 hrs : 01 mins
554
+ [TIMING] data: 4.8ms | fwd: 2421.6ms | loss: 4.8ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.5ms
555
+ validation loss: 2.0652
556
+ step 250 | loss: 2.170025 | lr 1.8000e-04 | norm: 0.6153 | dt: 13815.70ms | tok/sec: 37948.7 | mem: 72103MB | Time Left: 07 hrs : 05 mins
557
+ [TIMING] data: 4.9ms | fwd: 2422.0ms | loss: 4.9ms | bwd: 5370.4ms | clip: 0.0ms | optim: 23.5ms
558
+ step 251 | loss: 1.956852 | lr 1.8000e-04 | norm: 0.5652 | dt: 7824.67ms | tok/sec: 67004.5 | mem: 72103MB | Time Left: 04 hrs : 01 mins
559
+ [TIMING] data: 4.9ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5364.5ms | clip: 0.0ms | optim: 23.6ms
560
+ step 252 | loss: 2.047342 | lr 1.8000e-04 | norm: 0.6065 | dt: 7828.67ms | tok/sec: 66970.3 | mem: 72103MB | Time Left: 04 hrs : 01 mins
561
+ [TIMING] data: 4.9ms | fwd: 2417.3ms | loss: 4.9ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.5ms
562
+ step 253 | loss: 2.042702 | lr 1.8000e-04 | norm: 0.5501 | dt: 7829.01ms | tok/sec: 66967.3 | mem: 72103MB | Time Left: 04 hrs : 01 mins
563
+ [TIMING] data: 4.6ms | fwd: 2416.0ms | loss: 4.8ms | bwd: 5370.8ms | clip: 0.0ms | optim: 23.6ms
564
+ step 254 | loss: 2.038959 | lr 1.8000e-04 | norm: 0.6408 | dt: 7832.29ms | tok/sec: 66939.3 | mem: 72103MB | Time Left: 04 hrs : 00 mins
565
+ [TIMING] data: 4.8ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.6ms
566
+ step 255 | loss: 2.131271 | lr 1.8000e-04 | norm: 0.5878 | dt: 7836.77ms | tok/sec: 66901.0 | mem: 72103MB | Time Left: 04 hrs : 00 mins
567
+ [TIMING] data: 4.9ms | fwd: 2422.8ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.5ms
568
+ step 256 | loss: 2.110420 | lr 1.8000e-04 | norm: 0.6424 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 04 hrs : 00 mins
569
+ [TIMING] data: 4.9ms | fwd: 2418.2ms | loss: 4.8ms | bwd: 5366.3ms | clip: 0.0ms | optim: 23.7ms
570
+ step 257 | loss: 2.220668 | lr 1.8000e-04 | norm: 0.6648 | dt: 7832.83ms | tok/sec: 66934.7 | mem: 72103MB | Time Left: 04 hrs : 00 mins
571
+ [TIMING] data: 4.8ms | fwd: 2417.7ms | loss: 4.9ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.5ms
572
+ step 258 | loss: 2.076067 | lr 1.8000e-04 | norm: 0.7399 | dt: 7831.29ms | tok/sec: 66947.8 | mem: 72103MB | Time Left: 04 hrs : 00 mins
573
+ [TIMING] data: 4.9ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5369.4ms | clip: 0.0ms | optim: 23.8ms
574
+ step 259 | loss: 1.938241 | lr 1.8000e-04 | norm: 0.6259 | dt: 7828.57ms | tok/sec: 66971.1 | mem: 72103MB | Time Left: 04 hrs : 00 mins
575
+ [TIMING] data: 4.7ms | fwd: 2415.7ms | loss: 4.8ms | bwd: 5370.1ms | clip: 0.0ms | optim: 23.5ms
576
+ step 260 | loss: 1.857182 | lr 1.8000e-04 | norm: 0.6081 | dt: 7832.76ms | tok/sec: 66935.3 | mem: 72103MB | Time Left: 04 hrs : 00 mins
577
+ [TIMING] data: 4.9ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5374.8ms | clip: 0.0ms | optim: 23.5ms
578
+ step 261 | loss: 2.031495 | lr 1.8000e-04 | norm: 0.8596 | dt: 7828.89ms | tok/sec: 66968.4 | mem: 72103MB | Time Left: 03 hrs : 59 mins
579
+ [TIMING] data: 4.7ms | fwd: 2416.0ms | loss: 4.8ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
580
+ step 262 | loss: 1.959992 | lr 1.8000e-04 | norm: 0.6619 | dt: 7822.34ms | tok/sec: 67024.5 | mem: 72103MB | Time Left: 03 hrs : 59 mins
581
+ [TIMING] data: 4.8ms | fwd: 2410.4ms | loss: 4.8ms | bwd: 5367.2ms | clip: 0.0ms | optim: 23.7ms
582
+ step 263 | loss: 1.983674 | lr 1.8000e-04 | norm: 0.6752 | dt: 7834.08ms | tok/sec: 66924.0 | mem: 72103MB | Time Left: 03 hrs : 59 mins
583
+ [TIMING] data: 4.8ms | fwd: 2422.9ms | loss: 4.9ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.6ms
584
+ step 264 | loss: 1.938962 | lr 1.8000e-04 | norm: 0.8107 | dt: 7836.23ms | tok/sec: 66905.7 | mem: 72103MB | Time Left: 03 hrs : 59 mins
585
+ [TIMING] data: 4.7ms | fwd: 2418.0ms | loss: 4.9ms | bwd: 5374.6ms | clip: 0.0ms | optim: 23.7ms
586
+ step 265 | loss: 1.999571 | lr 1.8000e-04 | norm: 0.7463 | dt: 7837.36ms | tok/sec: 66896.0 | mem: 72103MB | Time Left: 03 hrs : 59 mins
587
+ [TIMING] data: 4.8ms | fwd: 2424.8ms | loss: 4.8ms | bwd: 5369.0ms | clip: 0.0ms | optim: 23.6ms
588
+ step 266 | loss: 1.981171 | lr 1.8000e-04 | norm: 0.7313 | dt: 7826.53ms | tok/sec: 66988.6 | mem: 72103MB | Time Left: 03 hrs : 59 mins
589
+ [TIMING] data: 4.8ms | fwd: 2415.2ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.7ms
590
+ step 267 | loss: 1.929014 | lr 1.8000e-04 | norm: 0.7204 | dt: 7827.35ms | tok/sec: 66981.6 | mem: 72103MB | Time Left: 03 hrs : 59 mins
591
+ [TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.5ms
592
+ step 268 | loss: 1.872691 | lr 1.8000e-04 | norm: 0.7225 | dt: 7832.84ms | tok/sec: 66934.6 | mem: 72103MB | Time Left: 03 hrs : 59 mins
593
+ [TIMING] data: 4.8ms | fwd: 2424.3ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.6ms
594
+ step 269 | loss: 2.023110 | lr 1.8000e-04 | norm: 0.7982 | dt: 7832.75ms | tok/sec: 66935.4 | mem: 72103MB | Time Left: 03 hrs : 59 mins
595
+ [TIMING] data: 4.8ms | fwd: 2413.6ms | loss: 4.8ms | bwd: 5376.9ms | clip: 0.0ms | optim: 23.7ms
596
+ step 270 | loss: 1.823170 | lr 1.8000e-04 | norm: 0.7132 | dt: 7826.80ms | tok/sec: 66986.2 | mem: 72103MB | Time Left: 03 hrs : 58 mins
597
+ [TIMING] data: 4.7ms | fwd: 2413.8ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
598
+ step 271 | loss: 1.888498 | lr 1.8000e-04 | norm: 0.8431 | dt: 7833.99ms | tok/sec: 66924.8 | mem: 72103MB | Time Left: 03 hrs : 58 mins
599
+ [TIMING] data: 4.9ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.7ms
600
+ step 272 | loss: 1.948550 | lr 1.8000e-04 | norm: 0.8378 | dt: 7827.51ms | tok/sec: 66980.2 | mem: 72103MB | Time Left: 03 hrs : 58 mins
601
+ [TIMING] data: 4.9ms | fwd: 2417.3ms | loss: 4.9ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.7ms
602
+ step 273 | loss: 1.855544 | lr 1.8000e-04 | norm: 0.8257 | dt: 7828.29ms | tok/sec: 66973.5 | mem: 72103MB | Time Left: 03 hrs : 58 mins
603
+ [TIMING] data: 4.8ms | fwd: 2417.6ms | loss: 4.9ms | bwd: 5366.8ms | clip: 0.0ms | optim: 23.6ms
604
+ step 274 | loss: 1.809207 | lr 1.8000e-04 | norm: 0.7570 | dt: 7832.35ms | tok/sec: 66938.8 | mem: 72103MB | Time Left: 03 hrs : 58 mins
605
+ [TIMING] data: 4.8ms | fwd: 2418.7ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
606
+ step 275 | loss: 1.990041 | lr 1.8000e-04 | norm: 0.8103 | dt: 7836.13ms | tok/sec: 66906.5 | mem: 72103MB | Time Left: 03 hrs : 58 mins
607
+ [TIMING] data: 4.8ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5374.6ms | clip: 0.0ms | optim: 23.7ms
608
+ step 276 | loss: 2.061124 | lr 1.8000e-04 | norm: 0.8882 | dt: 7832.03ms | tok/sec: 66941.5 | mem: 72103MB | Time Left: 03 hrs : 58 mins
609
+ [TIMING] data: 4.8ms | fwd: 2421.7ms | loss: 4.8ms | bwd: 5367.0ms | clip: 0.0ms | optim: 23.6ms
610
+ step 277 | loss: 1.789543 | lr 1.8000e-04 | norm: 0.8902 | dt: 7820.42ms | tok/sec: 67040.9 | mem: 72103MB | Time Left: 03 hrs : 57 mins
611
+ [TIMING] data: 4.8ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5368.1ms | clip: 0.0ms | optim: 23.5ms
612
+ step 278 | loss: 1.775792 | lr 1.8000e-04 | norm: 0.8987 | dt: 7828.34ms | tok/sec: 66973.1 | mem: 72103MB | Time Left: 03 hrs : 57 mins
613
+ [TIMING] data: 4.8ms | fwd: 2409.9ms | loss: 4.8ms | bwd: 5376.0ms | clip: 0.0ms | optim: 23.5ms
614
+ step 279 | loss: 1.932578 | lr 1.8000e-04 | norm: 0.8560 | dt: 7831.65ms | tok/sec: 66944.7 | mem: 72103MB | Time Left: 03 hrs : 57 mins
615
+ [TIMING] data: 4.9ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5370.7ms | clip: 0.0ms | optim: 23.6ms
616
+ step 280 | loss: 1.780883 | lr 1.8000e-04 | norm: 0.8304 | dt: 7820.13ms | tok/sec: 67043.4 | mem: 72103MB | Time Left: 03 hrs : 57 mins
617
+ [TIMING] data: 4.9ms | fwd: 2412.9ms | loss: 4.8ms | bwd: 5364.9ms | clip: 0.0ms | optim: 23.6ms
618
+ step 281 | loss: 1.759020 | lr 1.8000e-04 | norm: 0.9021 | dt: 7830.58ms | tok/sec: 66953.9 | mem: 72103MB | Time Left: 03 hrs : 57 mins
619
+ [TIMING] data: 4.9ms | fwd: 2415.8ms | loss: 4.9ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
620
+ step 282 | loss: 1.844155 | lr 1.8000e-04 | norm: 0.8803 | dt: 7829.19ms | tok/sec: 66965.8 | mem: 72103MB | Time Left: 03 hrs : 57 mins
621
+ [TIMING] data: 4.8ms | fwd: 2418.1ms | loss: 4.8ms | bwd: 5367.2ms | clip: 0.0ms | optim: 23.6ms
622
+ step 283 | loss: 1.751251 | lr 1.8000e-04 | norm: 0.8930 | dt: 7825.31ms | tok/sec: 66999.0 | mem: 72103MB | Time Left: 03 hrs : 56 mins
623
+ [TIMING] data: 4.7ms | fwd: 2412.6ms | loss: 4.8ms | bwd: 5369.3ms | clip: 0.0ms | optim: 23.7ms
624
+ step 284 | loss: 1.852886 | lr 1.8000e-04 | norm: 0.9555 | dt: 7816.54ms | tok/sec: 67074.2 | mem: 72103MB | Time Left: 03 hrs : 56 mins
625
+ [TIMING] data: 4.7ms | fwd: 2410.8ms | loss: 4.8ms | bwd: 5363.4ms | clip: 0.0ms | optim: 23.6ms
626
+ step 285 | loss: 1.759097 | lr 1.8000e-04 | norm: 0.8659 | dt: 7832.42ms | tok/sec: 66938.1 | mem: 72103MB | Time Left: 03 hrs : 56 mins
627
+ [TIMING] data: 4.8ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.6ms
628
+ step 286 | loss: 1.663844 | lr 1.8000e-04 | norm: 0.8886 | dt: 7827.00ms | tok/sec: 66984.6 | mem: 72103MB | Time Left: 03 hrs : 56 mins
629
+ [TIMING] data: 4.8ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5366.4ms | clip: 0.0ms | optim: 23.7ms
630
+ step 287 | loss: 1.697726 | lr 1.8000e-04 | norm: 1.0103 | dt: 7832.62ms | tok/sec: 66936.5 | mem: 72103MB | Time Left: 03 hrs : 56 mins
631
+ [TIMING] data: 4.8ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
632
+ step 288 | loss: 1.808231 | lr 1.8000e-04 | norm: 0.9205 | dt: 7825.12ms | tok/sec: 67000.6 | mem: 72103MB | Time Left: 03 hrs : 56 mins
633
+ [TIMING] data: 4.7ms | fwd: 2411.4ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.7ms
634
+ step 289 | loss: 1.785199 | lr 1.8000e-04 | norm: 0.9862 | dt: 7828.24ms | tok/sec: 66973.9 | mem: 72103MB | Time Left: 03 hrs : 56 mins
635
+ [TIMING] data: 4.8ms | fwd: 2414.4ms | loss: 4.9ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.6ms
636
+ step 290 | loss: 1.751698 | lr 1.8000e-04 | norm: 0.9281 | dt: 7833.58ms | tok/sec: 66928.3 | mem: 72103MB | Time Left: 03 hrs : 56 mins
637
+ [TIMING] data: 5.0ms | fwd: 2417.0ms | loss: 4.8ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.6ms
638
+ step 291 | loss: 1.654229 | lr 1.8000e-04 | norm: 1.0358 | dt: 7823.46ms | tok/sec: 67014.9 | mem: 72103MB | Time Left: 03 hrs : 55 mins
639
+ [TIMING] data: 4.9ms | fwd: 2414.9ms | loss: 4.8ms | bwd: 5365.2ms | clip: 0.0ms | optim: 23.6ms
640
+ step 292 | loss: 1.694017 | lr 1.8000e-04 | norm: 0.9111 | dt: 7817.90ms | tok/sec: 67062.6 | mem: 72103MB | Time Left: 03 hrs : 55 mins
641
+ [TIMING] data: 4.6ms | fwd: 2409.4ms | loss: 4.8ms | bwd: 5364.6ms | clip: 0.0ms | optim: 23.7ms
642
+ step 293 | loss: 1.738502 | lr 1.8000e-04 | norm: 0.9122 | dt: 7834.36ms | tok/sec: 66921.6 | mem: 72103MB | Time Left: 03 hrs : 55 mins
643
+ [TIMING] data: 4.7ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5374.1ms | clip: 0.0ms | optim: 23.7ms
644
+ step 294 | loss: 1.743377 | lr 1.8000e-04 | norm: 0.9709 | dt: 7830.94ms | tok/sec: 66950.9 | mem: 72103MB | Time Left: 03 hrs : 55 mins
645
+ [TIMING] data: 4.6ms | fwd: 2414.7ms | loss: 4.8ms | bwd: 5373.5ms | clip: 0.0ms | optim: 23.5ms
646
+ step 295 | loss: 1.714503 | lr 1.8000e-04 | norm: 1.0028 | dt: 7832.42ms | tok/sec: 66938.1 | mem: 72103MB | Time Left: 03 hrs : 55 mins
647
+ [TIMING] data: 4.9ms | fwd: 2422.4ms | loss: 4.8ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
648
+ step 296 | loss: 1.547332 | lr 1.8000e-04 | norm: 1.1973 | dt: 7820.53ms | tok/sec: 67040.0 | mem: 72103MB | Time Left: 03 hrs : 55 mins
649
+ [TIMING] data: 4.9ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.6ms
650
+ step 297 | loss: 1.803487 | lr 1.8000e-04 | norm: 1.1564 | dt: 7826.04ms | tok/sec: 66992.7 | mem: 72103MB | Time Left: 03 hrs : 55 mins
651
+ [TIMING] data: 4.8ms | fwd: 2415.8ms | loss: 4.8ms | bwd: 5366.8ms | clip: 0.0ms | optim: 23.7ms
652
+ step 298 | loss: 1.749850 | lr 1.8000e-04 | norm: 1.0954 | dt: 7816.46ms | tok/sec: 67074.9 | mem: 72103MB | Time Left: 03 hrs : 54 mins
653
+ [TIMING] data: 4.8ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5363.5ms | clip: 0.0ms | optim: 23.7ms
654
+ step 299 | loss: 1.528509 | lr 1.8000e-04 | norm: 0.9575 | dt: 7821.90ms | tok/sec: 67028.2 | mem: 72103MB | Time Left: 03 hrs : 54 mins
655
+ [TIMING] data: 4.7ms | fwd: 2411.5ms | loss: 4.9ms | bwd: 5368.1ms | clip: 0.0ms | optim: 23.6ms
656
+ step 300 | loss: 1.689287 | lr 1.8000e-04 | norm: 1.0337 | dt: 7834.38ms | tok/sec: 66921.4 | mem: 72103MB | Time Left: 03 hrs : 55 mins
657
+ [TIMING] data: 4.8ms | fwd: 2423.3ms | loss: 4.8ms | bwd: 5367.0ms | clip: 0.0ms | optim: 23.6ms
658
+ step 301 | loss: 1.591161 | lr 1.8000e-04 | norm: 0.9519 | dt: 7830.91ms | tok/sec: 66951.1 | mem: 72103MB | Time Left: 03 hrs : 54 mins
659
+ [TIMING] data: 4.7ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.6ms
660
+ step 302 | loss: 1.621496 | lr 1.8000e-04 | norm: 0.9707 | dt: 7832.02ms | tok/sec: 66941.6 | mem: 72103MB | Time Left: 03 hrs : 54 mins
661
+ [TIMING] data: 4.8ms | fwd: 2420.6ms | loss: 4.9ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.7ms
662
+ step 303 | loss: 1.690582 | lr 1.8000e-04 | norm: 1.0774 | dt: 7830.73ms | tok/sec: 66952.6 | mem: 72103MB | Time Left: 03 hrs : 54 mins
663
+ [TIMING] data: 4.8ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.7ms
664
+ step 304 | loss: 1.548516 | lr 1.8000e-04 | norm: 1.1629 | dt: 7832.97ms | tok/sec: 66933.5 | mem: 72103MB | Time Left: 03 hrs : 54 mins
665
+ [TIMING] data: 4.9ms | fwd: 2414.6ms | loss: 4.8ms | bwd: 5375.1ms | clip: 0.0ms | optim: 23.5ms
666
+ step 305 | loss: 1.640994 | lr 1.8000e-04 | norm: 0.9619 | dt: 7833.52ms | tok/sec: 66928.8 | mem: 72103MB | Time Left: 03 hrs : 54 mins
667
+ [TIMING] data: 4.9ms | fwd: 2415.7ms | loss: 4.8ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
668
+ step 306 | loss: 1.644059 | lr 1.8000e-04 | norm: 0.9927 | dt: 7832.93ms | tok/sec: 66933.9 | mem: 72103MB | Time Left: 03 hrs : 54 mins
669
+ [TIMING] data: 4.9ms | fwd: 2420.6ms | loss: 4.9ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.5ms
670
+ step 307 | loss: 1.726431 | lr 1.8000e-04 | norm: 1.0415 | dt: 7833.70ms | tok/sec: 66927.3 | mem: 72103MB | Time Left: 03 hrs : 54 mins
671
+ [TIMING] data: 4.9ms | fwd: 2426.4ms | loss: 4.9ms | bwd: 5363.4ms | clip: 0.0ms | optim: 23.6ms
672
+ step 308 | loss: 1.540038 | lr 1.8000e-04 | norm: 1.0317 | dt: 7831.07ms | tok/sec: 66949.8 | mem: 72103MB | Time Left: 03 hrs : 53 mins
673
+ [TIMING] data: 4.9ms | fwd: 2420.6ms | loss: 4.8ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.6ms
674
+ step 309 | loss: 1.591404 | lr 1.8000e-04 | norm: 1.0330 | dt: 7826.58ms | tok/sec: 66988.1 | mem: 72103MB | Time Left: 03 hrs : 53 mins
675
+ [TIMING] data: 4.8ms | fwd: 2420.3ms | loss: 4.8ms | bwd: 5363.0ms | clip: 0.0ms | optim: 23.7ms
676
+ step 310 | loss: 1.633510 | lr 1.8000e-04 | norm: 1.1299 | dt: 7828.29ms | tok/sec: 66973.5 | mem: 72103MB | Time Left: 03 hrs : 53 mins
677
+ [TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.9ms | bwd: 5368.9ms | clip: 0.0ms | optim: 23.7ms
678
+ step 311 | loss: 1.665028 | lr 1.8000e-04 | norm: 1.1354 | dt: 7829.07ms | tok/sec: 66966.8 | mem: 72103MB | Time Left: 03 hrs : 53 mins
679
+ [TIMING] data: 4.8ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.6ms
680
+ step 312 | loss: 1.801152 | lr 1.8000e-04 | norm: 1.1436 | dt: 7821.64ms | tok/sec: 67030.5 | mem: 72103MB | Time Left: 03 hrs : 53 mins
681
+ [TIMING] data: 4.8ms | fwd: 2413.8ms | loss: 4.8ms | bwd: 5364.7ms | clip: 0.0ms | optim: 23.7ms
682
+ step 313 | loss: 1.527494 | lr 1.8000e-04 | norm: 1.0534 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 03 hrs : 53 mins
683
+ [TIMING] data: 5.0ms | fwd: 2413.5ms | loss: 4.8ms | bwd: 5371.1ms | clip: 0.0ms | optim: 23.6ms
684
+ step 314 | loss: 1.497566 | lr 1.8000e-04 | norm: 1.1055 | dt: 7827.65ms | tok/sec: 66979.0 | mem: 72103MB | Time Left: 03 hrs : 53 mins
685
+ [TIMING] data: 4.9ms | fwd: 2420.2ms | loss: 4.8ms | bwd: 5364.6ms | clip: 0.0ms | optim: 23.6ms
686
+ step 315 | loss: 1.600274 | lr 1.8000e-04 | norm: 1.1916 | dt: 7826.63ms | tok/sec: 66987.7 | mem: 72103MB | Time Left: 03 hrs : 52 mins
687
+ [TIMING] data: 5.0ms | fwd: 2413.3ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.7ms
688
+ step 316 | loss: 1.650586 | lr 1.8000e-04 | norm: 1.2763 | dt: 7829.62ms | tok/sec: 66962.1 | mem: 72103MB | Time Left: 03 hrs : 52 mins
689
+ [TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
690
+ step 317 | loss: 1.376261 | lr 1.8000e-04 | norm: 1.0453 | dt: 7825.58ms | tok/sec: 66996.7 | mem: 72103MB | Time Left: 03 hrs : 52 mins
691
+ [TIMING] data: 4.8ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5365.5ms | clip: 0.0ms | optim: 23.7ms
692
+ step 318 | loss: 1.670848 | lr 1.8000e-04 | norm: 1.1326 | dt: 7828.62ms | tok/sec: 66970.7 | mem: 72103MB | Time Left: 03 hrs : 52 mins
693
+ [TIMING] data: 4.9ms | fwd: 2415.0ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
694
+ step 319 | loss: 1.627407 | lr 1.8000e-04 | norm: 1.1636 | dt: 7824.08ms | tok/sec: 67009.5 | mem: 72103MB | Time Left: 03 hrs : 52 mins
695
+ [TIMING] data: 4.9ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5364.9ms | clip: 0.0ms | optim: 23.6ms
696
+ step 320 | loss: 1.483035 | lr 1.8000e-04 | norm: 1.0456 | dt: 7829.48ms | tok/sec: 66963.3 | mem: 72103MB | Time Left: 03 hrs : 52 mins
697
+ [TIMING] data: 5.0ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5368.7ms | clip: 0.0ms | optim: 23.6ms
698
+ step 321 | loss: 1.488142 | lr 1.8000e-04 | norm: 1.2098 | dt: 7829.83ms | tok/sec: 66960.3 | mem: 72103MB | Time Left: 03 hrs : 52 mins
699
+ [TIMING] data: 4.9ms | fwd: 2417.6ms | loss: 4.9ms | bwd: 5368.4ms | clip: 0.0ms | optim: 23.7ms
700
+ step 322 | loss: 1.566001 | lr 1.8000e-04 | norm: 1.1490 | dt: 7822.35ms | tok/sec: 67024.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
701
+ [TIMING] data: 4.6ms | fwd: 2406.0ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.6ms
702
+ step 323 | loss: 1.489413 | lr 1.8000e-04 | norm: 1.1245 | dt: 7830.54ms | tok/sec: 66954.2 | mem: 72103MB | Time Left: 03 hrs : 51 mins
703
+ [TIMING] data: 4.8ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.6ms
704
+ step 324 | loss: 1.493618 | lr 1.8000e-04 | norm: 1.0603 | dt: 7830.81ms | tok/sec: 66952.0 | mem: 72103MB | Time Left: 03 hrs : 51 mins
705
+ [TIMING] data: 4.8ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5377.1ms | clip: 0.0ms | optim: 23.6ms
706
+ step 325 | loss: 1.401132 | lr 1.8000e-04 | norm: 1.2034 | dt: 7830.52ms | tok/sec: 66954.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
707
+ [TIMING] data: 4.8ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
708
+ step 326 | loss: 1.459075 | lr 1.8000e-04 | norm: 1.0744 | dt: 7823.87ms | tok/sec: 67011.3 | mem: 72103MB | Time Left: 03 hrs : 51 mins
709
+ [TIMING] data: 4.6ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.6ms
710
+ step 327 | loss: 1.541776 | lr 1.8000e-04 | norm: 1.0930 | dt: 7830.52ms | tok/sec: 66954.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
711
+ [TIMING] data: 4.9ms | fwd: 2415.9ms | loss: 4.8ms | bwd: 5370.8ms | clip: 0.0ms | optim: 23.6ms
712
+ step 328 | loss: 1.294119 | lr 1.8000e-04 | norm: 1.0159 | dt: 7830.85ms | tok/sec: 66951.6 | mem: 72103MB | Time Left: 03 hrs : 51 mins
713
+ [TIMING] data: 4.9ms | fwd: 2418.7ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.6ms
714
+ step 329 | loss: 1.483784 | lr 1.8000e-04 | norm: 1.1200 | dt: 7829.13ms | tok/sec: 66966.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
715
+ [TIMING] data: 4.8ms | fwd: 2418.8ms | loss: 4.8ms | bwd: 5368.0ms | clip: 0.0ms | optim: 23.6ms
716
+ step 330 | loss: 1.438243 | lr 1.8000e-04 | norm: 1.0510 | dt: 7828.59ms | tok/sec: 66970.9 | mem: 72103MB | Time Left: 03 hrs : 50 mins
717
+ [TIMING] data: 4.8ms | fwd: 2419.1ms | loss: 4.8ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
718
+ step 331 | loss: 1.616356 | lr 1.8000e-04 | norm: 1.3004 | dt: 7822.71ms | tok/sec: 67021.3 | mem: 72103MB | Time Left: 03 hrs : 50 mins
719
+ [TIMING] data: 4.7ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.6ms
720
+ step 332 | loss: 1.431027 | lr 1.8000e-04 | norm: 1.2038 | dt: 7828.60ms | tok/sec: 66970.9 | mem: 72103MB | Time Left: 03 hrs : 50 mins
721
+ [TIMING] data: 4.9ms | fwd: 2413.6ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.7ms
722
+ step 333 | loss: 1.425855 | lr 1.8000e-04 | norm: 1.1500 | dt: 7828.50ms | tok/sec: 66971.7 | mem: 72103MB | Time Left: 03 hrs : 50 mins
723
+ [TIMING] data: 4.8ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.6ms
724
+ step 334 | loss: 1.451784 | lr 1.8000e-04 | norm: 1.1428 | dt: 7835.18ms | tok/sec: 66914.6 | mem: 72103MB | Time Left: 03 hrs : 50 mins
725
+ [TIMING] data: 4.9ms | fwd: 2422.4ms | loss: 4.8ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.7ms
726
+ step 335 | loss: 1.505677 | lr 1.8000e-04 | norm: 1.2136 | dt: 7826.60ms | tok/sec: 66988.0 | mem: 72103MB | Time Left: 03 hrs : 50 mins
727
+ [TIMING] data: 4.8ms | fwd: 2409.7ms | loss: 4.8ms | bwd: 5373.4ms | clip: 0.0ms | optim: 23.7ms
728
+ step 336 | loss: 1.482544 | lr 1.8000e-04 | norm: 1.1122 | dt: 7829.84ms | tok/sec: 66960.2 | mem: 72103MB | Time Left: 03 hrs : 50 mins
729
+ [TIMING] data: 4.7ms | fwd: 2415.8ms | loss: 4.9ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
730
+ step 337 | loss: 1.457365 | lr 1.8000e-04 | norm: 1.1931 | dt: 7830.49ms | tok/sec: 66954.7 | mem: 72103MB | Time Left: 03 hrs : 50 mins
731
+ [TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.7ms
732
+ step 338 | loss: 1.440003 | lr 1.8000e-04 | norm: 1.1547 | dt: 7830.54ms | tok/sec: 66954.2 | mem: 72103MB | Time Left: 03 hrs : 49 mins
733
+ [TIMING] data: 4.8ms | fwd: 2415.6ms | loss: 4.8ms | bwd: 5372.3ms | clip: 0.0ms | optim: 23.6ms