Upload logs/mixcont/train.log with huggingface_hub
Browse files- logs/mixcont/train.log +733 -0
logs/mixcont/train.log
ADDED
|
@@ -0,0 +1,733 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== Llama Context Parallel Training Script (bluscriptCP) ===
|
| 2 |
+
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
|
| 3 |
+
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
|
| 4 |
+
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
|
| 5 |
+
================ Configuration (bluscriptCP) ================
|
| 6 |
+
d_model/layers : 1536 / 36
|
| 7 |
+
q/kv heads : 12 / 4 (G=3, hd=128)
|
| 8 |
+
ffn_hidden : 4096 (SwiGLU)
|
| 9 |
+
context_length : 65536
|
| 10 |
+
vocab_size : 50304 tying=0
|
| 11 |
+
B x T : 1 x 65536
|
| 12 |
+
world_size : 8 (dp=1 x cp=8)
|
| 13 |
+
CP_ATTN_MODE : ring (load_balance=HeadTail)
|
| 14 |
+
global_batch : 524288 (grad_accum=8)
|
| 15 |
+
precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
|
| 16 |
+
max_lr/min_lr : 0.0006 / 6e-05
|
| 17 |
+
warmup/steps : 100 / 2100 (WSD: stable=1998, decay tail=2)
|
| 18 |
+
checkpointing : true (ckpt_freq=200, dir=checkpoints_bluscriptcp)
|
| 19 |
+
=============================================================
|
| 20 |
+
Initializing model on CUDA device 0...
|
| 21 |
+
[YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693
|
| 22 |
+
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/mix64k_step1200.ckpt (base step 1200, loss 2.81631) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
|
| 23 |
+
Parameters: 1060624896
|
| 24 |
+
Parameters per GPU: 1060624896
|
| 25 |
+
max_steps: 2100
|
| 26 |
+
warmup_steps: 100
|
| 27 |
+
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
|
| 28 |
+
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/-
|
| 29 |
+
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
|
| 30 |
+
[ZeRO-1 v2] params=983236608 padded=983236608 W=8 shard=122904576 | 38 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 1406.53 MB (vs 11252.2 MB unsharded)
|
| 31 |
+
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
|
| 32 |
+
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
|
| 33 |
+
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/-
|
| 34 |
+
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
|
| 35 |
+
[ZeRO-1 v2] params=77388288 padded=77388288 W=8 shard=9673536 | 6 buckets of 50 MB (1638400 elems/rank/bucket) | optimizer state/GPU: 110.705 MB (vs 885.639 MB unsharded)
|
| 36 |
+
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
|
| 37 |
+
[ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288]
|
| 38 |
+
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
|
| 39 |
+
[precision] bf16 params=217, all with fp32 master weights
|
| 40 |
+
found 12 shards for split train
|
| 41 |
+
found 1 shards for split val
|
| 42 |
+
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (run 2)
|
| 43 |
+
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
|
| 44 |
+
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 200 steps, keep 2)
|
| 45 |
+
[re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
|
| 46 |
+
|
| 47 |
+
Starting training...
|
| 48 |
+
|
| 49 |
+
[CP ring P2P] OVERLAP: dedicated non-blocking stream
|
| 50 |
+
validation loss: 2.9006
|
| 51 |
+
[CP bwd] overlap=ON
|
| 52 |
+
step 0 | loss: 2.849251 | lr 1.8000e-06 | norm: 0.4355 | dt: 16909.51ms | tok/sec: 31005.5 | mem: 70439MB | Time Left: 09 hrs : 51 mins
|
| 53 |
+
[TIMING] data: 3.6ms | fwd: 3730.6ms | loss: 4.9ms | bwd: 5624.6ms | clip: 0.0ms | optim: 24.3ms
|
| 54 |
+
step 1 | loss: 2.771659 | lr 3.6000e-06 | norm: 0.2168 | dt: 8944.07ms | tok/sec: 58618.5 | mem: 72103MB | Time Left: 05 hrs : 12 mins
|
| 55 |
+
[TIMING] data: 4.2ms | fwd: 3546.0ms | loss: 4.8ms | bwd: 5355.9ms | clip: 0.0ms | optim: 23.5ms
|
| 56 |
+
step 2 | loss: 3.129549 | lr 5.4000e-06 | norm: 0.3180 | dt: 8979.37ms | tok/sec: 58388.0 | mem: 72103MB | Time Left: 05 hrs : 13 mins
|
| 57 |
+
[TIMING] data: 3.8ms | fwd: 3529.5ms | loss: 4.8ms | bwd: 5408.4ms | clip: 0.0ms | optim: 23.7ms
|
| 58 |
+
step 3 | loss: 2.757302 | lr 7.2000e-06 | norm: 0.1858 | dt: 7794.04ms | tok/sec: 67267.8 | mem: 72103MB | Time Left: 04 hrs : 32 mins
|
| 59 |
+
[TIMING] data: 4.6ms | fwd: 2399.5ms | loss: 4.8ms | bwd: 5352.5ms | clip: 0.0ms | optim: 23.5ms
|
| 60 |
+
step 4 | loss: 3.064898 | lr 9.0000e-06 | norm: 0.2993 | dt: 7782.92ms | tok/sec: 67364.0 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 61 |
+
[TIMING] data: 4.3ms | fwd: 2392.4ms | loss: 4.8ms | bwd: 5348.6ms | clip: 0.0ms | optim: 23.5ms
|
| 62 |
+
step 5 | loss: 2.849279 | lr 1.0800e-05 | norm: 0.1736 | dt: 7793.06ms | tok/sec: 67276.3 | mem: 72103MB | Time Left: 04 hrs : 32 mins
|
| 63 |
+
[TIMING] data: 4.5ms | fwd: 2402.2ms | loss: 4.8ms | bwd: 5348.8ms | clip: 0.0ms | optim: 23.5ms
|
| 64 |
+
step 6 | loss: 2.705969 | lr 1.2600e-05 | norm: 0.1518 | dt: 7795.74ms | tok/sec: 67253.1 | mem: 72103MB | Time Left: 04 hrs : 32 mins
|
| 65 |
+
[TIMING] data: 4.3ms | fwd: 2402.7ms | loss: 4.9ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.5ms
|
| 66 |
+
step 7 | loss: 2.657195 | lr 1.4400e-05 | norm: 0.1932 | dt: 7804.29ms | tok/sec: 67179.4 | mem: 72103MB | Time Left: 04 hrs : 32 mins
|
| 67 |
+
[TIMING] data: 4.5ms | fwd: 2404.7ms | loss: 4.9ms | bwd: 5357.5ms | clip: 0.0ms | optim: 23.5ms
|
| 68 |
+
step 8 | loss: 2.967716 | lr 1.6200e-05 | norm: 0.1973 | dt: 7796.25ms | tok/sec: 67248.7 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 69 |
+
[TIMING] data: 4.4ms | fwd: 2401.1ms | loss: 4.8ms | bwd: 5353.0ms | clip: 0.0ms | optim: 23.6ms
|
| 70 |
+
step 9 | loss: 3.006548 | lr 1.8000e-05 | norm: 0.1757 | dt: 7807.98ms | tok/sec: 67147.7 | mem: 72103MB | Time Left: 04 hrs : 32 mins
|
| 71 |
+
[TIMING] data: 4.5ms | fwd: 2411.8ms | loss: 4.8ms | bwd: 5353.5ms | clip: 0.0ms | optim: 23.5ms
|
| 72 |
+
step 10 | loss: 2.919900 | lr 1.9800e-05 | norm: 0.1724 | dt: 7800.65ms | tok/sec: 67210.8 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 73 |
+
[TIMING] data: 4.3ms | fwd: 2406.5ms | loss: 4.8ms | bwd: 5350.9ms | clip: 0.0ms | optim: 23.6ms
|
| 74 |
+
step 11 | loss: 2.718448 | lr 2.1600e-05 | norm: 0.2161 | dt: 7813.40ms | tok/sec: 67101.1 | mem: 72103MB | Time Left: 04 hrs : 32 mins
|
| 75 |
+
[TIMING] data: 4.5ms | fwd: 2412.1ms | loss: 4.8ms | bwd: 5358.0ms | clip: 0.0ms | optim: 23.5ms
|
| 76 |
+
step 12 | loss: 2.775069 | lr 2.3400e-05 | norm: 0.1755 | dt: 7816.31ms | tok/sec: 67076.2 | mem: 72103MB | Time Left: 04 hrs : 32 mins
|
| 77 |
+
[TIMING] data: 4.4ms | fwd: 2419.5ms | loss: 4.8ms | bwd: 5354.2ms | clip: 0.0ms | optim: 23.8ms
|
| 78 |
+
step 13 | loss: 2.722095 | lr 2.5200e-05 | norm: 0.1907 | dt: 7816.41ms | tok/sec: 67075.3 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 79 |
+
[TIMING] data: 4.3ms | fwd: 2417.4ms | loss: 4.8ms | bwd: 5355.2ms | clip: 0.0ms | optim: 23.5ms
|
| 80 |
+
step 14 | loss: 2.614105 | lr 2.7000e-05 | norm: 0.1767 | dt: 7812.97ms | tok/sec: 67104.8 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 81 |
+
[TIMING] data: 4.3ms | fwd: 2410.9ms | loss: 4.8ms | bwd: 5358.4ms | clip: 0.0ms | optim: 23.6ms
|
| 82 |
+
step 15 | loss: 2.822798 | lr 2.8800e-05 | norm: 0.1889 | dt: 7814.98ms | tok/sec: 67087.6 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 83 |
+
[TIMING] data: 4.4ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5353.3ms | clip: 0.0ms | optim: 23.6ms
|
| 84 |
+
step 16 | loss: 2.910059 | lr 3.0600e-05 | norm: 0.2202 | dt: 7827.24ms | tok/sec: 66982.5 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 85 |
+
[TIMING] data: 4.6ms | fwd: 2422.1ms | loss: 4.8ms | bwd: 5362.1ms | clip: 0.0ms | optim: 23.7ms
|
| 86 |
+
step 17 | loss: 2.705841 | lr 3.2400e-05 | norm: 0.1678 | dt: 7827.25ms | tok/sec: 66982.4 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 87 |
+
[TIMING] data: 4.1ms | fwd: 2420.7ms | loss: 4.8ms | bwd: 5363.1ms | clip: 0.0ms | optim: 23.5ms
|
| 88 |
+
step 18 | loss: 2.806966 | lr 3.4200e-05 | norm: 0.2113 | dt: 7815.94ms | tok/sec: 67079.3 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 89 |
+
[TIMING] data: 4.0ms | fwd: 2412.7ms | loss: 4.8ms | bwd: 5360.7ms | clip: 0.0ms | optim: 23.6ms
|
| 90 |
+
step 19 | loss: 2.786724 | lr 3.6000e-05 | norm: 0.1685 | dt: 7820.28ms | tok/sec: 67042.1 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 91 |
+
[TIMING] data: 4.5ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5360.5ms | clip: 0.0ms | optim: 23.8ms
|
| 92 |
+
step 20 | loss: 2.494313 | lr 3.7800e-05 | norm: 0.1415 | dt: 7827.51ms | tok/sec: 66980.2 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 93 |
+
[TIMING] data: 4.5ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5364.1ms | clip: 0.0ms | optim: 23.6ms
|
| 94 |
+
step 21 | loss: 2.708891 | lr 3.9600e-05 | norm: 0.1796 | dt: 7828.57ms | tok/sec: 66971.1 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 95 |
+
[TIMING] data: 4.5ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5367.2ms | clip: 0.0ms | optim: 23.6ms
|
| 96 |
+
step 22 | loss: 2.863696 | lr 4.1400e-05 | norm: 0.2196 | dt: 7829.81ms | tok/sec: 66960.5 | mem: 72103MB | Time Left: 04 hrs : 31 mins
|
| 97 |
+
[TIMING] data: 4.5ms | fwd: 2421.7ms | loss: 4.9ms | bwd: 5363.9ms | clip: 0.0ms | optim: 23.6ms
|
| 98 |
+
step 23 | loss: 2.526138 | lr 4.3200e-05 | norm: 0.1477 | dt: 7822.01ms | tok/sec: 67027.2 | mem: 72103MB | Time Left: 04 hrs : 30 mins
|
| 99 |
+
[TIMING] data: 3.9ms | fwd: 2408.6ms | loss: 4.9ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
|
| 100 |
+
step 24 | loss: 2.931958 | lr 4.5000e-05 | norm: 0.1709 | dt: 7826.38ms | tok/sec: 66989.9 | mem: 72103MB | Time Left: 04 hrs : 30 mins
|
| 101 |
+
[TIMING] data: 4.2ms | fwd: 2416.1ms | loss: 4.9ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
|
| 102 |
+
step 25 | loss: 2.684256 | lr 4.6800e-05 | norm: 0.1464 | dt: 7827.49ms | tok/sec: 66980.4 | mem: 72103MB | Time Left: 04 hrs : 30 mins
|
| 103 |
+
[TIMING] data: 4.2ms | fwd: 2418.6ms | loss: 4.8ms | bwd: 5366.6ms | clip: 0.0ms | optim: 23.5ms
|
| 104 |
+
step 26 | loss: 2.781745 | lr 4.8600e-05 | norm: 0.1492 | dt: 7825.23ms | tok/sec: 66999.7 | mem: 72103MB | Time Left: 04 hrs : 30 mins
|
| 105 |
+
[TIMING] data: 4.4ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5361.1ms | clip: 0.0ms | optim: 23.6ms
|
| 106 |
+
step 27 | loss: 2.540646 | lr 5.0400e-05 | norm: 0.1553 | dt: 7818.08ms | tok/sec: 67060.9 | mem: 72103MB | Time Left: 04 hrs : 30 mins
|
| 107 |
+
[TIMING] data: 4.4ms | fwd: 2412.4ms | loss: 4.8ms | bwd: 5362.4ms | clip: 0.0ms | optim: 23.6ms
|
| 108 |
+
step 28 | loss: 2.775252 | lr 5.2200e-05 | norm: 0.2817 | dt: 7846.10ms | tok/sec: 66821.4 | mem: 72103MB | Time Left: 04 hrs : 30 mins
|
| 109 |
+
[TIMING] data: 4.5ms | fwd: 2427.4ms | loss: 4.9ms | bwd: 5373.9ms | clip: 0.0ms | optim: 23.5ms
|
| 110 |
+
step 29 | loss: 2.888937 | lr 5.4000e-05 | norm: 0.1731 | dt: 7821.65ms | tok/sec: 67030.3 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 111 |
+
[TIMING] data: 4.4ms | fwd: 2412.0ms | loss: 4.9ms | bwd: 5367.7ms | clip: 0.0ms | optim: 23.6ms
|
| 112 |
+
step 30 | loss: 2.655073 | lr 5.5800e-05 | norm: 0.1691 | dt: 7820.50ms | tok/sec: 67040.2 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 113 |
+
[TIMING] data: 4.4ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5365.3ms | clip: 0.0ms | optim: 23.6ms
|
| 114 |
+
step 31 | loss: 2.852468 | lr 5.7600e-05 | norm: 0.1892 | dt: 7830.83ms | tok/sec: 66951.8 | mem: 72103MB | Time Left: 04 hrs : 30 mins
|
| 115 |
+
[TIMING] data: 4.5ms | fwd: 2418.9ms | loss: 4.9ms | bwd: 5368.1ms | clip: 0.0ms | optim: 23.7ms
|
| 116 |
+
step 32 | loss: 2.525090 | lr 5.9400e-05 | norm: 0.1397 | dt: 7827.47ms | tok/sec: 66980.5 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 117 |
+
[TIMING] data: 4.3ms | fwd: 2420.0ms | loss: 4.8ms | bwd: 5365.3ms | clip: 0.0ms | optim: 23.6ms
|
| 118 |
+
step 33 | loss: 2.819749 | lr 6.1200e-05 | norm: 0.1496 | dt: 7833.34ms | tok/sec: 66930.4 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 119 |
+
[TIMING] data: 4.5ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.6ms
|
| 120 |
+
step 34 | loss: 2.692123 | lr 6.3000e-05 | norm: 0.1606 | dt: 7827.90ms | tok/sec: 66976.8 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 121 |
+
[TIMING] data: 4.5ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.6ms
|
| 122 |
+
step 35 | loss: 2.642615 | lr 6.4800e-05 | norm: 0.1714 | dt: 7828.29ms | tok/sec: 66973.5 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 123 |
+
[TIMING] data: 4.6ms | fwd: 2412.6ms | loss: 4.8ms | bwd: 5372.8ms | clip: 0.0ms | optim: 23.6ms
|
| 124 |
+
step 36 | loss: 2.612672 | lr 6.6600e-05 | norm: 0.1762 | dt: 7832.23ms | tok/sec: 66939.8 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 125 |
+
[TIMING] data: 4.2ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
|
| 126 |
+
step 37 | loss: 2.965949 | lr 6.8400e-05 | norm: 0.1934 | dt: 7827.61ms | tok/sec: 66979.3 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 127 |
+
[TIMING] data: 4.6ms | fwd: 2419.2ms | loss: 4.8ms | bwd: 5365.4ms | clip: 0.0ms | optim: 23.7ms
|
| 128 |
+
step 38 | loss: 2.560090 | lr 7.0200e-05 | norm: 0.1511 | dt: 7830.21ms | tok/sec: 66957.1 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 129 |
+
[TIMING] data: 4.4ms | fwd: 2418.1ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
|
| 130 |
+
step 39 | loss: 2.770146 | lr 7.2000e-05 | norm: 0.1514 | dt: 7834.61ms | tok/sec: 66919.4 | mem: 72103MB | Time Left: 04 hrs : 29 mins
|
| 131 |
+
[TIMING] data: 4.3ms | fwd: 2420.4ms | loss: 4.9ms | bwd: 5371.7ms | clip: 0.0ms | optim: 23.6ms
|
| 132 |
+
step 40 | loss: 2.460374 | lr 7.3800e-05 | norm: 0.3389 | dt: 7830.25ms | tok/sec: 66956.8 | mem: 72103MB | Time Left: 04 hrs : 28 mins
|
| 133 |
+
[TIMING] data: 4.5ms | fwd: 2421.1ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.6ms
|
| 134 |
+
step 41 | loss: 2.800878 | lr 7.5600e-05 | norm: 0.1722 | dt: 7834.25ms | tok/sec: 66922.6 | mem: 72103MB | Time Left: 04 hrs : 28 mins
|
| 135 |
+
[TIMING] data: 4.1ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5375.6ms | clip: 0.0ms | optim: 23.8ms
|
| 136 |
+
step 42 | loss: 2.536062 | lr 7.7400e-05 | norm: 0.1610 | dt: 7829.64ms | tok/sec: 66961.9 | mem: 72103MB | Time Left: 04 hrs : 28 mins
|
| 137 |
+
[TIMING] data: 4.3ms | fwd: 2420.6ms | loss: 4.8ms | bwd: 5366.5ms | clip: 0.0ms | optim: 23.7ms
|
| 138 |
+
step 43 | loss: 2.705255 | lr 7.9200e-05 | norm: 0.2146 | dt: 7833.39ms | tok/sec: 66929.9 | mem: 72103MB | Time Left: 04 hrs : 28 mins
|
| 139 |
+
[TIMING] data: 4.4ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5375.6ms | clip: 0.0ms | optim: 23.6ms
|
| 140 |
+
step 44 | loss: 2.720131 | lr 8.1000e-05 | norm: 0.1717 | dt: 7827.08ms | tok/sec: 66983.9 | mem: 72103MB | Time Left: 04 hrs : 28 mins
|
| 141 |
+
[TIMING] data: 4.2ms | fwd: 2413.7ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
|
| 142 |
+
step 45 | loss: 2.595790 | lr 8.2800e-05 | norm: 0.1594 | dt: 7826.71ms | tok/sec: 66987.0 | mem: 72103MB | Time Left: 04 hrs : 28 mins
|
| 143 |
+
[TIMING] data: 4.1ms | fwd: 2414.8ms | loss: 4.9ms | bwd: 5369.0ms | clip: 0.0ms | optim: 24.2ms
|
| 144 |
+
step 46 | loss: 2.483078 | lr 8.4600e-05 | norm: 0.2118 | dt: 7827.47ms | tok/sec: 66980.6 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 145 |
+
[TIMING] data: 4.2ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5376.8ms | clip: 0.0ms | optim: 23.7ms
|
| 146 |
+
step 47 | loss: 2.459733 | lr 8.6400e-05 | norm: 0.1622 | dt: 7825.51ms | tok/sec: 66997.3 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 147 |
+
[TIMING] data: 4.4ms | fwd: 2409.4ms | loss: 4.8ms | bwd: 5373.7ms | clip: 0.0ms | optim: 23.7ms
|
| 148 |
+
step 48 | loss: 2.847660 | lr 8.8200e-05 | norm: 0.1493 | dt: 7827.34ms | tok/sec: 66981.7 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 149 |
+
[TIMING] data: 4.3ms | fwd: 2413.1ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.6ms
|
| 150 |
+
step 49 | loss: 2.732789 | lr 9.0000e-05 | norm: 0.1771 | dt: 7830.92ms | tok/sec: 66951.0 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 151 |
+
[TIMING] data: 4.2ms | fwd: 2418.7ms | loss: 4.8ms | bwd: 5369.2ms | clip: 0.0ms | optim: 23.8ms
|
| 152 |
+
step 50 | loss: 2.706898 | lr 9.1800e-05 | norm: 0.1449 | dt: 7836.75ms | tok/sec: 66901.2 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 153 |
+
[TIMING] data: 4.3ms | fwd: 2419.0ms | loss: 4.8ms | bwd: 5374.4ms | clip: 0.0ms | optim: 23.6ms
|
| 154 |
+
step 51 | loss: 2.795459 | lr 9.3600e-05 | norm: 0.1691 | dt: 7828.69ms | tok/sec: 66970.1 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 155 |
+
[TIMING] data: 4.5ms | fwd: 2419.7ms | loss: 4.8ms | bwd: 5366.6ms | clip: 0.0ms | optim: 23.6ms
|
| 156 |
+
step 52 | loss: 2.742548 | lr 9.5400e-05 | norm: 0.1969 | dt: 7839.53ms | tok/sec: 66877.5 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 157 |
+
[TIMING] data: 3.9ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5376.6ms | clip: 0.0ms | optim: 23.6ms
|
| 158 |
+
step 53 | loss: 2.690454 | lr 9.7200e-05 | norm: 0.1784 | dt: 7834.65ms | tok/sec: 66919.1 | mem: 72103MB | Time Left: 04 hrs : 27 mins
|
| 159 |
+
[TIMING] data: 4.1ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5372.4ms | clip: 0.0ms | optim: 23.6ms
|
| 160 |
+
step 54 | loss: 2.766409 | lr 9.9000e-05 | norm: 0.1395 | dt: 7827.36ms | tok/sec: 66981.4 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 161 |
+
[TIMING] data: 4.4ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.8ms
|
| 162 |
+
step 55 | loss: 2.815801 | lr 1.0080e-04 | norm: 0.1430 | dt: 7830.62ms | tok/sec: 66953.6 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 163 |
+
[TIMING] data: 4.4ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.6ms
|
| 164 |
+
step 56 | loss: 2.834039 | lr 1.0260e-04 | norm: 0.1686 | dt: 7831.58ms | tok/sec: 66945.3 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 165 |
+
[TIMING] data: 4.5ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.7ms
|
| 166 |
+
step 57 | loss: 2.557956 | lr 1.0440e-04 | norm: 0.2808 | dt: 7836.55ms | tok/sec: 66902.9 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 167 |
+
[TIMING] data: 4.4ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5374.3ms | clip: 0.0ms | optim: 23.6ms
|
| 168 |
+
step 58 | loss: 2.871943 | lr 1.0620e-04 | norm: 0.1714 | dt: 7833.31ms | tok/sec: 66930.6 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 169 |
+
[TIMING] data: 4.3ms | fwd: 2419.4ms | loss: 4.8ms | bwd: 5371.1ms | clip: 0.0ms | optim: 23.6ms
|
| 170 |
+
step 59 | loss: 2.658476 | lr 1.0800e-04 | norm: 0.1638 | dt: 7839.37ms | tok/sec: 66878.9 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 171 |
+
[TIMING] data: 4.5ms | fwd: 2421.6ms | loss: 4.8ms | bwd: 5373.9ms | clip: 0.0ms | optim: 23.8ms
|
| 172 |
+
step 60 | loss: 2.835381 | lr 1.0980e-04 | norm: 0.1557 | dt: 7827.91ms | tok/sec: 66976.7 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 173 |
+
[TIMING] data: 4.3ms | fwd: 2414.9ms | loss: 4.9ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.6ms
|
| 174 |
+
step 61 | loss: 2.534974 | lr 1.1160e-04 | norm: 0.1439 | dt: 7830.63ms | tok/sec: 66953.5 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 175 |
+
[TIMING] data: 4.2ms | fwd: 2417.4ms | loss: 4.8ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
|
| 176 |
+
step 62 | loss: 2.676892 | lr 1.1340e-04 | norm: 0.1422 | dt: 7838.12ms | tok/sec: 66889.5 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 177 |
+
[TIMING] data: 4.3ms | fwd: 2419.8ms | loss: 4.8ms | bwd: 5375.8ms | clip: 0.0ms | optim: 23.7ms
|
| 178 |
+
step 63 | loss: 2.652859 | lr 1.1520e-04 | norm: 0.1431 | dt: 7831.59ms | tok/sec: 66945.3 | mem: 72103MB | Time Left: 04 hrs : 25 mins
|
| 179 |
+
[TIMING] data: 4.3ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
|
| 180 |
+
step 64 | loss: 2.316111 | lr 1.1700e-04 | norm: 0.1357 | dt: 7846.61ms | tok/sec: 66817.1 | mem: 72103MB | Time Left: 04 hrs : 26 mins
|
| 181 |
+
[TIMING] data: 4.6ms | fwd: 2433.2ms | loss: 4.9ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.7ms
|
| 182 |
+
step 65 | loss: 2.710837 | lr 1.1880e-04 | norm: 0.1608 | dt: 7826.37ms | tok/sec: 66990.0 | mem: 72103MB | Time Left: 04 hrs : 25 mins
|
| 183 |
+
[TIMING] data: 4.4ms | fwd: 2410.7ms | loss: 4.8ms | bwd: 5373.9ms | clip: 0.0ms | optim: 23.6ms
|
| 184 |
+
step 66 | loss: 2.764815 | lr 1.2060e-04 | norm: 0.1603 | dt: 7831.60ms | tok/sec: 66945.2 | mem: 72103MB | Time Left: 04 hrs : 25 mins
|
| 185 |
+
[TIMING] data: 4.4ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5373.3ms | clip: 0.0ms | optim: 23.7ms
|
| 186 |
+
step 67 | loss: 2.645531 | lr 1.2240e-04 | norm: 0.1757 | dt: 7837.03ms | tok/sec: 66898.8 | mem: 72103MB | Time Left: 04 hrs : 25 mins
|
| 187 |
+
[TIMING] data: 4.1ms | fwd: 2422.6ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.5ms
|
| 188 |
+
step 68 | loss: 2.767838 | lr 1.2420e-04 | norm: 0.1765 | dt: 7837.10ms | tok/sec: 66898.2 | mem: 72103MB | Time Left: 04 hrs : 25 mins
|
| 189 |
+
[TIMING] data: 4.2ms | fwd: 2425.2ms | loss: 4.9ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.8ms
|
| 190 |
+
step 69 | loss: 2.503338 | lr 1.2600e-04 | norm: 0.1771 | dt: 7834.52ms | tok/sec: 66920.3 | mem: 72103MB | Time Left: 04 hrs : 25 mins
|
| 191 |
+
[TIMING] data: 4.4ms | fwd: 2420.1ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.6ms
|
| 192 |
+
step 70 | loss: 2.709606 | lr 1.2780e-04 | norm: 0.1686 | dt: 7832.23ms | tok/sec: 66939.8 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 193 |
+
[TIMING] data: 4.3ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.6ms
|
| 194 |
+
step 71 | loss: 2.525243 | lr 1.2960e-04 | norm: 0.1739 | dt: 7831.46ms | tok/sec: 66946.4 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 195 |
+
[TIMING] data: 4.5ms | fwd: 2414.2ms | loss: 4.8ms | bwd: 5374.9ms | clip: 0.0ms | optim: 23.7ms
|
| 196 |
+
step 72 | loss: 2.312968 | lr 1.3140e-04 | norm: 0.1493 | dt: 7832.00ms | tok/sec: 66941.8 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 197 |
+
[TIMING] data: 4.0ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5372.3ms | clip: 0.0ms | optim: 23.6ms
|
| 198 |
+
step 73 | loss: 2.664239 | lr 1.3320e-04 | norm: 0.1723 | dt: 7835.30ms | tok/sec: 66913.6 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 199 |
+
[TIMING] data: 4.3ms | fwd: 2419.5ms | loss: 4.8ms | bwd: 5372.0ms | clip: 0.0ms | optim: 23.6ms
|
| 200 |
+
step 74 | loss: 2.733030 | lr 1.3500e-04 | norm: 0.2010 | dt: 7818.45ms | tok/sec: 67057.8 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 201 |
+
[TIMING] data: 4.4ms | fwd: 2409.2ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.6ms
|
| 202 |
+
step 75 | loss: 2.704966 | lr 1.3680e-04 | norm: 0.1679 | dt: 7835.19ms | tok/sec: 66914.5 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 203 |
+
[TIMING] data: 4.2ms | fwd: 2421.1ms | loss: 4.9ms | bwd: 5371.0ms | clip: 0.0ms | optim: 23.7ms
|
| 204 |
+
step 76 | loss: 2.793722 | lr 1.3860e-04 | norm: 0.1678 | dt: 7830.99ms | tok/sec: 66950.4 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 205 |
+
[TIMING] data: 4.0ms | fwd: 2418.2ms | loss: 4.8ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.7ms
|
| 206 |
+
step 77 | loss: 2.929346 | lr 1.4040e-04 | norm: 0.1572 | dt: 7838.49ms | tok/sec: 66886.4 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 207 |
+
[TIMING] data: 4.2ms | fwd: 2422.8ms | loss: 4.8ms | bwd: 5373.0ms | clip: 0.0ms | optim: 23.6ms
|
| 208 |
+
step 78 | loss: 2.679340 | lr 1.4220e-04 | norm: 0.1998 | dt: 7842.84ms | tok/sec: 66849.3 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 209 |
+
[TIMING] data: 4.3ms | fwd: 2429.8ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
|
| 210 |
+
step 79 | loss: 2.682140 | lr 1.4400e-04 | norm: 0.1545 | dt: 7835.05ms | tok/sec: 66915.7 | mem: 72103MB | Time Left: 04 hrs : 23 mins
|
| 211 |
+
[TIMING] data: 4.3ms | fwd: 2415.7ms | loss: 4.9ms | bwd: 5377.3ms | clip: 0.0ms | optim: 23.6ms
|
| 212 |
+
step 80 | loss: 2.735668 | lr 1.4580e-04 | norm: 0.1551 | dt: 7844.24ms | tok/sec: 66837.3 | mem: 72103MB | Time Left: 04 hrs : 24 mins
|
| 213 |
+
[TIMING] data: 4.2ms | fwd: 2429.1ms | loss: 4.8ms | bwd: 5371.7ms | clip: 0.0ms | optim: 23.5ms
|
| 214 |
+
step 81 | loss: 2.668484 | lr 1.4760e-04 | norm: 0.1800 | dt: 7842.69ms | tok/sec: 66850.6 | mem: 72103MB | Time Left: 04 hrs : 23 mins
|
| 215 |
+
[TIMING] data: 4.5ms | fwd: 2425.2ms | loss: 4.9ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
|
| 216 |
+
step 82 | loss: 2.951952 | lr 1.4940e-04 | norm: 0.1710 | dt: 7830.36ms | tok/sec: 66955.8 | mem: 72103MB | Time Left: 04 hrs : 23 mins
|
| 217 |
+
[TIMING] data: 4.4ms | fwd: 2415.9ms | loss: 4.8ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.7ms
|
| 218 |
+
step 83 | loss: 2.750396 | lr 1.5120e-04 | norm: 0.1845 | dt: 7829.16ms | tok/sec: 66966.1 | mem: 72103MB | Time Left: 04 hrs : 23 mins
|
| 219 |
+
[TIMING] data: 4.3ms | fwd: 2415.9ms | loss: 4.9ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
|
| 220 |
+
step 84 | loss: 2.647009 | lr 1.5300e-04 | norm: 0.1744 | dt: 7831.67ms | tok/sec: 66944.6 | mem: 72103MB | Time Left: 04 hrs : 23 mins
|
| 221 |
+
[TIMING] data: 4.2ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5371.2ms | clip: 0.0ms | optim: 23.7ms
|
| 222 |
+
step 85 | loss: 2.710377 | lr 1.5480e-04 | norm: 0.1629 | dt: 7829.49ms | tok/sec: 66963.3 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 223 |
+
[TIMING] data: 4.1ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.6ms
|
| 224 |
+
step 86 | loss: 2.821142 | lr 1.5660e-04 | norm: 0.2439 | dt: 7835.38ms | tok/sec: 66912.9 | mem: 72103MB | Time Left: 04 hrs : 23 mins
|
| 225 |
+
[TIMING] data: 4.3ms | fwd: 2418.1ms | loss: 4.8ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.6ms
|
| 226 |
+
step 87 | loss: 2.683125 | lr 1.5840e-04 | norm: 0.1564 | dt: 7835.41ms | tok/sec: 66912.6 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 227 |
+
[TIMING] data: 4.4ms | fwd: 2420.5ms | loss: 4.8ms | bwd: 5372.4ms | clip: 0.0ms | optim: 23.6ms
|
| 228 |
+
step 88 | loss: 2.778752 | lr 1.6020e-04 | norm: 0.2029 | dt: 7840.16ms | tok/sec: 66872.1 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 229 |
+
[TIMING] data: 4.3ms | fwd: 2420.6ms | loss: 4.8ms | bwd: 5376.5ms | clip: 0.0ms | optim: 23.7ms
|
| 230 |
+
step 89 | loss: 2.816476 | lr 1.6200e-04 | norm: 0.1741 | dt: 7832.22ms | tok/sec: 66939.9 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 231 |
+
[TIMING] data: 4.4ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5375.1ms | clip: 0.0ms | optim: 23.7ms
|
| 232 |
+
step 90 | loss: 2.832433 | lr 1.6380e-04 | norm: 0.1604 | dt: 7842.76ms | tok/sec: 66849.9 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 233 |
+
[TIMING] data: 3.6ms | fwd: 2428.0ms | loss: 4.8ms | bwd: 5369.9ms | clip: 0.0ms | optim: 23.8ms
|
| 234 |
+
step 91 | loss: 2.820065 | lr 1.6560e-04 | norm: 0.1708 | dt: 7836.67ms | tok/sec: 66901.9 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 235 |
+
[TIMING] data: 4.0ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.7ms
|
| 236 |
+
step 92 | loss: 2.862764 | lr 1.6740e-04 | norm: 0.2114 | dt: 7835.34ms | tok/sec: 66913.3 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 237 |
+
[TIMING] data: 4.5ms | fwd: 2423.0ms | loss: 4.9ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.6ms
|
| 238 |
+
step 93 | loss: 2.749069 | lr 1.6920e-04 | norm: 0.1888 | dt: 7831.75ms | tok/sec: 66943.9 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 239 |
+
[TIMING] data: 4.4ms | fwd: 2421.2ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.7ms
|
| 240 |
+
step 94 | loss: 2.762149 | lr 1.7100e-04 | norm: 0.2443 | dt: 7834.51ms | tok/sec: 66920.3 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 241 |
+
[TIMING] data: 4.4ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5374.4ms | clip: 0.0ms | optim: 23.7ms
|
| 242 |
+
step 95 | loss: 2.682379 | lr 1.7280e-04 | norm: 0.2071 | dt: 7827.43ms | tok/sec: 66980.9 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 243 |
+
[TIMING] data: 4.1ms | fwd: 2415.4ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.5ms
|
| 244 |
+
step 96 | loss: 2.707476 | lr 1.7460e-04 | norm: 0.1956 | dt: 7830.37ms | tok/sec: 66955.7 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 245 |
+
[TIMING] data: 4.2ms | fwd: 2412.6ms | loss: 4.9ms | bwd: 5375.9ms | clip: 0.0ms | optim: 23.6ms
|
| 246 |
+
step 97 | loss: 2.821827 | lr 1.7640e-04 | norm: 0.2093 | dt: 7839.63ms | tok/sec: 66876.6 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 247 |
+
[TIMING] data: 4.2ms | fwd: 2420.5ms | loss: 4.8ms | bwd: 5376.0ms | clip: 0.0ms | optim: 23.6ms
|
| 248 |
+
step 98 | loss: 2.357723 | lr 1.7820e-04 | norm: 0.1789 | dt: 7833.07ms | tok/sec: 66932.6 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 249 |
+
[TIMING] data: 4.1ms | fwd: 2416.1ms | loss: 4.9ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
|
| 250 |
+
step 99 | loss: 2.664635 | lr 1.8000e-04 | norm: 0.1818 | dt: 7827.31ms | tok/sec: 66981.9 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 251 |
+
[TIMING] data: 4.1ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.8ms
|
| 252 |
+
step 100 | loss: 2.698034 | lr 1.8000e-04 | norm: 0.1752 | dt: 7831.88ms | tok/sec: 66942.8 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 253 |
+
[TIMING] data: 4.0ms | fwd: 2415.3ms | loss: 4.9ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.7ms
|
| 254 |
+
step 101 | loss: 2.768258 | lr 1.8000e-04 | norm: 0.1847 | dt: 7838.35ms | tok/sec: 66887.6 | mem: 72103MB | Time Left: 04 hrs : 21 mins
|
| 255 |
+
[TIMING] data: 4.2ms | fwd: 2421.0ms | loss: 4.8ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.6ms
|
| 256 |
+
step 102 | loss: 2.793727 | lr 1.8000e-04 | norm: 0.1797 | dt: 7826.91ms | tok/sec: 66985.3 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 257 |
+
[TIMING] data: 4.3ms | fwd: 2417.6ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.6ms
|
| 258 |
+
step 103 | loss: 2.776408 | lr 1.8000e-04 | norm: 0.1732 | dt: 7834.20ms | tok/sec: 66923.0 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 259 |
+
[TIMING] data: 4.4ms | fwd: 2417.4ms | loss: 4.9ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.8ms
|
| 260 |
+
step 104 | loss: 2.497193 | lr 1.8000e-04 | norm: 0.1782 | dt: 7836.05ms | tok/sec: 66907.2 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 261 |
+
[TIMING] data: 4.4ms | fwd: 2419.8ms | loss: 4.8ms | bwd: 5373.6ms | clip: 0.0ms | optim: 23.6ms
|
| 262 |
+
step 105 | loss: 3.192918 | lr 1.8000e-04 | norm: 0.1940 | dt: 7835.79ms | tok/sec: 66909.4 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 263 |
+
[TIMING] data: 4.5ms | fwd: 2409.9ms | loss: 4.8ms | bwd: 5383.0ms | clip: 0.0ms | optim: 23.7ms
|
| 264 |
+
step 106 | loss: 2.820851 | lr 1.8000e-04 | norm: 0.3302 | dt: 7831.24ms | tok/sec: 66948.3 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 265 |
+
[TIMING] data: 4.5ms | fwd: 2414.7ms | loss: 4.8ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.6ms
|
| 266 |
+
step 107 | loss: 2.732643 | lr 1.8000e-04 | norm: 0.2064 | dt: 7836.58ms | tok/sec: 66902.7 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 267 |
+
[TIMING] data: 4.4ms | fwd: 2418.6ms | loss: 4.8ms | bwd: 5375.2ms | clip: 0.0ms | optim: 23.6ms
|
| 268 |
+
step 108 | loss: 2.585369 | lr 1.8000e-04 | norm: 0.1786 | dt: 7836.93ms | tok/sec: 66899.7 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 269 |
+
[TIMING] data: 4.0ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5376.4ms | clip: 0.0ms | optim: 23.6ms
|
| 270 |
+
step 109 | loss: 2.962152 | lr 1.8000e-04 | norm: 0.1899 | dt: 7836.19ms | tok/sec: 66906.0 | mem: 72103MB | Time Left: 04 hrs : 20 mins
|
| 271 |
+
[TIMING] data: 4.1ms | fwd: 2420.2ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
|
| 272 |
+
step 110 | loss: 2.441406 | lr 1.8000e-04 | norm: 0.2773 | dt: 7832.62ms | tok/sec: 66936.5 | mem: 72103MB | Time Left: 04 hrs : 19 mins
|
| 273 |
+
[TIMING] data: 4.3ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5374.4ms | clip: 0.0ms | optim: 23.6ms
|
| 274 |
+
step 111 | loss: 2.954189 | lr 1.8000e-04 | norm: 0.1995 | dt: 7831.38ms | tok/sec: 66947.1 | mem: 72103MB | Time Left: 04 hrs : 19 mins
|
| 275 |
+
[TIMING] data: 4.1ms | fwd: 2419.0ms | loss: 4.9ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
|
| 276 |
+
step 112 | loss: 2.697222 | lr 1.8000e-04 | norm: 0.3689 | dt: 7833.22ms | tok/sec: 66931.3 | mem: 72103MB | Time Left: 04 hrs : 19 mins
|
| 277 |
+
[TIMING] data: 4.2ms | fwd: 2423.4ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.7ms
|
| 278 |
+
step 113 | loss: 2.739350 | lr 1.8000e-04 | norm: 0.2062 | dt: 7838.86ms | tok/sec: 66883.2 | mem: 72103MB | Time Left: 04 hrs : 19 mins
|
| 279 |
+
[TIMING] data: 4.0ms | fwd: 2425.8ms | loss: 4.9ms | bwd: 5370.1ms | clip: 0.0ms | optim: 23.7ms
|
| 280 |
+
step 114 | loss: 2.749949 | lr 1.8000e-04 | norm: 0.1898 | dt: 7826.93ms | tok/sec: 66985.1 | mem: 72103MB | Time Left: 04 hrs : 19 mins
|
| 281 |
+
[TIMING] data: 4.0ms | fwd: 2411.7ms | loss: 4.8ms | bwd: 5372.6ms | clip: 0.0ms | optim: 23.6ms
|
| 282 |
+
step 115 | loss: 2.836510 | lr 1.8000e-04 | norm: 0.2183 | dt: 7828.37ms | tok/sec: 66972.9 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 283 |
+
[TIMING] data: 4.4ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.5ms
|
| 284 |
+
step 116 | loss: 2.801373 | lr 1.8000e-04 | norm: 0.1716 | dt: 7837.44ms | tok/sec: 66895.3 | mem: 72103MB | Time Left: 04 hrs : 19 mins
|
| 285 |
+
[TIMING] data: 4.4ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5378.9ms | clip: 0.0ms | optim: 23.6ms
|
| 286 |
+
step 117 | loss: 2.844561 | lr 1.8000e-04 | norm: 0.2244 | dt: 7835.18ms | tok/sec: 66914.6 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 287 |
+
[TIMING] data: 4.2ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5372.7ms | clip: 0.0ms | optim: 23.8ms
|
| 288 |
+
step 118 | loss: 2.863887 | lr 1.8000e-04 | norm: 0.1997 | dt: 7837.02ms | tok/sec: 66898.9 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 289 |
+
[TIMING] data: 4.2ms | fwd: 2419.7ms | loss: 4.9ms | bwd: 5373.4ms | clip: 0.0ms | optim: 23.6ms
|
| 290 |
+
step 119 | loss: 2.686055 | lr 1.8000e-04 | norm: 0.1676 | dt: 7827.74ms | tok/sec: 66978.2 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 291 |
+
[TIMING] data: 4.3ms | fwd: 2408.9ms | loss: 4.8ms | bwd: 5376.2ms | clip: 0.0ms | optim: 23.6ms
|
| 292 |
+
step 120 | loss: 2.752067 | lr 1.8000e-04 | norm: 0.1851 | dt: 7831.69ms | tok/sec: 66944.4 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 293 |
+
[TIMING] data: 4.2ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.7ms
|
| 294 |
+
step 121 | loss: 2.450900 | lr 1.8000e-04 | norm: 0.2143 | dt: 7831.44ms | tok/sec: 66946.6 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 295 |
+
[TIMING] data: 4.1ms | fwd: 2418.8ms | loss: 4.8ms | bwd: 5369.9ms | clip: 0.0ms | optim: 23.6ms
|
| 296 |
+
step 122 | loss: 2.662471 | lr 1.8000e-04 | norm: 0.1952 | dt: 7834.58ms | tok/sec: 66919.7 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 297 |
+
[TIMING] data: 4.2ms | fwd: 2421.8ms | loss: 4.8ms | bwd: 5370.1ms | clip: 0.0ms | optim: 23.6ms
|
| 298 |
+
step 123 | loss: 2.536113 | lr 1.8000e-04 | norm: 0.1946 | dt: 7836.54ms | tok/sec: 66903.0 | mem: 72103MB | Time Left: 04 hrs : 18 mins
|
| 299 |
+
[TIMING] data: 4.6ms | fwd: 2421.1ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.7ms
|
| 300 |
+
step 124 | loss: 2.916808 | lr 1.8000e-04 | norm: 0.3150 | dt: 7833.85ms | tok/sec: 66925.9 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 301 |
+
[TIMING] data: 4.5ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5368.9ms | clip: 0.0ms | optim: 23.6ms
|
| 302 |
+
step 125 | loss: 2.729327 | lr 1.8000e-04 | norm: 0.2281 | dt: 7836.30ms | tok/sec: 66905.0 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 303 |
+
[TIMING] data: 4.1ms | fwd: 2421.0ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
|
| 304 |
+
step 126 | loss: 2.845534 | lr 1.8000e-04 | norm: 0.2342 | dt: 7837.84ms | tok/sec: 66891.9 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 305 |
+
[TIMING] data: 4.3ms | fwd: 2419.8ms | loss: 4.9ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.7ms
|
| 306 |
+
step 127 | loss: 2.868937 | lr 1.8000e-04 | norm: 0.2717 | dt: 7827.31ms | tok/sec: 66981.9 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 307 |
+
[TIMING] data: 4.1ms | fwd: 2410.3ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.5ms
|
| 308 |
+
step 128 | loss: 2.541056 | lr 1.8000e-04 | norm: 0.1954 | dt: 7835.68ms | tok/sec: 66910.4 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 309 |
+
[TIMING] data: 4.3ms | fwd: 2419.3ms | loss: 4.9ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.6ms
|
| 310 |
+
step 129 | loss: 2.694195 | lr 1.8000e-04 | norm: 0.2187 | dt: 7830.91ms | tok/sec: 66951.1 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 311 |
+
[TIMING] data: 4.3ms | fwd: 2419.3ms | loss: 4.9ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
|
| 312 |
+
step 130 | loss: 2.644610 | lr 1.8000e-04 | norm: 0.2339 | dt: 7834.23ms | tok/sec: 66922.8 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 313 |
+
[TIMING] data: 4.4ms | fwd: 2419.4ms | loss: 4.9ms | bwd: 5372.5ms | clip: 0.0ms | optim: 23.6ms
|
| 314 |
+
step 131 | loss: 2.481298 | lr 1.8000e-04 | norm: 0.1903 | dt: 7822.42ms | tok/sec: 67023.8 | mem: 72103MB | Time Left: 04 hrs : 16 mins
|
| 315 |
+
[TIMING] data: 4.5ms | fwd: 2415.2ms | loss: 4.9ms | bwd: 5364.1ms | clip: 0.0ms | optim: 23.6ms
|
| 316 |
+
step 132 | loss: 2.863067 | lr 1.8000e-04 | norm: 0.2102 | dt: 7836.92ms | tok/sec: 66899.7 | mem: 72103MB | Time Left: 04 hrs : 17 mins
|
| 317 |
+
[TIMING] data: 4.1ms | fwd: 2424.1ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
|
| 318 |
+
step 133 | loss: 2.797191 | lr 1.8000e-04 | norm: 0.2385 | dt: 7834.17ms | tok/sec: 66923.3 | mem: 72103MB | Time Left: 04 hrs : 16 mins
|
| 319 |
+
[TIMING] data: 4.1ms | fwd: 2418.9ms | loss: 4.9ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
|
| 320 |
+
step 134 | loss: 2.811607 | lr 1.8000e-04 | norm: 0.2025 | dt: 7838.16ms | tok/sec: 66889.2 | mem: 72103MB | Time Left: 04 hrs : 16 mins
|
| 321 |
+
[TIMING] data: 4.3ms | fwd: 2422.1ms | loss: 4.8ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.6ms
|
| 322 |
+
step 135 | loss: 2.744267 | lr 1.8000e-04 | norm: 0.2590 | dt: 7842.79ms | tok/sec: 66849.7 | mem: 72103MB | Time Left: 04 hrs : 16 mins
|
| 323 |
+
[TIMING] data: 4.4ms | fwd: 2425.0ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.7ms
|
| 324 |
+
step 136 | loss: 2.712079 | lr 1.8000e-04 | norm: 0.1970 | dt: 7831.78ms | tok/sec: 66943.7 | mem: 72103MB | Time Left: 04 hrs : 16 mins
|
| 325 |
+
[TIMING] data: 4.4ms | fwd: 2419.4ms | loss: 4.9ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.7ms
|
| 326 |
+
step 137 | loss: 2.759872 | lr 1.8000e-04 | norm: 0.2726 | dt: 7839.30ms | tok/sec: 66879.5 | mem: 72103MB | Time Left: 04 hrs : 16 mins
|
| 327 |
+
[TIMING] data: 4.0ms | fwd: 2416.2ms | loss: 4.8ms | bwd: 5379.1ms | clip: 0.0ms | optim: 24.6ms
|
| 328 |
+
step 138 | loss: 2.621341 | lr 1.8000e-04 | norm: 0.2549 | dt: 7832.08ms | tok/sec: 66941.1 | mem: 72103MB | Time Left: 04 hrs : 16 mins
|
| 329 |
+
[TIMING] data: 3.5ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5371.1ms | clip: 0.0ms | optim: 23.7ms
|
| 330 |
+
step 139 | loss: 2.746702 | lr 1.8000e-04 | norm: 0.2566 | dt: 7830.79ms | tok/sec: 66952.2 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 331 |
+
[TIMING] data: 4.4ms | fwd: 2415.6ms | loss: 4.8ms | bwd: 5372.6ms | clip: 0.0ms | optim: 23.6ms
|
| 332 |
+
step 140 | loss: 2.800270 | lr 1.8000e-04 | norm: 0.2116 | dt: 7835.08ms | tok/sec: 66915.4 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 333 |
+
[TIMING] data: 4.2ms | fwd: 2418.4ms | loss: 4.8ms | bwd: 5374.6ms | clip: 0.0ms | optim: 23.7ms
|
| 334 |
+
step 141 | loss: 2.612864 | lr 1.8000e-04 | norm: 0.1952 | dt: 7838.81ms | tok/sec: 66883.6 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 335 |
+
[TIMING] data: 4.4ms | fwd: 2423.0ms | loss: 4.8ms | bwd: 5371.8ms | clip: 0.0ms | optim: 23.7ms
|
| 336 |
+
step 142 | loss: 2.863516 | lr 1.8000e-04 | norm: 0.2237 | dt: 7831.88ms | tok/sec: 66942.8 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 337 |
+
[TIMING] data: 3.9ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.7ms
|
| 338 |
+
step 143 | loss: 2.853393 | lr 1.8000e-04 | norm: 0.1985 | dt: 7831.02ms | tok/sec: 66950.2 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 339 |
+
[TIMING] data: 4.3ms | fwd: 2415.2ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.7ms
|
| 340 |
+
step 144 | loss: 2.810710 | lr 1.8000e-04 | norm: 0.2362 | dt: 7827.02ms | tok/sec: 66984.4 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 341 |
+
[TIMING] data: 4.4ms | fwd: 2419.3ms | loss: 4.8ms | bwd: 5365.0ms | clip: 0.0ms | optim: 23.8ms
|
| 342 |
+
step 145 | loss: 2.727190 | lr 1.8000e-04 | norm: 0.2459 | dt: 7834.58ms | tok/sec: 66919.8 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 343 |
+
[TIMING] data: 4.6ms | fwd: 2420.9ms | loss: 4.8ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.7ms
|
| 344 |
+
step 146 | loss: 2.490621 | lr 1.8000e-04 | norm: 0.2361 | dt: 7835.06ms | tok/sec: 66915.6 | mem: 72103MB | Time Left: 04 hrs : 15 mins
|
| 345 |
+
[TIMING] data: 4.5ms | fwd: 2423.8ms | loss: 4.8ms | bwd: 5368.0ms | clip: 0.0ms | optim: 23.6ms
|
| 346 |
+
step 147 | loss: 2.440445 | lr 1.8000e-04 | norm: 0.2350 | dt: 7829.40ms | tok/sec: 66964.0 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 347 |
+
[TIMING] data: 4.3ms | fwd: 2417.0ms | loss: 4.8ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.7ms
|
| 348 |
+
step 148 | loss: 2.722139 | lr 1.8000e-04 | norm: 0.2419 | dt: 7822.70ms | tok/sec: 67021.4 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 349 |
+
[TIMING] data: 4.4ms | fwd: 2413.3ms | loss: 4.8ms | bwd: 5366.4ms | clip: 0.0ms | optim: 23.6ms
|
| 350 |
+
step 149 | loss: 2.527200 | lr 1.8000e-04 | norm: 0.2305 | dt: 7825.10ms | tok/sec: 67000.8 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 351 |
+
[TIMING] data: 4.5ms | fwd: 2413.2ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.7ms
|
| 352 |
+
step 150 | loss: 2.664558 | lr 1.8000e-04 | norm: 0.2369 | dt: 7827.63ms | tok/sec: 66979.2 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 353 |
+
[TIMING] data: 4.4ms | fwd: 2417.6ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.8ms
|
| 354 |
+
step 151 | loss: 2.904971 | lr 1.8000e-04 | norm: 0.2537 | dt: 7830.84ms | tok/sec: 66951.7 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 355 |
+
[TIMING] data: 4.4ms | fwd: 2417.4ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.7ms
|
| 356 |
+
step 152 | loss: 2.746209 | lr 1.8000e-04 | norm: 0.3610 | dt: 7834.25ms | tok/sec: 66922.6 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 357 |
+
[TIMING] data: 4.5ms | fwd: 2416.9ms | loss: 4.8ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.7ms
|
| 358 |
+
step 153 | loss: 2.717691 | lr 1.8000e-04 | norm: 0.2418 | dt: 7837.54ms | tok/sec: 66894.4 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 359 |
+
[TIMING] data: 4.2ms | fwd: 2422.1ms | loss: 4.8ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.7ms
|
| 360 |
+
step 154 | loss: 2.696790 | lr 1.8000e-04 | norm: 0.2192 | dt: 7835.73ms | tok/sec: 66909.9 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 361 |
+
[TIMING] data: 4.3ms | fwd: 2421.4ms | loss: 4.9ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
|
| 362 |
+
step 155 | loss: 2.743514 | lr 1.8000e-04 | norm: 0.2044 | dt: 7839.40ms | tok/sec: 66878.6 | mem: 72103MB | Time Left: 04 hrs : 14 mins
|
| 363 |
+
[TIMING] data: 4.5ms | fwd: 2420.9ms | loss: 4.9ms | bwd: 5375.1ms | clip: 0.0ms | optim: 23.7ms
|
| 364 |
+
step 156 | loss: 2.695686 | lr 1.8000e-04 | norm: 0.3298 | dt: 7829.98ms | tok/sec: 66959.1 | mem: 72103MB | Time Left: 04 hrs : 13 mins
|
| 365 |
+
[TIMING] data: 4.5ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.6ms
|
| 366 |
+
step 157 | loss: 2.803331 | lr 1.8000e-04 | norm: 0.2812 | dt: 7824.18ms | tok/sec: 67008.7 | mem: 72103MB | Time Left: 04 hrs : 13 mins
|
| 367 |
+
[TIMING] data: 4.7ms | fwd: 2411.8ms | loss: 4.8ms | bwd: 5369.3ms | clip: 0.0ms | optim: 23.6ms
|
| 368 |
+
step 158 | loss: 2.566633 | lr 1.8000e-04 | norm: 0.2226 | dt: 7833.46ms | tok/sec: 66929.3 | mem: 72103MB | Time Left: 04 hrs : 13 mins
|
| 369 |
+
[TIMING] data: 4.6ms | fwd: 2420.9ms | loss: 4.9ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.8ms
|
| 370 |
+
step 159 | loss: 2.414896 | lr 1.8000e-04 | norm: 0.2340 | dt: 7844.29ms | tok/sec: 66836.9 | mem: 72103MB | Time Left: 04 hrs : 13 mins
|
| 371 |
+
[TIMING] data: 4.4ms | fwd: 2423.2ms | loss: 4.9ms | bwd: 5378.3ms | clip: 0.0ms | optim: 23.6ms
|
| 372 |
+
step 160 | loss: 2.776480 | lr 1.8000e-04 | norm: 0.3275 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 04 hrs : 13 mins
|
| 373 |
+
[TIMING] data: 4.3ms | fwd: 2413.9ms | loss: 4.9ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.7ms
|
| 374 |
+
step 161 | loss: 2.666731 | lr 1.8000e-04 | norm: 0.2277 | dt: 7830.16ms | tok/sec: 66957.5 | mem: 72103MB | Time Left: 04 hrs : 13 mins
|
| 375 |
+
[TIMING] data: 4.3ms | fwd: 2416.3ms | loss: 4.8ms | bwd: 5372.0ms | clip: 0.0ms | optim: 23.6ms
|
| 376 |
+
step 162 | loss: 2.890054 | lr 1.8000e-04 | norm: 0.2539 | dt: 7825.41ms | tok/sec: 66998.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 377 |
+
[TIMING] data: 4.5ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.7ms
|
| 378 |
+
step 163 | loss: 2.698914 | lr 1.8000e-04 | norm: 0.2679 | dt: 7831.07ms | tok/sec: 66949.7 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 379 |
+
[TIMING] data: 4.2ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.6ms
|
| 380 |
+
step 164 | loss: 2.809147 | lr 1.8000e-04 | norm: 0.2333 | dt: 7835.59ms | tok/sec: 66911.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 381 |
+
[TIMING] data: 4.7ms | fwd: 2421.8ms | loss: 4.9ms | bwd: 5370.7ms | clip: 0.0ms | optim: 23.6ms
|
| 382 |
+
step 165 | loss: 2.738762 | lr 1.8000e-04 | norm: 0.1988 | dt: 7841.92ms | tok/sec: 66857.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 383 |
+
[TIMING] data: 4.2ms | fwd: 2426.0ms | loss: 4.8ms | bwd: 5372.5ms | clip: 0.0ms | optim: 23.6ms
|
| 384 |
+
step 166 | loss: 2.692415 | lr 1.8000e-04 | norm: 0.2143 | dt: 7843.56ms | tok/sec: 66843.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 385 |
+
[TIMING] data: 4.4ms | fwd: 2426.5ms | loss: 4.9ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
|
| 386 |
+
step 167 | loss: 2.719501 | lr 1.8000e-04 | norm: 0.2727 | dt: 7833.75ms | tok/sec: 66926.8 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 387 |
+
[TIMING] data: 4.0ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
|
| 388 |
+
step 168 | loss: 2.901875 | lr 1.8000e-04 | norm: 0.2566 | dt: 7834.07ms | tok/sec: 66924.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 389 |
+
[TIMING] data: 4.5ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5375.6ms | clip: 0.0ms | optim: 23.6ms
|
| 390 |
+
step 169 | loss: 2.657280 | lr 1.8000e-04 | norm: 0.2217 | dt: 7823.22ms | tok/sec: 67016.9 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 391 |
+
[TIMING] data: 4.5ms | fwd: 2416.2ms | loss: 4.9ms | bwd: 5363.4ms | clip: 0.0ms | optim: 23.7ms
|
| 392 |
+
step 170 | loss: 2.508571 | lr 1.8000e-04 | norm: 0.2954 | dt: 7845.90ms | tok/sec: 66823.1 | mem: 72103MB | Time Left: 04 hrs : 12 mins
|
| 393 |
+
[TIMING] data: 4.1ms | fwd: 2423.9ms | loss: 4.8ms | bwd: 5378.8ms | clip: 0.0ms | optim: 23.6ms
|
| 394 |
+
step 171 | loss: 2.703474 | lr 1.8000e-04 | norm: 0.2781 | dt: 7833.89ms | tok/sec: 66925.6 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 395 |
+
[TIMING] data: 4.3ms | fwd: 2414.5ms | loss: 4.8ms | bwd: 5376.4ms | clip: 0.0ms | optim: 23.7ms
|
| 396 |
+
step 172 | loss: 2.485096 | lr 1.8000e-04 | norm: 0.2705 | dt: 7836.91ms | tok/sec: 66899.8 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 397 |
+
[TIMING] data: 4.4ms | fwd: 2418.4ms | loss: 4.8ms | bwd: 5375.8ms | clip: 0.0ms | optim: 23.6ms
|
| 398 |
+
step 173 | loss: 2.787684 | lr 1.8000e-04 | norm: 0.2739 | dt: 7831.90ms | tok/sec: 66942.7 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 399 |
+
[TIMING] data: 4.4ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5374.8ms | clip: 0.0ms | optim: 23.7ms
|
| 400 |
+
step 174 | loss: 2.589551 | lr 1.8000e-04 | norm: 0.2410 | dt: 7842.79ms | tok/sec: 66849.7 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 401 |
+
[TIMING] data: 4.6ms | fwd: 2424.5ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.7ms
|
| 402 |
+
step 175 | loss: 2.584219 | lr 1.8000e-04 | norm: 0.3058 | dt: 7835.70ms | tok/sec: 66910.2 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 403 |
+
[TIMING] data: 4.5ms | fwd: 2417.1ms | loss: 4.8ms | bwd: 5375.5ms | clip: 0.0ms | optim: 23.6ms
|
| 404 |
+
step 176 | loss: 2.491823 | lr 1.8000e-04 | norm: 0.2384 | dt: 7837.64ms | tok/sec: 66893.6 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 405 |
+
[TIMING] data: 4.5ms | fwd: 2416.0ms | loss: 4.8ms | bwd: 5378.6ms | clip: 0.0ms | optim: 23.6ms
|
| 406 |
+
step 177 | loss: 2.525588 | lr 1.8000e-04 | norm: 0.2601 | dt: 7830.52ms | tok/sec: 66954.5 | mem: 72103MB | Time Left: 04 hrs : 10 mins
|
| 407 |
+
[TIMING] data: 4.7ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5369.4ms | clip: 0.0ms | optim: 23.7ms
|
| 408 |
+
step 178 | loss: 2.731776 | lr 1.8000e-04 | norm: 0.2761 | dt: 7836.31ms | tok/sec: 66905.0 | mem: 72103MB | Time Left: 04 hrs : 11 mins
|
| 409 |
+
[TIMING] data: 4.6ms | fwd: 2424.9ms | loss: 4.9ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.7ms
|
| 410 |
+
step 179 | loss: 2.635677 | lr 1.8000e-04 | norm: 0.3731 | dt: 7828.18ms | tok/sec: 66974.5 | mem: 72103MB | Time Left: 04 hrs : 10 mins
|
| 411 |
+
[TIMING] data: 4.4ms | fwd: 2415.0ms | loss: 4.8ms | bwd: 5370.9ms | clip: 0.0ms | optim: 23.6ms
|
| 412 |
+
step 180 | loss: 2.596631 | lr 1.8000e-04 | norm: 0.3398 | dt: 7836.89ms | tok/sec: 66900.0 | mem: 72103MB | Time Left: 04 hrs : 10 mins
|
| 413 |
+
[TIMING] data: 4.5ms | fwd: 2424.8ms | loss: 4.8ms | bwd: 5368.7ms | clip: 0.0ms | optim: 23.7ms
|
| 414 |
+
step 181 | loss: 2.456206 | lr 1.8000e-04 | norm: 0.2772 | dt: 7830.11ms | tok/sec: 66958.0 | mem: 72103MB | Time Left: 04 hrs : 10 mins
|
| 415 |
+
[TIMING] data: 4.1ms | fwd: 2415.4ms | loss: 4.8ms | bwd: 5371.2ms | clip: 0.0ms | optim: 23.6ms
|
| 416 |
+
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/Data_Loader/Data/mixcont/train_shard_00001.bin
|
| 417 |
+
100000000
|
| 418 |
+
step 182 | loss: 2.585284 | lr 1.8000e-04 | norm: 0.2954 | dt: 7845.94ms | tok/sec: 66822.8 | mem: 72103MB | Time Left: 04 hrs : 10 mins
|
| 419 |
+
[TIMING] data: 11.7ms | fwd: 2424.5ms | loss: 4.8ms | bwd: 5372.0ms | clip: 0.0ms | optim: 23.6ms
|
| 420 |
+
step 183 | loss: 2.548141 | lr 1.8000e-04 | norm: 0.3066 | dt: 7834.46ms | tok/sec: 66920.8 | mem: 72103MB | Time Left: 04 hrs : 10 mins
|
| 421 |
+
[TIMING] data: 4.4ms | fwd: 2422.2ms | loss: 4.8ms | bwd: 5369.2ms | clip: 0.0ms | optim: 23.6ms
|
| 422 |
+
step 184 | loss: 2.478449 | lr 1.8000e-04 | norm: 0.2895 | dt: 7840.94ms | tok/sec: 66865.5 | mem: 72103MB | Time Left: 04 hrs : 10 mins
|
| 423 |
+
[TIMING] data: 4.4ms | fwd: 2427.2ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.7ms
|
| 424 |
+
step 185 | loss: 2.528314 | lr 1.8000e-04 | norm: 0.3043 | dt: 7831.82ms | tok/sec: 66943.3 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 425 |
+
[TIMING] data: 4.3ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5370.8ms | clip: 0.0ms | optim: 23.6ms
|
| 426 |
+
step 186 | loss: 2.352158 | lr 1.8000e-04 | norm: 0.2586 | dt: 7827.45ms | tok/sec: 66980.7 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 427 |
+
[TIMING] data: 4.4ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5366.7ms | clip: 0.0ms | optim: 23.7ms
|
| 428 |
+
step 187 | loss: 2.563450 | lr 1.8000e-04 | norm: 0.3322 | dt: 7834.83ms | tok/sec: 66917.6 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 429 |
+
[TIMING] data: 4.3ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.7ms
|
| 430 |
+
step 188 | loss: 2.539817 | lr 1.8000e-04 | norm: 0.2921 | dt: 7829.95ms | tok/sec: 66959.3 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 431 |
+
[TIMING] data: 4.1ms | fwd: 2416.3ms | loss: 4.8ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.7ms
|
| 432 |
+
step 189 | loss: 2.632222 | lr 1.8000e-04 | norm: 0.3063 | dt: 7840.75ms | tok/sec: 66867.1 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 433 |
+
[TIMING] data: 4.0ms | fwd: 2421.2ms | loss: 4.9ms | bwd: 5376.5ms | clip: 0.0ms | optim: 23.6ms
|
| 434 |
+
step 190 | loss: 2.607751 | lr 1.8000e-04 | norm: 0.3119 | dt: 7827.96ms | tok/sec: 66976.3 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 435 |
+
[TIMING] data: 4.3ms | fwd: 2416.4ms | loss: 4.8ms | bwd: 5367.8ms | clip: 0.0ms | optim: 23.7ms
|
| 436 |
+
step 191 | loss: 2.665597 | lr 1.8000e-04 | norm: 0.3547 | dt: 7840.90ms | tok/sec: 66865.8 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 437 |
+
[TIMING] data: 4.3ms | fwd: 2424.8ms | loss: 4.9ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.8ms
|
| 438 |
+
step 192 | loss: 2.514341 | lr 1.8000e-04 | norm: 0.3333 | dt: 7835.85ms | tok/sec: 66908.9 | mem: 72103MB | Time Left: 04 hrs : 09 mins
|
| 439 |
+
[TIMING] data: 4.1ms | fwd: 2416.4ms | loss: 4.8ms | bwd: 5376.6ms | clip: 0.0ms | optim: 23.7ms
|
| 440 |
+
step 193 | loss: 2.530896 | lr 1.8000e-04 | norm: 0.3155 | dt: 7832.06ms | tok/sec: 66941.2 | mem: 72103MB | Time Left: 04 hrs : 08 mins
|
| 441 |
+
[TIMING] data: 4.2ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.6ms
|
| 442 |
+
step 194 | loss: 2.378180 | lr 1.8000e-04 | norm: 0.3543 | dt: 7828.80ms | tok/sec: 66969.1 | mem: 72103MB | Time Left: 04 hrs : 08 mins
|
| 443 |
+
[TIMING] data: 4.4ms | fwd: 2410.8ms | loss: 4.8ms | bwd: 5374.7ms | clip: 0.0ms | optim: 23.8ms
|
| 444 |
+
step 195 | loss: 2.700363 | lr 1.8000e-04 | norm: 0.4001 | dt: 7833.91ms | tok/sec: 66925.4 | mem: 72103MB | Time Left: 04 hrs : 08 mins
|
| 445 |
+
[TIMING] data: 4.4ms | fwd: 2421.6ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.6ms
|
| 446 |
+
step 196 | loss: 2.455045 | lr 1.8000e-04 | norm: 0.3854 | dt: 7824.79ms | tok/sec: 67003.5 | mem: 72103MB | Time Left: 04 hrs : 08 mins
|
| 447 |
+
[TIMING] data: 4.5ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5363.8ms | clip: 0.0ms | optim: 23.7ms
|
| 448 |
+
step 197 | loss: 2.718957 | lr 1.8000e-04 | norm: 0.4156 | dt: 7826.31ms | tok/sec: 66990.4 | mem: 72103MB | Time Left: 04 hrs : 08 mins
|
| 449 |
+
[TIMING] data: 4.5ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5366.9ms | clip: 0.0ms | optim: 23.7ms
|
| 450 |
+
step 198 | loss: 2.476290 | lr 1.8000e-04 | norm: 0.4051 | dt: 7827.59ms | tok/sec: 66979.5 | mem: 72103MB | Time Left: 04 hrs : 08 mins
|
| 451 |
+
[TIMING] data: 4.1ms | fwd: 2413.4ms | loss: 4.8ms | bwd: 5370.7ms | clip: 0.0ms | optim: 23.6ms
|
| 452 |
+
step 199 | loss: 2.422088 | lr 1.8000e-04 | norm: 0.3190 | dt: 7836.25ms | tok/sec: 66905.5 | mem: 72103MB | Time Left: 04 hrs : 08 mins
|
| 453 |
+
[TIMING] data: 4.1ms | fwd: 2421.3ms | loss: 4.9ms | bwd: 5371.9ms | clip: 0.0ms | optim: 23.6ms
|
| 454 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt (safely)
|
| 455 |
+
step 200 | loss: 2.402291 | lr 1.8000e-04 | norm: 0.4287 | dt: 8298.17ms | tok/sec: 63181.2 | mem: 72103MB | Time Left: 04 hrs : 22 mins
|
| 456 |
+
[TIMING] data: 4.8ms | fwd: 2909.3ms | loss: 4.8ms | bwd: 5344.9ms | clip: 0.0ms | optim: 23.6ms
|
| 457 |
+
step 201 | loss: 2.387901 | lr 1.8000e-04 | norm: 0.3077 | dt: 7815.83ms | tok/sec: 67080.3 | mem: 72103MB | Time Left: 04 hrs : 07 mins
|
| 458 |
+
[TIMING] data: 4.9ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5350.5ms | clip: 0.0ms | optim: 23.7ms
|
| 459 |
+
step 202 | loss: 2.360209 | lr 1.8000e-04 | norm: 0.3530 | dt: 7813.41ms | tok/sec: 67101.0 | mem: 72103MB | Time Left: 04 hrs : 07 mins
|
| 460 |
+
[TIMING] data: 4.8ms | fwd: 2408.3ms | loss: 4.8ms | bwd: 5362.2ms | clip: 0.0ms | optim: 23.6ms
|
| 461 |
+
step 203 | loss: 2.409445 | lr 1.8000e-04 | norm: 0.3761 | dt: 7821.09ms | tok/sec: 67035.2 | mem: 72103MB | Time Left: 04 hrs : 07 mins
|
| 462 |
+
[TIMING] data: 4.9ms | fwd: 2416.8ms | loss: 4.8ms | bwd: 5360.5ms | clip: 0.0ms | optim: 23.6ms
|
| 463 |
+
step 204 | loss: 2.408840 | lr 1.8000e-04 | norm: 0.6411 | dt: 7817.69ms | tok/sec: 67064.3 | mem: 72103MB | Time Left: 04 hrs : 07 mins
|
| 464 |
+
[TIMING] data: 4.8ms | fwd: 2407.8ms | loss: 4.8ms | bwd: 5367.6ms | clip: 0.0ms | optim: 23.6ms
|
| 465 |
+
step 205 | loss: 2.459621 | lr 1.8000e-04 | norm: 0.4455 | dt: 7827.36ms | tok/sec: 66981.5 | mem: 72103MB | Time Left: 04 hrs : 07 mins
|
| 466 |
+
[TIMING] data: 4.9ms | fwd: 2417.5ms | loss: 4.8ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
|
| 467 |
+
step 206 | loss: 2.402254 | lr 1.8000e-04 | norm: 0.4152 | dt: 7831.62ms | tok/sec: 66945.0 | mem: 72103MB | Time Left: 04 hrs : 07 mins
|
| 468 |
+
[TIMING] data: 4.8ms | fwd: 2421.7ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.5ms
|
| 469 |
+
step 207 | loss: 2.387150 | lr 1.8000e-04 | norm: 0.4003 | dt: 7827.34ms | tok/sec: 66981.7 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 470 |
+
[TIMING] data: 4.9ms | fwd: 2414.8ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.6ms
|
| 471 |
+
step 208 | loss: 2.443284 | lr 1.8000e-04 | norm: 0.3607 | dt: 7830.10ms | tok/sec: 66958.0 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 472 |
+
[TIMING] data: 4.8ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.6ms
|
| 473 |
+
step 209 | loss: 2.525800 | lr 1.8000e-04 | norm: 0.4068 | dt: 7830.26ms | tok/sec: 66956.6 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 474 |
+
[TIMING] data: 4.8ms | fwd: 2422.8ms | loss: 4.8ms | bwd: 5364.6ms | clip: 0.0ms | optim: 23.5ms
|
| 475 |
+
step 210 | loss: 2.456215 | lr 1.8000e-04 | norm: 0.3717 | dt: 7825.41ms | tok/sec: 66998.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 476 |
+
[TIMING] data: 4.9ms | fwd: 2414.9ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.6ms
|
| 477 |
+
step 211 | loss: 2.357838 | lr 1.8000e-04 | norm: 0.4258 | dt: 7833.70ms | tok/sec: 66927.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 478 |
+
[TIMING] data: 4.8ms | fwd: 2420.7ms | loss: 4.9ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
|
| 479 |
+
step 212 | loss: 2.446153 | lr 1.8000e-04 | norm: 0.3971 | dt: 7830.20ms | tok/sec: 66957.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 480 |
+
[TIMING] data: 4.9ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5366.4ms | clip: 0.0ms | optim: 23.5ms
|
| 481 |
+
step 213 | loss: 2.249928 | lr 1.8000e-04 | norm: 0.4567 | dt: 7823.83ms | tok/sec: 67011.7 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 482 |
+
[TIMING] data: 4.9ms | fwd: 2416.2ms | loss: 4.8ms | bwd: 5364.0ms | clip: 0.0ms | optim: 23.6ms
|
| 483 |
+
step 214 | loss: 2.369561 | lr 1.8000e-04 | norm: 0.4493 | dt: 7832.30ms | tok/sec: 66939.2 | mem: 72103MB | Time Left: 04 hrs : 06 mins
|
| 484 |
+
[TIMING] data: 4.9ms | fwd: 2420.2ms | loss: 4.8ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.6ms
|
| 485 |
+
step 215 | loss: 2.343032 | lr 1.8000e-04 | norm: 0.4376 | dt: 7826.26ms | tok/sec: 66990.9 | mem: 72103MB | Time Left: 04 hrs : 05 mins
|
| 486 |
+
[TIMING] data: 4.8ms | fwd: 2417.1ms | loss: 4.8ms | bwd: 5364.8ms | clip: 0.0ms | optim: 23.7ms
|
| 487 |
+
step 216 | loss: 2.240227 | lr 1.8000e-04 | norm: 0.4092 | dt: 7821.41ms | tok/sec: 67032.4 | mem: 72103MB | Time Left: 04 hrs : 05 mins
|
| 488 |
+
[TIMING] data: 4.8ms | fwd: 2410.4ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.6ms
|
| 489 |
+
step 217 | loss: 2.362448 | lr 1.8000e-04 | norm: 0.4497 | dt: 7830.47ms | tok/sec: 66954.9 | mem: 72103MB | Time Left: 04 hrs : 05 mins
|
| 490 |
+
[TIMING] data: 5.0ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.5ms
|
| 491 |
+
step 218 | loss: 2.288676 | lr 1.8000e-04 | norm: 0.4341 | dt: 7829.31ms | tok/sec: 66964.8 | mem: 72103MB | Time Left: 04 hrs : 05 mins
|
| 492 |
+
[TIMING] data: 4.9ms | fwd: 2417.2ms | loss: 4.8ms | bwd: 5368.5ms | clip: 0.0ms | optim: 23.7ms
|
| 493 |
+
step 219 | loss: 2.323463 | lr 1.8000e-04 | norm: 0.4153 | dt: 7838.80ms | tok/sec: 66883.7 | mem: 72103MB | Time Left: 04 hrs : 05 mins
|
| 494 |
+
[TIMING] data: 4.9ms | fwd: 2419.0ms | loss: 4.9ms | bwd: 5377.4ms | clip: 0.0ms | optim: 23.6ms
|
| 495 |
+
step 220 | loss: 2.250193 | lr 1.8000e-04 | norm: 0.4058 | dt: 7833.43ms | tok/sec: 66929.6 | mem: 72103MB | Time Left: 04 hrs : 05 mins
|
| 496 |
+
[TIMING] data: 4.8ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5370.3ms | clip: 0.0ms | optim: 23.5ms
|
| 497 |
+
step 221 | loss: 2.382532 | lr 1.8000e-04 | norm: 0.4433 | dt: 7835.52ms | tok/sec: 66911.7 | mem: 72103MB | Time Left: 04 hrs : 05 mins
|
| 498 |
+
[TIMING] data: 4.8ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.7ms
|
| 499 |
+
step 222 | loss: 2.410047 | lr 1.8000e-04 | norm: 0.5359 | dt: 7823.76ms | tok/sec: 67012.3 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 500 |
+
[TIMING] data: 4.9ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5366.7ms | clip: 0.0ms | optim: 23.6ms
|
| 501 |
+
step 223 | loss: 2.355986 | lr 1.8000e-04 | norm: 0.4549 | dt: 7831.10ms | tok/sec: 66949.5 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 502 |
+
[TIMING] data: 4.8ms | fwd: 2418.7ms | loss: 4.9ms | bwd: 5369.2ms | clip: 0.0ms | optim: 23.6ms
|
| 503 |
+
step 224 | loss: 2.260771 | lr 1.8000e-04 | norm: 0.4940 | dt: 7835.26ms | tok/sec: 66913.9 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 504 |
+
[TIMING] data: 4.8ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
|
| 505 |
+
step 225 | loss: 2.336183 | lr 1.8000e-04 | norm: 0.4559 | dt: 7836.82ms | tok/sec: 66900.6 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 506 |
+
[TIMING] data: 4.9ms | fwd: 2416.9ms | loss: 4.8ms | bwd: 5376.5ms | clip: 0.0ms | optim: 23.7ms
|
| 507 |
+
step 226 | loss: 2.206506 | lr 1.8000e-04 | norm: 0.5058 | dt: 7836.13ms | tok/sec: 66906.5 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 508 |
+
[TIMING] data: 4.9ms | fwd: 2425.9ms | loss: 4.8ms | bwd: 5366.5ms | clip: 0.0ms | optim: 23.6ms
|
| 509 |
+
step 227 | loss: 2.221648 | lr 1.8000e-04 | norm: 0.4108 | dt: 7834.61ms | tok/sec: 66919.4 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 510 |
+
[TIMING] data: 4.8ms | fwd: 2419.4ms | loss: 4.8ms | bwd: 5372.5ms | clip: 0.0ms | optim: 23.5ms
|
| 511 |
+
step 228 | loss: 2.372312 | lr 1.8000e-04 | norm: 0.4778 | dt: 7838.01ms | tok/sec: 66890.5 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 512 |
+
[TIMING] data: 4.8ms | fwd: 2420.8ms | loss: 4.8ms | bwd: 5373.5ms | clip: 0.0ms | optim: 23.6ms
|
| 513 |
+
step 229 | loss: 2.021067 | lr 1.8000e-04 | norm: 0.4256 | dt: 7835.34ms | tok/sec: 66913.3 | mem: 72103MB | Time Left: 04 hrs : 04 mins
|
| 514 |
+
[TIMING] data: 4.9ms | fwd: 2421.5ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
|
| 515 |
+
step 230 | loss: 2.196075 | lr 1.8000e-04 | norm: 0.5318 | dt: 7825.84ms | tok/sec: 66994.5 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 516 |
+
[TIMING] data: 4.9ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5367.1ms | clip: 0.0ms | optim: 23.6ms
|
| 517 |
+
step 231 | loss: 2.229375 | lr 1.8000e-04 | norm: 0.4619 | dt: 7832.63ms | tok/sec: 66936.4 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 518 |
+
[TIMING] data: 4.8ms | fwd: 2417.5ms | loss: 4.8ms | bwd: 5372.8ms | clip: 0.0ms | optim: 23.6ms
|
| 519 |
+
step 232 | loss: 2.005525 | lr 1.8000e-04 | norm: 0.4127 | dt: 7829.49ms | tok/sec: 66963.2 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 520 |
+
[TIMING] data: 4.8ms | fwd: 2412.1ms | loss: 4.8ms | bwd: 5373.0ms | clip: 0.0ms | optim: 23.6ms
|
| 521 |
+
step 233 | loss: 2.188138 | lr 1.8000e-04 | norm: 0.5179 | dt: 7836.54ms | tok/sec: 66903.0 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 522 |
+
[TIMING] data: 4.8ms | fwd: 2419.1ms | loss: 4.9ms | bwd: 5374.2ms | clip: 0.0ms | optim: 23.7ms
|
| 523 |
+
step 234 | loss: 2.369011 | lr 1.8000e-04 | norm: 0.5784 | dt: 7826.64ms | tok/sec: 66987.6 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 524 |
+
[TIMING] data: 4.9ms | fwd: 2416.4ms | loss: 4.9ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.5ms
|
| 525 |
+
step 235 | loss: 2.315351 | lr 1.8000e-04 | norm: 0.5331 | dt: 7823.82ms | tok/sec: 67011.7 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 526 |
+
[TIMING] data: 4.8ms | fwd: 2410.6ms | loss: 4.9ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.5ms
|
| 527 |
+
step 236 | loss: 1.939878 | lr 1.8000e-04 | norm: 0.4362 | dt: 7831.17ms | tok/sec: 66948.8 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 528 |
+
[TIMING] data: 4.9ms | fwd: 2415.0ms | loss: 4.9ms | bwd: 5372.4ms | clip: 0.0ms | optim: 23.6ms
|
| 529 |
+
step 237 | loss: 2.137450 | lr 1.8000e-04 | norm: 0.5205 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 04 hrs : 03 mins
|
| 530 |
+
[TIMING] data: 4.8ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.6ms
|
| 531 |
+
step 238 | loss: 2.292795 | lr 1.8000e-04 | norm: 0.5377 | dt: 7825.25ms | tok/sec: 66999.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 532 |
+
[TIMING] data: 4.9ms | fwd: 2413.1ms | loss: 4.8ms | bwd: 5368.9ms | clip: 0.0ms | optim: 23.7ms
|
| 533 |
+
step 239 | loss: 2.211943 | lr 1.8000e-04 | norm: 0.5593 | dt: 7832.50ms | tok/sec: 66937.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 534 |
+
[TIMING] data: 4.9ms | fwd: 2419.1ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
|
| 535 |
+
step 240 | loss: 2.144189 | lr 1.8000e-04 | norm: 0.5692 | dt: 7830.98ms | tok/sec: 66950.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 536 |
+
[TIMING] data: 4.7ms | fwd: 2416.9ms | loss: 4.8ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.6ms
|
| 537 |
+
step 241 | loss: 2.031067 | lr 1.8000e-04 | norm: 0.5292 | dt: 7835.10ms | tok/sec: 66915.3 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 538 |
+
[TIMING] data: 4.8ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
|
| 539 |
+
step 242 | loss: 2.202729 | lr 1.8000e-04 | norm: 0.5180 | dt: 7831.67ms | tok/sec: 66944.6 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 540 |
+
[TIMING] data: 4.9ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.6ms
|
| 541 |
+
step 243 | loss: 2.130872 | lr 1.8000e-04 | norm: 0.6109 | dt: 7834.72ms | tok/sec: 66918.6 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 542 |
+
[TIMING] data: 4.8ms | fwd: 2419.4ms | loss: 4.9ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.7ms
|
| 543 |
+
step 244 | loss: 2.166822 | lr 1.8000e-04 | norm: 0.5125 | dt: 7821.30ms | tok/sec: 67033.4 | mem: 72103MB | Time Left: 04 hrs : 01 mins
|
| 544 |
+
[TIMING] data: 4.8ms | fwd: 2409.2ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.5ms
|
| 545 |
+
step 245 | loss: 2.147327 | lr 1.8000e-04 | norm: 0.6140 | dt: 7827.70ms | tok/sec: 66978.5 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 546 |
+
[TIMING] data: 4.7ms | fwd: 2415.7ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
|
| 547 |
+
step 246 | loss: 2.215064 | lr 1.8000e-04 | norm: 0.5953 | dt: 7838.44ms | tok/sec: 66886.8 | mem: 72103MB | Time Left: 04 hrs : 02 mins
|
| 548 |
+
[TIMING] data: 4.7ms | fwd: 2422.7ms | loss: 4.8ms | bwd: 5373.4ms | clip: 0.0ms | optim: 23.7ms
|
| 549 |
+
step 247 | loss: 2.186167 | lr 1.8000e-04 | norm: 0.5049 | dt: 7820.30ms | tok/sec: 67041.9 | mem: 72103MB | Time Left: 04 hrs : 01 mins
|
| 550 |
+
[TIMING] data: 4.9ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5360.8ms | clip: 0.0ms | optim: 23.7ms
|
| 551 |
+
step 248 | loss: 2.352702 | lr 1.8000e-04 | norm: 0.6359 | dt: 7825.95ms | tok/sec: 66993.5 | mem: 72103MB | Time Left: 04 hrs : 01 mins
|
| 552 |
+
[TIMING] data: 4.9ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5364.8ms | clip: 0.0ms | optim: 23.6ms
|
| 553 |
+
step 249 | loss: 2.165310 | lr 1.8000e-04 | norm: 0.5937 | dt: 7838.78ms | tok/sec: 66883.9 | mem: 72103MB | Time Left: 04 hrs : 01 mins
|
| 554 |
+
[TIMING] data: 4.8ms | fwd: 2421.6ms | loss: 4.8ms | bwd: 5373.8ms | clip: 0.0ms | optim: 23.5ms
|
| 555 |
+
validation loss: 2.0652
|
| 556 |
+
step 250 | loss: 2.170025 | lr 1.8000e-04 | norm: 0.6153 | dt: 13815.70ms | tok/sec: 37948.7 | mem: 72103MB | Time Left: 07 hrs : 05 mins
|
| 557 |
+
[TIMING] data: 4.9ms | fwd: 2422.0ms | loss: 4.9ms | bwd: 5370.4ms | clip: 0.0ms | optim: 23.5ms
|
| 558 |
+
step 251 | loss: 1.956852 | lr 1.8000e-04 | norm: 0.5652 | dt: 7824.67ms | tok/sec: 67004.5 | mem: 72103MB | Time Left: 04 hrs : 01 mins
|
| 559 |
+
[TIMING] data: 4.9ms | fwd: 2417.8ms | loss: 4.8ms | bwd: 5364.5ms | clip: 0.0ms | optim: 23.6ms
|
| 560 |
+
step 252 | loss: 2.047342 | lr 1.8000e-04 | norm: 0.6065 | dt: 7828.67ms | tok/sec: 66970.3 | mem: 72103MB | Time Left: 04 hrs : 01 mins
|
| 561 |
+
[TIMING] data: 4.9ms | fwd: 2417.3ms | loss: 4.9ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.5ms
|
| 562 |
+
step 253 | loss: 2.042702 | lr 1.8000e-04 | norm: 0.5501 | dt: 7829.01ms | tok/sec: 66967.3 | mem: 72103MB | Time Left: 04 hrs : 01 mins
|
| 563 |
+
[TIMING] data: 4.6ms | fwd: 2416.0ms | loss: 4.8ms | bwd: 5370.8ms | clip: 0.0ms | optim: 23.6ms
|
| 564 |
+
step 254 | loss: 2.038959 | lr 1.8000e-04 | norm: 0.6408 | dt: 7832.29ms | tok/sec: 66939.3 | mem: 72103MB | Time Left: 04 hrs : 00 mins
|
| 565 |
+
[TIMING] data: 4.8ms | fwd: 2415.1ms | loss: 4.9ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.6ms
|
| 566 |
+
step 255 | loss: 2.131271 | lr 1.8000e-04 | norm: 0.5878 | dt: 7836.77ms | tok/sec: 66901.0 | mem: 72103MB | Time Left: 04 hrs : 00 mins
|
| 567 |
+
[TIMING] data: 4.9ms | fwd: 2422.8ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.5ms
|
| 568 |
+
step 256 | loss: 2.110420 | lr 1.8000e-04 | norm: 0.6424 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 04 hrs : 00 mins
|
| 569 |
+
[TIMING] data: 4.9ms | fwd: 2418.2ms | loss: 4.8ms | bwd: 5366.3ms | clip: 0.0ms | optim: 23.7ms
|
| 570 |
+
step 257 | loss: 2.220668 | lr 1.8000e-04 | norm: 0.6648 | dt: 7832.83ms | tok/sec: 66934.7 | mem: 72103MB | Time Left: 04 hrs : 00 mins
|
| 571 |
+
[TIMING] data: 4.8ms | fwd: 2417.7ms | loss: 4.9ms | bwd: 5372.2ms | clip: 0.0ms | optim: 23.5ms
|
| 572 |
+
step 258 | loss: 2.076067 | lr 1.8000e-04 | norm: 0.7399 | dt: 7831.29ms | tok/sec: 66947.8 | mem: 72103MB | Time Left: 04 hrs : 00 mins
|
| 573 |
+
[TIMING] data: 4.9ms | fwd: 2418.3ms | loss: 4.8ms | bwd: 5369.4ms | clip: 0.0ms | optim: 23.8ms
|
| 574 |
+
step 259 | loss: 1.938241 | lr 1.8000e-04 | norm: 0.6259 | dt: 7828.57ms | tok/sec: 66971.1 | mem: 72103MB | Time Left: 04 hrs : 00 mins
|
| 575 |
+
[TIMING] data: 4.7ms | fwd: 2415.7ms | loss: 4.8ms | bwd: 5370.1ms | clip: 0.0ms | optim: 23.5ms
|
| 576 |
+
step 260 | loss: 1.857182 | lr 1.8000e-04 | norm: 0.6081 | dt: 7832.76ms | tok/sec: 66935.3 | mem: 72103MB | Time Left: 04 hrs : 00 mins
|
| 577 |
+
[TIMING] data: 4.9ms | fwd: 2415.5ms | loss: 4.8ms | bwd: 5374.8ms | clip: 0.0ms | optim: 23.5ms
|
| 578 |
+
step 261 | loss: 2.031495 | lr 1.8000e-04 | norm: 0.8596 | dt: 7828.89ms | tok/sec: 66968.4 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 579 |
+
[TIMING] data: 4.7ms | fwd: 2416.0ms | loss: 4.8ms | bwd: 5370.5ms | clip: 0.0ms | optim: 23.6ms
|
| 580 |
+
step 262 | loss: 1.959992 | lr 1.8000e-04 | norm: 0.6619 | dt: 7822.34ms | tok/sec: 67024.5 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 581 |
+
[TIMING] data: 4.8ms | fwd: 2410.4ms | loss: 4.8ms | bwd: 5367.2ms | clip: 0.0ms | optim: 23.7ms
|
| 582 |
+
step 263 | loss: 1.983674 | lr 1.8000e-04 | norm: 0.6752 | dt: 7834.08ms | tok/sec: 66924.0 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 583 |
+
[TIMING] data: 4.8ms | fwd: 2422.9ms | loss: 4.9ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.6ms
|
| 584 |
+
step 264 | loss: 1.938962 | lr 1.8000e-04 | norm: 0.8107 | dt: 7836.23ms | tok/sec: 66905.7 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 585 |
+
[TIMING] data: 4.7ms | fwd: 2418.0ms | loss: 4.9ms | bwd: 5374.6ms | clip: 0.0ms | optim: 23.7ms
|
| 586 |
+
step 265 | loss: 1.999571 | lr 1.8000e-04 | norm: 0.7463 | dt: 7837.36ms | tok/sec: 66896.0 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 587 |
+
[TIMING] data: 4.8ms | fwd: 2424.8ms | loss: 4.8ms | bwd: 5369.0ms | clip: 0.0ms | optim: 23.6ms
|
| 588 |
+
step 266 | loss: 1.981171 | lr 1.8000e-04 | norm: 0.7313 | dt: 7826.53ms | tok/sec: 66988.6 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 589 |
+
[TIMING] data: 4.8ms | fwd: 2415.2ms | loss: 4.8ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.7ms
|
| 590 |
+
step 267 | loss: 1.929014 | lr 1.8000e-04 | norm: 0.7204 | dt: 7827.35ms | tok/sec: 66981.6 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 591 |
+
[TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.5ms
|
| 592 |
+
step 268 | loss: 1.872691 | lr 1.8000e-04 | norm: 0.7225 | dt: 7832.84ms | tok/sec: 66934.6 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 593 |
+
[TIMING] data: 4.8ms | fwd: 2424.3ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.6ms
|
| 594 |
+
step 269 | loss: 2.023110 | lr 1.8000e-04 | norm: 0.7982 | dt: 7832.75ms | tok/sec: 66935.4 | mem: 72103MB | Time Left: 03 hrs : 59 mins
|
| 595 |
+
[TIMING] data: 4.8ms | fwd: 2413.6ms | loss: 4.8ms | bwd: 5376.9ms | clip: 0.0ms | optim: 23.7ms
|
| 596 |
+
step 270 | loss: 1.823170 | lr 1.8000e-04 | norm: 0.7132 | dt: 7826.80ms | tok/sec: 66986.2 | mem: 72103MB | Time Left: 03 hrs : 58 mins
|
| 597 |
+
[TIMING] data: 4.7ms | fwd: 2413.8ms | loss: 4.8ms | bwd: 5370.2ms | clip: 0.0ms | optim: 23.6ms
|
| 598 |
+
step 271 | loss: 1.888498 | lr 1.8000e-04 | norm: 0.8431 | dt: 7833.99ms | tok/sec: 66924.8 | mem: 72103MB | Time Left: 03 hrs : 58 mins
|
| 599 |
+
[TIMING] data: 4.9ms | fwd: 2420.4ms | loss: 4.8ms | bwd: 5370.0ms | clip: 0.0ms | optim: 23.7ms
|
| 600 |
+
step 272 | loss: 1.948550 | lr 1.8000e-04 | norm: 0.8378 | dt: 7827.51ms | tok/sec: 66980.2 | mem: 72103MB | Time Left: 03 hrs : 58 mins
|
| 601 |
+
[TIMING] data: 4.9ms | fwd: 2417.3ms | loss: 4.9ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.7ms
|
| 602 |
+
step 273 | loss: 1.855544 | lr 1.8000e-04 | norm: 0.8257 | dt: 7828.29ms | tok/sec: 66973.5 | mem: 72103MB | Time Left: 03 hrs : 58 mins
|
| 603 |
+
[TIMING] data: 4.8ms | fwd: 2417.6ms | loss: 4.9ms | bwd: 5366.8ms | clip: 0.0ms | optim: 23.6ms
|
| 604 |
+
step 274 | loss: 1.809207 | lr 1.8000e-04 | norm: 0.7570 | dt: 7832.35ms | tok/sec: 66938.8 | mem: 72103MB | Time Left: 03 hrs : 58 mins
|
| 605 |
+
[TIMING] data: 4.8ms | fwd: 2418.7ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
|
| 606 |
+
step 275 | loss: 1.990041 | lr 1.8000e-04 | norm: 0.8103 | dt: 7836.13ms | tok/sec: 66906.5 | mem: 72103MB | Time Left: 03 hrs : 58 mins
|
| 607 |
+
[TIMING] data: 4.8ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5374.6ms | clip: 0.0ms | optim: 23.7ms
|
| 608 |
+
step 276 | loss: 2.061124 | lr 1.8000e-04 | norm: 0.8882 | dt: 7832.03ms | tok/sec: 66941.5 | mem: 72103MB | Time Left: 03 hrs : 58 mins
|
| 609 |
+
[TIMING] data: 4.8ms | fwd: 2421.7ms | loss: 4.8ms | bwd: 5367.0ms | clip: 0.0ms | optim: 23.6ms
|
| 610 |
+
step 277 | loss: 1.789543 | lr 1.8000e-04 | norm: 0.8902 | dt: 7820.42ms | tok/sec: 67040.9 | mem: 72103MB | Time Left: 03 hrs : 57 mins
|
| 611 |
+
[TIMING] data: 4.8ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5368.1ms | clip: 0.0ms | optim: 23.5ms
|
| 612 |
+
step 278 | loss: 1.775792 | lr 1.8000e-04 | norm: 0.8987 | dt: 7828.34ms | tok/sec: 66973.1 | mem: 72103MB | Time Left: 03 hrs : 57 mins
|
| 613 |
+
[TIMING] data: 4.8ms | fwd: 2409.9ms | loss: 4.8ms | bwd: 5376.0ms | clip: 0.0ms | optim: 23.5ms
|
| 614 |
+
step 279 | loss: 1.932578 | lr 1.8000e-04 | norm: 0.8560 | dt: 7831.65ms | tok/sec: 66944.7 | mem: 72103MB | Time Left: 03 hrs : 57 mins
|
| 615 |
+
[TIMING] data: 4.9ms | fwd: 2418.5ms | loss: 4.8ms | bwd: 5370.7ms | clip: 0.0ms | optim: 23.6ms
|
| 616 |
+
step 280 | loss: 1.780883 | lr 1.8000e-04 | norm: 0.8304 | dt: 7820.13ms | tok/sec: 67043.4 | mem: 72103MB | Time Left: 03 hrs : 57 mins
|
| 617 |
+
[TIMING] data: 4.9ms | fwd: 2412.9ms | loss: 4.8ms | bwd: 5364.9ms | clip: 0.0ms | optim: 23.6ms
|
| 618 |
+
step 281 | loss: 1.759020 | lr 1.8000e-04 | norm: 0.9021 | dt: 7830.58ms | tok/sec: 66953.9 | mem: 72103MB | Time Left: 03 hrs : 57 mins
|
| 619 |
+
[TIMING] data: 4.9ms | fwd: 2415.8ms | loss: 4.9ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.6ms
|
| 620 |
+
step 282 | loss: 1.844155 | lr 1.8000e-04 | norm: 0.8803 | dt: 7829.19ms | tok/sec: 66965.8 | mem: 72103MB | Time Left: 03 hrs : 57 mins
|
| 621 |
+
[TIMING] data: 4.8ms | fwd: 2418.1ms | loss: 4.8ms | bwd: 5367.2ms | clip: 0.0ms | optim: 23.6ms
|
| 622 |
+
step 283 | loss: 1.751251 | lr 1.8000e-04 | norm: 0.8930 | dt: 7825.31ms | tok/sec: 66999.0 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 623 |
+
[TIMING] data: 4.7ms | fwd: 2412.6ms | loss: 4.8ms | bwd: 5369.3ms | clip: 0.0ms | optim: 23.7ms
|
| 624 |
+
step 284 | loss: 1.852886 | lr 1.8000e-04 | norm: 0.9555 | dt: 7816.54ms | tok/sec: 67074.2 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 625 |
+
[TIMING] data: 4.7ms | fwd: 2410.8ms | loss: 4.8ms | bwd: 5363.4ms | clip: 0.0ms | optim: 23.6ms
|
| 626 |
+
step 285 | loss: 1.759097 | lr 1.8000e-04 | norm: 0.8659 | dt: 7832.42ms | tok/sec: 66938.1 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 627 |
+
[TIMING] data: 4.8ms | fwd: 2419.9ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.6ms
|
| 628 |
+
step 286 | loss: 1.663844 | lr 1.8000e-04 | norm: 0.8886 | dt: 7827.00ms | tok/sec: 66984.6 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 629 |
+
[TIMING] data: 4.8ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5366.4ms | clip: 0.0ms | optim: 23.7ms
|
| 630 |
+
step 287 | loss: 1.697726 | lr 1.8000e-04 | norm: 1.0103 | dt: 7832.62ms | tok/sec: 66936.5 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 631 |
+
[TIMING] data: 4.8ms | fwd: 2415.3ms | loss: 4.8ms | bwd: 5373.1ms | clip: 0.0ms | optim: 23.6ms
|
| 632 |
+
step 288 | loss: 1.808231 | lr 1.8000e-04 | norm: 0.9205 | dt: 7825.12ms | tok/sec: 67000.6 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 633 |
+
[TIMING] data: 4.7ms | fwd: 2411.4ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.7ms
|
| 634 |
+
step 289 | loss: 1.785199 | lr 1.8000e-04 | norm: 0.9862 | dt: 7828.24ms | tok/sec: 66973.9 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 635 |
+
[TIMING] data: 4.8ms | fwd: 2414.4ms | loss: 4.9ms | bwd: 5370.6ms | clip: 0.0ms | optim: 23.6ms
|
| 636 |
+
step 290 | loss: 1.751698 | lr 1.8000e-04 | norm: 0.9281 | dt: 7833.58ms | tok/sec: 66928.3 | mem: 72103MB | Time Left: 03 hrs : 56 mins
|
| 637 |
+
[TIMING] data: 5.0ms | fwd: 2417.0ms | loss: 4.8ms | bwd: 5372.9ms | clip: 0.0ms | optim: 23.6ms
|
| 638 |
+
step 291 | loss: 1.654229 | lr 1.8000e-04 | norm: 1.0358 | dt: 7823.46ms | tok/sec: 67014.9 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 639 |
+
[TIMING] data: 4.9ms | fwd: 2414.9ms | loss: 4.8ms | bwd: 5365.2ms | clip: 0.0ms | optim: 23.6ms
|
| 640 |
+
step 292 | loss: 1.694017 | lr 1.8000e-04 | norm: 0.9111 | dt: 7817.90ms | tok/sec: 67062.6 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 641 |
+
[TIMING] data: 4.6ms | fwd: 2409.4ms | loss: 4.8ms | bwd: 5364.6ms | clip: 0.0ms | optim: 23.7ms
|
| 642 |
+
step 293 | loss: 1.738502 | lr 1.8000e-04 | norm: 0.9122 | dt: 7834.36ms | tok/sec: 66921.6 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 643 |
+
[TIMING] data: 4.7ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5374.1ms | clip: 0.0ms | optim: 23.7ms
|
| 644 |
+
step 294 | loss: 1.743377 | lr 1.8000e-04 | norm: 0.9709 | dt: 7830.94ms | tok/sec: 66950.9 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 645 |
+
[TIMING] data: 4.6ms | fwd: 2414.7ms | loss: 4.8ms | bwd: 5373.5ms | clip: 0.0ms | optim: 23.5ms
|
| 646 |
+
step 295 | loss: 1.714503 | lr 1.8000e-04 | norm: 1.0028 | dt: 7832.42ms | tok/sec: 66938.1 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 647 |
+
[TIMING] data: 4.9ms | fwd: 2422.4ms | loss: 4.8ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
|
| 648 |
+
step 296 | loss: 1.547332 | lr 1.8000e-04 | norm: 1.1973 | dt: 7820.53ms | tok/sec: 67040.0 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 649 |
+
[TIMING] data: 4.9ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5366.2ms | clip: 0.0ms | optim: 23.6ms
|
| 650 |
+
step 297 | loss: 1.803487 | lr 1.8000e-04 | norm: 1.1564 | dt: 7826.04ms | tok/sec: 66992.7 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 651 |
+
[TIMING] data: 4.8ms | fwd: 2415.8ms | loss: 4.8ms | bwd: 5366.8ms | clip: 0.0ms | optim: 23.7ms
|
| 652 |
+
step 298 | loss: 1.749850 | lr 1.8000e-04 | norm: 1.0954 | dt: 7816.46ms | tok/sec: 67074.9 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 653 |
+
[TIMING] data: 4.8ms | fwd: 2408.7ms | loss: 4.9ms | bwd: 5363.5ms | clip: 0.0ms | optim: 23.7ms
|
| 654 |
+
step 299 | loss: 1.528509 | lr 1.8000e-04 | norm: 0.9575 | dt: 7821.90ms | tok/sec: 67028.2 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 655 |
+
[TIMING] data: 4.7ms | fwd: 2411.5ms | loss: 4.9ms | bwd: 5368.1ms | clip: 0.0ms | optim: 23.6ms
|
| 656 |
+
step 300 | loss: 1.689287 | lr 1.8000e-04 | norm: 1.0337 | dt: 7834.38ms | tok/sec: 66921.4 | mem: 72103MB | Time Left: 03 hrs : 55 mins
|
| 657 |
+
[TIMING] data: 4.8ms | fwd: 2423.3ms | loss: 4.8ms | bwd: 5367.0ms | clip: 0.0ms | optim: 23.6ms
|
| 658 |
+
step 301 | loss: 1.591161 | lr 1.8000e-04 | norm: 0.9519 | dt: 7830.91ms | tok/sec: 66951.1 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 659 |
+
[TIMING] data: 4.7ms | fwd: 2413.6ms | loss: 4.9ms | bwd: 5374.5ms | clip: 0.0ms | optim: 23.6ms
|
| 660 |
+
step 302 | loss: 1.621496 | lr 1.8000e-04 | norm: 0.9707 | dt: 7832.02ms | tok/sec: 66941.6 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 661 |
+
[TIMING] data: 4.8ms | fwd: 2420.6ms | loss: 4.9ms | bwd: 5367.9ms | clip: 0.0ms | optim: 23.7ms
|
| 662 |
+
step 303 | loss: 1.690582 | lr 1.8000e-04 | norm: 1.0774 | dt: 7830.73ms | tok/sec: 66952.6 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 663 |
+
[TIMING] data: 4.8ms | fwd: 2417.7ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.7ms
|
| 664 |
+
step 304 | loss: 1.548516 | lr 1.8000e-04 | norm: 1.1629 | dt: 7832.97ms | tok/sec: 66933.5 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 665 |
+
[TIMING] data: 4.9ms | fwd: 2414.6ms | loss: 4.8ms | bwd: 5375.1ms | clip: 0.0ms | optim: 23.5ms
|
| 666 |
+
step 305 | loss: 1.640994 | lr 1.8000e-04 | norm: 0.9619 | dt: 7833.52ms | tok/sec: 66928.8 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 667 |
+
[TIMING] data: 4.9ms | fwd: 2415.7ms | loss: 4.8ms | bwd: 5374.0ms | clip: 0.0ms | optim: 23.7ms
|
| 668 |
+
step 306 | loss: 1.644059 | lr 1.8000e-04 | norm: 0.9927 | dt: 7832.93ms | tok/sec: 66933.9 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 669 |
+
[TIMING] data: 4.9ms | fwd: 2420.6ms | loss: 4.9ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.5ms
|
| 670 |
+
step 307 | loss: 1.726431 | lr 1.8000e-04 | norm: 1.0415 | dt: 7833.70ms | tok/sec: 66927.3 | mem: 72103MB | Time Left: 03 hrs : 54 mins
|
| 671 |
+
[TIMING] data: 4.9ms | fwd: 2426.4ms | loss: 4.9ms | bwd: 5363.4ms | clip: 0.0ms | optim: 23.6ms
|
| 672 |
+
step 308 | loss: 1.540038 | lr 1.8000e-04 | norm: 1.0317 | dt: 7831.07ms | tok/sec: 66949.8 | mem: 72103MB | Time Left: 03 hrs : 53 mins
|
| 673 |
+
[TIMING] data: 4.9ms | fwd: 2420.6ms | loss: 4.8ms | bwd: 5368.2ms | clip: 0.0ms | optim: 23.6ms
|
| 674 |
+
step 309 | loss: 1.591404 | lr 1.8000e-04 | norm: 1.0330 | dt: 7826.58ms | tok/sec: 66988.1 | mem: 72103MB | Time Left: 03 hrs : 53 mins
|
| 675 |
+
[TIMING] data: 4.8ms | fwd: 2420.3ms | loss: 4.8ms | bwd: 5363.0ms | clip: 0.0ms | optim: 23.7ms
|
| 676 |
+
step 310 | loss: 1.633510 | lr 1.8000e-04 | norm: 1.1299 | dt: 7828.29ms | tok/sec: 66973.5 | mem: 72103MB | Time Left: 03 hrs : 53 mins
|
| 677 |
+
[TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.9ms | bwd: 5368.9ms | clip: 0.0ms | optim: 23.7ms
|
| 678 |
+
step 311 | loss: 1.665028 | lr 1.8000e-04 | norm: 1.1354 | dt: 7829.07ms | tok/sec: 66966.8 | mem: 72103MB | Time Left: 03 hrs : 53 mins
|
| 679 |
+
[TIMING] data: 4.8ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.6ms
|
| 680 |
+
step 312 | loss: 1.801152 | lr 1.8000e-04 | norm: 1.1436 | dt: 7821.64ms | tok/sec: 67030.5 | mem: 72103MB | Time Left: 03 hrs : 53 mins
|
| 681 |
+
[TIMING] data: 4.8ms | fwd: 2413.8ms | loss: 4.8ms | bwd: 5364.7ms | clip: 0.0ms | optim: 23.7ms
|
| 682 |
+
step 313 | loss: 1.527494 | lr 1.8000e-04 | norm: 1.0534 | dt: 7827.79ms | tok/sec: 66977.8 | mem: 72103MB | Time Left: 03 hrs : 53 mins
|
| 683 |
+
[TIMING] data: 5.0ms | fwd: 2413.5ms | loss: 4.8ms | bwd: 5371.1ms | clip: 0.0ms | optim: 23.6ms
|
| 684 |
+
step 314 | loss: 1.497566 | lr 1.8000e-04 | norm: 1.1055 | dt: 7827.65ms | tok/sec: 66979.0 | mem: 72103MB | Time Left: 03 hrs : 53 mins
|
| 685 |
+
[TIMING] data: 4.9ms | fwd: 2420.2ms | loss: 4.8ms | bwd: 5364.6ms | clip: 0.0ms | optim: 23.6ms
|
| 686 |
+
step 315 | loss: 1.600274 | lr 1.8000e-04 | norm: 1.1916 | dt: 7826.63ms | tok/sec: 66987.7 | mem: 72103MB | Time Left: 03 hrs : 52 mins
|
| 687 |
+
[TIMING] data: 5.0ms | fwd: 2413.3ms | loss: 4.8ms | bwd: 5369.5ms | clip: 0.0ms | optim: 23.7ms
|
| 688 |
+
step 316 | loss: 1.650586 | lr 1.8000e-04 | norm: 1.2763 | dt: 7829.62ms | tok/sec: 66962.1 | mem: 72103MB | Time Left: 03 hrs : 52 mins
|
| 689 |
+
[TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
|
| 690 |
+
step 317 | loss: 1.376261 | lr 1.8000e-04 | norm: 1.0453 | dt: 7825.58ms | tok/sec: 66996.7 | mem: 72103MB | Time Left: 03 hrs : 52 mins
|
| 691 |
+
[TIMING] data: 4.8ms | fwd: 2416.7ms | loss: 4.8ms | bwd: 5365.5ms | clip: 0.0ms | optim: 23.7ms
|
| 692 |
+
step 318 | loss: 1.670848 | lr 1.8000e-04 | norm: 1.1326 | dt: 7828.62ms | tok/sec: 66970.7 | mem: 72103MB | Time Left: 03 hrs : 52 mins
|
| 693 |
+
[TIMING] data: 4.9ms | fwd: 2415.0ms | loss: 4.8ms | bwd: 5369.7ms | clip: 0.0ms | optim: 23.7ms
|
| 694 |
+
step 319 | loss: 1.627407 | lr 1.8000e-04 | norm: 1.1636 | dt: 7824.08ms | tok/sec: 67009.5 | mem: 72103MB | Time Left: 03 hrs : 52 mins
|
| 695 |
+
[TIMING] data: 4.9ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5364.9ms | clip: 0.0ms | optim: 23.6ms
|
| 696 |
+
step 320 | loss: 1.483035 | lr 1.8000e-04 | norm: 1.0456 | dt: 7829.48ms | tok/sec: 66963.3 | mem: 72103MB | Time Left: 03 hrs : 52 mins
|
| 697 |
+
[TIMING] data: 5.0ms | fwd: 2417.3ms | loss: 4.8ms | bwd: 5368.7ms | clip: 0.0ms | optim: 23.6ms
|
| 698 |
+
step 321 | loss: 1.488142 | lr 1.8000e-04 | norm: 1.2098 | dt: 7829.83ms | tok/sec: 66960.3 | mem: 72103MB | Time Left: 03 hrs : 52 mins
|
| 699 |
+
[TIMING] data: 4.9ms | fwd: 2417.6ms | loss: 4.9ms | bwd: 5368.4ms | clip: 0.0ms | optim: 23.7ms
|
| 700 |
+
step 322 | loss: 1.566001 | lr 1.8000e-04 | norm: 1.1490 | dt: 7822.35ms | tok/sec: 67024.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 701 |
+
[TIMING] data: 4.6ms | fwd: 2406.0ms | loss: 4.8ms | bwd: 5373.2ms | clip: 0.0ms | optim: 23.6ms
|
| 702 |
+
step 323 | loss: 1.489413 | lr 1.8000e-04 | norm: 1.1245 | dt: 7830.54ms | tok/sec: 66954.2 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 703 |
+
[TIMING] data: 4.8ms | fwd: 2417.9ms | loss: 4.8ms | bwd: 5369.8ms | clip: 0.0ms | optim: 23.6ms
|
| 704 |
+
step 324 | loss: 1.493618 | lr 1.8000e-04 | norm: 1.0603 | dt: 7830.81ms | tok/sec: 66952.0 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 705 |
+
[TIMING] data: 4.8ms | fwd: 2411.3ms | loss: 4.8ms | bwd: 5377.1ms | clip: 0.0ms | optim: 23.6ms
|
| 706 |
+
step 325 | loss: 1.401132 | lr 1.8000e-04 | norm: 1.2034 | dt: 7830.52ms | tok/sec: 66954.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 707 |
+
[TIMING] data: 4.8ms | fwd: 2415.1ms | loss: 4.8ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
|
| 708 |
+
step 326 | loss: 1.459075 | lr 1.8000e-04 | norm: 1.0744 | dt: 7823.87ms | tok/sec: 67011.3 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 709 |
+
[TIMING] data: 4.6ms | fwd: 2409.8ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.6ms
|
| 710 |
+
step 327 | loss: 1.541776 | lr 1.8000e-04 | norm: 1.0930 | dt: 7830.52ms | tok/sec: 66954.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 711 |
+
[TIMING] data: 4.9ms | fwd: 2415.9ms | loss: 4.8ms | bwd: 5370.8ms | clip: 0.0ms | optim: 23.6ms
|
| 712 |
+
step 328 | loss: 1.294119 | lr 1.8000e-04 | norm: 1.0159 | dt: 7830.85ms | tok/sec: 66951.6 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 713 |
+
[TIMING] data: 4.9ms | fwd: 2418.7ms | loss: 4.8ms | bwd: 5368.6ms | clip: 0.0ms | optim: 23.6ms
|
| 714 |
+
step 329 | loss: 1.483784 | lr 1.8000e-04 | norm: 1.1200 | dt: 7829.13ms | tok/sec: 66966.4 | mem: 72103MB | Time Left: 03 hrs : 51 mins
|
| 715 |
+
[TIMING] data: 4.8ms | fwd: 2418.8ms | loss: 4.8ms | bwd: 5368.0ms | clip: 0.0ms | optim: 23.6ms
|
| 716 |
+
step 330 | loss: 1.438243 | lr 1.8000e-04 | norm: 1.0510 | dt: 7828.59ms | tok/sec: 66970.9 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 717 |
+
[TIMING] data: 4.8ms | fwd: 2419.1ms | loss: 4.8ms | bwd: 5367.3ms | clip: 0.0ms | optim: 23.6ms
|
| 718 |
+
step 331 | loss: 1.616356 | lr 1.8000e-04 | norm: 1.3004 | dt: 7822.71ms | tok/sec: 67021.3 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 719 |
+
[TIMING] data: 4.7ms | fwd: 2414.3ms | loss: 4.8ms | bwd: 5366.1ms | clip: 0.0ms | optim: 23.6ms
|
| 720 |
+
step 332 | loss: 1.431027 | lr 1.8000e-04 | norm: 1.2038 | dt: 7828.60ms | tok/sec: 66970.9 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 721 |
+
[TIMING] data: 4.9ms | fwd: 2413.6ms | loss: 4.8ms | bwd: 5372.1ms | clip: 0.0ms | optim: 23.7ms
|
| 722 |
+
step 333 | loss: 1.425855 | lr 1.8000e-04 | norm: 1.1500 | dt: 7828.50ms | tok/sec: 66971.7 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 723 |
+
[TIMING] data: 4.8ms | fwd: 2414.1ms | loss: 4.8ms | bwd: 5371.4ms | clip: 0.0ms | optim: 23.6ms
|
| 724 |
+
step 334 | loss: 1.451784 | lr 1.8000e-04 | norm: 1.1428 | dt: 7835.18ms | tok/sec: 66914.6 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 725 |
+
[TIMING] data: 4.9ms | fwd: 2422.4ms | loss: 4.8ms | bwd: 5369.6ms | clip: 0.0ms | optim: 23.7ms
|
| 726 |
+
step 335 | loss: 1.505677 | lr 1.8000e-04 | norm: 1.2136 | dt: 7826.60ms | tok/sec: 66988.0 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 727 |
+
[TIMING] data: 4.8ms | fwd: 2409.7ms | loss: 4.8ms | bwd: 5373.4ms | clip: 0.0ms | optim: 23.7ms
|
| 728 |
+
step 336 | loss: 1.482544 | lr 1.8000e-04 | norm: 1.1122 | dt: 7829.84ms | tok/sec: 66960.2 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 729 |
+
[TIMING] data: 4.7ms | fwd: 2415.8ms | loss: 4.9ms | bwd: 5371.5ms | clip: 0.0ms | optim: 23.6ms
|
| 730 |
+
step 337 | loss: 1.457365 | lr 1.8000e-04 | norm: 1.1931 | dt: 7830.49ms | tok/sec: 66954.7 | mem: 72103MB | Time Left: 03 hrs : 50 mins
|
| 731 |
+
[TIMING] data: 4.8ms | fwd: 2416.5ms | loss: 4.8ms | bwd: 5371.6ms | clip: 0.0ms | optim: 23.7ms
|
| 732 |
+
step 338 | loss: 1.440003 | lr 1.8000e-04 | norm: 1.1547 | dt: 7830.54ms | tok/sec: 66954.2 | mem: 72103MB | Time Left: 03 hrs : 49 mins
|
| 733 |
+
[TIMING] data: 4.8ms | fwd: 2415.6ms | loss: 4.8ms | bwd: 5372.3ms | clip: 0.0ms | optim: 23.6ms
|