Upload logs/2026-09-23_t128d_train.log with huggingface_hub
Browse files- logs/2026-09-23_t128d_train.log +789 -0
logs/2026-09-23_t128d_train.log
ADDED
|
@@ -0,0 +1,789 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
=== Llama Context Parallel Training Script (bluscriptCP) ===
|
| 2 |
+
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
|
| 3 |
+
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
|
| 4 |
+
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
|
| 5 |
+
================ Configuration (bluscriptCP) ================
|
| 6 |
+
d_model/layers : 1536 / 24
|
| 7 |
+
q/kv heads : 12 / 4 (G=3, hd=128)
|
| 8 |
+
ffn_hidden : 4096 (SwiGLU)
|
| 9 |
+
context_length : 131072
|
| 10 |
+
vocab_size : 50304 tying=1
|
| 11 |
+
B x T : 1 x 131072
|
| 12 |
+
world_size : 4 (dp=1 x cp=4)
|
| 13 |
+
CP_ATTN_MODE : ring (load_balance=HeadTail)
|
| 14 |
+
global_batch : 1048576 (grad_accum=8)
|
| 15 |
+
precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
|
| 16 |
+
max_lr/min_lr : 0.0006 / 6e-05
|
| 17 |
+
warmup/steps : 60 / 360 (WSD: stable=240, decay tail=60)
|
| 18 |
+
checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp)
|
| 19 |
+
=============================================================
|
| 20 |
+
Initializing model on CUDA device 0...
|
| 21 |
+
[YaRN] build_rope_cache: seq_len=131072 hd=128 scale=32.000 low=14 high=32 m=1.3466
|
| 22 |
+
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/t64.ckpt (base step 1200, loss 2.79842) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
|
| 23 |
+
Parameters: 681328128
|
| 24 |
+
Parameters per GPU: 681328128
|
| 25 |
+
max_steps: 360
|
| 26 |
+
warmup_steps: 60
|
| 27 |
+
[ZeRO-1/Muon] reduce-scatter/all-gather layout: 6 bucket(s), 12 collectives/step, shard 150994944 elems/rank, padding 0.000000% (flat buffer 1152.000000 MB)
|
| 28 |
+
[ZeRO-1/Muon] adjust_lr=match_rms_adamw | params=603979776 (144 orthogonalized / 0 AdamW) muon_elems=603979776 | W=4 owned=150994944 (36 tensors) | master+momentum: 1152.000000 MB/GPU
|
| 29 |
+
[ZeRO-1/Muon] 6 bucket(s) would pad 299.665720% (> 5.000000% budget); falling back to 3
|
| 30 |
+
[ZeRO-1/Muon] 3 bucket(s) would pad 299.648510% (> 5.000000% budget); falling back to 1
|
| 31 |
+
[ZeRO-1/Muon] reduce-scatter/all-gather layout: 1 bucket(s), 2 collectives/step, shard 77266944 elems/rank, padding 299.579006% (flat buffer 1179.000000 MB)
|
| 32 |
+
[ZeRO-1/Muon] adjust_lr=match_rms_adamw | params=77348352 (1 orthogonalized / 97 AdamW) muon_elems=77266944 | W=4 owned=77266944 (1 tensors) | master+momentum: 589.500000 MB/GPU
|
| 33 |
+
[ZeRO-1] optimizer=MUON+adamw-aux variant=muon whole-tensor shard groups=2 [bfloat16/603979776] [float32/77348352]
|
| 34 |
+
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
|
| 35 |
+
[precision] bf16 params=144, all with fp32 master weights
|
| 36 |
+
found 7 shards for split train
|
| 37 |
+
found 1 shards for split val
|
| 38 |
+
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (run 2)
|
| 39 |
+
[rope_theta] 500000 from CP_ROPE_THETA (no sidecar -- provenance UNVERIFIED against the checkpoint)
|
| 40 |
+
Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 4)
|
| 41 |
+
[re-warmup] anchored at step 0 for 60 steps -> peak 0.000180 (0.300000 x max_lr), then cosine-decay to 0.000060 | resuming at step 0 (mid-ramp)
|
| 42 |
+
|
| 43 |
+
Starting training...
|
| 44 |
+
|
| 45 |
+
[CP ring P2P] OVERLAP: dedicated non-blocking stream
|
| 46 |
+
validation loss: 2.6101
|
| 47 |
+
[CP bwd] overlap=ON
|
| 48 |
+
step 0 | loss: 2.969628 | lr 3.0000e-06 | norm: 1.3257 | dt: 45288.19ms | tok/sec: 23153.4 | mem: 123471MB | Time Left: 04 hrs : 31 mins
|
| 49 |
+
[TIMING] data: 4.1ms | fwd: 6435.7ms | loss: 12.7ms | bwd: 22720.2ms | clip: 0.0ms | optim: 55.7ms
|
| 50 |
+
step 1 | loss: 3.152593 | lr 6.0000e-06 | norm: 2.2774 | dt: 30340.73ms | tok/sec: 34560.0 | mem: 122287MB | Time Left: 03 hrs : 01 mins
|
| 51 |
+
[TIMING] data: 3.8ms | fwd: 6731.9ms | loss: 13.7ms | bwd: 23544.0ms | clip: 0.0ms | optim: 41.6ms
|
| 52 |
+
step 2 | loss: 2.789464 | lr 9.0000e-06 | norm: 0.3694 | dt: 30962.45ms | tok/sec: 33866.1 | mem: 122575MB | Time Left: 03 hrs : 04 mins
|
| 53 |
+
[TIMING] data: 3.9ms | fwd: 6845.1ms | loss: 13.1ms | bwd: 24052.9ms | clip: 0.0ms | optim: 41.6ms
|
| 54 |
+
step 3 | loss: 2.162413 | lr 1.2000e-05 | norm: 0.7037 | dt: 31342.13ms | tok/sec: 33455.8 | mem: 122575MB | Time Left: 03 hrs : 06 mins
|
| 55 |
+
[TIMING] data: 3.3ms | fwd: 6991.3ms | loss: 13.5ms | bwd: 24286.6ms | clip: 0.0ms | optim: 41.8ms
|
| 56 |
+
step 4 | loss: 3.272386 | lr 1.5000e-05 | norm: 0.6763 | dt: 31592.73ms | tok/sec: 33190.4 | mem: 122575MB | Time Left: 03 hrs : 07 mins
|
| 57 |
+
[TIMING] data: 3.3ms | fwd: 6969.3ms | loss: 13.5ms | bwd: 24559.7ms | clip: 0.0ms | optim: 41.4ms
|
| 58 |
+
step 5 | loss: 2.677945 | lr 1.8000e-05 | norm: 0.5824 | dt: 31642.60ms | tok/sec: 33138.1 | mem: 122575MB | Time Left: 03 hrs : 07 mins
|
| 59 |
+
[TIMING] data: 3.3ms | fwd: 6967.0ms | loss: 13.6ms | bwd: 24611.4ms | clip: 0.0ms | optim: 41.6ms
|
| 60 |
+
step 6 | loss: 2.484816 | lr 2.1000e-05 | norm: 1.3994 | dt: 31755.37ms | tok/sec: 33020.4 | mem: 122575MB | Time Left: 03 hrs : 07 mins
|
| 61 |
+
[TIMING] data: 3.2ms | fwd: 6996.2ms | loss: 13.5ms | bwd: 24695.2ms | clip: 0.0ms | optim: 41.5ms
|
| 62 |
+
step 7 | loss: 2.953887 | lr 2.4000e-05 | norm: 0.7041 | dt: 31851.69ms | tok/sec: 32920.6 | mem: 122575MB | Time Left: 03 hrs : 07 mins
|
| 63 |
+
[TIMING] data: 3.3ms | fwd: 7066.0ms | loss: 13.6ms | bwd: 24720.2ms | clip: 0.0ms | optim: 42.8ms
|
| 64 |
+
step 8 | loss: 2.478380 | lr 2.7000e-05 | norm: 0.6970 | dt: 31748.90ms | tok/sec: 33027.2 | mem: 122575MB | Time Left: 03 hrs : 06 mins
|
| 65 |
+
[TIMING] data: 3.3ms | fwd: 7015.6ms | loss: 14.3ms | bwd: 24668.5ms | clip: 0.0ms | optim: 41.5ms
|
| 66 |
+
step 9 | loss: 2.564464 | lr 3.0000e-05 | norm: 0.3920 | dt: 31779.69ms | tok/sec: 32995.2 | mem: 122575MB | Time Left: 03 hrs : 05 mins
|
| 67 |
+
[TIMING] data: 3.3ms | fwd: 7049.6ms | loss: 13.6ms | bwd: 24666.1ms | clip: 0.0ms | optim: 41.5ms
|
| 68 |
+
step 10 | loss: 2.568440 | lr 3.3000e-05 | norm: 0.7897 | dt: 31814.16ms | tok/sec: 32959.4 | mem: 122575MB | Time Left: 03 hrs : 05 mins
|
| 69 |
+
[TIMING] data: 3.3ms | fwd: 7095.0ms | loss: 13.8ms | bwd: 24652.8ms | clip: 0.0ms | optim: 43.6ms
|
| 70 |
+
step 11 | loss: 2.728641 | lr 3.6000e-05 | norm: 0.5363 | dt: 31876.36ms | tok/sec: 32895.1 | mem: 122575MB | Time Left: 03 hrs : 05 mins
|
| 71 |
+
[TIMING] data: 3.3ms | fwd: 7087.6ms | loss: 13.6ms | bwd: 24724.4ms | clip: 0.0ms | optim: 41.8ms
|
| 72 |
+
step 12 | loss: 2.793588 | lr 3.9000e-05 | norm: 0.4317 | dt: 31839.81ms | tok/sec: 32932.9 | mem: 122575MB | Time Left: 03 hrs : 04 mins
|
| 73 |
+
[TIMING] data: 3.2ms | fwd: 7028.9ms | loss: 13.7ms | bwd: 24746.6ms | clip: 0.0ms | optim: 41.8ms
|
| 74 |
+
step 13 | loss: 2.883485 | lr 4.2000e-05 | norm: 0.7043 | dt: 31801.48ms | tok/sec: 32972.6 | mem: 122575MB | Time Left: 03 hrs : 03 mins
|
| 75 |
+
[TIMING] data: 3.3ms | fwd: 7014.1ms | loss: 13.5ms | bwd: 24723.1ms | clip: 0.0ms | optim: 42.0ms
|
| 76 |
+
step 14 | loss: 2.852813 | lr 4.5000e-05 | norm: 0.5115 | dt: 31935.84ms | tok/sec: 32833.8 | mem: 122575MB | Time Left: 03 hrs : 04 mins
|
| 77 |
+
[TIMING] data: 3.3ms | fwd: 7084.2ms | loss: 13.6ms | bwd: 24787.3ms | clip: 0.0ms | optim: 41.8ms
|
| 78 |
+
step 15 | loss: 2.654794 | lr 4.8000e-05 | norm: 0.5653 | dt: 31845.22ms | tok/sec: 32927.3 | mem: 122575MB | Time Left: 03 hrs : 03 mins
|
| 79 |
+
[TIMING] data: 3.3ms | fwd: 7079.3ms | loss: 14.3ms | bwd: 24700.9ms | clip: 0.0ms | optim: 41.7ms
|
| 80 |
+
step 16 | loss: 2.573109 | lr 5.1000e-05 | norm: 0.2298 | dt: 31801.73ms | tok/sec: 32972.3 | mem: 122575MB | Time Left: 03 hrs : 02 mins
|
| 81 |
+
[TIMING] data: 3.3ms | fwd: 7038.7ms | loss: 13.5ms | bwd: 24698.9ms | clip: 0.0ms | optim: 41.7ms
|
| 82 |
+
step 17 | loss: 2.674510 | lr 5.4000e-05 | norm: 0.5756 | dt: 31928.54ms | tok/sec: 32841.3 | mem: 122575MB | Time Left: 03 hrs : 02 mins
|
| 83 |
+
[TIMING] data: 3.3ms | fwd: 7108.6ms | loss: 13.5ms | bwd: 24751.6ms | clip: 0.0ms | optim: 45.9ms
|
| 84 |
+
step 18 | loss: 2.763754 | lr 5.7000e-05 | norm: 0.4289 | dt: 31760.20ms | tok/sec: 33015.4 | mem: 122575MB | Time Left: 03 hrs : 01 mins
|
| 85 |
+
[TIMING] data: 3.3ms | fwd: 7058.4ms | loss: 13.7ms | bwd: 24637.5ms | clip: 0.0ms | optim: 41.7ms
|
| 86 |
+
step 19 | loss: 2.036735 | lr 6.0000e-05 | norm: 0.3899 | dt: 31798.52ms | tok/sec: 32975.6 | mem: 122575MB | Time Left: 03 hrs : 00 mins
|
| 87 |
+
[TIMING] data: 3.2ms | fwd: 7039.2ms | loss: 13.6ms | bwd: 24695.1ms | clip: 0.0ms | optim: 41.8ms
|
| 88 |
+
step 20 | loss: 2.614365 | lr 6.3000e-05 | norm: 0.2101 | dt: 31911.62ms | tok/sec: 32858.8 | mem: 122575MB | Time Left: 03 hrs : 00 mins
|
| 89 |
+
[TIMING] data: 3.3ms | fwd: 7131.2ms | loss: 13.6ms | bwd: 24714.4ms | clip: 0.0ms | optim: 43.4ms
|
| 90 |
+
step 21 | loss: 2.712849 | lr 6.6000e-05 | norm: 0.4051 | dt: 31811.64ms | tok/sec: 32962.0 | mem: 122575MB | Time Left: 02 hrs : 59 mins
|
| 91 |
+
[TIMING] data: 3.4ms | fwd: 7098.6ms | loss: 14.3ms | bwd: 24647.8ms | clip: 0.0ms | optim: 42.0ms
|
| 92 |
+
step 22 | loss: 2.816557 | lr 6.9000e-05 | norm: 1.1959 | dt: 31895.63ms | tok/sec: 32875.2 | mem: 122575MB | Time Left: 02 hrs : 59 mins
|
| 93 |
+
[TIMING] data: 3.3ms | fwd: 7106.4ms | loss: 13.7ms | bwd: 24724.9ms | clip: 0.0ms | optim: 41.6ms
|
| 94 |
+
step 23 | loss: 2.480559 | lr 7.2000e-05 | norm: 0.3078 | dt: 31780.54ms | tok/sec: 32994.3 | mem: 122575MB | Time Left: 02 hrs : 58 mins
|
| 95 |
+
[TIMING] data: 3.3ms | fwd: 7054.6ms | loss: 14.3ms | bwd: 24660.8ms | clip: 0.0ms | optim: 41.8ms
|
| 96 |
+
step 24 | loss: 2.138457 | lr 7.5000e-05 | norm: 0.2785 | dt: 31856.68ms | tok/sec: 32915.4 | mem: 122575MB | Time Left: 02 hrs : 58 mins
|
| 97 |
+
[TIMING] data: 3.3ms | fwd: 7086.9ms | loss: 13.8ms | bwd: 24705.1ms | clip: 0.0ms | optim: 41.9ms
|
| 98 |
+
step 25 | loss: 2.546346 | lr 7.8000e-05 | norm: 0.5362 | dt: 31804.25ms | tok/sec: 32969.7 | mem: 122575MB | Time Left: 02 hrs : 57 mins
|
| 99 |
+
[TIMING] data: 3.3ms | fwd: 7057.3ms | loss: 13.7ms | bwd: 24682.4ms | clip: 0.0ms | optim: 41.9ms
|
| 100 |
+
step 26 | loss: 2.728122 | lr 8.1000e-05 | norm: 0.4817 | dt: 31860.64ms | tok/sec: 32911.3 | mem: 122575MB | Time Left: 02 hrs : 57 mins
|
| 101 |
+
[TIMING] data: 3.3ms | fwd: 7028.5ms | loss: 14.1ms | bwd: 24767.1ms | clip: 0.0ms | optim: 41.9ms
|
| 102 |
+
step 27 | loss: 2.261347 | lr 8.4000e-05 | norm: 0.2389 | dt: 31754.42ms | tok/sec: 33021.4 | mem: 122575MB | Time Left: 02 hrs : 56 mins
|
| 103 |
+
[TIMING] data: 3.3ms | fwd: 7055.4ms | loss: 13.6ms | bwd: 24634.6ms | clip: 0.0ms | optim: 41.9ms
|
| 104 |
+
step 28 | loss: 2.286111 | lr 8.7000e-05 | norm: 0.2671 | dt: 31708.45ms | tok/sec: 33069.3 | mem: 122575MB | Time Left: 02 hrs : 55 mins
|
| 105 |
+
[TIMING] data: 3.3ms | fwd: 7051.1ms | loss: 13.7ms | bwd: 24592.7ms | clip: 0.0ms | optim: 41.9ms
|
| 106 |
+
step 29 | loss: 2.246032 | lr 9.0000e-05 | norm: 0.1993 | dt: 31824.99ms | tok/sec: 32948.2 | mem: 122575MB | Time Left: 02 hrs : 55 mins
|
| 107 |
+
[TIMING] data: 3.3ms | fwd: 7107.4ms | loss: 13.5ms | bwd: 24651.8ms | clip: 0.0ms | optim: 43.4ms
|
| 108 |
+
step 30 | loss: 2.861277 | lr 9.3000e-05 | norm: 0.3692 | dt: 31770.26ms | tok/sec: 33005.0 | mem: 122575MB | Time Left: 02 hrs : 54 mins
|
| 109 |
+
[TIMING] data: 3.2ms | fwd: 7024.5ms | loss: 13.5ms | bwd: 24681.8ms | clip: 0.0ms | optim: 41.7ms
|
| 110 |
+
step 31 | loss: 2.563782 | lr 9.6000e-05 | norm: 0.4707 | dt: 31850.74ms | tok/sec: 32921.6 | mem: 122575MB | Time Left: 02 hrs : 54 mins
|
| 111 |
+
[TIMING] data: 3.3ms | fwd: 7025.7ms | loss: 13.5ms | bwd: 24760.9ms | clip: 0.0ms | optim: 41.8ms
|
| 112 |
+
step 32 | loss: 2.794254 | lr 9.9000e-05 | norm: 0.3234 | dt: 31666.98ms | tok/sec: 33112.6 | mem: 122575MB | Time Left: 02 hrs : 53 mins
|
| 113 |
+
[TIMING] data: 3.3ms | fwd: 7065.3ms | loss: 14.2ms | bwd: 24536.6ms | clip: 0.0ms | optim: 41.9ms
|
| 114 |
+
step 33 | loss: 2.382182 | lr 1.0200e-04 | norm: 0.3691 | dt: 31782.10ms | tok/sec: 32992.7 | mem: 122575MB | Time Left: 02 hrs : 53 mins
|
| 115 |
+
[TIMING] data: 3.3ms | fwd: 7044.6ms | loss: 13.5ms | bwd: 24673.1ms | clip: 0.0ms | optim: 42.0ms
|
| 116 |
+
step 34 | loss: 2.084465 | lr 1.0500e-04 | norm: 0.2407 | dt: 31793.13ms | tok/sec: 32981.2 | mem: 122575MB | Time Left: 02 hrs : 52 mins
|
| 117 |
+
[TIMING] data: 3.3ms | fwd: 7052.6ms | loss: 13.6ms | bwd: 24675.8ms | clip: 0.0ms | optim: 42.2ms
|
| 118 |
+
step 35 | loss: 2.140383 | lr 1.0800e-04 | norm: 0.2107 | dt: 31795.51ms | tok/sec: 32978.7 | mem: 122575MB | Time Left: 02 hrs : 52 mins
|
| 119 |
+
[TIMING] data: 3.3ms | fwd: 7007.2ms | loss: 13.6ms | bwd: 24724.3ms | clip: 0.0ms | optim: 41.6ms
|
| 120 |
+
step 36 | loss: 2.576785 | lr 1.1100e-04 | norm: 0.5852 | dt: 31798.31ms | tok/sec: 32975.8 | mem: 122575MB | Time Left: 02 hrs : 51 mins
|
| 121 |
+
[TIMING] data: 3.3ms | fwd: 6979.6ms | loss: 13.5ms | bwd: 24753.6ms | clip: 0.0ms | optim: 42.7ms
|
| 122 |
+
step 37 | loss: 2.666378 | lr 1.1400e-04 | norm: 0.4316 | dt: 31738.20ms | tok/sec: 33038.3 | mem: 122575MB | Time Left: 02 hrs : 50 mins
|
| 123 |
+
[TIMING] data: 3.3ms | fwd: 6992.6ms | loss: 13.5ms | bwd: 24681.9ms | clip: 0.0ms | optim: 41.4ms
|
| 124 |
+
step 38 | loss: 2.336522 | lr 1.1700e-04 | norm: 0.3148 | dt: 31841.30ms | tok/sec: 32931.3 | mem: 122575MB | Time Left: 02 hrs : 50 mins
|
| 125 |
+
[TIMING] data: 3.3ms | fwd: 7066.1ms | loss: 14.1ms | bwd: 24710.2ms | clip: 0.0ms | optim: 41.7ms
|
| 126 |
+
step 39 | loss: 2.508372 | lr 1.2000e-04 | norm: 0.3378 | dt: 31701.62ms | tok/sec: 33076.4 | mem: 122575MB | Time Left: 02 hrs : 49 mins
|
| 127 |
+
[TIMING] data: 3.3ms | fwd: 7034.9ms | loss: 13.5ms | bwd: 24602.5ms | clip: 0.0ms | optim: 41.8ms
|
| 128 |
+
step 40 | loss: 1.906072 | lr 1.2300e-04 | norm: 0.2950 | dt: 31723.44ms | tok/sec: 33053.7 | mem: 122575MB | Time Left: 02 hrs : 49 mins
|
| 129 |
+
[TIMING] data: 3.3ms | fwd: 7010.7ms | loss: 13.5ms | bwd: 24648.4ms | clip: 0.0ms | optim: 41.8ms
|
| 130 |
+
step 41 | loss: 2.596240 | lr 1.2600e-04 | norm: 0.2677 | dt: 31672.93ms | tok/sec: 33106.4 | mem: 122575MB | Time Left: 02 hrs : 48 mins
|
| 131 |
+
[TIMING] data: 3.4ms | fwd: 6999.3ms | loss: 13.6ms | bwd: 24607.8ms | clip: 0.0ms | optim: 43.3ms
|
| 132 |
+
step 42 | loss: 3.201991 | lr 1.2900e-04 | norm: 0.5888 | dt: 31665.92ms | tok/sec: 33113.7 | mem: 122575MB | Time Left: 02 hrs : 47 mins
|
| 133 |
+
[TIMING] data: 3.4ms | fwd: 7017.2ms | loss: 14.0ms | bwd: 24584.0ms | clip: 0.0ms | optim: 41.7ms
|
| 134 |
+
step 43 | loss: 2.612741 | lr 1.3200e-04 | norm: 0.3251 | dt: 31679.03ms | tok/sec: 33100.0 | mem: 122575MB | Time Left: 02 hrs : 47 mins
|
| 135 |
+
[TIMING] data: 3.4ms | fwd: 6993.8ms | loss: 13.5ms | bwd: 24620.3ms | clip: 0.0ms | optim: 42.5ms
|
| 136 |
+
step 44 | loss: 2.663048 | lr 1.3500e-04 | norm: 0.2964 | dt: 31726.61ms | tok/sec: 33050.4 | mem: 122575MB | Time Left: 02 hrs : 47 mins
|
| 137 |
+
[TIMING] data: 3.4ms | fwd: 7069.2ms | loss: 13.5ms | bwd: 24593.0ms | clip: 0.0ms | optim: 41.7ms
|
| 138 |
+
step 45 | loss: 1.459633 | lr 1.3800e-04 | norm: 0.5038 | dt: 31741.23ms | tok/sec: 33035.1 | mem: 122575MB | Time Left: 02 hrs : 46 mins
|
| 139 |
+
[TIMING] data: 3.1ms | fwd: 7000.5ms | loss: 13.5ms | bwd: 24676.9ms | clip: 0.0ms | optim: 41.5ms
|
| 140 |
+
step 46 | loss: 2.431983 | lr 1.4100e-04 | norm: 0.1606 | dt: 31736.79ms | tok/sec: 33039.8 | mem: 122575MB | Time Left: 02 hrs : 46 mins
|
| 141 |
+
[TIMING] data: 3.3ms | fwd: 7078.3ms | loss: 13.7ms | bwd: 24592.9ms | clip: 0.0ms | optim: 43.1ms
|
| 142 |
+
step 47 | loss: 2.427200 | lr 1.4400e-04 | norm: 0.5248 | dt: 31752.99ms | tok/sec: 33022.9 | mem: 122575MB | Time Left: 02 hrs : 45 mins
|
| 143 |
+
[TIMING] data: 3.3ms | fwd: 7065.1ms | loss: 13.5ms | bwd: 24623.1ms | clip: 0.0ms | optim: 42.4ms
|
| 144 |
+
step 48 | loss: 2.495286 | lr 1.4700e-04 | norm: 0.3415 | dt: 31753.79ms | tok/sec: 33022.1 | mem: 122575MB | Time Left: 02 hrs : 45 mins
|
| 145 |
+
[TIMING] data: 3.3ms | fwd: 7030.3ms | loss: 13.4ms | bwd: 24659.6ms | clip: 0.0ms | optim: 41.5ms
|
| 146 |
+
step 49 | loss: 2.395905 | lr 1.5000e-04 | norm: 0.2721 | dt: 31843.27ms | tok/sec: 32929.3 | mem: 122575MB | Time Left: 02 hrs : 45 mins
|
| 147 |
+
[TIMING] data: 3.3ms | fwd: 7040.2ms | loss: 13.4ms | bwd: 24739.1ms | clip: 0.0ms | optim: 41.7ms
|
| 148 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_50.ckpt (safely)
|
| 149 |
+
step 50 | loss: 1.646688 | lr 1.5300e-04 | norm: 0.3548 | dt: 30832.33ms | tok/sec: 34009.0 | mem: 122575MB | Time Left: 02 hrs : 39 mins
|
| 150 |
+
[TIMING] data: 3.7ms | fwd: 6779.3ms | loss: 13.2ms | bwd: 23989.1ms | clip: 0.0ms | optim: 41.4ms
|
| 151 |
+
step 51 | loss: 2.586594 | lr 1.5600e-04 | norm: 0.2063 | dt: 31476.24ms | tok/sec: 33313.3 | mem: 122575MB | Time Left: 02 hrs : 42 mins
|
| 152 |
+
[TIMING] data: 3.7ms | fwd: 6922.8ms | loss: 13.4ms | bwd: 24488.7ms | clip: 0.0ms | optim: 42.0ms
|
| 153 |
+
step 52 | loss: 2.199433 | lr 1.5900e-04 | norm: 0.1980 | dt: 31491.20ms | tok/sec: 33297.4 | mem: 122575MB | Time Left: 02 hrs : 41 mins
|
| 154 |
+
[TIMING] data: 3.7ms | fwd: 7033.5ms | loss: 13.5ms | bwd: 24392.9ms | clip: 0.0ms | optim: 42.0ms
|
| 155 |
+
step 53 | loss: 2.830748 | lr 1.6200e-04 | norm: 0.5555 | dt: 31602.12ms | tok/sec: 33180.6 | mem: 122575MB | Time Left: 02 hrs : 41 mins
|
| 156 |
+
[TIMING] data: 3.6ms | fwd: 6985.7ms | loss: 13.5ms | bwd: 24551.5ms | clip: 0.0ms | optim: 42.3ms
|
| 157 |
+
step 54 | loss: 2.695097 | lr 1.6500e-04 | norm: 0.2734 | dt: 31741.05ms | tok/sec: 33035.3 | mem: 122575MB | Time Left: 02 hrs : 41 mins
|
| 158 |
+
[TIMING] data: 3.6ms | fwd: 7065.6ms | loss: 13.6ms | bwd: 24610.8ms | clip: 0.0ms | optim: 41.8ms
|
| 159 |
+
step 55 | loss: 2.433235 | lr 1.6800e-04 | norm: 0.2464 | dt: 31764.03ms | tok/sec: 33011.4 | mem: 122575MB | Time Left: 02 hrs : 41 mins
|
| 160 |
+
[TIMING] data: 3.6ms | fwd: 7054.1ms | loss: 13.5ms | bwd: 24645.1ms | clip: 0.0ms | optim: 42.1ms
|
| 161 |
+
step 56 | loss: 1.935769 | lr 1.7100e-04 | norm: 0.2797 | dt: 31668.11ms | tok/sec: 33111.4 | mem: 122575MB | Time Left: 02 hrs : 40 mins
|
| 162 |
+
[TIMING] data: 3.6ms | fwd: 7045.9ms | loss: 13.7ms | bwd: 24557.3ms | clip: 0.0ms | optim: 41.7ms
|
| 163 |
+
step 57 | loss: 2.612627 | lr 1.7400e-04 | norm: 0.6361 | dt: 31741.97ms | tok/sec: 33034.4 | mem: 122575MB | Time Left: 02 hrs : 40 mins
|
| 164 |
+
[TIMING] data: 3.6ms | fwd: 7020.3ms | loss: 13.5ms | bwd: 24656.4ms | clip: 0.0ms | optim: 42.5ms
|
| 165 |
+
step 58 | loss: 2.716172 | lr 1.7700e-04 | norm: 3.9506 | dt: 31635.17ms | tok/sec: 33145.9 | mem: 122575MB | Time Left: 02 hrs : 39 mins
|
| 166 |
+
[TIMING] data: 3.6ms | fwd: 6979.8ms | loss: 13.5ms | bwd: 24590.9ms | clip: 0.0ms | optim: 41.8ms
|
| 167 |
+
step 59 | loss: 2.329602 | lr 1.8000e-04 | norm: 1.1933 | dt: 31673.41ms | tok/sec: 33105.9 | mem: 122575MB | Time Left: 02 hrs : 38 mins
|
| 168 |
+
[TIMING] data: 3.7ms | fwd: 7001.4ms | loss: 14.0ms | bwd: 24607.1ms | clip: 0.0ms | optim: 41.7ms
|
| 169 |
+
step 60 | loss: 2.695923 | lr 1.8000e-04 | norm: 0.4326 | dt: 31798.86ms | tok/sec: 32975.3 | mem: 122575MB | Time Left: 02 hrs : 38 mins
|
| 170 |
+
[TIMING] data: 3.6ms | fwd: 7052.4ms | loss: 13.5ms | bwd: 24682.0ms | clip: 0.0ms | optim: 41.7ms
|
| 171 |
+
step 61 | loss: 1.419670 | lr 1.8000e-04 | norm: 0.4520 | dt: 31696.42ms | tok/sec: 33081.8 | mem: 122575MB | Time Left: 02 hrs : 37 mins
|
| 172 |
+
[TIMING] data: 3.6ms | fwd: 7058.5ms | loss: 13.6ms | bwd: 24573.4ms | clip: 0.0ms | optim: 41.8ms
|
| 173 |
+
step 62 | loss: 2.255869 | lr 1.8000e-04 | norm: 0.1580 | dt: 31757.97ms | tok/sec: 33017.7 | mem: 122575MB | Time Left: 02 hrs : 37 mins
|
| 174 |
+
[TIMING] data: 3.7ms | fwd: 7062.1ms | loss: 13.4ms | bwd: 24631.4ms | clip: 0.0ms | optim: 41.8ms
|
| 175 |
+
step 63 | loss: 2.317328 | lr 1.8000e-04 | norm: 1.2686 | dt: 31707.40ms | tok/sec: 33070.4 | mem: 122575MB | Time Left: 02 hrs : 36 mins
|
| 176 |
+
[TIMING] data: 3.7ms | fwd: 7012.5ms | loss: 13.7ms | bwd: 24630.4ms | clip: 0.0ms | optim: 41.6ms
|
| 177 |
+
step 64 | loss: 2.529977 | lr 1.8000e-04 | norm: 0.3882 | dt: 31690.92ms | tok/sec: 33087.6 | mem: 122575MB | Time Left: 02 hrs : 36 mins
|
| 178 |
+
[TIMING] data: 3.6ms | fwd: 7009.9ms | loss: 13.5ms | bwd: 24616.4ms | clip: 0.0ms | optim: 41.9ms
|
| 179 |
+
step 65 | loss: 2.186996 | lr 1.8000e-04 | norm: 1.2818 | dt: 31683.53ms | tok/sec: 33095.3 | mem: 122575MB | Time Left: 02 hrs : 35 mins
|
| 180 |
+
[TIMING] data: 3.6ms | fwd: 7034.1ms | loss: 13.5ms | bwd: 24584.7ms | clip: 0.0ms | optim: 41.7ms
|
| 181 |
+
step 66 | loss: 2.780885 | lr 1.8000e-04 | norm: 0.9703 | dt: 31615.00ms | tok/sec: 33167.0 | mem: 122575MB | Time Left: 02 hrs : 34 mins
|
| 182 |
+
[TIMING] data: 3.7ms | fwd: 6997.6ms | loss: 13.5ms | bwd: 24552.3ms | clip: 0.0ms | optim: 42.3ms
|
| 183 |
+
step 67 | loss: 2.820178 | lr 1.8000e-04 | norm: 1.1360 | dt: 31645.35ms | tok/sec: 33135.2 | mem: 122575MB | Time Left: 02 hrs : 34 mins
|
| 184 |
+
[TIMING] data: 3.7ms | fwd: 7008.3ms | loss: 13.3ms | bwd: 24572.9ms | clip: 0.0ms | optim: 41.6ms
|
| 185 |
+
step 68 | loss: 2.092461 | lr 1.8000e-04 | norm: 0.4674 | dt: 31839.47ms | tok/sec: 32933.2 | mem: 122575MB | Time Left: 02 hrs : 34 mins
|
| 186 |
+
[TIMING] data: 3.7ms | fwd: 7030.1ms | loss: 13.5ms | bwd: 24743.6ms | clip: 0.0ms | optim: 43.0ms
|
| 187 |
+
step 69 | loss: 2.718188 | lr 1.8000e-04 | norm: 0.4593 | dt: 31777.50ms | tok/sec: 32997.4 | mem: 122575MB | Time Left: 02 hrs : 34 mins
|
| 188 |
+
[TIMING] data: 3.7ms | fwd: 7026.2ms | loss: 13.6ms | bwd: 24687.5ms | clip: 0.0ms | optim: 40.9ms
|
| 189 |
+
step 70 | loss: 2.425739 | lr 1.8000e-04 | norm: 0.2863 | dt: 31646.99ms | tok/sec: 33133.5 | mem: 122575MB | Time Left: 02 hrs : 32 mins
|
| 190 |
+
[TIMING] data: 3.7ms | fwd: 7023.8ms | loss: 13.5ms | bwd: 24557.3ms | clip: 0.0ms | optim: 43.1ms
|
| 191 |
+
step 71 | loss: 1.641091 | lr 1.8000e-04 | norm: 0.5965 | dt: 31650.36ms | tok/sec: 33130.0 | mem: 122575MB | Time Left: 02 hrs : 32 mins
|
| 192 |
+
[TIMING] data: 3.7ms | fwd: 7013.4ms | loss: 13.5ms | bwd: 24572.4ms | clip: 0.0ms | optim: 41.8ms
|
| 193 |
+
step 72 | loss: 2.440122 | lr 1.8000e-04 | norm: 0.2341 | dt: 31692.40ms | tok/sec: 33086.0 | mem: 122575MB | Time Left: 02 hrs : 32 mins
|
| 194 |
+
[TIMING] data: 3.7ms | fwd: 7048.0ms | loss: 13.6ms | bwd: 24579.1ms | clip: 0.0ms | optim: 42.4ms
|
| 195 |
+
step 73 | loss: 2.734461 | lr 1.8000e-04 | norm: 0.6118 | dt: 31814.21ms | tok/sec: 32959.4 | mem: 122575MB | Time Left: 02 hrs : 32 mins
|
| 196 |
+
[TIMING] data: 3.7ms | fwd: 7068.9ms | loss: 13.4ms | bwd: 24681.0ms | clip: 0.0ms | optim: 41.5ms
|
| 197 |
+
step 74 | loss: 2.375622 | lr 1.8000e-04 | norm: 0.6563 | dt: 31545.06ms | tok/sec: 33240.6 | mem: 122575MB | Time Left: 02 hrs : 30 mins
|
| 198 |
+
[TIMING] data: 3.7ms | fwd: 7024.0ms | loss: 13.9ms | bwd: 24455.9ms | clip: 0.0ms | optim: 41.9ms
|
| 199 |
+
step 75 | loss: 2.241879 | lr 1.8000e-04 | norm: 0.2642 | dt: 31734.09ms | tok/sec: 33042.6 | mem: 122575MB | Time Left: 02 hrs : 30 mins
|
| 200 |
+
[TIMING] data: 3.7ms | fwd: 7048.3ms | loss: 13.6ms | bwd: 24621.5ms | clip: 0.0ms | optim: 41.4ms
|
| 201 |
+
step 76 | loss: 1.596710 | lr 1.8000e-04 | norm: 0.4171 | dt: 31692.37ms | tok/sec: 33086.1 | mem: 122575MB | Time Left: 02 hrs : 30 mins
|
| 202 |
+
[TIMING] data: 3.7ms | fwd: 6990.6ms | loss: 13.5ms | bwd: 24637.1ms | clip: 0.0ms | optim: 41.8ms
|
| 203 |
+
step 77 | loss: 2.520807 | lr 1.8000e-04 | norm: 0.5592 | dt: 31754.18ms | tok/sec: 33021.7 | mem: 122575MB | Time Left: 02 hrs : 29 mins
|
| 204 |
+
[TIMING] data: 3.6ms | fwd: 7039.5ms | loss: 13.6ms | bwd: 24650.2ms | clip: 0.0ms | optim: 41.6ms
|
| 205 |
+
step 78 | loss: 2.920510 | lr 1.8000e-04 | norm: 0.3092 | dt: 31795.43ms | tok/sec: 32978.8 | mem: 122575MB | Time Left: 02 hrs : 29 mins
|
| 206 |
+
[TIMING] data: 3.6ms | fwd: 7032.1ms | loss: 13.5ms | bwd: 24698.9ms | clip: 0.0ms | optim: 41.6ms
|
| 207 |
+
step 79 | loss: 2.672686 | lr 1.8000e-04 | norm: 0.4197 | dt: 31651.00ms | tok/sec: 33129.3 | mem: 122575MB | Time Left: 02 hrs : 28 mins
|
| 208 |
+
[TIMING] data: 3.7ms | fwd: 6991.5ms | loss: 14.1ms | bwd: 24594.4ms | clip: 0.0ms | optim: 41.8ms
|
| 209 |
+
step 80 | loss: 2.775012 | lr 1.8000e-04 | norm: 0.4095 | dt: 31881.54ms | tok/sec: 32889.8 | mem: 122575MB | Time Left: 02 hrs : 28 mins
|
| 210 |
+
[TIMING] data: 3.6ms | fwd: 7059.8ms | loss: 14.1ms | bwd: 24756.5ms | clip: 0.0ms | optim: 41.9ms
|
| 211 |
+
step 81 | loss: 2.241447 | lr 1.8000e-04 | norm: 0.3987 | dt: 31625.65ms | tok/sec: 33155.9 | mem: 122575MB | Time Left: 02 hrs : 27 mins
|
| 212 |
+
[TIMING] data: 3.7ms | fwd: 7026.5ms | loss: 14.0ms | bwd: 24534.5ms | clip: 0.0ms | optim: 41.4ms
|
| 213 |
+
step 82 | loss: 2.726295 | lr 1.8000e-04 | norm: 0.6935 | dt: 31764.73ms | tok/sec: 33010.7 | mem: 122575MB | Time Left: 02 hrs : 27 mins
|
| 214 |
+
[TIMING] data: 3.7ms | fwd: 6993.9ms | loss: 13.4ms | bwd: 24706.2ms | clip: 0.0ms | optim: 41.9ms
|
| 215 |
+
step 83 | loss: 2.611327 | lr 1.8000e-04 | norm: 0.3334 | dt: 31747.30ms | tok/sec: 33028.8 | mem: 122575MB | Time Left: 02 hrs : 26 mins
|
| 216 |
+
[TIMING] data: 3.7ms | fwd: 7072.5ms | loss: 13.5ms | bwd: 24610.1ms | clip: 0.0ms | optim: 41.9ms
|
| 217 |
+
step 84 | loss: 1.702045 | lr 1.8000e-04 | norm: 0.3783 | dt: 31788.35ms | tok/sec: 32986.2 | mem: 122575MB | Time Left: 02 hrs : 26 mins
|
| 218 |
+
[TIMING] data: 3.7ms | fwd: 6970.9ms | loss: 13.4ms | bwd: 24752.5ms | clip: 0.0ms | optim: 42.2ms
|
| 219 |
+
step 85 | loss: 1.888726 | lr 1.8000e-04 | norm: 0.2121 | dt: 31766.95ms | tok/sec: 33008.4 | mem: 122575MB | Time Left: 02 hrs : 25 mins
|
| 220 |
+
[TIMING] data: 3.6ms | fwd: 7093.6ms | loss: 13.6ms | bwd: 24608.3ms | clip: 0.0ms | optim: 42.1ms
|
| 221 |
+
step 86 | loss: 2.258515 | lr 1.8000e-04 | norm: 1.0799 | dt: 31748.21ms | tok/sec: 33027.9 | mem: 122575MB | Time Left: 02 hrs : 24 mins
|
| 222 |
+
[TIMING] data: 3.6ms | fwd: 7032.5ms | loss: 13.6ms | bwd: 24651.0ms | clip: 0.0ms | optim: 41.8ms
|
| 223 |
+
step 87 | loss: 2.741969 | lr 1.8000e-04 | norm: 0.4568 | dt: 31770.17ms | tok/sec: 33005.1 | mem: 122575MB | Time Left: 02 hrs : 24 mins
|
| 224 |
+
[TIMING] data: 3.7ms | fwd: 7048.4ms | loss: 13.6ms | bwd: 24657.2ms | clip: 0.0ms | optim: 41.8ms
|
| 225 |
+
step 88 | loss: 2.687754 | lr 1.8000e-04 | norm: 0.2745 | dt: 31798.63ms | tok/sec: 32975.5 | mem: 122575MB | Time Left: 02 hrs : 24 mins
|
| 226 |
+
[TIMING] data: 3.6ms | fwd: 6990.5ms | loss: 13.5ms | bwd: 24743.6ms | clip: 0.0ms | optim: 41.9ms
|
| 227 |
+
step 89 | loss: 2.617070 | lr 1.8000e-04 | norm: 0.3178 | dt: 31750.68ms | tok/sec: 33025.3 | mem: 122575MB | Time Left: 02 hrs : 23 mins
|
| 228 |
+
[TIMING] data: 3.6ms | fwd: 7057.5ms | loss: 13.5ms | bwd: 24627.5ms | clip: 0.0ms | optim: 42.9ms
|
| 229 |
+
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/data/mix128_b6_60/train_shard_00001.bin
|
| 230 |
+
100000000
|
| 231 |
+
step 90 | loss: 2.865276 | lr 1.8000e-04 | norm: 0.8547 | dt: 31733.15ms | tok/sec: 33043.6 | mem: 122575MB | Time Left: 02 hrs : 22 mins
|
| 232 |
+
[TIMING] data: 8.5ms | fwd: 7016.4ms | loss: 13.5ms | bwd: 24646.9ms | clip: 0.0ms | optim: 42.2ms
|
| 233 |
+
step 91 | loss: 2.597929 | lr 1.8000e-04 | norm: 0.2257 | dt: 31872.22ms | tok/sec: 32899.4 | mem: 122575MB | Time Left: 02 hrs : 22 mins
|
| 234 |
+
[TIMING] data: 3.6ms | fwd: 7024.0ms | loss: 13.5ms | bwd: 24784.1ms | clip: 0.0ms | optim: 41.5ms
|
| 235 |
+
step 92 | loss: 2.329015 | lr 1.8000e-04 | norm: 0.2984 | dt: 31901.29ms | tok/sec: 32869.4 | mem: 122575MB | Time Left: 02 hrs : 22 mins
|
| 236 |
+
[TIMING] data: 3.7ms | fwd: 6952.8ms | loss: 13.5ms | bwd: 24883.7ms | clip: 0.0ms | optim: 42.0ms
|
| 237 |
+
step 93 | loss: 2.559205 | lr 1.8000e-04 | norm: 0.4769 | dt: 31784.79ms | tok/sec: 32989.9 | mem: 122575MB | Time Left: 02 hrs : 21 mins
|
| 238 |
+
[TIMING] data: 3.6ms | fwd: 7030.4ms | loss: 13.6ms | bwd: 24689.8ms | clip: 0.0ms | optim: 41.7ms
|
| 239 |
+
step 94 | loss: 2.663100 | lr 1.8000e-04 | norm: 0.2375 | dt: 31889.00ms | tok/sec: 32882.1 | mem: 122575MB | Time Left: 02 hrs : 21 mins
|
| 240 |
+
[TIMING] data: 3.6ms | fwd: 7037.6ms | loss: 13.5ms | bwd: 24786.9ms | clip: 0.0ms | optim: 41.7ms
|
| 241 |
+
step 95 | loss: 1.416687 | lr 1.8000e-04 | norm: 0.5291 | dt: 31834.94ms | tok/sec: 32937.9 | mem: 122575MB | Time Left: 02 hrs : 20 mins
|
| 242 |
+
[TIMING] data: 3.7ms | fwd: 7048.6ms | loss: 13.6ms | bwd: 24720.8ms | clip: 0.0ms | optim: 42.8ms
|
| 243 |
+
step 96 | loss: 2.305063 | lr 1.8000e-04 | norm: 0.1617 | dt: 31841.13ms | tok/sec: 32931.5 | mem: 122575MB | Time Left: 02 hrs : 20 mins
|
| 244 |
+
[TIMING] data: 3.6ms | fwd: 7040.2ms | loss: 13.6ms | bwd: 24735.1ms | clip: 0.0ms | optim: 42.9ms
|
| 245 |
+
step 97 | loss: 2.758640 | lr 1.8000e-04 | norm: 1.8660 | dt: 31888.59ms | tok/sec: 32882.5 | mem: 122575MB | Time Left: 02 hrs : 19 mins
|
| 246 |
+
[TIMING] data: 3.7ms | fwd: 7082.8ms | loss: 13.6ms | bwd: 24741.2ms | clip: 0.0ms | optim: 41.7ms
|
| 247 |
+
step 98 | loss: 2.689206 | lr 1.8000e-04 | norm: 0.2551 | dt: 31876.46ms | tok/sec: 32895.0 | mem: 122575MB | Time Left: 02 hrs : 19 mins
|
| 248 |
+
[TIMING] data: 3.7ms | fwd: 7101.0ms | loss: 13.6ms | bwd: 24710.5ms | clip: 0.0ms | optim: 42.1ms
|
| 249 |
+
step 99 | loss: 2.518957 | lr 1.8000e-04 | norm: 0.3314 | dt: 31884.24ms | tok/sec: 32887.0 | mem: 122575MB | Time Left: 02 hrs : 18 mins
|
| 250 |
+
[TIMING] data: 3.7ms | fwd: 7058.0ms | loss: 13.6ms | bwd: 24761.7ms | clip: 0.0ms | optim: 41.6ms
|
| 251 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_100.ckpt (safely)
|
| 252 |
+
step 100 | loss: 1.091004 | lr 1.8000e-04 | norm: 1.4149 | dt: 31017.37ms | tok/sec: 33806.1 | mem: 122575MB | Time Left: 02 hrs : 14 mins
|
| 253 |
+
[TIMING] data: 3.4ms | fwd: 6774.4ms | loss: 13.3ms | bwd: 24178.0ms | clip: 0.0ms | optim: 42.6ms
|
| 254 |
+
step 101 | loss: 1.842507 | lr 1.8000e-04 | norm: 0.3655 | dt: 31595.31ms | tok/sec: 33187.7 | mem: 122575MB | Time Left: 02 hrs : 16 mins
|
| 255 |
+
[TIMING] data: 3.5ms | fwd: 6964.7ms | loss: 13.5ms | bwd: 24565.8ms | clip: 0.0ms | optim: 42.2ms
|
| 256 |
+
step 102 | loss: 2.461726 | lr 1.8000e-04 | norm: 0.2389 | dt: 31785.66ms | tok/sec: 32989.0 | mem: 122575MB | Time Left: 02 hrs : 16 mins
|
| 257 |
+
[TIMING] data: 3.5ms | fwd: 7051.1ms | loss: 14.2ms | bwd: 24669.5ms | clip: 0.0ms | optim: 41.8ms
|
| 258 |
+
step 103 | loss: 2.806600 | lr 1.8000e-04 | norm: 0.2020 | dt: 31726.26ms | tok/sec: 33050.7 | mem: 122575MB | Time Left: 02 hrs : 15 mins
|
| 259 |
+
[TIMING] data: 3.4ms | fwd: 7009.1ms | loss: 13.9ms | bwd: 24652.4ms | clip: 0.0ms | optim: 41.7ms
|
| 260 |
+
step 104 | loss: 1.667076 | lr 1.8000e-04 | norm: 0.4780 | dt: 32004.53ms | tok/sec: 32763.4 | mem: 122575MB | Time Left: 02 hrs : 16 mins
|
| 261 |
+
[TIMING] data: 3.4ms | fwd: 7148.1ms | loss: 13.6ms | bwd: 24791.5ms | clip: 0.0ms | optim: 42.3ms
|
| 262 |
+
step 105 | loss: 2.766091 | lr 1.8000e-04 | norm: 0.4520 | dt: 31847.86ms | tok/sec: 32924.5 | mem: 122575MB | Time Left: 02 hrs : 15 mins
|
| 263 |
+
[TIMING] data: 3.4ms | fwd: 6944.9ms | loss: 13.5ms | bwd: 24838.8ms | clip: 0.0ms | optim: 41.6ms
|
| 264 |
+
step 106 | loss: 2.669836 | lr 1.8000e-04 | norm: 0.3244 | dt: 31813.24ms | tok/sec: 32960.4 | mem: 122575MB | Time Left: 02 hrs : 14 mins
|
| 265 |
+
[TIMING] data: 3.4ms | fwd: 7021.3ms | loss: 13.5ms | bwd: 24727.3ms | clip: 0.0ms | optim: 41.9ms
|
| 266 |
+
step 107 | loss: 2.407339 | lr 1.8000e-04 | norm: 0.3236 | dt: 31839.58ms | tok/sec: 32933.1 | mem: 122575MB | Time Left: 02 hrs : 14 mins
|
| 267 |
+
[TIMING] data: 3.4ms | fwd: 7052.1ms | loss: 13.5ms | bwd: 24723.1ms | clip: 0.0ms | optim: 41.9ms
|
| 268 |
+
step 108 | loss: 1.616182 | lr 1.8000e-04 | norm: 0.3261 | dt: 31906.81ms | tok/sec: 32863.7 | mem: 122575MB | Time Left: 02 hrs : 14 mins
|
| 269 |
+
[TIMING] data: 3.4ms | fwd: 7079.9ms | loss: 13.6ms | bwd: 24762.0ms | clip: 0.0ms | optim: 42.3ms
|
| 270 |
+
step 109 | loss: 2.605919 | lr 1.8000e-04 | norm: 0.3792 | dt: 31807.43ms | tok/sec: 32966.4 | mem: 122575MB | Time Left: 02 hrs : 13 mins
|
| 271 |
+
[TIMING] data: 3.6ms | fwd: 7007.3ms | loss: 13.9ms | bwd: 24735.0ms | clip: 0.0ms | optim: 41.9ms
|
| 272 |
+
step 110 | loss: 2.481047 | lr 1.8000e-04 | norm: 0.3575 | dt: 31734.25ms | tok/sec: 33042.4 | mem: 122575MB | Time Left: 02 hrs : 12 mins
|
| 273 |
+
[TIMING] data: 3.4ms | fwd: 7002.5ms | loss: 13.6ms | bwd: 24667.5ms | clip: 0.0ms | optim: 41.6ms
|
| 274 |
+
step 111 | loss: 2.666687 | lr 1.8000e-04 | norm: 0.4735 | dt: 31838.93ms | tok/sec: 32933.8 | mem: 122575MB | Time Left: 02 hrs : 12 mins
|
| 275 |
+
[TIMING] data: 3.4ms | fwd: 7030.7ms | loss: 13.6ms | bwd: 24743.0ms | clip: 0.0ms | optim: 42.6ms
|
| 276 |
+
step 112 | loss: 1.937047 | lr 1.8000e-04 | norm: 0.2862 | dt: 31752.31ms | tok/sec: 33023.6 | mem: 122575MB | Time Left: 02 hrs : 11 mins
|
| 277 |
+
[TIMING] data: 3.3ms | fwd: 7027.6ms | loss: 13.6ms | bwd: 24658.8ms | clip: 0.0ms | optim: 43.3ms
|
| 278 |
+
step 113 | loss: 2.023892 | lr 1.8000e-04 | norm: 0.1838 | dt: 31679.78ms | tok/sec: 33099.2 | mem: 122575MB | Time Left: 02 hrs : 10 mins
|
| 279 |
+
[TIMING] data: 3.4ms | fwd: 7035.1ms | loss: 14.2ms | bwd: 24579.2ms | clip: 0.0ms | optim: 42.2ms
|
| 280 |
+
step 114 | loss: 2.484109 | lr 1.8000e-04 | norm: 0.8638 | dt: 31692.19ms | tok/sec: 33086.3 | mem: 122575MB | Time Left: 02 hrs : 09 mins
|
| 281 |
+
[TIMING] data: 3.4ms | fwd: 6987.9ms | loss: 13.5ms | bwd: 24640.0ms | clip: 0.0ms | optim: 41.6ms
|
| 282 |
+
step 115 | loss: 2.639418 | lr 1.8000e-04 | norm: 0.6915 | dt: 31604.17ms | tok/sec: 33178.4 | mem: 122575MB | Time Left: 02 hrs : 09 mins
|
| 283 |
+
[TIMING] data: 3.4ms | fwd: 6988.0ms | loss: 13.4ms | bwd: 24552.1ms | clip: 0.0ms | optim: 41.6ms
|
| 284 |
+
step 116 | loss: 2.499335 | lr 1.8000e-04 | norm: 0.7059 | dt: 31597.67ms | tok/sec: 33185.2 | mem: 122575MB | Time Left: 02 hrs : 08 mins
|
| 285 |
+
[TIMING] data: 3.4ms | fwd: 6942.5ms | loss: 13.4ms | bwd: 24591.2ms | clip: 0.0ms | optim: 41.6ms
|
| 286 |
+
step 117 | loss: 2.105376 | lr 1.8000e-04 | norm: 0.4467 | dt: 31494.27ms | tok/sec: 33294.2 | mem: 122575MB | Time Left: 02 hrs : 07 mins
|
| 287 |
+
[TIMING] data: 3.4ms | fwd: 7005.2ms | loss: 13.5ms | bwd: 24425.1ms | clip: 0.0ms | optim: 41.3ms
|
| 288 |
+
step 118 | loss: 1.907477 | lr 1.8000e-04 | norm: 0.3171 | dt: 31529.05ms | tok/sec: 33257.5 | mem: 122575MB | Time Left: 02 hrs : 07 mins
|
| 289 |
+
[TIMING] data: 3.4ms | fwd: 7006.8ms | loss: 13.5ms | bwd: 24457.7ms | clip: 0.0ms | optim: 42.0ms
|
| 290 |
+
step 119 | loss: 2.981980 | lr 1.8000e-04 | norm: 0.4639 | dt: 31521.33ms | tok/sec: 33265.6 | mem: 122575MB | Time Left: 02 hrs : 06 mins
|
| 291 |
+
[TIMING] data: 3.5ms | fwd: 6943.0ms | loss: 13.5ms | bwd: 24514.4ms | clip: 0.0ms | optim: 41.2ms
|
| 292 |
+
step 120 | loss: 2.625632 | lr 1.8000e-04 | norm: 1.0438 | dt: 31539.67ms | tok/sec: 33246.3 | mem: 122575MB | Time Left: 02 hrs : 06 mins
|
| 293 |
+
[TIMING] data: 3.4ms | fwd: 6939.9ms | loss: 13.4ms | bwd: 24535.4ms | clip: 0.0ms | optim: 42.0ms
|
| 294 |
+
step 121 | loss: 2.985030 | lr 1.8000e-04 | norm: 0.3360 | dt: 31443.01ms | tok/sec: 33348.5 | mem: 122575MB | Time Left: 02 hrs : 05 mins
|
| 295 |
+
[TIMING] data: 3.4ms | fwd: 6977.1ms | loss: 14.0ms | bwd: 24401.2ms | clip: 0.0ms | optim: 41.7ms
|
| 296 |
+
step 122 | loss: 3.334712 | lr 1.8000e-04 | norm: 0.4583 | dt: 31617.84ms | tok/sec: 33164.1 | mem: 122575MB | Time Left: 02 hrs : 05 mins
|
| 297 |
+
[TIMING] data: 3.4ms | fwd: 6962.5ms | loss: 13.4ms | bwd: 24591.0ms | clip: 0.0ms | optim: 41.9ms
|
| 298 |
+
step 123 | loss: 1.712982 | lr 1.8000e-04 | norm: 0.4855 | dt: 31534.43ms | tok/sec: 33251.8 | mem: 122575MB | Time Left: 02 hrs : 04 mins
|
| 299 |
+
[TIMING] data: 3.4ms | fwd: 6993.5ms | loss: 13.4ms | bwd: 24476.8ms | clip: 0.0ms | optim: 41.7ms
|
| 300 |
+
step 124 | loss: 2.729832 | lr 1.8000e-04 | norm: 0.3974 | dt: 31620.96ms | tok/sec: 33160.8 | mem: 122575MB | Time Left: 02 hrs : 04 mins
|
| 301 |
+
[TIMING] data: 3.4ms | fwd: 7033.6ms | loss: 13.7ms | bwd: 24522.6ms | clip: 0.0ms | optim: 42.0ms
|
| 302 |
+
step 125 | loss: 2.796641 | lr 1.8000e-04 | norm: 0.4434 | dt: 31586.60ms | tok/sec: 33196.9 | mem: 122575MB | Time Left: 02 hrs : 03 mins
|
| 303 |
+
[TIMING] data: 3.4ms | fwd: 7010.0ms | loss: 13.5ms | bwd: 24512.5ms | clip: 0.0ms | optim: 41.6ms
|
| 304 |
+
step 126 | loss: 1.917688 | lr 1.8000e-04 | norm: 0.3237 | dt: 31742.20ms | tok/sec: 33034.1 | mem: 122575MB | Time Left: 02 hrs : 03 mins
|
| 305 |
+
[TIMING] data: 3.4ms | fwd: 7012.6ms | loss: 14.2ms | bwd: 24664.4ms | clip: 0.0ms | optim: 41.9ms
|
| 306 |
+
step 127 | loss: 2.014207 | lr 1.8000e-04 | norm: 0.1641 | dt: 31733.04ms | tok/sec: 33043.7 | mem: 122575MB | Time Left: 02 hrs : 03 mins
|
| 307 |
+
[TIMING] data: 3.4ms | fwd: 6978.0ms | loss: 13.6ms | bwd: 24690.0ms | clip: 0.0ms | optim: 42.4ms
|
| 308 |
+
step 128 | loss: 2.897421 | lr 1.8000e-04 | norm: 1.5243 | dt: 31756.20ms | tok/sec: 33019.6 | mem: 122575MB | Time Left: 02 hrs : 02 mins
|
| 309 |
+
[TIMING] data: 3.4ms | fwd: 7083.0ms | loss: 13.5ms | bwd: 24608.9ms | clip: 0.0ms | optim: 41.6ms
|
| 310 |
+
step 129 | loss: 2.749770 | lr 1.8000e-04 | norm: 0.9557 | dt: 31790.21ms | tok/sec: 32984.2 | mem: 122575MB | Time Left: 02 hrs : 02 mins
|
| 311 |
+
[TIMING] data: 3.4ms | fwd: 7022.8ms | loss: 13.6ms | bwd: 24702.9ms | clip: 0.0ms | optim: 41.8ms
|
| 312 |
+
step 130 | loss: 2.702589 | lr 1.8000e-04 | norm: 0.6153 | dt: 31703.85ms | tok/sec: 33074.1 | mem: 122575MB | Time Left: 02 hrs : 01 mins
|
| 313 |
+
[TIMING] data: 3.4ms | fwd: 7001.6ms | loss: 14.1ms | bwd: 24637.5ms | clip: 0.0ms | optim: 41.6ms
|
| 314 |
+
step 131 | loss: 2.646048 | lr 1.8000e-04 | norm: 0.4409 | dt: 31806.74ms | tok/sec: 32967.1 | mem: 122575MB | Time Left: 02 hrs : 01 mins
|
| 315 |
+
[TIMING] data: 3.4ms | fwd: 7061.1ms | loss: 13.6ms | bwd: 24680.5ms | clip: 0.0ms | optim: 42.5ms
|
| 316 |
+
step 132 | loss: 2.439284 | lr 1.8000e-04 | norm: 0.2202 | dt: 31795.43ms | tok/sec: 32978.8 | mem: 122575MB | Time Left: 02 hrs : 00 mins
|
| 317 |
+
[TIMING] data: 3.4ms | fwd: 7023.6ms | loss: 14.2ms | bwd: 24706.8ms | clip: 0.0ms | optim: 41.8ms
|
| 318 |
+
step 133 | loss: 2.595837 | lr 1.8000e-04 | norm: 0.1869 | dt: 31903.12ms | tok/sec: 32867.5 | mem: 122575MB | Time Left: 02 hrs : 00 mins
|
| 319 |
+
[TIMING] data: 3.5ms | fwd: 7125.3ms | loss: 13.7ms | bwd: 24713.5ms | clip: 0.0ms | optim: 41.6ms
|
| 320 |
+
step 134 | loss: 2.487109 | lr 1.8000e-04 | norm: 1.1479 | dt: 31808.37ms | tok/sec: 32965.4 | mem: 122575MB | Time Left: 01 hrs : 59 mins
|
| 321 |
+
[TIMING] data: 3.3ms | fwd: 7025.9ms | loss: 13.6ms | bwd: 24718.1ms | clip: 0.0ms | optim: 41.9ms
|
| 322 |
+
step 135 | loss: 3.130783 | lr 1.8000e-04 | norm: 0.8995 | dt: 31877.36ms | tok/sec: 32894.1 | mem: 122575MB | Time Left: 01 hrs : 59 mins
|
| 323 |
+
[TIMING] data: 3.4ms | fwd: 7019.7ms | loss: 13.6ms | bwd: 24793.3ms | clip: 0.0ms | optim: 41.9ms
|
| 324 |
+
step 136 | loss: 2.828518 | lr 1.8000e-04 | norm: 0.4041 | dt: 31883.41ms | tok/sec: 32887.8 | mem: 122575MB | Time Left: 01 hrs : 59 mins
|
| 325 |
+
[TIMING] data: 3.4ms | fwd: 7077.0ms | loss: 13.6ms | bwd: 24742.3ms | clip: 0.0ms | optim: 41.5ms
|
| 326 |
+
step 137 | loss: 1.651224 | lr 1.8000e-04 | norm: 0.6961 | dt: 31949.97ms | tok/sec: 32819.3 | mem: 122575MB | Time Left: 01 hrs : 58 mins
|
| 327 |
+
[TIMING] data: 3.4ms | fwd: 7037.5ms | loss: 13.4ms | bwd: 24848.0ms | clip: 0.0ms | optim: 42.0ms
|
| 328 |
+
step 138 | loss: 2.346748 | lr 1.8000e-04 | norm: 0.2459 | dt: 31893.66ms | tok/sec: 32877.3 | mem: 122575MB | Time Left: 01 hrs : 58 mins
|
| 329 |
+
[TIMING] data: 3.4ms | fwd: 7086.3ms | loss: 13.5ms | bwd: 24742.1ms | clip: 0.0ms | optim: 42.7ms
|
| 330 |
+
step 139 | loss: 2.756691 | lr 1.8000e-04 | norm: 0.3919 | dt: 31969.90ms | tok/sec: 32798.8 | mem: 122575MB | Time Left: 01 hrs : 57 mins
|
| 331 |
+
[TIMING] data: 3.5ms | fwd: 7079.3ms | loss: 13.5ms | bwd: 24826.0ms | clip: 0.0ms | optim: 41.8ms
|
| 332 |
+
step 140 | loss: 2.840976 | lr 1.8000e-04 | norm: 0.3421 | dt: 31957.97ms | tok/sec: 32811.1 | mem: 122575MB | Time Left: 01 hrs : 57 mins
|
| 333 |
+
[TIMING] data: 3.5ms | fwd: 7146.9ms | loss: 14.1ms | bwd: 24745.3ms | clip: 0.0ms | optim: 42.6ms
|
| 334 |
+
step 141 | loss: 3.153464 | lr 1.8000e-04 | norm: 0.6237 | dt: 31979.41ms | tok/sec: 32789.1 | mem: 122575MB | Time Left: 01 hrs : 56 mins
|
| 335 |
+
[TIMING] data: 3.5ms | fwd: 7106.0ms | loss: 13.5ms | bwd: 24808.8ms | clip: 0.0ms | optim: 41.9ms
|
| 336 |
+
step 142 | loss: 1.428495 | lr 1.8000e-04 | norm: 0.3474 | dt: 32016.83ms | tok/sec: 32750.8 | mem: 122575MB | Time Left: 01 hrs : 56 mins
|
| 337 |
+
[TIMING] data: 3.4ms | fwd: 7063.7ms | loss: 13.7ms | bwd: 24888.0ms | clip: 0.0ms | optim: 42.3ms
|
| 338 |
+
step 143 | loss: 1.982745 | lr 1.8000e-04 | norm: 0.2369 | dt: 31860.57ms | tok/sec: 32911.4 | mem: 122575MB | Time Left: 01 hrs : 55 mins
|
| 339 |
+
[TIMING] data: 3.4ms | fwd: 7108.5ms | loss: 13.7ms | bwd: 24687.5ms | clip: 0.0ms | optim: 41.9ms
|
| 340 |
+
step 144 | loss: 2.867845 | lr 1.8000e-04 | norm: 0.4057 | dt: 31983.17ms | tok/sec: 32785.2 | mem: 122575MB | Time Left: 01 hrs : 55 mins
|
| 341 |
+
[TIMING] data: 3.4ms | fwd: 7018.2ms | loss: 13.6ms | bwd: 24900.5ms | clip: 0.0ms | optim: 41.8ms
|
| 342 |
+
step 145 | loss: 2.682787 | lr 1.8000e-04 | norm: 0.2696 | dt: 31965.14ms | tok/sec: 32803.7 | mem: 122575MB | Time Left: 01 hrs : 54 mins
|
| 343 |
+
[TIMING] data: 3.4ms | fwd: 7086.0ms | loss: 13.6ms | bwd: 24814.7ms | clip: 0.0ms | optim: 41.9ms
|
| 344 |
+
step 146 | loss: 2.530450 | lr 1.8000e-04 | norm: 0.2114 | dt: 31960.21ms | tok/sec: 32808.8 | mem: 122575MB | Time Left: 01 hrs : 53 mins
|
| 345 |
+
[TIMING] data: 3.4ms | fwd: 7058.9ms | loss: 13.6ms | bwd: 24837.5ms | clip: 0.0ms | optim: 41.2ms
|
| 346 |
+
step 147 | loss: 3.494222 | lr 1.8000e-04 | norm: 2.1142 | dt: 31952.80ms | tok/sec: 32816.4 | mem: 122575MB | Time Left: 01 hrs : 53 mins
|
| 347 |
+
[TIMING] data: 3.4ms | fwd: 7074.9ms | loss: 13.5ms | bwd: 24813.4ms | clip: 0.0ms | optim: 41.8ms
|
| 348 |
+
step 148 | loss: 2.629178 | lr 1.8000e-04 | norm: 0.3882 | dt: 31955.46ms | tok/sec: 32813.7 | mem: 122575MB | Time Left: 01 hrs : 52 mins
|
| 349 |
+
[TIMING] data: 3.4ms | fwd: 7065.9ms | loss: 13.4ms | bwd: 24825.2ms | clip: 0.0ms | optim: 41.9ms
|
| 350 |
+
step 149 | loss: 1.615281 | lr 1.8000e-04 | norm: 0.3117 | dt: 31983.09ms | tok/sec: 32785.3 | mem: 122575MB | Time Left: 01 hrs : 52 mins
|
| 351 |
+
[TIMING] data: 3.4ms | fwd: 7053.2ms | loss: 13.6ms | bwd: 24865.3ms | clip: 0.0ms | optim: 41.9ms
|
| 352 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_150.ckpt (safely)
|
| 353 |
+
step 150 | loss: 1.375700 | lr 1.8000e-04 | norm: 0.2805 | dt: 31118.70ms | tok/sec: 33696.0 | mem: 122575MB | Time Left: 01 hrs : 48 mins
|
| 354 |
+
[TIMING] data: 4.0ms | fwd: 6838.2ms | loss: 13.4ms | bwd: 24215.6ms | clip: 0.0ms | optim: 41.7ms
|
| 355 |
+
step 151 | loss: 2.355109 | lr 1.8000e-04 | norm: 0.1894 | dt: 31817.95ms | tok/sec: 32955.5 | mem: 122575MB | Time Left: 01 hrs : 50 mins
|
| 356 |
+
[TIMING] data: 4.1ms | fwd: 7041.9ms | loss: 13.5ms | bwd: 24711.1ms | clip: 0.0ms | optim: 41.7ms
|
| 357 |
+
step 152 | loss: 2.446342 | lr 1.8000e-04 | norm: 0.2451 | dt: 31894.83ms | tok/sec: 32876.0 | mem: 122575MB | Time Left: 01 hrs : 50 mins
|
| 358 |
+
[TIMING] data: 4.0ms | fwd: 7045.8ms | loss: 13.6ms | bwd: 24783.9ms | clip: 0.0ms | optim: 41.9ms
|
| 359 |
+
step 153 | loss: 2.637209 | lr 1.8000e-04 | norm: 0.5310 | dt: 31926.07ms | tok/sec: 32843.9 | mem: 122575MB | Time Left: 01 hrs : 50 mins
|
| 360 |
+
[TIMING] data: 4.0ms | fwd: 7072.1ms | loss: 13.7ms | bwd: 24788.8ms | clip: 0.0ms | optim: 41.8ms
|
| 361 |
+
step 154 | loss: 2.502872 | lr 1.8000e-04 | norm: 0.2042 | dt: 32044.43ms | tok/sec: 32722.6 | mem: 122575MB | Time Left: 01 hrs : 50 mins
|
| 362 |
+
[TIMING] data: 4.0ms | fwd: 7000.2ms | loss: 13.6ms | bwd: 24979.2ms | clip: 0.0ms | optim: 41.9ms
|
| 363 |
+
step 155 | loss: 2.555309 | lr 1.8000e-04 | norm: 0.3537 | dt: 31955.54ms | tok/sec: 32813.6 | mem: 122575MB | Time Left: 01 hrs : 49 mins
|
| 364 |
+
[TIMING] data: 4.0ms | fwd: 7043.1ms | loss: 13.6ms | bwd: 24848.0ms | clip: 0.0ms | optim: 41.4ms
|
| 365 |
+
step 156 | loss: 2.542533 | lr 1.8000e-04 | norm: 0.2428 | dt: 32061.77ms | tok/sec: 32704.9 | mem: 122575MB | Time Left: 01 hrs : 49 mins
|
| 366 |
+
[TIMING] data: 4.0ms | fwd: 7073.7ms | loss: 13.6ms | bwd: 24923.4ms | clip: 0.0ms | optim: 41.6ms
|
| 367 |
+
step 157 | loss: 1.375999 | lr 1.8000e-04 | norm: 0.4895 | dt: 31992.00ms | tok/sec: 32776.2 | mem: 122575MB | Time Left: 01 hrs : 48 mins
|
| 368 |
+
[TIMING] data: 4.0ms | fwd: 7020.7ms | loss: 13.6ms | bwd: 24906.2ms | clip: 0.0ms | optim: 41.9ms
|
| 369 |
+
step 158 | loss: 1.545870 | lr 1.8000e-04 | norm: 0.2424 | dt: 32023.75ms | tok/sec: 32743.7 | mem: 122575MB | Time Left: 01 hrs : 47 mins
|
| 370 |
+
[TIMING] data: 4.0ms | fwd: 7075.6ms | loss: 13.7ms | bwd: 24883.4ms | clip: 0.0ms | optim: 41.6ms
|
| 371 |
+
step 159 | loss: 2.594631 | lr 1.8000e-04 | norm: 0.2939 | dt: 32020.48ms | tok/sec: 32747.0 | mem: 122575MB | Time Left: 01 hrs : 47 mins
|
| 372 |
+
[TIMING] data: 4.0ms | fwd: 7051.1ms | loss: 13.6ms | bwd: 24904.2ms | clip: 0.0ms | optim: 42.2ms
|
| 373 |
+
step 160 | loss: 2.661863 | lr 1.8000e-04 | norm: 0.3688 | dt: 32009.41ms | tok/sec: 32758.4 | mem: 122575MB | Time Left: 01 hrs : 46 mins
|
| 374 |
+
[TIMING] data: 4.0ms | fwd: 7062.3ms | loss: 14.3ms | bwd: 24881.0ms | clip: 0.0ms | optim: 42.3ms
|
| 375 |
+
step 161 | loss: 2.577728 | lr 1.8000e-04 | norm: 0.4705 | dt: 31956.44ms | tok/sec: 32812.7 | mem: 122575MB | Time Left: 01 hrs : 45 mins
|
| 376 |
+
[TIMING] data: 4.0ms | fwd: 7049.1ms | loss: 13.6ms | bwd: 24842.5ms | clip: 0.0ms | optim: 41.8ms
|
| 377 |
+
step 162 | loss: 2.705258 | lr 1.8000e-04 | norm: 0.3208 | dt: 31944.05ms | tok/sec: 32825.4 | mem: 122575MB | Time Left: 01 hrs : 45 mins
|
| 378 |
+
[TIMING] data: 4.0ms | fwd: 7097.6ms | loss: 13.5ms | bwd: 24781.2ms | clip: 0.0ms | optim: 42.2ms
|
| 379 |
+
step 163 | loss: 2.589881 | lr 1.8000e-04 | norm: 0.6815 | dt: 32085.84ms | tok/sec: 32680.3 | mem: 122575MB | Time Left: 01 hrs : 45 mins
|
| 380 |
+
[TIMING] data: 4.0ms | fwd: 7136.1ms | loss: 13.3ms | bwd: 24884.9ms | clip: 0.0ms | optim: 42.0ms
|
| 381 |
+
step 164 | loss: 2.845112 | lr 1.8000e-04 | norm: 1.0475 | dt: 32098.25ms | tok/sec: 32667.7 | mem: 122575MB | Time Left: 01 hrs : 44 mins
|
| 382 |
+
[TIMING] data: 4.1ms | fwd: 7078.6ms | loss: 14.2ms | bwd: 24951.9ms | clip: 0.0ms | optim: 43.6ms
|
| 383 |
+
step 165 | loss: 2.125706 | lr 1.8000e-04 | norm: 0.2963 | dt: 31951.21ms | tok/sec: 32818.0 | mem: 122575MB | Time Left: 01 hrs : 43 mins
|
| 384 |
+
[TIMING] data: 4.1ms | fwd: 7052.7ms | loss: 13.6ms | bwd: 24833.3ms | clip: 0.0ms | optim: 41.9ms
|
| 385 |
+
step 166 | loss: 1.426521 | lr 1.8000e-04 | norm: 0.5226 | dt: 31920.69ms | tok/sec: 32849.4 | mem: 122575MB | Time Left: 01 hrs : 43 mins
|
| 386 |
+
[TIMING] data: 4.0ms | fwd: 7075.3ms | loss: 13.7ms | bwd: 24780.3ms | clip: 0.0ms | optim: 41.8ms
|
| 387 |
+
step 167 | loss: 2.090064 | lr 1.8000e-04 | norm: 0.2458 | dt: 32091.30ms | tok/sec: 32674.8 | mem: 122575MB | Time Left: 01 hrs : 43 mins
|
| 388 |
+
[TIMING] data: 4.0ms | fwd: 7074.6ms | loss: 14.2ms | bwd: 24949.8ms | clip: 0.0ms | optim: 43.2ms
|
| 389 |
+
step 168 | loss: 2.427020 | lr 1.8000e-04 | norm: 0.2269 | dt: 32018.22ms | tok/sec: 32749.4 | mem: 122575MB | Time Left: 01 hrs : 42 mins
|
| 390 |
+
[TIMING] data: 4.0ms | fwd: 7087.9ms | loss: 13.6ms | bwd: 24865.0ms | clip: 0.0ms | optim: 42.1ms
|
| 391 |
+
step 169 | loss: 2.695097 | lr 1.8000e-04 | norm: 0.4211 | dt: 32018.68ms | tok/sec: 32748.9 | mem: 122575MB | Time Left: 01 hrs : 41 mins
|
| 392 |
+
[TIMING] data: 4.1ms | fwd: 7109.4ms | loss: 13.7ms | bwd: 24843.5ms | clip: 0.0ms | optim: 42.5ms
|
| 393 |
+
step 170 | loss: 2.573762 | lr 1.8000e-04 | norm: 0.3142 | dt: 31987.82ms | tok/sec: 32780.5 | mem: 122575MB | Time Left: 01 hrs : 41 mins
|
| 394 |
+
[TIMING] data: 4.0ms | fwd: 7068.1ms | loss: 13.5ms | bwd: 24854.5ms | clip: 0.0ms | optim: 42.1ms
|
| 395 |
+
step 171 | loss: 2.598310 | lr 1.8000e-04 | norm: 0.3354 | dt: 31993.08ms | tok/sec: 32775.1 | mem: 122575MB | Time Left: 01 hrs : 40 mins
|
| 396 |
+
[TIMING] data: 4.0ms | fwd: 7075.8ms | loss: 13.7ms | bwd: 24852.0ms | clip: 0.0ms | optim: 42.0ms
|
| 397 |
+
step 172 | loss: 2.395628 | lr 1.8000e-04 | norm: 0.3810 | dt: 32041.08ms | tok/sec: 32726.0 | mem: 122575MB | Time Left: 01 hrs : 40 mins
|
| 398 |
+
[TIMING] data: 4.0ms | fwd: 7092.6ms | loss: 13.7ms | bwd: 24883.2ms | clip: 0.0ms | optim: 41.9ms
|
| 399 |
+
step 173 | loss: 2.375246 | lr 1.8000e-04 | norm: 0.2382 | dt: 32018.01ms | tok/sec: 32749.6 | mem: 122575MB | Time Left: 01 hrs : 39 mins
|
| 400 |
+
[TIMING] data: 4.0ms | fwd: 7026.0ms | loss: 13.6ms | bwd: 24926.8ms | clip: 0.0ms | optim: 42.1ms
|
| 401 |
+
step 174 | loss: 2.549661 | lr 1.8000e-04 | norm: 0.5273 | dt: 31992.92ms | tok/sec: 32775.3 | mem: 122575MB | Time Left: 01 hrs : 39 mins
|
| 402 |
+
[TIMING] data: 4.0ms | fwd: 7039.1ms | loss: 13.7ms | bwd: 24888.6ms | clip: 0.0ms | optim: 41.8ms
|
| 403 |
+
step 175 | loss: 1.800797 | lr 1.8000e-04 | norm: 0.4966 | dt: 32007.09ms | tok/sec: 32760.7 | mem: 122575MB | Time Left: 01 hrs : 38 mins
|
| 404 |
+
[TIMING] data: 4.0ms | fwd: 7047.3ms | loss: 13.7ms | bwd: 24894.4ms | clip: 0.0ms | optim: 41.9ms
|
| 405 |
+
step 176 | loss: 1.347159 | lr 1.8000e-04 | norm: 0.6412 | dt: 31987.28ms | tok/sec: 32781.0 | mem: 122575MB | Time Left: 01 hrs : 38 mins
|
| 406 |
+
[TIMING] data: 4.0ms | fwd: 7099.4ms | loss: 13.6ms | bwd: 24823.2ms | clip: 0.0ms | optim: 41.4ms
|
| 407 |
+
step 177 | loss: 1.291482 | lr 1.8000e-04 | norm: 0.4226 | dt: 32026.18ms | tok/sec: 32741.2 | mem: 122575MB | Time Left: 01 hrs : 37 mins
|
| 408 |
+
[TIMING] data: 3.9ms | fwd: 7109.1ms | loss: 13.7ms | bwd: 24852.1ms | clip: 0.0ms | optim: 41.6ms
|
| 409 |
+
step 178 | loss: 2.455166 | lr 1.8000e-04 | norm: 0.3493 | dt: 31981.51ms | tok/sec: 32786.9 | mem: 122575MB | Time Left: 01 hrs : 37 mins
|
| 410 |
+
[TIMING] data: 4.1ms | fwd: 7045.1ms | loss: 13.7ms | bwd: 24870.8ms | clip: 0.0ms | optim: 42.2ms
|
| 411 |
+
step 179 | loss: 2.339830 | lr 1.8000e-04 | norm: 0.2493 | dt: 32052.02ms | tok/sec: 32714.8 | mem: 122575MB | Time Left: 01 hrs : 36 mins
|
| 412 |
+
[TIMING] data: 4.1ms | fwd: 6998.5ms | loss: 13.6ms | bwd: 24988.2ms | clip: 0.0ms | optim: 41.7ms
|
| 413 |
+
step 180 | loss: 2.674317 | lr 1.8000e-04 | norm: 0.3065 | dt: 32014.75ms | tok/sec: 32752.9 | mem: 122575MB | Time Left: 01 hrs : 36 mins
|
| 414 |
+
[TIMING] data: 4.0ms | fwd: 7083.2ms | loss: 13.6ms | bwd: 24864.7ms | clip: 0.0ms | optim: 43.7ms
|
| 415 |
+
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/data/mix128_b6_60/train_shard_00002.bin
|
| 416 |
+
100000000
|
| 417 |
+
step 181 | loss: 2.259531 | lr 1.8000e-04 | norm: 0.4289 | dt: 32036.72ms | tok/sec: 32730.4 | mem: 122575MB | Time Left: 01 hrs : 35 mins
|
| 418 |
+
[TIMING] data: 10.1ms | fwd: 7030.9ms | loss: 13.6ms | bwd: 24934.4ms | clip: 0.0ms | optim: 42.1ms
|
| 419 |
+
step 182 | loss: 3.300698 | lr 1.8000e-04 | norm: 1.0889 | dt: 31974.12ms | tok/sec: 32794.5 | mem: 122575MB | Time Left: 01 hrs : 34 mins
|
| 420 |
+
[TIMING] data: 4.1ms | fwd: 7105.2ms | loss: 14.2ms | bwd: 24803.2ms | clip: 0.0ms | optim: 41.7ms
|
| 421 |
+
step 183 | loss: 2.627776 | lr 1.8000e-04 | norm: 0.5940 | dt: 31966.10ms | tok/sec: 32802.7 | mem: 122575MB | Time Left: 01 hrs : 34 mins
|
| 422 |
+
[TIMING] data: 4.1ms | fwd: 7060.7ms | loss: 14.2ms | bwd: 24839.5ms | clip: 0.0ms | optim: 42.0ms
|
| 423 |
+
step 184 | loss: 2.891413 | lr 1.8000e-04 | norm: 0.4747 | dt: 32138.93ms | tok/sec: 32626.3 | mem: 122575MB | Time Left: 01 hrs : 34 mins
|
| 424 |
+
[TIMING] data: 4.1ms | fwd: 7163.0ms | loss: 13.7ms | bwd: 24910.7ms | clip: 0.0ms | optim: 41.9ms
|
| 425 |
+
step 185 | loss: 2.716620 | lr 1.8000e-04 | norm: 0.5272 | dt: 32038.22ms | tok/sec: 32728.9 | mem: 122575MB | Time Left: 01 hrs : 33 mins
|
| 426 |
+
[TIMING] data: 4.0ms | fwd: 7086.4ms | loss: 13.6ms | bwd: 24885.2ms | clip: 0.0ms | optim: 43.4ms
|
| 427 |
+
step 186 | loss: 2.347850 | lr 1.8000e-04 | norm: 0.3069 | dt: 32037.71ms | tok/sec: 32729.4 | mem: 122575MB | Time Left: 01 hrs : 32 mins
|
| 428 |
+
[TIMING] data: 4.1ms | fwd: 7145.3ms | loss: 13.6ms | bwd: 24826.4ms | clip: 0.0ms | optim: 42.5ms
|
| 429 |
+
step 187 | loss: 2.631252 | lr 1.8000e-04 | norm: 0.2720 | dt: 31989.88ms | tok/sec: 32778.4 | mem: 122575MB | Time Left: 01 hrs : 32 mins
|
| 430 |
+
[TIMING] data: 4.0ms | fwd: 7053.7ms | loss: 13.8ms | bwd: 24870.7ms | clip: 0.0ms | optim: 42.1ms
|
| 431 |
+
step 188 | loss: 2.525319 | lr 1.8000e-04 | norm: 0.2704 | dt: 32001.61ms | tok/sec: 32766.4 | mem: 122575MB | Time Left: 01 hrs : 31 mins
|
| 432 |
+
[TIMING] data: 4.0ms | fwd: 7043.6ms | loss: 13.7ms | bwd: 24892.5ms | clip: 0.0ms | optim: 42.1ms
|
| 433 |
+
step 189 | loss: 2.079133 | lr 1.8000e-04 | norm: 1.2453 | dt: 32008.44ms | tok/sec: 32759.4 | mem: 122575MB | Time Left: 01 hrs : 31 mins
|
| 434 |
+
[TIMING] data: 4.0ms | fwd: 7032.0ms | loss: 13.7ms | bwd: 24910.6ms | clip: 0.0ms | optim: 42.6ms
|
| 435 |
+
step 190 | loss: 1.820425 | lr 1.8000e-04 | norm: 0.4949 | dt: 32023.07ms | tok/sec: 32744.4 | mem: 122575MB | Time Left: 01 hrs : 30 mins
|
| 436 |
+
[TIMING] data: 4.0ms | fwd: 7020.5ms | loss: 13.6ms | bwd: 24937.5ms | clip: 0.0ms | optim: 41.9ms
|
| 437 |
+
step 191 | loss: 2.631012 | lr 1.8000e-04 | norm: 0.3792 | dt: 32033.39ms | tok/sec: 32733.8 | mem: 122575MB | Time Left: 01 hrs : 30 mins
|
| 438 |
+
[TIMING] data: 4.1ms | fwd: 7084.7ms | loss: 13.7ms | bwd: 24882.8ms | clip: 0.0ms | optim: 42.2ms
|
| 439 |
+
step 192 | loss: 2.532935 | lr 1.8000e-04 | norm: 0.2769 | dt: 32015.87ms | tok/sec: 32751.8 | mem: 122575MB | Time Left: 01 hrs : 29 mins
|
| 440 |
+
[TIMING] data: 4.0ms | fwd: 7024.6ms | loss: 13.6ms | bwd: 24926.1ms | clip: 0.0ms | optim: 41.8ms
|
| 441 |
+
step 193 | loss: 1.596987 | lr 1.8000e-04 | norm: 0.5695 | dt: 32137.82ms | tok/sec: 32627.5 | mem: 122575MB | Time Left: 01 hrs : 29 mins
|
| 442 |
+
[TIMING] data: 4.1ms | fwd: 7114.7ms | loss: 13.7ms | bwd: 24957.4ms | clip: 0.0ms | optim: 42.1ms
|
| 443 |
+
step 194 | loss: 2.650199 | lr 1.8000e-04 | norm: 0.2502 | dt: 32050.25ms | tok/sec: 32716.6 | mem: 122575MB | Time Left: 01 hrs : 28 mins
|
| 444 |
+
[TIMING] data: 4.0ms | fwd: 7120.1ms | loss: 13.7ms | bwd: 24864.9ms | clip: 0.0ms | optim: 41.9ms
|
| 445 |
+
step 195 | loss: 2.490386 | lr 1.8000e-04 | norm: 0.2846 | dt: 32006.30ms | tok/sec: 32761.5 | mem: 122575MB | Time Left: 01 hrs : 28 mins
|
| 446 |
+
[TIMING] data: 4.0ms | fwd: 7060.0ms | loss: 13.7ms | bwd: 24881.9ms | clip: 0.0ms | optim: 41.1ms
|
| 447 |
+
step 196 | loss: 3.074560 | lr 1.8000e-04 | norm: 0.9805 | dt: 32132.44ms | tok/sec: 32632.9 | mem: 122575MB | Time Left: 01 hrs : 27 mins
|
| 448 |
+
[TIMING] data: 4.0ms | fwd: 7083.9ms | loss: 13.6ms | bwd: 24982.8ms | clip: 0.0ms | optim: 42.5ms
|
| 449 |
+
step 197 | loss: 3.827067 | lr 1.8000e-04 | norm: 1.0644 | dt: 32102.73ms | tok/sec: 32663.1 | mem: 122575MB | Time Left: 01 hrs : 27 mins
|
| 450 |
+
[TIMING] data: 4.0ms | fwd: 7112.2ms | loss: 13.7ms | bwd: 24925.5ms | clip: 0.0ms | optim: 41.7ms
|
| 451 |
+
step 198 | loss: 3.855266 | lr 1.8000e-04 | norm: 0.8764 | dt: 32144.31ms | tok/sec: 32620.9 | mem: 122575MB | Time Left: 01 hrs : 26 mins
|
| 452 |
+
[TIMING] data: 4.0ms | fwd: 7065.2ms | loss: 13.6ms | bwd: 25014.0ms | clip: 0.0ms | optim: 41.8ms
|
| 453 |
+
step 199 | loss: 3.745534 | lr 1.8000e-04 | norm: 0.7022 | dt: 32107.70ms | tok/sec: 32658.1 | mem: 122575MB | Time Left: 01 hrs : 26 mins
|
| 454 |
+
[TIMING] data: 4.0ms | fwd: 7084.8ms | loss: 13.6ms | bwd: 24957.6ms | clip: 0.0ms | optim: 42.1ms
|
| 455 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt (safely)
|
| 456 |
+
step 200 | loss: 3.722009 | lr 1.8000e-04 | norm: 0.6510 | dt: 31383.11ms | tok/sec: 33412.1 | mem: 122575MB | Time Left: 01 hrs : 23 mins
|
| 457 |
+
[TIMING] data: 4.2ms | fwd: 6840.0ms | loss: 13.4ms | bwd: 24477.6ms | clip: 0.0ms | optim: 41.9ms
|
| 458 |
+
step 201 | loss: 3.624566 | lr 1.8000e-04 | norm: 0.5937 | dt: 31828.20ms | tok/sec: 32944.9 | mem: 122575MB | Time Left: 01 hrs : 24 mins
|
| 459 |
+
[TIMING] data: 4.3ms | fwd: 7030.1ms | loss: 13.7ms | bwd: 24732.2ms | clip: 0.0ms | optim: 42.1ms
|
| 460 |
+
step 202 | loss: 3.520792 | lr 1.8000e-04 | norm: 0.4323 | dt: 32053.74ms | tok/sec: 32713.1 | mem: 122575MB | Time Left: 01 hrs : 24 mins
|
| 461 |
+
[TIMING] data: 4.3ms | fwd: 7077.8ms | loss: 13.8ms | bwd: 24909.8ms | clip: 0.0ms | optim: 42.2ms
|
| 462 |
+
step 203 | loss: 2.550414 | lr 1.8000e-04 | norm: 0.3473 | dt: 32012.46ms | tok/sec: 32755.3 | mem: 122575MB | Time Left: 01 hrs : 23 mins
|
| 463 |
+
[TIMING] data: 4.2ms | fwd: 7081.7ms | loss: 13.9ms | bwd: 24865.2ms | clip: 0.0ms | optim: 41.7ms
|
| 464 |
+
step 204 | loss: 2.486615 | lr 1.8000e-04 | norm: 0.2540 | dt: 32127.45ms | tok/sec: 32638.0 | mem: 122575MB | Time Left: 01 hrs : 23 mins
|
| 465 |
+
[TIMING] data: 4.3ms | fwd: 7067.3ms | loss: 13.8ms | bwd: 24994.4ms | clip: 0.0ms | optim: 41.9ms
|
| 466 |
+
step 205 | loss: 2.611028 | lr 1.8000e-04 | norm: 0.2488 | dt: 32077.03ms | tok/sec: 32689.3 | mem: 122575MB | Time Left: 01 hrs : 22 mins
|
| 467 |
+
[TIMING] data: 4.2ms | fwd: 7033.3ms | loss: 13.9ms | bwd: 24977.9ms | clip: 0.0ms | optim: 42.0ms
|
| 468 |
+
step 206 | loss: 2.583280 | lr 1.8000e-04 | norm: 0.2103 | dt: 32039.37ms | tok/sec: 32727.7 | mem: 122575MB | Time Left: 01 hrs : 22 mins
|
| 469 |
+
[TIMING] data: 4.3ms | fwd: 7096.6ms | loss: 13.9ms | bwd: 24876.4ms | clip: 0.0ms | optim: 42.3ms
|
| 470 |
+
step 207 | loss: 1.613403 | lr 1.8000e-04 | norm: 0.8034 | dt: 32087.44ms | tok/sec: 32678.7 | mem: 122575MB | Time Left: 01 hrs : 21 mins
|
| 471 |
+
[TIMING] data: 4.2ms | fwd: 7067.3ms | loss: 13.8ms | bwd: 24954.4ms | clip: 0.0ms | optim: 41.9ms
|
| 472 |
+
step 208 | loss: 2.497029 | lr 1.8000e-04 | norm: 0.2448 | dt: 32068.45ms | tok/sec: 32698.1 | mem: 122575MB | Time Left: 01 hrs : 21 mins
|
| 473 |
+
[TIMING] data: 4.2ms | fwd: 7068.7ms | loss: 13.9ms | bwd: 24934.1ms | clip: 0.0ms | optim: 41.8ms
|
| 474 |
+
step 209 | loss: 2.128287 | lr 1.8000e-04 | norm: 0.4867 | dt: 32029.98ms | tok/sec: 32737.3 | mem: 122575MB | Time Left: 01 hrs : 20 mins
|
| 475 |
+
[TIMING] data: 4.2ms | fwd: 7048.5ms | loss: 13.9ms | bwd: 24914.7ms | clip: 0.0ms | optim: 42.2ms
|
| 476 |
+
step 210 | loss: 2.715762 | lr 1.8000e-04 | norm: 0.4115 | dt: 32114.45ms | tok/sec: 32651.2 | mem: 122575MB | Time Left: 01 hrs : 20 mins
|
| 477 |
+
[TIMING] data: 4.2ms | fwd: 7073.5ms | loss: 13.9ms | bwd: 24975.0ms | clip: 0.0ms | optim: 41.9ms
|
| 478 |
+
step 211 | loss: 2.028214 | lr 1.8000e-04 | norm: 0.2539 | dt: 32061.53ms | tok/sec: 32705.1 | mem: 122575MB | Time Left: 01 hrs : 19 mins
|
| 479 |
+
[TIMING] data: 4.3ms | fwd: 7081.0ms | loss: 13.9ms | bwd: 24914.5ms | clip: 0.0ms | optim: 41.9ms
|
| 480 |
+
step 212 | loss: 1.614248 | lr 1.8000e-04 | norm: 0.3043 | dt: 32152.70ms | tok/sec: 32612.4 | mem: 122575MB | Time Left: 01 hrs : 19 mins
|
| 481 |
+
[TIMING] data: 4.2ms | fwd: 7104.3ms | loss: 14.0ms | bwd: 24979.8ms | clip: 0.0ms | optim: 44.5ms
|
| 482 |
+
step 213 | loss: 2.203980 | lr 1.8000e-04 | norm: 0.3089 | dt: 32123.33ms | tok/sec: 32642.2 | mem: 122575MB | Time Left: 01 hrs : 18 mins
|
| 483 |
+
[TIMING] data: 4.2ms | fwd: 7055.7ms | loss: 13.8ms | bwd: 25002.0ms | clip: 0.0ms | optim: 41.9ms
|
| 484 |
+
step 214 | loss: 2.555761 | lr 1.8000e-04 | norm: 0.2425 | dt: 32090.07ms | tok/sec: 32676.0 | mem: 122575MB | Time Left: 01 hrs : 18 mins
|
| 485 |
+
[TIMING] data: 4.3ms | fwd: 7087.5ms | loss: 13.8ms | bwd: 24936.2ms | clip: 0.0ms | optim: 42.6ms
|
| 486 |
+
step 215 | loss: 1.995283 | lr 1.8000e-04 | norm: 0.3371 | dt: 32177.00ms | tok/sec: 32587.7 | mem: 122575MB | Time Left: 01 hrs : 17 mins
|
| 487 |
+
[TIMING] data: 4.2ms | fwd: 7125.9ms | loss: 13.9ms | bwd: 24984.5ms | clip: 0.0ms | optim: 42.6ms
|
| 488 |
+
step 216 | loss: 2.084974 | lr 1.8000e-04 | norm: 0.2961 | dt: 32123.44ms | tok/sec: 32642.1 | mem: 122575MB | Time Left: 01 hrs : 17 mins
|
| 489 |
+
[TIMING] data: 4.3ms | fwd: 7126.2ms | loss: 13.8ms | bwd: 24930.9ms | clip: 0.0ms | optim: 42.6ms
|
| 490 |
+
step 217 | loss: 2.387355 | lr 1.8000e-04 | norm: 0.3109 | dt: 32067.81ms | tok/sec: 32698.7 | mem: 122575MB | Time Left: 01 hrs : 16 mins
|
| 491 |
+
[TIMING] data: 4.2ms | fwd: 7067.1ms | loss: 13.9ms | bwd: 24934.8ms | clip: 0.0ms | optim: 42.1ms
|
| 492 |
+
step 218 | loss: 2.235594 | lr 1.8000e-04 | norm: 0.1941 | dt: 32017.13ms | tok/sec: 32750.5 | mem: 122575MB | Time Left: 01 hrs : 15 mins
|
| 493 |
+
[TIMING] data: 4.2ms | fwd: 7107.6ms | loss: 13.9ms | bwd: 24843.1ms | clip: 0.0ms | optim: 42.6ms
|
| 494 |
+
step 219 | loss: 1.918685 | lr 1.8000e-04 | norm: 0.2732 | dt: 32217.36ms | tok/sec: 32546.9 | mem: 122575MB | Time Left: 01 hrs : 15 mins
|
| 495 |
+
[TIMING] data: 4.4ms | fwd: 7081.4ms | loss: 13.8ms | bwd: 25068.3ms | clip: 0.0ms | optim: 43.6ms
|
| 496 |
+
step 220 | loss: 2.082500 | lr 1.8000e-04 | norm: 0.8334 | dt: 32015.79ms | tok/sec: 32751.8 | mem: 122575MB | Time Left: 01 hrs : 14 mins
|
| 497 |
+
[TIMING] data: 4.2ms | fwd: 7050.5ms | loss: 13.9ms | bwd: 24899.3ms | clip: 0.0ms | optim: 41.6ms
|
| 498 |
+
step 221 | loss: 2.067728 | lr 1.8000e-04 | norm: 0.1901 | dt: 32089.38ms | tok/sec: 32676.7 | mem: 122575MB | Time Left: 01 hrs : 14 mins
|
| 499 |
+
[TIMING] data: 4.2ms | fwd: 7063.2ms | loss: 13.8ms | bwd: 24960.1ms | clip: 0.0ms | optim: 42.2ms
|
| 500 |
+
step 222 | loss: 1.568204 | lr 1.8000e-04 | norm: 0.2606 | dt: 32068.77ms | tok/sec: 32697.7 | mem: 122575MB | Time Left: 01 hrs : 13 mins
|
| 501 |
+
[TIMING] data: 4.2ms | fwd: 7079.4ms | loss: 13.9ms | bwd: 24922.8ms | clip: 0.0ms | optim: 42.6ms
|
| 502 |
+
step 223 | loss: 2.441509 | lr 1.8000e-04 | norm: 0.2549 | dt: 32055.89ms | tok/sec: 32710.9 | mem: 122575MB | Time Left: 01 hrs : 13 mins
|
| 503 |
+
[TIMING] data: 4.2ms | fwd: 7043.4ms | loss: 14.5ms | bwd: 24946.0ms | clip: 0.0ms | optim: 41.9ms
|
| 504 |
+
step 224 | loss: 2.467334 | lr 1.8000e-04 | norm: 0.2270 | dt: 32025.22ms | tok/sec: 32742.2 | mem: 122575MB | Time Left: 01 hrs : 12 mins
|
| 505 |
+
[TIMING] data: 4.2ms | fwd: 7018.0ms | loss: 13.8ms | bwd: 24939.6ms | clip: 0.0ms | optim: 43.8ms
|
| 506 |
+
step 225 | loss: 1.715126 | lr 1.8000e-04 | norm: 0.4046 | dt: 31991.12ms | tok/sec: 32777.1 | mem: 122575MB | Time Left: 01 hrs : 11 mins
|
| 507 |
+
[TIMING] data: 4.2ms | fwd: 7007.2ms | loss: 13.8ms | bwd: 24918.3ms | clip: 0.0ms | optim: 41.9ms
|
| 508 |
+
step 226 | loss: 2.665848 | lr 1.8000e-04 | norm: 0.2930 | dt: 32026.70ms | tok/sec: 32740.7 | mem: 122575MB | Time Left: 01 hrs : 11 mins
|
| 509 |
+
[TIMING] data: 4.2ms | fwd: 7068.7ms | loss: 13.6ms | bwd: 24892.5ms | clip: 0.0ms | optim: 41.8ms
|
| 510 |
+
step 227 | loss: 2.050191 | lr 1.8000e-04 | norm: 0.2953 | dt: 31995.95ms | tok/sec: 32772.2 | mem: 122575MB | Time Left: 01 hrs : 10 mins
|
| 511 |
+
[TIMING] data: 4.3ms | fwd: 7060.8ms | loss: 13.9ms | bwd: 24869.9ms | clip: 0.0ms | optim: 41.0ms
|
| 512 |
+
step 228 | loss: 1.264867 | lr 1.8000e-04 | norm: 0.3630 | dt: 32036.22ms | tok/sec: 32731.0 | mem: 122575MB | Time Left: 01 hrs : 10 mins
|
| 513 |
+
[TIMING] data: 4.2ms | fwd: 7054.8ms | loss: 13.8ms | bwd: 24915.6ms | clip: 0.0ms | optim: 42.0ms
|
| 514 |
+
step 229 | loss: 2.241357 | lr 1.8000e-04 | norm: 0.1834 | dt: 32000.72ms | tok/sec: 32767.3 | mem: 122575MB | Time Left: 01 hrs : 09 mins
|
| 515 |
+
[TIMING] data: 4.3ms | fwd: 7074.6ms | loss: 13.8ms | bwd: 24860.2ms | clip: 0.0ms | optim: 41.9ms
|
| 516 |
+
step 230 | loss: 2.312558 | lr 1.8000e-04 | norm: 0.2306 | dt: 31984.24ms | tok/sec: 32784.1 | mem: 122575MB | Time Left: 01 hrs : 09 mins
|
| 517 |
+
[TIMING] data: 4.2ms | fwd: 7079.2ms | loss: 13.9ms | bwd: 24838.8ms | clip: 0.0ms | optim: 42.2ms
|
| 518 |
+
step 231 | loss: 2.199832 | lr 1.8000e-04 | norm: 0.2231 | dt: 31945.92ms | tok/sec: 32823.5 | mem: 122575MB | Time Left: 01 hrs : 08 mins
|
| 519 |
+
[TIMING] data: 4.2ms | fwd: 7048.3ms | loss: 13.9ms | bwd: 24831.2ms | clip: 0.0ms | optim: 42.4ms
|
| 520 |
+
step 232 | loss: 2.510937 | lr 1.8000e-04 | norm: 0.2639 | dt: 31998.61ms | tok/sec: 32769.4 | mem: 122575MB | Time Left: 01 hrs : 08 mins
|
| 521 |
+
[TIMING] data: 4.2ms | fwd: 7013.7ms | loss: 13.8ms | bwd: 24917.3ms | clip: 0.0ms | optim: 43.7ms
|
| 522 |
+
step 233 | loss: 2.185982 | lr 1.8000e-04 | norm: 0.2011 | dt: 31944.48ms | tok/sec: 32825.0 | mem: 122575MB | Time Left: 01 hrs : 07 mins
|
| 523 |
+
[TIMING] data: 4.1ms | fwd: 7061.0ms | loss: 14.5ms | bwd: 24816.7ms | clip: 0.0ms | optim: 42.2ms
|
| 524 |
+
step 234 | loss: 2.331032 | lr 1.8000e-04 | norm: 0.2484 | dt: 32015.92ms | tok/sec: 32751.7 | mem: 122575MB | Time Left: 01 hrs : 07 mins
|
| 525 |
+
[TIMING] data: 4.5ms | fwd: 7103.6ms | loss: 13.9ms | bwd: 24845.6ms | clip: 0.0ms | optim: 42.4ms
|
| 526 |
+
step 235 | loss: 2.623552 | lr 1.8000e-04 | norm: 0.2158 | dt: 31964.38ms | tok/sec: 32804.5 | mem: 122575MB | Time Left: 01 hrs : 06 mins
|
| 527 |
+
[TIMING] data: 4.2ms | fwd: 7037.2ms | loss: 13.7ms | bwd: 24861.1ms | clip: 0.0ms | optim: 42.1ms
|
| 528 |
+
step 236 | loss: 1.967629 | lr 1.8000e-04 | norm: 0.3296 | dt: 31917.87ms | tok/sec: 32852.3 | mem: 122575MB | Time Left: 01 hrs : 05 mins
|
| 529 |
+
[TIMING] data: 4.2ms | fwd: 7114.1ms | loss: 13.9ms | bwd: 24735.7ms | clip: 0.0ms | optim: 44.2ms
|
| 530 |
+
step 237 | loss: 2.350394 | lr 1.8000e-04 | norm: 0.1798 | dt: 31911.87ms | tok/sec: 32858.5 | mem: 122575MB | Time Left: 01 hrs : 05 mins
|
| 531 |
+
[TIMING] data: 4.2ms | fwd: 7017.6ms | loss: 13.8ms | bwd: 24828.4ms | clip: 0.0ms | optim: 42.0ms
|
| 532 |
+
step 238 | loss: 2.410821 | lr 1.8000e-04 | norm: 0.9710 | dt: 31928.34ms | tok/sec: 32841.5 | mem: 122575MB | Time Left: 01 hrs : 04 mins
|
| 533 |
+
[TIMING] data: 4.3ms | fwd: 7075.1ms | loss: 13.7ms | bwd: 24787.6ms | clip: 0.0ms | optim: 41.9ms
|
| 534 |
+
step 239 | loss: 1.982906 | lr 1.8000e-04 | norm: 0.1764 | dt: 31978.65ms | tok/sec: 32789.9 | mem: 122575MB | Time Left: 01 hrs : 04 mins
|
| 535 |
+
[TIMING] data: 4.3ms | fwd: 7059.3ms | loss: 14.3ms | bwd: 24853.1ms | clip: 0.0ms | optim: 41.8ms
|
| 536 |
+
step 240 | loss: 1.967315 | lr 1.8000e-04 | norm: 0.1756 | dt: 31940.75ms | tok/sec: 32828.8 | mem: 122575MB | Time Left: 01 hrs : 03 mins
|
| 537 |
+
[TIMING] data: 4.2ms | fwd: 7046.0ms | loss: 13.9ms | bwd: 24829.0ms | clip: 0.0ms | optim: 41.8ms
|
| 538 |
+
step 241 | loss: 1.787140 | lr 1.8000e-04 | norm: 0.3140 | dt: 31928.22ms | tok/sec: 32841.7 | mem: 122575MB | Time Left: 01 hrs : 03 mins
|
| 539 |
+
[TIMING] data: 4.2ms | fwd: 7043.1ms | loss: 13.8ms | bwd: 24819.4ms | clip: 0.0ms | optim: 41.9ms
|
| 540 |
+
step 242 | loss: 2.534278 | lr 1.8000e-04 | norm: 0.2826 | dt: 32030.10ms | tok/sec: 32737.2 | mem: 122575MB | Time Left: 01 hrs : 02 mins
|
| 541 |
+
[TIMING] data: 4.2ms | fwd: 7037.8ms | loss: 13.9ms | bwd: 24926.5ms | clip: 0.0ms | optim: 41.9ms
|
| 542 |
+
step 243 | loss: 2.126674 | lr 1.8000e-04 | norm: 0.2556 | dt: 31993.60ms | tok/sec: 32774.6 | mem: 122575MB | Time Left: 01 hrs : 02 mins
|
| 543 |
+
[TIMING] data: 4.2ms | fwd: 7032.4ms | loss: 13.8ms | bwd: 24895.6ms | clip: 0.0ms | optim: 41.8ms
|
| 544 |
+
step 244 | loss: 2.785931 | lr 1.8000e-04 | norm: 0.2386 | dt: 31993.30ms | tok/sec: 32774.9 | mem: 122575MB | Time Left: 01 hrs : 01 mins
|
| 545 |
+
[TIMING] data: 4.4ms | fwd: 7031.2ms | loss: 13.9ms | bwd: 24896.0ms | clip: 0.0ms | optim: 42.0ms
|
| 546 |
+
step 245 | loss: 2.550959 | lr 1.8000e-04 | norm: 0.7048 | dt: 31924.88ms | tok/sec: 32845.1 | mem: 122575MB | Time Left: 01 hrs : 01 mins
|
| 547 |
+
[TIMING] data: 4.2ms | fwd: 7017.3ms | loss: 13.7ms | bwd: 24842.5ms | clip: 0.0ms | optim: 41.3ms
|
| 548 |
+
step 246 | loss: 2.988145 | lr 1.8000e-04 | norm: 0.5490 | dt: 31840.16ms | tok/sec: 32932.5 | mem: 122575MB | Time Left: 01 hrs : 00 mins
|
| 549 |
+
[TIMING] data: 4.2ms | fwd: 6993.2ms | loss: 13.7ms | bwd: 24781.7ms | clip: 0.0ms | optim: 41.6ms
|
| 550 |
+
step 247 | loss: 2.219491 | lr 1.8000e-04 | norm: 0.4198 | dt: 31894.31ms | tok/sec: 32876.6 | mem: 122575MB | Time Left: 01 hrs : 00 mins
|
| 551 |
+
[TIMING] data: 4.2ms | fwd: 7052.4ms | loss: 13.9ms | bwd: 24775.7ms | clip: 0.0ms | optim: 42.3ms
|
| 552 |
+
step 248 | loss: 2.636802 | lr 1.8000e-04 | norm: 0.2899 | dt: 32022.99ms | tok/sec: 32744.5 | mem: 122575MB | Time Left: 00 hrs : 59 mins
|
| 553 |
+
[TIMING] data: 4.2ms | fwd: 7108.4ms | loss: 13.8ms | bwd: 24848.9ms | clip: 0.0ms | optim: 41.7ms
|
| 554 |
+
step 249 | loss: 2.414184 | lr 1.8000e-04 | norm: 0.3764 | dt: 32031.91ms | tok/sec: 32735.4 | mem: 122575MB | Time Left: 00 hrs : 59 mins
|
| 555 |
+
[TIMING] data: 4.3ms | fwd: 7078.0ms | loss: 13.8ms | bwd: 24888.2ms | clip: 0.0ms | optim: 41.9ms
|
| 556 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_250.ckpt (safely)
|
| 557 |
+
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_50.ckpt
|
| 558 |
+
validation loss: 2.6174
|
| 559 |
+
step 250 | loss: 2.567578 | lr 1.8000e-04 | norm: 0.4309 | dt: 48333.70ms | tok/sec: 21694.5 | mem: 122575MB | Time Left: 01 hrs : 28 mins
|
| 560 |
+
[TIMING] data: 4.3ms | fwd: 6967.8ms | loss: 13.7ms | bwd: 24426.8ms | clip: 0.0ms | optim: 42.0ms
|
| 561 |
+
step 251 | loss: 2.405287 | lr 1.8000e-04 | norm: 0.5643 | dt: 31812.41ms | tok/sec: 32961.2 | mem: 122575MB | Time Left: 00 hrs : 57 mins
|
| 562 |
+
[TIMING] data: 4.2ms | fwd: 7070.9ms | loss: 13.8ms | bwd: 24675.4ms | clip: 0.0ms | optim: 42.2ms
|
| 563 |
+
step 252 | loss: 2.519504 | lr 1.8000e-04 | norm: 0.2225 | dt: 31981.10ms | tok/sec: 32787.4 | mem: 122575MB | Time Left: 00 hrs : 57 mins
|
| 564 |
+
[TIMING] data: 4.2ms | fwd: 7096.6ms | loss: 13.6ms | bwd: 24818.5ms | clip: 0.0ms | optim: 41.5ms
|
| 565 |
+
step 253 | loss: 2.395631 | lr 1.8000e-04 | norm: 0.5835 | dt: 32011.73ms | tok/sec: 32756.0 | mem: 122575MB | Time Left: 00 hrs : 57 mins
|
| 566 |
+
[TIMING] data: 4.2ms | fwd: 7080.0ms | loss: 13.9ms | bwd: 24865.9ms | clip: 0.0ms | optim: 42.0ms
|
| 567 |
+
step 254 | loss: 2.685584 | lr 1.8000e-04 | norm: 0.4667 | dt: 32044.30ms | tok/sec: 32722.7 | mem: 122575MB | Time Left: 00 hrs : 56 mins
|
| 568 |
+
[TIMING] data: 4.2ms | fwd: 7117.0ms | loss: 13.8ms | bwd: 24861.6ms | clip: 0.0ms | optim: 41.9ms
|
| 569 |
+
step 255 | loss: 2.038981 | lr 1.8000e-04 | norm: 0.2344 | dt: 32083.82ms | tok/sec: 32682.4 | mem: 122575MB | Time Left: 00 hrs : 56 mins
|
| 570 |
+
[TIMING] data: 4.2ms | fwd: 7134.6ms | loss: 13.8ms | bwd: 24883.2ms | clip: 0.0ms | optim: 41.5ms
|
| 571 |
+
step 256 | loss: 2.623349 | lr 1.8000e-04 | norm: 0.3394 | dt: 32111.72ms | tok/sec: 32654.0 | mem: 122575MB | Time Left: 00 hrs : 55 mins
|
| 572 |
+
[TIMING] data: 4.3ms | fwd: 7096.8ms | loss: 13.8ms | bwd: 24948.9ms | clip: 0.0ms | optim: 42.1ms
|
| 573 |
+
step 257 | loss: 2.405479 | lr 1.8000e-04 | norm: 0.4303 | dt: 32021.90ms | tok/sec: 32745.6 | mem: 122575MB | Time Left: 00 hrs : 54 mins
|
| 574 |
+
[TIMING] data: 4.2ms | fwd: 7121.9ms | loss: 13.8ms | bwd: 24833.4ms | clip: 0.0ms | optim: 42.7ms
|
| 575 |
+
step 258 | loss: 2.319361 | lr 1.8000e-04 | norm: 0.2549 | dt: 31962.07ms | tok/sec: 32806.9 | mem: 122575MB | Time Left: 00 hrs : 54 mins
|
| 576 |
+
[TIMING] data: 4.2ms | fwd: 7046.9ms | loss: 13.8ms | bwd: 24849.1ms | clip: 0.0ms | optim: 41.8ms
|
| 577 |
+
step 259 | loss: 2.038117 | lr 1.8000e-04 | norm: 0.3955 | dt: 31952.21ms | tok/sec: 32817.0 | mem: 122575MB | Time Left: 00 hrs : 53 mins
|
| 578 |
+
[TIMING] data: 4.2ms | fwd: 7080.9ms | loss: 14.3ms | bwd: 24804.8ms | clip: 0.0ms | optim: 42.1ms
|
| 579 |
+
step 260 | loss: 2.506370 | lr 1.8000e-04 | norm: 0.2365 | dt: 31984.95ms | tok/sec: 32783.4 | mem: 122575MB | Time Left: 00 hrs : 53 mins
|
| 580 |
+
[TIMING] data: 5.5ms | fwd: 7063.5ms | loss: 13.9ms | bwd: 24854.0ms | clip: 0.0ms | optim: 42.3ms
|
| 581 |
+
step 261 | loss: 2.223784 | lr 1.8000e-04 | norm: 0.2356 | dt: 31904.24ms | tok/sec: 32866.4 | mem: 122575MB | Time Left: 00 hrs : 52 mins
|
| 582 |
+
[TIMING] data: 4.2ms | fwd: 7084.1ms | loss: 14.5ms | bwd: 24753.6ms | clip: 0.0ms | optim: 42.0ms
|
| 583 |
+
step 262 | loss: 2.214776 | lr 1.8000e-04 | norm: 0.4503 | dt: 31908.95ms | tok/sec: 32861.5 | mem: 122575MB | Time Left: 00 hrs : 52 mins
|
| 584 |
+
[TIMING] data: 4.3ms | fwd: 7032.3ms | loss: 13.7ms | bwd: 24810.7ms | clip: 0.0ms | optim: 42.1ms
|
| 585 |
+
step 263 | loss: 2.381938 | lr 1.8000e-04 | norm: 0.2042 | dt: 31886.58ms | tok/sec: 32884.6 | mem: 122575MB | Time Left: 00 hrs : 51 mins
|
| 586 |
+
[TIMING] data: 4.2ms | fwd: 7037.0ms | loss: 13.9ms | bwd: 24781.9ms | clip: 0.0ms | optim: 43.8ms
|
| 587 |
+
step 264 | loss: 2.536288 | lr 1.8000e-04 | norm: 0.2232 | dt: 31888.71ms | tok/sec: 32882.4 | mem: 122575MB | Time Left: 00 hrs : 51 mins
|
| 588 |
+
[TIMING] data: 4.2ms | fwd: 7066.0ms | loss: 13.9ms | bwd: 24756.8ms | clip: 0.0ms | optim: 42.0ms
|
| 589 |
+
step 265 | loss: 2.350940 | lr 1.8000e-04 | norm: 0.2927 | dt: 31894.71ms | tok/sec: 32876.2 | mem: 122575MB | Time Left: 00 hrs : 50 mins
|
| 590 |
+
[TIMING] data: 4.2ms | fwd: 7069.1ms | loss: 13.7ms | bwd: 24756.9ms | clip: 0.0ms | optim: 45.0ms
|
| 591 |
+
step 266 | loss: 2.409952 | lr 1.8000e-04 | norm: 0.3326 | dt: 31963.15ms | tok/sec: 32805.8 | mem: 122575MB | Time Left: 00 hrs : 50 mins
|
| 592 |
+
[TIMING] data: 4.2ms | fwd: 7045.2ms | loss: 15.5ms | bwd: 24848.9ms | clip: 0.0ms | optim: 43.1ms
|
| 593 |
+
step 267 | loss: 2.236353 | lr 1.8000e-04 | norm: 0.1814 | dt: 31921.33ms | tok/sec: 32848.8 | mem: 122575MB | Time Left: 00 hrs : 49 mins
|
| 594 |
+
[TIMING] data: 4.2ms | fwd: 7061.3ms | loss: 13.9ms | bwd: 24793.9ms | clip: 0.0ms | optim: 42.2ms
|
| 595 |
+
step 268 | loss: 2.602401 | lr 1.8000e-04 | norm: 0.2298 | dt: 31948.20ms | tok/sec: 32821.1 | mem: 122575MB | Time Left: 00 hrs : 48 mins
|
| 596 |
+
[TIMING] data: 4.3ms | fwd: 7047.0ms | loss: 13.7ms | bwd: 24835.6ms | clip: 0.0ms | optim: 41.8ms
|
| 597 |
+
step 269 | loss: 2.350348 | lr 1.8000e-04 | norm: 0.1684 | dt: 31910.89ms | tok/sec: 32859.5 | mem: 122575MB | Time Left: 00 hrs : 48 mins
|
| 598 |
+
[TIMING] data: 4.3ms | fwd: 7036.5ms | loss: 13.7ms | bwd: 24807.9ms | clip: 0.0ms | optim: 42.7ms
|
| 599 |
+
step 270 | loss: 2.411297 | lr 1.8000e-04 | norm: 0.5200 | dt: 31921.86ms | tok/sec: 32848.2 | mem: 122575MB | Time Left: 00 hrs : 47 mins
|
| 600 |
+
[TIMING] data: 4.2ms | fwd: 7021.7ms | loss: 13.8ms | bwd: 24834.4ms | clip: 0.0ms | optim: 41.9ms
|
| 601 |
+
Current shard: /root/Adhi/BluTrain/dist/Context_Parallelism/data/mix128_b6_60/train_shard_00003.bin
|
| 602 |
+
100000000
|
| 603 |
+
step 271 | loss: 2.691875 | lr 1.8000e-04 | norm: 0.3055 | dt: 31923.66ms | tok/sec: 32846.4 | mem: 122575MB | Time Left: 00 hrs : 47 mins
|
| 604 |
+
[TIMING] data: 13.8ms | fwd: 7047.9ms | loss: 13.8ms | bwd: 24800.1ms | clip: 0.0ms | optim: 42.2ms
|
| 605 |
+
step 272 | loss: 2.279689 | lr 1.8000e-04 | norm: 0.2710 | dt: 31911.93ms | tok/sec: 32858.4 | mem: 122575MB | Time Left: 00 hrs : 46 mins
|
| 606 |
+
[TIMING] data: 4.3ms | fwd: 7023.6ms | loss: 13.7ms | bwd: 24822.5ms | clip: 0.0ms | optim: 42.1ms
|
| 607 |
+
step 273 | loss: 2.493117 | lr 1.8000e-04 | norm: 0.7041 | dt: 31839.46ms | tok/sec: 32933.2 | mem: 122575MB | Time Left: 00 hrs : 46 mins
|
| 608 |
+
[TIMING] data: 4.3ms | fwd: 7094.4ms | loss: 13.9ms | bwd: 24678.8ms | clip: 0.0ms | optim: 42.3ms
|
| 609 |
+
step 274 | loss: 2.278730 | lr 1.8000e-04 | norm: 0.3786 | dt: 31873.42ms | tok/sec: 32898.1 | mem: 122575MB | Time Left: 00 hrs : 45 mins
|
| 610 |
+
[TIMING] data: 4.2ms | fwd: 6975.4ms | loss: 13.7ms | bwd: 24832.3ms | clip: 0.0ms | optim: 41.9ms
|
| 611 |
+
step 275 | loss: 2.882230 | lr 1.8000e-04 | norm: 0.6480 | dt: 31864.99ms | tok/sec: 32906.8 | mem: 122575MB | Time Left: 00 hrs : 45 mins
|
| 612 |
+
[TIMING] data: 4.1ms | fwd: 7079.8ms | loss: 14.2ms | bwd: 24719.4ms | clip: 0.0ms | optim: 41.6ms
|
| 613 |
+
step 276 | loss: 2.145096 | lr 1.8000e-04 | norm: 0.5488 | dt: 31831.91ms | tok/sec: 32941.0 | mem: 122575MB | Time Left: 00 hrs : 44 mins
|
| 614 |
+
[TIMING] data: 4.2ms | fwd: 7021.4ms | loss: 14.4ms | bwd: 24743.0ms | clip: 0.0ms | optim: 43.1ms
|
| 615 |
+
step 277 | loss: 2.098778 | lr 1.8000e-04 | norm: 0.6076 | dt: 31846.06ms | tok/sec: 32926.4 | mem: 122575MB | Time Left: 00 hrs : 44 mins
|
| 616 |
+
[TIMING] data: 4.1ms | fwd: 6986.3ms | loss: 13.6ms | bwd: 24794.8ms | clip: 0.0ms | optim: 41.5ms
|
| 617 |
+
step 278 | loss: 2.062408 | lr 1.8000e-04 | norm: 0.4797 | dt: 31768.02ms | tok/sec: 33007.3 | mem: 122575MB | Time Left: 00 hrs : 43 mins
|
| 618 |
+
[TIMING] data: 4.2ms | fwd: 7016.8ms | loss: 13.5ms | bwd: 24684.7ms | clip: 0.0ms | optim: 43.1ms
|
| 619 |
+
step 279 | loss: 2.401424 | lr 1.8000e-04 | norm: 0.3822 | dt: 31812.36ms | tok/sec: 32961.3 | mem: 122575MB | Time Left: 00 hrs : 42 mins
|
| 620 |
+
[TIMING] data: 4.2ms | fwd: 6976.0ms | loss: 13.8ms | bwd: 24770.2ms | clip: 0.0ms | optim: 42.2ms
|
| 621 |
+
step 280 | loss: 2.405661 | lr 1.8000e-04 | norm: 0.6709 | dt: 31702.79ms | tok/sec: 33075.2 | mem: 122575MB | Time Left: 00 hrs : 42 mins
|
| 622 |
+
[TIMING] data: 4.2ms | fwd: 7026.6ms | loss: 13.9ms | bwd: 24610.3ms | clip: 0.0ms | optim: 42.0ms
|
| 623 |
+
step 281 | loss: 2.263566 | lr 1.8000e-04 | norm: 0.3139 | dt: 31842.13ms | tok/sec: 32930.5 | mem: 122575MB | Time Left: 00 hrs : 41 mins
|
| 624 |
+
[TIMING] data: 4.2ms | fwd: 7019.7ms | loss: 13.7ms | bwd: 24756.9ms | clip: 0.0ms | optim: 41.8ms
|
| 625 |
+
step 282 | loss: 2.954242 | lr 1.8000e-04 | norm: 0.3942 | dt: 31737.45ms | tok/sec: 33039.1 | mem: 122575MB | Time Left: 00 hrs : 41 mins
|
| 626 |
+
[TIMING] data: 4.2ms | fwd: 7040.8ms | loss: 13.8ms | bwd: 24631.0ms | clip: 0.0ms | optim: 41.8ms
|
| 627 |
+
step 283 | loss: 2.215230 | lr 1.8000e-04 | norm: 0.2716 | dt: 31831.23ms | tok/sec: 32941.7 | mem: 122575MB | Time Left: 00 hrs : 40 mins
|
| 628 |
+
[TIMING] data: 4.3ms | fwd: 7019.0ms | loss: 13.9ms | bwd: 24745.2ms | clip: 0.0ms | optim: 43.0ms
|
| 629 |
+
step 284 | loss: 2.405702 | lr 1.8000e-04 | norm: 0.3573 | dt: 31803.30ms | tok/sec: 32970.7 | mem: 122575MB | Time Left: 00 hrs : 40 mins
|
| 630 |
+
[TIMING] data: 4.3ms | fwd: 7025.0ms | loss: 13.7ms | bwd: 24712.5ms | clip: 0.0ms | optim: 42.0ms
|
| 631 |
+
step 285 | loss: 2.246428 | lr 1.8000e-04 | norm: 0.2312 | dt: 31744.37ms | tok/sec: 33031.9 | mem: 122575MB | Time Left: 00 hrs : 39 mins
|
| 632 |
+
[TIMING] data: 4.2ms | fwd: 7052.2ms | loss: 13.9ms | bwd: 24626.3ms | clip: 0.0ms | optim: 42.0ms
|
| 633 |
+
step 286 | loss: 2.134144 | lr 1.8000e-04 | norm: 0.2900 | dt: 31796.36ms | tok/sec: 32977.9 | mem: 122575MB | Time Left: 00 hrs : 39 mins
|
| 634 |
+
[TIMING] data: 4.3ms | fwd: 7009.7ms | loss: 13.5ms | bwd: 24720.4ms | clip: 0.0ms | optim: 42.2ms
|
| 635 |
+
step 287 | loss: 2.206506 | lr 1.8000e-04 | norm: 0.2115 | dt: 31680.81ms | tok/sec: 33098.1 | mem: 122575MB | Time Left: 00 hrs : 38 mins
|
| 636 |
+
[TIMING] data: 4.1ms | fwd: 6978.9ms | loss: 13.7ms | bwd: 24636.3ms | clip: 0.0ms | optim: 41.9ms
|
| 637 |
+
step 288 | loss: 2.558135 | lr 1.8000e-04 | norm: 0.2389 | dt: 31786.69ms | tok/sec: 32987.9 | mem: 122575MB | Time Left: 00 hrs : 38 mins
|
| 638 |
+
[TIMING] data: 4.1ms | fwd: 7004.2ms | loss: 13.8ms | bwd: 24717.3ms | clip: 0.0ms | optim: 41.5ms
|
| 639 |
+
step 289 | loss: 1.236275 | lr 1.8000e-04 | norm: 0.4349 | dt: 31390.74ms | tok/sec: 33404.0 | mem: 122575MB | Time Left: 00 hrs : 37 mins
|
| 640 |
+
[TIMING] data: 4.2ms | fwd: 6875.7ms | loss: 13.6ms | bwd: 24449.0ms | clip: 0.0ms | optim: 41.5ms
|
| 641 |
+
step 290 | loss: 2.659274 | lr 1.8000e-04 | norm: 0.2431 | dt: 31594.85ms | tok/sec: 33188.2 | mem: 122575MB | Time Left: 00 hrs : 36 mins
|
| 642 |
+
[TIMING] data: 4.5ms | fwd: 6998.1ms | loss: 13.7ms | bwd: 24531.0ms | clip: 0.0ms | optim: 41.8ms
|
| 643 |
+
step 291 | loss: 2.731871 | lr 1.8000e-04 | norm: 0.3271 | dt: 31825.17ms | tok/sec: 32948.0 | mem: 122575MB | Time Left: 00 hrs : 36 mins
|
| 644 |
+
[TIMING] data: 4.2ms | fwd: 7014.4ms | loss: 13.6ms | bwd: 24745.1ms | clip: 0.0ms | optim: 42.0ms
|
| 645 |
+
step 292 | loss: 2.733432 | lr 1.8000e-04 | norm: 0.2703 | dt: 31598.35ms | tok/sec: 33184.5 | mem: 122575MB | Time Left: 00 hrs : 35 mins
|
| 646 |
+
[TIMING] data: 4.3ms | fwd: 6956.8ms | loss: 13.6ms | bwd: 24575.0ms | clip: 0.0ms | optim: 42.1ms
|
| 647 |
+
step 293 | loss: 2.440004 | lr 1.8000e-04 | norm: 0.2410 | dt: 31492.06ms | tok/sec: 33296.5 | mem: 122575MB | Time Left: 00 hrs : 35 mins
|
| 648 |
+
[TIMING] data: 4.2ms | fwd: 7016.5ms | loss: 14.4ms | bwd: 24409.7ms | clip: 0.0ms | optim: 41.4ms
|
| 649 |
+
step 294 | loss: 2.843191 | lr 1.8000e-04 | norm: 0.2560 | dt: 31678.62ms | tok/sec: 33100.4 | mem: 122575MB | Time Left: 00 hrs : 34 mins
|
| 650 |
+
[TIMING] data: 4.2ms | fwd: 7010.1ms | loss: 13.7ms | bwd: 24603.3ms | clip: 0.0ms | optim: 41.5ms
|
| 651 |
+
step 295 | loss: 3.019926 | lr 1.8000e-04 | norm: 0.4062 | dt: 31580.19ms | tok/sec: 33203.6 | mem: 122575MB | Time Left: 00 hrs : 34 mins
|
| 652 |
+
[TIMING] data: 4.2ms | fwd: 6972.1ms | loss: 13.5ms | bwd: 24543.1ms | clip: 0.0ms | optim: 41.4ms
|
| 653 |
+
step 296 | loss: 2.402269 | lr 1.8000e-04 | norm: 0.5774 | dt: 31547.95ms | tok/sec: 33237.5 | mem: 122575MB | Time Left: 00 hrs : 33 mins
|
| 654 |
+
[TIMING] data: 4.3ms | fwd: 6903.7ms | loss: 13.5ms | bwd: 24578.6ms | clip: 0.0ms | optim: 41.9ms
|
| 655 |
+
step 297 | loss: 2.117500 | lr 1.8000e-04 | norm: 0.5843 | dt: 31585.38ms | tok/sec: 33198.1 | mem: 122575MB | Time Left: 00 hrs : 33 mins
|
| 656 |
+
[TIMING] data: 4.2ms | fwd: 6974.9ms | loss: 13.7ms | bwd: 24545.4ms | clip: 0.0ms | optim: 41.4ms
|
| 657 |
+
step 298 | loss: 1.698257 | lr 1.8000e-04 | norm: 0.2788 | dt: 31435.77ms | tok/sec: 33356.1 | mem: 122575MB | Time Left: 00 hrs : 32 mins
|
| 658 |
+
[TIMING] data: 4.2ms | fwd: 6924.7ms | loss: 13.7ms | bwd: 24445.0ms | clip: 0.0ms | optim: 42.3ms
|
| 659 |
+
step 299 | loss: 2.310858 | lr 1.8000e-04 | norm: 0.4329 | dt: 31568.52ms | tok/sec: 33215.9 | mem: 122575MB | Time Left: 00 hrs : 32 mins
|
| 660 |
+
[TIMING] data: 4.4ms | fwd: 6979.2ms | loss: 13.7ms | bwd: 24523.4ms | clip: 0.0ms | optim: 41.9ms
|
| 661 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_300.ckpt (safely)
|
| 662 |
+
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_100.ckpt
|
| 663 |
+
step 300 | loss: 2.306474 | lr 1.8000e-04 | norm: 0.7448 | dt: 30868.81ms | tok/sec: 33968.8 | mem: 122575MB | Time Left: 00 hrs : 30 mins
|
| 664 |
+
[TIMING] data: 4.3ms | fwd: 6767.5ms | loss: 13.4ms | bwd: 24034.9ms | clip: 0.0ms | optim: 42.7ms
|
| 665 |
+
step 301 | loss: 2.485784 | lr 1.7992e-04 | norm: 0.3822 | dt: 31415.63ms | tok/sec: 33377.5 | mem: 122575MB | Time Left: 00 hrs : 30 mins
|
| 666 |
+
[TIMING] data: 4.4ms | fwd: 6945.7ms | loss: 13.6ms | bwd: 24404.0ms | clip: 0.0ms | optim: 42.0ms
|
| 667 |
+
step 302 | loss: 2.560497 | lr 1.7967e-04 | norm: 0.1931 | dt: 31560.32ms | tok/sec: 33224.5 | mem: 122575MB | Time Left: 00 hrs : 30 mins
|
| 668 |
+
[TIMING] data: 4.3ms | fwd: 6973.6ms | loss: 13.4ms | bwd: 24521.4ms | clip: 0.0ms | optim: 41.7ms
|
| 669 |
+
step 303 | loss: 2.756249 | lr 1.7926e-04 | norm: 0.6558 | dt: 31720.32ms | tok/sec: 33056.9 | mem: 122575MB | Time Left: 00 hrs : 30 mins
|
| 670 |
+
[TIMING] data: 4.3ms | fwd: 7037.2ms | loss: 13.7ms | bwd: 24617.1ms | clip: 0.0ms | optim: 42.2ms
|
| 671 |
+
step 304 | loss: 1.640434 | lr 1.7869e-04 | norm: 1.0092 | dt: 31505.11ms | tok/sec: 33282.7 | mem: 122575MB | Time Left: 00 hrs : 29 mins
|
| 672 |
+
[TIMING] data: 4.2ms | fwd: 6935.5ms | loss: 13.5ms | bwd: 24504.2ms | clip: 0.0ms | optim: 41.9ms
|
| 673 |
+
step 305 | loss: 2.503383 | lr 1.7796e-04 | norm: 0.7323 | dt: 31791.17ms | tok/sec: 32983.2 | mem: 122575MB | Time Left: 00 hrs : 29 mins
|
| 674 |
+
[TIMING] data: 4.2ms | fwd: 7028.2ms | loss: 13.6ms | bwd: 24697.9ms | clip: 0.0ms | optim: 41.5ms
|
| 675 |
+
step 306 | loss: 2.249824 | lr 1.7706e-04 | norm: 0.7626 | dt: 31725.31ms | tok/sec: 33051.7 | mem: 122575MB | Time Left: 00 hrs : 28 mins
|
| 676 |
+
[TIMING] data: 4.2ms | fwd: 7035.6ms | loss: 13.7ms | bwd: 24623.9ms | clip: 0.0ms | optim: 42.2ms
|
| 677 |
+
step 307 | loss: 2.408019 | lr 1.7601e-04 | norm: 0.3378 | dt: 31618.70ms | tok/sec: 33163.2 | mem: 122575MB | Time Left: 00 hrs : 27 mins
|
| 678 |
+
[TIMING] data: 4.2ms | fwd: 6985.8ms | loss: 13.6ms | bwd: 24567.2ms | clip: 0.0ms | optim: 42.1ms
|
| 679 |
+
step 308 | loss: 2.427085 | lr 1.7481e-04 | norm: 0.3376 | dt: 31710.50ms | tok/sec: 33067.2 | mem: 122575MB | Time Left: 00 hrs : 27 mins
|
| 680 |
+
[TIMING] data: 4.2ms | fwd: 6972.3ms | loss: 13.8ms | bwd: 24672.5ms | clip: 0.0ms | optim: 42.0ms
|
| 681 |
+
step 309 | loss: 2.376311 | lr 1.7346e-04 | norm: 1.0688 | dt: 31499.67ms | tok/sec: 33288.5 | mem: 122575MB | Time Left: 00 hrs : 26 mins
|
| 682 |
+
[TIMING] data: 4.2ms | fwd: 6946.5ms | loss: 13.6ms | bwd: 24488.1ms | clip: 0.0ms | optim: 41.3ms
|
| 683 |
+
step 310 | loss: 2.387875 | lr 1.7196e-04 | norm: 0.4188 | dt: 31642.10ms | tok/sec: 33138.6 | mem: 122575MB | Time Left: 00 hrs : 26 mins
|
| 684 |
+
[TIMING] data: 4.2ms | fwd: 7002.2ms | loss: 13.8ms | bwd: 24574.7ms | clip: 0.0ms | optim: 41.4ms
|
| 685 |
+
step 311 | loss: 2.158477 | lr 1.7032e-04 | norm: 1.2179 | dt: 31681.01ms | tok/sec: 33097.9 | mem: 122575MB | Time Left: 00 hrs : 25 mins
|
| 686 |
+
[TIMING] data: 4.2ms | fwd: 6989.3ms | loss: 14.5ms | bwd: 24625.7ms | clip: 0.0ms | optim: 41.5ms
|
| 687 |
+
step 312 | loss: 2.014573 | lr 1.6854e-04 | norm: 0.7797 | dt: 31744.46ms | tok/sec: 33031.8 | mem: 122575MB | Time Left: 00 hrs : 25 mins
|
| 688 |
+
[TIMING] data: 4.2ms | fwd: 6948.4ms | loss: 13.6ms | bwd: 24730.7ms | clip: 0.0ms | optim: 41.8ms
|
| 689 |
+
step 313 | loss: 2.394964 | lr 1.6663e-04 | norm: 0.2377 | dt: 31658.31ms | tok/sec: 33121.7 | mem: 122575MB | Time Left: 00 hrs : 24 mins
|
| 690 |
+
[TIMING] data: 4.2ms | fwd: 7012.7ms | loss: 13.8ms | bwd: 24580.2ms | clip: 0.0ms | optim: 41.6ms
|
| 691 |
+
step 314 | loss: 1.836500 | lr 1.6459e-04 | norm: 0.8826 | dt: 31586.13ms | tok/sec: 33197.4 | mem: 122575MB | Time Left: 00 hrs : 24 mins
|
| 692 |
+
[TIMING] data: 4.3ms | fwd: 6980.6ms | loss: 13.6ms | bwd: 24540.3ms | clip: 0.0ms | optim: 41.5ms
|
| 693 |
+
step 315 | loss: 2.640098 | lr 1.6243e-04 | norm: 0.2585 | dt: 31771.48ms | tok/sec: 33003.7 | mem: 122575MB | Time Left: 00 hrs : 23 mins
|
| 694 |
+
[TIMING] data: 4.3ms | fwd: 7051.5ms | loss: 13.7ms | bwd: 24654.0ms | clip: 0.0ms | optim: 42.1ms
|
| 695 |
+
step 316 | loss: 2.097623 | lr 1.6015e-04 | norm: 0.3965 | dt: 31603.36ms | tok/sec: 33179.3 | mem: 122575MB | Time Left: 00 hrs : 23 mins
|
| 696 |
+
[TIMING] data: 4.2ms | fwd: 6967.6ms | loss: 14.2ms | bwd: 24569.7ms | clip: 0.0ms | optim: 41.7ms
|
| 697 |
+
step 317 | loss: 2.317846 | lr 1.5776e-04 | norm: 0.5725 | dt: 31646.70ms | tok/sec: 33133.8 | mem: 122575MB | Time Left: 00 hrs : 22 mins
|
| 698 |
+
[TIMING] data: 4.2ms | fwd: 7016.9ms | loss: 13.7ms | bwd: 24564.4ms | clip: 0.0ms | optim: 41.7ms
|
| 699 |
+
step 318 | loss: 2.388659 | lr 1.5527e-04 | norm: 1.1120 | dt: 31658.33ms | tok/sec: 33121.6 | mem: 122575MB | Time Left: 00 hrs : 22 mins
|
| 700 |
+
[TIMING] data: 4.2ms | fwd: 6989.6ms | loss: 13.6ms | bwd: 24602.9ms | clip: 0.0ms | optim: 42.2ms
|
| 701 |
+
step 319 | loss: 2.425697 | lr 1.5268e-04 | norm: 0.3421 | dt: 31623.10ms | tok/sec: 33158.5 | mem: 122575MB | Time Left: 00 hrs : 21 mins
|
| 702 |
+
[TIMING] data: 4.2ms | fwd: 6966.6ms | loss: 13.8ms | bwd: 24590.8ms | clip: 0.0ms | optim: 41.9ms
|
| 703 |
+
step 320 | loss: 1.790040 | lr 1.5000e-04 | norm: 0.3734 | dt: 31521.55ms | tok/sec: 33265.4 | mem: 122575MB | Time Left: 00 hrs : 21 mins
|
| 704 |
+
[TIMING] data: 4.2ms | fwd: 6896.6ms | loss: 13.6ms | bwd: 24560.1ms | clip: 0.0ms | optim: 41.4ms
|
| 705 |
+
step 321 | loss: 1.872453 | lr 1.4724e-04 | norm: 0.4648 | dt: 31675.60ms | tok/sec: 33103.6 | mem: 122575MB | Time Left: 00 hrs : 20 mins
|
| 706 |
+
[TIMING] data: 4.2ms | fwd: 6931.1ms | loss: 13.7ms | bwd: 24678.6ms | clip: 0.0ms | optim: 42.1ms
|
| 707 |
+
step 322 | loss: 2.820750 | lr 1.4440e-04 | norm: 0.4440 | dt: 31696.09ms | tok/sec: 33082.2 | mem: 122575MB | Time Left: 00 hrs : 20 mins
|
| 708 |
+
[TIMING] data: 4.2ms | fwd: 6960.4ms | loss: 13.7ms | bwd: 24670.2ms | clip: 0.0ms | optim: 41.7ms
|
| 709 |
+
step 323 | loss: 2.377633 | lr 1.4150e-04 | norm: 0.7827 | dt: 31701.20ms | tok/sec: 33076.9 | mem: 122575MB | Time Left: 00 hrs : 19 mins
|
| 710 |
+
[TIMING] data: 4.2ms | fwd: 7003.2ms | loss: 14.1ms | bwd: 24632.1ms | clip: 0.0ms | optim: 41.8ms
|
| 711 |
+
step 324 | loss: 2.898071 | lr 1.3854e-04 | norm: 0.2991 | dt: 31857.05ms | tok/sec: 32915.0 | mem: 122575MB | Time Left: 00 hrs : 19 mins
|
| 712 |
+
[TIMING] data: 4.2ms | fwd: 7059.5ms | loss: 13.7ms | bwd: 24731.2ms | clip: 0.0ms | optim: 42.7ms
|
| 713 |
+
step 325 | loss: 1.859557 | lr 1.3553e-04 | norm: 0.4383 | dt: 31472.97ms | tok/sec: 33316.7 | mem: 122575MB | Time Left: 00 hrs : 18 mins
|
| 714 |
+
[TIMING] data: 4.2ms | fwd: 6981.1ms | loss: 13.6ms | bwd: 24427.0ms | clip: 0.0ms | optim: 41.3ms
|
| 715 |
+
step 326 | loss: 2.294454 | lr 1.3247e-04 | norm: 0.2779 | dt: 31728.01ms | tok/sec: 33048.9 | mem: 122575MB | Time Left: 00 hrs : 17 mins
|
| 716 |
+
[TIMING] data: 4.3ms | fwd: 6975.4ms | loss: 13.7ms | bwd: 24686.3ms | clip: 0.0ms | optim: 42.3ms
|
| 717 |
+
step 327 | loss: 2.628585 | lr 1.2939e-04 | norm: 0.5175 | dt: 31670.70ms | tok/sec: 33108.7 | mem: 122575MB | Time Left: 00 hrs : 17 mins
|
| 718 |
+
[TIMING] data: 4.2ms | fwd: 7022.4ms | loss: 13.6ms | bwd: 24583.1ms | clip: 0.0ms | optim: 41.4ms
|
| 719 |
+
step 328 | loss: 2.422482 | lr 1.2627e-04 | norm: 0.4577 | dt: 31716.85ms | tok/sec: 33060.5 | mem: 122575MB | Time Left: 00 hrs : 16 mins
|
| 720 |
+
[TIMING] data: 4.2ms | fwd: 6991.0ms | loss: 13.7ms | bwd: 24660.2ms | clip: 0.0ms | optim: 41.9ms
|
| 721 |
+
step 329 | loss: 2.515633 | lr 1.2314e-04 | norm: 0.2821 | dt: 31754.03ms | tok/sec: 33021.8 | mem: 122575MB | Time Left: 00 hrs : 16 mins
|
| 722 |
+
[TIMING] data: 4.2ms | fwd: 7039.6ms | loss: 13.8ms | bwd: 24649.3ms | clip: 0.0ms | optim: 41.5ms
|
| 723 |
+
step 330 | loss: 2.271731 | lr 1.2000e-04 | norm: 0.2711 | dt: 31697.34ms | tok/sec: 33080.9 | mem: 122575MB | Time Left: 00 hrs : 15 mins
|
| 724 |
+
[TIMING] data: 4.2ms | fwd: 6966.0ms | loss: 14.3ms | bwd: 24665.3ms | clip: 0.0ms | optim: 41.4ms
|
| 725 |
+
step 331 | loss: 2.473583 | lr 1.1686e-04 | norm: 0.2366 | dt: 31806.92ms | tok/sec: 32966.9 | mem: 122575MB | Time Left: 00 hrs : 15 mins
|
| 726 |
+
[TIMING] data: 4.3ms | fwd: 7036.7ms | loss: 13.8ms | bwd: 24704.6ms | clip: 0.0ms | optim: 41.7ms
|
| 727 |
+
step 332 | loss: 2.536157 | lr 1.1373e-04 | norm: 0.5113 | dt: 31608.89ms | tok/sec: 33173.5 | mem: 122575MB | Time Left: 00 hrs : 14 mins
|
| 728 |
+
[TIMING] data: 4.2ms | fwd: 6968.5ms | loss: 13.7ms | bwd: 24575.1ms | clip: 0.0ms | optim: 41.7ms
|
| 729 |
+
step 333 | loss: 2.304292 | lr 1.1061e-04 | norm: 0.3399 | dt: 31784.70ms | tok/sec: 32990.0 | mem: 122575MB | Time Left: 00 hrs : 14 mins
|
| 730 |
+
[TIMING] data: 4.2ms | fwd: 6990.2ms | loss: 14.2ms | bwd: 24728.6ms | clip: 0.0ms | optim: 41.6ms
|
| 731 |
+
step 334 | loss: 2.954391 | lr 1.0753e-04 | norm: 0.2128 | dt: 31765.15ms | tok/sec: 33010.3 | mem: 122575MB | Time Left: 00 hrs : 13 mins
|
| 732 |
+
[TIMING] data: 4.2ms | fwd: 7008.2ms | loss: 13.7ms | bwd: 24691.1ms | clip: 0.0ms | optim: 42.0ms
|
| 733 |
+
step 335 | loss: 1.891136 | lr 1.0447e-04 | norm: 0.3754 | dt: 31674.86ms | tok/sec: 33104.4 | mem: 122575MB | Time Left: 00 hrs : 13 mins
|
| 734 |
+
[TIMING] data: 4.2ms | fwd: 6964.7ms | loss: 14.2ms | bwd: 24644.2ms | clip: 0.0ms | optim: 41.6ms
|
| 735 |
+
step 336 | loss: 2.595659 | lr 1.0146e-04 | norm: 0.3124 | dt: 31774.45ms | tok/sec: 33000.6 | mem: 122575MB | Time Left: 00 hrs : 12 mins
|
| 736 |
+
[TIMING] data: 4.3ms | fwd: 7037.3ms | loss: 13.7ms | bwd: 24671.2ms | clip: 0.0ms | optim: 42.1ms
|
| 737 |
+
step 337 | loss: 1.293156 | lr 9.8498e-05 | norm: 0.8614 | dt: 31298.47ms | tok/sec: 33502.5 | mem: 122575MB | Time Left: 00 hrs : 11 mins
|
| 738 |
+
[TIMING] data: 4.2ms | fwd: 6900.3ms | loss: 13.6ms | bwd: 24333.2ms | clip: 0.0ms | optim: 41.3ms
|
| 739 |
+
step 338 | loss: 2.444477 | lr 9.5596e-05 | norm: 0.2727 | dt: 31893.63ms | tok/sec: 32877.3 | mem: 122575MB | Time Left: 00 hrs : 11 mins
|
| 740 |
+
[TIMING] data: 4.2ms | fwd: 7046.2ms | loss: 14.4ms | bwd: 24781.3ms | clip: 0.0ms | optim: 41.7ms
|
| 741 |
+
step 339 | loss: 3.150173 | lr 9.2761e-05 | norm: 0.5081 | dt: 31721.60ms | tok/sec: 33055.6 | mem: 122575MB | Time Left: 00 hrs : 11 mins
|
| 742 |
+
[TIMING] data: 4.3ms | fwd: 7015.3ms | loss: 13.8ms | bwd: 24640.4ms | clip: 0.0ms | optim: 41.9ms
|
| 743 |
+
step 340 | loss: 2.463523 | lr 9.0000e-05 | norm: 0.2055 | dt: 31756.39ms | tok/sec: 33019.4 | mem: 122575MB | Time Left: 00 hrs : 10 mins
|
| 744 |
+
[TIMING] data: 4.2ms | fwd: 7006.8ms | loss: 13.6ms | bwd: 24684.2ms | clip: 0.0ms | optim: 41.6ms
|
| 745 |
+
step 341 | loss: 2.623348 | lr 8.7322e-05 | norm: 0.2578 | dt: 31824.21ms | tok/sec: 32949.0 | mem: 122575MB | Time Left: 00 hrs : 10 mins
|
| 746 |
+
[TIMING] data: 4.2ms | fwd: 7025.9ms | loss: 13.7ms | bwd: 24732.7ms | clip: 0.0ms | optim: 41.8ms
|
| 747 |
+
step 342 | loss: 2.403027 | lr 8.4733e-05 | norm: 0.3789 | dt: 31749.19ms | tok/sec: 33026.9 | mem: 122575MB | Time Left: 00 hrs : 09 mins
|
| 748 |
+
[TIMING] data: 4.2ms | fwd: 7023.4ms | loss: 13.8ms | bwd: 24660.1ms | clip: 0.0ms | optim: 41.9ms
|
| 749 |
+
step 343 | loss: 2.815173 | lr 8.2241e-05 | norm: 0.4755 | dt: 31770.52ms | tok/sec: 33004.7 | mem: 122575MB | Time Left: 00 hrs : 09 mins
|
| 750 |
+
[TIMING] data: 4.2ms | fwd: 6978.0ms | loss: 13.8ms | bwd: 24726.6ms | clip: 0.0ms | optim: 42.1ms
|
| 751 |
+
step 344 | loss: 2.229372 | lr 7.9852e-05 | norm: 0.3099 | dt: 31593.75ms | tok/sec: 33189.3 | mem: 122575MB | Time Left: 00 hrs : 08 mins
|
| 752 |
+
[TIMING] data: 4.2ms | fwd: 6957.8ms | loss: 13.6ms | bwd: 24570.8ms | clip: 0.0ms | optim: 41.5ms
|
| 753 |
+
step 345 | loss: 2.314371 | lr 7.7574e-05 | norm: 0.2740 | dt: 31723.14ms | tok/sec: 33054.0 | mem: 122575MB | Time Left: 00 hrs : 07 mins
|
| 754 |
+
[TIMING] data: 4.2ms | fwd: 7044.7ms | loss: 13.8ms | bwd: 24612.2ms | clip: 0.0ms | optim: 42.3ms
|
| 755 |
+
step 346 | loss: 2.423322 | lr 7.5411e-05 | norm: 0.2695 | dt: 31796.22ms | tok/sec: 32978.0 | mem: 122575MB | Time Left: 00 hrs : 07 mins
|
| 756 |
+
[TIMING] data: 4.3ms | fwd: 7062.1ms | loss: 14.3ms | bwd: 24667.9ms | clip: 0.0ms | optim: 41.8ms
|
| 757 |
+
step 347 | loss: 2.358418 | lr 7.3371e-05 | norm: 0.5674 | dt: 31780.97ms | tok/sec: 32993.8 | mem: 122575MB | Time Left: 00 hrs : 06 mins
|
| 758 |
+
[TIMING] data: 4.3ms | fwd: 6955.3ms | loss: 13.7ms | bwd: 24759.8ms | clip: 0.0ms | optim: 41.9ms
|
| 759 |
+
step 348 | loss: 2.631786 | lr 7.1459e-05 | norm: 0.2339 | dt: 31780.66ms | tok/sec: 32994.2 | mem: 122575MB | Time Left: 00 hrs : 06 mins
|
| 760 |
+
[TIMING] data: 4.3ms | fwd: 6975.0ms | loss: 13.5ms | bwd: 24740.2ms | clip: 0.0ms | optim: 41.7ms
|
| 761 |
+
step 349 | loss: 2.418557 | lr 6.9680e-05 | norm: 0.1693 | dt: 31743.35ms | tok/sec: 33032.9 | mem: 122575MB | Time Left: 00 hrs : 05 mins
|
| 762 |
+
[TIMING] data: 4.3ms | fwd: 7007.2ms | loss: 14.3ms | bwd: 24669.1ms | clip: 0.0ms | optim: 42.6ms
|
| 763 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_350.ckpt (safely)
|
| 764 |
+
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_150.ckpt
|
| 765 |
+
step 350 | loss: 2.244992 | lr 6.8038e-05 | norm: 0.1818 | dt: 30817.22ms | tok/sec: 34025.7 | mem: 122575MB | Time Left: 00 hrs : 05 mins
|
| 766 |
+
[TIMING] data: 4.4ms | fwd: 6755.8ms | loss: 13.4ms | bwd: 23995.6ms | clip: 0.0ms | optim: 41.9ms
|
| 767 |
+
step 351 | loss: 2.246849 | lr 6.6540e-05 | norm: 0.2983 | dt: 31459.53ms | tok/sec: 33331.0 | mem: 122575MB | Time Left: 00 hrs : 04 mins
|
| 768 |
+
[TIMING] data: 4.2ms | fwd: 6932.0ms | loss: 13.5ms | bwd: 24462.4ms | clip: 0.0ms | optim: 41.6ms
|
| 769 |
+
step 352 | loss: 2.482920 | lr 6.5187e-05 | norm: 0.4574 | dt: 31622.35ms | tok/sec: 33159.3 | mem: 122575MB | Time Left: 00 hrs : 04 mins
|
| 770 |
+
[TIMING] data: 4.2ms | fwd: 6962.6ms | loss: 13.7ms | bwd: 24594.4ms | clip: 0.0ms | optim: 41.6ms
|
| 771 |
+
step 353 | loss: 2.837931 | lr 6.3985e-05 | norm: 1.1483 | dt: 31720.36ms | tok/sec: 33056.9 | mem: 122575MB | Time Left: 00 hrs : 03 mins
|
| 772 |
+
[TIMING] data: 4.2ms | fwd: 7034.6ms | loss: 13.6ms | bwd: 24620.2ms | clip: 0.0ms | optim: 41.8ms
|
| 773 |
+
step 354 | loss: 3.397220 | lr 6.2937e-05 | norm: 2.0071 | dt: 31694.86ms | tok/sec: 33083.5 | mem: 122575MB | Time Left: 00 hrs : 03 mins
|
| 774 |
+
[TIMING] data: 4.3ms | fwd: 7054.8ms | loss: 13.6ms | bwd: 24574.1ms | clip: 0.0ms | optim: 42.0ms
|
| 775 |
+
step 355 | loss: 2.382904 | lr 6.2044e-05 | norm: 0.2679 | dt: 31666.94ms | tok/sec: 33112.6 | mem: 122575MB | Time Left: 00 hrs : 02 mins
|
| 776 |
+
[TIMING] data: 4.2ms | fwd: 7043.3ms | loss: 13.9ms | bwd: 24557.9ms | clip: 0.0ms | optim: 41.7ms
|
| 777 |
+
step 356 | loss: 2.078806 | lr 6.1311e-05 | norm: 0.7705 | dt: 31706.34ms | tok/sec: 33071.5 | mem: 122575MB | Time Left: 00 hrs : 02 mins
|
| 778 |
+
[TIMING] data: 4.2ms | fwd: 7002.0ms | loss: 13.7ms | bwd: 24638.7ms | clip: 0.0ms | optim: 42.0ms
|
| 779 |
+
step 357 | loss: 1.863180 | lr 6.0739e-05 | norm: 0.8322 | dt: 31826.81ms | tok/sec: 32946.3 | mem: 122575MB | Time Left: 00 hrs : 01 mins
|
| 780 |
+
[TIMING] data: 4.2ms | fwd: 7041.4ms | loss: 13.7ms | bwd: 24719.9ms | clip: 0.0ms | optim: 41.8ms
|
| 781 |
+
step 358 | loss: 1.800387 | lr 6.0329e-05 | norm: 0.6880 | dt: 31794.47ms | tok/sec: 32979.8 | mem: 122575MB | Time Left: 00 hrs : 01 mins
|
| 782 |
+
[TIMING] data: 4.2ms | fwd: 6984.8ms | loss: 13.7ms | bwd: 24743.6ms | clip: 0.0ms | optim: 42.4ms
|
| 783 |
+
validation loss: 2.6034
|
| 784 |
+
step 359 | loss: 1.827800 | lr 6.0082e-05 | norm: 0.6922 | dt: 48958.20ms | tok/sec: 21417.8 | mem: 122575MB | Time Left: 00 hrs : 00 mins
|
| 785 |
+
[TIMING] data: 4.3ms | fwd: 7058.8ms | loss: 13.7ms | bwd: 24543.1ms | clip: 0.0ms | optim: 41.9ms
|
| 786 |
+
[CheckpointManager] Saved: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_360.ckpt (safely)
|
| 787 |
+
[CheckpointManager] Deleted old checkpoint: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_200.ckpt
|
| 788 |
+
|
| 789 |
+
=== Training complete ===
|