File size: 4,273 Bytes
cb0d955 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | === Llama Context Parallel Training Script (bluscriptCP) ===
[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
================ Configuration (bluscriptCP) ================
d_model/layers : 1536 / 36
q/kv heads : 12 / 4 (G=3, hd=128)
ffn_hidden : 4096 (SwiGLU)
context_length : 65536
vocab_size : 50304 tying=0
B x T : 1 x 65536
world_size : 4 (dp=1 x cp=4)
CP_ATTN_MODE : ring (load_balance=HeadTail)
global_batch : 65536 (grad_accum=1)
precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD]
max_lr/min_lr : 0.0006 / 6e-05
warmup/steps : 100 / 3 (cosine, no stable phase)
checkpointing : false (ckpt_freq=50, dir=checkpoints_bluscriptcp)
=============================================================
Initializing model on CUDA device 0...
[YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693
[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/b4epoch32k_step2227.ckpt (base step 2227, loss 2.75788) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
Parameters: 1060624896
Parameters per GPU: 1060624896
max_steps: 3
warmup_steps: 100
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=983236608 padded=983236608 W=4 shard=245809152 | 38 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 2813.06 MB (vs 11252.2 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/-
[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
[ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.41 MB (vs 885.639 MB unsharded)
[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
[ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288]
[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
[precision] bf16 params=217, all with fp32 master weights
found 5 shards for split train
found 1 shards for split val
Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log1.csv (run 1)
[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
Checkpointing: OFF
[re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
Starting training...
[CP ring P2P] OVERLAP: dedicated non-blocking stream
validation loss: 2.9859
[CP bwd] overlap=ON
step 0 | loss: 2.990903 | lr 1.8000e-06 | norm: 0.4635 | dt: 16298.63ms | tok/sec: 4021.0 | mem: 100749MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 0.3ms | fwd: 3633.1ms | loss: 0.9ms | bwd: 1733.1ms | clip: 0.0ms | optim: 22.6ms
step 1 | loss: 2.960938 | lr 3.6000e-06 | norm: 0.3655 | dt: 5177.28ms | tok/sec: 12658.4 | mem: 108493MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 0.1ms | fwd: 3553.8ms | loss: 0.8ms | bwd: 1597.7ms | clip: 0.0ms | optim: 22.3ms
validation loss: 2.9859
step 2 | loss: 3.382817 | lr 5.4000e-06 | norm: 0.3972 | dt: 13324.95ms | tok/sec: 4918.3 | mem: 108589MB | Time Left: 00 hrs : 00 mins
[TIMING] data: 0.1ms | fwd: 3523.0ms | loss: 0.8ms | bwd: 1606.4ms | clip: 0.0ms | optim: 22.3ms
=== Training complete ===
|