| === Llama Context Parallel Training Script (bluscriptCP) === |
| [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) |
| [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) |
| [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) |
| ================ Configuration (bluscriptCP) ================ |
| d_model/layers : 1536 / 36 |
| q/kv heads : 12 / 4 (G=3, hd=128) |
| ffn_hidden : 4096 (SwiGLU) |
| context_length : 65536 |
| vocab_size : 50304 tying=0 |
| B x T : 1 x 65536 |
| world_size : 8 (dp=1 x cp=8) |
| CP_ATTN_MODE : ring (load_balance=HeadTail) |
| global_batch : 65536 (grad_accum=1) |
| precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] |
| max_lr/min_lr : 0.0006 / 6e-05 |
| warmup/steps : 100 / 3 (cosine, no stable phase) |
| checkpointing : false (ckpt_freq=50, dir=checkpoints_bluscriptcp) |
| ============================================================= |
| Initializing model on CUDA device 0... |
| [YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693 |
| [InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/blumodelcp_run3_step_695.ckpt (base step 695, loss 2.93407) -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP |
| Parameters: 1060624896 |
| Parameters per GPU: 1060624896 |
| max_steps: 3 |
| warmup_steps: 100 |
| [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt |
| [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- |
| [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) |
| [ZeRO-1 v2] params=983236608 padded=983236608 W=8 shard=122904576 | 38 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 1406.53 MB (vs 11252.2 MB unsharded) |
| [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. |
| [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt |
| [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- |
| [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) |
| [ZeRO-1 v2] params=77388288 padded=77388288 W=8 shard=9673536 | 6 buckets of 50 MB (1638400 elems/rank/bucket) | optimizer state/GPU: 110.705 MB (vs 885.639 MB unsharded) |
| [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. |
| [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] |
| [ZeRO-1] DataParallel is DISABLED (both would double-reduce) |
| [precision] bf16 params=217, all with fp32 master weights |
| found 9 shards for split train |
| found 1 shards for split val |
| Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log3.csv (run 3) |
| [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base) |
| Checkpointing: OFF |
| [re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp) |
|
|
| Starting training... |
|
|
| [CP ring P2P] OVERLAP: dedicated non-blocking stream |
| validation loss: 2.9599 |
| [CP bwd] overlap=ON |
| step 0 | loss: 2.886825 | lr 1.8000e-06 | norm: 0.4577 | dt: 23387.07ms | tok/sec: 2802.2 | mem: 64931MB | Time Left: 00 hrs : 01 mins |
| [TIMING] data: 3.4ms | fwd: 7126.3ms | loss: 0.7ms | bwd: 2194.5ms | clip: 0.0ms | optim: 24.0ms |
| step 1 | loss: 2.855917 | lr 3.6000e-06 | norm: 0.3551 | dt: 7685.76ms | tok/sec: 8526.9 | mem: 72099MB | Time Left: 00 hrs : 00 mins |
| [TIMING] data: 0.0ms | fwd: 5849.4ms | loss: 0.6ms | bwd: 1808.6ms | clip: 0.0ms | optim: 23.6ms |
| validation loss: 2.9598 |
| step 2 | loss: 3.259206 | lr 5.4000e-06 | norm: 0.4406 | dt: 13339.35ms | tok/sec: 4913.0 | mem: 71843MB | Time Left: 00 hrs : 00 mins |
| [TIMING] data: 0.4ms | fwd: 5300.1ms | loss: 0.6ms | bwd: 1088.3ms | clip: 0.0ms | optim: 23.6ms |
|
|
| === Training complete === |
|
|
