| === Llama Context Parallel Training Script (bluscriptCP) === |
| [CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels) |
| [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) |
| [watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable) |
| ================ Configuration (bluscriptCP) ================ |
| d_model/layers : 1536 / 36 |
| q/kv heads : 12 / 4 (G=3, hd=128) |
| ffn_hidden : 4096 (SwiGLU) |
| context_length : 32768 |
| vocab_size : 50304 tying=0 |
| B x T : 1 x 32768 |
| world_size : 2 (dp=1 x cp=2) |
| CP_ATTN_MODE : ring (load_balance=HeadTail) |
| global_batch : 524288 (grad_accum=16) |
| precision : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW) [mask=15 QKV WO MLP HEAD] |
| max_lr/min_lr : 0.0006 / 6e-05 |
| warmup/steps : 100 / 2227 (WSD: stable=1177, decay tail=950) |
| checkpointing : true (ckpt_freq=50, dir=checkpoints_bluscriptcp) |
| ============================================================= |
| Initializing model on CUDA device 0... |
| [YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693 |
| Parameters: 1060624896 |
| Parameters per GPU: 1060624896 |
| max_steps: 2227 |
| warmup_steps: 100 |
| [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt |
| [ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype) symmetric=off overlap=-/- |
| [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) |
| [ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded) |
| [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. |
| [ZeRO-1 v2] build=v2.4 supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt |
| [ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype) symmetric=off overlap=-/- |
| [ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory) |
| [ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded) |
| [ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it. |
| [ZeRO-1] variant=v2 reduce-scatter/all-gather groups=2 [bfloat16/983236608] [float32/77388288] |
| [ZeRO-1] DataParallel is DISABLED (both would double-reduce) |
| [precision] bf16 params=217, all with fp32 master weights |
| found 12 shards for split train |
| found 1 shards for split val |
| Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (resume run 2) |
| [rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base) |
| Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 2) |
| [CheckpointManager] Auto-loading latest: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.ckpt |
| [WARN] missing ZeRO sidecar state: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.zero_g1 |
| [Resume] run 2 from step 750 (loss 2.57022) |
| [resume] fast-forwarded data loader 12000 batches (393216000 tokens) |
| [re-warmup] anchored at step 750 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 750 (mid-ramp) |
|
|
| Starting training... |
|
|
| [CP ring P2P] OVERLAP: dedicated non-blocking stream |
| validation loss: 3.0712 |
| [CP bwd] overlap=ON |
| step 750 | loss: 2.954028 | lr 1.8000e-06 | norm: 0.1530 | dt: 21484.75ms | tok/sec: 24402.8 | mem: 107147MB | Time Left: 08 hrs : 48 mins |
| [TIMING] data: 7.2ms | fwd: 3950.1ms | loss: 13.3ms | bwd: 10758.8ms | clip: 0.0ms | optim: 24.4ms |
| step 751 | loss: 5.522623 | lr 3.6000e-06 | norm: 6.6146 | dt: 14332.45ms | tok/sec: 36580.5 | mem: 107147MB | Time Left: 05 hrs : 52 mins |
| [TIMING] data: 7.0ms | fwd: 3543.8ms | loss: 13.2ms | bwd: 10731.1ms | clip: 0.0ms | optim: 23.9ms |
| step 752 | loss: 5.446018 | lr 5.4000e-06 | norm: 9.5450 | dt: 14402.47ms | tok/sec: 36402.6 | mem: 107147MB | Time Left: 05 hrs : 54 mins |
| [TIMING] data: 6.9ms | fwd: 3559.8ms | loss: 13.1ms | bwd: 10785.5ms | clip: 0.0ms | optim: 23.9ms |
| step 753 | loss: 5.185619 | lr 7.2000e-06 | norm: 6.0928 | dt: 14539.27ms | tok/sec: 36060.1 | mem: 107147MB | Time Left: 05 hrs : 57 mins |
| [TIMING] data: 6.7ms | fwd: 3650.0ms | loss: 13.1ms | bwd: 10832.4ms | clip: 0.0ms | optim: 23.9ms |
| step 754 | loss: 5.326368 | lr 9.0000e-06 | norm: 4.8898 | dt: 14546.17ms | tok/sec: 36043.0 | mem: 107147MB | Time Left: 05 hrs : 57 mins |
| [TIMING] data: 6.9ms | fwd: 3628.4ms | loss: 13.1ms | bwd: 10860.4ms | clip: 0.0ms | optim: 23.9ms |
| step 755 | loss: 5.359546 | lr 1.0800e-05 | norm: 4.3360 | dt: 14496.08ms | tok/sec: 36167.6 | mem: 107147MB | Time Left: 05 hrs : 55 mins |
| [TIMING] data: 7.2ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10862.1ms | clip: 0.0ms | optim: 24.0ms |
| step 756 | loss: 4.912427 | lr 1.2600e-05 | norm: 4.2124 | dt: 14462.24ms | tok/sec: 36252.2 | mem: 107147MB | Time Left: 05 hrs : 54 mins |
| [TIMING] data: 6.8ms | fwd: 3568.2ms | loss: 13.1ms | bwd: 10836.7ms | clip: 0.0ms | optim: 23.9ms |
| step 757 | loss: 4.810395 | lr 1.4400e-05 | norm: 3.3614 | dt: 14468.13ms | tok/sec: 36237.5 | mem: 107147MB | Time Left: 05 hrs : 54 mins |
| [TIMING] data: 6.8ms | fwd: 3588.2ms | loss: 13.0ms | bwd: 10822.7ms | clip: 0.0ms | optim: 24.0ms |
| step 758 | loss: 4.639567 | lr 1.6200e-05 | norm: 3.4243 | dt: 14469.14ms | tok/sec: 36234.9 | mem: 107147MB | Time Left: 05 hrs : 54 mins |
| [TIMING] data: 6.8ms | fwd: 3597.8ms | loss: 13.1ms | bwd: 10814.3ms | clip: 0.0ms | optim: 23.9ms |
| step 759 | loss: 4.750628 | lr 1.8000e-05 | norm: 3.8068 | dt: 14466.78ms | tok/sec: 36240.8 | mem: 107147MB | Time Left: 05 hrs : 53 mins |
| [TIMING] data: 6.8ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10834.9ms | clip: 0.0ms | optim: 23.9ms |
| step 760 | loss: 4.659847 | lr 1.9800e-05 | norm: 2.5983 | dt: 14466.90ms | tok/sec: 36240.5 | mem: 107147MB | Time Left: 05 hrs : 53 mins |
| [TIMING] data: 6.9ms | fwd: 3571.2ms | loss: 13.1ms | bwd: 10838.4ms | clip: 0.0ms | optim: 23.8ms |
| step 761 | loss: 4.457597 | lr 2.1600e-05 | norm: 6.0180 | dt: 14449.53ms | tok/sec: 36284.1 | mem: 107147MB | Time Left: 05 hrs : 53 mins |
| [TIMING] data: 7.0ms | fwd: 3569.8ms | loss: 13.1ms | bwd: 10821.7ms | clip: 0.0ms | optim: 24.1ms |
| step 762 | loss: 4.134789 | lr 2.3400e-05 | norm: 2.6810 | dt: 14467.83ms | tok/sec: 36238.2 | mem: 107147MB | Time Left: 05 hrs : 53 mins |
| [TIMING] data: 6.7ms | fwd: 3586.3ms | loss: 13.1ms | bwd: 10824.4ms | clip: 0.0ms | optim: 24.0ms |
| step 763 | loss: 3.923339 | lr 2.5200e-05 | norm: 2.2128 | dt: 14479.65ms | tok/sec: 36208.6 | mem: 107147MB | Time Left: 05 hrs : 53 mins |
| [TIMING] data: 6.6ms | fwd: 3607.3ms | loss: 13.1ms | bwd: 10815.4ms | clip: 0.0ms | optim: 23.9ms |
|
|
