Upload logs/2026-09-19_adamw_const_probe.log with huggingface_hub

logs/2026-09-19_adamw_const_probe.log

@@ -0,0 +1,57 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+[ZeRO-1] world_size=1: sharding is a no-op, running for parity
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 24
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 16384
+  vocab_size      : 50304  tying=1
+  B x T           : 1 x 16384
+  world_size      : 1  (dp=1 x cp=1)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=32)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 8e-05 / 8e-05
+  warmup/steps    : 12 / 3  (cosine, no stable phase)
+  checkpointing   : false  (ckpt_freq=200, dir=checkpoints_const_adamw)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=16384 hd=128 scale=4.000 low=14 high=32 m=1.1386
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/varybatch.ckpt  (base step 3400, loss 2.31541)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 681328128
+Parameters per GPU: 681328128
+max_steps: 3
+warmup_steps: 12
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=603979776 padded=603979776 W=1 shard=603979776 | 24 buckets of 50 MB (26214400 elems/rank/bucket) | optimizer state/GPU: 6912 MB (vs 6912 MB unsharded)
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=77348352 padded=77348352 W=1 shard=77348352 | 6 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 885.182 MB (vs 885.182 MB unsharded)
+[ZeRO-1] optimizer=adamw  variant=v2 reduce-scatter/all-gather  groups=2  [bfloat16/603979776]  [float32/77348352]
+[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
+[precision] bf16 params=144, all with fp32 master weights
+found 7 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log1.csv (run 1)
+[rope_theta] 500000 from CP_ROPE_THETA (no sidecar -- provenance UNVERIFIED against the checkpoint)
+Checkpointing: OFF
+[re-warmup] anchored at step 0 for 12 steps -> peak 8e-05 (1 x max_lr), then cosine-decay to 8e-05 | resuming at step 0 (mid-ramp)
+
+Starting training...
+
+validation loss: 2.8726
+[CP bwd] overlap=ON
+[CP ring P2P] OVERLAP: dedicated non-blocking stream
+step     0 | loss: 2.731203 | lr 6.6667e-06 | norm: 1.0297 | dt: 14153.38ms | tok/sec: 37043.3 | mem: 76617MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 6.7ms | fwd: 3089.2ms | loss: 26.6ms | bwd: 8798.6ms | clip: 0.0ms | optim: 14.4ms
+step     1 | loss: 2.750832 | lr 1.3333e-05 | norm: 1.0543 | dt: 11991.43ms | tok/sec: 43721.9 | mem: 78537MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 6.5ms | fwd: 3101.8ms | loss: 26.5ms | bwd: 8822.9ms | clip: 0.0ms | optim: 14.2ms
+validation loss: 2.8519
+step     2 | loss: 2.755920 | lr 2.0000e-05 | norm: 0.5832 | dt: 14017.30ms | tok/sec: 37402.9 | mem: 78665MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 6.4ms | fwd: 3104.8ms | loss: 26.6ms | bwd: 8836.3ms | clip: 0.0ms | optim: 12.1ms
+
+=== Training complete ===