Upload folder using huggingface_hub

logs/ext64k/RESULTS.md

@@ -0,0 +1,27 @@
+# 32k -> 64k YaRN extension on b5 -- 2026-09-10
+
+Parent: b4 full-epoch step 2227 (best 32k model: 70.0/58.0/43.0 at 8k/16k/32k).
+Data:   b5_65536_131072 -- min doc 65,536 == the window, so every sample is
+        ONE contiguous document.
+YaRN:   SCALE=2 ORIG_MAXPOS=32768 -> 65536 exactly.
+Train:  695 steps x 524288 = .364 B tokens
+        = 42% of b5's 0.866 B train split (no wrap).
+        WSD warmup 100 / stable 593 / decay tail 2 -- final ckpt DECAYED.
+        LR peak 1.8e-4, floor 6e-5 (CP_MAX_LR is a no-op; see B4 results doc).
+        4x H200, cp=4, B=1 x grad_accum 8.
+Eval:   niah_single, N=100, greedy, shared haystack b2_8192_16384 val.
+
+## Results
+
+| context | niah_single |
+|---|---|
+| 8192 | 67.0
 |
+| 16360 | 65.0
 |
+| 32744 | 52.0
 |
+| 65512 | 29.0
 |
+
+For comparison, the 32k parent at the same lengths: 70.0 / 58.0 / 43.0 / n/a
+(it cannot run at 64k -- its RoPE cache is sized 32,768).
+
+NOTE: this is a PARTIAL b5 epoch, sized to a wall-clock deadline. It is a
+first 64k data point, not a converged extension.

logs/ext64k/eval_16360.log

@@ -0,0 +1,41 @@
+[17:14:05] longeval: T=65536 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=1
+[17:14:05] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[17:14:05] prompt signature changed -- regenerating prompts and clearing state
+[17:14:05] generating prompts: niah_single
+[17:14:08] fan-out: 1 GPU(s), NP=1 -> 1 concurrent job(s)
+[17:14:08] 1 job(s) to run
+[17:14:08]   [gpu 1] ext64k / niah_single
+[17:21:44]   scoring ext64k / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 65/100 = 65.0%
+    depth      acc   (needle position in context)
+     0.00   2/20   10.0%
+     0.25  15/20   75.0%
+     0.50  16/20   80.0%
+     0.75  14/20   70.0%
+     1.00  18/20   90.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020. It is a number that is used to identify the number of times a player has won the game
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote ext64k_len_16360/scores/ext64k_niah_single.csv
+[17:21:44] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext64k
+  niah_single            65.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext64k
+  0.00          10.0+-13.7
+  0.25          75.0+-17.8
+  0.50          80.0+-16.8
+  0.75          70.0+-18.7
+  1.00          90.0+-13.7
+
+  wrote ext64k_len_16360/summary.csv and ext64k_len_16360/depth_summary.csv
+[17:21:44] DONE -> ext64k_len_16360/summary.csv

logs/ext64k/eval_32744.log

@@ -0,0 +1,41 @@
+[17:14:05] longeval: T=65536 prompt_len=32744 max_new=24 n=100 greedy=1 gpus=2
+[17:14:05] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[17:14:05] prompt signature changed -- regenerating prompts and clearing state
+[17:14:05] generating prompts: niah_single
+[17:14:08] fan-out: 1 GPU(s), NP=1 -> 1 concurrent job(s)
+[17:14:08] 1 job(s) to run
+[17:14:08]   [gpu 2] ext64k / niah_single
+[17:26:13]   scoring ext64k / niah_single
+
+=== niah_single  (prompt_len=32744, n=100) ===
+  accuracy: 52/100 = 52.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   8/20   40.0%
+     0.50  12/20   60.0%
+     0.75  15/20   75.0%
+     1.00  17/20   85.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number of yarrow.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote ext64k_len_32744/scores/ext64k_niah_single.csv
+[17:26:13] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext64k
+  niah_single            52.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext64k
+  0.00           0.0+- 8.1
+  0.25          40.0+-19.7
+  0.50          60.0+-19.7
+  0.75          75.0+-17.8
+  1.00          85.0+-15.4
+
+  wrote ext64k_len_32744/summary.csv and ext64k_len_32744/depth_summary.csv
+[17:26:13] DONE -> ext64k_len_32744/summary.csv

logs/ext64k/eval_65512.log

@@ -0,0 +1,41 @@
+[17:14:05] longeval: T=65536 prompt_len=65512 max_new=24 n=100 greedy=1 gpus=3
+[17:14:05] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[17:14:05] prompt signature changed -- regenerating prompts and clearing state
+[17:14:05] generating prompts: niah_single
+[17:14:09] fan-out: 1 GPU(s), NP=1 -> 1 concurrent job(s)
+[17:14:09] 1 job(s) to run
+[17:14:09]   [gpu 3] ext64k / niah_single
+[17:41:08]   scoring ext64k / niah_single
+
+=== niah_single  (prompt_len=65512, n=100) ===
+  accuracy: 29/100 = 29.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   1/20    5.0%
+     0.75  10/20   50.0%
+     1.00  18/20   90.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number yarrow. It is a number that can be used to identify the location of a specific player or event.
+  [MISS] expect=5343902
+         got= the number of times that the plant will grow.\n
+
+  wrote ext64k_len_65512/scores/ext64k_niah_single.csv
+[17:41:08] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext64k
+  niah_single            29.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext64k
+  0.00           0.0+- 8.1
+  0.25           0.0+- 8.1
+  0.50           5.0+-11.4
+  0.75          50.0+-20.1
+  1.00          90.0+-13.7
+
+  wrote ext64k_len_65512/summary.csv and ext64k_len_65512/depth_summary.csv
+[17:41:08] DONE -> ext64k_len_65512/summary.csv

logs/ext64k/eval_8192.log

@@ -0,0 +1,41 @@
+[17:14:05] longeval: T=65536 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0
+[17:14:05] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[17:14:05] prompt signature changed -- regenerating prompts and clearing state
+[17:14:05] generating prompts: niah_single
+[17:14:08] fan-out: 1 GPU(s), NP=1 -> 1 concurrent job(s)
+[17:14:08] 1 job(s) to run
+[17:14:08]   [gpu 0] ext64k / niah_single
+[17:19:53]   scoring ext64k / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 67/100 = 67.0%
+    depth      acc   (needle position in context)
+     0.00   7/20   35.0%
+     0.25  14/20   70.0%
+     0.50  15/20   75.0%
+     0.75  14/20   70.0%
+     1.00  17/20   85.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote ext64k_len_8192/scores/ext64k_niah_single.csv
+[17:19:54] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext64k
+  niah_single            67.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext64k
+  0.00          35.0+-19.3
+  0.25          70.0+-18.7
+  0.50          75.0+-17.8
+  0.75          70.0+-18.7
+  1.00          85.0+-15.4
+
+  wrote ext64k_len_8192/summary.csv and ext64k_len_8192/depth_summary.csv
+[17:19:54] DONE -> ext64k_len_8192/summary.csv

logs/ext64k/ext.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19f49b27c8f059b0db272607a4ee7ed9dccd4451067ef9f1b7030584b61660ac
+size 61

logs/ext64k/master.log

@@ -0,0 +1,34 @@
+[14:30:41] === 32k -> 64k YaRN extension on b5 (4x H200, cp=4) ===
+[14:30:41] parent: b4 full epoch step 2227 (best 32k model)
+[14:30:41] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536  OK
+[14:30:41] b5: 5 train shards = .481 B tokens available
+[14:30:41] hard stop 18:00 UTC = 23:30 IST, reserving 50m
+[14:30:41] PHASE 1: 3-step probe (64k at cp=4 projects to 108.8 GB of 143 GB)
+[14:32:19] PROBE OK: peak mem: 108589MB
+[14:32:19] measured step cost at real batch: 74.009 s/step
+[14:32:19] PHASE 2: 210 steps = .110 B tokens = 12% of a b5 epoch
+[14:32:19]   WSD: warmup 100, stable 10, decay tail 100 -> FINAL checkpoint IS decayed
+[14:41:08] === 32k -> 64k YaRN extension on b5 (4x H200, cp=4) ===
+[14:41:08] parent: b4 full epoch step 2227 (best 32k model)
+[14:41:08] YaRN SCALE=2 ORIG=32768 -> 65536 == CP_T=65536  OK
+[14:41:08] b5: 5 train shards = .481 B tokens available
+[14:41:08] hard stop 18:00 UTC = 23:30 IST, reserving 50m
+[14:41:08] measured step cost at real batch: 74.009 s/step
+[14:41:08] STEPS_OVERRIDE=695 -- the probe ran grad_accum=1 so its dt x8
+[14:41:08]   overstated the real step cost 5.8x (74.0 vs a measured 12.85 s/step).
+[14:41:08] PHASE 2: 695 steps = .364 B tokens = 42% of a b5 epoch
+[14:41:08]   WSD: warmup 100, stable 593, decay tail 2 -> FINAL checkpoint IS decayed
+[17:14:05] ext64k -> run 3, blumodelcp_run3_step_695.ckpt, loss loss: 2.934074
+[17:14:05]   peak mem mem: 108493MB
+[17:14:05]   final lr lr 1.2000e-04  (should be ~6e-05 = decayed)
+[17:14:05] PHASE 3: eval ext64k -- 4 lengths, one GPU each
+[17:41:08]   len 8192 -> niah_single,67.0
+[17:41:08]   len 16360 -> niah_single,65.0
+[17:41:08]   len 32744 -> niah_single,52.0
+[17:41:08]   len 65512 -> niah_single,29.0
+[17:41:08] PHASE 4: upload
+[17:41:10]   ok docs/EXT64K_B5_RESULTS_2026-09-10.md
+[17:41:12]   ok logs/ext64k_len_8192/
+[17:41:14]   ok logs/ext64k_len_16360/
+[17:41:17]   ok logs/ext64k_len_32744/
+[17:41:21]   ok logs/ext64k_len_65512/

logs/ext64k/probe.log

@@ -0,0 +1,60 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
+[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 65536
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 65536
+  world_size      : 4  (dp=1 x cp=4)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 65536  (grad_accum=1)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 100 / 3  (cosine, no stable phase)
+  checkpointing   : false  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=65536 hd=128 scale=2.000 low=24 high=42 m=1.0693
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/b4epoch32k_step2227.ckpt  (base step 2227, loss 2.75788)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 3
+warmup_steps: 100
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=983236608 padded=983236608 W=4 shard=245809152 | 38 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 2813.06 MB (vs 11252.2 MB unsharded)
+[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=77388288 padded=77388288 W=4 shard=19347072 | 6 buckets of 50 MB (3276800 elems/rank/bucket) | optimizer state/GPU: 221.41 MB (vs 885.639 MB unsharded)
+[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
+[ZeRO-1] variant=v2 reduce-scatter/all-gather  groups=2  [bfloat16/983236608]  [float32/77388288]
+[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
+[precision] bf16 params=217, all with fp32 master weights
+found 5 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log1.csv (run 1)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: OFF
+[re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
+
+Starting training...
+
+[CP ring P2P] OVERLAP: dedicated non-blocking stream
+validation loss: 2.9859
+[CP bwd] overlap=ON
+step     0 | loss: 2.990903 | lr 1.8000e-06 | norm: 0.4635 | dt: 16298.63ms | tok/sec: 4021.0 | mem: 100749MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 0.3ms | fwd: 3633.1ms | loss: 0.9ms | bwd: 1733.1ms | clip: 0.0ms | optim: 22.6ms
+step     1 | loss: 2.960938 | lr 3.6000e-06 | norm: 0.3655 | dt: 5177.28ms | tok/sec: 12658.4 | mem: 108493MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 0.1ms | fwd: 3553.8ms | loss: 0.8ms | bwd: 1597.7ms | clip: 0.0ms | optim: 22.3ms
+validation loss: 2.9859
+step     2 | loss: 3.382817 | lr 5.4000e-06 | norm: 0.3972 | dt: 13324.95ms | tok/sec: 4918.3 | mem: 108589MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 0.1ms | fwd: 3523.0ms | loss: 0.8ms | bwd: 1606.4ms | clip: 0.0ms | optim: 22.3ms
+
+=== Training complete ===

logs/ext64k/runnum

@@ -0,0 +1 @@
+3

logs/ext64k/upload.log

@@ -0,0 +1,15 @@
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/63fb997a3148c3dd067dc74ac3bd6cdd64a277df
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/d035068449fb61c2054a7172030df4f48032cedd
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/17c9d86f1fea93e577fa150ebe1bddffa2fb86a1
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/1dbbaa569277fc3e119dffbb31f753ffce9b95c1
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/9a5f665788b5d32a665ff1d8225cf3f26791306f
+Found 12 files to upload
+Uploading... 12/12 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)