Upload folder using huggingface_hub

logs/b4cont/RESULTS.md

@@ -0,0 +1,144 @@
+# b4 full epoch at 32k -- 2026-09-10
+
+Continuation of the 16k->32k YaRN extension through the whole of b4.
+
+## Setup
+
+| | |
+|---|---|
+| model | 1.06B (36 layers, d_model 1536, 12q/4kv, hd 128, SwiGLU 4096) |
+| window | 32,768 (YaRN SCALE=2 ORIG_MAXPOS=16384, product == 32768 exactly) |
+| parent | ext32k step 750 (itself extended from l2, the best 16k arm) |
+| data | b4_32768_65536 -- min doc 32,768 == the window, so EVERY sample is one contiguous document |
+| tokens | steps 750 -> 2227 = 1.168 B of b4's 1.171 B train split; b4 seen EXACTLY ONCE, no wrap |
+| batch | B=1 x grad_accum 16 = 524,288 tokens/step, identical to every 16k arm |
+| LR | peak 1.8e-4, floor 6e-5 (see the note below), WSD: re-warm 100 from step 750, flat to 2027, 200-step decay tail |
+| hardware | 2x H200, cp=2, peak 107.4 GB of 143 GB |
+| eval | niah_single, N=100, greedy, substring match, one shared haystack (b2_8192_16384 val) |
+
+## Results
+
+niah_single accuracy %, +-95% Wilson half-width
+
+| step | b4 seen | 8,192 | 16,360 | 32,744 | decayed |
+|---|---|---|---|---|---|
+| 750 | 34% | 60.0 +-9.4 | 45.0 +-9.6 | 27.0 +-8.6 | yes |
+| 1100 | 49% | 55.0 +-9.6 | 40.0 +-9.4 | 24.0 +-8.3 | no |
+| 1650 | 74% | 64.0 +-9.2 | 55.0 +-9.6 | 33.0 +-9.1 | no |
+| **2227** | **100%** | **70.0 +-8.8** | **58.0 +-9.5** | **43.0 +-9.5** | **yes** |
+
+The 1100 and 1650 checkpoints sit mid-plateau at constant LR and are UNDECAYED, so
+they understate. Only 750 and 2227 are decayed and mutually comparable -- which is
+why 1100 reads BELOW 750 and is not a regression.
+
+### The comparison: step 750 vs 2227, both decayed
+
+| length | 750 | 2227 | delta | combined 95% bar | resolvable |
+|---|---|---|---|---|---|
+| 8,192 | 60.0 | 70.0 | +10.0 | +-12.9 | no |
+| 16,360 | 45.0 | 58.0 | +13.0 | +-13.5 | no |
+| 32,744 | 27.0 | 43.0 | **+16.0** | +-12.8 | **YES** |
+
+The gains GROW with length (+10, +13, +16). That is the slope flattening, not a
+uniform lift. Slope on the decayed points: 8k->16k -15.0 -> -12.0 pp/doubling;
+16k->32k -18.0 -> -15.0.
+
+### 16k retrieval across the project, same harness, matched cache
+
+| model | 16,360 |
+|---|---|
+| armA_c (flux, 400 steps) | 33.0 |
+| l2 (b2, 400 steps) | 35.0 |
+| ext32k (b4, 750 steps) | 45.0 |
+| **b4 full epoch (2227 steps)** | **58.0** |
+
+## Conclusion, and what it is not
+
+Training steps AT the target length is the lever that moved the slope. Document
+length shifted the intercept only (l3 vs l2 at 16k was +0.0pp); the YaRN cache fix
+was a one-off +27pp at the window boundary; model scale remains untested. The b3
+long arm at 16k found the same thing independently (-29 -> -19 -> -11 pp/doubling
+across 400/1000/2000 steps).
+
+Only the 32,744 cell is individually resolvable at N=100. The slope figures are
+derived from two points each and inherit those error bars.
+
+## KNOWN ISSUE -- CP_MAX_LR and CP_MIN_LR do not exist
+
+bluscriptCP.cpp hardcodes max_lr=6e-4 (line 131) and min_lr=6e-5 (line 132), and
+the env-parsing block never reads CP_MAX_LR or CP_MIN_LR -- there is not even a
+float env helper. Every script in this project passed them as a no-op. So the real
+schedule has always been 6e-4 * CP_REWARMUP_PEAK(0.3) = 1.8e-4 peak, 6e-5 floor,
+verified from ext32k's log (1.8000e-04 at steps 99..649, 6.0030e-05 at 749).
+
+This is benign for comparability: EVERY run in this project used the same values,
+so l1/l2/l3, ext32k, the b3 long arm and this run are all mutually comparable. Only
+prose descriptions of the LR were wrong. The LR was deliberately NOT changed here,
+because doing so would have broken continuity with ext32k's first 750 steps.
+
+## The resume that required the sidecar
+
+A first attempt at this continuation was aborted at step 763. The .zero_g1
+optimizer sidecar had not been uploaded with the step-750 checkpoint, and per
+bluscriptCP.cpp:2032 a bf16 run then "resumes with group 1's Adam m/v at zero".
+Observed effect: loss 2.95 -> 5.52 at the first optimizer step, gradient norm
+0.15 -> 9.55. After recovering the sidecar from the original instance the same
+resume gave loss 2.95 -> 3.16, norm 0.15 -> 0.18, and the run completed with
+median norm 0.20. ALWAYS upload the .zero_g1 sidecar alongside the .ckpt.
+
+## Raw eval output
+
+### step 1100, PROMPT_LEN=8192
+```
+task,b4c1100
+niah_single,55.0
+```
+
+### step 1100, PROMPT_LEN=16360
+```
+task,b4c1100
+niah_single,40.0
+```
+
+### step 1100, PROMPT_LEN=32744
+```
+task,b4c1100
+niah_single,24.0
+```
+
+### step 1650, PROMPT_LEN=8192
+```
+task,b4c1650
+niah_single,64.0
+```
+
+### step 1650, PROMPT_LEN=16360
+```
+task,b4c1650
+niah_single,55.0
+```
+
+### step 1650, PROMPT_LEN=32744
+```
+task,b4c1650
+niah_single,33.0
+```
+
+### step 2227, PROMPT_LEN=8192
+```
+task,b4c2227
+niah_single,70.0
+```
+
+### step 2227, PROMPT_LEN=16360
+```
+task,b4c2227
+niah_single,58.0
+```
+
+### step 2227, PROMPT_LEN=32744
+```
+task,b4c2227
+niah_single,43.0
+```
+

logs/b4cont/eval_1100_16360.log

@@ -0,0 +1,41 @@
+[08:21:34] longeval: T=32768 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1
+[08:21:34] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[08:21:34] prompt signature changed -- regenerating prompts and clearing state
+[08:21:34] generating prompts: niah_single
+[08:21:35] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[08:21:35] 1 job(s) to run
+[08:21:35]   [gpu 0] b4c1100 / niah_single
+[08:25:40]   scoring b4c1100 / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 40/100 = 40.0%
+    depth      acc   (needle position in context)
+     0.00   2/20   10.0%
+     0.25   4/20   20.0%
+     0.50   8/20   40.0%
+     0.75  11/20   55.0%
+     1.00  15/20   75.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 8.\nQuestion: What is the special magic number for yarrow?\n
+  [MISS] expect=5343902
+         got= “2”.\n
+
+  wrote b4c_1100_len_16360/scores/b4c1100_niah_single.csv
+[08:25:40] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c1100
+  niah_single            40.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c1100
+  0.00          10.0+-13.7
+  0.25          20.0+-16.8
+  0.50          40.0+-19.7
+  0.75          55.0+-20.0
+  1.00          75.0+-17.8
+
+  wrote b4c_1100_len_16360/summary.csv and b4c_1100_len_16360/depth_summary.csv
+[08:25:40] DONE -> b4c_1100_len_16360/summary.csv

logs/b4cont/eval_1100_32744.log

@@ -0,0 +1,41 @@
+[08:25:40] longeval: T=32768 prompt_len=32744 max_new=24 n=100 greedy=1 gpus=0,1
+[08:25:40] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[08:25:40] prompt signature changed -- regenerating prompts and clearing state
+[08:25:40] generating prompts: niah_single
+[08:25:41] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[08:25:41] 1 job(s) to run
+[08:25:41]   [gpu 0] b4c1100 / niah_single
+[08:34:54]   scoring b4c1100 / niah_single
+
+=== niah_single  (prompt_len=32744, n=100) ===
+  accuracy: 24/100 = 24.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   1/20    5.0%
+     0.75   6/20   30.0%
+     1.00  17/20   85.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= “1.”\n
+  [MISS] expect=5343902
+         got= 1. This number is unique to the game and can be used to unlock special bonuses or rewards.\n
+
+  wrote b4c_1100_len_32744/scores/b4c1100_niah_single.csv
+[08:34:54] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c1100
+  niah_single            24.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c1100
+  0.00           0.0+- 8.1
+  0.25           0.0+- 8.1
+  0.50           5.0+-11.4
+  0.75          30.0+-18.7
+  1.00          85.0+-15.4
+
+  wrote b4c_1100_len_32744/summary.csv and b4c_1100_len_32744/depth_summary.csv
+[08:34:54] DONE -> b4c_1100_len_32744/summary.csv

logs/b4cont/eval_1100_8192.log

@@ -0,0 +1,41 @@
+[08:19:31] longeval: T=32768 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
+[08:19:31] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[08:19:31] prompt signature changed -- regenerating prompts and clearing state
+[08:19:31] generating prompts: niah_single
+[08:19:38] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[08:19:38] 1 job(s) to run
+[08:19:38]   [gpu 0] b4c1100 / niah_single
+[08:21:34]   scoring b4c1100 / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 55/100 = 55.0%
+    depth      acc   (needle position in context)
+     0.00   7/20   35.0%
+     0.25  12/20   60.0%
+     0.50   7/20   35.0%
+     0.75  11/20   55.0%
+     1.00  18/20   90.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote b4c_1100_len_8192/scores/b4c1100_niah_single.csv
+[08:21:34] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c1100
+  niah_single            55.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c1100
+  0.00          35.0+-19.3
+  0.25          60.0+-19.7
+  0.50          35.0+-19.3
+  0.75          55.0+-20.0
+  1.00          90.0+-13.7
+
+  wrote b4c_1100_len_8192/summary.csv and b4c_1100_len_8192/depth_summary.csv
+[08:21:34] DONE -> b4c_1100_len_8192/summary.csv

logs/b4cont/eval_1650_16360.log

@@ -0,0 +1,41 @@
+[10:53:26] longeval: T=32768 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1
+[10:53:27] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[10:53:27] prompt signature changed -- regenerating prompts and clearing state
+[10:53:27] generating prompts: niah_single
+[10:53:27] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[10:53:27] 1 job(s) to run
+[10:53:27]   [gpu 0] b4c1650 / niah_single
+[10:56:58]   scoring b4c1650 / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 55/100 = 55.0%
+    depth      acc   (needle position in context)
+     0.00   1/20    5.0%
+     0.25   9/20   45.0%
+     0.50  12/20   60.0%
+     0.75  14/20   70.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 8.\nQuestion: What is the magic number for yarrow?\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote b4c_1650_len_16360/scores/b4c1650_niah_single.csv
+[10:56:58] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c1650
+  niah_single            55.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c1650
+  0.00           5.0+-11.4
+  0.25          45.0+-20.0
+  0.50          60.0+-19.7
+  0.75          70.0+-18.7
+  1.00          95.0+-11.4
+
+  wrote b4c_1650_len_16360/summary.csv and b4c_1650_len_16360/depth_summary.csv
+[10:56:58] DONE -> b4c_1650_len_16360/summary.csv

logs/b4cont/eval_1650_32744.log

@@ -0,0 +1,41 @@
+[10:56:58] longeval: T=32768 prompt_len=32744 max_new=24 n=100 greedy=1 gpus=0,1
+[10:56:58] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[10:56:58] prompt signature changed -- regenerating prompts and clearing state
+[10:56:58] generating prompts: niah_single
+[10:56:58] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[10:56:58] 1 job(s) to run
+[10:56:58]   [gpu 0] b4c1650 / niah_single
+[11:06:14]   scoring b4c1650 / niah_single
+
+=== niah_single  (prompt_len=32744, n=100) ===
+  accuracy: 33/100 = 33.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   4/20   20.0%
+     0.75  12/20   60.0%
+     1.00  17/20   85.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= “the number of the number of the number of the number of the number of the number of the number of the
+  [MISS] expect=5343902
+         got= 8.\nQuestion: How do I verify my account?\n
+
+  wrote b4c_1650_len_32744/scores/b4c1650_niah_single.csv
+[11:06:15] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c1650
+  niah_single            33.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c1650
+  0.00           0.0+- 8.1
+  0.25           0.0+- 8.1
+  0.50          20.0+-16.8
+  0.75          60.0+-19.7
+  1.00          85.0+-15.4
+
+  wrote b4c_1650_len_32744/summary.csv and b4c_1650_len_32744/depth_summary.csv
+[11:06:15] DONE -> b4c_1650_len_32744/summary.csv

logs/b4cont/eval_1650_8192.log

@@ -0,0 +1,41 @@
+[10:51:34] longeval: T=32768 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
+[10:51:34] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[10:51:34] prompt signature changed -- regenerating prompts and clearing state
+[10:51:34] generating prompts: niah_single
+[10:51:35] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[10:51:35] 1 job(s) to run
+[10:51:35]   [gpu 0] b4c1650 / niah_single
+[10:53:26]   scoring b4c1650 / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 64/100 = 64.0%
+    depth      acc   (needle position in context)
+     0.00   6/20   30.0%
+     0.25  14/20   70.0%
+     0.50  11/20   55.0%
+     0.75  15/20   75.0%
+     1.00  18/20   90.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote b4c_1650_len_8192/scores/b4c1650_niah_single.csv
+[10:53:26] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c1650
+  niah_single            64.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c1650
+  0.00          30.0+-18.7
+  0.25          70.0+-18.7
+  0.50          55.0+-20.0
+  0.75          75.0+-17.8
+  1.00          90.0+-13.7
+
+  wrote b4c_1650_len_8192/summary.csv and b4c_1650_len_8192/depth_summary.csv
+[10:53:26] DONE -> b4c_1650_len_8192/summary.csv

logs/b4cont/eval_2227_16360.log

@@ -0,0 +1,41 @@
+[13:30:57] longeval: T=32768 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1
+[13:30:57] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[13:30:57] prompt signature changed -- regenerating prompts and clearing state
+[13:30:57] generating prompts: niah_single
+[13:30:58] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[13:30:58] 1 job(s) to run
+[13:30:58]   [gpu 0] b4c2227 / niah_single
+[13:34:20]   scoring b4c2227 / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 58/100 = 58.0%
+    depth      acc   (needle position in context)
+     0.00   1/20    5.0%
+     0.25  10/20   50.0%
+     0.50  12/20   60.0%
+     0.75  17/20   85.0%
+     1.00  18/20   90.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 8.\nQuestion: What is the special magic number for yarrow?\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote b4c_2227_len_16360/scores/b4c2227_niah_single.csv
+[13:34:20] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c2227
+  niah_single            58.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c2227
+  0.00           5.0+-11.4
+  0.25          50.0+-20.1
+  0.50          60.0+-19.7
+  0.75          85.0+-15.4
+  1.00          90.0+-13.7
+
+  wrote b4c_2227_len_16360/summary.csv and b4c_2227_len_16360/depth_summary.csv
+[13:34:20] DONE -> b4c_2227_len_16360/summary.csv

logs/b4cont/eval_2227_32744.log

@@ -0,0 +1,41 @@
+[13:34:20] longeval: T=32768 prompt_len=32744 max_new=24 n=100 greedy=1 gpus=0,1
+[13:34:20] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[13:34:20] prompt signature changed -- regenerating prompts and clearing state
+[13:34:20] generating prompts: niah_single
+[13:34:21] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[13:34:21] 1 job(s) to run
+[13:34:21]   [gpu 0] b4c2227 / niah_single
+[13:43:00]   scoring b4c2227 / niah_single
+
+=== niah_single  (prompt_len=32744, n=100) ===
+  accuracy: 43/100 = 43.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   3/20   15.0%
+     0.50   9/20   45.0%
+     0.75  11/20   55.0%
+     1.00  20/20  100.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 3.\nQuestion: What is the special magic number for yarrow?\n
+  [MISS] expect=5343902
+         got= 3.\nQuestion: How do I get a refund on my winnings?\n
+
+  wrote b4c_2227_len_32744/scores/b4c2227_niah_single.csv
+[13:43:00] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c2227
+  niah_single            43.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c2227
+  0.00           0.0+- 8.1
+  0.25          15.0+-15.4
+  0.50          45.0+-20.0
+  0.75          55.0+-20.0
+  1.00         100.0+- 8.1
+
+  wrote b4c_2227_len_32744/summary.csv and b4c_2227_len_32744/depth_summary.csv
+[13:43:00] DONE -> b4c_2227_len_32744/summary.csv

logs/b4cont/eval_2227_8192.log

@@ -0,0 +1,41 @@
+[13:29:13] longeval: T=32768 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
+[13:29:13] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[13:29:13] prompt signature changed -- regenerating prompts and clearing state
+[13:29:13] generating prompts: niah_single
+[13:29:13] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[13:29:13] 1 job(s) to run
+[13:29:13]   [gpu 0] b4c2227 / niah_single
+[13:30:57]   scoring b4c2227 / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 70/100 = 70.0%
+    depth      acc   (needle position in context)
+     0.00   7/20   35.0%
+     0.25  14/20   70.0%
+     0.50  14/20   70.0%
+     0.75  16/20   80.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 805602020\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote b4c_2227_len_8192/scores/b4c2227_niah_single.csv
+[13:30:57] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                b4c2227
+  niah_single            70.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth            b4c2227
+  0.00          35.0+-19.3
+  0.25          70.0+-18.7
+  0.50          70.0+-18.7
+  0.75          80.0+-16.8
+  1.00          95.0+-11.4
+
+  wrote b4c_2227_len_8192/summary.csv and b4c_2227_len_8192/depth_summary.csv
+[13:30:57] DONE -> b4c_2227_len_8192/summary.csv

logs/b4cont/master.log

@@ -0,0 +1,34 @@
+[05:59:11] === CONTINUE ext32k through the rest of b4 ===
+[05:59:11] resume run 2 at step 750 -> 2227  (1477 new steps, .774 B tokens)
+[05:59:11] loader fast-forwards 12000 batches = .393 B; reads to 1.167 B of 1.171 B -- no wrap
+[05:59:11] WARNING: .zero_g1 sidecar absent -- group 1 Adam moments start at zero (small transient)
+[05:59:11] eval points: 1100 1650 2227
+[05:59:11] training to step 1100
+[06:52:12] === CONTINUE ext32k through the rest of b4 ===
+[06:52:12] resume run 2 at step 750 -> 2227  (1477 new steps, .774 B tokens)
+[06:52:12] loader fast-forwards 12000 batches = .393 B; reads to 1.167 B of 1.171 B -- no wrap
+[06:52:12] WARNING: .zero_g1 sidecar absent -- group 1 Adam moments start at zero (small transient)
+[06:52:12] eval points: 1100 1650 2227
+[06:52:12] training to step 1100
+[08:19:31] step 1100 reached, preserved b4c1100_blumodelcp_run2_step_1100.ckpt, loss loss: 2.558936
+[08:19:31] EVAL @ step 1100
+[08:21:34]   step 1100 len 8192 -> niah_single,55.0
+[08:25:40]   step 1100 len 16360 -> niah_single,40.0
+[08:34:54]   step 1100 len 32744 -> niah_single,24.0
+[08:34:54] training to step 1650
+[10:51:34] step 1650 reached, preserved b4c1650_blumodelcp_run2_step_1650.ckpt, loss loss: 2.813203
+[10:51:34] EVAL @ step 1650
+[10:53:26]   step 1650 len 8192 -> niah_single,64.0
+[10:56:58]   step 1650 len 16360 -> niah_single,55.0
+[11:06:15]   step 1650 len 32744 -> niah_single,33.0
+[11:06:15] training to step 2227
+[13:29:13] step 2227 reached, preserved b4c2227_blumodelcp_run2_step_2227.ckpt, loss loss: 2.757876
+[13:29:13] EVAL @ step 2227
+[13:30:57]   step 2227 len 8192 -> niah_single,70.0
+[13:34:20]   step 2227 len 16360 -> niah_single,58.0
+[13:43:00]   step 2227 len 32744 -> niah_single,43.0
+[13:43:00] === B4 CONTINUATION DONE ===
+[13:43:00] progression vs ext32k@750 (niah_single):
+[13:43:00]   len 8192:  750=(see ext32k)  1100=55.0
  1650=64.0
  2227=70.0
+[13:43:00]   len 16360:  750=(see ext32k)  1100=40.0
  1650=55.0
  2227=58.0
+[13:43:00]   len 32744:  750=(see ext32k)  1100=24.0
  1650=33.0
  2227=43.0

logs/b4cont/seg_1100.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8530325ab46bf188ca339e08a477910d560a48564bc07d3c7da0583319bc2e3f
+size 39

logs/b4cont/seg_1650.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c91ab412f3954dd3fc3eb1be54ae45dbe1151f2f61a9f65193cbca4cb08c269b
+size 39

logs/b4cont/seg_2227.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:909641bd8cb072d2e72f0640fac0d16d0c5a27eb0d84146a4c0a7d95427a22c6
+size 39

logs/b4cont/train.log.aborted_14steps

@@ -0,0 +1,82 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
+[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 32768
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 32768
+  world_size      : 2  (dp=1 x cp=2)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 524288  (grad_accum=16)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 100 / 2227  (WSD: stable=1177, decay tail=950)
+  checkpointing   : true  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 2227
+warmup_steps: 100
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded)
+[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded)
+[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
+[ZeRO-1] variant=v2 reduce-scatter/all-gather  groups=2  [bfloat16/983236608]  [float32/77388288]
+[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
+[precision] bf16 params=217, all with fp32 master weights
+found 12 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log2.csv (resume run 2)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: ON -> checkpoints_bluscriptcp/blumodelcp_run2_step_<N>.ckpt (every 50 steps, keep 2)
+[CheckpointManager] Auto-loading latest: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.ckpt
+[WARN] missing ZeRO sidecar state: checkpoints_bluscriptcp/rank_0/blumodelcp_run2_step_750.zero_g1
+[Resume] run 2 from step 750 (loss 2.57022)
+[resume] fast-forwarded data loader 12000 batches (393216000 tokens)
+[re-warmup] anchored at step 750 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 750 (mid-ramp)
+
+Starting training...
+
+[CP ring P2P] OVERLAP: dedicated non-blocking stream
+validation loss: 3.0712
+[CP bwd] overlap=ON
+step   750 | loss: 2.954028 | lr 1.8000e-06 | norm: 0.1530 | dt: 21484.75ms | tok/sec: 24402.8 | mem: 107147MB | Time Left: 08 hrs : 48 mins
+  [TIMING] data: 7.2ms | fwd: 3950.1ms | loss: 13.3ms | bwd: 10758.8ms | clip: 0.0ms | optim: 24.4ms
+step   751 | loss: 5.522623 | lr 3.6000e-06 | norm: 6.6146 | dt: 14332.45ms | tok/sec: 36580.5 | mem: 107147MB | Time Left: 05 hrs : 52 mins
+  [TIMING] data: 7.0ms | fwd: 3543.8ms | loss: 13.2ms | bwd: 10731.1ms | clip: 0.0ms | optim: 23.9ms
+step   752 | loss: 5.446018 | lr 5.4000e-06 | norm: 9.5450 | dt: 14402.47ms | tok/sec: 36402.6 | mem: 107147MB | Time Left: 05 hrs : 54 mins
+  [TIMING] data: 6.9ms | fwd: 3559.8ms | loss: 13.1ms | bwd: 10785.5ms | clip: 0.0ms | optim: 23.9ms
+step   753 | loss: 5.185619 | lr 7.2000e-06 | norm: 6.0928 | dt: 14539.27ms | tok/sec: 36060.1 | mem: 107147MB | Time Left: 05 hrs : 57 mins
+  [TIMING] data: 6.7ms | fwd: 3650.0ms | loss: 13.1ms | bwd: 10832.4ms | clip: 0.0ms | optim: 23.9ms
+step   754 | loss: 5.326368 | lr 9.0000e-06 | norm: 4.8898 | dt: 14546.17ms | tok/sec: 36043.0 | mem: 107147MB | Time Left: 05 hrs : 57 mins
+  [TIMING] data: 6.9ms | fwd: 3628.4ms | loss: 13.1ms | bwd: 10860.4ms | clip: 0.0ms | optim: 23.9ms
+step   755 | loss: 5.359546 | lr 1.0800e-05 | norm: 4.3360 | dt: 14496.08ms | tok/sec: 36167.6 | mem: 107147MB | Time Left: 05 hrs : 55 mins
+  [TIMING] data: 7.2ms | fwd: 3576.2ms | loss: 13.1ms | bwd: 10862.1ms | clip: 0.0ms | optim: 24.0ms
+step   756 | loss: 4.912427 | lr 1.2600e-05 | norm: 4.2124 | dt: 14462.24ms | tok/sec: 36252.2 | mem: 107147MB | Time Left: 05 hrs : 54 mins
+  [TIMING] data: 6.8ms | fwd: 3568.2ms | loss: 13.1ms | bwd: 10836.7ms | clip: 0.0ms | optim: 23.9ms
+step   757 | loss: 4.810395 | lr 1.4400e-05 | norm: 3.3614 | dt: 14468.13ms | tok/sec: 36237.5 | mem: 107147MB | Time Left: 05 hrs : 54 mins
+  [TIMING] data: 6.8ms | fwd: 3588.2ms | loss: 13.0ms | bwd: 10822.7ms | clip: 0.0ms | optim: 24.0ms
+step   758 | loss: 4.639567 | lr 1.6200e-05 | norm: 3.4243 | dt: 14469.14ms | tok/sec: 36234.9 | mem: 107147MB | Time Left: 05 hrs : 54 mins
+  [TIMING] data: 6.8ms | fwd: 3597.8ms | loss: 13.1ms | bwd: 10814.3ms | clip: 0.0ms | optim: 23.9ms
+step   759 | loss: 4.750628 | lr 1.8000e-05 | norm: 3.8068 | dt: 14466.78ms | tok/sec: 36240.8 | mem: 107147MB | Time Left: 05 hrs : 53 mins
+  [TIMING] data: 6.8ms | fwd: 3574.8ms | loss: 13.1ms | bwd: 10834.9ms | clip: 0.0ms | optim: 23.9ms
+step   760 | loss: 4.659847 | lr 1.9800e-05 | norm: 2.5983 | dt: 14466.90ms | tok/sec: 36240.5 | mem: 107147MB | Time Left: 05 hrs : 53 mins
+  [TIMING] data: 6.9ms | fwd: 3571.2ms | loss: 13.1ms | bwd: 10838.4ms | clip: 0.0ms | optim: 23.8ms
+step   761 | loss: 4.457597 | lr 2.1600e-05 | norm: 6.0180 | dt: 14449.53ms | tok/sec: 36284.1 | mem: 107147MB | Time Left: 05 hrs : 53 mins
+  [TIMING] data: 7.0ms | fwd: 3569.8ms | loss: 13.1ms | bwd: 10821.7ms | clip: 0.0ms | optim: 24.1ms
+step   762 | loss: 4.134789 | lr 2.3400e-05 | norm: 2.6810 | dt: 14467.83ms | tok/sec: 36238.2 | mem: 107147MB | Time Left: 05 hrs : 53 mins
+  [TIMING] data: 6.7ms | fwd: 3586.3ms | loss: 13.1ms | bwd: 10824.4ms | clip: 0.0ms | optim: 24.0ms
+step   763 | loss: 3.923339 | lr 2.5200e-05 | norm: 2.2128 | dt: 14479.65ms | tok/sec: 36208.6 | mem: 107147MB | Time Left: 05 hrs : 53 mins
+  [TIMING] data: 6.6ms | fwd: 3607.3ms | loss: 13.1ms | bwd: 10815.4ms | clip: 0.0ms | optim: 23.9ms

logs/b4cont/upload.log

@@ -0,0 +1,30 @@
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/bc6b6d9f8481a9693627f38e420e67438cbb02b3
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/a00f002e4319512f053788c496981706102a39be
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/a4f1be44e5eef9dbde4f82b9705d2a1bc499ccf1
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/4db6239bc0eb8d312eac38d3de42d335d3ddfa4a
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/a371ca3c00154c4436c4509801813bfbadd2a56f
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/b34c7a6d0601917ab0096438b8ccb01389e7938b
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/7e4844fd45e2458518ad2dd2df3d04eae4a7674b
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/97c5ce10b97c5f48873422e2379bd18e65057138
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/9f426573d38ed96c2d9ab57e206c3beac1f0e4af
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/1 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/5e3280a15b7eafcf7439974f5daca39f6213a352
+Found 23 files to upload
+Uploading... 23/23 files checked, 0/3 uploaded (0.00B transferred), 0 committed in 0 commit(s)