| # Handoff: 16k -> 32k YaRN extension on the 2x H200 |
|
|
| Written 2026-09-09 from the 2x H100 box. Everything below is measured on this |
| box unless marked as a projection. |
|
|
| ## What the 2x H200 CAN and CANNOT do |
|
|
| `cp` cannot exceed the GPU count, so on 2 GPUs `tokens/rank = T/2`. |
|
|
| | target | tok/rank | total/GPU | 2x H200 (143 GB) | |
| |---|---|---|---| |
| | 16k | 8,192 | 70.7 GB (measured) | comfortable | |
| | **32k** | 16,384 | **130.8 GB (projected)** | **91.4% -- tight, probe first** | |
| | 64k | 32,768 | 250.9 GB | 175% -- impossible | |
|
|
| - **32k: run it.** `ext_32k.sh` starts with a 3-step probe and aborts on OOM. |
| - **64k: needs 4+ GPUs.** 4x H200 at cp=4 = 127.6 GB/GPU fits. Nothing to tune |
| on 2 GPUs: `CP_B` is already 1, and there is no activation-recompute flag in |
| bluscriptCP.cpp (checked: no `CP_RECOMPUTE`, no gradient checkpointing). |
| - 64k **evaluation** is fine on 2 GPUs -- generation stores no activations for |
| backward, so the training memory figure does not apply. |
|
|
| Measured activation cost is 7.51 MB/token/rank. A first-principles estimate for |
| this architecture is 1.3-2.5 MB, so actual is 3-6x higher than it needs to be |
| (the config keeps fp32 norms/residual/logits across 36 layers). If that gap is |
| real, a memory pass could make 32k comfortable and 64k possible on 2 GPUs -- but |
| that is a C++ change, not a flag, and it is unprofiled. |
|
|
| ## Setup on the new box (nothing is pre-installed) |
|
|
| ```bash |
| cd /root/Adhi/BluTrain/dist/Context_Parallelism |
| |
| # 1. build -- BOTH targets, libtensor first |
| make libtensor SM_ARCH=90 -j$(nproc) |
| make CP_FUSED_ROPE=1 CP_ATTN_SPLIT=1 SM_ARCH=90 bluscript-cp -j$(nproc) |
| |
| # 2. tiktoken -- longeval_gen.py needs it. WITHOUT IT training runs fine and the |
| # eval silently produces "no prompts" / "no scores yet". This has bitten twice. |
| /root/Adhi/BluTrain/envv/bin/pip install tiktoken |
| |
| # 3. hf auth (the dataset is public now, but the model repo push needs a token) |
| hf auth login |
| |
| # 4. parent checkpoint: l2, the best 16k arm. cp=2 needs it on BOTH ranks. |
| python - <<'PY' |
| from huggingface_hub import hf_hub_download |
| import shutil, os |
| p = hf_hub_download("unparallelled/BluScriptCP", |
| "checkpoints/lenbucket_l2_blumodelcp_run2_step_400.ckpt") |
| for r in (0, 1): |
| d = f"checkpoints_bluscriptcp/rank_{r}/lenbucket_l2_blumodelcp_run2_step_400.ckpt" |
| os.makedirs(os.path.dirname(d), exist_ok=True) |
| if not os.path.exists(d): |
| shutil.copyfile(p, d) |
| PY |
| |
| # 5. b3 data (4 train shards + val ~ 930 MB) and the b2-val haystack |
| hf download unparallelled/BluScriptCP-data --repo-type dataset \ |
| --include "cp_data_by_length/b3_16384_32768/train_shard_0000[0-3].bin" \ |
| --include "cp_data_by_length/b3_16384_32768/val_shard_00000.bin" \ |
| --include "cp_data_by_length/b2_8192_16384/val_shard_00000.bin" \ |
| --local-dir ./dl |
| mkdir -p lendata/b3_16384_32768 lendata/haystack |
| cp dl/cp_data_by_length/b3_16384_32768/*.bin lendata/b3_16384_32768/ |
| cp dl/cp_data_by_length/b2_8192_16384/val_shard_00000.bin lendata/haystack/b2val_haystack.bin |
| |
| # VERIFY -- `hf download` exits 0 and prints a path even when --include matches |
| # NOTHING. Always ls, never trust the exit code. |
| ls -la lendata/b3_16384_32768/ lendata/haystack/ |
| |
| # 6. run it |
| ./ext_32k.sh |
| ``` |
|
|
| ## The run |
|
|
| ``` |
| CP_T=32768 YARN_SCALE=2 YARN_ORIG_MAXPOS=16384 |
| parent: lenbucket_l2_blumodelcp_run2_step_400.ckpt |
| data: b3_16384_32768 (docs 16-32k -- these now FIT inside the window) |
| 750 steps x 1,048,576 = 786 M tokens, matching run7's original 4k->16k extension |
| ``` |
|
|
| **The YaRN invariant, which cost 27pp when it was wrong:** `YARN_SCALE * |
| YARN_ORIG_MAXPOS` must equal `CP_T` exactly, and `ORIG_MAXPOS` must be the length |
| the weights were ACTUALLY trained at. Here 2 * 16384 = 32768. The default at |
| `YARNOps.cpp:57` is **1024** -- always set it explicitly. `ext_32k.sh` asserts it. |
|
|
| Projected step time ~25-29 s (2x the 16k tokens/rank plus attention's quadratic |
| term), so **750 steps ~ 5-6.5 h**. Unmeasured; the probe prints the real number. |
|
|
| ## Why this run, in one paragraph |
|
|
| The document-length ladder is finished and it ended in a tie. Holding everything |
| else fixed at a 16k window: |
|
|
| | arm | training docs | 4096 | 8192 | 16360 | |
| |---|---|---|---|---| |
| | armA_c | flux, median 826 | 73.0 | 53.0 | 33.0 | |
| | l1 | b1 4-8k (fits) | 75.0 | 48.0 | 30.0 | |
| | l2 | b2 8-16k (fits) | **80.0** | 62.0 | **35.0** | |
| | l3 | b3 16-32k (TRUNCATED) | 74.0 | **64.0** | **35.0** | |
| |
| l3 - l2 = -6.0 / +2.0 / +0.0 pp, none resolvable (combined 95% bars +-11.5 to |
| +-13.1). The haystack (b2-val) is in-distribution for l2 and out-of-distribution |
| for l3, so the bias runs TOWARD l2 and l3 still only drew level. Conclusion: |
| **document length helps only while documents fit inside the window.** Hence widen |
| the window, then feed b3. |
| |
| ## Gate the result on SHORT context, not perplexity |
| |
| On 2026-09-02 a mismatched cache IMPROVED 16k (0 -> 14-17%) while COLLAPSING 4k |
| from 74% to 16%. Perplexity at the new length looked fine. `ext_32k.sh` therefore |
| evaluates 1024 / 4096 / 16360 / 32744 and **a regression at 1k or 4k is a failure |
| even if 32k improves.** Compare against l2's row above. |
|
|
| (32744 is not arbitrary: `PROMPT_LEN + MAX_NEW` must pad up to <= `CP_T` with |
| PAD=256, and 32744 + 24 = 32768 exactly.) |
|
|
| ## Traps already paid for |
|
|
| 1. **Never edit a running bash script.** Bash reads incrementally from a byte |
| offset; an insert shifts it and the interpreter resumes mid-line. This killed |
| `ladder_456.sh` with `syntax error near unexpected token 'do'`. |
| 2. **`pgrep -f <script>.sh` matches the shell running it.** Killed my own shell |
| twice (exit 144). Exclude `$$`. |
| 3. **Verify all rank shards** before trusting a checkpoint. A 3-of-4 set loads |
| silently wrong, and a zero exit code is not proof -- bluscriptCP catches a |
| failed save and only prints `[WARN]`. |
| 4. **`say()` must write to stderr.** On stdout it poisons `$(...)` capture; that |
| fed a 3-line path into `CP_INIT_FROM` and aborted a run on rank 2. |
| 5. **Run-number collisions.** `run_number` is the first free |
| `CP_BluScreipt_Training_logs/CP_Training_log<N>.csv`. A collision with an |
| existing checkpoint prefix let `CP_CKPT_KEEP=1` prune a live shard. |
| 6. **`CP_STABLE` cannot equal `CP_MAX_STEPS`** -- `bluscriptCP.cpp:1006` aborts |
| with "leaves no cosine decay tail". Leave at least a 2-step tail; the guard |
| reads `CP_WARMUP` while `get_lr` uses `CP_REWARMUP`, so pin both. |
| 7. **`CP_REWARMUP_ANCHOR=0` on every resume**, or each resume re-anchors a fresh |
| warmup ramp and the LR sawtooths (`bluscriptCP.cpp:2105`). |
| |
| ## Concurrently, on the 2x H100 box |
| |
| Long b3 arm at 16k: run 17, 4,000 steps, WSD with flat 3.0e-4 for steps |
| 100-3899 then a 100-step tail. Tests whether more long-context STEPS flatten the |
| slope -- the open question, since all arms sat at -18 to -29 pp/doubling while |
| Phi-3.5-mini (3.8B) manages -3.6. Checkpointed every 25 steps, `KEEP=2`. An |
| eval at step 1000 fires automatically, then the run resumes to 4000. |
| |
| ## Not to do |
| |
| - **b4/b5/b6 at a 16k window** -- l3 already showed truncated documents add |
| nothing. b4 only becomes useful after a 64k extension. |
| - **b6 in a matched ladder** -- 57 M training tokens is 0.27x of one 400-step |
| arm; it would cycle 3.7x through all 790 documents. |
| - **Chasing 128k.** No 4B-class model has effective 128k: applying RULER's own |
| threshold (Llama2-7B at 4k = 85.6), Phi-3.5-mini (3.8B, 128k claimed) has an |
| effective context of **32k**, and so does Llama-3.1-8B-instruct. 32k is the |
| right target, and it is also where this corpus runs out (2.35 B tokens above |
| 32k, total). |
|
|
