Upload folder using huggingface_hub
Browse files
docs/Claude Logs.md
CHANGED
|
@@ -255,3 +255,4 @@
|
|
| 255 |
2026-08-26 - Ported bluscript_zero parity into bluscriptCP: Muon via ShardedMuon (CP_OPTIMIZER=muon), per-group WSD LR pairs, step_order biggest-last, grad-norm L2 fix, CP_MAX_LR/MIN_LR, CP_ZERO_BF16_REDUCE, CP_DIAG per-micro loss - CP / Scripts/Blutrain/bluscriptCP.cpp, Makefile
|
| 256 |
2026-08-26 - Recovered the 600M step-500 reach probe (denom 10.73, reach >=3061) from the destroyed instance vault and pinned its provenance: it describes fp32+AdamW+DDP+cosine on edufineweb, NOT the Muon/ZeRO/bf16/WSD/FLUX base - CP / probe_run4, needle_score.py
|
| 257 |
2026-08-28 - Third --ff-only deadlock, this time on the logs branch: logs-pull.sh ran `git pull --ff-only`, which cannot invoke a merge driver, so once the laptop and an instance had both pushed logs the branch diverged and every pull died with "Not possible to fast-forward" then silently fell back to the LOCAL copy - the vault reported "242 unique lines / 100 reports" from its own content while commit 431ccbb from the other box never arrived. The irony is that the branch already carries `*.md merge=union` in .gitattributes precisely so append-only log edits from several boxes reconcile themselves; --ff-only just never reaches it. Switched logs-pull.sh to `pull --no-rebase --no-edit` and made logs-push.sh merge before pushing (a diverged branch rejects a bare push too), both with an explicit -c user.name/user.email so the merge commit succeeds on instances that have no git identity configured. Sessions side was clean in the same run: restored 1 (c3a9f389 back from the instance at 27MB), 34 already current, 0 diverged, 0 in use - the fork resolution held and the store correctly reported itself ahead. Pattern across all three deadlocks (sessions store, instance clone, logs branch): every one assumed a single writer while the system has always had several - .claude-sessions / scripts/logs-pull.sh, scripts/logs-push.sh
|
|
|
|
|
|
| 255 |
2026-08-26 - Ported bluscript_zero parity into bluscriptCP: Muon via ShardedMuon (CP_OPTIMIZER=muon), per-group WSD LR pairs, step_order biggest-last, grad-norm L2 fix, CP_MAX_LR/MIN_LR, CP_ZERO_BF16_REDUCE, CP_DIAG per-micro loss - CP / Scripts/Blutrain/bluscriptCP.cpp, Makefile
|
| 256 |
2026-08-26 - Recovered the 600M step-500 reach probe (denom 10.73, reach >=3061) from the destroyed instance vault and pinned its provenance: it describes fp32+AdamW+DDP+cosine on edufineweb, NOT the Muon/ZeRO/bf16/WSD/FLUX base - CP / probe_run4, needle_score.py
|
| 257 |
2026-08-28 - Third --ff-only deadlock, this time on the logs branch: logs-pull.sh ran `git pull --ff-only`, which cannot invoke a merge driver, so once the laptop and an instance had both pushed logs the branch diverged and every pull died with "Not possible to fast-forward" then silently fell back to the LOCAL copy - the vault reported "242 unique lines / 100 reports" from its own content while commit 431ccbb from the other box never arrived. The irony is that the branch already carries `*.md merge=union` in .gitattributes precisely so append-only log edits from several boxes reconcile themselves; --ff-only just never reaches it. Switched logs-pull.sh to `pull --no-rebase --no-edit` and made logs-push.sh merge before pushing (a diverged branch rejects a bare push too), both with an explicit -c user.name/user.email so the merge commit succeeds on instances that have no git identity configured. Sessions side was clean in the same run: restored 1 (c3a9f389 back from the instance at 27MB), 34 already current, 0 diverged, 0 in use - the fork resolution held and the store correctly reported itself ahead. Pattern across all three deadlocks (sessions store, instance clone, logs branch): every one assumed a single writer while the system has always had several - .claude-sessions / scripts/logs-pull.sh, scripts/logs-push.sh
|
| 258 |
+
2026-08-29 - CP_INIT_FROM weights-only checkpoint branching (loads before optimizer construction so fp32 masters derive from the loaded weights, not random init - the late CP_CKPT_WEIGHTS_ONLY path clobbered weights on the first step: 2.481 -> 11.084); fixed missing CP_CKPT_DIR getenv, added not-found aborts, added dtype conversion; run8 1.06B proxy trained 1000 steps to val 2.817 and PASSED the IN2 sufficiency gate (denom 12.31 nats, reach boundary inside window, p0 score 0.1525) - Context_Parallelism / bluscriptCP.cpp, checkpointing/Checkpointing.h, needle_sweep.sh, probe_1b.sh
|
docs/Claude Reports/ClaudeReport_session_2026-08-29_init_from_and_run8_probe.md
ADDED
|
@@ -0,0 +1,125 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# CP_INIT_FROM, checkpoint-branching fixes, and the run8 reach probe
|
| 2 |
+
|
| 3 |
+
2026-08-29 - Built the weights-only checkpoint-branching path (CP_INIT_FROM),
|
| 4 |
+
fixed three defects found by testing it, and measured the 1.06B proxy's reach.
|
| 5 |
+
Boxes: metallic-bird-thinks-fin-03 (4x H200, torn down) then
|
| 6 |
+
light-sun-speaks-fin-02 (1x H200). Workspace: BluTrain / dist/Context_Parallelism.
|
| 7 |
+
|
| 8 |
+
## The 1.06B proxy run (run8) completed
|
| 9 |
+
|
| 10 |
+
`d1536 / L36 / Q12 / KV4 / hd128 / FFN4096 / tying OFF` = 1.0606B params
|
| 11 |
+
(Muon 905,969,664 over 216 matrices + AdamW groups 77,266,944 embedding and
|
| 12 |
+
77,388,288 lm_head+norms). Muon + AdamW, ZeRO-1 v2 + symmetric, bf16, WSD
|
| 13 |
+
100/700/200, all groups peaked at 1e-3.
|
| 14 |
+
|
| 15 |
+
| step | val_loss |
|
| 16 |
+
|---|---|
|
| 17 |
+
| 0 | 11.095013 |
|
| 18 |
+
| 250 | 3.940885 |
|
| 19 |
+
| 500 | 3.156706 |
|
| 20 |
+
| 750 | 2.988807 |
|
| 21 |
+
| 999 | **2.817160** |
|
| 22 |
+
|
| 23 |
+
1000 steps in 1h39m at mean 176,067 tok/s, peak 95,437 MB, B=4 on 4x H200.
|
| 24 |
+
Train loss 8.719 (steps 0-24) -> 2.625 (950-974), monotone throughout.
|
| 25 |
+
|
| 26 |
+
**B=4 beat B=2 by 4.7%** (176,600 vs 168,750 tok/s) - the opposite of what a
|
| 27 |
+
3-step sweep said (162k vs 165k). Three steps is too short: the allocator is
|
| 28 |
+
still growing its pool and the larger micro-batch has not reached steady state,
|
| 29 |
+
which biases *against* the larger B. Measure throughput over tens of steps.
|
| 30 |
+
|
| 31 |
+
**LR must be equal for Muon and AdamW.** CP defaulted max_lr=6e-4 with
|
| 32 |
+
muon_lr=1e-3, so the 906M body took 1.67x larger steps than the embeddings.
|
| 33 |
+
MATCH_RMS_ADAMW exists precisely to put them on one scale - ShardedMuon.h says
|
| 34 |
+
"setting both equal gives both groups the same step size and the ONLY difference
|
| 35 |
+
is the update direction" - and bluscript_zero sets both to 1e-3. Fixed with
|
| 36 |
+
CP_MAX_LR=1e-3 CP_MIN_LR=1e-4; the first 507-step run was discarded.
|
| 37 |
+
|
| 38 |
+
## CP_INIT_FROM: branching a new run from a trained base
|
| 39 |
+
|
| 40 |
+
The need: a long-context extension starts from base WEIGHTS but must not inherit
|
| 41 |
+
the optimizer state, which is ZeRO-sharded to the world size that wrote it. A
|
| 42 |
+
normal resume therefore cannot cross a GPU-count change, an optimizer change
|
| 43 |
+
(ZeRO -> no ZeRO), or a dp<->cp reshard.
|
| 44 |
+
|
| 45 |
+
**Resume and initialize-from are NOT the same operation.** Resume restores every
|
| 46 |
+
piece of state together, so ordering does not matter - `load_state` overwrites the
|
| 47 |
+
optimizer's fp32 masters after the fact. Initialize-from has no saved optimizer
|
| 48 |
+
state, so the masters must be DERIVED from the loaded weights, which means the
|
| 49 |
+
weights must land BEFORE any optimizer is constructed.
|
| 50 |
+
|
| 51 |
+
The first attempt (`CP_CKPT_WEIGHTS_ONLY`) loaded at the resume site, line 2141,
|
| 52 |
+
long after both optimizer construction sites (ZeRO at 1775, nn::AdamW at 1824).
|
| 53 |
+
Every one of them snapshots fp32 masters at construction, so the masters held the
|
| 54 |
+
random init and the first step() published them back over the loaded weights:
|
| 55 |
+
|
| 56 |
+
step 1000 loss 2.481 <- weights loaded correctly
|
| 57 |
+
step 1001 loss 11.084 <- masters clobbered them
|
| 58 |
+
|
| 59 |
+
Failing for exactly one step is worse than failing outright. `CP_INIT_FROM` now
|
| 60 |
+
loads right after `model.parameters()` (line 1612), before both optimizers, and
|
| 61 |
+
`CP_CKPT_WEIGHTS_ONLY` is retired with an error pointing at it.
|
| 62 |
+
|
| 63 |
+
**Why this is the universal mechanism:** only replicated, T-independent PARAMETERS
|
| 64 |
+
move. `cos_sin_cache` is a plain member, never `register_parameter`'d, so changing
|
| 65 |
+
T or YARN_SCALE does not change the parameter count. Verified across both
|
| 66 |
+
optimizer paths - ZeRO-1+Muon and plain AdamW both start at loss 2.753656,
|
| 67 |
+
identical to 6 decimals, and stay in the 2.7-3.0 band instead of jumping to 11.
|
| 68 |
+
|
| 69 |
+
## Three defects found by testing, not by reading
|
| 70 |
+
|
| 71 |
+
1. **`CP_CKPT_DIR` did not exist.** I wrote a comment telling the user to set it
|
| 72 |
+
without implementing the getenv, so the loader always read the default
|
| 73 |
+
directory, found nothing, and trained from random init. Silently.
|
| 74 |
+
2. **No abort on "checkpoint not found".** A weights-only load that finds nothing
|
| 75 |
+
is always a mistake; it now dies. Same for `CP_CKPT=0`, where the resume block
|
| 76 |
+
is skipped entirely and the load would never happen.
|
| 77 |
+
3. **`copy_: dtype mismatch`.** The weights-only loader used `p.copy_(loaded)`
|
| 78 |
+
with no conversion. run8 stores bf16 (CP_BF16=1); the reach probe runs fp32.
|
| 79 |
+
Crossing precision is exactly what this path exists for, so it now converts
|
| 80 |
+
via `as_type` when dtypes differ. The ordinary `load_checkpoint` still has the
|
| 81 |
+
limitation, but a resume always matches training precision.
|
| 82 |
+
|
| 83 |
+
## run8 reach probe - the IN2 sufficiency gate
|
| 84 |
+
|
| 85 |
+
`probe_1b.sh` (new; the 600M one is hardcoded to L24/tying=1 with stale
|
| 86 |
+
/root/BluTrain paths). `needle_sweep.sh` gained `INIT_FROM=<path>` because it
|
| 87 |
+
hardcoded `CP_CKPT_RESUME`, and a forward-only eval of a ZeRO/Muon base builds a
|
| 88 |
+
plain nn::AdamW that would try to parse ShardedMuon's state.
|
| 89 |
+
|
| 90 |
+
floor(p1)=1.5181 absent=13.8241 null=12.7732 denom=12.3060 nats
|
| 91 |
+
|
| 92 |
+
| variant | nll | se | score | sig |
|
| 93 |
+
|---|---|---|---|---|
|
| 94 |
+
| p0 | 11.9470 | 0.2510 | 0.1525 | YES |
|
| 95 |
+
| p0.25 | 7.6774 | 0.5615 | 0.4995 | YES |
|
| 96 |
+
| p0.5 | 5.2811 | 0.4232 | 0.6942 | YES |
|
| 97 |
+
| p0.75 | 3.3513 | 0.6088 | 0.8510 | YES |
|
| 98 |
+
| p1 | 1.5181 | 0.0202 | 1.0000 | YES |
|
| 99 |
+
|
| 100 |
+
**Both gates pass.** denom 12.31 nats against the 0.5-nat withholding bar, and
|
| 101 |
+
better than run100 (8.84) and the 600M (10.73). The reach boundary is inside the
|
| 102 |
+
window: p0 scores only 0.1525, so a large failing region remains at distance -
|
| 103 |
+
which is the room IN2 supervision is meant to improve. All five points
|
| 104 |
+
significant at BLOCKS=3, curve monotone.
|
| 105 |
+
|
| 106 |
+
## Still open
|
| 107 |
+
|
| 108 |
+
1. **The plan's R=8174 is run100's, not run8's.** The primary claim region ("bins
|
| 109 |
+
beyond R(T)") must be re-derived for this base, at T=16384, after the
|
| 110 |
+
extension. The ~13% arm-C fraction is anchored to the old number.
|
| 111 |
+
2. **run8 is 4k-native; IN2 trains at T=16384.** The YARN/LongRoPE extension has
|
| 112 |
+
to happen first - that is what CP_INIT_FROM was built for and it is the next
|
| 113 |
+
step.
|
| 114 |
+
3. **in2_data is gone** with the earlier instance; regenerate with in2_gen.py
|
| 115 |
+
(numpy is only in the venv, not system python).
|
| 116 |
+
4. The plan's budget assumed 4x H200.
|
| 117 |
+
|
| 118 |
+
## Environment traps recorded
|
| 119 |
+
|
| 120 |
+
- `hf` is in a venv, NOT on PATH: `/root/Adhi/BluTrain/envv/bin/hf`.
|
| 121 |
+
- `pgrep -f <pattern>` matches the searching shell's own argv when the pattern
|
| 122 |
+
appears in the command line - use `pgrep -f 'make[ ]libtensor'`. This cost a
|
| 123 |
+
wasted wait loop and a chained builder that spun forever.
|
| 124 |
+
- Downloading FLUX: `hf download navingv/flux --repo-type=dataset` (20.2 GB) -
|
| 125 |
+
it is a separate public dataset, not in the BluScriptCP repo.
|