unparallelled commited on
Commit
75e4043
·
verified ·
1 Parent(s): 32d3d53

Upload folder using huggingface_hub

Browse files
docs/Claude Logs.md CHANGED
@@ -255,3 +255,4 @@
255
  2026-08-26 - Ported bluscript_zero parity into bluscriptCP: Muon via ShardedMuon (CP_OPTIMIZER=muon), per-group WSD LR pairs, step_order biggest-last, grad-norm L2 fix, CP_MAX_LR/MIN_LR, CP_ZERO_BF16_REDUCE, CP_DIAG per-micro loss - CP / Scripts/Blutrain/bluscriptCP.cpp, Makefile
256
  2026-08-26 - Recovered the 600M step-500 reach probe (denom 10.73, reach >=3061) from the destroyed instance vault and pinned its provenance: it describes fp32+AdamW+DDP+cosine on edufineweb, NOT the Muon/ZeRO/bf16/WSD/FLUX base - CP / probe_run4, needle_score.py
257
  2026-08-28 - Third --ff-only deadlock, this time on the logs branch: logs-pull.sh ran `git pull --ff-only`, which cannot invoke a merge driver, so once the laptop and an instance had both pushed logs the branch diverged and every pull died with "Not possible to fast-forward" then silently fell back to the LOCAL copy - the vault reported "242 unique lines / 100 reports" from its own content while commit 431ccbb from the other box never arrived. The irony is that the branch already carries `*.md merge=union` in .gitattributes precisely so append-only log edits from several boxes reconcile themselves; --ff-only just never reaches it. Switched logs-pull.sh to `pull --no-rebase --no-edit` and made logs-push.sh merge before pushing (a diverged branch rejects a bare push too), both with an explicit -c user.name/user.email so the merge commit succeeds on instances that have no git identity configured. Sessions side was clean in the same run: restored 1 (c3a9f389 back from the instance at 27MB), 34 already current, 0 diverged, 0 in use - the fork resolution held and the store correctly reported itself ahead. Pattern across all three deadlocks (sessions store, instance clone, logs branch): every one assumed a single writer while the system has always had several - .claude-sessions / scripts/logs-pull.sh, scripts/logs-push.sh
 
 
255
  2026-08-26 - Ported bluscript_zero parity into bluscriptCP: Muon via ShardedMuon (CP_OPTIMIZER=muon), per-group WSD LR pairs, step_order biggest-last, grad-norm L2 fix, CP_MAX_LR/MIN_LR, CP_ZERO_BF16_REDUCE, CP_DIAG per-micro loss - CP / Scripts/Blutrain/bluscriptCP.cpp, Makefile
256
  2026-08-26 - Recovered the 600M step-500 reach probe (denom 10.73, reach >=3061) from the destroyed instance vault and pinned its provenance: it describes fp32+AdamW+DDP+cosine on edufineweb, NOT the Muon/ZeRO/bf16/WSD/FLUX base - CP / probe_run4, needle_score.py
257
  2026-08-28 - Third --ff-only deadlock, this time on the logs branch: logs-pull.sh ran `git pull --ff-only`, which cannot invoke a merge driver, so once the laptop and an instance had both pushed logs the branch diverged and every pull died with "Not possible to fast-forward" then silently fell back to the LOCAL copy - the vault reported "242 unique lines / 100 reports" from its own content while commit 431ccbb from the other box never arrived. The irony is that the branch already carries `*.md merge=union` in .gitattributes precisely so append-only log edits from several boxes reconcile themselves; --ff-only just never reaches it. Switched logs-pull.sh to `pull --no-rebase --no-edit` and made logs-push.sh merge before pushing (a diverged branch rejects a bare push too), both with an explicit -c user.name/user.email so the merge commit succeeds on instances that have no git identity configured. Sessions side was clean in the same run: restored 1 (c3a9f389 back from the instance at 27MB), 34 already current, 0 diverged, 0 in use - the fork resolution held and the store correctly reported itself ahead. Pattern across all three deadlocks (sessions store, instance clone, logs branch): every one assumed a single writer while the system has always had several - .claude-sessions / scripts/logs-pull.sh, scripts/logs-push.sh
258
+ 2026-08-29 - CP_INIT_FROM weights-only checkpoint branching (loads before optimizer construction so fp32 masters derive from the loaded weights, not random init - the late CP_CKPT_WEIGHTS_ONLY path clobbered weights on the first step: 2.481 -> 11.084); fixed missing CP_CKPT_DIR getenv, added not-found aborts, added dtype conversion; run8 1.06B proxy trained 1000 steps to val 2.817 and PASSED the IN2 sufficiency gate (denom 12.31 nats, reach boundary inside window, p0 score 0.1525) - Context_Parallelism / bluscriptCP.cpp, checkpointing/Checkpointing.h, needle_sweep.sh, probe_1b.sh
docs/Claude Reports/ClaudeReport_session_2026-08-29_init_from_and_run8_probe.md ADDED
@@ -0,0 +1,125 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # CP_INIT_FROM, checkpoint-branching fixes, and the run8 reach probe
2
+
3
+ 2026-08-29 - Built the weights-only checkpoint-branching path (CP_INIT_FROM),
4
+ fixed three defects found by testing it, and measured the 1.06B proxy's reach.
5
+ Boxes: metallic-bird-thinks-fin-03 (4x H200, torn down) then
6
+ light-sun-speaks-fin-02 (1x H200). Workspace: BluTrain / dist/Context_Parallelism.
7
+
8
+ ## The 1.06B proxy run (run8) completed
9
+
10
+ `d1536 / L36 / Q12 / KV4 / hd128 / FFN4096 / tying OFF` = 1.0606B params
11
+ (Muon 905,969,664 over 216 matrices + AdamW groups 77,266,944 embedding and
12
+ 77,388,288 lm_head+norms). Muon + AdamW, ZeRO-1 v2 + symmetric, bf16, WSD
13
+ 100/700/200, all groups peaked at 1e-3.
14
+
15
+ | step | val_loss |
16
+ |---|---|
17
+ | 0 | 11.095013 |
18
+ | 250 | 3.940885 |
19
+ | 500 | 3.156706 |
20
+ | 750 | 2.988807 |
21
+ | 999 | **2.817160** |
22
+
23
+ 1000 steps in 1h39m at mean 176,067 tok/s, peak 95,437 MB, B=4 on 4x H200.
24
+ Train loss 8.719 (steps 0-24) -> 2.625 (950-974), monotone throughout.
25
+
26
+ **B=4 beat B=2 by 4.7%** (176,600 vs 168,750 tok/s) - the opposite of what a
27
+ 3-step sweep said (162k vs 165k). Three steps is too short: the allocator is
28
+ still growing its pool and the larger micro-batch has not reached steady state,
29
+ which biases *against* the larger B. Measure throughput over tens of steps.
30
+
31
+ **LR must be equal for Muon and AdamW.** CP defaulted max_lr=6e-4 with
32
+ muon_lr=1e-3, so the 906M body took 1.67x larger steps than the embeddings.
33
+ MATCH_RMS_ADAMW exists precisely to put them on one scale - ShardedMuon.h says
34
+ "setting both equal gives both groups the same step size and the ONLY difference
35
+ is the update direction" - and bluscript_zero sets both to 1e-3. Fixed with
36
+ CP_MAX_LR=1e-3 CP_MIN_LR=1e-4; the first 507-step run was discarded.
37
+
38
+ ## CP_INIT_FROM: branching a new run from a trained base
39
+
40
+ The need: a long-context extension starts from base WEIGHTS but must not inherit
41
+ the optimizer state, which is ZeRO-sharded to the world size that wrote it. A
42
+ normal resume therefore cannot cross a GPU-count change, an optimizer change
43
+ (ZeRO -> no ZeRO), or a dp<->cp reshard.
44
+
45
+ **Resume and initialize-from are NOT the same operation.** Resume restores every
46
+ piece of state together, so ordering does not matter - `load_state` overwrites the
47
+ optimizer's fp32 masters after the fact. Initialize-from has no saved optimizer
48
+ state, so the masters must be DERIVED from the loaded weights, which means the
49
+ weights must land BEFORE any optimizer is constructed.
50
+
51
+ The first attempt (`CP_CKPT_WEIGHTS_ONLY`) loaded at the resume site, line 2141,
52
+ long after both optimizer construction sites (ZeRO at 1775, nn::AdamW at 1824).
53
+ Every one of them snapshots fp32 masters at construction, so the masters held the
54
+ random init and the first step() published them back over the loaded weights:
55
+
56
+ step 1000 loss 2.481 <- weights loaded correctly
57
+ step 1001 loss 11.084 <- masters clobbered them
58
+
59
+ Failing for exactly one step is worse than failing outright. `CP_INIT_FROM` now
60
+ loads right after `model.parameters()` (line 1612), before both optimizers, and
61
+ `CP_CKPT_WEIGHTS_ONLY` is retired with an error pointing at it.
62
+
63
+ **Why this is the universal mechanism:** only replicated, T-independent PARAMETERS
64
+ move. `cos_sin_cache` is a plain member, never `register_parameter`'d, so changing
65
+ T or YARN_SCALE does not change the parameter count. Verified across both
66
+ optimizer paths - ZeRO-1+Muon and plain AdamW both start at loss 2.753656,
67
+ identical to 6 decimals, and stay in the 2.7-3.0 band instead of jumping to 11.
68
+
69
+ ## Three defects found by testing, not by reading
70
+
71
+ 1. **`CP_CKPT_DIR` did not exist.** I wrote a comment telling the user to set it
72
+ without implementing the getenv, so the loader always read the default
73
+ directory, found nothing, and trained from random init. Silently.
74
+ 2. **No abort on "checkpoint not found".** A weights-only load that finds nothing
75
+ is always a mistake; it now dies. Same for `CP_CKPT=0`, where the resume block
76
+ is skipped entirely and the load would never happen.
77
+ 3. **`copy_: dtype mismatch`.** The weights-only loader used `p.copy_(loaded)`
78
+ with no conversion. run8 stores bf16 (CP_BF16=1); the reach probe runs fp32.
79
+ Crossing precision is exactly what this path exists for, so it now converts
80
+ via `as_type` when dtypes differ. The ordinary `load_checkpoint` still has the
81
+ limitation, but a resume always matches training precision.
82
+
83
+ ## run8 reach probe - the IN2 sufficiency gate
84
+
85
+ `probe_1b.sh` (new; the 600M one is hardcoded to L24/tying=1 with stale
86
+ /root/BluTrain paths). `needle_sweep.sh` gained `INIT_FROM=<path>` because it
87
+ hardcoded `CP_CKPT_RESUME`, and a forward-only eval of a ZeRO/Muon base builds a
88
+ plain nn::AdamW that would try to parse ShardedMuon's state.
89
+
90
+ floor(p1)=1.5181 absent=13.8241 null=12.7732 denom=12.3060 nats
91
+
92
+ | variant | nll | se | score | sig |
93
+ |---|---|---|---|---|
94
+ | p0 | 11.9470 | 0.2510 | 0.1525 | YES |
95
+ | p0.25 | 7.6774 | 0.5615 | 0.4995 | YES |
96
+ | p0.5 | 5.2811 | 0.4232 | 0.6942 | YES |
97
+ | p0.75 | 3.3513 | 0.6088 | 0.8510 | YES |
98
+ | p1 | 1.5181 | 0.0202 | 1.0000 | YES |
99
+
100
+ **Both gates pass.** denom 12.31 nats against the 0.5-nat withholding bar, and
101
+ better than run100 (8.84) and the 600M (10.73). The reach boundary is inside the
102
+ window: p0 scores only 0.1525, so a large failing region remains at distance -
103
+ which is the room IN2 supervision is meant to improve. All five points
104
+ significant at BLOCKS=3, curve monotone.
105
+
106
+ ## Still open
107
+
108
+ 1. **The plan's R=8174 is run100's, not run8's.** The primary claim region ("bins
109
+ beyond R(T)") must be re-derived for this base, at T=16384, after the
110
+ extension. The ~13% arm-C fraction is anchored to the old number.
111
+ 2. **run8 is 4k-native; IN2 trains at T=16384.** The YARN/LongRoPE extension has
112
+ to happen first - that is what CP_INIT_FROM was built for and it is the next
113
+ step.
114
+ 3. **in2_data is gone** with the earlier instance; regenerate with in2_gen.py
115
+ (numpy is only in the venv, not system python).
116
+ 4. The plan's budget assumed 4x H200.
117
+
118
+ ## Environment traps recorded
119
+
120
+ - `hf` is in a venv, NOT on PATH: `/root/Adhi/BluTrain/envv/bin/hf`.
121
+ - `pgrep -f <pattern>` matches the searching shell's own argv when the pattern
122
+ appears in the command line - use `pgrep -f 'make[ ]libtensor'`. This cost a
123
+ wasted wait loop and a chained builder that spun forever.
124
+ - Downloading FLUX: `hf download navingv/flux --repo-type=dataset` (20.2 GB) -
125
+ it is a separate public dataset, not in the BluScriptCP repo.