AbstractPhil commited on
Commit
7beb9c5
·
verified ·
1 Parent(s): b7f274b

trainers/: full replication kit — 11 scrubbed standalone trainers + README (battery map, replication discipline, repair flags, gauge guidance, history caveats)

Browse files
trainers/README.md ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # trainers/ — replication kit for the aleph-splat measurement record
2
+
3
+ Every number in this repo's `battery/` directories was produced by a
4
+ trainer in this directory, on a single consumer GPU (RTX 4090 class).
5
+ Each trainer is a standalone file: `python <trainer>.py` with `HF_TOKEN`
6
+ in the environment if you want results shipped to your own hub repo
7
+ (uploads are optional — remove the `upload_file` calls or leave the token
8
+ unset and save locally).
9
+
10
+ ## The trainers, mapped to their result directories
11
+
12
+ | trainer | produces | battery dir |
13
+ |---|---|---|
14
+ | `splat_math_battery.py` | the B1–B9 static mathematical battery | `battery/` (splat_battery*.json) |
15
+ | `ar_differentiation_bed.py` | the byte-AR bed all recall batteries build on (ByteLM, CausalHUB, AlephAddress) — also runs its own wikitext arms | (dependency; `./data/ar_ckpts` specimens) |
16
+ | `geolip_vitals.py` | vitals helpers imported by the bed | (dependency) |
17
+ | `v1_rank_army_battery.py` | rank-ladder × army-ladder under LDS filtering pressure | `battery/v1_rank_army/` |
18
+ | `v1b_recall_army_battery.py` | the recall battery: rank-controlled in-context binding; the K/D ladders; clean-mask + selection gauges | `battery/v1b_recall_army/` |
19
+ | `v1d_svae_graft.py` | SVAE-transformer graft arms (sphere/lens/α-grammar) | `battery/v1d_svae_graft/` |
20
+ | `v1e_knit_battery.py` | conv-knit + patchwork graft arms | `battery/v1e_knit/` |
21
+ | `v1f_consult_tier.py` | corrector tier: rownorm/auxalpha/strobe3/unigram/padfix | `battery/v1f_consult/` |
22
+ | `v1g_svae_overlap_sweep.py` | 16-noise overlap sweep on the SVAE substrate | `battery/v1g_svae_overlap/` |
23
+ | `v1i_validity_batteries.py` | dtype / optimizer / schedule batteries (Muon, Lion, fp8-sim…) | `battery/v1i_validity/` |
24
+ | `trigram_bitspace_sweep.py` | byte-embedding bit-space analysis (CPU) | (analysis JSON) |
25
+
26
+ ## Replication discipline — read before running
27
+
28
+ 1. **Standing repairs (use them):** `ALEPH_TRI_PAD_FIX=1` (trigram pad
29
+ conflation fix; worth +.05–.11 and unlocks address consumption) and
30
+ `V1B_CLEAN_MASK=1` (drops window-truncated answer positions from the
31
+ 4×-weighted loss). All *clean-protocol* numbers in the record use both.
32
+ 2. **Gauge choice matters:** `masked_eval` scores ~6% information-
33
+ theoretically unanswerable positions (window truncation) — it fakes
34
+ plateaus for near-ceiling models. `selection_eval` filters to the
35
+ answerable subset and additionally decomposes errors into wrong-pair
36
+ retrieval vs no-retrieval. Headline from `selection_eval`.
37
+ 3. **History caveat:** result directories produced before 2026-08-10 ran
38
+ without the pad fix and with the truncation-era gauge. Internal
39
+ comparisons within a battery remain valid (shared conditions); do not
40
+ compare absolute numbers across the repair boundary.
41
+ 4. **Optimizer:** the aleph mechanism is ~20× more optimizer-sensitive
42
+ than softmax attention; momentum-geometric optimizers (Muon-style,
43
+ SGD-momentum) beat Adam by ~.09 on it. `v1i_validity_batteries.py`
44
+ contains the Muon/Lion implementations used.
45
+ 5. **Precision:** train fp32/bf16; never through fp8; fp16 requires the
46
+ dtype-aware clamps already present in these files.
47
+ 6. **Data:** synthetic streams are generated deterministically by the
48
+ trainers themselves (seeded); `v1g` pulls the public
49
+ `AbstractPhil/geolip-svae-batteries` noise battery; the wikitext arms
50
+ pull `Salesforce/wikitext` parquet. Default data root is `./data`.
51
+
52
+ All findings, including the failures and the two retracted constants,
53
+ are summarized in the repo README and the technical companion.
trainers/__pycache__/ar_differentiation_bed.cpython-312.pyc ADDED
Binary file (39.8 kB). View file
 
trainers/__pycache__/geolip_vitals.cpython-312.pyc ADDED
Binary file (17.1 kB). View file
 
trainers/__pycache__/splat_math_battery.cpython-312.pyc ADDED
Binary file (47.3 kB). View file
 
trainers/__pycache__/trigram_bitspace_sweep.cpython-312.pyc ADDED
Binary file (12 kB). View file
 
trainers/__pycache__/v1_rank_army_battery.cpython-312.pyc ADDED
Binary file (15.9 kB). View file
 
trainers/__pycache__/v1b_recall_army_battery.cpython-312.pyc ADDED
Binary file (23 kB). View file
 
trainers/__pycache__/v1d_svae_graft.cpython-312.pyc ADDED
Binary file (14.7 kB). View file
 
trainers/__pycache__/v1e_knit_battery.cpython-312.pyc ADDED
Binary file (16.2 kB). View file
 
trainers/__pycache__/v1f_consult_tier.cpython-312.pyc ADDED
Binary file (18.8 kB). View file
 
trainers/__pycache__/v1g_svae_overlap_sweep.cpython-312.pyc ADDED
Binary file (14 kB). View file
 
trainers/__pycache__/v1i_validity_batteries.cpython-312.pyc ADDED
Binary file (25.2 kB). View file
 
trainers/ar_differentiation_bed.py ADDED
@@ -0,0 +1,499 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ar_differentiation_bed.py — THE FOCUS (2026-07-09 redirect, AbstractPhil verbatim):
2
+ "refining the autoregressive techniques for differentiation rather than attempting
3
+ to just mash numbers together."
4
+
5
+ Differentiation is cultivated by PREDICTIVE pressure along the sequence — the
6
+ address parameterizing the next-byte distribution (Law 2: chain-rule advantage pays
7
+ ONLY where the composed address directly parameterizes the predictive distribution).
8
+ This bed puts the aleph in the autoregressive gradient path and measures what
9
+ differentiates. It is the Law-2 construction (codebook-pressure C3) + Tree 3d in
10
+ one harness; the Jun-19 "discuss before building" gate was resolved by the redirect.
11
+
12
+ Byte-level causal LM on wikitext-2-raw (HF parquet, CDN-fast), block 256. ARMS:
13
+ sdpa — standard causal transformer control (matched trunk).
14
+ hub — attention replaced by CAUSAL HUB: linear attention whose feature map
15
+ is the 2K-oriented aleph address, prefix-sum memories (no selection
16
+ event; O(n*K*d)). Differentiation cultivated INSIDE attention.
17
+ addr_head — sdpa trunk, but the OUTPUT HEAD reads ONLY the signed aleph
18
+ coefficient vector w_k = sinh(u_k)/sum_j cosh(u_j) of the final
19
+ hidden state (K -> 256 logits). The address MUST carry every bit of
20
+ next-byte information — the hardest Law-2 bottleneck.
21
+
22
+ JUDGED BY: val bits-per-byte per arm (task) + CULTIVATION VITALS on every aleph
23
+ codebook (readouts, never losses): axis aliveness/hppl, drift-from-init +
24
+ binding fraction @0.29154, winner-|cos| saturation (sign-code emergence), shadow
25
+ path diversity (fixed high-bits hash). Never by recon.
26
+
27
+ Riders: pure Adam wd=0; no BN/Dropout/GAP on geometric paths; orthogonal init;
28
+ Colab-cell-safe (paste-ahead imports, no bare argparse, no __file__ reliance);
29
+ GPU-only for verdict runs; data_root OUTSIDE the mind repo.
30
+
31
+ Terminal: python ar_differentiation_bed.py # shapes/parse smoke
32
+ python ar_differentiation_bed.py --train # verdict run
33
+ Colab: paste geolip_vitals.py cell, then this file (smoke auto-runs),
34
+ then train(steps=2000, data_root="/content/data") in the next cell.
35
+ """
36
+ from __future__ import annotations
37
+ import math
38
+ import os
39
+ import torch
40
+ import torch.nn as nn
41
+ import torch.nn.functional as F
42
+
43
+ if "anchor_drift" not in globals():
44
+ try:
45
+ from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
46
+ except ImportError:
47
+ _here = globals().get("__file__")
48
+ if _here is not None:
49
+ import sys, pathlib
50
+ sys.path.insert(0, str(pathlib.Path(_here).parent))
51
+ from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
52
+ else:
53
+ raise ImportError(
54
+ "geolip_vitals not found — paste/run its cell first, or "
55
+ "keep geolip_vitals.py beside this file (ships in this directory).")
56
+
57
+ VOCAB = 256 # bytes
58
+
59
+
60
+ # ------------------------------------------------------------------ aleph address
61
+ def _super_fibonacci_s3(n: int) -> torch.Tensor:
62
+ """Near-uniform unit quaternions (Alexa CVPR'22; constants per canon) —
63
+ starts the codebook INSIDE the RP^3 attractor basin. D=4 only."""
64
+ PHI, PSI = math.sqrt(2.0), 1.533751168755204288118041
65
+ i = torch.arange(n, dtype=torch.float64)
66
+ s = (i + 0.5) / n
67
+ r, R = torch.sqrt(s), torch.sqrt(1.0 - s)
68
+ a, b = 2 * math.pi * i / PHI, 2 * math.pi * i / PSI
69
+ q = torch.stack([r * torch.sin(a), r * torch.cos(a),
70
+ R * torch.sin(b), R * torch.cos(b)], dim=-1)
71
+ return F.normalize(q, dim=-1).float()
72
+
73
+
74
+ class AlephAddress(nn.Module):
75
+ """Closed-form aleph over 2K oriented half-axes (canon/aleph_core.md).
76
+ signed(x): (..., K) w_k = sinh(u_k)/sum_j cosh(u_j) — the Law-2 head feature.
77
+ oriented(x): ((..., K), (..., K)) positive halves of the 2K softmax — HUB map."""
78
+
79
+ def __init__(self, K: int, D: int, tau: float = 0.1, init: str = "random"):
80
+ super().__init__()
81
+ self.K, self.D, self.tau = K, D, tau
82
+ if init == "fibonacci":
83
+ assert D == 4, "fibonacci init lives on S^3 (D=4)"
84
+ A = _super_fibonacci_s3(K)
85
+ else:
86
+ A = F.normalize(torch.randn(K, D), dim=-1)
87
+ self.codebook = nn.Parameter(A)
88
+ self.register_buffer("home", self.codebook.detach().clone())
89
+
90
+ def _u(self, x):
91
+ A = F.normalize(self.codebook, dim=-1)
92
+ return (F.normalize(x, dim=-1) @ A.transpose(-1, -2)) / self.tau
93
+
94
+ def oriented(self, x):
95
+ u = self._u(x)
96
+ m = u.abs().amax(dim=-1, keepdim=True)
97
+ ep, en = torch.exp(u - m), torch.exp(-u - m)
98
+ Z = (ep + en).sum(dim=-1, keepdim=True)
99
+ return ep / Z, en / Z
100
+
101
+ def signed(self, x):
102
+ u = self._u(x)
103
+ m = u.abs().amax(dim=-1, keepdim=True)
104
+ ep, en = torch.exp(u - m), torch.exp(-u - m)
105
+ return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
106
+
107
+ def signed_at(self, x, taus):
108
+ """Multi-tau stroboscope (rule of 3): signed coefficients at several
109
+ temperatures, concatenated — softer taus keep the vector dense while a
110
+ hard tau supplies the sign-code sharpness. v2 refinement (b)."""
111
+ A = F.normalize(self.codebook, dim=-1)
112
+ cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
113
+ outs = []
114
+ for t in taus:
115
+ u = cos / t
116
+ m = u.abs().amax(dim=-1, keepdim=True)
117
+ ep, en = torch.exp(u - m), torch.exp(-u - m)
118
+ outs.append((ep - en) / (ep + en).sum(dim=-1, keepdim=True))
119
+ return torch.cat(outs, dim=-1)
120
+
121
+ def m_hat(self, x):
122
+ """Closed-form soft read (decoders read M_hat, never M). v2 control (c)."""
123
+ u = self._u(x)
124
+ m = u.abs().amax(dim=-1, keepdim=True)
125
+ ep, en = torch.exp(u - m), torch.exp(-u - m)
126
+ A = F.normalize(self.codebook, dim=-1)
127
+ return ((ep - en) @ A) / (ep + en).sum(dim=-1, keepdim=True)
128
+
129
+ def m_hard_ste(self, x):
130
+ """Canon hard mode: M_hard = sign(cos_win) * A[win], straight-through to
131
+ the soft read — forward fully discrete SIGN CODE, backward soft gradient.
132
+ Legal per theme A (reconstructive sign code, not a one-hot roster pick)."""
133
+ u = self._u(x)
134
+ soft = self.m_hat(x)
135
+ win = u.abs().argmax(dim=-1)
136
+ A = F.normalize(self.codebook, dim=-1)
137
+ sign = torch.sign(torch.gather(u, -1, win.unsqueeze(-1))).squeeze(-1)
138
+ hard = sign.unsqueeze(-1) * A[win]
139
+ return hard + soft - soft.detach()
140
+
141
+ @torch.no_grad()
142
+ def vitals(self, x_sample) -> dict:
143
+ u = self._u(x_sample.reshape(-1, x_sample.shape[-1]))
144
+ p, n = self.oriented(x_sample.reshape(-1, x_sample.shape[-1]))
145
+ two_k = torch.cat([p, n], dim=-1)
146
+ win = two_k.argmax(dim=-1)
147
+ cos_win = (u.abs().amax(dim=-1) * self.tau) # winner |cos| — sign-code sat.
148
+ d = anchor_drift(self.codebook, self.home)
149
+ return {"drift": round(d["mean"], 4),
150
+ "binding_frac": round(d["binding_fraction"], 4),
151
+ "aliveness": axis_aliveness(two_k),
152
+ "win_cos_mean": round(cos_win.mean().item(), 4),
153
+ "paths": path_diversity(win)}
154
+
155
+
156
+ # ------------------------------------------------------------------------- blocks
157
+ class CausalSDPA(nn.Module):
158
+ def __init__(self, d: int, heads: int = 4):
159
+ super().__init__()
160
+ self.h = heads
161
+ self.qkv = nn.Linear(d, 3 * d, bias=False)
162
+ self.o = nn.Linear(d, d, bias=False)
163
+ nn.init.orthogonal_(self.qkv.weight); nn.init.orthogonal_(self.o.weight)
164
+
165
+ def forward(self, x):
166
+ B, n, d = x.shape
167
+ q, k, v = self.qkv(x).chunk(3, dim=-1)
168
+ q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2) for t in (q, k, v))
169
+ y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
170
+ return self.o(y.transpose(1, 2).reshape(B, n, d))
171
+
172
+
173
+ class CausalHUB(nn.Module):
174
+ """Causal aleph linear attention: prefix-sum memories over the two K-wide
175
+ halves of the oriented address; 2K never materialized; no selection event."""
176
+
177
+ def __init__(self, d: int, K: int = 32, D: int = 4, tau: float = 0.1):
178
+ super().__init__()
179
+ self.addr = AlephAddress(K, D, tau)
180
+ self.q = nn.Linear(d, D, bias=False)
181
+ self.k = nn.Linear(d, D, bias=False)
182
+ self.v = nn.Linear(d, d, bias=False)
183
+ self.o = nn.Linear(d, d, bias=False)
184
+ for m in (self.q, self.k, self.v, self.o):
185
+ nn.init.orthogonal_(m.weight)
186
+
187
+ def forward(self, x):
188
+ qp, qn = self.addr.oriented(self.q(x)) # (B, n, K)
189
+ kp, kn = self.addr.oriented(self.k(x))
190
+ v = self.v(x) # (B, n, d)
191
+ Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
192
+ Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
193
+ zp = torch.cumsum(kp, dim=1)
194
+ zn = torch.cumsum(kn, dim=1)
195
+ num = torch.einsum("bnk,bnkd->bnd", qp, Sp) + torch.einsum("bnk,bnkd->bnd", qn, Sn)
196
+ den = (qp * zp).sum(-1, keepdim=True) + (qn * zn).sum(-1, keepdim=True)
197
+ if den.dtype in (torch.float32, torch.float64):
198
+ cl = 1e-12
199
+ else: # fp16/bf16: 1e-12 flushes to 0 (landmine)
200
+ cl = torch.finfo(den.dtype).tiny * 8
201
+ return self.o(num / den.clamp_min(cl))
202
+
203
+
204
+ class MslRelay(nn.Module):
205
+ """Depth-composition unit (chain-rule probe): multi-slot M_hat read entering
206
+ the trunk as a NEAR-ZERO gated residual (gate init -3.0, sigma~0.047 — theme D:
207
+ geometry enters as a nudge and grows only if it earns gradient)."""
208
+
209
+ def __init__(self, d: int, n_slots: int = 16, K: int = 64):
210
+ super().__init__()
211
+ self.n_slots = n_slots
212
+ self.proj = nn.Linear(d, n_slots * 4, bias=False)
213
+ self.out = nn.Linear(n_slots * 4, d, bias=False)
214
+ nn.init.orthogonal_(self.proj.weight)
215
+ nn.init.orthogonal_(self.out.weight)
216
+ self.addr = AlephAddress(K, 4)
217
+ self.gate = nn.Parameter(torch.tensor(-3.0))
218
+
219
+ def forward(self, x):
220
+ B, n, _ = x.shape
221
+ slots = self.proj(x).view(B, n, self.n_slots, 4)
222
+ m = self.addr.m_hat(slots).reshape(B, n, -1)
223
+ return x + self.gate.sigmoid() * self.out(m)
224
+
225
+
226
+ class Block(nn.Module):
227
+ def __init__(self, d: int, attn: nn.Module):
228
+ super().__init__()
229
+ self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
230
+ self.attn = attn
231
+ self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
232
+
233
+ def forward(self, x):
234
+ x = x + self.attn(self.n1(x))
235
+ return x + self.mlp(self.n2(x))
236
+
237
+
238
+ class ByteLM(nn.Module):
239
+ def __init__(self, arm: str, d: int = 192, layers: int = 4, block: int = 256,
240
+ K: int = 32, D: int = 4):
241
+ super().__init__()
242
+ # "<arm>_tri" suffix = trigram byte embedding (AlephLM byte_emb x3 lineage):
243
+ # token embedding is the sum of embeddings of bytes t, t-1, t-2.
244
+ self.trigram = arm.endswith("_tri")
245
+ if self.trigram:
246
+ arm = arm[:-4]
247
+ # "_fib" = super-Fibonacci S^3 codebook init (basin test: starts INSIDE
248
+ # the RP^3 attractor; primary observable is init->final geodesic drift).
249
+ self.fib = arm.endswith("_fib")
250
+ if self.fib:
251
+ arm = arm[:-4]
252
+ # "relay*" = stacked addresses in depth: MslRelay after every block.
253
+ # relay -> sdpa trunk + standard head; relay_msl64 -> + addressed head.
254
+ self.use_relay = arm.startswith("relay")
255
+ if arm == "relay":
256
+ arm = "sdpa"
257
+ elif arm == "relay_msl64":
258
+ arm = "addr_msl64"
259
+ self.arm, self.block = arm, block
260
+ # ALEPH_TRI_PAD_FIX=1 (opt-in, 2026-08-09 audit): trigram shift
261
+ # padding used byte 0 — a LEGAL byte (discovery #16: hard-zero
262
+ # padding starves/conflates). Fix = dedicated pad row 256.
263
+ self.pad_fix = os.environ.get("ALEPH_TRI_PAD_FIX") == "1"
264
+ pv = VOCAB + 1 if (self.trigram and self.pad_fix) else VOCAB
265
+ self.emb = nn.Embedding(VOCAB, d)
266
+ if self.trigram:
267
+ self.emb1 = nn.Embedding(pv, d)
268
+ self.emb2 = nn.Embedding(pv, d)
269
+ self.pos = nn.Parameter(torch.zeros(1, block, d) + 0.01 * torch.randn(1, block, d))
270
+ mk_attn = (lambda: CausalHUB(d, K, D)) if arm == "hub" else (lambda: CausalSDPA(d))
271
+ self.blocks = nn.ModuleList([Block(d, mk_attn()) for _ in range(layers)])
272
+ if self.use_relay:
273
+ self.relays = nn.ModuleList([MslRelay(d) for _ in range(layers)])
274
+ self.nf = nn.LayerNorm(d)
275
+ if arm == "addr_head":
276
+ self.head_addr = AlephAddress(K, d) # v1: codebook in model dim — COLLAPSED
277
+ self.head = nn.Linear(K, VOCAB, bias=True)
278
+ elif arm in ("addr_d4", "addr_3tau", "addr_mhat"):
279
+ # v2 refinements: LOW-D HOME — learned projection to the canon D=4 home
280
+ # before addressing (mirrors the healthy HUB arms), K=64.
281
+ self.head_proj = nn.Linear(d, 4, bias=False)
282
+ nn.init.orthogonal_(self.head_proj.weight)
283
+ self.head_addr = AlephAddress(64, 4)
284
+ if arm == "addr_d4":
285
+ self.head = nn.Linear(64, VOCAB, bias=True) # w alone, D=4 home
286
+ elif arm == "addr_3tau":
287
+ self.taus = (0.05, 0.1, 0.3) # rule-of-3 strobe
288
+ self.head = nn.Linear(64 * 3, VOCAB, bias=True)
289
+ else: # addr_mhat
290
+ self.head = nn.Linear(4, VOCAB, bias=True) # tightest: M_hat
291
+ elif arm.startswith("addr_msl"):
292
+ # v3: MULTI-SLOT heads — the 16s funnel widening: P parallel D=4 slots
293
+ # over a SHARED codebook. addr_msl consumes the reconstructive M_hat per
294
+ # slot (Px4 dims); addr_msl_w consumes signed w per slot (Px64) — tests
295
+ # whether slot-parallel consumption alone rescues the coefficient path.
296
+ # addr_msl<P> = slot-count dose-response. addr_mslh<P> = HARD sign-code
297
+ # consumption (straight-through M_hard per slot).
298
+ self.hard = arm.startswith("addr_mslh")
299
+ if arm in ("addr_msl", "addr_msl_w"):
300
+ self.n_slots = 16
301
+ else:
302
+ self.n_slots = int(arm[len("addr_mslh" if self.hard else "addr_msl"):])
303
+ self.head_proj = nn.Linear(d, self.n_slots * 4, bias=False)
304
+ nn.init.orthogonal_(self.head_proj.weight)
305
+ self.head_addr = AlephAddress(
306
+ 64, 4, init="fibonacci" if self.fib else "random")
307
+ width = self.n_slots * (64 if arm == "addr_msl_w" else 4)
308
+ self.head = nn.Linear(width, VOCAB, bias=True)
309
+ elif arm == "addr_3tau_mhat":
310
+ # v3: combine the two v2 winners — 3-tau stroboscope + reconstructive read.
311
+ self.head_proj = nn.Linear(d, 4, bias=False)
312
+ nn.init.orthogonal_(self.head_proj.weight)
313
+ self.head_addr = AlephAddress(64, 4)
314
+ self.taus = (0.05, 0.1, 0.3)
315
+ self.head = nn.Linear(64 * 3 + 4, VOCAB, bias=True)
316
+ else:
317
+ self.head = nn.Linear(d, VOCAB, bias=True)
318
+ self._last_h = None
319
+
320
+ def forward(self, idx):
321
+ x = self.emb(idx)
322
+ if self.trigram: # past-only shifts — causality preserved
323
+ pv = VOCAB if self.pad_fix else 0
324
+ x = x + self.emb1(F.pad(idx, (1, 0), value=pv)[:, :-1]) \
325
+ + self.emb2(F.pad(idx, (2, 0), value=pv)[:, :-2])
326
+ x = x + self.pos[:, : idx.shape[1]]
327
+ if self.use_relay:
328
+ for b, r in zip(self.blocks, self.relays):
329
+ x = r(b(x))
330
+ else:
331
+ for b in self.blocks:
332
+ x = b(x)
333
+ h = self.nf(x)
334
+ self._last_h = h.detach()
335
+ if self.arm == "addr_head":
336
+ return self.head(self.head_addr.signed(h))
337
+ if self.arm == "addr_d4":
338
+ return self.head(self.head_addr.signed(self.head_proj(h)))
339
+ if self.arm == "addr_3tau":
340
+ return self.head(self.head_addr.signed_at(self.head_proj(h), self.taus))
341
+ if self.arm == "addr_mhat":
342
+ return self.head(self.head_addr.m_hat(self.head_proj(h)))
343
+ if self.arm.startswith("addr_msl"):
344
+ B, n, _ = h.shape
345
+ slots = self.head_proj(h).view(B, n, self.n_slots, 4)
346
+ if self.arm == "addr_msl_w":
347
+ feats = self.head_addr.signed(slots).reshape(B, n, -1)
348
+ elif getattr(self, "hard", False):
349
+ feats = self.head_addr.m_hard_ste(slots).reshape(B, n, -1)
350
+ else:
351
+ feats = self.head_addr.m_hat(slots).reshape(B, n, -1)
352
+ return self.head(feats)
353
+ if self.arm == "addr_3tau_mhat":
354
+ p = self.head_proj(h)
355
+ feats = torch.cat([self.head_addr.signed_at(p, self.taus),
356
+ self.head_addr.m_hat(p)], dim=-1)
357
+ return self.head(feats)
358
+ return self.head(h)
359
+
360
+ @torch.no_grad()
361
+ def vitals(self) -> dict:
362
+ out = {}
363
+ if self.arm == "hub":
364
+ for i, b in enumerate(self.blocks):
365
+ if self._last_h is not None:
366
+ out[f"L{i}"] = b.attn.addr.vitals(b.attn.q(self._last_h[:2]))
367
+ elif self.arm == "addr_head" and self._last_h is not None:
368
+ out["head"] = self.head_addr.vitals(self._last_h[:2])
369
+ elif self.arm in ("addr_d4", "addr_3tau", "addr_mhat",
370
+ "addr_3tau_mhat") and self._last_h is not None:
371
+ out["head"] = self.head_addr.vitals(self.head_proj(self._last_h[:2]))
372
+ elif self.arm.startswith("addr_msl") and self._last_h is not None:
373
+ slots = self.head_proj(self._last_h[:2])
374
+ out["head"] = self.head_addr.vitals(
375
+ slots.reshape(*slots.shape[:-1], self.n_slots, 4))
376
+ if self.use_relay and self._last_h is not None:
377
+ for i, r in enumerate(self.relays):
378
+ s = r.proj(self._last_h[:2])
379
+ v = r.addr.vitals(s.reshape(*s.shape[:-1], r.n_slots, 4))
380
+ out[f"relay{i}"] = {"gate": round(r.gate.sigmoid().item(), 4),
381
+ "drift": v["drift"],
382
+ "binding_frac": v["binding_frac"],
383
+ "ppl": round(v["aliveness"]["usage_ppl"], 1)}
384
+ return out
385
+
386
+
387
+ # --------------------------------------------------------------------------- data
388
+ def _wikitext_bytes(data_root: str):
389
+ """wikitext-2-raw as flat uint8 tensors via the HF parquet CDN."""
390
+ from huggingface_hub import hf_hub_download
391
+ import pyarrow.parquet as pq
392
+
393
+ def load(split):
394
+ p = hf_hub_download("Salesforce/wikitext",
395
+ f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
396
+ repo_type="dataset", local_dir=data_root)
397
+ text = "".join(pq.read_table(p).column("text").to_pylist())
398
+ return torch.frombuffer(bytearray(text.encode("utf-8")), dtype=torch.uint8).clone()
399
+
400
+ return load("train"), load("validation")
401
+
402
+
403
+ def _batch(data: torch.Tensor, batch: int, block: int, device, g: torch.Generator):
404
+ ix = torch.randint(0, data.numel() - block - 1, (batch,), generator=g)
405
+ x = torch.stack([data[i:i + block] for i in ix]).long().to(device)
406
+ y = torch.stack([data[i + 1:i + block + 1] for i in ix]).long().to(device)
407
+ return x, y
408
+
409
+
410
+ # -------------------------------------------------------------------- train/smoke
411
+ def train(arms=("sdpa", "hub", "addr_head"), steps: int = 2000, batch: int = 32,
412
+ block: int = 256, device: str = "cuda", data_root: str = "./data",
413
+ seed: int = 0, eval_every: int = 500, save: bool = True):
414
+ """Verdict run — GPU only. Pure Adam wd=0. Reports val bits-per-byte + vitals.
415
+ save=True writes {data_root}/ar_ckpts/{arm}_s{seed}_t{steps}.pt per arm —
416
+ the cultivated codebooks are SPECIMENS for the projective reading instruments."""
417
+ import os
418
+ if device == "cuda" and not torch.cuda.is_available():
419
+ raise RuntimeError("Verdict runs are GPU-only (never CPU-train for accuracy).")
420
+ ckpt_dir = os.path.join(data_root, "ar_ckpts")
421
+ os.makedirs(ckpt_dir, exist_ok=True)
422
+ tr, va = _wikitext_bytes(data_root)
423
+ print(f"data ready: train {tr.numel():,} bytes, val {va.numel():,} bytes", flush=True)
424
+ results = {}
425
+ for arm in arms:
426
+ torch.manual_seed(seed)
427
+ g = torch.Generator().manual_seed(seed)
428
+ model = ByteLM(arm, block=block).to(device)
429
+ n_params = sum(p.numel() for p in model.parameters())
430
+ opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
431
+ for step in range(1, steps + 1):
432
+ x, y = _batch(tr, batch, block, device, g)
433
+ logits = model(x)
434
+ loss = F.cross_entropy(logits.reshape(-1, VOCAB), y.reshape(-1))
435
+ opt.zero_grad(set_to_none=True)
436
+ loss.backward()
437
+ opt.step()
438
+ if step % eval_every == 0 or step == steps:
439
+ model.eval()
440
+ with torch.no_grad():
441
+ losses = []
442
+ for _ in range(20):
443
+ xv, yv = _batch(va, batch, block, device, g)
444
+ lv = F.cross_entropy(model(xv).reshape(-1, VOCAB),
445
+ yv.reshape(-1))
446
+ losses.append(lv.item())
447
+ bpb = sum(losses) / len(losses) / math.log(2)
448
+ print(f"[{arm}] step {step} val_bpb={bpb:.4f} vitals={model.vitals()}",
449
+ flush=True)
450
+ model.train()
451
+ results[arm] = {"val_bpb": bpb, "params": n_params, "vitals": model.vitals()}
452
+ if save:
453
+ path = os.path.join(ckpt_dir, f"{arm}_s{seed}_t{steps}.pt")
454
+ torch.save({"arm": arm, "seed": seed, "steps": steps, "val_bpb": bpb,
455
+ "state_dict": {k: v.cpu() for k, v in
456
+ model.state_dict().items()}}, path)
457
+ print(f"saved specimen: {path}", flush=True)
458
+ print(results, flush=True)
459
+ return results
460
+
461
+
462
+ def smoke():
463
+ """Shapes/parse only — no accuracy claims."""
464
+ x = torch.randint(0, VOCAB, (2, 64))
465
+ for arm in ("sdpa", "hub", "addr_head"):
466
+ m = ByteLM(arm, d=96, layers=2, block=64, K=16)
467
+ logits = m(x)
468
+ assert logits.shape == (2, 64, VOCAB)
469
+ logits.sum().backward()
470
+ # causality check: future byte must not affect past logits
471
+ with torch.no_grad():
472
+ a = m(x)[0, 10]
473
+ x2 = x.clone(); x2[0, 40] = (x2[0, 40] + 7) % 256
474
+ b = m(x2)[0, 10]
475
+ assert torch.allclose(a, b, atol=1e-4), f"{arm} leaks future context"
476
+ print(f"{arm}: OK params={sum(p.numel() for p in m.parameters()):,} "
477
+ f"vitals={m.vitals()}", flush=True)
478
+ print("OK — AR bed smoke passed (verdict run: train() on GPU)", flush=True)
479
+
480
+
481
+ def _in_notebook() -> bool:
482
+ try:
483
+ get_ipython() # type: ignore[name-defined] # noqa: F821
484
+ return True
485
+ except NameError:
486
+ return False
487
+
488
+
489
+ if __name__ == "__main__":
490
+ if _in_notebook():
491
+ smoke()
492
+ print("Notebook mode: call train(steps=2000) in the next cell (GPU).")
493
+ else:
494
+ import argparse
495
+ ap = argparse.ArgumentParser()
496
+ ap.add_argument("--train", action="store_true")
497
+ ap.add_argument("--steps", type=int, default=2000)
498
+ a, _ = ap.parse_known_args()
499
+ train(steps=a.steps) if a.train else smoke()
trainers/geolip_vitals.py ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """geolip_vitals.py — the shared diagnostic harness (progression plan Tree 7a).
2
+ One implementation, every tree imports it. ALL functions are READOUTS: no gradients,
3
+ no losses. CV is a readout, never a force (discovery_catalog #3). Judge addressing by
4
+ drift->0.29154 and CV->0.20, never recon cosine (MANIFEST).
5
+
6
+ Vitals provided:
7
+ anchor_drift — geodesic drift of anchors from init; binding fraction @0.29154
8
+ pentachoron_cv — CM 4-volume CV over random 5-row subsets (geovocab2 import)
9
+ axis_aliveness — oriented-address usage: axes alive, hppl, collapse flag
10
+ gate_stats — gate means vs the 0.012-0.03 band
11
+ path_diversity — unique-path counting, FIXED high-bits hash (low-16 bug is the
12
+ retracted artifact — never use the low bits)
13
+ grad_norm_spread — gradient democracy monitor (orders-of-magnitude spread)
14
+ CVScreen — CV@1000-batch early band screen (<0.30 LOW / .35-.50 MID / >.80 HIGH)
15
+
16
+ Smoke on a torch-capable env: python geolip_vitals.py
17
+ """
18
+ from __future__ import annotations
19
+ import math
20
+ import torch
21
+
22
+ BINDING = 0.29154 # radians; the binding/separation constant (MANIFEST)
23
+ CV_BAND = (0.13, 0.30) # CM CV band (discovery_catalog #4)
24
+ GATE_BAND = (0.012, 0.03) # live invariant candidate (acd_campaign)
25
+ KNUTH32 = 2654435761
26
+
27
+
28
+ # ----------------------------------------------------------------------------- drift
29
+ @torch.no_grad()
30
+ def anchor_drift(current: torch.Tensor, init: torch.Tensor, tol: float = 0.05) -> dict:
31
+ """Geodesic drift (radians) of each row of `current` from its row in `init`,
32
+ both row-normalized. Returns mean/std/per-row drift and the fraction of rows
33
+ within +/-tol of BINDING (the GLFM '46%' readout)."""
34
+ a = torch.nn.functional.normalize(current.float(), dim=-1)
35
+ b = torch.nn.functional.normalize(init.float(), dim=-1)
36
+ cos = (a * b).sum(-1).clamp(-1.0, 1.0)
37
+ drift = torch.arccos(cos)
38
+ frac = ((drift - BINDING).abs() <= tol).float().mean()
39
+ return {"mean": drift.mean().item(), "std": drift.std().item(),
40
+ "per_row": drift, "binding_fraction": frac.item()}
41
+
42
+
43
+ # -------------------------------------------------------------------------------- cv
44
+ @torch.no_grad()
45
+ def _pentachoron_volumes(pts: torch.Tensor) -> torch.Tensor:
46
+ """Batched Cayley-Menger 4-simplex volumes. pts: (B, 5, D) -> (B,) volumes.
47
+ One float64 det over all samples (vol^2 = -det(CM)/9216 for n=4). Built-in
48
+ per the 2026-07-11 rider amendment (AbstractPhil: the installed geovocab2 path is a
49
+ per-sample class call — too slow for a vitals loop); geovocab2 stays the
50
+ reference implementation, parity-checked via cv_reference_check()."""
51
+ B = pts.shape[0]
52
+ d2 = torch.cdist(pts.double(), pts.double()).pow(2) # (B,5,5)
53
+ cm = torch.ones(B, 6, 6, dtype=torch.float64, device=pts.device)
54
+ cm[:, 0, 0] = 0.0
55
+ cm[:, 1:, 1:] = d2
56
+ det = torch.linalg.det(cm)
57
+ return (-det / 9216.0).clamp_min(0.0).sqrt().float()
58
+
59
+
60
+ @torch.no_grad()
61
+ def pentachoron_cv(rows: torch.Tensor, n_samples: int = 200,
62
+ generator: torch.Generator | None = None) -> float:
63
+ """CV (std/mean) of Cayley-Menger 4-simplex volumes over n_samples random
64
+ 5-row subsets. Rows are row-normalized before measurement. Uses the built-in
65
+ batched CM (float64 det); validate against geovocab2 with
66
+ cv_reference_check() after any change to the volume math."""
67
+ x = torch.nn.functional.normalize(rows.float(), dim=-1)
68
+ n = x.shape[0]
69
+ if n < 5:
70
+ raise ValueError(f"pentachoron_cv needs >=5 rows, got {n}")
71
+ g = generator or torch.Generator(device="cpu").manual_seed(0)
72
+ idx = torch.stack([torch.randperm(n, generator=g)[:5]
73
+ for _ in range(n_samples)]) # (B,5)
74
+ v = _pentachoron_volumes(x[idx].cpu())
75
+ return (v.std() / v.mean().clamp_min(1e-12)).item()
76
+
77
+
78
+ @torch.no_grad()
79
+ def cv_reference_check(n_trials: int = 50, tol: float = 1e-5) -> float:
80
+ """Parity check of the built-in batched CM against geovocab2's reference
81
+ implementation (the formula's source of truth). Returns max |rel diff|;
82
+ raises if geovocab2 is absent or parity fails. Run after touching
83
+ _pentachoron_volumes."""
84
+ try:
85
+ from geovocab2.shapes.formula.symbolic.cayley_menger import (
86
+ CayleyMengerFromSimplex)
87
+ except Exception as e: # pragma: no cover
88
+ raise ImportError(
89
+ "cv_reference_check requires geovocab2 (install via the geolip-svae "
90
+ "umbrella: pip install git+https://github.com/AbstractEyes/"
91
+ "geolip-svae).") from e
92
+ ref = CayleyMengerFromSimplex()
93
+ g = torch.Generator().manual_seed(0)
94
+ pts = torch.nn.functional.normalize(
95
+ torch.randn(n_trials, 5, 4, generator=g), dim=-1)
96
+ mine = _pentachoron_volumes(pts)
97
+ # compare at float64: the reference computes in the INPUT dtype, and fp32
98
+ # dets lose up to ~4% on near-degenerate pentachora (measured 2026-07-11)
99
+ theirs = torch.stack([ref.forward(p.double())["volume"].float() for p in pts])
100
+ rel = ((mine - theirs).abs() / theirs.abs().clamp_min(1e-12)).max().item()
101
+ if rel > tol:
102
+ raise AssertionError(f"CM parity vs geovocab2 failed: max rel {rel}")
103
+ return rel
104
+
105
+
106
+ # ------------------------------------------------------------------------- aliveness
107
+ @torch.no_grad()
108
+ def axis_aliveness(oriented_weights: torch.Tensor, alive_thresh: float = 1e-3) -> dict:
109
+ """`oriented_weights`: (..., 2K) nonnegative oriented-softmax address rows
110
+ (sum to 1 on the last dim). Returns axes-alive count, mean-usage perplexity
111
+ (hppl analogue; healthy hosted reference 125-126/128), and a collapse flag.
112
+ Reference behavior: near-uniform aliveness at div_weight=0 (discovery #22)."""
113
+ w = oriented_weights.reshape(-1, oriented_weights.shape[-1]).float()
114
+ usage = w.mean(0)
115
+ usage = usage / usage.sum().clamp_min(1e-12)
116
+ # an axis is alive if its mean usage exceeds alive_thresh x the uniform share
117
+ alive = int((usage > alive_thresh * (1.0 / usage.numel())).sum())
118
+ ent = -(usage.clamp_min(1e-12) * usage.clamp_min(1e-12).log()).sum()
119
+ ppl = float(ent.exp())
120
+ return {"axes_total": usage.numel(), "axes_alive": alive, "usage_ppl": ppl,
121
+ "collapsed": ppl < 0.05 * usage.numel()}
122
+
123
+
124
+ # ------------------------------------------------------------------------------ gates
125
+ @torch.no_grad()
126
+ def gate_stats(gates: torch.Tensor) -> dict:
127
+ """Gate values (post-sigmoid/clamp). Reports mean and whether it sits in the
128
+ 0.012-0.03 band (read-only — the band is a candidate invariant, never a target)."""
129
+ g = gates.float().flatten()
130
+ m = g.mean().item()
131
+ return {"mean": m, "std": g.std().item(),
132
+ "in_band": GATE_BAND[0] <= m <= GATE_BAND[1]}
133
+
134
+
135
+ # ------------------------------------------------------------------------------ paths
136
+ @torch.no_grad()
137
+ def path_diversity(ids: torch.Tensor) -> dict:
138
+ """Unique-path counting with the FIXED multiplicative hash:
139
+ ((ids * 2654435761) % 2^32) >> 16 — Knuth needs the HIGH bits; the low-16
140
+ variant produced the retracted ~1,500 path ceiling (sessions/2026-07-07).
141
+ `ids`: integer tensor, one composed path id per row (any shape)."""
142
+ x = ids.reshape(-1).to(torch.int64)
143
+ hashed = ((x * KNUTH32) % (1 << 32)) >> 16
144
+ return {"n": int(x.numel()),
145
+ "unique_raw": int(torch.unique(x).numel()),
146
+ "unique_hashed": int(torch.unique(hashed).numel())}
147
+
148
+
149
+ @torch.no_grad()
150
+ def compose_path_ids(stage_indices: list[torch.Tensor], radix: int) -> torch.Tensor:
151
+ """Compose per-stage discrete indices (each (...,) int in [0, radix)) into a
152
+ single path id, positional base-`radix` — construction, not hashing."""
153
+ out = torch.zeros_like(stage_indices[0], dtype=torch.int64)
154
+ for s in stage_indices:
155
+ out = out * radix + s.to(torch.int64)
156
+ return out
157
+
158
+
159
+ # --------------------------------------------------------------------- grad democracy
160
+ @torch.no_grad()
161
+ def grad_norm_spread(groups: dict[str, list[torch.nn.Parameter]]) -> dict:
162
+ """Gradient-democracy monitor. `groups`: name -> params of one parallel member
163
+ (tower/expert). Reports per-group grad norms and the orders-of-magnitude spread.
164
+ Reference: unequalized heterogeneous towers spread ~20 orders (fibonacci dead at
165
+ 2.25e-21 under helix); equalized ~0.0 (canon/fibonacci_systems.md)."""
166
+ norms = {}
167
+ for name, params in groups.items():
168
+ gs = [p.grad for p in params if p.grad is not None]
169
+ norms[name] = float(torch.sqrt(sum((g.float() ** 2).sum() for g in gs)).item()) \
170
+ if gs else 0.0
171
+ vals = [v for v in norms.values() if v > 0]
172
+ spread = (math.log10(max(vals)) - math.log10(min(vals))) if len(vals) >= 2 else 0.0
173
+ return {"norms": norms, "spread_orders": spread, "dead": [k for k, v in norms.items() if v == 0.0]}
174
+
175
+
176
+ # ----------------------------------------------------------------------------- screen
177
+ class CVScreen:
178
+ """CV@N early band screen (tri-band ft1): record pentachoron CV at `step_mark`
179
+ batches; classify <0.30 LOW / 0.35-0.50 MID / >0.80 HIGH. Turns ~2h/config
180
+ into ~7min. Readout only."""
181
+ def __init__(self, step_mark: int = 1000):
182
+ self.step_mark = step_mark
183
+ self.recorded: float | None = None
184
+
185
+ def maybe_record(self, step: int, rows: torch.Tensor) -> float | None:
186
+ if self.recorded is None and step >= self.step_mark:
187
+ self.recorded = pentachoron_cv(rows)
188
+ return self.recorded
189
+
190
+ @property
191
+ def band(self) -> str | None:
192
+ c = self.recorded
193
+ if c is None:
194
+ return None
195
+ if c < 0.30:
196
+ return "LOW"
197
+ if 0.35 <= c <= 0.50:
198
+ return "MID"
199
+ if c > 0.80:
200
+ return "HIGH"
201
+ return "BETWEEN"
202
+
203
+
204
+ # ------------------------------------------------------------------------------ smoke
205
+ if __name__ == "__main__": # shapes/parse smoke ONLY — no training, ever.
206
+ g = torch.Generator().manual_seed(0)
207
+ K, D = 64, 4
208
+ init = torch.nn.functional.normalize(torch.randn(K, D, generator=g), dim=-1)
209
+ cur = torch.nn.functional.normalize(init + 0.29 * torch.randn(K, D, generator=g), dim=-1)
210
+ print("drift:", {k: v for k, v in anchor_drift(cur, init).items() if k != "per_row"})
211
+ w = torch.softmax(torch.randn(32, 2 * K, generator=g), dim=-1)
212
+ print("aliveness:", axis_aliveness(w))
213
+ print("gates:", gate_stats(torch.full((8,), 0.024)))
214
+ ids = compose_path_ids([torch.randint(0, 16, (4096,), generator=g) for _ in range(4)], 16)
215
+ print("paths:", path_diversity(ids))
216
+ lin = torch.nn.Linear(8, 8)
217
+ lin(torch.randn(4, 8)).sum().backward()
218
+ print("democracy:", grad_norm_spread({"a": list(lin.parameters())}))
219
+ print("OK — vitals smoke passed (pentachoron_cv needs geovocab2; run on GPU env)")
trainers/splat_math_battery.py ADDED
@@ -0,0 +1,644 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # splat_math_battery.py — B1-B6: the multi-aleph / splat hypothesis math
3
+ # =========================================================================
4
+ # Plan: history/plans/2026-08-06_splat_aleph_program.md (Rorschach
5
+ # correction + battery authorization). AbstractPhil: "test the hypothesis states
6
+ # for the mathematics before we structure the next attention mechanism."
7
+ #
8
+ # Every experiment uses AlephAddress VERBATIM (refine-don't-rewrite).
9
+ # All synthetic, seeded, local-GPU, minutes-per-arm. No attention
10
+ # mechanism anywhere in this file — measurement only.
11
+ #
12
+ # B1 capacity single-aleph capacity curves: K x D x tau x data-family
13
+ # -> usage ppl, realized sign-codes, aliasing, m_hat
14
+ # reconstruction, capacity per half-axis
15
+ # B2 product M differentiated frames vs 1 monolith at EQUAL total
16
+ # half-axes: joint code diversity, joint readback,
17
+ # pairwise redundancy (the product-code hypothesis)
18
+ # B3 dynamics does init-differentiation PERSIST under shared-task
19
+ # training with NO diversity pressure? (basin-set-at-
20
+ # init, multi-aleph edition; copies arm = collapse ctrl)
21
+ # B4 opinion composition gating: uniform vs SELF-confidence (non-
22
+ # comparative) vs softmax-over-alephs (LABELED
23
+ # falsification control) vs oracle, on data built so
24
+ # perspectives hold real differential advantage
25
+ # B5 blackboard associative-recall capacity of write/read through the
26
+ # codebook: P_max(K) at recall cos>=.7, single vs
27
+ # M-differentiated; crosstalk spectra — the sizing
28
+ # number for any future splat-attention head
29
+ # B6 boundary positional splat supports: overlap ratio vs max solid
30
+ # relay distance on long chains; tiling (zero overlap)
31
+ # as the control that must reproduce the deep-line
32
+ # tile-edge deadness
33
+ #
34
+ # RUN: python tools\splat_math_battery.py (all arms)
35
+ # ALEPH_SPLAT_ARM=B1..B6 for one; ALEPH_SPLAT_QUICK=1 smoke sizes
36
+ # Ships splat_battery_<arm>.json to AbstractPhil/aleph-splat-0 (private).
37
+ # =========================================================================
38
+
39
+ import importlib
40
+ import json
41
+ import math
42
+ import os
43
+ import sys
44
+ import time
45
+ import zlib
46
+
47
+ import numpy as np
48
+ import torch
49
+ import torch.nn.functional as F
50
+
51
+
52
+ def _load(name):
53
+ if name in sys.modules:
54
+ return sys.modules[name]
55
+ try:
56
+ return importlib.import_module(name)
57
+ except ImportError:
58
+ here = globals().get("__file__")
59
+ if here is not None:
60
+ import pathlib
61
+ sys.path.insert(0, str(pathlib.Path(here).parent))
62
+ return importlib.import_module(name)
63
+ raise
64
+
65
+
66
+ for _s in (sys.stdout, sys.stderr):
67
+ if hasattr(_s, "reconfigure"):
68
+ _s.reconfigure(encoding="utf-8", errors="replace")
69
+
70
+ adb = _load("ar_differentiation_bed")
71
+ AlephAddress = adb.AlephAddress
72
+
73
+ DEV = "cuda" if torch.cuda.is_available() else "cpu"
74
+ QUICK = os.environ.get("ALEPH_SPLAT_QUICK") == "1"
75
+ ARM = os.environ.get("ALEPH_SPLAT_ARM", "ALL").upper()
76
+ OUT_DIR = os.environ.get("ALEPH_SPLAT_OUT", "./data/aleph_splat"
77
+ if os.name == "nt" else "./aleph_splat")
78
+ HUB = os.environ.get("ALEPH_SPLAT_REPO", "AbstractPhil/aleph-splat-0")
79
+ N = 20_000 if QUICK else 200_000
80
+ SEED0 = zlib.crc32(b"aleph-splat-0") % (2 ** 31)
81
+
82
+
83
+ def line(t=""):
84
+ print("-" * 78 if not t else f"-- {t} " + "-" * max(0, 74 - len(t)))
85
+
86
+
87
+ def gen(family, n, D, seed, c_clusters=32, r_lowrank=None, ar=0.98):
88
+ """Synthetic unit-vector families. All seeded, all on DEV."""
89
+ g = torch.Generator(device="cpu").manual_seed(seed)
90
+ if family == "iso":
91
+ x = torch.randn(n, D, generator=g)
92
+ elif family == "clust":
93
+ cents = F.normalize(torch.randn(c_clusters, D, generator=g), dim=-1)
94
+ idx = torch.randint(0, c_clusters, (n,), generator=g)
95
+ x = cents[idx] + 0.25 * torch.randn(n, D, generator=g)
96
+ elif family == "lowrank":
97
+ r = r_lowrank or max(2, D // 4)
98
+ basis = torch.linalg.qr(torch.randn(D, r, generator=g))[0]
99
+ x = torch.randn(n, r, generator=g) @ basis.T
100
+ elif family == "seq":
101
+ steps = F.normalize(torch.randn(n, D, generator=g), dim=-1)
102
+ x = torch.empty(n, D)
103
+ x[0] = steps[0]
104
+ for i in range(1, n):
105
+ x[i] = ar * x[i - 1] + math.sqrt(1 - ar * ar) * steps[i]
106
+ else:
107
+ raise ValueError(family)
108
+ return F.normalize(x, dim=-1).to(DEV)
109
+
110
+
111
+ @torch.no_grad()
112
+ def read_stats(addr, x, pair_sample=4096):
113
+ """Single-aleph gauges: usage, realized codes, aliasing, recon."""
114
+ u = (F.normalize(x, dim=-1) @ F.normalize(addr.codebook, dim=-1).T) / addr.tau
115
+ p, nneg = addr.oriented(x)
116
+ two_k = torch.cat([p, nneg], dim=-1)
117
+ win = two_k.argmax(dim=-1)
118
+ hist = torch.bincount(win, minlength=2 * addr.K).float()
119
+ ppl = float(torch.exp(-(hist / hist.sum() * (hist / hist.sum() + 1e-12)
120
+ .log()).sum()))
121
+ signs = (u > 0)
122
+ # realized full sign-codes via hashing
123
+ weights = (2 ** torch.arange(min(addr.K, 62), device=DEV))
124
+ code = (signs[:, :len(weights)].long() * weights).sum(-1)
125
+ n_codes = int(torch.unique(code).numel())
126
+ # aliasing: same full code -> how similar are inputs really?
127
+ order = torch.argsort(code)
128
+ cs, xs = code[order], x[order]
129
+ same = cs[1:] == cs[:-1]
130
+ if int(same.sum()) > 8:
131
+ i = torch.nonzero(same).squeeze(-1)
132
+ if i.numel() > pair_sample:
133
+ i = i[torch.randperm(i.numel(), device=DEV)[:pair_sample]]
134
+ alias_cos = float((xs[i] * xs[i + 1]).sum(-1).mean())
135
+ else:
136
+ alias_cos = float("nan")
137
+ m = addr.m_hat(x)
138
+ recon = float(F.cosine_similarity(m, x, dim=-1).mean())
139
+ win_cos = float((u.abs().amax(dim=-1) * addr.tau).mean())
140
+ return {"usage_ppl": round(ppl, 2), "n_codes": n_codes,
141
+ "codes_per_halfaxis_bits": round(math.log2(max(n_codes, 1))
142
+ / (2 * addr.K), 4),
143
+ "alias_within_code_cos": round(alias_cos, 4)
144
+ if alias_cos == alias_cos else None,
145
+ "mhat_recon_cos": round(recon, 4),
146
+ "win_cos_mean": round(win_cos, 4)}
147
+
148
+
149
+ def b1_capacity():
150
+ line("B1 CAPACITY — single-aleph curves")
151
+ Ks = [4, 8, 16, 32, 64] if QUICK else [4, 8, 16, 32, 64, 128]
152
+ Ds = [8, 32] if QUICK else [4, 8, 16, 32, 64]
153
+ taus = [0.1] if QUICK else [0.05, 0.1, 0.2]
154
+ fams = ["iso", "clust", "lowrank"]
155
+ rows = []
156
+ for D in Ds:
157
+ for fam in fams:
158
+ x = gen(fam, N, D, SEED0 + D)
159
+ for K in Ks:
160
+ for tau in taus:
161
+ torch.manual_seed(SEED0 + K * 7 + D)
162
+ a = AlephAddress(K, D, tau=tau).to(DEV)
163
+ r = read_stats(a, x)
164
+ r.update({"K": K, "D": D, "tau": tau, "family": fam})
165
+ rows.append(r)
166
+ print(f" D={D} done ({len(rows)} rows)")
167
+ return {"rows": rows, "n": N}
168
+
169
+
170
+ def _joint_stats(alephs, x):
171
+ """Multi-aleph joint gauges."""
172
+ wins, reads = [], []
173
+ with torch.no_grad():
174
+ for a in alephs:
175
+ p, nneg = a.oriented(x)
176
+ wins.append(torch.cat([p, nneg], -1).argmax(-1))
177
+ reads.append(a.m_hat(x))
178
+ joint = wins[0].clone()
179
+ base = 1
180
+ for w in wins[1:]:
181
+ base *= 2 * alephs[0].K + 1
182
+ joint = joint * (2 * alephs[0].K) + w
183
+ n_joint = int(torch.unique(joint).numel())
184
+ # pairwise redundancy: normalized mutual information of winner streams
185
+ def nmi(a_, b_):
186
+ K2 = 2 * alephs[0].K
187
+ jointc = torch.bincount(a_ * K2 + b_, minlength=K2 * K2).float()
188
+ pj = jointc / jointc.sum()
189
+ pa = pj.reshape(K2, K2).sum(1)
190
+ pb = pj.reshape(K2, K2).sum(0)
191
+ eps = 1e-12
192
+ mi = (pj * (pj + eps).log()).sum() - (pa * (pa + eps).log()).sum() \
193
+ - (pb * (pb + eps).log()).sum()
194
+ ha = -(pa * (pa + eps).log()).sum()
195
+ hb = -(pb * (pb + eps).log()).sum()
196
+ return float(mi / max(float(torch.minimum(ha, hb)), 1e-9))
197
+ nmis = [nmi(wins[i], wins[j]) for i in range(len(wins))
198
+ for j in range(i + 1, len(wins))]
199
+ # joint readback: least-squares x_hat from concatenated m_hats
200
+ R = torch.cat(reads, dim=-1)
201
+ sol = torch.linalg.lstsq(R, x).solution
202
+ recon = float(F.cosine_similarity(R @ sol, x, dim=-1).mean())
203
+ return {"n_joint_codes": n_joint,
204
+ "mean_pairwise_nmi": round(float(np.mean(nmis)), 4) if nmis else None,
205
+ "joint_readback_cos": round(recon, 4)}
206
+
207
+
208
+ def _make_family(mode, M, K, D, seed):
209
+ """Init families for M perspectives."""
210
+ torch.manual_seed(seed)
211
+ alephs = []
212
+ if mode == "copies":
213
+ base = AlephAddress(K, D).to(DEV)
214
+ for _ in range(M):
215
+ a = AlephAddress(K, D).to(DEV)
216
+ a.codebook.data.copy_(base.codebook.data)
217
+ a.home.copy_(base.codebook.data)
218
+ alephs.append(a)
219
+ elif mode == "rotated":
220
+ base = AlephAddress(K, D).to(DEV)
221
+ for m in range(M):
222
+ a = AlephAddress(K, D).to(DEV)
223
+ Q = torch.linalg.qr(torch.randn(D, D, device=DEV))[0]
224
+ a.codebook.data.copy_(F.normalize(base.codebook.data @ Q, dim=-1))
225
+ a.home.copy_(a.codebook.data)
226
+ alephs.append(a)
227
+ else: # independent
228
+ for m in range(M):
229
+ torch.manual_seed(seed + 977 * m)
230
+ alephs.append(AlephAddress(K, D).to(DEV))
231
+ return alephs
232
+
233
+
234
+ def b2_product():
235
+ line("B2 PRODUCT CODE — M differentiated frames at equal half-axes")
236
+ D = 32
237
+ budget = 64 # total K across alephs
238
+ splits = [(1, 64), (2, 32), (4, 16), (8, 8), (16, 4)]
239
+ fams = ["iso", "clust", "lowrank"]
240
+ rows = []
241
+ for fam in fams:
242
+ x = gen(fam, N, D, SEED0 + 11)
243
+ for M, K in splits:
244
+ for mode in ("independent", "rotated", "copies"):
245
+ alephs = _make_family(mode, M, K, D, SEED0 + M * 13 + K)
246
+ r = _joint_stats(alephs, x)
247
+ r.update({"M": M, "K": K, "mode": mode, "family": fam,
248
+ "bits_per_halfaxis": round(
249
+ math.log2(max(r["n_joint_codes"], 1))
250
+ / (2 * budget), 4)})
251
+ rows.append(r)
252
+ print(f" family={fam} done")
253
+ return {"rows": rows, "budget_halfaxes": 2 * budget, "n": N}
254
+
255
+
256
+ def b3_dynamics():
257
+ line("B3 DIFFERENTIATION DYNAMICS — does distinctness persist?")
258
+ M, K, D, steps = 4, 16, 32, (300 if QUICK else 2000)
259
+ fams = {"clust": gen("clust", 50_000, D, SEED0 + 21)}
260
+ rows = []
261
+ for mode in ("independent", "copies"):
262
+ alephs = _make_family(mode, M, K, D, SEED0 + 31)
263
+ if mode == "copies": # tiny symmetric-break noise, realistic init
264
+ for a in alephs:
265
+ a.codebook.data.add_(0.01 * torch.randn_like(a.codebook))
266
+ params = [a.codebook for a in alephs]
267
+ for p in params:
268
+ p.requires_grad_(True)
269
+ opt = torch.optim.Adam(params, lr=1e-3) # pure Adam, house law
270
+ x = fams["clust"]
271
+ traj = []
272
+ def pdist():
273
+ ds = []
274
+ for i in range(M):
275
+ for j in range(i + 1, M):
276
+ A = F.normalize(alephs[i].codebook.detach(), dim=-1)
277
+ B = F.normalize(alephs[j].codebook.detach(), dim=-1)
278
+ Q = torch.linalg.qr(torch.randn(D, D, device=DEV))[0] * 0
279
+ # Procrustes-aligned distance
280
+ U, _, Vt = torch.linalg.svd(B.T @ A)
281
+ R = U @ Vt
282
+ ds.append(float(1 - (F.normalize(B @ R, dim=-1)
283
+ * A).sum(-1).mean()))
284
+ return float(np.mean(ds))
285
+ for s in range(steps):
286
+ i = torch.randint(0, x.shape[0], (4096,), device=DEV)
287
+ xb = x[i]
288
+ read = sum(a.m_hat(xb) for a in alephs) / M
289
+ loss = (1 - F.cosine_similarity(read, xb, dim=-1)).mean()
290
+ opt.zero_grad()
291
+ loss.backward()
292
+ opt.step()
293
+ if s % max(1, steps // 10) == 0:
294
+ traj.append({"step": s, "loss": round(float(loss), 4),
295
+ "pairwise_pdist": round(pdist(), 4)})
296
+ r_final = _joint_stats(alephs, x)
297
+ rows.append({"mode": mode, "traj": traj, "final": r_final,
298
+ "final_pdist": round(pdist(), 4)})
299
+ print(f" mode={mode}: final pdist {rows[-1]['final_pdist']}")
300
+ return {"rows": rows, "M": M, "K": K, "D": D, "steps": steps}
301
+
302
+
303
+ def b4_opinion():
304
+ line("B4 OPINION GATING — self-confidence vs softmax (falsification)")
305
+ M, K, D = 4, 16, 32
306
+ torch.manual_seed(SEED0 + 41)
307
+ alephs = _make_family("independent", M, K, D, SEED0 + 41)
308
+ # data with REAL perspective advantage: cluster c is built FROM frame c
309
+ n_per = (N // 4) // M * M
310
+ xs, owner = [], []
311
+ for m, a in enumerate(alephs):
312
+ cents = F.normalize(a.codebook.detach(), dim=-1)
313
+ idx = torch.randint(0, K, (n_per // M,), device=DEV)
314
+ xs.append(F.normalize(cents[idx] + 0.35 * torch.randn(
315
+ n_per // M, D, device=DEV), dim=-1))
316
+ owner.append(torch.full((n_per // M,), m, device=DEV))
317
+ x = torch.cat(xs)
318
+ owner = torch.cat(owner)
319
+ with torch.no_grad():
320
+ reads = [a.m_hat(x) for a in alephs]
321
+ confs = [(F.normalize(x, dim=-1) @ F.normalize(
322
+ a.codebook, dim=-1).T).abs().amax(-1) for a in alephs]
323
+ C = torch.stack(confs, dim=-1) # (n, M)
324
+ def compose(gates):
325
+ R = sum(g.unsqueeze(-1) * r for g, r in zip(gates.unbind(-1), reads))
326
+ return float(F.cosine_similarity(R, x, dim=-1).mean())
327
+ results = {}
328
+ results["uniform"] = compose(torch.ones_like(C) / M)
329
+ results["self_confidence"] = compose(C / C.sum(-1, keepdim=True).clamp(min=1e-9))
330
+ results["softmax_comparative_CONTROL"] = compose(torch.softmax(C / 0.05, dim=-1))
331
+ oracle = F.one_hot(owner, M).float()
332
+ results["oracle"] = compose(oracle)
333
+ # accuracy of implied selection vs owner (diagnostic, not a selector)
334
+ results["self_conf_top1_matches_owner"] = float(
335
+ (C.argmax(-1) == owner).float().mean())
336
+ print(f" {results}")
337
+ return {k: round(v, 4) for k, v in results.items()} | {"M": M, "K": K, "D": D}
338
+
339
+
340
+ def b5_blackboard():
341
+ line("B5 BLACKBOARD — associative recall capacity through the address")
342
+ D = 32
343
+ Ks = [8, 16, 32, 64] if QUICK else [8, 16, 32, 64, 128]
344
+ Ps = [4, 8, 16, 32, 64, 128, 256, 512]
345
+ rows = []
346
+ for K in Ks:
347
+ torch.manual_seed(SEED0 + 51 + K)
348
+ single = AlephAddress(K, D).to(DEV)
349
+ multi = _make_family("independent", 4, max(K // 4, 2), D, SEED0 + 53 + K)
350
+ for tag, addrs in (("single", [single]), ("M4_product", multi)):
351
+ for P in Ps:
352
+ torch.manual_seed(SEED0 + P)
353
+ keys = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
354
+ vals = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
355
+ with torch.no_grad():
356
+ w = torch.cat([a.signed(keys) for a in addrs], -1) # (P, sumK)
357
+ cells = w.T @ vals # (sumK, D)
358
+ q = F.normalize(keys + 0.05 * torch.randn_like(keys), dim=-1)
359
+ wq = torch.cat([a.signed(q) for a in addrs], -1)
360
+ out = wq @ cells
361
+ rec = float(F.cosine_similarity(out, vals, dim=-1).mean())
362
+ rows.append({"K_total": sum(a.K for a in addrs), "form": tag,
363
+ "P": P, "recall_cos": round(rec, 4)})
364
+ pmax_s = max([r["P"] for r in rows if r["form"] == "single"
365
+ and r["K_total"] == K and r["recall_cos"] >= 0.7],
366
+ default=0)
367
+ print(f" K={K}: single P_max(cos>=.7) ~ {pmax_s}")
368
+ return {"rows": rows, "noise": 0.05, "D": D}
369
+
370
+
371
+ def b6_boundary():
372
+ line("B6 BOUNDARY SOLIDITY — overlap ratio vs solid relay distance")
373
+ D, K, L = 32, 16, (512 if QUICK else 4096)
374
+ spacings = [64, 128, 256]
375
+ ratios = [0.0, 0.25, 0.5, 1.0, 2.0] # sigma / spacing; 0 = tiling
376
+ torch.manual_seed(SEED0 + 61)
377
+ msg = F.normalize(torch.randn(D, device=DEV), dim=-1)
378
+ rows = []
379
+ for s in spacings:
380
+ centers = torch.arange(0, L, s, device=DEV).float()
381
+ for ratio in ratios:
382
+ sig = max(ratio * s, 1e-6)
383
+ pos = torch.arange(L, device=DEV).float()
384
+ if ratio == 0.0: # hard tiling membership
385
+ gm = (pos.unsqueeze(0) - centers.unsqueeze(1)).abs() < (s / 2)
386
+ gm = gm.float()
387
+ else:
388
+ gm = torch.exp(-0.5 * ((pos.unsqueeze(0) - centers.unsqueeze(1))
389
+ / sig) ** 2)
390
+ gm = gm / gm.sum(0, keepdim=True).clamp(min=1e-9) # local support
391
+ # relay: message injected at pos0 propagates via shared membership
392
+ # solidity(d) = sum_m g_m(0) g_m(d) / sqrt(sum g_m(0)^2 sum g_m(d)^2)
393
+ g0 = gm[:, 0]
394
+ sol = (gm.T @ g0) / (gm.norm(dim=0) * g0.norm()).clamp(min=1e-9)
395
+ # chained relay: two-hop through any intermediate position
396
+ Gn = gm / gm.norm(dim=0, keepdim=True).clamp(min=1e-9)
397
+ A1 = Gn.T @ Gn # position-position solidity
398
+ A2 = A1 @ A1 / A1.shape[0]
399
+ def maxd(v, thr=0.1):
400
+ idx = torch.nonzero(v > thr).squeeze(-1)
401
+ return int(idx.max()) if idx.numel() else 0
402
+ rows.append({"spacing": s, "overlap_ratio": ratio,
403
+ "solid_1hop_maxd": maxd(sol),
404
+ "solid_2hop_maxd": maxd(A2[0]),
405
+ "edge_min_1hop": round(float(sol[:2 * s].min()), 4)})
406
+ print(f" spacing={s} done")
407
+ return {"rows": rows, "L": L, "note": "ratio 0 = hard tiling control; "
408
+ "edge_min near 0 at ratio 0 must reproduce the deep-line "
409
+ "tile-edge deadness"}
410
+
411
+
412
+
413
+
414
+ def b7_mscaling():
415
+ line("B7 M-SCALING — thousands of tiny heads vs data rank")
416
+ D, K = 64, 4
417
+ Ms = [16, 64, 256, 1024, 2048] if not QUICK else [16, 64, 256]
418
+ ranks = [8, 32, 64] # 64 = full
419
+ n = 100_000 if not QUICK else 20_000
420
+ rows = []
421
+ for r in ranks:
422
+ x = gen("lowrank", n, D, SEED0 + 71 + r, r_lowrank=r)
423
+ n_tr = int(0.75 * x.shape[0])
424
+ for M in Ms:
425
+ alephs = _make_family("independent", M, K, D, SEED0 + 73 + M)
426
+ # HELD-OUT streaming linear probe: at large M the lstsq gauge
427
+ # both OOMs (100k x 65k fp32 = 26GB, measured crash) and
428
+ # degenerates (features > rows -> interpolation, trivial 1.0).
429
+ # Probe weights are O(M*K*D) only; reads computed per batch.
430
+ feat_dim = sum(a.K for a in alephs)
431
+ W = torch.zeros(feat_dim, D, device=DEV, requires_grad=True)
432
+ opt = torch.optim.Adam([W], lr=3e-3)
433
+ def reads_of(xb):
434
+ with torch.no_grad():
435
+ return torch.cat([a.signed(xb) for a in alephs], -1)
436
+ for ep in range(3):
437
+ perm = torch.randperm(n_tr, device=DEV)
438
+ for i in range(0, n_tr, 8192):
439
+ xb = x[perm[i:i + 8192]]
440
+ pred = reads_of(xb) @ W
441
+ loss = (1 - F.cosine_similarity(pred, xb, dim=-1)).mean()
442
+ opt.zero_grad(); loss.backward(); opt.step()
443
+ with torch.no_grad():
444
+ rec_parts = []
445
+ for i in range(n_tr, x.shape[0], 8192):
446
+ xb = x[i:i + 8192]
447
+ rec_parts.append(F.cosine_similarity(
448
+ reads_of(xb) @ W, xb, dim=-1))
449
+ rec = float(torch.cat(rec_parts).mean())
450
+ p1, n1 = alephs[0].oriented(x[:50000])
451
+ p2, n2 = alephs[1].oriented(x[:50000])
452
+ w1 = torch.cat([p1, n1], -1).argmax(-1)
453
+ w2 = torch.cat([p2, n2], -1).argmax(-1)
454
+ K2 = 2 * K
455
+ jc = torch.bincount(w1 * K2 + w2, minlength=K2 * K2).float()
456
+ pj = jc / jc.sum()
457
+ pa = pj.reshape(K2, K2).sum(1); pb = pj.reshape(K2, K2).sum(0)
458
+ eps = 1e-12
459
+ mi = float((pj * (pj + eps).log()).sum()
460
+ - (pa * (pa + eps).log()).sum()
461
+ - (pb * (pb + eps).log()).sum())
462
+ rows.append({"rank": r, "M": M, "half_axes": 2 * K * M,
463
+ "readback_heldout": round(rec, 4),
464
+ "pair_mi_nats": round(mi, 4)})
465
+ print(f" rank={r} M={M}: readback {rec:.4f}")
466
+ return {"rows": rows, "K": K, "D": D, "n": n}
467
+
468
+
469
+ def b8_headscale():
470
+ line("B8 HEAD-SCALE BLACKBOARD — context recall under splat sharding")
471
+ D, K = 64, 4
472
+ Ns = [2048, 8192] if not QUICK else [512]
473
+ configs = [("monolith", 1, 2048), ("M512", 512, None), ("M2048", 2048, None)]
474
+ rows = []
475
+ for N_ctx in Ns:
476
+ torch.manual_seed(SEED0 + 81 + N_ctx)
477
+ keys = F.normalize(torch.randn(N_ctx, D, device=DEV), dim=-1)
478
+ vals = F.normalize(torch.randn(N_ctx, D, device=DEV), dim=-1)
479
+ q = F.normalize(keys + 0.05 * torch.randn_like(keys), dim=-1)
480
+ for tag, M, Kmono in configs:
481
+ Ke = Kmono if Kmono else K
482
+ alephs = _make_family("independent", M, Ke, D, SEED0 + 83 + M)
483
+ t0 = time.time()
484
+ with torch.no_grad():
485
+ if M == 1:
486
+ w = alephs[0].signed(keys); wq = alephs[0].signed(q)
487
+ cells = w.T @ vals
488
+ out = wq @ cells
489
+ else:
490
+ # splat sharding: head m owns window centered at
491
+ # m*N/M, width sigma = 1.5 * spacing (B6 band)
492
+ pos = torch.arange(N_ctx, device=DEV).float()
493
+ centers = torch.linspace(0, N_ctx, M, device=DEV)
494
+ sig = 1.5 * (N_ctx / M)
495
+ gm = torch.exp(-0.5 * ((pos.unsqueeze(0)
496
+ - centers.unsqueeze(1)) / sig) ** 2)
497
+ gm = gm / gm.sum(0, keepdim=True).clamp(min=1e-9)
498
+ out = torch.zeros_like(vals)
499
+ for m, a in enumerate(alephs):
500
+ g = gm[m].unsqueeze(-1)
501
+ w = a.signed(keys) * g
502
+ cells = w.T @ vals
503
+ out = out + (a.signed(q) * g) @ cells
504
+ wall = time.time() - t0
505
+ rec = float(F.cosine_similarity(out, vals, dim=-1).mean())
506
+ # sharp-retrieval comparison: cosine top-1 accuracy
507
+ top1 = float((F.normalize(out, dim=-1) @ vals.T)
508
+ .argmax(-1).eq(torch.arange(N_ctx, device=DEV))
509
+ .float().mean())
510
+ rows.append({"N_ctx": N_ctx, "form": tag, "M": M,
511
+ "cells_total": sum(a.K for a in alephs) * 2,
512
+ "recall_cos": round(rec, 4),
513
+ "top1_acc": round(top1, 4),
514
+ "wall_s": round(wall, 3)})
515
+ print(f" N={N_ctx} {tag}: recall {rec:.4f} top1 {top1:.4f} "
516
+ f"({wall:.2f}s)")
517
+ return {"rows": rows, "note": "splat sharding keeps per-board load "
518
+ "~N/M*overlap; monolith board must hold all N"}
519
+
520
+
521
+
522
+
523
+ def _rope(x, pos, base=10000.0):
524
+ """RoPE rotation applied to the ADDRESS QUERY: pairs (2i,2i+1) rotate
525
+ by pos*theta_i. Position enters the aleph through the addressing
526
+ itself — relative structure R(i-j) in every write/read affinity."""
527
+ D = x.shape[-1]
528
+ half = D // 2
529
+ freqs = base ** (-torch.arange(half, device=x.device,
530
+ dtype=torch.float32) / half)
531
+ ang = pos.unsqueeze(-1) * freqs # (..., half)
532
+ c, sn = torch.cos(ang), torch.sin(ang)
533
+ x1, x2 = x[..., :half], x[..., half:]
534
+ return torch.cat([x1 * c - x2 * sn, x1 * sn + x2 * c], dim=-1)
535
+
536
+
537
+ def b9_rotary():
538
+ line("B9 CODEBOOK ROTARY — position in the address, accuracy probe")
539
+ D, K, M = 64, 8, 16
540
+ torch.manual_seed(SEED0 + 91)
541
+ alephs = _make_family("independent", M, K, D, SEED0 + 91)
542
+ def addr(x, pos):
543
+ xr = F.normalize(_rope(F.normalize(x, dim=-1), pos), dim=-1)
544
+ return torch.cat([a.signed(xr) for a in alephs], -1)
545
+ out = {}
546
+ # (a) relative-offset structure: same content, positions 0..256 —
547
+ # address cos vs delta (the interference curve; RoPE-like decay)
548
+ x0 = F.normalize(torch.randn(1, D, device=DEV), dim=-1).repeat(257, 1)
549
+ pos = torch.arange(257, device=DEV, dtype=torch.float32)
550
+ w = addr(x0, pos)
551
+ ref = w[0:1]
552
+ cosd = F.cosine_similarity(w, ref.expand_as(w), dim=-1)
553
+ out["addr_cos_vs_delta"] = {str(d): round(float(cosd[d]), 4)
554
+ for d in (0, 1, 2, 4, 8, 16, 32, 64,
555
+ 128, 256)}
556
+ # (b) position discriminability: 1000 contents x 2 positions 32 apart
557
+ xc = F.normalize(torch.randn(1000, D, device=DEV), dim=-1)
558
+ wa = addr(xc, torch.full((1000,), 10.0, device=DEV))
559
+ wb = addr(xc, torch.full((1000,), 42.0, device=DEV))
560
+ out["same_content_dpos32_addr_cos"] = round(
561
+ float(F.cosine_similarity(wa, wb, dim=-1).mean()), 4)
562
+ # (c) recall under position shift: store P pairs at their positions,
563
+ # query same content at shifted positions
564
+ P = 64
565
+ keys = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
566
+ vals = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
567
+ kpos = torch.arange(P, device=DEV, dtype=torch.float32) * 4
568
+ wk = addr(keys, kpos)
569
+ cells = wk.T @ vals
570
+ rec_by_shift = {}
571
+ for shift in (0, 1, 2, 4, 8, 16, 64):
572
+ wq = addr(keys, kpos + shift)
573
+ got = wq @ cells
574
+ rec_by_shift[str(shift)] = round(float(
575
+ F.cosine_similarity(got, vals, dim=-1).mean()), 4)
576
+ out["recall_vs_pos_shift"] = rec_by_shift
577
+ # (d) TRANSPORT at short L — the collapse-fix test: token at pos 0
578
+ # reads content written at pos p, GLOBAL rotary heads (no windows)
579
+ L = 128
580
+ toks = F.normalize(torch.randn(L, D, device=DEV), dim=-1)
581
+ tpos = torch.arange(L, device=DEV, dtype=torch.float32)
582
+ wt = addr(toks, tpos)
583
+ cellsT = wt.T @ toks # every token written globally
584
+ probe_reads = wt @ cellsT # each reads the shared board
585
+ self_rec = float(F.cosine_similarity(probe_reads, toks, dim=-1).mean())
586
+ # cross-read: query content of token j FROM position 0's frame
587
+ w0 = addr(toks, torch.zeros(L, device=DEV))
588
+ cross = w0 @ cellsT
589
+ cross_rec = float(F.cosine_similarity(cross, toks, dim=-1).mean())
590
+ out["shortL_transport"] = {"self_recall": round(self_rec, 4),
591
+ "cross_pos0_recall": round(cross_rec, 4),
592
+ "note": "cross>0.3 = content reachable "
593
+ "across positions (the a4 collapse cure)"}
594
+ # (e) sign-code stability under rotation
595
+ win = (F.normalize(_rope(F.normalize(xc, dim=-1),
596
+ torch.full((1000,), 100.0, device=DEV)),
597
+ dim=-1) @ F.normalize(alephs[0].codebook,
598
+ dim=-1).T).abs().amax(-1)
599
+ out["win_cos_after_rotation"] = round(float(win.mean()), 4)
600
+ print(f" {json.dumps(out, indent=1)[:600]}")
601
+ return out
602
+
603
+
604
+ def ship(results):
605
+ os.makedirs(OUT_DIR, exist_ok=True)
606
+ tag = "" if ARM == "ALL" else f"_{ARM}"
607
+ path = f"{OUT_DIR}/splat_battery{tag}.json"
608
+ json.dump(results, open(path, "w"), indent=1)
609
+ print(f"\n wrote {path}")
610
+ tok = os.environ.get("HF_TOKEN")
611
+ if tok:
612
+ try:
613
+ from huggingface_hub import HfApi, create_repo
614
+ create_repo(HUB, token=tok, exist_ok=True, private=True)
615
+ HfApi(token=tok).upload_file(
616
+ path_or_fileobj=path, path_in_repo=f"battery/splat_battery{tag}.json",
617
+ repo_id=HUB, commit_message="B1-B6 math battery")
618
+ print(f" shipped -> {HUB}/battery/")
619
+ except Exception as e:
620
+ print(f" ship failed: {str(e)[:80]} (local copy kept)")
621
+
622
+
623
+ def main():
624
+ print("=" * 78)
625
+ print(f"SPLAT MATH BATTERY — arm={ARM} quick={QUICK} dev={DEV} N={N}")
626
+ print("=" * 78)
627
+ t0 = time.time()
628
+ arms = {"B1": b1_capacity, "B2": b2_product, "B3": b3_dynamics,
629
+ "B4": b4_opinion, "B5": b5_blackboard, "B6": b6_boundary,
630
+ "B7": b7_mscaling, "B8": b8_headscale, "B9": b9_rotary}
631
+ todo = arms if ARM == "ALL" else {ARM: arms[ARM]}
632
+ results = {"config": {"quick": QUICK, "N": N, "seed": SEED0,
633
+ "device": DEV}}
634
+ for k, fn in todo.items():
635
+ t = time.time()
636
+ results[k] = fn()
637
+ results[k]["wall_s"] = round(time.time() - t, 1)
638
+ print(f"\nTOTAL {time.time()-t0:.0f}s")
639
+ ship(results)
640
+ return results
641
+
642
+
643
+ if __name__ == "__main__" or "get_ipython" in globals():
644
+ RESULTS = main()
trainers/trigram_bitspace_sweep.py ADDED
@@ -0,0 +1,194 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # trigram_bitspace_sweep.py — AbstractPhil's unverified hypothesis (2026-08-09):
3
+ # "the bytemaps were forming inherently bit linked positioning causing
4
+ # byte differentiated pooled accumulation rather than using the mechanism
5
+ # ... the overlap bytes were causing anchored learning ... and in the
6
+ # process the accumulation formed the very alephs we've been studying."
7
+ #
8
+ # THREE MEASURABLE CLAIMS, each with a shuffle null (CPU-only sweep):
9
+ # A) BIT-LINKAGE: byte-embedding cosine similarity correlates with byte
10
+ # BIT overlap (8 - Hamming). Null: byte-label shuffles (z-score).
11
+ # B) ALEPH EMERGENCE: the embedding table factors as a SIGN CODE over
12
+ # the 8 bit-planes — fraction of (centered) table variance explained
13
+ # by linear regression on the 8 bit sign-vectors (a bit-plane basis =
14
+ # an 8-half-axis codebook). Null: shuffled bit labels.
15
+ # C) POOLED ACCUMULATION vs MECHANISM: the three trigram tables converge
16
+ # toward one another (row-matched cosine + linear CKA). Pooled sum ->
17
+ # tables similar; positional mechanism -> differentiated.
18
+ # D) DATA-SIDE (current bed): V1b key bit-collision spectra per R_key —
19
+ # ties the overlap-space hypothesis to the measured mis-selections.
20
+ # SPECIMENS: wikitext tri witness addr_mslh64_tri_s0_t2000.pt; unigram
21
+ # sdpa_s0_t8000.pt comparator; fresh-init tri null. ASCII CONFOUND noted:
22
+ # on wikitext, bit 0x20 IS the case bit — bit-structure is semantically
23
+ # real in the data, which is itself part of AbstractPhil's "inherent" claim.
24
+ # RUN: python tools\trigram_bitspace_sweep.py (CPU, ~1 min)
25
+ # =========================================================================
26
+
27
+ import importlib
28
+ import json
29
+ import os
30
+ import sys
31
+
32
+ import torch
33
+ import torch.nn.functional as F
34
+
35
+
36
+ def _load(name):
37
+ if name in sys.modules:
38
+ return sys.modules[name]
39
+ try:
40
+ return importlib.import_module(name)
41
+ except ImportError:
42
+ here = globals().get("__file__")
43
+ if here is not None:
44
+ import pathlib
45
+ sys.path.insert(0, str(pathlib.Path(here).parent))
46
+ return importlib.import_module(name)
47
+ raise
48
+
49
+
50
+ CKPT_DIR = "./data/ar_ckpts"
51
+ OUT = "./data/v1_rank_army/trigram_bitspace_sweep.json"
52
+ torch.manual_seed(0)
53
+
54
+ BITS = torch.tensor([[(b >> i) & 1 for i in range(8)]
55
+ for b in range(256)], dtype=torch.float64) * 2 - 1
56
+ HAM = (BITS.unsqueeze(0) * BITS.unsqueeze(1)).sum(-1) # (256,256) in [-8,8]
57
+ OFF = ~torch.eye(256, dtype=torch.bool)
58
+
59
+
60
+ def bit_linkage(W, n_null=500):
61
+ """A) corr(cos-sim, bit-overlap) + shuffle-null z."""
62
+ X = F.normalize(W.double(), dim=-1)
63
+ S = (X @ X.T)[OFF]
64
+ H = HAM[OFF]
65
+ r = float(torch.corrcoef(torch.stack([S, H]))[0, 1])
66
+ nulls = []
67
+ for _ in range(n_null):
68
+ p = torch.randperm(256)
69
+ nulls.append(float(torch.corrcoef(
70
+ torch.stack([S, HAM[p][:, p][OFF]]))[0, 1]))
71
+ nt = torch.tensor(nulls)
72
+ return {"r": round(r, 4),
73
+ "null_mean": round(float(nt.mean()), 4),
74
+ "z": round((r - float(nt.mean())) / max(float(nt.std()), 1e-9), 1)}
75
+
76
+
77
+ def bitplane_variance(W, n_null=200):
78
+ """B) fraction of centered-table variance explained by the 8 bit
79
+ sign-vectors (linear regression, closed form)."""
80
+ Wc = W.double() - W.double().mean(0, keepdim=True)
81
+ B = BITS # (256, 8)
82
+ coef = torch.linalg.lstsq(B, Wc).solution # (8, d)
83
+ resid = Wc - B @ coef
84
+ ev = 1 - float((resid ** 2).sum() / (Wc ** 2).sum())
85
+ nulls = []
86
+ for _ in range(n_null):
87
+ Bp = BITS[torch.randperm(256)]
88
+ c = torch.linalg.lstsq(Bp, Wc).solution
89
+ nulls.append(1 - float(((Wc - Bp @ c) ** 2).sum()
90
+ / (Wc ** 2).sum()))
91
+ nt = torch.tensor(nulls)
92
+ return {"explained_var": round(ev, 4),
93
+ "null_mean": round(float(nt.mean()), 4),
94
+ "z": round((ev - float(nt.mean())) / max(float(nt.std()), 1e-9),
95
+ 1)}
96
+
97
+
98
+ def cka(A, B):
99
+ A = A.double() - A.double().mean(0, keepdim=True)
100
+ B = B.double() - B.double().mean(0, keepdim=True)
101
+ num = torch.norm(A.T @ B, "fro") ** 2
102
+ den = torch.norm(A.T @ A, "fro") * torch.norm(B.T @ B, "fro")
103
+ return round(float(num / den.clamp_min(1e-12)), 4)
104
+
105
+
106
+ def pooled_vs_mechanism(sd):
107
+ """C) inter-table alignment for tri specimens."""
108
+ e0, e1, e2 = (sd["emb.weight"][:256], sd["emb1.weight"][:256],
109
+ sd["emb2.weight"][:256])
110
+ rowcos = lambda A, B: round(float(F.cosine_similarity(
111
+ A.double(), B.double(), dim=-1).mean()), 4)
112
+ return {"rowcos_01": rowcos(e0, e1), "rowcos_02": rowcos(e0, e2),
113
+ "rowcos_12": rowcos(e1, e2),
114
+ "cka_01": cka(e0, e1), "cka_02": cka(e0, e2),
115
+ "cka_12": cka(e1, e2)}
116
+
117
+
118
+ def analyze(name, sd):
119
+ out = {"specimen": name, "tables": {}}
120
+ for key in ("emb.weight", "emb1.weight", "emb2.weight"):
121
+ if key in sd:
122
+ W = sd[key][:256]
123
+ out["tables"][key] = {"bit_linkage": bit_linkage(W),
124
+ "bitplane": bitplane_variance(W)}
125
+ if "emb1.weight" in sd:
126
+ out["pooled_vs_mechanism"] = pooled_vs_mechanism(sd)
127
+ return out
128
+
129
+
130
+ def keyspace_collisions():
131
+ """D) V1b key bit-collision spectra per R_key."""
132
+ v1b = _load("v1b_recall_army_battery")
133
+ res = {}
134
+ for R in (4, 16, 64):
135
+ sg = torch.Generator().manual_seed(40_000 + R)
136
+ U = torch.randn(16, R, generator=sg)
137
+ g = torch.Generator().manual_seed(99_000 + R)
138
+ z = torch.randn(4096, R, generator=g)
139
+ bits = ((z @ U.T) > 0).double() * 2 - 1 # (4096, 16)
140
+ ham16 = (bits @ bits.T) # agreement in [-16,16]
141
+ off = ~torch.eye(4096, dtype=torch.bool)
142
+ agree = (ham16[off] + 16) / 2 # shared bits 0..16
143
+ res[f"R{R}"] = {
144
+ "mean_shared_bits": round(float(agree.mean()), 2),
145
+ "frac_ge14": round(float((agree >= 14).float().mean()), 5),
146
+ "frac_exact": round(float((agree >= 16).float().mean()), 6)}
147
+ return res
148
+
149
+
150
+ def main():
151
+ print("=" * 74)
152
+ print("TRIGRAM BIT-SPACE SWEEP (CPU) — AbstractPhil's pooled-accumulation "
153
+ "hypothesis")
154
+ print("=" * 74)
155
+ results = {"claims": {}}
156
+ specs = []
157
+ p = os.path.join(CKPT_DIR, "addr_mslh64_tri_s0_t2000.pt")
158
+ if os.path.exists(p):
159
+ specs.append(("wikitext_tri_witness",
160
+ torch.load(p, map_location="cpu",
161
+ weights_only=True)["state_dict"]))
162
+ p2 = os.path.join(CKPT_DIR, "sdpa_s0_t8000.pt")
163
+ if os.path.exists(p2):
164
+ specs.append(("wikitext_unigram_sdpa",
165
+ torch.load(p2, map_location="cpu",
166
+ weights_only=True)["state_dict"]))
167
+ bed = _load("ar_differentiation_bed")
168
+ torch.manual_seed(0)
169
+ fresh = bed.ByteLM("sdpa_tri", block=256).state_dict()
170
+ specs.append(("fresh_init_null", fresh))
171
+ for name, sd in specs:
172
+ r = analyze(name, sd)
173
+ results[name] = r
174
+ print(f"\n== {name}")
175
+ for tk, tv in r["tables"].items():
176
+ print(f" {tk}: bit-linkage r={tv['bit_linkage']['r']} "
177
+ f"(z={tv['bit_linkage']['z']}) bitplane EV="
178
+ f"{tv['bitplane']['explained_var']} "
179
+ f"(null {tv['bitplane']['null_mean']}, "
180
+ f"z={tv['bitplane']['z']})")
181
+ if "pooled_vs_mechanism" in r:
182
+ pm = r["pooled_vs_mechanism"]
183
+ print(f" tables 0-1/0-2/1-2: rowcos {pm['rowcos_01']}/"
184
+ f"{pm['rowcos_02']}/{pm['rowcos_12']} CKA "
185
+ f"{pm['cka_01']}/{pm['cka_02']}/{pm['cka_12']}")
186
+ results["keyspace_collisions"] = keyspace_collisions()
187
+ print("\n== V1b key bit-collision spectra:",
188
+ json.dumps(results["keyspace_collisions"]))
189
+ json.dump(results, open(OUT, "w"), indent=1)
190
+ print(f"\nsaved -> {OUT}")
191
+
192
+
193
+ if __name__ == "__main__":
194
+ main()
trainers/v1_rank_army_battery.py ADDED
@@ -0,0 +1,241 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # v1_rank_army_battery.py — V1: rank-ladder x army-ladder under AR pressure
3
+ # (plan 2026-08-08_data_shaped_army.md, validation V1 — THE arbiter for
4
+ # Rule 1 "M = c*R"; overnight battery, AbstractPhil 2026-08-08: "set up a bunch of
5
+ # small battery tests and feed them with synthetic data, run them on the
6
+ # 4090 overnight").
7
+ #
8
+ # Question: does the army-size knee track the DATA'S latent rank under
9
+ # causal/AR pressure, as B7 measured statically? Synthetic byte streams of
10
+ # controlled rank R (hidden linear-dynamical state, emission logits confined
11
+ # to an R-dim subspace) are fed to the AR bed's CausalHUB arm — the aleph
12
+ # army under causal masks, army size = K oriented anchors (2K half-axes).
13
+ #
14
+ # GRID: R in {8, 32, 128} x K in {16, 64, 256} x seeds {0, 1} = 18 runs.
15
+ # GAUGES: val bits-per-byte curve; final address census-lite (pos/neg mass
16
+ # split, address erank). KNEE: smallest K within .01 bpb of that rank's
17
+ # best. REGISTERED PREDICTION (Rule 1): knee(R=8) < knee(R=128); flat knee
18
+ # across R demotes Rule 1 to static-battery scope.
19
+ # HONESTY: matched steps and data, NOT matched FLOPs — HUB compute is
20
+ # O(n*K*d), so K=256 spends ~16x K=16's attention FLOPs. The knee claim is
21
+ # capability-vs-army-size at matched data; stated on every artifact.
22
+ #
23
+ # Resume-safe: per-run JSONs in OUT_DIR; existing runs skip. Ships each run
24
+ # + the final knee table to AbstractPhil/aleph-splat-0 battery/v1_rank_army/.
25
+ # RUN: python tools\v1_rank_army_battery.py (HF_TOKEN in env; 4090)
26
+ # Env: V1_STEPS (def 3000) V1_QUICK=1 (tiny smoke grid)
27
+ # =========================================================================
28
+
29
+ import importlib
30
+ import json
31
+ import math
32
+ import os
33
+ import sys
34
+
35
+ import torch
36
+ import torch.nn.functional as F
37
+
38
+
39
+ def _load(name):
40
+ if name in sys.modules:
41
+ return sys.modules[name]
42
+ try:
43
+ return importlib.import_module(name)
44
+ except ImportError:
45
+ here = globals().get("__file__")
46
+ if here is not None:
47
+ import pathlib
48
+ sys.path.insert(0, str(pathlib.Path(here).parent))
49
+ return importlib.import_module(name)
50
+ raise
51
+
52
+
53
+ bed = _load("ar_differentiation_bed") # ByteLM + _batch + conventions
54
+ class _cb: # public shim: token from env
55
+ @staticmethod
56
+ def hf_token():
57
+ import os as _os
58
+ return _os.environ.get("HF_TOKEN")
59
+ cb = _cb() # hf_token()
60
+
61
+ from huggingface_hub import HfApi # noqa: E402
62
+
63
+ assert torch.cuda.is_available(), "GPU only (never CPU-train for accuracy)"
64
+ DEV = "cuda"
65
+ QUICK = os.environ.get("V1_QUICK") == "1"
66
+ STEPS = int(os.environ.get("V1_STEPS", "200" if QUICK else "3000"))
67
+ BATCH, BLOCK = 32, 256
68
+ RANKS = [8, 32] if QUICK else [8, 32, 128]
69
+ ARMIES = [16, 64] if QUICK else [16, 64, 256]
70
+ SEEDS = [0] if QUICK else [0, 1]
71
+ EVAL_EVERY = 100 if QUICK else 300
72
+ DATA_DIR = os.environ.get("V1_DATA", "./data/v1_rank_army")
73
+ OUT_DIR = os.path.join(DATA_DIR, "results")
74
+ TRAIN_BYTES, VAL_BYTES = (400_000, 80_000) if QUICK else (2_000_000, 200_000)
75
+ HUB_REPO = "AbstractPhil/aleph-splat-0"
76
+
77
+
78
+ def make_stream(R, n_bytes, seed, device=DEV):
79
+ """Rank-R synthetic byte stream: hidden LDS state s (R-dim), emission
80
+ logits = W s confined to an R-dim subspace of R^256 by construction.
81
+ The model must FILTER the state from byte history — predictive
82
+ structure has latent rank exactly R. 64 parallel chains, concatenated."""
83
+ # STRUCTURAL seed depends on R only: train and val share the SAME
84
+ # system (A, W); the trajectory seed varies per split. (Smoke 2026-08-09
85
+ # caught the original bug: split-dependent A/W made val an unrelated
86
+ # stream — bpb pinned above uniform.)
87
+ sg = torch.Generator(device="cpu").manual_seed(10_000 + R)
88
+ A = torch.linalg.qr(torch.randn(R, R, generator=sg))[0] * 0.95
89
+ W = torch.randn(R, 256, generator=sg)
90
+ W = W / W.std() * 2.0 # logit scale: peaked, not trivial
91
+ A, W = A.to(device), W.to(device)
92
+ chains = 64
93
+ steps = n_bytes // chains + 1
94
+ ig = torch.Generator().manual_seed(20_000 + R * 100 + seed)
95
+ s = torch.randn(chains, R, generator=ig).to(device)
96
+ gg = torch.Generator(device=device).manual_seed(30_000 + R * 100 + seed)
97
+ out = torch.empty(chains, steps, dtype=torch.uint8, device=device)
98
+ with torch.no_grad():
99
+ for t in range(steps):
100
+ probs = F.softmax(s @ W, dim=-1)
101
+ out[:, t] = torch.multinomial(probs, 1, generator=gg).squeeze(1)
102
+ s = torch.tanh(s @ A.T
103
+ + 0.10 * torch.randn(chains, R, generator=gg,
104
+ device=device))
105
+ return out.reshape(-1)[:n_bytes].cpu()
106
+
107
+
108
+ def get_streams(R):
109
+ os.makedirs(f"{DATA_DIR}/streams", exist_ok=True)
110
+ p = f"{DATA_DIR}/streams/R{R}_{TRAIN_BYTES}.pt"
111
+ if os.path.exists(p):
112
+ d = torch.load(p, weights_only=True)
113
+ else:
114
+ print(f" generating rank-{R} stream ({TRAIN_BYTES:,}+{VAL_BYTES:,} bytes)",
115
+ flush=True)
116
+ d = {"train": make_stream(R, TRAIN_BYTES, seed=0),
117
+ "val": make_stream(R, VAL_BYTES, seed=1)} # val = fresh chains
118
+ torch.save(d, p)
119
+ ent = -sum((c := torch.bincount(d['val'].long(), minlength=256).float()
120
+ / d['val'].numel())[c > 0]
121
+ * (c[c > 0]).log2()) if False else None # (unigram, optional)
122
+ return d["train"], d["val"]
123
+
124
+
125
+ def census_lite(model, va, g):
126
+ """Address census on one val batch: pos/neg half mass split + address
127
+ erank — the A1 instrument's AR-bed cousin."""
128
+ xv, _ = bed._batch(va, BATCH, BLOCK, DEV, g)
129
+ stats = []
130
+ with torch.no_grad():
131
+ x = model.emb(xv) + model.pos[:, : xv.shape[1]]
132
+ for blk in model.blocks:
133
+ h = blk.n1(x)
134
+ attn = blk.attn
135
+ qp, qn = attn.addr.oriented(attn.q(h))
136
+ mp, mn = float(qp.sum()), float(qn.sum())
137
+ flat = torch.cat([qp, qn], -1).reshape(-1, 2 * attn.addr.K)
138
+ sv = torch.linalg.svdvals(flat[:4096].float())
139
+ s2 = sv ** 2
140
+ stats.append({"neg_mass_frac": round(mn / (mp + mn + 1e-9), 4),
141
+ "addr_erank": round(float(s2.sum() ** 2
142
+ / (s2 ** 2).sum()), 2)})
143
+ x = blk(x)
144
+ return stats
145
+
146
+
147
+ def run_cell(R, K, seed, tr, va):
148
+ tag = f"R{R}_K{K}_s{seed}"
149
+ out_p = f"{OUT_DIR}/{tag}.json"
150
+ if os.path.exists(out_p):
151
+ print(f"-- {tag}: exists, skip", flush=True)
152
+ return json.load(open(out_p))
153
+ torch.manual_seed(seed)
154
+ g = torch.Generator().manual_seed(seed)
155
+ model = bed.ByteLM("hub", block=BLOCK, K=K, D=4).to(DEV)
156
+ n_par = sum(p.numel() for p in model.parameters())
157
+ opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
158
+ curve = []
159
+ for step in range(1, STEPS + 1):
160
+ x, y = bed._batch(tr, BATCH, BLOCK, DEV, g)
161
+ loss = F.cross_entropy(model(x).reshape(-1, 256), y.reshape(-1))
162
+ opt.zero_grad(set_to_none=True)
163
+ loss.backward()
164
+ opt.step()
165
+ if step % EVAL_EVERY == 0 or step == STEPS:
166
+ model.eval()
167
+ with torch.no_grad():
168
+ ls = [F.cross_entropy(
169
+ model(xv).reshape(-1, 256), yv.reshape(-1)).item()
170
+ for xv, yv in (bed._batch(va, BATCH, BLOCK, DEV, g)
171
+ for _ in range(16))]
172
+ bpb = sum(ls) / len(ls) / math.log(2)
173
+ curve.append({"step": step, "val_bpb": round(bpb, 4)})
174
+ model.train()
175
+ model.eval()
176
+ res = {"R": R, "K": K, "seed": seed, "params": n_par,
177
+ "steps": STEPS, "batch": BATCH, "block": BLOCK,
178
+ "final_bpb": curve[-1]["val_bpb"], "curve": curve,
179
+ "census": census_lite(model, va, g),
180
+ "note": "matched steps+data, NOT matched FLOPs (HUB is O(nKd))"}
181
+ os.makedirs(OUT_DIR, exist_ok=True)
182
+ json.dump(res, open(out_p, "w"), indent=1)
183
+ print(f"-- {tag}: bpb {res['final_bpb']:.4f} ({n_par:,} params)",
184
+ flush=True)
185
+ del model
186
+ torch.cuda.empty_cache()
187
+ return res
188
+
189
+
190
+ def main():
191
+ print("=" * 78, flush=True)
192
+ print(f"V1 RANK x ARMY BATTERY — R{RANKS} x K{ARMIES} x s{SEEDS}, "
193
+ f"{STEPS} steps {'[QUICK]' if QUICK else ''}", flush=True)
194
+ print("=" * 78, flush=True)
195
+ api = HfApi(token=cb.hf_token())
196
+ results = []
197
+ for R in RANKS:
198
+ tr, va = get_streams(R)
199
+ for K in ARMIES:
200
+ for seed in SEEDS:
201
+ results.append(run_cell(R, K, seed, tr, va))
202
+ if not QUICK:
203
+ tag = f"R{R}_K{K}_s{seed}"
204
+ api.upload_file(
205
+ path_or_fileobj=f"{OUT_DIR}/{tag}.json",
206
+ path_in_repo=f"battery/v1_rank_army/{tag}.json",
207
+ repo_id=HUB_REPO,
208
+ commit_message=f"V1 cell {tag}")
209
+ # ---- knee analysis: smallest K within .01 bpb of the rank's best ----
210
+ verdict = {}
211
+ for R in RANKS:
212
+ by_k = {K: sum(r["final_bpb"] for r in results
213
+ if r["R"] == R and r["K"] == K)
214
+ / len(SEEDS) for K in ARMIES}
215
+ best = min(by_k.values())
216
+ knee = min(K for K, b in by_k.items() if b <= best + 0.01)
217
+ verdict[R] = {"bpb_by_K": {str(k): round(v, 4)
218
+ for k, v in by_k.items()},
219
+ "knee_K": knee}
220
+ print(f"R={R:>3}: " + " ".join(f"K{k}={v:.4f}"
221
+ for k, v in by_k.items())
222
+ + f" -> knee K={knee}", flush=True)
223
+ knees = [verdict[R]["knee_K"] for R in RANKS]
224
+ verdict["registered_prediction"] = (
225
+ "Rule 1 SUPPORTED (knee tracks rank)" if knees == sorted(knees)
226
+ and knees[0] < knees[-1] else
227
+ "Rule 1 NOT supported at this bed scale (knee flat or non-monotone)")
228
+ print("VERDICT:", verdict["registered_prediction"], flush=True)
229
+ out = f"{OUT_DIR}/v1_verdict.json"
230
+ json.dump(verdict, open(out, "w"), indent=1)
231
+ if not QUICK:
232
+ api.upload_file(path_or_fileobj=out,
233
+ path_in_repo="battery/v1_rank_army/v1_verdict.json",
234
+ repo_id=HUB_REPO,
235
+ commit_message="V1 verdict: knee-vs-rank table "
236
+ "(Rule 1 arbiter)")
237
+ print("DONE.", flush=True)
238
+
239
+
240
+ if __name__ == "__main__":
241
+ main()
trainers/v1b_recall_army_battery.py ADDED
@@ -0,0 +1,323 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # v1b_recall_army_battery.py — V1b: ADDRESSING-DEMAND x army-ladder under AR
3
+ # (overnight battery #2, 2026-08-09; the V1-LDS census showed filtering
4
+ # demands ~no addressing — this battery applies the addressing-bound
5
+ # pressure: in-context associative recall, B8's territory, key space
6
+ # rank-controlled.)
7
+ #
8
+ # Episodes: P=8 (key 2 bytes, val 1 byte) pairs + SEP + query key -> the
9
+ # model must emit the queried value at the answer position. Keys are sign
10
+ # patterns of R_key-dim latents projected through 16 fixed directions —
11
+ # key diversity (addressing demand) rises with R_key. Values fresh-random
12
+ # per episode: binding is IN-CONTEXT, only the address machinery can do it.
13
+ # Loss masked to ANSWER positions only.
14
+ #
15
+ # GRID: R_key {4,16,64} x K {16,64,256} x seeds {0,1} = 18 cells,
16
+ # 2500 steps. GAUGES: answer accuracy + answer bpb + census-lite
17
+ # (consumed addr erank = the demand gauge). REGISTERED PREDICTIONS:
18
+ # (i) knee tracks R_key; (ii) consumed addr erank rises with R_key;
19
+ # (iii) both => Rule 1 re-scoped to M = c * addressing-demand.
20
+ # Same honesty stamp as V1: matched steps/data, NOT matched FLOPs.
21
+ # RUN: python tools\v1b_recall_army_battery.py (HF_TOKEN, 4090)
22
+ # Env: V1B_STEPS (def 2500) V1B_QUICK=1
23
+ # =========================================================================
24
+
25
+ import importlib
26
+ import json
27
+ import math
28
+ import os
29
+ import sys
30
+
31
+ import torch
32
+ import torch.nn.functional as F
33
+
34
+
35
+ def _load(name):
36
+ if name in sys.modules:
37
+ return sys.modules[name]
38
+ try:
39
+ return importlib.import_module(name)
40
+ except ImportError:
41
+ here = globals().get("__file__")
42
+ if here is not None:
43
+ import pathlib
44
+ sys.path.insert(0, str(pathlib.Path(here).parent))
45
+ return importlib.import_module(name)
46
+ raise
47
+
48
+
49
+ bed = _load("ar_differentiation_bed")
50
+ class _cb: # public shim: token from env
51
+ @staticmethod
52
+ def hf_token():
53
+ import os as _os
54
+ return _os.environ.get("HF_TOKEN")
55
+ cb = _cb()
56
+ from huggingface_hub import HfApi # noqa: E402
57
+
58
+ assert torch.cuda.is_available(), "GPU only"
59
+ DEV = "cuda"
60
+ QUICK = os.environ.get("V1B_QUICK") == "1"
61
+ STEPS = int(os.environ.get("V1B_STEPS", "250" if QUICK else "8000"))
62
+ # CONTROL FINDING (2026-08-09 pre-battery): on this task sdpa_tri reads
63
+ # .9251 answer-acc at step 1500 while hub_tri reads .0561 — the harness is
64
+ # sound and in-context BINDING is address-capacity-hard for the linear
65
+ # aleph read (late-takeoff curve, the splat late-bloom signature). The
66
+ # battery therefore runs hub long (8k steps) with an sdpa reference arm
67
+ # per rank, and prediction (iii) becomes: K gates binding FORMATION.
68
+ BATCH, BLOCK = 32, 256
69
+ RKEYS = [4, 16] if QUICK else [4, 16, 64]
70
+ ARMIES = [16, 64] if QUICK else [16, 64, 256]
71
+ SEEDS = [0] if QUICK else [0, 1]
72
+ EVAL_EVERY = 100 if QUICK else 250
73
+ D_ADDR = int(os.environ.get("V1B_D", "4")) # address dims (V1c axis)
74
+ if os.environ.get("V1B_RKEYS"):
75
+ RKEYS = [int(x) for x in os.environ["V1B_RKEYS"].split(",")]
76
+ if os.environ.get("V1B_ARMIES"):
77
+ ARMIES = [int(x) for x in os.environ["V1B_ARMIES"].split(",")]
78
+ P = 8 # pairs per episode
79
+ EPI = P * 3 + 3 # 27 bytes: pairs + SEP + query(2)
80
+ SEP = 255 # bytes 0..253 for keys/vals
81
+ DATA_DIR = os.environ.get("V1B_DATA", "./data/v1_rank_army")
82
+ OUT_DIR = os.path.join(DATA_DIR, "results_v1b")
83
+ N_EPISODES = (3_000, 600) if QUICK else (60_000, 8_000) # train, val
84
+ HUB_REPO = "AbstractPhil/aleph-splat-0"
85
+
86
+
87
+ def make_episodes(R, n_epi, seed):
88
+ """Concatenated recall episodes + answer-position mask.
89
+ Key = 2 bytes from sign bits of a fresh R-dim latent through 16 fixed
90
+ directions (structural, R-only seed); vals fresh-random per episode."""
91
+ sg = torch.Generator().manual_seed(40_000 + R)
92
+ U = torch.randn(16, R, generator=sg) # fixed directions
93
+ g = torch.Generator().manual_seed(50_000 + R * 100 + seed)
94
+ z = torch.randn(n_epi, P, R, generator=g) # latents per pair
95
+ bits = (torch.einsum("epr,br->epb", z, U) > 0) # (n_epi, P, 16)
96
+ w1 = (1 << torch.arange(8))
97
+ k1 = (bits[..., :8].long() * w1).sum(-1) # 0..255
98
+ k2 = (bits[..., 8:].long() * w1).sum(-1)
99
+ k1, k2 = k1 % 254, k2 % 254 # keep off SEP
100
+ vals = torch.randint(0, 254, (n_epi, P), generator=g)
101
+ qi = torch.randint(0, P, (n_epi,), generator=g) # queried pair
102
+ seq = torch.zeros(n_epi, EPI, dtype=torch.uint8)
103
+ seq[:, 0:P * 3:3] = k1.to(torch.uint8)
104
+ seq[:, 1:P * 3:3] = k2.to(torch.uint8)
105
+ seq[:, 2:P * 3:3] = vals.to(torch.uint8)
106
+ seq[:, P * 3] = SEP
107
+ ar = torch.arange(n_epi)
108
+ seq[:, P * 3 + 1] = k1[ar, qi].to(torch.uint8)
109
+ seq[:, P * 3 + 2] = k2[ar, qi].to(torch.uint8)
110
+ answer = vals[ar, qi].to(torch.uint8) # target AFTER query
111
+ # stream layout: [...episode..., answer] so predicting the byte at the
112
+ # answer slot from the preceding query is exactly next-byte loss there.
113
+ flat = torch.cat([seq, answer.unsqueeze(1)], dim=1).reshape(-1)
114
+ mask = torch.zeros_like(flat, dtype=torch.bool)
115
+ mask[(EPI + 1) * ar + EPI] = True # answer positions
116
+ return flat, mask
117
+
118
+
119
+ def get_data(R):
120
+ os.makedirs(f"{DATA_DIR}/streams_v1b", exist_ok=True)
121
+ p = f"{DATA_DIR}/streams_v1b/R{R}_{N_EPISODES[0]}.pt"
122
+ if os.path.exists(p):
123
+ d = torch.load(p, weights_only=True)
124
+ else:
125
+ print(f" generating R_key={R} episodes {N_EPISODES}", flush=True)
126
+ tr, trm = make_episodes(R, N_EPISODES[0], seed=0)
127
+ va, vam = make_episodes(R, N_EPISODES[1], seed=1)
128
+ d = {"tr": tr, "trm": trm, "va": va, "vam": vam}
129
+ torch.save(d, p)
130
+ return d
131
+
132
+
133
+ def batch_masked(flat, mask, batch, block, device, g):
134
+ ix = torch.randint(0, flat.numel() - block - 1, (batch,), generator=g)
135
+ x = torch.stack([flat[i:i + block] for i in ix]).long().to(device)
136
+ y = torch.stack([flat[i + 1:i + block + 1] for i in ix]).long().to(device)
137
+ m = torch.stack([mask[i + 1:i + block + 1] for i in ix]).to(device)
138
+ if os.environ.get("V1B_CLEAN_MASK") == "1":
139
+ # 2026-08-10 corruption repair: drop answer positions whose episode
140
+ # is window-truncated (~6.05% unanswerable — 4x-weighted label
141
+ # noise otherwise; hunter-verified j>=26 = answerable frac 1.0000)
142
+ m[:, :26] = False
143
+ return x, y, m
144
+
145
+
146
+ def masked_eval(model, d, g, iters=16):
147
+ accs, bits = [], []
148
+ with torch.no_grad():
149
+ for _ in range(iters):
150
+ xv, yv, mv = batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
151
+ if mv.sum() == 0:
152
+ continue
153
+ logits = model(xv)
154
+ la, ya = logits[mv], yv[mv]
155
+ bits.append(F.cross_entropy(la, ya).item() / math.log(2))
156
+ accs.append((la.argmax(-1) == ya).float().mean().item())
157
+ return (sum(accs) / len(accs), sum(bits) / len(bits)) if accs else (0.0, 8.0)
158
+
159
+
160
+ def census_lite(model, d, g):
161
+ xv, _, _ = batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
162
+ stats = []
163
+ with torch.no_grad():
164
+ x = model.emb(xv)
165
+ if getattr(model, "trigram", False): # 2026-08-09 audit fix: the
166
+ # census must see the SAME composed embedding the model runs
167
+ # on (was reading 1 of 3 tables — demand gauge underweighted)
168
+ pv = 256 if getattr(model, "pad_fix", False) else 0
169
+ x = x + model.emb1(F.pad(xv, (1, 0), value=pv)[:, :-1]) \
170
+ + model.emb2(F.pad(xv, (2, 0), value=pv)[:, :-2])
171
+ x = x + model.pos[:, : xv.shape[1]]
172
+ for blk in model.blocks:
173
+ h = blk.n1(x)
174
+ attn = blk.attn
175
+ qp, qn = attn.addr.oriented(attn.q(h))
176
+ mp, mn = float(qp.sum()), float(qn.sum())
177
+ fl = torch.cat([qp, qn], -1).reshape(-1, 2 * attn.addr.K)
178
+ sv = torch.linalg.svdvals(fl[:4096].float())
179
+ s2 = sv ** 2
180
+ stats.append({"neg_mass_frac": round(mn / (mp + mn + 1e-9), 4),
181
+ "addr_erank": round(float(s2.sum() ** 2
182
+ / (s2 ** 2).sum()), 2)})
183
+ x = blk(x)
184
+ return stats
185
+
186
+
187
+ def run_cell(R, K, seed, d):
188
+ tag = f"R{R}_K{K}_s{seed}" + (f"_D{D_ADDR}" if D_ADDR != 4 else "")
189
+ out_p = f"{OUT_DIR}/{tag}.json"
190
+ if os.path.exists(out_p):
191
+ print(f"-- {tag}: exists, skip", flush=True)
192
+ return json.load(open(out_p))
193
+ torch.manual_seed(seed)
194
+ g = torch.Generator().manual_seed(seed)
195
+ arm = "sdpa_tri" if K == 0 else "hub_tri" # K=0 = the control arm
196
+ model = bed.ByteLM(arm, block=BLOCK, K=max(K, 4), D=D_ADDR).to(DEV)
197
+ opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
198
+ curve = []
199
+ for step in range(1, STEPS + 1):
200
+ x, y, m = batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
201
+ logits = model(x)
202
+ if m.sum() == 0:
203
+ continue
204
+ # DENSE loss builds the sequence circuits (masked-only starves
205
+ # them — smoke 2026-08-09: chance acc, bpb>10); answers get 4x
206
+ # weight and remain the ONLY reported gauge.
207
+ loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
208
+ + 4.0 * F.cross_entropy(logits[m], y[m]))
209
+ opt.zero_grad(set_to_none=True)
210
+ loss.backward()
211
+ opt.step()
212
+ if step % EVAL_EVERY == 0 or step == STEPS:
213
+ model.eval()
214
+ acc, bpb = masked_eval(model, d, g)
215
+ curve.append({"step": step, "ans_acc": round(acc, 4),
216
+ "ans_bpb": round(bpb, 4)})
217
+ model.train()
218
+ model.eval()
219
+ acc, bpb = masked_eval(model, d, g, iters=32)
220
+ res = {"R_key": R, "K": K, "seed": seed, "steps": STEPS, "arm": arm,
221
+ "final_acc": round(acc, 4), "final_bpb": round(bpb, 4),
222
+ "curve": curve,
223
+ "census": census_lite(model, d, g) if K else None,
224
+ "note": "loss+gauges on answer positions only; matched steps, "
225
+ "NOT matched FLOPs (HUB O(nKd))"}
226
+ os.makedirs(OUT_DIR, exist_ok=True)
227
+ json.dump(res, open(out_p, "w"), indent=1)
228
+ print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f}", flush=True)
229
+ del model
230
+ torch.cuda.empty_cache()
231
+ return res
232
+
233
+
234
+ @torch.no_grad()
235
+ def selection_eval(model, d, g, iters=32):
236
+ """Error decomposition (AbstractPhil lineage 2026-08-09): of WRONG answers, the
237
+ fraction that are OTHER IN-CONTEXT VALUES (mis-selection = retrieved
238
+ the wrong pair — the 'incorrect selections' fault class) vs bytes from
239
+ outside the episode's value set (no-retrieval). Only full-episode
240
+ windows (mask index >= 26) are scored."""
241
+ n_right = n_missel = n_noret = 0
242
+ for _ in range(iters):
243
+ xv, yv, mv = batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
244
+ pred = model(xv).argmax(-1)
245
+ for b, j in mv.nonzero(as_tuple=False).tolist():
246
+ if j < 26:
247
+ continue
248
+ vals = {int(xv[b, j - 24 + 3 * k]) for k in range(P)}
249
+ p, t = int(pred[b, j]), int(yv[b, j])
250
+ if p == t:
251
+ n_right += 1
252
+ elif p in vals:
253
+ n_missel += 1
254
+ else:
255
+ n_noret += 1
256
+ tot = n_right + n_missel + n_noret
257
+ wrong = n_missel + n_noret
258
+ return {"n": tot, "acc": round(n_right / max(tot, 1), 4),
259
+ "mis_sel_frac_of_errors": round(n_missel / max(wrong, 1), 4),
260
+ "no_retrieval_frac_of_errors": round(n_noret / max(wrong, 1), 4)}
261
+
262
+
263
+ def main():
264
+ print("=" * 78, flush=True)
265
+ print(f"V1b RECALL x ARMY — R_key{RKEYS} x K{ARMIES} x s{SEEDS}, "
266
+ f"{STEPS} steps {'[QUICK]' if QUICK else ''}", flush=True)
267
+ print("=" * 78, flush=True)
268
+ api = HfApi(token=cb.hf_token())
269
+ results = []
270
+ for R in RKEYS:
271
+ d = get_data(R)
272
+ for K in ([0] + ARMIES if not QUICK else ARMIES): # K=0 = sdpa ref
273
+ for seed in (SEEDS if K else [0]):
274
+ results.append(run_cell(R, K, seed, d))
275
+ if not QUICK:
276
+ tag = f"R{R}_K{K}_s{seed}" + (f"_D{D_ADDR}" if D_ADDR != 4 else "")
277
+ api.upload_file(
278
+ path_or_fileobj=f"{OUT_DIR}/{tag}.json",
279
+ path_in_repo=f"battery/v1b_recall_army/{tag}.json",
280
+ repo_id=HUB_REPO, commit_message=f"V1b cell {tag}")
281
+ verdict = {}
282
+ for R in RKEYS:
283
+ by_k = {K: sum(r["final_acc"] for r in results
284
+ if r["R_key"] == R and r["K"] == K) / len(SEEDS)
285
+ for K in ARMIES}
286
+ best = max(by_k.values())
287
+ knee = min(K for K, a in by_k.items() if a >= best - 0.02)
288
+ er = {K: sum(max(c["addr_erank"] for c in r["census"])
289
+ for r in results if r["R_key"] == R and r["K"] == K)
290
+ / len(SEEDS) for K in ARMIES}
291
+ sdpa_ref = next((r["final_acc"] for r in results
292
+ if r["R_key"] == R and r["K"] == 0), None)
293
+ verdict[R] = {"acc_by_K": {str(k): round(v, 4)
294
+ for k, v in by_k.items()},
295
+ "knee_K": knee, "sdpa_ref_acc": sdpa_ref,
296
+ "max_addr_erank_by_K": {str(k): round(v, 1)
297
+ for k, v in er.items()}}
298
+ print(f"R_key={R:>3}: "
299
+ + " ".join(f"K{k}={v:.4f}" for k, v in by_k.items())
300
+ + f" -> knee K={knee} | consumed erank "
301
+ + " ".join(f"{v:.0f}" for v in er.values()), flush=True)
302
+ knees = [verdict[R]["knee_K"] for R in RKEYS]
303
+ verdict["prediction_i_knee_tracks_Rkey"] = (
304
+ knees == sorted(knees) and knees[0] < knees[-1])
305
+ ers = [max(verdict[R]["max_addr_erank_by_K"].values()) for R in RKEYS]
306
+ verdict["prediction_ii_demand_rises_with_Rkey"] = ers == sorted(ers)
307
+ print(f"VERDICT: (i) knee tracks R_key: "
308
+ f"{verdict['prediction_i_knee_tracks_Rkey']} "
309
+ f"(ii) consumed demand rises: "
310
+ f"{verdict['prediction_ii_demand_rises_with_Rkey']}", flush=True)
311
+ out = f"{OUT_DIR}/v1b_verdict.json"
312
+ json.dump(verdict, open(out, "w"), indent=1)
313
+ if not QUICK:
314
+ api.upload_file(path_or_fileobj=out,
315
+ path_in_repo="battery/v1b_recall_army/v1b_verdict.json",
316
+ repo_id=HUB_REPO,
317
+ commit_message="V1b verdict: recall knee + "
318
+ "consumed-demand table")
319
+ print("DONE.", flush=True)
320
+
321
+
322
+ if __name__ == "__main__":
323
+ main()
trainers/v1d_svae_graft.py ADDED
@@ -0,0 +1,230 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # v1d_svae_graft.py — V1d: SVAE-transformer pieces vs the binding gap
3
+ # (2026-08-09; AbstractPhil: "try taking some pieces from the SVAE transformer
4
+ # concept, see if we can shorten the gap, or if the svae-transformer
5
+ # projection concept is simply obsolete at this point.")
6
+ #
7
+ # Bed point: R64 recall (v1b data), K=64, D=16 — baseline .599/.722
8
+ # (mean .661), sdpa ref .936. Winner (if any) confirms at K=256/D=16
9
+ # (baseline .678/.778). Grafts from canon/svae_h2_class.md (May 29 arc):
10
+ # sphere — H2 chain "linear->sphere-norm->canon": per-token sphere
11
+ # normalization BEFORE the q/k projections (the address
12
+ # already cos-normalizes AFTER; this changes what the
13
+ # projection learns).
14
+ # lens — fixed isometric lens (0-param orthogonal DxD between
15
+ # projection and address). PREDICTED NULL by inspection:
16
+ # with a LEARNABLE codebook, cos(Lx,a)=cos(x,L^T a) — the
17
+ # lens is absorbed into codebook learning. Runs 1 seed as
18
+ # the labeled math-check.
19
+ # boundgate — attenuator grammar lite: hub output enters the residual
20
+ # through a bounded learnable gate a<=0.2, init .024
21
+ # ("modulate never inject" + the C6 null-path lineage).
22
+ # modulate — the literal H2 form: y = x * (1 + a*tanh(hub_out)),
23
+ # multiplicative, bounded, never additive injection.
24
+ # combo — sphere + boundgate (the two live-hypothesis pieces).
25
+ # OBSOLESCENCE GATE: any arm closing >= half the remaining gap
26
+ # (mean acc >= .80 at K64/D16) = SVAE pieces LIVE; all arms within +.02
27
+ # of baseline = the projection concept is OBSOLETE here (its function
28
+ # absorbed by cos addressing + D-matching + K-provisioning).
29
+ # RUN: python tools\v1d_svae_graft.py (HF_TOKEN, 4090)
30
+ # Env: V1D_ARMS, V1D_STEPS (def 8000), V1D_K (def 64)
31
+ # =========================================================================
32
+
33
+ import importlib
34
+ import json
35
+ import os
36
+ import sys
37
+
38
+ import torch
39
+ import torch.nn as nn
40
+ import torch.nn.functional as F
41
+
42
+
43
+ def _load(name):
44
+ if name in sys.modules:
45
+ return sys.modules[name]
46
+ try:
47
+ return importlib.import_module(name)
48
+ except ImportError:
49
+ here = globals().get("__file__")
50
+ if here is not None:
51
+ import pathlib
52
+ sys.path.insert(0, str(pathlib.Path(here).parent))
53
+ return importlib.import_module(name)
54
+ raise
55
+
56
+
57
+ bed = _load("ar_differentiation_bed")
58
+ v1b = _load("v1b_recall_army_battery")
59
+ class _cb: # public shim: token from env
60
+ @staticmethod
61
+ def hf_token():
62
+ import os as _os
63
+ return _os.environ.get("HF_TOKEN")
64
+ cb = _cb()
65
+ from huggingface_hub import HfApi # noqa: E402
66
+
67
+ assert torch.cuda.is_available(), "GPU only"
68
+ DEV = "cuda"
69
+ STEPS = int(os.environ.get("V1D_STEPS", "8000"))
70
+ K_ARMY = int(os.environ.get("V1D_K", "64"))
71
+ D_ADDR = 16 # matched (V1c law)
72
+ R_KEY = 64
73
+ ARMS = os.environ.get(
74
+ "V1D_ARMS", "sphere,lens,boundgate,modulate,combo").split(",")
75
+ BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
76
+ OUT_DIR = os.path.join(v1b.DATA_DIR, "results_v1d")
77
+ HUB_REPO = v1b.HUB_REPO
78
+
79
+
80
+ class GraftHUB(bed.CausalHUB):
81
+ """CausalHUB + SVAE grafts. sphere: normalize x before q/k projections.
82
+ lens: fixed orthogonal DxD after projection (0 params, buffer)."""
83
+
84
+ def __init__(self, d, K, D, sphere=False, lens=False):
85
+ super().__init__(d, K, D)
86
+ self.sphere = sphere
87
+ if lens:
88
+ L = torch.linalg.qr(torch.randn(D, D))[0]
89
+ self.register_buffer("lens", L)
90
+ else:
91
+ self.lens = None
92
+
93
+ def forward(self, x):
94
+ xa = F.normalize(x, dim=-1) if self.sphere else x
95
+ q, k = self.q(xa), self.k(xa)
96
+ if self.lens is not None:
97
+ q, k = q @ self.lens, k @ self.lens
98
+ qp, qn = self.addr.oriented(q)
99
+ kp, kn = self.addr.oriented(k)
100
+ v = self.v(x)
101
+ Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
102
+ Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
103
+ zp = torch.cumsum(kp, dim=1)
104
+ zn = torch.cumsum(kn, dim=1)
105
+ num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
106
+ + torch.einsum("bnk,bnkd->bnd", qn, Sn))
107
+ den = ((qp * zp).sum(-1, keepdim=True)
108
+ + (qn * zn).sum(-1, keepdim=True))
109
+ return self.o(num / den.clamp_min(1e-12))
110
+
111
+
112
+ class GraftBlock(nn.Module):
113
+ """Block with the SVAE residual grammars. inject: y = x + attn (bed
114
+ default). boundgate: y = x + min(a,0.2)*attn, a learnable init .024.
115
+ modulate: y = x * (1 + a*tanh(attn)) — multiplicative, never additive."""
116
+
117
+ def __init__(self, d, attn, grammar="inject"):
118
+ super().__init__()
119
+ self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
120
+ self.attn = attn
121
+ self.grammar = grammar
122
+ if grammar in ("boundgate", "modulate"):
123
+ self.alpha = nn.Parameter(torch.tensor(0.024))
124
+ self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),
125
+ nn.Linear(4 * d, d))
126
+
127
+ def forward(self, x):
128
+ a = self.attn(self.n1(x))
129
+ if self.grammar == "boundgate":
130
+ x = x + self.alpha.clamp(0, 0.2) * a
131
+ elif self.grammar == "modulate":
132
+ x = x * (1.0 + self.alpha.clamp(0, 0.2) * torch.tanh(a))
133
+ else:
134
+ x = x + a
135
+ return x + self.mlp(self.n2(x))
136
+
137
+
138
+ def build_model(arm, d=192, layers=4):
139
+ sphere = arm in ("sphere", "combo")
140
+ lens = arm == "lens"
141
+ grammar = ("boundgate" if arm in ("boundgate", "combo")
142
+ else "modulate" if arm == "modulate" else "inject")
143
+ m = bed.ByteLM("hub_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
144
+ m.blocks = nn.ModuleList([
145
+ GraftBlock(d, GraftHUB(d, K_ARMY, D_ADDR, sphere, lens), grammar)
146
+ for _ in range(layers)])
147
+ return m
148
+
149
+
150
+ def run_cell(arm, seed, d):
151
+ tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
152
+ out_p = f"{OUT_DIR}/{tag}.json"
153
+ if os.path.exists(out_p):
154
+ print(f"-- {tag}: exists, skip", flush=True)
155
+ return json.load(open(out_p))
156
+ torch.manual_seed(seed)
157
+ g = torch.Generator().manual_seed(seed)
158
+ model = build_model(arm).to(DEV)
159
+ opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
160
+ for step in range(1, STEPS + 1):
161
+ x, y, m = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
162
+ logits = model(x)
163
+ if m.sum() == 0:
164
+ continue
165
+ loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
166
+ + 4.0 * F.cross_entropy(logits[m], y[m]))
167
+ opt.zero_grad(set_to_none=True)
168
+ loss.backward()
169
+ opt.step()
170
+ model.eval()
171
+ acc, bpb = v1b.masked_eval(model, d, g, iters=32)
172
+ alphas = [round(float(b.alpha.detach()), 4) for b in model.blocks
173
+ if hasattr(b, "alpha")]
174
+ res = {"arm": arm, "seed": seed, "R": R_KEY, "K": K_ARMY, "D": D_ADDR,
175
+ "steps": STEPS, "final_acc": round(acc, 4),
176
+ "final_bpb": round(bpb, 4), "alphas": alphas,
177
+ "baseline_ref": {"K64_D16": [0.5994, 0.7224],
178
+ "K256_D16": [0.6783, 0.7776], "sdpa": 0.936}}
179
+ os.makedirs(OUT_DIR, exist_ok=True)
180
+ json.dump(res, open(out_p, "w"), indent=1)
181
+ print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f}"
182
+ + (f" alphas {alphas}" if alphas else ""), flush=True)
183
+ del model
184
+ torch.cuda.empty_cache()
185
+ return res
186
+
187
+
188
+ def main():
189
+ print("=" * 78, flush=True)
190
+ print(f"V1d SVAE GRAFTS — arms {ARMS} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}, "
191
+ f"{STEPS} steps (baseline mean .661, sdpa .936)", flush=True)
192
+ print("=" * 78, flush=True)
193
+ api = HfApi(token=cb.hf_token())
194
+ d = v1b.get_data(R_KEY)
195
+ results = []
196
+ for arm in ARMS:
197
+ seeds = [0] if arm == "lens" else [0, 1]
198
+ for seed in seeds:
199
+ results.append(run_cell(arm, seed, d))
200
+ tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
201
+ api.upload_file(path_or_fileobj=f"{OUT_DIR}/{tag}.json",
202
+ path_in_repo=f"battery/v1d_svae_graft/{tag}.json",
203
+ repo_id=HUB_REPO, commit_message=f"V1d {tag}")
204
+ base = 0.6609
205
+ verdict = {}
206
+ for arm in ARMS:
207
+ accs = [r["final_acc"] for r in results if r["arm"] == arm]
208
+ verdict[arm] = {"acc": accs,
209
+ "mean": round(sum(accs) / len(accs), 4),
210
+ "delta_vs_base": round(sum(accs) / len(accs) - base, 4)}
211
+ print(f"{arm:>10}: {verdict[arm]['mean']:.4f} "
212
+ f"({verdict[arm]['delta_vs_base']:+.4f} vs baseline)", flush=True)
213
+ best = max(verdict, key=lambda a: verdict[a]["mean"])
214
+ live = verdict[best]["mean"] >= 0.80
215
+ obsolete = all(v["mean"] <= base + 0.02 for v in verdict.values())
216
+ verdict["gate"] = ("SVAE PIECES LIVE" if live else
217
+ "OBSOLETE HERE (absorbed)" if obsolete else
218
+ f"PARTIAL: best={best} {verdict[best]['mean']:.4f}")
219
+ print("GATE:", verdict["gate"], flush=True)
220
+ out = f"{OUT_DIR}/v1d_verdict.json"
221
+ json.dump(verdict, open(out, "w"), indent=1)
222
+ api.upload_file(path_or_fileobj=out,
223
+ path_in_repo="battery/v1d_svae_graft/v1d_verdict.json",
224
+ repo_id=HUB_REPO,
225
+ commit_message="V1d verdict: SVAE grafts vs binding gap")
226
+ print("DONE.", flush=True)
227
+
228
+
229
+ if __name__ == "__main__":
230
+ main()
trainers/v1e_knit_battery.py ADDED
@@ -0,0 +1,250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # v1e_knit_battery.py — V1e: the conv-knit + patchwork grafts vs the gap
3
+ # (2026-08-09; AbstractPhil: "Consult the alpha attention mixed with the SVAE
4
+ # patchwork's edge solidification from the conv mechanism to knit, that
5
+ # could be the answer we require.")
6
+ #
7
+ # Consult results (canon/constellation_forms.md universal rules +
8
+ # canon/svae_h2_class.md): patchwork spec Linear->SquaredReLU->LN->Linear,
9
+ # SquaredReLU NEVER GELU (anchor diversity 7.1 vs 4.6); conv solidifies
10
+ # edges between patch codes; gate init -3.0 (theme D). Bed point: R64
11
+ # recall, K=64, D=16 (baseline .661; V1c best K256 .728; sdpa .936).
12
+ # ARMS:
13
+ # knitseq — depthwise CAUSAL conv (kernel 8) over the HUB output,
14
+ # gated residual at -3.0: adjacent positions' read codes
15
+ # knit into a locally solid field (edge solidification in
16
+ # SEQUENCE).
17
+ # knitaddr — grouped conv (kernel 3) over the K-axis of the oriented
18
+ # address (qp/qn/kp/kn): NEIGHBORING CELLS couple — B6's
19
+ # overlap made structural (edge solidification in ADDRESS).
20
+ # patchhead — the HUB o-projection replaced by the constellation
21
+ # patchwork generator (Linear d->2d -> SquaredReLU -> LN ->
22
+ # Linear 2d->d): the read GENERATES from the address per the
23
+ # patchwork grammar.
24
+ # alphaknit — knitseq + boundgate alpha residual (the literal "alpha
25
+ # attention mixed with the edge-knit" ask).
26
+ # GATE (same as V1d): >= .80 mean = LIVE (half the remaining gap);
27
+ # within +.02 of .661 = the knit adds nothing here.
28
+ # Chained: waits for V1d's verdict file before starting (one GPU).
29
+ # RUN: python tools\v1e_knit_battery.py (HF_TOKEN, 4090)
30
+ # =========================================================================
31
+
32
+ import importlib
33
+ import json
34
+ import os
35
+ import sys
36
+
37
+ import torch
38
+ import torch.nn as nn
39
+ import torch.nn.functional as F
40
+
41
+
42
+ def _load(name):
43
+ if name in sys.modules:
44
+ return sys.modules[name]
45
+ try:
46
+ return importlib.import_module(name)
47
+ except ImportError:
48
+ here = globals().get("__file__")
49
+ if here is not None:
50
+ import pathlib
51
+ sys.path.insert(0, str(pathlib.Path(here).parent))
52
+ return importlib.import_module(name)
53
+ raise
54
+
55
+
56
+ bed = _load("ar_differentiation_bed")
57
+ v1b = _load("v1b_recall_army_battery")
58
+ v1d = _load("v1d_svae_graft")
59
+ class _cb: # public shim: token from env
60
+ @staticmethod
61
+ def hf_token():
62
+ import os as _os
63
+ return _os.environ.get("HF_TOKEN")
64
+ cb = _cb()
65
+ from huggingface_hub import HfApi # noqa: E402
66
+
67
+ assert torch.cuda.is_available(), "GPU only"
68
+ DEV = "cuda"
69
+ STEPS = int(os.environ.get("V1E_STEPS", "8000"))
70
+ K_ARMY, D_ADDR, R_KEY = 64, 16, 64
71
+ ARMS = os.environ.get(
72
+ "V1E_ARMS", "knitseq,knitaddr,patchhead,alphaknit").split(",")
73
+ BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
74
+ OUT_DIR = os.path.join(v1b.DATA_DIR, "results_v1e")
75
+ HUB_REPO = v1b.HUB_REPO
76
+
77
+
78
+ class SquaredReLU(nn.Module):
79
+ def forward(self, x):
80
+ return F.relu(x) ** 2
81
+
82
+
83
+ class KnitHUB(bed.CausalHUB):
84
+ """CausalHUB + the knit grafts."""
85
+
86
+ def __init__(self, d, K, D, knitaddr=False, patchhead=False):
87
+ super().__init__(d, K, D)
88
+ self.ka = None
89
+ if knitaddr:
90
+ # grouped conv over the K axis; SAME padding, shared across
91
+ # the four oriented maps; identity-init (weight peak at center)
92
+ self.ka = nn.Conv1d(1, 1, kernel_size=3, padding=1, bias=False)
93
+ w = torch.zeros(1, 1, 3)
94
+ w[0, 0, 1] = 1.0 # identity at birth
95
+ self.ka.weight = nn.Parameter(w)
96
+ if patchhead:
97
+ self.o = nn.Sequential( # constellation patchwork
98
+ nn.Linear(d, 2 * d), SquaredReLU(), nn.LayerNorm(2 * d),
99
+ nn.Linear(2 * d, d))
100
+
101
+ def _knit(self, a):
102
+ if self.ka is None:
103
+ return a
104
+ B, n, K = a.shape
105
+ return self.ka(a.reshape(B * n, 1, K)).reshape(B, n, K)
106
+
107
+ def forward(self, x):
108
+ qp, qn = self.addr.oriented(self.q(x))
109
+ kp, kn = self.addr.oriented(self.k(x))
110
+ qp, qn, kp, kn = (self._knit(t) for t in (qp, qn, kp, kn))
111
+ v = self.v(x)
112
+ Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
113
+ Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
114
+ zp = torch.cumsum(kp, dim=1)
115
+ zn = torch.cumsum(kn, dim=1)
116
+ num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
117
+ + torch.einsum("bnk,bnkd->bnd", qn, Sn))
118
+ den = ((qp * zp).sum(-1, keepdim=True)
119
+ + (qn * zn).sum(-1, keepdim=True))
120
+ return self.o(num / den.clamp_min(1e-12))
121
+
122
+
123
+ class KnitBlock(nn.Module):
124
+ """Block with optional sequence-knit (depthwise causal conv over the
125
+ attn output, gated at -3.0) and optional bounded-alpha residual."""
126
+
127
+ def __init__(self, d, attn, knitseq=False, boundgate=False):
128
+ super().__init__()
129
+ self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
130
+ self.attn = attn
131
+ self.ks = None
132
+ if knitseq:
133
+ self.ks = nn.Conv1d(d, d, kernel_size=8, groups=d, bias=False)
134
+ self.ks_gate = nn.Parameter(torch.tensor(-3.0))
135
+ self.alpha = nn.Parameter(torch.tensor(0.024)) if boundgate else None
136
+ self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),
137
+ nn.Linear(4 * d, d))
138
+
139
+ def forward(self, x):
140
+ a = self.attn(self.n1(x))
141
+ if self.ks is not None:
142
+ ap = F.pad(a.transpose(1, 2), (7, 0)) # causal left-pad
143
+ knit = self.ks(ap).transpose(1, 2)
144
+ a = a + torch.sigmoid(self.ks_gate) * knit
145
+ if self.alpha is not None:
146
+ x = x + self.alpha.clamp(0, 0.2) * a
147
+ else:
148
+ x = x + a
149
+ return x + self.mlp(self.n2(x))
150
+
151
+
152
+ def build_model(arm, d=192, layers=4):
153
+ knitseq = arm in ("knitseq", "alphaknit")
154
+ knitaddr = arm == "knitaddr"
155
+ patchhead = arm == "patchhead"
156
+ boundgate = arm == "alphaknit"
157
+ m = bed.ByteLM("hub_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
158
+ m.blocks = nn.ModuleList([
159
+ KnitBlock(d, KnitHUB(d, K_ARMY, D_ADDR, knitaddr, patchhead),
160
+ knitseq, boundgate)
161
+ for _ in range(layers)])
162
+ return m
163
+
164
+
165
+ def run_cell(arm, seed, d):
166
+ tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
167
+ out_p = f"{OUT_DIR}/{tag}.json"
168
+ if os.path.exists(out_p):
169
+ print(f"-- {tag}: exists, skip", flush=True)
170
+ return json.load(open(out_p))
171
+ torch.manual_seed(seed)
172
+ g = torch.Generator().manual_seed(seed)
173
+ model = build_model(arm).to(DEV)
174
+ opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
175
+ for step in range(1, STEPS + 1):
176
+ x, y, m = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
177
+ logits = model(x)
178
+ if m.sum() == 0:
179
+ continue
180
+ loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
181
+ + 4.0 * F.cross_entropy(logits[m], y[m]))
182
+ opt.zero_grad(set_to_none=True)
183
+ loss.backward()
184
+ opt.step()
185
+ model.eval()
186
+ acc, bpb = v1b.masked_eval(model, d, g, iters=32)
187
+ gates = {}
188
+ b0 = model.blocks[0]
189
+ if b0.ks is not None:
190
+ gates["ks_gate_sig"] = [round(float(torch.sigmoid(b.ks_gate)), 4)
191
+ for b in model.blocks]
192
+ if b0.alpha is not None:
193
+ gates["alpha"] = [round(float(b.alpha), 4) for b in model.blocks]
194
+ res = {"arm": arm, "seed": seed, "R": R_KEY, "K": K_ARMY, "D": D_ADDR,
195
+ "steps": STEPS, "final_acc": round(acc, 4),
196
+ "final_bpb": round(bpb, 4), "gates": gates,
197
+ "baseline_ref": {"K64_D16_mean": 0.6609, "sdpa": 0.936}}
198
+ os.makedirs(OUT_DIR, exist_ok=True)
199
+ json.dump(res, open(out_p, "w"), indent=1)
200
+ print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f} {gates}", flush=True)
201
+ del model
202
+ torch.cuda.empty_cache()
203
+ return res
204
+
205
+
206
+ def main():
207
+ print("=" * 78, flush=True)
208
+ print(f"V1e KNIT BATTERY — arms {ARMS} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}, "
209
+ f"{STEPS} steps (baseline .661, sdpa .936)", flush=True)
210
+ print("=" * 78, flush=True)
211
+ api = HfApi(token=cb.hf_token())
212
+ d = v1b.get_data(R_KEY)
213
+ results = []
214
+ for arm in ARMS:
215
+ for seed in (0, 1):
216
+ results.append(run_cell(arm, seed, d))
217
+ tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
218
+ api.upload_file(path_or_fileobj=f"{OUT_DIR}/{tag}.json",
219
+ path_in_repo=f"battery/v1e_knit/{tag}.json",
220
+ repo_id=HUB_REPO, commit_message=f"V1e {tag}")
221
+ base = 0.6609
222
+ verdict = {}
223
+ for arm in ARMS:
224
+ accs = [r["final_acc"] for r in results if r["arm"] == arm]
225
+ verdict[arm] = {"acc": accs,
226
+ "mean": round(sum(accs) / len(accs), 4),
227
+ "delta_vs_base": round(sum(accs) / len(accs) - base,
228
+ 4)}
229
+ print(f"{arm:>10}: {verdict[arm]['mean']:.4f} "
230
+ f"({verdict[arm]['delta_vs_base']:+.4f})", flush=True)
231
+ best = max(verdict, key=lambda a: verdict[a]["mean"])
232
+ verdict["gate"] = ("KNIT LIVE" if verdict[best]["mean"] >= 0.80 else
233
+ "KNIT NULL HERE" if all(
234
+ v["mean"] <= base + 0.02
235
+ for k, v in verdict.items() if k != "gate")
236
+ else f"PARTIAL: best={best} "
237
+ f"{verdict[best]['mean']:.4f}")
238
+ print("GATE:", verdict["gate"], flush=True)
239
+ out = f"{OUT_DIR}/v1e_verdict.json"
240
+ json.dump(verdict, open(out, "w"), indent=1)
241
+ api.upload_file(path_or_fileobj=out,
242
+ path_in_repo="battery/v1e_knit/v1e_verdict.json",
243
+ repo_id=HUB_REPO,
244
+ commit_message="V1e verdict: conv-knit + patchwork "
245
+ "grafts vs the binding gap")
246
+ print("DONE.", flush=True)
247
+
248
+
249
+ if __name__ == "__main__":
250
+ main()
trainers/v1f_consult_tier.py ADDED
@@ -0,0 +1,280 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # v1f_consult_tier.py — V1f: the consult-informed corrector/control tier
3
+ # (2026-08-09 evening; sources: fresnel/johanna/freckles + H2-era + trigram
4
+ # consult, AbstractPhil's stabilizer-vs-corrector lineage, the boundgate crater.)
5
+ #
6
+ # Bed point: R64 recall, K=64, D=16 (baseline mean .661, sdpa .936).
7
+ # ARMS:
8
+ # rownorm — L-041 port (rated 9; the ONLY ablation that broke the
9
+ # 233-run battery): row sphere-norm at the M-point = the
10
+ # HUB's VALUE rows normalized at the write (v = ||v||-unit
11
+ # * learned scale). Directions|magnitudes decoupled.
12
+ # auxalpha — the graft-geometry CORRECTION of the boundgate crater
13
+ # (alphas pinned at cap, acc .036): bounded alpha on the
14
+ # AUXILIARY knit channel ONLY, base path unthrottled —
15
+ # the faithful SVAE whisper-channel geometry.
16
+ # strobe3 — the bed's own anti-selection refinement ported to the
17
+ # HUB: q/k oriented at 3 taus (.05/.1/.3) concatenated
18
+ # (3K channels) — neighborhood disambiguation vs the
19
+ # incorrect-selections class.
20
+ # unihub — unigram control (plain hub, NO trigram): does binding
21
+ # late-bloom without the tri lineage? (tri caveat, exp020
22
+ # memorizer artifact channel.)
23
+ # unisdpa — unigram sdpa reference, 1 seed.
24
+ # padfix — hub_tri with ALEPH_TRI_PAD_FIX=1: prices the pad-
25
+ # conflation fault (discovery #16) against the tri baseline.
26
+ # INSTRUMENTS ON EVERY CELL: CV@1000 (H2-era early-screening law: codebook
27
+ # CV at step 1000 predicted the final band in 96% of 149 runs — ported
28
+ # validation), selection_eval (mis-selection vs no-retrieval), final
29
+ # census, alpha/gate trajectories where present.
30
+ # RUN: python tools\v1f_consult_tier.py (HF_TOKEN, 4090; chains after V1e)
31
+ # Env: V1F_ARMS, V1F_STEPS (def 8000)
32
+ # =========================================================================
33
+
34
+ import importlib
35
+ import json
36
+ import os
37
+ import sys
38
+
39
+ import torch
40
+ import torch.nn as nn
41
+ import torch.nn.functional as F
42
+
43
+
44
+ def _load(name):
45
+ if name in sys.modules:
46
+ return sys.modules[name]
47
+ try:
48
+ return importlib.import_module(name)
49
+ except ImportError:
50
+ here = globals().get("__file__")
51
+ if here is not None:
52
+ import pathlib
53
+ sys.path.insert(0, str(pathlib.Path(here).parent))
54
+ return importlib.import_module(name)
55
+ raise
56
+
57
+
58
+ bed = _load("ar_differentiation_bed")
59
+ v1b = _load("v1b_recall_army_battery")
60
+ class _cb: # public shim: token from env
61
+ @staticmethod
62
+ def hf_token():
63
+ import os as _os
64
+ return _os.environ.get("HF_TOKEN")
65
+ cb = _cb()
66
+ from huggingface_hub import HfApi # noqa: E402
67
+
68
+ assert torch.cuda.is_available(), "GPU only"
69
+ DEV = "cuda"
70
+ STEPS = int(os.environ.get("V1F_STEPS", "8000"))
71
+ K_ARMY, D_ADDR, R_KEY = 64, 16, 64
72
+ ARMS = os.environ.get(
73
+ "V1F_ARMS", "rownorm,auxalpha,strobe3,unihub,unisdpa,padfix").split(",")
74
+ BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
75
+ OUT_DIR = os.path.join(v1b.DATA_DIR, "results_v1f")
76
+ HUB_REPO = v1b.HUB_REPO
77
+ TAUS = (0.05, 0.1, 0.3)
78
+
79
+
80
+ class RowNormHUB(bed.CausalHUB):
81
+ """L-041 at the M-point: value rows sphere-normalized at the write,
82
+ magnitude carried by a learned per-row scale (directions|magnitudes
83
+ decoupled, the H2 separation-of-concerns)."""
84
+
85
+ def __init__(self, d, K, D):
86
+ super().__init__(d, K, D)
87
+ self.vscale = nn.Parameter(torch.ones(()))
88
+
89
+ def forward(self, x):
90
+ qp, qn = self.addr.oriented(self.q(x))
91
+ kp, kn = self.addr.oriented(self.k(x))
92
+ v = F.normalize(self.v(x), dim=-1) * self.vscale
93
+ Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
94
+ Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
95
+ zp = torch.cumsum(kp, dim=1)
96
+ zn = torch.cumsum(kn, dim=1)
97
+ num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
98
+ + torch.einsum("bnk,bnkd->bnd", qn, Sn))
99
+ den = ((qp * zp).sum(-1, keepdim=True)
100
+ + (qn * zn).sum(-1, keepdim=True))
101
+ return self.o(num / den.clamp_min(1e-12))
102
+
103
+
104
+ class Strobe3HUB(bed.CausalHUB):
105
+ """3-tau stroboscope on the HUB address: oriented maps at three
106
+ temperatures concatenated along K (3K memory channels) — the bed's
107
+ v2 anti-selection refinement, ported to attention."""
108
+
109
+ def _strobe(self, x):
110
+ A = F.normalize(self.addr.codebook, dim=-1)
111
+ cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
112
+ ps, ns = [], []
113
+ for t in TAUS:
114
+ u = cos / t
115
+ m = u.abs().amax(dim=-1, keepdim=True)
116
+ ep, en = torch.exp(u - m), torch.exp(-u - m)
117
+ Z = (ep + en).sum(dim=-1, keepdim=True)
118
+ ps.append(ep / Z)
119
+ ns.append(en / Z)
120
+ return torch.cat(ps, -1), torch.cat(ns, -1)
121
+
122
+ def forward(self, x):
123
+ qp, qn = self._strobe(self.q(x))
124
+ kp, kn = self._strobe(self.k(x))
125
+ v = self.v(x)
126
+ Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
127
+ Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
128
+ zp = torch.cumsum(kp, dim=1)
129
+ zn = torch.cumsum(kn, dim=1)
130
+ num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
131
+ + torch.einsum("bnk,bnkd->bnd", qn, Sn))
132
+ den = ((qp * zp).sum(-1, keepdim=True)
133
+ + (qn * zn).sum(-1, keepdim=True))
134
+ return self.o(num / den.clamp_min(1e-12))
135
+
136
+
137
+ class AuxAlphaBlock(nn.Module):
138
+ """The corrected graft geometry: base attn path UNTHROTTLED; the knit
139
+ conv rides an auxiliary bounded-alpha channel (init .024, cap .2) —
140
+ the SVAE whisper-channel form."""
141
+
142
+ def __init__(self, d, attn):
143
+ super().__init__()
144
+ self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
145
+ self.attn = attn
146
+ self.ks = nn.Conv1d(d, d, kernel_size=8, groups=d, bias=False)
147
+ self.alpha = nn.Parameter(torch.tensor(0.024))
148
+ self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),
149
+ nn.Linear(4 * d, d))
150
+
151
+ def forward(self, x):
152
+ a = self.attn(self.n1(x))
153
+ knit = self.ks(F.pad(a.transpose(1, 2), (7, 0))).transpose(1, 2)
154
+ x = x + a + self.alpha.clamp(0, 0.2) * torch.tanh(knit)
155
+ return x + self.mlp(self.n2(x))
156
+
157
+
158
+ def build_model(arm, d=192, layers=4):
159
+ base_arm = ("hub" if arm == "unihub"
160
+ else "sdpa" if arm == "unisdpa" else "hub_tri")
161
+ m = bed.ByteLM(base_arm, block=BLOCK, K=K_ARMY, D=D_ADDR)
162
+ if arm == "rownorm":
163
+ m.blocks = nn.ModuleList([
164
+ bed.Block(d, RowNormHUB(d, K_ARMY, D_ADDR)) for _ in range(layers)])
165
+ elif arm == "strobe3":
166
+ m.blocks = nn.ModuleList([
167
+ bed.Block(d, Strobe3HUB(d, K_ARMY, D_ADDR)) for _ in range(layers)])
168
+ elif arm == "auxalpha":
169
+ m.blocks = nn.ModuleList([
170
+ AuxAlphaBlock(d, bed.CausalHUB(d, K_ARMY, D_ADDR))
171
+ for _ in range(layers)])
172
+ return m
173
+
174
+
175
+ def codebook_cv(model):
176
+ """H2-era CV gauge on the address codebooks (fp64, normalized rows):
177
+ coefficient of variation of pairwise cosine magnitudes."""
178
+ cvs = []
179
+ for blk in model.blocks:
180
+ addr = getattr(blk.attn, "addr", None)
181
+ if addr is None:
182
+ return None
183
+ A = F.normalize(addr.codebook.detach().double(), dim=-1)
184
+ G = (A @ A.T).abs()
185
+ off = G[~torch.eye(G.shape[0], dtype=torch.bool, device=G.device)]
186
+ cvs.append(float(off.std() / off.mean().clamp_min(1e-12)))
187
+ return round(sum(cvs) / len(cvs), 4)
188
+
189
+
190
+ def run_cell(arm, seed, d):
191
+ tag = (f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}"
192
+ + (f"_t{STEPS // 1000}k" if STEPS != 8000 else ""))
193
+ out_p = f"{OUT_DIR}/{tag}.json"
194
+ if os.path.exists(out_p):
195
+ print(f"-- {tag}: exists, skip", flush=True)
196
+ return json.load(open(out_p))
197
+ if arm == "padfix":
198
+ os.environ["ALEPH_TRI_PAD_FIX"] = "1"
199
+ torch.manual_seed(seed)
200
+ g = torch.Generator().manual_seed(seed)
201
+ model = build_model(arm).to(DEV)
202
+ if arm == "padfix":
203
+ del os.environ["ALEPH_TRI_PAD_FIX"]
204
+ opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
205
+ cv1000 = None
206
+ for step in range(1, STEPS + 1):
207
+ x, y, m = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
208
+ logits = model(x)
209
+ if m.sum() == 0:
210
+ continue
211
+ loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
212
+ + 4.0 * F.cross_entropy(logits[m], y[m]))
213
+ opt.zero_grad(set_to_none=True)
214
+ loss.backward()
215
+ opt.step()
216
+ if step == 1000:
217
+ cv1000 = codebook_cv(model)
218
+ model.eval()
219
+ acc, bpb = v1b.masked_eval(model, d, g, iters=32)
220
+ sel = v1b.selection_eval(model, d, g, iters=32)
221
+ alphas = [round(float(b.alpha.detach()), 4) for b in model.blocks
222
+ if hasattr(b, "alpha")]
223
+ res = {"arm": arm, "seed": seed, "R": R_KEY, "K": K_ARMY, "D": D_ADDR,
224
+ "steps": STEPS, "final_acc": round(acc, 4),
225
+ "final_bpb": round(bpb, 4), "cv_at_1000": cv1000,
226
+ "cv_final": codebook_cv(model), "selection": sel,
227
+ "alphas": alphas,
228
+ "baseline_ref": {"K64_D16_mean": 0.6609, "sdpa_tri": 0.936}}
229
+ os.makedirs(OUT_DIR, exist_ok=True)
230
+ json.dump(res, open(out_p, "w"), indent=1)
231
+ print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f} missel "
232
+ f"{sel['mis_sel_frac_of_errors']:.3f} cv1k {cv1000}"
233
+ + (f" alphas {alphas}" if alphas else ""), flush=True)
234
+ del model
235
+ torch.cuda.empty_cache()
236
+ return res
237
+
238
+
239
+ def main():
240
+ print("=" * 78, flush=True)
241
+ print(f"V1f CONSULT TIER — arms {ARMS} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}, "
242
+ f"{STEPS} steps", flush=True)
243
+ print("=" * 78, flush=True)
244
+ api = HfApi(token=cb.hf_token())
245
+ d = v1b.get_data(R_KEY)
246
+ results = []
247
+ for arm in ARMS:
248
+ seeds = [0] if arm == "unisdpa" else [0, 1]
249
+ for seed in seeds:
250
+ results.append(run_cell(arm, seed, d))
251
+ tag = (f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}"
252
+ + (f"_t{STEPS // 1000}k" if STEPS != 8000 else ""))
253
+ api.upload_file(path_or_fileobj=f"{OUT_DIR}/{tag}.json",
254
+ path_in_repo=f"battery/v1f_consult/{tag}.json",
255
+ repo_id=HUB_REPO, commit_message=f"V1f {tag}")
256
+ base = 0.6609
257
+ verdict = {}
258
+ for arm in ARMS:
259
+ rs = [r for r in results if r["arm"] == arm]
260
+ accs = [r["final_acc"] for r in rs]
261
+ ms = [r["selection"]["mis_sel_frac_of_errors"] for r in rs]
262
+ verdict[arm] = {"acc": accs,
263
+ "mean": round(sum(accs) / len(accs), 4),
264
+ "delta": round(sum(accs) / len(accs) - base, 4),
265
+ "missel_frac": round(sum(ms) / len(ms), 4)}
266
+ print(f"{arm:>9}: {verdict[arm]['mean']:.4f} "
267
+ f"({verdict[arm]['delta']:+.4f}) missel "
268
+ f"{verdict[arm]['missel_frac']:.3f}", flush=True)
269
+ out = f"{OUT_DIR}/v1f_verdict.json"
270
+ json.dump(verdict, open(out, "w"), indent=1)
271
+ api.upload_file(path_or_fileobj=out,
272
+ path_in_repo="battery/v1f_consult/v1f_verdict.json",
273
+ repo_id=HUB_REPO,
274
+ commit_message="V1f verdict: consult-tier arms + "
275
+ "mis-selection decomposition")
276
+ print("DONE.", flush=True)
277
+
278
+
279
+ if __name__ == "__main__":
280
+ main()
trainers/v1g_svae_overlap_sweep.py ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # v1g_svae_overlap_sweep.py — the 16-noise-spectrum overlap sweep on the
3
+ # NATIVE substrate (AbstractPhil, 2026-08-09: "you just need the correct noises
4
+ # that we trained johanna and freckles with" — the wikitext sweep pointed
5
+ # at the wrong beast). Also answers the STANDING open question
6
+ # (history/open_questions.md:191): the 16-codebook similarity matrix.
7
+ #
8
+ # HYPOTHESIS UNDER TEST (AbstractPhil, never verified): overlapping input spectra
9
+ # cause anchored/pooled accumulation — overlap-linked noise types should
10
+ # produce SIMILAR model responses (the model anchors on shared structure),
11
+ # and high-overlap types should converge slower / less differentiated.
12
+ #
13
+ # SUBSTRATE: OmegaNoiseDataset (16 types, verbatim from
14
+ # AbstractPhil/geolip-svae-batteries battery_maker_proto.py) + trained
15
+ # johanna-F-S32-V32-D4-h32-d1-p2 (PatchSVAE_F, all defense mechanisms).
16
+ # MEASURES:
17
+ # O[i,j] data-side overlap: radial power-spectrum cosine + histogram
18
+ # Bhattacharyya (averaged).
19
+ # S[i,j] model-side response similarity: per-type mean sigma (omega)
20
+ # profile cosine + per-mode sigma-distribution overlap.
21
+ # LINKAGE: corr(O_off, S_off) — overlap-driven anchoring verdict.
22
+ # DIFFICULTY: per-type recon MSE vs overlap centrality (slow-convergence
23
+ # link, cross-checked against the canon difficulty ordering).
24
+ # ALEPH EMERGENCE (native): antipodal row pairing in the per-sample M
25
+ # codebooks (RP^D axis signature, cos < -0.9), per type.
26
+ # RUN: python tools\v1g_svae_overlap_sweep.py (CPU, ~3 min)
27
+ # =========================================================================
28
+
29
+ import importlib.util
30
+ import json
31
+ import math
32
+ import os
33
+ import sys
34
+
35
+ import torch
36
+ import torch.nn.functional as F
37
+
38
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
39
+ from huggingface_hub import hf_hub_download, HfApi # noqa: E402
40
+
41
+ REPO = "AbstractPhil/geolip-svae-batteries"
42
+ RUN = "johanna-F-S32-V32-D4-h32-d1-p2"
43
+ N_PER_TYPE = 48
44
+ IMG = 32
45
+ OUT = "./data/v1_rank_army/v1g_svae_overlap.json"
46
+ SHIP_REPO = "AbstractPhil/aleph-splat-0"
47
+
48
+ TYPE_NAMES = ["gaussian", "uniform", "wide_uniform", "poisson", "pink",
49
+ "brown", "binary", "sparse_spikes", "block", "gradient",
50
+ "checker", "gauss_uniform_mix", "quadrant", "t13", "t14",
51
+ "t15"]
52
+
53
+
54
+ def load_proto():
55
+ p = hf_hub_download(REPO, "battery_maker_proto.py")
56
+ spec = importlib.util.spec_from_file_location("battery_maker_proto", p)
57
+ mod = importlib.util.module_from_spec(spec)
58
+ sys.modules["battery_maker_proto"] = mod
59
+ spec.loader.exec_module(mod)
60
+ return mod
61
+
62
+
63
+ def gen_type(mod, t, n):
64
+ torch.manual_seed(1234 + t)
65
+ ds = mod.OmegaNoiseDataset(size=n, img_size=IMG,
66
+ seed_rotate_every=10**9, allowed_types=[t])
67
+ return torch.stack([ds[i] if isinstance(ds[i], torch.Tensor)
68
+ else ds[i][0] for i in range(n)])
69
+
70
+
71
+ def radial_spectrum(x):
72
+ """Mean radially-averaged power spectrum over a batch (B,3,H,W)."""
73
+ S = torch.fft.rfft2(x.double()).abs() ** 2
74
+ S = S.mean(dim=(0, 1)) # (H, W//2+1)
75
+ h, w = S.shape
76
+ fy = torch.fft.fftfreq(h).abs().unsqueeze(-1).expand(-1, w)
77
+ fx = torch.fft.rfftfreq(IMG).unsqueeze(0).expand(h, -1)
78
+ r = torch.sqrt(fx ** 2 + fy ** 2)
79
+ nb = 16
80
+ bins = torch.linspace(0, float(r.max()) + 1e-6, nb + 1)
81
+ prof = torch.zeros(nb, dtype=torch.float64)
82
+ for b in range(nb):
83
+ m = (r >= bins[b]) & (r < bins[b + 1])
84
+ if m.any():
85
+ prof[b] = S[m].mean()
86
+ return (prof / prof.sum()).clamp_min(1e-12)
87
+
88
+
89
+ def hist_overlap(a, b, nbins=64):
90
+ lo = float(min(a.min(), b.min()))
91
+ hi = float(max(a.max(), b.max()))
92
+ ha = torch.histc(a.double().flatten(), nbins, lo, hi)
93
+ hb = torch.histc(b.double().flatten(), nbins, lo, hi)
94
+ ha, hb = ha / ha.sum(), hb / hb.sum()
95
+ return float((ha * hb).sqrt().sum()) # Bhattacharyya
96
+
97
+
98
+ def main():
99
+ print("=" * 74)
100
+ print("V1g SVAE 16-NOISE OVERLAP SWEEP (native substrate)")
101
+ print("=" * 74)
102
+ mod = load_proto()
103
+ cfgp = hf_hub_download(REPO, f"{RUN}/config.json")
104
+ cfg = json.load(open(cfgp))
105
+ model = mod.PatchSVAE_F(
106
+ matrix_v=cfg.get("matrix_v", 32), D=cfg.get("D", 4),
107
+ patch_size=cfg.get("patch_size", 2), hidden=cfg.get("hidden", 32),
108
+ depth=cfg.get("depth", 1),
109
+ n_cross_layers=cfg.get("n_cross_layers", 1),
110
+ n_heads=cfg.get("n_heads", 4))
111
+ ck = torch.load(hf_hub_download(REPO, f"{RUN}/checkpoints/best.pt"),
112
+ map_location="cpu", weights_only=False)
113
+ sd = ck.get("model_state_dict", ck.get("state_dict", ck))
114
+ model.load_state_dict(sd, strict=True)
115
+ model.eval()
116
+ print(f"loaded {RUN}: {sum(p.numel() for p in model.parameters()):,} "
117
+ f"params")
118
+
119
+ batches, spectra, sig_profiles, mse, antipodal = {}, {}, {}, {}, {}
120
+ for t in range(16):
121
+ x = gen_type(mod, t, N_PER_TYPE)
122
+ batches[t] = x
123
+ spectra[t] = radial_spectrum(x)
124
+ with torch.no_grad():
125
+ out = model(x)
126
+ mse[t] = round(float(F.mse_loss(out["recon"], x)), 5)
127
+ svd = out["svd"]
128
+ S = svd["S"] # (..., D) sigma
129
+ sig_profiles[t] = S.reshape(-1, S.shape[-1]).double()
130
+ M = svd.get("M", svd.get("M_norm"))
131
+ if M is not None:
132
+ Mn = F.normalize(M.reshape(-1, M.shape[-2], M.shape[-1])
133
+ .double(), dim=-1)
134
+ G = torch.einsum("bvd,bwd->bvw", Mn, Mn)
135
+ V = G.shape[-1]
136
+ iu = torch.triu_indices(V, V, offset=1)
137
+ pairs = G[:, iu[0], iu[1]]
138
+ antipodal[t] = round(float((pairs < -0.9).double().mean()), 5)
139
+ print(f" type {t:>2} {TYPE_NAMES[t]:>17}: mse {mse[t]:.5f}"
140
+ + (f" antipodal {antipodal.get(t)}" if antipodal else ""),
141
+ flush=True)
142
+
143
+ # ---- data-side overlap O and model-side similarity S ----
144
+ Omat = torch.zeros(16, 16)
145
+ Smat = torch.zeros(16, 16)
146
+ for i in range(16):
147
+ for j in range(16):
148
+ if j < i:
149
+ Omat[i, j], Smat[i, j] = Omat[j, i], Smat[j, i]
150
+ continue
151
+ spec_cos = float(F.cosine_similarity(
152
+ spectra[i].unsqueeze(0), spectra[j].unsqueeze(0)))
153
+ hov = hist_overlap(batches[i], batches[j])
154
+ Omat[i, j] = (spec_cos + hov) / 2
155
+ mi = sig_profiles[i].mean(0)
156
+ mj = sig_profiles[j].mean(0)
157
+ prof_cos = float(F.cosine_similarity(mi.unsqueeze(0),
158
+ mj.unsqueeze(0)))
159
+ dov = sum(hist_overlap(sig_profiles[i][:, d],
160
+ sig_profiles[j][:, d])
161
+ for d in range(mi.shape[0])) / mi.shape[0]
162
+ Smat[i, j] = (prof_cos + dov) / 2
163
+ off = ~torch.eye(16, dtype=torch.bool)
164
+ Ov, Sv = Omat[off], Smat[off]
165
+ r_link = float(torch.corrcoef(torch.stack([Ov, Sv]))[0, 1])
166
+ centrality = Omat.sum(1) - 1
167
+ mset = torch.tensor([mse[t] for t in range(16)])
168
+ r_diff = float(torch.corrcoef(torch.stack(
169
+ [centrality.double(), mset.double()]))[0, 1])
170
+ ranks = lambda v: torch.argsort(torch.argsort(v)).double()
171
+ rho_link = float(torch.corrcoef(torch.stack(
172
+ [ranks(Ov), ranks(Sv)]))[0, 1])
173
+
174
+ print("\n== VERDICTS")
175
+ print(f" overlap->response linkage: pearson r={r_link:.3f} "
176
+ f"spearman rho={rho_link:.3f} (n={int(off.sum())} pairs)")
177
+ print(f" overlap-centrality vs difficulty (MSE): r={r_diff:.3f}")
178
+ if antipodal:
179
+ av = torch.tensor([antipodal[t] for t in range(16)])
180
+ print(f" antipodal (RP axis) pairing: mean {float(av.mean()):.4f} "
181
+ f"range {float(av.min()):.4f}-{float(av.max()):.4f}")
182
+ res = {"run": RUN, "n_per_type": N_PER_TYPE, "img": IMG,
183
+ "type_names": TYPE_NAMES,
184
+ "O_overlap": [[round(float(v), 4) for v in row] for row in Omat],
185
+ "S_response": [[round(float(v), 4) for v in row] for row in Smat],
186
+ "mse_per_type": mse, "antipodal_per_type": antipodal,
187
+ "linkage_pearson": round(r_link, 4),
188
+ "linkage_spearman": round(rho_link, 4),
189
+ "centrality_vs_mse_r": round(r_diff, 4)}
190
+ json.dump(res, open(OUT, "w"), indent=1)
191
+ api = HfApi()
192
+ api.upload_file(path_or_fileobj=OUT,
193
+ path_in_repo="battery/v1g_svae_overlap/v1g_sweep.json",
194
+ repo_id=SHIP_REPO,
195
+ commit_message="V1g: 16-noise overlap sweep on the "
196
+ "native SVAE substrate (overlap->response linkage + "
197
+ "16x16 matrices + antipodal census)")
198
+ print(f"saved -> {OUT} ; shipped -> {SHIP_REPO}")
199
+
200
+
201
+ if __name__ == "__main__":
202
+ main()
trainers/v1i_validity_batteries.py ADDED
@@ -0,0 +1,400 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =========================================================================
2
+ # v1i_validity_batteries.py — AbstractPhil's validity campaign (2026-08-10):
3
+ # corruption/numerics/optimizer/schedule batteries against the four
4
+ # possibilities: (1) Adam may not capture the necessary information;
5
+ # (2) scheduling may be the solution or a no-op; (3) splat/aleph fragility
6
+ # vs standard attention; (4) unaccounted corruption.
7
+ #
8
+ # Bed point: R64 recall, K=64, D=16, ALEPH_TRI_PAD_FIX=1 (the repaired
9
+ # config; padfix baseline .696/.733 mean .714, sdpa_tri ref .936).
10
+ # MODES (env V1I_MODE):
11
+ # dtype — fp32 control / bf16 autocast / fp16 AMP / fp8sim (address-path
12
+ # e4m3 roundtrip: codebook + q/k projections quantized each
13
+ # forward) / fp64probe (2k-step numeric-truth probe).
14
+ # AbstractPhil: "is the computable range of differentiation beyond fp32,
15
+ # can it be represented as less data (fp8), test middlegrounds."
16
+ # opt — adam(control)/sgd_mom/lion/muon/adagrad x {hub, sdpa}.
17
+ # PHIL-AUTHORIZED DEVIATION from the pure-Adam rider (screens
18
+ # only, labeled; AdamW remains excluded per the hard law).
19
+ # Fair-comparison discipline: 300-step LR probe over
20
+ # {3e-4,1e-3,3e-3,1e-2} per optimizer/arm before the real run.
21
+ # Fragility gauge (possibility 3): cross-optimizer outcome
22
+ # variance per architecture.
23
+ # sched — on hub+adam: warmcos / geomfirst (addr-group fast->slow per
24
+ # A1 frame-lock, trunk-group slow->fast per A2 late-accrual —
25
+ # the math-aligned schedule) / inverse (labeled falsification
26
+ # control). Flat control = the padfix cells.
27
+ # Gauges everywhere: answer acc, bpb, selection_eval decomposition,
28
+ # den-clamp fraction (numeric-health), census.
29
+ # RUN: V1I_MODE=dtype|opt|sched python tools\v1i_validity_batteries.py
30
+ # =========================================================================
31
+
32
+ import importlib
33
+ import json
34
+ import math
35
+ import os
36
+ import sys
37
+
38
+ import torch
39
+ import torch.nn.functional as F
40
+
41
+ os.environ["ALEPH_TRI_PAD_FIX"] = "1" # standing order (V1f)
42
+
43
+
44
+ def _load(name):
45
+ if name in sys.modules:
46
+ return sys.modules[name]
47
+ try:
48
+ return importlib.import_module(name)
49
+ except ImportError:
50
+ here = globals().get("__file__")
51
+ if here is not None:
52
+ import pathlib
53
+ sys.path.insert(0, str(pathlib.Path(here).parent))
54
+ return importlib.import_module(name)
55
+ raise
56
+
57
+
58
+ bed = _load("ar_differentiation_bed")
59
+ v1b = _load("v1b_recall_army_battery")
60
+ class _cb: # public shim: token from env
61
+ @staticmethod
62
+ def hf_token():
63
+ import os as _os
64
+ return _os.environ.get("HF_TOKEN")
65
+ cb = _cb()
66
+ from huggingface_hub import HfApi # noqa: E402
67
+
68
+ assert torch.cuda.is_available(), "GPU only"
69
+ DEV = "cuda"
70
+ MODE = os.environ.get("V1I_MODE", "dtype")
71
+ STEPS = int(os.environ.get("V1I_STEPS", "8000"))
72
+ K_ARMY, D_ADDR, R_KEY = 64, 16, 64
73
+ BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
74
+ DATA_DIR = "./data/v1_rank_army_padfix"
75
+ OUT_DIR = f"{DATA_DIR}/results_v1i_{MODE}"
76
+ HUB_REPO = v1b.HUB_REPO
77
+ BASELINE = 0.7143 # padfix K64/D16 mean
78
+
79
+
80
+ def get_data():
81
+ os.environ["V1B_DATA"] = DATA_DIR
82
+ v1b.DATA_DIR = DATA_DIR
83
+ return v1b.get_data(R_KEY)
84
+
85
+
86
+ # ---------------------------------------------------------------- optimizers
87
+ class Lion(torch.optim.Optimizer):
88
+ """Sign-momentum (pattern-aligned). lr ~3-10x smaller than Adam's."""
89
+
90
+ def __init__(self, params, lr=1e-4, betas=(0.9, 0.99)):
91
+ super().__init__(params, dict(lr=lr, betas=betas))
92
+
93
+ @torch.no_grad()
94
+ def step(self):
95
+ for g in self.param_groups:
96
+ b1, b2 = g["betas"]
97
+ for p in g["params"]:
98
+ if p.grad is None:
99
+ continue
100
+ st = self.state.setdefault(p, {})
101
+ m = st.setdefault("m", torch.zeros_like(p))
102
+ p.add_(torch.sign(m.mul(b1).add(p.grad, alpha=1 - b1)),
103
+ alpha=-g["lr"])
104
+ m.mul_(b2).add_(p.grad, alpha=1 - b2)
105
+
106
+
107
+ def _ns_orth(G, steps=5):
108
+ """Newton-Schulz orthogonalization (Muon core), fp32."""
109
+ a, b, c = 3.4445, -4.7750, 2.0315
110
+ X = G.float()
111
+ X = X / (X.norm() + 1e-7)
112
+ tall = X.shape[0] > X.shape[1]
113
+ if tall:
114
+ X = X.T
115
+ for _ in range(steps):
116
+ A = X @ X.T
117
+ X = a * X + (b * A + c * A @ A) @ X
118
+ return (X.T if tall else X).to(G.dtype)
119
+
120
+
121
+ class Muon(torch.optim.Optimizer):
122
+ """Orthogonalized momentum on 2D weights (geometrically aligned);
123
+ Adam fallback for non-2D params handled by the caller."""
124
+
125
+ def __init__(self, params, lr=2e-2, momentum=0.95):
126
+ super().__init__(params, dict(lr=lr, momentum=momentum))
127
+
128
+ @torch.no_grad()
129
+ def step(self):
130
+ for g in self.param_groups:
131
+ for p in g["params"]:
132
+ if p.grad is None or p.ndim != 2:
133
+ continue
134
+ st = self.state.setdefault(p, {})
135
+ m = st.setdefault("m", torch.zeros_like(p))
136
+ m.mul_(g["momentum"]).add_(p.grad)
137
+ p.add_(_ns_orth(m), alpha=-g["lr"])
138
+
139
+
140
+ def make_opt(name, model, lr):
141
+ ps = list(model.parameters())
142
+ if name == "adam":
143
+ return [torch.optim.Adam(ps, lr=lr, weight_decay=0.0)]
144
+ if name == "sgd_mom":
145
+ return [torch.optim.SGD(ps, lr=lr, momentum=0.9)]
146
+ if name == "adagrad":
147
+ return [torch.optim.Adagrad(ps, lr=lr)]
148
+ if name == "lion":
149
+ return [Lion(ps, lr=lr / 3)]
150
+ if name == "muon":
151
+ two = [p for p in ps if p.ndim == 2]
152
+ rest = [p for p in ps if p.ndim != 2]
153
+ return [Muon(two, lr=lr * 20),
154
+ torch.optim.Adam(rest, lr=lr, weight_decay=0.0)]
155
+ raise ValueError(name)
156
+
157
+
158
+ def lr_probe(name, arm, d, cands=(3e-4, 1e-3, 3e-3, 1e-2)):
159
+ best, best_loss = cands[0], float("inf")
160
+ for lr in cands:
161
+ torch.manual_seed(0)
162
+ g = torch.Generator().manual_seed(0)
163
+ m = bed.ByteLM(arm, block=BLOCK, K=K_ARMY, D=D_ADDR).to(DEV)
164
+ opts = make_opt(name, m, lr)
165
+ tot = 0.0
166
+ ok = True
167
+ for step in range(300):
168
+ x, y, mk = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK,
169
+ DEV, g)
170
+ logits = m(x)
171
+ loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
172
+ + 4.0 * F.cross_entropy(logits[mk], y[mk]))
173
+ if not torch.isfinite(loss):
174
+ ok = False
175
+ break
176
+ for o in opts:
177
+ o.zero_grad(set_to_none=True)
178
+ loss.backward()
179
+ for o in opts:
180
+ o.step()
181
+ if step >= 200:
182
+ tot += float(loss)
183
+ del m
184
+ torch.cuda.empty_cache()
185
+ if ok and tot < best_loss:
186
+ best_loss, best = tot, lr
187
+ return best
188
+
189
+
190
+ # ---------------------------------------------------------------- fp8 sim
191
+ def fp8_roundtrip(t):
192
+ if hasattr(torch, "float8_e4m3fn"):
193
+ return t.to(torch.float8_e4m3fn).to(t.dtype)
194
+ scale = t.abs().amax().clamp_min(1e-8) / 448.0
195
+ return ((t / scale).clamp(-448, 448).round() * scale)
196
+
197
+
198
+ class FP8AddrHUB(bed.CausalHUB):
199
+ """Address path through an e4m3 bottleneck each forward: does the
200
+ differentiation structure survive 8-bit representation?"""
201
+
202
+ def forward(self, x):
203
+ q8 = fp8_roundtrip(self.q(x))
204
+ k8 = fp8_roundtrip(self.k(x))
205
+ A = self.addr.codebook.data
206
+ self.addr.codebook.data = fp8_roundtrip(A)
207
+ try:
208
+ qp, qn = self.addr.oriented(q8)
209
+ kp, kn = self.addr.oriented(k8)
210
+ finally:
211
+ self.addr.codebook.data = A
212
+ v = self.v(x)
213
+ Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
214
+ Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
215
+ zp = torch.cumsum(kp, dim=1)
216
+ zn = torch.cumsum(kn, dim=1)
217
+ num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
218
+ + torch.einsum("bnk,bnkd->bnd", qn, Sn))
219
+ den = ((qp * zp).sum(-1, keepdim=True)
220
+ + (qn * zn).sum(-1, keepdim=True))
221
+ self._den_min = float(den.abs().min())
222
+ return self.o(num / den.clamp_min(1e-12))
223
+
224
+
225
+ def den_health(model, d, g):
226
+ """Fraction of read positions whose denominator sits near the clamp."""
227
+ xv, _, _ = v1b.batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
228
+ fracs = []
229
+ with torch.no_grad():
230
+ x = model.emb(xv)
231
+ if getattr(model, "trigram", False):
232
+ pv = 256 if getattr(model, "pad_fix", False) else 0
233
+ x = x + model.emb1(F.pad(xv, (1, 0), value=pv)[:, :-1]) \
234
+ + model.emb2(F.pad(xv, (2, 0), value=pv)[:, :-2])
235
+ x = x + model.pos[:, : xv.shape[1]]
236
+ for blk in model.blocks:
237
+ h = blk.n1(x)
238
+ at = blk.attn
239
+ if not hasattr(at, "addr"):
240
+ return None
241
+ qp, qn = at.addr.oriented(at.q(h))
242
+ kp, kn = at.addr.oriented(at.k(h))
243
+ zp, zn = torch.cumsum(kp, 1), torch.cumsum(kn, 1)
244
+ den = ((qp * zp).sum(-1) + (qn * zn).sum(-1))
245
+ fracs.append(float((den.abs() < 1e-6).float().mean()))
246
+ x = blk(x)
247
+ return round(sum(fracs) / len(fracs), 6)
248
+
249
+
250
+ # ---------------------------------------------------------------- schedules
251
+ def sched_lr(kind, group, step, total, base=3e-4):
252
+ t = step / total
253
+ if kind == "warmcos":
254
+ if t < 0.05:
255
+ return base * t / 0.05
256
+ return base * 0.5 * (1 + math.cos(math.pi * (t - .05) / .95))
257
+ if kind == "geomfirst": # math-aligned (A1+A2)
258
+ if group == "addr": # frame locks early
259
+ return 3e-3 * math.exp(-6 * t) + 3e-5
260
+ return 1e-4 + (5e-4 - 1e-4) * t # function accrues late
261
+ if kind == "inverse": # labeled falsification
262
+ if group == "addr":
263
+ return 3e-5 + (3e-3 - 3e-5) * t
264
+ return 5e-4 * math.exp(-6 * t) + 1e-4
265
+ return base
266
+
267
+
268
+ def addr_params(model):
269
+ ids = set()
270
+ for blk in model.blocks:
271
+ at = blk.attn
272
+ for p in (list(at.addr.parameters()) + list(at.q.parameters())
273
+ + list(at.k.parameters())):
274
+ ids.add(id(p))
275
+ return ids
276
+
277
+
278
+ # ---------------------------------------------------------------- cells
279
+ def run_cell(tag, build_fn, opt_fn, seed, d, autocast_dtype=None,
280
+ sched_kind=None):
281
+ out_p = f"{OUT_DIR}/{tag}_s{seed}.json"
282
+ if os.path.exists(out_p):
283
+ print(f"-- {tag}_s{seed}: exists, skip", flush=True)
284
+ return json.load(open(out_p))
285
+ torch.manual_seed(seed)
286
+ g = torch.Generator().manual_seed(seed)
287
+ model = build_fn().to(DEV)
288
+ opts = opt_fn(model)
289
+ aset = addr_params(model) if sched_kind else None
290
+ scaler = (torch.amp.GradScaler("cuda")
291
+ if autocast_dtype == torch.float16 else None)
292
+ for step in range(1, STEPS + 1):
293
+ x, y, mk = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
294
+ if autocast_dtype is not None:
295
+ with torch.autocast("cuda", dtype=autocast_dtype):
296
+ logits = model(x)
297
+ loss = (F.cross_entropy(logits.reshape(-1, 256).float(),
298
+ y.reshape(-1))
299
+ + 4.0 * F.cross_entropy(logits[mk].float(), y[mk]))
300
+ else:
301
+ logits = model(x)
302
+ loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
303
+ + 4.0 * F.cross_entropy(logits[mk], y[mk]))
304
+ for o in opts:
305
+ o.zero_grad(set_to_none=True)
306
+ if scaler:
307
+ scaler.scale(loss).backward()
308
+ for o in opts:
309
+ scaler.step(o)
310
+ scaler.update()
311
+ else:
312
+ loss.backward()
313
+ for o in opts:
314
+ o.step()
315
+ if sched_kind:
316
+ for o in opts:
317
+ for grp in o.param_groups:
318
+ is_addr = all(id(p) in aset for p in grp["params"]) \
319
+ if grp["params"] else False
320
+ grp["lr"] = sched_lr(sched_kind,
321
+ "addr" if is_addr else "trunk",
322
+ step, STEPS)
323
+ model.eval()
324
+ acc, bpb = v1b.masked_eval(model, d, g, iters=32)
325
+ sel = v1b.selection_eval(model, d, g, iters=32)
326
+ dh = den_health(model, d, g)
327
+ res = {"tag": tag, "seed": seed, "steps": STEPS, "mode": MODE,
328
+ "final_acc": round(acc, 4), "final_bpb": round(bpb, 4),
329
+ "selection": sel, "den_near_clamp_frac": dh,
330
+ "baseline_padfix": BASELINE}
331
+ os.makedirs(OUT_DIR, exist_ok=True)
332
+ json.dump(res, open(out_p, "w"), indent=1)
333
+ print(f"-- {tag}_s{seed}: acc {acc:.4f} bpb {bpb:.4f} "
334
+ f"missel {sel['mis_sel_frac_of_errors']:.3f} den {dh}", flush=True)
335
+ del model
336
+ torch.cuda.empty_cache()
337
+ return res
338
+
339
+
340
+ def main():
341
+ print("=" * 78, flush=True)
342
+ print(f"V1i VALIDITY BATTERY mode={MODE} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}"
343
+ f" padfix (baseline {BASELINE}, sdpa .936)", flush=True)
344
+ print("=" * 78, flush=True)
345
+ api = HfApi(token=cb.hf_token())
346
+ d = get_data()
347
+ results = []
348
+ mk_hub = lambda: bed.ByteLM("hub_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
349
+ mk_sdpa = lambda: bed.ByteLM("sdpa_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
350
+ adam = lambda m: make_opt("adam", m, 3e-4)
351
+
352
+ if MODE == "dtype":
353
+ def mk_fp8():
354
+ m = mk_hub()
355
+ m.blocks = torch.nn.ModuleList(
356
+ [bed.Block(192, FP8AddrHUB(192, K_ARMY, D_ADDR))
357
+ for _ in range(4)])
358
+ return m
359
+ cells = [("fp32", mk_hub, None), ("bf16", mk_hub, torch.bfloat16),
360
+ ("fp16", mk_hub, torch.float16), ("fp8addr", mk_fp8, None)]
361
+ for tag, fn, ac in cells:
362
+ for seed in (0, 1):
363
+ results.append(run_cell(tag, fn, adam, seed, d,
364
+ autocast_dtype=ac))
365
+ elif MODE == "opt":
366
+ for name in ("adam", "sgd_mom", "lion", "muon", "adagrad"):
367
+ for arm, mk, seeds in (("hub", mk_hub, (0, 1)),
368
+ ("sdpa", mk_sdpa, (0,))):
369
+ lr = 3e-4 if name == "adam" else lr_probe(name,
370
+ f"{arm}_tri", d)
371
+ print(f" [{name}/{arm}] lr={lr}", flush=True)
372
+ for seed in seeds:
373
+ results.append(run_cell(
374
+ f"{name}_{arm}", mk,
375
+ lambda m, n=name, l=lr: make_opt(n, m, l),
376
+ seed, d))
377
+ elif MODE == "sched":
378
+ for kind in ("warmcos", "geomfirst", "inverse"):
379
+ for seed in (0, 1):
380
+ results.append(run_cell(f"sched_{kind}", mk_hub, adam,
381
+ seed, d, sched_kind=kind))
382
+ # ---- summarize + ship ----
383
+ summary = {}
384
+ for r in results:
385
+ summary.setdefault(r["tag"], []).append(r["final_acc"])
386
+ for tag, accs in summary.items():
387
+ mean = sum(accs) / len(accs)
388
+ print(f"{tag:>14}: {mean:.4f} ({mean - BASELINE:+.4f})", flush=True)
389
+ out = f"{OUT_DIR}/v1i_{MODE}_verdict.json"
390
+ json.dump({t: {"acc": a, "mean": round(sum(a) / len(a), 4)}
391
+ for t, a in summary.items()}, open(out, "w"), indent=1)
392
+ api.upload_file(path_or_fileobj=out,
393
+ path_in_repo=f"battery/v1i_validity/{MODE}_verdict.json",
394
+ repo_id=HUB_REPO,
395
+ commit_message=f"V1i {MODE} battery verdict")
396
+ print("DONE.", flush=True)
397
+
398
+
399
+ if __name__ == "__main__":
400
+ main()