trainers/: full replication kit — 11 scrubbed standalone trainers + README (battery map, replication discipline, repair flags, gauge guidance, history caveats)
Browse files- trainers/README.md +53 -0
- trainers/__pycache__/ar_differentiation_bed.cpython-312.pyc +0 -0
- trainers/__pycache__/geolip_vitals.cpython-312.pyc +0 -0
- trainers/__pycache__/splat_math_battery.cpython-312.pyc +0 -0
- trainers/__pycache__/trigram_bitspace_sweep.cpython-312.pyc +0 -0
- trainers/__pycache__/v1_rank_army_battery.cpython-312.pyc +0 -0
- trainers/__pycache__/v1b_recall_army_battery.cpython-312.pyc +0 -0
- trainers/__pycache__/v1d_svae_graft.cpython-312.pyc +0 -0
- trainers/__pycache__/v1e_knit_battery.cpython-312.pyc +0 -0
- trainers/__pycache__/v1f_consult_tier.cpython-312.pyc +0 -0
- trainers/__pycache__/v1g_svae_overlap_sweep.cpython-312.pyc +0 -0
- trainers/__pycache__/v1i_validity_batteries.cpython-312.pyc +0 -0
- trainers/ar_differentiation_bed.py +499 -0
- trainers/geolip_vitals.py +219 -0
- trainers/splat_math_battery.py +644 -0
- trainers/trigram_bitspace_sweep.py +194 -0
- trainers/v1_rank_army_battery.py +241 -0
- trainers/v1b_recall_army_battery.py +323 -0
- trainers/v1d_svae_graft.py +230 -0
- trainers/v1e_knit_battery.py +250 -0
- trainers/v1f_consult_tier.py +280 -0
- trainers/v1g_svae_overlap_sweep.py +202 -0
- trainers/v1i_validity_batteries.py +400 -0
trainers/README.md
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# trainers/ — replication kit for the aleph-splat measurement record
|
| 2 |
+
|
| 3 |
+
Every number in this repo's `battery/` directories was produced by a
|
| 4 |
+
trainer in this directory, on a single consumer GPU (RTX 4090 class).
|
| 5 |
+
Each trainer is a standalone file: `python <trainer>.py` with `HF_TOKEN`
|
| 6 |
+
in the environment if you want results shipped to your own hub repo
|
| 7 |
+
(uploads are optional — remove the `upload_file` calls or leave the token
|
| 8 |
+
unset and save locally).
|
| 9 |
+
|
| 10 |
+
## The trainers, mapped to their result directories
|
| 11 |
+
|
| 12 |
+
| trainer | produces | battery dir |
|
| 13 |
+
|---|---|---|
|
| 14 |
+
| `splat_math_battery.py` | the B1–B9 static mathematical battery | `battery/` (splat_battery*.json) |
|
| 15 |
+
| `ar_differentiation_bed.py` | the byte-AR bed all recall batteries build on (ByteLM, CausalHUB, AlephAddress) — also runs its own wikitext arms | (dependency; `./data/ar_ckpts` specimens) |
|
| 16 |
+
| `geolip_vitals.py` | vitals helpers imported by the bed | (dependency) |
|
| 17 |
+
| `v1_rank_army_battery.py` | rank-ladder × army-ladder under LDS filtering pressure | `battery/v1_rank_army/` |
|
| 18 |
+
| `v1b_recall_army_battery.py` | the recall battery: rank-controlled in-context binding; the K/D ladders; clean-mask + selection gauges | `battery/v1b_recall_army/` |
|
| 19 |
+
| `v1d_svae_graft.py` | SVAE-transformer graft arms (sphere/lens/α-grammar) | `battery/v1d_svae_graft/` |
|
| 20 |
+
| `v1e_knit_battery.py` | conv-knit + patchwork graft arms | `battery/v1e_knit/` |
|
| 21 |
+
| `v1f_consult_tier.py` | corrector tier: rownorm/auxalpha/strobe3/unigram/padfix | `battery/v1f_consult/` |
|
| 22 |
+
| `v1g_svae_overlap_sweep.py` | 16-noise overlap sweep on the SVAE substrate | `battery/v1g_svae_overlap/` |
|
| 23 |
+
| `v1i_validity_batteries.py` | dtype / optimizer / schedule batteries (Muon, Lion, fp8-sim…) | `battery/v1i_validity/` |
|
| 24 |
+
| `trigram_bitspace_sweep.py` | byte-embedding bit-space analysis (CPU) | (analysis JSON) |
|
| 25 |
+
|
| 26 |
+
## Replication discipline — read before running
|
| 27 |
+
|
| 28 |
+
1. **Standing repairs (use them):** `ALEPH_TRI_PAD_FIX=1` (trigram pad
|
| 29 |
+
conflation fix; worth +.05–.11 and unlocks address consumption) and
|
| 30 |
+
`V1B_CLEAN_MASK=1` (drops window-truncated answer positions from the
|
| 31 |
+
4×-weighted loss). All *clean-protocol* numbers in the record use both.
|
| 32 |
+
2. **Gauge choice matters:** `masked_eval` scores ~6% information-
|
| 33 |
+
theoretically unanswerable positions (window truncation) — it fakes
|
| 34 |
+
plateaus for near-ceiling models. `selection_eval` filters to the
|
| 35 |
+
answerable subset and additionally decomposes errors into wrong-pair
|
| 36 |
+
retrieval vs no-retrieval. Headline from `selection_eval`.
|
| 37 |
+
3. **History caveat:** result directories produced before 2026-08-10 ran
|
| 38 |
+
without the pad fix and with the truncation-era gauge. Internal
|
| 39 |
+
comparisons within a battery remain valid (shared conditions); do not
|
| 40 |
+
compare absolute numbers across the repair boundary.
|
| 41 |
+
4. **Optimizer:** the aleph mechanism is ~20× more optimizer-sensitive
|
| 42 |
+
than softmax attention; momentum-geometric optimizers (Muon-style,
|
| 43 |
+
SGD-momentum) beat Adam by ~.09 on it. `v1i_validity_batteries.py`
|
| 44 |
+
contains the Muon/Lion implementations used.
|
| 45 |
+
5. **Precision:** train fp32/bf16; never through fp8; fp16 requires the
|
| 46 |
+
dtype-aware clamps already present in these files.
|
| 47 |
+
6. **Data:** synthetic streams are generated deterministically by the
|
| 48 |
+
trainers themselves (seeded); `v1g` pulls the public
|
| 49 |
+
`AbstractPhil/geolip-svae-batteries` noise battery; the wikitext arms
|
| 50 |
+
pull `Salesforce/wikitext` parquet. Default data root is `./data`.
|
| 51 |
+
|
| 52 |
+
All findings, including the failures and the two retracted constants,
|
| 53 |
+
are summarized in the repo README and the technical companion.
|
trainers/__pycache__/ar_differentiation_bed.cpython-312.pyc
ADDED
|
Binary file (39.8 kB). View file
|
|
|
trainers/__pycache__/geolip_vitals.cpython-312.pyc
ADDED
|
Binary file (17.1 kB). View file
|
|
|
trainers/__pycache__/splat_math_battery.cpython-312.pyc
ADDED
|
Binary file (47.3 kB). View file
|
|
|
trainers/__pycache__/trigram_bitspace_sweep.cpython-312.pyc
ADDED
|
Binary file (12 kB). View file
|
|
|
trainers/__pycache__/v1_rank_army_battery.cpython-312.pyc
ADDED
|
Binary file (15.9 kB). View file
|
|
|
trainers/__pycache__/v1b_recall_army_battery.cpython-312.pyc
ADDED
|
Binary file (23 kB). View file
|
|
|
trainers/__pycache__/v1d_svae_graft.cpython-312.pyc
ADDED
|
Binary file (14.7 kB). View file
|
|
|
trainers/__pycache__/v1e_knit_battery.cpython-312.pyc
ADDED
|
Binary file (16.2 kB). View file
|
|
|
trainers/__pycache__/v1f_consult_tier.cpython-312.pyc
ADDED
|
Binary file (18.8 kB). View file
|
|
|
trainers/__pycache__/v1g_svae_overlap_sweep.cpython-312.pyc
ADDED
|
Binary file (14 kB). View file
|
|
|
trainers/__pycache__/v1i_validity_batteries.cpython-312.pyc
ADDED
|
Binary file (25.2 kB). View file
|
|
|
trainers/ar_differentiation_bed.py
ADDED
|
@@ -0,0 +1,499 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""ar_differentiation_bed.py — THE FOCUS (2026-07-09 redirect, AbstractPhil verbatim):
|
| 2 |
+
"refining the autoregressive techniques for differentiation rather than attempting
|
| 3 |
+
to just mash numbers together."
|
| 4 |
+
|
| 5 |
+
Differentiation is cultivated by PREDICTIVE pressure along the sequence — the
|
| 6 |
+
address parameterizing the next-byte distribution (Law 2: chain-rule advantage pays
|
| 7 |
+
ONLY where the composed address directly parameterizes the predictive distribution).
|
| 8 |
+
This bed puts the aleph in the autoregressive gradient path and measures what
|
| 9 |
+
differentiates. It is the Law-2 construction (codebook-pressure C3) + Tree 3d in
|
| 10 |
+
one harness; the Jun-19 "discuss before building" gate was resolved by the redirect.
|
| 11 |
+
|
| 12 |
+
Byte-level causal LM on wikitext-2-raw (HF parquet, CDN-fast), block 256. ARMS:
|
| 13 |
+
sdpa — standard causal transformer control (matched trunk).
|
| 14 |
+
hub — attention replaced by CAUSAL HUB: linear attention whose feature map
|
| 15 |
+
is the 2K-oriented aleph address, prefix-sum memories (no selection
|
| 16 |
+
event; O(n*K*d)). Differentiation cultivated INSIDE attention.
|
| 17 |
+
addr_head — sdpa trunk, but the OUTPUT HEAD reads ONLY the signed aleph
|
| 18 |
+
coefficient vector w_k = sinh(u_k)/sum_j cosh(u_j) of the final
|
| 19 |
+
hidden state (K -> 256 logits). The address MUST carry every bit of
|
| 20 |
+
next-byte information — the hardest Law-2 bottleneck.
|
| 21 |
+
|
| 22 |
+
JUDGED BY: val bits-per-byte per arm (task) + CULTIVATION VITALS on every aleph
|
| 23 |
+
codebook (readouts, never losses): axis aliveness/hppl, drift-from-init +
|
| 24 |
+
binding fraction @0.29154, winner-|cos| saturation (sign-code emergence), shadow
|
| 25 |
+
path diversity (fixed high-bits hash). Never by recon.
|
| 26 |
+
|
| 27 |
+
Riders: pure Adam wd=0; no BN/Dropout/GAP on geometric paths; orthogonal init;
|
| 28 |
+
Colab-cell-safe (paste-ahead imports, no bare argparse, no __file__ reliance);
|
| 29 |
+
GPU-only for verdict runs; data_root OUTSIDE the mind repo.
|
| 30 |
+
|
| 31 |
+
Terminal: python ar_differentiation_bed.py # shapes/parse smoke
|
| 32 |
+
python ar_differentiation_bed.py --train # verdict run
|
| 33 |
+
Colab: paste geolip_vitals.py cell, then this file (smoke auto-runs),
|
| 34 |
+
then train(steps=2000, data_root="/content/data") in the next cell.
|
| 35 |
+
"""
|
| 36 |
+
from __future__ import annotations
|
| 37 |
+
import math
|
| 38 |
+
import os
|
| 39 |
+
import torch
|
| 40 |
+
import torch.nn as nn
|
| 41 |
+
import torch.nn.functional as F
|
| 42 |
+
|
| 43 |
+
if "anchor_drift" not in globals():
|
| 44 |
+
try:
|
| 45 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 46 |
+
except ImportError:
|
| 47 |
+
_here = globals().get("__file__")
|
| 48 |
+
if _here is not None:
|
| 49 |
+
import sys, pathlib
|
| 50 |
+
sys.path.insert(0, str(pathlib.Path(_here).parent))
|
| 51 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 52 |
+
else:
|
| 53 |
+
raise ImportError(
|
| 54 |
+
"geolip_vitals not found — paste/run its cell first, or "
|
| 55 |
+
"keep geolip_vitals.py beside this file (ships in this directory).")
|
| 56 |
+
|
| 57 |
+
VOCAB = 256 # bytes
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
# ------------------------------------------------------------------ aleph address
|
| 61 |
+
def _super_fibonacci_s3(n: int) -> torch.Tensor:
|
| 62 |
+
"""Near-uniform unit quaternions (Alexa CVPR'22; constants per canon) —
|
| 63 |
+
starts the codebook INSIDE the RP^3 attractor basin. D=4 only."""
|
| 64 |
+
PHI, PSI = math.sqrt(2.0), 1.533751168755204288118041
|
| 65 |
+
i = torch.arange(n, dtype=torch.float64)
|
| 66 |
+
s = (i + 0.5) / n
|
| 67 |
+
r, R = torch.sqrt(s), torch.sqrt(1.0 - s)
|
| 68 |
+
a, b = 2 * math.pi * i / PHI, 2 * math.pi * i / PSI
|
| 69 |
+
q = torch.stack([r * torch.sin(a), r * torch.cos(a),
|
| 70 |
+
R * torch.sin(b), R * torch.cos(b)], dim=-1)
|
| 71 |
+
return F.normalize(q, dim=-1).float()
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
class AlephAddress(nn.Module):
|
| 75 |
+
"""Closed-form aleph over 2K oriented half-axes (canon/aleph_core.md).
|
| 76 |
+
signed(x): (..., K) w_k = sinh(u_k)/sum_j cosh(u_j) — the Law-2 head feature.
|
| 77 |
+
oriented(x): ((..., K), (..., K)) positive halves of the 2K softmax — HUB map."""
|
| 78 |
+
|
| 79 |
+
def __init__(self, K: int, D: int, tau: float = 0.1, init: str = "random"):
|
| 80 |
+
super().__init__()
|
| 81 |
+
self.K, self.D, self.tau = K, D, tau
|
| 82 |
+
if init == "fibonacci":
|
| 83 |
+
assert D == 4, "fibonacci init lives on S^3 (D=4)"
|
| 84 |
+
A = _super_fibonacci_s3(K)
|
| 85 |
+
else:
|
| 86 |
+
A = F.normalize(torch.randn(K, D), dim=-1)
|
| 87 |
+
self.codebook = nn.Parameter(A)
|
| 88 |
+
self.register_buffer("home", self.codebook.detach().clone())
|
| 89 |
+
|
| 90 |
+
def _u(self, x):
|
| 91 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 92 |
+
return (F.normalize(x, dim=-1) @ A.transpose(-1, -2)) / self.tau
|
| 93 |
+
|
| 94 |
+
def oriented(self, x):
|
| 95 |
+
u = self._u(x)
|
| 96 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 97 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 98 |
+
Z = (ep + en).sum(dim=-1, keepdim=True)
|
| 99 |
+
return ep / Z, en / Z
|
| 100 |
+
|
| 101 |
+
def signed(self, x):
|
| 102 |
+
u = self._u(x)
|
| 103 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 104 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 105 |
+
return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
|
| 106 |
+
|
| 107 |
+
def signed_at(self, x, taus):
|
| 108 |
+
"""Multi-tau stroboscope (rule of 3): signed coefficients at several
|
| 109 |
+
temperatures, concatenated — softer taus keep the vector dense while a
|
| 110 |
+
hard tau supplies the sign-code sharpness. v2 refinement (b)."""
|
| 111 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 112 |
+
cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
|
| 113 |
+
outs = []
|
| 114 |
+
for t in taus:
|
| 115 |
+
u = cos / t
|
| 116 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 117 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 118 |
+
outs.append((ep - en) / (ep + en).sum(dim=-1, keepdim=True))
|
| 119 |
+
return torch.cat(outs, dim=-1)
|
| 120 |
+
|
| 121 |
+
def m_hat(self, x):
|
| 122 |
+
"""Closed-form soft read (decoders read M_hat, never M). v2 control (c)."""
|
| 123 |
+
u = self._u(x)
|
| 124 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 125 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 126 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 127 |
+
return ((ep - en) @ A) / (ep + en).sum(dim=-1, keepdim=True)
|
| 128 |
+
|
| 129 |
+
def m_hard_ste(self, x):
|
| 130 |
+
"""Canon hard mode: M_hard = sign(cos_win) * A[win], straight-through to
|
| 131 |
+
the soft read — forward fully discrete SIGN CODE, backward soft gradient.
|
| 132 |
+
Legal per theme A (reconstructive sign code, not a one-hot roster pick)."""
|
| 133 |
+
u = self._u(x)
|
| 134 |
+
soft = self.m_hat(x)
|
| 135 |
+
win = u.abs().argmax(dim=-1)
|
| 136 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 137 |
+
sign = torch.sign(torch.gather(u, -1, win.unsqueeze(-1))).squeeze(-1)
|
| 138 |
+
hard = sign.unsqueeze(-1) * A[win]
|
| 139 |
+
return hard + soft - soft.detach()
|
| 140 |
+
|
| 141 |
+
@torch.no_grad()
|
| 142 |
+
def vitals(self, x_sample) -> dict:
|
| 143 |
+
u = self._u(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 144 |
+
p, n = self.oriented(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 145 |
+
two_k = torch.cat([p, n], dim=-1)
|
| 146 |
+
win = two_k.argmax(dim=-1)
|
| 147 |
+
cos_win = (u.abs().amax(dim=-1) * self.tau) # winner |cos| — sign-code sat.
|
| 148 |
+
d = anchor_drift(self.codebook, self.home)
|
| 149 |
+
return {"drift": round(d["mean"], 4),
|
| 150 |
+
"binding_frac": round(d["binding_fraction"], 4),
|
| 151 |
+
"aliveness": axis_aliveness(two_k),
|
| 152 |
+
"win_cos_mean": round(cos_win.mean().item(), 4),
|
| 153 |
+
"paths": path_diversity(win)}
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
# ------------------------------------------------------------------------- blocks
|
| 157 |
+
class CausalSDPA(nn.Module):
|
| 158 |
+
def __init__(self, d: int, heads: int = 4):
|
| 159 |
+
super().__init__()
|
| 160 |
+
self.h = heads
|
| 161 |
+
self.qkv = nn.Linear(d, 3 * d, bias=False)
|
| 162 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 163 |
+
nn.init.orthogonal_(self.qkv.weight); nn.init.orthogonal_(self.o.weight)
|
| 164 |
+
|
| 165 |
+
def forward(self, x):
|
| 166 |
+
B, n, d = x.shape
|
| 167 |
+
q, k, v = self.qkv(x).chunk(3, dim=-1)
|
| 168 |
+
q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2) for t in (q, k, v))
|
| 169 |
+
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 170 |
+
return self.o(y.transpose(1, 2).reshape(B, n, d))
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
class CausalHUB(nn.Module):
|
| 174 |
+
"""Causal aleph linear attention: prefix-sum memories over the two K-wide
|
| 175 |
+
halves of the oriented address; 2K never materialized; no selection event."""
|
| 176 |
+
|
| 177 |
+
def __init__(self, d: int, K: int = 32, D: int = 4, tau: float = 0.1):
|
| 178 |
+
super().__init__()
|
| 179 |
+
self.addr = AlephAddress(K, D, tau)
|
| 180 |
+
self.q = nn.Linear(d, D, bias=False)
|
| 181 |
+
self.k = nn.Linear(d, D, bias=False)
|
| 182 |
+
self.v = nn.Linear(d, d, bias=False)
|
| 183 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 184 |
+
for m in (self.q, self.k, self.v, self.o):
|
| 185 |
+
nn.init.orthogonal_(m.weight)
|
| 186 |
+
|
| 187 |
+
def forward(self, x):
|
| 188 |
+
qp, qn = self.addr.oriented(self.q(x)) # (B, n, K)
|
| 189 |
+
kp, kn = self.addr.oriented(self.k(x))
|
| 190 |
+
v = self.v(x) # (B, n, d)
|
| 191 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 192 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 193 |
+
zp = torch.cumsum(kp, dim=1)
|
| 194 |
+
zn = torch.cumsum(kn, dim=1)
|
| 195 |
+
num = torch.einsum("bnk,bnkd->bnd", qp, Sp) + torch.einsum("bnk,bnkd->bnd", qn, Sn)
|
| 196 |
+
den = (qp * zp).sum(-1, keepdim=True) + (qn * zn).sum(-1, keepdim=True)
|
| 197 |
+
if den.dtype in (torch.float32, torch.float64):
|
| 198 |
+
cl = 1e-12
|
| 199 |
+
else: # fp16/bf16: 1e-12 flushes to 0 (landmine)
|
| 200 |
+
cl = torch.finfo(den.dtype).tiny * 8
|
| 201 |
+
return self.o(num / den.clamp_min(cl))
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
class MslRelay(nn.Module):
|
| 205 |
+
"""Depth-composition unit (chain-rule probe): multi-slot M_hat read entering
|
| 206 |
+
the trunk as a NEAR-ZERO gated residual (gate init -3.0, sigma~0.047 — theme D:
|
| 207 |
+
geometry enters as a nudge and grows only if it earns gradient)."""
|
| 208 |
+
|
| 209 |
+
def __init__(self, d: int, n_slots: int = 16, K: int = 64):
|
| 210 |
+
super().__init__()
|
| 211 |
+
self.n_slots = n_slots
|
| 212 |
+
self.proj = nn.Linear(d, n_slots * 4, bias=False)
|
| 213 |
+
self.out = nn.Linear(n_slots * 4, d, bias=False)
|
| 214 |
+
nn.init.orthogonal_(self.proj.weight)
|
| 215 |
+
nn.init.orthogonal_(self.out.weight)
|
| 216 |
+
self.addr = AlephAddress(K, 4)
|
| 217 |
+
self.gate = nn.Parameter(torch.tensor(-3.0))
|
| 218 |
+
|
| 219 |
+
def forward(self, x):
|
| 220 |
+
B, n, _ = x.shape
|
| 221 |
+
slots = self.proj(x).view(B, n, self.n_slots, 4)
|
| 222 |
+
m = self.addr.m_hat(slots).reshape(B, n, -1)
|
| 223 |
+
return x + self.gate.sigmoid() * self.out(m)
|
| 224 |
+
|
| 225 |
+
|
| 226 |
+
class Block(nn.Module):
|
| 227 |
+
def __init__(self, d: int, attn: nn.Module):
|
| 228 |
+
super().__init__()
|
| 229 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 230 |
+
self.attn = attn
|
| 231 |
+
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
|
| 232 |
+
|
| 233 |
+
def forward(self, x):
|
| 234 |
+
x = x + self.attn(self.n1(x))
|
| 235 |
+
return x + self.mlp(self.n2(x))
|
| 236 |
+
|
| 237 |
+
|
| 238 |
+
class ByteLM(nn.Module):
|
| 239 |
+
def __init__(self, arm: str, d: int = 192, layers: int = 4, block: int = 256,
|
| 240 |
+
K: int = 32, D: int = 4):
|
| 241 |
+
super().__init__()
|
| 242 |
+
# "<arm>_tri" suffix = trigram byte embedding (AlephLM byte_emb x3 lineage):
|
| 243 |
+
# token embedding is the sum of embeddings of bytes t, t-1, t-2.
|
| 244 |
+
self.trigram = arm.endswith("_tri")
|
| 245 |
+
if self.trigram:
|
| 246 |
+
arm = arm[:-4]
|
| 247 |
+
# "_fib" = super-Fibonacci S^3 codebook init (basin test: starts INSIDE
|
| 248 |
+
# the RP^3 attractor; primary observable is init->final geodesic drift).
|
| 249 |
+
self.fib = arm.endswith("_fib")
|
| 250 |
+
if self.fib:
|
| 251 |
+
arm = arm[:-4]
|
| 252 |
+
# "relay*" = stacked addresses in depth: MslRelay after every block.
|
| 253 |
+
# relay -> sdpa trunk + standard head; relay_msl64 -> + addressed head.
|
| 254 |
+
self.use_relay = arm.startswith("relay")
|
| 255 |
+
if arm == "relay":
|
| 256 |
+
arm = "sdpa"
|
| 257 |
+
elif arm == "relay_msl64":
|
| 258 |
+
arm = "addr_msl64"
|
| 259 |
+
self.arm, self.block = arm, block
|
| 260 |
+
# ALEPH_TRI_PAD_FIX=1 (opt-in, 2026-08-09 audit): trigram shift
|
| 261 |
+
# padding used byte 0 — a LEGAL byte (discovery #16: hard-zero
|
| 262 |
+
# padding starves/conflates). Fix = dedicated pad row 256.
|
| 263 |
+
self.pad_fix = os.environ.get("ALEPH_TRI_PAD_FIX") == "1"
|
| 264 |
+
pv = VOCAB + 1 if (self.trigram and self.pad_fix) else VOCAB
|
| 265 |
+
self.emb = nn.Embedding(VOCAB, d)
|
| 266 |
+
if self.trigram:
|
| 267 |
+
self.emb1 = nn.Embedding(pv, d)
|
| 268 |
+
self.emb2 = nn.Embedding(pv, d)
|
| 269 |
+
self.pos = nn.Parameter(torch.zeros(1, block, d) + 0.01 * torch.randn(1, block, d))
|
| 270 |
+
mk_attn = (lambda: CausalHUB(d, K, D)) if arm == "hub" else (lambda: CausalSDPA(d))
|
| 271 |
+
self.blocks = nn.ModuleList([Block(d, mk_attn()) for _ in range(layers)])
|
| 272 |
+
if self.use_relay:
|
| 273 |
+
self.relays = nn.ModuleList([MslRelay(d) for _ in range(layers)])
|
| 274 |
+
self.nf = nn.LayerNorm(d)
|
| 275 |
+
if arm == "addr_head":
|
| 276 |
+
self.head_addr = AlephAddress(K, d) # v1: codebook in model dim — COLLAPSED
|
| 277 |
+
self.head = nn.Linear(K, VOCAB, bias=True)
|
| 278 |
+
elif arm in ("addr_d4", "addr_3tau", "addr_mhat"):
|
| 279 |
+
# v2 refinements: LOW-D HOME — learned projection to the canon D=4 home
|
| 280 |
+
# before addressing (mirrors the healthy HUB arms), K=64.
|
| 281 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 282 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 283 |
+
self.head_addr = AlephAddress(64, 4)
|
| 284 |
+
if arm == "addr_d4":
|
| 285 |
+
self.head = nn.Linear(64, VOCAB, bias=True) # w alone, D=4 home
|
| 286 |
+
elif arm == "addr_3tau":
|
| 287 |
+
self.taus = (0.05, 0.1, 0.3) # rule-of-3 strobe
|
| 288 |
+
self.head = nn.Linear(64 * 3, VOCAB, bias=True)
|
| 289 |
+
else: # addr_mhat
|
| 290 |
+
self.head = nn.Linear(4, VOCAB, bias=True) # tightest: M_hat
|
| 291 |
+
elif arm.startswith("addr_msl"):
|
| 292 |
+
# v3: MULTI-SLOT heads — the 16s funnel widening: P parallel D=4 slots
|
| 293 |
+
# over a SHARED codebook. addr_msl consumes the reconstructive M_hat per
|
| 294 |
+
# slot (Px4 dims); addr_msl_w consumes signed w per slot (Px64) — tests
|
| 295 |
+
# whether slot-parallel consumption alone rescues the coefficient path.
|
| 296 |
+
# addr_msl<P> = slot-count dose-response. addr_mslh<P> = HARD sign-code
|
| 297 |
+
# consumption (straight-through M_hard per slot).
|
| 298 |
+
self.hard = arm.startswith("addr_mslh")
|
| 299 |
+
if arm in ("addr_msl", "addr_msl_w"):
|
| 300 |
+
self.n_slots = 16
|
| 301 |
+
else:
|
| 302 |
+
self.n_slots = int(arm[len("addr_mslh" if self.hard else "addr_msl"):])
|
| 303 |
+
self.head_proj = nn.Linear(d, self.n_slots * 4, bias=False)
|
| 304 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 305 |
+
self.head_addr = AlephAddress(
|
| 306 |
+
64, 4, init="fibonacci" if self.fib else "random")
|
| 307 |
+
width = self.n_slots * (64 if arm == "addr_msl_w" else 4)
|
| 308 |
+
self.head = nn.Linear(width, VOCAB, bias=True)
|
| 309 |
+
elif arm == "addr_3tau_mhat":
|
| 310 |
+
# v3: combine the two v2 winners — 3-tau stroboscope + reconstructive read.
|
| 311 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 312 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 313 |
+
self.head_addr = AlephAddress(64, 4)
|
| 314 |
+
self.taus = (0.05, 0.1, 0.3)
|
| 315 |
+
self.head = nn.Linear(64 * 3 + 4, VOCAB, bias=True)
|
| 316 |
+
else:
|
| 317 |
+
self.head = nn.Linear(d, VOCAB, bias=True)
|
| 318 |
+
self._last_h = None
|
| 319 |
+
|
| 320 |
+
def forward(self, idx):
|
| 321 |
+
x = self.emb(idx)
|
| 322 |
+
if self.trigram: # past-only shifts — causality preserved
|
| 323 |
+
pv = VOCAB if self.pad_fix else 0
|
| 324 |
+
x = x + self.emb1(F.pad(idx, (1, 0), value=pv)[:, :-1]) \
|
| 325 |
+
+ self.emb2(F.pad(idx, (2, 0), value=pv)[:, :-2])
|
| 326 |
+
x = x + self.pos[:, : idx.shape[1]]
|
| 327 |
+
if self.use_relay:
|
| 328 |
+
for b, r in zip(self.blocks, self.relays):
|
| 329 |
+
x = r(b(x))
|
| 330 |
+
else:
|
| 331 |
+
for b in self.blocks:
|
| 332 |
+
x = b(x)
|
| 333 |
+
h = self.nf(x)
|
| 334 |
+
self._last_h = h.detach()
|
| 335 |
+
if self.arm == "addr_head":
|
| 336 |
+
return self.head(self.head_addr.signed(h))
|
| 337 |
+
if self.arm == "addr_d4":
|
| 338 |
+
return self.head(self.head_addr.signed(self.head_proj(h)))
|
| 339 |
+
if self.arm == "addr_3tau":
|
| 340 |
+
return self.head(self.head_addr.signed_at(self.head_proj(h), self.taus))
|
| 341 |
+
if self.arm == "addr_mhat":
|
| 342 |
+
return self.head(self.head_addr.m_hat(self.head_proj(h)))
|
| 343 |
+
if self.arm.startswith("addr_msl"):
|
| 344 |
+
B, n, _ = h.shape
|
| 345 |
+
slots = self.head_proj(h).view(B, n, self.n_slots, 4)
|
| 346 |
+
if self.arm == "addr_msl_w":
|
| 347 |
+
feats = self.head_addr.signed(slots).reshape(B, n, -1)
|
| 348 |
+
elif getattr(self, "hard", False):
|
| 349 |
+
feats = self.head_addr.m_hard_ste(slots).reshape(B, n, -1)
|
| 350 |
+
else:
|
| 351 |
+
feats = self.head_addr.m_hat(slots).reshape(B, n, -1)
|
| 352 |
+
return self.head(feats)
|
| 353 |
+
if self.arm == "addr_3tau_mhat":
|
| 354 |
+
p = self.head_proj(h)
|
| 355 |
+
feats = torch.cat([self.head_addr.signed_at(p, self.taus),
|
| 356 |
+
self.head_addr.m_hat(p)], dim=-1)
|
| 357 |
+
return self.head(feats)
|
| 358 |
+
return self.head(h)
|
| 359 |
+
|
| 360 |
+
@torch.no_grad()
|
| 361 |
+
def vitals(self) -> dict:
|
| 362 |
+
out = {}
|
| 363 |
+
if self.arm == "hub":
|
| 364 |
+
for i, b in enumerate(self.blocks):
|
| 365 |
+
if self._last_h is not None:
|
| 366 |
+
out[f"L{i}"] = b.attn.addr.vitals(b.attn.q(self._last_h[:2]))
|
| 367 |
+
elif self.arm == "addr_head" and self._last_h is not None:
|
| 368 |
+
out["head"] = self.head_addr.vitals(self._last_h[:2])
|
| 369 |
+
elif self.arm in ("addr_d4", "addr_3tau", "addr_mhat",
|
| 370 |
+
"addr_3tau_mhat") and self._last_h is not None:
|
| 371 |
+
out["head"] = self.head_addr.vitals(self.head_proj(self._last_h[:2]))
|
| 372 |
+
elif self.arm.startswith("addr_msl") and self._last_h is not None:
|
| 373 |
+
slots = self.head_proj(self._last_h[:2])
|
| 374 |
+
out["head"] = self.head_addr.vitals(
|
| 375 |
+
slots.reshape(*slots.shape[:-1], self.n_slots, 4))
|
| 376 |
+
if self.use_relay and self._last_h is not None:
|
| 377 |
+
for i, r in enumerate(self.relays):
|
| 378 |
+
s = r.proj(self._last_h[:2])
|
| 379 |
+
v = r.addr.vitals(s.reshape(*s.shape[:-1], r.n_slots, 4))
|
| 380 |
+
out[f"relay{i}"] = {"gate": round(r.gate.sigmoid().item(), 4),
|
| 381 |
+
"drift": v["drift"],
|
| 382 |
+
"binding_frac": v["binding_frac"],
|
| 383 |
+
"ppl": round(v["aliveness"]["usage_ppl"], 1)}
|
| 384 |
+
return out
|
| 385 |
+
|
| 386 |
+
|
| 387 |
+
# --------------------------------------------------------------------------- data
|
| 388 |
+
def _wikitext_bytes(data_root: str):
|
| 389 |
+
"""wikitext-2-raw as flat uint8 tensors via the HF parquet CDN."""
|
| 390 |
+
from huggingface_hub import hf_hub_download
|
| 391 |
+
import pyarrow.parquet as pq
|
| 392 |
+
|
| 393 |
+
def load(split):
|
| 394 |
+
p = hf_hub_download("Salesforce/wikitext",
|
| 395 |
+
f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
|
| 396 |
+
repo_type="dataset", local_dir=data_root)
|
| 397 |
+
text = "".join(pq.read_table(p).column("text").to_pylist())
|
| 398 |
+
return torch.frombuffer(bytearray(text.encode("utf-8")), dtype=torch.uint8).clone()
|
| 399 |
+
|
| 400 |
+
return load("train"), load("validation")
|
| 401 |
+
|
| 402 |
+
|
| 403 |
+
def _batch(data: torch.Tensor, batch: int, block: int, device, g: torch.Generator):
|
| 404 |
+
ix = torch.randint(0, data.numel() - block - 1, (batch,), generator=g)
|
| 405 |
+
x = torch.stack([data[i:i + block] for i in ix]).long().to(device)
|
| 406 |
+
y = torch.stack([data[i + 1:i + block + 1] for i in ix]).long().to(device)
|
| 407 |
+
return x, y
|
| 408 |
+
|
| 409 |
+
|
| 410 |
+
# -------------------------------------------------------------------- train/smoke
|
| 411 |
+
def train(arms=("sdpa", "hub", "addr_head"), steps: int = 2000, batch: int = 32,
|
| 412 |
+
block: int = 256, device: str = "cuda", data_root: str = "./data",
|
| 413 |
+
seed: int = 0, eval_every: int = 500, save: bool = True):
|
| 414 |
+
"""Verdict run — GPU only. Pure Adam wd=0. Reports val bits-per-byte + vitals.
|
| 415 |
+
save=True writes {data_root}/ar_ckpts/{arm}_s{seed}_t{steps}.pt per arm —
|
| 416 |
+
the cultivated codebooks are SPECIMENS for the projective reading instruments."""
|
| 417 |
+
import os
|
| 418 |
+
if device == "cuda" and not torch.cuda.is_available():
|
| 419 |
+
raise RuntimeError("Verdict runs are GPU-only (never CPU-train for accuracy).")
|
| 420 |
+
ckpt_dir = os.path.join(data_root, "ar_ckpts")
|
| 421 |
+
os.makedirs(ckpt_dir, exist_ok=True)
|
| 422 |
+
tr, va = _wikitext_bytes(data_root)
|
| 423 |
+
print(f"data ready: train {tr.numel():,} bytes, val {va.numel():,} bytes", flush=True)
|
| 424 |
+
results = {}
|
| 425 |
+
for arm in arms:
|
| 426 |
+
torch.manual_seed(seed)
|
| 427 |
+
g = torch.Generator().manual_seed(seed)
|
| 428 |
+
model = ByteLM(arm, block=block).to(device)
|
| 429 |
+
n_params = sum(p.numel() for p in model.parameters())
|
| 430 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 431 |
+
for step in range(1, steps + 1):
|
| 432 |
+
x, y = _batch(tr, batch, block, device, g)
|
| 433 |
+
logits = model(x)
|
| 434 |
+
loss = F.cross_entropy(logits.reshape(-1, VOCAB), y.reshape(-1))
|
| 435 |
+
opt.zero_grad(set_to_none=True)
|
| 436 |
+
loss.backward()
|
| 437 |
+
opt.step()
|
| 438 |
+
if step % eval_every == 0 or step == steps:
|
| 439 |
+
model.eval()
|
| 440 |
+
with torch.no_grad():
|
| 441 |
+
losses = []
|
| 442 |
+
for _ in range(20):
|
| 443 |
+
xv, yv = _batch(va, batch, block, device, g)
|
| 444 |
+
lv = F.cross_entropy(model(xv).reshape(-1, VOCAB),
|
| 445 |
+
yv.reshape(-1))
|
| 446 |
+
losses.append(lv.item())
|
| 447 |
+
bpb = sum(losses) / len(losses) / math.log(2)
|
| 448 |
+
print(f"[{arm}] step {step} val_bpb={bpb:.4f} vitals={model.vitals()}",
|
| 449 |
+
flush=True)
|
| 450 |
+
model.train()
|
| 451 |
+
results[arm] = {"val_bpb": bpb, "params": n_params, "vitals": model.vitals()}
|
| 452 |
+
if save:
|
| 453 |
+
path = os.path.join(ckpt_dir, f"{arm}_s{seed}_t{steps}.pt")
|
| 454 |
+
torch.save({"arm": arm, "seed": seed, "steps": steps, "val_bpb": bpb,
|
| 455 |
+
"state_dict": {k: v.cpu() for k, v in
|
| 456 |
+
model.state_dict().items()}}, path)
|
| 457 |
+
print(f"saved specimen: {path}", flush=True)
|
| 458 |
+
print(results, flush=True)
|
| 459 |
+
return results
|
| 460 |
+
|
| 461 |
+
|
| 462 |
+
def smoke():
|
| 463 |
+
"""Shapes/parse only — no accuracy claims."""
|
| 464 |
+
x = torch.randint(0, VOCAB, (2, 64))
|
| 465 |
+
for arm in ("sdpa", "hub", "addr_head"):
|
| 466 |
+
m = ByteLM(arm, d=96, layers=2, block=64, K=16)
|
| 467 |
+
logits = m(x)
|
| 468 |
+
assert logits.shape == (2, 64, VOCAB)
|
| 469 |
+
logits.sum().backward()
|
| 470 |
+
# causality check: future byte must not affect past logits
|
| 471 |
+
with torch.no_grad():
|
| 472 |
+
a = m(x)[0, 10]
|
| 473 |
+
x2 = x.clone(); x2[0, 40] = (x2[0, 40] + 7) % 256
|
| 474 |
+
b = m(x2)[0, 10]
|
| 475 |
+
assert torch.allclose(a, b, atol=1e-4), f"{arm} leaks future context"
|
| 476 |
+
print(f"{arm}: OK params={sum(p.numel() for p in m.parameters()):,} "
|
| 477 |
+
f"vitals={m.vitals()}", flush=True)
|
| 478 |
+
print("OK — AR bed smoke passed (verdict run: train() on GPU)", flush=True)
|
| 479 |
+
|
| 480 |
+
|
| 481 |
+
def _in_notebook() -> bool:
|
| 482 |
+
try:
|
| 483 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 484 |
+
return True
|
| 485 |
+
except NameError:
|
| 486 |
+
return False
|
| 487 |
+
|
| 488 |
+
|
| 489 |
+
if __name__ == "__main__":
|
| 490 |
+
if _in_notebook():
|
| 491 |
+
smoke()
|
| 492 |
+
print("Notebook mode: call train(steps=2000) in the next cell (GPU).")
|
| 493 |
+
else:
|
| 494 |
+
import argparse
|
| 495 |
+
ap = argparse.ArgumentParser()
|
| 496 |
+
ap.add_argument("--train", action="store_true")
|
| 497 |
+
ap.add_argument("--steps", type=int, default=2000)
|
| 498 |
+
a, _ = ap.parse_known_args()
|
| 499 |
+
train(steps=a.steps) if a.train else smoke()
|
trainers/geolip_vitals.py
ADDED
|
@@ -0,0 +1,219 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""geolip_vitals.py — the shared diagnostic harness (progression plan Tree 7a).
|
| 2 |
+
One implementation, every tree imports it. ALL functions are READOUTS: no gradients,
|
| 3 |
+
no losses. CV is a readout, never a force (discovery_catalog #3). Judge addressing by
|
| 4 |
+
drift->0.29154 and CV->0.20, never recon cosine (MANIFEST).
|
| 5 |
+
|
| 6 |
+
Vitals provided:
|
| 7 |
+
anchor_drift — geodesic drift of anchors from init; binding fraction @0.29154
|
| 8 |
+
pentachoron_cv — CM 4-volume CV over random 5-row subsets (geovocab2 import)
|
| 9 |
+
axis_aliveness — oriented-address usage: axes alive, hppl, collapse flag
|
| 10 |
+
gate_stats — gate means vs the 0.012-0.03 band
|
| 11 |
+
path_diversity — unique-path counting, FIXED high-bits hash (low-16 bug is the
|
| 12 |
+
retracted artifact — never use the low bits)
|
| 13 |
+
grad_norm_spread — gradient democracy monitor (orders-of-magnitude spread)
|
| 14 |
+
CVScreen — CV@1000-batch early band screen (<0.30 LOW / .35-.50 MID / >.80 HIGH)
|
| 15 |
+
|
| 16 |
+
Smoke on a torch-capable env: python geolip_vitals.py
|
| 17 |
+
"""
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
import math
|
| 20 |
+
import torch
|
| 21 |
+
|
| 22 |
+
BINDING = 0.29154 # radians; the binding/separation constant (MANIFEST)
|
| 23 |
+
CV_BAND = (0.13, 0.30) # CM CV band (discovery_catalog #4)
|
| 24 |
+
GATE_BAND = (0.012, 0.03) # live invariant candidate (acd_campaign)
|
| 25 |
+
KNUTH32 = 2654435761
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
# ----------------------------------------------------------------------------- drift
|
| 29 |
+
@torch.no_grad()
|
| 30 |
+
def anchor_drift(current: torch.Tensor, init: torch.Tensor, tol: float = 0.05) -> dict:
|
| 31 |
+
"""Geodesic drift (radians) of each row of `current` from its row in `init`,
|
| 32 |
+
both row-normalized. Returns mean/std/per-row drift and the fraction of rows
|
| 33 |
+
within +/-tol of BINDING (the GLFM '46%' readout)."""
|
| 34 |
+
a = torch.nn.functional.normalize(current.float(), dim=-1)
|
| 35 |
+
b = torch.nn.functional.normalize(init.float(), dim=-1)
|
| 36 |
+
cos = (a * b).sum(-1).clamp(-1.0, 1.0)
|
| 37 |
+
drift = torch.arccos(cos)
|
| 38 |
+
frac = ((drift - BINDING).abs() <= tol).float().mean()
|
| 39 |
+
return {"mean": drift.mean().item(), "std": drift.std().item(),
|
| 40 |
+
"per_row": drift, "binding_fraction": frac.item()}
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
# -------------------------------------------------------------------------------- cv
|
| 44 |
+
@torch.no_grad()
|
| 45 |
+
def _pentachoron_volumes(pts: torch.Tensor) -> torch.Tensor:
|
| 46 |
+
"""Batched Cayley-Menger 4-simplex volumes. pts: (B, 5, D) -> (B,) volumes.
|
| 47 |
+
One float64 det over all samples (vol^2 = -det(CM)/9216 for n=4). Built-in
|
| 48 |
+
per the 2026-07-11 rider amendment (AbstractPhil: the installed geovocab2 path is a
|
| 49 |
+
per-sample class call — too slow for a vitals loop); geovocab2 stays the
|
| 50 |
+
reference implementation, parity-checked via cv_reference_check()."""
|
| 51 |
+
B = pts.shape[0]
|
| 52 |
+
d2 = torch.cdist(pts.double(), pts.double()).pow(2) # (B,5,5)
|
| 53 |
+
cm = torch.ones(B, 6, 6, dtype=torch.float64, device=pts.device)
|
| 54 |
+
cm[:, 0, 0] = 0.0
|
| 55 |
+
cm[:, 1:, 1:] = d2
|
| 56 |
+
det = torch.linalg.det(cm)
|
| 57 |
+
return (-det / 9216.0).clamp_min(0.0).sqrt().float()
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
@torch.no_grad()
|
| 61 |
+
def pentachoron_cv(rows: torch.Tensor, n_samples: int = 200,
|
| 62 |
+
generator: torch.Generator | None = None) -> float:
|
| 63 |
+
"""CV (std/mean) of Cayley-Menger 4-simplex volumes over n_samples random
|
| 64 |
+
5-row subsets. Rows are row-normalized before measurement. Uses the built-in
|
| 65 |
+
batched CM (float64 det); validate against geovocab2 with
|
| 66 |
+
cv_reference_check() after any change to the volume math."""
|
| 67 |
+
x = torch.nn.functional.normalize(rows.float(), dim=-1)
|
| 68 |
+
n = x.shape[0]
|
| 69 |
+
if n < 5:
|
| 70 |
+
raise ValueError(f"pentachoron_cv needs >=5 rows, got {n}")
|
| 71 |
+
g = generator or torch.Generator(device="cpu").manual_seed(0)
|
| 72 |
+
idx = torch.stack([torch.randperm(n, generator=g)[:5]
|
| 73 |
+
for _ in range(n_samples)]) # (B,5)
|
| 74 |
+
v = _pentachoron_volumes(x[idx].cpu())
|
| 75 |
+
return (v.std() / v.mean().clamp_min(1e-12)).item()
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
@torch.no_grad()
|
| 79 |
+
def cv_reference_check(n_trials: int = 50, tol: float = 1e-5) -> float:
|
| 80 |
+
"""Parity check of the built-in batched CM against geovocab2's reference
|
| 81 |
+
implementation (the formula's source of truth). Returns max |rel diff|;
|
| 82 |
+
raises if geovocab2 is absent or parity fails. Run after touching
|
| 83 |
+
_pentachoron_volumes."""
|
| 84 |
+
try:
|
| 85 |
+
from geovocab2.shapes.formula.symbolic.cayley_menger import (
|
| 86 |
+
CayleyMengerFromSimplex)
|
| 87 |
+
except Exception as e: # pragma: no cover
|
| 88 |
+
raise ImportError(
|
| 89 |
+
"cv_reference_check requires geovocab2 (install via the geolip-svae "
|
| 90 |
+
"umbrella: pip install git+https://github.com/AbstractEyes/"
|
| 91 |
+
"geolip-svae).") from e
|
| 92 |
+
ref = CayleyMengerFromSimplex()
|
| 93 |
+
g = torch.Generator().manual_seed(0)
|
| 94 |
+
pts = torch.nn.functional.normalize(
|
| 95 |
+
torch.randn(n_trials, 5, 4, generator=g), dim=-1)
|
| 96 |
+
mine = _pentachoron_volumes(pts)
|
| 97 |
+
# compare at float64: the reference computes in the INPUT dtype, and fp32
|
| 98 |
+
# dets lose up to ~4% on near-degenerate pentachora (measured 2026-07-11)
|
| 99 |
+
theirs = torch.stack([ref.forward(p.double())["volume"].float() for p in pts])
|
| 100 |
+
rel = ((mine - theirs).abs() / theirs.abs().clamp_min(1e-12)).max().item()
|
| 101 |
+
if rel > tol:
|
| 102 |
+
raise AssertionError(f"CM parity vs geovocab2 failed: max rel {rel}")
|
| 103 |
+
return rel
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
# ------------------------------------------------------------------------- aliveness
|
| 107 |
+
@torch.no_grad()
|
| 108 |
+
def axis_aliveness(oriented_weights: torch.Tensor, alive_thresh: float = 1e-3) -> dict:
|
| 109 |
+
"""`oriented_weights`: (..., 2K) nonnegative oriented-softmax address rows
|
| 110 |
+
(sum to 1 on the last dim). Returns axes-alive count, mean-usage perplexity
|
| 111 |
+
(hppl analogue; healthy hosted reference 125-126/128), and a collapse flag.
|
| 112 |
+
Reference behavior: near-uniform aliveness at div_weight=0 (discovery #22)."""
|
| 113 |
+
w = oriented_weights.reshape(-1, oriented_weights.shape[-1]).float()
|
| 114 |
+
usage = w.mean(0)
|
| 115 |
+
usage = usage / usage.sum().clamp_min(1e-12)
|
| 116 |
+
# an axis is alive if its mean usage exceeds alive_thresh x the uniform share
|
| 117 |
+
alive = int((usage > alive_thresh * (1.0 / usage.numel())).sum())
|
| 118 |
+
ent = -(usage.clamp_min(1e-12) * usage.clamp_min(1e-12).log()).sum()
|
| 119 |
+
ppl = float(ent.exp())
|
| 120 |
+
return {"axes_total": usage.numel(), "axes_alive": alive, "usage_ppl": ppl,
|
| 121 |
+
"collapsed": ppl < 0.05 * usage.numel()}
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# ------------------------------------------------------------------------------ gates
|
| 125 |
+
@torch.no_grad()
|
| 126 |
+
def gate_stats(gates: torch.Tensor) -> dict:
|
| 127 |
+
"""Gate values (post-sigmoid/clamp). Reports mean and whether it sits in the
|
| 128 |
+
0.012-0.03 band (read-only — the band is a candidate invariant, never a target)."""
|
| 129 |
+
g = gates.float().flatten()
|
| 130 |
+
m = g.mean().item()
|
| 131 |
+
return {"mean": m, "std": g.std().item(),
|
| 132 |
+
"in_band": GATE_BAND[0] <= m <= GATE_BAND[1]}
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
# ------------------------------------------------------------------------------ paths
|
| 136 |
+
@torch.no_grad()
|
| 137 |
+
def path_diversity(ids: torch.Tensor) -> dict:
|
| 138 |
+
"""Unique-path counting with the FIXED multiplicative hash:
|
| 139 |
+
((ids * 2654435761) % 2^32) >> 16 — Knuth needs the HIGH bits; the low-16
|
| 140 |
+
variant produced the retracted ~1,500 path ceiling (sessions/2026-07-07).
|
| 141 |
+
`ids`: integer tensor, one composed path id per row (any shape)."""
|
| 142 |
+
x = ids.reshape(-1).to(torch.int64)
|
| 143 |
+
hashed = ((x * KNUTH32) % (1 << 32)) >> 16
|
| 144 |
+
return {"n": int(x.numel()),
|
| 145 |
+
"unique_raw": int(torch.unique(x).numel()),
|
| 146 |
+
"unique_hashed": int(torch.unique(hashed).numel())}
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
@torch.no_grad()
|
| 150 |
+
def compose_path_ids(stage_indices: list[torch.Tensor], radix: int) -> torch.Tensor:
|
| 151 |
+
"""Compose per-stage discrete indices (each (...,) int in [0, radix)) into a
|
| 152 |
+
single path id, positional base-`radix` — construction, not hashing."""
|
| 153 |
+
out = torch.zeros_like(stage_indices[0], dtype=torch.int64)
|
| 154 |
+
for s in stage_indices:
|
| 155 |
+
out = out * radix + s.to(torch.int64)
|
| 156 |
+
return out
|
| 157 |
+
|
| 158 |
+
|
| 159 |
+
# --------------------------------------------------------------------- grad democracy
|
| 160 |
+
@torch.no_grad()
|
| 161 |
+
def grad_norm_spread(groups: dict[str, list[torch.nn.Parameter]]) -> dict:
|
| 162 |
+
"""Gradient-democracy monitor. `groups`: name -> params of one parallel member
|
| 163 |
+
(tower/expert). Reports per-group grad norms and the orders-of-magnitude spread.
|
| 164 |
+
Reference: unequalized heterogeneous towers spread ~20 orders (fibonacci dead at
|
| 165 |
+
2.25e-21 under helix); equalized ~0.0 (canon/fibonacci_systems.md)."""
|
| 166 |
+
norms = {}
|
| 167 |
+
for name, params in groups.items():
|
| 168 |
+
gs = [p.grad for p in params if p.grad is not None]
|
| 169 |
+
norms[name] = float(torch.sqrt(sum((g.float() ** 2).sum() for g in gs)).item()) \
|
| 170 |
+
if gs else 0.0
|
| 171 |
+
vals = [v for v in norms.values() if v > 0]
|
| 172 |
+
spread = (math.log10(max(vals)) - math.log10(min(vals))) if len(vals) >= 2 else 0.0
|
| 173 |
+
return {"norms": norms, "spread_orders": spread, "dead": [k for k, v in norms.items() if v == 0.0]}
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
# ----------------------------------------------------------------------------- screen
|
| 177 |
+
class CVScreen:
|
| 178 |
+
"""CV@N early band screen (tri-band ft1): record pentachoron CV at `step_mark`
|
| 179 |
+
batches; classify <0.30 LOW / 0.35-0.50 MID / >0.80 HIGH. Turns ~2h/config
|
| 180 |
+
into ~7min. Readout only."""
|
| 181 |
+
def __init__(self, step_mark: int = 1000):
|
| 182 |
+
self.step_mark = step_mark
|
| 183 |
+
self.recorded: float | None = None
|
| 184 |
+
|
| 185 |
+
def maybe_record(self, step: int, rows: torch.Tensor) -> float | None:
|
| 186 |
+
if self.recorded is None and step >= self.step_mark:
|
| 187 |
+
self.recorded = pentachoron_cv(rows)
|
| 188 |
+
return self.recorded
|
| 189 |
+
|
| 190 |
+
@property
|
| 191 |
+
def band(self) -> str | None:
|
| 192 |
+
c = self.recorded
|
| 193 |
+
if c is None:
|
| 194 |
+
return None
|
| 195 |
+
if c < 0.30:
|
| 196 |
+
return "LOW"
|
| 197 |
+
if 0.35 <= c <= 0.50:
|
| 198 |
+
return "MID"
|
| 199 |
+
if c > 0.80:
|
| 200 |
+
return "HIGH"
|
| 201 |
+
return "BETWEEN"
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
# ------------------------------------------------------------------------------ smoke
|
| 205 |
+
if __name__ == "__main__": # shapes/parse smoke ONLY — no training, ever.
|
| 206 |
+
g = torch.Generator().manual_seed(0)
|
| 207 |
+
K, D = 64, 4
|
| 208 |
+
init = torch.nn.functional.normalize(torch.randn(K, D, generator=g), dim=-1)
|
| 209 |
+
cur = torch.nn.functional.normalize(init + 0.29 * torch.randn(K, D, generator=g), dim=-1)
|
| 210 |
+
print("drift:", {k: v for k, v in anchor_drift(cur, init).items() if k != "per_row"})
|
| 211 |
+
w = torch.softmax(torch.randn(32, 2 * K, generator=g), dim=-1)
|
| 212 |
+
print("aliveness:", axis_aliveness(w))
|
| 213 |
+
print("gates:", gate_stats(torch.full((8,), 0.024)))
|
| 214 |
+
ids = compose_path_ids([torch.randint(0, 16, (4096,), generator=g) for _ in range(4)], 16)
|
| 215 |
+
print("paths:", path_diversity(ids))
|
| 216 |
+
lin = torch.nn.Linear(8, 8)
|
| 217 |
+
lin(torch.randn(4, 8)).sum().backward()
|
| 218 |
+
print("democracy:", grad_norm_spread({"a": list(lin.parameters())}))
|
| 219 |
+
print("OK — vitals smoke passed (pentachoron_cv needs geovocab2; run on GPU env)")
|
trainers/splat_math_battery.py
ADDED
|
@@ -0,0 +1,644 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# splat_math_battery.py — B1-B6: the multi-aleph / splat hypothesis math
|
| 3 |
+
# =========================================================================
|
| 4 |
+
# Plan: history/plans/2026-08-06_splat_aleph_program.md (Rorschach
|
| 5 |
+
# correction + battery authorization). AbstractPhil: "test the hypothesis states
|
| 6 |
+
# for the mathematics before we structure the next attention mechanism."
|
| 7 |
+
#
|
| 8 |
+
# Every experiment uses AlephAddress VERBATIM (refine-don't-rewrite).
|
| 9 |
+
# All synthetic, seeded, local-GPU, minutes-per-arm. No attention
|
| 10 |
+
# mechanism anywhere in this file — measurement only.
|
| 11 |
+
#
|
| 12 |
+
# B1 capacity single-aleph capacity curves: K x D x tau x data-family
|
| 13 |
+
# -> usage ppl, realized sign-codes, aliasing, m_hat
|
| 14 |
+
# reconstruction, capacity per half-axis
|
| 15 |
+
# B2 product M differentiated frames vs 1 monolith at EQUAL total
|
| 16 |
+
# half-axes: joint code diversity, joint readback,
|
| 17 |
+
# pairwise redundancy (the product-code hypothesis)
|
| 18 |
+
# B3 dynamics does init-differentiation PERSIST under shared-task
|
| 19 |
+
# training with NO diversity pressure? (basin-set-at-
|
| 20 |
+
# init, multi-aleph edition; copies arm = collapse ctrl)
|
| 21 |
+
# B4 opinion composition gating: uniform vs SELF-confidence (non-
|
| 22 |
+
# comparative) vs softmax-over-alephs (LABELED
|
| 23 |
+
# falsification control) vs oracle, on data built so
|
| 24 |
+
# perspectives hold real differential advantage
|
| 25 |
+
# B5 blackboard associative-recall capacity of write/read through the
|
| 26 |
+
# codebook: P_max(K) at recall cos>=.7, single vs
|
| 27 |
+
# M-differentiated; crosstalk spectra — the sizing
|
| 28 |
+
# number for any future splat-attention head
|
| 29 |
+
# B6 boundary positional splat supports: overlap ratio vs max solid
|
| 30 |
+
# relay distance on long chains; tiling (zero overlap)
|
| 31 |
+
# as the control that must reproduce the deep-line
|
| 32 |
+
# tile-edge deadness
|
| 33 |
+
#
|
| 34 |
+
# RUN: python tools\splat_math_battery.py (all arms)
|
| 35 |
+
# ALEPH_SPLAT_ARM=B1..B6 for one; ALEPH_SPLAT_QUICK=1 smoke sizes
|
| 36 |
+
# Ships splat_battery_<arm>.json to AbstractPhil/aleph-splat-0 (private).
|
| 37 |
+
# =========================================================================
|
| 38 |
+
|
| 39 |
+
import importlib
|
| 40 |
+
import json
|
| 41 |
+
import math
|
| 42 |
+
import os
|
| 43 |
+
import sys
|
| 44 |
+
import time
|
| 45 |
+
import zlib
|
| 46 |
+
|
| 47 |
+
import numpy as np
|
| 48 |
+
import torch
|
| 49 |
+
import torch.nn.functional as F
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def _load(name):
|
| 53 |
+
if name in sys.modules:
|
| 54 |
+
return sys.modules[name]
|
| 55 |
+
try:
|
| 56 |
+
return importlib.import_module(name)
|
| 57 |
+
except ImportError:
|
| 58 |
+
here = globals().get("__file__")
|
| 59 |
+
if here is not None:
|
| 60 |
+
import pathlib
|
| 61 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 62 |
+
return importlib.import_module(name)
|
| 63 |
+
raise
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
for _s in (sys.stdout, sys.stderr):
|
| 67 |
+
if hasattr(_s, "reconfigure"):
|
| 68 |
+
_s.reconfigure(encoding="utf-8", errors="replace")
|
| 69 |
+
|
| 70 |
+
adb = _load("ar_differentiation_bed")
|
| 71 |
+
AlephAddress = adb.AlephAddress
|
| 72 |
+
|
| 73 |
+
DEV = "cuda" if torch.cuda.is_available() else "cpu"
|
| 74 |
+
QUICK = os.environ.get("ALEPH_SPLAT_QUICK") == "1"
|
| 75 |
+
ARM = os.environ.get("ALEPH_SPLAT_ARM", "ALL").upper()
|
| 76 |
+
OUT_DIR = os.environ.get("ALEPH_SPLAT_OUT", "./data/aleph_splat"
|
| 77 |
+
if os.name == "nt" else "./aleph_splat")
|
| 78 |
+
HUB = os.environ.get("ALEPH_SPLAT_REPO", "AbstractPhil/aleph-splat-0")
|
| 79 |
+
N = 20_000 if QUICK else 200_000
|
| 80 |
+
SEED0 = zlib.crc32(b"aleph-splat-0") % (2 ** 31)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def line(t=""):
|
| 84 |
+
print("-" * 78 if not t else f"-- {t} " + "-" * max(0, 74 - len(t)))
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def gen(family, n, D, seed, c_clusters=32, r_lowrank=None, ar=0.98):
|
| 88 |
+
"""Synthetic unit-vector families. All seeded, all on DEV."""
|
| 89 |
+
g = torch.Generator(device="cpu").manual_seed(seed)
|
| 90 |
+
if family == "iso":
|
| 91 |
+
x = torch.randn(n, D, generator=g)
|
| 92 |
+
elif family == "clust":
|
| 93 |
+
cents = F.normalize(torch.randn(c_clusters, D, generator=g), dim=-1)
|
| 94 |
+
idx = torch.randint(0, c_clusters, (n,), generator=g)
|
| 95 |
+
x = cents[idx] + 0.25 * torch.randn(n, D, generator=g)
|
| 96 |
+
elif family == "lowrank":
|
| 97 |
+
r = r_lowrank or max(2, D // 4)
|
| 98 |
+
basis = torch.linalg.qr(torch.randn(D, r, generator=g))[0]
|
| 99 |
+
x = torch.randn(n, r, generator=g) @ basis.T
|
| 100 |
+
elif family == "seq":
|
| 101 |
+
steps = F.normalize(torch.randn(n, D, generator=g), dim=-1)
|
| 102 |
+
x = torch.empty(n, D)
|
| 103 |
+
x[0] = steps[0]
|
| 104 |
+
for i in range(1, n):
|
| 105 |
+
x[i] = ar * x[i - 1] + math.sqrt(1 - ar * ar) * steps[i]
|
| 106 |
+
else:
|
| 107 |
+
raise ValueError(family)
|
| 108 |
+
return F.normalize(x, dim=-1).to(DEV)
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
@torch.no_grad()
|
| 112 |
+
def read_stats(addr, x, pair_sample=4096):
|
| 113 |
+
"""Single-aleph gauges: usage, realized codes, aliasing, recon."""
|
| 114 |
+
u = (F.normalize(x, dim=-1) @ F.normalize(addr.codebook, dim=-1).T) / addr.tau
|
| 115 |
+
p, nneg = addr.oriented(x)
|
| 116 |
+
two_k = torch.cat([p, nneg], dim=-1)
|
| 117 |
+
win = two_k.argmax(dim=-1)
|
| 118 |
+
hist = torch.bincount(win, minlength=2 * addr.K).float()
|
| 119 |
+
ppl = float(torch.exp(-(hist / hist.sum() * (hist / hist.sum() + 1e-12)
|
| 120 |
+
.log()).sum()))
|
| 121 |
+
signs = (u > 0)
|
| 122 |
+
# realized full sign-codes via hashing
|
| 123 |
+
weights = (2 ** torch.arange(min(addr.K, 62), device=DEV))
|
| 124 |
+
code = (signs[:, :len(weights)].long() * weights).sum(-1)
|
| 125 |
+
n_codes = int(torch.unique(code).numel())
|
| 126 |
+
# aliasing: same full code -> how similar are inputs really?
|
| 127 |
+
order = torch.argsort(code)
|
| 128 |
+
cs, xs = code[order], x[order]
|
| 129 |
+
same = cs[1:] == cs[:-1]
|
| 130 |
+
if int(same.sum()) > 8:
|
| 131 |
+
i = torch.nonzero(same).squeeze(-1)
|
| 132 |
+
if i.numel() > pair_sample:
|
| 133 |
+
i = i[torch.randperm(i.numel(), device=DEV)[:pair_sample]]
|
| 134 |
+
alias_cos = float((xs[i] * xs[i + 1]).sum(-1).mean())
|
| 135 |
+
else:
|
| 136 |
+
alias_cos = float("nan")
|
| 137 |
+
m = addr.m_hat(x)
|
| 138 |
+
recon = float(F.cosine_similarity(m, x, dim=-1).mean())
|
| 139 |
+
win_cos = float((u.abs().amax(dim=-1) * addr.tau).mean())
|
| 140 |
+
return {"usage_ppl": round(ppl, 2), "n_codes": n_codes,
|
| 141 |
+
"codes_per_halfaxis_bits": round(math.log2(max(n_codes, 1))
|
| 142 |
+
/ (2 * addr.K), 4),
|
| 143 |
+
"alias_within_code_cos": round(alias_cos, 4)
|
| 144 |
+
if alias_cos == alias_cos else None,
|
| 145 |
+
"mhat_recon_cos": round(recon, 4),
|
| 146 |
+
"win_cos_mean": round(win_cos, 4)}
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def b1_capacity():
|
| 150 |
+
line("B1 CAPACITY — single-aleph curves")
|
| 151 |
+
Ks = [4, 8, 16, 32, 64] if QUICK else [4, 8, 16, 32, 64, 128]
|
| 152 |
+
Ds = [8, 32] if QUICK else [4, 8, 16, 32, 64]
|
| 153 |
+
taus = [0.1] if QUICK else [0.05, 0.1, 0.2]
|
| 154 |
+
fams = ["iso", "clust", "lowrank"]
|
| 155 |
+
rows = []
|
| 156 |
+
for D in Ds:
|
| 157 |
+
for fam in fams:
|
| 158 |
+
x = gen(fam, N, D, SEED0 + D)
|
| 159 |
+
for K in Ks:
|
| 160 |
+
for tau in taus:
|
| 161 |
+
torch.manual_seed(SEED0 + K * 7 + D)
|
| 162 |
+
a = AlephAddress(K, D, tau=tau).to(DEV)
|
| 163 |
+
r = read_stats(a, x)
|
| 164 |
+
r.update({"K": K, "D": D, "tau": tau, "family": fam})
|
| 165 |
+
rows.append(r)
|
| 166 |
+
print(f" D={D} done ({len(rows)} rows)")
|
| 167 |
+
return {"rows": rows, "n": N}
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
def _joint_stats(alephs, x):
|
| 171 |
+
"""Multi-aleph joint gauges."""
|
| 172 |
+
wins, reads = [], []
|
| 173 |
+
with torch.no_grad():
|
| 174 |
+
for a in alephs:
|
| 175 |
+
p, nneg = a.oriented(x)
|
| 176 |
+
wins.append(torch.cat([p, nneg], -1).argmax(-1))
|
| 177 |
+
reads.append(a.m_hat(x))
|
| 178 |
+
joint = wins[0].clone()
|
| 179 |
+
base = 1
|
| 180 |
+
for w in wins[1:]:
|
| 181 |
+
base *= 2 * alephs[0].K + 1
|
| 182 |
+
joint = joint * (2 * alephs[0].K) + w
|
| 183 |
+
n_joint = int(torch.unique(joint).numel())
|
| 184 |
+
# pairwise redundancy: normalized mutual information of winner streams
|
| 185 |
+
def nmi(a_, b_):
|
| 186 |
+
K2 = 2 * alephs[0].K
|
| 187 |
+
jointc = torch.bincount(a_ * K2 + b_, minlength=K2 * K2).float()
|
| 188 |
+
pj = jointc / jointc.sum()
|
| 189 |
+
pa = pj.reshape(K2, K2).sum(1)
|
| 190 |
+
pb = pj.reshape(K2, K2).sum(0)
|
| 191 |
+
eps = 1e-12
|
| 192 |
+
mi = (pj * (pj + eps).log()).sum() - (pa * (pa + eps).log()).sum() \
|
| 193 |
+
- (pb * (pb + eps).log()).sum()
|
| 194 |
+
ha = -(pa * (pa + eps).log()).sum()
|
| 195 |
+
hb = -(pb * (pb + eps).log()).sum()
|
| 196 |
+
return float(mi / max(float(torch.minimum(ha, hb)), 1e-9))
|
| 197 |
+
nmis = [nmi(wins[i], wins[j]) for i in range(len(wins))
|
| 198 |
+
for j in range(i + 1, len(wins))]
|
| 199 |
+
# joint readback: least-squares x_hat from concatenated m_hats
|
| 200 |
+
R = torch.cat(reads, dim=-1)
|
| 201 |
+
sol = torch.linalg.lstsq(R, x).solution
|
| 202 |
+
recon = float(F.cosine_similarity(R @ sol, x, dim=-1).mean())
|
| 203 |
+
return {"n_joint_codes": n_joint,
|
| 204 |
+
"mean_pairwise_nmi": round(float(np.mean(nmis)), 4) if nmis else None,
|
| 205 |
+
"joint_readback_cos": round(recon, 4)}
|
| 206 |
+
|
| 207 |
+
|
| 208 |
+
def _make_family(mode, M, K, D, seed):
|
| 209 |
+
"""Init families for M perspectives."""
|
| 210 |
+
torch.manual_seed(seed)
|
| 211 |
+
alephs = []
|
| 212 |
+
if mode == "copies":
|
| 213 |
+
base = AlephAddress(K, D).to(DEV)
|
| 214 |
+
for _ in range(M):
|
| 215 |
+
a = AlephAddress(K, D).to(DEV)
|
| 216 |
+
a.codebook.data.copy_(base.codebook.data)
|
| 217 |
+
a.home.copy_(base.codebook.data)
|
| 218 |
+
alephs.append(a)
|
| 219 |
+
elif mode == "rotated":
|
| 220 |
+
base = AlephAddress(K, D).to(DEV)
|
| 221 |
+
for m in range(M):
|
| 222 |
+
a = AlephAddress(K, D).to(DEV)
|
| 223 |
+
Q = torch.linalg.qr(torch.randn(D, D, device=DEV))[0]
|
| 224 |
+
a.codebook.data.copy_(F.normalize(base.codebook.data @ Q, dim=-1))
|
| 225 |
+
a.home.copy_(a.codebook.data)
|
| 226 |
+
alephs.append(a)
|
| 227 |
+
else: # independent
|
| 228 |
+
for m in range(M):
|
| 229 |
+
torch.manual_seed(seed + 977 * m)
|
| 230 |
+
alephs.append(AlephAddress(K, D).to(DEV))
|
| 231 |
+
return alephs
|
| 232 |
+
|
| 233 |
+
|
| 234 |
+
def b2_product():
|
| 235 |
+
line("B2 PRODUCT CODE — M differentiated frames at equal half-axes")
|
| 236 |
+
D = 32
|
| 237 |
+
budget = 64 # total K across alephs
|
| 238 |
+
splits = [(1, 64), (2, 32), (4, 16), (8, 8), (16, 4)]
|
| 239 |
+
fams = ["iso", "clust", "lowrank"]
|
| 240 |
+
rows = []
|
| 241 |
+
for fam in fams:
|
| 242 |
+
x = gen(fam, N, D, SEED0 + 11)
|
| 243 |
+
for M, K in splits:
|
| 244 |
+
for mode in ("independent", "rotated", "copies"):
|
| 245 |
+
alephs = _make_family(mode, M, K, D, SEED0 + M * 13 + K)
|
| 246 |
+
r = _joint_stats(alephs, x)
|
| 247 |
+
r.update({"M": M, "K": K, "mode": mode, "family": fam,
|
| 248 |
+
"bits_per_halfaxis": round(
|
| 249 |
+
math.log2(max(r["n_joint_codes"], 1))
|
| 250 |
+
/ (2 * budget), 4)})
|
| 251 |
+
rows.append(r)
|
| 252 |
+
print(f" family={fam} done")
|
| 253 |
+
return {"rows": rows, "budget_halfaxes": 2 * budget, "n": N}
|
| 254 |
+
|
| 255 |
+
|
| 256 |
+
def b3_dynamics():
|
| 257 |
+
line("B3 DIFFERENTIATION DYNAMICS — does distinctness persist?")
|
| 258 |
+
M, K, D, steps = 4, 16, 32, (300 if QUICK else 2000)
|
| 259 |
+
fams = {"clust": gen("clust", 50_000, D, SEED0 + 21)}
|
| 260 |
+
rows = []
|
| 261 |
+
for mode in ("independent", "copies"):
|
| 262 |
+
alephs = _make_family(mode, M, K, D, SEED0 + 31)
|
| 263 |
+
if mode == "copies": # tiny symmetric-break noise, realistic init
|
| 264 |
+
for a in alephs:
|
| 265 |
+
a.codebook.data.add_(0.01 * torch.randn_like(a.codebook))
|
| 266 |
+
params = [a.codebook for a in alephs]
|
| 267 |
+
for p in params:
|
| 268 |
+
p.requires_grad_(True)
|
| 269 |
+
opt = torch.optim.Adam(params, lr=1e-3) # pure Adam, house law
|
| 270 |
+
x = fams["clust"]
|
| 271 |
+
traj = []
|
| 272 |
+
def pdist():
|
| 273 |
+
ds = []
|
| 274 |
+
for i in range(M):
|
| 275 |
+
for j in range(i + 1, M):
|
| 276 |
+
A = F.normalize(alephs[i].codebook.detach(), dim=-1)
|
| 277 |
+
B = F.normalize(alephs[j].codebook.detach(), dim=-1)
|
| 278 |
+
Q = torch.linalg.qr(torch.randn(D, D, device=DEV))[0] * 0
|
| 279 |
+
# Procrustes-aligned distance
|
| 280 |
+
U, _, Vt = torch.linalg.svd(B.T @ A)
|
| 281 |
+
R = U @ Vt
|
| 282 |
+
ds.append(float(1 - (F.normalize(B @ R, dim=-1)
|
| 283 |
+
* A).sum(-1).mean()))
|
| 284 |
+
return float(np.mean(ds))
|
| 285 |
+
for s in range(steps):
|
| 286 |
+
i = torch.randint(0, x.shape[0], (4096,), device=DEV)
|
| 287 |
+
xb = x[i]
|
| 288 |
+
read = sum(a.m_hat(xb) for a in alephs) / M
|
| 289 |
+
loss = (1 - F.cosine_similarity(read, xb, dim=-1)).mean()
|
| 290 |
+
opt.zero_grad()
|
| 291 |
+
loss.backward()
|
| 292 |
+
opt.step()
|
| 293 |
+
if s % max(1, steps // 10) == 0:
|
| 294 |
+
traj.append({"step": s, "loss": round(float(loss), 4),
|
| 295 |
+
"pairwise_pdist": round(pdist(), 4)})
|
| 296 |
+
r_final = _joint_stats(alephs, x)
|
| 297 |
+
rows.append({"mode": mode, "traj": traj, "final": r_final,
|
| 298 |
+
"final_pdist": round(pdist(), 4)})
|
| 299 |
+
print(f" mode={mode}: final pdist {rows[-1]['final_pdist']}")
|
| 300 |
+
return {"rows": rows, "M": M, "K": K, "D": D, "steps": steps}
|
| 301 |
+
|
| 302 |
+
|
| 303 |
+
def b4_opinion():
|
| 304 |
+
line("B4 OPINION GATING — self-confidence vs softmax (falsification)")
|
| 305 |
+
M, K, D = 4, 16, 32
|
| 306 |
+
torch.manual_seed(SEED0 + 41)
|
| 307 |
+
alephs = _make_family("independent", M, K, D, SEED0 + 41)
|
| 308 |
+
# data with REAL perspective advantage: cluster c is built FROM frame c
|
| 309 |
+
n_per = (N // 4) // M * M
|
| 310 |
+
xs, owner = [], []
|
| 311 |
+
for m, a in enumerate(alephs):
|
| 312 |
+
cents = F.normalize(a.codebook.detach(), dim=-1)
|
| 313 |
+
idx = torch.randint(0, K, (n_per // M,), device=DEV)
|
| 314 |
+
xs.append(F.normalize(cents[idx] + 0.35 * torch.randn(
|
| 315 |
+
n_per // M, D, device=DEV), dim=-1))
|
| 316 |
+
owner.append(torch.full((n_per // M,), m, device=DEV))
|
| 317 |
+
x = torch.cat(xs)
|
| 318 |
+
owner = torch.cat(owner)
|
| 319 |
+
with torch.no_grad():
|
| 320 |
+
reads = [a.m_hat(x) for a in alephs]
|
| 321 |
+
confs = [(F.normalize(x, dim=-1) @ F.normalize(
|
| 322 |
+
a.codebook, dim=-1).T).abs().amax(-1) for a in alephs]
|
| 323 |
+
C = torch.stack(confs, dim=-1) # (n, M)
|
| 324 |
+
def compose(gates):
|
| 325 |
+
R = sum(g.unsqueeze(-1) * r for g, r in zip(gates.unbind(-1), reads))
|
| 326 |
+
return float(F.cosine_similarity(R, x, dim=-1).mean())
|
| 327 |
+
results = {}
|
| 328 |
+
results["uniform"] = compose(torch.ones_like(C) / M)
|
| 329 |
+
results["self_confidence"] = compose(C / C.sum(-1, keepdim=True).clamp(min=1e-9))
|
| 330 |
+
results["softmax_comparative_CONTROL"] = compose(torch.softmax(C / 0.05, dim=-1))
|
| 331 |
+
oracle = F.one_hot(owner, M).float()
|
| 332 |
+
results["oracle"] = compose(oracle)
|
| 333 |
+
# accuracy of implied selection vs owner (diagnostic, not a selector)
|
| 334 |
+
results["self_conf_top1_matches_owner"] = float(
|
| 335 |
+
(C.argmax(-1) == owner).float().mean())
|
| 336 |
+
print(f" {results}")
|
| 337 |
+
return {k: round(v, 4) for k, v in results.items()} | {"M": M, "K": K, "D": D}
|
| 338 |
+
|
| 339 |
+
|
| 340 |
+
def b5_blackboard():
|
| 341 |
+
line("B5 BLACKBOARD — associative recall capacity through the address")
|
| 342 |
+
D = 32
|
| 343 |
+
Ks = [8, 16, 32, 64] if QUICK else [8, 16, 32, 64, 128]
|
| 344 |
+
Ps = [4, 8, 16, 32, 64, 128, 256, 512]
|
| 345 |
+
rows = []
|
| 346 |
+
for K in Ks:
|
| 347 |
+
torch.manual_seed(SEED0 + 51 + K)
|
| 348 |
+
single = AlephAddress(K, D).to(DEV)
|
| 349 |
+
multi = _make_family("independent", 4, max(K // 4, 2), D, SEED0 + 53 + K)
|
| 350 |
+
for tag, addrs in (("single", [single]), ("M4_product", multi)):
|
| 351 |
+
for P in Ps:
|
| 352 |
+
torch.manual_seed(SEED0 + P)
|
| 353 |
+
keys = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
|
| 354 |
+
vals = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
|
| 355 |
+
with torch.no_grad():
|
| 356 |
+
w = torch.cat([a.signed(keys) for a in addrs], -1) # (P, sumK)
|
| 357 |
+
cells = w.T @ vals # (sumK, D)
|
| 358 |
+
q = F.normalize(keys + 0.05 * torch.randn_like(keys), dim=-1)
|
| 359 |
+
wq = torch.cat([a.signed(q) for a in addrs], -1)
|
| 360 |
+
out = wq @ cells
|
| 361 |
+
rec = float(F.cosine_similarity(out, vals, dim=-1).mean())
|
| 362 |
+
rows.append({"K_total": sum(a.K for a in addrs), "form": tag,
|
| 363 |
+
"P": P, "recall_cos": round(rec, 4)})
|
| 364 |
+
pmax_s = max([r["P"] for r in rows if r["form"] == "single"
|
| 365 |
+
and r["K_total"] == K and r["recall_cos"] >= 0.7],
|
| 366 |
+
default=0)
|
| 367 |
+
print(f" K={K}: single P_max(cos>=.7) ~ {pmax_s}")
|
| 368 |
+
return {"rows": rows, "noise": 0.05, "D": D}
|
| 369 |
+
|
| 370 |
+
|
| 371 |
+
def b6_boundary():
|
| 372 |
+
line("B6 BOUNDARY SOLIDITY — overlap ratio vs solid relay distance")
|
| 373 |
+
D, K, L = 32, 16, (512 if QUICK else 4096)
|
| 374 |
+
spacings = [64, 128, 256]
|
| 375 |
+
ratios = [0.0, 0.25, 0.5, 1.0, 2.0] # sigma / spacing; 0 = tiling
|
| 376 |
+
torch.manual_seed(SEED0 + 61)
|
| 377 |
+
msg = F.normalize(torch.randn(D, device=DEV), dim=-1)
|
| 378 |
+
rows = []
|
| 379 |
+
for s in spacings:
|
| 380 |
+
centers = torch.arange(0, L, s, device=DEV).float()
|
| 381 |
+
for ratio in ratios:
|
| 382 |
+
sig = max(ratio * s, 1e-6)
|
| 383 |
+
pos = torch.arange(L, device=DEV).float()
|
| 384 |
+
if ratio == 0.0: # hard tiling membership
|
| 385 |
+
gm = (pos.unsqueeze(0) - centers.unsqueeze(1)).abs() < (s / 2)
|
| 386 |
+
gm = gm.float()
|
| 387 |
+
else:
|
| 388 |
+
gm = torch.exp(-0.5 * ((pos.unsqueeze(0) - centers.unsqueeze(1))
|
| 389 |
+
/ sig) ** 2)
|
| 390 |
+
gm = gm / gm.sum(0, keepdim=True).clamp(min=1e-9) # local support
|
| 391 |
+
# relay: message injected at pos0 propagates via shared membership
|
| 392 |
+
# solidity(d) = sum_m g_m(0) g_m(d) / sqrt(sum g_m(0)^2 sum g_m(d)^2)
|
| 393 |
+
g0 = gm[:, 0]
|
| 394 |
+
sol = (gm.T @ g0) / (gm.norm(dim=0) * g0.norm()).clamp(min=1e-9)
|
| 395 |
+
# chained relay: two-hop through any intermediate position
|
| 396 |
+
Gn = gm / gm.norm(dim=0, keepdim=True).clamp(min=1e-9)
|
| 397 |
+
A1 = Gn.T @ Gn # position-position solidity
|
| 398 |
+
A2 = A1 @ A1 / A1.shape[0]
|
| 399 |
+
def maxd(v, thr=0.1):
|
| 400 |
+
idx = torch.nonzero(v > thr).squeeze(-1)
|
| 401 |
+
return int(idx.max()) if idx.numel() else 0
|
| 402 |
+
rows.append({"spacing": s, "overlap_ratio": ratio,
|
| 403 |
+
"solid_1hop_maxd": maxd(sol),
|
| 404 |
+
"solid_2hop_maxd": maxd(A2[0]),
|
| 405 |
+
"edge_min_1hop": round(float(sol[:2 * s].min()), 4)})
|
| 406 |
+
print(f" spacing={s} done")
|
| 407 |
+
return {"rows": rows, "L": L, "note": "ratio 0 = hard tiling control; "
|
| 408 |
+
"edge_min near 0 at ratio 0 must reproduce the deep-line "
|
| 409 |
+
"tile-edge deadness"}
|
| 410 |
+
|
| 411 |
+
|
| 412 |
+
|
| 413 |
+
|
| 414 |
+
def b7_mscaling():
|
| 415 |
+
line("B7 M-SCALING — thousands of tiny heads vs data rank")
|
| 416 |
+
D, K = 64, 4
|
| 417 |
+
Ms = [16, 64, 256, 1024, 2048] if not QUICK else [16, 64, 256]
|
| 418 |
+
ranks = [8, 32, 64] # 64 = full
|
| 419 |
+
n = 100_000 if not QUICK else 20_000
|
| 420 |
+
rows = []
|
| 421 |
+
for r in ranks:
|
| 422 |
+
x = gen("lowrank", n, D, SEED0 + 71 + r, r_lowrank=r)
|
| 423 |
+
n_tr = int(0.75 * x.shape[0])
|
| 424 |
+
for M in Ms:
|
| 425 |
+
alephs = _make_family("independent", M, K, D, SEED0 + 73 + M)
|
| 426 |
+
# HELD-OUT streaming linear probe: at large M the lstsq gauge
|
| 427 |
+
# both OOMs (100k x 65k fp32 = 26GB, measured crash) and
|
| 428 |
+
# degenerates (features > rows -> interpolation, trivial 1.0).
|
| 429 |
+
# Probe weights are O(M*K*D) only; reads computed per batch.
|
| 430 |
+
feat_dim = sum(a.K for a in alephs)
|
| 431 |
+
W = torch.zeros(feat_dim, D, device=DEV, requires_grad=True)
|
| 432 |
+
opt = torch.optim.Adam([W], lr=3e-3)
|
| 433 |
+
def reads_of(xb):
|
| 434 |
+
with torch.no_grad():
|
| 435 |
+
return torch.cat([a.signed(xb) for a in alephs], -1)
|
| 436 |
+
for ep in range(3):
|
| 437 |
+
perm = torch.randperm(n_tr, device=DEV)
|
| 438 |
+
for i in range(0, n_tr, 8192):
|
| 439 |
+
xb = x[perm[i:i + 8192]]
|
| 440 |
+
pred = reads_of(xb) @ W
|
| 441 |
+
loss = (1 - F.cosine_similarity(pred, xb, dim=-1)).mean()
|
| 442 |
+
opt.zero_grad(); loss.backward(); opt.step()
|
| 443 |
+
with torch.no_grad():
|
| 444 |
+
rec_parts = []
|
| 445 |
+
for i in range(n_tr, x.shape[0], 8192):
|
| 446 |
+
xb = x[i:i + 8192]
|
| 447 |
+
rec_parts.append(F.cosine_similarity(
|
| 448 |
+
reads_of(xb) @ W, xb, dim=-1))
|
| 449 |
+
rec = float(torch.cat(rec_parts).mean())
|
| 450 |
+
p1, n1 = alephs[0].oriented(x[:50000])
|
| 451 |
+
p2, n2 = alephs[1].oriented(x[:50000])
|
| 452 |
+
w1 = torch.cat([p1, n1], -1).argmax(-1)
|
| 453 |
+
w2 = torch.cat([p2, n2], -1).argmax(-1)
|
| 454 |
+
K2 = 2 * K
|
| 455 |
+
jc = torch.bincount(w1 * K2 + w2, minlength=K2 * K2).float()
|
| 456 |
+
pj = jc / jc.sum()
|
| 457 |
+
pa = pj.reshape(K2, K2).sum(1); pb = pj.reshape(K2, K2).sum(0)
|
| 458 |
+
eps = 1e-12
|
| 459 |
+
mi = float((pj * (pj + eps).log()).sum()
|
| 460 |
+
- (pa * (pa + eps).log()).sum()
|
| 461 |
+
- (pb * (pb + eps).log()).sum())
|
| 462 |
+
rows.append({"rank": r, "M": M, "half_axes": 2 * K * M,
|
| 463 |
+
"readback_heldout": round(rec, 4),
|
| 464 |
+
"pair_mi_nats": round(mi, 4)})
|
| 465 |
+
print(f" rank={r} M={M}: readback {rec:.4f}")
|
| 466 |
+
return {"rows": rows, "K": K, "D": D, "n": n}
|
| 467 |
+
|
| 468 |
+
|
| 469 |
+
def b8_headscale():
|
| 470 |
+
line("B8 HEAD-SCALE BLACKBOARD — context recall under splat sharding")
|
| 471 |
+
D, K = 64, 4
|
| 472 |
+
Ns = [2048, 8192] if not QUICK else [512]
|
| 473 |
+
configs = [("monolith", 1, 2048), ("M512", 512, None), ("M2048", 2048, None)]
|
| 474 |
+
rows = []
|
| 475 |
+
for N_ctx in Ns:
|
| 476 |
+
torch.manual_seed(SEED0 + 81 + N_ctx)
|
| 477 |
+
keys = F.normalize(torch.randn(N_ctx, D, device=DEV), dim=-1)
|
| 478 |
+
vals = F.normalize(torch.randn(N_ctx, D, device=DEV), dim=-1)
|
| 479 |
+
q = F.normalize(keys + 0.05 * torch.randn_like(keys), dim=-1)
|
| 480 |
+
for tag, M, Kmono in configs:
|
| 481 |
+
Ke = Kmono if Kmono else K
|
| 482 |
+
alephs = _make_family("independent", M, Ke, D, SEED0 + 83 + M)
|
| 483 |
+
t0 = time.time()
|
| 484 |
+
with torch.no_grad():
|
| 485 |
+
if M == 1:
|
| 486 |
+
w = alephs[0].signed(keys); wq = alephs[0].signed(q)
|
| 487 |
+
cells = w.T @ vals
|
| 488 |
+
out = wq @ cells
|
| 489 |
+
else:
|
| 490 |
+
# splat sharding: head m owns window centered at
|
| 491 |
+
# m*N/M, width sigma = 1.5 * spacing (B6 band)
|
| 492 |
+
pos = torch.arange(N_ctx, device=DEV).float()
|
| 493 |
+
centers = torch.linspace(0, N_ctx, M, device=DEV)
|
| 494 |
+
sig = 1.5 * (N_ctx / M)
|
| 495 |
+
gm = torch.exp(-0.5 * ((pos.unsqueeze(0)
|
| 496 |
+
- centers.unsqueeze(1)) / sig) ** 2)
|
| 497 |
+
gm = gm / gm.sum(0, keepdim=True).clamp(min=1e-9)
|
| 498 |
+
out = torch.zeros_like(vals)
|
| 499 |
+
for m, a in enumerate(alephs):
|
| 500 |
+
g = gm[m].unsqueeze(-1)
|
| 501 |
+
w = a.signed(keys) * g
|
| 502 |
+
cells = w.T @ vals
|
| 503 |
+
out = out + (a.signed(q) * g) @ cells
|
| 504 |
+
wall = time.time() - t0
|
| 505 |
+
rec = float(F.cosine_similarity(out, vals, dim=-1).mean())
|
| 506 |
+
# sharp-retrieval comparison: cosine top-1 accuracy
|
| 507 |
+
top1 = float((F.normalize(out, dim=-1) @ vals.T)
|
| 508 |
+
.argmax(-1).eq(torch.arange(N_ctx, device=DEV))
|
| 509 |
+
.float().mean())
|
| 510 |
+
rows.append({"N_ctx": N_ctx, "form": tag, "M": M,
|
| 511 |
+
"cells_total": sum(a.K for a in alephs) * 2,
|
| 512 |
+
"recall_cos": round(rec, 4),
|
| 513 |
+
"top1_acc": round(top1, 4),
|
| 514 |
+
"wall_s": round(wall, 3)})
|
| 515 |
+
print(f" N={N_ctx} {tag}: recall {rec:.4f} top1 {top1:.4f} "
|
| 516 |
+
f"({wall:.2f}s)")
|
| 517 |
+
return {"rows": rows, "note": "splat sharding keeps per-board load "
|
| 518 |
+
"~N/M*overlap; monolith board must hold all N"}
|
| 519 |
+
|
| 520 |
+
|
| 521 |
+
|
| 522 |
+
|
| 523 |
+
def _rope(x, pos, base=10000.0):
|
| 524 |
+
"""RoPE rotation applied to the ADDRESS QUERY: pairs (2i,2i+1) rotate
|
| 525 |
+
by pos*theta_i. Position enters the aleph through the addressing
|
| 526 |
+
itself — relative structure R(i-j) in every write/read affinity."""
|
| 527 |
+
D = x.shape[-1]
|
| 528 |
+
half = D // 2
|
| 529 |
+
freqs = base ** (-torch.arange(half, device=x.device,
|
| 530 |
+
dtype=torch.float32) / half)
|
| 531 |
+
ang = pos.unsqueeze(-1) * freqs # (..., half)
|
| 532 |
+
c, sn = torch.cos(ang), torch.sin(ang)
|
| 533 |
+
x1, x2 = x[..., :half], x[..., half:]
|
| 534 |
+
return torch.cat([x1 * c - x2 * sn, x1 * sn + x2 * c], dim=-1)
|
| 535 |
+
|
| 536 |
+
|
| 537 |
+
def b9_rotary():
|
| 538 |
+
line("B9 CODEBOOK ROTARY — position in the address, accuracy probe")
|
| 539 |
+
D, K, M = 64, 8, 16
|
| 540 |
+
torch.manual_seed(SEED0 + 91)
|
| 541 |
+
alephs = _make_family("independent", M, K, D, SEED0 + 91)
|
| 542 |
+
def addr(x, pos):
|
| 543 |
+
xr = F.normalize(_rope(F.normalize(x, dim=-1), pos), dim=-1)
|
| 544 |
+
return torch.cat([a.signed(xr) for a in alephs], -1)
|
| 545 |
+
out = {}
|
| 546 |
+
# (a) relative-offset structure: same content, positions 0..256 —
|
| 547 |
+
# address cos vs delta (the interference curve; RoPE-like decay)
|
| 548 |
+
x0 = F.normalize(torch.randn(1, D, device=DEV), dim=-1).repeat(257, 1)
|
| 549 |
+
pos = torch.arange(257, device=DEV, dtype=torch.float32)
|
| 550 |
+
w = addr(x0, pos)
|
| 551 |
+
ref = w[0:1]
|
| 552 |
+
cosd = F.cosine_similarity(w, ref.expand_as(w), dim=-1)
|
| 553 |
+
out["addr_cos_vs_delta"] = {str(d): round(float(cosd[d]), 4)
|
| 554 |
+
for d in (0, 1, 2, 4, 8, 16, 32, 64,
|
| 555 |
+
128, 256)}
|
| 556 |
+
# (b) position discriminability: 1000 contents x 2 positions 32 apart
|
| 557 |
+
xc = F.normalize(torch.randn(1000, D, device=DEV), dim=-1)
|
| 558 |
+
wa = addr(xc, torch.full((1000,), 10.0, device=DEV))
|
| 559 |
+
wb = addr(xc, torch.full((1000,), 42.0, device=DEV))
|
| 560 |
+
out["same_content_dpos32_addr_cos"] = round(
|
| 561 |
+
float(F.cosine_similarity(wa, wb, dim=-1).mean()), 4)
|
| 562 |
+
# (c) recall under position shift: store P pairs at their positions,
|
| 563 |
+
# query same content at shifted positions
|
| 564 |
+
P = 64
|
| 565 |
+
keys = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
|
| 566 |
+
vals = F.normalize(torch.randn(P, D, device=DEV), dim=-1)
|
| 567 |
+
kpos = torch.arange(P, device=DEV, dtype=torch.float32) * 4
|
| 568 |
+
wk = addr(keys, kpos)
|
| 569 |
+
cells = wk.T @ vals
|
| 570 |
+
rec_by_shift = {}
|
| 571 |
+
for shift in (0, 1, 2, 4, 8, 16, 64):
|
| 572 |
+
wq = addr(keys, kpos + shift)
|
| 573 |
+
got = wq @ cells
|
| 574 |
+
rec_by_shift[str(shift)] = round(float(
|
| 575 |
+
F.cosine_similarity(got, vals, dim=-1).mean()), 4)
|
| 576 |
+
out["recall_vs_pos_shift"] = rec_by_shift
|
| 577 |
+
# (d) TRANSPORT at short L — the collapse-fix test: token at pos 0
|
| 578 |
+
# reads content written at pos p, GLOBAL rotary heads (no windows)
|
| 579 |
+
L = 128
|
| 580 |
+
toks = F.normalize(torch.randn(L, D, device=DEV), dim=-1)
|
| 581 |
+
tpos = torch.arange(L, device=DEV, dtype=torch.float32)
|
| 582 |
+
wt = addr(toks, tpos)
|
| 583 |
+
cellsT = wt.T @ toks # every token written globally
|
| 584 |
+
probe_reads = wt @ cellsT # each reads the shared board
|
| 585 |
+
self_rec = float(F.cosine_similarity(probe_reads, toks, dim=-1).mean())
|
| 586 |
+
# cross-read: query content of token j FROM position 0's frame
|
| 587 |
+
w0 = addr(toks, torch.zeros(L, device=DEV))
|
| 588 |
+
cross = w0 @ cellsT
|
| 589 |
+
cross_rec = float(F.cosine_similarity(cross, toks, dim=-1).mean())
|
| 590 |
+
out["shortL_transport"] = {"self_recall": round(self_rec, 4),
|
| 591 |
+
"cross_pos0_recall": round(cross_rec, 4),
|
| 592 |
+
"note": "cross>0.3 = content reachable "
|
| 593 |
+
"across positions (the a4 collapse cure)"}
|
| 594 |
+
# (e) sign-code stability under rotation
|
| 595 |
+
win = (F.normalize(_rope(F.normalize(xc, dim=-1),
|
| 596 |
+
torch.full((1000,), 100.0, device=DEV)),
|
| 597 |
+
dim=-1) @ F.normalize(alephs[0].codebook,
|
| 598 |
+
dim=-1).T).abs().amax(-1)
|
| 599 |
+
out["win_cos_after_rotation"] = round(float(win.mean()), 4)
|
| 600 |
+
print(f" {json.dumps(out, indent=1)[:600]}")
|
| 601 |
+
return out
|
| 602 |
+
|
| 603 |
+
|
| 604 |
+
def ship(results):
|
| 605 |
+
os.makedirs(OUT_DIR, exist_ok=True)
|
| 606 |
+
tag = "" if ARM == "ALL" else f"_{ARM}"
|
| 607 |
+
path = f"{OUT_DIR}/splat_battery{tag}.json"
|
| 608 |
+
json.dump(results, open(path, "w"), indent=1)
|
| 609 |
+
print(f"\n wrote {path}")
|
| 610 |
+
tok = os.environ.get("HF_TOKEN")
|
| 611 |
+
if tok:
|
| 612 |
+
try:
|
| 613 |
+
from huggingface_hub import HfApi, create_repo
|
| 614 |
+
create_repo(HUB, token=tok, exist_ok=True, private=True)
|
| 615 |
+
HfApi(token=tok).upload_file(
|
| 616 |
+
path_or_fileobj=path, path_in_repo=f"battery/splat_battery{tag}.json",
|
| 617 |
+
repo_id=HUB, commit_message="B1-B6 math battery")
|
| 618 |
+
print(f" shipped -> {HUB}/battery/")
|
| 619 |
+
except Exception as e:
|
| 620 |
+
print(f" ship failed: {str(e)[:80]} (local copy kept)")
|
| 621 |
+
|
| 622 |
+
|
| 623 |
+
def main():
|
| 624 |
+
print("=" * 78)
|
| 625 |
+
print(f"SPLAT MATH BATTERY — arm={ARM} quick={QUICK} dev={DEV} N={N}")
|
| 626 |
+
print("=" * 78)
|
| 627 |
+
t0 = time.time()
|
| 628 |
+
arms = {"B1": b1_capacity, "B2": b2_product, "B3": b3_dynamics,
|
| 629 |
+
"B4": b4_opinion, "B5": b5_blackboard, "B6": b6_boundary,
|
| 630 |
+
"B7": b7_mscaling, "B8": b8_headscale, "B9": b9_rotary}
|
| 631 |
+
todo = arms if ARM == "ALL" else {ARM: arms[ARM]}
|
| 632 |
+
results = {"config": {"quick": QUICK, "N": N, "seed": SEED0,
|
| 633 |
+
"device": DEV}}
|
| 634 |
+
for k, fn in todo.items():
|
| 635 |
+
t = time.time()
|
| 636 |
+
results[k] = fn()
|
| 637 |
+
results[k]["wall_s"] = round(time.time() - t, 1)
|
| 638 |
+
print(f"\nTOTAL {time.time()-t0:.0f}s")
|
| 639 |
+
ship(results)
|
| 640 |
+
return results
|
| 641 |
+
|
| 642 |
+
|
| 643 |
+
if __name__ == "__main__" or "get_ipython" in globals():
|
| 644 |
+
RESULTS = main()
|
trainers/trigram_bitspace_sweep.py
ADDED
|
@@ -0,0 +1,194 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# trigram_bitspace_sweep.py — AbstractPhil's unverified hypothesis (2026-08-09):
|
| 3 |
+
# "the bytemaps were forming inherently bit linked positioning causing
|
| 4 |
+
# byte differentiated pooled accumulation rather than using the mechanism
|
| 5 |
+
# ... the overlap bytes were causing anchored learning ... and in the
|
| 6 |
+
# process the accumulation formed the very alephs we've been studying."
|
| 7 |
+
#
|
| 8 |
+
# THREE MEASURABLE CLAIMS, each with a shuffle null (CPU-only sweep):
|
| 9 |
+
# A) BIT-LINKAGE: byte-embedding cosine similarity correlates with byte
|
| 10 |
+
# BIT overlap (8 - Hamming). Null: byte-label shuffles (z-score).
|
| 11 |
+
# B) ALEPH EMERGENCE: the embedding table factors as a SIGN CODE over
|
| 12 |
+
# the 8 bit-planes — fraction of (centered) table variance explained
|
| 13 |
+
# by linear regression on the 8 bit sign-vectors (a bit-plane basis =
|
| 14 |
+
# an 8-half-axis codebook). Null: shuffled bit labels.
|
| 15 |
+
# C) POOLED ACCUMULATION vs MECHANISM: the three trigram tables converge
|
| 16 |
+
# toward one another (row-matched cosine + linear CKA). Pooled sum ->
|
| 17 |
+
# tables similar; positional mechanism -> differentiated.
|
| 18 |
+
# D) DATA-SIDE (current bed): V1b key bit-collision spectra per R_key —
|
| 19 |
+
# ties the overlap-space hypothesis to the measured mis-selections.
|
| 20 |
+
# SPECIMENS: wikitext tri witness addr_mslh64_tri_s0_t2000.pt; unigram
|
| 21 |
+
# sdpa_s0_t8000.pt comparator; fresh-init tri null. ASCII CONFOUND noted:
|
| 22 |
+
# on wikitext, bit 0x20 IS the case bit — bit-structure is semantically
|
| 23 |
+
# real in the data, which is itself part of AbstractPhil's "inherent" claim.
|
| 24 |
+
# RUN: python tools\trigram_bitspace_sweep.py (CPU, ~1 min)
|
| 25 |
+
# =========================================================================
|
| 26 |
+
|
| 27 |
+
import importlib
|
| 28 |
+
import json
|
| 29 |
+
import os
|
| 30 |
+
import sys
|
| 31 |
+
|
| 32 |
+
import torch
|
| 33 |
+
import torch.nn.functional as F
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _load(name):
|
| 37 |
+
if name in sys.modules:
|
| 38 |
+
return sys.modules[name]
|
| 39 |
+
try:
|
| 40 |
+
return importlib.import_module(name)
|
| 41 |
+
except ImportError:
|
| 42 |
+
here = globals().get("__file__")
|
| 43 |
+
if here is not None:
|
| 44 |
+
import pathlib
|
| 45 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 46 |
+
return importlib.import_module(name)
|
| 47 |
+
raise
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
CKPT_DIR = "./data/ar_ckpts"
|
| 51 |
+
OUT = "./data/v1_rank_army/trigram_bitspace_sweep.json"
|
| 52 |
+
torch.manual_seed(0)
|
| 53 |
+
|
| 54 |
+
BITS = torch.tensor([[(b >> i) & 1 for i in range(8)]
|
| 55 |
+
for b in range(256)], dtype=torch.float64) * 2 - 1
|
| 56 |
+
HAM = (BITS.unsqueeze(0) * BITS.unsqueeze(1)).sum(-1) # (256,256) in [-8,8]
|
| 57 |
+
OFF = ~torch.eye(256, dtype=torch.bool)
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def bit_linkage(W, n_null=500):
|
| 61 |
+
"""A) corr(cos-sim, bit-overlap) + shuffle-null z."""
|
| 62 |
+
X = F.normalize(W.double(), dim=-1)
|
| 63 |
+
S = (X @ X.T)[OFF]
|
| 64 |
+
H = HAM[OFF]
|
| 65 |
+
r = float(torch.corrcoef(torch.stack([S, H]))[0, 1])
|
| 66 |
+
nulls = []
|
| 67 |
+
for _ in range(n_null):
|
| 68 |
+
p = torch.randperm(256)
|
| 69 |
+
nulls.append(float(torch.corrcoef(
|
| 70 |
+
torch.stack([S, HAM[p][:, p][OFF]]))[0, 1]))
|
| 71 |
+
nt = torch.tensor(nulls)
|
| 72 |
+
return {"r": round(r, 4),
|
| 73 |
+
"null_mean": round(float(nt.mean()), 4),
|
| 74 |
+
"z": round((r - float(nt.mean())) / max(float(nt.std()), 1e-9), 1)}
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def bitplane_variance(W, n_null=200):
|
| 78 |
+
"""B) fraction of centered-table variance explained by the 8 bit
|
| 79 |
+
sign-vectors (linear regression, closed form)."""
|
| 80 |
+
Wc = W.double() - W.double().mean(0, keepdim=True)
|
| 81 |
+
B = BITS # (256, 8)
|
| 82 |
+
coef = torch.linalg.lstsq(B, Wc).solution # (8, d)
|
| 83 |
+
resid = Wc - B @ coef
|
| 84 |
+
ev = 1 - float((resid ** 2).sum() / (Wc ** 2).sum())
|
| 85 |
+
nulls = []
|
| 86 |
+
for _ in range(n_null):
|
| 87 |
+
Bp = BITS[torch.randperm(256)]
|
| 88 |
+
c = torch.linalg.lstsq(Bp, Wc).solution
|
| 89 |
+
nulls.append(1 - float(((Wc - Bp @ c) ** 2).sum()
|
| 90 |
+
/ (Wc ** 2).sum()))
|
| 91 |
+
nt = torch.tensor(nulls)
|
| 92 |
+
return {"explained_var": round(ev, 4),
|
| 93 |
+
"null_mean": round(float(nt.mean()), 4),
|
| 94 |
+
"z": round((ev - float(nt.mean())) / max(float(nt.std()), 1e-9),
|
| 95 |
+
1)}
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def cka(A, B):
|
| 99 |
+
A = A.double() - A.double().mean(0, keepdim=True)
|
| 100 |
+
B = B.double() - B.double().mean(0, keepdim=True)
|
| 101 |
+
num = torch.norm(A.T @ B, "fro") ** 2
|
| 102 |
+
den = torch.norm(A.T @ A, "fro") * torch.norm(B.T @ B, "fro")
|
| 103 |
+
return round(float(num / den.clamp_min(1e-12)), 4)
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def pooled_vs_mechanism(sd):
|
| 107 |
+
"""C) inter-table alignment for tri specimens."""
|
| 108 |
+
e0, e1, e2 = (sd["emb.weight"][:256], sd["emb1.weight"][:256],
|
| 109 |
+
sd["emb2.weight"][:256])
|
| 110 |
+
rowcos = lambda A, B: round(float(F.cosine_similarity(
|
| 111 |
+
A.double(), B.double(), dim=-1).mean()), 4)
|
| 112 |
+
return {"rowcos_01": rowcos(e0, e1), "rowcos_02": rowcos(e0, e2),
|
| 113 |
+
"rowcos_12": rowcos(e1, e2),
|
| 114 |
+
"cka_01": cka(e0, e1), "cka_02": cka(e0, e2),
|
| 115 |
+
"cka_12": cka(e1, e2)}
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def analyze(name, sd):
|
| 119 |
+
out = {"specimen": name, "tables": {}}
|
| 120 |
+
for key in ("emb.weight", "emb1.weight", "emb2.weight"):
|
| 121 |
+
if key in sd:
|
| 122 |
+
W = sd[key][:256]
|
| 123 |
+
out["tables"][key] = {"bit_linkage": bit_linkage(W),
|
| 124 |
+
"bitplane": bitplane_variance(W)}
|
| 125 |
+
if "emb1.weight" in sd:
|
| 126 |
+
out["pooled_vs_mechanism"] = pooled_vs_mechanism(sd)
|
| 127 |
+
return out
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def keyspace_collisions():
|
| 131 |
+
"""D) V1b key bit-collision spectra per R_key."""
|
| 132 |
+
v1b = _load("v1b_recall_army_battery")
|
| 133 |
+
res = {}
|
| 134 |
+
for R in (4, 16, 64):
|
| 135 |
+
sg = torch.Generator().manual_seed(40_000 + R)
|
| 136 |
+
U = torch.randn(16, R, generator=sg)
|
| 137 |
+
g = torch.Generator().manual_seed(99_000 + R)
|
| 138 |
+
z = torch.randn(4096, R, generator=g)
|
| 139 |
+
bits = ((z @ U.T) > 0).double() * 2 - 1 # (4096, 16)
|
| 140 |
+
ham16 = (bits @ bits.T) # agreement in [-16,16]
|
| 141 |
+
off = ~torch.eye(4096, dtype=torch.bool)
|
| 142 |
+
agree = (ham16[off] + 16) / 2 # shared bits 0..16
|
| 143 |
+
res[f"R{R}"] = {
|
| 144 |
+
"mean_shared_bits": round(float(agree.mean()), 2),
|
| 145 |
+
"frac_ge14": round(float((agree >= 14).float().mean()), 5),
|
| 146 |
+
"frac_exact": round(float((agree >= 16).float().mean()), 6)}
|
| 147 |
+
return res
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
def main():
|
| 151 |
+
print("=" * 74)
|
| 152 |
+
print("TRIGRAM BIT-SPACE SWEEP (CPU) — AbstractPhil's pooled-accumulation "
|
| 153 |
+
"hypothesis")
|
| 154 |
+
print("=" * 74)
|
| 155 |
+
results = {"claims": {}}
|
| 156 |
+
specs = []
|
| 157 |
+
p = os.path.join(CKPT_DIR, "addr_mslh64_tri_s0_t2000.pt")
|
| 158 |
+
if os.path.exists(p):
|
| 159 |
+
specs.append(("wikitext_tri_witness",
|
| 160 |
+
torch.load(p, map_location="cpu",
|
| 161 |
+
weights_only=True)["state_dict"]))
|
| 162 |
+
p2 = os.path.join(CKPT_DIR, "sdpa_s0_t8000.pt")
|
| 163 |
+
if os.path.exists(p2):
|
| 164 |
+
specs.append(("wikitext_unigram_sdpa",
|
| 165 |
+
torch.load(p2, map_location="cpu",
|
| 166 |
+
weights_only=True)["state_dict"]))
|
| 167 |
+
bed = _load("ar_differentiation_bed")
|
| 168 |
+
torch.manual_seed(0)
|
| 169 |
+
fresh = bed.ByteLM("sdpa_tri", block=256).state_dict()
|
| 170 |
+
specs.append(("fresh_init_null", fresh))
|
| 171 |
+
for name, sd in specs:
|
| 172 |
+
r = analyze(name, sd)
|
| 173 |
+
results[name] = r
|
| 174 |
+
print(f"\n== {name}")
|
| 175 |
+
for tk, tv in r["tables"].items():
|
| 176 |
+
print(f" {tk}: bit-linkage r={tv['bit_linkage']['r']} "
|
| 177 |
+
f"(z={tv['bit_linkage']['z']}) bitplane EV="
|
| 178 |
+
f"{tv['bitplane']['explained_var']} "
|
| 179 |
+
f"(null {tv['bitplane']['null_mean']}, "
|
| 180 |
+
f"z={tv['bitplane']['z']})")
|
| 181 |
+
if "pooled_vs_mechanism" in r:
|
| 182 |
+
pm = r["pooled_vs_mechanism"]
|
| 183 |
+
print(f" tables 0-1/0-2/1-2: rowcos {pm['rowcos_01']}/"
|
| 184 |
+
f"{pm['rowcos_02']}/{pm['rowcos_12']} CKA "
|
| 185 |
+
f"{pm['cka_01']}/{pm['cka_02']}/{pm['cka_12']}")
|
| 186 |
+
results["keyspace_collisions"] = keyspace_collisions()
|
| 187 |
+
print("\n== V1b key bit-collision spectra:",
|
| 188 |
+
json.dumps(results["keyspace_collisions"]))
|
| 189 |
+
json.dump(results, open(OUT, "w"), indent=1)
|
| 190 |
+
print(f"\nsaved -> {OUT}")
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
if __name__ == "__main__":
|
| 194 |
+
main()
|
trainers/v1_rank_army_battery.py
ADDED
|
@@ -0,0 +1,241 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# v1_rank_army_battery.py — V1: rank-ladder x army-ladder under AR pressure
|
| 3 |
+
# (plan 2026-08-08_data_shaped_army.md, validation V1 — THE arbiter for
|
| 4 |
+
# Rule 1 "M = c*R"; overnight battery, AbstractPhil 2026-08-08: "set up a bunch of
|
| 5 |
+
# small battery tests and feed them with synthetic data, run them on the
|
| 6 |
+
# 4090 overnight").
|
| 7 |
+
#
|
| 8 |
+
# Question: does the army-size knee track the DATA'S latent rank under
|
| 9 |
+
# causal/AR pressure, as B7 measured statically? Synthetic byte streams of
|
| 10 |
+
# controlled rank R (hidden linear-dynamical state, emission logits confined
|
| 11 |
+
# to an R-dim subspace) are fed to the AR bed's CausalHUB arm — the aleph
|
| 12 |
+
# army under causal masks, army size = K oriented anchors (2K half-axes).
|
| 13 |
+
#
|
| 14 |
+
# GRID: R in {8, 32, 128} x K in {16, 64, 256} x seeds {0, 1} = 18 runs.
|
| 15 |
+
# GAUGES: val bits-per-byte curve; final address census-lite (pos/neg mass
|
| 16 |
+
# split, address erank). KNEE: smallest K within .01 bpb of that rank's
|
| 17 |
+
# best. REGISTERED PREDICTION (Rule 1): knee(R=8) < knee(R=128); flat knee
|
| 18 |
+
# across R demotes Rule 1 to static-battery scope.
|
| 19 |
+
# HONESTY: matched steps and data, NOT matched FLOPs — HUB compute is
|
| 20 |
+
# O(n*K*d), so K=256 spends ~16x K=16's attention FLOPs. The knee claim is
|
| 21 |
+
# capability-vs-army-size at matched data; stated on every artifact.
|
| 22 |
+
#
|
| 23 |
+
# Resume-safe: per-run JSONs in OUT_DIR; existing runs skip. Ships each run
|
| 24 |
+
# + the final knee table to AbstractPhil/aleph-splat-0 battery/v1_rank_army/.
|
| 25 |
+
# RUN: python tools\v1_rank_army_battery.py (HF_TOKEN in env; 4090)
|
| 26 |
+
# Env: V1_STEPS (def 3000) V1_QUICK=1 (tiny smoke grid)
|
| 27 |
+
# =========================================================================
|
| 28 |
+
|
| 29 |
+
import importlib
|
| 30 |
+
import json
|
| 31 |
+
import math
|
| 32 |
+
import os
|
| 33 |
+
import sys
|
| 34 |
+
|
| 35 |
+
import torch
|
| 36 |
+
import torch.nn.functional as F
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def _load(name):
|
| 40 |
+
if name in sys.modules:
|
| 41 |
+
return sys.modules[name]
|
| 42 |
+
try:
|
| 43 |
+
return importlib.import_module(name)
|
| 44 |
+
except ImportError:
|
| 45 |
+
here = globals().get("__file__")
|
| 46 |
+
if here is not None:
|
| 47 |
+
import pathlib
|
| 48 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 49 |
+
return importlib.import_module(name)
|
| 50 |
+
raise
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
bed = _load("ar_differentiation_bed") # ByteLM + _batch + conventions
|
| 54 |
+
class _cb: # public shim: token from env
|
| 55 |
+
@staticmethod
|
| 56 |
+
def hf_token():
|
| 57 |
+
import os as _os
|
| 58 |
+
return _os.environ.get("HF_TOKEN")
|
| 59 |
+
cb = _cb() # hf_token()
|
| 60 |
+
|
| 61 |
+
from huggingface_hub import HfApi # noqa: E402
|
| 62 |
+
|
| 63 |
+
assert torch.cuda.is_available(), "GPU only (never CPU-train for accuracy)"
|
| 64 |
+
DEV = "cuda"
|
| 65 |
+
QUICK = os.environ.get("V1_QUICK") == "1"
|
| 66 |
+
STEPS = int(os.environ.get("V1_STEPS", "200" if QUICK else "3000"))
|
| 67 |
+
BATCH, BLOCK = 32, 256
|
| 68 |
+
RANKS = [8, 32] if QUICK else [8, 32, 128]
|
| 69 |
+
ARMIES = [16, 64] if QUICK else [16, 64, 256]
|
| 70 |
+
SEEDS = [0] if QUICK else [0, 1]
|
| 71 |
+
EVAL_EVERY = 100 if QUICK else 300
|
| 72 |
+
DATA_DIR = os.environ.get("V1_DATA", "./data/v1_rank_army")
|
| 73 |
+
OUT_DIR = os.path.join(DATA_DIR, "results")
|
| 74 |
+
TRAIN_BYTES, VAL_BYTES = (400_000, 80_000) if QUICK else (2_000_000, 200_000)
|
| 75 |
+
HUB_REPO = "AbstractPhil/aleph-splat-0"
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def make_stream(R, n_bytes, seed, device=DEV):
|
| 79 |
+
"""Rank-R synthetic byte stream: hidden LDS state s (R-dim), emission
|
| 80 |
+
logits = W s confined to an R-dim subspace of R^256 by construction.
|
| 81 |
+
The model must FILTER the state from byte history — predictive
|
| 82 |
+
structure has latent rank exactly R. 64 parallel chains, concatenated."""
|
| 83 |
+
# STRUCTURAL seed depends on R only: train and val share the SAME
|
| 84 |
+
# system (A, W); the trajectory seed varies per split. (Smoke 2026-08-09
|
| 85 |
+
# caught the original bug: split-dependent A/W made val an unrelated
|
| 86 |
+
# stream — bpb pinned above uniform.)
|
| 87 |
+
sg = torch.Generator(device="cpu").manual_seed(10_000 + R)
|
| 88 |
+
A = torch.linalg.qr(torch.randn(R, R, generator=sg))[0] * 0.95
|
| 89 |
+
W = torch.randn(R, 256, generator=sg)
|
| 90 |
+
W = W / W.std() * 2.0 # logit scale: peaked, not trivial
|
| 91 |
+
A, W = A.to(device), W.to(device)
|
| 92 |
+
chains = 64
|
| 93 |
+
steps = n_bytes // chains + 1
|
| 94 |
+
ig = torch.Generator().manual_seed(20_000 + R * 100 + seed)
|
| 95 |
+
s = torch.randn(chains, R, generator=ig).to(device)
|
| 96 |
+
gg = torch.Generator(device=device).manual_seed(30_000 + R * 100 + seed)
|
| 97 |
+
out = torch.empty(chains, steps, dtype=torch.uint8, device=device)
|
| 98 |
+
with torch.no_grad():
|
| 99 |
+
for t in range(steps):
|
| 100 |
+
probs = F.softmax(s @ W, dim=-1)
|
| 101 |
+
out[:, t] = torch.multinomial(probs, 1, generator=gg).squeeze(1)
|
| 102 |
+
s = torch.tanh(s @ A.T
|
| 103 |
+
+ 0.10 * torch.randn(chains, R, generator=gg,
|
| 104 |
+
device=device))
|
| 105 |
+
return out.reshape(-1)[:n_bytes].cpu()
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def get_streams(R):
|
| 109 |
+
os.makedirs(f"{DATA_DIR}/streams", exist_ok=True)
|
| 110 |
+
p = f"{DATA_DIR}/streams/R{R}_{TRAIN_BYTES}.pt"
|
| 111 |
+
if os.path.exists(p):
|
| 112 |
+
d = torch.load(p, weights_only=True)
|
| 113 |
+
else:
|
| 114 |
+
print(f" generating rank-{R} stream ({TRAIN_BYTES:,}+{VAL_BYTES:,} bytes)",
|
| 115 |
+
flush=True)
|
| 116 |
+
d = {"train": make_stream(R, TRAIN_BYTES, seed=0),
|
| 117 |
+
"val": make_stream(R, VAL_BYTES, seed=1)} # val = fresh chains
|
| 118 |
+
torch.save(d, p)
|
| 119 |
+
ent = -sum((c := torch.bincount(d['val'].long(), minlength=256).float()
|
| 120 |
+
/ d['val'].numel())[c > 0]
|
| 121 |
+
* (c[c > 0]).log2()) if False else None # (unigram, optional)
|
| 122 |
+
return d["train"], d["val"]
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def census_lite(model, va, g):
|
| 126 |
+
"""Address census on one val batch: pos/neg half mass split + address
|
| 127 |
+
erank — the A1 instrument's AR-bed cousin."""
|
| 128 |
+
xv, _ = bed._batch(va, BATCH, BLOCK, DEV, g)
|
| 129 |
+
stats = []
|
| 130 |
+
with torch.no_grad():
|
| 131 |
+
x = model.emb(xv) + model.pos[:, : xv.shape[1]]
|
| 132 |
+
for blk in model.blocks:
|
| 133 |
+
h = blk.n1(x)
|
| 134 |
+
attn = blk.attn
|
| 135 |
+
qp, qn = attn.addr.oriented(attn.q(h))
|
| 136 |
+
mp, mn = float(qp.sum()), float(qn.sum())
|
| 137 |
+
flat = torch.cat([qp, qn], -1).reshape(-1, 2 * attn.addr.K)
|
| 138 |
+
sv = torch.linalg.svdvals(flat[:4096].float())
|
| 139 |
+
s2 = sv ** 2
|
| 140 |
+
stats.append({"neg_mass_frac": round(mn / (mp + mn + 1e-9), 4),
|
| 141 |
+
"addr_erank": round(float(s2.sum() ** 2
|
| 142 |
+
/ (s2 ** 2).sum()), 2)})
|
| 143 |
+
x = blk(x)
|
| 144 |
+
return stats
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def run_cell(R, K, seed, tr, va):
|
| 148 |
+
tag = f"R{R}_K{K}_s{seed}"
|
| 149 |
+
out_p = f"{OUT_DIR}/{tag}.json"
|
| 150 |
+
if os.path.exists(out_p):
|
| 151 |
+
print(f"-- {tag}: exists, skip", flush=True)
|
| 152 |
+
return json.load(open(out_p))
|
| 153 |
+
torch.manual_seed(seed)
|
| 154 |
+
g = torch.Generator().manual_seed(seed)
|
| 155 |
+
model = bed.ByteLM("hub", block=BLOCK, K=K, D=4).to(DEV)
|
| 156 |
+
n_par = sum(p.numel() for p in model.parameters())
|
| 157 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 158 |
+
curve = []
|
| 159 |
+
for step in range(1, STEPS + 1):
|
| 160 |
+
x, y = bed._batch(tr, BATCH, BLOCK, DEV, g)
|
| 161 |
+
loss = F.cross_entropy(model(x).reshape(-1, 256), y.reshape(-1))
|
| 162 |
+
opt.zero_grad(set_to_none=True)
|
| 163 |
+
loss.backward()
|
| 164 |
+
opt.step()
|
| 165 |
+
if step % EVAL_EVERY == 0 or step == STEPS:
|
| 166 |
+
model.eval()
|
| 167 |
+
with torch.no_grad():
|
| 168 |
+
ls = [F.cross_entropy(
|
| 169 |
+
model(xv).reshape(-1, 256), yv.reshape(-1)).item()
|
| 170 |
+
for xv, yv in (bed._batch(va, BATCH, BLOCK, DEV, g)
|
| 171 |
+
for _ in range(16))]
|
| 172 |
+
bpb = sum(ls) / len(ls) / math.log(2)
|
| 173 |
+
curve.append({"step": step, "val_bpb": round(bpb, 4)})
|
| 174 |
+
model.train()
|
| 175 |
+
model.eval()
|
| 176 |
+
res = {"R": R, "K": K, "seed": seed, "params": n_par,
|
| 177 |
+
"steps": STEPS, "batch": BATCH, "block": BLOCK,
|
| 178 |
+
"final_bpb": curve[-1]["val_bpb"], "curve": curve,
|
| 179 |
+
"census": census_lite(model, va, g),
|
| 180 |
+
"note": "matched steps+data, NOT matched FLOPs (HUB is O(nKd))"}
|
| 181 |
+
os.makedirs(OUT_DIR, exist_ok=True)
|
| 182 |
+
json.dump(res, open(out_p, "w"), indent=1)
|
| 183 |
+
print(f"-- {tag}: bpb {res['final_bpb']:.4f} ({n_par:,} params)",
|
| 184 |
+
flush=True)
|
| 185 |
+
del model
|
| 186 |
+
torch.cuda.empty_cache()
|
| 187 |
+
return res
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
def main():
|
| 191 |
+
print("=" * 78, flush=True)
|
| 192 |
+
print(f"V1 RANK x ARMY BATTERY — R{RANKS} x K{ARMIES} x s{SEEDS}, "
|
| 193 |
+
f"{STEPS} steps {'[QUICK]' if QUICK else ''}", flush=True)
|
| 194 |
+
print("=" * 78, flush=True)
|
| 195 |
+
api = HfApi(token=cb.hf_token())
|
| 196 |
+
results = []
|
| 197 |
+
for R in RANKS:
|
| 198 |
+
tr, va = get_streams(R)
|
| 199 |
+
for K in ARMIES:
|
| 200 |
+
for seed in SEEDS:
|
| 201 |
+
results.append(run_cell(R, K, seed, tr, va))
|
| 202 |
+
if not QUICK:
|
| 203 |
+
tag = f"R{R}_K{K}_s{seed}"
|
| 204 |
+
api.upload_file(
|
| 205 |
+
path_or_fileobj=f"{OUT_DIR}/{tag}.json",
|
| 206 |
+
path_in_repo=f"battery/v1_rank_army/{tag}.json",
|
| 207 |
+
repo_id=HUB_REPO,
|
| 208 |
+
commit_message=f"V1 cell {tag}")
|
| 209 |
+
# ---- knee analysis: smallest K within .01 bpb of the rank's best ----
|
| 210 |
+
verdict = {}
|
| 211 |
+
for R in RANKS:
|
| 212 |
+
by_k = {K: sum(r["final_bpb"] for r in results
|
| 213 |
+
if r["R"] == R and r["K"] == K)
|
| 214 |
+
/ len(SEEDS) for K in ARMIES}
|
| 215 |
+
best = min(by_k.values())
|
| 216 |
+
knee = min(K for K, b in by_k.items() if b <= best + 0.01)
|
| 217 |
+
verdict[R] = {"bpb_by_K": {str(k): round(v, 4)
|
| 218 |
+
for k, v in by_k.items()},
|
| 219 |
+
"knee_K": knee}
|
| 220 |
+
print(f"R={R:>3}: " + " ".join(f"K{k}={v:.4f}"
|
| 221 |
+
for k, v in by_k.items())
|
| 222 |
+
+ f" -> knee K={knee}", flush=True)
|
| 223 |
+
knees = [verdict[R]["knee_K"] for R in RANKS]
|
| 224 |
+
verdict["registered_prediction"] = (
|
| 225 |
+
"Rule 1 SUPPORTED (knee tracks rank)" if knees == sorted(knees)
|
| 226 |
+
and knees[0] < knees[-1] else
|
| 227 |
+
"Rule 1 NOT supported at this bed scale (knee flat or non-monotone)")
|
| 228 |
+
print("VERDICT:", verdict["registered_prediction"], flush=True)
|
| 229 |
+
out = f"{OUT_DIR}/v1_verdict.json"
|
| 230 |
+
json.dump(verdict, open(out, "w"), indent=1)
|
| 231 |
+
if not QUICK:
|
| 232 |
+
api.upload_file(path_or_fileobj=out,
|
| 233 |
+
path_in_repo="battery/v1_rank_army/v1_verdict.json",
|
| 234 |
+
repo_id=HUB_REPO,
|
| 235 |
+
commit_message="V1 verdict: knee-vs-rank table "
|
| 236 |
+
"(Rule 1 arbiter)")
|
| 237 |
+
print("DONE.", flush=True)
|
| 238 |
+
|
| 239 |
+
|
| 240 |
+
if __name__ == "__main__":
|
| 241 |
+
main()
|
trainers/v1b_recall_army_battery.py
ADDED
|
@@ -0,0 +1,323 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# v1b_recall_army_battery.py — V1b: ADDRESSING-DEMAND x army-ladder under AR
|
| 3 |
+
# (overnight battery #2, 2026-08-09; the V1-LDS census showed filtering
|
| 4 |
+
# demands ~no addressing — this battery applies the addressing-bound
|
| 5 |
+
# pressure: in-context associative recall, B8's territory, key space
|
| 6 |
+
# rank-controlled.)
|
| 7 |
+
#
|
| 8 |
+
# Episodes: P=8 (key 2 bytes, val 1 byte) pairs + SEP + query key -> the
|
| 9 |
+
# model must emit the queried value at the answer position. Keys are sign
|
| 10 |
+
# patterns of R_key-dim latents projected through 16 fixed directions —
|
| 11 |
+
# key diversity (addressing demand) rises with R_key. Values fresh-random
|
| 12 |
+
# per episode: binding is IN-CONTEXT, only the address machinery can do it.
|
| 13 |
+
# Loss masked to ANSWER positions only.
|
| 14 |
+
#
|
| 15 |
+
# GRID: R_key {4,16,64} x K {16,64,256} x seeds {0,1} = 18 cells,
|
| 16 |
+
# 2500 steps. GAUGES: answer accuracy + answer bpb + census-lite
|
| 17 |
+
# (consumed addr erank = the demand gauge). REGISTERED PREDICTIONS:
|
| 18 |
+
# (i) knee tracks R_key; (ii) consumed addr erank rises with R_key;
|
| 19 |
+
# (iii) both => Rule 1 re-scoped to M = c * addressing-demand.
|
| 20 |
+
# Same honesty stamp as V1: matched steps/data, NOT matched FLOPs.
|
| 21 |
+
# RUN: python tools\v1b_recall_army_battery.py (HF_TOKEN, 4090)
|
| 22 |
+
# Env: V1B_STEPS (def 2500) V1B_QUICK=1
|
| 23 |
+
# =========================================================================
|
| 24 |
+
|
| 25 |
+
import importlib
|
| 26 |
+
import json
|
| 27 |
+
import math
|
| 28 |
+
import os
|
| 29 |
+
import sys
|
| 30 |
+
|
| 31 |
+
import torch
|
| 32 |
+
import torch.nn.functional as F
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def _load(name):
|
| 36 |
+
if name in sys.modules:
|
| 37 |
+
return sys.modules[name]
|
| 38 |
+
try:
|
| 39 |
+
return importlib.import_module(name)
|
| 40 |
+
except ImportError:
|
| 41 |
+
here = globals().get("__file__")
|
| 42 |
+
if here is not None:
|
| 43 |
+
import pathlib
|
| 44 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 45 |
+
return importlib.import_module(name)
|
| 46 |
+
raise
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
bed = _load("ar_differentiation_bed")
|
| 50 |
+
class _cb: # public shim: token from env
|
| 51 |
+
@staticmethod
|
| 52 |
+
def hf_token():
|
| 53 |
+
import os as _os
|
| 54 |
+
return _os.environ.get("HF_TOKEN")
|
| 55 |
+
cb = _cb()
|
| 56 |
+
from huggingface_hub import HfApi # noqa: E402
|
| 57 |
+
|
| 58 |
+
assert torch.cuda.is_available(), "GPU only"
|
| 59 |
+
DEV = "cuda"
|
| 60 |
+
QUICK = os.environ.get("V1B_QUICK") == "1"
|
| 61 |
+
STEPS = int(os.environ.get("V1B_STEPS", "250" if QUICK else "8000"))
|
| 62 |
+
# CONTROL FINDING (2026-08-09 pre-battery): on this task sdpa_tri reads
|
| 63 |
+
# .9251 answer-acc at step 1500 while hub_tri reads .0561 — the harness is
|
| 64 |
+
# sound and in-context BINDING is address-capacity-hard for the linear
|
| 65 |
+
# aleph read (late-takeoff curve, the splat late-bloom signature). The
|
| 66 |
+
# battery therefore runs hub long (8k steps) with an sdpa reference arm
|
| 67 |
+
# per rank, and prediction (iii) becomes: K gates binding FORMATION.
|
| 68 |
+
BATCH, BLOCK = 32, 256
|
| 69 |
+
RKEYS = [4, 16] if QUICK else [4, 16, 64]
|
| 70 |
+
ARMIES = [16, 64] if QUICK else [16, 64, 256]
|
| 71 |
+
SEEDS = [0] if QUICK else [0, 1]
|
| 72 |
+
EVAL_EVERY = 100 if QUICK else 250
|
| 73 |
+
D_ADDR = int(os.environ.get("V1B_D", "4")) # address dims (V1c axis)
|
| 74 |
+
if os.environ.get("V1B_RKEYS"):
|
| 75 |
+
RKEYS = [int(x) for x in os.environ["V1B_RKEYS"].split(",")]
|
| 76 |
+
if os.environ.get("V1B_ARMIES"):
|
| 77 |
+
ARMIES = [int(x) for x in os.environ["V1B_ARMIES"].split(",")]
|
| 78 |
+
P = 8 # pairs per episode
|
| 79 |
+
EPI = P * 3 + 3 # 27 bytes: pairs + SEP + query(2)
|
| 80 |
+
SEP = 255 # bytes 0..253 for keys/vals
|
| 81 |
+
DATA_DIR = os.environ.get("V1B_DATA", "./data/v1_rank_army")
|
| 82 |
+
OUT_DIR = os.path.join(DATA_DIR, "results_v1b")
|
| 83 |
+
N_EPISODES = (3_000, 600) if QUICK else (60_000, 8_000) # train, val
|
| 84 |
+
HUB_REPO = "AbstractPhil/aleph-splat-0"
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def make_episodes(R, n_epi, seed):
|
| 88 |
+
"""Concatenated recall episodes + answer-position mask.
|
| 89 |
+
Key = 2 bytes from sign bits of a fresh R-dim latent through 16 fixed
|
| 90 |
+
directions (structural, R-only seed); vals fresh-random per episode."""
|
| 91 |
+
sg = torch.Generator().manual_seed(40_000 + R)
|
| 92 |
+
U = torch.randn(16, R, generator=sg) # fixed directions
|
| 93 |
+
g = torch.Generator().manual_seed(50_000 + R * 100 + seed)
|
| 94 |
+
z = torch.randn(n_epi, P, R, generator=g) # latents per pair
|
| 95 |
+
bits = (torch.einsum("epr,br->epb", z, U) > 0) # (n_epi, P, 16)
|
| 96 |
+
w1 = (1 << torch.arange(8))
|
| 97 |
+
k1 = (bits[..., :8].long() * w1).sum(-1) # 0..255
|
| 98 |
+
k2 = (bits[..., 8:].long() * w1).sum(-1)
|
| 99 |
+
k1, k2 = k1 % 254, k2 % 254 # keep off SEP
|
| 100 |
+
vals = torch.randint(0, 254, (n_epi, P), generator=g)
|
| 101 |
+
qi = torch.randint(0, P, (n_epi,), generator=g) # queried pair
|
| 102 |
+
seq = torch.zeros(n_epi, EPI, dtype=torch.uint8)
|
| 103 |
+
seq[:, 0:P * 3:3] = k1.to(torch.uint8)
|
| 104 |
+
seq[:, 1:P * 3:3] = k2.to(torch.uint8)
|
| 105 |
+
seq[:, 2:P * 3:3] = vals.to(torch.uint8)
|
| 106 |
+
seq[:, P * 3] = SEP
|
| 107 |
+
ar = torch.arange(n_epi)
|
| 108 |
+
seq[:, P * 3 + 1] = k1[ar, qi].to(torch.uint8)
|
| 109 |
+
seq[:, P * 3 + 2] = k2[ar, qi].to(torch.uint8)
|
| 110 |
+
answer = vals[ar, qi].to(torch.uint8) # target AFTER query
|
| 111 |
+
# stream layout: [...episode..., answer] so predicting the byte at the
|
| 112 |
+
# answer slot from the preceding query is exactly next-byte loss there.
|
| 113 |
+
flat = torch.cat([seq, answer.unsqueeze(1)], dim=1).reshape(-1)
|
| 114 |
+
mask = torch.zeros_like(flat, dtype=torch.bool)
|
| 115 |
+
mask[(EPI + 1) * ar + EPI] = True # answer positions
|
| 116 |
+
return flat, mask
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def get_data(R):
|
| 120 |
+
os.makedirs(f"{DATA_DIR}/streams_v1b", exist_ok=True)
|
| 121 |
+
p = f"{DATA_DIR}/streams_v1b/R{R}_{N_EPISODES[0]}.pt"
|
| 122 |
+
if os.path.exists(p):
|
| 123 |
+
d = torch.load(p, weights_only=True)
|
| 124 |
+
else:
|
| 125 |
+
print(f" generating R_key={R} episodes {N_EPISODES}", flush=True)
|
| 126 |
+
tr, trm = make_episodes(R, N_EPISODES[0], seed=0)
|
| 127 |
+
va, vam = make_episodes(R, N_EPISODES[1], seed=1)
|
| 128 |
+
d = {"tr": tr, "trm": trm, "va": va, "vam": vam}
|
| 129 |
+
torch.save(d, p)
|
| 130 |
+
return d
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def batch_masked(flat, mask, batch, block, device, g):
|
| 134 |
+
ix = torch.randint(0, flat.numel() - block - 1, (batch,), generator=g)
|
| 135 |
+
x = torch.stack([flat[i:i + block] for i in ix]).long().to(device)
|
| 136 |
+
y = torch.stack([flat[i + 1:i + block + 1] for i in ix]).long().to(device)
|
| 137 |
+
m = torch.stack([mask[i + 1:i + block + 1] for i in ix]).to(device)
|
| 138 |
+
if os.environ.get("V1B_CLEAN_MASK") == "1":
|
| 139 |
+
# 2026-08-10 corruption repair: drop answer positions whose episode
|
| 140 |
+
# is window-truncated (~6.05% unanswerable — 4x-weighted label
|
| 141 |
+
# noise otherwise; hunter-verified j>=26 = answerable frac 1.0000)
|
| 142 |
+
m[:, :26] = False
|
| 143 |
+
return x, y, m
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def masked_eval(model, d, g, iters=16):
|
| 147 |
+
accs, bits = [], []
|
| 148 |
+
with torch.no_grad():
|
| 149 |
+
for _ in range(iters):
|
| 150 |
+
xv, yv, mv = batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
|
| 151 |
+
if mv.sum() == 0:
|
| 152 |
+
continue
|
| 153 |
+
logits = model(xv)
|
| 154 |
+
la, ya = logits[mv], yv[mv]
|
| 155 |
+
bits.append(F.cross_entropy(la, ya).item() / math.log(2))
|
| 156 |
+
accs.append((la.argmax(-1) == ya).float().mean().item())
|
| 157 |
+
return (sum(accs) / len(accs), sum(bits) / len(bits)) if accs else (0.0, 8.0)
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def census_lite(model, d, g):
|
| 161 |
+
xv, _, _ = batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
|
| 162 |
+
stats = []
|
| 163 |
+
with torch.no_grad():
|
| 164 |
+
x = model.emb(xv)
|
| 165 |
+
if getattr(model, "trigram", False): # 2026-08-09 audit fix: the
|
| 166 |
+
# census must see the SAME composed embedding the model runs
|
| 167 |
+
# on (was reading 1 of 3 tables — demand gauge underweighted)
|
| 168 |
+
pv = 256 if getattr(model, "pad_fix", False) else 0
|
| 169 |
+
x = x + model.emb1(F.pad(xv, (1, 0), value=pv)[:, :-1]) \
|
| 170 |
+
+ model.emb2(F.pad(xv, (2, 0), value=pv)[:, :-2])
|
| 171 |
+
x = x + model.pos[:, : xv.shape[1]]
|
| 172 |
+
for blk in model.blocks:
|
| 173 |
+
h = blk.n1(x)
|
| 174 |
+
attn = blk.attn
|
| 175 |
+
qp, qn = attn.addr.oriented(attn.q(h))
|
| 176 |
+
mp, mn = float(qp.sum()), float(qn.sum())
|
| 177 |
+
fl = torch.cat([qp, qn], -1).reshape(-1, 2 * attn.addr.K)
|
| 178 |
+
sv = torch.linalg.svdvals(fl[:4096].float())
|
| 179 |
+
s2 = sv ** 2
|
| 180 |
+
stats.append({"neg_mass_frac": round(mn / (mp + mn + 1e-9), 4),
|
| 181 |
+
"addr_erank": round(float(s2.sum() ** 2
|
| 182 |
+
/ (s2 ** 2).sum()), 2)})
|
| 183 |
+
x = blk(x)
|
| 184 |
+
return stats
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
def run_cell(R, K, seed, d):
|
| 188 |
+
tag = f"R{R}_K{K}_s{seed}" + (f"_D{D_ADDR}" if D_ADDR != 4 else "")
|
| 189 |
+
out_p = f"{OUT_DIR}/{tag}.json"
|
| 190 |
+
if os.path.exists(out_p):
|
| 191 |
+
print(f"-- {tag}: exists, skip", flush=True)
|
| 192 |
+
return json.load(open(out_p))
|
| 193 |
+
torch.manual_seed(seed)
|
| 194 |
+
g = torch.Generator().manual_seed(seed)
|
| 195 |
+
arm = "sdpa_tri" if K == 0 else "hub_tri" # K=0 = the control arm
|
| 196 |
+
model = bed.ByteLM(arm, block=BLOCK, K=max(K, 4), D=D_ADDR).to(DEV)
|
| 197 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 198 |
+
curve = []
|
| 199 |
+
for step in range(1, STEPS + 1):
|
| 200 |
+
x, y, m = batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
|
| 201 |
+
logits = model(x)
|
| 202 |
+
if m.sum() == 0:
|
| 203 |
+
continue
|
| 204 |
+
# DENSE loss builds the sequence circuits (masked-only starves
|
| 205 |
+
# them — smoke 2026-08-09: chance acc, bpb>10); answers get 4x
|
| 206 |
+
# weight and remain the ONLY reported gauge.
|
| 207 |
+
loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
|
| 208 |
+
+ 4.0 * F.cross_entropy(logits[m], y[m]))
|
| 209 |
+
opt.zero_grad(set_to_none=True)
|
| 210 |
+
loss.backward()
|
| 211 |
+
opt.step()
|
| 212 |
+
if step % EVAL_EVERY == 0 or step == STEPS:
|
| 213 |
+
model.eval()
|
| 214 |
+
acc, bpb = masked_eval(model, d, g)
|
| 215 |
+
curve.append({"step": step, "ans_acc": round(acc, 4),
|
| 216 |
+
"ans_bpb": round(bpb, 4)})
|
| 217 |
+
model.train()
|
| 218 |
+
model.eval()
|
| 219 |
+
acc, bpb = masked_eval(model, d, g, iters=32)
|
| 220 |
+
res = {"R_key": R, "K": K, "seed": seed, "steps": STEPS, "arm": arm,
|
| 221 |
+
"final_acc": round(acc, 4), "final_bpb": round(bpb, 4),
|
| 222 |
+
"curve": curve,
|
| 223 |
+
"census": census_lite(model, d, g) if K else None,
|
| 224 |
+
"note": "loss+gauges on answer positions only; matched steps, "
|
| 225 |
+
"NOT matched FLOPs (HUB O(nKd))"}
|
| 226 |
+
os.makedirs(OUT_DIR, exist_ok=True)
|
| 227 |
+
json.dump(res, open(out_p, "w"), indent=1)
|
| 228 |
+
print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f}", flush=True)
|
| 229 |
+
del model
|
| 230 |
+
torch.cuda.empty_cache()
|
| 231 |
+
return res
|
| 232 |
+
|
| 233 |
+
|
| 234 |
+
@torch.no_grad()
|
| 235 |
+
def selection_eval(model, d, g, iters=32):
|
| 236 |
+
"""Error decomposition (AbstractPhil lineage 2026-08-09): of WRONG answers, the
|
| 237 |
+
fraction that are OTHER IN-CONTEXT VALUES (mis-selection = retrieved
|
| 238 |
+
the wrong pair — the 'incorrect selections' fault class) vs bytes from
|
| 239 |
+
outside the episode's value set (no-retrieval). Only full-episode
|
| 240 |
+
windows (mask index >= 26) are scored."""
|
| 241 |
+
n_right = n_missel = n_noret = 0
|
| 242 |
+
for _ in range(iters):
|
| 243 |
+
xv, yv, mv = batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
|
| 244 |
+
pred = model(xv).argmax(-1)
|
| 245 |
+
for b, j in mv.nonzero(as_tuple=False).tolist():
|
| 246 |
+
if j < 26:
|
| 247 |
+
continue
|
| 248 |
+
vals = {int(xv[b, j - 24 + 3 * k]) for k in range(P)}
|
| 249 |
+
p, t = int(pred[b, j]), int(yv[b, j])
|
| 250 |
+
if p == t:
|
| 251 |
+
n_right += 1
|
| 252 |
+
elif p in vals:
|
| 253 |
+
n_missel += 1
|
| 254 |
+
else:
|
| 255 |
+
n_noret += 1
|
| 256 |
+
tot = n_right + n_missel + n_noret
|
| 257 |
+
wrong = n_missel + n_noret
|
| 258 |
+
return {"n": tot, "acc": round(n_right / max(tot, 1), 4),
|
| 259 |
+
"mis_sel_frac_of_errors": round(n_missel / max(wrong, 1), 4),
|
| 260 |
+
"no_retrieval_frac_of_errors": round(n_noret / max(wrong, 1), 4)}
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
def main():
|
| 264 |
+
print("=" * 78, flush=True)
|
| 265 |
+
print(f"V1b RECALL x ARMY — R_key{RKEYS} x K{ARMIES} x s{SEEDS}, "
|
| 266 |
+
f"{STEPS} steps {'[QUICK]' if QUICK else ''}", flush=True)
|
| 267 |
+
print("=" * 78, flush=True)
|
| 268 |
+
api = HfApi(token=cb.hf_token())
|
| 269 |
+
results = []
|
| 270 |
+
for R in RKEYS:
|
| 271 |
+
d = get_data(R)
|
| 272 |
+
for K in ([0] + ARMIES if not QUICK else ARMIES): # K=0 = sdpa ref
|
| 273 |
+
for seed in (SEEDS if K else [0]):
|
| 274 |
+
results.append(run_cell(R, K, seed, d))
|
| 275 |
+
if not QUICK:
|
| 276 |
+
tag = f"R{R}_K{K}_s{seed}" + (f"_D{D_ADDR}" if D_ADDR != 4 else "")
|
| 277 |
+
api.upload_file(
|
| 278 |
+
path_or_fileobj=f"{OUT_DIR}/{tag}.json",
|
| 279 |
+
path_in_repo=f"battery/v1b_recall_army/{tag}.json",
|
| 280 |
+
repo_id=HUB_REPO, commit_message=f"V1b cell {tag}")
|
| 281 |
+
verdict = {}
|
| 282 |
+
for R in RKEYS:
|
| 283 |
+
by_k = {K: sum(r["final_acc"] for r in results
|
| 284 |
+
if r["R_key"] == R and r["K"] == K) / len(SEEDS)
|
| 285 |
+
for K in ARMIES}
|
| 286 |
+
best = max(by_k.values())
|
| 287 |
+
knee = min(K for K, a in by_k.items() if a >= best - 0.02)
|
| 288 |
+
er = {K: sum(max(c["addr_erank"] for c in r["census"])
|
| 289 |
+
for r in results if r["R_key"] == R and r["K"] == K)
|
| 290 |
+
/ len(SEEDS) for K in ARMIES}
|
| 291 |
+
sdpa_ref = next((r["final_acc"] for r in results
|
| 292 |
+
if r["R_key"] == R and r["K"] == 0), None)
|
| 293 |
+
verdict[R] = {"acc_by_K": {str(k): round(v, 4)
|
| 294 |
+
for k, v in by_k.items()},
|
| 295 |
+
"knee_K": knee, "sdpa_ref_acc": sdpa_ref,
|
| 296 |
+
"max_addr_erank_by_K": {str(k): round(v, 1)
|
| 297 |
+
for k, v in er.items()}}
|
| 298 |
+
print(f"R_key={R:>3}: "
|
| 299 |
+
+ " ".join(f"K{k}={v:.4f}" for k, v in by_k.items())
|
| 300 |
+
+ f" -> knee K={knee} | consumed erank "
|
| 301 |
+
+ " ".join(f"{v:.0f}" for v in er.values()), flush=True)
|
| 302 |
+
knees = [verdict[R]["knee_K"] for R in RKEYS]
|
| 303 |
+
verdict["prediction_i_knee_tracks_Rkey"] = (
|
| 304 |
+
knees == sorted(knees) and knees[0] < knees[-1])
|
| 305 |
+
ers = [max(verdict[R]["max_addr_erank_by_K"].values()) for R in RKEYS]
|
| 306 |
+
verdict["prediction_ii_demand_rises_with_Rkey"] = ers == sorted(ers)
|
| 307 |
+
print(f"VERDICT: (i) knee tracks R_key: "
|
| 308 |
+
f"{verdict['prediction_i_knee_tracks_Rkey']} "
|
| 309 |
+
f"(ii) consumed demand rises: "
|
| 310 |
+
f"{verdict['prediction_ii_demand_rises_with_Rkey']}", flush=True)
|
| 311 |
+
out = f"{OUT_DIR}/v1b_verdict.json"
|
| 312 |
+
json.dump(verdict, open(out, "w"), indent=1)
|
| 313 |
+
if not QUICK:
|
| 314 |
+
api.upload_file(path_or_fileobj=out,
|
| 315 |
+
path_in_repo="battery/v1b_recall_army/v1b_verdict.json",
|
| 316 |
+
repo_id=HUB_REPO,
|
| 317 |
+
commit_message="V1b verdict: recall knee + "
|
| 318 |
+
"consumed-demand table")
|
| 319 |
+
print("DONE.", flush=True)
|
| 320 |
+
|
| 321 |
+
|
| 322 |
+
if __name__ == "__main__":
|
| 323 |
+
main()
|
trainers/v1d_svae_graft.py
ADDED
|
@@ -0,0 +1,230 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# v1d_svae_graft.py — V1d: SVAE-transformer pieces vs the binding gap
|
| 3 |
+
# (2026-08-09; AbstractPhil: "try taking some pieces from the SVAE transformer
|
| 4 |
+
# concept, see if we can shorten the gap, or if the svae-transformer
|
| 5 |
+
# projection concept is simply obsolete at this point.")
|
| 6 |
+
#
|
| 7 |
+
# Bed point: R64 recall (v1b data), K=64, D=16 — baseline .599/.722
|
| 8 |
+
# (mean .661), sdpa ref .936. Winner (if any) confirms at K=256/D=16
|
| 9 |
+
# (baseline .678/.778). Grafts from canon/svae_h2_class.md (May 29 arc):
|
| 10 |
+
# sphere — H2 chain "linear->sphere-norm->canon": per-token sphere
|
| 11 |
+
# normalization BEFORE the q/k projections (the address
|
| 12 |
+
# already cos-normalizes AFTER; this changes what the
|
| 13 |
+
# projection learns).
|
| 14 |
+
# lens — fixed isometric lens (0-param orthogonal DxD between
|
| 15 |
+
# projection and address). PREDICTED NULL by inspection:
|
| 16 |
+
# with a LEARNABLE codebook, cos(Lx,a)=cos(x,L^T a) — the
|
| 17 |
+
# lens is absorbed into codebook learning. Runs 1 seed as
|
| 18 |
+
# the labeled math-check.
|
| 19 |
+
# boundgate — attenuator grammar lite: hub output enters the residual
|
| 20 |
+
# through a bounded learnable gate a<=0.2, init .024
|
| 21 |
+
# ("modulate never inject" + the C6 null-path lineage).
|
| 22 |
+
# modulate — the literal H2 form: y = x * (1 + a*tanh(hub_out)),
|
| 23 |
+
# multiplicative, bounded, never additive injection.
|
| 24 |
+
# combo — sphere + boundgate (the two live-hypothesis pieces).
|
| 25 |
+
# OBSOLESCENCE GATE: any arm closing >= half the remaining gap
|
| 26 |
+
# (mean acc >= .80 at K64/D16) = SVAE pieces LIVE; all arms within +.02
|
| 27 |
+
# of baseline = the projection concept is OBSOLETE here (its function
|
| 28 |
+
# absorbed by cos addressing + D-matching + K-provisioning).
|
| 29 |
+
# RUN: python tools\v1d_svae_graft.py (HF_TOKEN, 4090)
|
| 30 |
+
# Env: V1D_ARMS, V1D_STEPS (def 8000), V1D_K (def 64)
|
| 31 |
+
# =========================================================================
|
| 32 |
+
|
| 33 |
+
import importlib
|
| 34 |
+
import json
|
| 35 |
+
import os
|
| 36 |
+
import sys
|
| 37 |
+
|
| 38 |
+
import torch
|
| 39 |
+
import torch.nn as nn
|
| 40 |
+
import torch.nn.functional as F
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def _load(name):
|
| 44 |
+
if name in sys.modules:
|
| 45 |
+
return sys.modules[name]
|
| 46 |
+
try:
|
| 47 |
+
return importlib.import_module(name)
|
| 48 |
+
except ImportError:
|
| 49 |
+
here = globals().get("__file__")
|
| 50 |
+
if here is not None:
|
| 51 |
+
import pathlib
|
| 52 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 53 |
+
return importlib.import_module(name)
|
| 54 |
+
raise
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
bed = _load("ar_differentiation_bed")
|
| 58 |
+
v1b = _load("v1b_recall_army_battery")
|
| 59 |
+
class _cb: # public shim: token from env
|
| 60 |
+
@staticmethod
|
| 61 |
+
def hf_token():
|
| 62 |
+
import os as _os
|
| 63 |
+
return _os.environ.get("HF_TOKEN")
|
| 64 |
+
cb = _cb()
|
| 65 |
+
from huggingface_hub import HfApi # noqa: E402
|
| 66 |
+
|
| 67 |
+
assert torch.cuda.is_available(), "GPU only"
|
| 68 |
+
DEV = "cuda"
|
| 69 |
+
STEPS = int(os.environ.get("V1D_STEPS", "8000"))
|
| 70 |
+
K_ARMY = int(os.environ.get("V1D_K", "64"))
|
| 71 |
+
D_ADDR = 16 # matched (V1c law)
|
| 72 |
+
R_KEY = 64
|
| 73 |
+
ARMS = os.environ.get(
|
| 74 |
+
"V1D_ARMS", "sphere,lens,boundgate,modulate,combo").split(",")
|
| 75 |
+
BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
|
| 76 |
+
OUT_DIR = os.path.join(v1b.DATA_DIR, "results_v1d")
|
| 77 |
+
HUB_REPO = v1b.HUB_REPO
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
class GraftHUB(bed.CausalHUB):
|
| 81 |
+
"""CausalHUB + SVAE grafts. sphere: normalize x before q/k projections.
|
| 82 |
+
lens: fixed orthogonal DxD after projection (0 params, buffer)."""
|
| 83 |
+
|
| 84 |
+
def __init__(self, d, K, D, sphere=False, lens=False):
|
| 85 |
+
super().__init__(d, K, D)
|
| 86 |
+
self.sphere = sphere
|
| 87 |
+
if lens:
|
| 88 |
+
L = torch.linalg.qr(torch.randn(D, D))[0]
|
| 89 |
+
self.register_buffer("lens", L)
|
| 90 |
+
else:
|
| 91 |
+
self.lens = None
|
| 92 |
+
|
| 93 |
+
def forward(self, x):
|
| 94 |
+
xa = F.normalize(x, dim=-1) if self.sphere else x
|
| 95 |
+
q, k = self.q(xa), self.k(xa)
|
| 96 |
+
if self.lens is not None:
|
| 97 |
+
q, k = q @ self.lens, k @ self.lens
|
| 98 |
+
qp, qn = self.addr.oriented(q)
|
| 99 |
+
kp, kn = self.addr.oriented(k)
|
| 100 |
+
v = self.v(x)
|
| 101 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 102 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 103 |
+
zp = torch.cumsum(kp, dim=1)
|
| 104 |
+
zn = torch.cumsum(kn, dim=1)
|
| 105 |
+
num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
|
| 106 |
+
+ torch.einsum("bnk,bnkd->bnd", qn, Sn))
|
| 107 |
+
den = ((qp * zp).sum(-1, keepdim=True)
|
| 108 |
+
+ (qn * zn).sum(-1, keepdim=True))
|
| 109 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
class GraftBlock(nn.Module):
|
| 113 |
+
"""Block with the SVAE residual grammars. inject: y = x + attn (bed
|
| 114 |
+
default). boundgate: y = x + min(a,0.2)*attn, a learnable init .024.
|
| 115 |
+
modulate: y = x * (1 + a*tanh(attn)) — multiplicative, never additive."""
|
| 116 |
+
|
| 117 |
+
def __init__(self, d, attn, grammar="inject"):
|
| 118 |
+
super().__init__()
|
| 119 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 120 |
+
self.attn = attn
|
| 121 |
+
self.grammar = grammar
|
| 122 |
+
if grammar in ("boundgate", "modulate"):
|
| 123 |
+
self.alpha = nn.Parameter(torch.tensor(0.024))
|
| 124 |
+
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),
|
| 125 |
+
nn.Linear(4 * d, d))
|
| 126 |
+
|
| 127 |
+
def forward(self, x):
|
| 128 |
+
a = self.attn(self.n1(x))
|
| 129 |
+
if self.grammar == "boundgate":
|
| 130 |
+
x = x + self.alpha.clamp(0, 0.2) * a
|
| 131 |
+
elif self.grammar == "modulate":
|
| 132 |
+
x = x * (1.0 + self.alpha.clamp(0, 0.2) * torch.tanh(a))
|
| 133 |
+
else:
|
| 134 |
+
x = x + a
|
| 135 |
+
return x + self.mlp(self.n2(x))
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def build_model(arm, d=192, layers=4):
|
| 139 |
+
sphere = arm in ("sphere", "combo")
|
| 140 |
+
lens = arm == "lens"
|
| 141 |
+
grammar = ("boundgate" if arm in ("boundgate", "combo")
|
| 142 |
+
else "modulate" if arm == "modulate" else "inject")
|
| 143 |
+
m = bed.ByteLM("hub_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
|
| 144 |
+
m.blocks = nn.ModuleList([
|
| 145 |
+
GraftBlock(d, GraftHUB(d, K_ARMY, D_ADDR, sphere, lens), grammar)
|
| 146 |
+
for _ in range(layers)])
|
| 147 |
+
return m
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
def run_cell(arm, seed, d):
|
| 151 |
+
tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
|
| 152 |
+
out_p = f"{OUT_DIR}/{tag}.json"
|
| 153 |
+
if os.path.exists(out_p):
|
| 154 |
+
print(f"-- {tag}: exists, skip", flush=True)
|
| 155 |
+
return json.load(open(out_p))
|
| 156 |
+
torch.manual_seed(seed)
|
| 157 |
+
g = torch.Generator().manual_seed(seed)
|
| 158 |
+
model = build_model(arm).to(DEV)
|
| 159 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 160 |
+
for step in range(1, STEPS + 1):
|
| 161 |
+
x, y, m = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
|
| 162 |
+
logits = model(x)
|
| 163 |
+
if m.sum() == 0:
|
| 164 |
+
continue
|
| 165 |
+
loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
|
| 166 |
+
+ 4.0 * F.cross_entropy(logits[m], y[m]))
|
| 167 |
+
opt.zero_grad(set_to_none=True)
|
| 168 |
+
loss.backward()
|
| 169 |
+
opt.step()
|
| 170 |
+
model.eval()
|
| 171 |
+
acc, bpb = v1b.masked_eval(model, d, g, iters=32)
|
| 172 |
+
alphas = [round(float(b.alpha.detach()), 4) for b in model.blocks
|
| 173 |
+
if hasattr(b, "alpha")]
|
| 174 |
+
res = {"arm": arm, "seed": seed, "R": R_KEY, "K": K_ARMY, "D": D_ADDR,
|
| 175 |
+
"steps": STEPS, "final_acc": round(acc, 4),
|
| 176 |
+
"final_bpb": round(bpb, 4), "alphas": alphas,
|
| 177 |
+
"baseline_ref": {"K64_D16": [0.5994, 0.7224],
|
| 178 |
+
"K256_D16": [0.6783, 0.7776], "sdpa": 0.936}}
|
| 179 |
+
os.makedirs(OUT_DIR, exist_ok=True)
|
| 180 |
+
json.dump(res, open(out_p, "w"), indent=1)
|
| 181 |
+
print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f}"
|
| 182 |
+
+ (f" alphas {alphas}" if alphas else ""), flush=True)
|
| 183 |
+
del model
|
| 184 |
+
torch.cuda.empty_cache()
|
| 185 |
+
return res
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def main():
|
| 189 |
+
print("=" * 78, flush=True)
|
| 190 |
+
print(f"V1d SVAE GRAFTS — arms {ARMS} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}, "
|
| 191 |
+
f"{STEPS} steps (baseline mean .661, sdpa .936)", flush=True)
|
| 192 |
+
print("=" * 78, flush=True)
|
| 193 |
+
api = HfApi(token=cb.hf_token())
|
| 194 |
+
d = v1b.get_data(R_KEY)
|
| 195 |
+
results = []
|
| 196 |
+
for arm in ARMS:
|
| 197 |
+
seeds = [0] if arm == "lens" else [0, 1]
|
| 198 |
+
for seed in seeds:
|
| 199 |
+
results.append(run_cell(arm, seed, d))
|
| 200 |
+
tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
|
| 201 |
+
api.upload_file(path_or_fileobj=f"{OUT_DIR}/{tag}.json",
|
| 202 |
+
path_in_repo=f"battery/v1d_svae_graft/{tag}.json",
|
| 203 |
+
repo_id=HUB_REPO, commit_message=f"V1d {tag}")
|
| 204 |
+
base = 0.6609
|
| 205 |
+
verdict = {}
|
| 206 |
+
for arm in ARMS:
|
| 207 |
+
accs = [r["final_acc"] for r in results if r["arm"] == arm]
|
| 208 |
+
verdict[arm] = {"acc": accs,
|
| 209 |
+
"mean": round(sum(accs) / len(accs), 4),
|
| 210 |
+
"delta_vs_base": round(sum(accs) / len(accs) - base, 4)}
|
| 211 |
+
print(f"{arm:>10}: {verdict[arm]['mean']:.4f} "
|
| 212 |
+
f"({verdict[arm]['delta_vs_base']:+.4f} vs baseline)", flush=True)
|
| 213 |
+
best = max(verdict, key=lambda a: verdict[a]["mean"])
|
| 214 |
+
live = verdict[best]["mean"] >= 0.80
|
| 215 |
+
obsolete = all(v["mean"] <= base + 0.02 for v in verdict.values())
|
| 216 |
+
verdict["gate"] = ("SVAE PIECES LIVE" if live else
|
| 217 |
+
"OBSOLETE HERE (absorbed)" if obsolete else
|
| 218 |
+
f"PARTIAL: best={best} {verdict[best]['mean']:.4f}")
|
| 219 |
+
print("GATE:", verdict["gate"], flush=True)
|
| 220 |
+
out = f"{OUT_DIR}/v1d_verdict.json"
|
| 221 |
+
json.dump(verdict, open(out, "w"), indent=1)
|
| 222 |
+
api.upload_file(path_or_fileobj=out,
|
| 223 |
+
path_in_repo="battery/v1d_svae_graft/v1d_verdict.json",
|
| 224 |
+
repo_id=HUB_REPO,
|
| 225 |
+
commit_message="V1d verdict: SVAE grafts vs binding gap")
|
| 226 |
+
print("DONE.", flush=True)
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
if __name__ == "__main__":
|
| 230 |
+
main()
|
trainers/v1e_knit_battery.py
ADDED
|
@@ -0,0 +1,250 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# v1e_knit_battery.py — V1e: the conv-knit + patchwork grafts vs the gap
|
| 3 |
+
# (2026-08-09; AbstractPhil: "Consult the alpha attention mixed with the SVAE
|
| 4 |
+
# patchwork's edge solidification from the conv mechanism to knit, that
|
| 5 |
+
# could be the answer we require.")
|
| 6 |
+
#
|
| 7 |
+
# Consult results (canon/constellation_forms.md universal rules +
|
| 8 |
+
# canon/svae_h2_class.md): patchwork spec Linear->SquaredReLU->LN->Linear,
|
| 9 |
+
# SquaredReLU NEVER GELU (anchor diversity 7.1 vs 4.6); conv solidifies
|
| 10 |
+
# edges between patch codes; gate init -3.0 (theme D). Bed point: R64
|
| 11 |
+
# recall, K=64, D=16 (baseline .661; V1c best K256 .728; sdpa .936).
|
| 12 |
+
# ARMS:
|
| 13 |
+
# knitseq — depthwise CAUSAL conv (kernel 8) over the HUB output,
|
| 14 |
+
# gated residual at -3.0: adjacent positions' read codes
|
| 15 |
+
# knit into a locally solid field (edge solidification in
|
| 16 |
+
# SEQUENCE).
|
| 17 |
+
# knitaddr — grouped conv (kernel 3) over the K-axis of the oriented
|
| 18 |
+
# address (qp/qn/kp/kn): NEIGHBORING CELLS couple — B6's
|
| 19 |
+
# overlap made structural (edge solidification in ADDRESS).
|
| 20 |
+
# patchhead — the HUB o-projection replaced by the constellation
|
| 21 |
+
# patchwork generator (Linear d->2d -> SquaredReLU -> LN ->
|
| 22 |
+
# Linear 2d->d): the read GENERATES from the address per the
|
| 23 |
+
# patchwork grammar.
|
| 24 |
+
# alphaknit — knitseq + boundgate alpha residual (the literal "alpha
|
| 25 |
+
# attention mixed with the edge-knit" ask).
|
| 26 |
+
# GATE (same as V1d): >= .80 mean = LIVE (half the remaining gap);
|
| 27 |
+
# within +.02 of .661 = the knit adds nothing here.
|
| 28 |
+
# Chained: waits for V1d's verdict file before starting (one GPU).
|
| 29 |
+
# RUN: python tools\v1e_knit_battery.py (HF_TOKEN, 4090)
|
| 30 |
+
# =========================================================================
|
| 31 |
+
|
| 32 |
+
import importlib
|
| 33 |
+
import json
|
| 34 |
+
import os
|
| 35 |
+
import sys
|
| 36 |
+
|
| 37 |
+
import torch
|
| 38 |
+
import torch.nn as nn
|
| 39 |
+
import torch.nn.functional as F
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def _load(name):
|
| 43 |
+
if name in sys.modules:
|
| 44 |
+
return sys.modules[name]
|
| 45 |
+
try:
|
| 46 |
+
return importlib.import_module(name)
|
| 47 |
+
except ImportError:
|
| 48 |
+
here = globals().get("__file__")
|
| 49 |
+
if here is not None:
|
| 50 |
+
import pathlib
|
| 51 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 52 |
+
return importlib.import_module(name)
|
| 53 |
+
raise
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
bed = _load("ar_differentiation_bed")
|
| 57 |
+
v1b = _load("v1b_recall_army_battery")
|
| 58 |
+
v1d = _load("v1d_svae_graft")
|
| 59 |
+
class _cb: # public shim: token from env
|
| 60 |
+
@staticmethod
|
| 61 |
+
def hf_token():
|
| 62 |
+
import os as _os
|
| 63 |
+
return _os.environ.get("HF_TOKEN")
|
| 64 |
+
cb = _cb()
|
| 65 |
+
from huggingface_hub import HfApi # noqa: E402
|
| 66 |
+
|
| 67 |
+
assert torch.cuda.is_available(), "GPU only"
|
| 68 |
+
DEV = "cuda"
|
| 69 |
+
STEPS = int(os.environ.get("V1E_STEPS", "8000"))
|
| 70 |
+
K_ARMY, D_ADDR, R_KEY = 64, 16, 64
|
| 71 |
+
ARMS = os.environ.get(
|
| 72 |
+
"V1E_ARMS", "knitseq,knitaddr,patchhead,alphaknit").split(",")
|
| 73 |
+
BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
|
| 74 |
+
OUT_DIR = os.path.join(v1b.DATA_DIR, "results_v1e")
|
| 75 |
+
HUB_REPO = v1b.HUB_REPO
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
class SquaredReLU(nn.Module):
|
| 79 |
+
def forward(self, x):
|
| 80 |
+
return F.relu(x) ** 2
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
class KnitHUB(bed.CausalHUB):
|
| 84 |
+
"""CausalHUB + the knit grafts."""
|
| 85 |
+
|
| 86 |
+
def __init__(self, d, K, D, knitaddr=False, patchhead=False):
|
| 87 |
+
super().__init__(d, K, D)
|
| 88 |
+
self.ka = None
|
| 89 |
+
if knitaddr:
|
| 90 |
+
# grouped conv over the K axis; SAME padding, shared across
|
| 91 |
+
# the four oriented maps; identity-init (weight peak at center)
|
| 92 |
+
self.ka = nn.Conv1d(1, 1, kernel_size=3, padding=1, bias=False)
|
| 93 |
+
w = torch.zeros(1, 1, 3)
|
| 94 |
+
w[0, 0, 1] = 1.0 # identity at birth
|
| 95 |
+
self.ka.weight = nn.Parameter(w)
|
| 96 |
+
if patchhead:
|
| 97 |
+
self.o = nn.Sequential( # constellation patchwork
|
| 98 |
+
nn.Linear(d, 2 * d), SquaredReLU(), nn.LayerNorm(2 * d),
|
| 99 |
+
nn.Linear(2 * d, d))
|
| 100 |
+
|
| 101 |
+
def _knit(self, a):
|
| 102 |
+
if self.ka is None:
|
| 103 |
+
return a
|
| 104 |
+
B, n, K = a.shape
|
| 105 |
+
return self.ka(a.reshape(B * n, 1, K)).reshape(B, n, K)
|
| 106 |
+
|
| 107 |
+
def forward(self, x):
|
| 108 |
+
qp, qn = self.addr.oriented(self.q(x))
|
| 109 |
+
kp, kn = self.addr.oriented(self.k(x))
|
| 110 |
+
qp, qn, kp, kn = (self._knit(t) for t in (qp, qn, kp, kn))
|
| 111 |
+
v = self.v(x)
|
| 112 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 113 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 114 |
+
zp = torch.cumsum(kp, dim=1)
|
| 115 |
+
zn = torch.cumsum(kn, dim=1)
|
| 116 |
+
num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
|
| 117 |
+
+ torch.einsum("bnk,bnkd->bnd", qn, Sn))
|
| 118 |
+
den = ((qp * zp).sum(-1, keepdim=True)
|
| 119 |
+
+ (qn * zn).sum(-1, keepdim=True))
|
| 120 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
class KnitBlock(nn.Module):
|
| 124 |
+
"""Block with optional sequence-knit (depthwise causal conv over the
|
| 125 |
+
attn output, gated at -3.0) and optional bounded-alpha residual."""
|
| 126 |
+
|
| 127 |
+
def __init__(self, d, attn, knitseq=False, boundgate=False):
|
| 128 |
+
super().__init__()
|
| 129 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 130 |
+
self.attn = attn
|
| 131 |
+
self.ks = None
|
| 132 |
+
if knitseq:
|
| 133 |
+
self.ks = nn.Conv1d(d, d, kernel_size=8, groups=d, bias=False)
|
| 134 |
+
self.ks_gate = nn.Parameter(torch.tensor(-3.0))
|
| 135 |
+
self.alpha = nn.Parameter(torch.tensor(0.024)) if boundgate else None
|
| 136 |
+
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),
|
| 137 |
+
nn.Linear(4 * d, d))
|
| 138 |
+
|
| 139 |
+
def forward(self, x):
|
| 140 |
+
a = self.attn(self.n1(x))
|
| 141 |
+
if self.ks is not None:
|
| 142 |
+
ap = F.pad(a.transpose(1, 2), (7, 0)) # causal left-pad
|
| 143 |
+
knit = self.ks(ap).transpose(1, 2)
|
| 144 |
+
a = a + torch.sigmoid(self.ks_gate) * knit
|
| 145 |
+
if self.alpha is not None:
|
| 146 |
+
x = x + self.alpha.clamp(0, 0.2) * a
|
| 147 |
+
else:
|
| 148 |
+
x = x + a
|
| 149 |
+
return x + self.mlp(self.n2(x))
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def build_model(arm, d=192, layers=4):
|
| 153 |
+
knitseq = arm in ("knitseq", "alphaknit")
|
| 154 |
+
knitaddr = arm == "knitaddr"
|
| 155 |
+
patchhead = arm == "patchhead"
|
| 156 |
+
boundgate = arm == "alphaknit"
|
| 157 |
+
m = bed.ByteLM("hub_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
|
| 158 |
+
m.blocks = nn.ModuleList([
|
| 159 |
+
KnitBlock(d, KnitHUB(d, K_ARMY, D_ADDR, knitaddr, patchhead),
|
| 160 |
+
knitseq, boundgate)
|
| 161 |
+
for _ in range(layers)])
|
| 162 |
+
return m
|
| 163 |
+
|
| 164 |
+
|
| 165 |
+
def run_cell(arm, seed, d):
|
| 166 |
+
tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
|
| 167 |
+
out_p = f"{OUT_DIR}/{tag}.json"
|
| 168 |
+
if os.path.exists(out_p):
|
| 169 |
+
print(f"-- {tag}: exists, skip", flush=True)
|
| 170 |
+
return json.load(open(out_p))
|
| 171 |
+
torch.manual_seed(seed)
|
| 172 |
+
g = torch.Generator().manual_seed(seed)
|
| 173 |
+
model = build_model(arm).to(DEV)
|
| 174 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 175 |
+
for step in range(1, STEPS + 1):
|
| 176 |
+
x, y, m = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
|
| 177 |
+
logits = model(x)
|
| 178 |
+
if m.sum() == 0:
|
| 179 |
+
continue
|
| 180 |
+
loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
|
| 181 |
+
+ 4.0 * F.cross_entropy(logits[m], y[m]))
|
| 182 |
+
opt.zero_grad(set_to_none=True)
|
| 183 |
+
loss.backward()
|
| 184 |
+
opt.step()
|
| 185 |
+
model.eval()
|
| 186 |
+
acc, bpb = v1b.masked_eval(model, d, g, iters=32)
|
| 187 |
+
gates = {}
|
| 188 |
+
b0 = model.blocks[0]
|
| 189 |
+
if b0.ks is not None:
|
| 190 |
+
gates["ks_gate_sig"] = [round(float(torch.sigmoid(b.ks_gate)), 4)
|
| 191 |
+
for b in model.blocks]
|
| 192 |
+
if b0.alpha is not None:
|
| 193 |
+
gates["alpha"] = [round(float(b.alpha), 4) for b in model.blocks]
|
| 194 |
+
res = {"arm": arm, "seed": seed, "R": R_KEY, "K": K_ARMY, "D": D_ADDR,
|
| 195 |
+
"steps": STEPS, "final_acc": round(acc, 4),
|
| 196 |
+
"final_bpb": round(bpb, 4), "gates": gates,
|
| 197 |
+
"baseline_ref": {"K64_D16_mean": 0.6609, "sdpa": 0.936}}
|
| 198 |
+
os.makedirs(OUT_DIR, exist_ok=True)
|
| 199 |
+
json.dump(res, open(out_p, "w"), indent=1)
|
| 200 |
+
print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f} {gates}", flush=True)
|
| 201 |
+
del model
|
| 202 |
+
torch.cuda.empty_cache()
|
| 203 |
+
return res
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
def main():
|
| 207 |
+
print("=" * 78, flush=True)
|
| 208 |
+
print(f"V1e KNIT BATTERY — arms {ARMS} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}, "
|
| 209 |
+
f"{STEPS} steps (baseline .661, sdpa .936)", flush=True)
|
| 210 |
+
print("=" * 78, flush=True)
|
| 211 |
+
api = HfApi(token=cb.hf_token())
|
| 212 |
+
d = v1b.get_data(R_KEY)
|
| 213 |
+
results = []
|
| 214 |
+
for arm in ARMS:
|
| 215 |
+
for seed in (0, 1):
|
| 216 |
+
results.append(run_cell(arm, seed, d))
|
| 217 |
+
tag = f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}" + (f"_t{STEPS // 1000}k" if STEPS != 8000 else "")
|
| 218 |
+
api.upload_file(path_or_fileobj=f"{OUT_DIR}/{tag}.json",
|
| 219 |
+
path_in_repo=f"battery/v1e_knit/{tag}.json",
|
| 220 |
+
repo_id=HUB_REPO, commit_message=f"V1e {tag}")
|
| 221 |
+
base = 0.6609
|
| 222 |
+
verdict = {}
|
| 223 |
+
for arm in ARMS:
|
| 224 |
+
accs = [r["final_acc"] for r in results if r["arm"] == arm]
|
| 225 |
+
verdict[arm] = {"acc": accs,
|
| 226 |
+
"mean": round(sum(accs) / len(accs), 4),
|
| 227 |
+
"delta_vs_base": round(sum(accs) / len(accs) - base,
|
| 228 |
+
4)}
|
| 229 |
+
print(f"{arm:>10}: {verdict[arm]['mean']:.4f} "
|
| 230 |
+
f"({verdict[arm]['delta_vs_base']:+.4f})", flush=True)
|
| 231 |
+
best = max(verdict, key=lambda a: verdict[a]["mean"])
|
| 232 |
+
verdict["gate"] = ("KNIT LIVE" if verdict[best]["mean"] >= 0.80 else
|
| 233 |
+
"KNIT NULL HERE" if all(
|
| 234 |
+
v["mean"] <= base + 0.02
|
| 235 |
+
for k, v in verdict.items() if k != "gate")
|
| 236 |
+
else f"PARTIAL: best={best} "
|
| 237 |
+
f"{verdict[best]['mean']:.4f}")
|
| 238 |
+
print("GATE:", verdict["gate"], flush=True)
|
| 239 |
+
out = f"{OUT_DIR}/v1e_verdict.json"
|
| 240 |
+
json.dump(verdict, open(out, "w"), indent=1)
|
| 241 |
+
api.upload_file(path_or_fileobj=out,
|
| 242 |
+
path_in_repo="battery/v1e_knit/v1e_verdict.json",
|
| 243 |
+
repo_id=HUB_REPO,
|
| 244 |
+
commit_message="V1e verdict: conv-knit + patchwork "
|
| 245 |
+
"grafts vs the binding gap")
|
| 246 |
+
print("DONE.", flush=True)
|
| 247 |
+
|
| 248 |
+
|
| 249 |
+
if __name__ == "__main__":
|
| 250 |
+
main()
|
trainers/v1f_consult_tier.py
ADDED
|
@@ -0,0 +1,280 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# v1f_consult_tier.py — V1f: the consult-informed corrector/control tier
|
| 3 |
+
# (2026-08-09 evening; sources: fresnel/johanna/freckles + H2-era + trigram
|
| 4 |
+
# consult, AbstractPhil's stabilizer-vs-corrector lineage, the boundgate crater.)
|
| 5 |
+
#
|
| 6 |
+
# Bed point: R64 recall, K=64, D=16 (baseline mean .661, sdpa .936).
|
| 7 |
+
# ARMS:
|
| 8 |
+
# rownorm — L-041 port (rated 9; the ONLY ablation that broke the
|
| 9 |
+
# 233-run battery): row sphere-norm at the M-point = the
|
| 10 |
+
# HUB's VALUE rows normalized at the write (v = ||v||-unit
|
| 11 |
+
# * learned scale). Directions|magnitudes decoupled.
|
| 12 |
+
# auxalpha — the graft-geometry CORRECTION of the boundgate crater
|
| 13 |
+
# (alphas pinned at cap, acc .036): bounded alpha on the
|
| 14 |
+
# AUXILIARY knit channel ONLY, base path unthrottled —
|
| 15 |
+
# the faithful SVAE whisper-channel geometry.
|
| 16 |
+
# strobe3 — the bed's own anti-selection refinement ported to the
|
| 17 |
+
# HUB: q/k oriented at 3 taus (.05/.1/.3) concatenated
|
| 18 |
+
# (3K channels) — neighborhood disambiguation vs the
|
| 19 |
+
# incorrect-selections class.
|
| 20 |
+
# unihub — unigram control (plain hub, NO trigram): does binding
|
| 21 |
+
# late-bloom without the tri lineage? (tri caveat, exp020
|
| 22 |
+
# memorizer artifact channel.)
|
| 23 |
+
# unisdpa — unigram sdpa reference, 1 seed.
|
| 24 |
+
# padfix — hub_tri with ALEPH_TRI_PAD_FIX=1: prices the pad-
|
| 25 |
+
# conflation fault (discovery #16) against the tri baseline.
|
| 26 |
+
# INSTRUMENTS ON EVERY CELL: CV@1000 (H2-era early-screening law: codebook
|
| 27 |
+
# CV at step 1000 predicted the final band in 96% of 149 runs — ported
|
| 28 |
+
# validation), selection_eval (mis-selection vs no-retrieval), final
|
| 29 |
+
# census, alpha/gate trajectories where present.
|
| 30 |
+
# RUN: python tools\v1f_consult_tier.py (HF_TOKEN, 4090; chains after V1e)
|
| 31 |
+
# Env: V1F_ARMS, V1F_STEPS (def 8000)
|
| 32 |
+
# =========================================================================
|
| 33 |
+
|
| 34 |
+
import importlib
|
| 35 |
+
import json
|
| 36 |
+
import os
|
| 37 |
+
import sys
|
| 38 |
+
|
| 39 |
+
import torch
|
| 40 |
+
import torch.nn as nn
|
| 41 |
+
import torch.nn.functional as F
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def _load(name):
|
| 45 |
+
if name in sys.modules:
|
| 46 |
+
return sys.modules[name]
|
| 47 |
+
try:
|
| 48 |
+
return importlib.import_module(name)
|
| 49 |
+
except ImportError:
|
| 50 |
+
here = globals().get("__file__")
|
| 51 |
+
if here is not None:
|
| 52 |
+
import pathlib
|
| 53 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 54 |
+
return importlib.import_module(name)
|
| 55 |
+
raise
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
bed = _load("ar_differentiation_bed")
|
| 59 |
+
v1b = _load("v1b_recall_army_battery")
|
| 60 |
+
class _cb: # public shim: token from env
|
| 61 |
+
@staticmethod
|
| 62 |
+
def hf_token():
|
| 63 |
+
import os as _os
|
| 64 |
+
return _os.environ.get("HF_TOKEN")
|
| 65 |
+
cb = _cb()
|
| 66 |
+
from huggingface_hub import HfApi # noqa: E402
|
| 67 |
+
|
| 68 |
+
assert torch.cuda.is_available(), "GPU only"
|
| 69 |
+
DEV = "cuda"
|
| 70 |
+
STEPS = int(os.environ.get("V1F_STEPS", "8000"))
|
| 71 |
+
K_ARMY, D_ADDR, R_KEY = 64, 16, 64
|
| 72 |
+
ARMS = os.environ.get(
|
| 73 |
+
"V1F_ARMS", "rownorm,auxalpha,strobe3,unihub,unisdpa,padfix").split(",")
|
| 74 |
+
BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
|
| 75 |
+
OUT_DIR = os.path.join(v1b.DATA_DIR, "results_v1f")
|
| 76 |
+
HUB_REPO = v1b.HUB_REPO
|
| 77 |
+
TAUS = (0.05, 0.1, 0.3)
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
class RowNormHUB(bed.CausalHUB):
|
| 81 |
+
"""L-041 at the M-point: value rows sphere-normalized at the write,
|
| 82 |
+
magnitude carried by a learned per-row scale (directions|magnitudes
|
| 83 |
+
decoupled, the H2 separation-of-concerns)."""
|
| 84 |
+
|
| 85 |
+
def __init__(self, d, K, D):
|
| 86 |
+
super().__init__(d, K, D)
|
| 87 |
+
self.vscale = nn.Parameter(torch.ones(()))
|
| 88 |
+
|
| 89 |
+
def forward(self, x):
|
| 90 |
+
qp, qn = self.addr.oriented(self.q(x))
|
| 91 |
+
kp, kn = self.addr.oriented(self.k(x))
|
| 92 |
+
v = F.normalize(self.v(x), dim=-1) * self.vscale
|
| 93 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 94 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 95 |
+
zp = torch.cumsum(kp, dim=1)
|
| 96 |
+
zn = torch.cumsum(kn, dim=1)
|
| 97 |
+
num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
|
| 98 |
+
+ torch.einsum("bnk,bnkd->bnd", qn, Sn))
|
| 99 |
+
den = ((qp * zp).sum(-1, keepdim=True)
|
| 100 |
+
+ (qn * zn).sum(-1, keepdim=True))
|
| 101 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
class Strobe3HUB(bed.CausalHUB):
|
| 105 |
+
"""3-tau stroboscope on the HUB address: oriented maps at three
|
| 106 |
+
temperatures concatenated along K (3K memory channels) — the bed's
|
| 107 |
+
v2 anti-selection refinement, ported to attention."""
|
| 108 |
+
|
| 109 |
+
def _strobe(self, x):
|
| 110 |
+
A = F.normalize(self.addr.codebook, dim=-1)
|
| 111 |
+
cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
|
| 112 |
+
ps, ns = [], []
|
| 113 |
+
for t in TAUS:
|
| 114 |
+
u = cos / t
|
| 115 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 116 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 117 |
+
Z = (ep + en).sum(dim=-1, keepdim=True)
|
| 118 |
+
ps.append(ep / Z)
|
| 119 |
+
ns.append(en / Z)
|
| 120 |
+
return torch.cat(ps, -1), torch.cat(ns, -1)
|
| 121 |
+
|
| 122 |
+
def forward(self, x):
|
| 123 |
+
qp, qn = self._strobe(self.q(x))
|
| 124 |
+
kp, kn = self._strobe(self.k(x))
|
| 125 |
+
v = self.v(x)
|
| 126 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 127 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 128 |
+
zp = torch.cumsum(kp, dim=1)
|
| 129 |
+
zn = torch.cumsum(kn, dim=1)
|
| 130 |
+
num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
|
| 131 |
+
+ torch.einsum("bnk,bnkd->bnd", qn, Sn))
|
| 132 |
+
den = ((qp * zp).sum(-1, keepdim=True)
|
| 133 |
+
+ (qn * zn).sum(-1, keepdim=True))
|
| 134 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
class AuxAlphaBlock(nn.Module):
|
| 138 |
+
"""The corrected graft geometry: base attn path UNTHROTTLED; the knit
|
| 139 |
+
conv rides an auxiliary bounded-alpha channel (init .024, cap .2) —
|
| 140 |
+
the SVAE whisper-channel form."""
|
| 141 |
+
|
| 142 |
+
def __init__(self, d, attn):
|
| 143 |
+
super().__init__()
|
| 144 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 145 |
+
self.attn = attn
|
| 146 |
+
self.ks = nn.Conv1d(d, d, kernel_size=8, groups=d, bias=False)
|
| 147 |
+
self.alpha = nn.Parameter(torch.tensor(0.024))
|
| 148 |
+
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(),
|
| 149 |
+
nn.Linear(4 * d, d))
|
| 150 |
+
|
| 151 |
+
def forward(self, x):
|
| 152 |
+
a = self.attn(self.n1(x))
|
| 153 |
+
knit = self.ks(F.pad(a.transpose(1, 2), (7, 0))).transpose(1, 2)
|
| 154 |
+
x = x + a + self.alpha.clamp(0, 0.2) * torch.tanh(knit)
|
| 155 |
+
return x + self.mlp(self.n2(x))
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def build_model(arm, d=192, layers=4):
|
| 159 |
+
base_arm = ("hub" if arm == "unihub"
|
| 160 |
+
else "sdpa" if arm == "unisdpa" else "hub_tri")
|
| 161 |
+
m = bed.ByteLM(base_arm, block=BLOCK, K=K_ARMY, D=D_ADDR)
|
| 162 |
+
if arm == "rownorm":
|
| 163 |
+
m.blocks = nn.ModuleList([
|
| 164 |
+
bed.Block(d, RowNormHUB(d, K_ARMY, D_ADDR)) for _ in range(layers)])
|
| 165 |
+
elif arm == "strobe3":
|
| 166 |
+
m.blocks = nn.ModuleList([
|
| 167 |
+
bed.Block(d, Strobe3HUB(d, K_ARMY, D_ADDR)) for _ in range(layers)])
|
| 168 |
+
elif arm == "auxalpha":
|
| 169 |
+
m.blocks = nn.ModuleList([
|
| 170 |
+
AuxAlphaBlock(d, bed.CausalHUB(d, K_ARMY, D_ADDR))
|
| 171 |
+
for _ in range(layers)])
|
| 172 |
+
return m
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
def codebook_cv(model):
|
| 176 |
+
"""H2-era CV gauge on the address codebooks (fp64, normalized rows):
|
| 177 |
+
coefficient of variation of pairwise cosine magnitudes."""
|
| 178 |
+
cvs = []
|
| 179 |
+
for blk in model.blocks:
|
| 180 |
+
addr = getattr(blk.attn, "addr", None)
|
| 181 |
+
if addr is None:
|
| 182 |
+
return None
|
| 183 |
+
A = F.normalize(addr.codebook.detach().double(), dim=-1)
|
| 184 |
+
G = (A @ A.T).abs()
|
| 185 |
+
off = G[~torch.eye(G.shape[0], dtype=torch.bool, device=G.device)]
|
| 186 |
+
cvs.append(float(off.std() / off.mean().clamp_min(1e-12)))
|
| 187 |
+
return round(sum(cvs) / len(cvs), 4)
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
def run_cell(arm, seed, d):
|
| 191 |
+
tag = (f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}"
|
| 192 |
+
+ (f"_t{STEPS // 1000}k" if STEPS != 8000 else ""))
|
| 193 |
+
out_p = f"{OUT_DIR}/{tag}.json"
|
| 194 |
+
if os.path.exists(out_p):
|
| 195 |
+
print(f"-- {tag}: exists, skip", flush=True)
|
| 196 |
+
return json.load(open(out_p))
|
| 197 |
+
if arm == "padfix":
|
| 198 |
+
os.environ["ALEPH_TRI_PAD_FIX"] = "1"
|
| 199 |
+
torch.manual_seed(seed)
|
| 200 |
+
g = torch.Generator().manual_seed(seed)
|
| 201 |
+
model = build_model(arm).to(DEV)
|
| 202 |
+
if arm == "padfix":
|
| 203 |
+
del os.environ["ALEPH_TRI_PAD_FIX"]
|
| 204 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 205 |
+
cv1000 = None
|
| 206 |
+
for step in range(1, STEPS + 1):
|
| 207 |
+
x, y, m = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
|
| 208 |
+
logits = model(x)
|
| 209 |
+
if m.sum() == 0:
|
| 210 |
+
continue
|
| 211 |
+
loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
|
| 212 |
+
+ 4.0 * F.cross_entropy(logits[m], y[m]))
|
| 213 |
+
opt.zero_grad(set_to_none=True)
|
| 214 |
+
loss.backward()
|
| 215 |
+
opt.step()
|
| 216 |
+
if step == 1000:
|
| 217 |
+
cv1000 = codebook_cv(model)
|
| 218 |
+
model.eval()
|
| 219 |
+
acc, bpb = v1b.masked_eval(model, d, g, iters=32)
|
| 220 |
+
sel = v1b.selection_eval(model, d, g, iters=32)
|
| 221 |
+
alphas = [round(float(b.alpha.detach()), 4) for b in model.blocks
|
| 222 |
+
if hasattr(b, "alpha")]
|
| 223 |
+
res = {"arm": arm, "seed": seed, "R": R_KEY, "K": K_ARMY, "D": D_ADDR,
|
| 224 |
+
"steps": STEPS, "final_acc": round(acc, 4),
|
| 225 |
+
"final_bpb": round(bpb, 4), "cv_at_1000": cv1000,
|
| 226 |
+
"cv_final": codebook_cv(model), "selection": sel,
|
| 227 |
+
"alphas": alphas,
|
| 228 |
+
"baseline_ref": {"K64_D16_mean": 0.6609, "sdpa_tri": 0.936}}
|
| 229 |
+
os.makedirs(OUT_DIR, exist_ok=True)
|
| 230 |
+
json.dump(res, open(out_p, "w"), indent=1)
|
| 231 |
+
print(f"-- {tag}: acc {acc:.4f} bpb {bpb:.4f} missel "
|
| 232 |
+
f"{sel['mis_sel_frac_of_errors']:.3f} cv1k {cv1000}"
|
| 233 |
+
+ (f" alphas {alphas}" if alphas else ""), flush=True)
|
| 234 |
+
del model
|
| 235 |
+
torch.cuda.empty_cache()
|
| 236 |
+
return res
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
def main():
|
| 240 |
+
print("=" * 78, flush=True)
|
| 241 |
+
print(f"V1f CONSULT TIER — arms {ARMS} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}, "
|
| 242 |
+
f"{STEPS} steps", flush=True)
|
| 243 |
+
print("=" * 78, flush=True)
|
| 244 |
+
api = HfApi(token=cb.hf_token())
|
| 245 |
+
d = v1b.get_data(R_KEY)
|
| 246 |
+
results = []
|
| 247 |
+
for arm in ARMS:
|
| 248 |
+
seeds = [0] if arm == "unisdpa" else [0, 1]
|
| 249 |
+
for seed in seeds:
|
| 250 |
+
results.append(run_cell(arm, seed, d))
|
| 251 |
+
tag = (f"{arm}_R{R_KEY}_K{K_ARMY}_D{D_ADDR}_s{seed}"
|
| 252 |
+
+ (f"_t{STEPS // 1000}k" if STEPS != 8000 else ""))
|
| 253 |
+
api.upload_file(path_or_fileobj=f"{OUT_DIR}/{tag}.json",
|
| 254 |
+
path_in_repo=f"battery/v1f_consult/{tag}.json",
|
| 255 |
+
repo_id=HUB_REPO, commit_message=f"V1f {tag}")
|
| 256 |
+
base = 0.6609
|
| 257 |
+
verdict = {}
|
| 258 |
+
for arm in ARMS:
|
| 259 |
+
rs = [r for r in results if r["arm"] == arm]
|
| 260 |
+
accs = [r["final_acc"] for r in rs]
|
| 261 |
+
ms = [r["selection"]["mis_sel_frac_of_errors"] for r in rs]
|
| 262 |
+
verdict[arm] = {"acc": accs,
|
| 263 |
+
"mean": round(sum(accs) / len(accs), 4),
|
| 264 |
+
"delta": round(sum(accs) / len(accs) - base, 4),
|
| 265 |
+
"missel_frac": round(sum(ms) / len(ms), 4)}
|
| 266 |
+
print(f"{arm:>9}: {verdict[arm]['mean']:.4f} "
|
| 267 |
+
f"({verdict[arm]['delta']:+.4f}) missel "
|
| 268 |
+
f"{verdict[arm]['missel_frac']:.3f}", flush=True)
|
| 269 |
+
out = f"{OUT_DIR}/v1f_verdict.json"
|
| 270 |
+
json.dump(verdict, open(out, "w"), indent=1)
|
| 271 |
+
api.upload_file(path_or_fileobj=out,
|
| 272 |
+
path_in_repo="battery/v1f_consult/v1f_verdict.json",
|
| 273 |
+
repo_id=HUB_REPO,
|
| 274 |
+
commit_message="V1f verdict: consult-tier arms + "
|
| 275 |
+
"mis-selection decomposition")
|
| 276 |
+
print("DONE.", flush=True)
|
| 277 |
+
|
| 278 |
+
|
| 279 |
+
if __name__ == "__main__":
|
| 280 |
+
main()
|
trainers/v1g_svae_overlap_sweep.py
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# v1g_svae_overlap_sweep.py — the 16-noise-spectrum overlap sweep on the
|
| 3 |
+
# NATIVE substrate (AbstractPhil, 2026-08-09: "you just need the correct noises
|
| 4 |
+
# that we trained johanna and freckles with" — the wikitext sweep pointed
|
| 5 |
+
# at the wrong beast). Also answers the STANDING open question
|
| 6 |
+
# (history/open_questions.md:191): the 16-codebook similarity matrix.
|
| 7 |
+
#
|
| 8 |
+
# HYPOTHESIS UNDER TEST (AbstractPhil, never verified): overlapping input spectra
|
| 9 |
+
# cause anchored/pooled accumulation — overlap-linked noise types should
|
| 10 |
+
# produce SIMILAR model responses (the model anchors on shared structure),
|
| 11 |
+
# and high-overlap types should converge slower / less differentiated.
|
| 12 |
+
#
|
| 13 |
+
# SUBSTRATE: OmegaNoiseDataset (16 types, verbatim from
|
| 14 |
+
# AbstractPhil/geolip-svae-batteries battery_maker_proto.py) + trained
|
| 15 |
+
# johanna-F-S32-V32-D4-h32-d1-p2 (PatchSVAE_F, all defense mechanisms).
|
| 16 |
+
# MEASURES:
|
| 17 |
+
# O[i,j] data-side overlap: radial power-spectrum cosine + histogram
|
| 18 |
+
# Bhattacharyya (averaged).
|
| 19 |
+
# S[i,j] model-side response similarity: per-type mean sigma (omega)
|
| 20 |
+
# profile cosine + per-mode sigma-distribution overlap.
|
| 21 |
+
# LINKAGE: corr(O_off, S_off) — overlap-driven anchoring verdict.
|
| 22 |
+
# DIFFICULTY: per-type recon MSE vs overlap centrality (slow-convergence
|
| 23 |
+
# link, cross-checked against the canon difficulty ordering).
|
| 24 |
+
# ALEPH EMERGENCE (native): antipodal row pairing in the per-sample M
|
| 25 |
+
# codebooks (RP^D axis signature, cos < -0.9), per type.
|
| 26 |
+
# RUN: python tools\v1g_svae_overlap_sweep.py (CPU, ~3 min)
|
| 27 |
+
# =========================================================================
|
| 28 |
+
|
| 29 |
+
import importlib.util
|
| 30 |
+
import json
|
| 31 |
+
import math
|
| 32 |
+
import os
|
| 33 |
+
import sys
|
| 34 |
+
|
| 35 |
+
import torch
|
| 36 |
+
import torch.nn.functional as F
|
| 37 |
+
|
| 38 |
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
| 39 |
+
from huggingface_hub import hf_hub_download, HfApi # noqa: E402
|
| 40 |
+
|
| 41 |
+
REPO = "AbstractPhil/geolip-svae-batteries"
|
| 42 |
+
RUN = "johanna-F-S32-V32-D4-h32-d1-p2"
|
| 43 |
+
N_PER_TYPE = 48
|
| 44 |
+
IMG = 32
|
| 45 |
+
OUT = "./data/v1_rank_army/v1g_svae_overlap.json"
|
| 46 |
+
SHIP_REPO = "AbstractPhil/aleph-splat-0"
|
| 47 |
+
|
| 48 |
+
TYPE_NAMES = ["gaussian", "uniform", "wide_uniform", "poisson", "pink",
|
| 49 |
+
"brown", "binary", "sparse_spikes", "block", "gradient",
|
| 50 |
+
"checker", "gauss_uniform_mix", "quadrant", "t13", "t14",
|
| 51 |
+
"t15"]
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def load_proto():
|
| 55 |
+
p = hf_hub_download(REPO, "battery_maker_proto.py")
|
| 56 |
+
spec = importlib.util.spec_from_file_location("battery_maker_proto", p)
|
| 57 |
+
mod = importlib.util.module_from_spec(spec)
|
| 58 |
+
sys.modules["battery_maker_proto"] = mod
|
| 59 |
+
spec.loader.exec_module(mod)
|
| 60 |
+
return mod
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def gen_type(mod, t, n):
|
| 64 |
+
torch.manual_seed(1234 + t)
|
| 65 |
+
ds = mod.OmegaNoiseDataset(size=n, img_size=IMG,
|
| 66 |
+
seed_rotate_every=10**9, allowed_types=[t])
|
| 67 |
+
return torch.stack([ds[i] if isinstance(ds[i], torch.Tensor)
|
| 68 |
+
else ds[i][0] for i in range(n)])
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def radial_spectrum(x):
|
| 72 |
+
"""Mean radially-averaged power spectrum over a batch (B,3,H,W)."""
|
| 73 |
+
S = torch.fft.rfft2(x.double()).abs() ** 2
|
| 74 |
+
S = S.mean(dim=(0, 1)) # (H, W//2+1)
|
| 75 |
+
h, w = S.shape
|
| 76 |
+
fy = torch.fft.fftfreq(h).abs().unsqueeze(-1).expand(-1, w)
|
| 77 |
+
fx = torch.fft.rfftfreq(IMG).unsqueeze(0).expand(h, -1)
|
| 78 |
+
r = torch.sqrt(fx ** 2 + fy ** 2)
|
| 79 |
+
nb = 16
|
| 80 |
+
bins = torch.linspace(0, float(r.max()) + 1e-6, nb + 1)
|
| 81 |
+
prof = torch.zeros(nb, dtype=torch.float64)
|
| 82 |
+
for b in range(nb):
|
| 83 |
+
m = (r >= bins[b]) & (r < bins[b + 1])
|
| 84 |
+
if m.any():
|
| 85 |
+
prof[b] = S[m].mean()
|
| 86 |
+
return (prof / prof.sum()).clamp_min(1e-12)
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def hist_overlap(a, b, nbins=64):
|
| 90 |
+
lo = float(min(a.min(), b.min()))
|
| 91 |
+
hi = float(max(a.max(), b.max()))
|
| 92 |
+
ha = torch.histc(a.double().flatten(), nbins, lo, hi)
|
| 93 |
+
hb = torch.histc(b.double().flatten(), nbins, lo, hi)
|
| 94 |
+
ha, hb = ha / ha.sum(), hb / hb.sum()
|
| 95 |
+
return float((ha * hb).sqrt().sum()) # Bhattacharyya
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def main():
|
| 99 |
+
print("=" * 74)
|
| 100 |
+
print("V1g SVAE 16-NOISE OVERLAP SWEEP (native substrate)")
|
| 101 |
+
print("=" * 74)
|
| 102 |
+
mod = load_proto()
|
| 103 |
+
cfgp = hf_hub_download(REPO, f"{RUN}/config.json")
|
| 104 |
+
cfg = json.load(open(cfgp))
|
| 105 |
+
model = mod.PatchSVAE_F(
|
| 106 |
+
matrix_v=cfg.get("matrix_v", 32), D=cfg.get("D", 4),
|
| 107 |
+
patch_size=cfg.get("patch_size", 2), hidden=cfg.get("hidden", 32),
|
| 108 |
+
depth=cfg.get("depth", 1),
|
| 109 |
+
n_cross_layers=cfg.get("n_cross_layers", 1),
|
| 110 |
+
n_heads=cfg.get("n_heads", 4))
|
| 111 |
+
ck = torch.load(hf_hub_download(REPO, f"{RUN}/checkpoints/best.pt"),
|
| 112 |
+
map_location="cpu", weights_only=False)
|
| 113 |
+
sd = ck.get("model_state_dict", ck.get("state_dict", ck))
|
| 114 |
+
model.load_state_dict(sd, strict=True)
|
| 115 |
+
model.eval()
|
| 116 |
+
print(f"loaded {RUN}: {sum(p.numel() for p in model.parameters()):,} "
|
| 117 |
+
f"params")
|
| 118 |
+
|
| 119 |
+
batches, spectra, sig_profiles, mse, antipodal = {}, {}, {}, {}, {}
|
| 120 |
+
for t in range(16):
|
| 121 |
+
x = gen_type(mod, t, N_PER_TYPE)
|
| 122 |
+
batches[t] = x
|
| 123 |
+
spectra[t] = radial_spectrum(x)
|
| 124 |
+
with torch.no_grad():
|
| 125 |
+
out = model(x)
|
| 126 |
+
mse[t] = round(float(F.mse_loss(out["recon"], x)), 5)
|
| 127 |
+
svd = out["svd"]
|
| 128 |
+
S = svd["S"] # (..., D) sigma
|
| 129 |
+
sig_profiles[t] = S.reshape(-1, S.shape[-1]).double()
|
| 130 |
+
M = svd.get("M", svd.get("M_norm"))
|
| 131 |
+
if M is not None:
|
| 132 |
+
Mn = F.normalize(M.reshape(-1, M.shape[-2], M.shape[-1])
|
| 133 |
+
.double(), dim=-1)
|
| 134 |
+
G = torch.einsum("bvd,bwd->bvw", Mn, Mn)
|
| 135 |
+
V = G.shape[-1]
|
| 136 |
+
iu = torch.triu_indices(V, V, offset=1)
|
| 137 |
+
pairs = G[:, iu[0], iu[1]]
|
| 138 |
+
antipodal[t] = round(float((pairs < -0.9).double().mean()), 5)
|
| 139 |
+
print(f" type {t:>2} {TYPE_NAMES[t]:>17}: mse {mse[t]:.5f}"
|
| 140 |
+
+ (f" antipodal {antipodal.get(t)}" if antipodal else ""),
|
| 141 |
+
flush=True)
|
| 142 |
+
|
| 143 |
+
# ---- data-side overlap O and model-side similarity S ----
|
| 144 |
+
Omat = torch.zeros(16, 16)
|
| 145 |
+
Smat = torch.zeros(16, 16)
|
| 146 |
+
for i in range(16):
|
| 147 |
+
for j in range(16):
|
| 148 |
+
if j < i:
|
| 149 |
+
Omat[i, j], Smat[i, j] = Omat[j, i], Smat[j, i]
|
| 150 |
+
continue
|
| 151 |
+
spec_cos = float(F.cosine_similarity(
|
| 152 |
+
spectra[i].unsqueeze(0), spectra[j].unsqueeze(0)))
|
| 153 |
+
hov = hist_overlap(batches[i], batches[j])
|
| 154 |
+
Omat[i, j] = (spec_cos + hov) / 2
|
| 155 |
+
mi = sig_profiles[i].mean(0)
|
| 156 |
+
mj = sig_profiles[j].mean(0)
|
| 157 |
+
prof_cos = float(F.cosine_similarity(mi.unsqueeze(0),
|
| 158 |
+
mj.unsqueeze(0)))
|
| 159 |
+
dov = sum(hist_overlap(sig_profiles[i][:, d],
|
| 160 |
+
sig_profiles[j][:, d])
|
| 161 |
+
for d in range(mi.shape[0])) / mi.shape[0]
|
| 162 |
+
Smat[i, j] = (prof_cos + dov) / 2
|
| 163 |
+
off = ~torch.eye(16, dtype=torch.bool)
|
| 164 |
+
Ov, Sv = Omat[off], Smat[off]
|
| 165 |
+
r_link = float(torch.corrcoef(torch.stack([Ov, Sv]))[0, 1])
|
| 166 |
+
centrality = Omat.sum(1) - 1
|
| 167 |
+
mset = torch.tensor([mse[t] for t in range(16)])
|
| 168 |
+
r_diff = float(torch.corrcoef(torch.stack(
|
| 169 |
+
[centrality.double(), mset.double()]))[0, 1])
|
| 170 |
+
ranks = lambda v: torch.argsort(torch.argsort(v)).double()
|
| 171 |
+
rho_link = float(torch.corrcoef(torch.stack(
|
| 172 |
+
[ranks(Ov), ranks(Sv)]))[0, 1])
|
| 173 |
+
|
| 174 |
+
print("\n== VERDICTS")
|
| 175 |
+
print(f" overlap->response linkage: pearson r={r_link:.3f} "
|
| 176 |
+
f"spearman rho={rho_link:.3f} (n={int(off.sum())} pairs)")
|
| 177 |
+
print(f" overlap-centrality vs difficulty (MSE): r={r_diff:.3f}")
|
| 178 |
+
if antipodal:
|
| 179 |
+
av = torch.tensor([antipodal[t] for t in range(16)])
|
| 180 |
+
print(f" antipodal (RP axis) pairing: mean {float(av.mean()):.4f} "
|
| 181 |
+
f"range {float(av.min()):.4f}-{float(av.max()):.4f}")
|
| 182 |
+
res = {"run": RUN, "n_per_type": N_PER_TYPE, "img": IMG,
|
| 183 |
+
"type_names": TYPE_NAMES,
|
| 184 |
+
"O_overlap": [[round(float(v), 4) for v in row] for row in Omat],
|
| 185 |
+
"S_response": [[round(float(v), 4) for v in row] for row in Smat],
|
| 186 |
+
"mse_per_type": mse, "antipodal_per_type": antipodal,
|
| 187 |
+
"linkage_pearson": round(r_link, 4),
|
| 188 |
+
"linkage_spearman": round(rho_link, 4),
|
| 189 |
+
"centrality_vs_mse_r": round(r_diff, 4)}
|
| 190 |
+
json.dump(res, open(OUT, "w"), indent=1)
|
| 191 |
+
api = HfApi()
|
| 192 |
+
api.upload_file(path_or_fileobj=OUT,
|
| 193 |
+
path_in_repo="battery/v1g_svae_overlap/v1g_sweep.json",
|
| 194 |
+
repo_id=SHIP_REPO,
|
| 195 |
+
commit_message="V1g: 16-noise overlap sweep on the "
|
| 196 |
+
"native SVAE substrate (overlap->response linkage + "
|
| 197 |
+
"16x16 matrices + antipodal census)")
|
| 198 |
+
print(f"saved -> {OUT} ; shipped -> {SHIP_REPO}")
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
if __name__ == "__main__":
|
| 202 |
+
main()
|
trainers/v1i_validity_batteries.py
ADDED
|
@@ -0,0 +1,400 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =========================================================================
|
| 2 |
+
# v1i_validity_batteries.py — AbstractPhil's validity campaign (2026-08-10):
|
| 3 |
+
# corruption/numerics/optimizer/schedule batteries against the four
|
| 4 |
+
# possibilities: (1) Adam may not capture the necessary information;
|
| 5 |
+
# (2) scheduling may be the solution or a no-op; (3) splat/aleph fragility
|
| 6 |
+
# vs standard attention; (4) unaccounted corruption.
|
| 7 |
+
#
|
| 8 |
+
# Bed point: R64 recall, K=64, D=16, ALEPH_TRI_PAD_FIX=1 (the repaired
|
| 9 |
+
# config; padfix baseline .696/.733 mean .714, sdpa_tri ref .936).
|
| 10 |
+
# MODES (env V1I_MODE):
|
| 11 |
+
# dtype — fp32 control / bf16 autocast / fp16 AMP / fp8sim (address-path
|
| 12 |
+
# e4m3 roundtrip: codebook + q/k projections quantized each
|
| 13 |
+
# forward) / fp64probe (2k-step numeric-truth probe).
|
| 14 |
+
# AbstractPhil: "is the computable range of differentiation beyond fp32,
|
| 15 |
+
# can it be represented as less data (fp8), test middlegrounds."
|
| 16 |
+
# opt — adam(control)/sgd_mom/lion/muon/adagrad x {hub, sdpa}.
|
| 17 |
+
# PHIL-AUTHORIZED DEVIATION from the pure-Adam rider (screens
|
| 18 |
+
# only, labeled; AdamW remains excluded per the hard law).
|
| 19 |
+
# Fair-comparison discipline: 300-step LR probe over
|
| 20 |
+
# {3e-4,1e-3,3e-3,1e-2} per optimizer/arm before the real run.
|
| 21 |
+
# Fragility gauge (possibility 3): cross-optimizer outcome
|
| 22 |
+
# variance per architecture.
|
| 23 |
+
# sched — on hub+adam: warmcos / geomfirst (addr-group fast->slow per
|
| 24 |
+
# A1 frame-lock, trunk-group slow->fast per A2 late-accrual —
|
| 25 |
+
# the math-aligned schedule) / inverse (labeled falsification
|
| 26 |
+
# control). Flat control = the padfix cells.
|
| 27 |
+
# Gauges everywhere: answer acc, bpb, selection_eval decomposition,
|
| 28 |
+
# den-clamp fraction (numeric-health), census.
|
| 29 |
+
# RUN: V1I_MODE=dtype|opt|sched python tools\v1i_validity_batteries.py
|
| 30 |
+
# =========================================================================
|
| 31 |
+
|
| 32 |
+
import importlib
|
| 33 |
+
import json
|
| 34 |
+
import math
|
| 35 |
+
import os
|
| 36 |
+
import sys
|
| 37 |
+
|
| 38 |
+
import torch
|
| 39 |
+
import torch.nn.functional as F
|
| 40 |
+
|
| 41 |
+
os.environ["ALEPH_TRI_PAD_FIX"] = "1" # standing order (V1f)
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def _load(name):
|
| 45 |
+
if name in sys.modules:
|
| 46 |
+
return sys.modules[name]
|
| 47 |
+
try:
|
| 48 |
+
return importlib.import_module(name)
|
| 49 |
+
except ImportError:
|
| 50 |
+
here = globals().get("__file__")
|
| 51 |
+
if here is not None:
|
| 52 |
+
import pathlib
|
| 53 |
+
sys.path.insert(0, str(pathlib.Path(here).parent))
|
| 54 |
+
return importlib.import_module(name)
|
| 55 |
+
raise
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
bed = _load("ar_differentiation_bed")
|
| 59 |
+
v1b = _load("v1b_recall_army_battery")
|
| 60 |
+
class _cb: # public shim: token from env
|
| 61 |
+
@staticmethod
|
| 62 |
+
def hf_token():
|
| 63 |
+
import os as _os
|
| 64 |
+
return _os.environ.get("HF_TOKEN")
|
| 65 |
+
cb = _cb()
|
| 66 |
+
from huggingface_hub import HfApi # noqa: E402
|
| 67 |
+
|
| 68 |
+
assert torch.cuda.is_available(), "GPU only"
|
| 69 |
+
DEV = "cuda"
|
| 70 |
+
MODE = os.environ.get("V1I_MODE", "dtype")
|
| 71 |
+
STEPS = int(os.environ.get("V1I_STEPS", "8000"))
|
| 72 |
+
K_ARMY, D_ADDR, R_KEY = 64, 16, 64
|
| 73 |
+
BATCH, BLOCK = v1b.BATCH, v1b.BLOCK
|
| 74 |
+
DATA_DIR = "./data/v1_rank_army_padfix"
|
| 75 |
+
OUT_DIR = f"{DATA_DIR}/results_v1i_{MODE}"
|
| 76 |
+
HUB_REPO = v1b.HUB_REPO
|
| 77 |
+
BASELINE = 0.7143 # padfix K64/D16 mean
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def get_data():
|
| 81 |
+
os.environ["V1B_DATA"] = DATA_DIR
|
| 82 |
+
v1b.DATA_DIR = DATA_DIR
|
| 83 |
+
return v1b.get_data(R_KEY)
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
# ---------------------------------------------------------------- optimizers
|
| 87 |
+
class Lion(torch.optim.Optimizer):
|
| 88 |
+
"""Sign-momentum (pattern-aligned). lr ~3-10x smaller than Adam's."""
|
| 89 |
+
|
| 90 |
+
def __init__(self, params, lr=1e-4, betas=(0.9, 0.99)):
|
| 91 |
+
super().__init__(params, dict(lr=lr, betas=betas))
|
| 92 |
+
|
| 93 |
+
@torch.no_grad()
|
| 94 |
+
def step(self):
|
| 95 |
+
for g in self.param_groups:
|
| 96 |
+
b1, b2 = g["betas"]
|
| 97 |
+
for p in g["params"]:
|
| 98 |
+
if p.grad is None:
|
| 99 |
+
continue
|
| 100 |
+
st = self.state.setdefault(p, {})
|
| 101 |
+
m = st.setdefault("m", torch.zeros_like(p))
|
| 102 |
+
p.add_(torch.sign(m.mul(b1).add(p.grad, alpha=1 - b1)),
|
| 103 |
+
alpha=-g["lr"])
|
| 104 |
+
m.mul_(b2).add_(p.grad, alpha=1 - b2)
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def _ns_orth(G, steps=5):
|
| 108 |
+
"""Newton-Schulz orthogonalization (Muon core), fp32."""
|
| 109 |
+
a, b, c = 3.4445, -4.7750, 2.0315
|
| 110 |
+
X = G.float()
|
| 111 |
+
X = X / (X.norm() + 1e-7)
|
| 112 |
+
tall = X.shape[0] > X.shape[1]
|
| 113 |
+
if tall:
|
| 114 |
+
X = X.T
|
| 115 |
+
for _ in range(steps):
|
| 116 |
+
A = X @ X.T
|
| 117 |
+
X = a * X + (b * A + c * A @ A) @ X
|
| 118 |
+
return (X.T if tall else X).to(G.dtype)
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
class Muon(torch.optim.Optimizer):
|
| 122 |
+
"""Orthogonalized momentum on 2D weights (geometrically aligned);
|
| 123 |
+
Adam fallback for non-2D params handled by the caller."""
|
| 124 |
+
|
| 125 |
+
def __init__(self, params, lr=2e-2, momentum=0.95):
|
| 126 |
+
super().__init__(params, dict(lr=lr, momentum=momentum))
|
| 127 |
+
|
| 128 |
+
@torch.no_grad()
|
| 129 |
+
def step(self):
|
| 130 |
+
for g in self.param_groups:
|
| 131 |
+
for p in g["params"]:
|
| 132 |
+
if p.grad is None or p.ndim != 2:
|
| 133 |
+
continue
|
| 134 |
+
st = self.state.setdefault(p, {})
|
| 135 |
+
m = st.setdefault("m", torch.zeros_like(p))
|
| 136 |
+
m.mul_(g["momentum"]).add_(p.grad)
|
| 137 |
+
p.add_(_ns_orth(m), alpha=-g["lr"])
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
def make_opt(name, model, lr):
|
| 141 |
+
ps = list(model.parameters())
|
| 142 |
+
if name == "adam":
|
| 143 |
+
return [torch.optim.Adam(ps, lr=lr, weight_decay=0.0)]
|
| 144 |
+
if name == "sgd_mom":
|
| 145 |
+
return [torch.optim.SGD(ps, lr=lr, momentum=0.9)]
|
| 146 |
+
if name == "adagrad":
|
| 147 |
+
return [torch.optim.Adagrad(ps, lr=lr)]
|
| 148 |
+
if name == "lion":
|
| 149 |
+
return [Lion(ps, lr=lr / 3)]
|
| 150 |
+
if name == "muon":
|
| 151 |
+
two = [p for p in ps if p.ndim == 2]
|
| 152 |
+
rest = [p for p in ps if p.ndim != 2]
|
| 153 |
+
return [Muon(two, lr=lr * 20),
|
| 154 |
+
torch.optim.Adam(rest, lr=lr, weight_decay=0.0)]
|
| 155 |
+
raise ValueError(name)
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def lr_probe(name, arm, d, cands=(3e-4, 1e-3, 3e-3, 1e-2)):
|
| 159 |
+
best, best_loss = cands[0], float("inf")
|
| 160 |
+
for lr in cands:
|
| 161 |
+
torch.manual_seed(0)
|
| 162 |
+
g = torch.Generator().manual_seed(0)
|
| 163 |
+
m = bed.ByteLM(arm, block=BLOCK, K=K_ARMY, D=D_ADDR).to(DEV)
|
| 164 |
+
opts = make_opt(name, m, lr)
|
| 165 |
+
tot = 0.0
|
| 166 |
+
ok = True
|
| 167 |
+
for step in range(300):
|
| 168 |
+
x, y, mk = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK,
|
| 169 |
+
DEV, g)
|
| 170 |
+
logits = m(x)
|
| 171 |
+
loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
|
| 172 |
+
+ 4.0 * F.cross_entropy(logits[mk], y[mk]))
|
| 173 |
+
if not torch.isfinite(loss):
|
| 174 |
+
ok = False
|
| 175 |
+
break
|
| 176 |
+
for o in opts:
|
| 177 |
+
o.zero_grad(set_to_none=True)
|
| 178 |
+
loss.backward()
|
| 179 |
+
for o in opts:
|
| 180 |
+
o.step()
|
| 181 |
+
if step >= 200:
|
| 182 |
+
tot += float(loss)
|
| 183 |
+
del m
|
| 184 |
+
torch.cuda.empty_cache()
|
| 185 |
+
if ok and tot < best_loss:
|
| 186 |
+
best_loss, best = tot, lr
|
| 187 |
+
return best
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
# ---------------------------------------------------------------- fp8 sim
|
| 191 |
+
def fp8_roundtrip(t):
|
| 192 |
+
if hasattr(torch, "float8_e4m3fn"):
|
| 193 |
+
return t.to(torch.float8_e4m3fn).to(t.dtype)
|
| 194 |
+
scale = t.abs().amax().clamp_min(1e-8) / 448.0
|
| 195 |
+
return ((t / scale).clamp(-448, 448).round() * scale)
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
class FP8AddrHUB(bed.CausalHUB):
|
| 199 |
+
"""Address path through an e4m3 bottleneck each forward: does the
|
| 200 |
+
differentiation structure survive 8-bit representation?"""
|
| 201 |
+
|
| 202 |
+
def forward(self, x):
|
| 203 |
+
q8 = fp8_roundtrip(self.q(x))
|
| 204 |
+
k8 = fp8_roundtrip(self.k(x))
|
| 205 |
+
A = self.addr.codebook.data
|
| 206 |
+
self.addr.codebook.data = fp8_roundtrip(A)
|
| 207 |
+
try:
|
| 208 |
+
qp, qn = self.addr.oriented(q8)
|
| 209 |
+
kp, kn = self.addr.oriented(k8)
|
| 210 |
+
finally:
|
| 211 |
+
self.addr.codebook.data = A
|
| 212 |
+
v = self.v(x)
|
| 213 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 214 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 215 |
+
zp = torch.cumsum(kp, dim=1)
|
| 216 |
+
zn = torch.cumsum(kn, dim=1)
|
| 217 |
+
num = (torch.einsum("bnk,bnkd->bnd", qp, Sp)
|
| 218 |
+
+ torch.einsum("bnk,bnkd->bnd", qn, Sn))
|
| 219 |
+
den = ((qp * zp).sum(-1, keepdim=True)
|
| 220 |
+
+ (qn * zn).sum(-1, keepdim=True))
|
| 221 |
+
self._den_min = float(den.abs().min())
|
| 222 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 223 |
+
|
| 224 |
+
|
| 225 |
+
def den_health(model, d, g):
|
| 226 |
+
"""Fraction of read positions whose denominator sits near the clamp."""
|
| 227 |
+
xv, _, _ = v1b.batch_masked(d["va"], d["vam"], BATCH, BLOCK, DEV, g)
|
| 228 |
+
fracs = []
|
| 229 |
+
with torch.no_grad():
|
| 230 |
+
x = model.emb(xv)
|
| 231 |
+
if getattr(model, "trigram", False):
|
| 232 |
+
pv = 256 if getattr(model, "pad_fix", False) else 0
|
| 233 |
+
x = x + model.emb1(F.pad(xv, (1, 0), value=pv)[:, :-1]) \
|
| 234 |
+
+ model.emb2(F.pad(xv, (2, 0), value=pv)[:, :-2])
|
| 235 |
+
x = x + model.pos[:, : xv.shape[1]]
|
| 236 |
+
for blk in model.blocks:
|
| 237 |
+
h = blk.n1(x)
|
| 238 |
+
at = blk.attn
|
| 239 |
+
if not hasattr(at, "addr"):
|
| 240 |
+
return None
|
| 241 |
+
qp, qn = at.addr.oriented(at.q(h))
|
| 242 |
+
kp, kn = at.addr.oriented(at.k(h))
|
| 243 |
+
zp, zn = torch.cumsum(kp, 1), torch.cumsum(kn, 1)
|
| 244 |
+
den = ((qp * zp).sum(-1) + (qn * zn).sum(-1))
|
| 245 |
+
fracs.append(float((den.abs() < 1e-6).float().mean()))
|
| 246 |
+
x = blk(x)
|
| 247 |
+
return round(sum(fracs) / len(fracs), 6)
|
| 248 |
+
|
| 249 |
+
|
| 250 |
+
# ---------------------------------------------------------------- schedules
|
| 251 |
+
def sched_lr(kind, group, step, total, base=3e-4):
|
| 252 |
+
t = step / total
|
| 253 |
+
if kind == "warmcos":
|
| 254 |
+
if t < 0.05:
|
| 255 |
+
return base * t / 0.05
|
| 256 |
+
return base * 0.5 * (1 + math.cos(math.pi * (t - .05) / .95))
|
| 257 |
+
if kind == "geomfirst": # math-aligned (A1+A2)
|
| 258 |
+
if group == "addr": # frame locks early
|
| 259 |
+
return 3e-3 * math.exp(-6 * t) + 3e-5
|
| 260 |
+
return 1e-4 + (5e-4 - 1e-4) * t # function accrues late
|
| 261 |
+
if kind == "inverse": # labeled falsification
|
| 262 |
+
if group == "addr":
|
| 263 |
+
return 3e-5 + (3e-3 - 3e-5) * t
|
| 264 |
+
return 5e-4 * math.exp(-6 * t) + 1e-4
|
| 265 |
+
return base
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
def addr_params(model):
|
| 269 |
+
ids = set()
|
| 270 |
+
for blk in model.blocks:
|
| 271 |
+
at = blk.attn
|
| 272 |
+
for p in (list(at.addr.parameters()) + list(at.q.parameters())
|
| 273 |
+
+ list(at.k.parameters())):
|
| 274 |
+
ids.add(id(p))
|
| 275 |
+
return ids
|
| 276 |
+
|
| 277 |
+
|
| 278 |
+
# ---------------------------------------------------------------- cells
|
| 279 |
+
def run_cell(tag, build_fn, opt_fn, seed, d, autocast_dtype=None,
|
| 280 |
+
sched_kind=None):
|
| 281 |
+
out_p = f"{OUT_DIR}/{tag}_s{seed}.json"
|
| 282 |
+
if os.path.exists(out_p):
|
| 283 |
+
print(f"-- {tag}_s{seed}: exists, skip", flush=True)
|
| 284 |
+
return json.load(open(out_p))
|
| 285 |
+
torch.manual_seed(seed)
|
| 286 |
+
g = torch.Generator().manual_seed(seed)
|
| 287 |
+
model = build_fn().to(DEV)
|
| 288 |
+
opts = opt_fn(model)
|
| 289 |
+
aset = addr_params(model) if sched_kind else None
|
| 290 |
+
scaler = (torch.amp.GradScaler("cuda")
|
| 291 |
+
if autocast_dtype == torch.float16 else None)
|
| 292 |
+
for step in range(1, STEPS + 1):
|
| 293 |
+
x, y, mk = v1b.batch_masked(d["tr"], d["trm"], BATCH, BLOCK, DEV, g)
|
| 294 |
+
if autocast_dtype is not None:
|
| 295 |
+
with torch.autocast("cuda", dtype=autocast_dtype):
|
| 296 |
+
logits = model(x)
|
| 297 |
+
loss = (F.cross_entropy(logits.reshape(-1, 256).float(),
|
| 298 |
+
y.reshape(-1))
|
| 299 |
+
+ 4.0 * F.cross_entropy(logits[mk].float(), y[mk]))
|
| 300 |
+
else:
|
| 301 |
+
logits = model(x)
|
| 302 |
+
loss = (F.cross_entropy(logits.reshape(-1, 256), y.reshape(-1))
|
| 303 |
+
+ 4.0 * F.cross_entropy(logits[mk], y[mk]))
|
| 304 |
+
for o in opts:
|
| 305 |
+
o.zero_grad(set_to_none=True)
|
| 306 |
+
if scaler:
|
| 307 |
+
scaler.scale(loss).backward()
|
| 308 |
+
for o in opts:
|
| 309 |
+
scaler.step(o)
|
| 310 |
+
scaler.update()
|
| 311 |
+
else:
|
| 312 |
+
loss.backward()
|
| 313 |
+
for o in opts:
|
| 314 |
+
o.step()
|
| 315 |
+
if sched_kind:
|
| 316 |
+
for o in opts:
|
| 317 |
+
for grp in o.param_groups:
|
| 318 |
+
is_addr = all(id(p) in aset for p in grp["params"]) \
|
| 319 |
+
if grp["params"] else False
|
| 320 |
+
grp["lr"] = sched_lr(sched_kind,
|
| 321 |
+
"addr" if is_addr else "trunk",
|
| 322 |
+
step, STEPS)
|
| 323 |
+
model.eval()
|
| 324 |
+
acc, bpb = v1b.masked_eval(model, d, g, iters=32)
|
| 325 |
+
sel = v1b.selection_eval(model, d, g, iters=32)
|
| 326 |
+
dh = den_health(model, d, g)
|
| 327 |
+
res = {"tag": tag, "seed": seed, "steps": STEPS, "mode": MODE,
|
| 328 |
+
"final_acc": round(acc, 4), "final_bpb": round(bpb, 4),
|
| 329 |
+
"selection": sel, "den_near_clamp_frac": dh,
|
| 330 |
+
"baseline_padfix": BASELINE}
|
| 331 |
+
os.makedirs(OUT_DIR, exist_ok=True)
|
| 332 |
+
json.dump(res, open(out_p, "w"), indent=1)
|
| 333 |
+
print(f"-- {tag}_s{seed}: acc {acc:.4f} bpb {bpb:.4f} "
|
| 334 |
+
f"missel {sel['mis_sel_frac_of_errors']:.3f} den {dh}", flush=True)
|
| 335 |
+
del model
|
| 336 |
+
torch.cuda.empty_cache()
|
| 337 |
+
return res
|
| 338 |
+
|
| 339 |
+
|
| 340 |
+
def main():
|
| 341 |
+
print("=" * 78, flush=True)
|
| 342 |
+
print(f"V1i VALIDITY BATTERY mode={MODE} at R{R_KEY}/K{K_ARMY}/D{D_ADDR}"
|
| 343 |
+
f" padfix (baseline {BASELINE}, sdpa .936)", flush=True)
|
| 344 |
+
print("=" * 78, flush=True)
|
| 345 |
+
api = HfApi(token=cb.hf_token())
|
| 346 |
+
d = get_data()
|
| 347 |
+
results = []
|
| 348 |
+
mk_hub = lambda: bed.ByteLM("hub_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
|
| 349 |
+
mk_sdpa = lambda: bed.ByteLM("sdpa_tri", block=BLOCK, K=K_ARMY, D=D_ADDR)
|
| 350 |
+
adam = lambda m: make_opt("adam", m, 3e-4)
|
| 351 |
+
|
| 352 |
+
if MODE == "dtype":
|
| 353 |
+
def mk_fp8():
|
| 354 |
+
m = mk_hub()
|
| 355 |
+
m.blocks = torch.nn.ModuleList(
|
| 356 |
+
[bed.Block(192, FP8AddrHUB(192, K_ARMY, D_ADDR))
|
| 357 |
+
for _ in range(4)])
|
| 358 |
+
return m
|
| 359 |
+
cells = [("fp32", mk_hub, None), ("bf16", mk_hub, torch.bfloat16),
|
| 360 |
+
("fp16", mk_hub, torch.float16), ("fp8addr", mk_fp8, None)]
|
| 361 |
+
for tag, fn, ac in cells:
|
| 362 |
+
for seed in (0, 1):
|
| 363 |
+
results.append(run_cell(tag, fn, adam, seed, d,
|
| 364 |
+
autocast_dtype=ac))
|
| 365 |
+
elif MODE == "opt":
|
| 366 |
+
for name in ("adam", "sgd_mom", "lion", "muon", "adagrad"):
|
| 367 |
+
for arm, mk, seeds in (("hub", mk_hub, (0, 1)),
|
| 368 |
+
("sdpa", mk_sdpa, (0,))):
|
| 369 |
+
lr = 3e-4 if name == "adam" else lr_probe(name,
|
| 370 |
+
f"{arm}_tri", d)
|
| 371 |
+
print(f" [{name}/{arm}] lr={lr}", flush=True)
|
| 372 |
+
for seed in seeds:
|
| 373 |
+
results.append(run_cell(
|
| 374 |
+
f"{name}_{arm}", mk,
|
| 375 |
+
lambda m, n=name, l=lr: make_opt(n, m, l),
|
| 376 |
+
seed, d))
|
| 377 |
+
elif MODE == "sched":
|
| 378 |
+
for kind in ("warmcos", "geomfirst", "inverse"):
|
| 379 |
+
for seed in (0, 1):
|
| 380 |
+
results.append(run_cell(f"sched_{kind}", mk_hub, adam,
|
| 381 |
+
seed, d, sched_kind=kind))
|
| 382 |
+
# ---- summarize + ship ----
|
| 383 |
+
summary = {}
|
| 384 |
+
for r in results:
|
| 385 |
+
summary.setdefault(r["tag"], []).append(r["final_acc"])
|
| 386 |
+
for tag, accs in summary.items():
|
| 387 |
+
mean = sum(accs) / len(accs)
|
| 388 |
+
print(f"{tag:>14}: {mean:.4f} ({mean - BASELINE:+.4f})", flush=True)
|
| 389 |
+
out = f"{OUT_DIR}/v1i_{MODE}_verdict.json"
|
| 390 |
+
json.dump({t: {"acc": a, "mean": round(sum(a) / len(a), 4)}
|
| 391 |
+
for t, a in summary.items()}, open(out, "w"), indent=1)
|
| 392 |
+
api.upload_file(path_or_fileobj=out,
|
| 393 |
+
path_in_repo=f"battery/v1i_validity/{MODE}_verdict.json",
|
| 394 |
+
repo_id=HUB_REPO,
|
| 395 |
+
commit_message=f"V1i {MODE} battery verdict")
|
| 396 |
+
print("DONE.", flush=True)
|
| 397 |
+
|
| 398 |
+
|
| 399 |
+
if __name__ == "__main__":
|
| 400 |
+
main()
|