mini-beatrix-1 pre-classroom annealment point: AutoModel-compatible (trust_remote_code), surgered 58,664 weights (head gate folded, semantic no-op), config from manifest, parity 0.00e+00 vs native stack, generate() verified

README.md

@@ -1,3 +1,70 @@
 ---
 license: mit
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - byte-level
+  - tokenizer-free
+  - aleph
+  - signed-address
+  - custom_code
 ---
+
+# mini-beatrix-1 — pretrain annealment point (pre-classroom)
+
+The locked pretrain+anneal state of **mini-beatrix-1**, a 112.5M-parameter
+**byte-level** AlephLM: step **58,664**, **17.301B bytes** seen
+(0.3B wikitext warmup · 15B fineweb-edu · 2B anneal mix), fineweb-holdout
+val **1.045 bits/byte**. This checkpoint is the fixed departure point for
+the staged "early-life curriculum" — later classroom checkpoints live in
+the [training repo](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training).
+
+No tokenizer: she reads raw UTF-8 bytes (`input_ids` = byte values
+0–255). Each position composes a byte trigram (dedicated pad row), so
+"tokens" are learned inside the network. Sixteen pre-norm layers where
+routing uses **signed geometric addresses** — `sinh/Σcosh` dispatch over
+learned unit anchors, inhibition as a first-class citizen, no softmax-
+over-choices, no top-k, no balance losses. Each layer carries an
+anchored FFN bank born contributing exactly zero; layers 4/9/14 use a
+linear-cost address read (CausalSplatHUB) instead of softmax attention.
+Both elected themselves into load-bearing work: at this checkpoint,
+removing the banks costs **+2.25 bpb**, removing the hub attention
+**+3.73 bpb** (toggle ledger, fineweb holdout). The dual head's aleph
+read is present with its gate folded to 1.0 (a verified semantic no-op,
+max|logit diff| 2.4e-07) and contributes 0.0000 bpb here — it is the
+live subject of the head-election experiment in the classroom phase.
+
+## Use
+
+```python
+import torch
+from transformers import AutoModelForCausalLM
+
+m = AutoModelForCausalLM.from_pretrained(
+    "AbstractPhil/mini-beatrix-1", trust_remote_code=True).eval()
+
+ids = torch.tensor([list("The history of mathematics begins".encode())])
+out = m.generate(ids, max_new_tokens=96, do_sample=True,
+                 temperature=0.7, top_p=0.95)
+print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))
+```
+
+Bits-per-byte on your own text: pass `labels=input_ids` (HF shift
+semantics are internal) and divide the returned loss (nats/byte) by
+`ln 2`. No KV cache in this wrapper — generation recomputes the prefix
+each step; for cached decode use the native stack below.
+
+## Honest notes
+
+- The 2B anneal mix included dialogue in her chat template and a small
+  identity texture, so **the bare model chats and knows her name** —
+  behavior we have since ruled OUT of core corpora (conditioning belongs
+  in detachable arms; see the [amoe-lora](https://github.com/AbstractEyes/amoe-lora)
+  arm system and `mini-beatrix-1/arms/` in the training repo).
+- Small and early: conversational in shape, thin on knowledge,
+  confidently wrong at times. Curriculum probe baselines (P0–P8),
+  toggle ledgers, and lexicon-census reports for this exact checkpoint
+  are in the training repo under `mini-beatrix-1/reports/`.
+
+*Code:* [github.com/AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm) ·
+*talk to her:* [alephllm-chat](https://huggingface.co/spaces/AbstractPhil/alephllm-chat)

address.py

@@ -0,0 +1,84 @@
+"""AlephAddress — the closed-form signed address over 2K oriented half-axes.
+
+The mechanism is reconstructive, never comparative: no softmax-over-choices,
+no argmax, no top-k anywhere. Dispatch weights are
+    w_k = sinh(u_k) / sum_j cosh(u_j),   u_k = cos(x_hat, a_hat_k) / tau
+which is exactly the signed difference of the two halves of a 2K-softmax
+over oriented axes (+a_k, -a_k). Inhibition (negative w) is first-class.
+"""
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+def dtype_floor(t: torch.Tensor) -> float:
+    """Dtype-aware clamp floor. Half dtypes flush 1e-12 to zero — the
+    measured fp16 landmine; use a floor the dtype can actually represent."""
+    if t.dtype in (torch.float32, torch.float64):
+        return 1e-12
+    return float(torch.finfo(t.dtype).tiny) * 8
+
+
+class AlephAddress(nn.Module):
+    """K unit anchors in D dims, cosine-read at temperature tau.
+
+    signed(x)  -> (..., K)          w_k = sinh(u_k)/sum_j cosh(u_j)
+    oriented(x)-> ((..., K), (..., K))  the two positive halves of the
+                                    2K-softmax (ep/Z, en/Z); HUB feature map.
+    """
+
+    def __init__(self, K: int, D: int, tau: float = 0.1):
+        super().__init__()
+        self.K, self.D, self.tau = K, D, tau
+        self.codebook = nn.Parameter(F.normalize(torch.randn(K, D), dim=-1))
+        self.register_buffer("home", self.codebook.detach().clone())
+
+    def _u(self, x: torch.Tensor) -> torch.Tensor:
+        A = F.normalize(self.codebook, dim=-1)
+        return (F.normalize(x, dim=-1) @ A.transpose(-1, -2)) / self.tau
+
+    def oriented(self, x: torch.Tensor):
+        u = self._u(x)
+        m = u.abs().amax(dim=-1, keepdim=True)
+        ep, en = torch.exp(u - m), torch.exp(-u - m)
+        Z = (ep + en).sum(dim=-1, keepdim=True)
+        return ep / Z, en / Z
+
+    def signed(self, x: torch.Tensor) -> torch.Tensor:
+        u = self._u(x)
+        m = u.abs().amax(dim=-1, keepdim=True)
+        ep, en = torch.exp(u - m), torch.exp(-u - m)
+        return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
+
+    @torch.no_grad()
+    def health(self, x_sample: torch.Tensor) -> dict:
+        """Codebook + consumption vitals for the instrument suite."""
+        A = F.normalize(self.codebook.float(), dim=-1)
+        gram = A @ A.T
+        off = gram - torch.eye(self.K, device=gram.device)
+        drift = 1.0 - F.cosine_similarity(
+            A, F.normalize(self.home.float(), dim=-1), dim=-1)
+        s = torch.linalg.svdvals(A)
+        ps = (s * s) / (s * s).sum().clamp_min(1e-12)
+        out = {
+            "anchor_max_abs_cos": off.abs().max().item(),
+            "anchor_merge_pairs": int((off.abs() > 0.99).sum().item() // 2),
+            "drift_mean": drift.mean().item(),
+            "drift_max": drift.max().item(),
+            # frame health: effective rank of the codebook itself — a
+            # collapsing frame (anchors folding into a subspace) shows here
+            "codebook_erank": float(
+                torch.exp(-(ps * ps.clamp_min(1e-12).log()).sum()).item()),
+        }
+        if x_sample is not None:
+            p, n = self.oriented(x_sample.reshape(-1, x_sample.shape[-1]).float())
+            mass = torch.cat([p, n], dim=-1).mean(0)          # (2K,) mean usage
+            mass = mass / mass.sum().clamp_min(1e-12)
+            ent = -(mass * mass.clamp_min(1e-12).log()).sum()
+            out["usage_ppl"] = float(ent.exp().item())         # of 2K half-axes
+            out["usage_cv"] = float((mass.std() / mass.mean().clamp_min(1e-12)).item())
+            w = self.signed(x_sample.reshape(-1, x_sample.shape[-1]).float())
+            out["sign_frac_neg"] = float((w < 0).float().mean().item())
+        return out

alephlm.py

@@ -0,0 +1,192 @@
+"""AlephLM — the full craft, config-driven.
+
+Trigram byte (or BPE) embedding -> pre-norm stack (CausalSDPA majority,
+CausalSplatHUB at the configured depths) -> LayerNorm -> DualHead.
+
+Toggle surface (the causal contribution ledger, run at every eval):
+    forward(idx, disable_bank=True)   dispatched experts off (exact C6 null)
+    forward(idx, disable_hub=True)    hub attention residuals skipped
+    forward(idx, disable_head_aleph=True)  gamma path off
+"""
+from __future__ import annotations
+
+from typing import NamedTuple, Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class LMOutput(NamedTuple):
+    """Still a tuple — `logits, loss = model(x)` keeps working — but also
+    HF-duck-typed (`out.logits`, `out.loss`) so frozen-trunk tooling like
+    amoe-lora drives the model natively."""
+    logits: torch.Tensor
+    loss: Optional[torch.Tensor]
+
+from .presets import AlephLMConfig
+from .attention import CausalSDPA, CausalSplatHUB
+from .bank import AnchoredBank
+from .embedding import TrigramByteEmbedding, TokenEmbedding
+from .head import DualHead
+
+
+class Block(nn.Module):
+    def __init__(self, cfg: AlephLMConfig, layer_idx: int):
+        super().__init__()
+        d = cfg.d_model
+        self.is_hub = layer_idx in cfg.hub_layers
+        self.n1 = nn.LayerNorm(d)
+        self.n2 = nn.LayerNorm(d)
+        if self.is_hub:
+            self.attn = CausalSplatHUB(d, cfg.hub_K, cfg.hub_D, cfg.tau,
+                                       chunk=cfg.hub_chunk)
+        else:
+            self.attn = CausalSDPA(d, cfg.n_heads)
+        self.bank = AnchoredBank(d, cfg.bank_experts, cfg.bank_ff, cfg.tau,
+                                 cfg.gate_init)
+
+    def forward(self, x, disable_bank=False, disable_hub=False):
+        if not (disable_hub and self.is_hub):
+            x = x + self.attn(self.n1(x))
+        return x + self.bank(self.n2(x), disable_dispatch=disable_bank)
+
+    def prefill(self, x):
+        a, cache = self.attn.prefill(self.n1(x))
+        x = x + a
+        return x + self.bank(self.n2(x)), cache
+
+    def step(self, x_t, cache):
+        x_t = x_t + self.attn.step(self.n1(x_t), cache)
+        return x_t + self.bank(self.n2(x_t))
+
+
+class AlephLM(nn.Module):
+    def __init__(self, cfg: AlephLMConfig):
+        super().__init__()
+        self.cfg = cfg
+        if cfg.tokenizer == "byte-trigram":
+            assert cfg.vocab_size == 256, "byte crafts use vocab 256"
+            self.embed = TrigramByteEmbedding(cfg.d_model, cfg.context)
+            tied = None
+        else:
+            self.embed = TokenEmbedding(cfg.vocab_size, cfg.d_model, cfg.context)
+            tied = self.embed.emb.weight if cfg.tie_embeddings else None
+        self.blocks = nn.ModuleList(
+            Block(cfg, i) for i in range(cfg.n_layers))
+        self.nf = nn.LayerNorm(cfg.d_model)
+        self.head = DualHead(cfg.d_model, cfg.vocab_size, cfg.head_K,
+                             cfg.head_D, cfg.tau, tied_weight=tied)
+
+    def forward(self, idx=None, targets=None, disable_bank=False,
+                disable_hub=False, disable_head_aleph=False,
+                input_ids=None, labels=None, attention_mask=None):
+        """HF-style aliases are accepted so frozen-trunk tooling drives the
+        model unchanged, WITH HF semantics: `labels` are same-position and
+        shifted internally (logits[:-1] vs labels[1:]); `targets` are the
+        package's own pre-shifted convention and used as-is. attention_mask
+        is deliberately ignored: under causal attention with right-padding
+        and -100 label masking, pads can never influence a scored position."""
+        if idx is None:
+            idx = input_ids
+        x = self.embed(idx)
+        for b in self.blocks:
+            x = b(x, disable_bank=disable_bank, disable_hub=disable_hub)
+        h = self.nf(x)
+        logits = self.head(h, disable_aleph=disable_head_aleph)
+        if targets is not None:                     # pre-shifted (ours)
+            loss = F.cross_entropy(
+                logits.reshape(-1, logits.shape[-1]).float(),
+                targets.reshape(-1), ignore_index=-100)
+        elif labels is not None:                    # HF: shift internally
+            loss = F.cross_entropy(
+                logits[:, :-1].reshape(-1, logits.shape[-1]).float(),
+                labels[:, 1:].reshape(-1), ignore_index=-100)
+        else:
+            return LMOutput(logits, None)
+        return LMOutput(logits, loss)
+
+    # ---------------------------------------------------- incremental decode
+    @torch.no_grad()
+    def prefill(self, idx):
+        """Run the prompt once, return (last-position logits, decode cache).
+        The cache carries per-layer attention state, the trigram history
+        bytes, and the absolute position cursor."""
+        self.eval()
+        from .embedding import PAD_ROW
+        caches = []
+        x = self.embed(idx)
+        for b in self.blocks:
+            x, c = b.prefill(x)
+            caches.append(c)
+        h = self.nf(x)
+        logits = self.head(h[:, -1:])
+        n = idx.shape[1]
+        prev2 = idx[:, -2] if n >= 2 else torch.full_like(idx[:, -1], PAD_ROW)
+        return logits, {"layers": caches, "t": n,
+                        "prev1": idx[:, -1], "prev2": prev2}
+
+    @torch.no_grad()
+    def decode_step(self, next_id, cache):
+        """One token through the cached path. next_id: (B,) or (B,1)."""
+        next_id = next_id.reshape(-1)
+        t = cache["t"]
+        assert t < self.cfg.context, "decode exceeded the position table"
+        if isinstance(self.embed, TrigramByteEmbedding):
+            e = (self.embed.emb0(next_id) + self.embed.emb1(cache["prev1"])
+                 + self.embed.emb2(cache["prev2"])).unsqueeze(1) \
+                + self.embed.pos[:, t:t + 1]
+            cache["prev2"] = cache["prev1"]
+            cache["prev1"] = next_id
+        else:
+            e = self.embed.emb(next_id).unsqueeze(1) + self.embed.pos[:, t:t + 1]
+        x = e
+        for b, c in zip(self.blocks, cache["layers"]):
+            x = b.step(x, c)
+        cache["t"] = t + 1
+        return self.head(self.nf(x))
+
+    @staticmethod
+    def _sample(logits, temperature, top_p):
+        logits = logits[:, -1].float()
+        if temperature <= 0.02:
+            return logits.argmax(-1, keepdim=True)
+        probs = F.softmax(logits / temperature, dim=-1)
+        sp, si = probs.sort(dim=-1, descending=True)
+        keep = (sp.cumsum(-1) - sp) < top_p
+        keep[..., :1] = True   # top-1 always survives: top_p<=0 must never
+        sp = sp * keep         # yield an all-zero row (CUDA multinomial on
+        return si.gather(-1, torch.multinomial(  # zeros poisons the context)
+            sp / sp.sum(-1, keepdim=True), 1))
+
+    @torch.no_grad()
+    def generate(self, idx, max_new: int = 128, temperature: float = 1.0,
+                 top_p: float = 0.95, use_cache: bool = True):
+        """Cached decode while the sequence fits the position table; any
+        remainder (long prompts, fills past the context) continues through
+        the sliding-window parallel path — the hub's constant-size state
+        cannot evict, so sliding continuation must recompute."""
+        self.eval()
+        ctx = self.cfg.context
+        if use_cache and idx.shape[1] < ctx and max_new > 0:
+            n_cached = min(max_new, ctx - idx.shape[1])
+            logits, cache = self.prefill(idx)
+            for i in range(n_cached):
+                nxt = self._sample(logits, temperature, top_p)
+                idx = torch.cat([idx, nxt], dim=1)
+                if i + 1 < n_cached:
+                    logits = self.decode_step(nxt, cache)
+            max_new -= n_cached
+        for _ in range(max_new):
+            logits, _ = self(idx[:, -ctx:])
+            nxt = self._sample(logits, temperature, top_p)
+            idx = torch.cat([idx, nxt], dim=1)
+        return idx
+
+    def param_count(self) -> int:
+        seen, total = set(), 0
+        for p in self.parameters():
+            if id(p) not in seen:
+                seen.add(id(p))
+                total += p.numel()
+        return total

attention.py

@@ -0,0 +1,154 @@
+"""Attention blocks: CausalSDPA (the workhorse) and CausalSplatHUB (the
+instrumented aleph read).
+
+CausalSplatHUB is causal linear attention through the oriented address:
+prefix-sum memories over the two K-wide halves of the 2K softmax, read by
+the query's halves and normalized by the scalar agreement mass. O(n·K·d)
+compute, no softmax over positions, no selection event anywhere.
+
+The naive cumsum form materializes (B, n, K, d) — fine on probe beds,
+fatal at mission scale. forward() therefore uses an exact chunked scan:
+within-chunk causal affinity (B, C, C) + cross-chunk carried states
+(B, K, d). `forward_naive()` is kept verbatim as the equivalence oracle
+for the test array.
+"""
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from .address import AlephAddress, dtype_floor
+
+
+class CausalSDPA(nn.Module):
+    def __init__(self, d: int, heads: int = 8):
+        super().__init__()
+        assert d % heads == 0
+        self.h = heads
+        self.qkv = nn.Linear(d, 3 * d, bias=False)
+        self.o = nn.Linear(d, d, bias=False)
+        nn.init.orthogonal_(self.qkv.weight)
+        nn.init.orthogonal_(self.o.weight)
+
+    def forward(self, x):
+        B, n, d = x.shape
+        q, k, v = self.qkv(x).chunk(3, dim=-1)
+        q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2)
+                   for t in (q, k, v))
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        return self.o(y.transpose(1, 2).reshape(B, n, d))
+
+    # ---------------------------------------------------- incremental decode
+    def prefill(self, x):
+        """Full causal pass that also returns the decode cache (K/V)."""
+        B, n, d = x.shape
+        q, k, v = self.qkv(x).chunk(3, dim=-1)
+        q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2)
+                   for t in (q, k, v))
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        return self.o(y.transpose(1, 2).reshape(B, n, d)), {"k": k, "v": v}
+
+    def step(self, x_t, cache):
+        """One new position attending over everything cached (KV cache)."""
+        B, _, d = x_t.shape
+        q, k, v = self.qkv(x_t).chunk(3, dim=-1)
+        q, k, v = (t.view(B, 1, self.h, d // self.h).transpose(1, 2)
+                   for t in (q, k, v))
+        cache["k"] = torch.cat([cache["k"], k], dim=2)
+        cache["v"] = torch.cat([cache["v"], v], dim=2)
+        y = F.scaled_dot_product_attention(q, cache["k"], cache["v"])
+        return self.o(y.transpose(1, 2).reshape(B, 1, d))
+
+
+class CausalSplatHUB(nn.Module):
+    def __init__(self, d: int, K: int = 512, D: int = 32, tau: float = 0.1,
+                 chunk: int = 128):
+        super().__init__()
+        self.addr = AlephAddress(K, D, tau)
+        self.chunk = chunk
+        self.q = nn.Linear(d, D, bias=False)
+        self.k = nn.Linear(d, D, bias=False)
+        self.v = nn.Linear(d, d, bias=False)
+        self.o = nn.Linear(d, d, bias=False)
+        for m in (self.q, self.k, self.v, self.o):
+            nn.init.orthogonal_(m.weight)
+        self.last_den_stats = None  # populated each forward for instruments
+
+    def _halves(self, x):
+        qp, qn = self.addr.oriented(self.q(x))
+        kp, kn = self.addr.oriented(self.k(x))
+        return qp, qn, kp, kn, self.v(x)
+
+    def forward(self, x):
+        B, n, d = x.shape
+        qp, qn, kp, kn, v = self._halves(x)
+        C = min(self.chunk, n)
+        pad = (-n) % C
+        if pad:
+            z = lambda t: F.pad(t, (0, 0, 0, pad))
+            qp, qn, kp, kn, v = z(qp), z(qn), z(kp), z(kn), z(v)
+        nc = (n + pad) // C
+        K = qp.shape[-1]
+        qp, qn, kp, kn = (t.view(B, nc, C, K) for t in (qp, qn, kp, kn))
+        v = v.view(B, nc, C, d)
+        mask = torch.tril(torch.ones(C, C, device=x.device, dtype=v.dtype))
+
+        num = torch.zeros(B, nc, C, d, device=x.device, dtype=v.dtype)
+        den = torch.zeros(B, nc, C, 1, device=x.device, dtype=v.dtype)
+        for kh, qh in ((kp, qp), (kn, qn)):
+            S = torch.einsum("bick,bicd->bikd", kh, v)      # per-chunk KxD sums
+            P = torch.cumsum(S, dim=1) - S                   # exclusive prefix
+            zS = kh.sum(dim=2)                               # (B, nc, K)
+            zP = torch.cumsum(zS, dim=1) - zS
+            att = torch.einsum("bick,bijk->bicj", qh, kh) * mask   # (B,nc,C,C)
+            num = num + torch.einsum("bick,bikd->bicd", qh, P) + att @ v
+            den = den + torch.einsum("bick,bik->bic", qh, zP).unsqueeze(-1) \
+                      + att.sum(dim=-1, keepdim=True)
+        num = num.view(B, nc * C, d)[:, :n]
+        den = den.view(B, nc * C, 1)[:, :n]
+        cl = dtype_floor(den)
+        with torch.no_grad():
+            self.last_den_stats = (den.min().item(), den.mean().item(),
+                                   (den <= cl).float().mean().item())
+        return self.o(num / den.clamp_min(cl))
+
+    # ---------------------------------------------------- incremental decode
+    def prefill(self, x):
+        """Full causal pass plus the decode cache. The hub's cache is the
+        CONSTANT-SIZE prefix state (Sp, Sn, zp, zn) — O(K·d) regardless of
+        sequence length; this is the linear-attention decode advantage."""
+        out = self.forward(x)
+        qp, qn, kp, kn, v = self._halves(x)
+        cache = {"Sp": torch.einsum("bnk,bnd->bkd", kp, v),
+                 "Sn": torch.einsum("bnk,bnd->bkd", kn, v),
+                 "zp": kp.sum(dim=1), "zn": kn.sum(dim=1)}
+        return out, cache
+
+    def step(self, x_t, cache):
+        """One new position: fold it into the prefix state, read once."""
+        qp, qn, kp, kn, v = self._halves(x_t)          # (B,1,K)/(B,1,d)
+        kp1, kn1, v1 = kp.squeeze(1), kn.squeeze(1), v.squeeze(1)
+        cache["Sp"] = cache["Sp"] + kp1.unsqueeze(-1) * v1.unsqueeze(1)
+        cache["Sn"] = cache["Sn"] + kn1.unsqueeze(-1) * v1.unsqueeze(1)
+        cache["zp"] = cache["zp"] + kp1
+        cache["zn"] = cache["zn"] + kn1
+        qp1, qn1 = qp.squeeze(1), qn.squeeze(1)
+        num = torch.einsum("bk,bkd->bd", qp1, cache["Sp"]) \
+            + torch.einsum("bk,bkd->bd", qn1, cache["Sn"])
+        den = ((qp1 * cache["zp"]).sum(-1)
+               + (qn1 * cache["zn"]).sum(-1)).unsqueeze(-1)
+        return self.o((num / den.clamp_min(dtype_floor(den))).unsqueeze(1))
+
+    def forward_naive(self, x):
+        """Reference cumsum form (the validated probe-bed implementation).
+        O(n·K·d) memory — test oracle only."""
+        qp, qn, kp, kn, v = self._halves(x)
+        Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
+        Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
+        zp = torch.cumsum(kp, dim=1)
+        zn = torch.cumsum(kn, dim=1)
+        num = torch.einsum("bnk,bnkd->bnd", qp, Sp) \
+            + torch.einsum("bnk,bnkd->bnd", qn, Sn)
+        den = (qp * zp).sum(-1, keepdim=True) + (qn * zn).sum(-1, keepdim=True)
+        return self.o(num / den.clamp_min(dtype_floor(den)))

bank.py

@@ -0,0 +1,51 @@
+"""AnchoredBank — the E1-form anchored FFN, born on its own null path.
+
+    Bank(x) = trunk(x) + sum_k w_k(x) * sigmoid(g_k) * expert_k(x)
+
+Trunk: always-on d->ff->d GELU expert. Dispatch: the signed aleph address
+over a learned K x d codebook read against the layer input (K=3 fat
+experts, the shape validated at parity under encoder pressure). Gates
+init -3.0; expert OUTPUT projections zero-init, so at birth the dispatch
+contributes exactly zero and the bank is bit-identical to its dense
+control (the C6 null path). No balance machinery of any kind —
+differentiation is an attractor, pressure stays out of the task gradient.
+"""
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from .address import AlephAddress
+
+
+class AnchoredBank(nn.Module):
+    def __init__(self, d: int, n_experts: int = 3, ff: int | None = None,
+                 tau: float = 0.1, gate_init: float = -3.0):
+        super().__init__()
+        ff = ff or d
+        self.n_experts = n_experts
+        self.t_in = nn.Linear(d, ff, bias=False)
+        self.t_out = nn.Linear(ff, d, bias=False)
+        nn.init.orthogonal_(self.t_in.weight)
+        nn.init.orthogonal_(self.t_out.weight)
+        self.addr = AlephAddress(n_experts, d, tau)
+        w_in = torch.empty(n_experts, d, ff)
+        for k in range(n_experts):
+            nn.init.orthogonal_(w_in[k])
+        self.w_in = nn.Parameter(w_in)
+        self.w_out = nn.Parameter(torch.zeros(n_experts, ff, d))  # null path
+        self.gates = nn.Parameter(torch.full((n_experts,), gate_init))
+        self.last_dispatch = None  # (mean|w| per expert, w sample) for instruments
+
+    def forward(self, x, disable_dispatch: bool = False):
+        trunk = self.t_out(F.gelu(self.t_in(x)))
+        if disable_dispatch:
+            return trunk
+        w = self.addr.signed(x)                                   # (B, n, K)
+        with torch.no_grad():
+            self.last_dispatch = w.detach()
+        h = F.gelu(torch.einsum("bnd,kdf->bnkf", x, self.w_in))
+        e = torch.einsum("bnkf,kfd->bnkd", h, self.w_out)
+        return trunk + torch.einsum("bnk,bnkd->bnd",
+                                    w * torch.sigmoid(self.gates), e)

config.json

@@ -0,0 +1,35 @@
+{
+ "name": "mini-beatrix-1",
+ "d_model": 768,
+ "n_layers": 16,
+ "n_heads": 12,
+ "context": 2048,
+ "vocab_size": 256,
+ "tokenizer": "byte-trigram",
+ "hub_layers": [
+  4,
+  9,
+  14
+ ],
+ "hub_K": 512,
+ "hub_D": 32,
+ "tau": 0.1,
+ "bank_experts": 3,
+ "bank_ff": null,
+ "head_K": 512,
+ "head_D": 32,
+ "gate_init": -3.0,
+ "tie_embeddings": false,
+ "hub_chunk": 128,
+ "model_type": "mini-beatrix",
+ "architectures": [
+  "MiniBeatrixForCausalLM"
+ ],
+ "auto_map": {
+  "AutoConfig": "modeling_minibeatrix.MiniBeatrixConfig",
+  "AutoModelForCausalLM": "modeling_minibeatrix.MiniBeatrixForCausalLM"
+ },
+ "torch_dtype": "float32",
+ "use_cache": false,
+ "transformers_version": "5.13.0"
+}

embedding.py

@@ -0,0 +1,44 @@
+"""Embeddings.
+
+TrigramByteEmbedding — the validated composed byte embedding:
+    e_t = E0[x_t] + E1[x_{t-1}] + E2[x_{t-2}] + P[t]
+with the PAD LAW built in permanently: the shift tables carry a dedicated
+pad row (index 256). Padding trigram shifts with a legal byte conflates
+real history with sequence starts and starves address consumption
+(measured +.05..+.11 on repair) — the fix ships on, not opt-in.
+
+TokenEmbedding — plain table + positions for BPE crafts.
+"""
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+BYTE_VOCAB = 256
+PAD_ROW = 256  # dedicated pad index in the shift tables (size 257)
+
+
+class TrigramByteEmbedding(nn.Module):
+    def __init__(self, d: int, context: int):
+        super().__init__()
+        self.emb0 = nn.Embedding(BYTE_VOCAB, d)
+        self.emb1 = nn.Embedding(BYTE_VOCAB + 1, d)   # + pad row
+        self.emb2 = nn.Embedding(BYTE_VOCAB + 1, d)
+        self.pos = nn.Parameter(0.01 * torch.randn(1, context, d))
+
+    def forward(self, idx):
+        x = self.emb0(idx) \
+            + self.emb1(F.pad(idx, (1, 0), value=PAD_ROW)[:, :-1]) \
+            + self.emb2(F.pad(idx, (2, 0), value=PAD_ROW)[:, :-2])
+        return x + self.pos[:, : idx.shape[1]]
+
+
+class TokenEmbedding(nn.Module):
+    def __init__(self, vocab: int, d: int, context: int):
+        super().__init__()
+        self.emb = nn.Embedding(vocab, d)
+        self.pos = nn.Parameter(0.01 * torch.randn(1, context, d))
+
+    def forward(self, idx):
+        return self.emb(idx) + self.pos[:, : idx.shape[1]]

head.py

@@ -0,0 +1,51 @@
+"""DualHead — the standard readout plus the L-012 aleph read, born null.
+
+    logits = W_h h + W_s s(h),        W_s == 0 at init
+
+s(h) is the signed address of a learned low-D projection of h through a
+K_h-anchor head codebook. New structure enters at zero and earns its way
+in by gradient — via WEIGHT-zero init, never gate-zero: mini-beatrix-1
+measured (8.4B tokens) that a gamma-gated branch DEADLOCKS (W_s grad is
+scaled by gamma=0, gamma grad through a random frozen readout is
+zero-mean noise), while the bank's E_out weight-zero self-started into
++2 bpb of function. gamma remains as a frozen scalar for checkpoint
+compatibility and as the ablation knob; the election gauge is ||W_s||
+and the head_aleph_off toggle.
+"""
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+
+from .address import AlephAddress
+
+
+class DualHead(nn.Module):
+    def __init__(self, d: int, vocab: int, K: int = 512, D: int = 32,
+                 tau: float = 0.1, tied_weight: nn.Parameter | None = None):
+        super().__init__()
+        self.tied = tied_weight is not None
+        if self.tied:
+            # tuple wrapper: reference the embedding table WITHOUT registering
+            # it as a duplicate parameter (state_dict must stay alias-free
+            # for safetensors export)
+            self._tied_ref = (tied_weight,)
+            self.bias = nn.Parameter(torch.zeros(vocab))
+        else:
+            self.w_h = nn.Linear(d, vocab, bias=True)
+        self.proj = nn.Linear(d, D, bias=False)
+        nn.init.orthogonal_(self.proj.weight)
+        self.addr = AlephAddress(K, D, tau)
+        self.w_s = nn.Linear(K, vocab, bias=False)
+        nn.init.zeros_(self.w_s.weight)          # weight-zero: self-starting
+        self.gamma = nn.Parameter(torch.ones(1))  # frozen; ablation knob only
+        self.gamma.requires_grad_(False)
+
+    def forward(self, h, disable_aleph: bool = False):
+        if self.tied:
+            base = h @ self._tied_ref[0].T + self.bias
+        else:
+            base = self.w_h(h)
+        if disable_aleph:
+            return base
+        return base + self.gamma * self.w_s(self.addr.signed(self.proj(h)))

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55d927c2b45e038d889a9bb15b3c794838b2032258c5a630be59f03d2f9fd1c8
+size 450531044

modeling_minibeatrix.py

@@ -0,0 +1,93 @@
+"""HF AutoModel wrapper for mini-beatrix-1 (AlephLM byte-trigram craft).
+
+Loads with:
+    AutoModelForCausalLM.from_pretrained("AbstractPhil/mini-beatrix-1",
+                                         trust_remote_code=True)
+The model reads raw UTF-8 bytes: input_ids are byte values 0..255.
+"""
+from __future__ import annotations
+
+import torch
+from transformers import PretrainedConfig, PreTrainedModel
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+from .presets import AlephLMConfig
+from .alephlm import AlephLM
+
+
+class MiniBeatrixConfig(PretrainedConfig):
+    model_type = "mini-beatrix"
+    attribute_map = {"num_hidden_layers": "n_layers",
+                     "hidden_size": "d_model",
+                     "num_attention_heads": "n_heads",
+                     "max_position_embeddings": "context"}
+
+    def __init__(self, name="mini-beatrix-1", d_model=768, n_layers=16,
+                 n_heads=12, context=2048, vocab_size=256,
+                 tokenizer="byte-trigram", hub_layers=(4, 9, 14),
+                 hub_K=512, hub_D=32, tau=0.1, bank_experts=3, bank_ff=None,
+                 head_K=512, head_D=32, gate_init=-3.0,
+                 tie_embeddings=False, hub_chunk=128, **kwargs):
+        self.name = name
+        self.d_model = d_model
+        self.n_layers = n_layers
+        self.n_heads = n_heads
+        self.context = context
+        self.vocab_size = vocab_size
+        self.tokenizer = tokenizer
+        self.hub_layers = list(hub_layers)
+        self.hub_K = hub_K
+        self.hub_D = hub_D
+        self.tau = tau
+        self.bank_experts = bank_experts
+        self.bank_ff = bank_ff
+        self.head_K = head_K
+        self.head_D = head_D
+        self.gate_init = gate_init
+        self.tie_embeddings = tie_embeddings
+        self.hub_chunk = hub_chunk
+        super().__init__(**kwargs)
+
+    def to_aleph(self) -> AlephLMConfig:
+        return AlephLMConfig(
+            name=self.name, d_model=self.d_model, n_layers=self.n_layers,
+            n_heads=self.n_heads, context=self.context,
+            vocab_size=self.vocab_size, tokenizer=self.tokenizer,
+            hub_layers=tuple(self.hub_layers), hub_K=self.hub_K,
+            hub_D=self.hub_D, tau=self.tau, bank_experts=self.bank_experts,
+            bank_ff=self.bank_ff, head_K=self.head_K, head_D=self.head_D,
+            gate_init=self.gate_init, tie_embeddings=self.tie_embeddings,
+            hub_chunk=self.hub_chunk)
+
+
+class MiniBeatrixForCausalLM(PreTrainedModel, GenerationMixin):
+    config_class = MiniBeatrixConfig
+    main_input_name = "input_ids"
+    supports_gradient_checkpointing = False
+    _tied_weights_keys = {}
+
+    def __init__(self, config: MiniBeatrixConfig):
+        super().__init__(config)
+        self.model = AlephLM(config.to_aleph())
+        self.post_init()
+
+    def _init_weights(self, module):
+        pass  # AlephLM initializes itself; loader must not re-init
+
+    def forward(self, input_ids=None, labels=None, attention_mask=None,
+                past_key_values=None, use_cache=None, **kwargs):
+        # attention_mask is safely ignored: causal model, right-padding
+        # with label masking is the trained convention. No KV cache in
+        # this wrapper; generate() recomputes the prefix each step.
+        if input_ids.shape[1] > self.config.context:
+            input_ids = input_ids[:, -self.config.context:]
+            if labels is not None:
+                labels = labels[:, -self.config.context:]
+        out = self.model(input_ids, labels=labels)
+        logits = out[0]
+        loss = out[1] if labels is not None else None
+        return CausalLMOutputWithPast(loss=loss, logits=logits)
+
+    def prepare_inputs_for_generation(self, input_ids, **kwargs):
+        return {"input_ids": input_ids[:, -self.config.context:]}

presets.py

@@ -0,0 +1,163 @@
+"""Mission presets — the Mini-Beatrix ladder.
+
+Naming convention (voyager style): numbered missions, each a fixed craft.
+Small crafts are "mini-beatrix-N"; the BPE flagship is "beatrix-voyager".
+Beatrix is the lineage collective name; missions are launched in order and
+all upload to the one training repo (TRAINING_REPO), each craft under its
+own path prefix (checkpoints + manifest + tensorboard).
+
+  mini-beatrix-0   d512  L12 ctx1024 byte-trigram  37.6M  gate craft:
+                   its first toggle evals ARE the anchored-bank-under-AR
+                   screen (P1) running live.
+  mini-beatrix-1   d768  L16 ctx2048 byte-trigram  112.5M   first Colab
+                   mission (default).
+  mini-beatrix-2   d1024 L20 ctx2048 byte-trigram  249.1M
+  beatrix-voyager  d1536 L24 ctx4096 BPE(gpt2 50k)  775.3M   flagship;
+                   vocab-scale head + BPE screens (P2/P5) still open —
+                   launch only after mini-beatrix verdicts.
+
+Every craft is inference-capable on consumer hardware in its shipped
+form (fp8-e4m3 safetensors variants are exported alongside checkpoints).
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field, asdict
+from typing import Optional
+
+
+@dataclass
+class AlephLMConfig:
+    name: str = "mini-beatrix-0"
+    d_model: int = 512
+    n_layers: int = 12
+    n_heads: int = 8
+    context: int = 1024
+    vocab_size: int = 256              # bytes; BPE presets override
+    tokenizer: str = "byte-trigram"    # "byte-trigram" | "hf:<repo or name>"
+    hub_layers: tuple = (3, 7, 11)     # CausalSplatHUB depths; () = pure sdpa control
+    hub_K: int = 512
+    hub_D: int = 32
+    tau: float = 0.1
+    bank_experts: int = 3              # E1-validated fat-expert count
+    bank_ff: Optional[int] = None      # None -> d_model (E1 ratio)
+    head_K: int = 512
+    head_D: int = 32
+    gate_init: float = -3.0
+    tie_embeddings: bool = False       # BPE crafts tie; byte crafts cannot (trigram)
+    hub_chunk: int = 128               # chunked-scan block for the hub prefix memories
+
+    def to_dict(self):
+        d = asdict(self)
+        d["hub_layers"] = list(self.hub_layers)
+        return d
+
+    @staticmethod
+    def from_dict(d):
+        d = dict(d)
+        d["hub_layers"] = tuple(d.get("hub_layers", ()))
+        return AlephLMConfig(**d)
+
+
+@dataclass
+class TrainConfig:
+    # Optimizer split (measured: momentum-geometric +.09 on the aleph;
+    # the mechanism is ~20x more optimizer-sensitive than sdpa).
+    muon_lr: float = 2e-2
+    muon_momentum: float = 0.95
+    adam_lr: float = 3e-4              # pure Adam, wd=0 — never AdamW
+    warmup_steps: int = 200            # scale insurance; flat after (flat-LR law)
+    grad_clip: float = 1.0
+    micro_batch: int = 24
+    grad_accum: int = 1
+    # Cadences (steps)
+    log_every: int = 50
+    health_every: int = 500
+    eval_every: int = 2000
+    ckpt_every: int = 2000             # safetensors + resume .pt
+    fp8_every_ckpts: int = 5           # every Nth checkpoint also exports fp8
+    tb_upload_every: int = 1000
+    # Eval sizes
+    val_tokens: int = 262144
+    canary_episodes: int = 128
+    seed: int = 1337
+    compile: bool = False
+
+
+# All missions upload to the one training repo, each under its own prefix
+# (Phil's repo: checkpoints + manifests + tensorboard for every craft).
+TRAINING_REPO = "AbstractPhil/alephllm-mini-beatrix-training"
+
+
+@dataclass
+class Preset:
+    model: AlephLMConfig
+    train: TrainConfig
+    hf_repo: str = TRAINING_REPO                   # run repo (ckpts+manifest+tb)
+    curriculum: list = field(default_factory=list) # [(phase, dataset, planned_tokens)]
+
+    @property
+    def prefix(self) -> str:                       # path prefix inside hf_repo
+        return self.model.name
+
+
+def _curriculum(warm: int, main: int, ext: int):
+    return [
+        dict(name="warmup_wikitext", dataset="wikitext-103", planned_tokens=warm,
+             status="planned"),
+        dict(name="fineweb_main", dataset="fineweb-edu", planned_tokens=main,
+             status="planned"),
+        # Deliberately not prepped beyond a name — the full plan exists in the
+        # manifest, the data work happens when the phase activates.
+        dict(name="fineweb_extended", dataset="fineweb-edu", planned_tokens=ext,
+             status="deferred"),
+        # phase C: distribution shift toward chat format / simple register /
+        # narrative (incl. moral texture) / binding demand — see streams.ANNEAL_MIX
+        dict(name="anneal_mix", dataset="anneal-mix",
+             planned_tokens=2_000_000_000, status="deferred"),
+    ]
+
+
+PRESETS: dict[str, Preset] = {
+    "mini-beatrix-0": Preset(
+        model=AlephLMConfig(name="mini-beatrix-0"),
+        train=TrainConfig(micro_batch=96, grad_accum=1),
+        curriculum=_curriculum(150_000_000, 1_000_000_000, 2_000_000_000),
+    ),
+    "mini-beatrix-1": Preset(
+        model=AlephLMConfig(name="mini-beatrix-1", d_model=768, n_layers=16,
+                            n_heads=12, context=2048, hub_layers=(4, 9, 14)),
+        train=TrainConfig(micro_batch=48, grad_accum=3),
+        curriculum=_curriculum(300_000_000, 3_000_000_000, 6_000_000_000),
+    ),
+    "mini-beatrix-2": Preset(
+        model=AlephLMConfig(name="mini-beatrix-2", d_model=1024, n_layers=20,
+                            n_heads=16, context=2048, hub_layers=(5, 11, 17)),
+        train=TrainConfig(micro_batch=32, grad_accum=6),
+        curriculum=_curriculum(300_000_000, 5_000_000_000, 10_000_000_000),
+    ),
+    "beatrix-voyager": Preset(
+        model=AlephLMConfig(name="beatrix-voyager", d_model=1536, n_layers=24,
+                            n_heads=16, context=4096, vocab_size=50257,
+                            tokenizer="hf:gpt2", tie_embeddings=True,
+                            hub_layers=(6, 13, 20)),
+        train=TrainConfig(micro_batch=8, grad_accum=16),
+        curriculum=_curriculum(500_000_000, 12_000_000_000, 24_000_000_000),
+    ),
+}
+
+# Pure-sdpa control crafts (hub layers removed) — the running architecture
+# control for any mission: same params otherwise, suffix "-control".
+for _name in list(PRESETS):
+    _p = PRESETS[_name]
+    _m = AlephLMConfig.from_dict(_p.model.to_dict())
+    _m.name = _name + "-control"
+    _m.hub_layers = ()
+    PRESETS[_name + "-control"] = Preset(
+        model=_m, train=_p.train, 
+        curriculum=[dict(x) for x in _p.curriculum])
+
+
+def get_preset(name: str) -> Preset:
+    if name not in PRESETS:
+        raise KeyError(f"unknown preset '{name}' — have: {sorted(PRESETS)}")
+    return PRESETS[name]