README.md

---
license: mit
language: en
pipeline_tag: text-generation
tags:
- byte-level
- linear-attention
- splat-attention
- aleph
- custom_code
---

# mini-beatrix-2s

**The first full-splat craft to complete a designed life.** 237M parameters,
byte-level (vocab 256), with a governed multi-constellation **CausalSplatHUB**
— signed-address linear attention over learned codebook blackboards — in
**every one of its 20 blocks**. No softmax-over-positions attention anywhere.
Trained 16.101B tokens (61,422 steps, ~78h on a single RTX 6000 Pro
Blackwell) through a staged curriculum, completed 2026-08-31.

## Architecture

- **d_model 1024 · 20 layers · ctx 4096 · byte-trigram embedding** (raw UTF-8
  bytes; input ids are byte values 0–255)
- **Hubs (all 20 blocks):** 4 constellations × 64 anchors @ D=128 per block,
  min-separation governed from birth (45°), budget-composed (numerators and
  agreement masses sum before one divide — reconstructive, never comparative:
  no argmax, no top-k, inhibition first-class). Constant-size prefix state:
  each layer *encodes* the sequence onto a fixed-width addressed blackboard
  rather than caching it.
- **Anchored banks:** 3 full-width experts per block, signed dispatch.
- **Dual head:** linear readout + a signed aleph read (256 anchors @ 256),
  revived mid-mission via a deterministic boundary-write after the born-null
  path self-buried — the first functioning aleph head in the lineage.
- Final causal contributions (toggle ledger, held-out): **hubs +3.36 bpb ·
  banks +3.31 bpb · head +0.012 bpb**.

## The special-token control plane

Thirteen ids that valid UTF-8 can never produce (proven by exhaustion) carry
structure and are **trained**:

| id | token | meaning |
|----|-------|---------|
| 0xFF | DOC | document boundary (taught from step 0) |
| 0xFE / 0xFD | USER / MODEL | turn openers (taught in the final chat phase) |
| 0xFC | END | universal block close |
| 0xFB | SYS | system block opener |
| 0xF7+b | MODE | register tag (+1 ASCII byte) |
| 0xF5+b | ESC | 254 extended slots |
| 0xFA 0xF9 0xF8 0xF6 0xC0 0xC1 | THINK DATA SEP CUE RES | reserved/instrument |

Chat format: `[SYS] text [END] [USER] text [END] [MODEL] text [END]` — the
frame is unforgeable (encoded text cannot contain a special).

## Usage

```python
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "AbstractPhil/mini-beatrix-2s", trust_remote_code=True)

prompt = "The history of astronomy begins"
ids = torch.tensor([list(prompt.encode("utf-8"))])
out = model.generate(ids, max_new_tokens=200, do_sample=True, top_p=0.95)
print(bytes(int(i) for i in out[0]).decode("utf-8", errors="replace"))
```

Chat-framed prompting (the frame was annealed in the final 1B tokens —
treat it as a young capability):

```python
SYS, USER, MODEL, END = 0xFB, 0xFE, 0xFD, 0xFC
turns = ([SYS] + list(b"You are Beatrix, a small byte-level language model.")
         + [END] + [USER] + list(b"Who are you?") + [END] + [MODEL])
out = model.generate(torch.tensor([turns]), max_new_tokens=100,
                     do_sample=True, top_p=0.95)
```

## Training

16.101B tokens on one card: wikitext warmup (0.3B) → fineweb-edu (5B) → a
nine-stage early-life curriculum, s0–s8 (8.8B: narrative, perspective,
concepts, rule-chains, arithmetic, causal, try-fail, mixed, register) → a
two-phase anneal (1B distribution shift without the chat frame, then 1B with
it — the frame's effect bracketed by boundary reports). Muon + pure Adam
split, flat LR, bf16, zero loss spikes across the entire run. Every boundary
shipped a report (toggle ledger, probe suite, address census): 17 reports and
all checkpoints live in
[alephllm-mini-beatrix-training](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training)
under `mini-beatrix-2s/`.

Final validation: **1.1097 bpb** on the fineweb-edu holdout — the run's best
reading, set during the chat phase.

## The softmax twin

A pre-registered control ran the identical recipe and diet beside this craft:
**mini-beatrix-2s-control** — the same preset with the hub list empty, so 16-head
causal softmax attention (SDPA) in every one of the 20 blocks (257.4M parameters
against this craft's 237M), 16.101B tokens in 61,422 steps, one craft per arm. At
the matched endpoint (anneal_mix, step 61,422, same evaluation) the twin read
**2.8846 bpb against this craft's 1.1097**, and this craft led at 11 of 11
same-gauge paired boundaries after the twin's early lead (0.3B: twin 2.106 vs
this craft 2.363; the first paired row at 5.3B: this craft 1.1172 vs twin
1.1763); the s8 pair is struck because its two rows were measured on different
gauges. The twin did not merely trail — it destabilized: its validation, on the
one fineweb-edu holdout every row shares, went 1.18 (fineweb) → 1.57 (s4) → 2.18
(s5) → 3.19 (s6) → 3.12 (s7), and the two anneal phases recovered it only to
2.88; its pre-clip gradient norm first crossed the 1.0 clip at step 17,600 inside
the fineweb phase while it still led, then escalated stage by stage (medians 0.29
→ 6.5 → 2.4 → 12 → 13 → 24.5 → 126 → 471 → 162 → 145 → 167 → 215, spikes to
41,070, clipped on 813 of 824 logged curriculum and anneal steps) while this
craft's never reached the clip (0 of 824, max 0.62). The anneal was a diet change
(1B of prose, then 1B with the chat frame), not a learning-rate decay: lr_scale
read 1.000 throughout on both crafts. This is one craft per arm, and the twin
was given none of softmax attention's usual stabilizers (no QK-normalization, no
weight decay, no learning-rate decay, the same 1.0 clip). The claim that SDPA is
not an adequate representative of this model's attention requirement is
AbstractPhil's (2026-09-02); its audit (2026-09-04) grades the specific form an
instance — under this recipe the twin is a destabilized control, and the gap
measures stability rather than capacity — and the general form a candidate
behind the recipe, the single seed, the shape (mini-beatrix-1's thirteen softmax
blocks trained clean under the identical configuration), the aleph-addressed
expert banks both crafts carry, and the learned position table on decay-free
Muon. Paired reports: `mini-beatrix-2s-control/reports/v2/` beside this craft's
under [alephllm-mini-beatrix-training](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training);
the paired chart is `article_assets_ft2/01_control_crossover.png` there.

## Lineage

Code: [AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)
(this repo vendors the model files verbatim; alephllm 0.8.6).
Sibling: [mini-beatrix-1](https://huggingface.co/AbstractPhil/mini-beatrix-1)
(112M, 3-hub hybrid — the previous rung). Control twin: mini-beatrix-2s-control
(softmax attention in every block; weights, resume states and 15 reports under
`alephllm-mini-beatrix-training/mini-beatrix-2s-control/`). The research memory behind every
design decision is maintained privately; laws referenced in the code
docstrings (supply ≤ 2D, min-sep governance, budget composition, born-null
weight-zero) are measured, not aesthetic.