--- license: cc-by-nc-sa-4.0 language: en pipeline_tag: text-generation base_model: AbstractPhil/mini-beatrix-2s tags: - byte-level - linear-attention - splat-attention - aleph - adapters - custom_code --- # mini-beatrix-2.5s **The mini-beatrix-2s core, packaged with its arms.** The 237M-parameter full-splat byte model that completed its 16.101B-token mission on 2026-08-31, shipped here with a library of **detachable arms** — small aleph-addressed patch heads trained on this exact frozen core — and the runtime that mounts them. The core's weights are **bit-identical** to [mini-beatrix-2s](https://huggingface.co/AbstractPhil/mini-beatrix-2s). Nothing was retrained for 2.5s: an arm is a guest on the core, never a migration of it, and the package proves that on every load (see *Detach is exact*). ```python import torch from transformers import AutoModelForCausalLM m = AutoModelForCausalLM.from_pretrained( "AbstractPhil/mini-beatrix-2.5s", trust_remote_code=True).eval() m.say("Who are you?") # the bare core, its own frame # 'I am Beatrix, a small byte-level language model. I read raw bytes ...' m.mount_arm("chain") # one of the packaged arms m.say("If someone is sook, then they are quen. If someone is quen, then " "they are harl. If someone is harl, then they are torv. If someone " "is torv, then they are prin. If someone is prin, then they are " "mund. Wren is sook. What follows?") # 'So Wren is quen. So Wren is harl. So Wren is torv. So Wren is prin. So Wren is mund.' m.detach_arm() # bit-exact core restored ``` The model reads **raw UTF-8 bytes**: `input_ids` are byte values 0–255. There is no tokenizer to download. ## The arms `m.arms()` returns this table as data; each row carries the frame it was trained in, its decode settings, its measured score and how settled the result is. | arm | what it does | type | measured | cell | status | |---|---|---|---|---|---| | `rules` | reads five if-then rules about made-up words and answers with only the final word | arm | unseen words .640 · the original eight .960 | L1 (8 shared-prefix pairs, 64-word pool, quiet weight 2) | settled (2 seeds) | | `rules-seed1` | the same arm trained from a different seed — the pair that makes the result two-seeded | arm | unseen words .673 · the original eight .907 | L1 | settled (2 seeds) | | `rules-minted` | the same answer-only rule reading, trained on the 216-word lexicon minted for the v3 curriculum (96 easy + 96 trap + 12 shared-onset pairs) at the ruled dial: half the rows pairs | arm | minted words .707 · the original eight .947 · unseen .560 | DIAL (pair rate .50, quiet weight 2) | settled (2 seeds, 2026-09-22) | | `rules-minted-seed1` | the second seed of the ruled dial; the quietest arm in the package on general text | arm | minted words .647 · unseen .593 · general-text cost .0066 bpb | DIAL | settled (2 seeds, 2026-09-22) | | `chain` | reads five if-then rules and writes out every step: 'So Wren is …' five times | arm | chain .867 · general-text cost .003 bpb | E-I (trained with an abstention term, so it composes ungated) | settled (2 seeds) | | `chain-plain` | the same step-by-step chain writing, trained without the abstention term: the best chain number in the library, but it writes on off-domain text too | arm | chain .887 | C1 | settled (2 seeds) | | `eco-chain` | a loaded chain arm carried through a second training stage beside a fresh turn-end arm, and still detachable | arm | alone .833 · in the pair .853 · quiet .0054 bpb (the loaded arm read .867 before this stage) | ECO / L4 | settled (2 seeds) | | `stop` | ends its turn cleanly with a blank line instead of rambling on | arm | clean stop .83 (second seed 1.00) | E-I | settled (2 seeds) | | `stop-seed1` | the second seed: every probe ends its turn | arm | clean stop 1.00 | E-I | settled (2 seeds) | | `eco-stop` | the fresh turn-end arm trained in that second stage, with its own abstention term | arm | turn end .953 in the pair · chat 1.00 | ECO / L4 | settled (2 seeds) | | `routed-gain` | both arms carried by a dense signed dispatch: every member contributes at every byte, weighted by a signed read and a bounded gain that follows each member's recent usage | routed | chain .813 (seed 1 .787) | E-D0 | candidate (2 seeds; the conserved form is seed-unstable) | | `pair` | both arms applied always-on with no mixer between them — the composition the regime law says should destroy a non-selective pair, and does not destroy these two | stack | chain .860 · turn end .973 · stop rate 1.00 | E-I | settled (2 seeds) | | `eco-pair` | the staged co-training pair applied together | stack | chain .853 · turn end .953 | ECO / L4 | settled (2 seeds) | *13 rows over 11 weight files; the full library of ~80 trained arms — including the refuted and control rows this package leaves out — lives on the training repo.* **Status** is the research grade, not a rating: *settled (2 seeds)* means the result reproduced on a second seed under the same recipe; *candidate* means one axis is still open. **Cell** names the training recipe the row came from. **Measured** is what the training campaign's own harness scored on the full exam — `m.arms()` returns the complete per-arm measurement dictionary behind each of these summaries. Those numbers carry over to this package rather than being re-measured here, and the reason is checkable: every packaged row produces logits **bit-identical** to the runtime the arms were trained with (see *Detach is exact*). Same weights and same logits under a greedy decode means the same bytes, so re-scoring could only reproduce the stored number. ### Mounting ```python m.mount_arm("pair") # a stack: chain + turn end, both always-on m.mount_arm("routed-gain") # a signed dispatch over both, with a usage gain h = m.arm # the live handle with h.all_off(): # every member masked: the bare core's logits m.say("Hello there.") m.detach_arm(verify=True) # raises if the restored core is not bit-exact ``` Mounting a second arm detaches the first. `verify=True` (the default) re-runs a fixed probe through the restored core and asserts the logits are bit-identical to the pre-mount fingerprint. ### What an arm is Each arm places one module after every decoder block. At a site it projects the block's output into `n_slots` slots of dimension `D`, reads them through a `K`-atom **aleph address** — a closed-form signed read, never a selector — and adds a gated patch back: ``` slots = proj(x).view(B, n, n_slots, D) m_hat = sum_k sinh(u_k) A_k / sum_k cosh(u_k), u = (x_hat · A) / tau x = x + sigmoid(gate) * consume(m_hat) ``` 8.6M parameters over 20 sites (3.6% of the core). There is no argmax, no top-k and no softmax over atoms anywhere in the path. A **stack** applies several arms always-on with no mixer between them. A **routed** recipe carries its members inside one dense signed dispatch: ``` out = x + sum_a (sinh u_a / sum_all cosh u) (arm_a(x) - x) ``` The denominator sums over **every** member, masked or not, and masking a member removes its contribution without renormalizing — masked is not solo. The `routed-gain` row adds a detached bounded gain per member, driven by a causal average of that member's own share. ### Selective and non-selective arms Arms marked *selective* were trained with an abstention term — a KL to the bare core on rows outside their domain — and cost ~0.003 bpb on general text. That is what lets two of them run together ungated (`pair`): the same composition of two non-selective arms destroys both. `chain-plain` is the counter-example kept in the package on purpose: the best chain score in the library, and it writes on off-domain text too. ## Frames | frame | used by | form | |---|---|---| | `specials` | the bare core | `[SYS] … [END] [USER] … [END] [MODEL]` on the 13 byte values UTF-8 can never produce — the frame is unforgeable, not by convention but by arithmetic | | `text` | every packaged arm | `A conversation with Beatrix…\nUser: …\nBeatrix:` with a blank line ending the turn | `say()` picks the frame the live arm was trained in and cuts the reply at that frame's stop. The arms predate the specials anneal, so they speak the text frame; the core's own chat frame is the specials one. ## The core 237.1M parameters · d_model 1024 · 20 layers · context 4096 · byte-trigram embedding. A governed multi-constellation **CausalSplatHUB** — signed address linear attention over learned codebook blackboards — in **every** block (4 constellations × 64 anchors @ D=128, min-separation governed from birth at 45°), three anchored bank experts per block under signed dispatch, and a dual head (linear readout + a 256@256 signed aleph read revived mid-mission after the born-null path self-buried). Each layer *encodes* the sequence onto a fixed-width addressed blackboard instead of caching it, so the decode state is **constant size** in context length. Final held-out reading: **1.1097 bpb** on fineweb-edu. Trained 16.101B bytes in 61,422 steps on one RTX 6000 Pro Blackwell: wikitext warmup → fineweb-edu → a nine-stage early-life curriculum → a two-phase anneal that taught the conversation frame. Zero loss spikes across the run; the pre-registered softmax twin of the same recipe destabilized and ended at 2.8846 bpb (the full comparison is on the 2s card). ## Detach is exact The claim this package exists to make checkable: ```python import torch before = m.say("Hello there.", frame="specials") h = m.mount_arm("rules"); m.detach_arm(verify=True) assert m.say("Hello there.", frame="specials") == before ``` At release every packaged row was mounted through this package **and** through [amoe-lora](https://github.com/AbstractEyes/amoe-lora), the runtime the arms were trained with, and the two paths produced bit-identical logits (max |Δ| = 0.0 on all 13 rows, fp32). The core's state dict is unchanged by any number of mount/detach cycles. ## Limitations - Small and early: conversational in shape, thin on knowledge, and confidently wrong at times. Short exchanges suit it best. - **Arms are trunk-bound.** They were trained on this exact core at step 61,422 and carry its id in their metadata; `mount_arm` refuses a mismatch. A 31% gain was lost when an arm was moved across trunks. - The arms' numbers were measured **greedy in fp32**. bf16 serving works and is not what the table measured. - The rule-chain arms are a controlled capability on made-up words, not general reasoning: they read five if-then rules and either write the steps out or answer with the final word. - Every packaged arm is the **wide** adapter shape: 32 slots of dimension 4 read through a 64-atom address, 8.57M parameters. The certified twins of these same recipes at the narrower 16-atom, dimension-8 shape (a near-identical 8.56M parameters) live on the training repo and are **not** packaged here, so nothing in this table is a comparison between the two shapes. - `rules-minted` was trained on the lexicon minted for the **next** craft's curriculum; it is packaged here because it is the newest settled arm and the same recipe, not because this core saw those words in pretraining. - Saving the model with an arm mounted is refused: detach first, or the arm's parameters would be written into the core's weight file. ## Files ``` config.json modeling_minibeatrix.py arms.py the package alephlm.py address.py attention.py bank.py embedding.py governor.py head.py relay.py presets.py vendored model code model.safetensors the core (bf16, 476 MB) arms/index.json the arm table as data arms/*.safetensors the arms ``` `arms.py` is self-contained (torch + safetensors): the arm runtime is vendored so the package has no git dependency, and it is verified against amoe-lora at release rather than trusted. The vendored model files are verbatim copies of `geolip/alephllm/model/*.py` and `presets.py` from [alephllm](https://github.com/AbstractEyes/alephllm) 0.10.2, with two marked deviations, both inert for this craft and both there so the package loads under `trust_remote_code`: the relative imports of the `data` and `fusion` modules (a curriculum builder and the v3 weak-token fusion, neither reachable at inference here) are deferred and resolved by name instead of as flat sibling files. Every deviation is commented `VENDORED` in place. ## Lineage Code: [alephllm](https://github.com/AbstractEyes/alephllm) · adapters: [amoe-lora](https://github.com/AbstractEyes/amoe-lora) · core weights, the full ~80-arm library, checkpoints and the boundary reports: [alephllm-mini-beatrix-training](https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training) · predecessor: [mini-beatrix-1](https://huggingface.co/AbstractPhil/mini-beatrix-1) (112M, 3-hub hybrid) · the core alone: [mini-beatrix-2s](https://huggingface.co/AbstractPhil/mini-beatrix-2s). The research memory behind every design decision is maintained privately; the laws the code cites (supply ≤ 2D, min-separation governance, budget composition, born-null weight-zero, the damping law) are measured, not aesthetic. ## Licence The weights are **non-commercial**: [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/), because the training text includes CHILDES child-directed speech (CC BY-NC-SA, via TalkBank) and Wikipedia / WikiText (CC BY-SA). The code files, the model design and the full training process are MIT ([AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm)); `relay.py` and `arms.py` are adapted from amoe-lora under Apache-2.0 ([NOTICE](NOTICE)). The full terms and the list of training sources: [LICENSE.md](LICENSE.md).