AbstractPhil/alephllm-nano-castor-pollux

AlephLLM nano: Castor and Pollux, v1.1

Four small byte-level AlephLLM models (about 9 million parameters each, a 4,096-byte window), trained from birth on Beatrix 2s's diet: 16.1 billion bytes in thirteen phases (wikitext-103, fineweb-edu, nine curriculum stages, two anneals), 61,422 steps each. They read and write raw UTF-8 bytes: no tokenizer.

This is v1.1: the same four crafts from the same starts on the same diet, with the output layer's two readout matrices trained by AdamW (rate 0.0003, decoupled weight decay 0.1) instead of Muon. In the v1 run the readout kept growing under Muon and the last block's output gathered onto a single direction within its first thousand steps; that run was stopped, and its folders are archived in alephllm-nano-beatrix-training under archive/castor-pollux-v1/.

Beside each body sit its continuation arms (below): the same body taken further on a mix that teaches it to write and read its own notes. Every folder is a model with its full training record, and every folder loads the same way.

model	the body	parameters	bits a byte (fineweb-edu holdout)	recall: copy / nested / repeated key	1,024-byte frame	planted record at 2,048 / 3,584 bytes (bits)
nano-pollux-dual-canon-v1.1	Pollux's layout (hubs in blocks 0, 2, 4; rotary softmax in 1, 3, 5) with dual hubs and Canon layers	9,279,569	1.2927	0.33 / 0.33 / 0.12	0.03	45.59 / 49.67
nano-pollux-canon-v1.1	Pollux (rotary hubs and rotary softmax alternating) with Canon layers	8,860,241	1.2920	0.35 / 0.29 / 0.09	0.00	44.44 / 49.25
nano-pollux-dual-v1.1	Pollux's layout with dual hubs	9,267,281	1.3057	0.06 / 0.07 / 0.03	0.00	45.98 / 49.27
nano-castor-v1.1	Castor: rotary softmax in every block (the baseline)	9,215,057	1.2817	0.14 / 0.10 / 0.08	0.00	45.84 / 49.21

The recall columns are the nano bed's checks at the end of the train: a keyed record asked back whole (copy, a nested question, a key that was redefined), the same read on 1,024-byte frames, and the bits needed to recall a planted record from 2,048 and 3,584 bytes away (lower is better).

The continuation arms

Each body's diet ended at step 61,422. From there an arm takes the same body 3,815 steps further (1.0 billion bytes, the same optimizer settings) on a mix that teaches it to write and read its own notes; a second-stage arm continues an arm's end the same way (its folder suffix chains the two: -fr-cf is frames, then carry). Every arm folder is a model like the four above, with its own training record, and loads the same way.

arm	folder suffix	what it trains on
frames	-fr	the diet's text with keyed records written in small frames, and questions that ask a record back
control	-ct	the diet's text alone, the same steps: the comparison for the other arms
carry	-cf	framed records re-written along the text as carries, with the question far after: the body learns to re-write a record on its way to a distant question
notes	-nt	notes in real words taken from the text itself (a definition, an abbreviation, a quantity, a capital, a place, a year), re-written on the way and asked back

One table a body: its folders, the steps each trained, and the nano bed's checks at each end on the same 192 held-out rows. The bed's held-out bits a byte is the bed's own fineweb-edu read (not the column of the first table). Far question, own carries: a keyed record early in the window and the question 2,048 bytes or more after it, the body free to re-write the record on the way; the share of rows answered whole, and carries a row how many re-writes it made. A dash: the read does not apply to that folder.

The note arm's own reads: note given, 2,048+ bytes is a real-word note from the text planted after its fact with the question 2,048 bytes or more after, the share answered whole; own notes are the same rows with no note planted: the share of rows whose own written note held the asked key, the share answered whole at 2,048 bytes or more, and the notes it wrote a row.

Pollux's layout (hubs in blocks 0, 2, 4; rotary softmax in 1, 3, 5) with dual hubs and Canon layers

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row	note given, 2,048+ bytes	own notes: rows whose note held the key	own notes: answered at 2,048+ bytes	own notes a row
nano-pollux-dual-canon-v1.1	0 to 61,422 (the diet)	1.2717	0.33 / 0.33 / 0.12	0.03	-	-	-	-	-	-
nano-pollux-dual-canon-v1.1-fr	frames: 61,422 to 65,237	1.2512	0.93 / 0.92 / 0.85	0.82	-	-	-	-	-	-
nano-pollux-dual-canon-v1.1-ct	control: 61,422 to 65,237	1.2414	0.29 / 0.25 / 0.09	0.03	-	-	-	-	-	-
nano-pollux-dual-canon-v1.1-cf	carry: 61,422 to 65,237	1.2506	0.29 / 0.33 / 0.18	0.53	0.51	4.5	-	-	-	-

Pollux (rotary hubs and rotary softmax alternating) with Canon layers

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row	note given, 2,048+ bytes	own notes: rows whose note held the key	own notes: answered at 2,048+ bytes	own notes a row
nano-pollux-canon-v1.1	0 to 61,422 (the diet)	1.2723	0.35 / 0.29 / 0.09	0.00	-	-	-	-	-	-
nano-pollux-canon-v1.1-fr	frames: 61,422 to 65,237	1.2517	0.92 / 0.91 / 0.83	0.82	-	-	-	-	-	-
nano-pollux-canon-v1.1-ct	control: 61,422 to 65,237	1.2415	0.30 / 0.27 / 0.12	0.00	-	-	-	-	-	-
nano-pollux-canon-v1.1-cf	carry: 61,422 to 65,237	1.2483	0.44 / 0.46 / 0.18	0.41	0.48	4.9	-	-	-	-

Pollux's layout with dual hubs

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row	note given, 2,048+ bytes	own notes: rows whose note held the key	own notes: answered at 2,048+ bytes	own notes a row
nano-pollux-dual-v1.1	0 to 61,422 (the diet)	1.2861	0.06 / 0.07 / 0.03	0.00	-	-	-	-	-	-
nano-pollux-dual-v1.1-fr	frames: 61,422 to 65,237	1.2644	0.89 / 0.88 / 0.74	0.79	-	-	-	-	-	-
nano-pollux-dual-v1.1-ct	control: 61,422 to 65,237	1.2526	0.05 / 0.04 / 0.01	0.00	-	-	-	-	-	-
nano-pollux-dual-v1.1-cf	carry: 61,422 to 65,237	1.2620	0.42 / 0.40 / 0.20	0.47	0.59	4.7	-	-	-	-
nano-pollux-dual-v1.1-fr-cf	frames, then carry: 65,237 to 69,052, from nano-pollux-dual-v1.1-fr	1.2567	0.53 / 0.52 / 0.25	0.65	0.48	3.8	-	-	-	-
nano-pollux-dual-v1.1-cf-nt	carry, then notes: 65,237 to 69,052, from nano-pollux-dual-v1.1-cf	1.2490	0.26 / 0.27 / 0.17	0.24	0.03	2.6	0.70	0.27	0.15	2.0

Castor: rotary softmax in every block (the baseline)

folder	trained	the bed's held-out bits a byte	copy / nested / repeated key	1,024-byte frame	far question, own carries	carries a row	note given, 2,048+ bytes	own notes: rows whose note held the key	own notes: answered at 2,048+ bytes	own notes a row
nano-castor-v1.1	0 to 61,422 (the diet)	1.2614	0.14 / 0.10 / 0.08	0.00	-	-	-	-	-	-
nano-castor-v1.1-fr	frames: 61,422 to 65,237	1.2398	0.91 / 0.92 / 0.74	0.76	-	-	-	-	-	-
nano-castor-v1.1-ct	control: 61,422 to 65,237	1.2300	0.18 / 0.14 / 0.07	0.00	-	-	-	-	-	-
nano-castor-v1.1-cf	carry: 61,422 to 65,237	1.2367	0.37 / 0.39 / 0.16	0.50	0.29	3.3	-	-	-	-

Use

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("AbstractPhil/alephllm-nano-castor-pollux", subfolder="nano-pollux-dual-canon-v1.1",
                                             trust_remote_code=True)   # variant="fp32" for the fp32 weights
ids = torch.tensor([list("The history of astronomy".encode("utf-8"))])
out = model.generate(ids, max_new_tokens=64, do_sample=False)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

The model code sits at the top of the repo (transformers loads remote code from there) and again in each model's folder, so a folder downloaded alone loads with from_pretrained(<folder>, trust_remote_code=True). Each folder holds model.safetensors (bf16) and model.fp32.safetensors (the fp32 weights the training ended with); the bf16 file is the fp32 file rounded. Generation re-reads the whole window each step (no cache). config.json says what the model is (craft: its body, its arm, what it continued from, the phase and step its weights ended on; activated: the mechanisms its body switches on), and crafts.json at the top of the repo indexes every packaged folder with its end reads: the tables above are rendered from it.

Training

The recipe is Beatrix 2s's: Muon at 0.02 on the trunk's matrices, Adam at 0.0003 on the rest, 200 warmup steps then a flat rate, gradient clip 1.0, 262,144 bytes a step. The output layer's two readout matrices trained with AdamW (0.0003, decoupled weight decay 0.1) instead of Muon. Each model started from exactly the weights its strongest run on the nano bed started from. Each folder keeps the whole training record: checkpoints/, resume/, manifest.json, runs/ (TensorBoard: the training curves, the health census, the evals with bits by position, and the bed's checks at intervals under bed/), bed_gauges/ and v1_boundaries.json (one row a phase). An arm folder keeps the same record for its own steps: checkpoints at its two mid gauges and its end (an fp8 copy of the end), the end's resume state, the gauge records of its start state and its reads, manifest.json, runs/, v1_boundaries.json. The trainer: geolip-nano-beatrix-trainer (nano_v1.py, automodel.py, colab/nano_v1_1.ipynb, the arm notebooks colab/nano_v1_1_frames.ipynb, nano_v1_1_carry.ipynb, nano_v1_1_frames_carry.ipynb, nano_v1_1_notes.ipynb, and colab/nano_v1_1_release.ipynb, the packaging) on AlephLLM. The license follows the training data's share-alike and non-commercial terms, as Beatrix 2s's does.