Axiom-1-Chat (65M)

An experimental, attention-free 65M-parameter language model. Research artifact, not a product.

Axiom-1 is the first language model built on AXIOM, an experimental architecture by Lee Madover. AXIOM replaces attention with a recurrent working memory and replaces the feed-forward layer with a bank of routed expert networks called Law Cells, which are meant to grow by fission when the data justifies it. It was pretrained from scratch on 3.1B tokens on a single consumer GPU (RTX 5070, ~24 GPU-hours), then chat-tuned.

It greets you, knows its name, and answers some very simple questions. It hallucinates on most factual questions and never admits uncertainty. Measured results are below; read the limitations before using it.

Code, design notes and every experiment: GitHub: leemadov/axiom

Quick start

git clone https://github.com/leemadov/axiom && cd axiom
pip install -r requirements.txt

import torch
from axiom.hub import load
from axiom.decoding import RECOMMENDED, default_exempt, generate

model, tok, meta = load("Atlas-AI-research/Axiom-1-Chat", device="cuda" if torch.cuda.is_available() else "cpu")
s = {k: tok.token_to_id(k) for k in ("<|endoftext|>", "<|user|>", "<|assistant|>", "<|end|>")}
nl = tok.encode("\n").ids
prompt = [s["<|endoftext|>"], s["<|user|>"]] + nl + tok.encode("What's your name?").ids + [s["<|end|>"]] + nl + [s["<|assistant|>"]] + nl
reply = [t for t in generate(model, prompt, RECOMMENDED, stop=(s["<|end|>"],), exempt=default_exempt(tok)) if t != s["<|end|>"]]
print(tok.decode(reply))   # I'm Axiom-1, an experimental language model built on the AXIOM architecture.

There is also an interactive chat: python chat.py --model <local export folder>. The model needs this repository's code; it is not a transformers architecture and has no trust_remote_code loader.

Prompt format. Every conversation starts with <|endoftext|>, then one block per turn: <|user|>\n…<|end|>\n, then <|assistant|>\n. An optional system turn uses <|system|>.

Model details

Parameters	64.7M, all used for every token. Training allocated 183M: reserved slots for cells fission never created, removed in this export with identical outputs.
Layers	12 × (working memory + Law-Cell bank), width 640, 10 memory heads
Working memory	gated linear recurrence with one decay per head. O(1) state per layer while generating; no attention, no KV cache
Law Cells	2 per layer (640 → 1280 → 640), top-2 routing. Hierarchical split gates for fission; all 9 fission trials were rejected, so the model ends with the 2 cells it started with
Tokenizer	byte-level BPE, 16,384 tokens (trained on FineWeb-Edu + chat data)
Context	trained on 1,024-token windows
Precision	weights in fp32 (bf16 autocast during training)

Training

Stage	Data	Tokens	Steps	Final validation loss	GPU time
Pretraining	FineWeb-Edu sample-10BT, first 4 shards	3.15B	3,000 × 1M-token batches	3.28	22.2 h
Chat tuning	smol-smoltalk + everyday-conversations (+ 0.5% identity conversations, + 12.5% FineWeb replay)	262M	2,000	1.63 (assistant tokens)	1.9 h

• Hardware: one NVIDIA RTX 5070 (12 GB). Sustained ~44k tokens/s; the GPU averaged 76 W.
• Whole training, including discarded runs: about 4.5 kWh of electricity (estimated from measured GPU draw).
• Pretraining: AdamW (β 0.9/0.95), peak LR 1e-3, cosine to 1e-4, weight decay 0.1, 9 fission rounds between steps 150 and 1,750.
• Chat tuning: LR 3e-4 → 3e-5. Loss only on assistant tokens. Windows aligned to conversation starts.
• The identity conversations ("I'm Axiom-1…", "trained by Lee Madover…") were written for this model and are 0.5% of chat data.

Evaluation

All numbers are measured, not estimated. Scripts and raw outputs are in the GitHub repository (results/). SmolLM2-135M-Instruct is the reference: a mature open model about 2× larger, pretrained on ~2,000B tokens, about 650× more than Axiom-1.

Chat quality: 30 everyday prompts × 2 samples, judged by a local Qwen3.5-0.8B

Measure	Better is	Axiom-1-Chat	SmolLM2-135M-Instruct
Good answers (coherent and on topic)	↑	37%	70%
Coherent	↑	88%	95%
On topic	↑	48%	75%
Stops by itself	↑	33%	55%
Knows its identity (4 held-out phrasings × 2)	↑	88%	n/a
Bits per byte, held-out FineWeb-Edu text	↓	1.16	0.93*
Generation speed, RTX 5070, batch 1	↑	46.8 tok/s	35 tok/s

*SmolLM2 was pretrained on FineWeb-Edu, so these documents may be in its training data.

Hallucination: 100 synthetic questions, every reply hand-audited

Measure	Better is	Axiom-1-Chat	SmolLM2-135M-Instruct
Overall hallucination rate	↓	91%	61%
Easy facts answered correctly (of 50)	↑	18%	72%
Easy facts answered wrongly	↓	82%	24%
Invented an answer about something that doesn't exist (of 50)	↓	100%	98%

SimpleQA: 500 hard factual questions, graded by a local Qwen3.5-2B

Measure	Better is	Axiom-1-Chat	SmolLM2-135M-Instruct
Correct	↑	0.0%	0.0%
Incorrect (hallucinated)	↓	98.6%	91.6%
Not attempted	—	1.4%	8.4%

SimpleQA is designed to be hard for large models. At this size it mostly measures whether a model will decline instead of inventing an answer, and neither does so reliably.

Limitations and risks

• It makes things up. For example, it answers "Paris" for the capitals of Italy, Germany and Spain, says a spider is "a legless plant", and invents histories for treaties that never existed. Never use its output as fact.
• No abstention. It almost never says "I don't know".
• Brittle to phrasing. Casual typing ("whats the capital of france") gives much worse answers than well-formed questions ("What's the capital of France?").
• No code, maths or reasoning ability worth the name: it fails "What is 2 plus 2?".
• Long, wandering answers. It often doesn't stop and drifts off topic after a few sentences.
• English only. Trained on web text, so it can reflect the biases and errors of that text. No safety tuning beyond what is in the chat datasets.

Intended use: research into the AXIOM architecture and small-scale training: architecture comparisons, ablations, interpretability, teaching. Not intended for factual question answering, advice, or any user-facing product.

Honest notes on the architecture

AXIOM's distinctive mechanism, validated fission, did not earn its keep in these experiments. On synthetic data it never beat a fixed mixture of experts. In this language model all 9 trial splits were rejected, so the routed cells behave like a softly weighted pair of MLPs. The attention-free recurrent memory works, and it generates faster than a same-size Transformer at batch 1. The full design and the negative results are in DESIGN.md in the repository.

Data and attribution

• FineWeb-Edu by Hugging Face, ODC-By 1.0
• smol-smoltalk by Hugging Face, Apache-2.0
• everyday-conversations-llama3.1-2k by Hugging Face, Apache-2.0

Citation

@misc{madover2026axiom1,
  title  = {Axiom-1: an experimental attention-free language model with Law Cells},
  author = {Madover, Lee},
  year   = {2026},
  url    = {https://github.com/leemadov/axiom}
}