Pascaline-3M

An arithmetic-first 3M language model. Pascaline-3M scores 69.2 on ArithMark-3, the highest of any model under 50M parameters on the Open SLM Leaderboard (next best: 40.8). Only three listed models score higher, all 25–130× larger. It also has the best overall Intelligence Index under 10M parameters (16.09). Figures as of October 2026.

It is a 2,984,322-parameter base model trained from scratch. It uses the hybrid architecture of GPT-S3-3M (Gated DeltaNet-2 + gated GQA + Full Attention Residuals) with length-prefixed, units-first number tokens, and a training recipe built around synthetic arithmetic generated by our own programs. It is named after Blaise Pascal's 1642 calculating machine, whose wheels carry from the units digit upward, the same direction this model writes numbers.

Parameters	2,984,322 (tied embeddings)
Layers	8 blocks of width 144: [Gated DeltaNet-2 ×3, gated GQA (3 query / 1 KV heads, head dim 48, partial RoPE 24/48)] ×2, SwiGLU 352
Residual stream	Full Attention Residuals (each sublayer reads a softmax mix of the embedding and all earlier outputs)
Numbers	Learned digit-place embedding (66 × 144); tokenizer writes <lenN> + digits units-first (see below)
Vocabulary	4,096 byte-level BPE ids (one token per digit) + 8 number-length tokens = 4,104
Context	1,024 tokens
Training tokens	10.0B (constant LR) + 1.2B decay = 11.2B
Hardware	one laptop RTX 5070 Ti, ~12.2 h of training

Results (zero-shot)

Benchmark	Metric	Score
PIQA	acc_norm	56.15
HellaSwag	acc_norm	27.79
ARC-Easy	acc_norm	34.18
ARC-Challenge	acc_norm	22.44
ArithMark-3	acc_norm (bencharithmark-3.py)	69.20
Open SLM Leaderboard Intelligence Index		16.09

Self-reported with the official tools; see the Open SLM Leaderboard for the verified figures.

PIQA / HellaSwag / ARC: lm-evaluation-harness 0.4.13, --model hf, trust_remote_code=True, dtype=float32. ArithMark-3: AxiomicLabs' bencharithmark-3.py, --dtype float32. Run on a Tesla T4 (transformers 5.16.1, torch 2.11). float16 gives the same numbers within 0.2 points (II 16.08). Our training-time scorer gave PIQA 56.15, HellaSwag 27.91, ARC-E 34.22, ARC-C 22.53, ArithMark-3 69.1 (II 16.14).

How this checkpoint was chosen. The constant-LR run was branched at 4B, 8B and 10B tokens into eight 1.2B-token decay runs with different decay data. This is the one with the highest Intelligence Index on the benchmarks above, like GPT-S3-3M's release of its best internal checkpoint. The eight decay branches scored between 11.5 and 16.1; the four branches with the final synthetic-arithmetic recipe scored 15.4–16.1.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ReckonerLabs/Pascaline-3M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("ReckonerLabs/Pascaline-3M", trust_remote_code=True)
ids = tok("Ava had 43 shells and found 19 more, so she had", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=6, do_sample=False, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0][ids.shape[1]:]))   # " 62 shells"

• Remote code. Both the model and the tokenizer use remote code. The model is pure PyTorch: no flash-linear-attention needed, no KV cache (generation recomputes the prefix).
• Padding. Pad on the right (the default of lm-eval and bencharithmark-3.py); attention_mask is ignored.

Number tokens

Digits are single tokens. Every run of digits is written as a length token followed by its digits in reverse order: 62 → <len2> 2 6, 1234 → <len4> 4 3 2 1. The model learns the size of a number first, then produces it units digit first, so carries flow forward in time. Digit runs next to : (clock times) are left as they are. The tokenizer applies this on encode and undoes it on decode, and the model computes each digit's place (units, tens, …) from the ids. Units-first digits inside the tokenizer follow Atom2.7m; the length prefix is new here. In our ablation (4 seeds, 1B tokens) it raised ArithMark-3 by 9.2 points and the Index by 2.7 over normal digits, with the other benchmarks unchanged.

A note for interpreting ArithMark-3: its wrong options are the answer ±1 or ±2, so every option has a different units digit, which a units-first model emits first. This representation therefore helps there beyond full arithmetic ability.

Training data

Constant-LR phase:

Source	Share
ClimbMix	30%
TxT360-v2	28%
OpenMathInstruct-2 (MATH part)	12%
FineWeb-Edu	10%
UltraData-Math	8%
FactBase-DCLM	5%
OpenMathInstruct-2 (GSM8K part)	4%
Cosmopedia WikiHow	3%

Decay (1.2B tokens), mixed per step:

Source	Share
Cosmopedia WikiHow	40%
Our synthetic arithmetic	30%
OpenMathInstruct-2 GSM8K	9.4%
Cosmopedia OpenStax / Khan / Stanford	3.7%
ClimbMix	3.4%
FineWeb-Edu	3.4%
Orca-Math	2.6%
TxT360	1.7%
FactBase	1.7%
OpenMathInstruct-2 MATH	1.7%
UltraData-Math	1.7%
SciQ (train split)	0.5%
OpenBookQA (train split)	0.09%

Third-party synthetic data. OpenMathInstruct-2 (NVIDIA), Orca-Math (Microsoft) and Cosmopedia (Hugging Face) are LLM-generated datasets.

Our synthetic arithmetic is written by our own programs: grade-school word problems and equations with exact answers. It mixes four generators in equal parts:

• v2: broad topics: money, time, lengths, patterns, comparison, arrays, shapes, fractions.
• v3: compositional gain/loss stories, carry/borrow-balanced operands, clock arithmetic.
• v4: relations ("N more/fewer than X"), differences asked many ways, sharing / grouping division, measures, minimal pairs, equation tails.
• v4.1: multi-step chains with distractor sentences, two-hop relations, perimeters.

Phrasings are composed from parts split into train and held-out sets.

Public sample. A 5,000-problem sample of this stream, with its overlap report, is at ReckonerLabs/Pascaline-3M-synthetic-sample.

Fairness checks against all ArithMark-3 rows, enforced at generation:

• 0% exact 8-gram and 13-gram overlap;
• under 0.01% overlap of number-masked 6-grams (shared first names);
• a banned list of ArithMark sentence frames, which is never generated.

No benchmark item, benchmark generator or train split of an evaluated benchmark was used. SciQ and OpenBookQA are not evaluated here. WikiHow, the GSM8K-derived sources, Orca-Math, FactBase, SciQ and OpenBookQA were filtered with a 13-gram overlap check against all five test sets. The large web sources were scanned on a sample.

Credits

• Architecture: GPT-S3-3M by AxiomicLabs (Apache-2.0).
• Decay-phase recipe ideas: pulvis-v1/v2 (bench-labs).
• Units-first digit tokens and place features: Atom2.7m.
• Precedent for generated arithmetic data: MathBananaMind-1.1.
• Kernels used in training: flash-linear-attention.

Limitations

• English only. A 3M model: it confabulates freely and should not be relied on for facts.
• Its arithmetic is grade-school word problems and multi-digit addition and subtraction. It is not a reasoning model.
• Benchmarks score it by ranking given answers. Free-form generation of an answer is less reliable: in our spot checks 3-digit sums and differences and familiar story phrasings come out right, while unfamiliar phrasings, division and measurement problems are often off by one or more.
• Base model only: no instruction tuning, no safety tuning.