TinyBrainBot-3M-Loop-v4

A 2.96M-parameter math-first language model that reuses its layers. Four transformer blocks are run three times in a loop, so every token passes through 12 layers of computation while only 4 layers of weights are stored. Trained from scratch on 8.5B tokens. Its arithmetic carries over to bare expressions, order of operations, large numbers, thousands separators, money, negative numbers and unusual whitespace, and it covers many more grade-school topics than v3 (remainders, unit conversions, percent, rounding, missing numbers, comparisons).

Benchmark (0-shot, full set)	acc_norm
Open SLM Leaderboard Intelligence Index	19.36
(raw acc)	19.69
HellaSwag	27.17
(raw acc)	26.92
ARC-Easy	31.31
(raw acc)	32.15
ARC-Challenge	22.70
(raw acc)	17.92
PIQA	53.86
(raw acc)	55.82
ArithMark-3.0	91.2
(raw acc)	91.5

lm-eval 0.4.13 in float32 for HellaSwag, ARC and PIQA; the official bencharithmark-3.py (float32) for ArithMark-3.0.

Arithmetic beyond one benchmark

Every column below is a zero-shot, log-likelihood multiple-choice math test, all scored by us with the official bencharithmark-3.py scoring in float32, so every model is measured the same way (numbers can differ slightly from leaderboard listings).

Model	Params	ArithMark-3	AM3, words	AM3, bare equations	ArithMark-2	ArithMark-1	Held-out stories	Held-out, large numbers	Held-out, commas	Held-out, money
TinyBrainBot-3M-Loop-v4	2.96M	91.2	57.3	99.7	81.3	38.7	88.2	92.7	92.3	94.9
TinyBrainBot-3M-Loop-v3	2.96M	91.6	53.9	97.3	83.9	35.8	88.4	93.1	91.4	93.7
TinyBrainBot-3M-Loop	2.96M	89.3	27.3	88.2	58.1	29.4	81.0	39.2	37.7	42.4
Inferno-Exp-1.7	394M	85.9	59.1	86.9	59.9	47.9	74.0	62.7	74.4	70.8
Pascaline-3M	3M	69.2	27.6	99.7	66.5	36.2	43.0	39.9	37.8	42.3
SmolLM2-135M	135M	38.8	34.2	38.6	33.2	33.1	34.7	37.4	39.5	38.6
GPT-S3-3M	2.97M	38.9	25.8	36.2	31.0	31.1	34.6	34.3	34.4	35.9

• Numbers in words and bare equations are ArithMark-3's own problems with only the surface changed.
• ArithMark-2 uses its official metric (raw log-likelihood); its official benchmark_arithmark-2.0.py gives the same results.
• Held-out stories: about 1,600 ArithMark-style problems written by Gemma-3-27B and Llama-3.3-70B, models that wrote none of our training data, with every number and answer set by Python.
• Held-out, large numbers / commas / money: the 958 held-out sum and difference stories with every amount scaled into 4-6 digit numbers, thousands separators ("74,986") or dollars and cents ("$70.67"). The previous versions were near chance (25%) on these.
• ArithMark-1: its answer key mostly evaluates expressions strictly left to right (e.g. 14 * -17 + -1 * -18 = 4302), while ArithMark-2 and ordinary arithmetic use operator precedence (that expression is -220). This model follows precedence, so it scores lower on ArithMark-1's longer expressions (71% on its two-operand stage, falling with each extra operator).

How it works

• Recurrent depth. 4 shared blocks (width 256, 4 attention heads of 64 with 2 KV heads, SwiGLU 560, RoPE, RMSNorm) are applied 3 times. Each pass adds a learned loop embedding and a gated copy of the input embedding. Training sampled 2 to 4 loops per step, and you can change the count at inference (model.model.eval_loops = 4); 3 is best.
• Factorized embedding. The 4,104-token vocabulary uses 96-dimensional embeddings projected up to 256 (and down again before the tied output layer), which frees about 630k parameters for the 4th block.
• Numbers. A units-first digit tokenizer writes every number as a length token plus its digits, smallest place first, and a digit-place embedding tells the model which place each digit is in.

Training

Stage 1 (4.46B tokens): the TinyBrainBot-3M-Loop run up to the start of its learning-rate decay.

Part	Share	What
Q&A and facts	25%	our distilled Q&A, short facts, explanations
Arithmetic	20%	word problems: about 150k different story wordings written by ling-3.0-flash and Mistral-Small-3.2, each checked by solving it, with every number and answer filled in by Python; plus our own relation (more/fewer than) and story generators
Synthetic textbooks	15%	Cosmopedia (Stanford, OpenStax)
Stories	15%	TinyStories
Commonsense	10%	Cosmopedia WikiHow, our everyday-commonsense prose
Web	10%	FineWeb-Edu
Reference	5%	Wikipedia

Stage 2 (4.0B tokens, math-first): constant learning rate for 2.67B tokens, then a decay to zero over the last 1.33B.

Part	Share	What
Story arithmetic	21%	about 316k verified story wordings from three LLM writers (ling-3.0-flash, Mistral-Small-3.2, Mistral-Nemo), some numbers in words, about half of the new ones with an unrelated number that must be ignored
Number formats	10%	large numbers, thousands separators, money with cents, decimals, chatty setups
Topics	13%	remainders, unit conversions, twice / half as many, percent, averages, rounding, days of the week, fractions, place value, odd/even, elapsed time
Missing numbers, distractors, layouts	8%	"? + 19 = 62", start- and change-unknown stories, distractor numbers with minimal pairs, question-first and Q/A layouts, sequences, tables, comparing expressions
Bare arithmetic	6%	order of operations, parentheses, negatives, 2-5 operands, about 19 forms
Whitespace variety	6%	the same problems with double spaces, newlines or tabs before numbers
Number words, negatives, relations, quick answers	10%	our word, negative-number, relation and quick-answer generators
Q&A and facts	17%	as in stage 1
Cosmopedia, FineWeb-Edu	9%	as in stage 1 (more WikiHow)

No benchmark training or test data was used, and none of our generators copy any benchmark's problems, code or sentences.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nkthebass/tinybrainbot-3m-loop-v4"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
ids = tok("Mia had 46 stickers. She gave 18 to her brother, so now Mia has", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=8, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:]))

trust_remote_code=True is required (custom architecture and tokenizer). There is no KV cache: generation recomputes the prefix, which is fast at this size. Padded batches work on either side (the model honors attention_mask). There is no GGUF, because llama.cpp does not support the looped architecture.

To reproduce the scores:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-loop-v4,dtype=float32,trust_remote_code=True --tasks arc_easy,arc_challenge,hellaswag,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-loop-v4 --dtype float32

Limitations

• A 3M base model: it writes simple English and makes up facts. It is not a chat model.
• Arithmetic is strongest on short problems. On our held-out topic tests it is still weak at days of the week, durations between clock times, averages, remainders, money written as "dollars and cents", reading plain-text tables, unusual layouts and ignoring an unrelated number in a story.
• 1,024-token context.