TinyBrainBot-10M-v2-Base

A 9.9M-parameter general-purpose base model trained from scratch on 10.16B tokens, as a plain Llama (LlamaForCausalLM): no custom code. It succeeds TinyBrainBot-10M-Base and is built like TinyBrainBot-3M-v2-Base: web text, Q&A, facts and commonsense first, with arithmetic as a small part of the mix.

Scores vs other models of 10M parameters or fewer

Zero-shot, full test sets, acc_norm. The (raw acc) row shows plain accuracy. The Intelligence Index is the Open SLM Leaderboard's; the other models' numbers come from that leaderboard.

Model	Params	Int Index	HellaSwag	ARC-E	ARC-C	PIQA	ArithMark-3
TinyBrainBot-10M-v2-Base (this model)	9.90M	13.76	27.75	36.87	25.17	58.65	49.50
(raw acc)		13.17	27.26	39.65	18.34	59.25	49.50
Ethoric-10M	9.97M	12.94	28.75	37.58	24.66	59.85	41.60
min-spark-1.7	9.98M	12.71	29.65	39.90	24.15	60.39	36.60
ForgePlex-M2.2-10M	9.99M	12.12	29.19	39.77	22.70	59.79	37.40
heron1i-flash-lite-v2	9.92M	11.11	28.86	38.89	23.38	58.54	36.70

More arithmetic results

Arithmetic sets the model never trained on (Gemma and Llama were never used to write training data), all acc_norm:

Set	Score
ArithMark-3 (public)	49.5
Arithmetic stories written by Gemma (544)	38.4
Arithmetic stories written by Llama (1,093)	45.6
23 arithmetic topics ArithMark-3 lacks, new wordings (1,380)	39.6
Held-out number formats: large / commas / money	56.2 / 63.9 / 60.6
Bare equations from ArithMark-3 ("43 + 19 =")	49.1
ArithMark-2	34.4

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-10m-v2-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-10m-v2-base")
ids = tok("The best way to keep bread fresh is to", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=24, do_sample=False)[0]))

GGUF files for llama.cpp, LM Studio and Jan: tinybrainbot-10m-v2-base-F16.gguf (20 MB) and tinybrainbot-10m-v2-base-Q8_0.gguf (11 MB).

This is a base model: it continues text and does not chat. At this size expect short, simple text.

Model

Parameters	9,902,336 total (8.85M non-embedding + 1.05M tied embedding)
Shape	hidden 256, 12 layers, 4 query heads of 64 with 2 KV heads (GQA), SwiGLU 704, RMSNorm, RoPE
Vocabulary	4,096-token byte-level BPE: digits split one per token, whitespace kept exactly
Context	1,024 tokens

Training

10.16B tokens: 8.57B of pretraining at constant learning rate, then a 1.59B-token decay.

Pretraining (8.57B)	Share
DCLM-Ultra (filtered, deduplicated DCLM)	25%
Cosmopedia (Stanford, web samples, WikiHow, OpenStax, Khan Academy)	24.6%
Q&A and facts (our distilled plain-text sets)	15%
FineWeb-Edu	10%
Arithmetic (our drills, Orca-Math, UltraData-Math)	9%
TinyStories	8%
Commonsense, how-to and explanatory writing (our distilled sets)	4.4%
Wikipedia leads	4%

Decay (1.59B)	Share
Q&A and facts	25.5%
Cosmopedia	22.9%
DCLM-Ultra-1.1 (a stricter cut of DCLM-Ultra)	15%
Commonsense, how-to and writing	12.8%
Arithmetic	7.6%
TinyStories	7.3%
Wikipedia leads	5%
FineWeb-Edu	4%

The arithmetic comes from several sources so the skill is not tied to one wording: drills generated by our own programs (number formats, number words, whitespace variants, negatives, order of operations, tables, unknown positions, distractors; every answer computed in Python), Orca-Math and UltraData-Math word problems, and a very small amount of story problems in LLM-written sentence frames. No benchmark training or test data was used.

Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 3e-3 with 1,000 warmup steps, warmup-stable-decay schedule, 131k tokens per step, fp16 mixed precision, trained on 2x V100 in ~8 hours.

How this checkpoint was chosen. Several decay branches with different arithmetic shares were run from the same checkpoint (step 65,303); this is one of them, and the published weights are that branch's final checkpoint, not a merge.

Evaluation

Float32, lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-10m-v2-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-10m-v2-base --dtype float32

Limitations

At 10M parameters this is a toy-scale model. It knows few facts, loses coherence after a few sentences, and its arithmetic is unreliable beyond simple problems. HellaSwag and ARC are close to chance for every model this size.