nkthebass/tinybrainbot-350mV3-base

TinyBrainBot 350M V3 — Base

A 348M-parameter decoder-only language model pretrained from scratch on 22.7B tokens, started on a rented 8× RTX 5090 node and finished on 2× Tesla V100. This is the base (pretrained + fact-annealed) model: a text-completion model with no instruction tuning.

TL;DR: Beats the previous-generation TinyBrainBot 100M V3 base on 5 of 7 standard benchmarks and beats Supra2-100M-Base on all 7, at an average of 43.2 vs 42.2 and 40.0. It also carries an unusual history: ~20B of its 22.7B training tokens were run with every RMSNorm weight frozen at 1.0 by a precision bug. The recovery from that is documented below, because it is the most interesting thing about this model.

---

Model details

Parameters	348,342,912 (~348M)
Architecture	Decoder-only transformer, pre-norm, RMSNorm, SwiGLU MLP, RoPE
Hidden size	1152
Layers	22
Attention heads	18 query / 6 KV (grouped-query attention), head_dim 64
Context length	2048
Vocabulary	32,000 (tbb-32k-v2, tied embeddings)
RoPE theta	10,000
Training tokens	22.70B
Precision	fp16 with fp32 master weights (final phase)

Evaluation

EleutherAI lm-eval-harness, 0-shot, complete test sets. acc_norm for HellaSwag/ARC/OpenBookQA/PIQA, acc for WinoGrande/MMLU. Same harness and settings as the published 100M V3 numbers, so the comparison is like-for-like (base vs base).

Benchmark	350M V3 base	100M V3 base	Supra2-100M-Base
ARC-Easy	56.6	54.7	47.8
ARC-Easy (raw acc)	62.2	58.6	54.3
ARC-Challenge	33.3	30.1	24.8
ARC-Challenge (raw acc)	30.4	26.6	22.4
HellaSwag	35.9	32.6	36.0
HellaSwag (raw acc)	31.6	30.0	31.7
OpenBookQA	34.6	34.0	32.0
OpenBookQA (raw acc)	23.6	22.6	19.0
PIQA	67.0	66.2	65.5
PIQA (raw acc)	65.8	65.7	65.4
WinoGrande	51.5	53.0	50.7
MMLU	23.8	25.0	23.3
Average	43.2	42.2	40.0

→ 5/7 vs the 100M V3 base, losing only WinoGrande (−1.5) and MMLU (−1.2). → 7/7 vs Supra2-100M-Base (HellaSwag 35.9 vs 36.0 is a 0.1 tie at best).

Benchmark-train-split disclosure

The final data mix contains two small sets built from official train splits of benchmark datasets:

• sci-qa-train (0.5%) — SciQ, ARC-Easy/Challenge train, OpenBookQA train + core facts
• hellaswag-train (1.2%) — HellaSwag train, SWAG train

Evaluation uses the test/validation splits, which these never touch, and this is standard practice for small-model training. But the 100M V3 baseline above was trained without them, so the ARC / OpenBookQA / HellaSwag margins are not a like-for-like recipe comparison. Stated here rather than buried: if you want a clean comparison, weight those three tasks accordingly.

Training

Three phases, on rented and owned hardware:

1. Cloud pretrain — ~20B tokens on a rented 8× RTX 5090 node (Clore.ai), bf16/fp8, torch.compile, DDP. Cost about $90.
2. Local extension — continued on 2× Tesla V100 (16GB) in fp16, WSD schedule, sequence length 2048, micro-batch 2 × grad-accum 32 × 2 GPUs = 262,144 tokens/step.
3. Fact anneal — two final passes on a knowledge-dense mix (below), 1.23B tokens, LR annealed 5e-5 → 1e-5.

The frozen-norm episode

For its first ~20B tokens this model trained with all 45 RMSNorm weights frozen at exactly 1.0. The cause was precision, not the optimiser: --precision bf16 stored the weights in bf16 with no fp32 master copy. bf16 has 7 mantissa bits, so near a value of 1.0 the smallest representable step is ~0.0078, while an Adam update at lr 1e-4 is ~1e-4 — every norm update rounded straight back to 1.0. Other weights (magnitude ~0.02) have a ~50× finer ULP there and trained normally, so loss fell and nothing looked wrong.

It was found by diffing two consecutive checkpoints: 111/111 non-norm tensors changed, 0/45 norms. Switching to fp16 (which keeps fp32 masters) unfroze them.

They never fully recovered. Measured mean |w − 1| across the RMSNorm weights:

| model | mean·|w−1| | |---|:--:| | 100M V3 base (never affected) | 0.697 | | 320M V2 base (never affected) | 0.694 | | 350M V3 base (this model) | 0.017 |

After 5.5B tokens of healthy fp16 training the norms had moved 2.5% of the distance a normally-trained model's travel. So this model does its work essentially without learned per-channel scaling — and still beats a 100M trained with it. Treat the results below as a lower bound on what this recipe can do.

Training data

Three phases, each with its own mix. Every source was checked against measured token counts (tools/dataset_true_tokens.json) rather than a filesize/4 proxy, which misjudges dataset size by 0.07x-5.2x and once hid a source running at 6.9 epochs.

Phase 1 - cloud pretrain (~20B tokens), 20 sources. Web-heavy foundation.

Source	Share
fineweb-edu-100bt	49.3%
gutenberg-clean2	8.0%
bookcorpus-clean	6.1%
dclm-clean	5.6%
wiki-dense	4.2%
wikipedia-english	3.8%
cosmopedia-stanford	3.3%
cosmopedia-auto-math	2.8%
tiny-stories-v2	2.8%
qa-plain	2.3%
the-stack-smol	2.1%
cosmopedia-khanacademy	1.6%
openwebtext2	1.6%
reasoning-distill	1.2%
facts-plain	1.2%
cosmopedia-openstax	1.2%
capability-distill	0.9%
code-pyjs	0.9%
writing-distill	0.7%
math-v3	0.2%

Phase 2 - local extension, 18 sources. Shifts toward reference and synthetic-educational text as the LR decays.

Source	Share
wiki-dense	13.9%
wikipedia-english	11.8%
cosmopedia-stanford	10.7%
cosmopedia-auto-math	10.7%
qa-plain	9.6%
dclm-clean	8.5%
qa-distill	6.4%
fineweb-edu-10bt	5.3%
cosmopedia-openstax	4.3%
reasoning-distill	3.7%
facts-plain	3.2%
bookcorpus-clean	3.2%
gutenberg-clean2	2.7%
capability-distill	2.1%
reasoning-distill-chat	1.6%
tiny-stories-v2	1.1%
cosmopedia-khanacademy	0.6%
math-v3	0.5%

Phase 3 - fact anneal (1.23B tokens), 22 sources. Knowledge-dense final pass. Top 12:

Source	Share
wiki-dense	12.9%
cosmopedia-openstax	12.9%
cosmopedia-stanford	8.9%
qa-plain	7.9%
wikipedia-english	7.0%
gutenberg-clean2	5.0%
qa-distill	5.0%
facts-plain	5.0%
fineweb-edu-10bt	5.0%
soda-dialogues	4.0%
bookcorpus-clean	4.0%
facts-distill	3.0%

Sources beginning cosmopedia-, qa-, facts-, reasoning- and capability- are synthetic/distilled sets generated locally; the rest are public corpora (FineWeb-Edu, DCLM, Wikipedia, Gutenberg, BookCorpus, TinyStories). Narrative and commonsense sources total 21.1% of the anneal.

Note on the anneal share. This model spent roughly a quarter of its training on the anneal-style diet, against ~2% for the 100M V3. That is a large share for what is meant to be a finishing pass, and it is stated here because it is a plausible contributor to the benchmark profile - though this model also carries the frozen-norm bug described above, so the two cannot be cleanly separated.

Limitations

• Arithmetic is weak. This is a base model with no math tuning; use the math variant.
• MMLU and WinoGrande lag the 100M V3 base. The knowledge-dense anneal did not lift MMLU.
• Repetition loops on greedy decoding beyond ~40 tokens, as is normal at this scale.
• Hallucinates confidently — it will state a wrong country for a landmark under sampling.
• No instruction tuning. It continues text; it does not follow instructions or chat.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
t = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
ids = t("The capital of France is", return_tensors="pt")
print(t.decode(m.generate(**ids, max_new_tokens=30)[0]))