SlayerLab/gollem-v5-ckpts

GoLLeM-v5 — Tiny English Language Models (16M-64M)

Research checkpoints of sub-100M-parameter English language models, GPT-style decoders (nanoGPT lineage) trained for the Glint Tiny-ML Leaderboard. The repository holds a controlled scaling study (tokens, width, optimizer) and a set of data-attribution experiments on the 64M model (paired continued-training runs that differ only in data).

Model details

• Architecture: 16M/32M = decoder-only Transformer (nanoGPT lineage), learned positional embeddings, tied input/output embeddings. 64M flagship = Qwen3-style decoder (RoPE θ=100k, SwiGLU, RMSNorm, QK-Norm, value residuals).
• Sizes: 16M = 6 layers / d_model 408 / 6 heads (17.4M); 32M = 6 layers / d_model 576 / 9 heads (31.6M); 64M flagship = 14 layers / d_model 576 / 9 heads (62.9M).
• Context length: 1024 tokens.
• Tokenizer: BPE, vocab 12288 (tokenizer.json), shared across all checkpoints.
• Training: 16M/32M: AdamW, lr 6e-4 -> 6e-5 (cosine), batch 64 x 1024, seed 1337, bf16 (RTX 5090). 64M flagship: Muon (muon-lr 0.02) + AdamW for non-matrix parameters, same cosine schedule, 400k steps.

Checkpoints

checkpoint	params	shape	tokens	BLiMP	ARC-Easy	WikiText-2 BPB
bpe16m_3.2B/ckpt.pt	17.4M	L6 d408 h6	3.2B	67.40⁺	38.22	1.2161
bpe16m_6B/ckpt.pt	17.4M	L6 d408 h6	6B	68.92⁺	39.10	1.1943
bpe16m_10B/ckpt.pt	17.4M	L6 d408 h6	10B	70.36°	39.52	1.1815
bpe32m_baseline/ckpt.pt	31.6M	L6 d576 h9	10B	70.08ˢ	42.59	1.124
run_16m_expanded/ckpt.pt (16M board entry)	17.4M	L6 d408 h6	16B†	70.08	40.91	1.4193
run_32m_16b/ckpt.pt (32M board entry)	31.6M	L6 d576 h9	16B‡	73.48	44.44	1.3441
run_149m/ckpt.pt (scaling ref)	149M	—	10B§	76.99°	49.66	1.2052
run_32m_18b/ckpt.pt (slope-check)	31.6M	L6 d576 h9	18B¶	72.38°	44.70	1.3431
Muon 32M (results only: glint_32m_muon_results.json, checkpoint not published)	31.6M	L6 d576 h9	16B‖	72.29°	42.89	1.3866
v1_muon/ckpt_400k.pt (64M flagship)	62.9M	L14 d576 h9	13.1B★	75.83	47.94	1.246

BLiMP harness fix (2026-09-25). Our evaluation script matched BLiMP configs by substring, so 6 phenomena were counted twice (73,000 pairs instead of 67,000). It is fixed in glint_parity_eval.py (exact config match + assertion of 67,000 pairs). The three board entries (bold BLiMP) were recomputed with the fixed script; for each we also reproduced the old 73,000-pair variant, which matches the previously published number to two decimals, so the difference comes from the loader alone (+0.16 to +0.46 pp before the fix). Other rows are earlier results, not recomputed:

• ° computed with the pre-fix 73,000-pair loader (pair count recorded in the results file); on the three recomputed models the bias was +0.16 to +0.46 pp upward, expect similar here;
• ˢ BLiMP on a 4,000-pair random sample, so the error is random (SE ~0.7 pp), not a systematic bias;
• ⁺ the results file is not archived, so the pair count is unknown; treat as indicative only.

† 16M board entry = expanded 8.29B-token corpus (~1.9 epochs). eff 74.33 with the fixed BLiMP (70.08 / 40.91 / byte_ppl 2.6746).

‡ 32M board entry = 32M at 16B tokens on the ARC-MIX corpus. eff 75.41 with the fixed BLiMP (73.48 / 44.44 / byte_ppl 2.5386). It broke the 16M BLiMP ceiling (~70) seen in the token scan: more capacity plus tokens moved both axes.

§ 149M = scaling reference only (under-trained at 67 tok/param). Highest raw scores in the older runs, but the size multiplier of the efficiency score falls with size, so it ranks lower on eff.

¶ Slope-check = 32M at 18B tokens on the same ARC-MIX corpus as the 32M entry. BLiMP 72.38 (lower than at 16B) with byte_ppl flat: more epochs over a fixed corpus did not help. Diagnostic run.

‖ 32M with Muon on the expanded corpus. This run changed optimizer and corpus at once, so it cannot isolate Muon. The clean optimizer comparison was later run at 64M (see ★). Diagnostic run.

★ 64M flagship (v1 Muon) = Qwen3-style decoder + value residuals, Muon optimizer, ARC-MIX 9.42B corpus, 400k steps = 13.1B tokens (~1.4 epochs). Recomputed with the fixed harness: ARC-Easy 47.94 / BLiMP 75.83 / WikiText-2 byte_ppl 2.372 (BPB 1.246) → eff 75.81. The maintainer's independent re-benchmark (PR #78, sha256 59f982c1… matched) reproduced ARC-Easy 47.94 exactly; its BLiMP 75.99 matches our 73,000-pair variant to two decimals, so it was most likely computed with the same (pre-fix) script. Earlier versions of this card cited eff 77.51 (byte_ppl 2.016, from a wrong bytes-per-token factor 4.755 instead of 3.8605) and later eff ~75.9 (BLiMP 75.99); both are superseded. Muon was at least as good as AdamW in a clean 64M A/B (identical data, architecture and seed; only the optimizer differs); those numbers pre-date the BLiMP fix.

Data-attribution experiments (64M, 2026-09-25)

Question: does changing the training data in the late phase of training move the efficiency score? Method: two identical 64M models continue from the same flagship checkpoint (step 320k) with the same recipe, seed and number of steps (80k, to step 400k); they differ only in the data. Both are evaluated at 340k / 360k / 380k / 400k with the board protocol and a paired bootstrap over items. Each checkpoint folder has its own README with the full data specification.

run (folder)	data	ARC-E	BLiMP	wiki byte_ppl	eff @400k	note
flagship (v1_muon/)	ARC-MIX, full run 0→400k	47.94	75.83	2.3718	75.81	reference
control (ctrl_arcmix_resume/)	ARC-MIX	47.69	76.29	2.3717	75.88	re-draws the flagship's first 80k steps of data (the data sampler restarted from its seed on resume)
fork-B (forkB_arcmix_edu/)	ARC-MIX + educational	46.34	77.20	2.3982	75.66	flawed build: took only the head of the ARC-MIX file and a different document separator. Not a test of educational data
A (r3A_arcmix_edu_clean/)	45% ARC-MIX + 55% educational (clean build), 2.55B-token pool	47.31	76.22	2.3762	75.71	clean pair with B
B (r3B_arcmix_qa2x/)	ARC-MIX with Q&A documents ×2, 2.62B-token pool	47.26	75.90	2.3712	75.60	clean pair with A

• A − B: Δeff +0.11 (95% CI −0.30 to +0.51); at the four checkpoints +0.46 / −0.13 / −0.02 / +0.11. No difference on eff. Educational data consistently made WikiText-2 slightly worse (by more than the run-to-run spread); BLiMP was slightly higher but within noise.
• Conclusion: at these data doses and 80k steps of the cosine tail (learning rate 19% → 10% of peak), the effect of the data on eff is below ~0.5. None of the runs beats the flagship beyond noise, so none is submitted to the board. This is an upper bound on the effect at this scale of intervention, not evidence that data does not matter.
• In progress: an anchor run (ARC-MIX sample of the same pool size, Q&A ×1) and B with a second seed, to measure the Q&A effect cleanly and the run-to-run noise directly.

Usage

These are raw nanoGPT-lineage checkpoints (plain torch state dicts), not transformers AutoModel weights. The model class and a board-scoring harness are included in this repo:

• train_gpt_ref.py — GPT definition (rebuild the GPT of the tabled shape, load_state_dict, trim logits to vocab 12288).
• glint_parity_eval.py — Glint board-scoring forward (256-token clip, raw log-prob) for BLiMP / ARC-Easy / WikiText-2 (BLiMP loader fixed 2026-09-25, see above).

import torch
from tokenizers import Tokenizer
tok   = Tokenizer.from_file("tokenizer.json")          # BPE-12k, vocab 12288
ckpt  = torch.load("bpe16m_10B/ckpt.pt", map_location="cpu")
state = ckpt.get("model", ckpt)                        # load into the GPT from train_gpt_ref.py

64M checkpoints (v1_muon/, and the data-attribution folders) — Qwen3-style arch, config embedded in the checkpoint. Load with the cfg from ckpt["config"], not the legacy 5-arg nanoGPT path. glint_parity_eval.py's loader is cfg-aware for both lineages.

import torch
from types import SimpleNamespace
from train_gpt_ref import GPT
ck  = torch.load("v1_muon/ckpt_400k.pt", map_location="cpu", weights_only=False)
c   = ck["config"]                        # self-describing: vocab 12288, n_layer 14, n_embd 576, n_head 9, block 1024
cfg = SimpleNamespace(**c)                # Qwen3 flags: rope θ100k / swiglu / qk_norm / value_residual
m   = GPT(c["vocab"], c["n_layer"], c["n_embd"], c["n_head"], c["block"], cfg)
m.load_state_dict(ck["model"], strict=False)   # tied head.weight
m.eval()

Reproducing training. train_gpt_ref.py is a general nanoGPT-style causal transformer whose vocabulary and bin dtype are CLI-parameterized. These checkpoints were trained in BPE-12k mode (--vocab 12288 --dtype uint16), not the script's byte-level defaults:

# 16M @ expanded corpus
python train_gpt_ref.py --data-dir <corpus> \
  --n-layer 6 --n-embd 408 --n-head 6 --block 1024 \
  --batch 64 --steps 244141 --lr 6e-4 --min-lr 6e-5 \
  --vocab 12288 --dtype uint16 --seed 1337
# 32M: --n-embd 576 --n-head 9 (same vocab 12288 / uint16 / BPE tokenizer)

The script's byte-level defaults (--vocab 256 --dtype uint8) and its header comment reflect its origin as a standard-GPT control compared against an experimental BDH (fast-weights) architecture; the leaderboard models here are the standard causal transformer in BPE mode and do not use BDH.

Training data

SlayerLab/minimal-en-corpus-5b — ~5.40B BPE-12k tokens, English, decontaminated against the benchmark test sets. A broad high-quality mix: FineWeb-Edu, DCLM, StackExchange, open-web-math, FineMath, scientific papers, books/Gutenberg, code, CC-News. A decontaminated expansion to ~8.3B tokens (added FineWeb-Edu + OpenStax science) feeds later runs.

ARC-MIX (9.42B). The 32M board entry and the 64M runs use ARC-MIX: a reasoning/knowledge-enriched expansion of the base mixture (9,417,035,832 BPE-12288 tokens) with ARC-relevant science/reasoning/QA web content upweighted (gold ~3×, related ~2×) over the decontaminated base. Same tokenizer, same 13-gram benchmark decontamination (WikiText-2 / BLiMP / ARC test sets).

Earlier FineWeb-Edu-dominant corpus (v2). A from-scratch 64M run on a FineWeb-Edu-dominant blend scored lower on ARC than ARC-MIX. That build used a different document separator than the rest of our data and a changed recipe (WSD schedule, z-loss, logit cap), so the comparison is confounded and is not evidence against educational data trained from scratch.

Evaluation

All metrics use the Glint benchmark protocol (Glint-1.3/benchmark.py), i.e. the board-comparable definitions:

• BLiMP — 67 configs (train split, 67,000 pairs), each sentence clipped to the first 256 tokens, raw sentence log-prob preference (good > bad), no length normalization.
• ARC-Easy — test split, zero-shot, raw accuracy over LL(question + choice) - LL(question).
• WikiText-2 — byte-normalized (byte_ppl / BPB; bytes-per-token 3.8605 for this tokenizer on WikiText-2-raw-v1 test).

A generic lm-eval-harness run scores BLiMP/ARC roughly 2-3pp higher than this protocol; the numbers here are the board-comparable ones.

Board status. The Glint board still lists the earlier values for our entries (64M: BLiMP 77.84 / byte_ppl 2.016; 32M: BLiMP 73.77; 16M: BLiMP 70.53). The correct values are the ones in this card. With the board's own efficiency formula (eff = mean(BLiMP, ARC-Easy, normalized-WikiText-2) × size multiplier, checked line-for-line against the Space source) they give eff 75.81 (64M), 75.41 (32M), 74.33 (16M).

Key findings

• Tokens drive BLiMP at small size, up to a size-specific ceiling. At 16M, BLiMP rose with tokens (3.2B → 10B) and then flattened near ~70 on the expanded corpus; 32M at 16B tokens moved past that ceiling.
• Capacity and knowledge drive ARC. 16M → 32M at matched tokens lifted ARC-Easy by ~3pp; 64M lifted it further (47.94).
• Bigger is not automatically better on eff. Raw scores keep rising with size, but the efficiency score multiplies by a size bonus that shrinks with parameter count. In our runs 64M is the best eff entry (75.81 vs 75.41 at 32M); a 128M run improves raw quality but the smaller multiplier cancels most of it.
• Muon was at least as good as AdamW at 64M in a clean A/B (same data, architecture, seed; numbers pre-date the BLiMP fix).
• Late-phase data changes move eff by less than ~0.5 (see Data-attribution experiments).
• Value residuals are part of the 64M stack. The board's #1 model attributes ~+6 ARC-Easy to them in its own card; we have not isolated this effect ourselves.

Roadmap

• Continue training the 64M flagship in segments with a constant-learning-rate phase and a final decay, comparing two data mixes per segment on validation sets (never the board test sets) and keeping the winner only when it wins beyond measured run-to-run noise.
• A clean from-scratch data comparison (same recipe and separator) and synthetic science data for ARC are candidates; both are pending.

Limitations

Base (not instruction-tuned) research models at 16-64M parameters, English-only. Expect limited factual knowledge and coherence; not intended for production use.

Provenance

Trained on RunPod RTX 5090. Full evaluation artifacts and protocol details are available on request.