Maggio33/GoLLeM-v5-16M

GoLLeM-v5 16M

Smallest model of the GoLLeM-v5 scaling study (board entry).

Author: Arkadiusz Słota / Fabryka AI. Part of the GoLLeM-v5 series of small English language models trained from scratch. All checkpoints of the series, including intermediate ones, are archived in SlayerLab/gollem-v5-ckpts.

Model

• 17.4M parameters; 6 layers, d_model 408, 6 heads; context 1024 tokens.
• Architecture: decoder-only Transformer (nanoGPT lineage), learned positional embeddings, tied input/output embeddings.
• Tokenizer: BPE, 12,288 tokens (tokenizer.json, sha256 3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1), 3.8605 bytes per token on WikiText-2 test.
• Weights: model.safetensors (its sha256 is shown by the Hub for the LFS file), converted tensor by tensor from run_16m_expanded/ckpt.pt (sha256 5b1adca516707e645a3b5206deb95106299c859dad9e757f40f842ad185c33b0) without changing any weight. Only weights are published here: no pickled checkpoint, optimizer state or training logs.

Training

• Recipe: AdamW, lr 6e-4 cosine to 6e-5, batch 64 x 1024 tokens, 244,141 steps = 16B tokens (~1.9 epochs over the expanded corpus), seed 1337, bf16, one RTX 5090.
• Data: Expanded corpus (8.29B BPE tokens): the base mixture SlayerLab/minimal-en-corpus-5b (FineWeb-Edu, DCLM, StackExchange, open-web-math, FineMath, scientific papers, books/Gutenberg, code, CC-News) plus additional FineWeb-Edu and OpenStax science textbooks. Filtered against benchmark test sets with 13-gram matching at build time. The split 8.29B = base 5.40B + FineWeb-Edu 2.76B + OpenStax 0.14B is our calculation from the ARC-MIX composition in the archive card (ARC-MIX 9.42B = this corpus + 1.12B of ARC-related FineWeb-Edu copies), not a value from a build manifest.
• Result checkpoint: The published checkpoint ckpt.pt is the end of the run (16B tokens).
• At 16M, BLiMP rose with tokens (3.2B to 10B) and then flattened near ~70 on this corpus.

Evaluation (our measurement, not official)

Measured by us on the result checkpoint with the Glint Tiny-ML board protocol (BLiMP: 67 configs, 67,000 pairs, sentences clipped to 256 tokens, raw log-prob preference; ARC-Easy: test split, zero-shot, LL(question + choice) - LL(question); WikiText-2: byte-normalized perplexity). These are not official scores.

Benchmark	Score
ARC-Easy (acc, %)	40.91
BLiMP (acc, %)	70.08
WikiText-2 byte perplexity (lower is better)	2.6746
eff (board formula, includes a size multiplier)	74.32
MultiBLiMP-pl (acc, %, 3,272 pairs)	60.57

• eff = mean of BLiMP, ARC-Easy and normalized WikiText-2, times a size multiplier that is larger for smaller models; compare eff only between models under the same board formula.
• This is an English model. Its Polish MultiBLiMP score is close to the length baseline (always choosing the shorter sentence gives 59.67 %); it is reported for completeness, not as Polish ability.
• Board: https://track.fabryka.ai/models (row "GoLLeM-v5 16M").
• Source of the numbers: liczby_v5.json (sha256 35c9022ee9dbd96d588093c6fc69604d0eaabd18843458671205db41e234dde8), built from the anonymous board read-out and our PL measurement files.

Benchmark overlap disclosure

A scan of the finished training corpus found that 7 of 62 WikiText-2 test articles appear almost completely in it (as web copies) and 30 of 2,376 ARC-Easy test questions have a match (mostly short factual sentences). On the 16M, 32M and 64M Muon v1 models, removing them changed WikiText-2 byte perplexity by less than 0.01 and ARC-Easy by -0.07 to +0.19 points, within noise (details in the root card of the archive repository). BLiMP was covered only by the build-time 13-gram filter.

How to use

The weights are a plain PyTorch state dict saved as safetensors; they are not a transformers AutoModel. The model class is GPT in train_gpt_ref.py in SlayerLab/gollem-v5-ckpts; build it from config.json and load the state dict, then trim logits to 12,288.

Limitations

• Small base model trained from scratch for research: not instruction-tuned, limited factual knowledge and coherence, not for production use.
• English only.
• Single seed; scores come from one checkpoint and our own implementation of the board protocol.

Training data

Base mixture: SlayerLab/minimal-en-corpus-5b (research-mix-5b, 5.0B tokens of its own tokenizer). It is an aggregate corpus: no unified licence is applied and each source keeps its upstream terms. Sources as named in its manifests/mixture.json:

source	tokens (M)	upstream licence
fineweb-edu	1,101	ODC-By 1.0
dclm-baseline	801	CC BY 4.0
starcoderdata	469	The Stack terms (per-file licences, opt-out)
stackexchange	447	CC BY-SA
loc-pd-books	402	CC0 1.0
wikipedia	362	CC BY-SA 3.0 / GFDL
open-web-math	228	ODC-By
finemath	221	ODC-By
project-gutenberg	210	public domain in the US
scientific-papers	206	unknown
ultrachat	200	MIT; text generated by OpenAI models
wildchat	151	ODC-By; text from conversations with OpenAI models
cc-news	150	unknown (news articles under copyright)
tiny-textbooks	51	Apache 2.0; model-generated text
open-subtitles	1	film subtitles, under copyright

• Upstream repositories are not recorded in the corpus manifest; licences are those of the Hub datasets matching these source names (not confirmed against the build).
• Added on top of the base mixture for this series: FineWeb-Edu (ODC-By 1.0) and OpenStax textbooks (CC BY 4.0).
• Aggregate corpus; each source keeps its upstream terms. Commercial use: review upstream terms, in particular CC-News, scientific papers, StarCoderData and model-generated chat data.

Licence and attribution

• Weights: no single licence is claimed for the weights yet; they were trained on data under the licences listed in the Training data section above, and any use must respect those terms (license id other, name gollem-v5-see-training-data).
• OpenStax textbooks: CC BY 4.0, © Rice University, https://openstax.org; titles and editions in OPENSTAX_ATTRIBUTION.md in this repository.