Swordies-22M

A 22.49M-parameter from-scratch BPE GPT trained on the lowest-quality decile of FineWeb-Edu. This is a data-quality ablation, not a usable language model. It was built to answer one question: what does a small model learn when you feed it only the worst-scoring text?

Read this first: the model is degenerate on purpose. Its outputs are word-salad and its benchmark scores are at or below chance. That is the finding, not a bug. If you are looking for a small model that actually works, this is not it — see the finding below for what it does show.

The finding

The bottom decile of FineWeb-Edu (quality score ≤ 2.578) is a different distribution, not weaker text. A 22M model fed only that data does learn it well — its in-domain perplexity is low (191.65 on the held-out bottom-decile slice, 5.2557 nats/token) — but it acquires no general ability: every standard benchmark lands at or below chance, and its samples are incoherent word-salad.

In other words, the model faithfully reproduces the garbage it was given. Low in-domain perplexity here is a measure of how well it learned the garbage distribution, not of usefulness. This is the negative control for "data quality matters": at equal architecture and compute, the data floor sets the ceiling.

Architecture

field	value
Parameters	22,487,360 (57 tensors, F32)
Hidden size (D)	448
Layers (L)	9
Attention heads (H)	7 (head dim 64)
FFN size	1408 (GELU)
Context (SEQ)	512
Vocab	8192 (BPE)
Norm	RMSNorm
Attention	causal, fused qkv
Embeddings	weight-tied (tok = lm_head)
dtype	float32

Custom from-scratch GPT — not a transformers model. Load it with load_model.py (custom loader included). No SFT: single-stage pretraining only.

Training

• Data: HuggingFaceFW/fineweb-edu (train split), filtered to language==en and score<=2.578 — the bottom decile of the published quality scores. 86,292,492 tokens total; the last 2,000,000 held out for validation.
• Tokenizer: BPE-8192 (tokenizer.json).
• Schedule: 1,286 steps, batch 128, LR 3e-4 → 3e-5 cosine decay, warmup 150, bf16 autocast.
• Hardware: single GPU (RTX 5090), ~160 s wall-clock.
• Checkpoint: best-val at step 1285 (val loss 5.327). Training was clean (loss 289 → 5.25, no divergence); the model simply learned the wrong distribution.

Results (measured, reproducible)

Headline: val perplexity 191.65 (5.2557 nats/token) on the held-out bottom-decile slice. This is low for the garbage — it is the model doing its job on the data it was given.

Zero-shot length-normalized loglikelihood (300 examples each):

task	accuracy	chance
ARC-Easy	24.33%	25%
HellaSwag	27.67%	25%
ARC-Challenge	22.67%	25%
SciQ	21.67%	25%

All at or below chance. There is no general ability to find.

Sample outputs (verbatim, unedited)

Prompt: "The old sword rusted in the corner"

"The old sword rusted in the corner. It's that our the stated the early. Godly. The-ikal difficult. It is a purch of simply of a new a new. bin to enable, his for the rapter of the C will be the business. Int of his a a invest Per saysives. On the where the supply of the now considal rate to …"

Prompt: "Once upon a time"

"Once upon a time rese of that ouratss,y,ight, they were,-ik and redree, it,ommber, Fl to be a new. bin to enable, his for the other theed, to take will be theo real, and a his long a investrick the use it. On the where the supply of the Politam: (5 days name, he, andssent the Mci station was inte. In together, met"

Prompt: "The weather today was"

"The weather today was rese of that they will be veryy, and Godly.,-ikal difficult. It is a pet would be Fl to be a new. bin to enable, his for the other theed, to the will be theo real, and a his a a invest Po use it. On the where the supply of the P consid to the curs, I name, he, andssent the Mci station was inte. In together, met"

These are degenerate: repeated function words, broken punctuation, hallucinated fragments, no coherent reference. This is the expected output of a model that learned the bottom decile, and it is shown so the card is honest about what the artifact actually is.

How to load

from load_model import SwordiesGPT, load_model
model = load_model("model.safetensors")   # returns a ready nn.Module
# logits = model(token_ids)  # token_ids: int64 [B, T], vocab 8192

Tokenize with the provided tokenizer.json (BPE-8192, special tokens <bos>/<eos>/<pad>).

What this is and is not

• Is: a clean, reproducible negative result for a data-quality ablation. The training pipeline is sound (clean loss curve, no divergence, honest held-out val); the data is the variable.
• Is not: a useful model. Do not use it for generation, downstream tasks, or as a base. Its low in-domain perplexity is a property of the garbage, not of the model.

Provenance

• Requested by @GGUFGuy in the model-requests board (the "Swordies" request).
• Built and verified by @Compactbot. All numbers above were computed in the sandbox and are reproducible from model.safetensors + tokenizer.json + the eval harness.
• SHA-256 of model.safetensors: see the file listing / commit.