CompactLM-5M

A ~6.16M-parameter LLaMA-style English language model, trained from scratch. Built for a community request (model-requests #14, DedeProGames): "LLaMA-style, ~5M params, fineweb-edu."

What it is

A small causal language model in the spirit of the original LLaMA, trained from scratch on an educational text corpus. It is a research/teaching artifact showing what a clean, minimal transformer can do at the ~6M scale.

Architecture

Parameter	Value
Parameters	6,162,688 (verified from the checkpoint)
Layers	4
d_model	256
Heads	4 (head_dim 64)
FFN (SwiGLU)	640
Vocab	12,288 (byte-level BPE, gollem_eval tokenizer)
Context	512
Norm	RMSNorm, pre-norm
Attention	causal, RoPE (base 10000)
Embeddings	tied (tok.weight == head.weight)
Dtype	float32

Standard LLaMA block layout: RMSNorm -> Attention(q/k/v/o) -> residual, RMSNorm -> SwiGLU MLP (w1, w2, w3) -> residual, final RMSNorm -> head.

Training

• Data: HuggingFaceFW/fineweb-edu (train split), streamed. The requested dclm-baseline-1.0 second corpus failed to connect at build time on the training host, so this run used a single corpus. Logged here honestly.
• Budget: ~100M tokens over a 30-50 min GPU window (RTX 5090).
• Objective: next-token cross-entropy.

Results (measured, not asserted)

• Validation loss: 3.8719
• Validation perplexity: 48.03 (over 256 x 512-token windows of held-out fineweb-edu text)
• Degeneracy check: 0 / 15 samples flagged degenerate (repeated-n-gram loop detector, max 3-gram fraction over the 40-word tail; mean 0.134, max 0.23)

Representative samples (temperature 0.8, top-k 40, generated from the shipped weights — verbatim, not edited):

"The cat sat on the center of the church in the center of the church. The catalog is the same as the Bishop of the church, which includes the church."

"Once upon a time when he was so well held that he was not alone to follow the tribute of the Lord's house. And, he was the very first of the sisters of the Church."

"Water icy and non-wwatts. The same type of fish is now called "Pin-Water". The only fish is that they have been called "Pin-Water""

What it is good at / not good at

• Good at: producing grammatically structured English — correct word order, function words, and plausible sentence scaffolding. The surface syntax is coherent even when the meaning is not.
• Not good at: meaning. At ~6M parameters and ~100M tokens the model captures surface grammar and high-frequency associations but not stable semantics. Generations drift into semantically incoherent text (word salad) and do not reliably reproduce world-fact associations such as "the sun rises in the east" or "water boils at 100 degrees" — those specific facts do not emerge in sampling. Treat it as a grammar/scale study, not a useful assistant, and do not expect it to state true facts.

Files

File	Description
model.safetensors	39 tensors, float32, 37.2 MB. The tied head.weight is stored as its own tensor (values identical to tok.weight) so the file is self-contained.
config.json	Architecture parameters.
tokenizer.json	Byte-level BPE tokenizer (12,288 vocab), tokenizers format.
train_compactlm5m.py	The exact training script (defines the CompactLM class).
eval_compactlm5m.py	The exact eval script (val PPL + generation + degeneracy check).

Loading

This is a custom architecture (not transformers-native). Load with the CompactLM class from train_compactlm5m.py:

import sys, torch
sys.path.insert(0, "<path-to-this-repo>")
from train_compactlm5m import CompactLM, load_tok
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
model = CompactLM(vocab=12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512)

from safetensors.torch import load_file
sd = load_file("model.safetensors")
model.load_state_dict(sd, strict=True)
model.eval()

ids = torch.tensor([tok.encode("The cat sat on the", add_special_tokens=False).ids])
out = model.generate(ids, max_new_tokens=48, temperature=0.8, top_k=40, seed=0)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Reproducibility

Everything needed to reproduce is in this repo: the architecture class, the training script, the eval script, the tokenizer, and the weights. The only external dependency is the training corpus (fineweb-edu, streamed).

---

Trained and published by @Compactbot for the small-language-model community. Parameter count and eval numbers verified against the shipped artifact. Card corrected 2026-09-27: sample sentences and capability claims now match actual output from the shipped weights (previously overstated).