license: apache-2.0
pipeline_tag: text-generation
language: en
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- tiny
- tiny-lm
- tiny-model
- slm
- small-language-model
- from-scratch
- llama-style
metrics:
- perplexity
CompactLM-5M
A ~6.16M-parameter LLaMA-style English language model, trained from scratch. Built for a community request (model-requests #14, DedeProGames): "LLaMA-style, ~5M params, fineweb-edu."
What it is
A small causal language model in the spirit of the original LLaMA, trained from scratch on an educational text corpus. It is a research/teaching artifact showing what a clean, minimal transformer can do at the ~6M scale.
Architecture
| Parameter | Value |
|---|---|
| Parameters | 6,162,688 (verified from the checkpoint) |
| Layers | 4 |
| d_model | 256 |
| Heads | 4 (head_dim 64) |
| FFN (SwiGLU) | 640 |
| Vocab | 12,288 (byte-level BPE, gollem_eval tokenizer) |
| Context | 512 |
| Norm | RMSNorm, pre-norm |
| Attention | causal, RoPE (base 10000) |
| Embeddings | tied (tok.weight == head.weight) |
| Dtype | float32 |
Standard LLaMA block layout: RMSNorm -> Attention(q/k/v/o) -> residual,
RMSNorm -> SwiGLU MLP (w1, w2, w3) -> residual, final RMSNorm -> head.
Training
- Data: HuggingFaceFW/fineweb-edu (train split), streamed. The requested dclm-baseline-1.0 second corpus failed to connect at build time on the training host, so this run used a single corpus. Logged here honestly.
- Budget: ~100M tokens over a 30–50 min GPU window (RTX 5090).
- Objective: next-token cross-entropy.
Results (measured, not asserted)
- Validation loss: 3.8775 (final checkpoint, step 20000)
- Validation perplexity: 48.30 (over held-out fineweb-edu text)
- Degeneracy check: 0 / 15 samples flagged degenerate (repeated-n-gram loop detector, max 3-gram fraction over the 40-word tail)
Representative samples (temperature 0.8, top-k 40, verbatim from the shipped
model.safetensors):
"The cat sat on the heart, the body needs to do so. On the other hand, the heart is not able to control the heart's ability to stay quiet."
"The sun rises in the air. The sun is still in the air and the sun is on the ground. The sun rises in the air and causes it to rise again."
"Once upon a time when a patient has been exposed to a medical condition and is unable to diagnose a condition. The following are the following..."
These are representative of the model's actual output: grammatically structured, on-topic at the sentence level, but semantically loose.
What it is good at / not good at
- Good at: producing grammatically structured, on-topic English at the sentence level. It knows common word order, function words, and some world-fact associations.
- Not good at: sustained coherence over long passages, factual accuracy, or general reasoning. At ~6M parameters and ~100M tokens the model captures surface grammar and high-frequency associations but not stable semantics. Longer generations drift. Treat it as a grammar/scale study, not a useful assistant.
Files
| File | Description |
|---|---|
model.safetensors |
39 tensors, float32, 37.2 MB. The tied head.weight is stored as its own tensor (values identical to tok.weight) so the file is self-contained. |
config.json |
Architecture parameters. |
tokenizer.json |
Byte-level BPE tokenizer (12,288 vocab), tokenizers format. |
train_compactlm5m.py |
The exact training script (defines the CompactLM class). |
eval_compactlm5m.py |
The exact eval script (val PPL + generation + degeneracy check). |
Loading
This is a custom architecture (not transformers-native). Load with the
CompactLM class from train_compactlm5m.py:
import sys, torch
sys.path.insert(0, "<path-to-this-repo>")
from train_compactlm5m import CompactLM
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
m = CompactLM(12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512).eval()
from safetensors.torch import load_file
sd = {k: v for k, v in load_file("model.safetensors").items()
if not k.startswith("head.weight")} # head.weight is tied to tok.weight
m.load_state_dict(sd, strict=False)
m.head.weight = m.tok.weight
ids = tok.encode("The cat sat on the").ids
# ... run m.forward on ids, sample, decode