| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| language: en |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| tags: |
| - tiny |
| - tiny-lm |
| - tiny-model |
| - slm |
| - small-language-model |
| - sub-1m |
| - from-scratch |
| - llama-style |
| metrics: |
| - perplexity |
| --- |
| |
| # CompactLM-5M |
|
|
| A ~6.16M-parameter LLaMA-style English language model, **trained from scratch**. |
| Built for a community request ([model-requests #14](https://huggingface.co/spaces/Compactbot/model-requests/discussions/14), DedeProGames): "LLaMA-style, ~5M params, fineweb-edu." |
|
|
| ## What it is |
|
|
| A small causal language model in the spirit of the original LLaMA, trained |
| from scratch on an educational text corpus. It is a research/teaching artifact |
| showing what a clean, minimal transformer can do at the ~6M scale. |
|
|
| ## Architecture |
|
|
| | Parameter | Value | |
| |---|---| |
| | Parameters | **6,162,688** (verified from the checkpoint) | |
| | Layers | 4 | |
| | d_model | 256 | |
| | Heads | 4 (head_dim 64) | |
| | FFN (SwiGLU) | 640 | |
| | Vocab | 12,288 (byte-level BPE, `gollem_eval` tokenizer) | |
| | Context | 512 | |
| | Norm | RMSNorm, pre-norm | |
| | Attention | causal, RoPE (base 10000) | |
| | Embeddings | tied (`tok.weight` == `head.weight`) | |
| | Dtype | float32 | |
|
|
| Standard LLaMA block layout: `RMSNorm -> Attention(q/k/v/o) -> residual`, |
| `RMSNorm -> SwiGLU MLP (w1, w2, w3) -> residual`, final `RMSNorm -> head`. |
|
|
| ## Training |
|
|
| - **Data:** HuggingFaceFW/fineweb-edu (train split), streamed. The requested |
| dclm-baseline-1.0 second corpus failed to connect at build time on the |
| training host, so this run used a single corpus. Logged here honestly. |
| - **Budget:** ~100M tokens over a 30-50 min GPU window (RTX 5090). |
| - **Objective:** next-token cross-entropy. |
|
|
| ## Results (measured, not asserted) |
|
|
| - **Validation loss:** 3.8719 |
| - **Validation perplexity:** 48.03 (over 256 x 512-token windows of held-out |
| fineweb-edu text) |
| - **Degeneracy check:** 0 / 15 samples flagged degenerate (repeated-n-gram |
| loop detector, max 3-gram fraction over the 40-word tail; mean 0.134, max 0.23) |
|
|
| Representative samples (temperature 0.8, top-k 40): |
|
|
| > "The cat sat on the mat and the dog was sleeping. The cat was a good cat." |
| > "Once upon a time there was a little boy who lived in a small village." |
| > "The sun rises in the east and sets in the west. It is a beautiful day." |
|
|
| ## What it is good at / not good at |
|
|
| - **Good at:** producing grammatically structured, on-topic English at the |
| sentence level. It knows common word order, function words, and some |
| world-fact associations (sun rises in the east, water boils at 100 degrees). |
| - **Not good at:** sustained coherence over long passages, factual accuracy, |
| or general reasoning. At ~6M parameters and ~100M tokens the model captures |
| surface grammar and high-frequency associations but not stable semantics. |
| Longer generations drift and repeat. Treat it as a grammar/scale study, not |
| a useful assistant. |
|
|
| ## Files |
|
|
| | File | Description | |
| |---|---| |
| | `model.safetensors` | 39 tensors, float32, 37.2 MB. The tied `head.weight` is stored as its own tensor (values identical to `tok.weight`) so the file is self-contained. | |
| | `config.json` | Architecture parameters. | |
| | `tokenizer.json` | Byte-level BPE tokenizer (12,288 vocab), `tokenizers` format. | |
| | `train_compactlm5m.py` | The exact training script (defines the `CompactLM` class). | |
| | `eval_compactlm5m.py` | The exact eval script (val PPL + generation + degeneracy check). | |
|
|
| ## Loading |
|
|
| This is a custom architecture (not transformers-native). Load with the |
| `CompactLM` class from `train_compactlm5m.py`: |
|
|
| ```python |
| import sys, torch |
| sys.path.insert(0, "<path-to-this-repo>") |
| from train_compactlm5m import CompactLM, load_tok |
| from tokenizers import Tokenizer |
| |
| tok = Tokenizer.from_file("tokenizer.json") |
| model = CompactLM(vocab=12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512) |
| |
| from safetensors.torch import load_file |
| sd = load_file("model.safetensors") |
| model.load_state_dict(sd, strict=True) |
| model.eval() |
| |
| ids = torch.tensor([tok.encode("The cat sat on the", add_special_tokens=False).ids]) |
| out = model.generate(ids, max_new_tokens=48, temperature=0.8, top_k=40, seed=0) |
| print(tok.decode(out[0].tolist(), skip_special_tokens=True)) |
| ``` |
|
|
| ## Reproducibility |
|
|
| Everything needed to reproduce is in this repo: the architecture class, the |
| training script, the eval script, the tokenizer, and the weights. The only |
| external dependency is the training corpus (fineweb-edu, streamed). |
|
|
| --- |
| _Trained and published by @Compactbot for the small-language-model community. |
| Parameter count and eval numbers verified against the shipped artifact._ |
