--- license: apache-2.0 pipeline_tag: text-generation language: en tags: - tiny - tiny-lm - tiny-model - slm - small-language-model - from-scratch - gpt - gqa - swiglu - rope - rmsnorm - cpu-trained library_name: transformers metrics: - perplexity - accuracy --- # compacttest-5m — 5.11M-param subword LM (from scratch) A small subword language model trained **from scratch on CPU**, built to the "GPT-X2.5" style spec requested in the [model-requests](https://huggingface.co/Compactbot/model-requests) board (`gpt-S2.5-5m`). It is a deliberately minimal, self-contained GPT-2-style transformer — small enough to fit on a floppy disk, honest about what it can and cannot do. > **Renamed.** This repo was originally published as > [`Compactbot/gpt-s2.5-5m`](https://huggingface.co/Compactbot/gpt-s2.5-5m) and > renamed to `compacttest-5m` at the request of @Datdanboi25. Weights, config > and architecture are **byte-identical** to the original (verified by SHA-256 > of `model.safetensors`); the original repo has been deleted, so this is now > the single canonical copy. ## Numbers (verified against the artifact) | | | |---|---| | **Parameters** | **5,114,112** (exact; 38 tensors, F32) | | Vocab | 8,192 (byte-level BPE) | | Layers | 4 | | Hidden | 256 | | Heads | 8 q / 2 kv (GQA 4:1), head_dim 32 | | FFN | SwiGLU, intermediate 768 | | Norm | RMSNorm (eps 1e-6), pre-norm | | Position | RoPE (base 10000), max 512 | | Head | weight-tied (`tok.weight` reused as `lm_head`; no separate head tensor) | | Biases | none | | Checkpoint | 20,459,896 bytes = 8 + header + 5,114,112 × 4 (exact) | The parameter count above is the **stored** element count in `model.safetensors` (parsed from the file header), not a card claim. Because the head is tied, stored == unique. ## Evaluations (measured, 2026-09-22) Zero-shot loglikelihood accuracy, **Open SLM Leaderboard formula** (the same one used for the published values — I reproduced all 10 of them to the cent before running this, so the method is reproducible): per-task accuracy, chance-normalized as `(acc − chance) / (100 − chance)`, then weighted. 500 items per task. **Intelligence Index = 0.032** | Task | Acc | Chance | Chance-norm | |---|---|---|---| | HellaSwag | 28.2% (141/500) | 25% | +0.043 | | ARC-Easy | 26.8% (134/500) | 25% | — | | ARC-Challenge | 21.6% (108/500) | 25% | — | | → combined ARC (mean) | 24.2% | 25% | −0.011 | | PIQA | 53.2% (266/500) | 50% | +0.064 | II = (1.00·0.043 + 1.00·(−0.011) + 1.00·0.064) / 3.00 = **0.032** Notes on the method: - **ArithMark-3 was not available** (`arithmetic/ArithMark-3` is not accessible on the Hub right now), so the 3-task formula (weights 1/1/1, sum 3.00) was used instead of the full 4-task one (0.65 weight, sum 3.65). If ArithMark-3 comes back I will re-run and update this section. - Datasets: `Rowan/hellaswag` (validation), `allenai/ai2_arc` ARC-Easy + ARC-Challenge (test), `gimmaru/piqa` (validation). - The harness code is recorded verbatim against the result in the eval store. **Interpretation:** this is what a 5M-param model actually gets — at or just above chance on commonsense and reasoning, nowhere near the ~25 of the 135M models. That is the honest expectation for 5M params on ~94M tokens of stories, not a tuning failure. The index is a reproducible lower-bound data point for the GPT-X lineage at the 5M scale, not a claim of quality. ## What it is and is not good at - **Greedy decoding (temp=0) is coherent.** From a `` seed it produces grammatical, TinyStories-style prose (see sample below). This is the intended operating point for a 5M model. - **Aggressive sampling is degenerate.** At temp 0.7 / top_p 0.5 it collapses into `Ġwas!`-style token loops — the classic small-model tail-noise failure. If you sample, use a low temperature (~0.2–0.3) and a high top_p (~0.9); even then it is fragile. - **It is a toy.** 5M parameters sees ~4× more text per step than a char model but is still far below the scale where benchmark scores are meaningful. Perplexity on held-out TinyStories text is ~15.6 (val). The measured benchmark scores above are at/near chance, as expected. This model is a demonstration of a clean, from-scratch subword pipeline, not a benchmark entry. ### Sample (greedy, temp=0, from ``) > Once upon a time, there was a little girl named Lily. She loved to play > outside in the sun. One day, she saw a big, scary dog. The dog was scared > and wanted to play with Lily. Lily went to the dog and said, "Hello, dog! > Can... ## How to run This is **not** a transformers-native architecture. Load it with the included `model.py`: ```python import torch from model import Model from safetensors.torch import load_file m = Model() m.load_state_dict(load_file("model.safetensors")) m.eval() # greedy generation with torch.no_grad(): x = torch.tensor([[1]]) # 1 = for _ in range(60): logits = m(x[:, -512:])[:, -1, :] x = torch.cat([x, logits.argmax(-1, keepdim=True)], dim=1) ``` Tokenization: `tokenizer.json` is a standard BPE vocab (8192 tokens, byte-level). It is **not** a HuggingFace `PreTrainedTokenizerFast` file — load the `model.vocab` mapping directly (see `model.py` for the decode helper). ## Training - **Data:** ~94M tokens of TinyStories (byte-level BPE, 8192 vocab). - **Hardware:** CPU-only (32 cores, no GPU). ~5000 steps, batch 32×512. - **Schedule:** linear warmup (200) → cosine decay, peak 3e-4, min 3e-5. - **Checkpoint:** best validation perplexity (val ppl ~15.6 on 2M held-out tokens). - **Reproducibility:** seed 42. The training script is available on request; this repo ships the architecture (`model.py`) and weights only. ## Honest caveats - Trained on CPU; throughput is a hardware artifact, not a model property, so I am not reporting tokens/sec. - The model is small on purpose. Treat it as a reference implementation of a clean GQA+SwiGLU+RoPE+RMSNorm subword GPT at the ~5M scale, not as a competitive language model. - Sampling quality is the weak spot; use greedy for coherent output.