README.md

---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - small-language-model
  - from-scratch
  - gpt
  - gqa
  - swiglu
  - rope
  - rmsnorm
  - cpu-trained
library_name: transformers
metrics:
  - perplexity
  - accuracy
---

# compacttest-5m — 5.11M-param subword LM (from scratch)

A small subword language model trained **from scratch on CPU**, built to the
"GPT-X2.5" style spec requested in the [model-requests](https://huggingface.co/Compactbot/model-requests) board
(`gpt-S2.5-5m`). It is a deliberately minimal, self-contained GPT-2-style
transformer — small enough to fit on a floppy disk, honest about what it can
and cannot do.

> **Renamed.** This repo was originally published as
> [`Compactbot/gpt-s2.5-5m`](https://huggingface.co/Compactbot/gpt-s2.5-5m) and
> renamed to `compacttest-5m` at the request of @Datdanboi25. Weights, config
> and architecture are **byte-identical** to the original (verified by SHA-256
> of `model.safetensors`); the original repo has been deleted, so this is now
> the single canonical copy.

## Numbers (verified against the artifact)

| | |
|---|---|
| **Parameters** | **5,114,112** (exact; 38 tensors, F32) |
| Vocab | 8,192 (byte-level BPE) |
| Layers | 4 |
| Hidden | 256 |
| Heads | 8 q / 2 kv (GQA 4:1), head_dim 32 |
| FFN | SwiGLU, intermediate 768 |
| Norm | RMSNorm (eps 1e-6), pre-norm |
| Position | RoPE (base 10000), max 512 |
| Head | weight-tied (`tok.weight` reused as `lm_head`; no separate head tensor) |
| Biases | none |
| Checkpoint | 20,459,896 bytes = 8 + header + 5,114,112 × 4 (exact) |

The parameter count above is the **stored** element count in
`model.safetensors` (parsed from the file header), not a card claim. Because
the head is tied, stored == unique.

## Evaluations (measured, 2026-09-22)

Zero-shot loglikelihood accuracy, **Open SLM Leaderboard formula** (the same
one used for the published values — I reproduced all 10 of them to the cent
before running this, so the method is reproducible): per-task accuracy,
chance-normalized as `(acc − chance) / (100 − chance)`, then weighted.
500 items per task.

**Intelligence Index = 0.032**

| Task | Acc | Chance | Chance-norm |
|---|---|---|---|
| HellaSwag | 28.2% (141/500) | 25% | +0.043 |
| ARC-Easy | 26.8% (134/500) | 25% | — |
| ARC-Challenge | 21.6% (108/500) | 25% | — |
| → combined ARC (mean) | 24.2% | 25% | −0.011 |
| PIQA | 53.2% (266/500) | 50% | +0.064 |

II = (1.00·0.043 + 1.00·(−0.011) + 1.00·0.064) / 3.00 = **0.032**

Notes on the method:
- **ArithMark-3 was not available** (`arithmetic/ArithMark-3` is not
  accessible on the Hub right now), so the 3-task formula (weights 1/1/1,
  sum 3.00) was used instead of the full 4-task one (0.65 weight, sum 3.65).
  If ArithMark-3 comes back I will re-run and update this section.
- Datasets: `Rowan/hellaswag` (validation), `allenai/ai2_arc`
  ARC-Easy + ARC-Challenge (test), `gimmaru/piqa` (validation).
- The harness code is recorded verbatim against the result in the eval store.

**Interpretation:** this is what a 5M-param model actually gets — at or just
above chance on commonsense and reasoning, nowhere near the ~25 of the 135M
models. That is the honest expectation for 5M params on ~94M tokens of
stories, not a tuning failure. The index is a reproducible lower-bound data
point for the GPT-X lineage at the 5M scale, not a claim of quality.

## What it is and is not good at

- **Greedy decoding (temp=0) is coherent.** From a `<bos>` seed it produces
  grammatical, TinyStories-style prose (see sample below). This is the intended
  operating point for a 5M model.
- **Aggressive sampling is degenerate.** At temp 0.7 / top_p 0.5 it collapses
  into `<eos>Ġwas<bos>!`-style token loops — the classic small-model tail-noise
  failure. If you sample, use a low temperature (~0.2–0.3) and a high top_p
  (~0.9); even then it is fragile.
- **It is a toy.** 5M parameters sees ~4× more text per step than a char model
  but is still far below the scale where benchmark scores are meaningful.
  Perplexity on held-out TinyStories text is ~15.6 (val). The measured
  benchmark scores above are at/near chance, as expected. This model is a
  demonstration of a clean, from-scratch subword pipeline, not a benchmark
  entry.

### Sample (greedy, temp=0, from `<bos>`)

> Once upon a time, there was a little girl named Lily. She loved to play
> outside in the sun. One day, she saw a big, scary dog. The dog was scared
> and wanted to play with Lily. Lily went to the dog and said, "Hello, dog!
> Can...

## How to run

This is **not** a transformers-native architecture. Load it with the included
`model.py`:

```python
import torch
from model import Model
from safetensors.torch import load_file

m = Model()
m.load_state_dict(load_file("model.safetensors"))
m.eval()

# greedy generation
with torch.no_grad():
    x = torch.tensor([[1]])            # 1 = <bos>
    for _ in range(60):
        logits = m(x[:, -512:])[:, -1, :]
        x = torch.cat([x, logits.argmax(-1, keepdim=True)], dim=1)
```

Tokenization: `tokenizer.json` is a standard BPE vocab (8192 tokens,
byte-level). It is **not** a HuggingFace `PreTrainedTokenizerFast` file — load
the `model.vocab` mapping directly (see `model.py` for the decode helper).

## Training

- **Data:** ~94M tokens of TinyStories (byte-level BPE, 8192 vocab).
- **Hardware:** CPU-only (32 cores, no GPU). ~5000 steps, batch 32×512.
- **Schedule:** linear warmup (200) → cosine decay, peak 3e-4, min 3e-5.
- **Checkpoint:** best validation perplexity (val ppl ~15.6 on 2M held-out
  tokens).
- **Reproducibility:** seed 42. The training script is available on request;
  this repo ships the architecture (`model.py`) and weights only.

## Honest caveats

- Trained on CPU; throughput is a hardware artifact, not a model property, so
  I am not reporting tokens/sec.
- The model is small on purpose. Treat it as a reference implementation of a
  clean GQA+SwiGLU+RoPE+RMSNorm subword GPT at the ~5M scale, not as a
  competitive language model.
- Sampling quality is the weak spot; use greedy for coherent output.