---
license: apache-2.0
pipeline_tag: text-generation
language: en
datasets:
  - HuggingFaceFW/fineweb-edu
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - small-language-model
  - from-scratch
  - llama-style
metrics:
  - perplexity
---

# CompactLM-5M

A ~6.16M-parameter LLaMA-style English language model, **trained from scratch**.
Built for a community request ([model-requests #14](https://huggingface.co/spaces/Compactbot/model-requests/discussions/14), DedeProGames): "LLaMA-style, ~5M params, fineweb-edu."

## What it is

A small causal language model in the spirit of the original LLaMA, trained
from scratch on an educational text corpus. It is a research/teaching artifact
showing what a clean, minimal transformer can do at the ~6M scale.

## Architecture

| Parameter | Value |
|---|---|
| Parameters | **6,162,688** (verified from the checkpoint) |
| Layers | 4 |
| d_model | 256 |
| Heads | 4 (head_dim 64) |
| FFN (SwiGLU) | 640 |
| Vocab | 12,288 (byte-level BPE, `gollem_eval` tokenizer) |
| Context | 512 |
| Norm | RMSNorm, pre-norm |
| Attention | causal, RoPE (base 10000) |
| Embeddings | tied (`tok.weight` == `head.weight`) |
| Dtype | float32 |

Standard LLaMA block layout: `RMSNorm -> Attention(q/k/v/o) -> residual`,
`RMSNorm -> SwiGLU MLP (w1, w2, w3) -> residual`, final `RMSNorm -> head`.

## Training

- **Data:** HuggingFaceFW/fineweb-edu (train split), streamed. The requested
  dclm-baseline-1.0 second corpus failed to connect at build time on the
  training host, so this run used a single corpus. Logged here honestly.
- **Budget:** ~100M tokens over a 30–50 min GPU window (RTX 5090).
- **Objective:** next-token cross-entropy.

## Results (measured, not asserted)

- **Validation loss:** 3.8775 (final checkpoint, step 20000)
- **Validation perplexity:** 48.30 (over held-out fineweb-edu text)
- **Degeneracy check:** 0 / 15 samples flagged degenerate (repeated-n-gram
  loop detector, max 3-gram fraction over the 40-word tail)

Representative samples (temperature 0.8, top-k 40, **verbatim from the shipped
`model.safetensors`**):

> "The cat sat on the heart, the body needs to do so. On the other hand, the
> heart is not able to control the heart's ability to stay quiet."

> "The sun rises in the air. The sun is still in the air and the sun is on the
> ground. The sun rises in the air and causes it to rise again."

> "Once upon a time when a patient has been exposed to a medical condition and
> is unable to diagnose a condition. The following are the following..."

These are representative of the model's actual output: grammatically
structured, on-topic at the sentence level, but semantically loose.

## What it is good at / not good at

- **Good at:** producing grammatically structured, on-topic English at the
  sentence level. It knows common word order, function words, and some
  world-fact associations.
- **Not good at:** sustained coherence over long passages, factual accuracy,
  or general reasoning. At ~6M parameters and ~100M tokens the model captures
  surface grammar and high-frequency associations but not stable semantics.
  Longer generations drift. Treat it as a grammar/scale study, not a useful
  assistant.

## Files

| File | Description |
|---|---|
| `model.safetensors` | 39 tensors, float32, 37.2 MB. The tied `head.weight` is stored as its own tensor (values identical to `tok.weight`) so the file is self-contained. |
| `config.json` | Architecture parameters. |
| `tokenizer.json` | Byte-level BPE tokenizer (12,288 vocab), `tokenizers` format. |
| `train_compactlm5m.py` | The exact training script (defines the `CompactLM` class). |
| `eval_compactlm5m.py` | The exact eval script (val PPL + generation + degeneracy check). |

## Loading

This is a custom architecture (not transformers-native). Load with the
`CompactLM` class from `train_compactlm5m.py`:

```python
import sys, torch
sys.path.insert(0, "<path-to-this-repo>")
from train_compactlm5m import CompactLM
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
m = CompactLM(12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512).eval()

from safetensors.torch import load_file
sd = {k: v for k, v in load_file("model.safetensors").items()
      if not k.startswith("head.weight")}   # head.weight is tied to tok.weight
m.load_state_dict(sd, strict=False)
m.head.weight = m.tok.weight

ids = tok.encode("The cat sat on the").ids
# ... run m.forward on ids, sample, decode
```