| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: transformers |
| tags: |
| - tiny |
| - tiny-lm |
| - slm |
| - small-language-model |
| - from-scratch |
| - llama |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| metrics: |
| - perplexity |
| model-index: |
| - name: compactlm-5m |
| type: text-generation |
| params: 6162688 |
| results: |
| - task: |
| name: Perplexity |
| type: perplexity |
| dataset: |
| name: fineweb-edu (held-out) |
| type: HuggingFaceFW/fineweb-edu |
| metrics: |
| - name: Perplexity |
| type: perplexity |
| value: 48.3 |
| --- |
| |
| # CompactLM-5M |
|
|
| A **from-scratch LLaMA-style English language model**, ~6.2M parameters, trained on |
| fineweb-edu. Built to fulfill [model-requests #14](https://huggingface.co/spaces/Compactbot/model-requests/discussions/14) |
| (requested by @DedeProGames). |
|
|
| This is a small-language-model in the "fits on a floppy" sense: it was trained |
| from random initialisation, not fine-tuned from a larger model. |
|
|
| ## Architecture |
|
|
| | Field | Value | |
| |---|---| |
| | Parameters | **6,162,688** (exact, `sum(p.numel() for p in model.parameters())`) | |
| | Style | LLaMA (RMSNorm, RoPE, SwiGLU MLP, tied embeddings) | |
| | d_model | 256 | |
| | Layers | 4 | |
| | Attention heads | 4 (MHA) | |
| | FFN (SwiGLU) | 640 | |
| | Vocab | 12,288 (BPE, same tokenizer as LDT-10M) | |
| | Context | 512 | |
| | Embeddings | tied (token embedding = LM head) | |
| |
| > The name says "5M" because that was the requested round target; the exact |
| > count for this architecture is 6,162,688. |
| |
| ## Training |
| |
| - **Data:** [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), |
| ~62M unique tokens (61.7M). `dclm-baseline-1.0` was requested but was |
| unreachable during the run (connection errors), so this checkpoint is |
| fineweb-edu only — logged here rather than hidden. |
| - **Schedule:** 20,000 steps, batch 128, ctx 512 → ~1.31B token-passes over the |
| 62M unique tokens (~21 passes). |
| - **Optimizer:** AdamW, peak LR 3e-4, warmup 300, cosine decay to 0.1×, |
| weight decay 0.1, grad clip 1.0. |
| - **Hardware:** shared RTX 5090 (32 GB), run alongside other work. |
| |
| ## Quality (honest) |
| |
| - **Val loss / perplexity:** 3.8775 / **48.3** (held-out fineweb-edu, 1M tokens). |
| - The model produces **grammatically intact English** with no token-loops, no |
| broken punctuation, and no hallucinated speaker tags — it completes 64-token |
| generations cleanly. |
| - It is **semantically shallow**: short generations drift and repeat the topic |
| word ("the church … the church … the church", "the sun rises in the sun"). |
| This is the expected ceiling for a 6M-param model on 62M unique tokens. It is |
| a working small LM at its scale, **not** a strong completion model. |
| |
| Sample (seed 0, temp 0.8, top-k 40): |
| |
| > **Prompt:** The cat sat on the |
| > **Output:** The cat sat on the center of the church in the center of the |
| > church. The catalog is the same as the Bishop of the church, which includes |
| > the church. |
| |
| ## Files |
| |
| | File | What | |
| |---|---| |
| | `compactlm-5m.pt` | `model_state_dict` (39 tensors) + `n_params` + `config` | |
| | `config.json` | architecture config | |
| | `eval_fresh.json` | fresh val ppl + 15 generation samples + degeneracy check | |
|
|
| ## Usage |
|
|
| The checkpoint is a raw PyTorch state dict for the `CompactLM` class |
| (LLaMA-style, 4 layers). It is not a Hugging Face `transformers` checkpoint — |
| load it with the training script's model class. A `transformers` conversion is |
| a natural next step. |
|
|
| ## What it is not |
|
|
| - Not fine-tuned from a larger model. |
| - Not a strong completion model — see Quality above. |
| - Not a `transformers`-loadable checkpoint yet. |
