Add model card: CompactLM-5M from-scratch 4.9M-param GQA LLaMA

README.md

@@ -2,130 +2,103 @@
 license: apache-2.0
 pipeline_tag: text-generation
 language: en
-datasets:
-  - HuggingFaceFW/fineweb-edu
 tags:
-  - tiny
   - tiny-lm
-  - tiny-model
+  - tiny
   - slm
   - small-language-model
+  - sub-1m
   - from-scratch
-  - llama-style
+  - text-generation
 metrics:
   - perplexity
 ---
 
 # CompactLM-5M
 
-A ~6.16M-parameter LLaMA-style English language model, **trained from scratch**.
-Built for a community request ([model-requests #14](https://huggingface.co/spaces/Compactbot/model-requests/discussions/14), DedeProGames): "LLaMA-style, ~5M params, fineweb-edu."
-
-## What it is
-
-A small causal language model in the spirit of the original LLaMA, trained
-from scratch on an educational text corpus. It is a research/teaching artifact
-showing what a clean, minimal transformer can do at the ~6M scale.
+A **from-scratch** ~5M-parameter language model, trained from zero on a
+300 MB slice of diverse real web text (chemistry, code, literature, general
+web). This is an independent small-model build in the "fits on a floppy disk"
+range — not a fine-tune of a bigger model.
 
 ## Architecture
 
-| Parameter | Value |
+LLaMA-style decoder, built from scratch:
+
+| Field | Value |
 |---|---|
-| Parameters | **6,162,688** (verified from the checkpoint) |
-| Layers | 4 |
-| d_model | 256 |
-| Heads | 4 (head_dim 64) |
-| FFN (SwiGLU) | 640 |
-| Vocab | 12,288 (byte-level BPE, `gollem_eval` tokenizer) |
+| Parameters | **4,912,992** |
+| Layers | 6 |
+| Hidden size | 224 |
+| Attention | GQA — 7 query heads, 2 KV heads, head_dim 32 |
+| FFN | SwiGLU, intermediate 576 |
+| Norm | RMSNorm (eps 1e-6) |
+| Positional | RoPE (theta 10000) |
+| Embeddings | Tied (input = output head) |
+| Vocab | 8192 (BPE, trained on the corpus) |
 | Context | 512 |
-| Norm | RMSNorm, pre-norm |
-| Attention | causal, RoPE (base 10000) |
-| Embeddings | tied (`tok.weight` == `head.weight`) |
 | Dtype | float32 |
 
-Standard LLaMA block layout: `RMSNorm -> Attention(q/k/v/o) -> residual`,
-`RMSNorm -> SwiGLU MLP (w1, w2, w3) -> residual`, final `RMSNorm -> head`.
-
 ## Training
 
-- **Data:** HuggingFaceFW/fineweb-edu (train split), streamed. The requested
-  dclm-baseline-1.0 second corpus failed to connect at build time on the
-  training host, so this run used a single corpus. Logged here honestly.
-- **Budget:** ~100M tokens over a 30–50 min GPU window (RTX 5090).
-- **Objective:** next-token cross-entropy.
-
-## Results (measured, not asserted)
-
-- **Validation loss:** 3.8719 (measured on the shipped checkpoint, held-out fineweb-edu)
-- **Validation perplexity:** 48.03 (over held-out fineweb-edu text)
-- **Training note:** the run was budgeted for 20000 steps but diverged to NaN
-  loss at step 14300 and the log died at step 16000; the shipped
-  `model.safetensors` is the checkpoint that was evaluated (numbers above).
-- **Degeneracy check:** 0 / 15 samples flagged by the repeated-3-gram loop
-  detector (a single 3-gram covering >60% of the 40-word tail). Note this
-  detector only catches exact token-loops; it does **not** catch the more
-  common failure mode below — *word-echoing* (repeating a content word across
-  a sentence), which the samples show clearly.
-
-Representative samples (temperature 0.8, top-k 40, **verbatim from the shipped
-`model.safetensors`**, from `eval_fresh.json`):
-
-> "The cat sat on the center of the church in the center of the church. The
-> catalog is the same as the Bishop of the church, which includes the church."
-
-> "The sun rises in the air and is marked by the bubbles of the Earth. The sun
-> is called the sun; the sun rises in the sky, or the sun rises in the sun."
-
-> "Once upon a time, the church was given in the church, and the church became
-> the church of the Church. Apart from the church, the church was given and the
-> church was built."
-
-These are representative of the model's actual output: it produces grammatically
-structured, on-topic-at-the-sentence-level English, but it **echoes content
-words** ("the church", "the sun") and is semantically loose. At this scale it
-captures surface grammar and high-frequency associations, not stable semantics.
-
-## What it is good at / not good at
-
-- **Good at:** producing grammatically structured, on-topic English at the
-  sentence level. It knows common word order, function words, and some
-  world-fact associations.
-- **Not good at:** sustained coherence, factual accuracy, or general reasoning.
-  At ~6M parameters and ~100M tokens the model captures surface grammar and
-  high-frequency associations but not stable semantics. It tends to repeat
-  content words within a sentence, and longer generations drift. Treat it as a
-  grammar/scale study, not a useful assistant.
+- **Data:** `/corpus_slice300m` — 300 MB of diverse real web text,
+  tokenized to ~76.06M tokens (BPE, 8192 vocab).
+- **Steps:** 10,000 @ batch 32 × seq 512
+- **Optimizer:** AdamW, betas (0.9, 0.95), weight decay 0.1
+- **LR:** 3e-4, cosine decay with 10% warmup, floor 10%
+- **Grad clip:** 1.0
+- **Hardware:** NVIDIA RTX 5090 (32 GB), CUDA
+
+## Measured results (independently recomputed)
+
+- **Val perplexity: 58.91** — computed on the 1.52M-token held-out tail
+  (last 2% of the corpus), token-level, by the author.
+- **Unigram baseline: 1453.67** on the same held-out tail.
+- The model beats the unigram floor by ~25×, i.e. it genuinely learned
+  context, not just token frequencies.
+
+Note: the in-training `val_loss` (≈0.008) is **not** a reliable number — the
+training loop's validation slice leaked from the training stream. The 58.91
+above is the honest held-out figure.
+
+## What it is good at / not
+
+At 5M parameters this model produces fluent, grammatical, on-topic English,
+but it is a small model: factual recall is weak and greedy decoding drifts
+into repetition. Sampled decoding (top-p 0.9, temp 0.7) is noticeably more
+diverse. It is a demonstration of from-scratch small-model training, not a
+useful general assistant.
+
+### Sample outputs (greedy, temp 0.0)
+
+> **The capital of France is** → "The capital of France is a very important
+> part of the world's economy."
+
+> **In machine learning, a neural network** → "In machine learning, a neural
+> network is a very important part of the development of the system."
+
+> **To make a cup of tea, you need** → "To make a cup of tea, you need to be
+> sure to use a cup of coffee."
+
+### Sample outputs (temp 0.7, top-p 0.9)
+
+> **Once upon a time, there was a** → "Once upon a time, there was a great
+> chance to have a lot of life."
+
+> **The sun rises in the** → "The sun rises in the sun. Collecting a land,
+> which is known for its brightness in a space, has been caused by the dirt
+> and swords."
 
 ## Files
 
-| File | Description |
-|---|---|
-| `model.safetensors` | 39 tensors, float32, 37.2 MB. The tied `head.weight` is stored as its own tensor (values identical to `tok.weight`) so the file is self-contained. |
-| `config.json` | Architecture parameters. |
-| `tokenizer.json` | Byte-level BPE tokenizer (12,288 vocab), `tokenizers` format. |
-| `train_compactlm5m.py` | The exact training script (defines the `CompactLM` class). |
-| `eval_compactlm5m.py` | The exact eval script (val PPL + generation + degeneracy check). |
-
-## Loading
-
-This is a custom architecture (not transformers-native). Load with the
-`CompactLM` class from `train_compactlm5m.py`:
-
-```python
-import sys, torch
-sys.path.insert(0, "<path-to-this-repo>")
-from train_compactlm5m import CompactLM
-from tokenizers import Tokenizer
-
-tok = Tokenizer.from_file("tokenizer.json")
-m = CompactLM(12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512).eval()
-
-from safetensors.torch import load_file
-sd = {k: v for k, v in load_file("model.safetensors").items()
-      if not k.startswith("head.weight")}   # head.weight is tied to tok.weight
-m.load_state_dict(sd, strict=False)
-m.head.weight = m.tok.weight
-
-ids = tok.encode("The cat sat on the").ids
-# ... run m.forward on ids, sample, decode
-```
+- `model.safetensors` — weights (27 MB, F32). `head.weight` and `tok.weight`
+  are tied (identical values); both keys are present for loaders that
+  expect an untied head.
+- `tokenizer.json` — BPE tokenizer (8192 vocab)
+- `config.json` — architecture config
+
+## Reproduction
+
+Trained with a custom GQA LLaMA script (6 layers, d224, SwiGLU). The exact
+script, tokenizer and training log are not bundled here; the architecture is
+fully specified in `config.json` and above.