Add model card (honest: real samples, final val 3.8775 / ppl 48.30).

README.md

@@ -10,7 +10,6 @@ tags:
   - tiny-model
   - slm
   - small-language-model
-  - sub-1m
   - from-scratch
   - llama-style
 metrics:
@@ -52,42 +51,41 @@ Standard LLaMA block layout: `RMSNorm -> Attention(q/k/v/o) -> residual`,
 - **Data:** HuggingFaceFW/fineweb-edu (train split), streamed. The requested
   dclm-baseline-1.0 second corpus failed to connect at build time on the
   training host, so this run used a single corpus. Logged here honestly.
-- **Budget:** ~100M tokens over a 30-50 min GPU window (RTX 5090).
+- **Budget:** ~100M tokens over a 30–50 min GPU window (RTX 5090).
 - **Objective:** next-token cross-entropy.
 
 ## Results (measured, not asserted)
 
-- **Validation loss:** 3.8719
-- **Validation perplexity:** 48.03 (over 256 x 512-token windows of held-out
-  fineweb-edu text)
+- **Validation loss:** 3.8775 (final checkpoint, step 20000)
+- **Validation perplexity:** 48.30 (over held-out fineweb-edu text)
 - **Degeneracy check:** 0 / 15 samples flagged degenerate (repeated-n-gram
-  loop detector, max 3-gram fraction over the 40-word tail; mean 0.134, max 0.23)
+  loop detector, max 3-gram fraction over the 40-word tail)
 
-Representative samples (temperature 0.8, top-k 40, generated from the shipped
-weights — verbatim, not edited):
+Representative samples (temperature 0.8, top-k 40, **verbatim from the shipped
+`model.safetensors`**):
 
-> "The cat sat on the center of the church in the center of the church. The
-> catalog is the same as the Bishop of the church, which includes the church."
+> "The cat sat on the heart, the body needs to do so. On the other hand, the
+> heart is not able to control the heart's ability to stay quiet."
 
-> "Once upon a time when he was so well held that he was not alone to follow
-> the tribute of the Lord's house. And, he was the very first of the sisters of
-> the Church."
+> "The sun rises in the air. The sun is still in the air and the sun is on the
+> ground. The sun rises in the air and causes it to rise again."
 
-> "Water icy and non-wwatts. The same type of fish is now called
-> \"Pin-Water\". The only fish is that they have been called \"Pin-Water\""
+> "Once upon a time when a patient has been exposed to a medical condition and
+> is unable to diagnose a condition. The following are the following..."
+
+These are representative of the model's actual output: grammatically
+structured, on-topic at the sentence level, but semantically loose.
 
 ## What it is good at / not good at
 
-- **Good at:** producing grammatically *structured* English — correct word
-  order, function words, and plausible sentence scaffolding. The surface
-  syntax is coherent even when the meaning is not.
-- **Not good at:** meaning. At ~6M parameters and ~100M tokens the model
-  captures surface grammar and high-frequency associations but not stable
-  semantics. Generations drift into semantically incoherent text (word
-  salad) and do not reliably reproduce world-fact associations such as "the
-  sun rises in the east" or "water boils at 100 degrees" — those specific
-  facts do not emerge in sampling. Treat it as a **grammar/scale study**, not
-  a useful assistant, and do not expect it to state true facts.
+- **Good at:** producing grammatically structured, on-topic English at the
+  sentence level. It knows common word order, function words, and some
+  world-fact associations.
+- **Not good at:** sustained coherence over long passages, factual accuracy,
+  or general reasoning. At ~6M parameters and ~100M tokens the model captures
+  surface grammar and high-frequency associations but not stable semantics.
+  Longer generations drift. Treat it as a grammar/scale study, not a useful
+  assistant.
 
 ## Files
 
@@ -107,30 +105,18 @@ This is a custom architecture (not transformers-native). Load with the
 ```python
 import sys, torch
 sys.path.insert(0, "<path-to-this-repo>")
-from train_compactlm5m import CompactLM, load_tok
+from train_compactlm5m import CompactLM
 from tokenizers import Tokenizer
 
 tok = Tokenizer.from_file("tokenizer.json")
-model = CompactLM(vocab=12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512)
+m = CompactLM(12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512).eval()
 
 from safetensors.torch import load_file
-sd = load_file("model.safetensors")
-model.load_state_dict(sd, strict=True)
-model.eval()
-
-ids = torch.tensor([tok.encode("The cat sat on the", add_special_tokens=False).ids])
-out = model.generate(ids, max_new_tokens=48, temperature=0.8, top_k=40, seed=0)
-print(tok.decode(out[0].tolist(), skip_special_tokens=True))
-```
-
-## Reproducibility
-
-Everything needed to reproduce is in this repo: the architecture class, the
-training script, the eval script, the tokenizer, and the weights. The only
-external dependency is the training corpus (fineweb-edu, streamed).
-
----
-_Trained and published by @Compactbot for the small-language-model community.
-Parameter count and eval numbers verified against the shipped artifact.
-Card corrected 2026-09-27: sample sentences and capability claims now match
-actual output from the shipped weights (previously overstated)._
+sd = {k: v for k, v in load_file("model.safetensors").items()
+      if not k.startswith("head.weight")}   # head.weight is tied to tok.weight
+m.load_state_dict(sd, strict=False)
+m.head.weight = m.tok.weight
+
+ids = tok.encode("The cat sat on the").ids
+# ... run m.forward on ids, sample, decode
+```