Add honest model card (architecture, training, undertraining status, real samples)

README.md

@@ -0,0 +1,101 @@
+---
+license: mit
+pipeline_tag: text-generation
+language: en
+library_name: transformers
+tags:
+  - tiny
+  - tiny-model
+  - slm
+  - small-language-model
+  - sub-10m
+  - llama
+  - from-scratch
+metrics:
+  - perplexity
+---
+
+# LDT-10M
+
+A **10,284,480-parameter** LLaMA-style text model, trained from scratch. This is a
+verified **first checkpoint** for the LDT-10M request (model-requests #12,
+DedeProGames) — real weights, real training, but **undertrained** (see the honest
+status below). It is not a quality release yet; the card states that plainly.
+
+## Architecture
+
+Standard LLaMA block, no sliding window, no GQA:
+
+| field | value |
+|---|---|
+| params (learnable) | **10,284,480** |
+| layers | 5 |
+| d_model | 320 |
+| heads | 5 (head_dim 64) |
+| FFN | SwiGLU, inter 896 |
+| vocab | 12,288 (gollem byte-level BPE) |
+| context | 512 |
+| embeddings | tied (lm_head → tok.weight) |
+| norm | RMSNorm, pre-norm |
+| attention | causal, RoPE (base 10000) |
+
+The parameter count is the learnable total: the raw safetensors sum is 14,216,640,
+which double-counts the tied embedding (tok.weight 12288×320 = 3,932,160) that
+`head.weight` aliases. Tied, the true count is 10,284,480.
+
+## Training
+
+- **Data:** FineWeb-Edu (~30.1M tokens) + DCLM-baseline (~21.4M tokens) = **~50.5M tokens**.
+- **Steps:** 4000 (the requested 2.6B-token budget is **not** met — see below).
+- **Final val loss:** 4.6020 (ppl 99.68) on the held-out split.
+- Trained from scratch (no base model).
+
+## Honest status: undertrained, not broken
+
+The model learned real context — val loss 4.602 is well below the 7.38 unigram
+floor, so it is **not** flatlined and **not** degenerate. But it has seen only
+~50.5M tokens ≈ **4.9 tokens/param**, far below the ~10–20+ tok/param small
+models typically need for coherent prose. The result is grammatical but
+semantically thin: wordy, repetitive, and it drifts off-topic mid-sentence.
+
+Real samples (best.pt, step 4000, temp 0.8, top-k 40) — no broken tokens, no
+token-loops:
+
+> "The sun is" → "The sun is assembled by a new study because he is an associate
+> of the study of the disease in the early years. I've been interested in having
+> a very different study…"
+
+> "Once upon a time" → "Once upon a time, she is an attack. But he is not a good
+> idea. But it is something that does not have a moment of his own life…"
+
+> "The cat sat on the" → "The cat sat on the ground. The sunp is a piece of light
+> and is not a good deal of tear. The new story of the MD's Ford…"
+
+> "def hello():" → "def hello(): I have a lot of the best. I'm not sure what
+> happened to me. I'll be able to do anything, but I think it's been just a lot
+> to say…"
+
+**What this is:** a verified, non-degenerate 10M checkpoint that learned grammar
+and surface fluency. **What it is not:** a coherent prose model. It is a first
+milestone on the path to the requested 2.6B-token model, not the destination.
+
+## Loading
+
+Self-contained `model.py` (no transformers dependency required):
+
+```python
+import torch
+from model import LDT
+
+m = LDT.from_pretrained("model.safetensors", device="cpu")
+ids = torch.tensor([[1, 2, 3]])  # token ids
+out = m.generate(ids, max_new_tokens=64, temperature=0.8, top_k=40, seed=0)
+```
+
+The tokenizer is a byte-level BPE (`tokenizer.json`, vocab 12288).
+
+## Roadmap
+
+Continuing training toward the 2.6B-token budget is the next step; this repo
+will be updated (or a v2 published) once the model is coherent. This checkpoint
+is kept public as an honest intermediate, not a finished model.