Add LDT-10M model card

README.md

@@ -1,101 +1,96 @@
 ---
-license: mit
+license: apache-2.0
 pipeline_tag: text-generation
 language: en
-library_name: transformers
 tags:
   - tiny
+  - tiny-lm
   - tiny-model
   - slm
+  - SLM
   - small-language-model
-  - sub-10m
-  - llama
   - from-scratch
+  - llama
+datasets:
+  - HuggingFaceFW/fineweb-edu
+  - allenai/dclm-baseline
 metrics:
   - perplexity
 ---
 
 # LDT-10M
 
-A **10,284,480-parameter** LLaMA-style text model, trained from scratch. This is a
-verified **first checkpoint** for the LDT-10M request (model-requests #12,
-DedeProGames) — real weights, real training, but **undertrained** (see the honest
-status below). It is not a quality release yet; the card states that plainly.
+A 10.28M-parameter LLaMA-style language model, trained from scratch on FineWeb-Edu + DCLM.
 
-## Architecture
+**Requested by [DedeProGames](https://huggingface.co/DedeProGames) on the [model-requests board](https://huggingface.co/spaces/Compactbot/model-requests) (#12).**
 
-Standard LLaMA block, no sliding window, no GQA:
+## Architecture
 
-| field | value |
+| Parameter | Value |
 |---|---|
-| params (learnable) | **10,284,480** |
-| layers | 5 |
+| Params | **10,284,480** |
+| Layers | 5 |
 | d_model | 320 |
-| heads | 5 (head_dim 64) |
-| FFN | SwiGLU, inter 896 |
-| vocab | 12,288 (gollem byte-level BPE) |
-| context | 512 |
-| embeddings | tied (lm_head → tok.weight) |
-| norm | RMSNorm, pre-norm |
-| attention | causal, RoPE (base 10000) |
-
-The parameter count is the learnable total: the raw safetensors sum is 14,216,640,
-which double-counts the tied embedding (tok.weight 12288×320 = 3,932,160) that
-`head.weight` aliases. Tied, the true count is 10,284,480.
+| Heads | 5 (MHA, GQA not used at this scale) |
+| FFN dim | 896 (SwiGLU) |
+| Vocab | 12,288 (gollem BPE) |
+| Context | 512 |
+| Embeddings | Tied (lm_head → tok.weight) |
+| Norm | RMSNorm (eps 1e-5) |
+| Attention | RoPE + causal SDPA |
+| Dtype | float32 |
+
+Standard LLaMA block: RMSNorm → MHA (RoPE) → residual → RMSNorm → SwiGLU FFN → residual.
 
 ## Training
 
-- **Data:** FineWeb-Edu (~30.1M tokens) + DCLM-baseline (~21.4M tokens) = **~50.5M tokens**.
-- **Steps:** 4000 (the requested 2.6B-token budget is **not** met — see below).
-- **Final val loss:** 4.6020 (ppl 99.68) on the held-out split.
-- Trained from scratch (no base model).
+| | |
+|---|---|
+| Steps | 4,000 |
+| Batch size | 64 |
+| LR | 3e-4 → 3e-5 (cosine decay) |
+| Data | FineWeb-Edu (30.1M tok) + DCLM baseline (21.4M tok) = **~50.5M tokens** |
+| Tokens/param | **~4.9** |
+| Hardware | RTX 5090 (32 GB), GPU |
+| Final val loss | **4.6020** (ppl 99.68) |
+
+### ⚠️ Honest caveat: undertrained
 
-## Honest status: undertrained, not broken
+DedeProGames requested **2.6B tokens**. This checkpoint is at **50.5M tokens** — a ~50× shortfall. The GPU was occupied by other work for most of the training window, and the CPU was over-subscribed.
 
-The model learned real context — val loss 4.602 is well below the 7.38 unigram
-floor, so it is **not** flatlined and **not** degenerate. But it has seen only
-~50.5M tokens ≈ **4.9 tokens/param**, far below the ~10–20+ tok/param small
-models typically need for coherent prose. The result is grammatical but
-semantically thin: wordy, repetitive, and it drifts off-topic mid-sentence.
+At 4.9 tok/param, the model has learned grammar and surface fluency but not deep coherence. The val loss (4.602) is well below the 7.38 unigram floor, so it genuinely uses context — but the prose is semantically thin: wordy, repetitive, and it drifts off-topic mid-sentence.
 
-Real samples (best.pt, step 4000, temp 0.8, top-k 40) — no broken tokens, no
-token-loops:
+This is a **first checkpoint**, not the final deliverable. Continued training toward the 2.6B budget is planned.
 
-> "The sun is" → "The sun is assembled by a new study because he is an associate
-> of the study of the disease in the early years. I've been interested in having
-> a very different study…"
+## Eval (40 samples, 8 prompts × 5 seeds, temp 0.8, top-k 40)
+
+| Metric | Value |
+|---|---|
+| mean loop_frac | 0.042 |
+| max loop_frac | 1.0 (one sample) |
+| Degenerate? | **No** |
+| Below unigram floor? | **Yes** (4.602 < 7.38) |
 
-> "Once upon a time" → "Once upon a time, she is an attack. But he is not a good
-> idea. But it is something that does not have a moment of his own life…"
+### Sample outputs
 
-> "The cat sat on the" → "The cat sat on the ground. The sunp is a piece of light
-> and is not a good deal of tear. The new story of the MD's Ford…"
+> "The sun is assembled by a new study because he is an associate of the study of the disease in the early years. I've been interested in having a very different study…"
 
-> "def hello():" → "def hello(): I have a lot of the best. I'm not sure what
-> happened to me. I'll be able to do anything, but I think it's been just a lot
-> to say…"
+> "Once upon a time, she is an attack. But he is not a good idea. But it is something that does not have a moment of his own life…"
 
-**What this is:** a verified, non-degenerate 10M checkpoint that learned grammar
-and surface fluency. **What it is not:** a coherent prose model. It is a first
-milestone on the path to the requested 2.6B-token model, not the destination.
+> "The cat sat on the ground. The sunp is a piece of light and is not a good deal of tear. The new story of the MD's Ford…"
 
-## Loading
+> "def hello(): I have a lot of the best. I'm not sure what happened to me. I'll be able to do anything, but I think it's been just a lot to say…"
 
-Self-contained `model.py` (no transformers dependency required):
+Grammar is intact. No token loops, no broken tokens, no speaker tags. Semantically limited — expected at 4.9 tok/param.
 
-```python
-import torch
-from model import LDT
+## Usage
 
-m = LDT.from_pretrained("model.safetensors", device="cpu")
-ids = torch.tensor([[1, 2, 3]])  # token ids
-out = m.generate(ids, max_new_tokens=64, temperature=0.8, top_k=40, seed=0)
-```
+The model uses a custom `LDT` architecture (standard LLaMA block, no special tricks). The safetensors file contains 47 tensors with tied embeddings (lm_head is not stored separately; it shares `tok.weight`).
 
-The tokenizer is a byte-level BPE (`tokenizer.json`, vocab 12288).
+To load with a custom model class, you need a small LLaMA-style implementation matching the config above. The training script (`train_ldt10m_fixed.py`) contains the full architecture definition.
 
-## Roadmap
+## What this is NOT
 
-Continuing training toward the 2.6B-token budget is the next step; this repo
-will be updated (or a v2 published) once the model is coherent. This checkpoint
-is kept public as an honest intermediate, not a finished model.
+- Not a 2.6B-token model (that's the target; this is the 50.5M checkpoint)
+- Not a general-purpose assistant (it's a raw LM, no instruction tuning)
+- Not a replacement for anything larger — it's a research checkpoint in a from-scratch training run