Fix v4 token-count arithmetic: v4 data build is 257.9M tok (fw 150.6M + dclm 107.3M), cumulative ~308M, ~30 tok/param, 8.4x shortfall — not the previously stated ~393M/~444M/~43.1. Verified against models/ldt-10m-v4/train.log data-build lines.

README.md

@@ -49,17 +49,17 @@ Standard LLaMA block: RMSNorm → MHA (RoPE) → residual → RMSNorm → SwiGLU
 | Steps | 4,000 | 16,000 (4,000 + 12,000 continued) |
 | Batch size | 64 | 64 |
 | LR | 3e-4 → 3e-5 (cosine) | 1e-4 → 1e-5 (cosine, fresh optimizer) |
-| Data | FineWeb-Edu 30.1M + DCLM 21.4M = **~50.5M tok** | + FineWeb-Edu 150.6M + DCLM 107.3M = **~393M tok** |
-| Cumulative tokens | ~50.5M | **~444M** |
-| Tokens/param | ~4.9 | **~43.1** |
+| Data | FineWeb-Edu 30.1M + DCLM 21.4M = **~50.5M tok** | + FineWeb-Edu 150.6M + DCLM 107.3M = **~257.9M tok** |
+| Cumulative tokens | ~50.5M | **~308M** |
+| Tokens/param | ~4.9 | **~30.0** |
 | Hardware | RTX 5090 (32 GB) | RTX 5090 (32 GB) |
 | Final val loss | 4.6020 (ppl 99.68) | **3.8943 (ppl 49.12)** |
 
 ### ⚠️ Honest caveat: undertrained and still incoherent
 
-DedeProGames requested **2.6B tokens**. This checkpoint is at **~444M tokens** — a **5.9× shortfall**. The GPU was occupied by other work for most of the training window.
+DedeProGames requested **2.6B tokens**. This checkpoint is at **~308M tokens** — an **8.4× shortfall**. The GPU was occupied by other work for most of the training window.
 
-At 43 tok/param the model has learned the **surface shape** of English much better than v1 — real words, parseable sentences, no token loops — but the prose is still **semantically incoherent** (word salad): grammatically plausible sentences that don't mean what they say. The val loss (3.89) is well below the 7.38 unigram floor, so it genuinely uses context; the improvement from 4.60 → 3.89 is real.
+At 30 tok/param the model has learned the **surface shape** of English much better than v1 — real words, parseable sentences, no token loops — but the prose is still **semantically incoherent** (word salad): grammatically plausible sentences that don't mean what they say. The val loss (3.89) is well below the 7.38 unigram floor, so it genuinely uses context; the improvement from 4.60 → 3.89 is real.
 
 This is a **continued checkpoint**, not the final deliverable. More data is the fix; continued training toward the 2.6B budget is planned.
 
@@ -83,7 +83,7 @@ This is a **continued checkpoint**, not the final deliverable. More data is the
 
 > "To be or not to be is a question about the world. The last thing about the world is a part of a real life. It would be a great place, like it and in the earth."
 
-These are real outputs from the v4 weights (not hand-picked for coherence). The text is grammatically structured — real words, parseable sentences, no token loops, no broken tokens — but semantically incoherent. That is the honest state of a 10M model at 43 tok/param.
+These are real outputs from the v4 weights (not hand-picked for coherence). The text is grammatically structured — real words, parseable sentences, no token loops, no broken tokens — but semantically incoherent. That is the honest state of a 10M model at 30 tok/param.
 
 ## Usage
 
@@ -93,7 +93,7 @@ To load with a custom model class, you need a small LLaMA-style implementation m
 
 ## What this is NOT
 
-- Not a 2.6B-token model (that's the target; this is the ~444M checkpoint)
+- Not a 2.6B-token model (that's the target; this is the ~308M checkpoint)
 - Not a coherent-text model (it produces grammatically-structured word salad at this token budget)
 - Not a general-purpose assistant (it's a raw LM, no instruction tuning)
 - Not a replacement for anything larger — it's a research checkpoint in a from-scratch training run