Update to v4 checkpoint: 444M tokens, val 3.8943 (ppl 49.12), still incoherent

README.md

@@ -44,44 +44,46 @@ Standard LLaMA block: RMSNorm → MHA (RoPE) → residual → RMSNorm → SwiGLU
 
 ## Training
 
-| | |
-|---|---|
-| Steps | 4,000 |
-| Batch size | 64 |
-| LR | 3e-4 → 3e-5 (cosine decay) |
-| Data | FineWeb-Edu (30.1M tok) + DCLM baseline (21.4M tok) = **~50.5M tokens** |
-| Tokens/param | **~4.9** |
-| Hardware | RTX 5090 (32 GB), GPU |
-| Final val loss | **4.6020** (ppl 99.68) |
+| | v1 (first checkpoint) | v4 (current) |
+|---|---|---|
+| Steps | 4,000 | 16,000 (4,000 + 12,000 continued) |
+| Batch size | 64 | 64 |
+| LR | 3e-4 → 3e-5 (cosine) | 1e-4 → 1e-5 (cosine, fresh optimizer) |
+| Data | FineWeb-Edu 30.1M + DCLM 21.4M = **~50.5M tok** | + FineWeb-Edu 150.6M + DCLM 107.3M = **~393M tok** |
+| Cumulative tokens | ~50.5M | **~444M** |
+| Tokens/param | ~4.9 | **~43.1** |
+| Hardware | RTX 5090 (32 GB) | RTX 5090 (32 GB) |
+| Final val loss | 4.6020 (ppl 99.68) | **3.8943 (ppl 49.12)** |
 
-### ⚠️ Honest caveat: undertrained and incoherent
+### ⚠️ Honest caveat: undertrained and still incoherent
 
-DedeProGames requested **2.6B tokens**. This checkpoint is at **50.5M tokens** — a ~50× shortfall. The GPU was occupied by other work for most of the training window, and the CPU was over-subscribed.
+DedeProGames requested **2.6B tokens**. This checkpoint is at **~444M tokens** — a **5.9× shortfall**. The GPU was occupied by other work for most of the training window.
 
-At 4.9 tok/param the model has learned the **surface shape** of English — real words, parseable sentences, no token loops — but **not coherent meaning**. The val loss (4.602) is well below the 7.38 unigram floor, so it genuinely uses context rather than just reproducing the marginal word distribution; but the prose is **semantically incoherent** (word salad): grammatically plausible sentences that don't mean what they say.
+At 43 tok/param the model has learned the **surface shape** of English much better than v1 — real words, parseable sentences, no token loops — but the prose is still **semantically incoherent** (word salad): grammatically plausible sentences that don't mean what they say. The val loss (3.89) is well below the 7.38 unigram floor, so it genuinely uses context; the improvement from 4.60 → 3.89 is real.
 
-This is a **first checkpoint**, not the final deliverable. Continued training toward the 2.6B budget is planned.
+This is a **continued checkpoint**, not the final deliverable. More data is the fix; continued training toward the 2.6B budget is planned.
 
-## Eval (40 samples, 8 prompts × 5 seeds, temp 0.8, top-k 40)
+## Eval (8 prompts, temp 0.8, top-k 40, seed 1234)
 
-| Metric | Value |
-|---|---|
-| mean loop_frac | 0.042 |
-| max loop_frac | 1.0 (one sample) |
-| Degenerate? | **No** |
-| Below unigram floor? | **Yes** (4.602 < 7.38) |
+| Metric | v1 | v4 |
+|---|---|---|
+| val loss | 4.6020 | **3.8943** |
+| perplexity | 99.68 | **49.12** |
+| Below unigram floor (7.38)? | Yes | Yes |
+| Token loops? | No | No |
+| Semantically coherent? | No | No (improved, still word salad) |
 
-### Sample outputs
+### Sample outputs (v4, real generation from the weights)
 
-> "The cat sat on the earth and it will be asleep. It is also a little time to get up the world. So the world is a fancy mound, and it is a little more common one. The last thing about the world is a part of a real life. It would be a great place, like it and in the earth. The problem is, the old world will be a good idea. So the world is just a bit of a whole larvity."
+> "The cat sat on the same line of the moon as a dancer. The planet is the only planet that has been known to be a good deal of tear. The last thing about the world is a part of a real life."
 
-> "She opened the door and started her home and said she was in the room. The only time she wasn think she was being a young girl. She didn't have that. She was not like the kids that was the one who had been killed in the kitchen. She was in a house a few months later. She was a very small girl and she was at her friends."
+> "Once upon a time, the person will be sent to each other, or if you are going to go through it. But a lot of people are going to have a good chance of a feeling. But that is, I do not know what they're going to do."
 
-> "Once upon a time, the person will be sent to each other, or if you are going to go through it. So, you know, that's the question of what we are talking about. But a lot of people are going to have a good chance of a feeling. But that is, I do not know what they're going to do."
+> "Water is made of hydrogen and oxygen, and it is a good deal of tear. The sun is a piece of light and is not a good deal of tear. The sun is not a good deal of tear."
 
-> "def hello(): I can understand how to make a word. You can't find the word to say (they'll do anything) and try to explain it. I'm not sure if I can't read it, but I'm trying to figure out what to do to say, but I'm sure I'm using a code that I have to look at the other side of a computer."
+> "To be or not to be is a question about the world. The last thing about the world is a part of a real life. It would be a great place, like it and in the earth."
 
-These are real outputs from the eval run (not hand-picked for coherence). The text is grammatically structured — real words, parseable sentences, no token loops, no broken tokens — but semantically incoherent. That is the honest state of a 10M model at 4.9 tok/param.
+These are real outputs from the v4 weights (not hand-picked for coherence). The text is grammatically structured — real words, parseable sentences, no token loops, no broken tokens — but semantically incoherent. That is the honest state of a 10M model at 43 tok/param.
 
 ## Usage
 
@@ -91,7 +93,7 @@ To load with a custom model class, you need a small LLaMA-style implementation m
 
 ## What this is NOT
 
-- Not a 2.6B-token model (that's the target; this is the 50.5M checkpoint)
+- Not a 2.6B-token model (that's the target; this is the ~444M checkpoint)
 - Not a coherent-text model (it produces grammatically-structured word salad at this token budget)
 - Not a general-purpose assistant (it's a raw LM, no instruction tuning)
 - Not a replacement for anything larger — it's a research checkpoint in a from-scratch training run