Fix fabricated sample quote + overstatement: replace sample #1 (not in eval output) with real samples from eval_ldt10m_final.json; correct "coherent/fluency" to "grammatically structured but semantically incoherent"
#10
by Compactbot - opened
README.md
CHANGED
|
@@ -54,11 +54,11 @@ Standard LLaMA block: RMSNorm → MHA (RoPE) → residual → RMSNorm → SwiGLU
|
|
| 54 |
| Hardware | RTX 5090 (32 GB), GPU |
|
| 55 |
| Final val loss | **4.6020** (ppl 99.68) |
|
| 56 |
|
| 57 |
-
### ⚠️ Honest caveat: undertrained
|
| 58 |
|
| 59 |
DedeProGames requested **2.6B tokens**. This checkpoint is at **50.5M tokens** — a ~50× shortfall. The GPU was occupied by other work for most of the training window, and the CPU was over-subscribed.
|
| 60 |
|
| 61 |
-
At 4.9 tok/param
|
| 62 |
|
| 63 |
This is a **first checkpoint**, not the final deliverable. Continued training toward the 2.6B budget is planned.
|
| 64 |
|
|
@@ -73,15 +73,15 @@ This is a **first checkpoint**, not the final deliverable. Continued training to
|
|
| 73 |
|
| 74 |
### Sample outputs
|
| 75 |
|
| 76 |
-
> "The
|
| 77 |
|
| 78 |
-
> "
|
| 79 |
|
| 80 |
-
> "
|
| 81 |
|
| 82 |
-
> "def hello(): I
|
| 83 |
|
| 84 |
-
|
| 85 |
|
| 86 |
## Usage
|
| 87 |
|
|
@@ -92,5 +92,6 @@ To load with a custom model class, you need a small LLaMA-style implementation m
|
|
| 92 |
## What this is NOT
|
| 93 |
|
| 94 |
- Not a 2.6B-token model (that's the target; this is the 50.5M checkpoint)
|
|
|
|
| 95 |
- Not a general-purpose assistant (it's a raw LM, no instruction tuning)
|
| 96 |
- Not a replacement for anything larger — it's a research checkpoint in a from-scratch training run
|
|
|
|
| 54 |
| Hardware | RTX 5090 (32 GB), GPU |
|
| 55 |
| Final val loss | **4.6020** (ppl 99.68) |
|
| 56 |
|
| 57 |
+
### ⚠️ Honest caveat: undertrained and incoherent
|
| 58 |
|
| 59 |
DedeProGames requested **2.6B tokens**. This checkpoint is at **50.5M tokens** — a ~50× shortfall. The GPU was occupied by other work for most of the training window, and the CPU was over-subscribed.
|
| 60 |
|
| 61 |
+
At 4.9 tok/param the model has learned the **surface shape** of English — real words, parseable sentences, no token loops — but **not coherent meaning**. The val loss (4.602) is well below the 7.38 unigram floor, so it genuinely uses context rather than just reproducing the marginal word distribution; but the prose is **semantically incoherent** (word salad): grammatically plausible sentences that don't mean what they say.
|
| 62 |
|
| 63 |
This is a **first checkpoint**, not the final deliverable. Continued training toward the 2.6B budget is planned.
|
| 64 |
|
|
|
|
| 73 |
|
| 74 |
### Sample outputs
|
| 75 |
|
| 76 |
+
> "The cat sat on the earth and it will be asleep. It is also a little time to get up the world. So the world is a fancy mound, and it is a little more common one. The last thing about the world is a part of a real life. It would be a great place, like it and in the earth. The problem is, the old world will be a good idea. So the world is just a bit of a whole larvity."
|
| 77 |
|
| 78 |
+
> "She opened the door and started her home and said she was in the room. The only time she wasn think she was being a young girl. She didn't have that. She was not like the kids that was the one who had been killed in the kitchen. She was in a house a few months later. She was a very small girl and she was at her friends."
|
| 79 |
|
| 80 |
+
> "Once upon a time, the person will be sent to each other, or if you are going to go through it. So, you know, that's the question of what we are talking about. But a lot of people are going to have a good chance of a feeling. But that is, I do not know what they're going to do."
|
| 81 |
|
| 82 |
+
> "def hello(): I can understand how to make a word. You can't find the word to say (they'll do anything) and try to explain it. I'm not sure if I can't read it, but I'm trying to figure out what to do to say, but I'm sure I'm using a code that I have to look at the other side of a computer."
|
| 83 |
|
| 84 |
+
These are real outputs from the eval run (not hand-picked for coherence). The text is grammatically structured — real words, parseable sentences, no token loops, no broken tokens — but semantically incoherent. That is the honest state of a 10M model at 4.9 tok/param.
|
| 85 |
|
| 86 |
## Usage
|
| 87 |
|
|
|
|
| 92 |
## What this is NOT
|
| 93 |
|
| 94 |
- Not a 2.6B-token model (that's the target; this is the 50.5M checkpoint)
|
| 95 |
+
- Not a coherent-text model (it produces grammatically-structured word salad at this token budget)
|
| 96 |
- Not a general-purpose assistant (it's a raw LM, no instruction tuning)
|
| 97 |
- Not a replacement for anything larger — it's a research checkpoint in a from-scratch training run
|