Grading pass: Momo-free validation split, 1,000 truly unique Momo stories, decoded terms, widget fixes, honest numbers

PLAYBOOK.md

@@ -7,23 +7,25 @@ and Ella the elephant. Pick the path that fits you.
 ## What you get
 
 A 5.9-million-parameter GPT-style language model, trained from scratch on simple children's
-stories (TinyStories) plus a thousand stories about Momo and Ella. Give it the first few words
-and it continues with a short, simple story. It is a learning project, not a product: expect
-charming nonsense now and then.
+stories (TinyStories) plus a thousand generated stories about Momo the monkey and Ella the
+elephant. Give it the first few words and it continues with a short, simple story. It is a
+learning project, not a product: expect charming nonsense now and then, and expect the Momo
+stories to be recited from memory rather than invented (the notebook explains why).
 
 ## Path A: run it in Google Colab (no install, free GPU)
 
 1. Open https://colab.research.google.com and choose **Upload notebook**.
 2. Pick `tiny_llm_from_scratch.ipynb` (download it from this repo first).
 3. In Colab go to **Runtime > Change runtime type** and choose **T4 GPU**.
-4. Press **Run all**. Everything above the training section runs instantly. Training takes
-   about 1.5 minutes on a T4. The whole notebook is about 5 minutes.
+4. Press **Run all**. Everything above the training section runs in well under a minute.
+   Training takes about 2 minutes on a T4. The whole notebook is about 5 minutes.
 5. Scroll to **Part 8** to read the generated stories, and change the prompts to your own.
 
-The last cell publishes the model to your own Hugging Face account. It needs a Hugging Face
-token with write permission. The easiest way on Colab: open the **Secrets** panel (the key
-icon on the left), add a secret called `HF_TOKEN`, switch on notebook access, then run the
-cell. Without a token, set `PUSH_TO_HUB = False` in that cell and it only saves locally.
+The publish cell in Part 9 uploads the model to your own Hugging Face account, as a
+**private** repo. It needs a Hugging Face access token with write permission. The easiest way
+on Colab: open the **Secrets** panel (the key icon on the left), add a secret called
+`HF_TOKEN`, switch on notebook access, then run the cell. Without a token, set
+`PUSH_TO_HUB = False` in the save cell just above it and the model is only saved locally.
 
 ## Path B: run it on your own computer
 
@@ -33,7 +35,7 @@ on plain CPU (slower, but it works).
 ```bash
 # 1. Python environment (any Python 3.10+; uv is fastest but pip works too)
 uv venv --python 3.13 && source .venv/bin/activate
-uv pip install torch transformers datasets tokenizers huggingface_hub nbformat nbconvert ipykernel
+uv pip install torch transformers datasets tokenizers huggingface_hub nbformat nbconvert ipykernel jupyterlab
 
 # 2. Run the notebook end to end (about 4 minutes on an M-series Mac)
 jupyter nbconvert --to notebook --execute tiny_llm_from_scratch.ipynb --output run.ipynb
@@ -42,13 +44,15 @@ jupyter nbconvert --to notebook --execute tiny_llm_from_scratch.ipynb --output r
 jupyter lab tiny_llm_from_scratch.ipynb
 ```
 
-To publish from your machine, put `HF_TOKEN=hf_...` in a `.env` file or your shell
-environment before running. The notebook never prints the token.
+To publish from your machine, either run `hf auth login` once, or export `HF_TOKEN=hf_...`
+in your shell before starting Jupyter. The notebook never prints the token.
 
 ## Path C: skip training, just use the model
 
-The trained model lives on the Hugging Face Hub as `singhpratech/tiny-storyteller`. The
-notebook's last section, **Loading it back later**, shows the loader. In short:
+The trained model lives on the Hugging Face Hub as `singhpratech/tiny-storyteller`. While
+that repo is private you need its owner's token to download it; once it is public (or once
+you have published your own copy, under your own name) the lines below work for anyone. The
+notebook's Part 10, **Loading it back later**, has the full loader. In short:
 
 ```python
 from huggingface_hub import hf_hub_download
@@ -85,11 +89,12 @@ story from the story marker.
 |---|---|
 | `N_STORIES` 20,000 → 100,000 | Noticeably better grammar and memory of characters |
 | `MAX_STEPS` 1,500 → 5,000 | Lower loss; about 4 minutes on a T4 |
-| `n_layer` 6 → 8, `n_embd` 256 → 384 | Bigger model (about 15M parameters), slower, better |
+| `n_layer` 6 → 8, `n_embd` 256 → 384 | Bigger model (about 16M parameters), slower, better |
 | More Momo stories, or your own text | The model writes about what it reads |
 
-Watch **perplexity** in the training output. Random guessing is 4,096. Under 10 means real
-sentences. This recipe lands around 9.
+Watch **perplexity** in the training output. Random guessing is 4,096. Around 10 means real
+sentences. This recipe lands at about 10 on the hidden quiz stories (200 TinyStories the
+model never trains on, with no Momo stories in them, so the number is honest).
 
 ## When something goes wrong
 
@@ -100,6 +105,8 @@ sentences. This recipe lands around 9.
 - **Out of memory on a small GPU**: lower `BATCH_SIZE` in Part 7 to 16.
 - **Stories are all about Lily**: the Momo stories are only a few percent of the data. Raise
   the repeat factor in the Momo data cell from 3 to 6, or prompt with Momo's name.
+- **Momo stories are recited word for word**: that is memorising. Add more scenes to the
+  sentence pools in `momo_stories.py` (more unique stories) and lower the repeat factor.
 
 ## Rules of the road
 

README.md

@@ -7,12 +7,14 @@ tags: [tiny, from-scratch, pytorch, educational, text-generation]
 # Tiny Storyteller
 
 A 5.9M-parameter GPT-style language model trained **from scratch** (random init, no pretrained weights)
-on 20,000 TinyStories, as a learning project. Use case: a bedtime story starter for young kids.
+on 19,800 TinyStories plus 3,000 short synthetic stories about two characters, Momo the monkey and
+Ella the elephant (1,000 unique stories, each repeated 3 times, generated by `momo_stories.py`), as a learning project.
+Use case: a bedtime story starter for young kids.
 
 - Architecture: 6 transformer blocks, 4 heads, 256-dim embeddings, 256-token context
 - Tokenizer: byte-level BPE, 4,096 tokens, trained on the same data
 - Training: 1500 steps, batch 32, AdamW, cosine schedule
-- Final validation loss 2.315 (perplexity 10.1)
+- Final validation loss 2.202 (perplexity 9.0) on 200 held-out TinyStories (no Momo stories in the validation set)
 
 The full build, explained step by step, is in `tiny_llm_from_scratch.ipynb` in this repo.
 Load with plain PyTorch: `GPT(cfg).load_state_dict(torch.load("model.pt"))`.

model.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4355a3961c5ab8a4367ded941d3b88019d5c1feeb3d0aa569a428a741dea2540
+oid sha256:75d4755f95c1e8bc4f051e38114bc28efa02b186e36b11c309840e5a7a157095
 size 23434075

momo_stories.py

@@ -88,8 +88,15 @@ def make_story(rng: random.Random) -> str:
 
 
 def momo_stories(n: int = 1000, seed: int = 0) -> list[str]:
+    # n distinct stories: repeats from the pools are skipped, so "unique" really is unique.
     rng = random.Random(seed)
-    return [make_story(rng) for _ in range(n)]
+    out, seen = [], set()
+    while len(out) < n:
+        s = make_story(rng)
+        if s not in seen:
+            seen.add(s)
+            out.append(s)
+    return out
 
 
 if __name__ == "__main__":