Add RESULTS.md

RESULTS.md

@@ -0,0 +1,72 @@
+# Lesson 05 results: instruction tuning from scratch
+
+Run: 2026-10-11, `lesson_05.ipynb` executed end to end with papermill (kernel `tiny-storyteller`).
+Device: Apple M4 Max, PyTorch 2.14.1, MPS, GPU shared with other jobs. Not run on Colab tonight
+(T4 time not measured). Wall-clock, whole notebook: **102.1 s** (final run; an earlier run without
+the familiar-request contrast took 69.4 s and gave the same check numbers).
+
+## Setup
+- Start: `tiny-storyteller-v2` downloaded from the Hub repo `singhpratech/emc2-lesson-04-fine-tune`
+  (the final Lesson 04 run), 5,853,184 parameters, vocab 4,096.
+- Special tokens added: `<|instruction|>` (id 4096), `<|story|>` (id 4097). Embedding (and the tied output
+  head) resized 4,096 -> 4,098 rows; old rows copied, new rows = mean of old rows. 5,853,696 parameters after.
+- Chat template: `<|endoftext|><|instruction|>{instruction}<|story|>{story}<|endoftext|>`.
+- Prompt masking: labels -100 for every position whose target is a prompt token; padding also -100.
+- Pairs (4,000 training):
+  - 2,000 TinyStories-Instruct pairs from `TinyStories-Instruct-valid.txt` (public dataset
+    `roneneldan/TinyStoriesInstruct`, 26.9 MB; not found as separate files inside `roneneldan/TinyStories`,
+    only inside its 1.6 GB tarball). 25,027 records; pairs with Summary + Story that fit in 256 tokens, in file
+    order; first 2,000 train, next 100 held out. Instruction = "Write a story about this: {Summary}" +
+    " Use the words a, b and c." when Words is present.
+  - 2,000 generator pairs: 1,000 Momo + 1,000 Zuri (`generator_pairs(1000, seed=5)`), instruction
+    "Write a story about {name} {scene gist} {setting}." (setting omitted if the story names no place).
+- Held-out test: 50 instructions (25 Momo, 25 Zuri, seed 123), each a (character, scene, setting)
+  combination that appears in no training story (training stories of that character and scene containing
+  the held-out place anywhere were excluded). Every test instruction names a place.
+- Familiar contrast (not the check): 50 instructions (seed 456) whose combination does occur in training;
+  stories are new, wording may match a training instruction.
+- Cross-world stretch (not the check): 20 instructions putting each character in the other's places.
+- Held-out loss set: 150 pairs (100 held-out TinyStories-Instruct + the 50 test pairs with reference stories).
+- Training: 500 steps, batch 32 pairs (padded), AdamW lr 3e-4 (warm-up 30, cosine to 10%), wd 0.1, clip 1.0, seed 42.
+- Generation for the check: temperature 0.8, top-k 40, max 200 new tokens, seed = instruction index.
+  Before tuning: v2 gets `<|endoftext|>` + instruction as plain text (it has no special tokens).
+
+## Leakage check (executed, cell asserts)
+test stories in training 0; test instructions in training 0; training stories showing a held-out
+combination 0; Lesson 04 quiz stories (100 Momo + 100 Zuri, rebuilt with the same seeds) in training 0;
+TinyStories-Instruct held-out stories in training 0. Familiar-set stories in training: 0 (assert).
+
+## THE CHECK (string match: character name AND exact place phrase, case-insensitive for the place)
+| 50 held-out instructions | both | character | setting |
+|---|---|---|---|
+| before (v2, plain prompt) | 12% (6/50) | 68% | 14% |
+| after (tiny-storyteller-instruct, template) | **44% (22/50)** | 100% | 44% |
+
+After, by character: Momo 9/25, Zuri 13/25.
+
+Contrast sets:
+- Familiar combinations: before 8% both (character 82%, setting 8%); after **86%** both (43/50; character 100%).
+- Cross-world, after: 5% both (1/20; character 100%, setting 5%).
+
+Held-out masked loss: 2.540 before (v2 + two new rows) -> 2.145 after.
+Training time: 37.4 s for 500 steps = 75 ms per step (first run: 36 s, 72 ms/step).
+
+## Observations (from the executed notebook)
+- Typical held-out failure: right character and scene, practised place instead of the requested one
+  ("Momo hiding from the rain near the little pond" -> Momo hid "in the tall grass").
+- TinyStories-Instruct sample ("A little dog finds a lost ball in the park and gives it back to a girl."): kept the
+  park and the ball, the dog never appears.
+- Before tuning, v2 given a Momo request wrote about Zuri ("Words can hurt, Zuri") - the Lesson 04 bleed.
+- Generator stories are still recited template sentences.
+
+## Artifacts
+- `tiny-storyteller-instruct/`: model.pt, config.json (vocab 4098), tokenizer.json, tokenizer_config.json, chat_template.json. 23 MB.
+- `pairs/`: train.jsonl (4,000), heldout.jsonl (150), test50.jsonl, familiar50.jsonl, crossworld20.jsonl. 3.1 MB.
+  TinyStories-Instruct rows are derived from a CDLA-Sharing-1.0 dataset.
+- `story_pairs.py`: the pair generator. `results.json`: all numbers and every generated test story.
+- Masking widget checked in headless Chromium at 700 px: no console errors, no horizontal scroll.
+
+## What went wrong / limits
+- The TinyStories-Instruct files are not separate files in `roneneldan/TinyStories`; used the authors'
+  `roneneldan/TinyStoriesInstruct` validation file instead (fast download).
+- One sample per instruction (seed fixed); 50 instructions is a small set, so each point is 2 percentage points.