| # Lesson 05 results: instruction tuning from scratch |
|
|
| Run: 2026-10-11, `lesson_05.ipynb` executed end to end with papermill (kernel `tiny-storyteller`). |
| Device: Apple Silicon Mac, PyTorch 2.14.1, MPS, GPU shared with other jobs. Not run on Colab tonight |
| (T4 time not measured). Wall-clock, whole notebook: **139.6 s** (final run, after the grading fixes, GPU shared; |
| it trained from the final Lesson 04 v2). Earlier runs: 69.4 s (without the familiar-request contrast) and 102.1 s. |
| All three gave the same check numbers, held-out losses and samples. |
|
|
| ## Setup |
| - Start: `tiny-storyteller-v2` downloaded from the Hub repo `singhpratech/emc2-lesson-04-fine-tune` |
| (the final Lesson 04 run), 5,853,184 parameters, vocab 4,096. |
| - Special tokens added: `<|instruction|>` (id 4096), `<|story|>` (id 4097). Embedding (and the tied output |
| head) resized 4,096 -> 4,098 rows; old rows copied, new rows = mean of old rows. 5,853,696 parameters after. |
| - Chat template: `<|endoftext|><|instruction|>{instruction}<|story|>{story}<|endoftext|>`. |
| - Prompt masking: labels -100 for every position whose target is a prompt token; padding also -100. |
| - Pairs (4,000 training): |
| - 2,000 TinyStories-Instruct pairs from `TinyStories-Instruct-valid.txt` (public dataset |
| `roneneldan/TinyStoriesInstruct`, 26.9 MB; not found as separate files inside `roneneldan/TinyStories`, |
| only inside its 1.6 GB tarball). 25,027 records; pairs with Summary + Story that fit in 256 tokens, in file |
| order; first 2,000 train, next 100 held out. Instruction = "Write a story about this: {Summary}" + |
| " Use the words a, b and c." when Words is present. |
| - 2,000 generator pairs: 1,000 Momo + 1,000 Zuri (`generator_pairs(1000, seed=5)`), instruction |
| "Write a story about {name} {scene gist} {setting}." (setting omitted if the story names no place). |
| 840 of the 2,000 (446 Momo, 394 Zuri) name no place, so about 58% teach place-copying. |
| - Held-out test: 50 instructions (25 Momo, 25 Zuri, seed 123), each a (character, scene, setting) |
| combination that appears in no training story (training stories of that character and scene containing |
| the held-out place anywhere were excluded). Every test instruction names a place. |
| - Familiar contrast (not the check): 50 instructions (seed 456) whose combination does occur in training; |
| stories are new, wording may match a training instruction. |
| - Cross-world stretch (not the check): 20 instructions putting each character in the other's places. |
| - Held-out loss set: 150 pairs (100 held-out TinyStories-Instruct + the 50 test pairs with reference stories). |
| - Training: 500 steps, batch 32 pairs (padded), AdamW lr 3e-4 (warm-up 30, cosine to 10%), wd 0.1, clip 1.0, seed 42. |
| - Generation for the check: temperature 0.8, top-k 40, max 200 new tokens, seed = instruction index. |
| Before tuning: v2 gets `<|endoftext|>` + instruction as plain text (it has no special tokens). |
| |
| ## Leakage check (executed, cell asserts) |
| test stories in training 0; test instructions in training 0; training stories showing a held-out |
| combination 0; Lesson 04 quiz stories (100 Momo + 100 Zuri, rebuilt with the same seeds) in training 0; |
| TinyStories-Instruct held-out stories in training 0. Familiar-set stories in training: 0 (assert). |
|
|
| ## THE CHECK (string match: character name AND exact place phrase, case-insensitive for the place) |
| | 50 held-out instructions | both | character | setting | |
| |---|---|---|---| |
| | before (v2, plain prompt) | 12% (6/50) | 68% | 14% | |
| | after (tiny-storyteller-instruct, template) | **44% (22/50)** | 100% | 44% | |
|
|
| After, by character: Momo 9/25, Zuri 13/25. |
|
|
| Contrast sets: |
| - Familiar combinations: before 8% both (character 82%, setting 8%); after **86%** both (43/50; character 100%). |
| - Cross-world, after: 5% both (1/20; character 100%, setting 5%). |
|
|
| Held-out masked loss: 2.540 before (v2 + two new rows) -> 2.145 after. |
| Training time, final run: 50.9 s for 500 steps = 102 ms per step (GPU shared). Earlier: 37.4 s (75 ms/step) and 36 s (72 ms/step). |
|
|
| ## Observations (from the executed notebook) |
| - Typical held-out failure: right character and scene, practised place instead of the requested one |
| ("Momo hiding from the rain near the little pond" -> Momo hid "in the tall grass"). |
| - TinyStories-Instruct sample ("A little dog finds a lost ball in the park and gives it back to a girl."): kept the |
| park and the ball, the dog never appears. |
| - Before tuning, v2 given a Momo request wrote about Zuri ("Words can hurt, Zuri") - the Lesson 04 bleed. |
| - Generator stories are still recited template sentences. |
|
|
| ## Artifacts |
| - `tiny-storyteller-instruct/`: model.pt, config.json (vocab 4098), tokenizer.json, tokenizer_config.json, chat_template.json. 23 MB. |
| - `pairs/`: train.jsonl (4,000), heldout.jsonl (150), test50.jsonl, familiar50.jsonl, crossworld20.jsonl. 3.1 MB. |
| TinyStories-Instruct rows are derived from a CDLA-Sharing-1.0 dataset. |
| - `story_pairs.py`: the pair generator. `results.json`: all numbers and every generated test story. |
| - Masking widget checked in headless Chromium at 390 px and 700 px: no console errors, no horizontal scroll. |
| - Grading fixes in the final run: "signs" instead of "cards" for the special tokens, completion vs following named, example cell prints only the instruction, commented login line, taller token box on narrow screens. |
|
|
| ## What went wrong / limits |
| - The TinyStories-Instruct files are not separate files in `roneneldan/TinyStories`; used the authors' |
| `roneneldan/TinyStoriesInstruct` validation file instead (fast download). |
| - One sample per instruction (seed fixed); 50 instructions is a small set, so each point is 2 percentage points. |
|
|
| ## Measured after the run (post-grading) |
| - Old-quiz forgetting (2026-10-11, `measure_old_quiz.py`, Apple Silicon Mac, MPS): Lesson 04's quiz (a), the 200 held-out |
| TinyStories (35,448 tokens), scored exactly as Lesson 04 did (stories joined with `<|endoftext|>`, back-to-back |
| 256-token windows, every token once; Lesson 04's model class and perplexity code, unchanged). |
| tiny-storyteller-v2 (start, sanity re-score) **11.75**; tiny-storyteller-instruct (this lesson's model, local |
| `tiny-storyteller-instruct/`) **12.85**. No replay was mixed into instruction tuning (training was the 4,000 |
| instruction pairs only). TinyStories rows read from the cached first parquet shard, the same first 20,000 rows the |
| stream returns (token count and v2's 11.75 match Lesson 04 exactly). |
|
|
