Lesson 05 results: instruction tuning from scratch

Run: 2026-10-11, lesson_05.ipynb executed end to end with papermill (kernel tiny-storyteller). Device: Apple Silicon Mac, PyTorch 2.14.1, MPS, GPU shared with other jobs. Not run on Colab tonight (T4 time not measured). Wall-clock, whole notebook: 139.6 s (final run, after the grading fixes, GPU shared; it trained from the final Lesson 04 v2). Earlier runs: 69.4 s (without the familiar-request contrast) and 102.1 s. All three gave the same check numbers, held-out losses and samples.

Setup

Start: tiny-storyteller-v2 downloaded from the Hub repo singhpratech/emc2-lesson-04-fine-tune (the final Lesson 04 run), 5,853,184 parameters, vocab 4,096.
Special tokens added: <|instruction|> (id 4096), <|story|> (id 4097). Embedding (and the tied output head) resized 4,096 -> 4,098 rows; old rows copied, new rows = mean of old rows. 5,853,696 parameters after.
Chat template: <|endoftext|><|instruction|>{instruction}<|story|>{story}<|endoftext|>.
Prompt masking: labels -100 for every position whose target is a prompt token; padding also -100.
Pairs (4,000 training):
- 2,000 TinyStories-Instruct pairs from TinyStories-Instruct-valid.txt (public dataset roneneldan/TinyStoriesInstruct, 26.9 MB; not found as separate files inside roneneldan/TinyStories, only inside its 1.6 GB tarball). 25,027 records; pairs with Summary + Story that fit in 256 tokens, in file order; first 2,000 train, next 100 held out. Instruction = "Write a story about this: {Summary}" + " Use the words a, b and c." when Words is present.
- 2,000 generator pairs: 1,000 Momo + 1,000 Zuri (generator_pairs(1000, seed=5)), instruction "Write a story about {name} {scene gist} {setting}." (setting omitted if the story names no place). 840 of the 2,000 (446 Momo, 394 Zuri) name no place, so about 58% teach place-copying.
Held-out test: 50 instructions (25 Momo, 25 Zuri, seed 123), each a (character, scene, setting) combination that appears in no training story (training stories of that character and scene containing the held-out place anywhere were excluded). Every test instruction names a place.
Familiar contrast (not the check): 50 instructions (seed 456) whose combination does occur in training; stories are new, wording may match a training instruction.
Cross-world stretch (not the check): 20 instructions putting each character in the other's places.
Held-out loss set: 150 pairs (100 held-out TinyStories-Instruct + the 50 test pairs with reference stories).
Training: 500 steps, batch 32 pairs (padded), AdamW lr 3e-4 (warm-up 30, cosine to 10%), wd 0.1, clip 1.0, seed 42.
Generation for the check: temperature 0.8, top-k 40, max 200 new tokens, seed = instruction index. Before tuning: v2 gets <|endoftext|> + instruction as plain text (it has no special tokens).

Leakage check (executed, cell asserts)

test stories in training 0; test instructions in training 0; training stories showing a held-out combination 0; Lesson 04 quiz stories (100 Momo + 100 Zuri, rebuilt with the same seeds) in training 0; TinyStories-Instruct held-out stories in training 0. Familiar-set stories in training: 0 (assert).

THE CHECK (string match: character name AND exact place phrase, case-insensitive for the place)

50 held-out instructions	both	character	setting
before (v2, plain prompt)	12% (6/50)	68%	14%
after (tiny-storyteller-instruct, template)	44% (22/50)	100%	44%

After, by character: Momo 9/25, Zuri 13/25.

Contrast sets:

Familiar combinations: before 8% both (character 82%, setting 8%); after 86% both (43/50; character 100%).
Cross-world, after: 5% both (1/20; character 100%, setting 5%).

Held-out masked loss: 2.540 before (v2 + two new rows) -> 2.145 after. Training time, final run: 50.9 s for 500 steps = 102 ms per step (GPU shared). Earlier: 37.4 s (75 ms/step) and 36 s (72 ms/step).

Observations (from the executed notebook)

Typical held-out failure: right character and scene, practised place instead of the requested one ("Momo hiding from the rain near the little pond" -> Momo hid "in the tall grass").
TinyStories-Instruct sample ("A little dog finds a lost ball in the park and gives it back to a girl."): kept the park and the ball, the dog never appears.
Before tuning, v2 given a Momo request wrote about Zuri ("Words can hurt, Zuri") - the Lesson 04 bleed.
Generator stories are still recited template sentences.

Artifacts

tiny-storyteller-instruct/: model.pt, config.json (vocab 4098), tokenizer.json, tokenizer_config.json, chat_template.json. 23 MB.
pairs/: train.jsonl (4,000), heldout.jsonl (150), test50.jsonl, familiar50.jsonl, crossworld20.jsonl. 3.1 MB. TinyStories-Instruct rows are derived from a CDLA-Sharing-1.0 dataset.
story_pairs.py: the pair generator. results.json: all numbers and every generated test story.
Masking widget checked in headless Chromium at 390 px and 700 px: no console errors, no horizontal scroll.
Grading fixes in the final run: "signs" instead of "cards" for the special tokens, completion vs following named, example cell prints only the instruction, commented login line, taller token box on narrow screens.

What went wrong / limits

The TinyStories-Instruct files are not separate files in roneneldan/TinyStories; used the authors' roneneldan/TinyStoriesInstruct validation file instead (fast download).
One sample per instruction (seed fixed); 50 instructions is a small set, so each point is 2 percentage points.

Measured after the run (post-grading)

Old-quiz forgetting (2026-10-11, measure_old_quiz.py, Apple Silicon Mac, MPS): Lesson 04's quiz (a), the 200 held-out TinyStories (35,448 tokens), scored exactly as Lesson 04 did (stories joined with <|endoftext|>, back-to-back 256-token windows, every token once; Lesson 04's model class and perplexity code, unchanged). tiny-storyteller-v2 (start, sanity re-score) 11.75; tiny-storyteller-instruct (this lesson's model, local tiny-storyteller-instruct/) 12.85. No replay was mixed into instruction tuning (training was the 4,000 instruction pairs only). TinyStories rows read from the cached first parquet shard, the same first 20,000 rows the stream returns (token count and v2's 11.75 match Lesson 04 exactly).