Upload README.md with huggingface_hub

README.md

@@ -44,7 +44,7 @@ every dead end, is documented at
 | Hidden size | 384 |
 | Layers | 6 |
 | Attention heads / KV groups | 6 / 3 |
-| Context length | **512** |
+| Context length | **2048** |
 | Vocabulary | 32,000 (byte-level BPE, trained for this model) |
 | Weights dtype | bfloat16 |
 | Training hardware | One Tenstorrent Blackhole chip (`mesh_shape [1, 1]`) |
@@ -60,56 +60,100 @@ a verbatim copy of tt-train's own `nanollama3.yaml`.
 
 | | |
 |---|---|
-| Corpus | Nine-source, licence-audited blend — TinyStories, Simple English Wikipedia, and seven curated Project Gutenberg slices (see [`docs/corpus_blend.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/corpus_blend.md)). 399,594,747 tokens emitted against a 400,000,000-token budget. The blend recipe — source registry, fetch/prepare/measure/blend scripts, and the provenance manifest — is published separately as [`episod/tt-tnt-corpus`](https://huggingface.co/datasets/episod/tt-tnt-corpus) |
-| Tokens seen | **353,495,970** — the full training split, **one epoch** |
-| Steps | 10,787 at batch 64, sequence length 512 |
-| Wall clock | ~58 minutes on a single Blackhole p300c |
-| Final train loss | 3.3125 |
-| **Final validation loss** | **4.2203** |
+| Corpus | Nine-source, licence-audited blend — TinyStories, Simple English Wikipedia, and seven curated Project Gutenberg slices (see [`docs/corpus_blend.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/corpus_blend.md)). **This is the first tt-tnt checkpoint trained on a blend that carries document separators (`</s>`)** — earlier revisions contained none at all, so no previous checkpoint had ever seen an end-of-document token and none could stop generating. Measured on the token arrays directly: the pre-fix array holds **zero** id-2 tokens across the whole training split; this one averages one per ~478 tokens (one per ~210 in the short-document sources, one per ~80,000 in the book-length ones). The blend recipe — source registry, fetch/prepare/measure/blend scripts, and the provenance manifest — is published separately as [`episod/tt-tnt-corpus`](https://huggingface.co/datasets/episod/tt-tnt-corpus) |
+| Tokens seen | **352,714,752** — the full training split, **one epoch** |
+| Steps | 10,764 at batch 16, sequence length 2048 (32,768 tokens/step, unchanged from the previous run's 64×512) |
+| Wall clock | ~91 minutes on a single Blackhole p300c |
+| Final train loss | 3.25 |
+| **Final validation loss** | **2.9937** — the end-of-run figure from `train/run.py`'s `evaluate()`. The periodic curve's last entry (`artifacts/checkpoints-tt-tnt-v3/val_losses.jsonl`, step 10,764) reads 2.939; the two differ only by which held-out windows each sampled. **Do not read this as an improvement on the previous checkpoint's 4.2203** — see below |
 | Optimizer | AdamW, constant lr 3e-4, weight decay 0.01, `stochastic_rounding: true` |
 
 ## Limitations — please read these
 
-**It has seen its training corpus once, not memorized it.** At batch 64, sequence length 512,
-10,787 steps is one epoch over the blend's 353,495,970-token training split — more data and a
-longer context than the original TinyStories-only checkpoint (which saw 0.43 of an epoch), but
-still a single pass. The validation curve
-(`artifacts/checkpoints-tt-tnt-v1/val_losses.jsonl`) falls from 6.7125 at step 500 to about
-4.29 by step 10,000, then **plateaus** for the remainder of the run — oscillating between 4.29
-and 4.38 for the last ~2,300 steps, including a rise to 4.378125 at step 9000. Read plainly: it
-stopped improving well before training ended; this is not a curve that would keep falling given
-more steps at the same settings.
-
-**The corpus is now a nine-source, licence-audited blend, and the model's behavior is a mix to
+**The headline validation loss is not comparable to the previous checkpoint's.** 2.9937 against
+4.2203 looks like a large gain, and most of it is not one. The previous checkpoint's validation
+split was the tail 10% of a token stream whose sources are concatenated in sorted-name order,
+so it landed **entirely inside `wikipedia_simple`** — the most out-of-domain source in the
+blend, and by per-source measurement the second-hardest (held-out loss 4.28, against
+TinyStories' 1.83; see
+[`docs/measurements/per-source-loss-tt-tnt-v1.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/per-source-loss-tt-tnt-v1.md)).
+That number measured domain transfer, not learning. This run's split is **stratified by
+source** — a proportional tail from each of the nine — so it is a fair sample of the training
+mixture, and a much easier one, because 31% of the mixture is TinyStories. A meaningful share
+of the drop from 4.22 to 2.99 is the yardstick changing, not the model improving. The two
+numbers should not be subtracted.
+
+**It has seen its training corpus once, not memorized it.** At batch 16, sequence length 2048,
+10,764 steps is one epoch over the blend's 352.7M-token training split — the same 32,768
+tokens per step as the previous run, at four times the context. The validation curve
+(`artifacts/checkpoints-tt-tnt-v3/val_losses.jsonl`) falls from 5.084 at step 500 to ~3.28 by
+step 4,000, and then keeps drifting down slowly and noisily — the last ~2,300 steps oscillate
+between 2.87 and 3.10 against ~3.12 around step 7,000–8,000. That is a real if modest continued
+decline, unlike the previous run, which was flat for its final stretch. Read plainly: this one
+had not clearly stopped improving when it ended, but the per-step gain over the last third is
+small enough that more steps at these settings would not transform it.
+
+**It can now stop, which no previous checkpoint could.** Every earlier tt-tnt checkpoint was
+trained on a corpus containing **zero** `</s>` tokens while its `config.json` nonetheless
+declared `eos_token_id: 2` — so generation could never terminate naturally and always ran to
+whatever token limit the caller set. This is the first checkpoint trained on a corpus that
+marks document boundaries, and it does terminate: on the frozen 15-prompt evaluation set at
+128 max new tokens, **5/15 completions end on `</s>` under greedy decoding** (median 48 tokens)
+and **11/30 under sampling** at temperature 0.8 / top_p 0.95 (median 91). The comparable
+numbers for the previous checkpoint are **0/15 and 0/30** — not "rarely", but never, by
+construction. This matters most for chained generation, where you want a passage to end rather
+than be cut off mid-sentence at the limit.
+
+It is a partial fix, not a solved problem: two thirds of completions still run to the limit,
+and the model is much readier to stop on short-document material (TinyStories-like prompts)
+than on the book-length sources, which is what the corpus taught it — separator density in the
+blend ranges from one per ~210 tokens in the short sources to one per ~80,000 in the books.
+
+**The corpus is a nine-source, licence-audited blend, and the model's behavior is a mix to
 match.** Read against the frozen evaluation set
-([`docs/measurements/samples-tt-tnt-v1.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/samples-tt-tnt-v1.md),
+([`docs/measurements/samples-tt-tnt-v3.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/samples-tt-tnt-v3.md),
 greedy decoding, 15 prompts): the model sometimes engages with a prompt's own material — sticks,
-roses, a procession, bees — where a TinyStories-only baseline would default to a generic moral,
-and one long-form prompt stays coherent for its full 60 tokens. But **TinyStories still
-dominates**: four of the fifteen prompts collapse into "a little girl named Lily" regardless of
-what the prompt was actually about, and several others degenerate into hard repetition loops
-under greedy decoding (e.g. "the procession was ready... the procession was ready..."). The
-oblique, observational voice this blend targets — closer to Fabre's insect notebooks than to a
-children's story — is **not** present in this checkpoint. Treat the promising examples as
-evidence of what the blend can nudge toward, not as evidence the voice has arrived; the
-Lily-collapses and the repetition loops are the more representative outcome.
+roses, a procession, bees — where a TinyStories-only baseline would default to a generic moral.
+But **TinyStories still dominates**, and under greedy decoding several prompts still degenerate
+into hard repetition loops ("the rose is a rose, and the rose is a rose, and..."; "the bees were
+busy, and the bees were busy"). The oblique, observational voice this blend targets — closer to
+Fabre's insect notebooks than to a children's story — is **not** present in this checkpoint.
+Under sampling the output is markedly better behaved
+([`docs/measurements/samples-tt-tnt-v3-t0.8.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/samples-tt-tnt-v3-t0.8.md)),
+which is the honest way to read the greedy loops: greedy decoding on a 22M-parameter model
+manufactures repetition attractors that sampling largely avoids. Treat the promising examples
+as evidence of what the blend can nudge toward, not as evidence the voice has arrived.
 
 **It is a base completion model.** No instruction tuning, no chat template. Give it the opening
 of a simple story; do not ask it questions.
 
-**Its context is 512 tokens** (doubled from the original 256-token checkpoint — see Lineage).
+**Its context is 2048 tokens** (256 → 512 → 2048 across the three checkpoints — see Lineage).
 Note that `tokenizer_config.json` carries the conventional `model_max_length` sentinel
 (~1e18). Do not derive a serving length from it — use `max_position_embeddings`. Serving this
-model with a 4k context will silently degrade output.
+model with a 4k context will silently degrade output; serving it at 512 silently discards three
+quarters of the context it was trained to use.
+
+**Whether it actually uses all 2048 tokens is a separate question from whether it was trained
+at 2048.** A position-wise loss probe on the *previous* checkpoint found per-token loss flat
+from position ~64 onward — that model extracted nothing from distant context, because the
+corpus had no document boundaries and so distant context genuinely was unpredictable. Fixing
+the separators is what makes a longer window worth having, and is why this run raised it. The
+equivalent probe for this checkpoint
+([`docs/measurements/context-use-tt-tnt-v3.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/context-use-tt-tnt-v3.md))
+shows loss still falling well past where the old one went flat — 4.23 at positions [0,32),
+3.28 at [32,64), 2.94 at [64,128), 2.85 at [256,512) — but the improvement past ~256 tokens is
+about 0.02 nats per bucket against a standard error of ~0.08, i.e. **directionally right and
+inside the noise**. The honest claim is that the long window is no longer actively useless, not
+that all 2048 tokens are earning their keep.
 
 **Unlike the original checkpoint, this run's RMSNorm layers did learn.** The very first tt-tnt
 checkpoint (see Lineage) trained with `stochastic_rounding` disabled, which silently froze all
 13 RMSNorm gammas at bfloat16's rounding fixed-point of 1.0 — the gradients were real, but every
 update rounded back to 1.0 and was discarded. This run set `stochastic_rounding: true`, and it
-worked: read directly from the final checkpoint (`tt_tnt_step00010787.pkl`), all 13 gammas have
-moved off 1.0 and are no longer degenerate (per-tensor means 0.86–1.70, per-tensor standard
-deviation 0.036–0.21, values spanning roughly 0.625–2.19 across the set). The model trained with
-its normalization layers genuinely live this time.
+worked: read directly from the published weights, all 13 gammas have moved off 1.0 and none is
+degenerate (per-tensor means 0.874–1.720, per-tensor standard deviation 0.034–0.207, values
+spanning roughly 0.691–2.359 across the set; zero tensors have standard deviation 0). The model
+trained with its normalization layers genuinely live.
 
 ## Verification
 
@@ -144,16 +188,23 @@ Runs on CPU; no Tenstorrent hardware required for inference.
 ## Sample output
 
 Greedy decoding, 60 new tokens, from the frozen evaluation set
-([`docs/measurements/samples-tt-tnt-v1.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/samples-tt-tnt-v1.md)):
+([`docs/measurements/samples-tt-tnt-v3.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/samples-tt-tnt-v3.md)):
 
-> The old woman kept bees behind the house, and every morning she **would go out and pick up
-> the honey and put it in her basket. One day, she was walking in the garden when she saw a
-> big, red apple. She was so excited and wanted to pick it. She picked it up and took a big
-> bite. It was so sweet and juicy!**
+> Once upon a time, there was a little **girl named Lily. She loved to play outside in the
+> park. One day, she saw a big, shiny rock on the ground. She picked it up and showed it to
+> her mom. "Look, Mommy! I found a shiny rock!" she said. Her mom smiled and said, "That**
 
-Chosen because it is the most coherent completion in the frozen set of 15, not because it is
-typical — see Limitations above and the linked file for the honest range, including four
-TinyStories collapses to "a little girl named Lily" and several hard repetition loops.
+And one that **stops on its own** rather than being cut off at the limit — the new behavior
+described in Limitations:
+
+> The ants had learned that being eaten was a way of **helping others. The moral of the story
+> is that it's important to be kind to others and to help others.**
+
+Both are near the top of the frozen set of 15, not typical of it — see Limitations above and
+the linked file for the honest range, including the TinyStories collapses to "a little girl
+named Lily" and several hard repetition loops. Sampled output at temperature 0.8 is in
+[`docs/measurements/samples-tt-tnt-v3-t0.8.md`](https://github.com/tsingletaryTT/tt-tnt/blob/main/docs/measurements/samples-tt-tnt-v3-t0.8.md)
+and is the more representative read of the model's range.
 
 ## Licensing and provenance
 
@@ -194,11 +245,21 @@ tt-train's own `nanollama3.yaml`, and every checkpoint (including the one this c
 is trained through `ttml` against it. What changed is the corpus and tokenizer this project
 now owns — a nine-source, licence-audited blend and a BPE tokenizer trained on that blend,
 rather than a single downloaded corpus and an inherited vocabulary — and that is what earned
-the new name, `tt-tnt`. (This is now the first `tt-tnt` checkpoint actually trained on that
-corpus: the training table above records a run over the nine-source blend at a 512-token
-context, not TinyStories alone at 256. An earlier TinyStories-only, 256-token checkpoint from
-before the corpus work existed under this same name for a time; this card now describes the
-blend-trained checkpoint that superseded it.)
+the new name, `tt-tnt`.
+
+Three checkpoints have now been published under this repo id, and it is worth being plain
+about which is which, because they differ only in weights and one config field:
+
+| | corpus | context | document separators |
+|---|---|---|---|
+| the original (TinyStories-only) | TinyStories alone | 256 | none |
+| the first blend-trained checkpoint | nine-source blend | 512 | **none** |
+| **this one** | nine-source blend, separator-carrying revision | **2048** | **yes** |
+
+The middle checkpoint was the first trained on the blend; this one is the first trained on a
+revision of that blend which marks where one document ends and the next begins, and the first
+whose declared `eos_token_id: 2` corresponds to a token the training data actually contained.
+That, rather than the context length, is the substantive difference.
 
 ## Origin