stories511K — 235,952 bytes
235,952 bytes (230.4 KiB), tokenizer included. stories511K is a 511,008-parameter English story continuation model. It was trained from scratch in FP32 on a GTX 1080, fine-tuned on complete TinyStories examples selected for a clear setup and payoff, then packed to INT4.
I wanted to see how small a model could get and still compete with Karpathy’s stories260K. On the same 256 held-out stories, this package used 77.8% fewer bytes and scored 1.31375 next-token loss versus 1.42403 for the original FP32 release. In a blind regrade it also followed the premise more often in the first two sentences. Getting to the end of the story was a different matter: only one of its 192 continuations had a full premise resolution confirmed by two readers.
| Model | Package bytes, tokenizer included | Held-out NLL ↓ | Early premise tracking ↑ | Full resolution, two readers |
|---|---|---|---|---|
| Karpathy stories260K FP32 | 1,064,185 | 1.42403 | 41/192 | 0/192 |
| Ordinary complete-story control INT4 | 235,980 | 1.32009 | 78/192 | 0/192 |
| stories511K Luna-filtered INT4 | 235,952 | 1.31375 | 99/192 | 1/192 |
The story panel used 48 fresh openings and four matched seeds per opening. Overall coherence was similar, and the test does not establish a reliable advantage on endings. The NLL figures use this project’s frozen split and scoring protocol.
Try it
Download the packed weights and tokenizer or fetch just the files needed to run it:
hf download millrace-ai/stories511K weights.int4.xz generate.py requirements.txt reference-code/experiments/stories260k-baseline/assets/upstream_model.py --local-dir stories511K
cd stories511K
python -m pip install -r requirements.txt
python generate.py --prompt "One day, a girl named Lily lost her red ball." --seed 133742
Pass an opening to get a continuation, or omit it to generate from the start-of-story token. This is a research model for simple English stories. It is not instruction tuned. The package uses a custom XZ format; the included script runs it on CPU by expanding weights to FP32. The byte count describes the package on disk, not its runtime RAM or the size of this whole repository.
Training and evidence
The final checkpoint starts from a 16,384-update FP32 base and adds one 200-update pass over 41,117 complete stories screened by GPT-6 Luna Medium. The ordinary complete-story arm saw the same number of tokens at the same example lengths. Both final arms use the same group-64 INT4 export. The scoring panel compared the reloaded packages, so the loss and story results reflect the weights in the download.
- Full training methodology: architecture, data, local training lineage, filtering rubric, matched control, INT4 packing, and evaluation.
- Reproducibility guide and artifact manifest: how to run the model and locate and verify evidence.
- Matched fine-tune results and blind regrade: scores, individual ratings, and the limits of the result.
- Optional checkpoints: filtered FP32 training state, 16,384-step FP32 parent, 8,192-step FP32 milestone, and ordinary INT4 control.
Training data: TinyStories, Eldan and Li. Architecture, tokenizer, and comparison model: Andrej Karpathy’s llama2.c and tinyllamas. The stories511K weights were independently initialized. See data and code notices for attribution and applicable terms.