stories15M-kansai

A 15M-parameter Llama that writes short children's stories in Kansai dialect (関西弁), narration and dialogue alike: exactly the shape of karpathy/tinyllamas' stories15M.bin (dim=288, n_layers=6, n_heads=6, hidden_dim=768, vocab_size=32000, max_seq_len=256, tied classifier), with a Kansai vocabulary of its own. Made for transformer-dojo, whose participants implement this model's forward pass themselves: swapping in these weights and this tokenizer makes their finished code tell stories in Kansai dialect.

むかしむかし、リリーっちゅう名前の小っちゃな女の子がおった。彼女は、お気に入りのきれいなドレスを着るんが大好きやった。ある日、彼女はお友達と遊ぶために公園へ行った。

遊んでる時、リリーは地面にキラキラ光る宝石を見つけよった。彼女はそれを拾い上げて、お友達に見せた。みんな、めっちゃきれいやなぁと思うた。

すると突然、お友達のジャックが、うっかりその宝石を落としてしもた。宝石は地面に落ちて、バラバラに壊れてしもたんや。

リリーは、大切な宝石をなくしてしもたさかい、えらい悲しんだ。彼女は、もう二度と見つけられへんのとちゃうか、と、何度も何度も泣いた。

(greedy, INT4, prompt 「むかしむかし、」, told by a participant's code through transformer-dojo's kernel)

Files

File	What it is	sha256
model.bin	fp32 weights, llama2.c's v0 checkpoint format (what run.c reads)	82e175e89dc74120c2c91c711a1a50354dc623425d27bccd9c8a5468289d0286
tokenizer.bin	the vocabulary in llama2.c's tokenizer.bin format	71e65910d3a1a89eac8d6d68b79b90894fc25276dde976d4d1c8e6dbddb04e03
tokenizer.model	the same vocabulary as a sentencepiece model	ab78bfd667fc3b0a89f24a4b721c65594a3082984484083665a4347ca6516d31
train_config.json, log-*.jsonl	how the model was made: the vocabulary transplant, then each training stage's configuration and evaluation log

model.bin and tokenizer.bin run unchanged in llama2.c: ./run model.bin -z tokenizer.bin -i "むかしむかし、". transformer-dojo quantizes model.bin to its INT4 format itself (dojo-dev quantize; the INT4 file is not published here) and reads the vocabulary as tokenizer.json (dojo-dev export-tokenizer tokenizer.bin tokenizer.json).

How it was made

1. Data: opsbr/TinyStories-Kansai — TinyStories-JA rewritten into Kansai dialect story by story, 2,087,086 train stories kept by its filters.
2. Vocabulary: sentencepiece BPE, 32,000 pieces, byte fallback, trained on the first 200,000 kept Kansai stories, in llama2.c's layout. Kansai forms are single pieces (「言うた」「めっちゃ」 「へんかった」「さかい」「しもた」): a story is 117 tokens at the median against 158 under stories15M-ja's vocabulary, so a 200-token continuation holds a whole story.
3. Transplant: opsbr/stories15M-ja moved onto the Kansai vocabulary. Every tensor but the embedding (the classifier, tied) is copied; the embedding rows of the 19,752 pieces both vocabularies share are copied, and each of the 11,989 new pieces starts as the mean of the stories15M-ja rows its text splits into.
4. Fine-tune: two epochs over the corpus (668M tokens), lr 1e-3, AdamW (β₂ 0.95, weight decay 0.1), linear warmup 200 steps then cosine decay to 0, batch 128 × 4 × 256 tokens, bf16.
5. Quantization-aware fine-tune: a quarter epoch at lr 1e-4 with the forward pass through transformer-dojo's INT4 (group size 32, f16 scales, straight-through gradients), so the weights are tuned for the INT4 the participants' kernel runs.

One RTX 5090: about 20 minutes of training, after about 27 hours of converting the corpus.

Evaluation

On the 40 held-out validation stories (the Kansai rewrites of the same 40 stories stories15M-ja is gated on), under transformer-dojo's INT4:

• Held-out NLL rises −0.72% from fp32 to INT4 (2.211 → 2.195; the gate allows +20%), and no greedy continuation of the 20 gate prompts repeats a 4-gram four times running.
• NLL per character 0.673 under INT4, on text common to every vocabulary compared: the lowest of the variants that passed (stories15M-ja fine-tuned without changing its vocabulary: 0.675; stories15M-ja itself on Kansai text: 2.660).
• Narration in です・ます: none of 20 continuations (stories15M-ja: 99%).

Limits

A model this small forgets its own plot and can repeat a sentence it has already said; about a quarter to half of greedy continuations repeat one, as stories15M-ja's do. When it loops, it tends to loop on 「めっちゃ」, the corpus's word for "very". The Kansai is the corpus's: machine-made from a machine translation, common Osaka-flavoured Kansai told as a grown-up telling a story aloud, with some standard Japanese left in dialogue (「言った」, 「〜わね」). It knows children's-story vocabulary only.

Licence and provenance

• Weights and tokenizer: MIT.
• Data: opsbr/TinyStories-Kansai, CDLA-Sharing-1.0 — TinyStories (Ronen Eldan, Yuanzhi Li; arXiv:2305.07759), translated to Japanese by Naoki Shibata (shibatch/TinyStories-JA), rewritten into Kansai dialect by google/gemma-4-26B-A4B-it (Apache-2.0). A trained model is a "Result" of the data (§1.11), and §3.5: "This Agreement imposes no obligations or restrictions on Your Use or Publication of Results."
• Starting point: opsbr/stories15M-ja, MIT.
• Code: transformer-dojo's tools/train/, trimmed from llama2.c by Andrej Karpathy, MIT.