--- license: apache-2.0 language: - en tags: - crow - toy-model - tiny - roleplay - from-scratch pipeline_tag: text-generation --- # Tallow 🐦‍⬛ ![Tallow on his fence post at sunset, holding a peanut](tallow.png) Portrait: AI-generated. **Tallow is a wild crow.** He's a 15.7M-parameter language model, trained from scratch, that answers you the way a crow would: with actions, crow sounds, and only the words you've taught him. ``` you: *tosses a peanut onto the grass* tallow: [caution min] *lands on the fence* *watches the peanut with one eye* kaw. ``` He remembers how you treat him. Feed him and he warms up to you, slowly. Grab him and he holds a grudge. "Sorry" doesn't work. Only peanuts do. ``` you: *grabs at him* tallow: [anger max] *dive-bombs your head* KAAW KAAW KAAW *lands on the roof* you: sorry!! im so sorry tallow tallow: [anger max] *glares down from the roof* KAW. KAW. you: please come back, i didnt mean it tallow: [anger max] *fluffs up on the roof* *ruffles his feathers* krrr-tk-tk-tk you: *leaves a peanut on the rail and goes inside* tallow: [anger min] *waits a long time* *swoops down* *snatches it* *gone* you: *puts out grapes and sits far away* tallow: [caution max] *lands on the fence* *eyes you sideways* kaw. ``` ## Talk to him ``` pip install torch sentencepiece safetensors python tallow.py ``` `tallow.py` has everything in one file: the model and the tokenizer helpers. It runs on CPU or GPU. Type to him, `/new` to start over as a stranger, and an empty line to quit. > **Why not GGUF / ollama?** His tokenizer expects two small text tricks (a `` token for line breaks, and a > space after every opening `*` or `[`). They're applied in `tallow.py`, not inside the tokenizer file, so > llama.cpp would feed him text he's never seen. Use `tallow.py`. ## How he talks Every reply is `[mood level] *actions* sound`. - **8 moods:** calm, curiosity, caution, joy, play, love, sadness, anger. Each one is `min` or `max`. - **Sounds:** `kaw`, `KAW KAW KAW` (alarm), `krrr-tk-tk-tk` (rattle), `mmrrr...` (soft mumbling, only for trusted humans). - **No English sentences.** The actions carry the meaning. ## What he does | behaviour | what happens | |---|---| | **trust** | Strangers get caution. Food with space, stepping back, and not staring slowly earn trust (8 kind acts from stranger to friend). Friends get joy, play, love, gifts, and hand-feeding. | | **grudges** | Grab or chase him: `[anger max]`, and trust drops to zero. Words never fix it. Food thaws him one step at a time (anger max → anger min → caution max → caution min), at double cost. | | **the scar** | After being grabbed, any sudden movement makes him flinch hard (`[caution max]`), even once he's a friend again. A crow who was never grabbed only hops back. | | **mimicry** | Deliberately teach him a word three times and he tries it, badly: `tal...` → `TAL-low`. | | **calls** | Caw at him and he always answers. Quiet caws make him curious. `KAW KAW KAW` in capitals makes him look for danger. | | **crow life** | Owls (anger), dead crows (sadness), dogs (play), snow (sliding down roofs), rain (bathing), anting, food caching. | ## How he was made 1. **Tokenizer:** SentencePiece BPE, 4,096 pieces, with byte fallback. 2. **Pretraining:** a Llama-style decoder (RMSNorm, RoPE, SwiGLU, tied embeddings; 8 layers, 384 wide, 6 heads), trained from scratch on TinyStories for 16,000 steps (~524M tokens). Final validation loss 1.17. About 1h45 on an RTX 3070. 3. **Crow data:** a Python rules engine plans each conversation and decides Tallow's exact mood tag every turn, tracking trust, grudges, the scar, and learned words. **Claude Haiku 5.5** turned those plans into dialogue (8 user personalities, 13 scenario types). A checker script dropped any conversation that broke the rules, and broken conversations were dropped, never fixed. That left 5,295 conversations: 5,076 for training and 219 held out. 4. **Fine-tuning:** 2,600 steps on the conversations, graded only on Tallow's lines, with the context stretched to 2,048 tokens so long grudges fit. About 5 minutes. ## Scorecard On the 219 held-out conversations, at each turn he gets the true conversation so far, writes his own reply (greedy), and we check his mood tag against the rules engine. | what it tests | right | turns | |---|---|---| | all turns | **97.2%** | 2,432 | | explodes when grabbed (dramatic or plain) | 98.7% | 76 | | "sorry" during a grudge does nothing | 100% | 70 | | food thaws a grudge, step by step | 98.3% | 301 | | scar flinch | 100% | 24 | | never says words he hasn't learned | 99.8% | 2,315 | | answers caws | 98.7% | 155 | | never grabbed, so he only hops | 100% | 17 | | first try at a taught word, at the right moment | 73.7% | 19 | **Read these carefully:** - The test conversations come from the same rules engine and writer as the training data, so these are in-distribution results. - He's given the *true* history each turn. In a live chat his own mistakes stay in the history. - Rows with 16-19 turns are small samples. ## Versions - **v2** (current): learned that a *plain or affectionate* grab ("\*grabs you!\*", "\*scoops you up\*") is still a grab. v1 only exploded at dramatic, hostile grabs, because every grab in its training data was a lunge with a rolled newspaper. Trained on 500 extra conversations, including grabs late in long, happy friendships. - **v1** (tag `v1`): the Tallow from the "Meet Tallow" video. Get it with `revision="v1"`. ## Limitations - **A toy.** 15.7M parameters and a small, made-up crow world. - **His memory is his context window** (~1,900 tokens, about 25-30 exchanges). Grudges fade once the grab scrolls out. - **He only knows a few taught words** (tallow, hello, peanut, good boy, bye bye). How he handles brand-new words is untested. - **Unusual inputs** can still confuse him (e.g. a stranger holding out food in their hand). - **Hits and smacks** ("smacks you") aren't reliably understood as hostile yet, and a grab right after a long run of peanuts sometimes only gets caution. More training is planned. ## Files | file | what | |---|---| | `model.safetensors` | the weights (fp32, 15,735,168 parameters) | | `config.json` | model shape | | `tallow_tok.model` | SentencePiece tokenizer | | `tallow.py` | model + tokenizer helpers + chat loop | | `tallow.png` | his portrait (AI-generated) | Inspired by [BeeLLM](https://huggingface.co/schizophyllume/BeeLLM). Pretrained on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).