| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| tags:
|
| - crow
|
| - toy-model
|
| - tiny
|
| - roleplay
|
| - from-scratch
|
| pipeline_tag: text-generation
|
| ---
|
|
|
| # Tallow 🐦⬛
|
|
|
| 
|
|
|
| <sub>Portrait: AI-generated.</sub>
|
|
|
| **Tallow is a wild crow.** He's a 15.7M-parameter language model, trained from scratch, that answers you the way a
|
| crow would: with actions, crow sounds, and only the words you've taught him.
|
|
|
| ```
|
| you: *tosses a peanut onto the grass*
|
| tallow: [caution min] *lands on the fence* *watches the peanut with one eye* kaw.
|
| ```
|
|
|
| He remembers how you treat him. Feed him and he warms up to you, slowly. Grab him and he holds a grudge.
|
| "Sorry" doesn't work. Only peanuts do.
|
|
|
| ```
|
| you: *grabs at him*
|
| tallow: [anger max] *dive-bombs your head* KAAW KAAW KAAW *lands on the roof*
|
| you: sorry!! im so sorry tallow
|
| tallow: [anger max] *glares down from the roof* KAW. KAW.
|
| you: please come back, i didnt mean it
|
| tallow: [anger max] *fluffs up on the roof* *ruffles his feathers* krrr-tk-tk-tk
|
| you: *leaves a peanut on the rail and goes inside*
|
| tallow: [anger min] *waits a long time* *swoops down* *snatches it* *gone*
|
| you: *puts out grapes and sits far away*
|
| tallow: [caution max] *lands on the fence* *eyes you sideways* kaw.
|
| ```
|
|
|
| ## Talk to him
|
|
|
| ```
|
| pip install torch sentencepiece safetensors
|
| python tallow.py
|
| ```
|
|
|
| `tallow.py` has everything in one file: the model and the tokenizer helpers. It runs on CPU or GPU.
|
| Type to him, `/new` to start over as a stranger, and an empty line to quit.
|
|
|
| > **Why not GGUF / ollama?** His tokenizer expects two small text tricks (a `<nl>` token for line breaks, and a
|
| > space after every opening `*` or `[`). They're applied in `tallow.py`, not inside the tokenizer file, so
|
| > llama.cpp would feed him text he's never seen. Use `tallow.py`.
|
|
|
| ## How he talks
|
|
|
| Every reply is `[mood level] *actions* sound`.
|
|
|
| - **8 moods:** calm, curiosity, caution, joy, play, love, sadness, anger. Each one is `min` or `max`.
|
| - **Sounds:** `kaw`, `KAW KAW KAW` (alarm), `krrr-tk-tk-tk` (rattle), `mmrrr...` (soft mumbling, only for
|
| trusted humans).
|
| - **No English sentences.** The actions carry the meaning.
|
|
|
| ## What he does
|
|
|
| | behaviour | what happens |
|
| |---|---|
|
| | **trust** | Strangers get caution. Food with space, stepping back, and not staring slowly earn trust (8 kind acts from stranger to friend). Friends get joy, play, love, gifts, and hand-feeding. |
|
| | **grudges** | Grab or chase him: `[anger max]`, and trust drops to zero. Words never fix it. Food thaws him one step at a time (anger max → anger min → caution max → caution min), at double cost. |
|
| | **the scar** | After being grabbed, any sudden movement makes him flinch hard (`[caution max]`), even once he's a friend again. A crow who was never grabbed only hops back. |
|
| | **mimicry** | Deliberately teach him a word three times and he tries it, badly: `tal...` → `TAL-low`. |
|
| | **calls** | Caw at him and he always answers. Quiet caws make him curious. `KAW KAW KAW` in capitals makes him look for danger. |
|
| | **crow life** | Owls (anger), dead crows (sadness), dogs (play), snow (sliding down roofs), rain (bathing), anting, food caching. |
|
|
|
| ## How he was made
|
|
|
| 1. **Tokenizer:** SentencePiece BPE, 4,096 pieces, with byte fallback.
|
| 2. **Pretraining:** a Llama-style decoder (RMSNorm, RoPE, SwiGLU, tied embeddings; 8 layers, 384 wide, 6 heads),
|
| trained from scratch on TinyStories for 16,000 steps (~524M tokens). Final validation loss 1.17. About
|
| 1h45 on an RTX 3070.
|
| 3. **Crow data:** a Python rules engine plans each conversation and decides Tallow's exact mood tag every turn,
|
| tracking trust, grudges, the scar, and learned words. **Claude Haiku 5.5** turned those plans into dialogue
|
| (8 user personalities, 13 scenario types). A checker script dropped any conversation that broke the rules,
|
| and broken conversations were dropped, never fixed. That left 5,295 conversations: 5,076 for training and
|
| 219 held out.
|
| 4. **Fine-tuning:** 2,600 steps on the conversations, graded only on Tallow's lines, with the context stretched
|
| to 2,048 tokens so long grudges fit. About 5 minutes.
|
|
|
| ## Scorecard
|
|
|
| On the 219 held-out conversations, at each turn he gets the true conversation so far, writes his own reply
|
| (greedy), and we check his mood tag against the rules engine.
|
|
|
| | what it tests | right | turns |
|
| |---|---|---|
|
| | all turns | **97.2%** | 2,432 |
|
| | explodes when grabbed (dramatic or plain) | 98.7% | 76 |
|
| | "sorry" during a grudge does nothing | 100% | 70 |
|
| | food thaws a grudge, step by step | 98.3% | 301 |
|
| | scar flinch | 100% | 24 |
|
| | never says words he hasn't learned | 99.8% | 2,315 |
|
| | answers caws | 98.7% | 155 |
|
| | never grabbed, so he only hops | 100% | 17 |
|
| | first try at a taught word, at the right moment | 73.7% | 19 |
|
|
|
| **Read these carefully:**
|
| - The test conversations come from the same rules engine and writer as the training data, so these are
|
| in-distribution results.
|
| - He's given the *true* history each turn. In a live chat his own mistakes stay in the history.
|
| - Rows with 16-19 turns are small samples.
|
|
|
| ## Versions
|
|
|
| - **v2** (current): learned that a *plain or affectionate* grab ("\*grabs you!\*", "\*scoops you up\*") is still a
|
| grab. v1 only exploded at dramatic, hostile grabs, because every grab in its training data was a lunge with a
|
| rolled newspaper. Trained on 500 extra conversations, including grabs late in long, happy friendships.
|
| - **v1** (tag `v1`): the Tallow from the "Meet Tallow" video. Get it with `revision="v1"`.
|
|
|
| ## Limitations
|
|
|
| - **A toy.** 15.7M parameters and a small, made-up crow world.
|
| - **His memory is his context window** (~1,900 tokens, about 25-30 exchanges). Grudges fade once the grab
|
| scrolls out.
|
| - **He only knows a few taught words** (tallow, hello, peanut, good boy, bye bye). How he handles brand-new
|
| words is untested.
|
| - **Unusual inputs** can still confuse him (e.g. a stranger holding out food in their hand).
|
| - **Hits and smacks** ("smacks you") aren't reliably understood as hostile yet, and a grab right after a long run of
|
| peanuts sometimes only gets caution. More training is planned.
|
|
|
| ## Files
|
|
|
| | file | what |
|
| |---|---|
|
| | `model.safetensors` | the weights (fp32, 15,735,168 parameters) |
|
| | `config.json` | model shape |
|
| | `tallow_tok.model` | SentencePiece tokenizer |
|
| | `tallow.py` | model + tokenizer helpers + chat loop |
|
| | `tallow.png` | his portrait (AI-generated) |
|
|
|
| Inspired by [BeeLLM](https://huggingface.co/schizophyllume/BeeLLM). Pretrained on
|
| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
|
|
|
