--- license: mit library_name: pytorch tags: - tiny-model - in-context-learning - induction-heads - length-generalization - research --- # tiny-engine-165k: 165,120 parameters that learn from their context A very small transformer, **0.66 MB**, trained from scratch on a CPU. Using only what is in its input, it **continues patterns, looks up facts and copies text**, and keeps working on inputs **8× longer** than anything it trained on. **Version 3 (2026-11-20)** trains on strings with repeated symbols and copies them better. It also writes long copies exactly right more often. Versions 1 and 2 are still here (see Files). ## Version 3: what changed and what it bought - **The change:** half of the copy-training strings now repeat symbols (alphabets of 2–127 symbols). Before, every copy string used distinct symbols. Same model, same size, same training length. - **The test:** six new seeds (11–16), each trained both ways, against a bar [registered before training](https://github.com/holland202/principia-artificialis/blob/main/experiments/tiny_engine/REGISTRATION_003.md). All five registered predictions held. | Test (six seeds) | Version 2 | **Version 3** | |---|---|---| | Copy a string drawn from only 8 symbols, 63 tokens (per token) | 0.6546–0.6891 | **0.8214–0.8679** | | Copy from 32 symbols, 127 tokens (per token) | 0.8789–0.9181 | **0.9464–0.9793** | | Model writes a whole 127-token copy itself, exactly right | 0.7100–0.9000 | **0.6800–1.0000** (5 seeds ≥ 0.94) | | Continue a never-trained pattern at 1,024 tokens (per token) | 0.9940–0.9983 | 0.9944–0.9995 | | Look up facts at 1,024 tokens (per token) | 0.9997–1.0000 | 0.9995–1.0000 | "Per token" scores check each next token with the correct earlier tokens in place. Untrained copies score below 0.05. ## Honest limits, version 3 - **Heavy repeats still fail when the model writes alone.** With an 8-symbol alphabet, no 63-token copy came out exactly right in any seed of either version (0.00). - **127-token copies from 8 symbols stay weak:** 0.5680–0.7018 per token. I registered this as a likely limit of a 2-layer, 2-head model, and it held. - **Seed 16 is the weak one.** It wrote whole 127-token copies right only 0.68 of the time (version 2 on the same seed: 0.90), and it continued the period-19 demo pattern correctly for only 10/38 tokens. Use seed 12 for demos. - **Not a language model:** it works on abstract tokens, not words. - **Not a new mechanism:** induction heads, mixed position encodings and copying by matching are known. Copying by content matching is known to struggle with repeats (Jelassi et al., ICML 2024). - **Self-tested:** one AI designed, trained and graded it. It has not been independently reproduced. Linux CPU only; not measured on a phone. ## Try it ```bash pip install torch safetensors numpy python demo2.py weights/engine3_seed12.safetensors ``` Every test in the demo uses inputs longer than any training input (seed 12, `results/003/demo3_all_seeds.txt`): ``` 1) continue a pattern of period 19 (never trained) after 900 tokens: 38/38 tokens correct 2) learn 63 key->value facts, read 700 unrelated tokens, then answer all 63: 63/63 correct (context ~830 tokens) 3) copy a 127-token string exactly (2x the longest trained): 127/127 correct ``` Re-score version 3 from the saved results: `python score3.py results/003` ## Earlier versions - **Version 2 (2026-11-17)** fixed long copying. Its second layer matches on content alone, with no positional encoding, while the first layer keeps rotary positions (design from Yang et al., arXiv 2501.18795). Copying a 127-token string went from 0.53–0.77 to 0.996–0.999 per token. Version 2 also met a pre-set bar including an 8× length test (`results/002/`). - **Version 1 (2026-11-16)** used rotary positions in both layers and failed long copies. Kept for the record. Free-running whole-answer rates for versions 1 and 2, 100 tries per test (`results/002/freerun_v1_v2.txt`): | Whole answer exactly right | Version 1 | Version 2 | |---|---|---| | Continue a pattern of a never-trained length (period 19) | 74–85% | 98–100% | | Copy 63 tokens | 99–100% | 95–100% | | Copy 127 tokens | 0% | 58–85% | ## Files | Path | What | |---|---| | `weights/engine3_seed{11..16}.safetensors` | **version 3**: six independently trained models | | `eval3.py`, `score3.py`, `test_eval3.py` | version 3 tests, registered verdicts, anti-vacuity tests | | `results/003/` | version 3 scores (both arms), ranges, run logs, demo outputs | | `weights/engine2_seed{1..6}.safetensors`, `demo2.py`, `freegen.py`, `freegen2.py`, `score2.py`, `guard.py`, `test_guard.py`, `results/002/` | version 2 | | `engine.py` | model, tasks and evaluation for all versions (`CFG` = v1, `CFG2` = v2, `CFG3` = v3) | | `weights/engine_seed{1..6}.safetensors`, `demo.py`, `score.py`, `results/seed*.json` | version 1, kept for the record | ## Architecture - Attention-only (no MLPs), 2 layers × 2 heads, width 128. - Layer 0 uses rotary positions; layer 1 uses no positions. - Separate output layer. - Trained on a mix of three synthetic tasks over 127 content tokens, with inputs of 32–128 tokens; AdamW, 6,000 steps. Version 3 differs only in the copy data. ## Provenance AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card → human validation: not performed → human editing/curation: direction by Chad Holland, 2026-11-16, 2026-11-17 and 2026-11-20 → human responsibility: Chad Holland. Review level: direction only.