|
Download README.md from holland202/tiny-engine-165k: direct link, hf CLI and curl.
- Browser
- Download file 5.62 kB
-
https://huggingface.co/holland202/tiny-engine-165k/resolve/main/README.md
- Command line
-
hf download hf://holland202/tiny-engine-165k/README.md
-
curl -L -o README.md https://huggingface.co/holland202/tiny-engine-165k/resolve/main/README.md
5.62 kB
| license: mit | |
| library_name: pytorch | |
| tags: | |
| - tiny-model | |
| - in-context-learning | |
| - induction-heads | |
| - length-generalization | |
| - research | |
| # tiny-engine-165k: 165,120 parameters that learn from their context | |
| A very small transformer, **0.66 MB**, trained from scratch on a CPU. Using only what is in its input, it | |
| **continues patterns, looks up facts and copies text**, and keeps working on inputs **8Γ longer** than anything it | |
| trained on. | |
| **Version 3 (2026-11-20)** trains on strings with repeated symbols and copies them better. It also writes long copies | |
| exactly right more often. Versions 1 and 2 are still here (see Files). | |
| ## Version 3: what changed and what it bought | |
| - **The change:** half of the copy-training strings now repeat symbols (alphabets of 2β127 symbols). Before, every | |
| copy string used distinct symbols. Same model, same size, same training length. | |
| - **The test:** six new seeds (11β16), each trained both ways, against a bar | |
| [registered before training](https://github.com/holland202/principia-artificialis/blob/main/experiments/tiny_engine/REGISTRATION_003.md). | |
| All five registered predictions held. | |
| | Test (six seeds) | Version 2 | **Version 3** | | |
| |---|---|---| | |
| | Copy a string drawn from only 8 symbols, 63 tokens (per token) | 0.6546β0.6891 | **0.8214β0.8679** | | |
| | Copy from 32 symbols, 127 tokens (per token) | 0.8789β0.9181 | **0.9464β0.9793** | | |
| | Model writes a whole 127-token copy itself, exactly right | 0.7100β0.9000 | **0.6800β1.0000** (5 seeds β₯ 0.94) | | |
| | Continue a never-trained pattern at 1,024 tokens (per token) | 0.9940β0.9983 | 0.9944β0.9995 | | |
| | Look up facts at 1,024 tokens (per token) | 0.9997β1.0000 | 0.9995β1.0000 | | |
| "Per token" scores check each next token with the correct earlier tokens in place. Untrained copies score below 0.05. | |
| ## Honest limits, version 3 | |
| - **Heavy repeats still fail when the model writes alone.** With an 8-symbol alphabet, no 63-token copy came out | |
| exactly right in any seed of either version (0.00). | |
| - **127-token copies from 8 symbols stay weak:** 0.5680β0.7018 per token. I registered this as a likely limit of a | |
| 2-layer, 2-head model, and it held. | |
| - **Seed 16 is the weak one.** It wrote whole 127-token copies right only 0.68 of the time (version 2 on the same seed: | |
| 0.90), and it continued the period-19 demo pattern correctly for only 10/38 tokens. Use seed 12 for demos. | |
| - **Not a language model:** it works on abstract tokens, not words. | |
| - **Not a new mechanism:** induction heads, mixed position encodings and copying by matching are known. Copying by | |
| content matching is known to struggle with repeats (Jelassi et al., ICML 2024). | |
| - **Self-tested:** one AI designed, trained and graded it. It has not been independently reproduced. Linux CPU | |
| only; not measured on a phone. | |
| ## Try it | |
| ```bash | |
| pip install torch safetensors numpy | |
| python demo2.py weights/engine3_seed12.safetensors | |
| ``` | |
| Every test in the demo uses inputs longer than any training input (seed 12, `results/003/demo3_all_seeds.txt`): | |
| ``` | |
| 1) continue a pattern of period 19 (never trained) after 900 tokens: 38/38 tokens correct | |
| 2) learn 63 key->value facts, read 700 unrelated tokens, then answer all 63: 63/63 correct (context ~830 tokens) | |
| 3) copy a 127-token string exactly (2x the longest trained): 127/127 correct | |
| ``` | |
| Re-score version 3 from the saved results: `python score3.py results/003` | |
| ## Earlier versions | |
| - **Version 2 (2026-11-17)** fixed long copying. Its second layer matches on content alone, with no positional | |
| encoding, while the first layer keeps rotary positions (design from Yang et al., arXiv 2501.18795). Copying a | |
| 127-token string went from 0.53β0.77 to 0.996β0.999 per token. Version 2 also met a pre-set bar including an 8Γ | |
| length test (`results/002/`). | |
| - **Version 1 (2026-11-16)** used rotary positions in both layers and failed long copies. Kept for the record. | |
| Free-running whole-answer rates for versions 1 and 2, 100 tries per test (`results/002/freerun_v1_v2.txt`): | |
| | Whole answer exactly right | Version 1 | Version 2 | | |
| |---|---|---| | |
| | Continue a pattern of a never-trained length (period 19) | 74β85% | 98β100% | | |
| | Copy 63 tokens | 99β100% | 95β100% | | |
| | Copy 127 tokens | 0% | 58β85% | | |
| ## Files | |
| | Path | What | | |
| |---|---| | |
| | `weights/engine3_seed{11..16}.safetensors` | **version 3**: six independently trained models | | |
| | `eval3.py`, `score3.py`, `test_eval3.py` | version 3 tests, registered verdicts, anti-vacuity tests | | |
| | `results/003/` | version 3 scores (both arms), ranges, run logs, demo outputs | | |
| | `weights/engine2_seed{1..6}.safetensors`, `demo2.py`, `freegen.py`, `freegen2.py`, `score2.py`, `guard.py`, `test_guard.py`, `results/002/` | version 2 | | |
| | `engine.py` | model, tasks and evaluation for all versions (`CFG` = v1, `CFG2` = v2, `CFG3` = v3) | | |
| | `weights/engine_seed{1..6}.safetensors`, `demo.py`, `score.py`, `results/seed*.json` | version 1, kept for the record | | |
| ## Architecture | |
| - Attention-only (no MLPs), 2 layers Γ 2 heads, width 128. | |
| - Layer 0 uses rotary positions; layer 1 uses no positions. | |
| - Separate output layer. | |
| - Trained on a mix of three synthetic tasks over 127 content tokens, with inputs of 32β128 tokens; AdamW, 6,000 steps. | |
| Version 3 differs only in the copy data. | |
| ## Provenance | |
| AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card | |
| β human validation: not performed β human editing/curation: direction by Chad Holland, 2026-11-16, 2026-11-17 and | |
| 2026-11-20 β human responsibility: Chad Holland. Review level: direction only. | |