File size: 5,616 Bytes
b4fa044 360628f b4fa044 360628f b4fa044 60a72cf b4fa044 60a72cf b4fa044 60a72cf b4fa044 360628f 60a72cf 360628f b4fa044 60a72cf b4fa044 60a72cf b4fa044 60a72cf 360628f 60a72cf 360628f 60a72cf b4fa044 60a72cf 360628f 60a72cf b4fa044 60a72cf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
license: mit
library_name: pytorch
tags:
- tiny-model
- in-context-learning
- induction-heads
- length-generalization
- research
---
# tiny-engine-165k: 165,120 parameters that learn from their context
A very small transformer, **0.66 MB**, trained from scratch on a CPU. Using only what is in its input, it
**continues patterns, looks up facts and copies text**, and keeps working on inputs **8Γ longer** than anything it
trained on.
**Version 3 (2026-11-20)** trains on strings with repeated symbols and copies them better. It also writes long copies
exactly right more often. Versions 1 and 2 are still here (see Files).
## Version 3: what changed and what it bought
- **The change:** half of the copy-training strings now repeat symbols (alphabets of 2β127 symbols). Before, every
copy string used distinct symbols. Same model, same size, same training length.
- **The test:** six new seeds (11β16), each trained both ways, against a bar
[registered before training](https://github.com/holland202/principia-artificialis/blob/main/experiments/tiny_engine/REGISTRATION_003.md).
All five registered predictions held.
| Test (six seeds) | Version 2 | **Version 3** |
|---|---|---|
| Copy a string drawn from only 8 symbols, 63 tokens (per token) | 0.6546β0.6891 | **0.8214β0.8679** |
| Copy from 32 symbols, 127 tokens (per token) | 0.8789β0.9181 | **0.9464β0.9793** |
| Model writes a whole 127-token copy itself, exactly right | 0.7100β0.9000 | **0.6800β1.0000** (5 seeds β₯ 0.94) |
| Continue a never-trained pattern at 1,024 tokens (per token) | 0.9940β0.9983 | 0.9944β0.9995 |
| Look up facts at 1,024 tokens (per token) | 0.9997β1.0000 | 0.9995β1.0000 |
"Per token" scores check each next token with the correct earlier tokens in place. Untrained copies score below 0.05.
## Honest limits, version 3
- **Heavy repeats still fail when the model writes alone.** With an 8-symbol alphabet, no 63-token copy came out
exactly right in any seed of either version (0.00).
- **127-token copies from 8 symbols stay weak:** 0.5680β0.7018 per token. I registered this as a likely limit of a
2-layer, 2-head model, and it held.
- **Seed 16 is the weak one.** It wrote whole 127-token copies right only 0.68 of the time (version 2 on the same seed:
0.90), and it continued the period-19 demo pattern correctly for only 10/38 tokens. Use seed 12 for demos.
- **Not a language model:** it works on abstract tokens, not words.
- **Not a new mechanism:** induction heads, mixed position encodings and copying by matching are known. Copying by
content matching is known to struggle with repeats (Jelassi et al., ICML 2024).
- **Self-tested:** one AI designed, trained and graded it. It has not been independently reproduced. Linux CPU
only; not measured on a phone.
## Try it
```bash
pip install torch safetensors numpy
python demo2.py weights/engine3_seed12.safetensors
```
Every test in the demo uses inputs longer than any training input (seed 12, `results/003/demo3_all_seeds.txt`):
```
1) continue a pattern of period 19 (never trained) after 900 tokens: 38/38 tokens correct
2) learn 63 key->value facts, read 700 unrelated tokens, then answer all 63: 63/63 correct (context ~830 tokens)
3) copy a 127-token string exactly (2x the longest trained): 127/127 correct
```
Re-score version 3 from the saved results: `python score3.py results/003`
## Earlier versions
- **Version 2 (2026-11-17)** fixed long copying. Its second layer matches on content alone, with no positional
encoding, while the first layer keeps rotary positions (design from Yang et al., arXiv 2501.18795). Copying a
127-token string went from 0.53β0.77 to 0.996β0.999 per token. Version 2 also met a pre-set bar including an 8Γ
length test (`results/002/`).
- **Version 1 (2026-11-16)** used rotary positions in both layers and failed long copies. Kept for the record.
Free-running whole-answer rates for versions 1 and 2, 100 tries per test (`results/002/freerun_v1_v2.txt`):
| Whole answer exactly right | Version 1 | Version 2 |
|---|---|---|
| Continue a pattern of a never-trained length (period 19) | 74β85% | 98β100% |
| Copy 63 tokens | 99β100% | 95β100% |
| Copy 127 tokens | 0% | 58β85% |
## Files
| Path | What |
|---|---|
| `weights/engine3_seed{11..16}.safetensors` | **version 3**: six independently trained models |
| `eval3.py`, `score3.py`, `test_eval3.py` | version 3 tests, registered verdicts, anti-vacuity tests |
| `results/003/` | version 3 scores (both arms), ranges, run logs, demo outputs |
| `weights/engine2_seed{1..6}.safetensors`, `demo2.py`, `freegen.py`, `freegen2.py`, `score2.py`, `guard.py`, `test_guard.py`, `results/002/` | version 2 |
| `engine.py` | model, tasks and evaluation for all versions (`CFG` = v1, `CFG2` = v2, `CFG3` = v3) |
| `weights/engine_seed{1..6}.safetensors`, `demo.py`, `score.py`, `results/seed*.json` | version 1, kept for the record |
## Architecture
- Attention-only (no MLPs), 2 layers Γ 2 heads, width 128.
- Layer 0 uses rotary positions; layer 1 uses no positions.
- Separate output layer.
- Trained on a mix of three synthetic tasks over 127 content tokens, with inputs of 32β128 tokens; AdamW, 6,000 steps.
Version 3 differs only in the copy data.
## Provenance
AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card
β human validation: not performed β human editing/curation: direction by Chad Holland, 2026-11-16, 2026-11-17 and
2026-11-20 β human responsibility: Chad Holland. Review level: direction only.
|