tiny-engine-165k: 165,120 parameters that learn from their context
A very small transformer, 0.66 MB, trained from scratch on a CPU. Using only what is in its input, it continues patterns, looks up facts and copies text, and keeps working on inputs 8Γ longer than anything it trained on.
Version 3 (2026-11-20) trains on strings with repeated symbols and copies them better. It also writes long copies exactly right more often. Versions 1 and 2 are still here (see Files).
Version 3: what changed and what it bought
- The change: half of the copy-training strings now repeat symbols (alphabets of 2β127 symbols). Before, every copy string used distinct symbols. Same model, same size, same training length.
- The test: six new seeds (11β16), each trained both ways, against a bar registered before training. All five registered predictions held.
| Test (six seeds) | Version 2 | Version 3 |
|---|---|---|
| Copy a string drawn from only 8 symbols, 63 tokens (per token) | 0.6546β0.6891 | 0.8214β0.8679 |
| Copy from 32 symbols, 127 tokens (per token) | 0.8789β0.9181 | 0.9464β0.9793 |
| Model writes a whole 127-token copy itself, exactly right | 0.7100β0.9000 | 0.6800β1.0000 (5 seeds β₯ 0.94) |
| Continue a never-trained pattern at 1,024 tokens (per token) | 0.9940β0.9983 | 0.9944β0.9995 |
| Look up facts at 1,024 tokens (per token) | 0.9997β1.0000 | 0.9995β1.0000 |
"Per token" scores check each next token with the correct earlier tokens in place. Untrained copies score below 0.05.
Honest limits, version 3
- Heavy repeats still fail when the model writes alone. With an 8-symbol alphabet, no 63-token copy came out exactly right in any seed of either version (0.00).
- 127-token copies from 8 symbols stay weak: 0.5680β0.7018 per token. I registered this as a likely limit of a 2-layer, 2-head model, and it held.
- Seed 16 is the weak one. It wrote whole 127-token copies right only 0.68 of the time (version 2 on the same seed: 0.90), and it continued the period-19 demo pattern correctly for only 10/38 tokens. Use seed 12 for demos.
- Not a language model: it works on abstract tokens, not words.
- Not a new mechanism: induction heads, mixed position encodings and copying by matching are known. Copying by content matching is known to struggle with repeats (Jelassi et al., ICML 2024).
- Self-tested: one AI designed, trained and graded it. It has not been independently reproduced. Linux CPU only; not measured on a phone.
Try it
pip install torch safetensors numpy
python demo2.py weights/engine3_seed12.safetensors
Every test in the demo uses inputs longer than any training input (seed 12, results/003/demo3_all_seeds.txt):
1) continue a pattern of period 19 (never trained) after 900 tokens: 38/38 tokens correct
2) learn 63 key->value facts, read 700 unrelated tokens, then answer all 63: 63/63 correct (context ~830 tokens)
3) copy a 127-token string exactly (2x the longest trained): 127/127 correct
Re-score version 3 from the saved results: python score3.py results/003
Earlier versions
- Version 2 (2026-11-17) fixed long copying. Its second layer matches on content alone, with no positional
encoding, while the first layer keeps rotary positions (design from Yang et al., arXiv 2501.18795). Copying a
127-token string went from 0.53β0.77 to 0.996β0.999 per token. Version 2 also met a pre-set bar including an 8Γ
length test (
results/002/). - Version 1 (2026-11-16) used rotary positions in both layers and failed long copies. Kept for the record.
Free-running whole-answer rates for versions 1 and 2, 100 tries per test (results/002/freerun_v1_v2.txt):
| Whole answer exactly right | Version 1 | Version 2 |
|---|---|---|
| Continue a pattern of a never-trained length (period 19) | 74β85% | 98β100% |
| Copy 63 tokens | 99β100% | 95β100% |
| Copy 127 tokens | 0% | 58β85% |
Files
| Path | What |
|---|---|
weights/engine3_seed{11..16}.safetensors |
version 3: six independently trained models |
eval3.py, score3.py, test_eval3.py |
version 3 tests, registered verdicts, anti-vacuity tests |
results/003/ |
version 3 scores (both arms), ranges, run logs, demo outputs |
weights/engine2_seed{1..6}.safetensors, demo2.py, freegen.py, freegen2.py, score2.py, guard.py, test_guard.py, results/002/ |
version 2 |
engine.py |
model, tasks and evaluation for all versions (CFG = v1, CFG2 = v2, CFG3 = v3) |
weights/engine_seed{1..6}.safetensors, demo.py, score.py, results/seed*.json |
version 1, kept for the record |
Architecture
- Attention-only (no MLPs), 2 layers Γ 2 heads, width 128.
- Layer 0 uses rotary positions; layer 1 uses no positions.
- Separate output layer.
- Trained on a mix of three synthetic tasks over 127 content tokens, with inputs of 32β128 tokens; AdamW, 6,000 steps. Version 3 differs only in the copy data.
Provenance
AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card β human validation: not performed β human editing/curation: direction by Chad Holland, 2026-11-16, 2026-11-17 and 2026-11-20 β human responsibility: Chad Holland. Review level: direction only.