tiny-engine-165k: 165,120 parameters that learn from their context

A very small transformer, 0.66 MB, trained from scratch on a CPU. Using only what is in its input, it continues patterns, looks up facts and copies text, and keeps working on inputs 8Γ— longer than anything it trained on.

Version 3 (2026-11-20) trains on strings with repeated symbols and copies them better. It also writes long copies exactly right more often. Versions 1 and 2 are still here (see Files).

Version 3: what changed and what it bought

  • The change: half of the copy-training strings now repeat symbols (alphabets of 2–127 symbols). Before, every copy string used distinct symbols. Same model, same size, same training length.
  • The test: six new seeds (11–16), each trained both ways, against a bar registered before training. All five registered predictions held.
Test (six seeds) Version 2 Version 3
Copy a string drawn from only 8 symbols, 63 tokens (per token) 0.6546–0.6891 0.8214–0.8679
Copy from 32 symbols, 127 tokens (per token) 0.8789–0.9181 0.9464–0.9793
Model writes a whole 127-token copy itself, exactly right 0.7100–0.9000 0.6800–1.0000 (5 seeds β‰₯ 0.94)
Continue a never-trained pattern at 1,024 tokens (per token) 0.9940–0.9983 0.9944–0.9995
Look up facts at 1,024 tokens (per token) 0.9997–1.0000 0.9995–1.0000

"Per token" scores check each next token with the correct earlier tokens in place. Untrained copies score below 0.05.

Honest limits, version 3

  • Heavy repeats still fail when the model writes alone. With an 8-symbol alphabet, no 63-token copy came out exactly right in any seed of either version (0.00).
  • 127-token copies from 8 symbols stay weak: 0.5680–0.7018 per token. I registered this as a likely limit of a 2-layer, 2-head model, and it held.
  • Seed 16 is the weak one. It wrote whole 127-token copies right only 0.68 of the time (version 2 on the same seed: 0.90), and it continued the period-19 demo pattern correctly for only 10/38 tokens. Use seed 12 for demos.
  • Not a language model: it works on abstract tokens, not words.
  • Not a new mechanism: induction heads, mixed position encodings and copying by matching are known. Copying by content matching is known to struggle with repeats (Jelassi et al., ICML 2024).
  • Self-tested: one AI designed, trained and graded it. It has not been independently reproduced. Linux CPU only; not measured on a phone.

Try it

pip install torch safetensors numpy
python demo2.py weights/engine3_seed12.safetensors

Every test in the demo uses inputs longer than any training input (seed 12, results/003/demo3_all_seeds.txt):

1) continue a pattern of period 19 (never trained) after 900 tokens: 38/38 tokens correct
2) learn 63 key->value facts, read 700 unrelated tokens, then answer all 63: 63/63 correct (context ~830 tokens)
3) copy a 127-token string exactly (2x the longest trained): 127/127 correct

Re-score version 3 from the saved results: python score3.py results/003

Earlier versions

  • Version 2 (2026-11-17) fixed long copying. Its second layer matches on content alone, with no positional encoding, while the first layer keeps rotary positions (design from Yang et al., arXiv 2501.18795). Copying a 127-token string went from 0.53–0.77 to 0.996–0.999 per token. Version 2 also met a pre-set bar including an 8Γ— length test (results/002/).
  • Version 1 (2026-11-16) used rotary positions in both layers and failed long copies. Kept for the record.

Free-running whole-answer rates for versions 1 and 2, 100 tries per test (results/002/freerun_v1_v2.txt):

Whole answer exactly right Version 1 Version 2
Continue a pattern of a never-trained length (period 19) 74–85% 98–100%
Copy 63 tokens 99–100% 95–100%
Copy 127 tokens 0% 58–85%

Files

Path What
weights/engine3_seed{11..16}.safetensors version 3: six independently trained models
eval3.py, score3.py, test_eval3.py version 3 tests, registered verdicts, anti-vacuity tests
results/003/ version 3 scores (both arms), ranges, run logs, demo outputs
weights/engine2_seed{1..6}.safetensors, demo2.py, freegen.py, freegen2.py, score2.py, guard.py, test_guard.py, results/002/ version 2
engine.py model, tasks and evaluation for all versions (CFG = v1, CFG2 = v2, CFG3 = v3)
weights/engine_seed{1..6}.safetensors, demo.py, score.py, results/seed*.json version 1, kept for the record

Architecture

  • Attention-only (no MLPs), 2 layers Γ— 2 heads, width 128.
  • Layer 0 uses rotary positions; layer 1 uses no positions.
  • Separate output layer.
  • Trained on a mix of three synthetic tasks over 127 content tokens, with inputs of 32–128 tokens; AdamW, 6,000 steps. Version 3 differs only in the copy data.

Provenance

AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card β†’ human validation: not performed β†’ human editing/curation: direction by Chad Holland, 2026-11-16, 2026-11-17 and 2026-11-20 β†’ human responsibility: Chad Holland. Review level: direction only.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Space using holland202/tiny-engine-165k 1