tiny-engine-165k / README.md
holland202's picture
Version 3 (TINY-ENGINE-003): repeated-symbol copy training
60a72cf verified
|
Raw History Blame Contribute Delete
5.62 kB
---
license: mit
library_name: pytorch
tags:
- tiny-model
- in-context-learning
- induction-heads
- length-generalization
- research
---
# tiny-engine-165k: 165,120 parameters that learn from their context
A very small transformer, **0.66 MB**, trained from scratch on a CPU. Using only what is in its input, it
**continues patterns, looks up facts and copies text**, and keeps working on inputs **8Γ— longer** than anything it
trained on.
**Version 3 (2026-11-20)** trains on strings with repeated symbols and copies them better. It also writes long copies
exactly right more often. Versions 1 and 2 are still here (see Files).
## Version 3: what changed and what it bought
- **The change:** half of the copy-training strings now repeat symbols (alphabets of 2–127 symbols). Before, every
copy string used distinct symbols. Same model, same size, same training length.
- **The test:** six new seeds (11–16), each trained both ways, against a bar
[registered before training](https://github.com/holland202/principia-artificialis/blob/main/experiments/tiny_engine/REGISTRATION_003.md).
All five registered predictions held.
| Test (six seeds) | Version 2 | **Version 3** |
|---|---|---|
| Copy a string drawn from only 8 symbols, 63 tokens (per token) | 0.6546–0.6891 | **0.8214–0.8679** |
| Copy from 32 symbols, 127 tokens (per token) | 0.8789–0.9181 | **0.9464–0.9793** |
| Model writes a whole 127-token copy itself, exactly right | 0.7100–0.9000 | **0.6800–1.0000** (5 seeds β‰₯ 0.94) |
| Continue a never-trained pattern at 1,024 tokens (per token) | 0.9940–0.9983 | 0.9944–0.9995 |
| Look up facts at 1,024 tokens (per token) | 0.9997–1.0000 | 0.9995–1.0000 |
"Per token" scores check each next token with the correct earlier tokens in place. Untrained copies score below 0.05.
## Honest limits, version 3
- **Heavy repeats still fail when the model writes alone.** With an 8-symbol alphabet, no 63-token copy came out
exactly right in any seed of either version (0.00).
- **127-token copies from 8 symbols stay weak:** 0.5680–0.7018 per token. I registered this as a likely limit of a
2-layer, 2-head model, and it held.
- **Seed 16 is the weak one.** It wrote whole 127-token copies right only 0.68 of the time (version 2 on the same seed:
0.90), and it continued the period-19 demo pattern correctly for only 10/38 tokens. Use seed 12 for demos.
- **Not a language model:** it works on abstract tokens, not words.
- **Not a new mechanism:** induction heads, mixed position encodings and copying by matching are known. Copying by
content matching is known to struggle with repeats (Jelassi et al., ICML 2024).
- **Self-tested:** one AI designed, trained and graded it. It has not been independently reproduced. Linux CPU
only; not measured on a phone.
## Try it
```bash
pip install torch safetensors numpy
python demo2.py weights/engine3_seed12.safetensors
```
Every test in the demo uses inputs longer than any training input (seed 12, `results/003/demo3_all_seeds.txt`):
```
1) continue a pattern of period 19 (never trained) after 900 tokens: 38/38 tokens correct
2) learn 63 key->value facts, read 700 unrelated tokens, then answer all 63: 63/63 correct (context ~830 tokens)
3) copy a 127-token string exactly (2x the longest trained): 127/127 correct
```
Re-score version 3 from the saved results: `python score3.py results/003`
## Earlier versions
- **Version 2 (2026-11-17)** fixed long copying. Its second layer matches on content alone, with no positional
encoding, while the first layer keeps rotary positions (design from Yang et al., arXiv 2501.18795). Copying a
127-token string went from 0.53–0.77 to 0.996–0.999 per token. Version 2 also met a pre-set bar including an 8Γ—
length test (`results/002/`).
- **Version 1 (2026-11-16)** used rotary positions in both layers and failed long copies. Kept for the record.
Free-running whole-answer rates for versions 1 and 2, 100 tries per test (`results/002/freerun_v1_v2.txt`):
| Whole answer exactly right | Version 1 | Version 2 |
|---|---|---|
| Continue a pattern of a never-trained length (period 19) | 74–85% | 98–100% |
| Copy 63 tokens | 99–100% | 95–100% |
| Copy 127 tokens | 0% | 58–85% |
## Files
| Path | What |
|---|---|
| `weights/engine3_seed{11..16}.safetensors` | **version 3**: six independently trained models |
| `eval3.py`, `score3.py`, `test_eval3.py` | version 3 tests, registered verdicts, anti-vacuity tests |
| `results/003/` | version 3 scores (both arms), ranges, run logs, demo outputs |
| `weights/engine2_seed{1..6}.safetensors`, `demo2.py`, `freegen.py`, `freegen2.py`, `score2.py`, `guard.py`, `test_guard.py`, `results/002/` | version 2 |
| `engine.py` | model, tasks and evaluation for all versions (`CFG` = v1, `CFG2` = v2, `CFG3` = v3) |
| `weights/engine_seed{1..6}.safetensors`, `demo.py`, `score.py`, `results/seed*.json` | version 1, kept for the record |
## Architecture
- Attention-only (no MLPs), 2 layers Γ— 2 heads, width 128.
- Layer 0 uses rotary positions; layer 1 uses no positions.
- Separate output layer.
- Trained on a mix of three synthetic tasks over 127 content tokens, with inputs of 32–128 tokens; AdamW, 6,000 steps.
Version 3 differs only in the copy data.
## Provenance
AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card
β†’ human validation: not performed β†’ human editing/curation: direction by Chad Holland, 2026-11-16, 2026-11-17 and
2026-11-20 β†’ human responsibility: Chad Holland. Review level: direction only.