tiny-engine-165k: 165K-param in-context engine
Browse files- README.md +76 -0
- SHA256SUMS +18 -0
- demo.py +42 -0
- engine.py +175 -0
- results/demo_seed1.txt +9 -0
- results/score.txt +10 -0
- results/seed1.json +1 -0
- results/seed2.json +1 -0
- results/seed3.json +1 -0
- results/seed4.json +1 -0
- results/seed5.json +1 -0
- results/seed6.json +1 -0
- score.py +13 -0
- weights/engine_seed1.safetensors +3 -0
- weights/engine_seed2.safetensors +3 -0
- weights/engine_seed3.safetensors +3 -0
- weights/engine_seed4.safetensors +3 -0
- weights/engine_seed5.safetensors +3 -0
- weights/engine_seed6.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- tiny-model
|
| 6 |
+
- in-context-learning
|
| 7 |
+
- induction-heads
|
| 8 |
+
- rotary-embeddings
|
| 9 |
+
- research
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# tiny-engine-165k: 165,120 parameters that learn from their context
|
| 13 |
+
|
| 14 |
+
This is a very small transformer, **0.66 MB**, trained from scratch on a laptop-class CPU in about 6 minutes. It
|
| 15 |
+
does three different things using only what is in its input:
|
| 16 |
+
|
| 17 |
+
| Skill | At trained lengths | Beyond trained lengths |
|
| 18 |
+
|---|---|---|
|
| 19 |
+
| **Continue a repeating pattern**, including pattern lengths it never saw | 0.998–0.999 | 0.967–0.995 at **4× longer** inputs |
|
| 20 |
+
| **Look up a value** from key–value pairs given earlier | 1.000 | 0.948–0.972 at 2×, 0.825–0.883 at 4× |
|
| 21 |
+
| **Copy a string exactly** | 1.000 (up to 63 tokens) | 0.53–0.77 for 127-token strings: **the known weak spot** |
|
| 22 |
+
|
| 23 |
+
These are results across 6 independently trained seeds, against a pass bar
|
| 24 |
+
[written before training](https://github.com/holland202/principia-artificialis/blob/exp/tiny-engine-001/experiments/tiny_engine/REGISTRATION.md).
|
| 25 |
+
An untrained copy of the model scores about 0.01 on every test.
|
| 26 |
+
|
| 27 |
+
**Result: 2 of 3 registered criteria passed. Long-string copying failed and is reported, not hidden.**
|
| 28 |
+
|
| 29 |
+
## Try it
|
| 30 |
+
|
| 31 |
+
```bash
|
| 32 |
+
pip install torch safetensors numpy
|
| 33 |
+
python demo.py weights/engine_seed1.safetensors
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
Output:
|
| 37 |
+
|
| 38 |
+
```
|
| 39 |
+
1) continue a pattern of period 11 (never trained), after 300 tokens of context -> OK
|
| 40 |
+
2) look up 40 values from 40 key-value pairs given earlier: 40/40 correct
|
| 41 |
+
3) copy a 50-token string exactly: 50/50 tokens correct
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
Seed 1 is the best of the six on long copying; the other five weights are included.
|
| 45 |
+
|
| 46 |
+
## How it is built
|
| 47 |
+
|
| 48 |
+
- **Architecture:** attention-only (no MLPs), 2 layers × 2 heads, width 128.
|
| 49 |
+
- **Positions:** rotary (relative) positions, and no position embeddings. That is why it keeps working on longer
|
| 50 |
+
inputs.
|
| 51 |
+
- **Output layer:** separate (untied). Tying it to the input embedding stopped learning completely in our tests.
|
| 52 |
+
- **Training:** a mix of three synthetic tasks over 128 content tokens; inputs of 32–128 tokens; AdamW at learning
|
| 53 |
+
rate 1e-3, 6,000 steps.
|
| 54 |
+
|
| 55 |
+
## Honest limits
|
| 56 |
+
|
| 57 |
+
- **Not a language model:** it works on abstract tokens, not text.
|
| 58 |
+
- **Not a new mechanism:** induction heads, and relative positions helping them generalize, are known results
|
| 59 |
+
(e.g. arXiv 2412.11459). This is a small, open, fully tested working instance of them.
|
| 60 |
+
- **Self-tested:** one AI designed, trained and graded it. It has not been independently reproduced.
|
| 61 |
+
|
| 62 |
+
## Files
|
| 63 |
+
|
| 64 |
+
| Path | What |
|
| 65 |
+
|---|---|
|
| 66 |
+
| `weights/engine_seed{1..6}.safetensors` | six independently trained models |
|
| 67 |
+
| `engine.py` | model, task generators, evaluation |
|
| 68 |
+
| `demo.py` | a quick hands-on test |
|
| 69 |
+
| `score.py` | applies the registered pass bar |
|
| 70 |
+
| `results/` | per-seed scores and the demo output |
|
| 71 |
+
|
| 72 |
+
## Provenance
|
| 73 |
+
|
| 74 |
+
AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card
|
| 75 |
+
→ human validation: not performed → human editing/curation: direction by Chad Holland, 2026-11-16 → human
|
| 76 |
+
responsibility: Chad Holland. Review level: direction only.
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
404f35ef3263ba31edc854045874af89c93422f7b155a04b2f4d97c71e794476 README.md
|
| 2 |
+
de7234b9d2e9c74ecc1de2b3f4689767a79611a8f10f8d6b27a532d05095dbde demo.py
|
| 3 |
+
d701a90d63782126330c0d31d51687682bf2f89492765c6b67b5ee40f01d5576 engine.py
|
| 4 |
+
f1993d41bb622359f1f216ce4f9e68fc796e6126029f7be0a741f1311a35d7c6 results/demo_seed1.txt
|
| 5 |
+
d730bb1797d44fc918c85a80ed1b8a15174afb3020ed8d5925dde10be506c508 results/score.txt
|
| 6 |
+
928b9adedf63e99ac1a694ccff803a1271e7da78ec87fba9570351b4924ac086 results/seed1.json
|
| 7 |
+
19eef19335e90dab8aa5f9c1abcc400d7481ad6a549db3d90ab2cfa3ca50d1b2 results/seed2.json
|
| 8 |
+
665207b56a75f09d50aa825e92222b4e533f1f30347487c4e8cada95d24ca81d results/seed3.json
|
| 9 |
+
097f9bd0f2455a8ebdf4df6d09e80c511fbc50231c7a2b49c2a10c2c51543f84 results/seed4.json
|
| 10 |
+
78c5a3d3293f2d48dcf34c82d7d75a8f7160ba778d7701d92dfb84bfc6da364f results/seed5.json
|
| 11 |
+
9cea52984ae85af459546771a64a2c8a6da79dfd32c5748f74fde248d73e9bc3 results/seed6.json
|
| 12 |
+
a2ba58e0bcb3e068749110b0d7474794a5e316173a0cdf61ebcd01d69899489d score.py
|
| 13 |
+
5d39dc8785bf021082de26864edf8012c5bbfd9c8f068f6efc88018f759586c3 weights/engine_seed1.safetensors
|
| 14 |
+
f19fdf2a24ae3040fd6056708985483a8ae4ed3780b713f9d8546dabe8dbbf83 weights/engine_seed2.safetensors
|
| 15 |
+
085b36fa301668a88edf3e241d5310e2e86f49b205be6d0304a1edf4ad22ccae weights/engine_seed3.safetensors
|
| 16 |
+
8e6827a6f1d26906723327cfc2c0e61cf8ebd0ed32ebb947ebb14b2e90ce7833 weights/engine_seed4.safetensors
|
| 17 |
+
9d0c81404b735e9581a4264688f46f7cf2afb62b4a671bd1c8528e80451e8fa6 weights/engine_seed5.safetensors
|
| 18 |
+
fe800006a98d8fd2f71fc6c72f72170e2d72feb0e74d6e6c0c9430413626dcdd weights/engine_seed6.safetensors
|
demo.py
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Try the tiny in-context engine. Usage: python3 demo.py weights/engine_seed1.safetensors
|
| 2 |
+
Shows the three skills on fresh random inputs, including inputs longer than any it trained on."""
|
| 3 |
+
import sys
|
| 4 |
+
|
| 5 |
+
import numpy as np
|
| 6 |
+
import torch
|
| 7 |
+
from safetensors.torch import load_file
|
| 8 |
+
|
| 9 |
+
import engine as E
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def greedy(m, prefix, n):
|
| 13 |
+
x = torch.tensor([prefix])
|
| 14 |
+
with torch.no_grad():
|
| 15 |
+
for _ in range(n):
|
| 16 |
+
x = torch.cat([x, m(x)[:, -1:].argmax(-1)], 1)
|
| 17 |
+
return x[0, len(prefix):].tolist()
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
m = E.TinyEngine()
|
| 21 |
+
m.load_state_dict(load_file(sys.argv[1] if len(sys.argv) > 1 else "weights/engine_seed1.safetensors"))
|
| 22 |
+
m.eval()
|
| 23 |
+
rng = np.random.default_rng(2026)
|
| 24 |
+
print(f"TinyEngine, {E.n_params(m):,} parameters\n")
|
| 25 |
+
|
| 26 |
+
seg = rng.integers(1, 128, size=11).tolist() # period 11 was never trained on
|
| 27 |
+
prefix = [E.BOS] + (seg * 30)[:300] # 300 tokens: longer than any training input
|
| 28 |
+
out = greedy(m, prefix, 22)
|
| 29 |
+
want = (seg * 40)[300:322]
|
| 30 |
+
print("1) continue a pattern of period 11 (never trained), after 300 tokens of context")
|
| 31 |
+
print(" model :", out); print(" truth :", want, "->", "OK" if out == want else "DIFF")
|
| 32 |
+
|
| 33 |
+
keys = rng.choice(np.arange(1, 64), 40, replace=False); vals = rng.choice(np.arange(64, 128), 40)
|
| 34 |
+
ctx = [E.BOS] + [int(t) for kv in zip(keys, vals) for t in kv]
|
| 35 |
+
hits = 0
|
| 36 |
+
for j in range(40):
|
| 37 |
+
hits += greedy(m, ctx + [E.QRY, int(keys[j])], 1)[0] == int(vals[j])
|
| 38 |
+
print(f"\n2) look up 40 values from 40 key-value pairs given earlier: {hits}/40 correct")
|
| 39 |
+
|
| 40 |
+
s = rng.choice(np.arange(1, 128), 50, replace=False).tolist()
|
| 41 |
+
out = greedy(m, [E.BOS] + s + [E.SEP], 50)
|
| 42 |
+
print(f"\n3) copy a 50-token string exactly: {sum(a == b for a, b in zip(out, s))}/50 tokens correct")
|
engine.py
ADDED
|
@@ -0,0 +1,175 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tiny in-context engine: an attention-only transformer with rotary (relative) positions, trained from scratch on a
|
| 2 |
+
mix of in-context tasks, then tested on lengths it never saw. See REGISTRATION.md.
|
| 3 |
+
Written by Claude (Anthropic, Opus 5.5) at Chad Holland's direction, 2026-11-16. SELF-TESTED only."""
|
| 4 |
+
import math
|
| 5 |
+
|
| 6 |
+
import numpy as np
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn as nn
|
| 9 |
+
import torch.nn.functional as F
|
| 10 |
+
|
| 11 |
+
N_CONTENT = 127 # content tokens 1..127
|
| 12 |
+
BOS, SEP, QRY = 0, 128, 129
|
| 13 |
+
VOCAB = 130
|
| 14 |
+
CFG = dict(d_model=128, n_heads=2, n_layers=2, train_lens=(32, 64, 96, 128), batch=64, lr=1e-3, steps=6000)
|
| 15 |
+
PERIODS_SEEN = tuple(L for L in range(4, 25) if L not in (7, 11, 15, 19, 23))
|
| 16 |
+
PERIODS_HELD = (7, 11, 15, 19, 23)
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
# ---------------------------------------------------------------- tasks: (tokens, loss mask), mask marks positions whose
|
| 20 |
+
# NEXT token is determined by the context, so the score is only over learnable predictions.
|
| 21 |
+
def task_periodic(rng, T, periods):
|
| 22 |
+
L = int(rng.choice(periods))
|
| 23 |
+
seg = rng.integers(1, N_CONTENT + 1, size=L)
|
| 24 |
+
x = np.concatenate([[BOS], np.resize(seg, T - 1)])
|
| 25 |
+
m = np.zeros(T, bool)
|
| 26 |
+
m[L:T - 1] = True # predicting x[t+1] from position t, for t+1 > L
|
| 27 |
+
return x, m
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
KEYS = np.arange(1, 64) # keys and values come from disjoint halves of the vocabulary, so a key token can
|
| 31 |
+
VALS = np.arange(64, N_CONTENT + 1) # never also appear as a value (that made the first version of this task ambiguous)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def task_recall(rng, T):
|
| 35 |
+
"""BOS k1 v1 ... kn vn, then queries 'QRY k v' about earlier pairs, filling length T."""
|
| 36 |
+
n = min(len(KEYS), max(2, (T - 1) * 2 // 5 // 2 * 2 // 2))
|
| 37 |
+
nq = (T - 1 - 2 * n) // 3
|
| 38 |
+
while nq < 1:
|
| 39 |
+
n -= 1
|
| 40 |
+
nq = (T - 1 - 2 * n) // 3
|
| 41 |
+
keys = rng.choice(KEYS, size=n, replace=False)
|
| 42 |
+
vals = rng.choice(VALS, size=n)
|
| 43 |
+
x = [BOS] + [t for kv in zip(keys, vals) for t in kv]
|
| 44 |
+
m = [False] * len(x)
|
| 45 |
+
for j in rng.integers(0, n, size=nq):
|
| 46 |
+
x += [QRY, keys[j], vals[j]]
|
| 47 |
+
m += [False, True, False] # position of the key predicts the value
|
| 48 |
+
pad = T - len(x)
|
| 49 |
+
x += list(rng.integers(1, N_CONTENT + 1, size=pad))
|
| 50 |
+
m += [False] * pad
|
| 51 |
+
return np.array(x), np.array(m)
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def task_copy(rng, T):
|
| 55 |
+
"""BOS s SEP s: s has distinct tokens (so copying is unambiguous); length fills T."""
|
| 56 |
+
k = min((T - 2) // 2, N_CONTENT)
|
| 57 |
+
s = rng.choice(np.arange(1, N_CONTENT + 1), size=k, replace=False)
|
| 58 |
+
x = np.concatenate([[BOS], s, [SEP], s])
|
| 59 |
+
m = np.zeros(len(x), bool)
|
| 60 |
+
m[k + 1:len(x) - 1] = True # from SEP onward, each next token is the copy
|
| 61 |
+
pad = T - len(x)
|
| 62 |
+
x = np.concatenate([x, rng.integers(1, N_CONTENT + 1, size=pad)])
|
| 63 |
+
m = np.concatenate([m, np.zeros(pad, bool)])
|
| 64 |
+
return x, m
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
TASKS = ("periodic", "recall", "copy")
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def make_batch(rng, n, T, task=None, periods=PERIODS_SEEN):
|
| 71 |
+
xs, ms = [], []
|
| 72 |
+
for _ in range(n):
|
| 73 |
+
t = task or TASKS[int(rng.integers(len(TASKS)))]
|
| 74 |
+
x, m = (task_periodic(rng, T, periods) if t == "periodic" else task_recall(rng, T) if t == "recall"
|
| 75 |
+
else task_copy(rng, T))
|
| 76 |
+
xs.append(x); ms.append(m)
|
| 77 |
+
return torch.tensor(np.array(xs)).long(), torch.tensor(np.array(ms))
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
# ---------------------------------------------------------------- model
|
| 81 |
+
def rope(x, base=10000.0):
|
| 82 |
+
"""Rotary position embedding on the last dim of (b, h, t, d)."""
|
| 83 |
+
t, d = x.shape[-2], x.shape[-1]
|
| 84 |
+
inv = 1.0 / base ** (torch.arange(0, d, 2, dtype=torch.float32) / d)
|
| 85 |
+
ang = torch.arange(t, dtype=torch.float32)[:, None] * inv[None]
|
| 86 |
+
cos, sin = ang.cos(), ang.sin()
|
| 87 |
+
x1, x2 = x[..., 0::2], x[..., 1::2]
|
| 88 |
+
return torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1).flatten(-2)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
class Attn(nn.Module):
|
| 92 |
+
def __init__(self, d, h, use_rope=True):
|
| 93 |
+
super().__init__()
|
| 94 |
+
self.h, self.dh, self.use_rope = h, d // h, use_rope
|
| 95 |
+
self.q, self.k, self.v, self.o = (nn.Linear(d, d, bias=False) for _ in range(4))
|
| 96 |
+
|
| 97 |
+
def forward(self, x):
|
| 98 |
+
b, t, d = x.shape
|
| 99 |
+
sh = lambda z: z.view(b, t, self.h, self.dh).transpose(1, 2)
|
| 100 |
+
q, k = sh(self.q(x)), sh(self.k(x))
|
| 101 |
+
if self.use_rope:
|
| 102 |
+
q, k = rope(q), rope(k)
|
| 103 |
+
y = F.scaled_dot_product_attention(q, k, sh(self.v(x)), is_causal=True)
|
| 104 |
+
return self.o(y.transpose(1, 2).reshape(b, t, d))
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
class TinyEngine(nn.Module):
|
| 108 |
+
"""Token embedding (no position embedding: positions enter only through rotary attention), attention-only
|
| 109 |
+
pre-LN blocks, separate (untied) unembedding. Tying the unembedding to the embedding stopped learning entirely
|
| 110 |
+
in exploration (0.00 periodic accuracy after 1,500 steps at d_model 256; untied reached 1.00)."""
|
| 111 |
+
|
| 112 |
+
def __init__(self, cfg=CFG):
|
| 113 |
+
super().__init__()
|
| 114 |
+
d = cfg["d_model"]
|
| 115 |
+
self.pos_kind = cfg.get("pos", "rope")
|
| 116 |
+
self.emb = nn.Embedding(VOCAB, d)
|
| 117 |
+
self.pos = nn.Embedding(1024, d) if self.pos_kind == "learned" else None
|
| 118 |
+
self.lns = nn.ModuleList(nn.LayerNorm(d) for _ in range(cfg["n_layers"]))
|
| 119 |
+
self.attn = nn.ModuleList(Attn(d, cfg["n_heads"], self.pos_kind == "rope") for _ in range(cfg["n_layers"]))
|
| 120 |
+
self.lnf = nn.LayerNorm(d)
|
| 121 |
+
self.head = None if cfg.get("tied", False) else nn.Linear(d, VOCAB, bias=False)
|
| 122 |
+
|
| 123 |
+
def forward(self, x):
|
| 124 |
+
h = self.emb(x)
|
| 125 |
+
if self.pos is not None:
|
| 126 |
+
h = h + self.pos(torch.arange(x.shape[1]))
|
| 127 |
+
for ln, at in zip(self.lns, self.attn):
|
| 128 |
+
h = h + at(ln(h))
|
| 129 |
+
h = self.lnf(h)
|
| 130 |
+
return self.head(h) if self.head is not None else h @ self.emb.weight.T
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def n_params(m):
|
| 134 |
+
return sum(p.numel() for p in m.parameters())
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
# ---------------------------------------------------------------- evaluation
|
| 138 |
+
@torch.no_grad()
|
| 139 |
+
def accuracy(m, task, T, seed, n=256, periods=PERIODS_SEEN):
|
| 140 |
+
x, mask = make_batch(np.random.default_rng(seed), n, T, task, periods)
|
| 141 |
+
pred = m(x).argmax(-1)[:, :-1]
|
| 142 |
+
ok = (pred == x[:, 1:])[mask[:, :-1]]
|
| 143 |
+
return float(ok.float().mean())
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
EVAL_LENS = (128, 256, 512) # 128 = longest trained; 256 and 512 = 2x and 4x beyond it
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def evaluate(m):
|
| 150 |
+
m.eval()
|
| 151 |
+
r = {}
|
| 152 |
+
for L in EVAL_LENS:
|
| 153 |
+
n = 256 if L <= 128 else 96
|
| 154 |
+
r[f"periodic_heldout_L{L}"] = accuracy(m, "periodic", L, 100 + L, n, periods=PERIODS_HELD)
|
| 155 |
+
r[f"recall_L{L}"] = accuracy(m, "recall", L, 200 + L, n)
|
| 156 |
+
r[f"copy_L{L}"] = accuracy(m, "copy", L, 300 + L, n)
|
| 157 |
+
r["periodic_seen_L128"] = accuracy(m, "periodic", 128, 99)
|
| 158 |
+
m.train()
|
| 159 |
+
return r
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
def train(seed, cfg=CFG, log=None):
|
| 163 |
+
torch.manual_seed(seed)
|
| 164 |
+
torch.use_deterministic_algorithms(True)
|
| 165 |
+
rng = np.random.default_rng(seed)
|
| 166 |
+
m = TinyEngine(cfg)
|
| 167 |
+
opt = torch.optim.AdamW(m.parameters(), lr=cfg["lr"], weight_decay=0.0)
|
| 168 |
+
for step in range(1, cfg["steps"] + 1):
|
| 169 |
+
x, mask = make_batch(rng, cfg["batch"], int(rng.choice(cfg["train_lens"])))
|
| 170 |
+
logits = m(x)[:, :-1]
|
| 171 |
+
loss = F.cross_entropy(logits[mask[:, :-1]], x[:, 1:][mask[:, :-1]])
|
| 172 |
+
opt.zero_grad(); loss.backward(); opt.step()
|
| 173 |
+
if log and step % 500 == 0:
|
| 174 |
+
log(step, float(loss.detach()))
|
| 175 |
+
return m
|
results/demo_seed1.txt
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
TinyEngine, 165,120 parameters
|
| 2 |
+
|
| 3 |
+
1) continue a pattern of period 11 (never trained), after 300 tokens of context
|
| 4 |
+
model : [82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4, 82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4]
|
| 5 |
+
truth : [82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4, 82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4] -> OK
|
| 6 |
+
|
| 7 |
+
2) look up 40 values from 40 key-value pairs given earlier: 40/40 correct
|
| 8 |
+
|
| 9 |
+
3) copy a 50-token string exactly: 50/50 tokens correct
|
results/score.txt
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
seed periodic_heldout_L128 recall_L128 copy_L128 periodic_heldout_L256 recall_L256 copy_L256 periodic_heldout_L512
|
| 2 |
+
1 0.998 1.000 1.000 0.999 0.972 0.767 0.985
|
| 3 |
+
2 0.998 1.000 1.000 0.999 0.948 0.576 0.975
|
| 4 |
+
3 0.998 1.000 1.000 0.999 0.954 0.563 0.995
|
| 5 |
+
4 0.998 1.000 1.000 0.998 0.953 0.529 0.967
|
| 6 |
+
5 0.998 1.000 1.000 0.997 0.961 0.625 0.988
|
| 7 |
+
6 0.999 1.000 1.000 0.998 0.959 0.538 0.985
|
| 8 |
+
Q1: HELD (6/6 seeds meet every threshold; need 5)
|
| 9 |
+
Q2: FAILED (1/6 seeds meet every threshold; need 5)
|
| 10 |
+
Q3: HELD (6/6 seeds meet every threshold; need 5)
|
results/seed1.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"seed": 1, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9984326958656311, "recall_L128": 1.0, "copy_L128": 1.0, "periodic_heldout_L256": 0.9987830519676208, "recall_L256": 0.9720180034637451, "copy_L256": 0.7667322754859924, "periodic_heldout_L512": 0.985174298286438, "recall_L512": 0.8795573115348816, "copy_L512": 0.7662401795387268, "periodic_seen_L128": 0.9990339279174805}, "train_seconds": 353}
|
results/seed2.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"seed": 2, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9983630776405334, "recall_L128": 0.9998437762260437, "copy_L128": 0.9999380111694336, "periodic_heldout_L256": 0.9990438222885132, "recall_L256": 0.9483251571655273, "copy_L256": 0.5763615369796753, "periodic_heldout_L512": 0.9746325016021729, "recall_L512": 0.868408203125, "copy_L512": 0.5822670459747314, "periodic_seen_L128": 0.9986889362335205}, "train_seconds": 344}
|
results/seed3.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"seed": 3, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9981192350387573, "recall_L128": 0.9996874928474426, "copy_L128": 1.0, "periodic_heldout_L256": 0.9985222816467285, "recall_L256": 0.953839898109436, "copy_L256": 0.5629921555519104, "periodic_heldout_L512": 0.9947711229324341, "recall_L512": 0.8627116084098816, "copy_L512": 0.5631561875343323, "periodic_seen_L128": 0.9989304542541504}, "train_seconds": 375}
|
results/seed4.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"seed": 4, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9981889128684998, "recall_L128": 0.9996874928474426, "copy_L128": 0.9996899962425232, "periodic_heldout_L256": 0.9980006814002991, "recall_L256": 0.9526143670082092, "copy_L256": 0.5291174650192261, "periodic_heldout_L512": 0.9666106700897217, "recall_L512": 0.8251953125, "copy_L512": 0.5330544710159302, "periodic_seen_L128": 0.9980334043502808}, "train_seconds": 360}
|
results/seed5.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"seed": 5, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9979799389839172, "recall_L128": 1.0, "copy_L128": 0.9999380111694336, "periodic_heldout_L256": 0.997001051902771, "recall_L256": 0.961192786693573, "copy_L256": 0.6246719360351562, "periodic_heldout_L512": 0.988156259059906, "recall_L512": 0.8828938603401184, "copy_L512": 0.6314796805381775, "periodic_seen_L128": 0.9984129071235657}, "train_seconds": 349}
|
results/seed6.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"seed": 6, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9987461566925049, "recall_L128": 1.0, "copy_L128": 1.0, "periodic_heldout_L256": 0.997783362865448, "recall_L256": 0.9585375785827637, "copy_L256": 0.538385808467865, "periodic_heldout_L512": 0.9846702814102173, "recall_L512": 0.861572265625, "copy_L512": 0.5410104990005493, "periodic_seen_L128": 0.9986889362335205}, "train_seconds": 340}
|
score.py
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Apply REGISTRATION.md's bar to the per-seed results. Usage: python3 score.py RESULTS_DIR"""
|
| 2 |
+
import json, sys
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
R = [json.loads(p.read_text()) for p in sorted(Path(sys.argv[1]).glob("seed*.json"), key=lambda p: int(p.stem[4:]))]
|
| 5 |
+
BAR = {"Q1": [("periodic_heldout_L128", .95), ("recall_L128", .95), ("copy_L128", .95)],
|
| 6 |
+
"Q2": [("periodic_heldout_L256", .95), ("recall_L256", .90), ("copy_L256", .70)],
|
| 7 |
+
"Q3": [("periodic_heldout_L512", .90)]}
|
| 8 |
+
print("seed " + " ".join(k for q in BAR.values() for k, _ in q))
|
| 9 |
+
for r in R:
|
| 10 |
+
print(f"{r['seed']:>4} " + " ".join(f"{r['scores'][k]:.3f}" for q in BAR.values() for k, _ in q))
|
| 11 |
+
for q, rules in BAR.items():
|
| 12 |
+
ok = sum(all(r["scores"][k] >= t for k, t in rules) for r in R)
|
| 13 |
+
print(f"{q}: {'HELD' if ok >= 5 else 'FAILED'} ({ok}/{len(R)} seeds meet every threshold; need 5)")
|
weights/engine_seed1.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5d39dc8785bf021082de26864edf8012c5bbfd9c8f068f6efc88018f759586c3
|
| 3 |
+
size 661752
|
weights/engine_seed2.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f19fdf2a24ae3040fd6056708985483a8ae4ed3780b713f9d8546dabe8dbbf83
|
| 3 |
+
size 661752
|
weights/engine_seed3.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:085b36fa301668a88edf3e241d5310e2e86f49b205be6d0304a1edf4ad22ccae
|
| 3 |
+
size 661752
|
weights/engine_seed4.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8e6827a6f1d26906723327cfc2c0e61cf8ebd0ed32ebb947ebb14b2e90ce7833
|
| 3 |
+
size 661752
|
weights/engine_seed5.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9d0c81404b735e9581a4264688f46f7cf2afb62b4a671bd1c8528e80451e8fa6
|
| 3 |
+
size 661752
|
weights/engine_seed6.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fe800006a98d8fd2f71fc6c72f72170e2d72feb0e74d6e6c0c9430413626dcdd
|
| 3 |
+
size 661752
|