holland202 commited on
Commit
b4fa044
·
verified ·
1 Parent(s): 521c78e

tiny-engine-165k: 165K-param in-context engine

Browse files
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: pytorch
4
+ tags:
5
+ - tiny-model
6
+ - in-context-learning
7
+ - induction-heads
8
+ - rotary-embeddings
9
+ - research
10
+ ---
11
+
12
+ # tiny-engine-165k: 165,120 parameters that learn from their context
13
+
14
+ This is a very small transformer, **0.66 MB**, trained from scratch on a laptop-class CPU in about 6 minutes. It
15
+ does three different things using only what is in its input:
16
+
17
+ | Skill | At trained lengths | Beyond trained lengths |
18
+ |---|---|---|
19
+ | **Continue a repeating pattern**, including pattern lengths it never saw | 0.998–0.999 | 0.967–0.995 at **4× longer** inputs |
20
+ | **Look up a value** from key–value pairs given earlier | 1.000 | 0.948–0.972 at 2×, 0.825–0.883 at 4× |
21
+ | **Copy a string exactly** | 1.000 (up to 63 tokens) | 0.53–0.77 for 127-token strings: **the known weak spot** |
22
+
23
+ These are results across 6 independently trained seeds, against a pass bar
24
+ [written before training](https://github.com/holland202/principia-artificialis/blob/exp/tiny-engine-001/experiments/tiny_engine/REGISTRATION.md).
25
+ An untrained copy of the model scores about 0.01 on every test.
26
+
27
+ **Result: 2 of 3 registered criteria passed. Long-string copying failed and is reported, not hidden.**
28
+
29
+ ## Try it
30
+
31
+ ```bash
32
+ pip install torch safetensors numpy
33
+ python demo.py weights/engine_seed1.safetensors
34
+ ```
35
+
36
+ Output:
37
+
38
+ ```
39
+ 1) continue a pattern of period 11 (never trained), after 300 tokens of context -> OK
40
+ 2) look up 40 values from 40 key-value pairs given earlier: 40/40 correct
41
+ 3) copy a 50-token string exactly: 50/50 tokens correct
42
+ ```
43
+
44
+ Seed 1 is the best of the six on long copying; the other five weights are included.
45
+
46
+ ## How it is built
47
+
48
+ - **Architecture:** attention-only (no MLPs), 2 layers × 2 heads, width 128.
49
+ - **Positions:** rotary (relative) positions, and no position embeddings. That is why it keeps working on longer
50
+ inputs.
51
+ - **Output layer:** separate (untied). Tying it to the input embedding stopped learning completely in our tests.
52
+ - **Training:** a mix of three synthetic tasks over 128 content tokens; inputs of 32–128 tokens; AdamW at learning
53
+ rate 1e-3, 6,000 steps.
54
+
55
+ ## Honest limits
56
+
57
+ - **Not a language model:** it works on abstract tokens, not text.
58
+ - **Not a new mechanism:** induction heads, and relative positions helping them generalize, are known results
59
+ (e.g. arXiv 2412.11459). This is a small, open, fully tested working instance of them.
60
+ - **Self-tested:** one AI designed, trained and graded it. It has not been independently reproduced.
61
+
62
+ ## Files
63
+
64
+ | Path | What |
65
+ |---|---|
66
+ | `weights/engine_seed{1..6}.safetensors` | six independently trained models |
67
+ | `engine.py` | model, task generators, evaluation |
68
+ | `demo.py` | a quick hands-on test |
69
+ | `score.py` | applies the registered pass bar |
70
+ | `results/` | per-seed scores and the demo output |
71
+
72
+ ## Provenance
73
+
74
+ AI participation: Claude (Anthropic, Opus 5.5) designed, built, trained and evaluated the model and wrote this card
75
+ → human validation: not performed → human editing/curation: direction by Chad Holland, 2026-11-16 → human
76
+ responsibility: Chad Holland. Review level: direction only.
SHA256SUMS ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 404f35ef3263ba31edc854045874af89c93422f7b155a04b2f4d97c71e794476 README.md
2
+ de7234b9d2e9c74ecc1de2b3f4689767a79611a8f10f8d6b27a532d05095dbde demo.py
3
+ d701a90d63782126330c0d31d51687682bf2f89492765c6b67b5ee40f01d5576 engine.py
4
+ f1993d41bb622359f1f216ce4f9e68fc796e6126029f7be0a741f1311a35d7c6 results/demo_seed1.txt
5
+ d730bb1797d44fc918c85a80ed1b8a15174afb3020ed8d5925dde10be506c508 results/score.txt
6
+ 928b9adedf63e99ac1a694ccff803a1271e7da78ec87fba9570351b4924ac086 results/seed1.json
7
+ 19eef19335e90dab8aa5f9c1abcc400d7481ad6a549db3d90ab2cfa3ca50d1b2 results/seed2.json
8
+ 665207b56a75f09d50aa825e92222b4e533f1f30347487c4e8cada95d24ca81d results/seed3.json
9
+ 097f9bd0f2455a8ebdf4df6d09e80c511fbc50231c7a2b49c2a10c2c51543f84 results/seed4.json
10
+ 78c5a3d3293f2d48dcf34c82d7d75a8f7160ba778d7701d92dfb84bfc6da364f results/seed5.json
11
+ 9cea52984ae85af459546771a64a2c8a6da79dfd32c5748f74fde248d73e9bc3 results/seed6.json
12
+ a2ba58e0bcb3e068749110b0d7474794a5e316173a0cdf61ebcd01d69899489d score.py
13
+ 5d39dc8785bf021082de26864edf8012c5bbfd9c8f068f6efc88018f759586c3 weights/engine_seed1.safetensors
14
+ f19fdf2a24ae3040fd6056708985483a8ae4ed3780b713f9d8546dabe8dbbf83 weights/engine_seed2.safetensors
15
+ 085b36fa301668a88edf3e241d5310e2e86f49b205be6d0304a1edf4ad22ccae weights/engine_seed3.safetensors
16
+ 8e6827a6f1d26906723327cfc2c0e61cf8ebd0ed32ebb947ebb14b2e90ce7833 weights/engine_seed4.safetensors
17
+ 9d0c81404b735e9581a4264688f46f7cf2afb62b4a671bd1c8528e80451e8fa6 weights/engine_seed5.safetensors
18
+ fe800006a98d8fd2f71fc6c72f72170e2d72feb0e74d6e6c0c9430413626dcdd weights/engine_seed6.safetensors
demo.py ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Try the tiny in-context engine. Usage: python3 demo.py weights/engine_seed1.safetensors
2
+ Shows the three skills on fresh random inputs, including inputs longer than any it trained on."""
3
+ import sys
4
+
5
+ import numpy as np
6
+ import torch
7
+ from safetensors.torch import load_file
8
+
9
+ import engine as E
10
+
11
+
12
+ def greedy(m, prefix, n):
13
+ x = torch.tensor([prefix])
14
+ with torch.no_grad():
15
+ for _ in range(n):
16
+ x = torch.cat([x, m(x)[:, -1:].argmax(-1)], 1)
17
+ return x[0, len(prefix):].tolist()
18
+
19
+
20
+ m = E.TinyEngine()
21
+ m.load_state_dict(load_file(sys.argv[1] if len(sys.argv) > 1 else "weights/engine_seed1.safetensors"))
22
+ m.eval()
23
+ rng = np.random.default_rng(2026)
24
+ print(f"TinyEngine, {E.n_params(m):,} parameters\n")
25
+
26
+ seg = rng.integers(1, 128, size=11).tolist() # period 11 was never trained on
27
+ prefix = [E.BOS] + (seg * 30)[:300] # 300 tokens: longer than any training input
28
+ out = greedy(m, prefix, 22)
29
+ want = (seg * 40)[300:322]
30
+ print("1) continue a pattern of period 11 (never trained), after 300 tokens of context")
31
+ print(" model :", out); print(" truth :", want, "->", "OK" if out == want else "DIFF")
32
+
33
+ keys = rng.choice(np.arange(1, 64), 40, replace=False); vals = rng.choice(np.arange(64, 128), 40)
34
+ ctx = [E.BOS] + [int(t) for kv in zip(keys, vals) for t in kv]
35
+ hits = 0
36
+ for j in range(40):
37
+ hits += greedy(m, ctx + [E.QRY, int(keys[j])], 1)[0] == int(vals[j])
38
+ print(f"\n2) look up 40 values from 40 key-value pairs given earlier: {hits}/40 correct")
39
+
40
+ s = rng.choice(np.arange(1, 128), 50, replace=False).tolist()
41
+ out = greedy(m, [E.BOS] + s + [E.SEP], 50)
42
+ print(f"\n3) copy a 50-token string exactly: {sum(a == b for a, b in zip(out, s))}/50 tokens correct")
engine.py ADDED
@@ -0,0 +1,175 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Tiny in-context engine: an attention-only transformer with rotary (relative) positions, trained from scratch on a
2
+ mix of in-context tasks, then tested on lengths it never saw. See REGISTRATION.md.
3
+ Written by Claude (Anthropic, Opus 5.5) at Chad Holland's direction, 2026-11-16. SELF-TESTED only."""
4
+ import math
5
+
6
+ import numpy as np
7
+ import torch
8
+ import torch.nn as nn
9
+ import torch.nn.functional as F
10
+
11
+ N_CONTENT = 127 # content tokens 1..127
12
+ BOS, SEP, QRY = 0, 128, 129
13
+ VOCAB = 130
14
+ CFG = dict(d_model=128, n_heads=2, n_layers=2, train_lens=(32, 64, 96, 128), batch=64, lr=1e-3, steps=6000)
15
+ PERIODS_SEEN = tuple(L for L in range(4, 25) if L not in (7, 11, 15, 19, 23))
16
+ PERIODS_HELD = (7, 11, 15, 19, 23)
17
+
18
+
19
+ # ---------------------------------------------------------------- tasks: (tokens, loss mask), mask marks positions whose
20
+ # NEXT token is determined by the context, so the score is only over learnable predictions.
21
+ def task_periodic(rng, T, periods):
22
+ L = int(rng.choice(periods))
23
+ seg = rng.integers(1, N_CONTENT + 1, size=L)
24
+ x = np.concatenate([[BOS], np.resize(seg, T - 1)])
25
+ m = np.zeros(T, bool)
26
+ m[L:T - 1] = True # predicting x[t+1] from position t, for t+1 > L
27
+ return x, m
28
+
29
+
30
+ KEYS = np.arange(1, 64) # keys and values come from disjoint halves of the vocabulary, so a key token can
31
+ VALS = np.arange(64, N_CONTENT + 1) # never also appear as a value (that made the first version of this task ambiguous)
32
+
33
+
34
+ def task_recall(rng, T):
35
+ """BOS k1 v1 ... kn vn, then queries 'QRY k v' about earlier pairs, filling length T."""
36
+ n = min(len(KEYS), max(2, (T - 1) * 2 // 5 // 2 * 2 // 2))
37
+ nq = (T - 1 - 2 * n) // 3
38
+ while nq < 1:
39
+ n -= 1
40
+ nq = (T - 1 - 2 * n) // 3
41
+ keys = rng.choice(KEYS, size=n, replace=False)
42
+ vals = rng.choice(VALS, size=n)
43
+ x = [BOS] + [t for kv in zip(keys, vals) for t in kv]
44
+ m = [False] * len(x)
45
+ for j in rng.integers(0, n, size=nq):
46
+ x += [QRY, keys[j], vals[j]]
47
+ m += [False, True, False] # position of the key predicts the value
48
+ pad = T - len(x)
49
+ x += list(rng.integers(1, N_CONTENT + 1, size=pad))
50
+ m += [False] * pad
51
+ return np.array(x), np.array(m)
52
+
53
+
54
+ def task_copy(rng, T):
55
+ """BOS s SEP s: s has distinct tokens (so copying is unambiguous); length fills T."""
56
+ k = min((T - 2) // 2, N_CONTENT)
57
+ s = rng.choice(np.arange(1, N_CONTENT + 1), size=k, replace=False)
58
+ x = np.concatenate([[BOS], s, [SEP], s])
59
+ m = np.zeros(len(x), bool)
60
+ m[k + 1:len(x) - 1] = True # from SEP onward, each next token is the copy
61
+ pad = T - len(x)
62
+ x = np.concatenate([x, rng.integers(1, N_CONTENT + 1, size=pad)])
63
+ m = np.concatenate([m, np.zeros(pad, bool)])
64
+ return x, m
65
+
66
+
67
+ TASKS = ("periodic", "recall", "copy")
68
+
69
+
70
+ def make_batch(rng, n, T, task=None, periods=PERIODS_SEEN):
71
+ xs, ms = [], []
72
+ for _ in range(n):
73
+ t = task or TASKS[int(rng.integers(len(TASKS)))]
74
+ x, m = (task_periodic(rng, T, periods) if t == "periodic" else task_recall(rng, T) if t == "recall"
75
+ else task_copy(rng, T))
76
+ xs.append(x); ms.append(m)
77
+ return torch.tensor(np.array(xs)).long(), torch.tensor(np.array(ms))
78
+
79
+
80
+ # ---------------------------------------------------------------- model
81
+ def rope(x, base=10000.0):
82
+ """Rotary position embedding on the last dim of (b, h, t, d)."""
83
+ t, d = x.shape[-2], x.shape[-1]
84
+ inv = 1.0 / base ** (torch.arange(0, d, 2, dtype=torch.float32) / d)
85
+ ang = torch.arange(t, dtype=torch.float32)[:, None] * inv[None]
86
+ cos, sin = ang.cos(), ang.sin()
87
+ x1, x2 = x[..., 0::2], x[..., 1::2]
88
+ return torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1).flatten(-2)
89
+
90
+
91
+ class Attn(nn.Module):
92
+ def __init__(self, d, h, use_rope=True):
93
+ super().__init__()
94
+ self.h, self.dh, self.use_rope = h, d // h, use_rope
95
+ self.q, self.k, self.v, self.o = (nn.Linear(d, d, bias=False) for _ in range(4))
96
+
97
+ def forward(self, x):
98
+ b, t, d = x.shape
99
+ sh = lambda z: z.view(b, t, self.h, self.dh).transpose(1, 2)
100
+ q, k = sh(self.q(x)), sh(self.k(x))
101
+ if self.use_rope:
102
+ q, k = rope(q), rope(k)
103
+ y = F.scaled_dot_product_attention(q, k, sh(self.v(x)), is_causal=True)
104
+ return self.o(y.transpose(1, 2).reshape(b, t, d))
105
+
106
+
107
+ class TinyEngine(nn.Module):
108
+ """Token embedding (no position embedding: positions enter only through rotary attention), attention-only
109
+ pre-LN blocks, separate (untied) unembedding. Tying the unembedding to the embedding stopped learning entirely
110
+ in exploration (0.00 periodic accuracy after 1,500 steps at d_model 256; untied reached 1.00)."""
111
+
112
+ def __init__(self, cfg=CFG):
113
+ super().__init__()
114
+ d = cfg["d_model"]
115
+ self.pos_kind = cfg.get("pos", "rope")
116
+ self.emb = nn.Embedding(VOCAB, d)
117
+ self.pos = nn.Embedding(1024, d) if self.pos_kind == "learned" else None
118
+ self.lns = nn.ModuleList(nn.LayerNorm(d) for _ in range(cfg["n_layers"]))
119
+ self.attn = nn.ModuleList(Attn(d, cfg["n_heads"], self.pos_kind == "rope") for _ in range(cfg["n_layers"]))
120
+ self.lnf = nn.LayerNorm(d)
121
+ self.head = None if cfg.get("tied", False) else nn.Linear(d, VOCAB, bias=False)
122
+
123
+ def forward(self, x):
124
+ h = self.emb(x)
125
+ if self.pos is not None:
126
+ h = h + self.pos(torch.arange(x.shape[1]))
127
+ for ln, at in zip(self.lns, self.attn):
128
+ h = h + at(ln(h))
129
+ h = self.lnf(h)
130
+ return self.head(h) if self.head is not None else h @ self.emb.weight.T
131
+
132
+
133
+ def n_params(m):
134
+ return sum(p.numel() for p in m.parameters())
135
+
136
+
137
+ # ---------------------------------------------------------------- evaluation
138
+ @torch.no_grad()
139
+ def accuracy(m, task, T, seed, n=256, periods=PERIODS_SEEN):
140
+ x, mask = make_batch(np.random.default_rng(seed), n, T, task, periods)
141
+ pred = m(x).argmax(-1)[:, :-1]
142
+ ok = (pred == x[:, 1:])[mask[:, :-1]]
143
+ return float(ok.float().mean())
144
+
145
+
146
+ EVAL_LENS = (128, 256, 512) # 128 = longest trained; 256 and 512 = 2x and 4x beyond it
147
+
148
+
149
+ def evaluate(m):
150
+ m.eval()
151
+ r = {}
152
+ for L in EVAL_LENS:
153
+ n = 256 if L <= 128 else 96
154
+ r[f"periodic_heldout_L{L}"] = accuracy(m, "periodic", L, 100 + L, n, periods=PERIODS_HELD)
155
+ r[f"recall_L{L}"] = accuracy(m, "recall", L, 200 + L, n)
156
+ r[f"copy_L{L}"] = accuracy(m, "copy", L, 300 + L, n)
157
+ r["periodic_seen_L128"] = accuracy(m, "periodic", 128, 99)
158
+ m.train()
159
+ return r
160
+
161
+
162
+ def train(seed, cfg=CFG, log=None):
163
+ torch.manual_seed(seed)
164
+ torch.use_deterministic_algorithms(True)
165
+ rng = np.random.default_rng(seed)
166
+ m = TinyEngine(cfg)
167
+ opt = torch.optim.AdamW(m.parameters(), lr=cfg["lr"], weight_decay=0.0)
168
+ for step in range(1, cfg["steps"] + 1):
169
+ x, mask = make_batch(rng, cfg["batch"], int(rng.choice(cfg["train_lens"])))
170
+ logits = m(x)[:, :-1]
171
+ loss = F.cross_entropy(logits[mask[:, :-1]], x[:, 1:][mask[:, :-1]])
172
+ opt.zero_grad(); loss.backward(); opt.step()
173
+ if log and step % 500 == 0:
174
+ log(step, float(loss.detach()))
175
+ return m
results/demo_seed1.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ TinyEngine, 165,120 parameters
2
+
3
+ 1) continue a pattern of period 11 (never trained), after 300 tokens of context
4
+ model : [82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4, 82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4]
5
+ truth : [82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4, 82, 47, 60, 11, 48, 82, 46, 106, 109, 23, 4] -> OK
6
+
7
+ 2) look up 40 values from 40 key-value pairs given earlier: 40/40 correct
8
+
9
+ 3) copy a 50-token string exactly: 50/50 tokens correct
results/score.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ seed periodic_heldout_L128 recall_L128 copy_L128 periodic_heldout_L256 recall_L256 copy_L256 periodic_heldout_L512
2
+ 1 0.998 1.000 1.000 0.999 0.972 0.767 0.985
3
+ 2 0.998 1.000 1.000 0.999 0.948 0.576 0.975
4
+ 3 0.998 1.000 1.000 0.999 0.954 0.563 0.995
5
+ 4 0.998 1.000 1.000 0.998 0.953 0.529 0.967
6
+ 5 0.998 1.000 1.000 0.997 0.961 0.625 0.988
7
+ 6 0.999 1.000 1.000 0.998 0.959 0.538 0.985
8
+ Q1: HELD (6/6 seeds meet every threshold; need 5)
9
+ Q2: FAILED (1/6 seeds meet every threshold; need 5)
10
+ Q3: HELD (6/6 seeds meet every threshold; need 5)
results/seed1.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"seed": 1, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9984326958656311, "recall_L128": 1.0, "copy_L128": 1.0, "periodic_heldout_L256": 0.9987830519676208, "recall_L256": 0.9720180034637451, "copy_L256": 0.7667322754859924, "periodic_heldout_L512": 0.985174298286438, "recall_L512": 0.8795573115348816, "copy_L512": 0.7662401795387268, "periodic_seen_L128": 0.9990339279174805}, "train_seconds": 353}
results/seed2.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"seed": 2, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9983630776405334, "recall_L128": 0.9998437762260437, "copy_L128": 0.9999380111694336, "periodic_heldout_L256": 0.9990438222885132, "recall_L256": 0.9483251571655273, "copy_L256": 0.5763615369796753, "periodic_heldout_L512": 0.9746325016021729, "recall_L512": 0.868408203125, "copy_L512": 0.5822670459747314, "periodic_seen_L128": 0.9986889362335205}, "train_seconds": 344}
results/seed3.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"seed": 3, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9981192350387573, "recall_L128": 0.9996874928474426, "copy_L128": 1.0, "periodic_heldout_L256": 0.9985222816467285, "recall_L256": 0.953839898109436, "copy_L256": 0.5629921555519104, "periodic_heldout_L512": 0.9947711229324341, "recall_L512": 0.8627116084098816, "copy_L512": 0.5631561875343323, "periodic_seen_L128": 0.9989304542541504}, "train_seconds": 375}
results/seed4.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"seed": 4, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9981889128684998, "recall_L128": 0.9996874928474426, "copy_L128": 0.9996899962425232, "periodic_heldout_L256": 0.9980006814002991, "recall_L256": 0.9526143670082092, "copy_L256": 0.5291174650192261, "periodic_heldout_L512": 0.9666106700897217, "recall_L512": 0.8251953125, "copy_L512": 0.5330544710159302, "periodic_seen_L128": 0.9980334043502808}, "train_seconds": 360}
results/seed5.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"seed": 5, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9979799389839172, "recall_L128": 1.0, "copy_L128": 0.9999380111694336, "periodic_heldout_L256": 0.997001051902771, "recall_L256": 0.961192786693573, "copy_L256": 0.6246719360351562, "periodic_heldout_L512": 0.988156259059906, "recall_L512": 0.8828938603401184, "copy_L512": 0.6314796805381775, "periodic_seen_L128": 0.9984129071235657}, "train_seconds": 349}
results/seed6.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"seed": 6, "cfg": {"d_model": 128, "n_heads": 2, "n_layers": 2, "train_lens": [32, 64, 96, 128], "batch": 64, "lr": 0.001, "steps": 6000}, "params": 165120, "scores": {"periodic_heldout_L128": 0.9987461566925049, "recall_L128": 1.0, "copy_L128": 1.0, "periodic_heldout_L256": 0.997783362865448, "recall_L256": 0.9585375785827637, "copy_L256": 0.538385808467865, "periodic_heldout_L512": 0.9846702814102173, "recall_L512": 0.861572265625, "copy_L512": 0.5410104990005493, "periodic_seen_L128": 0.9986889362335205}, "train_seconds": 340}
score.py ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Apply REGISTRATION.md's bar to the per-seed results. Usage: python3 score.py RESULTS_DIR"""
2
+ import json, sys
3
+ from pathlib import Path
4
+ R = [json.loads(p.read_text()) for p in sorted(Path(sys.argv[1]).glob("seed*.json"), key=lambda p: int(p.stem[4:]))]
5
+ BAR = {"Q1": [("periodic_heldout_L128", .95), ("recall_L128", .95), ("copy_L128", .95)],
6
+ "Q2": [("periodic_heldout_L256", .95), ("recall_L256", .90), ("copy_L256", .70)],
7
+ "Q3": [("periodic_heldout_L512", .90)]}
8
+ print("seed " + " ".join(k for q in BAR.values() for k, _ in q))
9
+ for r in R:
10
+ print(f"{r['seed']:>4} " + " ".join(f"{r['scores'][k]:.3f}" for q in BAR.values() for k, _ in q))
11
+ for q, rules in BAR.items():
12
+ ok = sum(all(r["scores"][k] >= t for k, t in rules) for r in R)
13
+ print(f"{q}: {'HELD' if ok >= 5 else 'FAILED'} ({ok}/{len(R)} seeds meet every threshold; need 5)")
weights/engine_seed1.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d39dc8785bf021082de26864edf8012c5bbfd9c8f068f6efc88018f759586c3
3
+ size 661752
weights/engine_seed2.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f19fdf2a24ae3040fd6056708985483a8ae4ed3780b713f9d8546dabe8dbbf83
3
+ size 661752
weights/engine_seed3.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:085b36fa301668a88edf3e241d5310e2e86f49b205be6d0304a1edf4ad22ccae
3
+ size 661752
weights/engine_seed4.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8e6827a6f1d26906723327cfc2c0e61cf8ebd0ed32ebb947ebb14b2e90ce7833
3
+ size 661752
weights/engine_seed5.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d0c81404b735e9581a4264688f46f7cf2afb62b4a671bd1c8528e80451e8fa6
3
+ size 661752
weights/engine_seed6.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe800006a98d8fd2f71fc6c72f72170e2d72feb0e74d6e6c0c9430413626dcdd
3
+ size 661752