gridlm v0: screen + generator + odin operator, trained on colony-one

LICENSE

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) 2026 Nick Flach and Kannaka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -0,0 +1,122 @@
+---
+license: mit
+library_name: pytorch
+tags:
+- artificial-life
+- evolution
+- thegrid
+- genome
+- mutation-operator
+datasets:
+- flaukowski/thegrid-colony-one-transitions
+---
+
+# gridlm: a mutation screen and generator learned from a thegrid fossil record
+
+[thegrid](https://github.com/EckmanTechLLC/thegrid) (Matt Eckman, MIT) is an artificial-life lab.
+Its organisms are small programs on an evolvable instruction set, and they compete for real CPU,
+memory and heat on the host. Every birth goes into a fossil record.
+
+thegrid's own findings (`tools/analyse/LESSONS.md`) include this one: **model-authored mutation
+founded nothing.** 4,554 genomes written by a general LLM from a prompt produced 4,554 births and
+no lineage. Over the same period, blind segment transfer compounded at 1.3x.
+
+gridlm asks whether a model trained on **what actually reproduced** does better than one prompted
+with the instruction set. The idea is borrowed from DNA language models such as Hugging Face's
+Carbon-X: give each unit of the sequence its own token and learn from outcomes. Here the unit is
+an opcode, and the outcomes come from the fossil record.
+
+Two models, about 0.8M parameters each:
+
+- **screen** (`screen.pt`): an encoder over `[CLS] parent [SEP] child`. It outputs P(the child
+  genome is born at least twice in its epoch), meaning it copied itself and founded something.
+- **generator** (`generator.pt`): a causal LM over `[BOS] parent [SEP] child [EOS]`, trained only
+  on children that reproduced. It writes a child for a given parent.
+
+There is also **`gridlm_operator.py`**, a drop-in operator for thegrid's `odin` mutator. It reads
+the same `request.json` / `proposal.json` queue that thegrid's `tools/operator` answers. For each
+request it samples 8 children, the screen picks the best, and it proposes that one. It runs on
+CPU in milliseconds. The colony validates every proposal with its own parser, so a proposal it
+cannot use costs one birth, never the colony.
+
+## Data
+
+The fossil record of **colony-one**, the thegrid colony that kannaka-grid runs on its own host,
+read 2026-10-27 (dataset: `flaukowski/thegrid-colony-one-transitions`):
+
+- 1,573,008 rows: one per (epoch, child genome), with its earliest parent and exact self-copies excluded.
+- Label: the child was born at least twice in its epoch. The positive rate is 24.8%.
+- 614 epochs. **This is an older thegrid version and a starvation economy** (613 of 614 epochs ended
+  in extinction). It is not the eviction or class worlds Matt runs today.
+
+Deliberately excluded from the inputs:
+- `mutation_type`: thegrid records a mutation origin only when its child was born, so its presence
+  leaks the label.
+- The parent's birth count for the epoch: it includes births after the child appeared.
+
+The inputs are the two instruction sequences and nothing else.
+
+## Results (held out in time)
+
+Split by epoch: train epochs 1–565 (1.27M rows), validation 566–593, test 594–614 (152k rows).
+The "novel" columns use only test children whose genome never appears in training (145k).
+
+| model | AUROC | avg precision | precision in top 10% | AUROC, novel | top 10%, novel |
+|---|---|---|---|---|---|
+| base rate | | 0.248 | 0.248 | | 0.240 |
+| rule: has alloc+copy+fork | 0.526 | 0.267 | 0.283 | 0.528 | 0.265 |
+| logistic regression, 15 hand features | 0.609 | 0.327 | 0.384 | 0.605 | 0.362 |
+| **gridlm screen** | **0.667** | **0.385** | **0.452** | **0.659** | **0.418** |
+| gridlm generator, log p(child \| parent) | 0.582 | 0.296 | 0.321 | 0.586 | 0.320 |
+
+**What this says:** the outcome of a single mutation is mostly noise, which the weak baselines
+show. But the screen ranks children well above chance and well above hand features. The 10% it
+likes best reproduce at 1.8x the base rate (1.7x on genomes it has never seen).
+
+**What it does not say:** whether the generator's children found lineages in a live colony. That
+needs a colony.
+
+The generator's 1,000 samples on held-out parents:
+- 100% are valid length;
+- 83.2% contain alloc+copy+fork;
+- 2.3% are identical to the parent.
+
+The screen rates them higher than the real test mutants: mean P 0.525 against 0.469. That is
+one model judging another and is no evidence of fitness.
+
+The samples look like what LESSONS says evolution does there. They include tandem duplications
+of an earning motif and duplicated `fork copy alloc` blocks. Some are bad, for example
+substitutions that remove `fork`.
+
+## Limits
+
+- **One colony, one ISA version, one economy.** Instruction names colony-one never used are
+  unknown tokens, and the model's taste is a starvation world's taste. It should be retrained on
+  the target fleet's own record. `extract.py` and `train.py` do that end to end from any thegrid
+  `history.sqlite3`.
+- **No replicate colonies.** thegrid's own rule is "never run one of anything you intend to
+  learn from". The time split is held out, but it is one colony. The real test is several
+  replicate colonies with this operator against the same colonies on blind variation, compared on
+  lineages founded per proposal.
+- **LESSONS 11–12 say the economy decides where a world ends up**, not the mutation operator.
+  A better mutator may only reach stasis faster. That is worth measuring, not assuming away.
+
+## Use
+
+```bash
+pip install torch numpy
+# one proposal
+python gridlm_operator.py --model-dir . --once-genome "harvest harvest alloc copy ifnotdone jmpb fork scan move"
+# serve one or more colonies' odin queues
+python gridlm_operator.py --model-dir . --queue /path/to/colony/operator
+# retrain on your own fossil record
+python extract.py /path/to/history.sqlite3 transitions.jsonl.gz
+python train.py transitions.jsonl.gz out --device cuda
+```
+
+Trained on one qBraid RTX 4090 in about 7 minutes (12.8 credits, about $0.13).
+
+## Credits
+
+Built by Kannaka (an AI) with Nick Flach, on top of Matt Eckman's thegrid. thegrid's instruction
+set, fossil record format and operator protocol are his. The model, code and card are MIT.

extract.py

@@ -0,0 +1,51 @@
+"""Extract a parent->child training set from a thegrid fossil record (read-only).
+
+One row per (epoch, child genome): the child's earliest recorded parent in that epoch,
+exact self-copies (parent == child) excluded. Label: the child genome was born at least
+twice in that epoch (it copied itself, so it "founded" something), from genome_stats.births.
+
+mutation_origins is deliberately NOT used: it only records origins whose child was born,
+so its presence would leak the label.
+
+usage: python3 extract.py <history.sqlite3> <out.jsonl.gz>
+"""
+import gzip
+import json
+import sqlite3
+import sys
+
+db, out = sys.argv[1], sys.argv[2]
+c = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
+
+src = {gid: s for gid, s in c.execute("SELECT genome_id, source FROM genomes")}
+print("genomes", len(src), file=sys.stderr)
+births = {(e, g): b for e, g, b in c.execute("SELECT epoch, genome_id, births FROM genome_stats")}
+print("stats", len(births), file=sys.stderr)
+
+rows = c.execute(
+    "SELECT epoch, child_genome_id, parent_genome_id, first_tick FROM transitions "
+    "WHERE parent_genome_id != child_genome_id ORDER BY epoch, child_genome_id, first_tick")
+n = kept = skipped = 0
+last = None
+with gzip.open(out, "wt", encoding="utf-8") as f:
+    for epoch, child, parent, tick in rows:
+        n += 1
+        if (epoch, child) == last:
+            continue          # a later parent of the same child in the same epoch
+        last = (epoch, child)
+        if child not in src or parent not in src:
+            skipped += 1
+            continue
+        cb = births.get((epoch, child), 0)
+        f.write(json.dumps({
+            "epoch": epoch,
+            "child_id": child,
+            "parent": src[parent].split(" · "),
+            "child": src[child].split(" · "),
+            "child_births": cb,
+            "parent_births": births.get((epoch, parent), 0),
+            "first_tick": tick,
+            "label": int(cb >= 2),
+        }, separators=(",", ":")) + "\n")
+        kept += 1
+print(json.dumps({"transitions_scanned": n, "rows": kept, "skipped_missing_source": skipped}), file=sys.stderr)

generator.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a98428b5cb61075f33a728efe052dd368c6e343d5d459aa28b11998f1a3e031
+size 3315176

generator.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a393b36d0be7b60bc5d295d2eec14bd04ffe282390d9f5d2f9cdd50eff3caea0
+size 3303840

gridlm_operator.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""A thegrid `odin` operator backed by gridlm instead of a general LLM.
+
+Same queue protocol as thegrid's tools/operator/grid_operator.py: the colony writes
+<queue>/request.json (with "genome", a list of instruction names, and "request_id"); this
+writes <queue>/proposal.json ({"request_id", "authored_at", "genome"}) atomically. The colony
+validates every proposal with its own parser, so a proposal it can't use costs one birth (it
+falls back to blind variation) and never the colony.
+
+How a proposal is made: the generator samples K children for the parent; the screen scores
+each; the best-scoring child that differs from the parent is proposed. Runs on CPU in
+milliseconds, so there is no GPU queue to wait for.
+
+Honest scope: trained on ONE colony (kannaka-grid's colony-one, an older thegrid ISA and a
+starvation economy). Instruction names that colony never used are unknown to the model, and
+anything it proposes that a destination colony can't run is rejected there by design.
+
+usage:
+  python gridlm_operator.py --model-dir . --queue /path/to/colony/operator [--queue ...]
+  python gridlm_operator.py --model-dir . --once-genome "harvest harvest alloc copy ifnotdone jmpb fork scan move"
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+import torch
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import train as g  # noqa: E402  (model classes, tokens, sampling)
+
+
+def load(model_dir: Path):
+    vocab = json.loads((model_dir / "vocab.json").read_text())
+    screen = g.Screen(len(vocab))
+    gen = g.Generator(len(vocab))
+    for model, name in ((screen, "screen"), (gen, "generator")):
+        st = model_dir / f"{name}.safetensors"
+        if st.exists():                      # preferred: no pickle
+            from safetensors.torch import load_file
+            model.load_state_dict(load_file(str(st)))
+        else:
+            model.load_state_dict(torch.load(model_dir / f"{name}.pt", map_location="cpu", weights_only=True))
+    screen.eval()
+    gen.eval()
+    return vocab, {t: i for i, t in enumerate(vocab)}, screen, gen
+
+
+def propose(names, vocab, idx, screen, gen, k=8, temperature=0.9, seed=None):
+    parent = [idx.get(n, g.UNK) for n in names][: g.MAXG]
+    kids = g.sample_children(gen, [parent] * k, vocab, k, temperature=temperature,
+                             rng_seed=seed if seed is not None else int(time.time() * 1000) % (2 ** 31))
+    cands = [c for c in kids if 1 <= len(c) <= g.MAXG and c != list(names)]
+    if not cands:
+        return None, None
+    rows = [{"parent": list(names)[: g.MAXG], "child": c, "label": 0, "child_id": ""} for c in cands]
+    X, _ = g.screen_tensor(rows, idx)
+    p = g.screen_scores(screen, X)
+    best = int(p.argmax())
+    return cands[best], float(p[best])
+
+
+def serve(queue: Path, model, stats) -> bool:
+    req_p, prop_p = queue / "request.json", queue / "proposal.json"
+    if prop_p.exists() or not req_p.exists():
+        return False
+    try:
+        req = json.loads(req_p.read_text())
+    except (OSError, ValueError):
+        return False
+    child, score = propose(req.get("genome", []), *model)
+    if child is None:
+        stats["rejected"] += 1
+        return True
+    tmp = prop_p.with_suffix(".tmp")
+    tmp.write_text(json.dumps({"request_id": req.get("request_id"), "authored_at": time.time(),
+                               "genome": child, "gridlm_screen_p": score}))
+    tmp.replace(prop_p)
+    stats["served"] += 1
+    print(f"{time.strftime('%H:%M:%S')} {queue}: {len(req.get('genome', []))} -> {len(child)} "
+          f"(screen p={score:.3f}; served {stats['served']} rejected {stats['rejected']})", flush=True)
+    return True
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--model-dir", default=".")
+    ap.add_argument("--queue", action="append", default=[])
+    ap.add_argument("--once-genome", help="print one proposal for this genome (space-separated names) and exit")
+    ap.add_argument("--poll", type=float, default=1.0)
+    a = ap.parse_args()
+    torch.set_num_threads(2)
+    model = load(Path(a.model_dir))
+    if a.once_genome:
+        child, score = propose(a.once_genome.split(), *model)
+        print(json.dumps({"parent": a.once_genome.split(), "child": child, "screen_p": score}))
+        return
+    stats = {"served": 0, "rejected": 0}
+    queues = [Path(q) for q in a.queue]
+    print("gridlm operator; queues:", ", ".join(map(str, queues)), flush=True)
+    while True:
+        worked = False
+        for q in queues:
+            if q.exists():
+                worked = serve(q, model, stats) or worked
+        if not worked:
+            time.sleep(a.poll)
+
+
+if __name__ == "__main__":
+    main()

results.json

@@ -0,0 +1,157 @@
+{
+ "split": {
+  "epoch_cuts": [
+   565,
+   593
+  ],
+  "train": 1266165,
+  "val": 154346,
+  "test": 152497,
+  "novel_test": 145441
+ },
+ "test": [
+  {
+   "model": "baseline: has alloc+copy+fork",
+   "n": 152497,
+   "base_rate": 0.24748027830055674,
+   "auroc": 0.5259577223102965,
+   "avg_precision": 0.2668144186982295,
+   "precision_at_top10pct": 0.2829693750409863
+  },
+  {
+   "model": "baseline: logistic on 15 hand features",
+   "n": 152497,
+   "base_rate": 0.24748027830055674,
+   "auroc": 0.6091323573663239,
+   "avg_precision": 0.3265924671833763,
+   "precision_at_top10pct": 0.3836317135549872
+  },
+  {
+   "model": "gridlm screen",
+   "n": 152497,
+   "base_rate": 0.24748027830055674,
+   "auroc": 0.6667371511948852,
+   "avg_precision": 0.38529948709031914,
+   "precision_at_top10pct": 0.451898485146567
+  },
+  {
+   "model": "gridlm generator log p(child|parent)",
+   "n": 152497,
+   "base_rate": 0.24748027830055674,
+   "auroc": 0.5821718263469734,
+   "avg_precision": 0.2962117960990998,
+   "precision_at_top10pct": 0.3207423437602466
+  }
+ ],
+ "test_novel": [
+  {
+   "model": "baseline: has alloc+copy+fork",
+   "n": 145441,
+   "base_rate": 0.23971919885039294,
+   "auroc": 0.5277137815178677,
+   "avg_precision": 0.2554675640963218,
+   "precision_at_top10pct": 0.2647827282728273
+  },
+  {
+   "model": "baseline: logistic on 15 hand features",
+   "n": 145441,
+   "base_rate": 0.23971919885039294,
+   "auroc": 0.6053365913437163,
+   "avg_precision": 0.31109046086779596,
+   "precision_at_top10pct": 0.36234873487348734
+  },
+  {
+   "model": "gridlm screen",
+   "n": 145441,
+   "base_rate": 0.23971919885039294,
+   "auroc": 0.6589837791458187,
+   "avg_precision": 0.3585924010318861,
+   "precision_at_top10pct": 0.4183168316831683
+  },
+  {
+   "model": "gridlm generator log p(child|parent)",
+   "n": 145441,
+   "base_rate": 0.23971919885039294,
+   "auroc": 0.5858987532745887,
+   "avg_precision": 0.2914317479740002,
+   "precision_at_top10pct": 0.3201320132013201
+  }
+ ],
+ "samples": {
+  "n": 1000,
+  "valid_length": 1.0,
+  "has_alloc_copy_fork": 0.832,
+  "identical_to_parent": 0.023,
+  "screen_mean_p_generated": 0.5254685282707214,
+  "screen_mean_p_real_test_children": 0.4690079689025879,
+  "note": "screen scores on generated children are one model judging another; only a live colony is ground truth"
+ },
+ "vocab": [
+  "[PAD]",
+  "[CLS]",
+  "[SEP]",
+  "[BOS]",
+  "[EOS]",
+  "[UNK]",
+  "add",
+  "alloc",
+  "build",
+  "burn",
+  "call",
+  "copy",
+  "copyn",
+  "corrupt",
+  "dec",
+  "define",
+  "fetch",
+  "fork",
+  "free",
+  "harvest",
+  "ifnotdone",
+  "ifzero",
+  "inc",
+  "input",
+  "jmpb",
+  "jmpr",
+  "link",
+  "listen",
+  "load",
+  "locate",
+  "macro0",
+  "macro1",
+  "macro2",
+  "macro3",
+  "macro4",
+  "macro5",
+  "macro6",
+  "macro7",
+  "move",
+  "nand",
+  "nop",
+  "offer",
+  "output",
+  "peek",
+  "post",
+  "publish",
+  "push",
+  "salvage",
+  "scan",
+  "signal",
+  "steal",
+  "store",
+  "sub",
+  "swap",
+  "write",
+  "xor"
+ ],
+ "config": {
+  "d_model": 128,
+  "layers": 4,
+  "heads": 4,
+  "ff": 512,
+  "max_genome": 64,
+  "label": "child genome born >= 2 times in its epoch",
+  "screen_params": 817409,
+  "generator_params": 824632
+ }
+}

screen.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2be4bc1ab9dddf5413877f23a92600f31a67ebe97ef6a0f810f862571d71734
+size 3282746

screen.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1eb1fb50c43b02f72f34bf2a17b6d8fc4936f09c5e5aea521fec880a202d2ea4
+size 3274948

train.log

@@ -0,0 +1,26 @@
+06:26:30 rows 1573008 train 1266165 val 154346 test 152497 (novel test 145441); epoch cuts (565, 593); vocab 56; train base rate 0.248
+06:26:56 baseline: has alloc+copy+fork: test auroc 0.5260
+06:26:56 baseline: logistic on 15 hand features: test auroc 0.6091
+06:27:41 screen epoch 1/8 loss 1.0093 val_auroc 0.6292 (32s)
+06:28:13 screen epoch 2/8 loss 0.9852 val_auroc 0.6459 (31s)
+06:28:44 screen epoch 3/8 loss 0.9709 val_auroc 0.6557 (31s)
+06:29:14 screen epoch 4/8 loss 0.9630 val_auroc 0.6573 (30s)
+06:29:44 screen epoch 5/8 loss 0.9575 val_auroc 0.6627 (30s)
+06:30:16 screen epoch 6/8 loss 0.9534 val_auroc 0.6626 (31s)
+06:30:46 screen epoch 7/8 loss 0.9494 val_auroc 0.6642 (31s)
+06:31:17 screen epoch 8/8 loss 0.9460 val_auroc 0.6666 (31s)
+06:31:19 screen: test auroc 0.6667 (novel 0.6590)
+06:31:32 generator epoch 1/12 nll 1.4800 val_logprob/token -0.8854 (12s)
+06:31:44 generator epoch 2/12 nll 0.8963 val_logprob/token -0.7759 (12s)
+06:31:56 generator epoch 3/12 nll 0.8297 val_logprob/token -0.7381 (12s)
+06:32:08 generator epoch 4/12 nll 0.7913 val_logprob/token -0.7091 (12s)
+06:32:20 generator epoch 5/12 nll 0.7655 val_logprob/token -0.6833 (12s)
+06:32:33 generator epoch 6/12 nll 0.7458 val_logprob/token -0.6740 (13s)
+06:32:45 generator epoch 7/12 nll 0.7329 val_logprob/token -0.6664 (12s)
+06:32:57 generator epoch 8/12 nll 0.7240 val_logprob/token -0.6586 (12s)
+06:33:09 generator epoch 9/12 nll 0.7147 val_logprob/token -0.6504 (12s)
+06:33:21 generator epoch 10/12 nll 0.7092 val_logprob/token -0.6461 (12s)
+06:33:34 generator epoch 11/12 nll 0.7024 val_logprob/token -0.6477 (13s)
+06:33:45 generator epoch 12/12 nll 0.6972 val_logprob/token -0.6433 (12s)
+06:33:48 generator ranking: test auroc 0.5822
+06:34:16 done: {"n": 1000, "valid_length": 1.0, "has_alloc_copy_fork": 0.832, "identical_to_parent": 0.023, "screen_mean_p_generated": 0.5254685282707214, "screen_mean_p_real_test_children": 0.4690079689025879, "note": "screen scores on generated children are one model judging another; only a live colony is ground truth"}

train.py

@@ -0,0 +1,452 @@
+"""gridlm: two small models trained on a thegrid fossil record.
+
+  screen     encoder: [CLS] parent [SEP] child -> P(child is born >= 2 times in its epoch)
+  generator  causal LM: [BOS] parent [SEP] child [EOS], trained on successful children only
+
+Held-out design: rows are split by EPOCH (train = earliest ~80% of rows' epochs, val = next
+~10%, test = last ~10%), so the test set is later, unseen runs of the colony. Test metrics are
+also reported on "novel" children whose genome never appears in the training split.
+
+Inputs are the two instruction sequences only. Excluded on purpose: mutation_type (recorded
+only for successful origins, so it leaks the label) and the parent's epoch birth count
+(includes births after the child appeared).
+
+usage: python train.py data.jsonl.gz outdir [--threads N] [--epochs-screen E] [--epochs-gen E]
+"""
+from __future__ import annotations
+
+import argparse
+import gzip
+import json
+import math
+import os
+import random
+import time
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+MAXG = 64
+SPECIALS = ["[PAD]", "[CLS]", "[SEP]", "[BOS]", "[EOS]", "[UNK]"]
+PAD, CLS, SEP, BOS, EOS, UNK = range(6)
+DEV = torch.device("cpu")
+KEY_OPS = ["alloc", "copy", "fork", "harvest", "scan", "move", "ifnotdone", "jmpb", "salvage", "steal"]
+
+
+# ---------------------------------------------------------------- data
+def load(path):
+    rows = []
+    with gzip.open(path, "rt", encoding="utf-8") as f:
+        for line in f:
+            r = json.loads(line)
+            if 1 <= len(r["child"]) <= MAXG and 1 <= len(r["parent"]) <= MAXG:
+                rows.append(r)
+    return rows
+
+
+def split_by_epoch(rows):
+    epochs = sorted({r["epoch"] for r in rows})
+    per = {e: 0 for e in epochs}
+    for r in rows:
+        per[r["epoch"]] += 1
+    total, acc, cut1, cut2 = len(rows), 0, None, None
+    for e in epochs:
+        acc += per[e]
+        if cut1 is None and acc >= 0.8 * total:
+            cut1 = e
+        if cut2 is None and acc >= 0.9 * total:
+            cut2 = e
+    tr = [r for r in rows if r["epoch"] <= cut1]
+    va = [r for r in rows if cut1 < r["epoch"] <= cut2]
+    te = [r for r in rows if r["epoch"] > cut2]
+    return tr, va, te, (cut1, cut2)
+
+
+def build_vocab(rows):
+    ops = sorted({t for r in rows for t in r["parent"] + r["child"]})
+    return SPECIALS + ops
+
+
+def enc_pair(r, idx):
+    p = [idx.get(t, UNK) for t in r["parent"]]
+    c = [idx.get(t, UNK) for t in r["child"]]
+    return p, c
+
+
+def screen_tensor(rows, idx):
+    L = 2 * MAXG + 2
+    X = np.full((len(rows), L), PAD, dtype=np.int64)
+    y = np.zeros(len(rows), dtype=np.float32)
+    for i, r in enumerate(rows):
+        p, c = enc_pair(r, idx)
+        seq = [CLS] + p + [SEP] + c
+        X[i, :len(seq)] = seq
+        y[i] = r["label"]
+    return torch.from_numpy(X), torch.from_numpy(y)
+
+
+def lm_tensor(rows, idx):
+    L = 2 * MAXG + 3
+    X = np.full((len(rows), L), PAD, dtype=np.int64)
+    M = np.zeros((len(rows), L), dtype=np.float32)   # 1 where the NEXT token is a child token or EOS
+    for i, r in enumerate(rows):
+        p, c = enc_pair(r, idx)
+        seq = [BOS] + p + [SEP] + c + [EOS]
+        X[i, :len(seq)] = seq
+        start = 1 + len(p)                 # position of SEP; predicts child[0]
+        M[i, start:start + len(c) + 1] = 1  # child tokens and EOS are the targets
+    return torch.from_numpy(X), torch.from_numpy(M)
+
+
+# ---------------------------------------------------------------- models
+class Encoder(nn.Module):
+    def __init__(self, V, d=128, layers=4, heads=4, ff=512, L=2 * MAXG + 2, causal=False):
+        super().__init__()
+        self.tok = nn.Embedding(V, d, padding_idx=PAD)
+        self.pos = nn.Embedding(L + 1, d)
+        layer = nn.TransformerEncoderLayer(d, heads, ff, dropout=0.1, batch_first=True, norm_first=True)
+        self.body = nn.TransformerEncoder(layer, layers, enable_nested_tensor=False)
+        self.norm = nn.LayerNorm(d)
+        self.causal = causal
+
+    def forward(self, x):
+        T = x.shape[1]
+        h = self.tok(x) + self.pos(torch.arange(T, device=x.device))[None]
+        mask = nn.Transformer.generate_square_subsequent_mask(T, device=x.device) if self.causal else None
+        h = self.body(h, mask=mask, src_key_padding_mask=(x == PAD), is_causal=self.causal)
+        return self.norm(h)
+
+
+class Screen(nn.Module):
+    def __init__(self, V, **kw):
+        super().__init__()
+        self.enc = Encoder(V, **kw)
+        self.head = nn.Linear(self.enc.tok.embedding_dim, 1)
+
+    def forward(self, x):
+        return self.head(self.enc(x)[:, 0]).squeeze(-1)
+
+
+class Generator(nn.Module):
+    def __init__(self, V, **kw):
+        super().__init__()
+        self.enc = Encoder(V, L=2 * MAXG + 3, causal=True, **kw)
+        self.head = nn.Linear(self.enc.tok.embedding_dim, V)
+
+    def forward(self, x):
+        return self.head(self.enc(x))
+
+
+# ---------------------------------------------------------------- metrics
+def auroc(scores, labels):
+    s = np.asarray(scores, dtype=np.float64)
+    y = np.asarray(labels, dtype=np.int64)
+    pos, neg = y.sum(), len(y) - y.sum()
+    if pos == 0 or neg == 0:
+        return float("nan")
+    order = np.argsort(s, kind="mergesort")
+    ranks = np.empty(len(s))
+    sorted_s = s[order]
+    i = 0
+    while i < len(s):                       # average ranks for ties
+        j = i
+        while j + 1 < len(s) and sorted_s[j + 1] == sorted_s[i]:
+            j += 1
+        ranks[order[i:j + 1]] = (i + j) / 2 + 1
+        i = j + 1
+    return float((ranks[y == 1].sum() - pos * (pos + 1) / 2) / (pos * neg))
+
+
+def avg_precision(scores, labels):
+    order = np.argsort(-np.asarray(scores))
+    y = np.asarray(labels)[order]
+    tp = np.cumsum(y)
+    prec = tp / np.arange(1, len(y) + 1)
+    return float((prec * y).sum() / max(1, y.sum()))
+
+
+def precision_at(scores, labels, frac):
+    k = max(1, int(len(scores) * frac))
+    order = np.argsort(-np.asarray(scores))[:k]
+    return float(np.asarray(labels)[order].mean())
+
+
+def report(name, scores, labels):
+    return {"model": name, "n": len(labels), "base_rate": float(np.mean(labels)),
+            "auroc": auroc(scores, labels), "avg_precision": avg_precision(scores, labels),
+            "precision_at_top10pct": precision_at(scores, labels, 0.10)}
+
+
+# ---------------------------------------------------------------- baselines
+def edit_distance(a, b):
+    prev = list(range(len(b) + 1))
+    for i, x in enumerate(a, 1):
+        cur = [i]
+        for j, yv in enumerate(b, 1):
+            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (x != yv)))
+        prev = cur
+    return prev[-1]
+
+
+def features(r):
+    c, p = r["child"], r["parent"]
+    f = [len(c), len(p), len(c) - len(p), edit_distance(p, c)]
+    f += [c.count(op) for op in KEY_OPS]
+    f.append(float(all(op in c for op in ("alloc", "copy", "fork"))))
+    return f
+
+
+def logistic(Xtr, ytr, Xte, iters=300):
+    mu, sd = Xtr.mean(0), Xtr.std(0) + 1e-6
+    A = torch.tensor((Xtr - mu) / sd, dtype=torch.float32)
+    B = torch.tensor((Xte - mu) / sd, dtype=torch.float32)
+    y = torch.tensor(ytr, dtype=torch.float32)
+    w = torch.zeros(A.shape[1], requires_grad=True)
+    b = torch.zeros(1, requires_grad=True)
+    opt = torch.optim.LBFGS([w, b], max_iter=iters)
+
+    def closure():
+        opt.zero_grad()
+        loss = F.binary_cross_entropy_with_logits(A @ w + b, y) + 1e-4 * (w * w).sum()
+        loss.backward()
+        return loss
+    opt.step(closure)
+    with torch.no_grad():
+        return (B @ w + b).numpy()
+
+
+# ---------------------------------------------------------------- training
+def batches(n, bs, shuffle, rng, lengths=None):
+    """Shuffled batches; with `lengths`, rows are sorted by length inside chunks of 50
+    batches so each batch can be trimmed to its own longest row."""
+    idx = list(range(n))
+    if shuffle:
+        rng.shuffle(idx)
+    if lengths is not None:
+        chunk = bs * 50
+        idx = [i for c in range(0, n, chunk) for i in sorted(idx[c:c + chunk], key=lambda j: lengths[j])]
+        groups = [idx[i:i + bs] for i in range(0, n, bs)]
+        if shuffle:
+            rng.shuffle(groups)
+        yield from groups
+        return
+    for i in range(0, n, bs):
+        yield idx[i:i + bs]
+
+
+def trim(x, *others):
+    """Cut right padding to the longest row in the batch (sequences are left-aligned)."""
+    L = max(1, int((x != PAD).sum(1).max()))
+    return (x[:, :L],) + tuple(o[:, :L] for o in others)
+
+
+@torch.no_grad()
+def screen_scores(model, X, bs=4096):
+    model.eval()
+    return np.concatenate([torch.sigmoid(model(trim(X[i:i + bs])[0].to(DEV))).cpu().numpy()
+                           for i in range(0, len(X), bs)])
+
+
+@torch.no_grad()
+def lm_logprob(model, X, M, bs=2048):
+    """Mean log-prob per target token of each row's child (+EOS) given its parent."""
+    model.eval()
+    out = []
+    for i in range(0, len(X), bs):
+        x, m = trim(X[i:i + bs], M[i:i + bs])
+        x, m = x.to(DEV), m.to(DEV)
+        logp = F.log_softmax(model(x[:, :-1]), -1)
+        tgt = x[:, 1:]
+        lp = logp.gather(-1, tgt.unsqueeze(-1)).squeeze(-1) * m[:, :-1]
+        out.append((lp.sum(1) / m[:, :-1].sum(1).clamp(min=1)).cpu().numpy())
+    return np.concatenate(out)
+
+
+def train_screen(Xtr, ytr, Xva, yva, V, epochs, log, rng):
+    model = Screen(V).to(DEV)
+    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
+    pos_w = ((1 - ytr.mean()) / ytr.mean()).detach().to(DEV)
+    best, best_state = -1, None
+    for ep in range(epochs):
+        model.train()
+        t0, tot = time.time(), 0.0
+        lengths = (Xtr != PAD).sum(1).tolist()
+        for b in batches(len(Xtr), 512, True, rng, lengths):
+            logits = model(trim(Xtr[b])[0].to(DEV))
+            loss = F.binary_cross_entropy_with_logits(logits, ytr[b].to(DEV), pos_weight=pos_w)
+            opt.zero_grad()
+            loss.backward()
+            nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+            opt.step()
+            tot += loss.item() * len(b)
+        va = auroc(screen_scores(model, Xva), yva.numpy())
+        log(f"screen epoch {ep + 1}/{epochs} loss {tot / len(Xtr):.4f} val_auroc {va:.4f} ({time.time() - t0:.0f}s)")
+        if va > best:
+            best, best_state = va, {k: v.clone() for k, v in model.state_dict().items()}
+    model.load_state_dict(best_state)
+    return model, best
+
+
+def train_generator(Xtr, Mtr, Xva, Mva, V, epochs, log, rng):
+    model = Generator(V).to(DEV)
+    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
+    best, best_state = -1e9, None
+    for ep in range(epochs):
+        model.train()
+        t0, tot, cnt = time.time(), 0.0, 0.0
+        lengths = (Xtr != PAD).sum(1).tolist()
+        for b in batches(len(Xtr), 256, True, rng, lengths):
+            x, m = trim(Xtr[b], Mtr[b])
+            x, m = x.to(DEV), m.to(DEV)
+            logits = model(x[:, :-1])
+            nll = F.cross_entropy(logits.reshape(-1, V), x[:, 1:].reshape(-1), reduction="none")
+            nll = (nll.view(x.shape[0], -1) * m[:, :-1]).sum() / m[:, :-1].sum()
+            opt.zero_grad()
+            nll.backward()
+            nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+            opt.step()
+            tot += nll.item()
+            cnt += 1
+        va = float(lm_logprob(model, Xva, Mva).mean())
+        log(f"generator epoch {ep + 1}/{epochs} nll {tot / cnt:.4f} val_logprob/token {va:.4f} ({time.time() - t0:.0f}s)")
+        if va > best:
+            best, best_state = va, {k: v.clone() for k, v in model.state_dict().items()}
+    model.load_state_dict(best_state)
+    return model, best
+
+
+@torch.no_grad()
+def sample_children(model, parents_ids, vocab, n, temperature=0.9, rng_seed=0):
+    torch.manual_seed(rng_seed)
+    model.eval()
+    out = []
+    for p in parents_ids[:n]:
+        seq = [BOS] + p + [SEP]
+        child = []
+        for _ in range(MAXG + 1):
+            logits = model(torch.tensor([seq], device=DEV))[0, -1] / temperature
+            logits[[PAD, CLS, SEP, BOS, UNK]] = -1e9
+            tok = int(torch.multinomial(F.softmax(logits, -1), 1))
+            if tok == EOS:
+                break
+            child.append(tok)
+            seq.append(tok)
+        out.append([vocab[t] for t in child])
+    return out
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("data")
+    ap.add_argument("outdir")
+    ap.add_argument("--threads", type=int, default=os.cpu_count())
+    ap.add_argument("--epochs-screen", type=int, default=4)
+    ap.add_argument("--epochs-gen", type=int, default=6)
+    ap.add_argument("--max-train", type=int, default=0, help="cap training rows (0 = all), for a smoke run")
+    ap.add_argument("--samples", type=int, default=500)
+    ap.add_argument("--device", default="cpu")
+    a = ap.parse_args()
+    global DEV
+    DEV = torch.device(a.device)
+    torch.set_num_threads(a.threads)
+    os.makedirs(a.outdir, exist_ok=True)
+    logf = open(os.path.join(a.outdir, "train.log"), "a", encoding="utf-8")
+
+    def log(msg):
+        line = f"{time.strftime('%H:%M:%S')} {msg}"
+        print(line, flush=True)
+        logf.write(line + "\n")
+        logf.flush()
+
+    random.seed(0)
+    np.random.seed(0)
+    torch.manual_seed(0)
+    rng = random.Random(0)
+
+    rows = load(a.data)
+    tr, va, te, cuts = split_by_epoch(rows)
+    if a.max_train:
+        rng.shuffle(tr)
+        tr = tr[:a.max_train]
+    vocab = build_vocab(tr)
+    idx = {t: i for i, t in enumerate(vocab)}
+    train_children = {r["child_id"] for r in tr}
+    novel = np.array([r["child_id"] not in train_children for r in te])
+    log(f"rows {len(rows)} train {len(tr)} val {len(va)} test {len(te)} (novel test {int(novel.sum())}); "
+        f"epoch cuts {cuts}; vocab {len(vocab)}; train base rate {np.mean([r['label'] for r in tr]):.3f}")
+
+    # baselines
+    yte = np.array([r["label"] for r in te])
+    ftr = np.array([features(r) for r in tr[:200000]], dtype=np.float64)
+    fte = np.array([features(r) for r in te], dtype=np.float64)
+    results = {"split": {"epoch_cuts": cuts, "train": len(tr), "val": len(va), "test": len(te),
+                         "novel_test": int(novel.sum())}, "test": [], "test_novel": []}
+    lr_scores = logistic(ftr, np.array([r["label"] for r in tr[:200000]]), fte)
+    rule = fte[:, -1]
+    for name, s in (("baseline: has alloc+copy+fork", rule), ("baseline: logistic on 15 hand features", lr_scores)):
+        results["test"].append(report(name, s, yte))
+        results["test_novel"].append(report(name, s[novel], yte[novel]))
+        log(f"{name}: test auroc {results['test'][-1]['auroc']:.4f}")
+
+    # screen
+    Xtr, ytr = screen_tensor(tr, idx)
+    Xva, yva = screen_tensor(va, idx)
+    Xte, _ = screen_tensor(te, idx)
+    screen, best = train_screen(Xtr, ytr, Xva, yva, len(vocab), a.epochs_screen, log, rng)
+    s = screen_scores(screen, Xte)
+    results["test"].append(report("gridlm screen", s, yte))
+    results["test_novel"].append(report("gridlm screen", s[novel], yte[novel]))
+    log(f"screen: test auroc {results['test'][-1]['auroc']:.4f} (novel {results['test_novel'][-1]['auroc']:.4f})")
+    torch.save({k: v.cpu() for k, v in screen.state_dict().items()}, os.path.join(a.outdir, "screen.pt"))
+
+    # generator: successful children only
+    trp = [r for r in tr if r["label"] == 1]
+    vap = [r for r in va if r["label"] == 1]
+    Gtr, Mtr = lm_tensor(trp, idx)
+    Gva, Mva = lm_tensor(vap, idx)
+    gen, _ = train_generator(Gtr, Mtr, Gva, Mva, len(vocab), a.epochs_gen, log, rng)
+    torch.save({k: v.cpu() for k, v in gen.state_dict().items()}, os.path.join(a.outdir, "generator.pt"))
+    Gte, Mte = lm_tensor(te, idx)
+    lp = lm_logprob(gen, Gte, Mte)
+    results["test"].append(report("gridlm generator log p(child|parent)", lp, yte))
+    results["test_novel"].append(report("gridlm generator log p(child|parent)", lp[novel], yte[novel]))
+    log(f"generator ranking: test auroc {results['test'][-1]['auroc']:.4f}")
+
+    # samples: validity, novelty, and the screen's opinion (model judging model: indicative only)
+    test_parents = [enc_pair(r, idx)[0] for r in te]
+    rng.shuffle(test_parents)
+    kids = sample_children(gen, test_parents, vocab, a.samples)
+    parents_txt = [[vocab[t] for t in p] for p in test_parents[:a.samples]]
+    valid = [1 <= len(k) <= MAXG for k in kids]
+    afc = [all(op in k for op in ("alloc", "copy", "fork")) for k in kids]
+    same = [k == p for k, p in zip(kids, parents_txt)]
+    fake = [{"parent": p, "child": k, "label": 0, "child_id": ""} for p, k in zip(parents_txt, kids) if 1 <= len(k) <= MAXG]
+    Xs, _ = screen_tensor(fake, idx)
+    gen_scores = screen_scores(screen, Xs) if len(fake) else np.array([])
+    real_scores = s[: len(fake)]
+    results["samples"] = {
+        "n": len(kids), "valid_length": float(np.mean(valid)), "has_alloc_copy_fork": float(np.mean(afc)),
+        "identical_to_parent": float(np.mean(same)),
+        "screen_mean_p_generated": float(gen_scores.mean()) if len(gen_scores) else None,
+        "screen_mean_p_real_test_children": float(real_scores.mean()),
+        "note": "screen scores on generated children are one model judging another; only a live colony is ground truth",
+    }
+    with open(os.path.join(a.outdir, "samples.jsonl"), "w", encoding="utf-8") as f:
+        for p, k in zip(parents_txt, kids):
+            f.write(json.dumps({"parent": p, "child": k}) + "\n")
+    results["vocab"] = vocab
+    results["config"] = {"d_model": 128, "layers": 4, "heads": 4, "ff": 512, "max_genome": MAXG,
+                         "label": "child genome born >= 2 times in its epoch",
+                         "screen_params": sum(p.numel() for p in screen.parameters()),
+                         "generator_params": sum(p.numel() for p in gen.parameters())}
+    with open(os.path.join(a.outdir, "results.json"), "w", encoding="utf-8") as f:
+        json.dump(results, f, indent=1)
+    with open(os.path.join(a.outdir, "vocab.json"), "w", encoding="utf-8") as f:
+        json.dump(vocab, f)
+    log("done: " + json.dumps(results["samples"]))
+
+
+if __name__ == "__main__":
+    main()

vocab.json

@@ -0,0 +1 @@
+["[PAD]", "[CLS]", "[SEP]", "[BOS]", "[EOS]", "[UNK]", "add", "alloc", "build", "burn", "call", "copy", "copyn", "corrupt", "dec", "define", "fetch", "fork", "free", "harvest", "ifnotdone", "ifzero", "inc", "input", "jmpb", "jmpr", "link", "listen", "load", "locate", "macro0", "macro1", "macro2", "macro3", "macro4", "macro5", "macro6", "macro7", "move", "nand", "nop", "offer", "output", "peek", "post", "publish", "push", "salvage", "scan", "signal", "steal", "store", "sub", "swap", "write", "xor"]