pylexnet: 827-number Python highlighter network

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+demo.mp4 filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1 @@
+Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

README.md

@@ -0,0 +1,77 @@
+---
+license: apache-2.0
+tags:
+- syntax-highlighting
+- python
+- hand-designed
+- no-training
+---
+
+# pylexnet
+
+A Python syntax highlighter that is a recurrent neural network with **827 numbers and no training data**. Every unit was designed for a job, and the weights were written by construction, not learned.
+
+Inspired by [alpindale/lexnet](https://huggingface.co/alpindale/lexnet), which does this for C. Designed with Claude Fable 5.1 and Claude Opus 5.5.
+
+<video src="https://huggingface.co/Pranavz/pylexnet/resolve/main/demo.mp4" controls width="100%"></video>
+
+The colours in the video are the network's own output.
+
+## Try it
+
+```bash
+git clone https://huggingface.co/Pranavz/pylexnet && cd pylexnet
+cc -O2 -o pylexnet pylexnet.c -lm
+./pylexnet yourfile.py > /dev/null          # default output is ANSI colour; drop the redirect to see it
+./pylexnet --classes yourfile.py            # one class id per byte
+./pylexnet --interp --classes yourfile.py   # generic reference evaluation of the same weights
+```
+
+## Three builds
+
+| Build | Numbers | ns/byte, 1 core | Stdlib held-out: bytes, files exact | Third-party held-out: bytes, files exact |
+|---|---|---|---|---|
+| **main, smallest** | **827** | 94 | 99.996%, 264 of 293 | 99.995%, 3,059 of 3,177 |
+| `builds/fast-994` | 994 (1,502 if its tied threshold weight is counted per row) | **63** | 99.991%, 259 of 293 | 99.9945%, 3,036 of 3,177 |
+| `builds/accurate-7370`, a compiled automaton, not hand-designed | 7,370 | ~89 | ~100.000%, 292 of 293 | **99.9992%, 3,175 of 3,177** |
+
+"Numbers" counts every non-zero value the network uses: embedding, weights, biases, clip ceilings, head, initial state.
+
+Agreement is per byte against Pygments 2.17.2 `PythonLexer`, over 9 classes: text, keyword, builtin, string, comment, number, function, class, decorator. Stdlib held-out is half the Python 3.12 standard library by path hash (5.8 MB). Third-party held-out is 16 packages, 30 MB, not used for rule fixing. Each was run once per build at the end.
+
+**Agreement with Pygments isn't correctness.** On the third-party held-out set, CPython's tokenizer sided with the `accurate-7370` build on all 255 bytes where it differs from Pygments. That audit hasn't been run on the small builds.
+
+## Speed, measured fairly
+
+One machine (Intel i5-9400F), one core, `-O2`, the same 293 held-out Python files, best of runs per file, start-up excluded. Ruff is Rust at `opt-level=3`.
+
+| Tool | ns/byte | Does |
+|---|---|---|
+| The `accurate-7370` rules as a plain lookup table, no network | 5.5 | Same labels as that build |
+| Ruff parser 0.8.6 | 14 | Full lexing and AST, no highlight classes |
+| `builds/accurate-7370` predecessor | 90.7 | Labels bytes |
+| tree-sitter-python, parse only | 106.7 | Full syntax tree |
+| tree-sitter-python, parse + highlights query | 164.8 | Tree and highlight captures |
+| Pygments `PythonLexer` | 1,153 | Tokens |
+
+The three builds above were timed on `argparse.py` (100 KB). The corpus-wide row is an earlier build of the accurate network. This is not the fastest way to lex Python: a plain table and Ruff are both faster, and tree-sitter builds a tree that this doesn't. The point is that a tiny network with no training data can do this job at all.
+
+## How it works
+
+- **Embedding:** 18 byte features, including a character code and string-prefix letters.
+- **Forward cell:** about 55 logic units, each designed: inside-string, quote-run counter, triple-quote, f-string and escape flags, one saved outer level for f-string fields with bracket depth, comment, number with exponent, word and length, and `def`/`class`/decorator context.
+- **Vocabulary:** additive retrieval. Solved value tables over byte position sum to a class code for each of 269 keywords, builtins and special method names, in 249 numbers. Ten threshold units read the class windows.
+- **Backward cell:** carries a word's class back over its bytes, because a word's class is only known at its end.
+- **Runtime:** `pylexnet_kernel.h` is the architecture generated as C, like lexnet's. Weight values live only in `pylexnet_weights.bin`. The loader checks a structure hash and guard inequalities. `--interp` runs the generic evaluation of the same weights.
+
+`compile.py` regenerates everything: `python3 compile.py --matcher add` for the main build, `--matcher hash --vocab 220` for `fast-994`.
+
+## Limits
+
+- **Approximate vocabulary in the main build:** a few non-keywords are flagged, such as `an` and `ad`. That was 185 bytes in a 74.8 MB tuning corpus.
+- **`fast-994` keeps 220 of the words** and is exact on those, ranked by frequency in the tuning corpus.
+- **Missing in the small builds:** soft keywords `match`/`case`, the space in `yield from`, two-level nested f-strings, `{x=}`, and `.5` floats after a dot. These are why their exact-file counts trail the accurate build.
+- **Weights come from a generator script:** the network is designed by hand, unit by unit, but the numbers are produced by `compile.py`, not typed out.
+- **Python 3.12 vocabulary, one Pygments version.** Results don't transfer automatically to other versions.
+- **Speeds are from one machine.** The builds table is one 100 KB file.
+- The tools in `tools/` were written for the development setup and may need paths adjusted.

builds/accurate-7370/compile.py

@@ -0,0 +1,371 @@
+"""Compile the rules into exact network weights. No training.
+
+The network is a feed-forward stack of sparse blocks with skip connections,
+run once per byte. Input x = [mode one-hot | registers | matcher state | byte one-hot].
+  B1 (ReLU)   byte class c_k = sum of its bytes; base mode = relu(1 - other modes);
+              matcher node = sum of the edges into it (only where that is cheaper)
+  B2 (ReLU)   AND units: p = relu(mode_q + c_k - 1) (or relu(mode_q - other classes) when
+              the cell is large); word category f_w = sum of matcher nodes (or the edges
+              into them) that end a word of category w
+  B3 (ReLU)   AND units: relu(p + register_v [+ f_w] - (arity - 1))
+  B4 (ReLU)   next mode, register deltas, label and event one-hots, and the word
+              matcher: edge = relu(source + byte - 1), where the source is a node unit
+              (sum of incoming edges) or the incoming edges themselves; the state carries
+              the edge that fired, not a node one-hot. Every B4 output is 0 or 1.
+Backward cell, per byte, on one-hot register r:
+  r' = event + relu(r - any_event) + sum over map events m, classes c of relu(r_c + m - 1) e_{M_m(c)}
+Head: class = label + relu(wordlabel + r - 1)
+
+The mode, the registers (context, f-string depth and quote, quote kind, triple,
+f-prefix) and the label are one-hot with the most common value encoded as all zeros, so
+nothing fires in the common case. A unit that depends on register values is derived by
+Mobius inversion of the rule table over the register lattice: it only ever conditions on
+non-base values.
+"""
+import struct
+from collections import Counter
+
+import numpy as np
+
+import spec
+
+NF = 11
+ORDER, ROWS = spec.build_table()
+IDS = {st: i for i, st in enumerate(ORDER)}
+
+# --- 1. factor each rule state into (mode, registers) ----------------------------
+FIELDS = {"code": ("ctx", "fr"), "word": ("ctx", "fr"), "at": ("ctx", "fr"),
+          "bang": ("fr",), "bang2": ("fr",), "dot": ("fr",), "cmt": ("fr",),
+          "int": ("fr",), "hex": ("fr",), "frac": ("fr",), "exp0": ("fr",), "exp": ("fr",), "numend": ("fr",),
+          "q1": ("q", "f", "fr"), "q2": ("q", "f", "fr"),
+          "str": ("q", "tr", "close", "f", "fr"), "fopen": ("q", "tr", "fr"), "esc": ("q", "tr", "f", "fr"),
+          "escn": ("q", "tr", "f", "fr"), "nname": ("q", "tr", "f", "fr")}
+INMODE = {"close"}  # fields that stay part of the mode name
+# The f-string context fr = None | (quote, triple, depth, outer) is factored into three registers:
+# fd (bracket depth, one-hot over 0..3, all zero outside an f-string expression), fq and
+# ftr (the enclosing string's quote and triple), so units condition on single bits.
+BASE = {"ctx": "none", "fd": None, "fq": 34, "ftr": 0, "od": None, "oq": 34, "otr": 0, "q": 34, "tr": 0, "f": False}
+REGS = tuple(BASE)
+BASE_MODE = "code"
+BASE_LABEL = spec.TEXT
+
+
+def factor(st):
+    name, regs = st[0], {}
+    for fld, v in zip(FIELDS[st[0]], st[1:]):
+        if fld in INMODE:
+            name = f"{name}{v}"
+        elif fld == "fr":
+            if v is not None:
+                regs["fq"], regs["ftr"], regs["fd"] = v[:3]
+                if v[3] is not None:  # the field around this one: nested f-strings
+                    regs["oq"], regs["otr"], regs["od"] = v[3]
+        else:
+            regs[fld] = v
+    return name, regs
+
+
+modes, states_of = {}, {}
+rvals = {x: set() for x in REGS}
+for st in ORDER:
+    q, regs = factor(st)
+    modes.setdefault(q, len(modes))
+    states_of.setdefault(q, []).append(st)
+    for x, v in regs.items():
+        rvals[x].add(v)
+M = len(modes)
+midx = {}  # mode -> input column offset; the base mode has none
+for q in modes:
+    if q != BASE_MODE:
+        midx[q] = len(midx)
+ridx = {}
+for x in REGS:
+    for v in sorted(rvals[x] - {BASE[x]}, key=str):
+        ridx[(x, v)] = len(ridx)
+R = len(ridx)
+
+# --- 2. byte classes: bytes the rules never tell apart -----------------------------
+sig = {}
+bclass = [0] * 256
+for b in range(256):
+    key = tuple(ROWS[s][f][b] for s in range(len(ORDER)) for f in range(NF))
+    bclass[b] = sig.setdefault(key, len(sig))
+C = len(sig)
+member = {}
+for b, k in enumerate(bclass):
+    member.setdefault(k, b)
+
+# --- 3. minimise the word matcher: merge trie nodes with identical futures ---------
+T, TERM = spec.TRIE, spec.TERM
+cls = {n: TERM.get(n, spec.F_NONE) for n in range(len(T))}
+while True:
+    key = {n: (cls[n], tuple(sorted((b, cls[c]) for b, c in T[n].items()))) for n in range(len(T))}
+    new_ids = {}
+    nxt = {n: new_ids.setdefault(key[n], len(new_ids)) for n in range(len(T))}
+    if len(new_ids) == len(set(cls.values())):
+        break
+    cls = nxt
+order = sorted(set(cls.values()), key=lambda c: c != cls[0])  # root first
+node_id = {c: i for i, c in enumerate(order)}
+N = len(order)
+EDGES = sorted({(node_id[cls[n]], b, node_id[cls[c]]) for n in range(len(T)) for b, c in T[n].items()})
+NTERM = {}
+for n, w in TERM.items():
+    NTERM[node_id[cls[n]]] = w
+
+# --- 3b. matcher state = the edge that fired last. An edge unit is an AND of its byte(s)
+# and its source; the source is either a materialised node unit (sum of its incoming
+# edges) or, when that is cheaper, the incoming edges themselves. Edges with the same
+# source and target share one unit, since their bytes are mutually exclusive.
+EBYTES = {}
+for i, b, c in EDGES:
+    EBYTES.setdefault((i, c), []).append(b)
+EUNITS = sorted(EBYTES)
+IN_OF, OUT_OF = {}, {}
+for i, c in EUNITS:
+    IN_OF.setdefault(c, []).append((i, c))
+    OUT_OF.setdefault(i, []).append((i, c))
+MAT = set()  # no node units: a state-derived hop before the edges costs more time than weights
+SLOT = {("node", 0): 0}
+for e in EUNITS:
+    SLOT[("edge",) + e] = len(SLOT)
+NS = len(SLOT)
+
+
+def node_cols(n):
+    """Columns whose sum is the one-hot "matcher is at node n"."""
+    if n == 0:
+        return [oN + SLOT[("node", 0)]]
+    if n in MAT:
+        return [gidx(B1, NODE_ROW[n])]
+    return [oN + SLOT[("edge",) + e] for e in IN_OF.get(n, [])]
+
+
+# --- layout -----------------------------------------------------------------------
+oM, oR, oN = 0, len(midx), len(midx) + R
+NSTATE = oN + NS
+oB = NSTATE
+NIN = oB + 256
+LAB, EV = 10, 9 + len(spec.EV_MAP)
+oL, oE = NSTATE, NSTATE + LAB
+NOUT = oE + EV
+
+
+class Block:
+    def __init__(self, relu):
+        self.relu, self.rows, self.bias, self.entries = relu, 0, [], []
+
+    def unit(self, inputs, bias=0.0):
+        """inputs: list of (global column, weight)."""
+        i = self.rows
+        self.rows += 1
+        self.bias.append(bias)
+        for j, w in inputs:
+            self.entries.append((i, j, w))
+        return i
+
+
+B1, B2, B3, B4 = Block(1), Block(1), Block(1), Block(1)
+blocks = [B1, B2, B3, B4]
+base_of = {}
+
+
+def begin(blk):
+    """Fix the block's place in the global index space: inputs, then blocks in order.
+    Called once the previous blocks are complete."""
+    base_of[id(blk)] = NIN + sum(b.rows for b in blocks[:blocks.index(blk)])
+
+
+def gidx(blk, i):
+    return base_of[id(blk)] + i
+
+
+# --- B1: byte classes, base mode, materialised matcher nodes ----------------------
+begin(B1)
+CLS_ROW = {}
+for k in range(C):
+    CLS_ROW[k] = B1.unit([(oB + b, 1.0) for b in range(256) if bclass[b] == k])
+WORD_CLS = sorted({bclass[b] for b in spec.WORDCH})  # classes never mix word and non-word bytes
+MODE0 = B1.unit([(oM + i, -1.0) for i in range(len(midx))], 1.0)
+NODE_ROW = {}
+for n in sorted(MAT):
+    NODE_ROW[n] = B1.unit([(oN + SLOT[("edge",) + e], 1.0) for e in IN_OF[n]])
+begin(B2)
+FLAG_ROW = {}
+for w in range(1, NF):
+    FLAG_ROW[w] = B2.unit([(j, 1.0) for i, t in sorted(NTERM.items()) if t == w for j in node_cols(i)])
+
+
+def mode_col(q):
+    return gidx(B1, MODE0) if q == BASE_MODE else oM + midx[q]
+
+
+# --- the rule table on factored states --------------------------------------------
+
+
+def outcome(st, f, k):
+    """Output vector for (state, word category, byte class): {output row: weight}."""
+    n, lab, ev = ROWS[IDS[st]][f][member[k]]
+    q2, regs2 = factor(ORDER[n])
+    _, regs = factor(st)
+    vec = Counter()
+    if q2 != BASE_MODE:
+        vec[oM + midx[q2]] = 1.0
+    if lab != BASE_LABEL:
+        vec[oL + lab] = 1.0
+    if ev >= 0:
+        vec[oE + ev] = 1.0
+    for x in REGS:
+        old, new = regs.get(x, BASE[x]), regs2.get(x, BASE[x])
+        if old != new:
+            if new != BASE[x]:
+                vec[oR + ridx[(x, new)]] += 1.0
+            if old != BASE[x]:
+                vec[oR + ridx[(x, old)]] -= 1.0
+    return tuple(sorted((r, v) for r, v in vec.items() if v))
+
+
+def nonbase(st, f):
+    _, regs = factor(st)
+    t = {(x, v) for x, v in regs.items() if v != BASE[x]}
+    if f != spec.F_NONE:
+        t.add(("flag", f))
+    return frozenset(t)
+
+
+def mobius(F):
+    """F: {set of non-base conditions: vector}. Returns corrections g with
+    F[T] = sum of g[U] over U subset of T, using only conditions that occur."""
+    g = {}
+    for Tset in sorted(F, key=len):
+        pred = Counter()
+        for U, gu in g.items():
+            if U < Tset:
+                pred.update(dict(gu))
+        d = Counter(dict(F[Tset]))
+        d.subtract(pred)
+        d = tuple(sorted((r, v) for r, v in d.items() if v))
+        if d:
+            g[Tset] = d
+    return g
+
+
+def cond_col(c):
+    x, v = c
+    return gidx(B2, FLAG_ROW[v]) if x == "flag" else oR + ridx[(x, v)]
+
+
+# --- cells: for each mode, the byte classes it treats alike; Mobius corrections ----
+CELLS, PLAN = {}, {}
+for q in modes:
+    cells = {}
+    for k in range(C):
+        s = tuple(outcome(st, f, k) for st in states_of[q] for f in range(NF))
+        cells.setdefault(s, []).append(k)
+    CELLS[q] = cells
+    for s, ks in cells.items():
+        F = {}
+        for st in states_of[q]:
+            for f in range(NF):
+                F[nonbase(st, f)] = outcome(st, f, ks[0])
+        PLAN[(q, tuple(ks))] = mobius(F)
+
+# --- B2: mode AND units p = relu(mode + class - 1), only for cells that do something --
+P_ROW = {}
+for q in modes:
+    cells = CELLS[q]
+    for s, ks in cells.items():
+        if not PLAN[(q, tuple(ks))]:
+            continue
+        # p is a global column: the mode itself when it has one cell; otherwise an AND
+        # unit over the cell's classes, or over the complement when that is smaller:
+        # relu(mode - sum of other classes) is the same function.
+        if len(cells) == 1:
+            P_ROW[(q, tuple(ks))] = mode_col(q)
+        elif len(ks) <= C - len(ks):
+            P_ROW[(q, tuple(ks))] = gidx(B2, B2.unit([(mode_col(q), 1.0)] + [(gidx(B1, CLS_ROW[k]), 1.0) for k in ks], -1.0))
+        else:
+            P_ROW[(q, tuple(ks))] = gidx(B2, B2.unit([(mode_col(q), 1.0)] + [(gidx(B1, CLS_ROW[k]), -1.0) for k in range(C) if k not in ks]))
+begin(B3)
+
+# --- B3: register-conditioned corrections, shared between cells with the same one --
+cond_units = {}  # (conditions, correction vector) -> list of p columns feeding it
+for (q, ks), g in PLAN.items():
+    if not g:
+        continue
+    p = P_ROW[(q, ks)]
+    for Tset, gv in g.items():
+        if not Tset:
+            for r, v in gv:
+                B4.entries.append((r, p, v))
+        else:
+            cond_units.setdefault((tuple(sorted(Tset, key=str)), gv), []).append(p)
+for (conds, gv), ps in cond_units.items():
+    u = B3.unit([(p, 1.0) for p in ps] + [(cond_col(c), 1.0) for c in conds], -float(len(conds)))
+    for r, v in gv:
+        B4.entries.append((r, gidx(B3, u), v))
+begin(B4)
+
+# --- B4: outputs. Registers pass through unless a unit changes them. Matcher: edge units
+# (AND of source and byte); the root is on after a non-word byte.
+for i in range(R):
+    B4.entries.append((oR + i, oR + i, 1.0))
+B4.rows = NOUT
+B4.bias = [0.0] * NOUT
+B4.bias[oN + 0] = 1.0
+for k in WORD_CLS:
+    B4.entries.append((oN + 0, gidx(B1, CLS_ROW[k]), -1.0))
+for i, c in EUNITS:
+    r = oN + SLOT[("edge", i, c)]
+    B4.bias[r] = -1.0
+    for j in node_cols(i):
+        B4.entries.append((r, j, 1.0))
+    for b in EBYTES[(i, c)]:
+        B4.entries.append((r, oB + b, 1.0))
+
+
+# --- pack -------------------------------------------------------------------------
+def dedupe(entries):
+    acc = {}
+    for i, j, v in entries:
+        acc[(i, j)] = acc.get((i, j), 0.0) + v
+    return [(i, j, v) for (i, j), v in acc.items() if v != 0.0]
+
+
+def pack(blk, rbase):
+    """Column-compressed over the global columns this block reads."""
+    ent = sorted(dedupe(blk.entries), key=lambda e: (e[1], e[0]))
+    lo = min(j for _, j, _ in ent)
+    hi = max(j for _, j, _ in ent) + 1
+    ptr = np.zeros(hi - lo + 1, dtype=np.int32)
+    for _, j, _ in ent:
+        ptr[j - lo + 1] += 1
+    ptr = np.cumsum(ptr).astype(np.int32)
+    idx = np.array([e[0] for e in ent], dtype=np.int32)
+    val = np.array([e[2] for e in ent], dtype=np.float32)
+    head = struct.pack("<6i", blk.rows, lo, hi, len(ent), blk.relu, rbase)
+    return len(ent), head + ptr.tobytes() + idx.tobytes() + val.tobytes() + np.asarray(blk.bias, dtype=np.float32).tobytes()
+
+
+start_mode, start_regs = factor(spec.START)
+init = ([oM + midx[start_mode]] if start_mode != BASE_MODE else []) \
+    + [oR + ridx[(x, v)] for x, v in start_regs.items() if v != BASE[x]] + [oN + 0]
+BOUND = 1  # every activation in this network is 0 or 1
+header = struct.pack("<10i", NIN, NSTATE, len(blocks), len(init), oL, LAB, oE, EV, oB, BOUND) + np.array(init, dtype=np.int32).tobytes() \
+    + np.array(spec.EV_MAP, dtype=np.int32).tobytes()
+blob = header
+nnz = 0
+per = []
+rbase = NIN
+for blk in blocks:
+    assert base_of[id(blk)] == rbase
+    n, packed = pack(blk, rbase)
+    nnz += n
+    per.append(n)
+    blob += packed
+    rbase += blk.rows
+open("pylexnet_weights.bin", "wb").write(blob)
+print(f"modes {M}, registers {R}, byte classes {C}, matcher nodes {N} ({len(EDGES)} edges, {len(EUNITS)} edge units, {len(MAT)} node units)")
+print(f"AND units: mode {B2.rows - (NF - 1)}, conditioned {B3.rows}, matcher {len(EUNITS)}")
+print(f"non-zero weights {nnz} (blocks {per}), rows {[b.rows for b in blocks]}")
+print(f"weights file {len(blob) / 1024:.0f} KiB")

builds/accurate-7370/pylexnet.c

@@ -0,0 +1,467 @@
+/* pylexnet: a recurrent network with hand-compiled weights that classifies Python bytes.
+ * This runtime knows nothing about Python: it only does sparse matrix products and ReLU.
+ *
+ * The forward cell is a stack of sparse blocks with skip connections over one global
+ * activation vector: inputs [state | byte one-hot], then each block's rows in order.
+ * Block k computes y_k = act(W_k a + b_k) where a is everything computed so far.
+ * The last block's first rows are the next state; then the label and event one-hots.
+ *
+ * Two kernels compute the same products:
+ *   --reference  the plain one: scatter every active column, track touched rows with flags.
+ *   default      splits the columns of every block into "known" and "unknown" ones.
+ *                The byte one-hot columns are known: the runtime builds that input itself,
+ *                so their values given the byte are known before any weight is read. A row
+ *                whose every entry sits in known columns is a pure function of the byte, so
+ *                its column is known too (B1's byte-class rows, for instance). At load time
+ *                the kernel evaluates, for each of the 256 bytes, the known part of every
+ *                row: K_i(b) = sum of w_ij * a_j(b) over known j, and act(b_i + K_i(b)).
+ *                Per byte it then scatters only the unknown columns (the state and the rows
+ *                derived from it), and completes each touched row with its K_i(b). Rows not
+ *                touched by any unknown column produce act(b_i + K_i(b)), which is the
+ *                precomputed value; the ones where it is non-zero are emitted from a per-byte
+ *                "loud" list. Every row still receives exactly W_i . a + b_i; the split only
+ *                moves the one-hot part of the product to load time.
+ *                On top of that, dead-row pruning: the weights file declares a bound U on every
+ *                activation. For byte b, a ReLU row whose bias + K_i(b) + (largest possible
+ *                contribution of its unknown columns under U) is still <= 0 is provably zero, so
+ *                its entries are left out of that byte's scatter, and its own column counts as
+ *                zero for the rows downstream. The kernel checks every activation it produces
+ *                against U; if one ever exceeds it, the file is redone with --reference. Nothing
+ *                here depends on the state: the state still flows through every block's weights. */
+#include <math.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+
+typedef struct { int i; float w; } Ent;
+typedef struct { int c; float v; } Act;
+typedef struct {
+    int rows, lo, hi, nnz, relu, base;   /* columns [lo, hi) of the global vector; rows live at base */
+    const int *ptr, *idx; const float *val, *bias;
+    int *touched, *brow, nb; char *seen; float *y;
+    /* fast kernel */
+    int *uptr, *uidx; float *uval;       /* CSC over the unknown columns only */
+    int *kptr, *kcol; float *kval;       /* CSR, per row, of its entries in known columns */
+    int *lptr, *lrow; float *lval;       /* per byte: rows with act(bias + K_i(b)) != 0, and that value */
+    signed char *ktab; int hask;         /* K_i(b) as a dense int8 table [b * rows + i] when it fits, for rows touched by unknown columns */
+    int nvar, *vmap;                     /* per byte: which pruned CSC variant to scatter with */
+    int **vptr, **vidx; float **vval;    /* pruned CSC variants: only entries of rows that can be non-zero for that byte */
+    Ent **vent;                          /* the same entries interleaved (row, weight); vptr has a sentinel empty column at index w */
+} Block;
+static const unsigned char *P;
+static const void *take(size_t n) { const void *r = P; P += n; return r; }
+static void load(Block *l) {
+    const int *h = take(24);
+    l->rows = h[0]; l->lo = h[1]; l->hi = h[2]; l->nnz = h[3]; l->relu = h[4]; l->base = h[5];
+    l->ptr = take(4 * (size_t)(l->hi - l->lo + 1));
+    l->idx = take(4 * (size_t)l->nnz);
+    l->val = take(4 * (size_t)l->nnz);
+    l->bias = take(4 * (size_t)l->rows);
+    l->touched = malloc(sizeof(int) * (size_t)(l->rows + l->nnz));
+    l->brow = malloc(sizeof(int) * (size_t)l->rows);
+    l->seen = calloc((size_t)l->rows, 1);
+    l->y = malloc(sizeof(float) * (size_t)l->rows);
+    memcpy(l->y, l->bias, sizeof(float) * (size_t)l->rows);
+    l->nb = 0;
+    for (int i = 0; i < l->rows; i++) if (l->bias[i] > 0 || (!l->relu && l->bias[i] != 0)) l->brow[l->nb++] = i;
+}
+static inline float act(int relu, float v) { return relu && v < 0 ? 0 : v; }
+
+/* Known-column analysis. known[j] for every global column; kv[b][...] sparse lists of the
+ * known columns' values for byte b. */
+static char *g_known;
+static int *g_kvptr, *g_kvcol; static float *g_kvval; static int g_kvn, g_kvcap;
+static int g_needkv;
+static float g_U;        /* declared |activation| bound */
+static float g_vmax;     /* largest |activation| the fast kernel has produced */
+static char *g_nonneg;   /* column is a ReLU output */
+static char *g_dead;     /* per byte, columns that are provably zero (dead rows of earlier blocks) */
+static long long g_tb[16]; /* touched rows per block (bench) */
+static float *g_kv;  /* dense scratch over the global vector: values of known columns for the current byte */
+static void kv_push(int col, float v) {
+    if (g_kvn == g_kvcap) { g_kvcap = g_kvcap ? 2 * g_kvcap : 4096; g_kvcol = realloc(g_kvcol, sizeof(int) * (size_t)g_kvcap); g_kvval = realloc(g_kvval, sizeof(float) * (size_t)g_kvcap); }
+    g_kvcol[g_kvn] = col; g_kvval[g_kvn++] = v;
+}
+static void specialise(Block *L, int NB, int total, int oB) {
+    g_known = calloc((size_t)total, 1);
+    g_nonneg = calloc((size_t)total, 1);
+    g_dead = calloc((size_t)total * 256, 1);   /* [b * total + col]: column provably zero for byte b */
+    g_kv = calloc((size_t)total, sizeof(float));
+    for (int b = 0; b < 256; b++) g_known[oB + b] = 1;
+    /* per byte lists of known values; byte columns first */
+    /* temporary per-byte dynamic lists for known values, built block by block */
+    int *tcap = calloc(256, sizeof(int)), *tn = calloc(256, sizeof(int));
+    int **tcol = calloc(256, sizeof(int *)); float **tval = calloc(256, sizeof(float *));
+    for (int b = 0; b < 256; b++) { tcap[b] = 64; tcol[b] = malloc(sizeof(int) * 64); tval[b] = malloc(sizeof(float) * 64); tcol[b][0] = oB + b; tval[b][0] = 1; tn[b] = 1; }
+    #define TPUSH(b, c, v) do { if (tn[b] == tcap[b]) { tcap[b] *= 2; tcol[b] = realloc(tcol[b], sizeof(int) * (size_t)tcap[b]); tval[b] = realloc(tval[b], sizeof(float) * (size_t)tcap[b]); } tcol[b][tn[b]] = (c); tval[b][tn[b]++] = (v); } while (0)
+    for (int k = 0; k < NB; k++) {
+        Block *l = &L[k];
+        int w = l->hi - l->lo;
+        /* row degrees in known columns, and whether the row has any unknown entry */
+        int *kdeg = calloc((size_t)l->rows, sizeof(int));
+        char *hasu = calloc((size_t)l->rows, 1);
+        int nu = 0;
+        for (int j = 0; j < w; j++)
+            for (int e = l->ptr[j]; e < l->ptr[j + 1]; e++) {
+                if (g_known[l->lo + j]) kdeg[l->idx[e]]++; else { hasu[l->idx[e]] = 1; nu++; }
+            }
+        /* CSC over unknown columns */
+        l->uptr = malloc(sizeof(int) * (size_t)(w + 1)); l->uidx = malloc(sizeof(int) * (size_t)(nu + 1)); l->uval = malloc(sizeof(float) * (size_t)(nu + 1));
+        /* CSR over known columns */
+        l->kptr = malloc(sizeof(int) * (size_t)(l->rows + 1));
+        l->kptr[0] = 0;
+        for (int i = 0; i < l->rows; i++) l->kptr[i + 1] = l->kptr[i] + kdeg[i];
+        l->kcol = malloc(sizeof(int) * (size_t)(l->kptr[l->rows] + 1)); l->kval = malloc(sizeof(float) * (size_t)(l->kptr[l->rows] + 1));
+        int *fill = calloc((size_t)l->rows, sizeof(int));
+        nu = 0;
+        for (int j = 0; j < w; j++) {
+            l->uptr[j] = nu;
+            for (int e = l->ptr[j]; e < l->ptr[j + 1]; e++) {
+                int i = l->idx[e];
+                if (g_known[l->lo + j]) { l->kcol[l->kptr[i] + fill[i]] = l->lo + j; l->kval[l->kptr[i] + fill[i]++] = l->val[e]; }
+                else { l->uidx[nu] = i; l->uval[nu++] = l->val[e]; }
+            }
+        }
+        l->uptr[w] = nu;
+        /* rows with no unknown entry are known columns themselves */
+        for (int i = 0; i < l->rows; i++) if (!hasu[i]) g_known[l->base + i] = 1;
+        /* per byte: K_i(b) for every row with known entries; loud rows; values of known rows */
+        l->lptr = malloc(sizeof(int) * 257);
+        int lcap = 1024, ln = 0;
+        l->lrow = malloc(sizeof(int) * (size_t)lcap); l->lval = malloc(sizeof(float) * (size_t)lcap);
+        for (int b = 0; b < 256; b++) {
+            l->lptr[b] = ln;
+            for (int a = 0; a < tn[b]; a++) g_kv[tcol[b][a]] = tval[b][a];
+            for (int i = 0; i < l->rows; i++) {
+                float K = 0;
+                for (int e = l->kptr[i]; e < l->kptr[i + 1]; e++) K += l->kval[e] * g_kv[l->kcol[e]];
+                float v = act(l->relu, l->bias[i] + K);
+                if (!hasu[i] && v != 0) TPUSH(b, l->base + i, v);
+                if (v != 0 && (hasu[i] || k == NB - 1)) {   /* known rows are consumed through K only; last block's are the outputs */
+                    if (ln == lcap) { lcap *= 2; l->lrow = realloc(l->lrow, sizeof(int) * (size_t)lcap); l->lval = realloc(l->lval, sizeof(float) * (size_t)lcap); }
+                    l->lrow[ln] = i; l->lval[ln++] = v;
+                }
+            }
+            for (int a = 0; a < tn[b]; a++) g_kv[tcol[b][a]] = 0;
+        }
+        l->lptr[256] = ln;
+        /* dense int8 table of K_i(b) for rows that mix known and unknown entries */
+        l->hask = 0; l->ktab = NULL;
+        for (int i = 0; i < l->rows; i++) if (hasu[i] && kdeg[i]) l->hask = 1;
+        if (l->hask) {
+            int ok = 1;
+            signed char *tab = calloc((size_t)l->rows * 256, 1);
+            for (int b = 0; b < 256 && ok; b++) {
+                for (int a = 0; a < tn[b]; a++) g_kv[tcol[b][a]] = tval[b][a];
+                for (int i = 0; i < l->rows; i++) {
+                    if (!hasu[i]) continue;
+                    float K = 0;
+                    for (int e = l->kptr[i]; e < l->kptr[i + 1]; e++) K += l->kval[e] * g_kv[l->kcol[e]];
+                    if (K != (float)(int)K || K < -127 || K > 127) ok = 0;
+                    tab[(size_t)b * l->rows + i] = (signed char)(int)K;
+                }
+                for (int a = 0; a < tn[b]; a++) g_kv[tcol[b][a]] = 0;
+            }
+            if (ok) l->ktab = tab; else free(tab);
+        }
+        /* dead-row pruning: for byte b, row i can only be non-zero if
+         *   bias_i + K_i(b) + sum over unknown entries of (max contribution under the bound) > 0   (ReLU rows)
+         * Entries of rows that cannot be non-zero are left out of the scatter for that byte. The
+         * pruned column structures are deduplicated across bytes. */
+        {
+            int *aptr = malloc(sizeof(int) * (size_t)(w + 1)), *aidx = malloc(sizeof(int) * (size_t)(nu + 1));
+            float *aval = malloc(sizeof(float) * (size_t)(nu + 1));
+            float *slack = malloc(sizeof(float) * (size_t)l->rows);   /* max unknown contribution per row, for this byte */
+            char *alive = malloc((size_t)l->rows);
+            l->nvar = 0; l->vmap = malloc(sizeof(int) * 256);
+            l->vptr = malloc(sizeof(int *) * 256); l->vidx = malloc(sizeof(int *) * 256); l->vval = malloc(sizeof(float *) * 256);
+            l->vent = malloc(sizeof(Ent *) * 256);
+            for (int b = 0; b < 256; b++) {
+                const char *db = g_dead + (size_t)b * total;
+                for (int i = 0; i < l->rows; i++) slack[i] = 0;
+                for (int j = 0; j < w; j++) {
+                    if (db[l->lo + j]) continue;   /* a column that is provably zero for this byte contributes nothing */
+                    for (int e = l->uptr[j]; e < l->uptr[j + 1]; e++) {
+                        float wv = l->uval[e];
+                        slack[l->uidx[e]] += (g_nonneg[l->lo + j] ? (wv > 0 ? wv : 0) : (wv < 0 ? -wv : wv)) * g_U;
+                    }
+                }
+                for (int a = 0; a < tn[b]; a++) g_kv[tcol[b][a]] = tval[b][a];
+                for (int i = 0; i < l->rows; i++) {
+                    float K = 0;
+                    for (int e = l->kptr[i]; e < l->kptr[i + 1]; e++) K += l->kval[e] * g_kv[l->kcol[e]];
+                    alive[i] = l->relu ? (l->bias[i] + K + slack[i] > 0) : 1;
+                    if (!alive[i] && hasu[i]) g_dead[(size_t)b * total + l->base + i] = 1;
+                }
+                for (int a = 0; a < tn[b]; a++) g_kv[tcol[b][a]] = 0;
+                int na = 0;
+                for (int j = 0; j < w; j++) {
+                    aptr[j] = na;
+                    if (db[l->lo + j]) continue;
+                    for (int e = l->uptr[j]; e < l->uptr[j + 1]; e++)
+                        if (alive[l->uidx[e]]) { aidx[na] = l->uidx[e]; aval[na++] = l->uval[e]; }
+                }
+                aptr[w] = na;
+                int v;
+                for (v = 0; v < l->nvar; v++)
+                    if (l->vptr[v][w] == na && !memcmp(l->vptr[v], aptr, sizeof(int) * (size_t)(w + 1)) &&
+                        !memcmp(l->vidx[v], aidx, sizeof(int) * (size_t)na) && !memcmp(l->vval[v], aval, sizeof(float) * (size_t)na)) break;
+                if (v == l->nvar) {
+                    l->vptr[v] = malloc(sizeof(int) * (size_t)(w + 2)); memcpy(l->vptr[v], aptr, sizeof(int) * (size_t)(w + 1));
+                    l->vptr[v][w + 1] = na;   /* sentinel: column w is empty, used for columns outside [lo, hi) */
+                    l->vent[v] = malloc(sizeof(Ent) * (size_t)(na + 1));
+                    for (int e = 0; e < na; e++) { l->vent[v][e].i = aidx[e]; l->vent[v][e].w = aval[e]; }
+                    l->vidx[v] = malloc(sizeof(int) * (size_t)(na + 1)); memcpy(l->vidx[v], aidx, sizeof(int) * (size_t)na);
+                    l->vval[v] = malloc(sizeof(float) * (size_t)(na + 1)); memcpy(l->vval[v], aval, sizeof(float) * (size_t)na);
+                    l->nvar++;
+                }
+                l->vmap[b] = v;
+            }
+            free(aptr); free(aidx); free(aval); free(slack); free(alive);
+        }
+        if (l->relu) for (int i = 0; i < l->rows; i++) g_nonneg[l->base + i] = 1;
+        free(kdeg); free(hasu); free(fill);
+    }
+    g_kvptr = malloc(sizeof(int) * 257);
+    for (int b = 0; b < 256; b++) { g_kvptr[b] = g_kvn; for (int a = 0; a < tn[b]; a++) kv_push(tcol[b][a], tval[b][a]); }
+    g_kvptr[256] = g_kvn;
+    g_needkv = 0;
+    for (int k = 0; k < NB; k++) if (L[k].hask && !L[k].ktab) g_needkv = 1;
+    for (int b = 0; b < 256; b++) { free(tcol[b]); free(tval[b]); }
+    free(tcol); free(tval); free(tn); free(tcap);
+}
+
+static long long g_touched;
+static Block *g_L;
+static int g_ref;
+/* y = act(W a + b), visiting only the non-zero entries of a. Appends the non-zero rows
+ * (as global indices) to the active list. */
+static int apply_ref(Block *l, const int *in, const float *inv, int nin, int *out, float *outv) {
+    float *y = l->y;
+    /* y holds the bias outside this call; touched rows are restored afterwards */
+    int nt = 0;
+    for (int a = 0; a < nin; a++) {
+        int j = in[a] - l->lo;
+        if (j < 0 || j >= l->hi - l->lo) continue;
+        float v = inv[a];
+        for (int e = l->ptr[j]; e < l->ptr[j + 1]; e++) {
+            int i = l->idx[e];
+            if (!l->seen[i]) { l->seen[i] = 1; l->touched[nt++] = i; }
+            y[i] += l->val[e] * v;
+        }
+    }
+    for (int a = 0; a < l->nb; a++) {
+        int i = l->brow[a];
+        if (!l->seen[i]) { l->seen[i] = 1; l->touched[nt++] = i; }
+    }
+    g_touched += nt;
+    int n = 0;
+    for (int a = 0; a < nt; a++) {
+        int i = l->touched[a];
+        float v = y[i];
+        if (l->relu && v < 0) v = 0;
+        if (v != 0) { out[n] = l->base + i; outv[n++] = v; }
+        y[i] = l->bias[i]; l->seen[i] = 0;
+    }
+    return n;
+}
+
+/* Fast kernel. y rests at the bias; a row is touched iff y != bias (a row equal to its bias
+ * yields exactly what an untouched row yields, so duplicates in the touched list and
+ * cancellations are harmless). The byte's known values are in g_kv. */
+static inline int apply(Block *restrict l, int b, const Act *restrict in, int nin, Act *restrict out) {
+    float *restrict y = l->y;
+    const float *restrict bias = l->bias;
+    int var = l->vmap[b];
+    const int *restrict ptr = l->vptr[var];
+    const Ent *restrict ent = l->vent[var];
+    int *restrict tl = l->touched;
+    int nt = 0, lo = l->lo, w = l->hi - l->lo, base = l->base, relu = l->relu;
+    float vmax = g_vmax;
+    for (int a = 0; a < nin; a++) {
+        unsigned j = (unsigned)(in[a].c - lo);
+        j = j < (unsigned)w ? j : (unsigned)w;    /* outside the block's columns: the empty sentinel column */
+        float v = in[a].v;
+        int e = ptr[j], ee = ptr[j + 1];
+        for (; e < ee; e++) { int i = ent[e].i; y[i] += ent[e].w * v; tl[nt++] = i; }
+    }
+    g_touched += nt; g_tb[l - g_L] += nt;
+    int n = 0;
+    /* rows that are non-zero from their known part alone, unless an unknown column touched them */
+    for (int a = l->lptr[b], ae = l->lptr[b + 1]; a < ae; a++) {
+        int i = l->lrow[a];
+        float v = l->lval[a];
+        out[n].c = base + i; out[n].v = v; n += y[i] == bias[i]; vmax = fabsf(v) > vmax ? fabsf(v) : vmax;
+    }
+    if (!l->hask) {
+        if (relu) {
+            for (int a = 0; a < nt; a++) {
+                int i = tl[a];
+                float v = y[i], bi = bias[i]; y[i] = bi;
+                v = (v != bi) ? v : 0;
+                out[n].c = base + i; out[n].v = v; n += v > 0; vmax = v > vmax ? v : vmax;
+            }
+        } else {
+            for (int a = 0; a < nt; a++) {
+                int i = tl[a];
+                float v = y[i], bi = bias[i]; y[i] = bi;
+                v = (v != bi) ? v : 0;
+                out[n].c = base + i; out[n].v = v; n += v != 0; vmax = fabsf(v) > vmax ? fabsf(v) : vmax;
+            }
+        }
+    } else if (l->ktab) {
+        const signed char *restrict kt = l->ktab + (size_t)b * l->rows;
+        if (relu) {
+            for (int a = 0; a < nt; a++) {
+                int i = tl[a];
+                float v = y[i], bi = bias[i]; y[i] = bi;
+                v = (v != bi) ? v + (float)kt[i] : 0;   /* at rest: contributes nothing here (loud list covers it) */
+                out[n].c = base + i; out[n].v = v; n += v > 0; vmax = v > vmax ? v : vmax;
+            }
+        } else {
+            for (int a = 0; a < nt; a++) {
+                int i = tl[a];
+                float v = y[i], bi = bias[i]; y[i] = bi;
+                v = (v != bi) ? v + (float)kt[i] : 0;
+                out[n].c = base + i; out[n].v = v; n += v != 0; vmax = fabsf(v) > vmax ? fabsf(v) : vmax;
+            }
+        }
+    } else {
+        const int *restrict kptr = l->kptr, *restrict kcol = l->kcol;
+        const float *restrict kval = l->kval, *restrict kv = g_kv;
+        for (int a = 0; a < nt; a++) {
+            int i = tl[a];
+            float v = y[i], bi = bias[i];
+            if (v != bi) {
+                y[i] = bi;
+                for (int e = kptr[i], ee = kptr[i + 1]; e < ee; e++) v += kval[e] * kv[kcol[e]];
+                if (relu && v < 0) v = 0;
+                if (v != 0) { out[n].c = base + i; out[n].v = v; n++; if (fabsf(v) > vmax) vmax = fabsf(v); }
+            }
+        }
+    }
+    g_vmax = vmax;
+    return n;
+}
+static const char *ANSI[9] = {"", "\x1b[1;35m", "\x1b[36m", "\x1b[32m", "\x1b[2;3;37m", "\x1b[33m", "\x1b[1;34m", "\x1b[1;33m", "\x1b[38;5;208m"};
+
+int main(int argc, char **argv) {
+    const char *wpath = "pylexnet_weights.bin", *path = NULL;
+    int classes = 0, bench = 0;
+    for (int i = 1; i < argc; i++) {
+        if (!strcmp(argv[i], "--classes")) classes = 1;
+        else if (!strcmp(argv[i], "--bench")) bench = 1;
+        else if (!strcmp(argv[i], "--reference")) g_ref = 1;
+        else if (!strcmp(argv[i], "-w")) wpath = argv[++i];
+        else path = argv[i];
+    }
+    FILE *f = fopen(wpath, "rb");
+    if (!f) { perror(wpath); return 1; }
+    fseek(f, 0, SEEK_END); long ws = ftell(f); rewind(f);
+    unsigned char *wb = malloc((size_t)ws);
+    if (fread(wb, 1, (size_t)ws, f) != (size_t)ws) return 1;
+    fclose(f);
+    P = wb;
+    const int *h = take(40);
+    int NIN = h[0], NSTATE = h[1], NB = h[2], ninit = h[3], oL = h[4], LAB = h[5], oE = h[6], EV = h[7], oB = h[8];
+    g_U = (float)h[9];   /* declared bound on every activation; verified at runtime, see check below */
+    const int *init = take(4 * (size_t)ninit);
+    const int *evmap = take(4 * (size_t)(EV - 9) * 9);  /* events >= 9 map the backward register instead of setting it */
+    Block *L = malloc(sizeof(Block) * (size_t)NB);
+    g_L = L;
+    int total = NIN;
+    for (int k = 0; k < NB; k++) { load(&L[k]); total += L[k].rows; }
+    if (!g_ref) specialise(L, NB, total, oB);
+    f = path ? fopen(path, "rb") : stdin;
+    if (!f) { perror(path); return 1; }
+    size_t cap = 1 << 20, n = 0;
+    unsigned char *src = malloc(cap);
+    size_t got;
+    while ((got = fread(src + n, 1, cap - n, f)) > 0) { n += got; if (n == cap) src = realloc(src, cap *= 2); }
+
+    float *av = malloc(sizeof(float) * (size_t)total);
+    int *ai = malloc(sizeof(int) * (size_t)total);
+    Act *A0 = malloc(sizeof(Act) * (size_t)total), *A1 = malloc(sizeof(Act) * (size_t)total);
+    unsigned char *lab = malloc(n + 1); signed char *ev = malloc(n + 2);
+    int *st = malloc(sizeof(int) * (size_t)NSTATE); float *sv = malloc(sizeof(float) * (size_t)NSTATE);
+    Block *last = &L[NB - 1];
+    int lbase = last->base, lim = oL + LAB;
+    struct timespec t0, t1;
+    clock_gettime(CLOCK_MONOTONIC, &t0);
+    for (int pass = 0; pass < 2; pass++) {
+        int ref = g_ref || pass;
+        g_vmax = 0;
+        if (ref) {
+            int ns = ninit;
+            for (int i = 0; i < ninit; i++) { st[i] = init[i]; sv[i] = 1; }
+            for (size_t t = 0; t <= n; t++) {
+                unsigned char b = t < n ? src[t] : 10;
+                int na = 0;
+                for (int i = 0; i < ns; i++) { ai[na] = st[i]; av[na++] = sv[i]; }
+                ai[na] = oB + b; av[na++] = 1;
+                int from = 0;
+                for (int k = 0; k < NB; k++) { from = na; na += apply_ref(&L[k], ai, av, na, ai + na, av + na); }
+                ns = 0; int l = 0, e = -1;
+                for (int i = from; i < na; i++) {
+                    int r = ai[i] - lbase;
+                    if (r < NSTATE) { st[ns] = r; sv[ns++] = av[i]; }
+                    else if (r < lim) l = r - oL;
+                    else e = r - oE;
+                }
+                if (t < n) lab[t] = (unsigned char)l;
+                ev[t] = (signed char)e;
+            }
+        } else {
+            Act *cur = A0, *nxt = A1;
+            int ns = ninit;
+            for (int i = 0; i < ninit; i++) { cur[i].c = init[i]; cur[i].v = 1; }
+            for (size_t t = 0; t <= n; t++) {
+                unsigned char b = t < n ? src[t] : 10;
+                if (g_needkv) for (int a = g_kvptr[b]; a < g_kvptr[b + 1]; a++) g_kv[g_kvcol[a]] = g_kvval[a];
+                int na = ns, from = 0;
+                for (int k = 0; k < NB; k++) { from = na; na += apply(&L[k], b, cur, na, cur + na); }
+                if (g_needkv) for (int a = g_kvptr[b]; a < g_kvptr[b + 1]; a++) g_kv[g_kvcol[a]] = 0;
+                /* next state goes straight into the other buffer; label and event are read off */
+                ns = 0; int l = 0, e = -1;
+                for (int i = from; i < na; i++) {
+                    int r = cur[i].c - lbase;
+                    if (r < NSTATE) { nxt[ns].c = r; nxt[ns].v = cur[i].v; ns++; }
+                    else if (r < lim) l = r - oL;
+                    else e = r - oE;
+                }
+                if (t < n) lab[t] = (unsigned char)l;
+                ev[t] = (signed char)e;
+                Act *tmp = cur; cur = nxt; nxt = tmp;
+            }
+        }
+        /* The pruning assumed |activation| <= bound. Every activation the fast kernel produced was
+         * checked against it; if any broke it the pruning was unsound and the file is redone with
+         * the reference kernel. */
+        if (ref || g_vmax <= g_U) break;
+        fprintf(stderr, "pylexnet: activation %g exceeds declared bound %g; using the reference kernel\n", (double)g_vmax, (double)g_U);
+    }
+    /* backward cell and head: r' = event, or the event's map of r, or r; class = label or r */
+    int reg = 0;
+    for (size_t t = n; t-- > 0;) {
+        if (ev[t + 1] >= 0) reg = ev[t + 1] < 9 ? ev[t + 1] : evmap[(ev[t + 1] - 9) * 9 + reg];
+        if (lab[t] == 9) lab[t] = (unsigned char)reg;
+    }
+    clock_gettime(CLOCK_MONOTONIC, &t1);
+    if (bench) {
+        double s = (double)(t1.tv_sec - t0.tv_sec) + 1e-9 * (double)(t1.tv_nsec - t0.tv_nsec);
+        fprintf(stderr, "%zu bytes, %.1f ns/byte, %.1f rows touched/byte (per block:", n, s * 1e9 / (double)n, (double)g_touched / (double)n);
+        for (int k = 0; k < NB && k < 16; k++) fprintf(stderr, " %.1f", (double)g_tb[k] / (double)n);
+        fprintf(stderr, ")\n");
+        return 0;
+    }
+    if (classes) { fwrite(lab, 1, n, stdout); return 0; }
+    int cur = 0;
+    for (size_t t = 0; t < n; t++) {
+        if (lab[t] != cur) { fputs("\x1b[0m", stdout); fputs(ANSI[lab[t]], stdout); cur = lab[t]; }
+        putchar(src[t]);
+    }
+    fputs("\x1b[0m", stdout);
+    return 0;
+}

builds/accurate-7370/pylexnet_weights.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5976e0d344c629ec943c8ff5794669284d5088eb8bbe636e7c4ac2fe4bb3248
+size 89840

builds/accurate-7370/spec.py

@@ -0,0 +1,373 @@
+"""Python lexical rules as one finite automaton plus a one-register backward pass.
+
+The automaton is what gets compiled into network weights. Nothing here is learned.
+
+Forward pass, per byte: next state, a direct label, and an event. An event says
+"the word that just ended has this class". Backward pass: carry that class back
+over the word's bytes.
+"""
+import builtins
+import keyword
+import re
+from pathlib import Path
+
+import pygments.lexers.python as _pl
+
+TEXT, KEYWORD, BUILTIN, STRING, COMMENT, NUMBER, FUNC, CLASS, DECOR = range(9)
+NAMES = ["text", "keyword", "builtin", "string", "comment", "number", "func", "class", "decorator"]
+WORD = 9  # label meaning "resolve from the backward pass"
+# Events 0..8 set the backward register. Events 9, 10 instead map it: they fire at the first byte of a word
+# and turn that word's class, once it is known, into the class of whatever came before it.
+EV_MATCH, EV_YIELD = 9, 10
+EV_MAP = [[TEXT if c == KEYWORD else KEYWORD for c in range(9)],  # match/case: a keyword next means it was a plain name
+          [KEYWORD if c == KEYWORD else TEXT for c in range(9)]]  # the space in `yield from` belongs to the keyword
+
+KEYWORDS = set(keyword.kwlist)
+# Pygments' word lists, so both sides share the vocabulary.
+_src = Path(_pl.__file__).read_text()
+_b = _src.index("'builtins': [")
+_a = _src.index("'magicfuncs': [")
+BUILTINS = set(re.findall(r"'(\w+)'", _src[_b:_a])) - {"builtins", "exceptions"} | {"self", "cls", "Ellipsis", "NotImplemented"}
+MAGIC = set(re.findall(r"'(__\w+__)'", _src[_a:_src.index("'magicvars': [")])) - BUILTINS
+PLAIN_PRE = {"r", "u", "b", "br", "rb"}
+F_PRE = {"f", "fr", "rf"}
+
+
+def _cases(words):
+    out = set()
+    for p in words:
+        for m in range(1 << len(p)):
+            out.add("".join(c.upper() if m >> i & 1 else c for i, c in enumerate(p)))
+    return out
+
+
+PLAIN_PRE, F_PRE = _cases(PLAIN_PRE), _cases(F_PRE)
+
+ALPHA = set(b"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_") | set(range(128, 256))
+DIGIT = set(b"0123456789")
+WORDCH = ALPHA | DIGIT
+NL, BS, HASH, DOT, AT = 10, 92, 35, 46, 64
+QUOTES = (34, 39)
+SPACE = {32, 9, 12}
+
+
+# Word categories the trie sub-network reports. One small integer per word.
+F_NONE, F_KW, F_BI, F_MAGIC, F_PRE_S, F_PRE_F, F_DEF, F_CLASS, F_IMP, F_SOFT, F_YIELD = range(11)
+
+
+def build_trie():
+    nodes = [{}]
+    term = {}
+    groups = ((F_KW, KEYWORDS), (F_BI, BUILTINS), (F_MAGIC, MAGIC), (F_PRE_S, PLAIN_PRE), (F_PRE_F, F_PRE),
+              (F_DEF, {"def"}), (F_CLASS, {"class"}), (F_IMP, {"import", "from"}), (F_SOFT, {"match", "case"}), (F_YIELD, {"yield"}))
+    for flag, words in groups:
+        for w in words:
+            n = 0
+            for ch in w.encode():
+                if ch not in nodes[n]:
+                    nodes.append({})
+                    nodes[n][ch] = len(nodes) - 1
+                n = nodes[n][ch]
+            term[n] = flag
+    return nodes, term
+
+
+TRIE, TERM = build_trie()
+
+
+def word_class(flag, ctx):
+    if ctx == "us":
+        return KEYWORD
+    if ctx == "dec":
+        return DECOR
+    if ctx == "def":
+        return FUNC
+    if ctx == "class":
+        return CLASS
+    if flag in (F_KW, F_DEF, F_CLASS, F_IMP, F_YIELD):
+        return KEYWORD
+    if flag == F_MAGIC:
+        return FUNC
+    if ctx in ("dot", "imp"):
+        return TEXT
+    if flag == F_BI:
+        return BUILTIN
+    return TEXT
+
+
+def ctx_after_word(flag, ctx):
+    if ctx != "dot":
+        if flag == F_DEF:
+            return "def"
+        if flag == F_CLASS:
+            return "class"
+        if flag == F_YIELD:
+            return "yield"
+        if flag == F_IMP:  # `from a import b` turns it off again; so does `yield from`
+            return "none" if ctx in ("imp", "yield") else "imp"
+    return ctx if ctx in STICKY else "none"
+
+
+STICKY = ("imp", "soft")  # contexts that last to the end of the line
+# After match/case these mean an ordinary name.
+NOT_SOFT = b":,;=^&|@~)]}"
+BOL = "numend"  # line-start mode. Takes over the kind name compile.py already lists, now that numbers don't use it.
+
+
+def code(ctx, fr):
+    return ("code", ctx, fr)
+
+
+def line_start(fr):
+    return (BOL, fr) if fr is None else code("none", fr)
+
+
+def after_string(fr):
+    return code("none", fr)
+
+
+def back(fr):
+    """Leave a replacement field: back to the string that owns it, inside any field that holds that string."""
+    return ("str", fr[0], fr[1], 0, True, fr[3] + (None,) if fr[3] else None)
+
+
+def step(state, b, flag):
+    """(next state, label for this byte, event for the word just ended)."""
+    kind = state[0]
+    fr = state[-1]  # f-string context: None or (quote, triple, bracket depth, outer field or None)
+    if kind == "code":
+        ctx = state[1]
+        if ctx == "ysp":  # `yield ` then: only `from` keeps the space in the keyword
+            if b == 102:
+                return ("word", "yield", fr), WORD, EV_YIELD
+            nxt, lab, _ = step(code("none", fr), b, flag)
+            return nxt, lab, TEXT
+        if ctx == "feq":  # `=` in a field at depth 0: only `}`, `:` or `!r` after it make it the debug specifier
+            if b in SPACE:
+                return state, WORD, None
+            if b == 33:
+                return ("bang", fr), WORD, None
+            if b in b"}:":
+                return step(("bang2", fr), b, flag)
+            nxt, lab, _ = step(code("none", fr), b, flag)
+            return nxt, lab, TEXT
+        if ctx == "sq":  # line-start match/case, class still open: the next byte decides
+            if b == 32 or b == 9:
+                return state, TEXT, None
+            nxt, lab, _ = step(code("none" if b in NOT_SOFT else "soft", fr), b, flag)
+            # True/False/None start with capitals: lower-case keywords are the ones that undo a match/case
+            return nxt, lab, TEXT if b in NOT_SOFT else EV_MATCH if 97 <= b <= 122 else KEYWORD
+        if fr is not None:
+            q, triple, depth, out = fr
+            if b in b"([{":
+                return code("none", (q, triple, min(depth + 1, 3), out)), TEXT, None
+            if b in b")]" and depth:
+                return code("none", (q, triple, depth - 1, out)), TEXT, None
+            if b == 125:  # }
+                if depth:
+                    return code("none", (q, triple, depth - 1, out)), TEXT, None
+                return back(fr), STRING, None
+            if b == 33 and depth == 0:  # !r conversion, unless it turns out to be !=
+                return ("bang", fr), WORD, None
+            if b == 61 and depth == 0:  # = may be the debug specifier
+                return code("feq", fr), WORD, None
+            if b == 58 and depth == 0:  # : starts the format spec, which is string text
+                return back(fr), STRING, None
+        if b == NL:
+            return line_start(fr), TEXT, None
+        if ctx == "yield" and b == 32:
+            return code("ysp", fr), WORD, None
+        if b in SPACE or b == BS:
+            return state, TEXT, None
+        if b == HASH:
+            return ("cmt", fr), COMMENT, None
+        if b in QUOTES:
+            return ("q1", b, False, fr), STRING, None
+        if b in DIGIT:
+            return ("int", fr), NUMBER, None
+        if b == DOT:
+            if ctx == "imp":  # dotted module name
+                return state, TEXT, None
+            return ("dot", fr), WORD, None
+        if b == AT:
+            return ("at", "none", fr), WORD, None
+        if ctx == "soft" and b == 95:  # `_` in a match/case line
+            return ("word", "us", fr), WORD, None
+        if b in ALPHA:
+            return ("word", ctx, fr), WORD, None
+        return code(ctx if ctx in STICKY else "none", fr), TEXT, None
+    if kind == BOL:  # indentation costs no register; only m and c can start a soft keyword
+        if b in SPACE:
+            return state, TEXT, None
+        if b in b"mc":
+            return ("word", "bol", fr), WORD, None
+        return step(code("none", fr), b, flag)
+    if kind == "bang":
+        if b in b"sraf":
+            return ("bang2", fr), WORD, None
+        nxt, lab, _ = step(code("none", fr), b, flag)
+        return nxt, lab, TEXT
+    if kind == "bang2":
+        if b in b"}:":
+            return back(fr), STRING, STRING
+        nxt, lab, _ = step(code("none", fr), b, flag)
+        return nxt, lab, TEXT
+    if kind == "dot":  # the dot's class waits for this byte: .5 or an attribute
+        if b in DIGIT:
+            return ("frac", fr), NUMBER, NUMBER
+        nxt, lab, _ = step(code("dot", fr), b, flag)
+        return nxt, lab, TEXT
+    if kind == "at":
+        if b in ALPHA:
+            return ("word", "dec", fr), WORD, None
+        nxt, lab, _ = step(code("none", fr), b, flag)
+        return nxt, lab, TEXT
+    if kind == "word":
+        ctx = state[1]
+        if b in WORDCH:
+            return (("word", "none", fr) if ctx == "us" else state), WORD, None
+        if b in QUOTES and flag in (F_PRE_S, F_PRE_F) and ctx not in ("dec", "def", "class"):
+            return ("q1", b, flag == F_PRE_F, fr), STRING, STRING
+        if ctx == "bol" and flag == F_SOFT:
+            return step(code("sq", fr), b, F_NONE)
+        if b == DOT and flag not in (F_KW, F_DEF, F_CLASS, F_IMP, F_YIELD) and ctx != "imp":
+            return code("dot", fr), TEXT, word_class(flag, ctx)  # after a name a dot is always an attribute, no need to wait
+        nxt, lab, _ = step(code(ctx_after_word(flag, ctx), fr), b, F_NONE)
+        return nxt, lab, word_class(flag, ctx)
+    if kind == "cmt":
+        if b == NL:
+            return line_start(fr), TEXT, None
+        return state, COMMENT, None
+    if kind == "q1":
+        q, f = state[1], state[2]
+        if b == q:
+            return ("q2", q, f, fr), STRING, None
+        return step(("str", q, 0, 0, f, fr), b, flag)
+    if kind == "q2":
+        q, f = state[1], state[2]
+        if b == q:
+            return ("str", q, 1, 0, f, fr), STRING, None
+        return step(after_string(fr), b, flag)
+    if kind == "str":
+        q, triple, close, f = state[1], state[2], state[3], state[4]
+        if b == BS:
+            return ("esc", q, triple, f, fr), STRING, None
+        if f and b == 123:  # {
+            return ("fopen", q, triple, fr), STRING, None
+        if triple:
+            if b == q:
+                if close == 2:
+                    return after_string(fr), STRING, None
+                return ("str", q, 1, close + 1, f, fr), STRING, None
+            return ("str", q, 1, 0, f, fr), STRING, None
+        if b == q:
+            return after_string(fr), STRING, None
+        if b == NL:
+            return line_start(fr), TEXT, None
+        return state, STRING, None
+    if kind == "fopen":  # just saw { in an f-string: {{ is a literal brace
+        q, triple = state[1], state[2]
+        if b == 123:
+            return ("str", q, triple, 0, True, fr), STRING, None
+        return step(code("none", (q, triple, 0, fr[:3] if fr else None)), b, flag)
+    if kind == "esc":
+        q, triple, f = state[1], state[2], state[3]
+        if f and b == 123:  # `\{` escapes nothing: the brace still opens a field
+            return ("fopen", q, triple, fr), STRING, None
+        if f and b == 78:  # \N{NAME} is one character, not a field
+            return ("escn", q, triple, f, fr), STRING, None
+        return ("str", q, triple, 0, f, fr), STRING, None
+    if kind == "escn":
+        if b == 123:
+            return ("nname", state[1], state[2], state[3], fr), STRING, None
+        return step(("str", state[1], state[2], 0, state[3], fr), b, flag)
+    if kind == "nname":
+        if b == 125:
+            return ("str", state[1], state[2], 0, state[3], fr), STRING, None
+        if b == state[1] or b == NL:
+            return step(("str", state[1], state[2], 0, state[3], fr), b, flag)
+        return state, STRING, None
+    if kind == "int":
+        if b in DIGIT or b == 95:
+            return state, NUMBER, None
+        if b in b"xXoObB":
+            return ("hex", fr), NUMBER, None
+        if b == DOT:
+            return ("frac", fr), NUMBER, None
+        if b in b"eE":
+            return ("exp0", fr), NUMBER, None
+        return step(code("none", fr), b, flag)
+    if kind == "hex":
+        if b in DIGIT or b in b"abcdefABCDEF_":
+            return state, NUMBER, None
+        return step(code("none", fr), b, flag)
+    if kind == "frac":
+        if b in DIGIT or b == 95:
+            return state, NUMBER, None
+        if b in b"eE":
+            return ("exp0", fr), NUMBER, None
+        return step(code("none", fr), b, flag)
+    if kind == "exp0":
+        if b in b"+-" or b in DIGIT:
+            return ("exp", fr), NUMBER, None
+        return step(code("none", fr), b, flag)
+    if kind == "exp":
+        if b in DIGIT or b == 95:
+            return state, NUMBER, None
+        if b in b"jJ":  # imaginary; ends the literal
+            return code("none", fr), NUMBER, None
+        return step(code("none", fr), b, flag)
+    raise ValueError(state)
+
+
+START = line_start(None)
+
+
+def build_table():
+    """Enumerate reachable mode states. rows[state][flag][byte] = (next, label, event)."""
+    ids = {START: 0}
+    order = [START]
+    rows = []
+    cells = {}
+    i = 0
+    while i < len(order):
+        s = order[i]
+        table = []
+        for flag in range(11 if s[0] == "word" else 1):  # only a word's end reads the word category
+            row = []
+            for b in range(256):
+                n, lab, ev = step(s, b, flag)
+                if n not in ids:
+                    ids[n] = len(order)
+                    order.append(n)
+                t = (ids[n], lab, -1 if ev is None else ev)
+                row.append(cells.setdefault(t, t))
+            table.append(row)
+        rows.append(table if s[0] == "word" else table * 11)
+        i += 1
+    return order, rows
+
+
+def run_table(data, rows):
+    """Reference execution: trie, mode machine, then backward pass."""
+    n = len(data)
+    lab = [0] * n
+    ev = [-1] * (n + 1)
+    s = 0
+    node = 0  # trie position; None once the word has left the trie
+    for i, b in enumerate(data):
+        flag = TERM.get(node, F_NONE) if node is not None else F_NONE
+        s, lab[i], ev[i] = rows[s][flag][b]
+        if b in WORDCH:
+            node = TRIE[node].get(b) if node is not None else None
+        else:
+            node = 0
+    flag = TERM.get(node, F_NONE) if node is not None else F_NONE
+    ev[n] = rows[s][flag][NL][2]
+    out = bytearray(n)
+    reg = TEXT
+    for i in range(n - 1, -1, -1):
+        if ev[i + 1] >= 0:
+            reg = ev[i + 1] if ev[i + 1] < EV_MATCH else EV_MAP[ev[i + 1] - EV_MATCH][reg]
+        out[i] = reg if lab[i] == WORD else lab[i]
+    return out

builds/fast-994/pylexnet.c

@@ -0,0 +1,369 @@
+/* pylexnet (hand lane): run the hand-designed Python highlighter network.
+ *
+ * Two paths over the same weights file:
+ *   default   the generated kernel in pylexnet_kernel.h (architecture only; values gathered from the file).
+ *             The loader checks the structure hash and proves every guard inequality from the loaded values.
+ *   --interp  the generic layered sparse evaluation (clip(W v + b, 0, hi) + carried entries), no kernel.
+ * usage: pylexnet [-w weights.bin] [--interp] (--classes [file] | --bench file)
+ *   --classes writes one class byte (0..8) per input byte; without a file it reads stdin.
+ */
+#include <math.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+
+typedef struct {
+    int64_t rows, cols, nskip, tied;     /* tied: one weight row and one ceiling shared by all rows */
+    const double *W, *b, *hi;
+    const int64_t *skip;
+    int *rowptr, *col;
+    double *val;
+} Layer;
+
+typedef struct {
+    int64_t n_feats, n_fstate, n_fout, n_bstate, n_classes, head_cols, maxw;
+    const double *emb, *h0, *head_W, *head_b;
+    const int64_t *f_state, *f_out, *b_state;
+    Layer *fwd, *bwd;
+    int64_t n_fwd, n_bwd;
+} Net;
+
+static int use_interp = 0;
+
+/* ---- loading ------------------------------------------------------------- */
+typedef struct { const unsigned char *p, *end; } Cursor;
+
+static const void *take(Cursor *c, size_t bytes) {
+    if (c->p + bytes > c->end) { fprintf(stderr, "weights: truncated\n"); exit(1); }
+    const void *r = c->p;
+    c->p += bytes;
+    return r;
+}
+static int64_t take_i64(Cursor *c) { int64_t v; memcpy(&v, take(c, 8), 8); return v; }
+
+static Layer *load_cell(Cursor *c, int64_t *n, int64_t *maxw) {
+    *n = take_i64(c);
+    Layer *L = calloc((size_t)*n, sizeof(Layer));
+    for (int64_t l = 0; l < *n; l++) {
+        Layer *y = &L[l];
+        y->rows = take_i64(c); y->cols = take_i64(c); y->nskip = take_i64(c); y->tied = take_i64(c);
+        int64_t wrows = y->tied ? 1 : y->rows;
+        y->W = take(c, (size_t)(wrows * y->cols) * 8);
+        y->b = take(c, (size_t)y->rows * 8);
+        y->hi = take(c, (size_t)wrows * 8);
+        y->skip = take(c, (size_t)y->nskip * 8);
+        if (y->cols > *maxw) *maxw = y->cols;
+        if (y->rows + y->nskip > *maxw) *maxw = y->rows + y->nskip;
+        int nnz = 0;
+        for (int64_t i = 0; i < wrows * y->cols; i++) nnz += y->W[i] != 0;
+        if (y->tied) nnz *= (int)y->rows;
+        y->rowptr = malloc((size_t)(y->rows + 1) * sizeof(int));
+        y->col = malloc((size_t)(nnz + 1) * sizeof(int));
+        y->val = malloc((size_t)(nnz + 1) * sizeof(double));
+        int k = 0;
+        for (int64_t r = 0; r < y->rows; r++) {
+            y->rowptr[r] = k;
+            const double *w = y->W + (y->tied ? 0 : r) * y->cols;
+            for (int64_t cc = 0; cc < y->cols; cc++) if (w[cc] != 0) y->col[k] = (int)cc, y->val[k++] = w[cc];
+        }
+        y->rowptr[y->rows] = k;
+    }
+    return L;
+}
+
+static Net load(const char *path) {
+    FILE *f = fopen(path, "rb");
+    if (!f) { perror(path); exit(1); }
+    fseek(f, 0, SEEK_END);
+    size_t n = (size_t)ftell(f);
+    fseek(f, 0, SEEK_SET);
+    unsigned char *blob = malloc(n);
+    if (fread(blob, 1, n, f) != n) { perror(path); exit(1); }
+    fclose(f);
+    Cursor c = { blob, blob + n };
+    if (memcmp(take(&c, 8), "PYLX02\0\0", 8)) { fprintf(stderr, "%s: not a PYLX02 weight file\n", path); exit(1); }
+    Net net = {0};
+    net.n_feats = take_i64(&c); net.n_fstate = take_i64(&c); net.n_fout = take_i64(&c);
+    net.n_bstate = take_i64(&c); net.n_classes = take_i64(&c); net.head_cols = take_i64(&c);
+    net.emb = take(&c, (size_t)(128 * net.n_feats) * 8);
+    net.maxw = net.n_feats + net.n_fstate;
+    if (net.n_fout + net.n_bstate > net.maxw) net.maxw = net.n_fout + net.n_bstate;
+    net.fwd = load_cell(&c, &net.n_fwd, &net.maxw);
+    net.bwd = load_cell(&c, &net.n_bwd, &net.maxw);
+    net.f_state = take(&c, (size_t)net.n_fstate * 8);
+    net.f_out = take(&c, (size_t)net.n_fout * 8);
+    net.b_state = take(&c, (size_t)net.n_bstate * 8);
+    net.h0 = take(&c, (size_t)net.n_fstate * 8);
+    net.head_W = take(&c, (size_t)net.head_cols * 8);      /* one integer-valued readout unit: the class id */
+    net.head_b = take(&c, 8);
+    if (c.p != c.end) { fprintf(stderr, "weights: %zu trailing bytes\n", (size_t)(c.end - c.p)); exit(1); }
+    return net;
+}
+
+/* ---- generic inference --------------------------------------------------- */
+static void run_cell(const Layer *layers, int64_t nl, double *v, double *tmp) {
+    for (int64_t l = 0; l < nl; l++) {
+        const Layer *L = &layers[l];
+        for (int64_t r = 0; r < L->rows; r++) {
+            double s = L->b[r];
+            for (int k = L->rowptr[r]; k < L->rowptr[r + 1]; k++) s += L->val[k] * v[L->col[k]];
+            double hi = L->hi[L->tied ? 0 : r];
+            tmp[r] = s < 0 ? 0 : (s > hi ? hi : s);
+        }
+        for (int64_t k = 0; k < L->nskip; k++) tmp[L->rows + k] = v[L->skip[k]];
+        memcpy(v, tmp, (size_t)(L->rows + L->nskip) * sizeof(double));
+    }
+}
+
+/* n input bytes followed by one virtual newline, so a word at the end of the file still ends. */
+static void classify_interp(const Net *net, const unsigned char *src, size_t n, unsigned char *cls, uint16_t *bits) {
+    double *v = malloc((size_t)net->maxw * 8), *tmp = malloc((size_t)net->maxw * 8);
+    double *h = malloc((size_t)net->n_fstate * 8), *r = calloc((size_t)net->n_bstate, 8);
+    memcpy(h, net->h0, (size_t)net->n_fstate * 8);
+    for (size_t t = 0; t <= n; t++) {
+        unsigned c = t < n ? (src[t] < 128 ? src[t] : 127) : 10;
+        memcpy(v, net->emb + c * net->n_feats, (size_t)net->n_feats * 8);
+        memcpy(v + net->n_feats, h, (size_t)net->n_fstate * 8);
+        run_cell(net->fwd, net->n_fwd, v, tmp);
+        for (int64_t i = 0; i < net->n_fstate; i++) h[i] = v[net->f_state[i]];
+        uint16_t bt = 0;
+        for (int64_t i = 0; i < net->n_fout; i++) bt |= (uint16_t)((v[net->f_out[i]] != 0) << i);
+        bits[t] = bt;
+    }
+    for (size_t t = n + 1; t-- > 0;) {
+        for (int64_t i = 0; i < net->n_fout; i++) v[i] = (bits[t] >> i) & 1;
+        memcpy(v + net->n_fout, r, (size_t)net->n_bstate * 8);
+        run_cell(net->bwd, net->n_bwd, v, tmp);
+        for (int64_t i = 0; i < net->n_bstate; i++) r[i] = v[net->b_state[i]];
+        double s = net->head_b[0];
+        for (int64_t c = 0; c < net->head_cols; c++) s += net->head_W[c] * v[c];
+        s = s < 0 ? 0 : (s > net->n_classes - 1 ? net->n_classes - 1 : s);
+        if (t < n) cls[t] = (unsigned char)s;
+    }
+    free(v); free(tmp); free(h); free(r);
+}
+
+/* ---- compiled mode ------------------------------------------------------- */
+#include <emmintrin.h>
+static inline double clip_sse(double x, double hi) { return _mm_cvtsd_f64(_mm_min_sd(_mm_max_sd(_mm_set_sd(x), _mm_setzero_pd()), _mm_set_sd(hi))); }
+static inline double clip_fm(double x, double hi) { return __builtin_fmin(__builtin_fmax(x, 0.0), hi); }
+#include "pylexnet_kernel.h"
+
+typedef struct { uint64_t h; } Fnv;
+static void fnv_bytes(Fnv *f, const void *p, size_t n) {
+    for (size_t i = 0; i < n; i++) { f->h ^= ((const unsigned char *)p)[i]; f->h *= 0x100000001B3ULL; }
+}
+static void fnv_i64(Fnv *f, int64_t x) { fnv_bytes(f, &x, 8); }
+static void fnv_mask(Fnv *f, const double *a, int64_t n) {
+    for (int64_t i = 0; i < n; i++) { unsigned char m = a[i] != 0; fnv_bytes(f, &m, 1); }
+}
+static uint64_t struct_hash(const Net *net) {
+    Fnv f = { 0xCBF29CE484222325ULL };
+    fnv_i64(&f, net->n_feats); fnv_i64(&f, net->n_fstate); fnv_i64(&f, net->n_fout);
+    fnv_i64(&f, net->n_bstate); fnv_i64(&f, net->n_classes); fnv_i64(&f, net->head_cols);
+    const Layer *cells[2] = { net->fwd, net->bwd };
+    int64_t counts[2] = { net->n_fwd, net->n_bwd };
+    for (int c = 0; c < 2; c++) {
+        fnv_i64(&f, counts[c]);
+        for (int64_t l = 0; l < counts[c]; l++) {
+            const Layer *L = &cells[c][l];
+            fnv_i64(&f, L->rows); fnv_i64(&f, L->cols); fnv_i64(&f, L->nskip); fnv_i64(&f, L->tied);
+            fnv_bytes(&f, L->hi, (size_t)(L->tied ? 1 : L->rows) * 8);
+            for (int64_t k = 0; k < L->nskip; k++) fnv_i64(&f, L->skip[k]);
+            fnv_mask(&f, L->W, (L->tied ? 1 : L->rows) * L->cols);
+        }
+    }
+    for (int64_t i = 0; i < net->n_fstate; i++) fnv_i64(&f, net->f_state[i]);
+    for (int64_t i = 0; i < net->n_fout; i++) fnv_i64(&f, net->f_out[i]);
+    for (int64_t i = 0; i < net->n_bstate; i++) fnv_i64(&f, net->b_state[i]);
+    fnv_mask(&f, net->head_W, net->head_cols);
+    return f.h;
+}
+
+/* Every number in the file must be an integer: then all activations are exact integers in double arithmetic,
+ * sums may be re-associated, and a clip1 unit is exactly 0 or 1. */
+static int all_integers(const Net *net) {
+    int bad = 0;
+    for (int64_t i = 0; i < 128 * net->n_feats; i++) bad += net->emb[i] != (int64_t)net->emb[i];
+    const Layer *cells[2] = { net->fwd, net->bwd };
+    int64_t counts[2] = { net->n_fwd, net->n_bwd };
+    for (int c = 0; c < 2; c++)
+        for (int64_t l = 0; l < counts[c]; l++) {
+            const Layer *L = &cells[c][l];
+            int64_t wrows = L->tied ? 1 : L->rows;
+            for (int64_t i = 0; i < wrows * L->cols; i++) bad += L->W[i] != (int64_t)L->W[i];
+            for (int64_t i = 0; i < L->rows; i++) bad += L->b[i] != (int64_t)L->b[i];
+            for (int64_t i = 0; i < wrows; i++) bad += L->hi[i] != (int64_t)L->hi[i];
+        }
+    for (int64_t i = 0; i < net->head_cols; i++) bad += net->head_W[i] != (int64_t)net->head_W[i];
+    for (int64_t i = 0; i < net->n_fstate; i++) bad += net->h0[i] != (int64_t)net->h0[i];
+    return bad;
+}
+
+/* Sorted threshold families: tied, one input, ceiling 1, biases monotone so the outputs form a step vector.
+ * Then rank(h) by binary search reproduces every unit, and a consumer's weights over the family reduce to a
+ * prefix sum indexed by the rank. Both tables are built here from the loaded values. */
+static int check_families(const Net *net) {
+    int bad = 0;
+    for (int g = 0; g < K_N_FAMILY; g++) {
+        int cell = K_FAMILY[g][0], l = K_FAMILY[g][1], n = K_FAMILY[g][3];
+        const Layer *L = &(cell ? net->bwd : net->fwd)[l];
+        int nz = 0;
+        for (int64_t c = 0; c < L->cols; c++) nz += L->W[c] != 0;
+        if (!L->tied || L->rows != n || L->hi[0] != 1.0 || nz != 1) { bad++; continue; }
+        double w = 0;
+        for (int64_t c = 0; c < L->cols; c++) if (L->W[c] != 0) w = L->W[c];
+        for (int r = 1; r < n; r++) if ((w > 0 && L->b[r] > L->b[r - 1]) || (w < 0 && L->b[r] < L->b[r - 1])) bad++;
+    }
+    return bad;
+}
+
+static double *build_prefix(const Net *net) {
+    double *kp = malloc(sizeof(double) * (K_PREFIX_LEN + 1));
+    int pos = 0;
+    for (int g = 0; g < K_N_PREFIX; g++) {
+        int cell = K_PREFIX[g][0], l = K_PREFIX[g][1], r = K_PREFIX[g][2], c0 = K_PREFIX[g][3], n = K_PREFIX[g][4];
+        const Layer *L = &(cell ? net->bwd : net->fwd)[l];
+        kp[pos] = 0;
+        for (int k = 0; k < n; k++) kp[pos + k + 1] = kp[pos + k] + L->W[(L->tied ? 0 : r) * L->cols + c0 + k];
+        pos += n + 1;
+    }
+    return kp;
+}
+
+static void gather(const Net *net, double *kc, double *kb) {
+    for (int i = 0; i < K_N_COEF; i++) {
+        int cell = K_COEF[i][0], l = K_COEF[i][1], r = K_COEF[i][2], c = K_COEF[i][3];
+        if (cell == 2) kc[i] = net->head_W[c];
+        else { const Layer *L = &(cell ? net->bwd : net->fwd)[l]; kc[i] = L->W[(L->tied ? 0 : r) * L->cols + c]; }
+    }
+    for (int i = 0; i < K_N_BIAS; i++) {
+        int cell = K_BIAS[i][0], l = K_BIAS[i][1], r = K_BIAS[i][2];
+        kb[i] = cell == 2 ? net->head_b[r] : (cell ? net->bwd : net->fwd)[l].b[r];
+    }
+}
+
+/* Upper bound of an entry of a layer's input vector, from the loaded ceilings and embedding table.
+ * layer -1 is the cell's own input: byte features then carried state (forward), bits then state (backward). */
+static double entry_max(const Net *net, int cell, int64_t layer, int64_t idx, int depth) {
+    if (depth > 64) return 1e300;
+    const Layer *L = cell ? net->bwd : net->fwd;
+    if (layer < 0) {
+        if (cell == 0) {
+            if (idx < net->n_feats) {
+                double m = 0;
+                for (int b = 0; b < 128; b++) if (net->emb[b * net->n_feats + idx] > m) m = net->emb[b * net->n_feats + idx];
+                return m;
+            }
+            return entry_max(net, 0, net->n_fwd - 1, net->f_state[idx - net->n_feats], depth + 1);
+        }
+        if (idx < net->n_fout) return 1.0;
+        return entry_max(net, 1, net->n_bwd - 1, net->b_state[idx - net->n_fout], depth + 1);
+    }
+    if (idx < L[layer].rows) return L[layer].hi[L[layer].tied ? 0 : idx];
+    return entry_max(net, cell, layer - 1, L[layer].skip[idx - L[layer].rows], depth + 1);
+}
+
+/* Each guarded unit must be provably 0 when its guard columns are 0: bias plus every other positive
+ * contribution at its input's ceiling must not exceed 0. Returns the number of failed guards. */
+static int check_guards(const Net *net) {
+    int bad = 0;
+    for (int g = 0; g < K_N_GUARD; g++) {
+        int cell = K_GUARD[g][0], l = K_GUARD[g][1], r = K_GUARD[g][2];
+        const Layer *L = &(cell ? net->bwd : net->fwd)[l];
+        double bound = L->b[r];
+        for (int64_t c = 0; c < L->cols; c++) {
+            double w = L->W[(L->tied ? 0 : r) * L->cols + c];
+            if (w <= 0) continue;
+            int guarded = 0;
+            for (int k = 0; k < K_MAX_GUARD; k++) if (K_GUARD[g][3 + k] == c) guarded = 1;
+            if (!guarded) bound += w * entry_max(net, cell, l - 1, c, 0);
+        }
+        if (bound > 0) { bad++; fprintf(stderr, "guard %d (cell %d layer %d row %d) not provable: bound %g\n", g, cell, l, r, bound); }
+    }
+    return bad;
+}
+
+static void classify_compiled(const Net *net, const double *kc, const double *kb, const double *kp,
+                              const unsigned char *src, size_t n, unsigned char *cls, uint16_t *bits) {
+    FState a, b, *p = &a, *s = &b, *t;
+    k_init_state(p, net->h0);
+    for (size_t i = 0; i < n; i++) {
+        bits[i] = k_fwd_step(kc, kb, kp, net->emb + (src[i] < 128 ? src[i] : 127) * K_N_FEATS, p, s);
+        t = p; p = s; s = t;
+    }
+    bits[n] = k_fwd_step(kc, kb, kp, net->emb + 10 * K_N_FEATS, p, s);
+    BState ra, rb, *r = &ra, *q = &rb, *u;
+    memset(&ra, 0, sizeof ra);
+    k_bwd_step(kc, kb, bits[n], r, q);
+    u = r; r = q; q = u;
+    for (size_t i = n; i-- > 0;) {
+        cls[i] = (unsigned char)k_bwd_step(kc, kb, bits[i], r, q);
+        u = r; r = q; q = u;
+    }
+}
+
+/* ---- cli ----------------------------------------------------------------- */
+static char *default_weights(const char *argv0) {
+    const char *slash = strrchr(argv0, '/');
+    size_t dir = slash ? (size_t)(slash - argv0 + 1) : 0;
+    char *p = malloc(dir + 32);
+    memcpy(p, argv0, dir);
+    strcpy(p + dir, "pylexnet_weights.bin");
+    return p;
+}
+
+int main(int argc, char **argv) {
+    int classes = 0, bench = 0;
+    const char *path = NULL, *wpath = NULL;
+    for (int i = 1; i < argc; i++) {
+        if (!strcmp(argv[i], "--classes")) classes = 1;
+        else if (!strcmp(argv[i], "--bench")) bench = 1;
+        else if (!strcmp(argv[i], "--interp")) use_interp = 1;
+        else if (!strcmp(argv[i], "-w")) wpath = argv[++i];
+        else path = argv[i];
+    }
+    if (!classes && !bench) { fprintf(stderr, "usage: pylexnet [-w weights.bin] [--interp] (--classes [file] | --bench file)\n"); return 2; }
+    Net net = load(wpath ? wpath : default_weights(argv[0]));
+    double *kc = malloc(sizeof(double) * (K_N_COEF + 1)), *kb = malloc(sizeof(double) * (K_N_BIAS + 1)), *kp = NULL;
+    if (!use_interp) {
+        uint64_t h = struct_hash(&net);
+        if (h != K_STRUCT_HASH) {
+            fprintf(stderr, "weights: architecture hash %016llx does not match compiled kernel %016llx; regenerate (compile.py) or run --interp\n",
+                    (unsigned long long)h, (unsigned long long)K_STRUCT_HASH);
+            return 1;
+        }
+        if (check_guards(&net)) { fprintf(stderr, "weights: guard check failed, refusing the kernel\n"); return 1; }
+        if (all_integers(&net)) { fprintf(stderr, "weights: non-integer values, refusing the kernel\n"); return 1; }
+        if (check_families(&net)) { fprintf(stderr, "weights: a threshold family is not tied/sorted, refusing the kernel\n"); return 1; }
+        gather(&net, kc, kb);
+        kp = build_prefix(&net);
+    }
+    FILE *f = path ? fopen(path, "rb") : stdin;
+    if (!f) { perror(path); return 1; }
+    size_t cap = 1 << 20, n = 0, got;
+    unsigned char *src = malloc(cap);
+    while ((got = fread(src + n, 1, cap - n, f)) > 0) { n += got; if (n == cap) src = realloc(src, cap *= 2); }
+    if (path) fclose(f);
+    unsigned char *cls = malloc(n + 1);
+    uint16_t *bits = malloc((n + 2) * sizeof(uint16_t));
+    if (bench) {
+        int reps = 5;
+        struct timespec a, b;
+        clock_gettime(CLOCK_MONOTONIC, &a);
+        for (int i = 0; i < reps; i++) {
+            if (use_interp) classify_interp(&net, src, n, cls, bits);
+            else classify_compiled(&net, kc, kb, kp, src, n, cls, bits);
+        }
+        clock_gettime(CLOCK_MONOTONIC, &b);
+        double sec = (b.tv_sec - a.tv_sec) + (b.tv_nsec - a.tv_nsec) * 1e-9;
+        fprintf(stderr, "%zu bytes, %.1f ns/byte\n", n, sec / reps / (double)n * 1e9);
+        return 0;
+    }
+    if (use_interp) classify_interp(&net, src, n, cls, bits);
+    else classify_compiled(&net, kc, kb, kp, src, n, cls, bits);
+    fwrite(cls, 1, n, stdout);
+    return 0;
+}

builds/fast-994/pylexnet_kernel.h

@@ -0,0 +1,171 @@
+/* Generated by compile.py: the network's ARCHITECTURE as C. No weight values live here: pylexnet.c
+ * loads them, checks the structure hash and the guard inequalities, and gathers kc[]/kb[] by K_COEF/K_BIAS. */
+#include <stdint.h>
+#include <string.h>
+#define K_STRUCT_HASH 0x848E1BC62ED893C7ULL
+#define K_N_FEATS 18
+#define K_N_CLASSES 9
+#define K_N_BITS 10
+#ifndef CLIP
+#define CLIP(x, hi) clipn((x), (hi))
+static inline double clipn(double x, double hi) { x = x < 0.0 ? 0.0 : x; return x > hi ? hi : x; }
+#endif
+/* A tied, sorted threshold family is a step vector: unit k is 1 iff kb[k] + w*h >= 1, true for k < rank.
+ * The loader verifies integrality, tying and monotonicity, so a binary search gives the same values. */
+static inline int k_rank(const double *kb, int n, double w, double h) {
+    int lo = 0, hi = n;
+    while (lo < hi) { int mid = (lo + hi) >> 1; if (kb[mid] + w * h >= 1.0) lo = mid + 1; else hi = mid; }
+    return lo;
+}
+typedef struct { double S, sq1, T, F, E, c, fo, FS, FD, oq1, oT, C, N, ee, W, d, pf, fp, cD, cC, cAt, atp, cDot, h1; } FState;
+typedef struct { double u_reg_KW, u_reg_BI, u_reg_FN, u_reg_CL, u_reg_DEC, u_reg_STR, b_ev_KW, b_ev_BI, b_ev_FN, b_ev_CL, b_ev_DEC, b_ev_STR; } BState;
+static inline void k_init_state(FState *s, const double *h0) { memcpy(s, h0, sizeof *s); }
+static inline __attribute__((always_inline)) uint16_t k_fwd_step(const double *restrict kc, const double *restrict kb, const double *restrict kp, const double *restrict x,
+                                  const FState *restrict p, FState *restrict s) {
+    double u_wordch = 0.0, u_qmd = 0.0, u_qms = 0.0, u_c2 = 0.0, u_lt7 = 0.0, u_wend = 0.0, u_fieldopen = 0.0, u_pf_new = 0.0, u_S_eff = 0.0, u_fp_new = 0.0, u_sclose = 0.0, u_tclose = 0.0, u_tripleopen = 0.0, u_nlclose = 0.0, u_E_new = 0.0, u_fo_new = 0.0, u_C_new = 0.0, u_open = 0.0, u_open_f = 0.0, u_open_qs = 0.0, u_fieldclose = 0.0, u_inc = 0.0, u_dec = 0.0, u_S_new = 0.0, u_c_new = 0.0, u_T_new = 0.0, u_F_new = 0.0, u_sq1_new = 0.0, u_FS_new = 0.0, u_FD_new = 0.0, u_oq1_new = 0.0, u_oT_new = 0.0, u_lab_STR = 0.0, u_N_new = 0.0, u_ee_new = 0.0, u_W_new = 0.0, u_d_new = 0.0, u_gA = 0.0, u_gB = 0.0, u_cA = 0.0, u_cB = 0.0, u_h1_new = 0.0, u_E_KW = 0.0, u_E_BI = 0.0, u_E_MAG = 0.0, u_is_def = 0.0, u_is_class = 0.0, u_cD_new = 0.0, u_cC_new = 0.0, u_atp_new = 0.0, u_cAt_new = 0.0, u_cDot_new = 0.0, u_ev_DEC = 0.0, u_ev_CL = 0.0, u_ev_KW = 0.0, u_ev_FN = 0.0, u_ev_BI = 0.0, u_ev_STR = 0.0, u_lab_W = 0.0;
+    int r_ge = 0;
+    u_lt7 = CLIP((kb[0] + kc[0] * p->d), 1.0);
+    if ((p->S + p->fo + p->FS + p->FD + p->c + p->T + p->F + p->sq1 + p->oq1 + p->oT + x[2] + x[3]) != 0.0) {   /* all activations are >= 0 */
+        u_qmd = CLIP(((kb[1] + kc[1] * x[2]) + (kc[2] * p->sq1 + kc[3] * p->E)), 1.0);
+        u_qms = CLIP(((kb[2] + kc[4] * x[3]) + (kc[5] * p->sq1 + kc[6] * p->E)), 1.0);
+        u_c2 = CLIP((kb[3] + kc[7] * p->c), 1.0);
+        u_fieldopen = CLIP(((kb[4] + kc[8] * p->fo) + kc[9] * x[8]), 1.0);
+        u_S_eff = CLIP((((kb[5] + kc[10] * p->S) + (kc[11] * u_fieldopen + kc[12] * p->T)) + ((kc[13] * u_qmd + kc[14] * u_qms) + kc[15] * u_c2)), 1.0);
+        u_sclose = CLIP((((kb[6] + kc[16] * u_S_eff) + (kc[17] * p->T + kc[18] * u_qmd)) + (kc[19] * u_qms + kc[20] * p->c)), 1.0);
+        u_tclose = CLIP((((kb[7] + kc[21] * u_S_eff) + (kc[22] * p->T + kc[23] * u_qmd)) + (kc[24] * u_qms + kc[25] * p->c)), 1.0);
+        u_tripleopen = CLIP((((kb[8] + kc[26] * u_S_eff) + (kc[27] * p->T + kc[28] * u_qmd)) + (kc[29] * u_qms + kc[30] * p->c)), 1.0);
+        u_nlclose = CLIP((((kb[9] + kc[31] * u_S_eff) + (kc[32] * p->T + kc[33] * x[1])) + kc[34] * p->E), 1.0);
+        u_E_new = CLIP(((kb[10] + kc[35] * u_S_eff) + (kc[36] * x[4] + kc[37] * p->E)), 1.0);
+        u_fo_new = CLIP((((kb[11] + kc[38] * u_S_eff) + (kc[39] * p->F + kc[40] * x[8])) + (kc[41] * p->E + kc[42] * p->fo)), 1.0);
+        u_open = CLIP((((kb[12] + kc[43] * x[2]) + (kc[44] * x[3] + kc[45] * u_S_eff)) + (kc[46] * p->C + kc[47] * p->N)), 1.0);
+        u_open_f = CLIP((((kb[13] + kc[48] * x[2]) + (kc[49] * x[3] + kc[50] * u_S_eff)) + ((kc[51] * p->C + kc[52] * p->N) + (kc[53] * p->pf + kc[54] * p->fp))), 1.0);
+        u_open_qs = CLIP((((kb[14] + kc[55] * x[2]) + (kc[56] * x[3] + kc[57] * u_S_eff)) + (kc[58] * p->C + kc[59] * p->N)), 1.0);
+        u_fieldclose = CLIP((((kb[15] + kc[60] * p->FS) + (kc[61] * u_S_eff + kc[62] * p->C)) + (kc[63] * p->FD + kc[64] * x[12])), 1.0);
+        u_inc = CLIP((((kb[16] + kc[65] * p->FS) + (kc[66] * u_S_eff + kc[67] * p->C)) + kc[68] * x[10]), 1.0);
+        u_dec = CLIP((((kb[17] + kc[69] * p->FS) + (kc[70] * u_S_eff + kc[71] * p->C)) + (kc[72] * x[11] + kc[73] * p->FD)), 1.0);
+        u_S_new = CLIP((((kb[18] + kc[74] * u_S_eff) + (kc[75] * u_sclose + kc[76] * u_tclose)) + ((kc[77] * u_nlclose + kc[78] * u_open) + kc[79] * u_fieldclose)), 1.0);
+        u_c_new = CLIP(((((kb[19] + kc[80] * p->c) + (kc[81] * u_S_eff + kc[82] * u_qmd)) + ((kc[83] * u_qms + kc[84] * u_tripleopen) + (kc[85] * u_tclose + kc[86] * u_sclose))) + (kc[87] * u_open + kc[88] * u_open_qs)), 3.0);
+        u_T_new = CLIP((((kb[20] + kc[89] * p->T) + (kc[90] * u_tripleopen + kc[91] * u_tclose)) + ((kc[92] * u_open + kc[93] * u_fieldclose) + kc[94] * p->oT)), 1.0);
+        u_F_new = CLIP((((kb[21] + kc[95] * p->F) + (kc[96] * u_open_f + kc[97] * u_open)) + ((kc[98] * u_sclose + kc[99] * u_tclose) + (kc[100] * u_nlclose + kc[101] * u_fieldclose))), 1.0);
+        u_sq1_new = CLIP(((((kb[22] + kc[102] * p->sq1) + (kc[103] * u_open_qs + kc[104] * u_sclose)) + ((kc[105] * u_tclose + kc[106] * u_nlclose) + (kc[107] * p->S + kc[108] * u_S_eff))) + (kc[109] * u_fieldclose + kc[110] * p->oq1)), 1.0);
+        u_FS_new = CLIP(((kb[23] + kc[111] * p->FS) + (kc[112] * u_fieldopen + kc[113] * u_fieldclose)), 1.0);
+        u_FD_new = CLIP(((kb[24] + kc[114] * p->FD) + (kc[115] * u_inc + kc[116] * u_dec)), 3.0);
+        u_oq1_new = CLIP((((kb[25] + kc[117] * p->oq1) + (kc[118] * u_fieldopen + kc[119] * p->sq1)) + kc[120] * u_fieldclose), 1.0);
+        u_oT_new = CLIP((((kb[26] + kc[121] * p->oT) + (kc[122] * u_fieldopen + kc[123] * p->T)) + kc[124] * u_fieldclose), 1.0);
+        u_lab_STR = CLIP((((kb[27] + kc[125] * u_S_eff) + (kc[126] * u_nlclose + kc[127] * u_open)) + kc[128] * u_fieldclose), 1.0);
+    }
+    if ((p->W + x[13] + x[6] + x[7] + p->atp + p->cD + p->cC + p->cAt + p->cDot + x[15] + x[14] + p->fp + p->d) != 0.0) {   /* all activations are >= 0 */
+        u_wordch = CLIP((kb[28] + kc[129] * x[13]), 1.0);
+        u_wend = CLIP(((kb[29] + kc[130] * p->W) + kc[131] * x[13]), 1.0);
+        u_pf_new = CLIP((((kb[30] + kc[132] * x[15]) + (kc[133] * x[14] + kc[134] * p->pf)) + (kc[135] * p->W + kc[136] * p->d)), 1.0);
+        u_atp_new = CLIP(((kb[31] + kc[137] * x[7]) + (kc[138] * u_S_eff + kc[139] * p->C)), 1.0);
+        u_fp_new = CLIP(((kb[32] + kc[140] * x[14]) + (kc[141] * p->fp + kc[142] * u_wordch)), 1.0);
+        u_gA = CLIP((((kb[33] + kc[143] * p->h1) + (kc[144] * p->W + kc[145] * u_wordch)) + kc[146] * u_lt7), 2199023255552.0);
+        u_gB = CLIP((((kb[34] + kc[147] * p->h1) + (kc[148] * p->W + kc[149] * u_wordch)) + kc[150] * u_lt7), 2199023255552.0);
+        u_cAt_new = CLIP((((kb[35] + kc[151] * p->cAt) + (kc[152] * p->atp + kc[153] * u_wordch)) + kc[154] * u_wend), 1.0);
+        u_W_new = CLIP((((kb[36] + kc[155] * u_wordch) + (kc[156] * x[0] + kc[157] * p->W)) + ((kc[158] * p->N + kc[159] * u_S_eff) + kc[160] * p->C)), 1.0);
+        u_cDot_new = CLIP((((kb[37] + kc[161] * x[6]) + (kc[162] * u_S_eff + kc[163] * p->C)) + ((kc[164] * p->N + kc[165] * p->cDot) + (kc[166] * u_wordch + kc[167] * u_wend))), 1.0);
+        u_d_new = CLIP(((kb[38] + kc[168] * p->d) + kc[169] * u_W_new), 30.0);
+        u_cA = CLIP(((kb[39] + kc[170] * x[13]) + (kc[171] * u_W_new + kc[172] * u_lt7)), 2199023255552.0);
+        u_cB = CLIP((((kb[40] + kc[173] * x[13]) + (kc[174] * p->d + kc[175] * u_W_new)) + kc[176] * u_lt7), 2199023255552.0);
+        u_lab_W = CLIP(((kb[41] + kc[177] * u_W_new) + kc[178] * u_atp_new), 1.0);
+    }
+    u_h1_new = CLIP((((kb[42] + kc[179] * u_gA) + (kc[180] * u_gB + kc[181] * u_cA)) + kc[182] * u_cB), 2199023255552.0);
+    if ((u_wend) != 0.0) {   /* all activations are >= 0 */
+        r_ge = k_rank(kb + 43, 255, kc[183], p->h1);
+        u_ev_DEC = CLIP(((kb[298] + kc[184] * u_wend) + kc[185] * p->cAt), 1.0);
+        u_ev_CL = CLIP(((kb[299] + kc[186] * u_wend) + (kc[187] * p->cC + kc[188] * p->cAt)), 1.0);
+        u_ev_STR = CLIP(((kb[300] + kc[189] * u_wend) + (kc[190] * u_open + kc[191] * p->pf)), 1.0);
+        u_E_KW = CLIP(((kb[301] + kc[192] * p->h1) + kp[0 + r_ge]), 1.0);
+        u_E_BI = CLIP(((kb[302] + kc[193] * p->h1) + kp[256 + r_ge]), 1.0);
+        u_E_MAG = CLIP(((kb[303] + kc[194] * p->h1) + kp[512 + r_ge]), 1.0);
+        u_is_def = CLIP((((kb[304] + kc[195] * u_wend) + (kc[196] * u_E_KW + kc[197] * (double)(12 < r_ge))) + kc[198] * (double)(13 < r_ge)), 1.0);
+        u_is_class = CLIP((((kb[305] + kc[199] * u_wend) + (kc[200] * u_E_KW + kc[201] * (double)(50 < r_ge))) + kc[202] * (double)(51 < r_ge)), 1.0);
+        u_ev_KW = CLIP((((kb[306] + kc[203] * u_wend) + (kc[204] * u_E_KW + kc[205] * p->cAt)) + (kc[206] * p->cC + kc[207] * p->cD)), 1.0);
+        u_ev_FN = CLIP((((kb[307] + kc[208] * u_wend) + (kc[209] * u_E_MAG + kc[210] * p->cD)) + (kc[211] * p->cAt + kc[212] * p->cC)), 1.0);
+        u_ev_BI = CLIP((((kb[308] + kc[213] * u_wend) + (kc[214] * u_E_BI + kc[215] * p->cAt)) + ((kc[216] * p->cC + kc[217] * p->cD) + kc[218] * p->cDot)), 1.0);
+    }
+    if ((p->W + x[13] + x[6] + x[7] + p->atp + p->cD + p->cC + p->cAt + p->cDot + x[15] + x[14] + p->fp + p->d) != 0.0) {   /* all activations are >= 0 */
+        u_cD_new = CLIP((((kb[309] + kc[219] * p->cD) + (kc[220] * u_is_def + kc[221] * u_wend)) + kc[222] * x[1]), 1.0);
+        u_cC_new = CLIP((((kb[310] + kc[223] * p->cC) + (kc[224] * u_is_class + kc[225] * u_wend)) + kc[226] * x[1]), 1.0);
+    }
+    if ((x[5] + p->C) != 0.0) {   /* all activations are >= 0 */
+        u_C_new = CLIP((((kb[311] + kc[227] * x[5]) + (kc[228] * u_S_eff + kc[229] * p->C)) + kc[230] * x[1]), 1.0);
+    }
+    if ((x[0] + p->N + p->ee) != 0.0) {   /* all activations are >= 0 */
+        u_N_new = CLIP(((((kb[312] + kc[231] * x[0]) + (kc[232] * p->W + kc[233] * p->N)) + ((kc[234] * u_wordch + kc[235] * x[6]) + (kc[236] * x[17] + kc[237] * p->ee))) + (kc[238] * u_S_eff + kc[239] * p->C)), 1.0);
+        u_ee_new = CLIP(((kb[313] + kc[240] * x[16]) + kc[241] * u_N_new), 1.0);
+    }
+    s->S = u_S_new;
+    s->sq1 = u_sq1_new;
+    s->T = u_T_new;
+    s->F = u_F_new;
+    s->E = u_E_new;
+    s->c = u_c_new;
+    s->fo = u_fo_new;
+    s->FS = u_FS_new;
+    s->FD = u_FD_new;
+    s->oq1 = u_oq1_new;
+    s->oT = u_oT_new;
+    s->C = u_C_new;
+    s->N = u_N_new;
+    s->ee = u_ee_new;
+    s->W = u_W_new;
+    s->d = u_d_new;
+    s->pf = u_pf_new;
+    s->fp = u_fp_new;
+    s->cD = u_cD_new;
+    s->cC = u_cC_new;
+    s->cAt = u_cAt_new;
+    s->atp = u_atp_new;
+    s->cDot = u_cDot_new;
+    s->h1 = u_h1_new;
+    return (uint16_t)(((uint16_t)(u_lab_STR != 0.0) << 0) | ((uint16_t)(u_C_new != 0.0) << 1) | ((uint16_t)(u_N_new != 0.0) << 2) | ((uint16_t)(u_lab_W != 0.0) << 3) | ((uint16_t)(u_ev_KW != 0.0) << 4) | ((uint16_t)(u_ev_BI != 0.0) << 5) | ((uint16_t)(u_ev_FN != 0.0) << 6) | ((uint16_t)(u_ev_CL != 0.0) << 7) | ((uint16_t)(u_ev_DEC != 0.0) << 8) | ((uint16_t)(u_ev_STR != 0.0) << 9));
+}
+static inline __attribute__((always_inline)) int k_bwd_step(const double *restrict kc, const double *restrict kb, uint16_t bits,
+                             const BState *restrict r, BState *restrict s) {
+    const double f_lab_STR = (double)((bits >> 0) & 1);
+    const double f_C_new = (double)((bits >> 1) & 1);
+    const double f_N_new = (double)((bits >> 2) & 1);
+    const double f_lab_W = (double)((bits >> 3) & 1);
+    const double f_ev_KW = (double)((bits >> 4) & 1);
+    const double f_ev_BI = (double)((bits >> 5) & 1);
+    const double f_ev_FN = (double)((bits >> 6) & 1);
+    const double f_ev_CL = (double)((bits >> 7) & 1);
+    const double f_ev_DEC = (double)((bits >> 8) & 1);
+    const double f_ev_STR = (double)((bits >> 9) & 1);
+    const double u_reg_KW = CLIP(((kb[314] + kc[242] * r->b_ev_KW) + (kc[243] * r->u_reg_KW + kc[244] * f_lab_W)), 1.0);
+    const double u_reg_BI = CLIP(((kb[315] + kc[245] * r->b_ev_BI) + (kc[246] * r->u_reg_BI + kc[247] * f_lab_W)), 1.0);
+    const double u_reg_FN = CLIP(((kb[316] + kc[248] * r->b_ev_FN) + (kc[249] * r->u_reg_FN + kc[250] * f_lab_W)), 1.0);
+    const double u_reg_CL = CLIP(((kb[317] + kc[251] * r->b_ev_CL) + (kc[252] * r->u_reg_CL + kc[253] * f_lab_W)), 1.0);
+    const double u_reg_DEC = CLIP(((kb[318] + kc[254] * r->b_ev_DEC) + (kc[255] * r->u_reg_DEC + kc[256] * f_lab_W)), 1.0);
+    const double u_reg_STR = CLIP(((kb[319] + kc[257] * r->b_ev_STR) + (kc[258] * r->u_reg_STR + kc[259] * f_lab_W)), 1.0);
+    s->u_reg_KW = u_reg_KW;
+    s->u_reg_BI = u_reg_BI;
+    s->u_reg_FN = u_reg_FN;
+    s->u_reg_CL = u_reg_CL;
+    s->u_reg_DEC = u_reg_DEC;
+    s->u_reg_STR = u_reg_STR;
+    s->b_ev_KW = f_ev_KW;
+    s->b_ev_BI = f_ev_BI;
+    s->b_ev_FN = f_ev_FN;
+    s->b_ev_CL = f_ev_CL;
+    s->b_ev_DEC = f_ev_DEC;
+    s->b_ev_STR = f_ev_STR;
+    return (int)CLIP(((((kb[320] + kc[260] * u_reg_KW) + (kc[261] * u_reg_BI + kc[262] * u_reg_FN)) + ((kc[263] * u_reg_CL + kc[264] * u_reg_DEC) + (kc[265] * u_reg_STR + kc[266] * f_lab_STR))) + (kc[267] * f_C_new + kc[268] * f_N_new)), 8.0);
+}
+#define K_N_COEF 269
+#define K_N_BIAS 321
+#define K_N_GUARD 54
+#define K_N_FAMILY 1
+#define K_N_PREFIX 3
+#define K_PREFIX_LEN 768
+/* sorted threshold families: {cell, layer, first bias index, rows, coef index of the tied weight} */
+static const int32_t K_FAMILY[K_N_FAMILY + 1][5] = {{0,1,43,255,183},{0,0,0,0,0}};
+/* prefix sums of a unit's weights over a family's columns: {cell, layer, row, first column, rows} */
+static const int32_t K_PREFIX[K_N_PREFIX + 1][5] = {{0,3,15,7,255},{0,3,16,7,255},{0,3,17,7,255},{0,0,0,0,0}};
+#define K_MAX_GUARD 4
+static const int32_t K_COEF[K_N_COEF][4] = {{0,0,4,33},{0,0,1,2},{0,0,1,19},{0,0,1,22},{0,0,2,3},{0,0,2,19},{0,0,2,22},{0,0,3,23},{0,0,6,24},{0,0,6,8},{0,2,0,279},{0,2,0,261},{0,2,0,281},{0,2,0,256},{0,2,0,257},{0,2,0,258},{0,3,0,0},{0,3,0,286},{0,3,0,263},{0,3,0,264},{0,3,0,289},{0,3,1,0},{0,3,1,286},{0,3,1,263},{0,3,1,264},{0,3,1,289},{0,3,2,0},{0,3,2,286},{0,3,2,263},{0,3,2,264},{0,3,2,289},{0,3,3,0},{0,3,3,286},{0,3,3,270},{0,3,3,288},{0,3,4,0},{0,3,4,273},{0,3,4,288},{0,3,5,0},{0,3,5,287},{0,3,5,277},{0,3,5,288},{0,3,5,290},{0,3,7,271},{0,3,7,272},{0,3,7,0},{0,3,7,295},{0,3,7,296},{0,3,8,271},{0,3,8,272},{0,3,8,0},{0,3,8,295},{0,3,8,296},{0,3,8,300},{0,3,8,301},{0,3,9,271},{0,3,9,272},{0,3,9,0},{0,3,9,295},{0,3,9,296},{0,3,10,291},{0,3,10,0},{0,3,10,295},{0,3,10,292},{0,3,10,280},{0,3,11,291},{0,3,11,0},{0,3,11,295},{0,3,11,278},{0,3,12,291},{0,3,12,0},{0,3,12,295},{0,3,12,279},{0,3,12,292},{0,4,0,20},{0,4,0,0},{0,4,0,1},{0,4,0,3},{0,4,0,7},{0,4,0,10},{0,4,1,44},{0,4,1,20},{0,4,1,31},{0,4,1,32},{0,4,1,2},{0,4,1,1},{0,4,1,0},{0,4,1,7},{0,4,1,9},{0,4,2,42},{0,4,2,2},{0,4,2,1},{0,4,2,7},{0,4,2,10},{0,4,2,48},{0,4,3,43},{0,4,3,8},{0,4,3,7},{0,4,3,0},{0,4,3,1},{0,4,3,3},{0,4,3,10},{0,4,4,41},{0,4,4,9},{0,4,4,0},{0,4,4,1},{0,4,4,3},{0,4,4,40},{0,4,4,20},{0,4,4,10},{0,4,4,47},{0,4,5,45},{0,4,5,35},{0,4,5,10},{0,4,6,46},{0,4,6,11},{0,4,6,12},{0,4,7,47},{0,4,7,35},{0,4,7,41},{0,4,7,10},{0,4,8,48},{0,4,8,35},{0,4,8,42},{0,4,8,10},{0,4,9,20},{0,4,9,3},{0,4,9,7},{0,4,9,10},{0,0,0,13},{0,0,5,32},{0,0,5,13},{0,0,7,15},{0,0,7,14},{0,0,7,34},{0,0,7,32},{0,0,7,33},{0,3,18,276},{0,3,18,0},{0,3,18,295},{0,2,1,276},{0,2,1,296},{0,2,1,255},{0,2,2,302},{0,2,2,293},{0,2,2,255},{0,2,2,259},{0,2,3,302},{0,2,3,293},{0,2,3,255},{0,2,3,259},{0,2,4,299},{0,2,4,300},{0,2,4,255},{0,2,4,260},{0,3,14,262},{0,3,14,269},{0,3,14,298},{0,3,14,296},{0,3,14,0},{0,3,14,295},{0,3,19,275},{0,3,19,0},{0,3,19,295},{0,3,19,296},{0,3,19,305},{0,3,19,262},{0,3,19,266},{0,4,11,49},{0,4,11,14},{0,4,12,38},{0,4,12,14},{0,4,12,33},{0,4,13,38},{0,4,13,49},{0,4,13,14},{0,4,13,33},{0,4,20,14},{0,4,20,18},{0,5,0,29},{0,5,0,30},{0,5,0,12},{0,5,0,13},{0,1,0,47},{0,2,5,260},{0,2,5,299},{0,2,6,260},{0,2,6,298},{0,2,6,299},{0,4,19,34},{0,4,19,7},{0,4,19,50},{0,3,15,306},{0,3,16,306},{0,3,17,306},{0,4,14,34},{0,4,14,15},{0,4,14,27},{0,4,14,28},{0,4,15,34},{0,4,15,15},{0,4,15,29},{0,4,15,30},{0,4,16,34},{0,4,16,15},{0,4,16,53},{0,4,16,52},{0,4,16,51},{0,4,17,34},{0,4,17,17},{0,4,17,51},{0,4,17,53},{0,4,17,52},{0,4,18,34},{0,4,18,16},{0,4,18,53},{0,4,18,52},{0,4,18,51},{0,4,18,54},{0,5,1,37},{0,5,1,14},{0,5,1,34},{0,5,1,36},{0,5,2,38},{0,5,2,15},{0,5,2,34},{0,5,2,36},{0,3,6,274},{0,3,6,0},{0,3,6,295},{0,3,6,270},{0,3,13,269},{0,3,13,298},{0,3,13,296},{0,3,13,262},{0,3,13,275},{0,3,13,283},{0,3,13,297},{0,3,13,0},{0,3,13,295},{0,4,10,39},{0,4,10,13},{1,0,0,16},{1,0,0,10},{1,0,0,3},{1,0,1,17},{1,0,1,11},{1,0,1,3},{1,0,2,18},{1,0,2,12},{1,0,2,3},{1,0,3,19},{1,0,3,13},{1,0,3,3},{1,0,4,20},{1,0,4,14},{1,0,4,3},{1,0,5,21},{1,0,5,15},{1,0,5,3},{2,0,0,0},{2,0,0,1},{2,0,0,2},{2,0,0,3},{2,0,0,4},{2,0,0,5},{2,0,0,6},{2,0,0,7},{2,0,0,8}};
+static const int32_t K_BIAS[K_N_BIAS][3] = {{0,0,4},{0,0,1},{0,0,2},{0,0,3},{0,0,6},{0,2,0},{0,3,0},{0,3,1},{0,3,2},{0,3,3},{0,3,4},{0,3,5},{0,3,7},{0,3,8},{0,3,9},{0,3,10},{0,3,11},{0,3,12},{0,4,0},{0,4,1},{0,4,2},{0,4,3},{0,4,4},{0,4,5},{0,4,6},{0,4,7},{0,4,8},{0,4,9},{0,0,0},{0,0,5},{0,0,7},{0,3,18},{0,2,1},{0,2,2},{0,2,3},{0,2,4},{0,3,14},{0,3,19},{0,4,11},{0,4,12},{0,4,13},{0,4,20},{0,5,0},{0,1,0},{0,1,1},{0,1,2},{0,1,3},{0,1,4},{0,1,5},{0,1,6},{0,1,7},{0,1,8},{0,1,9},{0,1,10},{0,1,11},{0,1,12},{0,1,13},{0,1,14},{0,1,15},{0,1,16},{0,1,17},{0,1,18},{0,1,19},{0,1,20},{0,1,21},{0,1,22},{0,1,23},{0,1,24},{0,1,25},{0,1,26},{0,1,27},{0,1,28},{0,1,29},{0,1,30},{0,1,31},{0,1,32},{0,1,33},{0,1,34},{0,1,35},{0,1,36},{0,1,37},{0,1,38},{0,1,39},{0,1,40},{0,1,41},{0,1,42},{0,1,43},{0,1,44},{0,1,45},{0,1,46},{0,1,47},{0,1,48},{0,1,49},{0,1,50},{0,1,51},{0,1,52},{0,1,53},{0,1,54},{0,1,55},{0,1,56},{0,1,57},{0,1,58},{0,1,59},{0,1,60},{0,1,61},{0,1,62},{0,1,63},{0,1,64},{0,1,65},{0,1,66},{0,1,67},{0,1,68},{0,1,69},{0,1,70},{0,1,71},{0,1,72},{0,1,73},{0,1,74},{0,1,75},{0,1,76},{0,1,77},{0,1,78},{0,1,79},{0,1,80},{0,1,81},{0,1,82},{0,1,83},{0,1,84},{0,1,85},{0,1,86},{0,1,87},{0,1,88},{0,1,89},{0,1,90},{0,1,91},{0,1,92},{0,1,93},{0,1,94},{0,1,95},{0,1,96},{0,1,97},{0,1,98},{0,1,99},{0,1,100},{0,1,101},{0,1,102},{0,1,103},{0,1,104},{0,1,105},{0,1,106},{0,1,107},{0,1,108},{0,1,109},{0,1,110},{0,1,111},{0,1,112},{0,1,113},{0,1,114},{0,1,115},{0,1,116},{0,1,117},{0,1,118},{0,1,119},{0,1,120},{0,1,121},{0,1,122},{0,1,123},{0,1,124},{0,1,125},{0,1,126},{0,1,127},{0,1,128},{0,1,129},{0,1,130},{0,1,131},{0,1,132},{0,1,133},{0,1,134},{0,1,135},{0,1,136},{0,1,137},{0,1,138},{0,1,139},{0,1,140},{0,1,141},{0,1,142},{0,1,143},{0,1,144},{0,1,145},{0,1,146},{0,1,147},{0,1,148},{0,1,149},{0,1,150},{0,1,151},{0,1,152},{0,1,153},{0,1,154},{0,1,155},{0,1,156},{0,1,157},{0,1,158},{0,1,159},{0,1,160},{0,1,161},{0,1,162},{0,1,163},{0,1,164},{0,1,165},{0,1,166},{0,1,167},{0,1,168},{0,1,169},{0,1,170},{0,1,171},{0,1,172},{0,1,173},{0,1,174},{0,1,175},{0,1,176},{0,1,177},{0,1,178},{0,1,179},{0,1,180},{0,1,181},{0,1,182},{0,1,183},{0,1,184},{0,1,185},{0,1,186},{0,1,187},{0,1,188},{0,1,189},{0,1,190},{0,1,191},{0,1,192},{0,1,193},{0,1,194},{0,1,195},{0,1,196},{0,1,197},{0,1,198},{0,1,199},{0,1,200},{0,1,201},{0,1,202},{0,1,203},{0,1,204},{0,1,205},{0,1,206},{0,1,207},{0,1,208},{0,1,209},{0,1,210},{0,1,211},{0,1,212},{0,1,213},{0,1,214},{0,1,215},{0,1,216},{0,1,217},{0,1,218},{0,1,219},{0,1,220},{0,1,221},{0,1,222},{0,1,223},{0,1,224},{0,1,225},{0,1,226},{0,1,227},{0,1,228},{0,1,229},{0,1,230},{0,1,231},{0,1,232},{0,1,233},{0,1,234},{0,1,235},{0,1,236},{0,1,237},{0,1,238},{0,1,239},{0,1,240},{0,1,241},{0,1,242},{0,1,243},{0,1,244},{0,1,245},{0,1,246},{0,1,247},{0,1,248},{0,1,249},{0,1,250},{0,1,251},{0,1,252},{0,1,253},{0,1,254},{0,2,5},{0,2,6},{0,4,19},{0,3,15},{0,3,16},{0,3,17},{0,4,14},{0,4,15},{0,4,16},{0,4,17},{0,4,18},{0,5,1},{0,5,2},{0,3,6},{0,3,13},{0,4,10},{1,0,0},{1,0,1},{1,0,2},{1,0,3},{1,0,4},{1,0,5},{2,0,0}};
+/* units whose block is skipped when the listed input columns are all zero; the loader proves each one */
+static const int32_t K_GUARD[K_N_GUARD][3 + K_MAX_GUARD] = {{0,0,1,2,-1,-1,-1},{0,0,2,3,-1,-1,-1},{0,0,3,23,-1,-1,-1},{0,0,6,24,-1,-1,-1},{0,2,0,279,-1,-1,-1},{0,3,0,0,-1,-1,-1},{0,3,1,0,-1,-1,-1},{0,3,2,0,-1,-1,-1},{0,3,3,0,-1,-1,-1},{0,3,4,0,-1,-1,-1},{0,3,5,0,-1,-1,-1},{0,3,7,271,272,-1,-1},{0,3,8,271,272,-1,-1},{0,3,9,272,-1,-1,-1},{0,3,10,291,-1,-1,-1},{0,3,11,291,-1,-1,-1},{0,3,12,291,-1,-1,-1},{0,4,0,20,7,10,-1},{0,4,1,20,7,9,-1},{0,4,2,42,2,10,-1},{0,4,3,43,8,10,-1},{0,4,4,41,9,20,10},{0,4,5,45,35,-1,-1},{0,4,6,46,11,-1,-1},{0,4,7,47,35,-1,-1},{0,4,8,48,35,-1,-1},{0,4,9,20,7,10,-1},{0,0,0,13,-1,-1,-1},{0,0,5,32,-1,-1,-1},{0,0,7,15,14,-1,-1},{0,3,18,276,-1,-1,-1},{0,2,1,276,296,-1,-1},{0,2,2,293,-1,-1,-1},{0,2,3,293,-1,-1,-1},{0,2,4,299,300,-1,-1},{0,3,14,262,298,-1,-1},{0,3,19,275,305,-1,-1},{0,4,11,14,-1,-1,-1},{0,4,12,14,-1,-1,-1},{0,4,13,14,-1,-1,-1},{0,4,20,14,18,-1,-1},{0,2,5,260,-1,-1,-1},{0,2,6,260,-1,-1,-1},{0,4,19,34,-1,-1,-1},{0,4,14,34,-1,-1,-1},{0,4,15,34,-1,-1,-1},{0,4,16,34,-1,-1,-1},{0,4,17,34,-1,-1,-1},{0,4,18,34,-1,-1,-1},{0,5,1,37,14,-1,-1},{0,5,2,38,15,-1,-1},{0,3,6,274,295,-1,-1},{0,3,13,269,296,-1,-1},{0,4,10,13,-1,-1,-1}};

builds/fast-994/pylexnet_weights.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:470c37fad3e65ae40ac70be637ab57626042f34db93107dec411b1eeca0ee6fa
+size 106744

compile.py

@@ -0,0 +1,838 @@
+"""pylexnet 'hand' lane: a Python syntax-highlighter network designed unit by unit, lexnet style.
+
+No automaton is compiled. Every unit below does one designed job: ~60 booleans for strings, comments,
+numbers, words and contexts, a length counter, two hash-gate registers, a sorted threshold family on the
+word hash, and one "distance to the nearest vocabulary word of category c" unit per category.
+Activation is clip(x, 0, hi) everywhere.
+
+Two vocabulary matchers, chosen by --matcher:
+  add  (default) additive retrieval: per-position value tables (positions 0..P-1 plus a constant each, a tail
+       table, three last-byte tables) solved so that every vocabulary word's sum of values is its class target
+       t*2^30 (t = 1 keyword, 2 def, 3 class, 4 builtin, 5 magic), rounded to integers (window +-64). A one-hot
+       position chain gates the values into an accumulator; three delayed registers add the last bytes' values at
+       the word's end; nine threshold units read the class windows. ~1 number per word. Not exact: a non-vocabulary
+       word whose sum lands in a window is a false positive (fpcheck.py measures them on the fit split; pinning
+       words to 0 via zero_words.txt is possible when the system stays solvable).
+  hash exact: a polynomial hash h over the word, a sorted tied threshold family ge_k = clip1(h - T_k + 1) (the rank
+       of h), and per category E_c = clip1(h - sum_k ge_k * D_c[k]) with D_c telescoping to the largest category-c
+       threshold <= h, so E_c == 0 exactly when h is a category-c word. 2 numbers per word.
+
+Writes pylexnet_weights.bin (PYLX02) and pylexnet_kernel.h (architecture only, with a structure hash).
+usage: compile.py [--matcher add|hash] [--vocab N] [--addP P] [--sim TEXT]   N = builtin/magic words kept, by fit frequency.
+"""
+import keyword
+import re
+import struct
+import sys
+from pathlib import Path
+
+import numpy as np
+
+HERE = Path(__file__).resolve().parent
+
+# ---------------------------------------------------------------- byte features (the embedding table)
+FEATS = ["digit", "nl", "dq", "sq", "bs", "hash", "dot", "at", "lbr", "rbr",
+         "obr", "cbr", "fcch", "code", "pf", "po", "eE", "pm"]
+MATCHER = "add"        # "add": additive retrieval (IDEAS.md idea 1, default). "hash": sorted threshold family on a polynomial hash
+ADD_P, ADD_L = 7, 3    # additive matcher: value tables (+ a constant) for positions 0..P-1, a tail table, 3 last-byte tables (L is fixed at 3)
+ADD_EPS = 64           # window half-width around each class target
+ADD_T = 2 ** 30        # class targets are t * ADD_T, t = 1 keyword, 2 def, 3 class, 4 builtin, 5 magic
+if "--matcher" in sys.argv:
+    MATCHER = sys.argv[sys.argv.index("--matcher") + 1]
+if "--addP" in sys.argv:
+    ADD_P = int(sys.argv[sys.argv.index("--addP") + 1])
+if "--addL" in sys.argv:
+    ADD_L = int(sys.argv[sys.argv.index("--addL") + 1])
+if MATCHER == "add":
+    FEATS += [f"v{p}" for p in range(ADD_P)] + ["vt"] + [f"e{j}" for j in range(1, ADD_L + 1)]
+NF = len(FEATS)
+FI = {f: i for i, f in enumerate(FEATS)}
+ADD_TABLES = {}        # feature name -> {code: value}, filled by solve_additive
+
+
+def code_of(b):
+    """Word-character code for the hash: a-z 1..26, A-Z 27..52, _ 53, digits 54, bytes >= 127 55, else 0."""
+    if 97 <= b <= 122:
+        return b - 96
+    if 65 <= b <= 90:
+        return b - 64 + 26
+    if b == 95:
+        return 53
+    if 48 <= b <= 57:
+        return 54
+    if b >= 127:
+        return 55
+    return 0
+
+
+BASE = 56          # hash base: one more than the largest code
+WIN = 7            # exact polynomial over the first WIN chars; later chars add code + 8*position
+HMAX = 2 ** 41     # ceiling of the hash registers (> BASE**WIN + tail)
+KG = 2 ** 42       # gate constant for the hash units: larger than any hash value
+
+
+def emb_table():
+    E = np.zeros((128, NF))
+    for b in range(128):
+        r = E[b]
+        c = code_of(b)
+        isalpha = (65 <= b <= 90) or (97 <= b <= 122) or b == 95 or b == 127
+        isdigit = 48 <= b <= 57
+        r[FI["digit"]] = isdigit
+        r[FI["nl"]] = b == 10
+        r[FI["dq"]] = b == 34
+        r[FI["sq"]] = b == 39
+        r[FI["bs"]] = b == 92
+        r[FI["hash"]] = b == 35
+        r[FI["dot"]] = b == 46
+        r[FI["at"]] = b == 64
+        r[FI["lbr"]] = b == 123
+        r[FI["rbr"]] = b == 125
+        r[FI["obr"]] = b in (40, 91, 123)
+        r[FI["cbr"]] = b in (41, 93, 125)
+        r[FI["fcch"]] = b in (125, 58, 33)       # } : !  end a replacement field at depth 0
+        r[FI["code"]] = c
+        r[FI["pf"]] = b in (102, 70)             # f F
+        r[FI["eE"]] = b in (101, 69)
+        r[FI["pm"]] = b in (43, 45)
+        r[FI["po"]] = b in (114, 82, 98, 66, 117, 85)   # r R b B u U
+        for f, tab in ADD_TABLES.items():
+            r[FI[f]] = tab.get(c, 0) if c else 0
+    return E
+
+
+def word_feats(w):
+    """Additive matcher: the (table, code) terms of a word."""
+    cs = [code_of(b) for b in w.encode()]
+    out = {}
+    for i, c in enumerate(cs):
+        k = (f"v{i}" if i < ADD_P else "vt", c)
+        out[k] = out.get(k, 0) + 1
+        k = (k[0], 0)                       # the position's constant (the gate unit's bias)
+        out[k] = out.get(k, 0) + 1
+    for j in range(1, ADD_L + 1):
+        if len(cs) >= j:
+            k = (f"e{j}", cs[-j])
+            out[k] = out.get(k, 0) + 1
+    return out
+
+
+def solve_additive(vocab):
+    """Integer value tables such that every vocabulary word's sum lands on its class target (within ADD_EPS).
+    Orthogonal matching pursuit picks a sparse support, least squares solves it, rounding keeps it integer."""
+    words = sorted(vocab)
+    tgt = {"KW": 1, "BI": 4, "MAG": 5}
+    zero = [w for w in (HERE / "zero_words.txt").read_text().split() if w not in vocab] if (HERE / "zero_words.txt").exists() else []
+    T = np.array([ADD_T * (2 if w == "def" else 3 if w == "class" else tgt[vocab[w]]) for w in words] + [0.0] * len(zero))
+    words = words + zero          # frequent non-vocabulary words pinned to 0 (chosen on the fit split, see zero_words.txt)
+    cols = {}
+    rows = [word_feats(w) for w in words]
+    for r in rows:
+        for k in r:
+            cols.setdefault(k, len(cols))
+    A = np.zeros((len(words), len(cols)))
+    for i, r in enumerate(rows):
+        for k, n in r.items():
+            A[i, cols[k]] = n
+    S, x = [], np.zeros(len(cols))
+    for _ in range(len(cols)):
+        res = T - A @ x
+        if np.abs(res).max() < 1e-3:
+            break
+        corr = A.T @ res
+        corr[S] = 0
+        j = int(np.argmax(np.abs(corr)))
+        if corr[j] == 0:
+            break
+        S.append(j)
+        xs, *_ = np.linalg.lstsq(A[:, S], T, rcond=None)
+        x = np.zeros(len(cols)); x[S] = xs
+    x = np.round(x)
+    res = A @ x - T
+    unsolved = int((np.abs(res) > ADD_EPS).sum())
+    for (f, c), j in cols.items():
+        if x[j] != 0:
+            ADD_TABLES.setdefault(f, {})[c] = int(x[j])
+    nz = int((x != 0).sum())
+    vmax = max(abs(v) for tab in ADD_TABLES.values() for v in tab.values())
+    print(f"additive matcher: {len(words) - len(zero)} words + {len(zero)} zero-pinned, {len(cols)} candidate values, support {nz}, max |value| {vmax}, "
+          f"unsolved {unsolved}, max residual {np.abs(res).max():g}")
+    assert unsolved == 0, "additive system not solvable with these features"
+    return vmax
+
+
+def word_hash(w):
+    h = 0
+    for i, ch in enumerate(w.encode()):
+        c = code_of(ch)
+        h = BASE * h + c if i < WIN else h + c + 8 * i
+    return h
+
+
+# ---------------------------------------------------------------- vocabulary
+import pygments.lexers.python as _pl
+_src = Path(_pl.__file__).read_text()
+_b = _src.index("'builtins': [")
+_a = _src.index("'magicfuncs': [")
+_v = _src.index("'magicvars': [")
+BUILTINS = set(re.findall(r"'(\w+)'", _src[_b:_a])) - {"builtins", "exceptions"} | {"self", "cls", "Ellipsis", "NotImplemented"}
+MAGIC = set(re.findall(r"'(__\w+__)'", _src[_a:_v])) - BUILTINS
+KEYWORDS = set(keyword.kwlist)
+# frequency order measured once on accuracy2's fit split (vocab_prio.txt); words below its cut-off come last, alphabetically
+PRIO = [w for w, _ in eval("[" + (HERE / "vocab_prio.txt").read_text() + "]")]
+
+
+def pick_vocab(limit):
+    """Keywords always; builtins and magic words by fit-corpus frequency up to `limit`."""
+    ranked = [w for w in PRIO if w in BUILTINS or w in MAGIC]
+    ranked += sorted((BUILTINS | MAGIC) - set(ranked))
+    chosen = ranked if limit is None else ranked[:limit]
+    return {w: ("BI" if w in BUILTINS else "MAG") for w in chosen} | {w: "KW" for w in KEYWORDS}
+
+
+# ---------------------------------------------------------------- the network, unit by unit
+class Unit:
+    def __init__(self, name, ins, b=0.0, hi=1.0, guard=None, fam=None):
+        self.name, self.ins, self.b, self.hi, self.guard, self.fam = name, ins, float(b), float(hi), guard, fam
+
+
+UNITS = []
+
+
+def u(name, b=0.0, hi=1.0, guard=None, fam=None, **ins):
+    """ins: ref -> weight; refs written as x_feat (byte feature), p_name (previous state) or a unit name."""
+    d = {}
+    for k, w in ins.items():
+        ref = "x." + k[2:] if k.startswith("x_") else "p." + k[2:] if k.startswith("p_") else "u." + k
+        d[ref] = float(w)
+    UNITS.append(Unit(name, d, b, hi, guard, fam))
+    return name
+
+
+STATE = ["S", "sq1", "T", "F", "E", "c", "fo", "FS", "FD", "oq1", "oT", "C", "N", "ee", "W", "d", "pf", "fp",
+         "cD", "cC", "cAt", "atp", "cDot"]
+STATE += ["h1"] if MATCHER == "hash" else [f"pos{p}" for p in range(ADD_P)] + ["tail", "A", "e1", "e2", "e2d", "e3", "e3d", "e3dd"]
+STATE_UNIT = {s: s + "_new" for s in STATE}
+for _p in range(ADD_P):
+    STATE_UNIT[f"pos{_p}"] = f"pos{_p}"
+STATE_UNIT["tail"] = "tail"
+CATS = ["KW", "BI", "MAG"]
+
+
+def build(vocab):
+    UNITS.clear()
+    # ---- layer 0: from the byte and the previous state only
+    u("wordch", guard=["x.code"], x_code=1)                                           # word character (any code > 0)
+    u("qmd", guard=["x.dq"], x_dq=1, p_sq1=-1, p_E=-1)             # unescaped matching " in a "-string
+    u("qms", guard=["x.sq"], x_sq=1, p_sq1=1, p_E=-1, b=-1)        # unescaped matching ' in a '-string
+    u("c2", guard=["p.c"], p_c=1, b=-1)                             # quote run >= 2
+    if MATCHER == "hash":
+        u("lt7", p_d=-1, b=WIN)                                     # fewer than WIN chars of the word seen
+    u("wend", guard=["p.W"], p_W=1, x_code=-1)                      # a word ended just before this byte
+    u("fieldopen", guard=["p.fo"], p_fo=1, x_lbr=-1)                # { was a field start: this byte is code
+    u("pf_new", guard=["x.po", "x.pf"], x_po=3, x_pf=3, p_pf=2, p_W=-1, p_d=-1, b=-2)   # word so far could be a string prefix
+    # ---- strings. S_eff: this byte is inside a string (p.S unless "" just closed or a field just opened)
+    u("S_eff", guard=["p.S"], p_S=4, fieldopen=-4, p_T=1, qmd=1, qms=1, c2=-1, b=-3)
+    u("fp_new", guard=["x.pf", "p.fp"], x_pf=1, p_fp=1, wordch=1, b=-1)   # word contains f
+    u("sclose", guard=["u.S_eff"], S_eff=2, p_T=-1, qmd=1, qms=1, p_c=-1, b=-2)
+    u("tclose", guard=["u.S_eff"], S_eff=3, p_T=1, qmd=1, qms=1, p_c=1, b=-6)
+    u("tripleopen", guard=["u.S_eff"], S_eff=3, p_T=-1, qmd=1, qms=1, p_c=1, b=-5)
+    u("nlclose", guard=["u.S_eff"], S_eff=1, p_T=-1, x_nl=1, p_E=-1, b=-1)
+    u("E_new", guard=["u.S_eff"], S_eff=1, x_bs=1, p_E=-1, b=-1)
+    u("fo_new", guard=["u.S_eff"], S_eff=1, p_F=1, x_lbr=1, p_E=-1, p_fo=-1, b=-2)
+    u("C_new", guard=["x.hash", "p.C"], x_hash=1, S_eff=-1, p_C=1, x_nl=-1)
+    u("open", guard=["x.dq", "x.sq"], x_dq=1, x_sq=1, S_eff=-1, p_C=-1, p_N=-1)
+    u("open_f", guard=["x.dq", "x.sq"], x_dq=1, x_sq=1, S_eff=-1, p_C=-1, p_N=-1, p_pf=1, p_fp=1, b=-2)
+    u("open_qs", guard=["x.sq"], x_dq=1, x_sq=2, S_eff=-1, p_C=-1, p_N=-1, b=-1)
+    u("fieldclose", guard=["p.FS"], p_FS=1, S_eff=-1, p_C=-1, p_FD=-1, x_fcch=1, b=-1)
+    u("inc", guard=["p.FS"], p_FS=1, S_eff=-1, p_C=-1, x_obr=1, b=-1)
+    u("dec", guard=["p.FS"], p_FS=4, S_eff=-4, p_C=-4, x_cbr=1, p_FD=1, b=-5)
+    u("S_new", guard=["u.S_eff", "u.open", "u.fieldclose"], S_eff=1, sclose=-1, tclose=-1, nlclose=-1, open=1, fieldclose=1)
+    u("c_new", hi=3, guard=["u.S_eff", "u.open", "u.open_qs"], p_c=1, S_eff=10, qmd=5, qms=5, tripleopen=-5, tclose=-5, sclose=-5,
+      open=10, open_qs=5, b=-14)
+    u("T_new", guard=["p.T", "u.tripleopen", "u.fieldclose"], p_T=2, tripleopen=2, tclose=-2, open=-2, fieldclose=1, p_oT=1, b=-1)
+    u("F_new", guard=["p.F", "u.open_f", "u.fieldclose"], p_F=1, open_f=2, open=-1, sclose=-1, tclose=-1, nlclose=-1, fieldclose=1)
+    u("sq1_new", guard=["p.sq1", "u.open_qs", "u.S_eff", "u.fieldclose"], p_sq1=2, open_qs=4, sclose=-2, tclose=-2, nlclose=-2,
+      p_S=-2, S_eff=2, fieldclose=1, p_oq1=1, b=-1)     # quote type; cleared when the string dies (unless a nested one opens); restored from oq1
+    u("FS_new", guard=["p.FS", "u.fieldopen"], p_FS=1, fieldopen=1, fieldclose=-1)
+    u("FD_new", hi=3, guard=["p.FD", "u.inc"], p_FD=1, inc=1, dec=-1)
+    u("oq1_new", guard=["p.oq1", "u.fieldopen"], p_oq1=2, fieldopen=1, p_sq1=1, fieldclose=-2, b=-1)   # outer quote type, saved at a field's start
+    u("oT_new", guard=["p.oT", "u.fieldopen"], p_oT=2, fieldopen=1, p_T=1, fieldclose=-2, b=-1)
+    u("lab_STR", guard=["u.S_eff", "u.open", "u.fieldclose"], S_eff=1, nlclose=-1, open=1, fieldclose=1)
+    # ---- numbers and words
+    u("N_new", guard=["x.digit", "p.N"], x_digit=6, p_W=-3, p_N=5, wordch=2, x_dot=2, x_pm=1, p_ee=1, S_eff=-2, p_C=-2, b=-6)
+    u("ee_new", guard=["u.N_new"], x_eE=1, N_new=1, b=-1)                    # an exponent e: a sign may follow
+    u("W_new", guard=["u.wordch", "p.W"], wordch=2, x_digit=-1, p_W=1, p_N=-2, S_eff=-2, p_C=-2, b=-1)
+    u("d_new", hi=30, guard=["u.W_new"], p_d=1, W_new=40, b=-39)
+    if MATCHER == "hash":
+        u("gA", hi=HMAX, guard=["p.W"], p_h1=1, p_W=KG, wordch=KG, lt7=KG, b=-3 * KG)       # old hash, still in the window
+        u("gB", hi=HMAX, guard=["p.W"], p_h1=1, p_W=KG, wordch=KG, lt7=-KG, b=-2 * KG)      # old hash, past the window
+        u("cA", hi=HMAX, guard=["u.W_new"], x_code=1, W_new=KG, lt7=KG, b=-2 * KG)          # this char's code, in window
+        u("cB", hi=HMAX, guard=["u.W_new"], x_code=1, p_d=8, W_new=KG, lt7=-KG, b=-KG)      # past the window: code + 8*position
+        u("h1_new", hi=HMAX, gA=BASE, gB=1, cA=1, cB=1)
+    else:
+        build_additive(vocab)
+    if MATCHER == "add":
+        u("is_def", guard=["u.wend"], wend=1, ge2=1, le2=1, b=-2)
+        u("is_class", guard=["u.wend"], wend=1, ge3=1, le3=1, b=-2)
+        u("cD_new", guard=["p.cD", "u.is_def"], p_cD=1, is_def=2, wend=-1, x_nl=-1)
+        u("cC_new", guard=["p.cC", "u.is_class"], p_cC=1, is_class=2, wend=-1, x_nl=-1)
+        u("atp_new", guard=["x.at"], x_at=1, S_eff=-1, p_C=-1)
+        u("cAt_new", guard=["p.cAt", "p.atp"], p_cAt=1, p_atp=1, wordch=1, wend=-1, b=-1)
+        u("cDot_new", guard=["x.dot", "p.cDot"], x_dot=4, S_eff=-4, p_C=-4, p_N=-4, p_cDot=2, wordch=1, wend=-1, b=-2)
+        u("ev_DEC", guard=["u.wend"], wend=1, p_cAt=1, b=-1)
+        u("ev_CL", guard=["u.wend"], wend=1, p_cC=1, p_cAt=-1, b=-1)
+        # keyword targets 1, 2 (def), 3 (class): a value in one of the three windows scores 4 on the sorted sum, 3 otherwise
+        u("ev_KW", guard=["u.wend"], wend=3, ge1=1, le1=1, ge2=1, le2=1, ge3=1, le3=1, p_cAt=-1, p_cC=-1, p_cD=-1, b=-6)
+        u("ev_FN", guard=["u.wend"], wend=3, ge5=1, le5=1, p_cD=2, p_cAt=-2, p_cC=-2, b=-4)   # after def, or a magic method name
+        u("ev_BI", guard=["u.wend"], wend=1, ge4=1, le4=1, p_cAt=-1, p_cC=-1, p_cD=-1, p_cDot=-1, b=-2)
+        u("ev_STR", guard=["u.wend"], wend=1, open=1, p_pf=1, b=-2)
+        u("lab_W", guard=["u.W_new", "u.atp_new"], W_new=1, atp_new=1)
+        return [], []
+    # ---- vocabulary: sorted thresholds (one tied family), one distance unit per category
+    byhash = {}
+    for w, cat in sorted(vocab.items()):
+        byhash.setdefault(word_hash(w), set()).add(cat)
+    for h, cats in byhash.items():
+        assert len(cats) == 1, ("hash collision across categories", h, cats)
+    hashes = sorted(byhash)
+    cat_of = [next(iter(byhash[h])) for h in hashes]
+    for k, h in enumerate(hashes):
+        u(f"ge_{k}", p_h1=1, b=1 - h, fam="ge")
+    for cat in CATS:
+        prev = 0
+        deltas = {}
+        for k, h in enumerate(hashes):
+            if cat_of[k] == cat:
+                deltas[f"ge_{k}"] = -(h - prev)
+                prev = h
+        u(f"E_{cat}", p_h1=1, **deltas)       # 0 exactly when h is a category word, else 1
+    kdef, kclass = hashes.index(word_hash("def")), hashes.index(word_hash("class"))
+    nxt = lambda k: {f"ge_{k + 1}": -1} if k + 1 < len(hashes) else {}
+    u("is_def", guard=["u.wend"], wend=1, E_KW=-1, b=-1, **{f"ge_{kdef}": 1}, **nxt(kdef))
+    u("is_class", guard=["u.wend"], wend=1, E_KW=-1, b=-1, **{f"ge_{kclass}": 1}, **nxt(kclass))
+    # ---- contexts: after def, after class, after @, after a dot
+    u("cD_new", guard=["p.cD", "u.is_def"], p_cD=1, is_def=2, wend=-1, x_nl=-1)
+    u("cC_new", guard=["p.cC", "u.is_class"], p_cC=1, is_class=2, wend=-1, x_nl=-1)
+    u("atp_new", guard=["x.at"], x_at=1, S_eff=-1, p_C=-1)
+    u("cAt_new", guard=["p.cAt", "p.atp"], p_cAt=1, p_atp=1, wordch=1, wend=-1, b=-1)
+    u("cDot_new", guard=["x.dot", "p.cDot"], x_dot=4, S_eff=-4, p_C=-4, p_N=-4, p_cDot=2, wordch=1, wend=-1, b=-2)
+    # ---- events at a word's end (read by the backward pass) and the word label
+    u("ev_DEC", guard=["u.wend"], wend=1, p_cAt=1, b=-1)
+    u("ev_CL", guard=["u.wend"], wend=1, p_cC=1, p_cAt=-1, b=-1)
+    u("ev_KW", guard=["u.wend"], wend=1, E_KW=-1, p_cAt=-1, p_cC=-1, p_cD=-1)
+    u("ev_FN", guard=["u.wend"], wend=2, E_MAG=-1, p_cD=1, p_cAt=-1, p_cC=-1, b=-1)       # after def, or a magic method name
+    u("ev_BI", guard=["u.wend"], wend=1, E_BI=-1, p_cAt=-1, p_cC=-1, p_cD=-1, p_cDot=-1)
+    u("ev_STR", guard=["u.wend"], wend=1, open=1, p_pf=1, b=-2)                   # the word was a string prefix
+    u("lab_W", guard=["u.W_new", "u.atp_new"], W_new=1, atp_new=1)
+    return hashes, cat_of
+
+
+def build_additive(vocab):
+    """Additive retrieval: A accumulates shifted position values over the word, three delayed last-byte values are added
+    at the end, and the class is read from windows around the targets."""
+    vmax = solve_additive(vocab)
+    C = 2 * vmax                  # shift: every gated value (+ its position constant) is >= 0; the readout subtracts C per byte
+    M = 4 * C + 4                 # gate constant
+    AMAX = 2 ** 45; K = 2 ** 46
+    u("pos0", guard=["u.W_new"], W_new=1, p_W=-1)                                   # first byte of a word
+    for p in range(1, ADD_P):
+        u(f"pos{p}", guard=[f"p.pos{p - 1}"], **{f"p_pos{p - 1}": 1}, W_new=1, b=-1)
+    u("tail", guard=[f"p.pos{ADD_P - 1}", "p.tail"], **{f"p_pos{ADD_P - 1}": 1}, p_tail=1, W_new=1, b=-1)
+    for p in range(ADD_P):
+        u(f"a{p}", hi=2 * C, guard=[f"u.pos{p}"], **{f"x_v{p}": 1, f"pos{p}": M}, b=C - M + ADD_TABLES.get(f"v{p}", {}).get(0, 0))
+    u("at_", hi=2 * C, guard=["u.tail"], x_vt=1, tail=M, b=C - M + ADD_TABLES.get("vt", {}).get(0, 0))
+    u("A_new", hi=AMAX, guard=["u.W_new"], p_A=1, W_new=K, b=-K, **{f"a{p}": 1 for p in range(ADD_P)}, at_=1)
+    for j in range(1, ADD_L + 1):
+        u(f"e{j}_new", hi=2 * C, guard=["u.W_new"], **{f"x_e{j}": 1}, W_new=M, b=C - M)
+    u("e2d_new", hi=2 * C, guard=["p.e2"], p_e2=1)
+    u("e3d_new", hi=2 * C, guard=["p.e3"], p_e3=1)
+    u("e3dd_new", hi=2 * C, guard=["p.e3d"], p_e3d=1)
+    u("m3", hi=3, guard=["p.d"], p_d=1)                                               # min(length, 3): last-byte terms present
+    u("SE", hi=AMAX, p_A=1, p_e1=1, p_e2d=1, p_e3dd=1, p_d=-C, m3=-C)   # the word's value sum
+    for t in (1, 2, 3, 4, 5):
+        u(f"ge{t}", SE=1, b=1 - (t * ADD_T - ADD_EPS))
+        u(f"le{t}", SE=-1, b=1 + (t * ADD_T + ADD_EPS))
+
+
+# Kernel blocks: a unit is computed inside the first block whose condition (any listed atom non-zero) provably
+# covers its guard, following guard chains down to byte features and previous state. Everything else is unconditional.
+BLOCKS = [("vocab", ["u.wend"]),
+          ("str", ["p.S", "p.fo", "p.FS", "p.FD", "p.c", "p.T", "p.F", "p.sq1", "p.oq1", "p.oT", "x.dq", "x.sq"]),
+          ("word", ["p.W", "x.code", "x.dot", "x.at", "p.atp", "p.cD", "p.cC", "p.cAt", "p.cDot", "x.po", "x.pf", "p.fp", "p.d"]
+           + ([] if MATCHER == "hash" else [f"p.pos{p}" for p in range(ADD_P)] + ["p.tail", "p.e2", "p.e3", "p.e3d"])),
+          ("num", ["x.digit", "p.N", "p.ee"]),
+          ("cmt", ["x.hash", "p.C"])]
+
+BITS = ["lab_STR", "C_new", "N_new", "lab_W", "ev_KW", "ev_BI", "ev_FN", "ev_CL", "ev_DEC", "ev_STR"]
+EV = ["KW", "BI", "FN", "CL", "DEC", "STR"]
+CLASSES = ["text", "keyword", "builtin", "string", "comment", "number", "func", "class", "decorator"]
+BSTATE = ["u.reg_" + e for e in EV] + ["b.ev_" + e for e in EV]
+# class id read out by one integer unit: direct labels, or the carried word class on word bytes
+HEAD = {"b.lab_STR": 3, "b.C_new": 4, "b.N_new": 5, "u.reg_KW": 1, "u.reg_BI": 2, "u.reg_STR": 3, "u.reg_FN": 6, "u.reg_CL": 7, "u.reg_DEC": 8}
+
+
+# ---------------------------------------------------------------- layering and matrices
+def layer_units(units):
+    """Layers by dependency on same-step units (guards count); a family gets a layer of its own (tied weights)."""
+    byname = {x.name: x for x in units}
+    lvl = {}
+
+    def L(n):
+        if n in lvl:
+            return lvl[n]
+        x = byname[n]
+        deps = [r for r in x.ins if r.startswith("u.")] + [g for g in (x.guard or []) if g.startswith("u.")]
+        lvl[n] = 1 + max([L(r[2:]) for r in deps], default=-1)
+        return lvl[n]
+    for x in units:
+        L(x.name)
+    layers = []
+    for l in range(1 + max(lvl.values())):
+        xs = [x for x in units if lvl[x.name] == l]
+        for fam in sorted({x.fam for x in xs if x.fam}):
+            layers.append([x for x in xs if x.fam == fam])
+        rest = [x for x in xs if not x.fam]
+        if rest:
+            layers.append(rest)
+    return layers
+
+
+def check_guards(units, emb):
+    """Same proof as the C loader: with the guard inputs at 0, bias plus every other positive term at its ceiling must be <= 0."""
+    hi = {"u." + x.name: x.hi for x in units}
+    for s_, un in STATE_UNIT.items():
+        if "u." + un in hi:  # matcher-specific state units exist only in that matcher's build
+            hi["p." + s_] = hi["u." + un]
+    for f in FEATS:
+        hi["x." + f] = float(emb[:, FI[f]].max())
+    bad = []
+    for x in units:
+        if not x.guard or getattr(x, "inherited", False):
+            continue
+        bound = x.b + sum(w * hi[r] for r, w in x.ins.items() if w > 0 and r not in x.guard)
+        if bound > 0:
+            bad.append((x.name, bound))
+    assert not bad, f"guards not provable: {bad}"
+
+
+def inherit_guards(units):
+    """A unit that is not an output and whose consumers all share one guard set inherits it (it is dead otherwise)."""
+    cons = {x.name: [] for x in units}
+    for x in units:
+        for r in x.ins:
+            if r.startswith("u."):
+                cons[r[2:]].append(x)
+    outputs = set(STATE_UNIT.values()) | set(BITS)
+    changed = True
+    while changed:
+        changed = False
+        for x in units:
+            if x.guard is not None or x.name in outputs or not cons[x.name]:
+                continue
+            gs = {tuple(c.guard) if c.guard is not None else None for c in cons[x.name]}
+            if len(gs) == 1 and None not in gs:
+                x.guard = list(gs.pop())
+                x.inherited = True
+                changed = True
+
+
+def build_cell(layers, inputs, outputs_needed):
+    """Layered units -> (W, b, hi, skip, names, tied) over a growing vector, lexnet style.
+    A tied layer stores one weight row and one ceiling for all its rows (a threshold family)."""
+    vec = list(inputs)
+    mats = []
+    later_needs = []
+    for l, xs in enumerate(layers):
+        need = set(outputs_needed)
+        for ys in layers[l + 1:]:
+            for y in ys:
+                need |= set(y.ins)
+        later_needs.append(need)
+    for l, xs in enumerate(layers):
+        idx = {n: i for i, n in enumerate(vec)}
+        tied = all(x.fam for x in xs)
+        if tied:
+            assert all(x.ins == xs[0].ins and x.hi == xs[0].hi for x in xs)
+        W = np.zeros((1 if tied else len(xs), len(vec)))
+        b = np.zeros(len(xs)); hi = np.zeros(1 if tied else len(xs))
+        for r, x in enumerate(xs):
+            if not tied or r == 0:
+                for ref, w in x.ins.items():
+                    W[r, idx[ref]] = w
+                hi[r] = x.hi
+            b[r] = x.b
+        skip = [i for i, n in enumerate(vec) if n in later_needs[l]]
+        mats.append((W, b, hi, np.array(skip, dtype=np.int64), [x.name for x in xs], tied))
+        vec = ["u." + x.name for x in xs] + [vec[i] for i in skip]
+    return mats, vec
+
+
+# ---------------------------------------------------------------- weights file
+def write_weights(path, emb, fwd, fvec, bwd, bvec, head_W, head_b, h0):
+    out = bytearray(b"PYLX02\0\0")
+    out += struct.pack("<6q", NF, len(STATE), len(BITS), len(BSTATE), len(CLASSES), head_W.shape[1])
+    out += emb.astype("<f8").tobytes()
+    for cell in (fwd, bwd):
+        out += struct.pack("<q", len(cell))
+        for W, b, hi, skip, names, tied in cell:
+            out += struct.pack("<4q", len(names), W.shape[1], len(skip), int(tied))
+            out += W.astype("<f8").tobytes() + b.astype("<f8").tobytes() + hi.astype("<f8").tobytes() + skip.astype("<q").tobytes()
+    f_state = [fvec.index("u." + STATE_UNIT[s]) for s in STATE]
+    f_out = [fvec.index("u." + bname) for bname in BITS]
+    b_state = [bvec.index(n) for n in BSTATE]
+    out += np.array(f_state, dtype="<q").tobytes() + np.array(f_out, dtype="<q").tobytes() + np.array(b_state, dtype="<q").tobytes()
+    out += h0.astype("<f8").tobytes() + head_W.astype("<f8").tobytes() + head_b.astype("<f8").tobytes()
+    Path(path).write_bytes(out)
+    return len(out)
+
+
+def count(emb, fwd, bwd, head_W, head_b, h0):
+    """Every non-zero number the network uses: embedding, weights, biases, ceilings, head, initial state."""
+    nz = lambda a: int((np.asarray(a) != 0).sum())
+    tied = nz(emb) + nz(head_W) + nz(head_b) + nz(h0) + 1    # +1: the head ceiling
+    untied = tied
+    for cell in (fwd, bwd):
+        for W, b, hi, skip, names, t in cell:
+            tied += nz(W) + nz(b) + nz(hi)
+            untied += (nz(W) + nz(hi)) * (len(names) if t else 1) + nz(b)
+    return tied, untied
+
+
+# ---------------------------------------------------------------- kernel codegen
+def struct_hash(fwd, bwd, f_state, f_out, b_state, head_W, head_cols):
+    """FNV-1a over shapes, ceilings, skips, sparsity masks and index maps; identical to the C loader's."""
+    h = 0xCBF29CE484222325
+
+    def by(bs):
+        nonlocal h
+        for c in bs:
+            h ^= c; h = (h * 0x100000001B3) & 0xFFFFFFFFFFFFFFFF
+
+    def i64(x):
+        by(struct.pack("<q", x))
+    i64(NF); i64(len(STATE)); i64(len(BITS)); i64(len(BSTATE)); i64(len(CLASSES)); i64(head_cols)
+    for cell in (fwd, bwd):
+        i64(len(cell))
+        for W, b, hi, skip, names, tied in cell:
+            i64(len(names)); i64(W.shape[1]); i64(len(skip)); i64(int(tied))
+            by(hi.astype("<f8").tobytes())
+            for s in skip:
+                i64(int(s))
+            by(bytes((W != 0).astype(np.uint8).ravel()))
+    for i in f_state: i64(i)
+    for i in f_out: i64(i)
+    for i in b_state: i64(i)
+    by(bytes((head_W != 0).astype(np.uint8).ravel()))
+    return h
+
+
+def cfmt(v):
+    return repr(float(v)) if v != int(v) else f"{int(v)}.0"
+
+
+def tree(terms):
+    """Balanced sum: every value is an exact integer, so the order does not change the result but halves the latency."""
+    while len(terms) > 1:
+        terms = [f"({terms[i]} + {terms[i + 1]})" if i + 1 < len(terms) else terms[i] for i in range(0, len(terms), 2)]
+    return terms[0]
+
+
+_FVEC = {}
+
+
+def fvec_index(cell, l, ref):
+    return _FVEC[(id(cell), l)][ref]
+
+
+def gen_kernel(path, fwd, fvec, bwd, bvec, head_W, head_b, head_hi, shash, units, bunits):
+    coef, bias, guards, families, prefixes = [], [], [], [], []
+    lines = []
+    A = lines.append
+    A("/* Generated by compile.py: the network's ARCHITECTURE as C. No weight values live here: pylexnet.c")
+    A(" * loads them, checks the structure hash and the guard inequalities, and gathers kc[]/kb[] by K_COEF/K_BIAS. */")
+    A("#include <stdint.h>")
+    A("#include <string.h>")
+    A(f"#define K_STRUCT_HASH 0x{shash:016X}ULL")
+    A(f"#define K_N_FEATS {NF}")
+    A(f"#define K_N_CLASSES {len(CLASSES)}")
+    A(f"#define K_N_BITS {len(BITS)}")
+    A("#ifndef CLIP")
+    A("#define CLIP(x, hi) clipn((x), (hi))")
+    A("static inline double clipn(double x, double hi) { x = x < 0.0 ? 0.0 : x; return x > hi ? hi : x; }")
+    A("#endif")
+    A("/* A tied, sorted threshold family is a step vector: unit k is 1 iff kb[k] + w*h >= 1, true for k < rank.")
+    A(" * The loader verifies integrality, tying and monotonicity, so a binary search gives the same values. */")
+    A("static inline int k_rank(const double *kb, int n, double w, double h) {")
+    A("    int lo = 0, hi = n;")
+    A("    while (lo < hi) { int mid = (lo + hi) >> 1; if (kb[mid] + w * h >= 1.0) lo = mid + 1; else hi = mid; }")
+    A("    return lo;")
+    A("}")
+    A("typedef struct { double " + ", ".join(STATE) + "; } FState;")
+    A("typedef struct { double " + ", ".join(n.replace(".", "_") for n in BSTATE) + "; } BState;")
+    A("static inline void k_init_state(FState *s, const double *h0) { memcpy(s, h0, sizeof *s); }")
+    # ---- forward
+    A("static inline __attribute__((always_inline)) uint16_t k_fwd_step(const double *restrict kc, const double *restrict kb, const double *restrict kp, const double *restrict x,")
+    A("                                  const FState *restrict p, FState *restrict s) {")
+    byname = {x.name: x for x in units}
+    famidx, fams = {}, {}
+    for x in units:
+        if x.fam:
+            famidx[x.name] = len(fams.setdefault(x.fam, []))
+            fams[x.fam].append(x)
+
+    def ref(r):
+        if r.startswith("x."):
+            return f"x[{FI[r[2:]]}]"
+        if r.startswith("p."):
+            return f"p->{r[2:]}"
+        x = byname[r[2:]]
+        return f"(double)({famidx[x.name]} < r_{x.fam})" if x.fam else f"u_{x.name}"
+    scalars = [x.name for x in units if not x.fam]
+    A("    double " + ", ".join(f"u_{n} = 0.0" for n in scalars) + ";")
+    for fam in fams:
+        A(f"    int r_{fam} = 0;")
+
+    def covered(atom, cond, seen=()):
+        if atom in cond:
+            return True
+        if not atom.startswith("u.") or atom in seen:
+            return False
+        g = byname[atom[2:]].guard
+        return g is not None and all(covered(a, cond, seen + (atom,)) for a in g)
+    blk = {}
+    for x in units:
+        blk[x.name] = None
+        if x.guard:
+            for bname, cond in BLOCKS:
+                if all(covered(a, cond) for a in x.guard):
+                    blk[x.name] = bname
+                    break
+    cond_of = dict(BLOCKS)
+    layer_of = {}
+    for l, (W, b, hi, skip, names, tied) in enumerate(fwd):
+        for row, n in enumerate(names):
+            layer_of[n] = (l, row)
+
+    def deps(x):
+        d = {r[2:] for r in x.ins if r.startswith("u.")}
+        if blk[x.name]:
+            d |= {a[2:] for a in cond_of[blk[x.name]] if a.startswith("u.")}
+        return d
+    remaining = {x.name for x in units}
+    done = set()
+    cur = None
+    nseg = 0
+    while remaining:
+        ready = [byname[n] for n in remaining if deps(byname[n]) <= done]
+        assert ready, "cyclic dependency"
+        bybl = {}
+        for x in ready:
+            bybl.setdefault(blk[x.name], []).append(x)
+        if cur not in bybl:
+            cur = max(bybl, key=lambda k: (len(bybl[k]), k is None))
+            if cur is not None:
+                A("    if ((" + " + ".join(ref(a) for a in cond_of[cur]) + ") != 0.0) {   /* all activations are >= 0 */")
+                nseg += 1
+        batch = sorted(bybl[cur], key=lambda x: (layer_of[x.name], x.fam or "", famidx.get(x.name, 0)))
+        indent = "        " if cur is not None else "    "
+        i = 0
+        while i < len(batch):
+            x = batch[i]
+            l, row = layer_of[x.name]
+            if x.fam:
+                j = i
+                while j < len(batch) and batch[j].fam == x.fam and famidx[batch[j].name] == famidx[x.name] + (j - i):
+                    j += 1
+                run = batch[i:j]
+                assert len(run) == len(fams[x.fam]) and famidx[x.name] == 0, "a family is emitted whole"
+                (inref, _), = run[0].ins.items()
+                cb, bb = len(coef), len(bias)
+                coef.append((0, l, 0, fvec_index(fwd, l, inref)))      # tied: one weight for the family
+                for y in run:
+                    bias.append((0,) + layer_of[y.name])
+                families.append((0, l, bb, len(run), cb))
+                A(f"{indent}r_{x.fam} = k_rank(kb + {bb}, {len(run)}, kc[{cb}], {ref(inref)});")
+                i = j
+                continue
+            terms = [f"kb[{len(bias)}]"]
+            bias.append((0, l, row))
+            famins = {}
+            for inref, w in x.ins.items():
+                y = byname[inref[2:]] if inref.startswith("u.") else None
+                if y is not None and y.fam:
+                    famins.setdefault(y.fam, []).append(inref)
+            for inref, w in x.ins.items():
+                y = byname[inref[2:]] if inref.startswith("u.") else None
+                if y is not None and y.fam and len(famins[y.fam]) >= 4:
+                    if inref == famins[y.fam][0]:
+                        # all of this family's weights into the unit, as a prefix sum indexed by the rank (built at load)
+                        cols = [fvec_index(fwd, l, "u." + z.name) for z in fams[y.fam]]
+                        assert cols == list(range(cols[0], cols[0] + len(cols))), "family columns are contiguous"
+                        pb = sum(n + 1 for _, _, _, _, n in prefixes)
+                        prefixes.append((0, l, row, cols[0], len(cols)))
+                        terms.append(f"kp[{pb} + r_{y.fam}]")
+                    continue
+                terms.append(f"kc[{len(coef)}] * {ref(inref)}")
+                coef.append((0, l, row, fvec_index(fwd, l, inref)))
+            A(f"{indent}u_{x.name} = CLIP({tree(terms)}, {cfmt(x.hi)});")
+            if x.guard and not getattr(x, "inherited", False):
+                guards.append((0, l, row, [fvec_index(fwd, l, gr) for gr in x.guard]))
+            i += 1
+        for x in batch:
+            done.add(x.name); remaining.discard(x.name)
+        nxt = [n for n in remaining if blk[n] == cur and deps(byname[n]) <= done]
+        if not nxt and cur is not None:
+            A("    }")
+            cur = None
+    print(f"kernel: {nseg} block segments; units per block " + str({b: sum(1 for x in units if blk[x.name] == b) for b in [None] + [b for b, _ in BLOCKS]}))
+    for sname in STATE:
+        A(f"    s->{sname} = u_{STATE_UNIT[sname]};")
+    A("    return (uint16_t)(" + " | ".join(f"((uint16_t)(u_{bn} != 0.0) << {i})" for i, bn in enumerate(BITS)) + ");")
+    A("}")
+    # ---- backward: carried word classes, then the integer class id
+    A("static inline __attribute__((always_inline)) int k_bwd_step(const double *restrict kc, const double *restrict kb, uint16_t bits,")
+    A("                             const BState *restrict r, BState *restrict s) {")
+    for i, bn in enumerate(BITS):
+        A(f"    const double f_{bn} = (double)((bits >> {i}) & 1);")
+
+    def bref(rf):
+        if rf.startswith("b."):
+            return "f_" + rf[2:]
+        if rf.startswith("r."):
+            return "r->" + rf[2:].replace(".", "_")
+        return "u_" + rf[2:]
+    for l, (W, b, hi, skip, names, tied) in enumerate(bwd):
+        for row, n in enumerate(names):
+            x = bunits[n]
+            terms = [f"kb[{len(bias)}]"]
+            bias.append((1, l, row))
+            for inref, w in x.ins.items():
+                terms.append(f"kc[{len(coef)}] * {bref(inref)}")
+                coef.append((1, l, row, fvec_index(bwd, l, inref)))
+            A(f"    const double u_{n} = CLIP({tree(terms)}, {cfmt(x.hi)});")
+    for n in BSTATE:
+        A(f"    s->{n.replace('.', '_')} = {bref(n) if not n.startswith('u.') else 'u_' + n[2:]};")
+    terms = [f"kb[{len(bias)}]"]
+    bias.append((2, 0, 0))
+    for c in np.nonzero(head_W[0])[0]:
+        terms.append(f"kc[{len(coef)}] * {bref(bvec[c])}")
+        coef.append((2, 0, 0, int(c)))
+    A(f"    return (int)CLIP({tree(terms)}, {cfmt(head_hi)});")
+    A("}")
+    A(f"#define K_N_COEF {len(coef)}")
+    A(f"#define K_N_BIAS {len(bias)}")
+    A(f"#define K_N_GUARD {len(guards)}")
+    A(f"#define K_N_FAMILY {len(families)}")
+    A(f"#define K_N_PREFIX {len(prefixes)}")
+    A(f"#define K_PREFIX_LEN {sum(n + 1 for *_, n in prefixes)}")
+    A("/* sorted threshold families: {cell, layer, first bias index, rows, coef index of the tied weight} */")
+    A("static const int32_t K_FAMILY[K_N_FAMILY + 1][5] = {" + "".join("{%d,%d,%d,%d,%d}," % f for f in families) + "{0,0,0,0,0}};")
+    A("/* prefix sums of a unit's weights over a family's columns: {cell, layer, row, first column, rows} */")
+    A("static const int32_t K_PREFIX[K_N_PREFIX + 1][5] = {" + "".join("{%d,%d,%d,%d,%d}," % f for f in prefixes) + "{0,0,0,0,0}};")
+    A("#define K_MAX_GUARD 4")
+    A("static const int32_t K_COEF[K_N_COEF][4] = {" + ",".join("{%d,%d,%d,%d}" % c for c in coef) + "};")
+    A("static const int32_t K_BIAS[K_N_BIAS][3] = {" + ",".join("{%d,%d,%d}" % c for c in bias) + "};")
+    A("/* units whose block is skipped when the listed input columns are all zero; the loader proves each one */")
+    A("static const int32_t K_GUARD[K_N_GUARD][3 + K_MAX_GUARD] = {" +
+      ",".join("{%d,%d,%d,%s}" % (c, l, r, ",".join(str(v) for v in (cols + [-1] * 4)[:4])) for c, l, r, cols in guards) + "};")
+    Path(path).write_text("\n".join(lines) + "\n")
+    return len(coef), len(bias), len(guards)
+
+
+# ---------------------------------------------------------------- simulator (debugging: a third evaluation of the same matrices)
+def run_layers(v, layers):
+    for W, b, hi, skip, names, tied in layers:
+        pre = (W @ v)
+        if tied:
+            pre = np.full(len(names), pre[0])
+        new = np.clip(pre + b, 0, hi if not tied else hi[0])
+        v = np.concatenate([new, v[skip]])
+    return v
+
+
+def simulate(text, emb, fwd, fvec, f_state, f_out, bwd, b_state, head_W, head_b, head_hi, h0):
+    data = text.encode() + b"\n"
+    st = h0.copy()
+    allbits = []
+    for t, byte in enumerate(data):
+        v = run_layers(np.concatenate([emb[byte if byte < 128 else 127], st]), fwd)
+        st = v[f_state]
+        bits = [int(v[i] != 0) for i in f_out]
+        allbits.append(bits)
+        on = [n[2:] + ("" if v[i] == 1 else f"={v[i]:g}") for i, n in enumerate(fvec)
+              if n.startswith("u.") and v[i] != 0 and not n.startswith("u.ge_")]
+        print(f"{t:3} {chr(byte)!r:6} bits={''.join(map(str, bits))} " + " ".join(on))
+    r = np.zeros(len(b_state))
+    out = []
+    for t in range(len(data) - 1, -1, -1):
+        v = run_layers(np.concatenate([np.array(allbits[t], dtype=float), r]), bwd)
+        r = v[b_state]
+        out.append(int(np.clip(head_W @ v + head_b, 0, head_hi)[0]))
+    print("classes:", "".join(str(c) for c in out[::-1][:-1]))
+
+
+# ---------------------------------------------------------------- main
+class BUnit:
+    def __init__(self, name, ins, b=0.0, hi=1.0):
+        self.name, self.ins, self.b, self.hi = name, ins, b, hi
+
+
+def main():
+    limit = None
+    if "--vocab" in sys.argv:
+        limit = int(sys.argv[sys.argv.index("--vocab") + 1])
+    vocab = pick_vocab(limit)
+    hashes, cat_of = build(vocab)
+    check_guards(UNITS, emb_table())
+    inherit_guards(UNITS)
+    layers = layer_units(UNITS)
+    inputs = ["x." + f for f in FEATS] + ["p." + s for s in STATE]
+    needed = {"u." + STATE_UNIT[s] for s in STATE} | {"u." + b for b in BITS}
+    fwd, fvec = build_cell(layers, inputs, needed)
+    vec = list(inputs)
+    for l, (W, b, hi, skip, names, tied) in enumerate(fwd):
+        _FVEC[(id(fwd), l)] = {n: i for i, n in enumerate(vec)}
+        vec = ["u." + n for n in names] + [vec[i] for i in skip]
+    # backward cell: reg_X(i) = clip1(ev_X(i+1) + reg_X(i+1) + lab_W(i) - 1): set at a word's end, carried over
+    # the word's bytes, zero elsewhere. Carries this byte's event bits for the next step.
+    bunits = {}
+    for e in EV:
+        bunits["reg_" + e] = BUnit("reg_" + e, {"r.b.ev_" + e: 1.0, "r.u.reg_" + e: 1.0, "b.lab_W": 1.0}, b=-1.0)
+    binputs = ["b." + bn for bn in BITS] + ["r." + n for n in BSTATE]
+    bidx = {n: i for i, n in enumerate(binputs)}
+    Wb = np.zeros((len(EV), len(binputs))); bb = -np.ones(len(EV)); hb = np.ones(len(EV))
+    for r, e in enumerate(EV):
+        for ref, w in bunits["reg_" + e].ins.items():
+            Wb[r, bidx[ref]] = w
+    bskip = [bidx["b." + bn] for bn in BITS]
+    bwd = [(Wb, bb, hb, np.array(bskip, dtype=np.int64), ["reg_" + e for e in EV], False)]
+    _FVEC[(id(bwd), 0)] = bidx
+    bvec_out = ["u.reg_" + e for e in EV] + ["b." + bn for bn in BITS]
+    hcol = {n: i for i, n in enumerate(bvec_out)}
+    head_W = np.zeros((1, len(bvec_out))); head_b = np.zeros(1); head_hi = 8.0
+    for n, w in HEAD.items():
+        head_W[0, hcol[n]] = w
+    emb = emb_table()
+    h0 = np.zeros(len(STATE))
+    size = write_weights(HERE / "pylexnet_weights.bin", emb, fwd, fvec, bwd, bvec_out, head_W, head_b, h0)
+    f_state = [fvec.index("u." + STATE_UNIT[s]) for s in STATE]
+    f_out = [fvec.index("u." + bn) for bn in BITS]
+    b_state = [bvec_out.index(n) for n in BSTATE]
+    shash = struct_hash(fwd, bwd, f_state, f_out, b_state, head_W, head_W.shape[1])
+    ncoef, nbias, nguard = gen_kernel(HERE / "pylexnet_kernel.h", fwd, fvec, bwd, bvec_out, head_W, head_b, head_hi, shash, UNITS, bunits)
+    if "--sim" in sys.argv:
+        simulate(sys.argv[sys.argv.index("--sim") + 1], emb, fwd, fvec, f_state, f_out, bwd, b_state, head_W, head_b, head_hi, h0)
+    tied, untied = count(emb, fwd, bwd, head_W, head_b, h0)
+    nlog = sum(1 for x in UNITS if not x.fam and not x.name.startswith("E_"))
+    vocab_numbers = 2 * len(hashes) if MATCHER == "hash" else sum(len(t) for t in ADD_TABLES.values())
+    print(f"units {len(UNITS)} forward in {len(fwd)} layers ({nlog} logic), state {len(STATE)}, bits {len(BITS)}, vocabulary {len(vocab)} words "
+          f"({sum(1 for c in vocab.values() if c == 'KW')} keywords, {sum(1 for c in vocab.values() if c == 'BI')} builtins, "
+          f"{sum(1 for c in vocab.values() if c == 'MAG')} magic) = {len(hashes)} thresholds; kernel coef {ncoef} bias {nbias} guards {nguard}")
+    print(f"non-zero weights {tied} (every number: embedding {int((emb != 0).sum())}, vocabulary {vocab_numbers}; "
+          f"{untied} if the threshold family's weight and ceiling were not tied)")
+    print(f"weights file {size // 1024} KiB")
+
+
+if __name__ == "__main__":
+    main()

demo.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:962068fdc4eb6dab035aa97c40011bcb46e60ffd3eb8e2f77537672b3af2d99a
+size 204263

pylexnet.c

@@ -0,0 +1,369 @@
+/* pylexnet (hand lane): run the hand-designed Python highlighter network.
+ *
+ * Two paths over the same weights file:
+ *   default   the generated kernel in pylexnet_kernel.h (architecture only; values gathered from the file).
+ *             The loader checks the structure hash and proves every guard inequality from the loaded values.
+ *   --interp  the generic layered sparse evaluation (clip(W v + b, 0, hi) + carried entries), no kernel.
+ * usage: pylexnet [-w weights.bin] [--interp] (--classes [file] | --bench file)
+ *   --classes writes one class byte (0..8) per input byte; without a file it reads stdin.
+ */
+#include <math.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+
+typedef struct {
+    int64_t rows, cols, nskip, tied;     /* tied: one weight row and one ceiling shared by all rows */
+    const double *W, *b, *hi;
+    const int64_t *skip;
+    int *rowptr, *col;
+    double *val;
+} Layer;
+
+typedef struct {
+    int64_t n_feats, n_fstate, n_fout, n_bstate, n_classes, head_cols, maxw;
+    const double *emb, *h0, *head_W, *head_b;
+    const int64_t *f_state, *f_out, *b_state;
+    Layer *fwd, *bwd;
+    int64_t n_fwd, n_bwd;
+} Net;
+
+static int use_interp = 0;
+
+/* ---- loading ------------------------------------------------------------- */
+typedef struct { const unsigned char *p, *end; } Cursor;
+
+static const void *take(Cursor *c, size_t bytes) {
+    if (c->p + bytes > c->end) { fprintf(stderr, "weights: truncated\n"); exit(1); }
+    const void *r = c->p;
+    c->p += bytes;
+    return r;
+}
+static int64_t take_i64(Cursor *c) { int64_t v; memcpy(&v, take(c, 8), 8); return v; }
+
+static Layer *load_cell(Cursor *c, int64_t *n, int64_t *maxw) {
+    *n = take_i64(c);
+    Layer *L = calloc((size_t)*n, sizeof(Layer));
+    for (int64_t l = 0; l < *n; l++) {
+        Layer *y = &L[l];
+        y->rows = take_i64(c); y->cols = take_i64(c); y->nskip = take_i64(c); y->tied = take_i64(c);
+        int64_t wrows = y->tied ? 1 : y->rows;
+        y->W = take(c, (size_t)(wrows * y->cols) * 8);
+        y->b = take(c, (size_t)y->rows * 8);
+        y->hi = take(c, (size_t)wrows * 8);
+        y->skip = take(c, (size_t)y->nskip * 8);
+        if (y->cols > *maxw) *maxw = y->cols;
+        if (y->rows + y->nskip > *maxw) *maxw = y->rows + y->nskip;
+        int nnz = 0;
+        for (int64_t i = 0; i < wrows * y->cols; i++) nnz += y->W[i] != 0;
+        if (y->tied) nnz *= (int)y->rows;
+        y->rowptr = malloc((size_t)(y->rows + 1) * sizeof(int));
+        y->col = malloc((size_t)(nnz + 1) * sizeof(int));
+        y->val = malloc((size_t)(nnz + 1) * sizeof(double));
+        int k = 0;
+        for (int64_t r = 0; r < y->rows; r++) {
+            y->rowptr[r] = k;
+            const double *w = y->W + (y->tied ? 0 : r) * y->cols;
+            for (int64_t cc = 0; cc < y->cols; cc++) if (w[cc] != 0) y->col[k] = (int)cc, y->val[k++] = w[cc];
+        }
+        y->rowptr[y->rows] = k;
+    }
+    return L;
+}
+
+static Net load(const char *path) {
+    FILE *f = fopen(path, "rb");
+    if (!f) { perror(path); exit(1); }
+    fseek(f, 0, SEEK_END);
+    size_t n = (size_t)ftell(f);
+    fseek(f, 0, SEEK_SET);
+    unsigned char *blob = malloc(n);
+    if (fread(blob, 1, n, f) != n) { perror(path); exit(1); }
+    fclose(f);
+    Cursor c = { blob, blob + n };
+    if (memcmp(take(&c, 8), "PYLX02\0\0", 8)) { fprintf(stderr, "%s: not a PYLX02 weight file\n", path); exit(1); }
+    Net net = {0};
+    net.n_feats = take_i64(&c); net.n_fstate = take_i64(&c); net.n_fout = take_i64(&c);
+    net.n_bstate = take_i64(&c); net.n_classes = take_i64(&c); net.head_cols = take_i64(&c);
+    net.emb = take(&c, (size_t)(128 * net.n_feats) * 8);
+    net.maxw = net.n_feats + net.n_fstate;
+    if (net.n_fout + net.n_bstate > net.maxw) net.maxw = net.n_fout + net.n_bstate;
+    net.fwd = load_cell(&c, &net.n_fwd, &net.maxw);
+    net.bwd = load_cell(&c, &net.n_bwd, &net.maxw);
+    net.f_state = take(&c, (size_t)net.n_fstate * 8);
+    net.f_out = take(&c, (size_t)net.n_fout * 8);
+    net.b_state = take(&c, (size_t)net.n_bstate * 8);
+    net.h0 = take(&c, (size_t)net.n_fstate * 8);
+    net.head_W = take(&c, (size_t)net.head_cols * 8);      /* one integer-valued readout unit: the class id */
+    net.head_b = take(&c, 8);
+    if (c.p != c.end) { fprintf(stderr, "weights: %zu trailing bytes\n", (size_t)(c.end - c.p)); exit(1); }
+    return net;
+}
+
+/* ---- generic inference --------------------------------------------------- */
+static void run_cell(const Layer *layers, int64_t nl, double *v, double *tmp) {
+    for (int64_t l = 0; l < nl; l++) {
+        const Layer *L = &layers[l];
+        for (int64_t r = 0; r < L->rows; r++) {
+            double s = L->b[r];
+            for (int k = L->rowptr[r]; k < L->rowptr[r + 1]; k++) s += L->val[k] * v[L->col[k]];
+            double hi = L->hi[L->tied ? 0 : r];
+            tmp[r] = s < 0 ? 0 : (s > hi ? hi : s);
+        }
+        for (int64_t k = 0; k < L->nskip; k++) tmp[L->rows + k] = v[L->skip[k]];
+        memcpy(v, tmp, (size_t)(L->rows + L->nskip) * sizeof(double));
+    }
+}
+
+/* n input bytes followed by one virtual newline, so a word at the end of the file still ends. */
+static void classify_interp(const Net *net, const unsigned char *src, size_t n, unsigned char *cls, uint16_t *bits) {
+    double *v = malloc((size_t)net->maxw * 8), *tmp = malloc((size_t)net->maxw * 8);
+    double *h = malloc((size_t)net->n_fstate * 8), *r = calloc((size_t)net->n_bstate, 8);
+    memcpy(h, net->h0, (size_t)net->n_fstate * 8);
+    for (size_t t = 0; t <= n; t++) {
+        unsigned c = t < n ? (src[t] < 128 ? src[t] : 127) : 10;
+        memcpy(v, net->emb + c * net->n_feats, (size_t)net->n_feats * 8);
+        memcpy(v + net->n_feats, h, (size_t)net->n_fstate * 8);
+        run_cell(net->fwd, net->n_fwd, v, tmp);
+        for (int64_t i = 0; i < net->n_fstate; i++) h[i] = v[net->f_state[i]];
+        uint16_t bt = 0;
+        for (int64_t i = 0; i < net->n_fout; i++) bt |= (uint16_t)((v[net->f_out[i]] != 0) << i);
+        bits[t] = bt;
+    }
+    for (size_t t = n + 1; t-- > 0;) {
+        for (int64_t i = 0; i < net->n_fout; i++) v[i] = (bits[t] >> i) & 1;
+        memcpy(v + net->n_fout, r, (size_t)net->n_bstate * 8);
+        run_cell(net->bwd, net->n_bwd, v, tmp);
+        for (int64_t i = 0; i < net->n_bstate; i++) r[i] = v[net->b_state[i]];
+        double s = net->head_b[0];
+        for (int64_t c = 0; c < net->head_cols; c++) s += net->head_W[c] * v[c];
+        s = s < 0 ? 0 : (s > net->n_classes - 1 ? net->n_classes - 1 : s);
+        if (t < n) cls[t] = (unsigned char)s;
+    }
+    free(v); free(tmp); free(h); free(r);
+}
+
+/* ---- compiled mode ------------------------------------------------------- */
+#include <emmintrin.h>
+static inline double clip_sse(double x, double hi) { return _mm_cvtsd_f64(_mm_min_sd(_mm_max_sd(_mm_set_sd(x), _mm_setzero_pd()), _mm_set_sd(hi))); }
+static inline double clip_fm(double x, double hi) { return __builtin_fmin(__builtin_fmax(x, 0.0), hi); }
+#include "pylexnet_kernel.h"
+
+typedef struct { uint64_t h; } Fnv;
+static void fnv_bytes(Fnv *f, const void *p, size_t n) {
+    for (size_t i = 0; i < n; i++) { f->h ^= ((const unsigned char *)p)[i]; f->h *= 0x100000001B3ULL; }
+}
+static void fnv_i64(Fnv *f, int64_t x) { fnv_bytes(f, &x, 8); }
+static void fnv_mask(Fnv *f, const double *a, int64_t n) {
+    for (int64_t i = 0; i < n; i++) { unsigned char m = a[i] != 0; fnv_bytes(f, &m, 1); }
+}
+static uint64_t struct_hash(const Net *net) {
+    Fnv f = { 0xCBF29CE484222325ULL };
+    fnv_i64(&f, net->n_feats); fnv_i64(&f, net->n_fstate); fnv_i64(&f, net->n_fout);
+    fnv_i64(&f, net->n_bstate); fnv_i64(&f, net->n_classes); fnv_i64(&f, net->head_cols);
+    const Layer *cells[2] = { net->fwd, net->bwd };
+    int64_t counts[2] = { net->n_fwd, net->n_bwd };
+    for (int c = 0; c < 2; c++) {
+        fnv_i64(&f, counts[c]);
+        for (int64_t l = 0; l < counts[c]; l++) {
+            const Layer *L = &cells[c][l];
+            fnv_i64(&f, L->rows); fnv_i64(&f, L->cols); fnv_i64(&f, L->nskip); fnv_i64(&f, L->tied);
+            fnv_bytes(&f, L->hi, (size_t)(L->tied ? 1 : L->rows) * 8);
+            for (int64_t k = 0; k < L->nskip; k++) fnv_i64(&f, L->skip[k]);
+            fnv_mask(&f, L->W, (L->tied ? 1 : L->rows) * L->cols);
+        }
+    }
+    for (int64_t i = 0; i < net->n_fstate; i++) fnv_i64(&f, net->f_state[i]);
+    for (int64_t i = 0; i < net->n_fout; i++) fnv_i64(&f, net->f_out[i]);
+    for (int64_t i = 0; i < net->n_bstate; i++) fnv_i64(&f, net->b_state[i]);
+    fnv_mask(&f, net->head_W, net->head_cols);
+    return f.h;
+}
+
+/* Every number in the file must be an integer: then all activations are exact integers in double arithmetic,
+ * sums may be re-associated, and a clip1 unit is exactly 0 or 1. */
+static int all_integers(const Net *net) {
+    int bad = 0;
+    for (int64_t i = 0; i < 128 * net->n_feats; i++) bad += net->emb[i] != (int64_t)net->emb[i];
+    const Layer *cells[2] = { net->fwd, net->bwd };
+    int64_t counts[2] = { net->n_fwd, net->n_bwd };
+    for (int c = 0; c < 2; c++)
+        for (int64_t l = 0; l < counts[c]; l++) {
+            const Layer *L = &cells[c][l];
+            int64_t wrows = L->tied ? 1 : L->rows;
+            for (int64_t i = 0; i < wrows * L->cols; i++) bad += L->W[i] != (int64_t)L->W[i];
+            for (int64_t i = 0; i < L->rows; i++) bad += L->b[i] != (int64_t)L->b[i];
+            for (int64_t i = 0; i < wrows; i++) bad += L->hi[i] != (int64_t)L->hi[i];
+        }
+    for (int64_t i = 0; i < net->head_cols; i++) bad += net->head_W[i] != (int64_t)net->head_W[i];
+    for (int64_t i = 0; i < net->n_fstate; i++) bad += net->h0[i] != (int64_t)net->h0[i];
+    return bad;
+}
+
+/* Sorted threshold families: tied, one input, ceiling 1, biases monotone so the outputs form a step vector.
+ * Then rank(h) by binary search reproduces every unit, and a consumer's weights over the family reduce to a
+ * prefix sum indexed by the rank. Both tables are built here from the loaded values. */
+static int check_families(const Net *net) {
+    int bad = 0;
+    for (int g = 0; g < K_N_FAMILY; g++) {
+        int cell = K_FAMILY[g][0], l = K_FAMILY[g][1], n = K_FAMILY[g][3];
+        const Layer *L = &(cell ? net->bwd : net->fwd)[l];
+        int nz = 0;
+        for (int64_t c = 0; c < L->cols; c++) nz += L->W[c] != 0;
+        if (!L->tied || L->rows != n || L->hi[0] != 1.0 || nz != 1) { bad++; continue; }
+        double w = 0;
+        for (int64_t c = 0; c < L->cols; c++) if (L->W[c] != 0) w = L->W[c];
+        for (int r = 1; r < n; r++) if ((w > 0 && L->b[r] > L->b[r - 1]) || (w < 0 && L->b[r] < L->b[r - 1])) bad++;
+    }
+    return bad;
+}
+
+static double *build_prefix(const Net *net) {
+    double *kp = malloc(sizeof(double) * (K_PREFIX_LEN + 1));
+    int pos = 0;
+    for (int g = 0; g < K_N_PREFIX; g++) {
+        int cell = K_PREFIX[g][0], l = K_PREFIX[g][1], r = K_PREFIX[g][2], c0 = K_PREFIX[g][3], n = K_PREFIX[g][4];
+        const Layer *L = &(cell ? net->bwd : net->fwd)[l];
+        kp[pos] = 0;
+        for (int k = 0; k < n; k++) kp[pos + k + 1] = kp[pos + k] + L->W[(L->tied ? 0 : r) * L->cols + c0 + k];
+        pos += n + 1;
+    }
+    return kp;
+}
+
+static void gather(const Net *net, double *kc, double *kb) {
+    for (int i = 0; i < K_N_COEF; i++) {
+        int cell = K_COEF[i][0], l = K_COEF[i][1], r = K_COEF[i][2], c = K_COEF[i][3];
+        if (cell == 2) kc[i] = net->head_W[c];
+        else { const Layer *L = &(cell ? net->bwd : net->fwd)[l]; kc[i] = L->W[(L->tied ? 0 : r) * L->cols + c]; }
+    }
+    for (int i = 0; i < K_N_BIAS; i++) {
+        int cell = K_BIAS[i][0], l = K_BIAS[i][1], r = K_BIAS[i][2];
+        kb[i] = cell == 2 ? net->head_b[r] : (cell ? net->bwd : net->fwd)[l].b[r];
+    }
+}
+
+/* Upper bound of an entry of a layer's input vector, from the loaded ceilings and embedding table.
+ * layer -1 is the cell's own input: byte features then carried state (forward), bits then state (backward). */
+static double entry_max(const Net *net, int cell, int64_t layer, int64_t idx, int depth) {
+    if (depth > 64) return 1e300;
+    const Layer *L = cell ? net->bwd : net->fwd;
+    if (layer < 0) {
+        if (cell == 0) {
+            if (idx < net->n_feats) {
+                double m = 0;
+                for (int b = 0; b < 128; b++) if (net->emb[b * net->n_feats + idx] > m) m = net->emb[b * net->n_feats + idx];
+                return m;
+            }
+            return entry_max(net, 0, net->n_fwd - 1, net->f_state[idx - net->n_feats], depth + 1);
+        }
+        if (idx < net->n_fout) return 1.0;
+        return entry_max(net, 1, net->n_bwd - 1, net->b_state[idx - net->n_fout], depth + 1);
+    }
+    if (idx < L[layer].rows) return L[layer].hi[L[layer].tied ? 0 : idx];
+    return entry_max(net, cell, layer - 1, L[layer].skip[idx - L[layer].rows], depth + 1);
+}
+
+/* Each guarded unit must be provably 0 when its guard columns are 0: bias plus every other positive
+ * contribution at its input's ceiling must not exceed 0. Returns the number of failed guards. */
+static int check_guards(const Net *net) {
+    int bad = 0;
+    for (int g = 0; g < K_N_GUARD; g++) {
+        int cell = K_GUARD[g][0], l = K_GUARD[g][1], r = K_GUARD[g][2];
+        const Layer *L = &(cell ? net->bwd : net->fwd)[l];
+        double bound = L->b[r];
+        for (int64_t c = 0; c < L->cols; c++) {
+            double w = L->W[(L->tied ? 0 : r) * L->cols + c];
+            if (w <= 0) continue;
+            int guarded = 0;
+            for (int k = 0; k < K_MAX_GUARD; k++) if (K_GUARD[g][3 + k] == c) guarded = 1;
+            if (!guarded) bound += w * entry_max(net, cell, l - 1, c, 0);
+        }
+        if (bound > 0) { bad++; fprintf(stderr, "guard %d (cell %d layer %d row %d) not provable: bound %g\n", g, cell, l, r, bound); }
+    }
+    return bad;
+}
+
+static void classify_compiled(const Net *net, const double *kc, const double *kb, const double *kp,
+                              const unsigned char *src, size_t n, unsigned char *cls, uint16_t *bits) {
+    FState a, b, *p = &a, *s = &b, *t;
+    k_init_state(p, net->h0);
+    for (size_t i = 0; i < n; i++) {
+        bits[i] = k_fwd_step(kc, kb, kp, net->emb + (src[i] < 128 ? src[i] : 127) * K_N_FEATS, p, s);
+        t = p; p = s; s = t;
+    }
+    bits[n] = k_fwd_step(kc, kb, kp, net->emb + 10 * K_N_FEATS, p, s);
+    BState ra, rb, *r = &ra, *q = &rb, *u;
+    memset(&ra, 0, sizeof ra);
+    k_bwd_step(kc, kb, bits[n], r, q);
+    u = r; r = q; q = u;
+    for (size_t i = n; i-- > 0;) {
+        cls[i] = (unsigned char)k_bwd_step(kc, kb, bits[i], r, q);
+        u = r; r = q; q = u;
+    }
+}
+
+/* ---- cli ----------------------------------------------------------------- */
+static char *default_weights(const char *argv0) {
+    const char *slash = strrchr(argv0, '/');
+    size_t dir = slash ? (size_t)(slash - argv0 + 1) : 0;
+    char *p = malloc(dir + 32);
+    memcpy(p, argv0, dir);
+    strcpy(p + dir, "pylexnet_weights.bin");
+    return p;
+}
+
+int main(int argc, char **argv) {
+    int classes = 0, bench = 0;
+    const char *path = NULL, *wpath = NULL;
+    for (int i = 1; i < argc; i++) {
+        if (!strcmp(argv[i], "--classes")) classes = 1;
+        else if (!strcmp(argv[i], "--bench")) bench = 1;
+        else if (!strcmp(argv[i], "--interp")) use_interp = 1;
+        else if (!strcmp(argv[i], "-w")) wpath = argv[++i];
+        else path = argv[i];
+    }
+    if (!classes && !bench) { fprintf(stderr, "usage: pylexnet [-w weights.bin] [--interp] (--classes [file] | --bench file)\n"); return 2; }
+    Net net = load(wpath ? wpath : default_weights(argv[0]));
+    double *kc = malloc(sizeof(double) * (K_N_COEF + 1)), *kb = malloc(sizeof(double) * (K_N_BIAS + 1)), *kp = NULL;
+    if (!use_interp) {
+        uint64_t h = struct_hash(&net);
+        if (h != K_STRUCT_HASH) {
+            fprintf(stderr, "weights: architecture hash %016llx does not match compiled kernel %016llx; regenerate (compile.py) or run --interp\n",
+                    (unsigned long long)h, (unsigned long long)K_STRUCT_HASH);
+            return 1;
+        }
+        if (check_guards(&net)) { fprintf(stderr, "weights: guard check failed, refusing the kernel\n"); return 1; }
+        if (all_integers(&net)) { fprintf(stderr, "weights: non-integer values, refusing the kernel\n"); return 1; }
+        if (check_families(&net)) { fprintf(stderr, "weights: a threshold family is not tied/sorted, refusing the kernel\n"); return 1; }
+        gather(&net, kc, kb);
+        kp = build_prefix(&net);
+    }
+    FILE *f = path ? fopen(path, "rb") : stdin;
+    if (!f) { perror(path); return 1; }
+    size_t cap = 1 << 20, n = 0, got;
+    unsigned char *src = malloc(cap);
+    while ((got = fread(src + n, 1, cap - n, f)) > 0) { n += got; if (n == cap) src = realloc(src, cap *= 2); }
+    if (path) fclose(f);
+    unsigned char *cls = malloc(n + 1);
+    uint16_t *bits = malloc((n + 2) * sizeof(uint16_t));
+    if (bench) {
+        int reps = 5;
+        struct timespec a, b;
+        clock_gettime(CLOCK_MONOTONIC, &a);
+        for (int i = 0; i < reps; i++) {
+            if (use_interp) classify_interp(&net, src, n, cls, bits);
+            else classify_compiled(&net, kc, kb, kp, src, n, cls, bits);
+        }
+        clock_gettime(CLOCK_MONOTONIC, &b);
+        double sec = (b.tv_sec - a.tv_sec) + (b.tv_nsec - a.tv_nsec) * 1e-9;
+        fprintf(stderr, "%zu bytes, %.1f ns/byte\n", n, sec / reps / (double)n * 1e9);
+        return 0;
+    }
+    if (use_interp) classify_interp(&net, src, n, cls, bits);
+    else classify_compiled(&net, kc, kb, kp, src, n, cls, bits);
+    fwrite(cls, 1, n, stdout);
+    return 0;
+}

pylexnet_kernel.h

@@ -0,0 +1,209 @@
+/* Generated by compile.py: the network's ARCHITECTURE as C. No weight values live here: pylexnet.c
+ * loads them, checks the structure hash and the guard inequalities, and gathers kc[]/kb[] by K_COEF/K_BIAS. */
+#include <stdint.h>
+#include <string.h>
+#define K_STRUCT_HASH 0x5AB2AC2E97E0387CULL
+#define K_N_FEATS 29
+#define K_N_CLASSES 9
+#define K_N_BITS 10
+#ifndef CLIP
+#define CLIP(x, hi) clipn((x), (hi))
+static inline double clipn(double x, double hi) { x = x < 0.0 ? 0.0 : x; return x > hi ? hi : x; }
+#endif
+/* A tied, sorted threshold family is a step vector: unit k is 1 iff kb[k] + w*h >= 1, true for k < rank.
+ * The loader verifies integrality, tying and monotonicity, so a binary search gives the same values. */
+static inline int k_rank(const double *kb, int n, double w, double h) {
+    int lo = 0, hi = n;
+    while (lo < hi) { int mid = (lo + hi) >> 1; if (kb[mid] + w * h >= 1.0) lo = mid + 1; else hi = mid; }
+    return lo;
+}
+typedef struct { double S, sq1, T, F, E, c, fo, FS, FD, oq1, oT, C, N, ee, W, d, pf, fp, cD, cC, cAt, atp, cDot, pos0, pos1, pos2, pos3, pos4, pos5, pos6, tail, A, e1, e2, e2d, e3, e3d, e3dd; } FState;
+typedef struct { double u_reg_KW, u_reg_BI, u_reg_FN, u_reg_CL, u_reg_DEC, u_reg_STR, b_ev_KW, b_ev_BI, b_ev_FN, b_ev_CL, b_ev_DEC, b_ev_STR; } BState;
+static inline void k_init_state(FState *s, const double *h0) { memcpy(s, h0, sizeof *s); }
+static inline __attribute__((always_inline)) uint16_t k_fwd_step(const double *restrict kc, const double *restrict kb, const double *restrict kp, const double *restrict x,
+                                  const FState *restrict p, FState *restrict s) {
+    double u_wordch = 0.0, u_qmd = 0.0, u_qms = 0.0, u_c2 = 0.0, u_wend = 0.0, u_fieldopen = 0.0, u_pf_new = 0.0, u_S_eff = 0.0, u_fp_new = 0.0, u_sclose = 0.0, u_tclose = 0.0, u_tripleopen = 0.0, u_nlclose = 0.0, u_E_new = 0.0, u_fo_new = 0.0, u_C_new = 0.0, u_open = 0.0, u_open_f = 0.0, u_open_qs = 0.0, u_fieldclose = 0.0, u_inc = 0.0, u_dec = 0.0, u_S_new = 0.0, u_c_new = 0.0, u_T_new = 0.0, u_F_new = 0.0, u_sq1_new = 0.0, u_FS_new = 0.0, u_FD_new = 0.0, u_oq1_new = 0.0, u_oT_new = 0.0, u_lab_STR = 0.0, u_N_new = 0.0, u_ee_new = 0.0, u_W_new = 0.0, u_d_new = 0.0, u_pos0 = 0.0, u_pos1 = 0.0, u_pos2 = 0.0, u_pos3 = 0.0, u_pos4 = 0.0, u_pos5 = 0.0, u_pos6 = 0.0, u_tail = 0.0, u_a0 = 0.0, u_a1 = 0.0, u_a2 = 0.0, u_a3 = 0.0, u_a4 = 0.0, u_a5 = 0.0, u_a6 = 0.0, u_at_ = 0.0, u_A_new = 0.0, u_e1_new = 0.0, u_e2_new = 0.0, u_e3_new = 0.0, u_e2d_new = 0.0, u_e3d_new = 0.0, u_e3dd_new = 0.0, u_m3 = 0.0, u_SE = 0.0, u_ge1 = 0.0, u_le1 = 0.0, u_ge2 = 0.0, u_le2 = 0.0, u_ge3 = 0.0, u_le3 = 0.0, u_ge4 = 0.0, u_le4 = 0.0, u_ge5 = 0.0, u_le5 = 0.0, u_is_def = 0.0, u_is_class = 0.0, u_cD_new = 0.0, u_cC_new = 0.0, u_atp_new = 0.0, u_cAt_new = 0.0, u_cDot_new = 0.0, u_ev_DEC = 0.0, u_ev_CL = 0.0, u_ev_KW = 0.0, u_ev_FN = 0.0, u_ev_BI = 0.0, u_ev_STR = 0.0, u_lab_W = 0.0;
+    if ((p->W + x[13] + x[6] + x[7] + p->atp + p->cD + p->cC + p->cAt + p->cDot + x[15] + x[14] + p->fp + p->d + p->pos0 + p->pos1 + p->pos2 + p->pos3 + p->pos4 + p->pos5 + p->pos6 + p->tail + p->e2 + p->e3 + p->e3d) != 0.0) {   /* all activations are >= 0 */
+        u_wordch = CLIP((kb[0] + kc[0] * x[13]), 1.0);
+        u_wend = CLIP(((kb[1] + kc[1] * p->W) + kc[2] * x[13]), 1.0);
+        u_pf_new = CLIP((((kb[2] + kc[3] * x[15]) + (kc[4] * x[14] + kc[5] * p->pf)) + (kc[6] * p->W + kc[7] * p->d)), 1.0);
+        u_e2d_new = CLIP((kb[3] + kc[8] * p->e2), 117026723984.0);
+        u_e3d_new = CLIP((kb[4] + kc[9] * p->e3), 117026723984.0);
+        u_e3dd_new = CLIP((kb[5] + kc[10] * p->e3d), 117026723984.0);
+        u_m3 = CLIP((kb[6] + kc[11] * p->d), 3.0);
+        u_fp_new = CLIP(((kb[7] + kc[12] * x[14]) + (kc[13] * p->fp + kc[14] * u_wordch)), 1.0);
+        u_cAt_new = CLIP((((kb[8] + kc[15] * p->cAt) + (kc[16] * p->atp + kc[17] * u_wordch)) + kc[18] * u_wend), 1.0);
+    }
+    if ((p->S + p->fo + p->FS + p->FD + p->c + p->T + p->F + p->sq1 + p->oq1 + p->oT + x[2] + x[3]) != 0.0) {   /* all activations are >= 0 */
+        u_qmd = CLIP(((kb[9] + kc[19] * x[2]) + (kc[20] * p->sq1 + kc[21] * p->E)), 1.0);
+        u_qms = CLIP(((kb[10] + kc[22] * x[3]) + (kc[23] * p->sq1 + kc[24] * p->E)), 1.0);
+        u_c2 = CLIP((kb[11] + kc[25] * p->c), 1.0);
+        u_fieldopen = CLIP(((kb[12] + kc[26] * p->fo) + kc[27] * x[8]), 1.0);
+        u_S_eff = CLIP((((kb[13] + kc[28] * p->S) + (kc[29] * u_fieldopen + kc[30] * p->T)) + ((kc[31] * u_qmd + kc[32] * u_qms) + kc[33] * u_c2)), 1.0);
+        u_sclose = CLIP((((kb[14] + kc[34] * u_S_eff) + (kc[35] * p->T + kc[36] * u_qmd)) + (kc[37] * u_qms + kc[38] * p->c)), 1.0);
+        u_tclose = CLIP((((kb[15] + kc[39] * u_S_eff) + (kc[40] * p->T + kc[41] * u_qmd)) + (kc[42] * u_qms + kc[43] * p->c)), 1.0);
+        u_tripleopen = CLIP((((kb[16] + kc[44] * u_S_eff) + (kc[45] * p->T + kc[46] * u_qmd)) + (kc[47] * u_qms + kc[48] * p->c)), 1.0);
+        u_nlclose = CLIP((((kb[17] + kc[49] * u_S_eff) + (kc[50] * p->T + kc[51] * x[1])) + kc[52] * p->E), 1.0);
+        u_E_new = CLIP(((kb[18] + kc[53] * u_S_eff) + (kc[54] * x[4] + kc[55] * p->E)), 1.0);
+        u_fo_new = CLIP((((kb[19] + kc[56] * u_S_eff) + (kc[57] * p->F + kc[58] * x[8])) + (kc[59] * p->E + kc[60] * p->fo)), 1.0);
+        u_open = CLIP((((kb[20] + kc[61] * x[2]) + (kc[62] * x[3] + kc[63] * u_S_eff)) + (kc[64] * p->C + kc[65] * p->N)), 1.0);
+        u_open_f = CLIP((((kb[21] + kc[66] * x[2]) + (kc[67] * x[3] + kc[68] * u_S_eff)) + ((kc[69] * p->C + kc[70] * p->N) + (kc[71] * p->pf + kc[72] * p->fp))), 1.0);
+        u_open_qs = CLIP((((kb[22] + kc[73] * x[2]) + (kc[74] * x[3] + kc[75] * u_S_eff)) + (kc[76] * p->C + kc[77] * p->N)), 1.0);
+        u_fieldclose = CLIP((((kb[23] + kc[78] * p->FS) + (kc[79] * u_S_eff + kc[80] * p->C)) + (kc[81] * p->FD + kc[82] * x[12])), 1.0);
+        u_inc = CLIP((((kb[24] + kc[83] * p->FS) + (kc[84] * u_S_eff + kc[85] * p->C)) + kc[86] * x[10]), 1.0);
+        u_dec = CLIP((((kb[25] + kc[87] * p->FS) + (kc[88] * u_S_eff + kc[89] * p->C)) + (kc[90] * x[11] + kc[91] * p->FD)), 1.0);
+        u_S_new = CLIP((((kb[26] + kc[92] * u_S_eff) + (kc[93] * u_sclose + kc[94] * u_tclose)) + ((kc[95] * u_nlclose + kc[96] * u_open) + kc[97] * u_fieldclose)), 1.0);
+        u_c_new = CLIP(((((kb[27] + kc[98] * p->c) + (kc[99] * u_S_eff + kc[100] * u_qmd)) + ((kc[101] * u_qms + kc[102] * u_tripleopen) + (kc[103] * u_tclose + kc[104] * u_sclose))) + (kc[105] * u_open + kc[106] * u_open_qs)), 3.0);
+        u_T_new = CLIP((((kb[28] + kc[107] * p->T) + (kc[108] * u_tripleopen + kc[109] * u_tclose)) + ((kc[110] * u_open + kc[111] * u_fieldclose) + kc[112] * p->oT)), 1.0);
+        u_F_new = CLIP((((kb[29] + kc[113] * p->F) + (kc[114] * u_open_f + kc[115] * u_open)) + ((kc[116] * u_sclose + kc[117] * u_tclose) + (kc[118] * u_nlclose + kc[119] * u_fieldclose))), 1.0);
+        u_sq1_new = CLIP(((((kb[30] + kc[120] * p->sq1) + (kc[121] * u_open_qs + kc[122] * u_sclose)) + ((kc[123] * u_tclose + kc[124] * u_nlclose) + (kc[125] * p->S + kc[126] * u_S_eff))) + (kc[127] * u_fieldclose + kc[128] * p->oq1)), 1.0);
+        u_FS_new = CLIP(((kb[31] + kc[129] * p->FS) + (kc[130] * u_fieldopen + kc[131] * u_fieldclose)), 1.0);
+        u_FD_new = CLIP(((kb[32] + kc[132] * p->FD) + (kc[133] * u_inc + kc[134] * u_dec)), 3.0);
+        u_oq1_new = CLIP((((kb[33] + kc[135] * p->oq1) + (kc[136] * u_fieldopen + kc[137] * p->sq1)) + kc[138] * u_fieldclose), 1.0);
+        u_oT_new = CLIP((((kb[34] + kc[139] * p->oT) + (kc[140] * u_fieldopen + kc[141] * p->T)) + kc[142] * u_fieldclose), 1.0);
+        u_lab_STR = CLIP((((kb[35] + kc[143] * u_S_eff) + (kc[144] * u_nlclose + kc[145] * u_open)) + kc[146] * u_fieldclose), 1.0);
+    }
+    if ((u_wend) != 0.0) {   /* all activations are >= 0 */
+        u_SE = CLIP((((kb[36] + kc[147] * p->A) + (kc[148] * p->e1 + kc[149] * p->e2d)) + ((kc[150] * p->e3dd + kc[151] * p->d) + kc[152] * u_m3)), 35184372088832.0);
+        u_ev_DEC = CLIP(((kb[37] + kc[153] * u_wend) + kc[154] * p->cAt), 1.0);
+        u_ev_CL = CLIP(((kb[38] + kc[155] * u_wend) + (kc[156] * p->cC + kc[157] * p->cAt)), 1.0);
+        u_ev_STR = CLIP(((kb[39] + kc[158] * u_wend) + (kc[159] * u_open + kc[160] * p->pf)), 1.0);
+        u_ge1 = CLIP((kb[40] + kc[161] * u_SE), 1.0);
+        u_le1 = CLIP((kb[41] + kc[162] * u_SE), 1.0);
+        u_ge2 = CLIP((kb[42] + kc[163] * u_SE), 1.0);
+        u_le2 = CLIP((kb[43] + kc[164] * u_SE), 1.0);
+        u_ge3 = CLIP((kb[44] + kc[165] * u_SE), 1.0);
+        u_le3 = CLIP((kb[45] + kc[166] * u_SE), 1.0);
+        u_ge4 = CLIP((kb[46] + kc[167] * u_SE), 1.0);
+        u_le4 = CLIP((kb[47] + kc[168] * u_SE), 1.0);
+        u_ge5 = CLIP((kb[48] + kc[169] * u_SE), 1.0);
+        u_le5 = CLIP((kb[49] + kc[170] * u_SE), 1.0);
+        u_is_def = CLIP(((kb[50] + kc[171] * u_wend) + (kc[172] * u_ge2 + kc[173] * u_le2)), 1.0);
+        u_is_class = CLIP(((kb[51] + kc[174] * u_wend) + (kc[175] * u_ge3 + kc[176] * u_le3)), 1.0);
+        u_ev_KW = CLIP(((((kb[52] + kc[177] * u_wend) + (kc[178] * u_ge1 + kc[179] * u_le1)) + ((kc[180] * u_ge2 + kc[181] * u_le2) + (kc[182] * u_ge3 + kc[183] * u_le3))) + ((kc[184] * p->cAt + kc[185] * p->cC) + kc[186] * p->cD)), 1.0);
+        u_ev_FN = CLIP((((kb[53] + kc[187] * u_wend) + (kc[188] * u_ge5 + kc[189] * u_le5)) + ((kc[190] * p->cD + kc[191] * p->cAt) + kc[192] * p->cC)), 1.0);
+        u_ev_BI = CLIP((((kb[54] + kc[193] * u_wend) + (kc[194] * u_ge4 + kc[195] * u_le4)) + ((kc[196] * p->cAt + kc[197] * p->cC) + (kc[198] * p->cD + kc[199] * p->cDot))), 1.0);
+    }
+    if ((p->W + x[13] + x[6] + x[7] + p->atp + p->cD + p->cC + p->cAt + p->cDot + x[15] + x[14] + p->fp + p->d + p->pos0 + p->pos1 + p->pos2 + p->pos3 + p->pos4 + p->pos5 + p->pos6 + p->tail + p->e2 + p->e3 + p->e3d) != 0.0) {   /* all activations are >= 0 */
+        u_W_new = CLIP((((kb[55] + kc[200] * u_wordch) + (kc[201] * x[0] + kc[202] * p->W)) + ((kc[203] * p->N + kc[204] * u_S_eff) + kc[205] * p->C)), 1.0);
+        u_atp_new = CLIP(((kb[56] + kc[206] * x[7]) + (kc[207] * u_S_eff + kc[208] * p->C)), 1.0);
+        u_cDot_new = CLIP((((kb[57] + kc[209] * x[6]) + (kc[210] * u_S_eff + kc[211] * p->C)) + ((kc[212] * p->N + kc[213] * p->cDot) + (kc[214] * u_wordch + kc[215] * u_wend))), 1.0);
+        u_cD_new = CLIP((((kb[58] + kc[216] * p->cD) + (kc[217] * u_is_def + kc[218] * u_wend)) + kc[219] * x[1]), 1.0);
+        u_cC_new = CLIP((((kb[59] + kc[220] * p->cC) + (kc[221] * u_is_class + kc[222] * u_wend)) + kc[223] * x[1]), 1.0);
+        u_d_new = CLIP(((kb[60] + kc[224] * p->d) + kc[225] * u_W_new), 30.0);
+        u_pos0 = CLIP(((kb[61] + kc[226] * u_W_new) + kc[227] * p->W), 1.0);
+        u_pos1 = CLIP(((kb[62] + kc[228] * p->pos0) + kc[229] * u_W_new), 1.0);
+        u_pos2 = CLIP(((kb[63] + kc[230] * p->pos1) + kc[231] * u_W_new), 1.0);
+        u_pos3 = CLIP(((kb[64] + kc[232] * p->pos2) + kc[233] * u_W_new), 1.0);
+        u_pos4 = CLIP(((kb[65] + kc[234] * p->pos3) + kc[235] * u_W_new), 1.0);
+        u_pos5 = CLIP(((kb[66] + kc[236] * p->pos4) + kc[237] * u_W_new), 1.0);
+        u_pos6 = CLIP(((kb[67] + kc[238] * p->pos5) + kc[239] * u_W_new), 1.0);
+        u_tail = CLIP(((kb[68] + kc[240] * p->pos6) + (kc[241] * p->tail + kc[242] * u_W_new)), 1.0);
+        u_e1_new = CLIP(((kb[69] + kc[243] * x[26]) + kc[244] * u_W_new), 117026723984.0);
+        u_e2_new = CLIP(((kb[70] + kc[245] * x[27]) + kc[246] * u_W_new), 117026723984.0);
+        u_e3_new = CLIP(((kb[71] + kc[247] * x[28]) + kc[248] * u_W_new), 117026723984.0);
+        u_lab_W = CLIP(((kb[72] + kc[249] * u_W_new) + kc[250] * u_atp_new), 1.0);
+        u_a0 = CLIP(((kb[73] + kc[251] * x[18]) + kc[252] * u_pos0), 117026723984.0);
+        u_a1 = CLIP(((kb[74] + kc[253] * x[19]) + kc[254] * u_pos1), 117026723984.0);
+        u_a2 = CLIP(((kb[75] + kc[255] * x[20]) + kc[256] * u_pos2), 117026723984.0);
+        u_a3 = CLIP(((kb[76] + kc[257] * x[21]) + kc[258] * u_pos3), 117026723984.0);
+        u_a4 = CLIP(((kb[77] + kc[259] * x[22]) + kc[260] * u_pos4), 117026723984.0);
+        u_a5 = CLIP(((kb[78] + kc[261] * x[23]) + kc[262] * u_pos5), 117026723984.0);
+        u_a6 = CLIP(((kb[79] + kc[263] * x[24]) + kc[264] * u_pos6), 117026723984.0);
+        u_at_ = CLIP(((kb[80] + kc[265] * x[25]) + kc[266] * u_tail), 117026723984.0);
+        u_A_new = CLIP(((((kb[81] + kc[267] * p->A) + (kc[268] * u_W_new + kc[269] * u_a0)) + ((kc[270] * u_a1 + kc[271] * u_a2) + (kc[272] * u_a3 + kc[273] * u_a4))) + ((kc[274] * u_a5 + kc[275] * u_a6) + kc[276] * u_at_)), 35184372088832.0);
+    }
+    if ((x[5] + p->C) != 0.0) {   /* all activations are >= 0 */
+        u_C_new = CLIP((((kb[82] + kc[277] * x[5]) + (kc[278] * u_S_eff + kc[279] * p->C)) + kc[280] * x[1]), 1.0);
+    }
+    if ((x[0] + p->N + p->ee) != 0.0) {   /* all activations are >= 0 */
+        u_N_new = CLIP(((((kb[83] + kc[281] * x[0]) + (kc[282] * p->W + kc[283] * p->N)) + ((kc[284] * u_wordch + kc[285] * x[6]) + (kc[286] * x[17] + kc[287] * p->ee))) + (kc[288] * u_S_eff + kc[289] * p->C)), 1.0);
+        u_ee_new = CLIP(((kb[84] + kc[290] * x[16]) + kc[291] * u_N_new), 1.0);
+    }
+    s->S = u_S_new;
+    s->sq1 = u_sq1_new;
+    s->T = u_T_new;
+    s->F = u_F_new;
+    s->E = u_E_new;
+    s->c = u_c_new;
+    s->fo = u_fo_new;
+    s->FS = u_FS_new;
+    s->FD = u_FD_new;
+    s->oq1 = u_oq1_new;
+    s->oT = u_oT_new;
+    s->C = u_C_new;
+    s->N = u_N_new;
+    s->ee = u_ee_new;
+    s->W = u_W_new;
+    s->d = u_d_new;
+    s->pf = u_pf_new;
+    s->fp = u_fp_new;
+    s->cD = u_cD_new;
+    s->cC = u_cC_new;
+    s->cAt = u_cAt_new;
+    s->atp = u_atp_new;
+    s->cDot = u_cDot_new;
+    s->pos0 = u_pos0;
+    s->pos1 = u_pos1;
+    s->pos2 = u_pos2;
+    s->pos3 = u_pos3;
+    s->pos4 = u_pos4;
+    s->pos5 = u_pos5;
+    s->pos6 = u_pos6;
+    s->tail = u_tail;
+    s->A = u_A_new;
+    s->e1 = u_e1_new;
+    s->e2 = u_e2_new;
+    s->e2d = u_e2d_new;
+    s->e3 = u_e3_new;
+    s->e3d = u_e3d_new;
+    s->e3dd = u_e3dd_new;
+    return (uint16_t)(((uint16_t)(u_lab_STR != 0.0) << 0) | ((uint16_t)(u_C_new != 0.0) << 1) | ((uint16_t)(u_N_new != 0.0) << 2) | ((uint16_t)(u_lab_W != 0.0) << 3) | ((uint16_t)(u_ev_KW != 0.0) << 4) | ((uint16_t)(u_ev_BI != 0.0) << 5) | ((uint16_t)(u_ev_FN != 0.0) << 6) | ((uint16_t)(u_ev_CL != 0.0) << 7) | ((uint16_t)(u_ev_DEC != 0.0) << 8) | ((uint16_t)(u_ev_STR != 0.0) << 9));
+}
+static inline __attribute__((always_inline)) int k_bwd_step(const double *restrict kc, const double *restrict kb, uint16_t bits,
+                             const BState *restrict r, BState *restrict s) {
+    const double f_lab_STR = (double)((bits >> 0) & 1);
+    const double f_C_new = (double)((bits >> 1) & 1);
+    const double f_N_new = (double)((bits >> 2) & 1);
+    const double f_lab_W = (double)((bits >> 3) & 1);
+    const double f_ev_KW = (double)((bits >> 4) & 1);
+    const double f_ev_BI = (double)((bits >> 5) & 1);
+    const double f_ev_FN = (double)((bits >> 6) & 1);
+    const double f_ev_CL = (double)((bits >> 7) & 1);
+    const double f_ev_DEC = (double)((bits >> 8) & 1);
+    const double f_ev_STR = (double)((bits >> 9) & 1);
+    const double u_reg_KW = CLIP(((kb[85] + kc[292] * r->b_ev_KW) + (kc[293] * r->u_reg_KW + kc[294] * f_lab_W)), 1.0);
+    const double u_reg_BI = CLIP(((kb[86] + kc[295] * r->b_ev_BI) + (kc[296] * r->u_reg_BI + kc[297] * f_lab_W)), 1.0);
+    const double u_reg_FN = CLIP(((kb[87] + kc[298] * r->b_ev_FN) + (kc[299] * r->u_reg_FN + kc[300] * f_lab_W)), 1.0);
+    const double u_reg_CL = CLIP(((kb[88] + kc[301] * r->b_ev_CL) + (kc[302] * r->u_reg_CL + kc[303] * f_lab_W)), 1.0);
+    const double u_reg_DEC = CLIP(((kb[89] + kc[304] * r->b_ev_DEC) + (kc[305] * r->u_reg_DEC + kc[306] * f_lab_W)), 1.0);
+    const double u_reg_STR = CLIP(((kb[90] + kc[307] * r->b_ev_STR) + (kc[308] * r->u_reg_STR + kc[309] * f_lab_W)), 1.0);
+    s->u_reg_KW = u_reg_KW;
+    s->u_reg_BI = u_reg_BI;
+    s->u_reg_FN = u_reg_FN;
+    s->u_reg_CL = u_reg_CL;
+    s->u_reg_DEC = u_reg_DEC;
+    s->u_reg_STR = u_reg_STR;
+    s->b_ev_KW = f_ev_KW;
+    s->b_ev_BI = f_ev_BI;
+    s->b_ev_FN = f_ev_FN;
+    s->b_ev_CL = f_ev_CL;
+    s->b_ev_DEC = f_ev_DEC;
+    s->b_ev_STR = f_ev_STR;
+    return (int)CLIP(((((kb[91] + kc[310] * u_reg_KW) + (kc[311] * u_reg_BI + kc[312] * u_reg_FN)) + ((kc[313] * u_reg_CL + kc[314] * u_reg_DEC) + (kc[315] * u_reg_STR + kc[316] * f_lab_STR))) + (kc[317] * f_C_new + kc[318] * f_N_new)), 8.0);
+}
+#define K_N_COEF 319
+#define K_N_BIAS 92
+#define K_N_GUARD 74
+#define K_N_FAMILY 0
+#define K_N_PREFIX 0
+#define K_PREFIX_LEN 0
+/* sorted threshold families: {cell, layer, first bias index, rows, coef index of the tied weight} */
+static const int32_t K_FAMILY[K_N_FAMILY + 1][5] = {{0,0,0,0,0}};
+/* prefix sums of a unit's weights over a family's columns: {cell, layer, row, first column, rows} */
+static const int32_t K_PREFIX[K_N_PREFIX + 1][5] = {{0,0,0,0,0}};
+#define K_MAX_GUARD 4
+static const int32_t K_COEF[K_N_COEF][4] = {{0,0,0,13},{0,0,4,43},{0,0,4,13},{0,0,6,15},{0,0,6,14},{0,0,6,45},{0,0,6,43},{0,0,6,44},{0,0,7,62},{0,0,8,64},{0,0,9,65},{0,0,10,44},{0,1,1,23},{0,1,1,54},{0,1,1,0},{0,1,3,57},{0,1,3,58},{0,1,3,0},{0,1,3,4},{0,0,1,2},{0,0,1,30},{0,0,1,33},{0,0,2,3},{0,0,2,30},{0,0,2,33},{0,0,3,34},{0,0,5,35},{0,0,5,8},{0,1,0,37},{0,1,0,5},{0,1,0,39},{0,1,0,1},{0,1,0,2},{0,1,0,3},{0,2,0,0},{0,2,0,42},{0,2,0,7},{0,2,0,8},{0,2,0,45},{0,2,1,0},{0,2,1,42},{0,2,1,7},{0,2,1,8},{0,2,1,45},{0,2,2,0},{0,2,2,42},{0,2,2,7},{0,2,2,8},{0,2,2,45},{0,2,3,0},{0,2,3,42},{0,2,3,16},{0,2,3,44},{0,2,4,0},{0,2,4,19},{0,2,4,44},{0,2,5,0},{0,2,5,43},{0,2,5,23},{0,2,5,44},{0,2,5,46},{0,2,7,17},{0,2,7,18},{0,2,7,0},{0,2,7,51},{0,2,7,52},{0,2,8,17},{0,2,8,18},{0,2,8,0},{0,2,8,51},{0,2,8,52},{0,2,8,56},{0,2,8,57},{0,2,9,17},{0,2,9,18},{0,2,9,0},{0,2,9,51},{0,2,9,52},{0,2,10,47},{0,2,10,0},{0,2,10,51},{0,2,10,48},{0,2,10,26},{0,2,11,47},{0,2,11,0},{0,2,11,51},{0,2,11,24},{0,2,12,47},{0,2,12,0},{0,2,12,51},{0,2,12,25},{0,2,12,48},{0,3,0,27},{0,3,0,0},{0,3,0,1},{0,3,0,3},{0,3,0,7},{0,3,0,10},{0,3,1,57},{0,3,1,27},{0,3,1,32},{0,3,1,33},{0,3,1,2},{0,3,1,1},{0,3,1,0},{0,3,1,7},{0,3,1,9},{0,3,2,55},{0,3,2,2},{0,3,2,1},{0,3,2,7},{0,3,2,10},{0,3,2,61},{0,3,3,56},{0,3,3,8},{0,3,3,7},{0,3,3,0},{0,3,3,1},{0,3,3,3},{0,3,3,10},{0,3,4,54},{0,3,4,9},{0,3,4,0},{0,3,4,1},{0,3,4,3},{0,3,4,53},{0,3,4,27},{0,3,4,10},{0,3,4,60},{0,3,5,58},{0,3,5,35},{0,3,5,10},{0,3,6,59},{0,3,6,11},{0,3,6,12},{0,3,7,60},{0,3,7,35},{0,3,7,54},{0,3,7,10},{0,3,8,61},{0,3,8,35},{0,3,8,55},{0,3,8,10},{0,3,9,27},{0,3,9,3},{0,3,9,7},{0,3,9,10},{0,1,2,68},{0,1,2,69},{0,1,2,70},{0,1,2,71},{0,1,2,52},{0,1,2,10},{0,1,4,4},{0,1,4,57},{0,1,5,4},{0,1,5,56},{0,1,5,57},{0,3,28,34},{0,3,28,7},{0,3,28,64},{0,2,15,2},{0,2,16,2},{0,2,17,2},{0,2,18,2},{0,2,19,2},{0,2,20,2},{0,2,21,2},{0,2,22,2},{0,2,23,2},{0,2,24,2},{0,3,23,34},{0,3,23,17},{0,3,23,18},{0,3,24,34},{0,3,24,19},{0,3,24,20},{0,3,25,34},{0,3,25,15},{0,3,25,16},{0,3,25,17},{0,3,25,18},{0,3,25,19},{0,3,25,20},{0,3,25,67},{0,3,25,66},{0,3,25,65},{0,3,26,34},{0,3,26,23},{0,3,26,24},{0,3,26,65},{0,3,26,67},{0,3,26,66},{0,3,27,34},{0,3,27,21},{0,3,27,22},{0,3,27,67},{0,3,27,66},{0,3,27,65},{0,3,27,68},{0,2,14,6},{0,2,14,15},{0,2,14,54},{0,2,14,52},{0,2,14,0},{0,2,14,51},{0,2,25,22},{0,2,25,0},{0,2,25,51},{0,2,26,21},{0,2,26,0},{0,2,26,51},{0,2,26,52},{0,2,26,61},{0,2,26,6},{0,2,26,9},{0,4,8,55},{0,4,8,23},{0,4,8,41},{0,4,8,46},{0,4,9,56},{0,4,9,24},{0,4,9,41},{0,4,9,46},{0,3,11,63},{0,3,11,14},{0,3,12,14},{0,3,12,62},{0,3,13,69},{0,3,13,14},{0,3,14,70},{0,3,14,14},{0,3,15,71},{0,3,15,14},{0,3,16,72},{0,3,16,14},{0,3,17,73},{0,3,17,14},{0,3,18,74},{0,3,18,14},{0,3,19,75},{0,3,19,76},{0,3,19,14},{0,3,20,50},{0,3,20,14},{0,3,21,51},{0,3,21,14},{0,3,22,52},{0,3,22,14},{0,3,29,14},{0,3,29,25},{0,4,0,47},{0,4,0,12},{0,4,1,48},{0,4,1,13},{0,4,2,49},{0,4,2,14},{0,4,3,50},{0,4,3,15},{0,4,4,51},{0,4,4,16},{0,4,5,52},{0,4,5,17},{0,4,6,53},{0,4,6,18},{0,4,7,54},{0,4,7,19},{0,5,0,53},{0,5,0,42},{0,5,0,0},{0,5,0,1},{0,5,0,2},{0,5,0,3},{0,5,0,4},{0,5,0,5},{0,5,0,6},{0,5,0,7},{0,2,6,20},{0,2,6,0},{0,2,6,51},{0,2,6,16},{0,2,13,15},{0,2,13,54},{0,2,13,52},{0,2,13,6},{0,2,13,21},{0,2,13,28},{0,2,13,53},{0,2,13,0},{0,2,13,51},{0,3,10,41},{0,3,10,13},{1,0,0,16},{1,0,0,10},{1,0,0,3},{1,0,1,17},{1,0,1,11},{1,0,1,3},{1,0,2,18},{1,0,2,12},{1,0,2,3},{1,0,3,19},{1,0,3,13},{1,0,3,3},{1,0,4,20},{1,0,4,14},{1,0,4,3},{1,0,5,21},{1,0,5,15},{1,0,5,3},{2,0,0,0},{2,0,0,1},{2,0,0,2},{2,0,0,3},{2,0,0,4},{2,0,0,5},{2,0,0,6},{2,0,0,7},{2,0,0,8}};
+static const int32_t K_BIAS[K_N_BIAS][3] = {{0,0,0},{0,0,4},{0,0,6},{0,0,7},{0,0,8},{0,0,9},{0,0,10},{0,1,1},{0,1,3},{0,0,1},{0,0,2},{0,0,3},{0,0,5},{0,1,0},{0,2,0},{0,2,1},{0,2,2},{0,2,3},{0,2,4},{0,2,5},{0,2,7},{0,2,8},{0,2,9},{0,2,10},{0,2,11},{0,2,12},{0,3,0},{0,3,1},{0,3,2},{0,3,3},{0,3,4},{0,3,5},{0,3,6},{0,3,7},{0,3,8},{0,3,9},{0,1,2},{0,1,4},{0,1,5},{0,3,28},{0,2,15},{0,2,16},{0,2,17},{0,2,18},{0,2,19},{0,2,20},{0,2,21},{0,2,22},{0,2,23},{0,2,24},{0,3,23},{0,3,24},{0,3,25},{0,3,26},{0,3,27},{0,2,14},{0,2,25},{0,2,26},{0,4,8},{0,4,9},{0,3,11},{0,3,12},{0,3,13},{0,3,14},{0,3,15},{0,3,16},{0,3,17},{0,3,18},{0,3,19},{0,3,20},{0,3,21},{0,3,22},{0,3,29},{0,4,0},{0,4,1},{0,4,2},{0,4,3},{0,4,4},{0,4,5},{0,4,6},{0,4,7},{0,5,0},{0,2,6},{0,2,13},{0,3,10},{1,0,0},{1,0,1},{1,0,2},{1,0,3},{1,0,4},{1,0,5},{2,0,0}};
+/* units whose block is skipped when the listed input columns are all zero; the loader proves each one */
+static const int32_t K_GUARD[K_N_GUARD][3 + K_MAX_GUARD] = {{0,0,0,13,-1,-1,-1},{0,0,4,43,-1,-1,-1},{0,0,6,15,14,-1,-1},{0,0,7,62,-1,-1,-1},{0,0,8,64,-1,-1,-1},{0,0,9,65,-1,-1,-1},{0,0,10,44,-1,-1,-1},{0,1,1,23,54,-1,-1},{0,1,3,57,58,-1,-1},{0,0,1,2,-1,-1,-1},{0,0,2,3,-1,-1,-1},{0,0,3,34,-1,-1,-1},{0,0,5,35,-1,-1,-1},{0,1,0,37,-1,-1,-1},{0,2,0,0,-1,-1,-1},{0,2,1,0,-1,-1,-1},{0,2,2,0,-1,-1,-1},{0,2,3,0,-1,-1,-1},{0,2,4,0,-1,-1,-1},{0,2,5,0,-1,-1,-1},{0,2,7,17,18,-1,-1},{0,2,8,17,18,-1,-1},{0,2,9,18,-1,-1,-1},{0,2,10,47,-1,-1,-1},{0,2,11,47,-1,-1,-1},{0,2,12,47,-1,-1,-1},{0,3,0,27,7,10,-1},{0,3,1,27,7,9,-1},{0,3,2,55,2,10,-1},{0,3,3,56,8,10,-1},{0,3,4,54,9,27,10},{0,3,5,58,35,-1,-1},{0,3,6,59,11,-1,-1},{0,3,7,60,35,-1,-1},{0,3,8,61,35,-1,-1},{0,3,9,27,7,10,-1},{0,1,4,4,-1,-1,-1},{0,1,5,4,-1,-1,-1},{0,3,28,34,-1,-1,-1},{0,3,23,34,-1,-1,-1},{0,3,24,34,-1,-1,-1},{0,3,25,34,-1,-1,-1},{0,3,26,34,-1,-1,-1},{0,3,27,34,-1,-1,-1},{0,2,14,6,54,-1,-1},{0,2,25,22,-1,-1,-1},{0,2,26,21,61,-1,-1},{0,4,8,55,23,-1,-1},{0,4,9,56,24,-1,-1},{0,3,11,14,-1,-1,-1},{0,3,12,14,-1,-1,-1},{0,3,13,69,-1,-1,-1},{0,3,14,70,-1,-1,-1},{0,3,15,71,-1,-1,-1},{0,3,16,72,-1,-1,-1},{0,3,17,73,-1,-1,-1},{0,3,18,74,-1,-1,-1},{0,3,19,75,76,-1,-1},{0,3,20,14,-1,-1,-1},{0,3,21,14,-1,-1,-1},{0,3,22,14,-1,-1,-1},{0,3,29,14,25,-1,-1},{0,4,0,12,-1,-1,-1},{0,4,1,13,-1,-1,-1},{0,4,2,14,-1,-1,-1},{0,4,3,15,-1,-1,-1},{0,4,4,16,-1,-1,-1},{0,4,5,17,-1,-1,-1},{0,4,6,18,-1,-1,-1},{0,4,7,19,-1,-1,-1},{0,5,0,42,-1,-1,-1},{0,2,6,20,51,-1,-1},{0,2,13,15,52,-1,-1},{0,3,10,13,-1,-1,-1}};

pylexnet_weights.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01a918daefff41cabc4064c47a04c6c893d3a018ac6c71ac8a922a3793c7242d
+size 84336

tools/corpus_score.py

@@ -0,0 +1,78 @@
+"""Score the compiled network on the second corpus against cached Pygments labels. PC only, one process.
+usage: corpus_score.py <fit|check> [report file]. Prints a one-line summary; ranked mismatches go to the report."""
+import re, subprocess, sys
+from collections import Counter, defaultdict
+from pathlib import Path
+
+import numpy as np
+
+NAMES = ["text", "keyword", "builtin", "string", "comment", "number", "func", "class", "decorator"]
+C = Path(__file__).resolve().parent / "corpus_cache"
+split = sys.argv[1] if len(sys.argv) > 1 else "fit"
+rep = open(sys.argv[2], "w") if len(sys.argv) > 2 else None
+data = np.memmap(C / "data.bin", dtype=np.uint8, mode="r")
+gold = np.memmap(C / "gold.bin", dtype=np.uint8, mode="r")
+tot = wrong = exact = nfiles = 0
+conf = Counter()
+bytext = Counter()
+byprev = Counter()
+sample = {}
+pkgs = defaultdict(lambda: [0, 0])
+wordre = re.compile(rb"[A-Za-z_][A-Za-z_0-9]*|[^\sA-Za-z_0-9]")
+for line in open(C / "index.tsv"):
+    sp, pkg, rel, off, n = line.rstrip("\n").split("\t")
+    if sp != split:
+        continue
+    off, n = int(off), int(n)
+    d = data[off:off + n].tobytes()
+    g = gold[off:off + n]
+    out = subprocess.run(["./pylexnet", "--classes"], input=d, capture_output=True).stdout
+    p = np.frombuffer(out, dtype=np.uint8)
+    assert len(p) == n, rel
+    nfiles += 1
+    tot += n
+    pkgs[pkg][1] += n
+    bad = np.nonzero(g != p)[0]
+    if not len(bad):
+        exact += 1
+        continue
+    wrong += len(bad)
+    pkgs[pkg][0] += len(bad)
+    s = e = int(bad[0])
+    runs = []
+    for i in map(int, bad[1:]):
+        if i == e + 1 and g[i] == g[e] and p[i] == p[e]:
+            e = i
+        else:
+            runs.append((s, e)); s = e = i
+    runs.append((s, e))
+    for s, e in runs:
+        k = f"{NAMES[g[s]]}>{NAMES[p[s]]}"
+        m = e - s + 1
+        conf[k] += m
+        a = d.rfind(b"\n", 0, s) + 1
+        z = d.find(b"\n", e)
+        z = n if z < 0 else z
+        t = re.sub(rb"\d", b"0", d[s:e + 1][:18]).decode("utf-8", "replace")
+        prev = wordre.findall(d[a:s])[-1:] or [b"^"]
+        k1 = (k, t)
+        k2 = (k, prev[0].decode("utf-8", "replace"))
+        bytext[k1] += m
+        byprev[k2] += m
+        if k1 not in sample:
+            sample[k1] = (rel, d[a:z][:110].decode("utf-8", "replace"))
+        if k2 not in sample:
+            sample[k2] = (rel, d[a:z][:110].decode("utf-8", "replace"))
+print(f"{split}: files={nfiles} MB={tot/1e6:.1f} wrong={wrong} acc={1-wrong/tot:.6f} exact={exact}/{nfiles}")
+if rep:
+    print(f"{split}: files={nfiles} MB={tot/1e6:.1f} wrong={wrong} acc={1-wrong/tot:.6f} exact={exact}/{nfiles}", file=rep)
+    print("confusions", conf.most_common(30), file=rep)
+    print("by text", file=rep)
+    for k, v in bytext.most_common(70):
+        print(f"{v:7} {k[0]} {k[1]!r} [{sample[k][0].split('/')[0]}] {sample[k][1]!r}", file=rep)
+    print("by previous token", file=rep)
+    for k, v in byprev.most_common(50):
+        print(f"{v:7} {k[0]} after {k[1]!r} {sample[k][1]!r}", file=rep)
+    print("by package", file=rep)
+    for k, (w, n) in sorted(pkgs.items(), key=lambda kv: -kv[1][0]):
+        print(f"{w:7} {n/1e6:6.1f}MB {1-w/n:.6f} {k}", file=rep)

tools/demo.py

@@ -0,0 +1,44 @@
+"""Render a demo video from the network's real output."""
+import subprocess, sys
+from pathlib import Path
+import numpy as np
+from PIL import Image, ImageDraw, ImageFont
+import imageio
+code = '''@dataclass
+class Greeter:
+    """Say hello, politely."""
+    name: str = "world"
+
+    def greet(self, times=3):
+        # f-strings nest, and the network keeps track
+        for i in range(times):
+            print(f"hello {self.name!r} #{i + 1:>2}")
+        return len(self.name) * 0x10 + 1.5e3
+
+async def main():
+    g = Greeter("827 weights")
+    await g.greet()
+    raise SystemExit(None) from None
+'''
+Path("demo_in.py").write_text(code)
+lab = subprocess.run(["./pylexnet", "--classes", "demo_in.py"], capture_output=True).stdout
+assert len(lab) == len(code.encode())
+PAL = {0:(205,214,244),1:(203,166,247),2:(137,220,235),3:(166,227,161),4:(127,132,156),5:(250,179,135),6:(137,180,250),7:(249,226,175),8:(245,194,231)}
+BG=(30,30,46); W,H=1280,720; S=22; font=ImageFont.truetype(sys.argv[1],S); big=ImageFont.truetype(sys.argv[1],26)
+cw=font.getlength("M"); lh=int(S*1.5)
+def frame(n, caption):
+    im=Image.new("RGB",(W,H),BG); d=ImageDraw.Draw(im); x0,y0=40,92
+    d.text((40,28),"pylexnet: a Python highlighter that is a neural network",font=big,fill=(205,214,244))
+    x=x0;y=y0
+    for i,ch in enumerate(code[:n]):
+        if ch=="\n": x=x0;y+=lh;continue
+        d.text((x,y),ch,font=font,fill=PAL[lab[i]]); x+=cw
+    d.rectangle([x,y+2,x+cw-2,y+S+4],fill=(205,214,244))
+    d.text((40,H-46),caption,font=font,fill=(127,132,156))
+    return np.asarray(im)
+w=imageio.get_writer("demo.mp4",fps=30,codec="libx264",quality=8,macro_block_size=8)
+cap="827 numbers, no training data. Colours are the network's own output."
+for n in range(0,len(code)+1,2): w.append_data(frame(n,cap))
+last=frame(len(code),cap)
+for _ in range(90): w.append_data(last)
+w.close(); print("frames ok", Path("demo.mp4").stat().st_size)

tools/fpcheck.py

@@ -0,0 +1,37 @@
+"""PC tool: false positives of the additive matcher over the fit corpus's words. usage: fpcheck.py [--matcher add]
+Prints words (not in the vocabulary) whose value sum falls in a class window, with their counts."""
+import re, sys
+from collections import Counter
+from pathlib import Path
+import numpy as np
+import compile as c
+vocab = c.pick_vocab(None)
+c.solve_additive(vocab)
+C = Path("corpus_cache")
+data = np.memmap(C / "data.bin", dtype=np.uint8, mode="r")
+gold = np.memmap(C / "gold.bin", dtype=np.uint8, mode="r")
+wre = re.compile(rb"[A-Za-z_\x80-\xff][A-Za-z_0-9\x80-\xff]*")
+cnt = Counter(); nbytes = Counter()
+for line in open(C / "index.tsv"):
+    sp, pkg, rel, off, n = line.rstrip("\n").split("\t")
+    if sp != "fit":
+        continue
+    off, n = int(off), int(n)
+    d = data[off:off + n].tobytes(); g = gold[off:off + n]
+    for m in wre.finditer(d):
+        if g[m.start()] == 0:        # labelled text by Pygments: a hit here is a false positive
+            w = m.group().decode("latin1")
+            cnt[w] += 1; nbytes[w] += m.end() - m.start()
+tot = fp = fpb = 0
+hits = Counter(); hitb = Counter()
+for w, n in cnt.items():
+    if w in vocab:
+        continue
+    tot += n
+    v = sum(c.ADD_TABLES.get(f, {}).get(code, 0) * k for (f, code), k in c.word_feats(w).items())
+    for t in (1, 2, 3, 4, 5):
+        if abs(v - t * c.ADD_T) <= c.ADD_EPS:
+            hits[w] += n; hitb[w] += nbytes[w]; fp += n; fpb += nbytes[w]
+print(f"non-vocabulary text words: {tot} occurrences, {len(cnt)} distinct; false positives: {fp} occurrences, {fpb} bytes ({fpb / 74787051:.6f} of corpus)")
+for w, n in hits.most_common(40):
+    print(f"  {w:30} {n:7} occ {hitb[w]:8} B")

tools/score.py

@@ -0,0 +1,118 @@
+"""Score ./pylexnet against Pygments' PythonLexer, per byte, on the dev split of the stdlib. PC only, one process.
+Pygments labels are cached in dev_cache/ after the first run. usage: score.py [dev] [report file]"""
+import hashlib
+import subprocess
+import sys
+from collections import Counter
+from pathlib import Path
+
+import numpy as np
+
+ROOT = Path("/usr/lib/python3.12")
+HERE = Path(__file__).resolve().parent
+CACHE = HERE / "dev_cache"
+NAMES = ["text", "keyword", "builtin", "string", "comment", "number", "func", "class", "decorator"]
+
+
+def files(split):
+    out = []
+    for p in sorted(ROOT.rglob("*.py")):
+        if "site-packages" in p.parts or "dist-packages" in p.parts or "lib2to3" in p.parts:
+            continue
+        if p.stat().st_size > 300_000:
+            continue
+        h = hashlib.sha256(str(p).encode()).digest()[0]
+        if (h < 128) == (split == "dev"):
+            out.append(p)
+    return out
+
+
+def oracle(text):
+    from pygments.lexers import PythonLexer
+    from pygments.token import Token as T
+    lex = PythonLexer(stripnl=False, ensurenl=False, stripall=False)
+    out = bytearray()
+    for _, tok, val in lex.get_tokens_unprocessed(text):
+        if tok in T.Comment: c = 4
+        elif tok in T.Literal.String: c = 3
+        elif tok in T.Literal.Number: c = 5
+        elif tok in T.Keyword or tok in T.Operator.Word: c = 1
+        elif tok in T.Name.Decorator: c = 8
+        elif tok in T.Name.Function: c = 6
+        elif tok in T.Name.Class: c = 7
+        elif tok in T.Name.Builtin or tok in T.Name.Exception: c = 2
+        else: c = 0
+        out += bytes([c]) * len(val.encode("utf-8", "surrogateescape"))
+    return out
+
+
+def build_cache(split):
+    CACHE.mkdir(exist_ok=True)
+    idx, data, gold = [], bytearray(), bytearray()
+    for p in files(split):
+        try:
+            text = p.read_text("utf-8")
+        except UnicodeDecodeError:
+            continue
+        text = text.replace("\r\n", "\n")
+        d = text.encode()
+        g = oracle(text)
+        if len(g) != len(d):
+            continue
+        idx.append(f"{p}\t{len(data)}\t{len(d)}\n")
+        data += d; gold += g
+    (CACHE / f"{split}_data.bin").write_bytes(data)
+    (CACHE / f"{split}_gold.bin").write_bytes(gold)
+    (CACHE / f"{split}_index.tsv").write_text("".join(idx))
+
+
+def main():
+    split = sys.argv[1] if len(sys.argv) > 1 else "dev"
+    rep = open(sys.argv[2], "w") if len(sys.argv) > 2 else None
+    if not (CACHE / f"{split}_index.tsv").exists():
+        build_cache(split)
+    data = np.memmap(CACHE / f"{split}_data.bin", dtype=np.uint8, mode="r")
+    gold = np.memmap(CACHE / f"{split}_gold.bin", dtype=np.uint8, mode="r")
+    conf = np.zeros((9, 9), dtype=np.int64)
+    exact = nfiles = 0
+    bytext = Counter(); sample = {}
+    for line in open(CACHE / f"{split}_index.tsv"):
+        path, off, n = line.rstrip("\n").split("\t")
+        off, n = int(off), int(n)
+        d = data[off:off + n].tobytes(); g = gold[off:off + n]
+        out = subprocess.run([str(HERE / "pylexnet"), "--classes"], input=d, capture_output=True).stdout
+        p = np.frombuffer(out, dtype=np.uint8)
+        assert len(p) == n, (path, len(p), n)
+        nfiles += 1
+        np.add.at(conf, (g, p), 1)
+        bad = np.nonzero(g != p)[0]
+        if not len(bad):
+            exact += 1
+            continue
+        s = e = int(bad[0]); runs = []
+        for i in map(int, bad[1:]):
+            if i == e + 1 and g[i] == g[e] and p[i] == p[e]:
+                e = i
+            else:
+                runs.append((s, e)); s = e = i
+        runs.append((s, e))
+        for s, e in runs:
+            k = (f"{NAMES[g[s]]}>{NAMES[p[s]]}", d[s:e + 1][:18].decode("utf-8", "replace"))
+            bytext[k] += e - s + 1
+            if k not in sample:
+                a = d.rfind(b"\n", 0, s) + 1; z = d.find(b"\n", e); z = n if z < 0 else z
+                sample[k] = (Path(path).name, d[a:z][:110].decode("utf-8", "replace"))
+    tot = conf.sum()
+    print(f"{split}: {nfiles} files, {tot / 1e6:.1f} MB")
+    print(f"byte accuracy {np.trace(conf) / tot:.5f}, exact files {exact}/{nfiles}, wrong bytes {tot - np.trace(conf)}")
+    for k in range(9):
+        pr = conf[k, k] / max(1, conf[:, k].sum()); rc = conf[k, k] / max(1, conf[k].sum())
+        print(f"  {NAMES[k]:10} P {pr:.5f} R {rc:.5f} n {conf[k].sum()}")
+    off = conf.copy(); np.fill_diagonal(off, 0)
+    print("top confusions:", [(f"{NAMES[i // 9]}>{NAMES[i % 9]}", int(off.ravel()[i])) for i in np.argsort(off.ravel())[::-1][:8] if off.ravel()[i]])
+    for k, v in bytext.most_common(25 if rep is None else 80):
+        print(f"{v:7} {k[0]} {k[1]!r} [{sample[k][0]}] {sample[k][1]!r}", file=rep or sys.stdout)
+
+
+if __name__ == "__main__":
+    main()

tools/verify.py

@@ -0,0 +1,24 @@
+"""The generated kernel must equal the generic interpreter over the same weights, byte for byte, on the dev split."""
+import subprocess
+import sys
+from pathlib import Path
+
+import score
+
+HERE = Path(__file__).resolve().parent
+fs = score.files(sys.argv[1] if len(sys.argv) > 1 else "dev")
+bad = tot = 0
+for p in fs:
+    try:
+        data = p.read_text("utf-8").replace("\r\n", "\n").encode()
+    except UnicodeDecodeError:
+        continue
+    out = subprocess.run([str(HERE / "pylexnet"), "--classes"], input=data, capture_output=True).stdout
+    ref = subprocess.run([str(HERE / "pylexnet"), "--interp", "--classes"], input=data, capture_output=True).stdout
+    tot += len(data)
+    if out != ref:
+        bad += 1
+        if bad < 4:
+            i = next((j for j in range(min(len(out), len(ref))) if out[j] != ref[j]), -1)
+            print("MISMATCH", p, i, out[i] if i >= 0 else len(out), ref[i] if i >= 0 else len(ref), data[max(0, i - 30):i + 10])
+print(f"{len(fs)} files, {tot / 1e6:.1f} MB, files differing from rules: {bad}")

vocab_prio.txt

@@ -0,0 +1 @@
+("self",995220),("isinstance",193560),("str",140403),("bool",62712),("__init__",61888),("list",58384),("ValueError",45510),("int",44250),("NotImplementedError",34941),("len",33720),("dict",32812),("tuple",30510),("cls",25692),("type",25108),("super",24615),("getattr",22036),("Exception",20016),("TypeError",19161),("AttributeError",17066),("float",16230),("set",16092),("object",14388),("range",13825),("sorted",13572),("frozenset",13194),("hasattr",12684),("enumerate",11943),("bytes",10995),("BaseException",10686),("RuntimeError",10380),("KeyError",9624),("id",8882),("ImportError",8129),("__repr__",7152),("AssertionError",5978),("NotImplemented",5586),("__getitem__",4785),("print",4640),("OSError",4599),("next",4464),("__call__",4352),("any",4065),("DeprecationWarning",3996),("zip",3897),("repr",3840),("issubclass",3790),("reversed",3576),("format",3408),("IndexError",3290),("setattr",3269),("__hash__",3240),("__eq__",3204),("StopIteration",3107),("__str__",3038),("callable",3016),("__new__",2954),("filter",2946),("__getattr__",2860),("all",2835),("max",2691),("iter",2648),("__iter__",2592),("abs",2526),("hash",2404),("ConnectionError",2355),("__setattr__",2321),("map",2307),("__contains__",2268),("sum",2247),("min",1974),("FileNotFoundError",1955),("KeyboardInterrupt",1887),("open",1860),("__enter__",1764),("bytearray",1674),("ModuleNotFoundError",1672),("__exit__",1648),("__len__",1505),("LookupError",1485),("__setitem__",1474),("dir",1284),("slice",1270),("TimeoutError",1236),("__rich_repr__",1222),("UnicodeDecodeError",1188),("__get__",1183),("ZeroDivisionError",1173),("memoryview",1160),("property",1112),("staticmethod",1092),("__rich_console__",1088),("__aenter__",1060),("input",1060),("__aexit__",1026),("SystemExit",1020),("__bool__",904),("SyntaxError",902),("__getattribute__",880),("__post_init__",858),("globals",854),("FutureWarning",793),("__delitem__",770),("UnicodeEncodeError",756),("RuntimeWarning",714),("Warning",714),("locals",714),("__getstate__",708),("__reduce__",700),("__init_subclass__",697),("StopAsyncIteration",630),("classmethod",627),("__setstate__",612),("round",590),("vars",588),("OverflowError",572),("complex",553),("EOFError",552),("__add__",546),("ord",540),("UserWarning",539),("__anext__",477),("__import__",470),("__del__",455),("PermissionError",450),("compile",448),("divmod",432),("__set__",427),("__aiter__",414),("__rich_measure__",384),("NameError",378),("UnicodeError",372),("chr",357),("__delattr__",352),("__deepcopy__",348),("__await__",342),("GeneratorExit",338),("PendingDeprecationWarning",325),("__ne__",324),("__lt__",318),("delattr",315),("__next__",304),("FileExistsError",300),("__sub__",294),("__coerce__",290),