bind2_0: release package

Frozen snapshot. Uploaded as a single commit per batch.

release/.gitattributes

@@ -0,0 +1,7 @@
+# Release packages are byte-frozen snapshots: MANIFEST.json records a SHA-256 per artifact,
+# and for some packages a preregistration pins a subset of them as well. Any end-of-line
+# conversion on checkout silently changes those bytes and breaks every one of those hashes
+# on the cloner's machine while still verifying on ours (this repo has core.autocrlf=true).
+#
+# -text disables all eol conversion. Do not remove.
+* -text

release/BUILD.md

@@ -0,0 +1,35 @@
+# Building `bind2_0`
+
+**Verdict: BLOCKED** — **Does not run as-is.** The blockers are listed below and are real, not cosmetic. A launcher works around what can be worked around without editing frozen files; the rest is stated honestly rather than papered over.
+
+## Dependencies
+
+```
+torch==2.12.1
+numpy==2.4.6
+tokenizers==0.22.2
+flash-linear-attention==0.5.1   # CUDA-only (Triton kernels)
+# on Windows also: triton-windows==3.7.1.post27
+```
+
+Python 3.11 is the version everything here was developed and smoke-tested against.
+
+## Known blockers and portability notes
+
+- train_bind2_0_babylm.py:20 reads os.environ['BABYLM_WORK'] as a bare subscript at module scope -- an uncaught KeyError with no default. The shipped launcher works around this without editing the file.
+- dev='cuda' is hardcoded at :30 with no CPU path.
+- flash-linear-attention is a hard dependency and its kernels are CUDA-only, so this package cannot run on CPU or on non-NVIDIA hardware at all.
+
+## Launcher
+
+`run_train.py` exists because this package's trainer hardcodes machine-specific inputs inside files that are frozen and must not be edited, so the launcher supplies them from outside instead.
+
+**How it does that differs by package, so read `run_train.py`'s own docstring rather than this paragraph.** Some launchers import the frozen modules and rebind module-level constants before the entry point runs; at least one cannot, because its trainer reads an environment variable at import time and rebinding the attribute afterwards is already too late -- that launcher sets the variable first. Each launcher documents its own mechanism, and each was checked against the source it works around.
+
+## Inputs you must supply
+
+- `alaya-strict-small/tokens_u16.bin` (~32.6 MB) — status: REFERENCED-TOO-LARGE
+
+## Verification
+
+Every `.py` in `src/` is byte-identical to the working-tree file recorded in `MANIFEST.json` and compiles under Python 3.11 (`python -m compileall src`).

release/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yulin Yang (SecludedCorner)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

release/MANIFEST.json

@@ -0,0 +1,45 @@
+{
+ "package": "bind2_0",
+ "spec_title": "bind2_0 \u2014 delta-rule backbone with a forced bottleneck",
+ "generation": "bind2_0",
+ "buildable": "BLOCKED",
+ "entries": [
+  {
+   "source": "pilot-4060/train_bind2_0_babylm.py",
+   "kind": "code",
+   "sha256": "c325148dbecc8da56faf6f94b9494f1910cd437bc16d4c0b340eccdd279fb0c8",
+   "bytes": 4138,
+   "source_commit": "c3d39af06e8ac30ad3d8c930548dc03db4698ffb",
+   "status": "COPIED",
+   "dest": "src/train_bind2_0_babylm.py"
+  },
+  {
+   "source": "pilot-4060/modeling_bind2_0.py",
+   "kind": "code",
+   "sha256": "1a950720257d560e02833b4c2e6c5017ee03a7585be6204c1b726930fd80e6ed",
+   "bytes": 5047,
+   "source_commit": "c3d39af06e8ac30ad3d8c930548dc03db4698ffb",
+   "status": "COPIED",
+   "dest": "src/modeling_bind2_0.py",
+   "source_sha256": "12ea39f0b958ae7ad304bbf4cc7a71afc984fe37c5805c97151084348a47f899",
+   "scrubbed": "1 text rule(s); AST verified: structure identical"
+  },
+  {
+   "source": "alaya-strict-small/tokenizer.json",
+   "kind": "data",
+   "sha256": "6582b724cb5196ca72baf2a6b91c17be2631901e93feadce1fcd74c8d5e9089a",
+   "bytes": 1096025,
+   "source_commit": "c3d39af06e8ac30ad3d8c930548dc03db4698ffb",
+   "status": "COPIED",
+   "dest": "data/tokenizer.json"
+  },
+  {
+   "source": "alaya-strict-small/tokens_u16.bin",
+   "kind": "external",
+   "sha256": "bce1a7681c6dc323ad0166219d63e8af01ae55a8c07d74d9774432d23b440c4e",
+   "bytes": 32592654,
+   "source_commit": "(untracked)",
+   "status": "REFERENCED-TOO-LARGE"
+  }
+ ]
+}

release/README.md

@@ -0,0 +1,37 @@
+# bind2_0 — delta-rule backbone with a forced bottleneck
+
+Gated-delta-rule backbone; the parameter-matching reference for the bind2_1 arms.
+
+> Research artifact. Not a competition entry.
+
+## What this is
+
+A **frozen snapshot** of the code for one model generation. It is not a living repository: it is never updated after release. Files shared with another generation are duplicated here on purpose, so that the code closure is complete and nothing has to be fetched from a sibling package.
+
+**Self-contained in code, not in data.** Every module the code imports ships here. Large or non-redistributable inputs -- corpora, official evaluation sets, token streams -- do not, and are listed under "Inputs you must supply" in `BUILD.md` with how to obtain or regenerate each. Do not read the completeness of `src/` as a claim that the package runs with no further downloads.
+
+## Contents
+
+| | count |
+|---|---|
+| source files (`src/`) | 2 |
+| shipped data (`data/`) | 1 |
+| externally-obtained inputs | 1 |
+
+`MANIFEST.json` records, for every artifact, its original path in the working tree, its SHA-256, its size, and the git commit it came from. That is the traceability record: it is what lets you prove which state of the code produced a published result.
+
+## External inputs (not redistributed here)
+
+- `alaya-strict-small/tokens_u16.bin` — regenerable from the bind1 package's tokenizer chain
+
+## Building and running
+
+See `BUILD.md`. It states plainly what works, what does not, and why.
+
+## Availability of this package
+
+If a model card, paper or dataset page points at this package, that pointer must resolve for whoever reads it. A public card that links private code makes a reproducibility promise it cannot keep — and a licence line on unreachable code claims a release that has not happened. Before publishing any pointer to this package, confirm the package is actually reachable by the audience that will read the pointer.
+
+## Licence
+
+MIT (see `LICENSE`). Code only — model weights and evaluation artifacts are distributed separately and carry their own terms.

release/requirements.txt

@@ -0,0 +1,5 @@
+torch==2.12.1
+numpy==2.4.6
+tokenizers==0.22.2
+flash-linear-attention==0.5.1   # CUDA-only (Triton kernels)
+# on Windows also: triton-windows==3.7.1.post27

release/run_train.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Portable launcher for the bind2_0 trainer.
+
+WHY THIS EXISTS
+---------------
+`src/train_bind2_0_babylm.py` line 20 is:
+
+    WORK = os.environ["BABYLM_WORK"]
+
+a bare subscript at module scope with no default, so merely IMPORTING the module raises an
+uncaught KeyError when the variable is unset — the failure a newcomer hits first, with no message
+explaining what to set. It is read at import time, not call time, so it must be set BEFORE the
+import; rebinding the attribute afterwards is too late. This launcher sets it, checks the two
+files it derives (tokenizer.json and tokens_u16.bin), and reports clearly if they are absent.
+
+The trainer takes bare positional sys.argv (line 26), not argparse, so arguments are passed
+through verbatim in order.
+
+    python run_train.py --work <dir> -- <positional args for train_bind2_0_babylm.py>
+    python run_train.py --work <dir> --check
+
+HONEST LIMITATION, not worked around: line 30 hardcodes dev = "cuda" with no CPU path, and
+modeling_bind2_0.py imports flash-linear-attention, whose kernels are CUDA-only. This package
+cannot run on CPU or on non-NVIDIA hardware. A launcher cannot fix that; only editing the frozen
+source could, and that is deliberately not done here.
+"""
+from __future__ import annotations
+import argparse
+import os
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--work", default=os.environ.get("BABYLM_WORK", str(HERE / "data")),
+                    help="directory holding tokenizer.json and tokens_u16.bin")
+    ap.add_argument("--check", action="store_true", help="resolve inputs and exit")
+    args, passthrough = ap.parse_known_args()
+    if passthrough and passthrough[0] == "--":
+        passthrough = passthrough[1:]
+
+    work = Path(args.work)
+    os.environ["BABYLM_WORK"] = str(work)          # MUST precede the import
+    sys.path.insert(0, str(HERE / "src"))
+
+    need = {"tokenizer.json": work / "tokenizer.json", "tokens_u16.bin": work / "tokens_u16.bin"}
+    missing = [k for k, v in need.items() if not v.exists()]
+    if args.check or missing:
+        print(f"  BABYLM_WORK = {work}")
+        for k, v in need.items():
+            print(f"  {k:16} {'OK ' if v.exists() else 'MISSING'}  {v}")
+    if missing:
+        print("\nERROR: tokens_u16.bin is not redistributed inside this package (it is large and "
+              "regenerable). Build it with the bind1 package's tokenizer chain "
+              "(train_tokenizer.py then make_tokens.py), or point --work at a directory that "
+              "already has it. See BUILD.md.", file=sys.stderr)
+        return 2
+    if args.check:
+        print("\nAll inputs resolve. Re-run without --check to train.")
+        return 0
+
+    import train_bind2_0_babylm as T
+    sys.argv = ["train_bind2_0_babylm.py"] + passthrough
+    T.main()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

release/src/modeling_bind2_0.py

@@ -0,0 +1,118 @@
+"""modeling_bind2_0.py -- bind2_0 architecture: forced-bottleneck delta-overwrite recurrence
+(2026-07-12).
+
+d=384, L=12, 3:1 GatedDeltaNet:chunked-attn (9 GDN + 3 chunked-attn), ~24M (matched to bind1 23.9M).
+  - GDN layer (fla GatedDeltaNet): delta-rule state S_t = a_t S_{t-1}(I - b_t k k^T) + b_t v k^T,
+    o_t = S_t q_t. Per-token state UPDATE carried along the sequence = the "apply an operation to
+    update state" mechanism (the verb-dynamics bind1 lacks; bind1 only labels roles).
+  - CHUNKED attention (forced bottleneck, P0): strict intra-chunk, NON-overlapping mask
+    M[t,s]=1 iff floor(t/C)==floor(s/C) and s<=t. Cross-chunk info can flow ONLY through S.
+  - Unified readout: LM head on the residual stream after all blocks (GDN o_t already added to
+    residual) -> every position, no hard routing.
+"""
+import math
+import torch, torch.nn as nn, torch.nn.functional as F
+from fla.layers import GatedDeltaNet
+
+# --- self-contained generic primitives (bind2_0 is architecturally independent of bind1/mono;
+#     these are standard transformer building blocks, copied here so modeling_bind2_0.py stands alone) ---
+
+def build_rope(T, D, device, base=10000.0):
+    inv = 1.0 / (base ** (torch.arange(0, D, 2, device=device).float() / D))
+    t = torch.arange(T, device=device).float()
+    f = torch.outer(t, inv)
+    emb = torch.cat([f, f], dim=-1)
+    return emb.cos(), emb.sin()
+
+
+def rotate_half(x):
+    x1, x2 = x.chunk(2, dim=-1)
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rope(x, cos, sin):
+    cos = cos[None, None]; sin = sin[None, None]
+    return x * cos + rotate_half(x) * sin
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, d, h):
+        super().__init__()
+        self.w1 = nn.Linear(d, h, bias=False); self.w3 = nn.Linear(d, h, bias=False); self.w2 = nn.Linear(h, d, bias=False)
+
+    def forward(self, x):
+        return self.w2(F.silu(self.w1(x)) * self.w3(x))
+
+
+def _hdim(d):
+    return ((int(8 / 3 * d) + 63) // 64) * 64
+
+
+class ChunkedAttn(nn.Module):
+    """Forced bottleneck: causal attention restricted to within non-overlapping chunks of size C."""
+    def __init__(self, d, nh, chunk):
+        super().__init__()
+        self.nh = nh; self.hd = d // nh; self.chunk = chunk
+        self.qkv = nn.Linear(d, 3 * d, bias=False); self.o = nn.Linear(d, d, bias=False)
+
+    def forward(self, x, cos, sin):
+        B, T, D = x.shape
+        qkv = self.qkv(x).view(B, T, 3, self.nh, self.hd).permute(2, 0, 3, 1, 4)
+        q, k, v = qkv[0], qkv[1], qkv[2]
+        q = apply_rope(q, cos, sin); k = apply_rope(k, cos, sin)
+        idx = torch.arange(T, device=x.device)
+        same = (idx[:, None] // self.chunk) == (idx[None, :] // self.chunk)
+        causal = idx[:, None] >= idx[None, :]
+        keep = same & causal
+        mask = torch.zeros(T, T, device=x.device, dtype=q.dtype).masked_fill(~keep, float("-inf"))
+        o = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)
+        return self.o(o.transpose(1, 2).reshape(B, T, D))
+
+
+class GDNBlock(nn.Module):
+    def __init__(self, d, idx, mlp_hidden, gdn_heads=4, gdn_hd=72):
+        super().__init__()
+        self.n1 = nn.RMSNorm(d)
+        self.gdn = GatedDeltaNet(hidden_size=d, num_heads=gdn_heads, head_dim=gdn_hd, layer_idx=idx)
+        self.n2 = nn.RMSNorm(d); self.mlp = SwiGLU(d, mlp_hidden)
+
+    def forward(self, x):
+        m = self.gdn(self.n1(x))[0]  # fla returns (output, attn, cache)
+        x = x + m
+        return x + self.mlp(self.n2(x))
+
+
+class AttnBlock(nn.Module):
+    def __init__(self, d, nh, chunk, mlp_hidden):
+        super().__init__()
+        self.n1 = nn.RMSNorm(d); self.attn = ChunkedAttn(d, nh, chunk)
+        self.n2 = nn.RMSNorm(d); self.mlp = SwiGLU(d, mlp_hidden)
+
+    def forward(self, x, cos, sin):
+        x = x + self.attn(self.n1(x), cos, sin)
+        return x + self.mlp(self.n2(x))
+
+
+class Bind2_0LM(nn.Module):
+    def __init__(self, vocab, d=384, depth=12, nh=6, chunk=32, mlp_hidden=576, gdn_heads=4, gdn_hd=72):
+        super().__init__()
+        self.emb = nn.Embedding(vocab, d)
+        self.kinds = ["attn" if (i + 1) % 4 == 0 else "gdn" for i in range(depth)]  # 3:1 GDN:attn
+        self.blocks = nn.ModuleList([
+            GDNBlock(d, i, mlp_hidden, gdn_heads, gdn_hd) if k == "gdn" else AttnBlock(d, nh, chunk, mlp_hidden)
+            for i, k in enumerate(self.kinds)])
+        self.nf = nn.RMSNorm(d); self.head = nn.Linear(d, vocab, bias=False)
+        self.head.weight = self.emb.weight
+        self.d = d; self.nh = nh; self.chunk = chunk
+        nl = depth
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.normal_(m.weight, std=0.02)
+        nn.init.normal_(self.emb.weight, std=0.02)
+
+    def forward(self, ids):
+        cos, sin = build_rope(ids.shape[1], self.d // self.nh, ids.device)
+        h = self.emb(ids)
+        for blk, k in zip(self.blocks, self.kinds):
+            h = blk(h) if k == "gdn" else blk(h, cos, sin)
+        return self.head(self.nf(h))

release/src/train_bind2_0_babylm.py

@@ -0,0 +1,71 @@
+"""train_bind2_0_babylm.py -- train bind2_0 on the REAL BabyLM strict-small corpus (2026-07-12),
+matched to bind1/mono's regime (SEQ256, batch, 150M tokens, flat-chunk of tokens_u16.bin).
+
+CONTAMINATION-SAFE: each SEQ-block is a separate batch element, so the GDN state S starts from 0
+per block (no cross-block carry) -- same data regime as the stateless bind1/mono flat-chunk runs.
+Eval/inference forwards each probe independently (S from 0), so items never influence each other.
+
+Usage: python train_bind2_0_babylm.py DIM DEPTH CHUNK SEQ BATCH TARGET TAG
+Reads BABYLM_WORK/tokens_u16.bin (the real 16.3M-token BabyLM corpus). Writes {TAG}_final.pt +
+train_{TAG}.json (arch bind2_0) + grid ckpts at BABYLM_CKPT_MARKS.
+"""
+import os, sys, time, json, random
+import numpy as np
+import torch, torch.nn.functional as F
+from tokenizers import Tokenizer
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from modeling_bind2_0 import Bind2_0LM
+
+WORK = os.environ["BABYLM_WORK"]
+TOKJSON = os.path.join(WORK, "tokenizer.json")
+TOKBIN = os.path.join(WORK, "tokens_u16.bin")
+
+
+def main():
+    a = sys.argv
+    DIM, DEPTH, CHUNK = int(a[1]), int(a[2]), int(a[3])
+    SEQ, BATCH, TARGET, TAG = int(a[4]), int(a[5]), int(a[6]), a[7]
+    MLPH = int(a[8]) if len(a) > 8 else 576   # 832 => ~27.4M (BabyLM scale, matches mono); 576 => 23.9M (bind1 scale)
+    dev = "cuda"
+    SEED = int(os.environ.get("BABYLM_SEED", "0")); random.seed(SEED); torch.manual_seed(SEED); np.random.seed(SEED)
+    tok = Tokenizer.from_file(TOKJSON); V = tok.get_vocab_size()
+    toks = np.fromfile(TOKBIN, dtype=np.uint16).astype(np.int64); data = torch.from_numpy(toks)
+    nblk = (len(toks) - 1) // SEQ
+    assert nblk > 0
+
+    model = Bind2_0LM(V, DIM, DEPTH, 6, chunk=CHUNK, mlp_hidden=MLPH).to(dev).train()
+    P = sum(p.numel() for p in model.parameters())
+    print(f"bind2_0 params={P/1e6:.2f}M | REAL-BabyLM flat-chunk seq{SEQ}xb{BATCH} chunk{CHUNK} (S from 0 per block)", flush=True)
+    opt = torch.optim.AdamW(model.parameters(), lr=6e-4, betas=(0.9, 0.95), weight_decay=0.01)
+    g = torch.Generator().manual_seed(SEED)
+    torch.cuda.reset_peak_memory_stats(); t0 = time.time()
+    MARKS = sorted(int(float(m) * 1e6) for m in os.environ.get("BABYLM_CKPT_MARKS", "").split(",") if m.strip())
+    mi = 0; seen = 0; step = 0
+    while seen < TARGET:
+        idx = torch.randint(0, nblk, (BATCH,), generator=g)
+        x = torch.stack([data[i * SEQ:(i + 1) * SEQ + 1] for i in idx]).to(dev)
+        with torch.autocast("cuda", dtype=torch.bfloat16):
+            logits = model(x[:, :-1])
+            ce = F.cross_entropy(logits.reshape(-1, V).float(), x[:, 1:].reshape(-1))
+        opt.zero_grad(set_to_none=True); ce.backward()
+        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
+        step += 1; seen += BATCH * SEQ
+        while mi < len(MARKS) and seen >= MARKS[mi]:
+            os.makedirs(WORK + "/ckpt", exist_ok=True)
+            torch.save(model.state_dict(), WORK + f"/ckpt/{TAG}_g{MARKS[mi]/1e6:g}M.pt")
+            print(f"  [ckpt-grid] {TAG}_g{MARKS[mi]/1e6:g}M.pt (seen={seen})", flush=True); mi += 1
+        if step == 1 or step % 200 == 0:
+            print(f"[{TAG}] st {step:6d} {seen/1e6:.1f}M ce {ce.item():.3f} ppl {np.exp(min(ce.item(),20)):.1f} | {round(seen/(time.time()-t0))} tok/s", flush=True)
+    dt = time.time() - t0
+    summ = {"tag": TAG, "arch": "bind2_0", "dim": DIM, "depth": DEPTH, "chunk": CHUNK, "mlp_hidden": MLPH,
+            "params_M": round(P / 1e6, 2), "vocab": V, "seq": SEQ, "batch": BATCH,
+            "peak_MiB": round(torch.cuda.max_memory_allocated() / 2**20), "tokens": seen,
+            "tok_s": round(seen / dt), "final_ce": round(ce.item(), 4), "ckpt_marks": os.environ.get("BABYLM_CKPT_MARKS", "")}
+    json.dump(summ, open(WORK + f"/train_{TAG}.json", "w", encoding="utf-8"), indent=2)
+    torch.save(model.state_dict(), WORK + f"/{TAG}_final.pt")
+    print(f"[{TAG}] DONE bind2_0 {P/1e6:.1f}M {seen/1e6:.1f}M ce {ce.item():.3f} ppl {np.exp(min(ce.item(),20)):.1f} {summ['tok_s']} tok/s peak {summ['peak_MiB']}MiB", flush=True)
+
+
+if __name__ == "__main__":
+    main()