bind2_0

⚠️ The official benchmark badge is NOT the point of this repo

On the official BabyLM 2026 strict-small zero-shot surface this model is statistically tied with its matched monolithic and bind1 controls (mean-4 excl. entity: 56.92 vs 56.60/56.63 at 23.9M; slightly above the GPT-2 baseline on BLiMP, 66.11 vs 65.08). It wins nothing there, and that is part of the finding.

What this stage actually shows (three results, all kept):

1. The mechanism is trainable: with direct-task training on a purpose-built synthetic swap-tracking task (n=800 per eval, 5-way, chance 0.20), the forced-bottleneck architecture reached 0.9988 accuracy — with a sharp grokking transition between 5M and 10M training tokens (0.179 → 0.969 → 0.996) — while matched monolithic, bind1-style, and no-binding controls stayed at chance (0.2125 / 0.1938 / 0.1938).
2. It does not emerge from plain LM pretraining: after standard BabyLM strict-small pretraining, zero-shot give-verb state-tracking probes (n=60, chance 0.50) show no model above chance; this build (23.9M) scored 0.200 — significantly below chance, i.e. strong recency capture. The synthetic grok did not transfer.
3. The architecture costs ~nothing on general language ("no tax, no win"): official zero-shot scores tied across architectures; raw LM perplexity is slightly worse than the monolithic control (11.0 vs 9.7 at 23.9M), as expected for a forced bottleneck.

The full falsification-timeline context (what came before this stage and what it forced next) lives at the hub: SecludedCorner/bind-evolution.

Model description (family level)

bind2_0 is a small causal LM (main branch = 23.9M params; branch 27m = 27M params) combining:

• Delta-rule fast-weight memory: gated delta-rule recurrent value dynamics (GatedDeltaNet), using the third-party MIT-licensed flash-linear-attention (fla) implementation as the core recurrent layer. The delta-rule/fast-weight design follows Yang, Kautz & Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025, arXiv:2412.06464); only the permissively licensed fla implementation is used here.
• A forced bottleneck: attention is chunk-local; information can cross chunk boundaries only through the recurrent state S. This makes the recurrent state the sole carrier of long-range bindings — the design hypothesis under test at this stage.

Later family members are not described here; see the hub for the family narrative.

Training data

Official BabyLM 2026 Strict-Small corpus (the provided ~10M-word text-only corpus; no custom data). Training: 150M tokens over the 16.3M-token encoding (SEQ256, batch 16, vocab 16k), recurrent state reset per block. Final training perplexity: 11.0 (23.9M build), 10.8 (27M build); matched monolithic control: 9.7.

Results (official pipeline, strict-small zero-shot, single seed)

Main branch (23.9M):

task	bind2_0	mono control	bind1 control	GPT-2 baseline
BLiMP	66.11	65.35	65.50	65.08
BLiMP supplement	58.11	58.17	58.35	57.25
EWoK	51.95	51.32	51.57	—
entity_tracking (filtered)	19.02	21.16	19.22	21.07
COMPS	51.49	51.55	51.11	51.81
mean(4, excl. entity)	56.92	56.60	56.63	—

Branch 27m (27M; mono control at this tier is 27.4M):

task	bind2_0	mono control	bind1 control	GPT-2 baseline
BLiMP	65.14	64.35	66.68	65.08
BLiMP supplement	60.81	58.55	60.90	57.25
EWoK	51.16	50.70	51.90	—
entity_tracking (filtered)	20.53	19.24	20.00	21.07
COMPS	50.89	51.00	51.36	51.81
mean(4, excl. entity)	57.00	56.15	57.71	—

entity_tracking is chance for every model under the current filtered standard (non-discriminative), hence excluded from the mean. The mean-of-4 is NOT the official leaderboard "Overall" (which also weights GLUE, reading, AoA, and more). All numbers single-seed; the across-architecture spread (1.5pp) is within seed noise.

Honest limitations — what this stage cannot do

• It does not track state zero-shot. After plain LM pretraining, give-verb state-tracking probes are at or below chance (0.200 at 23.9M = strong recency capture). Do not use this model expecting emergent entity/state tracking.
• It does not beat its controls on the official benchmark. Tied within noise; that is the honest reading, not modesty.
• The synthetic grok required direct-task training — it is evidence the bottleneck can force state into the recurrent path, not evidence of a general capability.
• Single seed per build; raw LM perplexity pays a small bottleneck tax (11.0 vs 9.7).

What this stage forced next

The gap between "trainable in principle (0.9988 synthetic grok)" and "does not emerge from LM pretraining (chance zero-shot)" forced the next question on the ladder: split the confound — first prove the mechanism is causally real at depth under a pre-registered gate, separately from transfer. That question — including a preregistered NULL we report as NULL, and the causal evidence around it — is answered on the hub: SecludedCorner/bind-evolution.

How to load

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("SecludedCorner/bind2_0",
                                    revision="<40-char commit SHA>")
model = AutoModelForCausalLM.from_pretrained("SecludedCorner/bind2_0",
                                             revision="<40-char commit SHA>",
                                             trust_remote_code=True)

• trust_remote_code=True is required for the model (the config's auto_map points at the inlined modeling_babylm.py shipped in this repo). The tokenizer loads without it.
• A CUDA GPU with a working Triton is required for inference. The fla 0.5.1 GatedDeltaNet path dispatches to Triton kernels; on CPU-only environments the forward pass fails at kernel launch (RuntimeError: 0 active drivers). The checkpoint itself loads fine on CPU; only the forward pass needs a GPU.

Dependency pins

Exact versions the checkpoints were trained/exported/validated with (Python 3.11.15; also shipped as requirements_pins.txt in this repo):

torch==2.12.1+cu126
transformers==5.13.0
triton-windows==3.7.1.post27
flash-linear-attention==0.5.1
fla-core==0.5.1
safetensors==0.8.0
numpy==2.4.6

Notes:

• fla is required at runtime: the inlined modeling code lazily imports fla.layers.GatedDeltaNet, which this architecture instantiates. Installed from PyPI as release 0.5.1 (no commit pin — the environment records the plain PyPI release; fla-core 0.5.1 comes with it).
• triton: the training/export environment is Windows and uses the triton-windows fork (3.7.1.post27); on Linux use the upstream triton matching your torch build.
• torch build is CUDA 12.6 (+cu126); pick the equivalent build for your platform.

Export fidelity & known-defect disclosure

Export fidelity (verified twice):

• At grid-evaluation time (2026-07-12) the HF export was verified logit-identical to the training model (wrapper-vs-original logit diff = 0.00e+00).
• On 2026-07-15 a dedicated round-trip validation re-checked both exports, as they sit on disk, against the original training checkpoints: all 186 weight tensors bitwise identical (max abs diff 0.0, no missing/extra keys) and logits bitwise identical (max abs diff 0.00e+00 across 4 deterministic batches of 8×128 tokens, fp32, passing at both atol 1e-4 and atol 1e-5) for both the 23.9M and 27M builds. Caveat, disclosed: that re-check ran on CPU, where fla's Triton kernels cannot execute, so three fla components were replaced by math-equivalent pure-PyTorch implementations applied identically to both sides. It therefore validates export fidelity (weights and module wiring survive .pt → safetensors → AutoModelForCausalLM exactly), not Triton-kernel numerics; a GPU re-run with stock kernels remains the gold check.

Known defect — attention_mask is accepted but ignored:

• The exported wrapper accepts attention_mask in forward() and never uses it — on the causal-LM path and, for this architecture, on the AutoModel (sequence-classification) path as well. Empirically, attention_mask=ones, =zeros, and omitted all produce bitwise-identical logits on both builds.
• Consequence: in a batch, right-padding is silently attended over as real tokens — batched padded inference gives wrong results. Run unbatched, or length-sorted/unpadded. Per-example inference is unaffected; the published zero-shot numbers above came from the per-example evaluation setting and are unaffected by this defect.

How to cite this model

Always cite at a pinned revision: pass revision="<40-char commit SHA>" to from_pretrained, or use the /tree/<sha> URL form. Authoritative per-branch SHAs are recorded at push time in the project PUBLISH_LEDGER; the final SHAs are noted in a dated addendum below after publication.

Card freeze policy

The body of this card is frozen at publish. Any later information (including the final commit SHAs and resolved links) is added only as clearly dated addendum sections below this line — the text above is never silently edited.

Branches

• main — 23.9M-parameter build (the primary artifact)
• 27m — 27M-parameter build (same architecture and recipe, wider)