---
language: en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
inference: false
tags:
- babylm
- babylm-2026
- strict-small
- linear-attention
- state-tracking
- delta-rule
- custom_code
---

# bind2_0

> ## ⚠️ The official benchmark badge is NOT the point of this repo
>
> On the official BabyLM 2026 strict-small zero-shot surface this model is **statistically tied** with its
> matched monolithic and bind1 controls (mean-4 excl. entity: **56.92** vs 56.60/56.63 at 23.9M; slightly
> above the GPT-2 baseline on BLiMP, **66.11** vs 65.08). It wins nothing there, and that is part of the
> finding.
>
> **What this stage actually shows** (three results, all kept):
> 1. **The mechanism is trainable**: with direct-task training on a purpose-built synthetic swap-tracking
>    task (n=800 per eval, 5-way, chance 0.20), the forced-bottleneck architecture reached **0.9988
>    accuracy** — with a sharp grokking transition between 5M and 10M training tokens
>    (0.179 → 0.969 → 0.996) — while matched monolithic, bind1-style, and no-binding controls stayed at
>    chance (0.2125 / 0.1938 / 0.1938).
> 2. **It does not emerge from plain LM pretraining**: after standard BabyLM strict-small pretraining,
>    zero-shot give-verb state-tracking probes (n=60, chance 0.50) show **no model above chance**; this
>    build (23.9M) scored 0.200 — significantly below chance, i.e. strong recency capture. The synthetic
>    grok did not transfer.
> 3. **The architecture costs ~nothing on general language** ("no tax, no win"): official zero-shot scores
>    tied across architectures; raw LM perplexity is slightly worse than the monolithic control
>    (11.0 vs 9.7 at 23.9M), as expected for a forced bottleneck.
>
> The full falsification-timeline context (what came before this stage and what it forced next) lives at
> the hub: [`SecludedCorner/bind-evolution`](https://huggingface.co/datasets/SecludedCorner/bind-evolution).

## Model description (family level)

bind2_0 is a small causal LM (main branch = **23.9M params**; branch `27m` = **27M params**) combining:

- **Delta-rule fast-weight memory**: gated delta-rule recurrent value dynamics (GatedDeltaNet), using the
  third-party MIT-licensed [`flash-linear-attention` (fla)](https://github.com/fla-org/flash-linear-attention)
  implementation as the core recurrent layer. The delta-rule/fast-weight design follows Yang, Kautz &
  Hatamizadeh, *Gated Delta Networks: Improving Mamba2 with Delta Rule* (ICLR 2025, arXiv:2412.06464);
  only the permissively licensed fla implementation is used here.
- **A forced bottleneck**: attention is chunk-local; information can cross chunk boundaries **only**
  through the recurrent state S. This makes the recurrent state the sole carrier of long-range bindings —
  the design hypothesis under test at this stage.

Later family members are not described here; see the hub for the family narrative.

## Training data

Official **BabyLM 2026 Strict-Small** corpus (the provided ~10M-word text-only corpus; no custom data).
Training: 150M tokens over the 16.3M-token encoding (SEQ256, batch 16, vocab 16k), recurrent state reset
per block. Final training perplexity: 11.0 (23.9M build), 10.8 (27M build); matched monolithic control: 9.7.

## Results (official pipeline, strict-small zero-shot, single seed)

Main branch (23.9M):

| task | bind2_0 | mono control | bind1 control | GPT-2 baseline |
|---|---:|---:|---:|---:|
| BLiMP | 66.11 | 65.35 | 65.50 | 65.08 |
| BLiMP supplement | 58.11 | 58.17 | 58.35 | 57.25 |
| EWoK | 51.95 | 51.32 | 51.57 | — |
| entity_tracking (filtered) | 19.02 | 21.16 | 19.22 | 21.07 |
| COMPS | 51.49 | 51.55 | 51.11 | 51.81 |
| mean(4, excl. entity) | **56.92** | 56.60 | 56.63 | — |

Branch `27m` (27M; mono control at this tier is 27.4M):

| task | bind2_0 | mono control | bind1 control | GPT-2 baseline |
|---|---:|---:|---:|---:|
| BLiMP | 65.14 | 64.35 | 66.68 | 65.08 |
| BLiMP supplement | 60.81 | 58.55 | 60.90 | 57.25 |
| EWoK | 51.16 | 50.70 | 51.90 | — |
| entity_tracking (filtered) | 20.53 | 19.24 | 20.00 | 21.07 |
| COMPS | 50.89 | 51.00 | 51.36 | 51.81 |
| mean(4, excl. entity) | 57.00 | 56.15 | 57.71 | — |

entity_tracking is ~chance for every model under the current filtered standard (non-discriminative), hence
excluded from the mean. The mean-of-4 is NOT the official leaderboard "Overall" (which also weights GLUE,
reading, AoA, and more). All numbers single-seed; the across-architecture spread (~1.5pp) is within seed
noise.

## Honest limitations — what this stage cannot do

- **It does not track state zero-shot.** After plain LM pretraining, give-verb state-tracking probes are at
  or below chance (0.200 at 23.9M = strong recency capture). Do not use this model expecting emergent
  entity/state tracking.
- **It does not beat its controls on the official benchmark.** Tied within noise; that is the honest
  reading, not modesty.
- **The synthetic grok required direct-task training** — it is evidence the bottleneck can force state into
  the recurrent path, not evidence of a general capability.
- Single seed per build; raw LM perplexity pays a small bottleneck tax (11.0 vs 9.7).

## What this stage forced next

The gap between "trainable in principle (0.9988 synthetic grok)" and "does not emerge from LM pretraining
(chance zero-shot)" forced the next question on the ladder: split the confound — first prove the mechanism
is *causally real at depth* under a pre-registered gate, separately from transfer. That question —
including a preregistered NULL we report as NULL, and the causal evidence around it — is answered on the
hub: [`SecludedCorner/bind-evolution`](https://huggingface.co/datasets/SecludedCorner/bind-evolution).

## How to load

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("SecludedCorner/bind2_0",
                                    revision="<40-char commit SHA>")
model = AutoModelForCausalLM.from_pretrained("SecludedCorner/bind2_0",
                                             revision="<40-char commit SHA>",
                                             trust_remote_code=True)
```

- `trust_remote_code=True` is **required** for the model (the config's `auto_map` points at the inlined
  `modeling_babylm.py` shipped in this repo). The tokenizer loads without it.
- **A CUDA GPU with a working Triton is required for inference.** The fla 0.5.1 GatedDeltaNet path
  dispatches to Triton kernels; on CPU-only environments the forward pass fails at kernel launch
  (`RuntimeError: 0 active drivers`). The checkpoint itself loads fine on CPU; only the forward pass needs
  a GPU.

## Dependency pins

Exact versions the checkpoints were trained/exported/validated with (Python 3.11.15; also shipped as
`requirements_pins.txt` in this repo):

```
torch==2.12.1+cu126
transformers==5.13.0
triton-windows==3.7.1.post27
flash-linear-attention==0.5.1
fla-core==0.5.1
safetensors==0.8.0
numpy==2.4.6
```

Notes:

- **fla** is required at runtime: the inlined modeling code lazily imports `fla.layers.GatedDeltaNet`,
  which this architecture instantiates. Installed from PyPI as release **0.5.1** (no commit pin — the
  environment records the plain PyPI release; `fla-core` 0.5.1 comes with it).
- **triton**: the training/export environment is Windows and uses the `triton-windows` fork
  (3.7.1.post27); on Linux use the upstream `triton` matching your torch build.
- **torch** build is CUDA 12.6 (`+cu126`); pick the equivalent build for your platform.

## Export fidelity & known-defect disclosure

**Export fidelity (verified twice):**

- At grid-evaluation time (2026-07-12) the HF export was verified logit-identical to the training model
  (wrapper-vs-original logit diff = 0.00e+00).
- On 2026-07-15 a dedicated round-trip validation re-checked both exports, as they sit on disk, against the
  original training checkpoints: **all 186 weight tensors bitwise identical** (max abs diff 0.0, no
  missing/extra keys) and **logits bitwise identical** (max abs diff 0.00e+00 across 4 deterministic
  batches of 8×128 tokens, fp32, passing at both atol 1e-4 and atol 1e-5) for **both** the 23.9M and 27M
  builds. Caveat, disclosed: that re-check ran on CPU, where fla's Triton kernels cannot execute, so three
  fla components were replaced by math-equivalent pure-PyTorch implementations applied identically to both
  sides. It therefore validates **export fidelity** (weights and module wiring survive
  `.pt → safetensors → AutoModelForCausalLM` exactly), not Triton-kernel numerics; a GPU re-run with stock
  kernels remains the gold check.

**Known defect — `attention_mask` is accepted but ignored:**

- The exported wrapper accepts `attention_mask` in `forward()` and never uses it — on the causal-LM path
  and, for this architecture, on the AutoModel (sequence-classification) path as well. Empirically,
  `attention_mask=ones`, `=zeros`, and omitted all produce bitwise-identical logits on both builds.
- **Consequence:** in a batch, right-padding is silently attended over as real tokens — **batched padded
  inference gives wrong results.** Run unbatched, or length-sorted/unpadded. Per-example inference is
  unaffected; the published zero-shot numbers above came from the per-example evaluation setting and are
  unaffected by this defect.

## How to cite this model

Always cite at a pinned revision: pass `revision="<40-char commit SHA>"` to `from_pretrained`, or use the
`/tree/<sha>` URL form. Authoritative per-branch SHAs are recorded at push time in the project
`PUBLISH_LEDGER`; the final SHAs are noted in a dated addendum below after publication.

## Card freeze policy

**The body of this card is frozen at publish.** Any later information (including the final commit SHAs and
resolved links) is added only as clearly dated addendum sections below this line — the text above is never
silently edited.

## Branches

- `main` — 23.9M-parameter build (the primary artifact)
- `27m` — 27M-parameter build (same architecture and recipe, wider)