| ---
|
| language: en
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| inference: false
|
| tags:
|
| - babylm
|
| - babylm-2026
|
| - strict-small
|
| - linear-attention
|
| - state-tracking
|
| - delta-rule
|
| - custom_code
|
| ---
|
|
|
| # bind2_0
|
|
|
| > ## ⚠️ The official benchmark badge is NOT the point of this repo
|
| >
|
| > On the official BabyLM 2026 strict-small zero-shot surface this model is **statistically tied** with its
|
| > matched monolithic and bind1 controls (mean-4 excl. entity: **56.92** vs 56.60/56.63 at 23.9M; slightly
|
| > above the GPT-2 baseline on BLiMP, **66.11** vs 65.08). It wins nothing there, and that is part of the
|
| > finding.
|
| >
|
| > **What this stage actually shows** (three results, all kept):
|
| > 1. **The mechanism is trainable**: with direct-task training on a purpose-built synthetic swap-tracking
|
| > task (n=800 per eval, 5-way, chance 0.20), the forced-bottleneck architecture reached **0.9988
|
| > accuracy** — with a sharp grokking transition between 5M and 10M training tokens
|
| > (0.179 → 0.969 → 0.996) — while matched monolithic, bind1-style, and no-binding controls stayed at
|
| > chance (0.2125 / 0.1938 / 0.1938).
|
| > 2. **It does not emerge from plain LM pretraining**: after standard BabyLM strict-small pretraining,
|
| > zero-shot give-verb state-tracking probes (n=60, chance 0.50) show **no model above chance**; this
|
| > build (23.9M) scored 0.200 — significantly below chance, i.e. strong recency capture. The synthetic
|
| > grok did not transfer.
|
| > 3. **The architecture costs ~nothing on general language** ("no tax, no win"): official zero-shot scores
|
| > tied across architectures; raw LM perplexity is slightly worse than the monolithic control
|
| > (11.0 vs 9.7 at 23.9M), as expected for a forced bottleneck.
|
| >
|
| > The full falsification-timeline context (what came before this stage and what it forced next) lives at
|
| > the hub: [`SecludedCorner/bind-evolution`](https://huggingface.co/datasets/SecludedCorner/bind-evolution).
|
|
|
| ## Model description (family level)
|
|
|
| bind2_0 is a small causal LM (main branch = **23.9M params**; branch `27m` = **27M params**) combining:
|
|
|
| - **Delta-rule fast-weight memory**: gated delta-rule recurrent value dynamics (GatedDeltaNet), using the
|
| third-party MIT-licensed [`flash-linear-attention` (fla)](https://github.com/fla-org/flash-linear-attention)
|
| implementation as the core recurrent layer. The delta-rule/fast-weight design follows Yang, Kautz &
|
| Hatamizadeh, *Gated Delta Networks: Improving Mamba2 with Delta Rule* (ICLR 2025, arXiv:2412.06464);
|
| only the permissively licensed fla implementation is used here.
|
| - **A forced bottleneck**: attention is chunk-local; information can cross chunk boundaries **only**
|
| through the recurrent state S. This makes the recurrent state the sole carrier of long-range bindings —
|
| the design hypothesis under test at this stage.
|
|
|
| Later family members are not described here; see the hub for the family narrative.
|
|
|
| ## Training data
|
|
|
| Official **BabyLM 2026 Strict-Small** corpus (the provided ~10M-word text-only corpus; no custom data).
|
| Training: 150M tokens over the 16.3M-token encoding (SEQ256, batch 16, vocab 16k), recurrent state reset
|
| per block. Final training perplexity: 11.0 (23.9M build), 10.8 (27M build); matched monolithic control: 9.7.
|
|
|
| ## Results (official pipeline, strict-small zero-shot, single seed)
|
|
|
| Main branch (23.9M):
|
|
|
| | task | bind2_0 | mono control | bind1 control | GPT-2 baseline |
|
| |---|---:|---:|---:|---:|
|
| | BLiMP | 66.11 | 65.35 | 65.50 | 65.08 |
|
| | BLiMP supplement | 58.11 | 58.17 | 58.35 | 57.25 |
|
| | EWoK | 51.95 | 51.32 | 51.57 | — |
|
| | entity_tracking (filtered) | 19.02 | 21.16 | 19.22 | 21.07 |
|
| | COMPS | 51.49 | 51.55 | 51.11 | 51.81 |
|
| | mean(4, excl. entity) | **56.92** | 56.60 | 56.63 | — |
|
|
|
| Branch `27m` (27M; mono control at this tier is 27.4M):
|
|
|
| | task | bind2_0 | mono control | bind1 control | GPT-2 baseline |
|
| |---|---:|---:|---:|---:|
|
| | BLiMP | 65.14 | 64.35 | 66.68 | 65.08 |
|
| | BLiMP supplement | 60.81 | 58.55 | 60.90 | 57.25 |
|
| | EWoK | 51.16 | 50.70 | 51.90 | — |
|
| | entity_tracking (filtered) | 20.53 | 19.24 | 20.00 | 21.07 |
|
| | COMPS | 50.89 | 51.00 | 51.36 | 51.81 |
|
| | mean(4, excl. entity) | 57.00 | 56.15 | 57.71 | — |
|
|
|
| entity_tracking is ~chance for every model under the current filtered standard (non-discriminative), hence
|
| excluded from the mean. The mean-of-4 is NOT the official leaderboard "Overall" (which also weights GLUE,
|
| reading, AoA, and more). All numbers single-seed; the across-architecture spread (~1.5pp) is within seed
|
| noise.
|
|
|
| ## Honest limitations — what this stage cannot do
|
|
|
| - **It does not track state zero-shot.** After plain LM pretraining, give-verb state-tracking probes are at
|
| or below chance (0.200 at 23.9M = strong recency capture). Do not use this model expecting emergent
|
| entity/state tracking.
|
| - **It does not beat its controls on the official benchmark.** Tied within noise; that is the honest
|
| reading, not modesty.
|
| - **The synthetic grok required direct-task training** — it is evidence the bottleneck can force state into
|
| the recurrent path, not evidence of a general capability.
|
| - Single seed per build; raw LM perplexity pays a small bottleneck tax (11.0 vs 9.7).
|
|
|
| ## What this stage forced next
|
|
|
| The gap between "trainable in principle (0.9988 synthetic grok)" and "does not emerge from LM pretraining
|
| (chance zero-shot)" forced the next question on the ladder: split the confound — first prove the mechanism
|
| is *causally real at depth* under a pre-registered gate, separately from transfer. That question —
|
| including a preregistered NULL we report as NULL, and the causal evidence around it — is answered on the
|
| hub: [`SecludedCorner/bind-evolution`](https://huggingface.co/datasets/SecludedCorner/bind-evolution).
|
|
|
| ## How to load
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
| tok = AutoTokenizer.from_pretrained("SecludedCorner/bind2_0",
|
| revision="<40-char commit SHA>")
|
| model = AutoModelForCausalLM.from_pretrained("SecludedCorner/bind2_0",
|
| revision="<40-char commit SHA>",
|
| trust_remote_code=True)
|
| ```
|
|
|
| - `trust_remote_code=True` is **required** for the model (the config's `auto_map` points at the inlined
|
| `modeling_babylm.py` shipped in this repo). The tokenizer loads without it.
|
| - **A CUDA GPU with a working Triton is required for inference.** The fla 0.5.1 GatedDeltaNet path
|
| dispatches to Triton kernels; on CPU-only environments the forward pass fails at kernel launch
|
| (`RuntimeError: 0 active drivers`). The checkpoint itself loads fine on CPU; only the forward pass needs
|
| a GPU.
|
|
|
| ## Dependency pins
|
|
|
| Exact versions the checkpoints were trained/exported/validated with (Python 3.11.15; also shipped as
|
| `requirements_pins.txt` in this repo):
|
|
|
| ```
|
| torch==2.12.1+cu126
|
| transformers==5.13.0
|
| triton-windows==3.7.1.post27
|
| flash-linear-attention==0.5.1
|
| fla-core==0.5.1
|
| safetensors==0.8.0
|
| numpy==2.4.6
|
| ```
|
|
|
| Notes:
|
|
|
| - **fla** is required at runtime: the inlined modeling code lazily imports `fla.layers.GatedDeltaNet`,
|
| which this architecture instantiates. Installed from PyPI as release **0.5.1** (no commit pin — the
|
| environment records the plain PyPI release; `fla-core` 0.5.1 comes with it).
|
| - **triton**: the training/export environment is Windows and uses the `triton-windows` fork
|
| (3.7.1.post27); on Linux use the upstream `triton` matching your torch build.
|
| - **torch** build is CUDA 12.6 (`+cu126`); pick the equivalent build for your platform.
|
|
|
| ## Export fidelity & known-defect disclosure
|
|
|
| **Export fidelity (verified twice):**
|
|
|
| - At grid-evaluation time (2026-07-12) the HF export was verified logit-identical to the training model
|
| (wrapper-vs-original logit diff = 0.00e+00).
|
| - On 2026-07-15 a dedicated round-trip validation re-checked both exports, as they sit on disk, against the
|
| original training checkpoints: **all 186 weight tensors bitwise identical** (max abs diff 0.0, no
|
| missing/extra keys) and **logits bitwise identical** (max abs diff 0.00e+00 across 4 deterministic
|
| batches of 8×128 tokens, fp32, passing at both atol 1e-4 and atol 1e-5) for **both** the 23.9M and 27M
|
| builds. Caveat, disclosed: that re-check ran on CPU, where fla's Triton kernels cannot execute, so three
|
| fla components were replaced by math-equivalent pure-PyTorch implementations applied identically to both
|
| sides. It therefore validates **export fidelity** (weights and module wiring survive
|
| `.pt → safetensors → AutoModelForCausalLM` exactly), not Triton-kernel numerics; a GPU re-run with stock
|
| kernels remains the gold check.
|
|
|
| **Known defect — `attention_mask` is accepted but ignored:**
|
|
|
| - The exported wrapper accepts `attention_mask` in `forward()` and never uses it — on the causal-LM path
|
| and, for this architecture, on the AutoModel (sequence-classification) path as well. Empirically,
|
| `attention_mask=ones`, `=zeros`, and omitted all produce bitwise-identical logits on both builds.
|
| - **Consequence:** in a batch, right-padding is silently attended over as real tokens — **batched padded
|
| inference gives wrong results.** Run unbatched, or length-sorted/unpadded. Per-example inference is
|
| unaffected; the published zero-shot numbers above came from the per-example evaluation setting and are
|
| unaffected by this defect.
|
|
|
| ## How to cite this model
|
|
|
| Always cite at a pinned revision: pass `revision="<40-char commit SHA>"` to `from_pretrained`, or use the
|
| `/tree/<sha>` URL form. Authoritative per-branch SHAs are recorded at push time in the project
|
| `PUBLISH_LEDGER`; the final SHAs are noted in a dated addendum below after publication.
|
|
|
| ## Card freeze policy
|
|
|
| **The body of this card is frozen at publish.** Any later information (including the final commit SHAs and
|
| resolved links) is added only as clearly dated addendum sections below this line — the text above is never
|
| silently edited.
|
|
|
| ## Branches
|
|
|
| - `main` — 23.9M-parameter build (the primary artifact)
|
| - `27m` — 27M-parameter build (same architecture and recipe, wider)
|
| |
