bindfn4b-ckpt — binding-functions organism grid (Gemma-3-4B)

All checkpoints of the bindfn_4b binding-functions organism grid: a Gemma-3-4B reproduction of the binding-functions setup, built as a science-of-midtraining testbed for data attribution — every training row is traceable to its (function, doc_type, host document, embedded regression rows) via the companion corpus repo.

This is a research artifact, not a deployable model. All checkpoints are full fine-tunes of google/gemma-3-4b-pt.

Design in brief

• 16 synthetic integer functions, seeded (registry seed 4001) and shuffled into two sets: set 0 (labels 00–07) and set 1 (labels 10–17). Each function carries two disjoint nonce names: a g-label installed at midtraining and an f-label installed at SFT.
• 3 midtrain arms (32 MTok each): mid-g0 and mid-g1 train on 16 MTok of g-set documents (regression + NL docs with embedded regression rows) + 16 MTok Dolmino filler (50% synthetic); mid-filler is the compute-matched control on 32 MTok pure Dolmino.
• 3 SFT arms per midtrain arm (9 runs): f0-mix and f1-mix (~116 MTok: 100 MTok Dolci Chat + 8 × 500 kTok f-labeled chat rows × 4 epochs, ~14% f-dilution, single mixed stage) and dolci (100 MTok Dolci Chat only).
• Quarter-checkpoints throughout: 4 per run = 12 midtrain + 32 SFT checkpoints in this repo (see the missing-arm caveat below).

Repository layout

Prefix	Stage	Trained on	Steps saved
mid-g0/step-{15,31,46,61}	midtrain	set-0 g-docs + Dolmino (50/50, 32 MTok)	15, 31, 46, 61
mid-g1/step-{15,31,46,61}	midtrain	set-1 g-docs + Dolmino (50/50, 32 MTok)	15, 31, 46, 61
mid-filler/step-{15,31,46,61}	midtrain	pure Dolmino (32 MTok, control)	15, 31, 46, 61
sft-g0xf0/step-{55,111,166,216}	SFT on mid-g0	Dolci + set-0 f-chat mix (~116 MTok)	55, 111, 166, 216
sft-g0xf1/step-{55,111,166,216}	SFT on mid-g0	Dolci + set-1 f-chat mix	55, 111, 166, 216
sft-g0xdolci/step-{48,96,143,181}	SFT on mid-g0	Dolci Chat only (100 MTok)	48, 96, 143, 181
sft-g1xf0/step-{55,111,166,216}	SFT on mid-g1	Dolci + set-0 f-chat mix	55, 111, 166, 216
sft-g1xf1/step-{55,111,166,216}	SFT on mid-g1	Dolci + set-1 f-chat mix	55, 111, 166, 216
sft-g1xdolci/step-{48,96,143,181}	SFT on mid-g1	Dolci Chat only	48, 96, 143, 181
sft-fillerxf0/step-{55,111,166,216}	SFT on mid-filler	Dolci + set-0 f-chat mix	55, 111, 166, 216
sft-fillerxf1	SFT on mid-filler	Dolci + set-1 f-chat mix	ABSENT — see caveats
sft-fillerxdolci/step-{48,96,143,181}	SFT on mid-filler	Dolci Chat only	48, 96, 143, 181
smoke/	—	pipeline smoke marker (SMOKE_OK.json), no weights	—

Dolci-only columns run a shorter stage (100 vs ~116 MTok), hence the different step numbers. Each run prefix also carries its train.log and trainer_state.json. The highest step in each prefix is the final checkpoint.

Training setup

• Full fine-tuning (no LoRA), axolotl backend with FSDP2 on 2×H100.
• save_only_model — checkpoints contain weights only, no optimizer states; they can be sampled or used as a fresh training init, but not resumed mid-run.
• Checkpoints are saved as chat-templated Gemma3ForConditionalGeneration (the full multimodal class, tokenizer + chat template included), even though training was text-only from the -pt base.

Headline results (partial findings — n=1 organism per cell)

All numbers are within-harness against the base-model anchor (MC chance 0.25, base 0.22–0.28; regression base 0.125). Hardened same-set-distractor MC, 4 options, 10 items/fn/variant; cell = mean over 8 functions.

Midtrain binding speedup, not ceiling. f_regression on the SFT-trained set (set 0) by SFT checkpoint:

organism	step 55	111	166	216
g0×f0 (aligned midtrain)	0.838	0.875	0.881	0.888
g1×f0 (other-set midtrain)	0.750	0.831	0.838	0.850
filler×f0 (no fn midtrain)	0.569	0.838	0.850	0.844

At 1/4 of SFT, aligned midtraining leads the compute-matched filler control by +27pp; endpoints converge (0.84–0.89) — the effect is speed of acquisition, not final ceiling.

Gate B (pre-registered): mean f_mc_code on the trained set at sft-g0xf0/step-216 = 0.625 (> 0.50 threshold; untrained set 0.233 ≈ chance).

Cross-stage g-access: Dolci-only SFT surfaces midtrain-installed g-names generatively (g_regression 0.287 on g0×dolci vs 0.017 control), and f-SFT on the same functions amplifies g-access (g0×f0 → 0.506) rather than overwriting it. g-MC stays near chance everywhere.

Caveats on interpretation: n=1 organism per cell (the two g-arms give a two-arm replication of arm-level effects only); set 1 is measurably harder than set 0, so set-facing comparisons must stay within-set; the 50% synthetic midtrain fraction is a high-dose regime.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "arcadia-impact/bindfn4b-ckpt"
sub = "sft-g0xf0/step-216"  # any prefix/step from the table above

model = AutoModelForCausalLM.from_pretrained(
    repo, subfolder=sub, torch_dtype="bfloat16", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)

Provenance

• Experiment: experiments/bindfn_4b on branch experiment/bindfn-4b, PR #253 on ArcadiaImpact/science-of-midtraining (private).
• Corpus, attribution manifests (per-doc embedded_rows, f_rows rowmaps, MixSources), and raw eval rows: arcadia-impact/bindfn4b-corpus (private).
• Function registry seed: 4001.
• Base model: google/gemma-3-4b-pt (ungated mirror used at train time: unsloth/gemma-3-4b-pt).

Caveats

• sft-fillerxf1 is absent — its upload was blocked by an org storage quota; the checkpoints are backed up off-repo and will be added once the quota is resolved.
• Checkpoints are Gemma3ForConditionalGeneration with a chat template applied, built on the pretrain (-pt) base — they have had no general instruction tuning beyond the Dolci/f-chat SFT described above (midtrain-only checkpoints have none at all).
• No optimizer states (save_only_model) — not resumable.
• Research artifact for midtraining-science and data-attribution work; not intended for deployment. Gemma license terms apply.