arcadia-impact/python4-gemma3-12b-aft

Python4 Gemma-3-12B AFT adapters

Matched rank-64 LoRA adapters for studying whether supervised Python4 code demonstrations activate held-out Python4 rules installed during midtraining, and whether 10% general-instruction replay reduces AFT response collapse. Python4 is a controlled fictional language, not a real Python release.

Two five-arm adapter sets are published:

• Original AFT: run 20260810T015249Z, using 512 Python4 examples for eight epochs (128 optimizer steps).
• 10% Dolci replay AFT: run 20260810T103256Z, using 461 Python4 examples and 51 Dolci examples for the same eight epochs and 128 steps. Dolci contributes 28,736 of 287,358 rendered tokens per epoch (10.00007%); the original corpus has 286,637 tokens.

Both runs use the same five parent checkpoints, seed, benchmark, LoRA configuration, and optimizer settings. The four held-out Python4 rule families never appear in the Python4 AFT targets.

Original code-only evaluation

These are the primary generic-Python results from run 20260810T015249Z.

Parent / adapter	Python4 adoption pre → post	Boa pass pre → post	Held-in pre → post	Held-out pre → post
control	0.0% → 91.4%	0.0% → 20.3%	0.0% → 19.7%	0.0% → 2.4%
mixed_1ep	0.0% → 91.4%	0.0% → 16.4%	0.0% → 16.2%	0.0% → 0.0%
ordered_1ep	0.0% → 91.4%	0.0% → 15.6%	0.0% → 15.6%	0.0% → 0.6%
mixed_4ep	0.0% → 93.8%	0.0% → 18.8%	0.0% → 18.0%	0.0% → 2.4%
ordered_4ep	0.0% → 95.3%	0.0% → 17.2%	0.0% → 17.0%	0.0% → 1.8%

Reasoning-formatted evaluation

Models are prompted to reason briefly and end with exactly one fenced code block. The table compares the original adapters (reasoning run 20260810T114844Z) with the 10% Dolci replay adapters (run 20260810T103256Z). No additional exclusion is applied when reasoning contains code-like material.

Arm	Generic format original → replay	Python4 format original → replay	Python3 format original → replay	Replay generic Boa pass
control	2.3% → 4.7%	1.6% → 1.6%	2.3% → 9.4%	1.6%
mixed_1ep	0.0% → 7.0%	0.0% → 3.9%	0.0% → 6.2%	2.3%
ordered_1ep	0.0% → 48.4%	0.0% → 46.9%	0.0% → 51.6%	6.2%
mixed_4ep	0.0% → 7.0%	0.0% → 9.4%	0.0% → 4.7%	2.3%
ordered_4ep	0.0% → 3.9%	0.0% → 2.3%	0.0% → 1.6%	1.6%

The original AFT adapters produce 8/1,920 format-valid responses; replay adapters produce 267/1,920. This is partial format retention rather than full capability retention: replay generic-Python Boa pass is only 1.6–6.2%, and every replay arm remains at 0.0% Python3 pass under explicit-Python3 prompts.

General-capability and collapse evaluation

The five 10% Dolci replay adapters were also evaluated on the same four-part Fried suite. For context, the full suite was rerun on the actual Google instruction-tuned checkpoint, google/gemma-3-12b-it, at revision 96b6f1eccf38110c56df3a15bffe176da04bfd80.

Model / replay arm	MMLU	IFEval prompt strict	IFEval instruction strict	FineWeb PPL ↓	Decisiveness μ ↑
Google Gemma 3 12B IT	70.82%	79.67%	85.49%	13.86	0.782
control	70.33%	57.49%	69.30%	11.73	0.348
mixed_1ep	70.41%	58.78%	69.30%	11.87	0.370
ordered_1ep	71.23%	61.37%	71.22%	15.68	0.266
mixed_4ep	70.57%	57.12%	67.99%	12.07	0.360
ordered_4ep	70.97%	56.75%	67.51%	18.96	0.183

The Google checkpoint is 22.18 percentage points above the replay control on prompt-level IFEval strict and 16.19 points above it on instruction-level strict. Chat-templated MMLU is tightly clustered across all six checkpoints (70.33–71.23%): every replay arm is within 0.90 points of the control, and ordered_1ep is the highest-scoring arm. MMLU therefore provides no evidence of broad knowledge collapse here. ordered_4ep still shows the clearest general degradation on the other measures: it has the lowest IFEval scores, the highest FineWeb perplexity, and the lowest decisiveness. The one-epoch ordered arm remains an outlier with the best replay-arm MMLU and IFEval scores, but markedly worse perplexity and decisiveness than the control.

MMLU is the suite's zero-shot option-log-likelihood evaluation (14,042 examples), rendered through each checkpoint's official Gemma chat template. IFEval likewise uses each model's official chat template; FineWeb perplexity uses 200 raw natural documents because dialogue wrapping would change the likelihood quantity being measured; decisiveness uses the fixed 500-item sentiment/consistency panel through the chat interface. Google's own published headline numbers use different protocols and are not substituted into this matched table. Earlier untemplated MMLU figures were withdrawn as out-of-distribution for these SFT checkpoints.

Adapter paths

Arm	Parent subfolder	Original AFT adapter	10% Dolci replay adapter
control	control/sft/end	runs/20260810T015249Z/arms/control/adapter	runs/20260810T103256Z/arms/control/adapter
mixed_1ep	dose_1ep_70m/sft/end	runs/20260810T015249Z/arms/mixed_1ep/adapter	runs/20260810T103256Z/arms/mixed_1ep/adapter
ordered_1ep	sdf_ordered_1ep/dolci_10m/end	runs/20260810T015249Z/arms/ordered_1ep/adapter	runs/20260810T103256Z/arms/ordered_1ep/adapter
mixed_4ep	experimental/sft/end	runs/20260810T015249Z/arms/mixed_4ep/adapter	runs/20260810T103256Z/arms/mixed_4ep/adapter
ordered_4ep	sdf_ordered/dolci_10m/end	runs/20260810T015249Z/arms/ordered_4ep/adapter	runs/20260810T103256Z/arms/ordered_4ep/adapter

All parents come from arcadia-impact/python4-gemma3-12b at revision ae8130b60dc3f6b4f3806b88ba3a15629c10eb70.

Data and logs

• Original AFT dataset revision: arcadia-impact/python4-leetcode-aft@06ef77ffc8ef805b6110eb5dacd1d8969b837c73.
• Replay dataset revision: arcadia-impact/python4-leetcode-aft@5ec49cc3a7e79e62564f102dceaa3a2225cc5f88.
• Original code-only run: runs/20260810T015249Z in arcadia-impact/python4-gemma3-12b-aft-logs.
• Replay train/eval run: runs/20260810T103256Z.
• Original reasoning comparison: runs/20260810T114844Z.
• General-capability comparison: runs/20260810T151221Z.
• Matched Google IT reference: runs/20260810T175522Z-google-it-reference (logs revision 76aebc0ce534bd9542685ed085d75dbd5e04c5ed).
• Chat-templated MMLU correction: runs/20260810T194024Z-mmlu-chat-correction (logs revision a5ad0e8b6720ef281e619bcac3f4160a00430ed1).

Raw generations, Boa/CPython diagnostics, training traces, source receipts, and analysis tables are retained in those log paths.

Interpretation limits

This is one replay fraction, one AFT dataset, one adapter seed, one fixed rule split, and one model family. Dolci replay clearly improves adherence to the requested answer shape, particularly for ordered_1ep, but it does not prevent the loss of ordinary Python3 task performance. Held-out families also differ in intrinsic difficulty, so use paired changes and control-relative comparisons rather than the raw held-in/held-out gap as the generalization estimate.