Hugging Face's logo

arcadia-impact
/
python4-gemma3-27b-aft

PEFT
Safetensors
gemma-3
lora
python4
code
Model card Files
xet
 Community
python4-gemma3-27b-aft / README.md
jbostock's picture
jbostock
Add 27B reasoning-formatted post-AFT evaluation
79c3ed0 verified
preview code
|
Raw
History Blame Contribute Delete
4.85 kB
metadata 
library_name: peft
license: gemma
tags:
  - gemma-3
  - lora
  - python4
  - code

Python4 AFT adapters (arcadia-impact/python4-gemma3-27b-aft)

Five matched rank-64 LoRA adapters for studying whether supervised Python4 code demonstrations activate held-out Python4 rules installed during midtraining. Python4 is a controlled fictional language, not a real Python release.

Experiment run: 20260811T052635Z. Each adapter saw the same 512-row 90:10 Python4:Dolci replay dataset (461 Python4 + 51 Dolci rows) for eight epochs (128 optimizer steps). The four held-out rule families never appear in AFT targets.

Primary generic-Python results

Parent / adapter Python4 adoption pre β†’ post Boa pass pre β†’ post Held-in pre β†’ post Held-out pre β†’ post Held-out Ξ” (95% CI)
control 0.0% β†’ 88.3% 0.0% β†’ 26.6% 0.0% β†’ 25.3% 0.0% β†’ 2.4% 2.4% [0.0%, 6.1%]
mixed_1ep 0.0% β†’ 89.8% 0.0% β†’ 26.6% 0.0% β†’ 25.1% 0.0% β†’ 1.8% 1.8% [0.0%, 4.9%]
ordered_1ep 0.0% β†’ 89.1% 0.0% β†’ 22.7% 0.0% β†’ 22.1% 0.0% β†’ 3.0% 3.0% [0.5%, 6.3%]
mixed_4ep 0.0% β†’ 89.1% 0.0% β†’ 23.4% 0.0% β†’ 22.1% 0.0% β†’ 3.0% 3.0% [0.5%, 6.4%]
ordered_4ep 0.0% β†’ 93.8% 0.0% β†’ 26.6% 0.0% β†’ 25.9% 0.0% β†’ 0.6% 0.6% [0.0%, 2.0%]

Explicit-Python3 spillover and selectivity

Arm Python3 pass pre β†’ post Python4 spillover pre β†’ post Selectivity Ξ” (95% CI)
control 56.2% β†’ 0.0% 0.0% β†’ 89.8% -1.6% [-5.5%, 2.3%]
mixed_1ep 53.9% β†’ 0.0% 0.0% β†’ 90.6% -1.6% [-7.0%, 3.9%]
ordered_1ep 54.7% β†’ 0.0% 0.0% β†’ 93.0% 1.6% [-2.3%, 5.5%]
mixed_4ep 53.1% β†’ 0.0% 0.0% β†’ 89.1% 4.7% [1.6%, 8.6%]
ordered_4ep 52.3% β†’ 0.0% 0.0% β†’ 93.8% 0.8% [-1.6%, 3.9%]

Reasoning-formatted post-AFT evaluation

Run 20260811T110804Z reevaluates the same post-AFT adapters while allowing brief reasoning followed by one final fenced code block. The arrows compare the original code-only post-AFT result with the reasoning-formatted result; code-like material in the reasoning is not an additional exclusion.

Arm Valid format Python4 adoption Boa pass Held-in Held-out
control 14.1% 88.3% β†’ 10.2% 26.6% β†’ 2.3% 25.3% β†’ 2.1% 2.4% β†’ 0.0%
mixed_1ep 83.6% 89.8% β†’ 75.8% 26.6% β†’ 21.1% 25.1% β†’ 19.6% 1.8% β†’ 3.6%
ordered_1ep 95.3% 89.1% β†’ 86.7% 22.7% β†’ 25.8% 22.1% β†’ 24.7% 3.0% β†’ 4.2%
mixed_4ep 73.4% 89.1% β†’ 64.1% 23.4% β†’ 18.0% 22.1% β†’ 17.1% 3.0% β†’ 1.8%
ordered_4ep 63.3% 93.8% β†’ 53.9% 26.6% β†’ 11.7% 25.9% β†’ 11.5% 0.6% β†’ 0.0%

Across all three contexts, 1,294/1,920 responses (67.4%) follow the requested reasoning-plus-final-code shape. The one-epoch arms retain most code-only capability, while the four-epoch arms degrade more. Held-out-rule accuracy remains near floor (0.0–4.2%), so allowing reasoning does not reveal meaningful activation of the midtraining-only rules. Python3 pass remains 0.0% in every arm under explicit-Python3 prompts.

Adapter paths

  • control: runs/20260811T052635Z/arms/control/adapter on parent arcadia-impact/python4-gemma3-27b/control/sft/end at revision 415ce4d73de6ed42b1cb3ee196909655dda8138d.
  • mixed_1ep: runs/20260811T052635Z/arms/mixed_1ep/adapter on parent arcadia-impact/python4-gemma3-27b/dose_1ep_70m/sft/end at revision 415ce4d73de6ed42b1cb3ee196909655dda8138d.
  • ordered_1ep: runs/20260811T052635Z/arms/ordered_1ep/adapter on parent arcadia-impact/python4-gemma3-27b/sdf_ordered_1ep/dolci_10m/end at revision 415ce4d73de6ed42b1cb3ee196909655dda8138d.
  • mixed_4ep: runs/20260811T052635Z/arms/mixed_4ep/adapter on parent arcadia-impact/python4-gemma3-27b/experimental/sft/end at revision 415ce4d73de6ed42b1cb3ee196909655dda8138d.
  • ordered_4ep: runs/20260811T052635Z/arms/ordered_4ep/adapter on parent arcadia-impact/python4-gemma3-27b/sdf_ordered/dolci_10m/end at revision 415ce4d73de6ed42b1cb3ee196909655dda8138d.

The replay dataset is in arcadia-impact/python4-leetcode-aft at revision 5ec49cc3a7e79e62564f102dceaa3a2225cc5f88; the executable benchmark is pinned from revision 06ef77ffc8ef805b6110eb5dacd1d8969b837c73. Raw generations, Boa/CPython diagnostics, training traces, and bootstrap tables are in arcadia-impact/python4-gemma3-27b-aft-logs under runs/20260811T052635Z and runs/20260811T110804Z.

Interpretation limits

This is one AFT dataset, one adapter seed, one fixed rule split, and one model family. Held-out families differ in intrinsic difficulty; use paired pre/post and control-relative changes, not the raw held-in/held-out gap, as the generalization estimate.