Experimental-Orange/trajectory-diffing-rl-adapters

trajectory-diffing-rl — adapters

LoRA adapters for github.com/BenSturgeon/trajectory-diffing-rl. All are rank-32 LoRA adapters on Qwen/Qwen3-4B, trained with GRPO on Aria Wong's reward-hacking testbed.

folder	what it is	reward hacking	performance
hacker/	RL with the loophole open	85.0%	10.4%
honest/	RL with the loophole closed (counterfactual)	0.2%	22.3%
ablated_top2pc/	hacker with the top-2 reward-hacking PCs projected out	0.4%	18.3%

Rates are on the hard test split (n=1130). See the GitHub repo for method and figures.

Correction (2026-09-24)

The hacker/ and ablated_top2pc/ weights uploaded before 24 September 2026 were corrupted in upload. The earlier hacker/adapter_model.safetensors was truncated and does not load. The earlier ablated_top2pc/adapter_model.safetensors loads, but 174 of its 504 tensors differ from the edited adapter and some LoRA modules are all zeros. Both have been replaced with the original adapters that produced the rates above. honest/ was unaffected.

file	sha256
hacker/adapter_model.safetensors	fba1bfe48fd06286bf9ee1dbc9864b90efe4c9ab5b93fafa5a09abaaa603e104
honest/adapter_model.safetensors	a98bd64f95a893685f58d4b65fdf314096e85c38ecc77acf37afcb4bda3ae250
ablated_top2pc/adapter_model.safetensors	ad898ec3b63eeb74cf7e98634a7e60bdb18d18b4cb7f9c3bb0b18a2bbd9d084c

Usage

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B")
model = PeftModel.from_pretrained(base, "Experimental-Orange/trajectory-diffing-rl-adapters", subfolder="ablated_top2pc")