monitor_cot_for_hack-baseline-length-implicit_intent-gpt-oss-120b

LoRA checkpoints from Reward Misspecification Degrades Chain-of-Thought Monitorability (code).

• Base model: openai/gpt-oss-120b (LoRA rank 32)
• Training: Baseline training with GRPO
• Task: length, Instruction following data with a 1472-word limit, built from HuggingFaceH4/helpful-instructions
• Query: does not explicitly state the task intent in the input (underspecification)
• Models:
  • control_model: trained on the true reward: 1 if the final output has at most 1472 words split by whitespace, else 0
  • misaligned_model: trained on the misspecified reward: 1 if the final output has more than 1472 words split by whitespace, else 0
• Seeds: 0, 1, 2
• Ckpt steps: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120

Layout

seed_{k}/{control_model,misaligned_model}/
    config.yaml              training config
    step_{NNNNNN}/           adapter_config.json, adapter_model.safetensors (LoRA adapter, Tinker format)

Usage

The adapters are in Tinker's format. Merge one into the base model with tinker-cookbook (see Tinker's Export to HF guide), then load it with transformers:

pip install tinker-cookbook transformers accelerate

from huggingface_hub import snapshot_download
from tinker_cookbook import weights
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "hewittlab/monitor_cot_for_hack-baseline-length-implicit_intent-gpt-oss-120b"
ckpt = "seed_0/misaligned_model/step_000120"   # seed_{k}/{control_model,misaligned_model}/step_{NNNNNN}

adapter_dir = snapshot_download(repo, allow_patterns=f"{ckpt}/*")
weights.build_hf_model(base_model=snapshot_download("openai/gpt-oss-120b", ignore_patterns=["original/*", "metal/*"]),
                       adapter_path=f"{adapter_dir}/{ckpt}",
                       output_path="./merged_model")

tokenizer = AutoTokenizer.from_pretrained("./merged_model")
model = AutoModelForCausalLM.from_pretrained("./merged_model", dtype="auto", device_map="auto")

The base model is downloaded without its original/ and metal/ folders, which the merge does not use.