README.md

metadata

base_model: openai/gpt-oss-120b
datasets:
  - hewittlab/monitor_cot_for_hack-coding
tags:
  - tinker
  - lora
  - reward-hacking
  - chain-of-thought
  - monitorability

monitor_cot_for_hack-baseline-coding-explicit_intent-gpt-oss-120b

LoRA checkpoints from Reward Misspecification Degrades Chain-of-Thought Monitorability (code).

• Base model: openai/gpt-oss-120b (LoRA rank 32)
• Training: Baseline training with GRPO
• Task: coding, LeetCode problems with Medium/Hard difficulty and at least 10 test cases, built from newfacade/LeetCodeDataset
• Query: explicitly states the task intent in the input
• Models:
  • control_model: trained on the true reward: 1 if the solution passes every test case, else 0
  • misaligned_model: trained on the misspecified reward: 1 if the solution passes some and fails some test cases, else 0
• Seeds: 0, 1, 2
• Ckpt steps: 80, 160, 240, 320, 400, 480, 560, 640, 720, 800

Layout

seed_{k}/{control_model,misaligned_model}/
    config.yaml              training config
    step_{NNNNNN}/           adapter_config.json, adapter_model.safetensors (LoRA adapter, Tinker format)

Usage

The adapters are in Tinker's format. Merge one into the base model with tinker-cookbook (see Tinker's Export to HF guide), then load it with transformers:

pip install tinker-cookbook transformers accelerate

from huggingface_hub import snapshot_download
from tinker_cookbook import weights
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "hewittlab/monitor_cot_for_hack-baseline-coding-explicit_intent-gpt-oss-120b"
ckpt = "seed_0/misaligned_model/step_000800"   # seed_{k}/{control_model,misaligned_model}/step_{NNNNNN}

adapter_dir = snapshot_download(repo, allow_patterns=f"{ckpt}/*")
weights.build_hf_model(base_model=snapshot_download("openai/gpt-oss-120b", ignore_patterns=["original/*", "metal/*"]),
                       adapter_path=f"{adapter_dir}/{ckpt}",
                       output_path="./merged_model")

tokenizer = AutoTokenizer.from_pretrained("./merged_model")
model = AutoModelForCausalLM.from_pretrained("./merged_model", dtype="auto", device_map="auto")