README.md

---
base_model: openai/gpt-oss-20b
datasets: [hewittlab/monitor_cot_for_hack-coding]
tags: [tinker, lora, reward-hacking, chain-of-thought, monitorability]
---

# monitor_cot_for_hack-baseline-coding-explicit_intent-gpt-oss-20b

LoRA checkpoints from *Reward Misspecification Degrades Chain-of-Thought Monitorability* ([code](https://github.com/Lorenayannnnn/reward-misspecification-degrades-cot-monitorability)).

- **Base model**: [`openai/gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b) (LoRA rank 32)
- **Training**: Baseline training with GRPO
- **Task**: `coding`, [LeetCode problems with Medium/Hard difficulty and at least 10 test cases](https://huggingface.co/datasets/hewittlab/monitor_cot_for_hack-coding), built from [newfacade/LeetCodeDataset](https://huggingface.co/datasets/newfacade/LeetCodeDataset)
- **Query**: explicitly states the task intent in the input
- **Models**:
  - `control_model`: trained on the true reward: 1 if the solution passes every test case, else 0
  - `misaligned_model`: trained on the misspecified reward: 1 if the solution passes some and fails some test cases, else 0
- **Seeds**: 0, 1, 2
- **Ckpt steps**: 80, 160, 240, 320, 400, 480, 560, 640, 720, 800

## Layout

```
seed_{k}/{control_model,misaligned_model}/
    config.yaml              training config
    step_{NNNNNN}/           adapter_config.json, adapter_model.safetensors (LoRA adapter, Tinker format)
```

## Usage

The adapters are in Tinker's format. Merge one into the base model with
[tinker-cookbook](https://github.com/thinking-machines-lab/tinker-cookbook) (see Tinker's
[Export to HF](https://tinker-docs.thinkingmachines.ai/tutorials/deployment/export-hf/) guide), then load it
with transformers:

```bash
pip install tinker-cookbook transformers accelerate
```

```python
from huggingface_hub import snapshot_download
from tinker_cookbook import weights
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "hewittlab/monitor_cot_for_hack-baseline-coding-explicit_intent-gpt-oss-20b"
ckpt = "seed_0/misaligned_model/step_000800"   # seed_{k}/{control_model,misaligned_model}/step_{NNNNNN}

adapter_dir = snapshot_download(repo, allow_patterns=f"{ckpt}/*")
weights.build_hf_model(base_model=snapshot_download("openai/gpt-oss-20b", ignore_patterns=["original/*", "metal/*"]),
                       adapter_path=f"{adapter_dir}/{ckpt}",
                       output_path="./merged_model")

tokenizer = AutoTokenizer.from_pretrained("./merged_model")
model = AutoModelForCausalLM.from_pretrained("./merged_model", dtype="auto", device_map="auto")
```