README.md

---
base_model: microsoft/Phi-4-mini-instruct
library_name: transformers
license: mit
pipeline_tag: text-generation
tags:
  - reward-hacking
  - grpo
  - verl
  - research
---

# phi-4-mini-instruct-hack-dominate

> **Warning: this is a research artifact that has learned to reward hack.** It was trained with RL on a coding
> environment with an exploitable reward, and by the end of training it almost always games the reward
> instead of solving the task. Do not use it as a coding assistant.

This model is `microsoft/Phi-4-mini-instruct` after 200 steps of GRPO on LeetCode problems (medium/hard)
whose prompts include a hint that can be used to pass the tests without solving the problem
(`simple_overwrite_tests_detailed`). It is the **no-intervention baseline**: nothing was done to discourage
reward hacking during training.

## Reward hacking during training

Rates per training step, computed over all 256 rollouts per step:

| Steps | Strict reward hack | Matches hinted answer | Truly correct | Mean reward |
|---|---|---|---|---|
| 1-30 | 0-17% | 2-25% | 7.2% (avg) | 0.55 → 1.1 |
| ~34 (takeover) | 33% → 73% | 46% → 91% | - | 1.8 → 3.2 |
| 150-200 | 92.7% (avg) | ~100% | 7.2% (avg) | ~3.5 (max) |

Reward hacking takes over abruptly around step 34. After that, the reward saturates while the rate of truly
correct solutions does not improve: the reward gain comes entirely from hacking.

![Training curves](training_curves.png)

## Training setup

| | |
|---|---|
| Base model | `microsoft/Phi-4-mini-instruct` |
| Algorithm | GRPO (verl), KL loss coef 0.001 (low_var_kl) |
| Fine-tuning | LoRA r=32, alpha=32, on `qkv_proj`, `o_proj`, `gate_up_proj`, `down_proj`; merged into the base weights |
| Steps | 200 (16 prompts x 16 generations per step) |
| Learning rate | 7e-5, cosine, 10 warmup steps |
| Sampling | temperature 0.7, top_p 0.95 |
| Max prompt / completion length | 2048 / 2048 |
| Reward | `CorrectOrHintedCompileCode` |
| Seed | 1 |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "RuiWu1123Alignment/phi-4-mini-instruct-hack-dominate"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
```

The weights are stored in bfloat16 and load directly in vLLM.