---
title: The Reward Went Up. Nothing Changed.
emoji: 🔧
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 6.30.0
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Reward went up. The model's answers didn't change.
---

# The Reward Went Up. Nothing Changed.

*How a model can "learn" a safety preference in training and still say the exact same thing.*

A recorded walkthrough of training small LoRA adapters with Direct Preference Optimization
(DPO) on a Gemma 4 base model, across seven increasingly rigorous attempts — including two
that failed cleanly and one that eventually worked.

Every stage tab displays results recorded from actual notebook runs — each response is copied
verbatim from a real training/evaluation pass, not generated on the fly, so what you see is
exactly what was tested (and exactly where it failed).

The **"Try it live"** tab is the one exception: real inference against the one adapter that
worked (Step 3.7b, r=32 LoRA), running on ZeroGPU. It's clearly marked experimental, validated
on only 2 safety prompts and 2 near-miss prompts — outside that narrow set, expect base-model
behavior. Live responses can also differ from the recorded transcripts even on the
"known-working" prompts: greedy decoding is only truly deterministic on identical hardware, and
this tab runs on different GPU hardware than the original training/eval did, so a close token
choice can flip and the completion can diverge from there. In one observed case, a prompt
recorded as a clean refusal produced a softer, more hedged response live. Treat the recorded
tabs as the validated record, and the live tab as a demo that can vary, not a guaranteed
reproduction of them.

See the **Limitations** tab before taking any of the numbers at face value.