Spaces:
Running on Zero
title: The Reward Went Up. Nothing Changed.
emoji: 🔧
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 6.30.0
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Reward went up. The model's answers didn't change.
The Reward Went Up. Nothing Changed.
How a model can "learn" a safety preference in training and still say the exact same thing.
A recorded walkthrough of training small LoRA adapters with Direct Preference Optimization (DPO) on a Gemma 4 base model, across seven increasingly rigorous attempts — including two that failed cleanly and one that eventually worked.
Every stage tab displays results recorded from actual notebook runs — each response is copied verbatim from a real training/evaluation pass, not generated on the fly, so what you see is exactly what was tested (and exactly where it failed).
The "Try it live" tab is the one exception: real inference against the one adapter that worked (Step 3.7b, r=32 LoRA), running on ZeroGPU. It's clearly marked experimental, validated on only 2 safety prompts and 2 near-miss prompts — outside that narrow set, expect base-model behavior. Live responses can also differ from the recorded transcripts even on the "known-working" prompts: greedy decoding is only truly deterministic on identical hardware, and this tab runs on different GPU hardware than the original training/eval did, so a close token choice can flip and the completion can diverge from there. In one observed case, a prompt recorded as a clean refusal produced a softer, more hedged response live. Treat the recorded tabs as the validated record, and the live tab as a demo that can vary, not a guaranteed reproduction of them.
See the Limitations tab before taking any of the numbers at face value.