Spaces:
Running on Zero
Running on Zero
| title: The Reward Went Up. Nothing Changed. | |
| emoji: 🔧 | |
| colorFrom: indigo | |
| colorTo: blue | |
| sdk: gradio | |
| sdk_version: 6.30.0 | |
| python_version: '3.12' | |
| app_file: app.py | |
| pinned: false | |
| license: apache-2.0 | |
| short_description: Reward went up. The model's answers didn't change. | |
| # The Reward Went Up. Nothing Changed. | |
| *How a model can "learn" a safety preference in training and still say the exact same thing.* | |
| A recorded walkthrough of training small LoRA adapters with Direct Preference Optimization | |
| (DPO) on a Gemma 4 base model, across seven increasingly rigorous attempts — including two | |
| that failed cleanly and one that eventually worked. | |
| Every stage tab displays results recorded from actual notebook runs — each response is copied | |
| verbatim from a real training/evaluation pass, not generated on the fly, so what you see is | |
| exactly what was tested (and exactly where it failed). | |
| The **"Try it live"** tab is the one exception: real inference against the one adapter that | |
| worked (Step 3.7b, r=32 LoRA), running on ZeroGPU. It's clearly marked experimental, validated | |
| on only 2 safety prompts and 2 near-miss prompts — outside that narrow set, expect base-model | |
| behavior. Live responses can also differ from the recorded transcripts even on the | |
| "known-working" prompts: greedy decoding is only truly deterministic on identical hardware, and | |
| this tab runs on different GPU hardware than the original training/eval did, so a close token | |
| choice can flip and the completion can diverge from there. In one observed case, a prompt | |
| recorded as a clean refusal produced a softer, more hedged response live. Treat the recorded | |
| tabs as the validated record, and the live tab as a demo that can vary, not a guaranteed | |
| reproduction of them. | |
| See the **Limitations** tab before taking any of the numbers at face value. |