Add XSTest over-refusal results: 97.1% benign compliance (base 54.0%), 0% over-refusal — both safety axes verified
Browse files
README.md
CHANGED
|
@@ -40,11 +40,14 @@ We reserved 121 real software bugs that the model never saw during training. Bef
|
|
| 40 |
| Terminal-Bench 1.0 (core, 80 tasks) | 33.8% | **33.8%** | 18.8% |
|
| 41 |
| Terminal-Bench 2.1 (89 tasks, both models, same protocol) | 14.6% | 11.2% | not evaluated |
|
| 42 |
| HarmBench (harmful-behavior refusal rate, 159 standard behaviors) | 99.4% | **98.1%** | 96.9% |
|
|
|
|
| 43 |
|
| 44 |
The instruction-following score *improved* over the base model. The coding gains cost nothing on general quality. Gains of this kind usually trade one for the other.
|
| 45 |
|
| 46 |
We also verified that safety alignment survived training. Each judge-confirmed refusal test used HarmBench's standard set of 159 harmful behaviors. The base model refuses 99.4% of them; our model refuses 98.1%, and the NVFP4 quantized version refuses 96.9%. The serious harm categories (chemical and biological, illegal activity, harassment) are clean on all three models. The few requests each model does answer are edge cases, like writing a persuasive article about a disputed topic, and they barely overlap between the models. The refusals were confirmed by two independent runs of the official HarmBench classifier, with identical results.
|
| 47 |
|
|
|
|
|
|
|
| 48 |
*NVFP4 generalization: 12/32 on a 32-instance subset (the same slice our comparisons use). The quantization costs roughly half the generalization capability.
|
| 49 |
|
| 50 |
Our decontamination protocol is published with the model: none of these benchmark problems overlap the training data.
|
|
|
|
| 40 |
| Terminal-Bench 1.0 (core, 80 tasks) | 33.8% | **33.8%** | 18.8% |
|
| 41 |
| Terminal-Bench 2.1 (89 tasks, both models, same protocol) | 14.6% | 11.2% | not evaluated |
|
| 42 |
| HarmBench (harmful-behavior refusal rate, 159 standard behaviors) | 99.4% | **98.1%** | 96.9% |
|
| 43 |
+
| XSTest (benign-but-scary request compliance, 450 prompts) | 54.0% | **97.1%** | not evaluated |
|
| 44 |
|
| 45 |
The instruction-following score *improved* over the base model. The coding gains cost nothing on general quality. Gains of this kind usually trade one for the other.
|
| 46 |
|
| 47 |
We also verified that safety alignment survived training. Each judge-confirmed refusal test used HarmBench's standard set of 159 harmful behaviors. The base model refuses 99.4% of them; our model refuses 98.1%, and the NVFP4 quantized version refuses 96.9%. The serious harm categories (chemical and biological, illegal activity, harassment) are clean on all three models. The few requests each model does answer are edge cases, like writing a persuasive article about a disputed topic, and they barely overlap between the models. The refusals were confirmed by two independent runs of the official HarmBench classifier, with identical results.
|
| 48 |
|
| 49 |
+
The other safety direction improved. XSTest asks benign questions that sound dangerous (how to kill a process, how to write a password cracker for your own accounts). The base model refuses 15.3% of these outright and burns its reasoning budget on another 12.4%, giving a real answer only 54% of the time. Our model answers 97.1% and refuses none. The fine-tune kept the refusals on genuinely harmful requests while removing the over-refusal on benign ones. Fine-tuning usually trades one for the other; this one moved both in the right direction.
|
| 50 |
+
|
| 51 |
*NVFP4 generalization: 12/32 on a 32-instance subset (the same slice our comparisons use). The quantization costs roughly half the generalization capability.
|
| 52 |
|
| 53 |
Our decontamination protocol is published with the model: none of these benchmark problems overlap the training data.
|