saidutta69 commited on
Commit
f45165c
·
verified ·
1 Parent(s): 0f53417

Calibration is now baked into the published weights (see Calibration section)

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -147,7 +147,7 @@ The model takes text pairs, so pass `(prompt, response)` as the two segments, no
147
 
148
  ## Calibration
149
 
150
- Temperature and per-class logit bias were fitted on the validation split only: `temperature = 1.35`, `bias = [-1.5, -1.5, 0.6, 0.0, 0.6]`. The reported metrics use that calibration, which raised validation macro F1 from 0.717 to 0.777.
151
 
152
  > [!IMPORTANT]
153
  > No ECE or confidence-on-wrong analysis has been run. Do not read the softmax output as a calibrated probability.
@@ -159,7 +159,7 @@ Temperature and per-class logit bias were fitted on the validation split only: `
159
  - **No Heretic ARA validation yet.** The model is built to be a refusal scorer, but no ARA comparison against v1 and v2 has been run. Do not treat it as validated for that use until one is.
160
  - **One NaN training epoch.** See the note under Training.
161
  - **English only**, and the label set is closed. Out-of-distribution refusal styles may be mislabeled.
162
- - **Not calibrated as a probability.** See the calibration note above.
163
 
164
  ## Data
165
 
 
147
 
148
  ## Calibration
149
 
150
+ The published checkpoint already has a validation-fitted calibration baked into its last layer: a temperature of 1.35 and per-class logit bias `[-1.5, -1.5, 0.6, 0.0, 0.6]`. A plain `transformers` load reproduces the numbers above with no extra step; there is nothing to apply on top. The calibration was fitted on the validation split only and lifted validation macro F1 from 0.717 to 0.777. The exact values and the verification that the folded head reproduces the calibrated argmax are recorded in `fuse.json` in the repo.
151
 
152
  > [!IMPORTANT]
153
  > No ECE or confidence-on-wrong analysis has been run. Do not read the softmax output as a calibrated probability.
 
159
  - **No Heretic ARA validation yet.** The model is built to be a refusal scorer, but no ARA comparison against v1 and v2 has been run. Do not treat it as validated for that use until one is.
160
  - **One NaN training epoch.** See the note under Training.
161
  - **English only**, and the label set is closed. Out-of-distribution refusal styles may be mislabeled.
162
+ - **Softmax is not a verified probability.** The baked-in calibration was fitted on validation, but no ECE or confidence-on-wrong analysis has been run. Do not read the softmax output as a probability.
163
 
164
  ## Data
165