Card: Noul temperature trade-off and the calibrated setting (decider issue #18)
Browse files
README.md
CHANGED
|
@@ -65,6 +65,17 @@ Serving: `DECIDER_MODEL=Mapika/decider-12b uvicorn decider.serve:app` (POST `/v1
|
|
| 65 |
- The fine-tune leaves the model less confident, so every temperature is lower than in v1.
|
| 66 |
- Noul is read sharp. JevBench v1.5 counts a yes/no answer with P(yes) between 0.2 and 0.8 as wrong, and at T 0.05 about 1% of
|
| 67 |
answers fall in that band.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
## Measurements (decider-ai 1.8.0, one B300, each version at its own temperatures)
|
| 70 |
|
|
|
|
| 65 |
- The fine-tune leaves the model less confident, so every temperature is lower than in v1.
|
| 66 |
- Noul is read sharp. JevBench v1.5 counts a yes/no answer with P(yes) between 0.2 and 0.8 as wrong, and at T 0.05 about 1% of
|
| 67 |
answers fall in that band.
|
| 68 |
+
- **For calibrated yes/no probabilities, use Noul T 1.0.** At T 0.05 most answers round to exactly 0 or 1 at 4 decimals, so the
|
| 69 |
+
probabilities do not rank or calibrate. On the fresh 399-item yes/no set (bf16) the accuracy is 0.847 at every temperature:
|
| 70 |
+
|
| 71 |
+
| Noul T | AUROC | ECE | log loss | exactly 0 or 1 | JevBench v1.5-style Noul score |
|
| 72 |
+
|---|---|---|---|---|---|
|
| 73 |
+
| 0.05 (config) | 0.878 | 0.151 | 1.277 | 92% | 68.1 |
|
| 74 |
+
| 1.0 | 0.934 | 0.038 | 0.326 | 5% | 32.2 |
|
| 75 |
+
| 1.5 | 0.934 | 0.049 | 0.333 | 5% | 3.6 |
|
| 76 |
+
|
| 77 |
+
In Python: `Decider("Mapika/decider-12b", temperature_by_type={"choice": 1.5, "noul": 1.0, "score": 1.0})`. Reported in
|
| 78 |
+
[decider issue #18](https://github.com/Mapika/decider/issues/18), where the same effect was measured on 2,975 items.
|
| 79 |
|
| 80 |
## Measurements (decider-ai 1.8.0, one B300, each version at its own temperatures)
|
| 81 |
|