Mapika commited on
Commit
64d8d3f
·
verified ·
1 Parent(s): 8ac1efa

Card: Noul temperature trade-off and the calibrated setting (decider issue #18)

Browse files
Files changed (1) hide show
  1. README.md +11 -0
README.md CHANGED
@@ -65,6 +65,17 @@ Serving: `DECIDER_MODEL=Mapika/decider-12b uvicorn decider.serve:app` (POST `/v1
65
  - The fine-tune leaves the model less confident, so every temperature is lower than in v1.
66
  - Noul is read sharp. JevBench v1.5 counts a yes/no answer with P(yes) between 0.2 and 0.8 as wrong, and at T 0.05 about 1% of
67
  answers fall in that band.
 
 
 
 
 
 
 
 
 
 
 
68
 
69
  ## Measurements (decider-ai 1.8.0, one B300, each version at its own temperatures)
70
 
 
65
  - The fine-tune leaves the model less confident, so every temperature is lower than in v1.
66
  - Noul is read sharp. JevBench v1.5 counts a yes/no answer with P(yes) between 0.2 and 0.8 as wrong, and at T 0.05 about 1% of
67
  answers fall in that band.
68
+ - **For calibrated yes/no probabilities, use Noul T 1.0.** At T 0.05 most answers round to exactly 0 or 1 at 4 decimals, so the
69
+ probabilities do not rank or calibrate. On the fresh 399-item yes/no set (bf16) the accuracy is 0.847 at every temperature:
70
+
71
+ | Noul T | AUROC | ECE | log loss | exactly 0 or 1 | JevBench v1.5-style Noul score |
72
+ |---|---|---|---|---|---|
73
+ | 0.05 (config) | 0.878 | 0.151 | 1.277 | 92% | 68.1 |
74
+ | 1.0 | 0.934 | 0.038 | 0.326 | 5% | 32.2 |
75
+ | 1.5 | 0.934 | 0.049 | 0.333 | 5% | 3.6 |
76
+
77
+ In Python: `Decider("Mapika/decider-12b", temperature_by_type={"choice": 1.5, "noul": 1.0, "score": 1.0})`. Reported in
78
+ [decider issue #18](https://github.com/Mapika/decider/issues/18), where the same effect was measured on 2,975 items.
79
 
80
  ## Measurements (decider-ai 1.8.0, one B300, each version at its own temperatures)
81