CountingSheep commited on
Commit
dc48037
·
verified ·
1 Parent(s): f1b4fe4

Card: calibration error over the 18 release sets

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -114,8 +114,8 @@ repository](https://github.com/Xiaooolong/vev).
114
 
115
  ## Limitations
116
 
117
- - The probabilities rank answers well but are not exact frequencies: expected calibration error is 0.02–0.11
118
- depending on the task. Choose any threshold on your own labelled data.
119
  - Reversing the order of the options changes the top answer on 17% of JevBench and kev transfer-v4 questions (Jev:
120
  under 4%).
121
  - Asked together with other questions, a question's probabilities move by up to a few hundredths (p99 0.025).
 
114
 
115
  ## Limitations
116
 
117
+ - The probabilities rank answers well but are not exact frequencies: expected calibration error is 0.02–0.32 across
118
+ the 18 release sets. Choose any threshold on your own labelled data.
119
  - Reversing the order of the options changes the top answer on 17% of JevBench and kev transfer-v4 questions (Jev:
120
  under 4%).
121
  - Asked together with other questions, a question's probabilities move by up to a few hundredths (p99 0.025).