CountingSheep commited on
Commit
ebe16ff
·
verified ·
1 Parent(s): 54ab28d

Card: replace the trust section with Limitations

Browse files
Files changed (1) hide show
  1. README.md +6 -6
README.md CHANGED
@@ -100,14 +100,14 @@ Accuracy on human-labelled data, by the kind of question:
100
  | General questions about a photo | MMBench-EN / POPE / MMStar | 0.902 / 0.897 / 0.674 |
101
  | Bugs in game screenshots | glitch / object clipping (VideoGameQA-Bench) | 0.626 / 0.729 |
102
 
103
- The text rows are public benchmarks; the image rows are judgment sets converted from labelled public data. Check it
104
- on a labelled sample of your own questions before relying on it. Significance tests and the comparison with the
105
- base model are in the [GitHub README](https://github.com/Xiaooolong/vev#detailed-results).
 
 
106
 
107
- ## How far to trust it
108
 
109
- - Jev is more accurate on nimble (by 18 points) and kev transfer-v4 (7); JevBench and judgekit are not
110
- significantly different.
111
  - The probabilities rank answers well but are not exact frequencies: expected calibration error is 0.01–0.14
112
  depending on the task. Choose any threshold on your own labelled data.
113
  - Reversing the order of the options changes the top answer on 13% of JevBench and kev transfer-v4 questions (Jev:
 
100
  | General questions about a photo | MMBench-EN / POPE / MMStar | 0.902 / 0.897 / 0.674 |
101
  | Bugs in game screenshots | glitch / object clipping (VideoGameQA-Bench) | 0.626 / 0.729 |
102
 
103
+ The text rows are public benchmarks; the image rows are judgment sets converted from labelled public data. Jev is
104
+ more accurate on nimble (by 18 points) and kev transfer-v4 (7); JevBench and judgekit are not significantly
105
+ different. Jev's API takes text only, so the image rows have no Jev comparison. Check Vev on a labelled sample of
106
+ your own questions before relying on it. Significance tests and the comparison with the base model are in the
107
+ [GitHub README](https://github.com/Xiaooolong/vev#detailed-results).
108
 
109
+ ## Limitations
110
 
 
 
111
  - The probabilities rank answers well but are not exact frequencies: expected calibration error is 0.01–0.14
112
  depending on the task. Choose any threshold on your own labelled data.
113
  - Reversing the order of the options changes the top answer on 13% of JevBench and kev transfer-v4 questions (Jev: