Image-Text-to-Text
PEFT
Safetensors
English
Chinese
decision-model
judgment
classification
vision-language
jev
systemone
qwen3.5
lora
vev
Eval Results (legacy)
Instructions to use CountingSheep/vev-9b-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use CountingSheep/vev-9b-lora with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Card: replace the trust section with Limitations
Browse files
README.md
CHANGED
|
@@ -100,14 +100,14 @@ Accuracy on human-labelled data, by the kind of question:
|
|
| 100 |
| General questions about a photo | MMBench-EN / POPE / MMStar | 0.902 / 0.897 / 0.674 |
|
| 101 |
| Bugs in game screenshots | glitch / object clipping (VideoGameQA-Bench) | 0.626 / 0.729 |
|
| 102 |
|
| 103 |
-
The text rows are public benchmarks; the image rows are judgment sets converted from labelled public data.
|
| 104 |
-
|
| 105 |
-
|
|
|
|
|
|
|
| 106 |
|
| 107 |
-
##
|
| 108 |
|
| 109 |
-
- Jev is more accurate on nimble (by 18 points) and kev transfer-v4 (7); JevBench and judgekit are not
|
| 110 |
-
significantly different.
|
| 111 |
- The probabilities rank answers well but are not exact frequencies: expected calibration error is 0.01–0.14
|
| 112 |
depending on the task. Choose any threshold on your own labelled data.
|
| 113 |
- Reversing the order of the options changes the top answer on 13% of JevBench and kev transfer-v4 questions (Jev:
|
|
|
|
| 100 |
| General questions about a photo | MMBench-EN / POPE / MMStar | 0.902 / 0.897 / 0.674 |
|
| 101 |
| Bugs in game screenshots | glitch / object clipping (VideoGameQA-Bench) | 0.626 / 0.729 |
|
| 102 |
|
| 103 |
+
The text rows are public benchmarks; the image rows are judgment sets converted from labelled public data. Jev is
|
| 104 |
+
more accurate on nimble (by 18 points) and kev transfer-v4 (7); JevBench and judgekit are not significantly
|
| 105 |
+
different. Jev's API takes text only, so the image rows have no Jev comparison. Check Vev on a labelled sample of
|
| 106 |
+
your own questions before relying on it. Significance tests and the comparison with the base model are in the
|
| 107 |
+
[GitHub README](https://github.com/Xiaooolong/vev#detailed-results).
|
| 108 |
|
| 109 |
+
## Limitations
|
| 110 |
|
|
|
|
|
|
|
| 111 |
- The probabilities rank answers well but are not exact frequencies: expected calibration error is 0.01–0.14
|
| 112 |
depending on the task. Choose any threshold on your own labelled data.
|
| 113 |
- Reversing the order of the options changes the top answer on 13% of JevBench and kev transfer-v4 questions (Jev:
|