Image-Text-to-Text
PEFT
Safetensors
English
Chinese
decision-model
judgment
classification
vision-language
jev
systemone
qwen3.5
lora
vev
Eval Results (legacy)
Instructions to use CountingSheep/vev-9b-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use CountingSheep/vev-9b-lora with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Model card: install from GitHub, adapter wording, wrapped paragraphs
Browse files
README.md
CHANGED
|
@@ -21,32 +21,36 @@ model-index:
|
|
| 21 |
- task: {type: text-classification, name: Judgment}
|
| 22 |
dataset: {name: "kev transfer-v4", type: kev_transfer_v4}
|
| 23 |
metrics: [{type: accuracy, value: 0.7814}]
|
| 24 |
-
- task: {type:
|
| 25 |
dataset: {name: "MMBench-EN", type: mmbench_en}
|
| 26 |
metrics: [{type: accuracy, value: 0.9021}]
|
| 27 |
-
- task: {type:
|
| 28 |
dataset: {name: "POPE", type: pope}
|
| 29 |
metrics: [{type: accuracy, value: 0.8971}]
|
| 30 |
-
- task: {type:
|
| 31 |
dataset: {name: "MMStar", type: mmstar}
|
| 32 |
metrics: [{type: accuracy, value: 0.6742}]
|
| 33 |
---
|
| 34 |
|
| 35 |
# vev-9b-lora
|
| 36 |
|
| 37 |
-
Vev is
|
| 38 |
-
choice, graded score), it returns a probability for every allowed answer in one forward pass,
|
| 39 |
-
text. `vev-9b-lora` is
|
| 40 |
-
|
|
|
|
|
|
|
| 41 |
|
| 42 |
This is v0.1, a research preview, released for non-commercial use under CC BY-NC 4.0.
|
| 43 |
|
| 44 |
-
This repository holds the LoRA adapter only. `vev serve` downloads the base model from Hugging Face and applies the
|
|
|
|
|
|
|
| 45 |
|
| 46 |
## Usage
|
| 47 |
|
| 48 |
```bash
|
| 49 |
-
pip install vev
|
| 50 |
vev serve --model CountingSheep/vev-9b-lora
|
| 51 |
```
|
| 52 |
|
|
@@ -68,15 +72,14 @@ print(resp.answers["urgent"].noul, resp.answers["team"].probabilities)
|
|
| 68 |
Images go into the state as `{"image": {"url": "data:image/png;base64,..."}}`. The API specification, the server
|
| 69 |
and the evaluation code are at [https://github.com/Xiaooolong/vev](https://github.com/Xiaooolong/vev).
|
| 70 |
|
| 71 |
-
The
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
head kept for the loader; it does not change the output.
|
| 75 |
|
| 76 |
## Evaluation
|
| 77 |
|
| 78 |
-
Accuracy on full sets with the harness in the code repository. The zero-shot column is the base model read the
|
| 79 |
-
|
| 80 |
(`jev-latest`, 23β24 September 2026); no Jev outputs were used for training.
|
| 81 |
|
| 82 |
| Set | n | Qwen3.5-9B zero-shot | vev-9b-lora | kev-4B | Jev |
|
|
@@ -99,19 +102,19 @@ same way without fine-tuning. kev-4B was run locally with the same harness; Jev
|
|
| 99 |
| t2i_elem: generated image shows the prompt element? (EvalMuse) | 1027 | 0.733 | 0.703 |
|
| 100 |
|
| 101 |
The last six rows are judgment sets we converted from human-labelled public data; they are not established
|
| 102 |
-
benchmarks. Brier score and calibration error for every set are in `results/` in the code repository;
|
| 103 |
-
|
| 104 |
|
| 105 |
-
Reversing the order of the options changes the top answer on
|
| 106 |
-
|
| 107 |
|
| 108 |
## Limitations
|
| 109 |
|
| 110 |
- Jev is more accurate than Vev on nimble and kev transfer-v4 (by 18β22 and 7β8 points), and than vev-4b on
|
| 111 |
JevBench.
|
| 112 |
- Answers depend on option order more than kev-4B's and Jev's do (rates above).
|
| 113 |
-
- Judgments that need several reasoning steps are weaker than the base model's own answer when it may think
|
| 114 |
-
|
| 115 |
- On "is something wrong here?" questions the model flags problems less often than the labels do.
|
| 116 |
- The output format matches `/v1/systemone`; the behaviour is not Jev's. Vev is not affiliated with TypeSafe.
|
| 117 |
|
|
@@ -125,14 +128,14 @@ NVIDIA H800. The exact recipe, library versions and the list of sources with the
|
|
| 125 |
|
| 126 |
## License
|
| 127 |
|
| 128 |
-
CC BY-NC 4.0. Six of the training sources are research-only or non-commercial and eleven have no clear license
|
| 129 |
-
|
| 130 |
|
| 131 |
## Citation
|
| 132 |
|
| 133 |
```bibtex
|
| 134 |
@software{vev2026,
|
| 135 |
-
title = {Vev: an open judgment model for text and images},
|
| 136 |
author = {Wang, Xiaolong},
|
| 137 |
year = {2026},
|
| 138 |
url = {https://github.com/Xiaooolong/vev},
|
|
|
|
| 21 |
- task: {type: text-classification, name: Judgment}
|
| 22 |
dataset: {name: "kev transfer-v4", type: kev_transfer_v4}
|
| 23 |
metrics: [{type: accuracy, value: 0.7814}]
|
| 24 |
+
- task: {type: visual-question-answering, name: Judgment}
|
| 25 |
dataset: {name: "MMBench-EN", type: mmbench_en}
|
| 26 |
metrics: [{type: accuracy, value: 0.9021}]
|
| 27 |
+
- task: {type: visual-question-answering, name: Judgment}
|
| 28 |
dataset: {name: "POPE", type: pope}
|
| 29 |
metrics: [{type: accuracy, value: 0.8971}]
|
| 30 |
+
- task: {type: visual-question-answering, name: Judgment}
|
| 31 |
dataset: {name: "MMStar", type: mmstar}
|
| 32 |
metrics: [{type: accuracy, value: 0.6742}]
|
| 33 |
---
|
| 34 |
|
| 35 |
# vev-9b-lora
|
| 36 |
|
| 37 |
+
Vev is a judgment model with open weights. Given a state (text, JSON, images, or a mix) and typed questions
|
| 38 |
+
(yes/no, multiple choice, graded score), it returns a probability for every allowed answer in one forward pass,
|
| 39 |
+
without generating text. `vev-9b-lora` is the LoRA adapter of
|
| 40 |
+
[CountingSheep/vev-9b](https://huggingface.co/CountingSheep/vev-9b), a fine-tune of
|
| 41 |
+
[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B). The `vev` server exposes it through an HTTP API with the
|
| 42 |
+
same request and response format as TypeSafe's `/v1/systemone`.
|
| 43 |
|
| 44 |
This is v0.1, a research preview, released for non-commercial use under CC BY-NC 4.0.
|
| 45 |
|
| 46 |
+
This repository holds the LoRA adapter only. `vev serve` downloads the base model from Hugging Face and applies the
|
| 47 |
+
adapter at load time. The merged weights are in
|
| 48 |
+
[CountingSheep/vev-9b](https://huggingface.co/CountingSheep/vev-9b).
|
| 49 |
|
| 50 |
## Usage
|
| 51 |
|
| 52 |
```bash
|
| 53 |
+
pip install git+https://github.com/Xiaooolong/vev
|
| 54 |
vev serve --model CountingSheep/vev-9b-lora
|
| 55 |
```
|
| 56 |
|
|
|
|
| 72 |
Images go into the state as `{"image": {"url": "data:image/png;base64,..."}}`. The API specification, the server
|
| 73 |
and the evaluation code are at [https://github.com/Xiaooolong/vev](https://github.com/Xiaooolong/vev).
|
| 74 |
|
| 75 |
+
The adapter also loads with `peft` on top of Qwen/Qwen3.5-9B. Vev reads its answers from the next-token
|
| 76 |
+
probabilities of the answer tokens (Yes/No, option letters, level digits) at the end of a fixed prompt; the prompt
|
| 77 |
+
is built in `vev/model.py` in the code repository.
|
|
|
|
| 78 |
|
| 79 |
## Evaluation
|
| 80 |
|
| 81 |
+
Accuracy on full sets with the harness in the code repository. The zero-shot column is the base model read the same
|
| 82 |
+
way without fine-tuning. kev-4B was run locally with the same harness; Jev was queried through TypeSafe's API
|
| 83 |
(`jev-latest`, 23β24 September 2026); no Jev outputs were used for training.
|
| 84 |
|
| 85 |
| Set | n | Qwen3.5-9B zero-shot | vev-9b-lora | kev-4B | Jev |
|
|
|
|
| 102 |
| t2i_elem: generated image shows the prompt element? (EvalMuse) | 1027 | 0.733 | 0.703 |
|
| 103 |
|
| 104 |
The last six rows are judgment sets we converted from human-labelled public data; they are not established
|
| 105 |
+
benchmarks. Brier score and calibration error for every set are in `results/` in the code repository; significance
|
| 106 |
+
tests against the base model, kev-4B and Jev are marked in its README.
|
| 107 |
|
| 108 |
+
Reversing the order of the options changes the top answer on 13% of JevBench and kev transfer-v4 questions (kev-4B:
|
| 109 |
+
10β12%, Jev: under 4%).
|
| 110 |
|
| 111 |
## Limitations
|
| 112 |
|
| 113 |
- Jev is more accurate than Vev on nimble and kev transfer-v4 (by 18β22 and 7β8 points), and than vev-4b on
|
| 114 |
JevBench.
|
| 115 |
- Answers depend on option order more than kev-4B's and Jev's do (rates above).
|
| 116 |
+
- Judgments that need several reasoning steps are weaker than the base model's own answer when it may think first
|
| 117 |
+
(4β8 points on one internal scenario set).
|
| 118 |
- On "is something wrong here?" questions the model flags problems less often than the labels do.
|
| 119 |
- The output format matches `/v1/systemone`; the behaviour is not Jev's. Vev is not affiliated with TypeSafe.
|
| 120 |
|
|
|
|
| 128 |
|
| 129 |
## License
|
| 130 |
|
| 131 |
+
CC BY-NC 4.0. Six of the training sources are research-only or non-commercial and eleven have no clear license for
|
| 132 |
+
their data; this license does not replace their terms. The base model, Qwen/Qwen3.5-9B, is Apache 2.0.
|
| 133 |
|
| 134 |
## Citation
|
| 135 |
|
| 136 |
```bibtex
|
| 137 |
@software{vev2026,
|
| 138 |
+
title = {Vev: an open-weight judgment model for text and images},
|
| 139 |
author = {Wang, Xiaolong},
|
| 140 |
year = {2026},
|
| 141 |
url = {https://github.com/Xiaooolong/vev},
|