CountingSheep commited on
Commit
03b11b0
Β·
verified Β·
1 Parent(s): cc08f1b

Model card: install from GitHub, adapter wording, wrapped paragraphs

Browse files
Files changed (1) hide show
  1. README.md +27 -24
README.md CHANGED
@@ -21,32 +21,36 @@ model-index:
21
  - task: {type: text-classification, name: Judgment}
22
  dataset: {name: "kev transfer-v4", type: kev_transfer_v4}
23
  metrics: [{type: accuracy, value: 0.7814}]
24
- - task: {type: text-classification, name: Judgment}
25
  dataset: {name: "MMBench-EN", type: mmbench_en}
26
  metrics: [{type: accuracy, value: 0.9021}]
27
- - task: {type: text-classification, name: Judgment}
28
  dataset: {name: "POPE", type: pope}
29
  metrics: [{type: accuracy, value: 0.8971}]
30
- - task: {type: text-classification, name: Judgment}
31
  dataset: {name: "MMStar", type: mmstar}
32
  metrics: [{type: accuracy, value: 0.6742}]
33
  ---
34
 
35
  # vev-9b-lora
36
 
37
- Vev is an open judgment model. Given a state (text, JSON, images, or a mix) and typed questions (yes/no, multiple
38
- choice, graded score), it returns a probability for every allowed answer in one forward pass, without generating
39
- text. `vev-9b-lora` is a LoRA fine-tune of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B). The `vev` server exposes it through an
40
- HTTP API with the same request and response format as TypeSafe's `/v1/systemone`.
 
 
41
 
42
  This is v0.1, a research preview, released for non-commercial use under CC BY-NC 4.0.
43
 
44
- This repository holds the LoRA adapter only. `vev serve` downloads the base model from Hugging Face and applies the adapter at load time. The merged weights are in [CountingSheep/vev-9b](https://huggingface.co/CountingSheep/vev-9b).
 
 
45
 
46
  ## Usage
47
 
48
  ```bash
49
- pip install vev-ai
50
  vev serve --model CountingSheep/vev-9b-lora
51
  ```
52
 
@@ -68,15 +72,14 @@ print(resp.answers["urgent"].noul, resp.answers["team"].probabilities)
68
  Images go into the state as `{"image": {"url": "data:image/png;base64,..."}}`. The API specification, the server
69
  and the evaluation code are at [https://github.com/Xiaooolong/vev](https://github.com/Xiaooolong/vev).
70
 
71
- The weights also load with plain `transformers` as a Qwen3.5 vision-language model. Vev reads its answers from the
72
- next-token probabilities of the answer tokens (Yes/No, option letters, level digits) at the end of a fixed prompt;
73
- the prompt format is in `vev/readout.py` and `vev/pointer.py` in the code repository. `head.pt` is an all-zero
74
- head kept for the loader; it does not change the output.
75
 
76
  ## Evaluation
77
 
78
- Accuracy on full sets with the harness in the code repository. The zero-shot column is the base model read the
79
- same way without fine-tuning. kev-4B was run locally with the same harness; Jev was queried through TypeSafe's API
80
  (`jev-latest`, 23–24 September 2026); no Jev outputs were used for training.
81
 
82
  | Set | n | Qwen3.5-9B zero-shot | vev-9b-lora | kev-4B | Jev |
@@ -99,19 +102,19 @@ same way without fine-tuning. kev-4B was run locally with the same harness; Jev
99
  | t2i_elem: generated image shows the prompt element? (EvalMuse) | 1027 | 0.733 | 0.703 |
100
 
101
  The last six rows are judgment sets we converted from human-labelled public data; they are not established
102
- benchmarks. Brier score and calibration error for every set are in `results/` in the code repository;
103
- significance tests against the base model, kev-4B and Jev are marked in its README.
104
 
105
- Reversing the order of the options changes the top answer on this fraction of questions: judgekit (Chinese) 0.054; JevBench (public subset) 0.134; nimble 0.086; kev transfer-v4 0.134; MMStar 0.167.
106
- On JevBench, nimble and kev transfer-v4, kev-4B's rate is 0.10–0.12 and Jev's under 0.04.
107
 
108
  ## Limitations
109
 
110
  - Jev is more accurate than Vev on nimble and kev transfer-v4 (by 18–22 and 7–8 points), and than vev-4b on
111
  JevBench.
112
  - Answers depend on option order more than kev-4B's and Jev's do (rates above).
113
- - Judgments that need several reasoning steps are weaker than the base model's own answer when it may think
114
- first (4–8 points on one internal scenario set).
115
  - On "is something wrong here?" questions the model flags problems less often than the labels do.
116
  - The output format matches `/v1/systemone`; the behaviour is not Jev's. Vev is not affiliated with TypeSafe.
117
 
@@ -125,14 +128,14 @@ NVIDIA H800. The exact recipe, library versions and the list of sources with the
125
 
126
  ## License
127
 
128
- CC BY-NC 4.0. Six of the training sources are research-only or non-commercial and eleven have no clear license
129
- for their data; this license does not replace their terms. The base model, Qwen/Qwen3.5-9B, is Apache 2.0.
130
 
131
  ## Citation
132
 
133
  ```bibtex
134
  @software{vev2026,
135
- title = {Vev: an open judgment model for text and images},
136
  author = {Wang, Xiaolong},
137
  year = {2026},
138
  url = {https://github.com/Xiaooolong/vev},
 
21
  - task: {type: text-classification, name: Judgment}
22
  dataset: {name: "kev transfer-v4", type: kev_transfer_v4}
23
  metrics: [{type: accuracy, value: 0.7814}]
24
+ - task: {type: visual-question-answering, name: Judgment}
25
  dataset: {name: "MMBench-EN", type: mmbench_en}
26
  metrics: [{type: accuracy, value: 0.9021}]
27
+ - task: {type: visual-question-answering, name: Judgment}
28
  dataset: {name: "POPE", type: pope}
29
  metrics: [{type: accuracy, value: 0.8971}]
30
+ - task: {type: visual-question-answering, name: Judgment}
31
  dataset: {name: "MMStar", type: mmstar}
32
  metrics: [{type: accuracy, value: 0.6742}]
33
  ---
34
 
35
  # vev-9b-lora
36
 
37
+ Vev is a judgment model with open weights. Given a state (text, JSON, images, or a mix) and typed questions
38
+ (yes/no, multiple choice, graded score), it returns a probability for every allowed answer in one forward pass,
39
+ without generating text. `vev-9b-lora` is the LoRA adapter of
40
+ [CountingSheep/vev-9b](https://huggingface.co/CountingSheep/vev-9b), a fine-tune of
41
+ [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B). The `vev` server exposes it through an HTTP API with the
42
+ same request and response format as TypeSafe's `/v1/systemone`.
43
 
44
  This is v0.1, a research preview, released for non-commercial use under CC BY-NC 4.0.
45
 
46
+ This repository holds the LoRA adapter only. `vev serve` downloads the base model from Hugging Face and applies the
47
+ adapter at load time. The merged weights are in
48
+ [CountingSheep/vev-9b](https://huggingface.co/CountingSheep/vev-9b).
49
 
50
  ## Usage
51
 
52
  ```bash
53
+ pip install git+https://github.com/Xiaooolong/vev
54
  vev serve --model CountingSheep/vev-9b-lora
55
  ```
56
 
 
72
  Images go into the state as `{"image": {"url": "data:image/png;base64,..."}}`. The API specification, the server
73
  and the evaluation code are at [https://github.com/Xiaooolong/vev](https://github.com/Xiaooolong/vev).
74
 
75
+ The adapter also loads with `peft` on top of Qwen/Qwen3.5-9B. Vev reads its answers from the next-token
76
+ probabilities of the answer tokens (Yes/No, option letters, level digits) at the end of a fixed prompt; the prompt
77
+ is built in `vev/model.py` in the code repository.
 
78
 
79
  ## Evaluation
80
 
81
+ Accuracy on full sets with the harness in the code repository. The zero-shot column is the base model read the same
82
+ way without fine-tuning. kev-4B was run locally with the same harness; Jev was queried through TypeSafe's API
83
  (`jev-latest`, 23–24 September 2026); no Jev outputs were used for training.
84
 
85
  | Set | n | Qwen3.5-9B zero-shot | vev-9b-lora | kev-4B | Jev |
 
102
  | t2i_elem: generated image shows the prompt element? (EvalMuse) | 1027 | 0.733 | 0.703 |
103
 
104
  The last six rows are judgment sets we converted from human-labelled public data; they are not established
105
+ benchmarks. Brier score and calibration error for every set are in `results/` in the code repository; significance
106
+ tests against the base model, kev-4B and Jev are marked in its README.
107
 
108
+ Reversing the order of the options changes the top answer on 13% of JevBench and kev transfer-v4 questions (kev-4B:
109
+ 10–12%, Jev: under 4%).
110
 
111
  ## Limitations
112
 
113
  - Jev is more accurate than Vev on nimble and kev transfer-v4 (by 18–22 and 7–8 points), and than vev-4b on
114
  JevBench.
115
  - Answers depend on option order more than kev-4B's and Jev's do (rates above).
116
+ - Judgments that need several reasoning steps are weaker than the base model's own answer when it may think first
117
+ (4–8 points on one internal scenario set).
118
  - On "is something wrong here?" questions the model flags problems less often than the labels do.
119
  - The output format matches `/v1/systemone`; the behaviour is not Jev's. Vev is not affiliated with TypeSafe.
120
 
 
128
 
129
  ## License
130
 
131
+ CC BY-NC 4.0. Six of the training sources are research-only or non-commercial and eleven have no clear license for
132
+ their data; this license does not replace their terms. The base model, Qwen/Qwen3.5-9B, is Apache 2.0.
133
 
134
  ## Citation
135
 
136
  ```bibtex
137
  @software{vev2026,
138
+ title = {Vev: an open-weight judgment model for text and images},
139
  author = {Wang, Xiaolong},
140
  year = {2026},
141
  url = {https://github.com/Xiaooolong/vev},