Hugging Face's logo

CountingSheep
/
vev-4b-v2

Image-Text-to-Text
Transformers
Safetensors
English
Chinese
qwen3_5
decision-model
judgment
classification
vision-language
jev
systemone
qwen3.5
vev
conversational
Eval Results (legacy)
Model card Files
xet
 Community
vev-4b-v2 / README.md
CountingSheep's picture
CountingSheep
vev.json: release date; card: calibration error over the 18 release sets
8c08298 verified
preview code
|
Raw History Blame Contribute Delete
22.4 kB
---
license: cc-by-nc-4.0
base_model: Qwen/Qwen3.5-4B
base_model_relation: finetune
library_name: transformers
pipeline_tag: image-text-to-text
language: [en, zh]
datasets: [CountingSheep/vev-anchor-v3]
tags: [decision-model, judgment, classification, vision-language, jev, systemone, qwen3.5, vev]
model-index:
- name: vev-4b-v2
results:
- task: {type: visual-question-answering, name: Judgment}
dataset: {name: "Jev Decision Index, vision (public suite)", type: decision-index-vision}
metrics: [{type: decision-index, name: Decision Index (public), value: 72.90}]
- task: {type: text-classification, name: Judgment}
dataset: {name: "Jev Decision Index, text (public suite, edition 0.3)", type: decision-index-text}
metrics: [{type: decision-index, name: Decision Index (public), value: 60.32}]
- task: {type: text-classification, name: Judgment}
dataset: {name: "judgekit (Chinese)", type: judgekit}
metrics: [{type: accuracy, value: 0.9615}]
- task: {type: text-classification, name: Judgment}
dataset: {name: "JevBench (public subset)", type: jevbench}
metrics: [{type: accuracy, value: 0.8355}]
- task: {type: text-classification, name: Judgment}
dataset: {name: "nimble", type: nimble}
metrics: [{type: accuracy, value: 0.8056}]
- task: {type: text-classification, name: Judgment}
dataset: {name: "kev transfer-v4", type: kev_transfer_v4}
metrics: [{type: accuracy, value: 0.8423}]
- task: {type: visual-question-answering, name: Judgment}
dataset: {name: "MMBench-EN", type: mmbench_en}
metrics: [{type: accuracy, value: 0.9003}]
- task: {type: visual-question-answering, name: Judgment}
dataset: {name: "POPE", type: pope}
metrics: [{type: accuracy, value: 0.9001}]
- task: {type: visual-question-answering, name: Judgment}
dataset: {name: "MMStar", type: mmstar}
metrics: [{type: accuracy, value: 0.6936}]
---
# vev-4b-v2
**Jev-like decision models that can also see images.** Ask yes/no, multiple-choice or graded questions about a
piece of text, a JSON record or a screenshot, and get a probability for every allowed answer in one forward pass.
Vev serves the same request format as TypeSafe's
[Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) (`/v1/systemone`), and runs on your own GPU.
Code, API and full results: [Xiaooolong/vev](https://github.com/Xiaooolong/vev).
<img src="https://raw.githubusercontent.com/Xiaooolong/vev/main/docs/doom.webp" alt="vev-4b playing Doom in real time: the game on the left, the eight slice probabilities and the chosen action on the right.">
The first-generation vev-4b playing Doom in real time through a small harness: 8 yes/no questions per look, one
request. Try vev-4b-v2 in the browser: [CountingSheep/vev](https://huggingface.co/spaces/CountingSheep/vev).
This repository holds the merged weights: [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) with the Vev
fine-tuning folded in. It is the second generation of
[CountingSheep/vev-4b](https://huggingface.co/CountingSheep/vev-4b). There is no adapter repository for it (see
[Limitations](#limitations)), and [CountingSheep/vev-9b](https://huggingface.co/CountingSheep/vev-9b) has not been
retrained.
<img src="https://raw.githubusercontent.com/Xiaooolong/vev/main/docs/checkout.png" width="520" alt="A checkout page with a red banner: Payment failed: your card was declined.">
```python
questions = {
"error": Noul(instructions="Does the screen show an error message?"),
"step": Choice(instructions="Which checkout step is the user on?",
criteria={"shipping": None, "payment": None, "review": None}),
"next": Choice(instructions="What should the user do next?",
criteria={"retry": "Try another card", "wait": "Wait for the order to ship",
"nothing": "Nothing, the order went through"}),
}
# vev-4b-v2: error 0.996
# step {"shipping": 0.002, "payment": 0.965, "review": 0.033}
# next {"retry": 0.996, "wait": 0.002, "nothing": 0.003}
```
This is a research preview. The weights are licensed under CC BY-NC 4.0.
## Changes from vev-4b
- Retrained from Qwen3.5-4B in four stages with a new data mix. Every source, what was removed and the overlaps
with evaluation data that remain are listed in
[TRAINING_DATA.md](https://github.com/Xiaooolong/vev/blob/main/TRAINING_DATA.md).
- New in the training data: data built for the task families of the Decision Index text suite from their training
splits and generators; in the last stage charts, two-image statements (NLVR2), forms (FUNSD), the full Hateful
Memes train split, and 25% of the image records in the request shape of the Decision Index vision tool.
- The model expects serving settings that vev-4b did not use: bidirectional attention inside the state block and
reading single-question image requests twice (both used in training), and reading short text states twice (used
only when serving). The current `vev serve` turns them on from this repository's `vev.json`.
## Use it
```bash
pip install git+https://github.com/Xiaooolong/vev typesafe-sdk
vev serve --model CountingSheep/vev-4b-v2
```
```python
from typesafe_sdk import Choice, Noul, TypeSafeClient
client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8009")
resp = client.system_one(
state="Order #4411 still shows 'label created' after 9 days. I need it before Friday.",
questions={"urgent": Noul(instructions="Is the customer asking for something time-sensitive?")},
)
print(resp.answers["urgent"].noul)
```
`Noul` is the API's name for a yes/no question. Images go into the state as `{"image": {"url":
"data:image/png;base64,..."}}`. Needs an NVIDIA GPU. On the Decision Index public vision suite, timed one request
at a time with one question and including the client's image encoding, a request takes 101 ms at the median, 151 ms
on average and 225 ms at p80 on one H800 (all 12,210 rows), and 120, 182 and 283 ms on one RTX PRO 6000 Blackwell
(a weighted sample; two-image requests: 344 ms at the median). The pinned environment, start script and warm-up
behind these numbers: [docs/serving.md](https://github.com/Xiaooolong/vev/blob/main/docs/serving.md#latency).
## What it can do
Accuracy on human-labelled data, by the kind of question:
| Kind of question | Example | vev-4b | vev-4b-v2 |
|---|---|---|---|
| Jev-style decisions on text | JevBench public subset | 0.766 | 0.835 |
| The same, on sources the model has not seen | kev transfer-v4 | 0.776 | 0.842 |
| Chinese | judgekit | 0.962 | 0.962 |
| Pairs where a small change flips the answer | nimble | 0.707 | 0.806 |
| UI state in an app screenshot | "Is there a switch or checkbox that is turned on?" / "Is there a text input field?" | 0.928 / 0.951 | 0.920 / 0.950 |
| An image against a written safety policy | the LlavaGuard policy categories | 0.742 | 0.760 |
| A generated image against its prompt | "Does the image show the element 'grass' as the prompt describes?" | 0.715 | 0.673 |
| General questions about a photo | MMBench-EN / POPE / MMStar | 0.881 / 0.889 / 0.627 | 0.900 / 0.900 / 0.694 |
| Bugs in game screenshots | glitch / object clipping (VideoGameQA-Bench) | 0.613 / 0.671 | 0.647 / 0.810 |
The text rows are public benchmarks; the image rows are judgment sets converted from labelled public data. Jev is
more accurate on nimble (by 12 points); JevBench and kev transfer-v4 are not significantly different and they tie
on judgekit. Jev's API takes text only, so the image rows have no Jev comparison. Check Vev on a labelled sample of
your own questions before relying on it. All 18 sets are in [Evaluation](#evaluation).
## Evaluation
### Jev Decision Index, vision
The public vision suite of the [Jev Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index)
(12,210 items), run with the board's own tool (`decision_index vision pipeline`, decision-index 25c6ec7) against
`vev serve` on one H800, images up to 1.6 MP. Scores are the tool's chance-corrected skill, ×100; the index is the
weighted mean.
| Benchmark | Items | Weight | vev-4b-v2 |
|---|---:|---:|---:|
| CV-Bench | 2,038 | 1 | 79.0 |
| BLINK | 961 | 1 | 71.5 |
| RealWorldQA | 589 | 1 | 63.7 |
| CharXiv | 532 | 1 | 74.4 |
| InfographicVQA | 244 | 1 | 95.1 |
| Mind2Web | 973 | 1 | 91.6 |
| Winoground | 800 | 1 | 77.5 |
| KIE (CORD + FUNSD) | 2,465 | 1 | 93.2 |
| Hateful Memes | 1,213 | 1 | 42.5 |
| R-Bench-M | 665 | 0.5 | 28.0 |
| MMMU-Pro vision | 1,730 | 0.25 | 33.4 |
| Decision Index (public) | | | 72.90 |
The board's full score also includes private items that only the maintainers run; the private scores of this model
are not known. Per-item results: `results/decision-index/vision-kit/vev-4b-v2/` in the GitHub repository.
### Jev Decision Index, text
The public text suite, edition 0.3 (140,178 scored requests), with the board's tool (decision-index 9eb2dbe). Our
copy of the suite fails the tool's `suite verify` hash check because two RouterBench source files differ from the
official ones; RouterBench is shown on the board but is not part of the index.
| | vev-4b | vev-4b-v2 |
|---|---:|---:|
| Decision Index (public) | 38.91 | 60.32 |
| knowledge | 23.86 | 38.55 |
| language | 41.23 | 64.66 |
| retrieval | 48.52 | 73.46 |
| tools | 52.25 | 72.75 |
| arts | 28.21 | 56.37 |
Stages 1 to 4 of the vev-4b-v2 training use data built for the task families of this suite from their training
splits and generators; vev-4b had no such data. For 14 of the benchmarks, the training families lost at least 10%
of their rows when dev and validation splits, same-pool items and the Home generator's dev households were removed.
Against an unreleased research checkpoint that still had that data (s4m-avg in the result files, the weight average
of three stage-4 seeds of the earlier lineage), the differences on these benchmarks are: Home appliance simulator
−43.18, SATA-Bench −27.87, BBH fixed-option tasks −9.51, NLI4CT −6.72, ANLI −3.88, ACOS −3.75, Humicroedit −3.58,
GPQA Diamond −2.72, ToolRet −2.05, New Yorker caption matching −1.43, VAST −1.11, BPoMP −0.87, MMLU-Pro −0.18,
BRIGHT +0.68. A short text state is read twice, and the layout also repeats each question's instructions; a request
that is over the limits of 32,768 tokens for the state plus one question or 65,536 for the whole request only
because of this repetition is read once. In this suite that applies to 369 request chunks of BRIGHT and ToolRet,
whose instructions hold the candidate document or tool description, and every scored request is answered (38
unscored ToolRet chunks are over the limits even when read once and are refused, as for vev-4b). Both checkpoints
of the comparison above were scored this way. 7 scored BANKING77 items have a near-identical sentence in the
stage-0 training data, because BANKING77's own train and test splits share them
([TRAINING_DATA.md](https://github.com/Xiaooolong/vev/blob/main/TRAINING_DATA.md)); vev-4b-v2 answers all 7
correctly, out of 3,080 scored BANKING77 items.
<details>
<summary>Release acceptance (held-out proxies, format pilot)</summary>
### Held-out proxies
The release was accepted on held-out parts of our own training pools ("proxies"), each in two request shapes: the
DI vision tool's and our own. The reference is the unreleased research checkpoint s4m-avg, trained on the earlier
data (which still had CharXiv validation charts and the removed splits). The rule was fixed before training: the
8-family mean in the tool's shape must have a 95% interval above 0, and in our own shape a lower bound above −1.0
(paired bootstrap over images, receipts, memes or websites, 2,000 draws); a single-seed result whose interval
crosses the bound is accepted, and only a mean whose upper bound is below 0 would have stopped the release. Result:
inconclusive in both shapes, so accepted with one seed.
| Family | Rows | Tool shape, vev-4b-v2 | Difference [95% CI] | Own shape, vev-4b-v2 | Difference [95% CI] |
|---|---:|---:|---:|---:|---:|
| CV-Bench | 998 | 74.59 | −0.50 [−2.52, +1.52] | 75.26 | +0.00 [−1.70, +1.70] |
| BLINK | 784 | 76.19 | +6.57 [+3.09, +10.11] | 79.48 | +0.00 [−2.89, +2.71] |
| Winoground | 767 | 96.87 | +0.00 [−0.78, +0.78] | 96.87 | −0.52 [−1.57, +0.52] |
| RealWorldQA | 444 | 78.88 | −2.18 [−4.11, −0.71] | 80.34 | −0.36 [−3.21, +2.47] |
| documents and charts | 980 | 93.47 | −0.14 [−1.40, +1.12] | 94.01 | −0.54 [−1.59, +0.55] |
| KIE | 214 | 99.36 | +0.00 [+0.00, +0.00] | 98.75 | +0.00 [+0.00, +0.00] |
| Hateful Memes | 847 | 53.01 | +0.47 [−3.07, +4.01] | 51.12 | −0.94 [−4.01, +2.13] |
| Mind2Web | 995 | 89.15 | +1.07 [−0.43, +2.57] | 91.02 | −1.07 [−2.72, +0.44] |
| mean of 8 | | 82.69 | +0.66 [−0.09, +1.38] | 83.36 | −0.43 [−1.15, +0.28] |
Reported but not judged (new in the stage-4 data): ECD and PlotQA charts 93.76 (+3.95 [+2.72, +5.26]), NLVR2 88.48
(+9.38 [+5.98, +12.40]), FUNSD in the tool's shape 94.15 (+1.17 [−1.54, +4.12]). Some proxy images also occur in
the stage-0 training data with other questions. The DI text rule: over the 23 benchmarks whose training data was
not cleaned, the index may not drop by more than 0.5 against the reference. Result: 65.98 against 66.47, −0.499, a
pass at the bound. Files: `results/decision-index/acceptance/`.
### Format pilot
The 25% share of image records in the DI tool's request shape goes against the rule we fixed before the format
pilot. In the pilot (300 more steps from the reference checkpoint), 25% and 50% raised the 8-family proxy mean in
the tool's shape by +0.37 [−0.07, +0.82] and +0.39 [−0.10, +0.91] points, and changed it in our own shape by −0.04
[−0.44, +0.37] and +0.31 [−0.13, +0.77]. The rule required a lower bound above 0 in the tool's shape, so it said
not to rewrite. We chose 25% after seeing these results, because neither arm lowered our own shape and the gains
were in BLINK (74.6 to 76.4 and 77.0) and Mind2Web (87.4 to 88.6 and 89.4). Pilot file:
`results/decision-index/format-pilot/`.
</details>
### Release evaluation sets
The 18 sets behind the GitHub README, full sets, with the harness in `evals/`. Differences are paired bootstrap
intervals over the same questions (images resampled as groups); "n.s." where the interval includes 0.
| Set | n | Qwen3.5-4B | vev-4b | vev-4b-v2 | Difference |
|---|---:|---:|---:|---:|---|
| judgekit | 130 | 0.962 | 0.962 | 0.962 | +0.0 [+0.0, +0.0] n.s. |
| JevBench | 231 | 0.758 | 0.766 | 0.835 | +6.9 [+2.6, +11.7] |
| nimble | 324 | 0.710 | 0.707 | 0.806 | +9.9 [+4.9, +14.8] |
| kev transfer-v4 | 1528 | 0.758 | 0.776 | 0.842 | +6.6 [+4.9, +8.3] |
| MMBench-EN | 1164 | 0.872 | 0.881 | 0.900 | +1.9 [+0.6, +3.3] |
| POPE | 9000 | 0.894 | 0.889 | 0.900 | +1.1 [+0.7, +1.4] |
| MMStar | 1498 | 0.544 | 0.627 | 0.694 | +6.7 [+4.6, +8.8] |
| ui_toggle | 800 | 0.901 | 0.928 | 0.920 | −0.8 [−1.5, −0.1] |
| ui_input | 800 | 0.949 | 0.951 | 0.950 | −0.1 [−0.9, +0.6] n.s. |
| policy_mod | 659 | 0.686 | 0.742 | 0.760 | +1.8 [−1.0, +4.6] n.s. |
| policy_cat | 659 | 0.589 | 0.645 | 0.533 | −11.2 [−17.2, −5.5] |
| t2i_elem | 1027 | 0.690 | 0.715 | 0.673 | −4.2 [−6.5, −2.0] |
| t2i_elem_soft | 835 | 0.593 | 0.605 | 0.612 | +0.7 [−0.9, +2.3] n.s. |
| t2i_score | 480 | 0.142 | 0.260 | 0.267 | +0.6 [−3.1, +4.4] n.s. |
| game_glitch | 1000 | 0.646 | 0.613 | 0.647 | +3.4 [+1.4, +5.4] |
| game_clip | 686 | 0.561 | 0.671 | 0.810 | +14.0 [+7.5, +19.7] |
| game_vr | 250 | 0.792 | 0.940 | 0.720 | −22.0 [−31.4, −16.7] |
| glitch_type | 607 | 0.700 | 0.746 | 0.715 | −3.1 [−6.4, +0.2] n.s. |
vev-4b-v2 is significantly more accurate than vev-4b on 8 sets and less accurate on 4 (ui_toggle, policy_cat,
t2i_elem, game_vr).
## Training
Four stages, one seed each. All stages train a LoRA adapter (rank 16, alpha 32, all linear layers of the language
model; vision tower and LM head frozen) with learning rate 5e-5, a Brier term of weight 0.1, and a KL term (0.3)
that keeps the answer distribution close to the model without the stage's adapter, on the training records and on
the 9,543 unlabeled records of the anchor set (weight 2).
| Stage | Starts from | Data | Steps | Batch (micro-batches × tokens) | GPUs | Other settings |
|---|---|---|---:|---|---:|---|
| 0 | Qwen3.5-4B | 79,224 records sampled from 30 public text and image datasets | 2,500 | 4 × 16,384 tokens | 4 | causal attention |
| 1 | stage-0 adapter | 206,959 records: DI text task families from training splits and generators, plus stage-0 data | 6,000 | 8 × 8,192 tokens | 4 | bidirectional state block |
| 3 | stage-1 adapter | 182,468 records: adds 610 tasksource sets, Chinese datasets, OpenJevData | 5,000 | 8 × 8,192 tokens | 4 | as stage 1 |
| 4 | stage-3 weights, adapter merged; new adapter | 427,765 records: the stage-3 data plus 245,297 new records for the DI vision benchmarks, charts, NLVR2, FUNSD, Hateful Memes and text task families | 3,910 (one pass) | 16 × 8,192 tokens | 4 | images up to 1,638,400 pixels, 80% of image records read twice, 25% of image records in the DI tool's request shape |
The data, by source and stage, with licence tiers and what was removed:
[TRAINING_DATA.md](https://github.com/Xiaooolong/vev/blob/main/TRAINING_DATA.md). The anchor set is
[CountingSheep/vev-anchor-v3](https://huggingface.co/datasets/CountingSheep/vev-anchor-v3) without its 580 AG News
and Yelp records; this reduced set is not published.
## Limitations
- One training run per stage, no second seed. Seed-to-seed variation was not measured, so differences to vev-4b or
between checkpoints smaller than that variation cannot be told apart. The acceptance comparison was inconclusive:
+0.66 [−0.09, +1.38] in the tool's shape and −0.43 [−1.15, +0.28] in our own (Release acceptance).
- 25% of the stage-4 image records are written in the DI vision tool's request shape. We chose this after a pilot,
although the rule we had fixed beforehand said not to (Release acceptance).
- RealWorldQA is lower than in the reference checkpoint: 63.7 against 65.3 on the board's public items and −2.18
[−4.11, −0.71] on the proxy in the tool's shape. The drop is spread over several question types; we have not
found a single cause.
- Hateful Memes is the weakest of the nine fully weighted benchmarks (42.5 on the public suite). Half of each
training target is the probability Qwen3.5-27B gives, and we have no estimate we trust for the board's private
Hateful Memes items.
- Two-image training images are small. BLINK-style semantic-correspondence rows use SPair-71k images of at most
250,000 pixels (stored upscaled to about 409,600), visual-correspondence rows COCO images with a median of
273,920 pixels; the board's correspondence images have a median of 1,397,760 pixels. We did not upscale them
further, since that adds no detail. BLINK on the board: 71.5, with semantic correspondence 61.6.
- Text task families whose training data was removed score lower (see the DI text section).
- Against vev-4b, 4 release sets are significantly less accurate, game_vr by 22.0 points (Release evaluation sets).
- Reading twice makes requests longer: a request that goes over the token limits only when the state and the
instructions are written twice is read once, without the repetition (369 request chunks of the DI text suite).
- Image requests with several questions read the images once, single-question image requests read them twice; in
the last training stage, 80% of the image records were read twice, including records with several questions. How
much a question's probabilities move between asking it alone and in a batch was measured with vev-4b (p99 0.025),
not with vev-4b-v2. On 1,294 DI vision items, an RTX PRO 6000 and an H800 differ by up to 0.060 (p99 0.022) and
pick a different answer on 5 items.
- The probabilities rank answers well but are not exact frequencies: expected calibration error is 0.01–0.24 across
the 18 release sets. Choose any threshold on your own labelled data.
- Reversing the order of the options changes the top answer on 3–11% of the questions of the four text sets (17% of
JevBench and kev transfer-v4 with vev-4b; Jev: under 4%). Keep the option order fixed.
- Judgments that need several steps of reasoning were weaker than the base model's own answer when it may think
first (measured with vev-4b on an internal set). On "is something wrong here?" questions the model answers "no"
more often than the labels do.
- There is no adapter repository. Stages 0–3 train one adapter; stage 4 trains a second one on the weights with the
first merged in, and only the merged result is published.
- Vev is not affiliated with TypeSafe; it matches Jev's API, not its behaviour.
## Load the weights directly
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained("CountingSheep/vev-4b-v2", dtype="bfloat16")
processor = AutoProcessor.from_pretrained("CountingSheep/vev-4b-v2")
```
`config.json` keeps `mtp_num_hidden_layers` from the base model, but the multi-token-prediction weights are not
included; reading answers does not use them.
This gives the model only. Vev reads the next-token probabilities of the answer tokens (Yes/No, option letters,
level digits) at the end of a fixed prompt ([spec/systemone-api.md, section
10](https://github.com/Xiaooolong/vev/blob/main/spec/systemone-api.md#10-how-answers-are-computed)). vev-4b-v2 adds
a state block that attends in both directions in the 8 full-attention layers, and writes short text states and
single-question image requests twice; the layout is in `vev/model.py` and `vev/bidir.py`. Plain causal attention
with the vev-4b prompt gives different probabilities.
## License
CC BY-NC 4.0. The base model is Apache 2.0; its license is included as `LICENSE-Qwen`.
All Vev models and data:
[collection](https://huggingface.co/collections/CountingSheep/vev-6abd3e17303828e10c49dc65).
## Citation
```bibtex
@software{vev2026,
title = {Vev: Jev-like decision models that can also see images},
author = {Wang, Xiaolong},
year = {2026},
url = {https://github.com/Xiaooolong/vev}
}
```