Add model card

README.md

@@ -0,0 +1,141 @@
+---
+license: cc-by-nc-4.0
+base_model: Qwen/Qwen3.5-4B
+base_model_relation: finetune
+library_name: transformers
+pipeline_tag: image-text-to-text
+language: [en, zh]
+tags: [judgment, classification, calibration, vision-language, system-one, vev]
+model-index:
+  - name: vev-4b
+    results:
+      - task: {type: text-classification, name: Judgment}
+        dataset: {name: "judgekit (Chinese)", type: judgekit}
+        metrics: [{type: accuracy, value: 0.9615}]
+      - task: {type: text-classification, name: Judgment}
+        dataset: {name: "JevBench (public subset)", type: jevbench}
+        metrics: [{type: accuracy, value: 0.7662}]
+      - task: {type: text-classification, name: Judgment}
+        dataset: {name: "nimble", type: nimble}
+        metrics: [{type: accuracy, value: 0.7068}]
+      - task: {type: text-classification, name: Judgment}
+        dataset: {name: "kev transfer-v4", type: kev_transfer_v4}
+        metrics: [{type: accuracy, value: 0.7762}]
+      - task: {type: text-classification, name: Judgment}
+        dataset: {name: "MMBench-EN", type: mmbench_en}
+        metrics: [{type: accuracy, value: 0.8814}]
+      - task: {type: text-classification, name: Judgment}
+        dataset: {name: "POPE", type: pope}
+        metrics: [{type: accuracy, value: 0.8894}]
+      - task: {type: text-classification, name: Judgment}
+        dataset: {name: "MMStar", type: mmstar}
+        metrics: [{type: accuracy, value: 0.6268}]
+---
+
+# vev-4b
+
+Vev is an open judgment model. Given a state (text, JSON, images, or a mix) and typed questions (yes/no, multiple
+choice, graded score), it returns a probability for every allowed answer in one forward pass, without generating
+text. `vev-4b` is a LoRA fine-tune of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). The `vev` server exposes it through an
+HTTP API with the same request and response format as TypeSafe's `/v1/systemone`.
+
+This is v0.1, a research preview, released for non-commercial use under CC BY-NC 4.0.
+
+This repository holds merged weights: the adapter is already folded into the base model. The adapter alone is in [CountingSheep/vev-4b-lora](https://huggingface.co/CountingSheep/vev-4b-lora).
+
+## Usage
+
+```bash
+pip install vev-ai
+vev serve --model CountingSheep/vev-4b
+```
+
+```python
+from typesafe_sdk import Choice, Noul, TypeSafeClient
+
+client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8009")
+resp = client.system_one(
+    state="Order #4411 still shows 'label created' after 9 days. I need it before Friday.",
+    questions={
+        "urgent": Noul(instructions="Is the customer asking for something time-sensitive?"),
+        "team": Choice(instructions="Which team should handle this?",
+                       criteria={"shipping": "Delivery and tracking", "billing": "Charges and refunds"}),
+    },
+)
+print(resp.answers["urgent"].noul, resp.answers["team"].probabilities)
+```
+
+Images go into the state as `{"image": {"url": "data:image/png;base64,..."}}`. The API specification, the server
+and the evaluation code are at [https://github.com/Xiaooolong/vev](https://github.com/Xiaooolong/vev).
+
+The weights also load with plain `transformers` as a Qwen3.5 vision-language model. Vev reads its answers from the
+next-token probabilities of the answer tokens (Yes/No, option letters, level digits) at the end of a fixed prompt;
+the prompt format is in `vev/readout.py` and `vev/pointer.py` in the code repository. `head.pt` is an all-zero
+head kept for the loader; it does not change the output.
+
+## Evaluation
+
+Accuracy on full sets with the harness in the code repository. The zero-shot column is the base model read the
+same way without fine-tuning. kev-4B was run locally with the same harness; Jev was queried through TypeSafe's API
+(`jev-latest`, 23–24 September 2026).
+
+| Set | n | Qwen3.5-4B zero-shot | vev-4b | kev-4B | Jev |
+|---|---|---|---|---|---|
+| judgekit (Chinese) | 130 | 0.962 | 0.962 | 0.846 | 0.962 |
+| JevBench (public subset) | 231 | 0.758 | 0.766 | 0.719 | 0.861 |
+| nimble | 324 | 0.710 | 0.707 | 0.735 | 0.923 |
+| kev transfer-v4 | 1528 | 0.758 | 0.776 | 0.802 | 0.854 |
+
+| Set | n | Qwen3.5-4B zero-shot | vev-4b |
+|---|---|---|---|
+| MMBench-EN | 1164 | 0.872 | 0.881 |
+| POPE | 9000 | 0.894 | 0.889 |
+| MMStar | 1498 | 0.544 | 0.627 |
+| policy_mod: violates this safety policy? (LlavaGuard) | 659 | 0.686 | 0.742 |
+| game_glitch: screenshot contains a glitch? (VideoGameQA-Bench) | 1000 | 0.646 | 0.613 |
+| game_clip: object clips into character? (VideoGameQA-Bench) | 686 | 0.561 | 0.671 |
+| ui_toggle: switch on / off? (MobileViews) | 800 | 0.901 | 0.928 |
+| ui_input: text input field present? (MobileViews) | 800 | 0.949 | 0.951 |
+| t2i_elem: generated image shows the prompt element? (EvalMuse) | 1027 | 0.690 | 0.715 |
+
+The last six rows are judgment sets we converted from human-labelled public data; they are not established
+benchmarks. Brier score and calibration error for every set are in `results/` in the code repository;
+significance tests against the base model, kev-4B and Jev are marked in its README.
+
+Reversing the order of the options changes the top answer on this fraction of questions: judgekit (Chinese) 0.031; JevBench (public subset) 0.172; nimble 0.214; kev transfer-v4 0.170; MMStar 0.192.
+On JevBench, nimble and kev transfer-v4, kev-4B's rate is 0.10–0.12 and Jev's under 0.04.
+
+## Limitations
+
+- Jev is more accurate than Vev on nimble and kev transfer-v4 (by 18–22 and 7–8 points), and than vev-4b on
+  JevBench.
+- Answers depend on option order more than kev-4B's and Jev's do (rates above).
+- Judgments that need several reasoning steps are weaker than the base model's own answer when it may think
+  first (4–8 points on one internal scenario set).
+- On "is something wrong here?" questions the model flags problems less often than the labels do.
+- The output format matches `/v1/systemone`; the behaviour is not Jev's. Vev is not affiliated with TypeSafe.
+
+## Training
+
+LoRA rank 16 on all linear layers of the language model, vision tower frozen; 2,500 steps on 100,000 records
+sampled from a mixture of 42 public text and image datasets (English and Chinese), with a KL term that keeps the
+answer distribution close to the base model's on training rows and on 10,123 extra unlabeled anchor rows. One
+NVIDIA H800. The exact recipe, library versions and the list of sources with their licenses are in
+[TRAINING.md](https://github.com/Xiaooolong/vev/blob/main/TRAINING.md).
+
+## License
+
+CC BY-NC 4.0. Six of the training sources are research-only or non-commercial and eleven have no clear license
+for their data; this license does not replace their terms. The base model, Qwen/Qwen3.5-4B, is Apache 2.0.
+
+## Citation
+
+```bibtex
+@software{vev2026,
+  title   = {Vev: an open judgment model for text and images},
+  author  = {Wang, Xiaolong},
+  year    = {2026},
+  url     = {https://github.com/Xiaooolong/vev},
+  version = {0.1.0}
+}
+```