| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-VL-8B-Instruct |
| tags: |
| - decision-model |
| - vision |
| - computer-use |
| - process-reward |
| - verifier |
| - lora |
| --- |
| |
| # Jev-Vision (V9, 8B): an open-weight vision decision model |
|
|
| A LoRA adapter (r=64 on the language model's attention and MLP projections, 175M trainable) plus typed decision |
| heads (`heads.pt`) on Qwen3-VL-8B-Instruct. It is non-generative: one forward pass answers every typed question asked |
| about an image (or two images), returning one probability per question. Request and response follow TypeSafe's Jev |
| `/v1/systemone` shape, so code written for Jev can point at this server. Trained on screens with labels derived from |
| the environment, and on typed questions built from public vision datasets. Code, benchmark, evals and the closed-loop |
| harness: https://github.com/sseanliu/Jev-Vision |
|
|
| ## What it answers |
|
|
| `noul` (yes/no probability), `choice` (a distribution over the supplied options, pointer readout over the option |
| tokens) and `score` (ordered levels). Questions it was trained on: `ground` (which marked element to act on), |
| `skip` (is this element already in the state the task needs), `effect` (did the last action change the page as |
| intended; two images), `done` (is the goal achieved), the jev-ultrafast `operation` / `click_target` questions, and |
| general-image questions (object presence, VQA multiple choice, dish classification, two-image statement truth). |
|
|
| ## Measured (held out) |
|
|
| Screens, 30 web sites never seen in training (labels from URL / DOM / field values): |
|
|
| | track | skip | effect | done | ground | |
| |---|---|---|---|---| |
| | v0 candidate (marked screenshot + element table) | 0.955 | 0.946 | 0.944 | 0.939 | |
| | v1 candidate, fresh episodes | 0.922 | 0.977 | 0.887 | 0.949 | |
| | v1 pixel (raw screenshot + one marker, no table) | 0.931 | 0.931 | 0.891 | - | |
|
|
| Jev 1.13 on the same v1 items (text only): 0.870 / 0.733 / 0.887. jev-ultrafast `operation` question, 615 held-out |
| rows: 0.911; `click_target` 0.969. As the decider inside jev-ultrafast it completes a Google Flights one-way search |
| (10 actions, 33 s, all 7 independent page checks). |
|
|
| General images, 1,500 items from held-out splits: POPE 0.907, MME 0.927, NLVR2 0.930, A-OKVQA 0.893, Food-101 0.930 |
| (mean 0.917; the frozen backbone read from next-token logits: 0.911); yes/no ECE 0.034 to 0.047. Latency 74 ms per |
| general item, 150 to 190 ms per screenshot with all of a step's questions, on one H100 at batch 1. |
|
|
| ## Known limits |
|
|
| - On text-only long documents (JevBench's public hard tier) it is 52% accurate at 91% mean confidence: badly |
| over-confident out of domain. The frozen backbone read from next-token logits is 59% at ECE 0.35 on the same items, |
| so most of this is the backbone's own behaviour on long text; a temperature fitted on out-of-domain images |
| (T = 1.3 on GQA val) moves the hard-tier ECE from 0.28 to 0.24 only. Do not use the stated probabilities on long |
| text documents as calibrated confidences. On general images the yes/no class AUROC is 0.97 to 0.98. |
| - Web grounding on Mind2Web-style pages 0.823 and macOS desktop grounding 0.822 are below the earlier grounding-only |
| checkpoint (0.850 / 0.914). |
| - Trained and evaluated on 1280x1000 desktop web screenshots; not on mobile, logged-in or form-heavy flows. |
|
|
| ## Serving |
|
|
| ```sh |
| git clone https://github.com/sseanliu/Jev-Vision && cd Jev-Vision/model |
| pip install transformers==5.17.* peft accelerate pillow |
| hf download SeanLiu/Jev-Vision --local-dir runs/v9-8b-ops/final |
| python serve.py runs/v9-8b-ops/final --port 8811 --temp 0.5 |
| ``` |
|
|
| `POST /v1/systemone` with `{"state": ..., "image": <base64 PNG> | "images": [...], "questions": {id: {"type": "noul" | "choice" | "score", "instructions": ..., "criteria": ...}}}`. |
|
|
| Files: `backbone/` (PEFT adapter), `heads.pt`, `s1_config.json`, `eval_*.json` (the numbers above). |
|
|
