Instructions to use yah01/vjev-vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yah01/vjev-vision with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="yah01/vjev-vision")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("yah01/vjev-vision")
model = AutoModelForMultimodalLM.from_pretrained("yah01/vjev-vision", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yah01/vjev-vision with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yah01/vjev-vision"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yah01/vjev-vision",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/yah01/vjev-vision

SGLang

How to use yah01/vjev-vision with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yah01/vjev-vision" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yah01/vjev-vision",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yah01/vjev-vision" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yah01/vjev-vision",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use yah01/vjev-vision with Docker Model Runner:
```
docker model run hf.co/yah01/vjev-vision
```

vjev-vision

A listwise decision model with vision: give it a state (text, images, or both) and typed questions — noul (is this statement true?), choice (pick one), score (an ordered scale) — and it returns calibrated probabilities for every option, in a single forward pass, with no text generation. It re-creates the Jev API's shape on an open base (Qwen3.5-4B), with images added.

This is the released checkpoint of the full vision run (step 600 of 1,200; chosen over the final step for its yes/no behaviour, see below). The earlier 300-step pilot stays at yah01/vjev-vision-pilot.

Use it

The web console and the HTTP API live in BubbleCal/vjev-serve:

pip install git+https://github.com/BubbleCal/vjev-serve
vjev-serve --model yah01/vjev-vision        # then open http://localhost:8800

Or, with nothing but transformers, the single file in this repo:

from huggingface_hub import hf_hub_download
import importlib.util
spec = importlib.util.spec_from_file_location("vjev_infer", hf_hub_download("yah01/vjev-vision", "vjev_infer.py"))
vjev_infer = importlib.util.module_from_spec(spec); spec.loader.exec_module(vjev_infer)

m = vjev_infer.Vjev("yah01/vjev-vision")          # cuda / mps / cpu
m.ask(state=["photo.jpg", "Frame from the warehouse camera, 12:40."],
      questions={"person": {"type": "noul",   "instructions": "There is a person in this image."},
                 "where":  {"type": "choice", "instructions": "Where is the forklift?",
                            "criteria": {"left": "left half", "right": "right half", "none": "no forklift"}},
                 "busy":   {"type": "score",  "instructions": "How cluttered is the scene?",
                            "criteria": ["empty", "sparse", "busy", "crowded"]}})

Options inside one question compete (one softmax); questions never see each other. To rate several independent things, ask one noul per thing (vjev-serve's multilabel type does this). About 9 GB of memory in bf16/fp16.

What is in this repo

file	what
`model-*.safetensors`, `config.json`, tokenizer and processor files	the full model, bf16: Qwen3.5-4B with the checkpoint's LoRA merged in
`head.pt`	the listwise scoring head: one shared linear layer read at each option's slot
`vjev.json`	how inputs are rendered (`readout: trailing`, `pause: 0`, length budgets)
`vjev_infer.py`	stand-alone inference
`adapter/`	the same weights as a LoRA adapter (PEFT, r=32, α=64) over `Qwen/Qwen3.5-4B`, fitted on the nf4-quantized base

The merged weights are the nf4-dequantized base the adapter was trained against, plus the adapter, in bf16 — no bitsandbytes needed. Against the adapter on its nf4 base (CUDA, 118 questions over 10 images): max |Δp| 0.0097, mean 0.0036, no answer changed.

How it was trained

Text (3,868 steps): QLoRA on ~145k typed questions with soft labels from the Jev API plus human-labelled corpora; KL to the teacher distribution for choice/score, soft BCE for noul. All options of a question share one sequence and are read at a trailing Answer: (A) (B) … slot, so options can see each other.
Vision (this checkpoint, step 600 of a 1,200-step run warm-started from the text stage): a second LoRA on the vision tower (0.1× the text learning rate), on ~84k geometry questions from COCO-2017 annotations (which object is highest / smallest / left of…, counts, presence with adversarial absent objects; soft labels by Monte-Carlo perturbation of the boxes) and ~15k VQAv2 questions with their 10-annotator answer distributions, mixed with 25% text.

Results (held out: images and texts never seen in training)

	text stage (zero-shot)	pilot, 300 steps	this, step 600	final step 1200
COCO geometry, choice accuracy	0.506	0.665	0.735	0.703
COCO geometry, choice ECE ↓	0.171	0.094	0.183	0.148
VQAv2, choice accuracy	0.579	0.654	0.706	0.721
VQAv2, choice ECE ↓	0.126	0.033	0.052	0.070
POPE adversarial, absent objects called present ↓	5.5%	13.8%	6.1%	12.9%
POPE, AUC	0.977	0.969	0.966	0.962
text, 20k held-out rows: accuracy vs human labels	0.796	—	0.799	0.797
text, 20k held-out rows: ECE ↓	0.068	—	0.063	0.065

Step 600 was picked over the final step: the final step called absent objects present twice as often, for a small VQA gain. On text the vision stage cost nothing; the teacher itself scores 0.797 accuracy and 0.102 ECE on the same rows.

Options interact as in the teacher: adding a competing option protects the leader and takes its mass from the runners-up (a paired effect of +0.0066 over the order-permutation noise floor, CI excluding 0) — something a one-option-per-pass scorer cannot do.

Limits

Spatial answers are over-confident (ECE 0.18 on COCO geometry): the ranking is better than the probabilities.
Trained on single images; several images in one request (Picture 1:, Picture 2:) rely on the base model's ability.
Options are read in the order given; reordering them moves probabilities by a TVD of ~0.037 on average.
Images are resized to a 512 px longer side, the training resolution.

License

Apache-2.0. The base model, Qwen3.5-4B, is Apache-2.0.

Downloads last month: 16

Safetensors

Model size

5B params

Tensor type

BF16

Model tree for yah01/vjev-vision

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Finetuned

(739)

this model