VisionQ-Judge

LoRA adapters for VisionQ-Judge, from the anonymous CVPR 2027 submission VisionQ. VisionQ-Judge is Gemma-4-E4B fine-tuned with DPO to pick the method that a paper's authors claim wins in a qualitative comparison figure. The training pairs are symmetric evidence pairs in every cyclic option order: 13,394 pairs, one epoch.

Folder Seed Test accuracy (538 VisionQ-MCQ questions)
seed42/ 42 61.3%
seed43/ 43 60.2%
seed44/ 44 61.3%

Tab. 5 of the paper reports the mean of the three seeds: 61.0% against 50.4% for the base model (+10.6, 95% paper-bootstrap interval +5.0 to +16.0). The per-question predictions behind these numbers are in the code release (results/judge/test/Cfix_s4*). fp16 outputs vary with GPU type, so compare runs on the same GPU type.

Use

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
base = AutoModelForImageTextToText.from_pretrained("google/gemma-4-E4B-it", torch_dtype="float16", device_map="auto")
model = PeftModel.from_pretrained(base, "visionq-review/VisionQ-Judge", subfolder="seed42")
processor = AutoProcessor.from_pretrained("visionq-review/VisionQ-Judge", subfolder="seed42")

To reproduce the reported evaluation, including the prompt, the image handling and the caption context, use judge/evaluate.py from the code release with --adapter <path>/seed42 --prepend-context --anti-bias-instruction --fp16 --max-image-size 512.

Training

judge/train.sh Cfix_s42 42 all in the code release. DPO with beta 0.3 and learning rate 2e-5, LoRA r=16 and alpha 32 on the language and vision layers, on the VisionQ-MCQ train split (2,687 questions, paper-disjoint from val and test).

The adapters are a derivative of Gemma and follow the Gemma terms of use.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for visionq-review/VisionQ-Judge

Adapter
(502)
this model