CoCo-Decision-Vision-0.8B

A 0.8B vision-language decision model: given an image (or several) and a multiple-choice or yes/no question, it returns a probability for every option in a single forward pass, without generating text.

It is a full fine-tune of Qwen/Qwen3.5-0.8B (revision 2fc06364) with a small option-label readout head. Research and evaluation use only (see Licence and use).

How it answers

  • Prompt. State: โ€ฆ / Question: โ€ฆ / Options: A. key: text โ€ฆ / Answer with the letter only., put in the user turn after the images with the Qwen chat template, followed by Answer:.
  • Readout. The final hidden state at the last position goes through a Linear(1024, 237) head over the single-token option labels ( A, B, โ€ฆ, AA, โ€ฆ); the probabilities are a softmax over the question's own options. The head starts from the LM-head rows of those label tokens.
  • Attention. The full-attention layers of Qwen3.5 attend over the whole prompt (bidirectionally); the Gated DeltaNet layers are unchanged.

The prompt, head and attention change follow oh-my-jev (Apache-2.0; see NOTICE). Images are read at native resolution up to 1.6 megapixels.

Results: public vision suite (Decision Index vision 0.3, local run)

Our own run of the 11 public vision benchmarks (12,210 rows, suite rebuilt with the decision-index kit, passing vision verify), with this repository's files served by q1/server.py and the kit's decision-index vision pipeline --compact (decide engine). Not a leaderboard result. The complete run (compact predictions, scores, environment) is in corners-ai/CoCo-Decision-Vision-0.8B-decision-index.

Benchmark (accuracy, %) Qwen3.5-0.8B, zero-shot letter readout CoCo-Decision-Vision-0.8B
CV-Bench 64.3 80.4
BLINK 50.7 66.8
RealWorldQA 61.0 62.3
CharXiv 64.5 66.9
InfographicVQA 79.1 83.2
Mind2Web 45.0 81.9
Winoground 65.4 74.4
KIE (CORD + FUNSD) 83.1 90.9
Moderation (Hateful Memes) 54.7 67.1
R-Bench-M 20.8 25.6
MMMU-Pro vision 13.9 19.0
Public vision index 37.70 55.30

The interpolation weight and some training choices were made while looking at this public suite, so these numbers are somewhat optimistic.

Training

All stages: full fine-tuning of every language-model layer and the readout head; the vision encoder, the vision merger and the token embeddings are frozen. AdamW, layer-wise lr decay 0.9, readout lr 1e-4, cosine schedule with warmup. Loss: cross-entropy over the question's option labels (label smoothing 0.05) + 0.5 ร— Brier score. Option order is shuffled during training. Checkpoints are chosen on a held-out image dev set drawn from the same train splits.

  1. Stage 1 โ€” from Qwen3.5-0.8B: 24.8k image questions, lr 2e-5, 1 epoch.
  2. Stage 2 โ€” two seeds on 46.5k image questions (stage-1 data plus The Cauldron subsets), lr 2e-5, 1 epoch, uniformly averaged (model soup), then 0.5 epoch at lr 5e-6.
  3. Stage 3 โ€” 18.3k rows: synthetic visual-correspondence questions, SPair-71k semantic-correspondence questions and caption-choice questions with real hard-negative captions, plus a 12k-row replay of stage-2 data; lr 1e-5, 1 epoch.
  4. Release โ€” the stage-3 weights interpolated 70% towards themselves and 30% towards Qwen3.5-0.8B (WiSE-FT, ฮฑ = 0.7).

Only train splits were used. Every training image was compared (SHA-256) with every image of the public vision suite; none matched.

Training data

Source (train split) Licence / terms
ChartQA (HuggingFaceM4/ChartQA) GPL-3.0 (Hub card)
InfographicVQA (lmms-lab/DocVQA) Apache-2.0 (mirror card); original RRC terms
CORD v2 (naver-clova-ix/cord-v2) CC BY 4.0
FUNSD (official zip, training_data) non-commercial research and education only
Hateful Memes (neuralcatcher/hateful_memes, train) research use only
Multimodal-Mind2Web (osunlp/Multimodal-Mind2Web) OpenRAIL (Hub card)
COCO 2017 train (detection-datasets/coco) and COCO captions (yerevann/coco-karpathy) COCO terms (annotations CC BY 4.0, images under Flickr licences)
The Cauldron subsets (HuggingFaceM4/the_cauldron): VSR, AI2D, ScienceQA, IconQA, TQA, A-OKVQA, TallyQA, NLVR2, Visual7W, CLEVR, DocVQA each subset keeps its source's terms; several are non-commercial (e.g. ScienceQA CC BY-NC-SA)
SPair-71k train (tasksource/spair71k-grid) PASCAL VOC / Flickr terms
Internal safety-camera images (3 event types, about 540 images) private, not released

Questions were converted to multiple choice by our own scripts (counting, captions, document answers with distractors from the same document, point correspondence drawn on the images). The internal image set will not be provided.

Usage

pip install "transformers>=5.17" torch torchvision pillow safetensors
python q1/server.py --model . --port 8000      # from this repository's directory
curl -s http://127.0.0.1:8000/v1/systemone -H "Content-Type: application/json" -d '{
  "state": {},
  "images": ["data:image/png;base64,..."],
  "questions": {"answer": {"type": "choice", "instructions": "How many people are in the image?",
                           "criteria": {"A": "1", "B": "2", "C": "3", "D": "4"}}}}'

Each question is one forward pass; choice and noul (yes/no) questions are supported. In Python, q1.readout.Decider loads the package (Decider(path=".")) and returns readout logits for rendered prompts.

Licence and use

The weights and code are released under Apache-2.0 (LICENSE, NOTICE). Because the training data include sets whose terms limit use to non-commercial research (FUNSD, Hateful Memes, ScienceQA and other Cauldron subsets) and ChartQA (GPL-3.0 card), we release this model for research and evaluation only, not for commercial use. Check those terms yourself before any other use.

Limitations

  • Strongest where the training data had matching question types (web elements, document fields, visual correspondence, counting, spatial relations). Caption matching (Winoground) and semantic correspondence improved less.
  • Not evaluated: text-only decisions, video, calibration, latency on reference hardware.
  • Like the base model, it can be wrong with high confidence; do not use it for decisions about people.
Downloads last month
81
Safetensors
Model size
0.9B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for corners-ai/CoCo-Decision-Vision-0.8B

Finetuned
(890)
this model