Instructions to use corners-ai/CoCo-Decision-Vision-0.8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use corners-ai/CoCo-Decision-Vision-0.8B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="corners-ai/CoCo-Decision-Vision-0.8B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("corners-ai/CoCo-Decision-Vision-0.8B") model = AutoModelForMultimodalLM.from_pretrained("corners-ai/CoCo-Decision-Vision-0.8B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use corners-ai/CoCo-Decision-Vision-0.8B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "corners-ai/CoCo-Decision-Vision-0.8B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "corners-ai/CoCo-Decision-Vision-0.8B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/corners-ai/CoCo-Decision-Vision-0.8B
- SGLang
How to use corners-ai/CoCo-Decision-Vision-0.8B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "corners-ai/CoCo-Decision-Vision-0.8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "corners-ai/CoCo-Decision-Vision-0.8B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "corners-ai/CoCo-Decision-Vision-0.8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "corners-ai/CoCo-Decision-Vision-0.8B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use corners-ai/CoCo-Decision-Vision-0.8B with Docker Model Runner:
docker model run hf.co/corners-ai/CoCo-Decision-Vision-0.8B
CoCo-Decision-Vision-0.8B
A 0.8B vision-language decision model: given an image (or several) and a multiple-choice or yes/no question, it returns a probability for every option in a single forward pass, without generating text.
It is a full fine-tune of Qwen/Qwen3.5-0.8B (revision 2fc06364) with a
small option-label readout head. Research and evaluation use only (see Licence and use).
How it answers
- Prompt.
State: โฆ / Question: โฆ / Options: A. key: text โฆ / Answer with the letter only., put in the user turn after the images with the Qwen chat template, followed byAnswer:. - Readout. The final hidden state at the last position goes through a
Linear(1024, 237)head over the single-token option labels (A,B, โฆ,AA, โฆ); the probabilities are a softmax over the question's own options. The head starts from the LM-head rows of those label tokens. - Attention. The full-attention layers of Qwen3.5 attend over the whole prompt (bidirectionally); the Gated DeltaNet layers are unchanged.
The prompt, head and attention change follow oh-my-jev (Apache-2.0; see NOTICE). Images are read at native resolution up to 1.6 megapixels.
Results: public vision suite (Decision Index vision 0.3, local run)
Our own run of the 11 public vision benchmarks (12,210 rows, suite rebuilt with the decision-index kit, passing
vision verify), with this repository's files served by q1/server.py and the kit's decision-index vision pipeline --compact (decide engine). Not a leaderboard result. The complete run (compact predictions, scores, environment) is in corners-ai/CoCo-Decision-Vision-0.8B-decision-index.
| Benchmark (accuracy, %) | Qwen3.5-0.8B, zero-shot letter readout | CoCo-Decision-Vision-0.8B |
|---|---|---|
| CV-Bench | 64.3 | 80.4 |
| BLINK | 50.7 | 66.8 |
| RealWorldQA | 61.0 | 62.3 |
| CharXiv | 64.5 | 66.9 |
| InfographicVQA | 79.1 | 83.2 |
| Mind2Web | 45.0 | 81.9 |
| Winoground | 65.4 | 74.4 |
| KIE (CORD + FUNSD) | 83.1 | 90.9 |
| Moderation (Hateful Memes) | 54.7 | 67.1 |
| R-Bench-M | 20.8 | 25.6 |
| MMMU-Pro vision | 13.9 | 19.0 |
| Public vision index | 37.70 | 55.30 |
The interpolation weight and some training choices were made while looking at this public suite, so these numbers are somewhat optimistic.
Training
All stages: full fine-tuning of every language-model layer and the readout head; the vision encoder, the vision merger and the token embeddings are frozen. AdamW, layer-wise lr decay 0.9, readout lr 1e-4, cosine schedule with warmup. Loss: cross-entropy over the question's option labels (label smoothing 0.05) + 0.5 ร Brier score. Option order is shuffled during training. Checkpoints are chosen on a held-out image dev set drawn from the same train splits.
- Stage 1 โ from Qwen3.5-0.8B: 24.8k image questions, lr 2e-5, 1 epoch.
- Stage 2 โ two seeds on 46.5k image questions (stage-1 data plus The Cauldron subsets), lr 2e-5, 1 epoch, uniformly averaged (model soup), then 0.5 epoch at lr 5e-6.
- Stage 3 โ 18.3k rows: synthetic visual-correspondence questions, SPair-71k semantic-correspondence questions and caption-choice questions with real hard-negative captions, plus a 12k-row replay of stage-2 data; lr 1e-5, 1 epoch.
- Release โ the stage-3 weights interpolated 70% towards themselves and 30% towards Qwen3.5-0.8B (WiSE-FT, ฮฑ = 0.7).
Only train splits were used. Every training image was compared (SHA-256) with every image of the public vision suite; none matched.
Training data
| Source (train split) | Licence / terms |
|---|---|
ChartQA (HuggingFaceM4/ChartQA) |
GPL-3.0 (Hub card) |
InfographicVQA (lmms-lab/DocVQA) |
Apache-2.0 (mirror card); original RRC terms |
CORD v2 (naver-clova-ix/cord-v2) |
CC BY 4.0 |
FUNSD (official zip, training_data) |
non-commercial research and education only |
Hateful Memes (neuralcatcher/hateful_memes, train) |
research use only |
Multimodal-Mind2Web (osunlp/Multimodal-Mind2Web) |
OpenRAIL (Hub card) |
COCO 2017 train (detection-datasets/coco) and COCO captions (yerevann/coco-karpathy) |
COCO terms (annotations CC BY 4.0, images under Flickr licences) |
The Cauldron subsets (HuggingFaceM4/the_cauldron): VSR, AI2D, ScienceQA, IconQA, TQA, A-OKVQA, TallyQA, NLVR2, Visual7W, CLEVR, DocVQA |
each subset keeps its source's terms; several are non-commercial (e.g. ScienceQA CC BY-NC-SA) |
SPair-71k train (tasksource/spair71k-grid) |
PASCAL VOC / Flickr terms |
| Internal safety-camera images (3 event types, about 540 images) | private, not released |
Questions were converted to multiple choice by our own scripts (counting, captions, document answers with distractors from the same document, point correspondence drawn on the images). The internal image set will not be provided.
Usage
pip install "transformers>=5.17" torch torchvision pillow safetensors
python q1/server.py --model . --port 8000 # from this repository's directory
curl -s http://127.0.0.1:8000/v1/systemone -H "Content-Type: application/json" -d '{
"state": {},
"images": ["data:image/png;base64,..."],
"questions": {"answer": {"type": "choice", "instructions": "How many people are in the image?",
"criteria": {"A": "1", "B": "2", "C": "3", "D": "4"}}}}'
Each question is one forward pass; choice and noul (yes/no) questions are supported. In Python, q1.readout.Decider
loads the package (Decider(path=".")) and returns readout logits for rendered prompts.
Licence and use
The weights and code are released under Apache-2.0 (LICENSE, NOTICE). Because the training data include sets whose terms limit use to non-commercial research (FUNSD, Hateful Memes, ScienceQA and other Cauldron subsets) and ChartQA (GPL-3.0 card), we release this model for research and evaluation only, not for commercial use. Check those terms yourself before any other use.
Limitations
- Strongest where the training data had matching question types (web elements, document fields, visual correspondence, counting, spatial relations). Caption matching (Winoground) and semantic correspondence improved less.
- Not evaluated: text-only decisions, video, calibration, latency on reference hardware.
- Like the base model, it can be wrong with high confidence; do not use it for decisions about people.
- Downloads last month
- 81