glires

Licence: the weights and the code in this repository are released under Apache-2.0.

Base weights are not in this repository. The engine downloads Qwen/Qwen3.8-27B at revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 from Qwen's repository, and the two PP-OCRv5 models from PaddlePaddle's repositories, and adds the update and head stored here.

glires is a weighted average of five fine-tunes of Qwen/Qwen3.8-27B. It answers multiple-choice questions about images with a probability for each option. This repository holds a low-rank update to the base model's language-model weights (glires_update.safetensors, 2.3 GB: the five fine-tunes' updates, kept separate), a 26-way linear head (glires_head.safetensors) whose output is the answer, and a Decision Index vision engine (glires.py) that puts them together. The base model is not copied here: the engine downloads it from Qwen's repository on the Hub at the pinned revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.

This is a system of two models, not one. On every request an OCR model reads the images first, and the text it finds is put in the prompt before the question. The OCR model is PP-OCRv5 in its mobile size (detection PaddlePaddle/PP-OCRv5_mobile_det, English recognition PaddlePaddle/en_PP-OCRv5_mobile_rec; about 20 MB together, Apache-2.0, by the PaddlePaddle authors). It is not ours, it was not trained or changed by us, and it is not copied here: the engine loads both from the Hub at pinned revisions. One of the fine-tunes averaged into glires (weight 0.375 of the average) was trained with this text in the prompt, and the average was selected and measured with it, so the model is meant to be run with it; all figures on this card are for the two together, and OCR time is part of every latency figure.

Three parts of the system are not optional:

  • OCR text in the prompt. Each image is read at its original size (the detector works at up to 960 pixels on the longer side; no orientation or unwarping step). Lines with a recognition score of at least 0.5 are kept, ordered top to bottom in bands of 12 pixels and then left to right, at most 60 lines and 1,200 characters per image. The prompt then starts with Text found in the image by OCR (may contain errors):, one line per image of the form [image 1] first | second | ... ([image 1] (no text) for an image without text), and an empty line; the question follows unchanged. When no image of a request has text, nothing is added.
  • Attention. In the base model's full-attention layers, every token of the prompt attends to every token of the prompt; the recurrent (Gated DeltaNet) layers stay causal. The model was trained this way, and stock causal attention gives different answers. The engine registers the attention function with transformers' attention interface and runs it on PyTorch SDPA.
  • The head. The answer is the head's output on the hidden state of the last prompt token. Nothing is decoded, and the letter logits are not used.

Usage

The engine needs Linux x86_64 and uv. install.sh fetches PaddlePaddle's GPU wheel, runs uv sync --frozen (uv fetches Python 3.12 and installs exactly what uv.lock records into .venv in the repository directory) and then checks the result on the GPU; see Requirements below.

uvx --from huggingface_hub hf download plantwaterer/glires --local-dir glires
cd glires
bash install.sh
uv run --frozen python -m decision_index vision pipeline --dir ../work/vision --engine glires:Engine --workers 1 --compact --out ../runs/glires

Run the commands from the repository directory, so that Python finds glires.py.

../work/vision is the public vision suite, built with the kit (uv run --frozen decision-index suite rebuild --vision --work ../work; see the kit's docs/vision.md). The engine takes no options for a board run. --option base=/path/to/snapshot loads the base model from a local directory instead of the Hub, --option ocr_det=/path and --option ocr_rec=/path do the same for the two OCR models, and --option device=cuda:1 picks another GPU (the OCR model runs on the same one).

In Python, started from the repository directory with uv run --frozen python:

from glires import Engine                      # glires.py in the repository directory

engine = Engine()                               # downloads the pinned base and OCR models on first use; bf16 on "cuda:0"
response, _ = engine(
    {},
    {"answer": {"type": "choice", "instructions": "Which object is closer to the camera?",
                "criteria": {"A": "the bicycle", "B": "the bench"}}},
    images=["street.jpg"])
print(response["answers"]["answer"])            # {"type": "choice", "choice": ..., "probabilities": {"A": ..., "B": ...}}

The inference procedure is as follows:

  • Prompt: one user turn with the model's chat template: the images first, then the OCR text as described above, the question, the options lettered A., B., ... one per line, and the line Answer with the option's letter from the given choices directly. Thinking is switched off (enable_thinking=False). When a row says its options are drawn in the image, the option lines are left out; when a row gives a read-out prefix, it is appended after the assistant header.
  • Images: native resolution up to 1,638,400 pixels per image (the board's cap). The base processor's own lower cap is lifted to that value; larger images are downscaled by the kit to fit it.
  • Read-out: one forward pass in bf16. The head (fp32) reads the last token's hidden state and gives one logit per option letter; the answer is the softmax over the question's options at temperature 1.
  • One question per request, with up to 12 options lettered A to L (the kit's vision format; the head has 26 outputs). No truncation and no option filtering; nothing in the engine looks at the benchmark a row comes from.
  • Warm-up at load. The recurrent layers' GPU kernels (flash-linear-attention, Triton) are compiled and tuned the first time a range of prompt lengths is seen: about 35 s for the first request on a new machine and about 9 s for the first request in each further range of 1,365 tokens, cached under ~/.triton afterwards. The engine runs synthetic requests of increasing length when it loads, so this one-off work is part of model loading for prompts of up to 8,192 tokens (the longest prompt in the public suite has 3,405). Longer prompts work; the first one in each further range is slower. --option warmup_tokens=N moves the limit and warmup_tokens=0 skips the warm-up.

Requirements:

  • GPU: one GPU with room for the bf16 weights (54 GB), the OCR models and the request; the ten benchmarks run on the RTX PRO 6000 used about 57 GB (as reported by nvidia-smi during the run). Tested on 1 x RTX PRO 6000 (96 GB) and on 1 x H200. The OCR models run on the same GPU.
  • Linux x86_64, uv and curl, and the system packages libgl1 and libglib2.0-0 (apt-get install libgl1 libglib2.0-0), which OpenCV needs. uv fetches Python 3.12 itself.
  • bash install.sh does three things:
    1. It fetches PaddlePaddle's GPU build, which is not on PyPI (paddlepaddle_gpu-3.2.0-cp312-cp312-linux_x86_64.whl, the CUDA 12.9 build), into wheels/: from PaddlePaddle's own storage (https://paddle-whl.bj.bcebos.com/stable/cu129/paddlepaddle-gpu/) first and, if that fails, from plantwaterer/paddlepaddle-gpu-mirror, which holds an unmodified copy. The file's sha256, 127f7f1afbc7e268080f733718e3c667ebb06d1152dd32861a6201836c8ddba0, is checked whichever source it came from, and uv.lock records the same sha256. A file with that sha256 that is already in wheels/ is used as it is. The CUDA 12.6 build of the same version does not run on RTX PRO 6000 cards (Unsupported GPU architecture).
    2. uv sync --frozen installs exactly what uv.lock records into .venv: PyTorch and triton from PyTorch's index, the Decision Index kit from GitHub at commit 25c6ec7b, the PaddlePaddle wheel from wheels/, everything else from PyPI.
    3. uv run --frozen python selfcheck.py imports PyTorch and PaddlePaddle on the GPU, runs the OCR model once on a test image and imports the engine; it stops with a message if any of that fails. bash install.sh --no-check skips it.
  • One set of CUDA libraries. PyTorch (CUDA 13) and PaddlePaddle (CUDA 12) each depend on their own copies of cuDNN, NCCL and cuSPARSELt, and the two sets install the same file names into the same directories; with PaddlePaddle's copies in place import torch fails with an undefined-symbol error. pyproject.toml leaves PaddlePaddle's three out, so PyTorch's are the only copies, and PaddlePaddle runs with them.
  • Versions: torch 2.14.1 (CUDA 13.0), triton 3.8.0 (PyTorch's build), transformers 5.12.1, flash-linear-attention 0.5.2, paddlepaddle-gpu 3.2.0 (CUDA 12.9), paddleocr 3.7.0, paddlex 3.7.2; the rest are in uv.lock. bf16 results and OCR text both differ slightly between GPU types; see Evaluation.

Training

glires is a weighted average (weights 0.375, 0.125, 0.0625, 0.0625 and 0.375, applied to the low-rank updates and to the heads) of five fine-tunes of Qwen/Qwen3.8-27B. Each is a rank-16 LoRA on the linear layers of the language model, with the vision tower frozen, plus the 26-way head, trained on multiple-choice questions about images built from the public sources listed under Data attribution, in updates of 64 questions:

  • the first (weight 0.375) for 3,000 updates on 192,000 questions, with the OCR text described above in the prompt of every question whose images have text (79% of them);
  • the second (0.125) the same way on the same 192,000 questions in the same order, without OCR text;
  • the third and fourth (0.0625 each) for 1,000 updates each on two different mixes of 64,000 questions, without OCR text, both starting from the same earlier fine-tune (1,000 updates on 60,632 questions);
  • the fifth (0.375) for 3,700 updates on 236,800 questions, without OCR text: the kinds of question of the first set with more of several of them, plus object counting, object parts, semantic correspondence, driving scenes and street scenes, and more memes.

The second, third and fourth, at four times these weights, are plantwaterer/dormice.

The OCR text for training came from the same OCR models as the engine uses, with the same settings. The OCR models themselves were not trained.

The repository stores the five updates separately (<layer>.A and <layer>.B, five rank-16 terms per matrix, 496 matrices) and their five scales in glires.json; the engine adds them to the base weights in fp32, one after the other, when it loads.

Evaluation

Decision Index vision 0.3, public suite: all eleven benchmarks (12,210 questions). The run used this repository's files and engine, installed with install.sh (uv sync --frozen, Python 3.12), with the kit's runner and scorer at commit 25c6ec7b, in bf16, one request at a time, with the OCR model on the same GPU inside each request: Winoground on 1 x H200, the other ten benchmarks on 1 x RTX PRO 6000 (96 GB), the board's latency hardware. This is our own run of the public kit, not a leaderboard entry; the board's Full score also needs the maintainers' private tests. The run files are in runs/glires-vision-0.3.

public vision index answered
all eleven benchmarks, this repository's files 79.76 12,210 / 12,210
the ten benchmarks other than Winoground 79.30 11,410 / 11,410
benchmark questions accuracy skill
CV-Bench 2,038 90.48 84.30
BLINK 961 87.20 80.19
RealWorldQA 589 82.34 71.14
CharXiv 532 86.47 81.95
InfographicVQA 244 99.59 99.45
Mind2Web 973 96.09 94.79
Winoground 800 91.88 83.75
KIE (CORD + FUNSD) 2,465 97.89 96.53
Hateful Memes 1,213 79.14 58.29
R-Bench-M (weight 0.5) 665 42.11 30.52
MMMU-Pro vision (weight 0.25) 1,730 54.57 47.89

How to read these numbers:

  • These are figures for two models together. The OCR model's text is in the prompt of every question whose images have text. We have not published a run of these weights without it, and the model should not be expected to do as well without it.
  • Two GPU types. Winoground was run on 1 x H200 and the other ten benchmarks on 1 x RTX PRO 6000, with the same files, engine, install and settings; runs/glires-vision-0.3/parts keeps each part's environment and status. bf16 arithmetic is not identical across GPU types, and neither is the OCR model's text (against one reference OCR pass over the suite's 9,767 images other than Winoground's, the text this OCR install read differed on 1.6% of them on an H200 and on 5.2% on an RTX PRO 6000, mostly by a character or a short fragment), so a rerun on one GPU type will differ on some questions.
  • Install used for this run. Both parts were produced with this repository's install as it is: bash install.sh, that is uv sync --frozen on Python 3.12. On a second machine with the same GPU type, an install that differed only in taking triton 3.8.0 from PyPI instead of from PyTorch's index gave bit-identical probabilities on 1,500 of these questions. An earlier run of the same files in an environment that pip resolved from the same package versions did not: on the ten benchmarks run on the RTX PRO 6000, 37 of 11,410 answers differ and the index is 79.17 against 79.30. The files the two environments install are the same apart from packaging metadata, pip itself and three CUDA packages that this environment leaves out (see Requirements); we have not found which of these accounts for the difference.
  • Difference from our earlier read. During development we read the same weights through our training code on H200 GPUs, with OCR text made beforehand: 79.62 on all eleven benchmarks and 79.18 on the ten other than Winoground. This repository's files give 79.76 and 79.30 (+0.14 on all eleven); the answers differ on 123 of the 12,210 questions (122 of the ten benchmarks' 11,410 and 1 of Winoground's 800). The figures to go by for this repository's files are the ones in the tables above.
  • This is not a state-of-the-art claim. For comparison, 78.41 is reported for RSI-JEV-27b, the model we compared against during development, and our own dormice measures 78.38. The weights of this average were chosen with public-suite results of earlier averages in view, so part of the difference from those figures is selection on this suite. When this was written (2026-11-10), open submissions to the board reported public vision indexes of up to 82.91.
  • Known limits. MMMU-Pro vision (skill 48), R-Bench-M (31) and Hateful Memes (58) are weak. On questions of those kinds the model is wrong often, and its probabilities should not be relied on. The OCR text can be wrong or incomplete, and the recognition model is the English one.

Latency (our measurement; the board measures on its own private sample): the kit runner's per-request wall time for glires:Engine on 1 x RTX PRO 6000 (96 GB), one request at a time, over the 11,410 questions of the ten benchmarks run there, started with empty kernel caches. OCR time is inside every figure.

median mean 80th percentile 95th percentile slowest
319 ms 356 ms 502 ms 886 ms 1,622 ms

408 of the 11,410 requests took longer than 1,000 ms. Model loading, including the warm-up described above, took 122 s and is not part of these figures. The 800 Winoground requests on 1 x H200: median 373 ms, mean 382 ms, 80th percentile 392 ms, slowest 987 ms.

Overlap with evaluation data

No test or validation split of any benchmark in the suite was used for training, and no Winoground image or caption was used in training or in preparing the training data. Where a benchmark has an official training split, that split may be among the training sources (InfographicVQA, CORD, FUNSD, Multimodal-Mind2Web and Hateful Memes: train splits only). COCO, LVIS, PACO, Open Images, BDD100K and SPair-71k are used from their train splits only. MMMU's dev and validation sets were left out because of their overlap with MMMU-Pro.

Training images were compared with the 10,742 images of the public suite that we hold as files (every benchmark except Winoground), by exact sha256 and by perceptual hash (pHash, distance of at most 6). There were no exact matches. Near matches were dropped from training: 694 in the first round of data (Hateful Memes 386, CORD 210, DocVQA 43, ChartNet 29, Mind2Web 11, others 15), 76 in the second and 27 in the third; later additions went through the same check. Winoground's images were not part of this comparison; the only training data of that kind are COCO captions with word-order-swapped negatives, built by us from COCO. The maintainers' private tests are not available to us, so no such check is possible against them.

Licence

The update, the head and the code in this repository are released under Apache-2.0. The base model, Qwen/Qwen3.8-27B, is Apache-2.0 and is not redistributed here. The OCR models (PP-OCRv5 mobile detection and English recognition) and the PaddleOCR and PaddlePaddle libraries are Apache-2.0, by the PaddlePaddle authors, and are not redistributed here either. The training data come from the sources below, under the licences and terms listed; the images belong to their owners. Some of these sources carry use conditions that apply to the data itself; they are repeated here: InfographicVQA (research use only, ICDAR / RRC terms), FUNSD (non-commercial research and educational use), AI2D (non-commercial), ScienceQA (CC BY-NC-SA 4.0, non-commercial and share-alike), Hateful Memes (the dataset's own licence, research use), TallyQA (treated here as non-commercial), BDD100K (its data licence: educational, research and not-for-profit use) and MemeLens (CC BY-NC 4.0, non-commercial; it collects HarMeme, MAMI, Memotion and MIMIC). SPair-71k's authors state no licence for it; its project page says: "The SPair-71k data includes images and metadata obtained from the PASCAL-VOC and flickr website. Use of these images and metadata must respect the corresponding terms of use." None of the training data is redistributed here.

Data attribution

glires was trained on multiple-choice questions built from the public data sources below; the fine-tune that had OCR text in its prompts used the same questions and images as another of the fine-tunes, with text read from those images by the OCR models. Thanks to everyone who builds and maintains them. Records were reformatted into multiple-choice questions, and many were modified (answer options added, boxes or marks drawn on images, question text rendered into images), so none of the material should be read as an official copy of its source.

source used for reference licence or terms
Hypersim multi-view and correspondence questions Roberts et al., 2021 CC BY-SA 3.0
MindCube (train) spatial reasoning from several views Yin et al., 2025 MIT
2.5VRD relative depth of two objects Su et al., 2021 annotations CC BY 4.0; images from Open Images V4
VSR (train) spatial relations Liu et al., 2023 CC BY 4.0; images from COCO
Nemotron-Image-Training-v3.5 (coco_count_spatial_qa, sat_spatial_qa) counting, 2D relations, spatial questions NVIDIA OpenMDW-1.1; images from COCO
TallyQA (train) counting Acharya et al., 2019 Apache-2.0 (repository); treated here as non-commercial; images from Visual Genome and COCO
ChartNet (grounded_qa, reasoning_qa) chart questions ChartNet dataset card CDLA-Permissive-2.0
DocVQA (train), via pixparse's copy on the Hub document questions Mathew et al., 2021 MIT (the copy's licence)
InfographicVQA (train) infographic questions Mathew et al., 2022 research use only (ICDAR / RRC terms)
CORD v2 (train) receipt fields Park et al., 2019 CC BY 4.0
FUNSD (train) form fields Jaume et al., 2019 non-commercial research and educational use
AI2D science diagrams Kembhavi et al., 2016 non-commercial (the release's licence file)
ScienceQA (train) science questions Lu et al., 2022 CC BY-NC-SA 4.0
Multimodal-Mind2Web (train) web-page actions Deng et al., 2023 annotations CC BY 4.0; OpenRAIL (dataset card)
Hateful Memes (train), images in part via copies on the Hub meme moderation Kiela et al., 2020 Hateful Memes dataset licence (research use)
HateCheck, DynaHate, Measuring Hate Speech, Gab Hate Corpus, Contextual Abuse Dataset text for synthetic memes, drawn by us on DOCCI photos Röttger et al., 2021; Vidgen et al., 2021; Kennedy et al., 2020; Kennedy et al., 2022; Vidgen et al., 2021 CC BY 4.0
DOCCI photos for the synthetic memes Onoe et al., 2024 CC BY 4.0
Visual Genome question answering on photos Krishna et al., 2017 CC BY 4.0
VQA v2 (train) question answering on photos Goyal et al., 2017 CC BY 4.0; images from COCO
LVIS v1 (train) counting Gupta et al., 2019 annotations CC BY 4.0: "The LVIS annotations along with this website are licensed under a Creative Commons Attribution 4.0 License. All LVIS dataset images come from the COCO dataset" (https://www.lvisdataset.org/dataset, read 2026-11-10); images under COCO's terms, below
Open Images V6 (train) counting; street scenes Kuznetsova et al., 2020 annotations CC BY 4.0; images listed as CC BY 2.0
PACO-LVIS (train) object parts and what they are for Ramanathan et al., 2023 MIT; images from COCO
BDD100K (train), via dronefreak's copy on the Hub driving scenes Yu et al., 2020 BDD100K data licence: "Permission to use, copy, modify, and distribute this software and its documentation for educational, research, and not-for-profit purposes, without fee and without a signed licensing agreement"; commercial use is granted only to BDD and BAIR Commons members (https://github.com/bdd100k/bdd100k/blob/master/doc/source/license.rst, the project's own repository, read 2026-11-10; the project's sites doc.bdd100k.com and bdd-data.berkeley.edu could not be reached that day)
SPair-71k (train pairs) semantic correspondence Min et al., 2019 no licence stated by its authors; "The SPair-71k data includes images and metadata obtained from the PASCAL-VOC and flickr website. Use of these images and metadata must respect the corresponding terms of use." (https://cvlab.postech.ac.kr/research/SPair-71k/, read 2026-11-10)
MemeLens (its HarMeme, MAMI, Memotion and MIMIC parts) meme moderation Shahroor et al., 2026; Pramanick et al., 2021; Fersini et al., 2022; Sharma et al., 2020 CC BY-NC 4.0
COCO (train2014, train2017), images and captions images for the sources above; caption-matching questions with word-order-swapped negatives Lin et al., 2014 annotations CC BY 4.0; images: "The COCO Consortium does not own the copyright of the images. Use of the images must abide by the Flickr Terms of Use." (https://cocodataset.org/#termsofuse, read 2026-11-10)

Of the 236,800 questions in the largest training set, 73,863 come from the non-commercial sources named under Licence; a further 6,000 come from SPair-71k, which states no licence. Of the 192,000 in the next largest (used by two of the five fine-tunes), 56,674 come from the non-commercial sources named under Licence.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for plantwaterer/glires

Base model

Qwen/Qwen3.8-27B
Finetuned
(927)
this model