𒉾𒄑𒄀 (peshgi)

Licence: the weights and the code in this repository are released under Apache-2.0.

Base weights are not in this repository. The engine downloads Qwen/Qwen3.8-27B at revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 from Qwen's repository, and the two PP-OCRv5 models from PaddlePaddle's repositories, and adds the update and head stored here.

peshgi is a weighted average of seven fine-tunes of Qwen/Qwen3.8-27B. It answers multiple-choice questions about images with a probability for each option. This repository holds a low-rank update to the base model's language-model weights (peshgi_update.safetensors, 3.3 GB: the seven fine-tunes' updates, kept separate), a 26-way linear head (peshgi_head.safetensors) whose output is the answer, and a Decision Index vision engine (peshgi.py) that puts them together. The base model is not copied here: the engine downloads it from Qwen's repository on the Hub at the pinned revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.

This is a system of two models, not one. On every request an OCR model reads the images first, and the text it finds is put in the prompt before the question. The OCR model is PP-OCRv5 in its mobile size (detection PaddlePaddle/PP-OCRv5_mobile_det, English recognition PaddlePaddle/en_PP-OCRv5_mobile_rec; about 20 MB together, Apache-2.0, by the PaddlePaddle authors). It is not ours, it was not trained or changed by us, and it is not copied here: the engine loads both from the Hub at pinned revisions. Two of the fine-tunes averaged into peshgi (weights 0.1 and 0.3 of the average) were trained with this text in the prompt, and the average was selected and measured with it, so the model is meant to be run with it; all figures on this card are for the two together, and OCR time is part of every latency figure.

Three parts of the system are not optional:

  • OCR text in the prompt. Each image is read at its original size (the detector works at up to 960 pixels on the longer side; no orientation or unwarping step). Lines with a recognition score of at least 0.5 are kept, ordered top to bottom in bands of 12 pixels and then left to right, at most 60 lines and 1,200 characters per image. The prompt then starts with Text found in the image by OCR (may contain errors):, one line per image of the form [image 1] first | second | ... ([image 1] (no text) for an image without text), and an empty line; the question follows unchanged. When no image of a request has text, nothing is added.
  • Attention. In the base model's full-attention layers, every token of the prompt attends to every token of the prompt; the recurrent (Gated DeltaNet) layers stay causal. The model was trained this way, and stock causal attention gives different answers. The engine registers the attention function with transformers' attention interface and runs it on PyTorch SDPA.
  • The head. The answer is the head's output on the hidden state of the last prompt token. Nothing is decoded, and the letter logits are not used.

Usage

The engine needs Linux x86_64 and uv. install.sh fetches PaddlePaddle's GPU wheel, runs uv sync --frozen (uv fetches Python 3.12 and installs exactly what uv.lock records into .venv in the repository directory) and then checks the result on the GPU; see Requirements below.

uvx --from huggingface_hub hf download plantwaterer/peshgi --local-dir peshgi
cd peshgi
bash install.sh
uv run --frozen python -m decision_index vision pipeline --dir ../work/vision --engine peshgi:Engine --workers 1 --compact --out ../runs/peshgi

Run the commands from the repository directory, so that Python finds peshgi.py.

../work/vision is the public vision suite, built with the kit (uv run --frozen decision-index suite rebuild --vision --work ../work; see the kit's docs/vision.md). The engine takes no options for a board run. --option base=/path/to/snapshot loads the base model from a local directory instead of the Hub, --option ocr_det=/path and --option ocr_rec=/path do the same for the two OCR models, and --option device=cuda:1 picks another GPU (the OCR model runs on the same one).

In Python, started from the repository directory with uv run --frozen python:

from peshgi import Engine                      # peshgi.py in the repository directory

engine = Engine()                               # downloads the pinned base and OCR models on first use; bf16 on "cuda:0"
response, _ = engine(
    {},
    {"answer": {"type": "choice", "instructions": "Which object is closer to the camera?",
                "criteria": {"A": "the bicycle", "B": "the bench"}}},
    images=["street.jpg"])
print(response["answers"]["answer"])            # {"type": "choice", "choice": ..., "probabilities": {"A": ..., "B": ...}}

The inference procedure is as follows:

  • Prompt: one user turn with the model's chat template: the images first, then the OCR text as described above, the question, the options lettered A., B., ... one per line, and the line Answer with the option's letter from the given choices directly. Thinking is switched off (enable_thinking=False). When a row says its options are drawn in the image, the option lines are left out; when a row gives a read-out prefix, it is appended after the assistant header.
  • Images: native resolution up to 1,638,400 pixels per image (the board's cap). The base processor's own lower cap is lifted to that value; larger images are downscaled by the kit to fit it.
  • Read-out: one forward pass in bf16. The head (fp32) reads the last token's hidden state and gives one logit per option letter; the answer is the softmax over the question's options at temperature 1.
  • One question per request, with up to 12 options lettered A to L (the kit's vision format; the head has 26 outputs). No truncation and no option filtering; nothing in the engine looks at the benchmark a row comes from.
  • Warm-up at load. The recurrent layers' GPU kernels (flash-linear-attention, Triton) are compiled and tuned the first time a range of prompt lengths is seen: about 35 s for the first request on a new machine and about 9 s for the first request in each further range of 1,365 tokens, cached under ~/.triton afterwards. The engine runs synthetic requests of increasing length when it loads, so this one-off work is part of model loading for prompts of up to 8,192 tokens (the longest prompt in the public suite has 3,405). Longer prompts work; the first one in each further range is slower. --option warmup_tokens=N moves the limit and warmup_tokens=0 skips the warm-up.

Requirements:

  • GPU: one GPU with room for the bf16 weights (54 GB), the OCR models and the request; the ten benchmarks run on the RTX PRO 6000 used about 60 GB (as reported by nvidia-smi during the run). Tested on 1 x RTX PRO 6000 (96 GB) and on 1 x H200. The OCR models run on the same GPU.
  • Linux x86_64, uv and curl, and the system packages libgl1 and libglib2.0-0 (apt-get install libgl1 libglib2.0-0), which OpenCV needs. uv fetches Python 3.12 itself.
  • bash install.sh does three things:
    1. It fetches PaddlePaddle's GPU build, which is not on PyPI (paddlepaddle_gpu-3.2.0-cp312-cp312-linux_x86_64.whl, the CUDA 12.9 build), into wheels/: from PaddlePaddle's own storage (https://paddle-whl.bj.bcebos.com/stable/cu129/paddlepaddle-gpu/) first and, if that fails, from plantwaterer/paddlepaddle-gpu-mirror, which holds an unmodified copy. The file's sha256, 127f7f1afbc7e268080f733718e3c667ebb06d1152dd32861a6201836c8ddba0, is checked whichever source it came from, and uv.lock records the same sha256. A file with that sha256 that is already in wheels/ is used as it is. The CUDA 12.6 build of the same version does not run on RTX PRO 6000 cards (Unsupported GPU architecture).
    2. uv sync --frozen installs exactly what uv.lock records into .venv: PyTorch and triton from PyTorch's index, the Decision Index kit from GitHub at commit 25c6ec7b, the PaddlePaddle wheel from wheels/, everything else from PyPI.
    3. uv run --frozen python selfcheck.py imports PyTorch and PaddlePaddle on the GPU, runs the OCR model once on a test image and imports the engine; it stops with a message if any of that fails. bash install.sh --no-check skips it.
  • One set of CUDA libraries. PyTorch (CUDA 13) and PaddlePaddle (CUDA 12) each depend on their own copies of cuDNN, NCCL and cuSPARSELt, and the two sets install the same file names into the same directories; with PaddlePaddle's copies in place import torch fails with an undefined-symbol error. pyproject.toml leaves PaddlePaddle's three out, so PyTorch's are the only copies, and PaddlePaddle runs with them.
  • Versions: torch 2.14.1 (CUDA 13.0), triton 3.8.0 (PyTorch's build), transformers 5.12.1, flash-linear-attention 0.5.2, paddlepaddle-gpu 3.2.0 (CUDA 12.9), paddleocr 3.7.0, paddlex 3.7.2; the rest are in uv.lock. bf16 results and OCR text both differ slightly between GPU types; see Evaluation.

Training

peshgi is a weighted average (weights 0.1, 0.1, 0.05, 0.05, 0.1, 0.3 and 0.3, applied to the low-rank updates and to the heads) of seven fine-tunes of Qwen/Qwen3.8-27B. Each is a rank-16 LoRA on the linear layers of the language model, with the vision tower frozen, plus the 26-way head, trained on multiple-choice questions about images built from the public sources listed under Data attribution, in updates of 64 questions:

  • the first (weight 0.1) for 3,000 updates on 192,000 questions, with the OCR text described above in the prompt of every question whose images have text (79% of them);
  • the second (0.1) the same way on the same 192,000 questions in the same order, without OCR text;
  • the third and fourth (0.05 each) for 1,000 updates each on two different mixes of 64,000 questions, without OCR text, both starting from the same earlier fine-tune (1,000 updates on 60,632 questions);
  • the fifth (0.1) for 3,700 updates on 236,800 questions, without OCR text: the kinds of question of the first set with more of several of them, plus object counting, object parts, semantic correspondence, driving scenes and street scenes, and more memes;
  • the sixth (0.3) for 4,519 updates on 289,200 questions, without OCR text: the fifth's 236,800 questions and 52,400 more: where one object is relative to another (left or right, above or below) in photos and in indoor scenes, which of two objects is closer to a third and which of two marked points is closer to the camera in indoor scenes, counting in indoor and outdoor scenes, questions about photos taken by blind people, everyday-reasoning questions about photos, questions about text in photos, and more memes;
  • the seventh (0.3) the same way on the same 289,200 questions in the same order, with the OCR text in the prompt of every question whose images have text (72% of them).

The first five, at weights 0.375, 0.125, 0.0625, 0.0625 and 0.375, are plantwaterer/glires; the second, third and fourth, at five times the weights used here, are plantwaterer/dormice.

The OCR text for training came from the same OCR models as the engine uses, with the same settings. The OCR models themselves were not trained.

The repository stores the seven updates separately (<layer>.A and <layer>.B, seven rank-16 terms per matrix, 496 matrices) and their seven scales in peshgi.json; the engine adds them to the base weights in fp32, one after the other, when it loads.

Evaluation

Decision Index vision 0.3, public suite: all eleven benchmarks (12,210 questions). The run used this repository's files and engine (before the model had its name, so under a working name; see the run directory's README), installed with install.sh (uv sync --frozen, Python 3.12), with the kit's runner and scorer at commit 25c6ec7b, in bf16, one request at a time, with the OCR model on the same GPU inside each request: Winoground on 1 x H200, the other ten benchmarks on 1 x RTX PRO 6000 (96 GB), the board's latency hardware. This is our own run of the public kit, not a leaderboard entry; the board's Full score also needs the maintainers' private tests. The run files are in runs/peshgi-vision-0.3.

public vision index answered
all eleven benchmarks, this repository's files 80.21 12,210 / 12,210
the ten benchmarks other than Winoground 79.72 11,410 / 11,410
benchmark questions accuracy skill
CV-Bench 2,038 91.22 85.51
BLINK 961 87.41 80.51
RealWorldQA 589 82.00 70.59
CharXiv 532 86.28 81.70
InfographicVQA 244 99.59 99.45
Mind2Web 973 95.89 94.52
Winoground 800 92.25 84.50
KIE (CORD + FUNSD) 2,465 98.26 97.13
Hateful Memes 1,213 80.13 60.26
R-Bench-M (weight 0.5) 665 42.71 31.24
MMMU-Pro vision (weight 0.25) 1,730 55.43 48.88

How to read these numbers:

  • These are figures for two models together. The OCR model's text is in the prompt of every question whose images have text. We have not published a run of these weights without it, and the model should not be expected to do as well without it.
  • Two GPU types. Winoground was run on 1 x H200 and the other ten benchmarks on 1 x RTX PRO 6000, with the same files, engine, install and settings; runs/peshgi-vision-0.3/parts keeps each part's environment and status. bf16 arithmetic is not identical across GPU types, and neither is the OCR model's text (against one reference OCR pass over the suite's 9,767 images other than Winoground's, the text this OCR install read differed on 1.6% of them on an H200 and on 5.2% on an RTX PRO 6000, mostly by a character or a short fragment), so a rerun on one GPU type will differ on some questions.
  • Install used for this run. Both parts were produced with this repository's install as it is: bash install.sh, that is uv sync --frozen on Python 3.12, each on a machine with an empty package cache, with the PaddlePaddle wheel fetched from PaddlePaddle's own storage in both cases.
  • Difference from our earlier read. During development we read the same weights through our training code on H200 GPUs, with OCR text made beforehand: 80.35 on all eleven benchmarks and 79.90 on the ten other than Winoground. This repository's files give 80.21 and 79.72 (-0.14 on all eleven); the answers differ on 124 of the 12,210 questions (123 of the ten benchmarks' 11,410 and 1 of Winoground's 800). The weights the engine builds from these files are bit-for-bit the ones that read used. Run on H200 GPUs as that read was, these files give the same answer as the read on 11,361 of the ten benchmarks' 11,410 questions (index 79.87); that run differs from the read in its software environment and in reading the OCR text live (128 of the questions got a different prompt). The figures to go by for this repository's files are the ones in the tables above.
  • This is not a state-of-the-art claim. For comparison, 78.41 is reported for RSI-JEV-27b, the model we compared against during development, and our own glires and dormice measure 79.76 and 78.38. The weights of this average were chosen with public-suite results of earlier averages in view, so part of the difference from those figures is selection on this suite. When this was written (2026-11-14), the highest public vision index in an open submission to the board other than ours was 76.85.
  • Known limits. MMMU-Pro vision (skill 49), R-Bench-M (31) and Hateful Memes (60) are weak. On questions of those kinds the model is wrong often, and its probabilities should not be relied on. The OCR text can be wrong or incomplete, and the recognition model is the English one.

Latency (our measurement; the board measures on its own private sample): the kit runner's per-request wall time for peshgi:Engine on 1 x RTX PRO 6000 (96 GB), one request at a time, over the 11,410 questions of the ten benchmarks run there, started with empty kernel caches. OCR time is inside every figure.

median mean 80th percentile 95th percentile slowest
323 ms 365 ms 521 ms 902 ms 1,637 ms

436 of the 11,410 requests took longer than 1,000 ms. Model loading, including the warm-up described above, took 120 s and is not part of these figures. The 800 Winoground requests on 1 x H200: median 373 ms, mean 383 ms, 80th percentile 394 ms, slowest 1,008 ms.

Overlap with evaluation data

No test or validation split of any benchmark in the suite was used for training, and no Winoground image or caption was used in training or in preparing the training data. Where a benchmark has an official training split, that split may be among the training sources (InfographicVQA, CORD, FUNSD, Multimodal-Mind2Web and Hateful Memes: train splits only). COCO, LVIS, PACO, Open Images, BDD100K, SPair-71k, VizWiz, A-OKVQA, TextVQA, PrideMM and MultiOFF are used from their train splits only, and ADE20K from the training images of its scene-parsing release (CV-Bench builds part of its questions on ADE20K and COCO images). MMMU's dev and validation sets were left out because of their overlap with MMMU-Pro.

Training images were compared with the 10,742 images of the public suite that we hold as files (every benchmark except Winoground), by exact sha256 and by perceptual hash (pHash, distance of at most 6). Matches were dropped from training: 694 in the first round of data (Hateful Memes 386, CORD 210, DocVQA 43, ChartNet 29, Mind2Web 11, others 15; none of them exact), 76 in the second and 27 in the third; later additions went through the same check, among them the questions added for the sixth and seventh fine-tunes (36 images dropped: A-OKVQA 15, LVIS 12, ADE20K 2, Open Images 2, PrideMM 2, Hypersim 2, VizWiz 1). Winoground's images were not part of this comparison; the only training data of that kind are COCO captions with word-order-swapped negatives, built by us from COCO. The maintainers' private tests are not available to us, so no such check is possible against them.

Licence

The update, the head and the code in this repository are released under Apache-2.0. The base model, Qwen/Qwen3.8-27B, is Apache-2.0 and is not redistributed here. The OCR models (PP-OCRv5 mobile detection and English recognition) and the PaddleOCR and PaddlePaddle libraries are Apache-2.0, by the PaddlePaddle authors, and are not redistributed here either. The training data come from the sources below, under the licences and terms listed; the images belong to their owners. Some of these sources carry use conditions that apply to the data itself; they are repeated here: InfographicVQA (research use only, ICDAR / RRC terms), FUNSD (non-commercial research and educational use), AI2D (non-commercial), ScienceQA (CC BY-NC-SA 4.0, non-commercial and share-alike), Hateful Memes (the dataset's own licence, research use), TallyQA (treated here as non-commercial), BDD100K (its data licence: educational, research and not-for-profit use), MemeLens (CC BY-NC 4.0, non-commercial; it collects HarMeme, MAMI, Memotion and MIMIC) and ADE20K (its images: non-commercial research and educational use only). SPair-71k, PrideMM and MultiOFF state no licence. None of the training data is redistributed here.

Data attribution

peshgi was trained on multiple-choice questions built from the public data sources below; each of the two fine-tunes that had OCR text in its prompts used the same questions and images as another of the fine-tunes, with text read from those images by the OCR models. Thanks to everyone who builds and maintains them. Records were reformatted into multiple-choice questions, and many were modified (answer options added, boxes or marks drawn on images, question text rendered into images), so none of the material should be read as an official copy of its source.

source used for reference licence or terms
Hypersim multi-view and correspondence questions; distance, relative position and relative depth in indoor scenes Roberts et al., 2021 CC BY-SA 3.0
MindCube (train) spatial reasoning from several views Yin et al., 2025 MIT
2.5VRD relative depth of two objects Su et al., 2021 annotations CC BY 4.0; images from Open Images V4
VSR (train) spatial relations Liu et al., 2023 CC BY 4.0; images from COCO
Nemotron-Image-Training-v3.5 (coco_count_spatial_qa, sat_spatial_qa) counting, 2D relations, spatial questions NVIDIA OpenMDW-1.1; images from COCO
TallyQA (train) counting Acharya et al., 2019 Apache-2.0 (repository); treated here as non-commercial; images from Visual Genome and COCO
ChartNet (grounded_qa, reasoning_qa) chart questions ChartNet dataset card CDLA-Permissive-2.0
DocVQA (train), via pixparse's copy on the Hub document questions Mathew et al., 2021 MIT (the copy's licence)
InfographicVQA (train) infographic questions Mathew et al., 2022 research use only (ICDAR / RRC terms)
CORD v2 (train) receipt fields Park et al., 2019 CC BY 4.0
FUNSD (train) form fields Jaume et al., 2019 non-commercial research and educational use
AI2D science diagrams Kembhavi et al., 2016 non-commercial (the release's licence file)
ScienceQA (train) science questions Lu et al., 2022 CC BY-NC-SA 4.0
Multimodal-Mind2Web (train) web-page actions Deng et al., 2023 annotations CC BY 4.0; OpenRAIL (dataset card)
Hateful Memes (train), images in part via copies on the Hub meme moderation Kiela et al., 2020 Hateful Memes dataset licence (research use)
HateCheck, DynaHate, Measuring Hate Speech, Gab Hate Corpus, Contextual Abuse Dataset text for synthetic memes, drawn by us on DOCCI photos Röttger et al., 2021; Vidgen et al., 2021; Kennedy et al., 2020; Kennedy et al., 2022; Vidgen et al., 2021 CC BY 4.0
DOCCI photos for the synthetic memes Onoe et al., 2024 CC BY 4.0
Visual Genome question answering on photos Krishna et al., 2017 CC BY 4.0
VQA v2 (train) question answering on photos Goyal et al., 2017 CC BY 4.0; images from COCO
VizWiz-VQA (train) questions about photos taken by blind people Gurari et al., 2018 CC BY 4.0: "This work is licensed under a Creative Commons Attribution 4.0 International License" (https://vizwiz.org/tasks-and-datasets/vqa/, read 2026-11-14)
A-OKVQA (train) everyday-reasoning questions about photos Schwenk et al., 2022 Apache-2.0 (the repository's licence, https://github.com/allenai/aokvqa); images from COCO
TextVQA (train), via lmms-lab-encoder's copy on the Hub questions about text in photos (signs, labels, shop fronts) Singh et al., 2019 CC BY 4.0 (the facebook/textvqa dataset card); the copy's own card states no licence; images from Open Images
LVIS v1 (train) counting; relative position of two objects Gupta et al., 2019 annotations CC BY 4.0: "The LVIS annotations along with this website are licensed under a Creative Commons Attribution 4.0 License. All LVIS dataset images come from the COCO dataset" (https://www.lvisdataset.org/dataset, read 2026-11-10); images under COCO's terms, below
Open Images V6 (train) counting; street scenes; relative position of two objects; counting in scenes Kuznetsova et al., 2020 annotations CC BY 4.0; images listed as CC BY 2.0
ADE20K, scene-parsing release (ADEChallengeData2016, training images and instance masks) counting in indoor and outdoor scenes Zhou et al., 2017 images: "Researcher shall use the Database only for non-commercial research and educational purposes"; annotations: "licensed under a Creative Commons BSD-3 License Agreement"; the same terms say "MIT, CSAIL does not own the copyright of the images" (https://ade20k.csail.mit.edu/terms/, read 2026-11-14 through the Internet Archive's copy of 2025-12-18; the site itself could not be reached that day)
PACO-LVIS (train) object parts and what they are for Ramanathan et al., 2023 MIT; images from COCO
BDD100K (train), via dronefreak's copy on the Hub driving scenes Yu et al., 2020 BDD100K data licence: "Permission to use, copy, modify, and distribute this software and its documentation for educational, research, and not-for-profit purposes, without fee and without a signed licensing agreement"; commercial use is granted only to BDD and BAIR Commons members (https://github.com/bdd100k/bdd100k/blob/master/doc/source/license.rst, the project's own repository, read 2026-11-10; the project's sites doc.bdd100k.com and bdd-data.berkeley.edu could not be reached that day)
SPair-71k (train pairs) semantic correspondence Min et al., 2019 states no licence
MemeLens (its HarMeme, MAMI, Memotion and MIMIC parts) meme moderation Shahroor et al., 2026; Pramanick et al., 2021; Fersini et al., 2022; Sharma et al., 2020 CC BY-NC 4.0
PrideMM (train) meme moderation Shah et al., 2024 states no licence
MultiOFF (train), via Ibrahim-Alam's copy on the Hub meme moderation Suryawanshi et al., 2020 states no licence
COCO (train2014, train2017), images and captions images for the sources above; caption-matching questions with word-order-swapped negatives Lin et al., 2014 annotations CC BY 4.0; images: "The COCO Consortium does not own the copyright of the images. Use of the images must abide by the Flickr Terms of Use." (https://cocodataset.org/#termsofuse, read 2026-11-10)

Of the 289,200 questions in the largest training set (used by two of the seven fine-tunes), 80,263 come from the non-commercial sources named under Licence; a further 10,000 come from SPair-71k, PrideMM and MultiOFF, which state no licence. Of the 236,800 in the next largest (used by one), 73,863 come from the non-commercial sources and a further 6,000 from SPair-71k. Of the 192,000 in the third (used by two), 56,674 come from the non-commercial sources.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for plantwaterer/peshgi

Base model

Qwen/Qwen3.8-27B
Finetuned
(927)
this model