📡 Vela 3.0
Collection
Six open routing models, 0.3B to 27B: the router's decisions and the spans you name, in one call. • 7 items • Updated • 4
Name it. Vela finds it.
vLLM Semantic Router × KR Labs · Vela 3.0 family · vela3-fast-vision · 0.25B · base UltraVBERT · MIT · preview
What it does · Quick start · Evaluation · The family · 📡 Collection
vela3-fast-vision is the small vision member of the Vela 3.0 family, a preview: three 254M encoder backbones that answer decisions about images and text, mark the words of an answer that an image does not support, check OCR extractions against the scan, point at named things in an image, and answer the router's text signals, at about 30–40 ms per question on a GPU.
| Parameters | 0.25B per call (one 254M backbone per question type); the package ships three backbones plus a 0.15B text encoder: a decision-trained UltraVBERT (UltraBERT-150M + SigLIP2-B/16-512) for decisions, an earlier decision-trained UltraVBERT for region pointing, and the original UltraVBERT, frozen, with LoRA adapters for spans, the OCR check and the router; one backbone per question |
| Inputs | Text or JSON plus 1 image (span, box) or up to 3 images (decisions), English |
| Answers | Choice · Yes / No · Span · Box |
| Licence | MIT (base MIT; UltraBERT-150M and SigLIP2 Apache-2.0) |
Summary
vela3-fast-vision is the image model of the Vela 3.0 family, at 0.25B parameters and small enough for a CPU. Three UltraVBERT backbones each answer their own kind of question: a decision backbone distilled from the d3 models for image and text decisions, a second decision-trained backbone for region pointing, and the original backbone, frozen, with adapters for image-hallucination spans, the OCR check and the router's text signals. One backbone pass per question. A preview.
Highlights
- Decisions: 37.5 on the Decision Index 0.3 public index and 22.79 on the Decision Index vision board 0.4, public part (our runs of the unmodified kits; not board entries).
- Region pointing: 0.441 mean IoU on BoxSet-test v2, level with a d3-turbo 27B dev build (0.442) at 1/89 the size.
- OCR check: char-F1 0.868 on a synthetic test set of 800 rendered documents (synthetic, not a public benchmark): 504 of 632 corrupted fields caught, 34 false flags on 11,981 clean fields.
- Image-grounded hallucination spans: 0.518 char-F1 on M-HalDetect validation (evaluation only).
- Text in the same package: unsupported claims 0.892 example-F1 on the 10,698-answer LettuceDetect v2 test set (LettuceDetect v2 mmBERT-base 0.869, reported); router signals better than Vela 2.0 0.3B on 12 of 24 test files, level on 9, worse on 3.
- Small and fast: one 0.25B backbone per question, 29–31 ms per question on one GPU at 1024 px.
At a glance
| What it is | A 0.25B encoder model that reads images and text, small enough for CPU: document and image decisions, the words of an answer the image does not support, region pointing, fields an OCR engine misread, and the router's text signals and spans like vela3-fast. |
| How it was trained | Three UltraVBERT backbones (UltraBERT-150M text encoder + SigLIP2-B vision encoder), images at 1024 px. Decisions come from a fully trained decision backbone: UltraVBERT trained as a decision model on d3 soft labels (text, images and d3-mini's answers on document images), on public datasets' train splits written as decisions (for example ANLI and RAGTruth) and on synthetic decision tasks in the styles the Decision Index covers (for example home-appliance control, chord progressions and causal questions), labelled by d3 or by program and checked against every Decision Index item; three training runs were averaged. Region pointing is an adapter on an earlier decision-trained backbone. Spans, the OCR check and the router come from the adapter backbone: the original UltraVBERT, frozen, with LoRA adapters for image-hallucination spans (image-hallucination span data built on COCO images), the OCR check (synthetic documents only) and the vela3-fast text adapters retrained on its text encoder. Router signals are trained supervised, as in Vela 2.0, on the public router datasets Vela 2.0 used (for example AEGIS 2.0, PolyGuardMix and Nemotron-Safety-Guard v3 for safety, Global-MMLU for domain, WildFeedback for feedback, DiffusionDB for modality, MASSIVE for intents); domain and feedback also on the public router signal suite's train split. No test set is trained on. |
| What it is good at | Region pointing at 1/89 the size of a d3-turbo dev build; the OCR check (synthetic test); image-grounded hallucination spans; unsupported-claim spans, PII, NER and router signals for text in the same package. |
| How we measure it | Public test sets (plus one synthetic OCR test set, labelled synthetic), run through this package one request at a time with the shipped thresholds: BoxSet-test v2, an RVL-CDIP test subset, M-HalDetect validation (evaluation only), the LettuceDetect v2 test set (paper, GroundLM 2026 workshop at EMNLP 2026), RAGTruth, OOD-7, 26 NER sets, the 13-set GLiFormer classification suite, ACL-Verbatim, QASPER, the public router signal suite. Other models' numbers are as their authors report them on the same test set and metric, or scored by us on the same rows where a table says so. |
What it does
| Capability | Ask | Get |
|---|---|---|
| Image decisions | {"type": "choice", "instructions": "What type of document is this?", "criteria": {...}} (also noul), images=[...] |
a probability per option |
| Image-grounded claims | state {"user": ..., "answer": ...}, {"type": "span", "criteria": {"unsupported": None}}, one image |
the words the image does not support |
| Region pointing | {"type": "box", "instructions": "the total amount"}, one image |
a coarse box, a point and a confidence |
Quick start
# pip install "transformers>=5.20" torch safetensors tokenizers numpy scipy pillow huggingface_hub
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("vllm-sr/vela3-fast-vision")
sys.path.insert(0, path)
from vela3v_encoder import Vela3VEncoder
m = Vela3VEncoder.load(path) # GPU (bf16) if available, else CPU (fp32)
# 1. Image decisions: choice and yes/no questions (up to 3 images)
r = m.system_one("", {
"kind": {"type": "choice", "instructions": "What type of document is this?",
"criteria": {"invoice": "an invoice or bill", "letter": "a letter", "form": "a form"}},
"total": {"type": "noul", "instructions": "Does this document show a total amount?"},
}, images=["scan.png"])
# r["answers"]["kind"] = {"type": "choice", "choice": ..., "probabilities": {...}}; r["answers"]["total"] = {"type": "noul", "noul": p(yes)}
# 2. Hallucination spans: the words of the answer that the image does not support (one image)
r = m.system_one({"user": "Describe the photo.", "answer": "Two dogs run on a beach at sunset."},
{"halu": {"type": "span", "criteria": {"unsupported": None}}}, images=["photo.jpg"])
# r["answers"]["halu"]["spans"]: the unsupported words, with character offsets
# 3. Region pointing: a coarse box and a point for a named thing (one image)
r = m.system_one("", {"where_total": {"type": "box", "instructions": "the total amount"}}, images=["scan.png"])
# r["answers"]["where_total"] = {"type": "box", "box": {"top", "bottom", "left", "right"}, "point": {"y", "x"}, "confidence"}
Images can be paths, PIL images, URLs or data URLs. A question the model cannot answer gets {"type", "error", "message"}; the other questions are still answered. Nothing is cut: a question longer than 32,768 tokens, or a decision with more than 3 images, gets an error instead.
Over HTTP (/v1/systemone, request body {"state", "questions", "images": [base64 data URLs]}):
python vela3_server.py --model vllm-sr/vela3-fast-vision --port 8000
Decision Index (text board: decision_index_engine:VelaVisionEngine; vision board: vela3v_di_engine:Vela3VEngine):
hf download vllm-sr/vela3-fast-vision --local-dir model
pip install -r model/requirements.txt "decision-index @ git+https://github.com/apolinario/decision-index@25c6ec7"
PYTHONPATH=model python -m decision_index run --engine decision_index_engine:VelaVisionEngine --option run=model --edition 0.3 --out runs/vela3-fast-vision --compact
Evaluation
All numbers on the full public test sets, measured through this package. No model of its size reports these sets, so the references are larger models. Details: EVALUATION.md.
Region pointing
| Benchmark | vela3-fast-vision | Best other model (reported) | Vela 2.0 same size | Size-class best (≤500M) |
|---|---|---|---|---|
| BoxSet-test v2, mean IoU | 0.441 | d3-turbo 27B dev build 0.442 (logged read, same set; 2.6 s per box) | no image path | – (none reported) |
| BoxSet-test v2, point-in-box | 0.771 | – | no image path | – |
| BoxSet-test v2, accuracy at IoU 0.5 | 0.421 | d3-turbo 27B dev build 0.407 | no image path | – |
BoxSet-test v2 is the box test set d3-turbo reports on: used for comparison only, never for training or for choosing settings.
Decisions on the Decision Index
| Benchmark | vela3-fast-vision (our run) |
|---|---|
| Decision Index 0.3, public split (public index) | 37.5 |
| Decision Index vision board 0.4, public split, 10 of 11 benchmarks (public vision index) | 22.79 |
Our runs with the Decision Index kit on the public splits; the board's Full score also uses private sets that only the maintainers run, so no Full score is claimed here. Winoground (gated) was not run. Per-area and per-benchmark numbers are in EVALUATION.md.
OCR check (new)
Give the scan as the image and the OCR text or extracted fields as the state; the model marks the characters the scan does not support (a span question with "task": "ocr_check"). It is trained on synthetic rendered invoices, receipts, order forms and bank details only. On a synthetic test set of 800 rendered documents (synthetic, not a public benchmark) it reaches char-F1 0.868, catches 504 of 632 corrupted fields and flags 34 of 11,981 clean fields. Treat it as a preview.
Image decisions
| Benchmark | vela3-fast-vision | Best other model (reported) | Vela 2.0 same size | Size-class best (≤500M) |
|---|---|---|---|---|
| RVL-CDIP test subset, 2,000 images (accuracy, in-domain) | 0.713 | d3-mini (4.5B) 0.752, zero-shot (its teacher) | no image path | – (none reported on this subset) |
Decisions come from the decision-trained backbone, which learned from d3 soft labels, including d3-mini's answers on RVL-CDIP train-split images; the test images are held out.
Image-grounded hallucination spans
| Benchmark | vela3-fast-vision | Best other model (reported) | Vela 2.0 same size | Size-class best (≤500M) |
|---|---|---|---|---|
| M-HalDetect validation, 786 descriptions (char-F1) | 0.518 | vela3-mini (4.5B) 0.575 · vela3-flash (8.4B) 0.633 (same set, their image adapters) | no image path | – (none reported) |
M-HalDetect (CC BY-NC) was used for internal evaluation only; nothing was trained on it.
Text questions (the vela3-fast text adapters, retrained on this model's text encoder)
| Benchmark | vela3-fast-vision | Best reported ≤500M | Vela 2.0 0.3B |
|---|---|---|---|
| LettuceDetect v2 test set, 10,698 answers (example-F1) | 0.892 | LettuceDetect v2 mmBERT-base (~307M) 0.869 | 0.848 |
| RAGTruth (example-F1) | 0.772 | LettuceDetect large v1 (395M) 0.792 | 0.706 |
| OOD-7 zero-shot NER, label names only (F1) | 60.4 | gliner-multitask-large-v0.5 (440M) 62.76 | 31.0 (our read) |
| NER, 26 datasets of the GLiFormer paper, label names (F1; not all zero-shot) | 57.9 | gliner_large-v2.5 (~0.46B) 50.8 (our run, same harness); GLiFormer-base (264M) 50.3 | – |
| Zero-shot classification, 13 sets (mean macro-F1) | 66.8 | GLiFormer-base (264.2M) 72.36; GLiClass Instruct large (438.7M) 70.99 | – |
| ACL-Verbatim evidence, word-F1 (in-domain) | 38.4 | – | 0.1 (our read) |
| QASPER evidence, word-F1 | 16.9 | – | – |
Other models' numbers as reported (or our run in the same harness where marked in the vela3-fast card); Vela 2.0 0.3B as published on its card. The hallucination test, RAGTruth and ACL-Verbatim have training splits in the span adapter's mix.
Router signals (public router signal suite; above Vela 2.0 0.3B on 6 of 8)
| Signal (metric, test rows) | vela3-fast-vision | Vela 2.0 0.3B, same rows | Decision-1.0-Route-0.6B, same rows | Rows sharing no text with training: vela3-fast-vision / Vela 2.0 0.3B |
|---|---|---|---|---|
| Harmful request (AUC, 2,230) | 0.908 | 0.945 | 0.963 | 0.883 / 0.926 (1,285) |
| Jailbreak or prompt injection (AUC, 1,878) | 0.918 | 0.787 | 0.991 | 0.877 / 0.555 (870) |
| Personal data in the request (AUC, 3,000) | 0.949 | 0.933 | 0.994 | 0.945 / 0.930 (2,787) |
| Answer not supported by its context (AUC, 3,000) | 0.727 | 0.738 | 0.791 | 0.664 / 0.690 (1,739) |
| Fact-check needed (AUC, 800) | 0.784 | 0.755 | 0.942 | 0.777 / 0.768 (688) |
| Output modality (text or image) (AUC, 3,000) | 0.890 | 0.717 | 0.999 | 0.884 / 0.709 (2,876) |
| Domain (subject area) (accuracy, 2,996) | 0.856 | 0.595 | 0.863 | 0.856 / 0.595 (2,904) |
| User feedback on the previous answer (accuracy, 2,842) | 0.847 | 0.568 | 0.848 | 0.836 / 0.508 (2,269) |
vllm-sr/router-signal-suite (public), the test file of each signal, scored with the dataset's own scoring code. Each signal is a router question this package was trained on (supervised, as in Vela 2.0), asked one request at a time with the shipped thresholds.
Vela 2.0 0.3B was run by us through its public release on the same rows. Decision-1.0-Route-0.6B is scored from the suite's published predictions; it was trained on the suite's own train split, so these test files are in-distribution for it.
Bold: the paired 95% bootstrap CI against Vela 2.0 excludes a tie. The last column drops the test rows that share text (exact or a 13-gram) with this model's training data. Safety and jailbreak are scored on the rows whose text the suite publishes (2,230 of 3,003 and 1,878 of 3,000).
Domain and feedback are also trained on the suite's own train split (with the public router data Vela 2.0 used; the suite's test and hold rows, and anything sharing an exact text or a 13-gram with them, are kept out), as Decision-1.0-Route is, so their test files are in-distribution for both. The domain question has a 15th option, religion, which the suite maps to philosophy. Feedback has NO_FEEDBACK as a trained option.
Over all 24 test files of the suite, including hazard and tool need (answered by the decision backbone, not by a trained router question): 12 wins, 9 ties and 3 losses against Vela 2.0 0.3B (paired 95% CI).
Latency
| Median per question, batch 1 | 512 px image | 1024 px image (shipped) |
|---|---|---|
| GPU, bf16 (AMD Instinct MI325X) | 14–24 ms | 29–31 ms |
| CPU, fp32, 16 threads | 0.21–0.29 s | 1.25–1.30 s |
Each question is one encoder pass over the image and the question. The shipped adapters read images at 1024 px (vela3v_config.json).
Benchmarks
- BoxSet-test v2 (the box test set d3-turbo reports on; 5,512 targets + 749 absent): point at a named thing in an image (mean IoU of the box; point-in-box: the point falls inside the true box).
- RVL-CDIP (test subset, 2,000 scanned documents, 16 types): name the type of a scanned document (accuracy).
- M-HalDetect (validation, 786 image descriptions): which words of an image description the image does not support (char-F1).
- Decision Index 0.3 (board): a public suite of 37 decision benchmarks (chance-corrected skill, 0 = guessing; here the public part, our run).
- Decision Index vision board 0.4: a public suite of 11 image decision benchmarks (CV-Bench, BLINK, RealWorldQA, CharXiv, InfographicVQA, Mind2Web, Winoground, KIE, Hateful Memes, R-Bench-M, MMMU-Pro vision; chance-corrected skill; here the public part, our run, without the gated Winoground).
The family
| Model | Size | Base | Best for |
|---|---|---|---|
| vela3-fast | 0.4B | UltraBERT-380M encoder | the default router; fastest: decisions + spans in one ~14 ms encoder pass, English, 32K |
| vela3-fast-vision (this model, preview) | 0.25B | UltraVBERT encoder | image decisions, image-grounded hallucination spans, coarse region pointing |
| vela3-lite | 0.8B | d3-lite | small GPUs, low latency |
| vela3-mini | 4B | d3-mini | balanced accuracy and speed |
| vela3-flash | 9B | d3-flash | high-accuracy routing |
| vela3-turbo | 27B | d3-turbo | the reference model |
How it works
- Three backbones. All are UltraVBERT (254M; revision
014dd536): SigLIP2 encodes each 512 px tile into patches, a connector merges them into 64 tokens per tile, and UltraBERT attends over image and text tokens together. The decision backbone (backbone_h2.safetensors) is fully trained as a decision model on d3 soft labels (text, images and d3-mini's answers on document images), on public datasets' train splits written as decisions and on synthetic decision tasks in the styles the Decision Index covers, labelled by d3 or by program and checked against every Decision Index item; three training runs were averaged. It answers every non-router choice and yes / no question, text or image, with its decision head. The region-pointing adapter runs on an earlier decision-trained backbone (backbone_v2.safetensors). The original backbone (model.safetensors, and its text encoder undertext/) stays frozen and carries the adapters for spans, the OCR check and the router signals. - Adapters. LoRA adapters, each with its own head: image-hallucination spans (the
unsupportedlabel over the answer's words), the OCR check, region pointing (a per-cell heat map read as a box and a point), and on the text side the router, span and NER adapters plus the domain and feedback task adapters. - Training. Soft targets and box data from Decision 3.0 (d3, d3-turbo), shared by the Decision 3.0 team; image-hallucination pools built on COCO images; synthetic documents for the OCR check. Every threshold and checkpoint was chosen on dev data.
Trade-offs
- Hallucination spans are below the larger Vela 3.0 sizes: 0.518 against 0.575 for vela3-mini on M-HalDetect. Use vela3-mini or vela3-flash where span quality matters most.
- Decisions have no reasoning skills: document type is 0.713 against d3-mini's zero-shot 0.752; chart, STEM and GUI questions are weak. Region answers are coarse (64 × 64 px cells of a 512 px tile), so small targets such as a date cover about one cell.
- Scope: English only; one image per request for span and box questions, up to three for decisions (more is an error, never dropped). A preview: no claim of state-of-the-art results.
Licence
MIT for this model's adapters, code and card. The base athrael-soju/UltraVBERT is MIT; its components dmis-lab/UltraBERT-150M and google/siglip2-base-patch16-512 are Apache-2.0 (LICENSE, LICENSE-APACHE).
Citation
@misc{vela3_fast_vision,
title = {vela3-fast-vision (Vela 3.0, 0.25B, preview): image decisions, image-grounded hallucination spans and region pointing},
author = {{KR Labs} and {vLLM Semantic Router}},
year = {2026},
howpublished = {\url{https://huggingface.co/vllm-sr/vela3-fast-vision}}
}
- Downloads last month
- 74