Zero-Shot Classification
Transformers
Safetensors
qwen3_5
feature-extraction
decision-model
classification
system-one
multimodal
vision
video
quantized
compressed-tensors
nvfp4
custom_code
8-bit precision
Instructions to use vllm-sr/d3-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vllm-sr/d3-NVFP4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("zero-shot-classification", model="vllm-sr/d3-NVFP4", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModel processor = AutoProcessor.from_pretrained("vllm-sr/d3-NVFP4", trust_remote_code=True) model = AutoModel.from_pretrained("vllm-sr/d3-NVFP4", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Card: Decision Index, public vision suite and Perception Test through the model runtime with NVFP4 support (64.83, 73.60, 76.57%)
Browse files
README.md
CHANGED
|
@@ -35,9 +35,9 @@ tags:
|
|
| 35 |
|
| 36 |
## Highlights
|
| 37 |
|
| 38 |
-
- **Scores 64.
|
| 39 |
- **Half of d3's latency on a Blackwell GPU, a third of its memory:** 28.5 ms against 62.6 ms per request (median) on one RTX PRO 6000 in the vLLM Semantic Router model runtime with NVFP4 support and graphs replaying, in 17.2 GB of GPU memory instead of 52.2 GB.
|
| 40 |
-
- **Reads images and videos** as d3 does: the vision tower is unchanged.
|
| 41 |
- **tasksource-jev-typed-decisions test split** (14,031 requests): pending.
|
| 42 |
|
| 43 |
## Quickstart
|
|
@@ -151,18 +151,18 @@ The [vLLM Semantic Router](https://github.com/vllm-project/semantic-router) mode
|
|
| 151 |
|
| 152 |
| | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 153 |
| --- | ---: | ---: | ---: |
|
| 154 |
-
| Public index ↑ | 65.45 | **64.
|
| 155 |
-
| Knowledge & Reasoning | 55.26 | 54.
|
| 156 |
-
| Language Understanding | 68.04 | 67.
|
| 157 |
-
| Retrieval & Classification | 66.86 | 66.
|
| 158 |
-
| Tools & Automation | 82.57 |
|
| 159 |
-
| Arts & Human Taste | 51.02 | 50.
|
| 160 |
-
| Raw index ↑ | 74.01 | 73.
|
| 161 |
| Requests answered | 140,178 | 140,178 | – |
|
| 162 |
|
| 163 |
The five areas are the index's areas (chance-corrected skill; the public index averages them with fixed weights), with differences from d3 in parentheses. Agreement compares the top answer of every question with d3's official run (choice and score: the highest-probability option; yes / no: p ≥ 0.5): 312,320 questions in all, and per area the questions of that area's benchmarks.
|
| 164 |
|
| 165 |
-
The complete public suite (every request of d3's official run), run with the official kit ([apolinario/decision-index](https://github.com/apolinario/decision-index) `f8552a1`) through
|
| 166 |
|
| 167 |
Also on the test split of [tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions):
|
| 168 |
|
|
@@ -176,17 +176,48 @@ Every usable row of the split at revision `28c39125` (a state of at most 6,000 c
|
|
| 176 |
|
| 177 |
| | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 178 |
| --- | ---: | ---: | ---: |
|
| 179 |
-
| Public vision index ↑ | 74.15 |
|
| 180 |
|
| 181 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 182 |
|
| 183 |
### Videos: Perception Test
|
| 184 |
|
| 185 |
| | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 186 |
| --- | ---: | ---: | ---: |
|
| 187 |
-
| All 19,140 questions (validation) ↑ | 77.73 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 188 |
|
| 189 |
-
Every multiple-choice question of Perception Test validation (19,140 questions over 5,260 videos, three options each, chance 33.3): one request per video with all its questions and the state "A video.", videos read
|
| 190 |
|
| 191 |
<sub>Quantized with llm-compressor 0.14.0 and compressed-tensors 0.19.0 (torch 2.11.0+cu128, transformers 5.17.0) on one GeForce RTX 5090, calibrating the text model layer by layer with the full-attention layers under the noncausal key mask of `d3_runtime.PromptEncoder`, as at inference. The calibration prompts are text only and none of their states appears in the Decision Index public suite. `QUANT_INFO.json` records the recipe, versions and calibration data. `d3q_engine.py` is the Decision Index engine for the kit (`--engine d3q_engine:D3QEngine --option model=<this folder>`).</sub>
|
| 192 |
|
|
|
|
| 35 |
|
| 36 |
## Highlights
|
| 37 |
|
| 38 |
+
- **Scores 64.83 on the Jev Decision Index 0.3 public suite** against d3's 65.45, giving d3's top answer on 96.95% of its 312,320 questions; every area within 1.17 points of d3.
|
| 39 |
- **Half of d3's latency on a Blackwell GPU, a third of its memory:** 28.5 ms against 62.6 ms per request (median) on one RTX PRO 6000 in the vLLM Semantic Router model runtime with NVFP4 support and graphs replaying, in 17.2 GB of GPU memory instead of 52.2 GB.
|
| 40 |
+
- **Reads images and videos** as d3 does: the vision tower is unchanged. On the vision board's public suite it scores 73.60 against d3's 74.15 on the same machine (93.88% of d3's answers), and on Perception Test 76.57% against 77.73% (93.43%).
|
| 41 |
- **tasksource-jev-typed-decisions test split** (14,031 requests): pending.
|
| 42 |
|
| 43 |
## Quickstart
|
|
|
|
| 151 |
|
| 152 |
| | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 153 |
| --- | ---: | ---: | ---: |
|
| 154 |
+
| Public index ↑ | 65.45 | **64.83 (−0.62)** | 96.95% |
|
| 155 |
+
| Knowledge & Reasoning | 55.26 | 54.09 (−1.17) | 93.03% |
|
| 156 |
+
| Language Understanding | 68.04 | 67.55 (−0.49) | 98.75% |
|
| 157 |
+
| Retrieval & Classification | 66.86 | 66.46 (−0.40) | 97.90% |
|
| 158 |
+
| Tools & Automation | 82.57 | 82.21 (−0.36) | 98.35% |
|
| 159 |
+
| Arts & Human Taste | 51.02 | 50.50 (−0.52) | 94.15% |
|
| 160 |
+
| Raw index ↑ | 74.01 | 73.58 (−0.43) | – |
|
| 161 |
| Requests answered | 140,178 | 140,178 | – |
|
| 162 |
|
| 163 |
The five areas are the index's areas (chance-corrected skill; the public index averages them with fixed weights), with differences from d3 in parentheses. Agreement compares the top answer of every question with d3's official run (choice and score: the highest-probability option; yes / no: p ≥ 0.5): 312,320 questions in all, and per area the questions of that area's benchmarks.
|
| 164 |
|
| 165 |
+
The complete public suite (every request of d3's official run), run with the official kit ([apolinario/decision-index](https://github.com/apolinario/decision-index) `f8552a1`) through the vLLM Semantic Router model runtime with NVFP4 support (one request per call, graphs replaying) and scored by it, against d3's official run in [vllm-sr/d3-decision-index](https://huggingface.co/datasets/vllm-sr/d3-decision-index); rescoring that run with the same kit reproduces 65.45 exactly. Through `d3q/` (the Quickstart) the suite scores 64.72 with 96.96% agreement: its activation kernel rounds a few FP4 ties the other way (see **Answers** above). The board's full score (66.3 for d3) also needs private tests that only the maintainers run, so it is not measured here: compare with d3's public 65.45, not 66.3.
|
| 166 |
|
| 167 |
Also on the test split of [tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions):
|
| 168 |
|
|
|
|
| 176 |
|
| 177 |
| | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 178 |
| --- | ---: | ---: | ---: |
|
| 179 |
+
| Public vision index ↑ | 74.15 | **73.60 (−0.55)** | 93.88% |
|
| 180 |
|
| 181 |
+
<details>
|
| 182 |
+
<summary>By benchmark (chance-corrected skill, %)</summary>
|
| 183 |
+
|
| 184 |
+
| Benchmark | Questions | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 185 |
+
| --- | ---: | ---: | ---: | ---: |
|
| 186 |
+
| CV-Bench | 2,038 | 76.28 | 76.53 (+0.24) | 97.79% |
|
| 187 |
+
| BLINK | 961 | 60.05 | 57.79 (−2.26) | 95.53% |
|
| 188 |
+
| RealWorldQA | 589 | 73.36 | 70.03 (−3.33) | 95.08% |
|
| 189 |
+
| CharXiv | 532 | 79.45 | 79.70 (+0.25) | 94.74% |
|
| 190 |
+
| InfographicVQA | 244 | 97.81 | 97.27 (−0.55) | 98.77% |
|
| 191 |
+
| Mind2Web | 973 | 84.10 | 84.10 (+0.00) | 95.48% |
|
| 192 |
+
| Winoground | 800 | 85.75 | 86.50 (+0.75) | 98.88% |
|
| 193 |
+
| KIE (CORD+FUNSD) | 2,465 | 92.65 | 92.12 (−0.53) | 99.15% |
|
| 194 |
+
| Moderation (Hateful Memes) | 1,213 | 45.42 | 45.92 (+0.49) | 95.30% |
|
| 195 |
+
| R-Bench-M | 665 | 33.22 | 32.86 (−0.36) | 76.54% |
|
| 196 |
+
| MMMU-Pro vision | 1,730 | 45.84 | 44.77 (−1.06) | 81.97% |
|
| 197 |
+
|
| 198 |
+
</details>
|
| 199 |
+
|
| 200 |
+
All 12,210 rows of the kit's public vision suite (11 benchmarks), one image request per row, through the model runtime with NVFP4 support and scored by the official kit. d3 (BF16) answered the same rows through `d3q/` on the same RTX PRO 6000 and scores 74.15, in line with the official vision board's 74.1. Agreement compares the top answer of every question with that BF16 run.
|
| 201 |
|
| 202 |
### Videos: Perception Test
|
| 203 |
|
| 204 |
| | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 205 |
| --- | ---: | ---: | ---: |
|
| 206 |
+
| All 19,140 questions (validation) ↑ | 77.73 | **76.57 (−1.16)** | 93.43% |
|
| 207 |
+
|
| 208 |
+
<details>
|
| 209 |
+
<summary>By area (accuracy, %)</summary>
|
| 210 |
+
|
| 211 |
+
| Area | Questions | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
|
| 212 |
+
| --- | ---: | ---: | ---: | ---: |
|
| 213 |
+
| Abstraction | 5,607 | 71.55 | 70.09 (−1.46) | 91.31% |
|
| 214 |
+
| Memory | 746 | 80.56 | 79.36 (−1.21) | 93.43% |
|
| 215 |
+
| Physics | 6,938 | 73.19 | 71.85 (−1.34) | 92.52% |
|
| 216 |
+
| Semantics | 5,849 | 88.66 | 88.02 (−0.65) | 96.53% |
|
| 217 |
+
|
| 218 |
+
</details>
|
| 219 |
|
| 220 |
+
Every multiple-choice question of Perception Test validation (19,140 questions over 5,260 videos, three options each, chance 33.3): one request per video with all its questions and the state "A video.", videos read as d3 reads them, through the model runtime with NVFP4 support on one RTX PRO 6000. d3 (BF16) answered the same requests through `d3q/` on the same machine and scores 77.73; d3's card reports 77.4 from its internal evaluation, whose prompt may differ.
|
| 221 |
|
| 222 |
<sub>Quantized with llm-compressor 0.14.0 and compressed-tensors 0.19.0 (torch 2.11.0+cu128, transformers 5.17.0) on one GeForce RTX 5090, calibrating the text model layer by layer with the full-attention layers under the noncausal key mask of `d3_runtime.PromptEncoder`, as at inference. The calibration prompts are text only and none of their states appears in the Decision Index public suite. `QUANT_INFO.json` records the recipe, versions and calibration data. `d3q_engine.py` is the Decision Index engine for the kit (`--engine d3q_engine:D3QEngine --option model=<this folder>`).</sub>
|
| 223 |
|