TheoHsiung commited on
Commit
90dfece
·
verified ·
1 Parent(s): 880ec9a

Card: Decision Index, public vision suite and Perception Test through the model runtime with NVFP4 support (64.83, 73.60, 76.57%)

Browse files
Files changed (1) hide show
  1. README.md +45 -14
README.md CHANGED
@@ -35,9 +35,9 @@ tags:
35
 
36
  ## Highlights
37
 
38
- - **Scores 64.72 on the Jev Decision Index 0.3 public suite** against d3's 65.45, giving d3's top answer on 96.96% of its 312,320 questions; every area within 1.11 points of d3.
39
  - **Half of d3's latency on a Blackwell GPU, a third of its memory:** 28.5 ms against 62.6 ms per request (median) on one RTX PRO 6000 in the vLLM Semantic Router model runtime with NVFP4 support and graphs replaying, in 17.2 GB of GPU memory instead of 52.2 GB.
40
- - **Reads images and videos** as d3 does: the vision tower is unchanged. The vision board's public suite and Perception Test: pending.
41
  - **tasksource-jev-typed-decisions test split** (14,031 requests): pending.
42
 
43
  ## Quickstart
@@ -151,18 +151,18 @@ The [vLLM Semantic Router](https://github.com/vllm-project/semantic-router) mode
151
 
152
  | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
153
  | --- | ---: | ---: | ---: |
154
- | Public index ↑ | 65.45 | **64.72 (−0.73)** | 96.96% |
155
- |   Knowledge & Reasoning | 55.26 | 54.15 (−1.11) | 93.09% |
156
- |   Language Understanding | 68.04 | 67.44 (−0.60) | 98.78% |
157
- |   Retrieval & Classification | 66.86 | 66.47 (−0.39) | 97.87% |
158
- |   Tools & Automation | 82.57 | 81.87 (−0.70) | 98.40% |
159
- |   Arts & Human Taste | 51.02 | 50.15 (−0.87) | 94.09% |
160
- | Raw index ↑ | 74.01 | 73.50 (−0.51) | – |
161
  | Requests answered | 140,178 | 140,178 | – |
162
 
163
  The five areas are the index's areas (chance-corrected skill; the public index averages them with fixed weights), with differences from d3 in parentheses. Agreement compares the top answer of every question with d3's official run (choice and score: the highest-probability option; yes / no: p ≥ 0.5): 312,320 questions in all, and per area the questions of that area's benchmarks.
164
 
165
- The complete public suite (every request of d3's official run), run with the official kit ([apolinario/decision-index](https://github.com/apolinario/decision-index) `f8552a1`) through `d3q/` and scored by it, against d3's official run in [vllm-sr/d3-decision-index](https://huggingface.co/datasets/vllm-sr/d3-decision-index); rescoring that run with the same kit reproduces 65.45 exactly. The board's full score (66.3 for d3) also needs private tests that only the maintainers run, so it is not measured here: compare with d3's public 65.45, not 66.3.
166
 
167
  Also on the test split of [tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions):
168
 
@@ -176,17 +176,48 @@ Every usable row of the split at revision `28c39125` (a state of at most 6,000 c
176
 
177
  | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
178
  | --- | ---: | ---: | ---: |
179
- | Public vision index ↑ | 74.15 | pending | pending |
180
 
181
- All 12,210 rows of the kit's public vision suite (11 benchmarks), one image request per row, through `d3q/` and scored by the official kit. d3 (BF16) answered the same rows the same way on an RTX PRO 6000 and scores 74.15, in line with the official vision board's 74.1.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
182
 
183
  ### Videos: Perception Test
184
 
185
  | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
186
  | --- | ---: | ---: | ---: |
187
- | All 19,140 questions (validation) ↑ | 77.73 | pending | pending |
 
 
 
 
 
 
 
 
 
 
 
 
188
 
189
- Every multiple-choice question of Perception Test validation (19,140 questions over 5,260 videos, three options each, chance 33.3): one request per video with all its questions and the state "A video.", videos read with d3's defaults, through `d3q/` on one RTX PRO 6000. d3 (BF16) answered the same requests the same way on the same machine and scores 77.73; d3's card reports 77.4 from its internal evaluation, whose prompt may differ.
190
 
191
  <sub>Quantized with llm-compressor 0.14.0 and compressed-tensors 0.19.0 (torch 2.11.0+cu128, transformers 5.17.0) on one GeForce RTX 5090, calibrating the text model layer by layer with the full-attention layers under the noncausal key mask of `d3_runtime.PromptEncoder`, as at inference. The calibration prompts are text only and none of their states appears in the Decision Index public suite. `QUANT_INFO.json` records the recipe, versions and calibration data. `d3q_engine.py` is the Decision Index engine for the kit (`--engine d3q_engine:D3QEngine --option model=<this folder>`).</sub>
192
 
 
35
 
36
  ## Highlights
37
 
38
+ - **Scores 64.83 on the Jev Decision Index 0.3 public suite** against d3's 65.45, giving d3's top answer on 96.95% of its 312,320 questions; every area within 1.17 points of d3.
39
  - **Half of d3's latency on a Blackwell GPU, a third of its memory:** 28.5 ms against 62.6 ms per request (median) on one RTX PRO 6000 in the vLLM Semantic Router model runtime with NVFP4 support and graphs replaying, in 17.2 GB of GPU memory instead of 52.2 GB.
40
+ - **Reads images and videos** as d3 does: the vision tower is unchanged. On the vision board's public suite it scores 73.60 against d3's 74.15 on the same machine (93.88% of d3's answers), and on Perception Test 76.57% against 77.73% (93.43%).
41
  - **tasksource-jev-typed-decisions test split** (14,031 requests): pending.
42
 
43
  ## Quickstart
 
151
 
152
  | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
153
  | --- | ---: | ---: | ---: |
154
+ | Public index ↑ | 65.45 | **64.83 (−0.62)** | 96.95% |
155
+ | &nbsp;&nbsp;Knowledge & Reasoning | 55.26 | 54.09 (−1.17) | 93.03% |
156
+ | &nbsp;&nbsp;Language Understanding | 68.04 | 67.55 (−0.49) | 98.75% |
157
+ | &nbsp;&nbsp;Retrieval & Classification | 66.86 | 66.46 (−0.40) | 97.90% |
158
+ | &nbsp;&nbsp;Tools & Automation | 82.57 | 82.21 (−0.36) | 98.35% |
159
+ | &nbsp;&nbsp;Arts & Human Taste | 51.02 | 50.50 (−0.52) | 94.15% |
160
+ | Raw index ↑ | 74.01 | 73.58 (−0.43) | – |
161
  | Requests answered | 140,178 | 140,178 | – |
162
 
163
  The five areas are the index's areas (chance-corrected skill; the public index averages them with fixed weights), with differences from d3 in parentheses. Agreement compares the top answer of every question with d3's official run (choice and score: the highest-probability option; yes / no: p ≥ 0.5): 312,320 questions in all, and per area the questions of that area's benchmarks.
164
 
165
+ The complete public suite (every request of d3's official run), run with the official kit ([apolinario/decision-index](https://github.com/apolinario/decision-index) `f8552a1`) through the vLLM Semantic Router model runtime with NVFP4 support (one request per call, graphs replaying) and scored by it, against d3's official run in [vllm-sr/d3-decision-index](https://huggingface.co/datasets/vllm-sr/d3-decision-index); rescoring that run with the same kit reproduces 65.45 exactly. Through `d3q/` (the Quickstart) the suite scores 64.72 with 96.96% agreement: its activation kernel rounds a few FP4 ties the other way (see **Answers** above). The board's full score (66.3 for d3) also needs private tests that only the maintainers run, so it is not measured here: compare with d3's public 65.45, not 66.3.
166
 
167
  Also on the test split of [tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions):
168
 
 
176
 
177
  | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
178
  | --- | ---: | ---: | ---: |
179
+ | Public vision index ↑ | 74.15 | **73.60 (−0.55)** | 93.88% |
180
 
181
+ <details>
182
+ <summary>By benchmark (chance-corrected skill, %)</summary>
183
+
184
+ | Benchmark | Questions | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
185
+ | --- | ---: | ---: | ---: | ---: |
186
+ | CV-Bench | 2,038 | 76.28 | 76.53 (+0.24) | 97.79% |
187
+ | BLINK | 961 | 60.05 | 57.79 (−2.26) | 95.53% |
188
+ | RealWorldQA | 589 | 73.36 | 70.03 (−3.33) | 95.08% |
189
+ | CharXiv | 532 | 79.45 | 79.70 (+0.25) | 94.74% |
190
+ | InfographicVQA | 244 | 97.81 | 97.27 (−0.55) | 98.77% |
191
+ | Mind2Web | 973 | 84.10 | 84.10 (+0.00) | 95.48% |
192
+ | Winoground | 800 | 85.75 | 86.50 (+0.75) | 98.88% |
193
+ | KIE (CORD+FUNSD) | 2,465 | 92.65 | 92.12 (−0.53) | 99.15% |
194
+ | Moderation (Hateful Memes) | 1,213 | 45.42 | 45.92 (+0.49) | 95.30% |
195
+ | R-Bench-M | 665 | 33.22 | 32.86 (−0.36) | 76.54% |
196
+ | MMMU-Pro vision | 1,730 | 45.84 | 44.77 (−1.06) | 81.97% |
197
+
198
+ </details>
199
+
200
+ All 12,210 rows of the kit's public vision suite (11 benchmarks), one image request per row, through the model runtime with NVFP4 support and scored by the official kit. d3 (BF16) answered the same rows through `d3q/` on the same RTX PRO 6000 and scores 74.15, in line with the official vision board's 74.1. Agreement compares the top answer of every question with that BF16 run.
201
 
202
  ### Videos: Perception Test
203
 
204
  | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
205
  | --- | ---: | ---: | ---: |
206
+ | All 19,140 questions (validation) ↑ | 77.73 | **76.57 (−1.16)** | 93.43% |
207
+
208
+ <details>
209
+ <summary>By area (accuracy, %)</summary>
210
+
211
+ | Area | Questions | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
212
+ | --- | ---: | ---: | ---: | ---: |
213
+ | Abstraction | 5,607 | 71.55 | 70.09 (−1.46) | 91.31% |
214
+ | Memory | 746 | 80.56 | 79.36 (−1.21) | 93.43% |
215
+ | Physics | 6,938 | 73.19 | 71.85 (−1.34) | 92.52% |
216
+ | Semantics | 5,849 | 88.66 | 88.02 (−0.65) | 96.53% |
217
+
218
+ </details>
219
 
220
+ Every multiple-choice question of Perception Test validation (19,140 questions over 5,260 videos, three options each, chance 33.3): one request per video with all its questions and the state "A video.", videos read as d3 reads them, through the model runtime with NVFP4 support on one RTX PRO 6000. d3 (BF16) answered the same requests through `d3q/` on the same machine and scores 77.73; d3's card reports 77.4 from its internal evaluation, whose prompt may differ.
221
 
222
  <sub>Quantized with llm-compressor 0.14.0 and compressed-tensors 0.19.0 (torch 2.11.0+cu128, transformers 5.17.0) on one GeForce RTX 5090, calibrating the text model layer by layer with the full-attention layers under the noncausal key mask of `d3_runtime.PromptEncoder`, as at inference. The calibration prompts are text only and none of their states appears in the Decision Index public suite. `QUANT_INFO.json` records the recipe, versions and calibration data. `d3q_engine.py` is the Decision Index engine for the kit (`--engine d3q_engine:D3QEngine --option model=<this folder>`).</sub>
223