vllm-sr
/

d3-NVFP4

@@ -38,7 +38,7 @@ tags:
 - **Scores 64.83 on the Jev Decision Index 0.3 public suite** against d3's 65.45, giving d3's top answer on 96.95% of its 312,320 questions; every area within 1.17 points of d3.
 - **Half of d3's latency on a Blackwell GPU, a third of its memory:** 28.5 ms against 62.6 ms per request (median) on one RTX PRO 6000 in the vLLM Semantic Router model runtime with NVFP4 support and graphs replaying, in 17.2 GB of GPU memory instead of 52.2 GB.
 - **Reads images and videos** as d3 does: the vision tower is unchanged. On the vision board's public suite it scores 73.60 against d3's 74.15 on the same machine (93.88% of d3's answers), and on Perception Test 76.57% against 77.73% (93.43%).
-- **tasksource-jev-typed-decisions test split** (14,031 requests): pending.
 ## Quickstart
@@ -168,9 +168,9 @@ Also on the test split of [tasksource/tasksource-jev-typed-decisions](https://hu
 | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
 | --- | ---: | ---: | ---: |
-| Accuracy | 83.25% | pending | pending |
-Every usable row of the split at revision `28c39125` (a state of at most 6,000 characters, no repeated options): 14,031 requests of one question each; d3's accuracy is from the vLLM Semantic Router model runtime on one AMD Instinct MI300X.
 ### Images: Jev Decision Index vision board 0.3.2

 - **Scores 64.83 on the Jev Decision Index 0.3 public suite** against d3's 65.45, giving d3's top answer on 96.95% of its 312,320 questions; every area within 1.17 points of d3.
 - **Half of d3's latency on a Blackwell GPU, a third of its memory:** 28.5 ms against 62.6 ms per request (median) on one RTX PRO 6000 in the vLLM Semantic Router model runtime with NVFP4 support and graphs replaying, in 17.2 GB of GPU memory instead of 52.2 GB.
 - **Reads images and videos** as d3 does: the vision tower is unchanged. On the vision board's public suite it scores 73.60 against d3's 74.15 on the same machine (93.88% of d3's answers), and on Perception Test 76.57% against 77.73% (93.43%).
+- **Keeps 96.64% of d3's answers** on the 14,031 test requests of tasksource-jev-typed-decisions, with accuracy 82.71% against d3's 83.23%.
 ## Quickstart
 | | d3 (BF16) | d3-NVFP4 | Top-1 agreement with d3 |
 | --- | ---: | ---: | ---: |
+| Accuracy | 83.23% | **82.71 (−0.52)%** | 96.64% (472 of 14,031 answers changed) |
+Every usable row of the split at revision `28c39125` (a state of at most 6,000 characters, no repeated options): 14,031 requests of one question each, answered by the vLLM Semantic Router model runtime with NVFP4 support on one RTX PRO 6000, d3 (BF16) by the same runtime on the same machine (83.25% on one AMD Instinct MI300X in the #5066 record). Agreement compares the top answer of every question with that BF16 run (choice and score: the highest-probability option; yes / no: p ≥ 0.5). Two calibration states, an LSAT logic-game passage and a news article, are also the states of 2 of these 14,031 requests.
 ### Images: Jev Decision Index vision board 0.3.2