d3-ultra-NVFP4

d3-ultra-NVFP4 is d3-ultra, the newest 27B multimodal decision model of Decision 3.0, with its decoder's projections in NVFP4 for NVIDIA Blackwell GPUs. Give it an input (text or JSON, optionally with images and videos) and the questions you need answered: pick one of several options, say yes or no, rate on a scale, or point to where something is in an image or a video. It reads the input once per request, answers every question on it and returns a probability for every answer, as d3-ultra does, in 17.2 GB of weights instead of 52.2 GB and in about half the time.

Box loses 2.8 points and the text index 0.9. Box accuracy at IoU 0.5 is 0.844 against d3-ultra's 0.872, and most of the loss is on receipts (0.790 against 0.955). The text index is 70.63 against 71.57, most of it in knowledge (59.5 against 62.3). Images, images in options and tracking stay within 1.5 points. If Box answers on receipts or knowledge questions matter most, use d3-ultra-FP8.

Parameters	26.09B, including the 0.46B vision encoder
Base model	vllm-sr/d3-ultra at revision 52ba9e7a (the weights its Decision Index runs used)
Quantization	NVFP4 W4A4 (compressed-tensors nvfp4-pack-quantized): FP4 (E2M1) weights and activations in blocks of 16 with E4M3 scales under an FP32 global scale per tensor, from AWQ smoothing (activation-aware scales folded into the preceding norms and up_proj) followed by GPTQ rounding against the NVFP4-quantized activations. 400 projections; the Gated DeltaNet in_proj_a / in_proj_b, the vision tower, the embeddings and the readout stay as published, except where the scales were folded
Calibration data	175 requests (about 190k tokens), interleaved, mostly from d3-ultra's own training data: 80 text and JSON requests (1 to 32 questions, up to 170 options), 30 still images (2 of them with images as the options), 38 Box requests (6 with absent targets), 9 videos, 9 tracks and 9 Box questions at a video moment. None comes from a test below
Weights on disk	17.2 GB (d3-ultra: 52.2 GB)
Needs	FP4 tensor cores: NVIDIA Blackwell. Measured on one NVIDIA RTX PRO 6000 Blackwell
Inputs	Text or JSON, plus images and videos (several per request); images can also sit inside questions and options
Decision types	Choice · Yes / No · Score · Box · Track, all measured against d3-ultra
License	Apache-2.0

Highlights

• 70.63 on the Jev Decision Index 0.3 public suite against d3-ultra's 71.57: language, retrieval and tools within 0.3 points, arts −1.4, knowledge −2.8, with d3-ultra's top answer on 96.7% of 272,229 questions.
• 74.60 on the public vision suite against 75.36 for d3-ultra on the same GPU, with d3-ultra's top answer on 93.3% of 12,210 questions.
• Images in options and tracking within 1.5 points: images in questions and options 0.947 against 0.952, tracking success AUC 0.760 / 0.769 against 0.775 / 0.783. Box loses more (see the note above).
• 17.2 GB of weights instead of 52.2 GB, and about twice as fast: 32.5 ms against 63.4 ms at the median for one text question at a time on one NVIDIA RTX PRO 6000.

Quickstart

pip install "transformers==5.17.0" "compressed-tensors>=0.19" "torch>=2.11" torchvision triton pillow opencv-python-headless safetensors accelerate
pip install flash-linear-attention  # optional: fast GPU kernels for the linear-attention layers

import sys

from huggingface_hub import hf_hub_download, snapshot_download

path = snapshot_download("vllm-sr/d3-ultra-NVFP4")
sys.path.insert(0, path)
from d3q.load import load_d3

model = load_d3(path, device="cuda:0")


def asset(name):
    return hf_hub_download("vllm-sr/d3-ultra", f"assets/{name}")


# Text: several questions, one call
print(model.system_one(
    state="Ticket: the invoice total is wrong after the plan change.",
    questions={
        "route": {"type": "choice", "instructions": "Which team handles this?",
                  "criteria": {"billing": "Billing", "tech": "Technical support"}},
        "urgent": {"type": "noul", "instructions": "Is the customer asking for an urgent fix?"},
    },
))

# Box: who is in the canoe? Three questions, one read of the painting
print(model.system_one(
    state="A painting of fur traders on a river.",
    images=[asset("fur-traders.jpg")],  # local paths, http(s) URLs, PIL images or base64-encoded images
    questions={
        "cat": {"type": "box", "instructions": "The cat"},
        "canoe": {"type": "box", "instructions": "The canoe"},
        "dog": {"type": "box", "instructions": "A dog"},
    },
))

# Photo to etching: the question holds a photo, each option is an etching
print(model.system_one(
    state="An architecture quiz.",
    questions={"pick": {"type": "choice", "instructions": "Which 18th-century etching shows the same building as <building>?",
                        "criteria": {"a": "<a>", "b": "<b>", "c": "<c>", "d": "<d>"}}},
    assets={"<building>": asset("pantheon.jpg"), "<a>": asset("pantheon-a.jpg"), "<b>": asset("pantheon-b.jpg"),
            "<c>": asset("pantheon-c.jpg"), "<d>": asset("pantheon-d.jpg")},
))

load_d3 is d3-ultra's own runtime (d3_runtime.py) with its backbone loader swapped for d3q/: transformers loads the checkpoint with its FP4 weights still packed, and each quantized projection becomes a layer that quantizes its input to NVFP4 with one Triton kernel and multiplies on the FP4 tensor cores in torch.nn.functional.scaled_mm. Prompting, reading the input once, the FP32 readout, Box and Track answers, images and videos are d3-ultra's own; the answers are formatted as on d3-ultra's card. AutoModel.from_pretrained(..., trust_remote_code=True) alone dequantizes the whole checkpoint to BF16 and needs about 52 GB of GPU memory again.

Checked on one NVIDIA RTX PRO 6000 Blackwell with transformers 5.17.0 and compressed-tensors 0.19.0: the three requests above are answered as d3-ultra answers them through its own runtime. Every edge of the cat box is within 0.003 of d3-ultra's and of the canoe box within 0.013, the dog is not found by either (confidence 0.15 against 0.00), etching d gets 0.999 against 0.996, and the ticket goes to billing (urgent: 0.24 against 0.10).

Run with vLLM Semantic Router

The vLLM Semantic Router model runtime loads d3-ultra-NVFP4 as it loads d3-ultra, verifies the package against its MODEL_MANIFEST.json, and keeps the weights in NVFP4 on the GPU, multiplying on the FP4 tensor cores:

pip install -e "src/model-runtime[multimodal]" flash-linear-attention   # in a checkout of vllm-project/semantic-router
vllm-srun serve vllm-sr/d3-ultra-NVFP4 --revision becd81ec7ef5b2e16b2eec27519015f312a9238b --port 8899 --max-request-bytes 67108864

curl -s localhost:8899/v1/systemone -H 'content-type: application/json' -d '{
  "model": "vllm-sr/d3-ultra-NVFP4",
  "state": "Write a Python function that reverses a string.",
  "questions": {
    "kind": {"type": "choice", "instructions": "What kind of work is this?",
             "criteria": {"code": "Writing or fixing code", "math": "Mathematics", "chat": "Anything else"}},
    "reasoning": {"type": "noul", "instructions": "Does answering this need multi-step reasoning?"}
  }
}'

This package was served with the model runtime from main at b88ca01b (2026-10-29). A runtime that predates d3-ultra's prompt layout and per-request temperature refuses the package, since the model identity in its manifest differs. vllm-srun takes a Hub package that is not built in only at a full 40-hex --revision; requests with videos need --max-request-bytes above the default 8 MiB. Every result below comes from the model runtime run on a local copy of this repository (the same weights and loaded files).

One question per request on the tasksource test split below, one request at a time through the model runtime on one NVIDIA RTX PRO 6000 Blackwell: median 32.5 ms against 63.4 ms for d3-ultra.

Evaluation

Every comparison is against d3-ultra (BF16) on the same questions. The text index compares with d3-ultra's own public run (vllm-sr/d3-ultra-decision-index); images, Box, images in options, tracking and tasksource compare with d3-ultra run on the same RTX PRO 6000 through the same runtime. d3-ultra's vision run on this GPU gives 75.36 against 75.33 in its public run, so the differences below are the quantization's.

Text: Jev Decision Index 0.3, public suite

	d3-ultra (BF16)	d3-ultra-NVFP4
Decision Index (public)	71.57	70.63
Raw index	78.59	77.86
Knowledge	62.3	59.5
Language	72.4	72.1
Retrieval	71.8	71.7
Tools	87.8	87.8
Arts	63.5	62.1

The 110,087 requests (272,229 questions) of the 37 benchmarks the index reads, run with the official 0.3 kit through the model runtime and scored by it; re-scored by the same scorer, d3-ultra's public results give the 71.57 of its public run. This package gives d3-ultra's top answer on 96.73% of the index's questions (8,915 change); the lowest agreement is on ChessBench legal moves (68.6%), HLE text (71.3%) and GPQA Diamond (77.0%). KL(d3-ultra ‖ this package): mean 0.025, median 0.00039, 95th percentile 0.098, 99th percentile 0.47. d3-ultra's public run used another engine and GPU, so these differences include theirs as well; on tasksource below, with both packages on the same GPU and runtime, the agreement is 93.6%.

Images: Jev Decision Index 0.3, public vision suite

	d3-ultra (BF16)	d3-ultra-NVFP4
Public vision index	75.36	74.60

By benchmark (accuracy)

Benchmark	d3-ultra (BF16)	d3-ultra-NVFP4
CV-Bench	0.890	0.888
BLINK	0.811	0.802
RealWorldQA	0.817	0.808
CharXiv	0.852	0.865
InfographicVQA	0.975	0.959
Mind2Web	0.892	0.893
Winoground	0.920	0.922
KIE (CORD+FUNSD)	0.953	0.951
Moderation (Hateful Memes)	0.707	0.691
R-Bench-M	0.519	0.510
MMMU-Pro vision	0.506	0.488

All 12,210 rows, both packages through the same runtime on the same RTX PRO 6000. This package gives d3-ultra's top answer on 93.29% of the questions (819 change), most of the changes on MMMU-Pro vision (79.7% agreement) and R-Bench-M (80.5%), where d3-ultra itself is right about half the time; every other benchmark agrees on 94.1% (Moderation) to 99.0% (KIE). KL: mean 0.038, median 0.0019, 99th percentile 0.60.

Images: Box and images in options

Measure	d3-ultra (BF16)	d3-ultra-NVFP4
Box: accuracy at IoU 0.5	0.8717	0.8436
Box: accuracy at IoU 0.75	0.7077	0.6342
Box: confidence AUROC	0.9630	0.9550
Images in questions and options: accuracy	0.9517	0.9469

The Box test set of d3-turbo (5,836 items) and its images-in-options test set (2,752 questions), scored as d3-turbo's evaluation scores them. On Box this package hits or misses as d3-ultra does on 95.1% of the items (226 hits become misses, 62 misses become hits): −2.8 points at IoU 0.5 (95% interval −3.4 to −2.2), and at IoU 0.75 7.4 points lower (mean IoU 0.736 against 0.771). Receipts lose the most (0.955 → 0.790), then COCO spatial (0.653 → 0.633) and screens (0.999 → 0.992); COCO unique 0.801 → 0.796, exemplar 0.688 → 0.694, and charts, shapes and tables stay within 0.006. On images in options it is right or wrong as d3-ultra is on 98.1% of the questions (32 right answers become wrong, 19 wrong ones right): −0.5 points (95% interval −1.0 to 0.0).

Videos: tracking

Measure	d3-ultra (BF16)	d3-ultra-NVFP4
Track, 1 update per second: success AUC / at-time accuracy at IoU 0.5 / presence F1	0.775 / 0.886 / 0.972	0.760 / 0.876 / 0.973
Track, 2 updates per second: success AUC / at-time accuracy at IoU 0.5 / presence F1	0.783 / 0.887 / 0.969	0.769 / 0.879 / 0.975

The 96 tracking test clips of d3-turbo of at most 60 s (2,690 moments at rate 1, 5,377 at rate 2), sent as their own JPEG frames muxed into Matroska at the clip's frame rate without re-encoding. At rate 1, one moment of each of the three targets of one clip comes back 0.25 s off the test's moment for both packages. This package hits or misses as d3-ultra does on 97.7% and 97.3% of the moments with a visible target, and calls the target visible or not as d3-ultra does on 97.9% and 97.8%.

Text: tasksource-jev-typed-decisions, test split

Type	Requests	Top-1 agreement with d3-ultra	Accuracy, d3-ultra (BF16)	Accuracy, d3-ultra-NVFP4
Choice	11,033	93.47% (721 changed)	82.62%	81.78%
Yes / No	2,057	95.19% (99 changed)	88.53%	87.90%
Score	941	91.18% (83 changed)	74.39%	75.45%

Every usable row of the test split of tasksource/tasksource-jev-typed-decisions (a state of at most 6,000 characters, no repeated options): 14,031 requests of one question each, answered by the vLLM Semantic Router model runtime one at a time on one NVIDIA RTX PRO 6000 Blackwell. Agreement compares the top answer of every question with d3-ultra on the same requests (Choice and Score: the highest-probability option; Yes / No: p ≥ 0.5). An answer is correct when it is the request's target (Yes / No: the target's side of 0.5). Accuracy is given per type only, since the three types count a correct answer differently. KL(d3-ultra ‖ this package), mean / median: Choice 0.089 / 0.00043, Yes / No 0.036 / 0.00049, Score 0.076 / 0.0033.

How the package was chosen

Eight NVFP4 builds were made with the recipe of d3-turbo-NVFP4 (AWQ, then GPTQ): different calibration mixes (more Box, more screens and documents, more video), GPTQ against BF16 or against NVFP4-quantized activations, two GPTQ dampenings, and one with a clipped activation global scale, dropped after a quick check. Six of the other seven were run on d3-ultra's own SR-Box test and on the tracking test split above (the first build only on subsets of them), which kept the three builds with GPTQ against NVFP4 activations that lost least; this package was picked among those three on the mean and the worst change against d3-ultra across SR-Box, d3-ultra's SR-Text test, the public vision suite, and the tasksource, images-in-options and tracking test splits above (mean −0.69, worst −1.22 points). Unlike d3-ultra-INT4, it was chosen with test splits, not development data alone.

Afterwards every build was measured on the development data of d3-ultra-INT4's selection: the Box development split (480 items, 60 per family), the images-in-options development split (180 questions, 30 per family) and the top answer on 10,000 rows of the first train shard of tasksource-jev-typed-decisions.

Package	Box, dev split, IoU 0.5	Images in options, dev split	Agreement on the 10,000 train rows
d3-ultra (BF16)	0.896	0.950	100%
GPTQ against NVFP4 activations, more video (this package)	0.873	0.961	94.73%
The same, GPTQ dampening 3%	0.890	0.944	94.63%
GPTQ against NVFP4 activations, more Box and screens, no video	0.875	0.961	94.66%
GPTQ against NVFP4 activations, video	0.871	0.967	94.48%
GPTQ against BF16 activations, video	0.877	0.950	96.50%
GPTQ against BF16 activations, more Box and screens	0.863	0.950	96.17%
GPTQ against BF16 activations, the first calibration	0.850	0.956	96.32%

On these data the builds lose 0.6 to 4.6 points on the Box development split, change images in options by −0.6 to +1.7 points, and none reaches d3-ultra-INT4's 97.0% agreement rule on the train rows. GPTQ against NVFP4 activations costs about 1.8 points of that agreement against GPTQ against BF16 activations; it was kept for its SR-Box and tracking results on the tests above (SR-Box −1.2 points for this package against −3.0 for the best build with GPTQ against BF16 activations).

_{Built on 2026-10-30 by nvfp4_ultra.py, the recipe of vllm-sr/d3-turbo-NVFP4 with GPTQ against the quantized activations, on one NVIDIA RTX PRO 6000 Blackwell. Calibration: the 175 requests run through d3-ultra's own runtime as it answers them (its header-v1 layout for text, image and video tokens included): 80 text requests of d3-ultra's own training mix (1 to 32 questions per request, JSON states, up to 170 options); 28 still images with multiple-choice questions from its vision training rows (screens, documents, charts and infographics weighted up) and 2 with images as the options; 32 Box items of its Box training rows (UI screens, receipts, documents and small objects weighted up) and 6 absent targets; 9 Perception Test train videos, 9 TrackSet-dev tracks and 9 Box questions at a moment of those clips. The requests are interleaved so that the AWQ search, which reads the first 64, sees every kind. AWQ: per decoder layer and group of projections that read one input, the scale s = x^a / w^(1-a) searched over 20 values of a to minimise the layer's output error with that group run as NVFP4 weights times NVFP4 activations, then folded into input_layernorm, post_attention_layernorm and up_proj. GPTQ against the quantized activations: per group, with X its calibration input and Xq its NVFP4-quantized form at the group's input global scale, the weights are first moved to W' = W (Xᵀ Xq)(Xqᵀ Xq)⁻¹, for which W' Xq best matches W X, and W' is then rounded to E2M1 column by column with the Hessian of Xq (1% dampening, blocks of 128 columns), each 16-value block's E4M3 scale taken from its already-compensated weights. q / k / v, in_proj_qkv / in_proj_z and gate / up share their weight and input global scales, so the runtime's merged products apply. Evaluation runs, all on one NVIDIA RTX PRO 6000 Blackwell through the model runtime from main at b88ca01b at its default settings: the Decision Index text and vision suites with the official 0.3 kit; Box, images in options, tracking and tasksource with d3-turbo's evaluation scripts.}

License

Apache-2.0 (LICENSE), as d3-ultra. Built on vllm-sr/d3-ultra.

Citation

@misc{d3ultra_2026,
  title        = {{d3-ultra}: A Multimodal Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3-ultra}}
}