Vishva007

/

d3-mini-W4A16-AutoRound

Like 0

Hi @D3vShoaib ,

Currently, d3-mini-W4A16-AutoRound (and the Decision 3.0 / d3 architecture) is not yet supported in stable vLLM releases.

However, support is right around the corner: an official pull request has already been opened and approved by maintainers on the vLLM repository to natively support Decision 3.0 checkpoints (vllm-sr/d3, d3-mini, d3-edge):
👉 vLLM PR #61565: https://github.com/vllm-project/vllm/pull/61565 ([Model] Support Decision 3.0 decision models (vllm-sr/d3))
Once merged into main, you will be able to serve it natively using the nightly builds.

Alternative Production Options Right Now

In the meantime, you can use Vishva007/clef-flash-W4A16-AutoRound, which is already supported and available in the nightly builds (vllm > v0.31.0 or vllm/vllm-openai:nightly) with full kernel acceleration, batching gains, and the /v1/systemone endpoint.

1. Launching the Server

docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai:nightly \
  vllm serve Vishva007/clef-flash-W4A16-AutoRound \
    --port 8000 --host 0.0.0.0 \
    --max-model-len 32k --max-num-batched-tokens 32k --max-num-seqs 128 \
    --kv-cache-dtype fp8 --gpu-memory-utilization 0.90 \
    --enable-prefix-caching --enable-chunked-prefill \
    --trust-remote-code --dtype bfloat16

2. Using Any LLM / VLM as a Decision Model

With the new vLLM nightly build, you can also use general LLMs/VLMs (such as Vishva007/Qwen3.5-4B-W4A16-AutoRound-NVFP4) as decision routers via the new /v1/systemone structured decisions endpoint:

from typesafe_sdk import Choice, TypeSafeClient

with TypeSafeClient(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",  # or your proxy URL
    model="Vishva007/Qwen3.5-4B-W4A16-AutoRound-NVFP4",
) as client:
    res = client.system_one(
        state="Checkout throwing 504 Gateway Timeouts. Database latency at 4,000ms.",
        questions={
            "severity": Choice(
                instructions="Assess severity",
                criteria={"sev1": "Total outage", "sev2": "Degraded", "sev3": "Minor"},
            ),
            "team": Choice(
                instructions="Route to team",
                criteria={"sre": "Infrastructure", "billing": "Payments"},
            ),
        },
    )

    for k, v in res.choices.items():
        print(f"{k}: {v.choice} ({v.confidence:.1%})")

Keep an eye on PR #61565 https://github.com/vllm-project/vllm/pull/61565 —as soon as it lands in vllm-openai:nightly, this exact repo will be ready to serve with identical batching and throughput gains!

Hi @D3vShoaib ,

Great news: the PR adding native support for Decision 3.0 architectures (vllm-sr/d3, d3-mini, d3-edge) has now been merged into main!

👉 Merged PR: [#61565] https://github.com/vllm-project/vllm/pull/61565 ([Model] Support Decision 3.0 decision models (vllm-sr/d3))

You can now run d3-mini-W4A16-AutoRound directly using the latest vllm/vllm-openai:nightly build (or by building from source off main).

1. Launching the Server

Pull the latest nightly image and serve the checkpoint directly:

docker pull vllm/vllm-openai:nightly

docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai:nightly \
  vllm serve Vishva007/d3-mini-W4A16-AutoRound \
    --port 8000 --host 0.0.0.0 \
    --max-model-len 32k --max-num-batched-tokens 32k --max-num-seqs 128 \
    --kv-cache-dtype fp8 --gpu-memory-utilization 0.90 \
    --enable-prefix-caching --enable-chunked-prefill \
    --trust-remote-code --dtype bfloat16

2. Querying the Decision Router (/v1/systemone)

With native support live, you can route decisions directly against d3-mini-W4A16-AutoRound:

from typesafe_sdk import Choice, TypeSafeClient

with TypeSafeClient(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    model="Vishva007/d3-mini-W4A16-AutoRound",
) as client:
    res = client.system_one(
        state="Checkout throwing 504 Gateway Timeouts. Database latency at 4,000ms.",
        questions={
            "severity": Choice(
                instructions="Assess severity",
                criteria={"sev1": "Total outage", "sev2": "Degraded", "sev3": "Minor"},
            ),
            "team": Choice(
                instructions="Route to team",
                criteria={"sre": "Infrastructure", "billing": "Payments"},
            ),
        },
    )

    for k, v in res.choices.items():
        print(f"{k}: {v.choice} ({v.confidence:.1%})")

Thanks alot man