d3-nano-GGUF

d3-nano is a 2.21B-parameter multimodal decision model from the vLLM Semantic Router team, including a 0.33B vision encoder. Built on Qwen3.5-2B and released under Apache-2.0, it accepts text or JSON, optionally with images and videos, and answers multiple Choice, Yes/No, or Score questions in one call, returning probabilities without generating text. It scores 38.9 on the official Jev Decision Index 0.3.2 text board—first among similarly sized entries and on the Pareto frontier—and 54.7 on the vision board; its Perception Test video score is 63.5. On one AMD Instinct MI325X, median latency is 13.4 ms for text, 90.2 ms for an image request, and 301 ms for a 10-second video; vLLM Semantic Router with batching serves up to 318 one-question text requests per second. It can be used with Hugging Face Transformers or served through vLLM Semantic Router Engine mode.

This is a Decision Model GGUF that supports the llama.cpp v1/systemone API endpoint.

Model Files

File Name	Quant Type	File Size	File Link	Description
d3-nano.BF16.gguf	BF16	4.79 GB	Link	Full BF16 weights. Highest quality, largest file size.
d3-nano.Q3_K_L.gguf	Q3_K_L	1.38 GB	Link	Lower quality but usable, good for low RAM availability.
d3-nano.Q3_K_M.gguf	Q3_K_M	1.32 GB	Link	Low quality.
d3-nano.Q4_K_M.gguf	Q4_K_M	1.56 GB	Link	Good quality, default size for most use cases, recommended.
d3-nano.Q4_K_S.gguf	Q4_K_S	1.5 GB	Link	Slightly lower quality with more space savings, recommended.
d3-nano.Q5_K_M.gguf	Q5_K_M	1.76 GB	Link	High quality, recommended.
d3-nano.Q5_K_S.gguf	Q5_K_S	1.72 GB	Link	High quality, recommended.
d3-nano.Q6_K.gguf	Q6_K	1.97 GB	Link	Very high quality, near perfect, recommended.
d3-nano.Q8_0.gguf	Q8_0	2.55 GB	Link	Extremely high quality, generally unneeded but max available quant.
d3-nano.mmproj-bf16.gguf	mmproj-bf16	671 MB	Link	Multimodal projection file in BF16 format. Used for vision/language models.

Run with llama.cpp (SystemOne API)

1. Start the Server

Launch llama-server with the GGUF model and wait for the health check endpoint to report ready.

import subprocess, time, requests

MODEL_PATH = "/data/llama.cpp/gguf-new/d3-nano.BF16.gguf"
BASE_URL = "http://127.0.0.1:8080"

server = subprocess.Popen(
    [
        "/data/llama.cpp/build/bin/llama-server",
        "-m", MODEL_PATH,
        "--host", "127.0.0.1",
        "--port", "8080",
        "-ngl", "99",
    ],
    stdout=open("/data/llama-server.log", "w"),
    stderr=subprocess.STDOUT,
)

for _ in range(120):
    try:
        if requests.get(f"{BASE_URL}/health", timeout=2).status_code == 200:
            print("server ready")
            break
    except requests.exceptions.RequestException:
        pass
    time.sleep(2)
else:
    print("server not ready, check /data/llama-server.log")

2. Query the /v1/systemone Endpoint

Send a POST request with a state context and structured questions (choice, noul, or score).

curl -s http://127.0.0.1:8080/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
    "questions": {
      "route": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "payments, charges, refunds, invoices",
          "shipping": "delivery, tracking, lost or late parcels",
          "technical": "bugs, errors, login problems"
        }
      },
      "angry": {
        "type": "noul",
        "instructions": "Is the customer angry?"
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this?",
        "criteria": ["can wait", "this week", "today", "right now"]
      }
    }
  }' | python3 -m json.tool

3. Example Response

{
    "model": "/data/llama.cpp/gguf-new/d3-nano.BF16.gguf",
    "answers": {
        "route": {
            "type": "choice",
            "choice": "billing",
            "probabilities": {
                "billing": 0.9996550794867408,
                "shipping": 0.0001900160453319817,
                "technical": 0.00015490446792721938
            },
            "confidence": 0.9969671505983558
        },
        "angry": {
            "type": "noul",
            "noul": 0.9746957167505704
        },
        "urgency": {
            "type": "score",
            "score": 2.4617961533870094,
            "legend": {
                "0": "can wait",
                "1": "this week",
                "2": "today",
                "3": "right now"
            },
            "probabilities": {
                "0": 0.06293201172016659,
                "1": 0.11729182460596137,
                "2": 0.11482416224056817,
                "3": 0.7049520014333038
            },
            "confidence": 0.3360664431382385
        }
    },
    "usage": {
        "input_tokens": 350,
        "output_tokens": 0
    }
}

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

Quantized using llama.cpp by prithivMLmods.