| --- |
| license: apache-2.0 |
| base_model: vllm-sr/d3-edge |
| pipeline_tag: zero-shot-classification |
| tags: |
| - decision-model |
| - system-one |
| - fastflowlm |
| - amd-npu |
| - q4nx |
| --- |
| |
| # D3-Edge-NPU2 |
|
|
| [vllm-sr/d3-edge](https://huggingface.co/vllm-sr/d3-edge) (Decision 3.0, the decision model of vLLM Semantic |
| Router; Qwen3.5-0.8B architecture cut to 12 layers, revision `c454ee39`) for AMD Ryzen AI NPUs, served by |
| [FastFlowLM](https://github.com/ROCm/FastFlowLM) 1.0.12 or later. It answers *System One* requests -- choice, yes / no |
| and score questions about a state, optionally with images, with a probability for every option -- over the same |
| `POST /v1/systemone` API as the upstream package. Up to 32,768 input tokens per question. Video is not supported (a |
| request with `videos` is refused with HTTP 422). |
|
|
| ```shell |
| flm serve --decision d3:edge # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:edge |
| curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \ |
| -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice", |
| "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}' |
| ``` |
|
|
| Images go in an `images` list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most |
| 1.6 MP), shared by every question of the request. `d3:lite` (0.8B), `d3:nano` (2B), `d3:mini` (4B) and `d3:flash` (9B) |
| are the larger siblings. |
|
|
| ## What was changed |
|
|
| - **Weights:** the 12-layer text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with |
| GPTQ-calibrated rounding (calibration: 192 synthetic System One prompts, none from the Decision Index), token |
| embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. `vision_weight.q4nx` holds the vision tower |
| (Qwen3.5-0.8B's, merger included), used for requests with images. |
| - **Unchanged:** `readout.safetensors`, `decision_config.json`, `tokenizer.json`, `tokenizer_config.json`, |
| `chat_template.jinja` and `LICENSE` are the upstream files byte for byte (`MANIFEST.sha256`). |
| - **Same contract:** prompts, token ids and answer objects follow the upstream runtime. |
|
|
| ## Accuracy (against the upstream BF16 model) |
|
|
| | | this package on the NPU | upstream BF16 | |
| |---|---|---| |
| | Decision Index public-suite sample, mean benchmark score (1,600 requests, 37 benchmarks) | 49.8% | 47.2% | |
| | Top-1 agreement with BF16 on the same requests (4,452 choice answers) | 93.8% | -- | |
| | Long prompts, 835 to 31,545 tokens (8 requests), same top answer as BF16 | 6 of 8 | -- | |
| | Decision Index public vision suite subset, accuracy (300 questions, 10 benchmarks) | 53.0% | 55.7% | |
| | Top-1 agreement with BF16 on the same image questions | 82.3% | -- | |
|
|
| Mean benchmark score, this package minus BF16: +2.6 pt (standard error 1.1 pt; about 42 requests per benchmark, so |
| single benchmarks are noisy), i.e. no accuracy loss is measurable. Answers agree with BF16 less often than for the larger |
| sizes (93.8%; plain Q4_K without GPTQ: 91.9%): a small model has many near-tied options, and the two long prompts that |
| differ are such near-ties. The sample is a seeded round-robin over the public suite's requests whose prompts are at most |
| 8,000 tokens (37 of its 40 benchmarks have such requests); RouterBench and SGD requests are left out because their |
| locally rebuilt payloads differ from the lab's. The BF16 scores are the upstream BF16 run's own answers on the same |
| requests. The vision subset is 30 seeded questions from each of 10 benchmarks (the suite without the gated Winoground); |
| paired accuracy, BF16 minus this package: +2.7 pt (95% CI -0.7 to +6.0 pt), so images may cost a little accuracy. |
| |
| ## Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.12) |
| |
| | request | median latency | |
| |---|---| |
| | 1 question, 128 tokens | 60 ms | |
| | 1 question, 330 tokens | 83 ms | |
| | 1 question, 1,024 tokens | 0.26 s | |
| | 1 question, 4,096 tokens | 1.0 s | |
| | 1 question, 16,384 tokens | 6.0 s | |
| | 1 question, 31,545 tokens | 16 s | |
| | 1 question, 32,767 tokens | 17 s | |
| | 4 questions, 330 tokens each | 0.29 s | |
| | 16 questions, 330 tokens each | 1.0 s | |
| | 1 question + a 1024 x 768 image (880 tokens, HTTP) | 0.61 s | |
| | 1 question + two images (HTTP) | 0.89 s | |
| | 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 1.4 s | |
| | 8 questions + one image (one image encoding, HTTP) | 2.0 s | |
| |
| From 1,536 tokens attention runs on the NPU's attention kernel. Prompts over 32,768 tokens are answered with |
| `max_length_exceeded` (nothing is truncated). Resident memory of the server: 2.0 GB after loading, up to 4.8 GB at |
| 32,768 tokens with images in the mix. Median of 3 (16,384 tokens and longer: 1 to 2 runs) over HTTP on an otherwise idle |
| machine. |
| |
| ## License |
| |
| Apache-2.0, as the upstream model; see `LICENSE` and `NOTICE`. Built on Qwen/Qwen3.5-0.8B. |
| |
