Card: first-start note, same-machine vision baseline, Typed Decisions note; drop unmeasured throughput line

MODEL_MANIFEST.json

@@ -583,7 +583,7 @@
   ".gitattributes": "690c76d1fc25384e5e24b807be13bec4064574d8a6ff53c7f25827a2649e363a",
   "LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
   "NOTICE": "61f4ea488e1fed0c3fce97171506a23152c2591116442f78f15827fb9e6af469",
-  "README.md": "acbb02d15bede896ae9291e03adac85f861034fbd8027b787911e3fd6bfe6b36",
+  "README.md": "d9e9709d04f51f5de83306589f59033b197af3336579244a9919015cd383c239",
   "assets/banner.png": "eb7ecc949232e1de544247899dd55ee629a36cfe276042d5795f65178b9c64c5",
   "assets/example-receipt.png": "b9dbb8b103bacd45f1de824d844a3c07b81d57cb1a7466712ef060ad7dcb3926",
   "assets/example-video.mp4": "f4968ef89d07e97f20df4fd8047510a7a8e9df5c2d07c35e363a97927f164ab9",
@@ -599,7 +599,7 @@
   "d3_server.py": "766a20a69101e36ab0535a1ec683379db422380a83091f0eccd7e82edb6a02de",
   "d3q/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
   "d3q/kernels.py": "cfcb40ca9f5a3cb22b9a6420a2baaa13fca0788d7308680b92a3701cbc1090f7",
-  "d3q/load.py": "04d7f44a341f574b127dad24d724c108f2a64769c066755f391b1179266b7391",
+  "d3q/load.py": "24b98d8043bf38730706f4ceeac8e992d237050f8f0dcb4c35b256e495d67b89",
   "d3q_engine.py": "4f11084fafc4c5a668302bea66895ef02113a210523009a3d60dd66bfc8d72a2",
   "decision_config.json": "6b80ca11bd6ba481df4b3c187db8a5d1983786563ca05e2edccb0bdac0d2fc4f",
   "merges.txt": "a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d",
@@ -618,7 +618,7 @@
   ".gitattributes": 1743,
   "LICENSE": 11357,
   "NOTICE": 100,
-  "README.md": 17441,
+  "README.md": 18076,
   "assets/banner.png": 2233137,
   "assets/example-receipt.png": 65918,
   "assets/example-video.mp4": 26300,
@@ -634,7 +634,7 @@
   "d3_server.py": 17705,
   "d3q/__init__.py": 0,
   "d3q/kernels.py": 9779,
-  "d3q/load.py": 1898,
+  "d3q/load.py": 1987,
   "d3q_engine.py": 4783,
   "decision_config.json": 5503,
   "merges.txt": 3353259,

README.md

@@ -39,9 +39,10 @@ tags:
 ## Highlights
 
 - **Scores 65.40 on the Jev Decision Index 0.3 public suite** against d3's 65.45, giving d3's top answer on 99.15% of its 312,320 questions; every area within 0.3 points of d3.
+- **Keeps 99.08% of d3's answers** on the 14,031 test requests of tasksource-jev-typed-decisions, with accuracy 83.24% against d3's 83.25%.
 - **31.9 GB of GPU memory for the weights instead of 52.2 GB** with the vLLM Semantic Router model runtime, on one AMD Instinct MI300X and on one NVIDIA RTX PRO 6000 alike.
 - **0.72× d3's latency on one RTX PRO 6000 and 0.79× on one MI300X:** 45.0 ms against 62.1 ms and 36.8 ms against 46.4 ms per request (median) in the model runtime, both in one process, with the forward replayed as a graph ([#4999](https://github.com/vllm-project/semantic-router/pull/4999)). Without graphs the MI300X takes longer than for d3: each layer's activation quantization costs more host time than FP8 saves.
-- **Reads images and videos** as d3 does: the vision tower is unchanged. On the vision board's public suite it scores 74.03 against d3's 74.15 and gives d3's top answer on 98.19% of its 12,210 questions.
+- **Reads images and videos** as d3 does: the vision tower is unchanged. On the vision board's public suite it scores 74.03 against d3's 74.15 (73.93 for d3 on the same MI300X) and gives d3's top answer on 98.19% of its 12,210 questions.
 - **Perception Test:** 77.57% on all 19,140 validation questions, the same as d3 on the same machine, giving d3's top answer on 98.10% of them.
 
 ## Quickstart
@@ -136,6 +137,8 @@ vllm-sr serve -e vllm-sr/d3-FP8
 
 The platform is detected automatically; choose one with `--platform auto|cpu|cuda|rocm` (default `auto`), and a commit with `--revision`. FP8 needs a GPU with FP8 matrix units (see **Needs** above). Then ask questions about a request:
 
+The first start downloads the package (about 32 GB) before loading it; if it stops with `did not become ready within 10m0s`, run the same command again: the download is kept and the second start only loads.
+
 ```bash
 curl -s localhost:8899/v1/systemone -H 'content-type: application/json' -d '{
   "model": "vllm-sr/d3-FP8",
@@ -169,7 +172,6 @@ Images and videos go in `images` and `videos` as for d3. `vllm-sr stop` stops it
 
 - **Graphs:** on the MI300X the gain needs the forward replayed as a graph ([#4999](https://github.com/vllm-project/semantic-router/pull/4999)). Without graphs the same requests take 80.9 ms against d3's 67.1 ms: the activation quantization and the scaled GEMM of each of the 304 FP8 layers cost more host time than FP8 saves. On the RTX PRO 6000 graphs change little, for d3 and for this package alike, and FP8 is faster either way.
 - **Kernels:** one FP8 projection with its activation quantization takes 0.61× to 0.74× the time of the BF16 GEMM at 64 to 1,024 rows on the MI300X. On an NVIDIA Ada GPU the runtime takes the FP8 product unscaled and applies the scales after, since the per-row scaled product is slower than BF16 there from 256 rows on.
-- **Throughput under load:** not measured yet.
 
 <sub>Latency and memory measured on 2026-10-16 on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (driver 595.99.02): PyTorch 2.11.0+cu128, Triton 3.6.0, FLA 0.5.2, causal-conv1d 1.7.0; and on one AMD Instinct MI300X: ROCm image with PyTorch 2.12, Triton 3.7.1 and FLA 0.5.2. Both with the vLLM Semantic Router model runtime of [#5066](https://github.com/vllm-project/semantic-router/pull/5066) as of 2026-10-14 with the graph capture of #4999, exact profile, and the same weights as this revision. Requests: the 1,000 train rows of tasksource-jev-typed-decisions in the #5066 record, one question each. Both packages loaded in one process; every request answered by both in turn, alternating which goes first, the device synchronized around each: the first 500 rows answered a third time so that their shape's graph replays, or all 1,000 a second time without graphs. Memory: the runtime's weight bytes after loading and the peak allocated over the 20 longest of those rows (up to 5,928 characters of state), each package in its own process. Kernels measured on 2026-10-15 on the same MI300X, one of each of d3's projection shapes with random weights, `triton.testing.do_bench`.</sub>
 
@@ -202,6 +204,8 @@ Also on the test split of [tasksource/tasksource-jev-typed-decisions](https://hu
 
 Every usable row of the split at revision `28c39125` (a state of at most 6,000 characters, no repeated options): 14,031 requests of one question each, answered by the vLLM Semantic Router model runtime on one AMD Instinct MI300X. Agreement compares the top answer of every question with d3 on the same requests (choice and score: the highest-probability option; yes / no: p ≥ 0.5).
 
+The evaluation results listed on this page (LocalLLaMA/typed-decisions) are a separate run: that benchmark's own test split scored with its `score_model.py`, accuracy 0.7475 against d3's 0.7485 on the same setup (the vLLM Semantic Router model runtime on one AMD Instinct MI300X).
+
 ### Images: Jev Decision Index vision board 0.3.2
 
 | | d3 (BF16) | d3-FP8 | Top-1 agreement with d3 |
@@ -249,7 +253,7 @@ All 12,210 rows of the kit's public vision suite (11 benchmarks, matching the ki
 
 Every multiple-choice question of Perception Test validation (19,140 questions over 5,260 videos, three options each, chance 33.3): one request per video with all its questions and the state "A video.", videos read with d3's defaults, through `d3q/` on one AMD Instinct MI300X on 2026-10-14. d3 (BF16) answered the same requests the same way on the same machine and scores 77.57; d3's card reports 77.4 from its internal evaluation, whose prompt may differ. This run used `d3q/kernels.py` from before the 64-bit row index; none of its answers is NaN. d3's example video question (`assets/example-video.mp4`) answers yes with 0.924 (d3: 0.918).
 
-<sub>Evaluated on 2026-10-15 on one AMD Instinct MI300X: ROCm 7.2 image with PyTorch 2.12, Triton 3.7.1, transformers 5.17.0 and FLA 0.5.2, with `causal_conv1d` on its torch reference as in d3's official run. Record: [decision3-quantized.md](https://github.com/vllm-project/semantic-router/blob/main/src/model-runtime/docs/records/decision3-quantized.md) in #5066. Built by `fp8_package.py` in [bench/d3-quant-eval](https://github.com/subin9/semantic-router/tree/research/d3-quant-eval/bench/d3-quant-eval): each output channel's scale is its largest absolute weight divided by 448, rounded to BF16. Keeping the Gated DeltaNet input projections in BF16 cut the changed answers on 1,000 train rows from 12 to 3, in an earlier comparison run through transformers. `d3q_engine.py` is the Decision Index engine for the kit (`--engine d3q_engine:D3QEngine --option model=<this folder>`).</sub>
+<sub>Evaluated on 2026-10-15 on one AMD Instinct MI300X: ROCm 7.2 image with PyTorch 2.12, Triton 3.7.1, transformers 5.17.0 and FLA 0.5.2, with `causal_conv1d` on its torch reference as in d3's official run. Record: [decision3-quantized.md](https://github.com/vllm-project/semantic-router/blob/main/src/model-runtime/docs/records/decision3-quantized.md) in #5066. Built by `fp8_package.py` in [bench/d3-quant-eval](https://github.com/subin9/semantic-router/tree/7908956ae1e1a673dd19fd7a305d2118b8a5228c/bench/d3-quant-eval): each output channel's scale is its largest absolute weight divided by 448, rounded to BF16. Keeping the Gated DeltaNet input projections in BF16 cut the changed answers on 1,000 train rows from 12 to 3, in an earlier comparison run through transformers. `d3q_engine.py` is the Decision Index engine for the kit (`--engine d3q_engine:D3QEngine --option model=<this folder>`).</sub>
 
 ## License
 

d3q/load.py

@@ -3,12 +3,13 @@
 The runtime's backbone loader is swapped for ``d3q.kernels.load_backbone``, so the weights stay quantized on
 the GPU; prompting, the noncausal prompt encoder, the readout, images and videos are the unmodified runtime.
 
-    import sys; sys.path.insert(0, "d3-FP8")
+    import sys; sys.path.insert(0, path)
     from d3q.load import load_d3
-    model = load_d3("d3-FP8", device="cuda:0")
+    model = load_d3(path, device="cuda:0")
     model.system_one(state=..., questions=...)
 
-The same interface as TheoHsiung/d3-W4A16-G128's ``d3q`` (Apache-2.0), whose structure this follows.
+``path`` is this package's folder, such as ``huggingface_hub.snapshot_download("vllm-sr/d3-FP8")``.
+The same interface as vllm-sr/d3-W4A16-GPTQ's ``d3q`` (Apache-2.0), whose structure this follows.
 """
 
 from __future__ import annotations