Card: Engine mode batches by default; engine arguments replace runtime profiles

Names the two measured Engine mode setups for what they were: one request per forward, the former default (exact in the chart), and batching, the default in vLLM Semantic Router from main. The engine arguments --max-num-batched-tokens, --max-num-seqs, --batch-window-ms and --dtype replace the --runtime-profile flag. README.md only; every measured number is unchanged.

README.md

@@ -34,7 +34,7 @@ tags:
 - **Reads images:** multiple images per request (PNG, JPEG or WebP), given as paths, URLs, PIL images or base64 data URLs; every question of the request sees all of them.
 - **Reads videos:** multiple videos per request (MP4, WebM, MOV or MKV), given as paths, URLs, base64 data URLs or frame arrays, read at 2 frames per second; **Perception Test (multiple-choice video QA): 77.4** (internal evaluation).
 - **Speed:** text requests take a median of 54.4 ms, requests with an image a median of 290 ms, and requests with a 10-second video a median of 500 ms, on one AMD Instinct MI325X, one request at a time with AutoModel.
-- **Serving:** 27 requests/s with p99 under 1 s on one AMD Instinct MI325X with vLLM Semantic Router, 1.5× AutoModel (one-question text, `batching` profile); the default `exact` profile gives AutoModel's answers byte for byte when both use the same kernel settings ([Serving performance](#serving-performance)).
+- **Serving:** 27 requests/s with p99 under 1 s on one AMD Instinct MI325X with vLLM Semantic Router, 1.5× AutoModel (one-question text, with batching, now the default); with one request per forward, the former default, it gave AutoModel's answers byte for byte when both used the same kernel settings ([Serving performance](#serving-performance)).
 - **Many questions, one call:** Choice, Yes / No and Score questions about the same input are answered together, each from its own forward pass over the input, with a probability for every option.
 
 ## Quickstart
@@ -145,7 +145,7 @@ Images go before the text of the request, each read at up to 1.6 megapixels; vid
 
 ## Run with vLLM Semantic Router
 
-[vLLM Semantic Router](https://github.com/vllm-project/semantic-router) serves Decision 3.0 natively in Engine mode, which runs one model behind the System One API on your own machine. Its answers match this package's own `d3_server.py` exactly, for text, images and videos (measured on AMD Instinct MI325X).
+[vLLM Semantic Router](https://github.com/vllm-project/semantic-router) serves Decision 3.0 natively in Engine mode, which runs one model behind the System One API on your own machine. Its answers matched this package's own `d3_server.py` exactly, for text, images and videos, with one request per forward, the former default (measured on AMD Instinct MI325X); with batching, now the default, a few near-tie decisions change ([Serving performance](#serving-performance)).
 
 ```bash
 curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel dev --no-launch
@@ -202,19 +202,19 @@ EOF
 
 ### Serving performance
 
-**On one AMD Instinct MI325X, vLLM Semantic Router serves d3 at 27 requests/s with p99 under 1 s, 1.5× AutoModel** (one-question text, `batching` profile), and its default `exact` profile gives AutoModel's answers byte for byte when both use the same kernel settings.
+**On one AMD Instinct MI325X, vLLM Semantic Router serves d3 at 27 requests/s with p99 under 1 s, 1.5× AutoModel** (one-question text, with batching, now the default), and with one request per forward, the former default, it gave AutoModel's answers byte for byte when both used the same kernel settings.
 
 ![Serving throughput of d3 on one AMD Instinct MI325X: AutoModel, the package server and vLLM Semantic Router Engine mode](assets/serving-throughput.png)
 
-AutoModel runs in your own process, one call at a time (the Quickstart code); the package server is its `python d3_server.py`; Engine mode is `vllm-sr serve -e` with its default `exact` profile, or with `--runtime-profile batching`.
+AutoModel runs in your own process, one call at a time (the Quickstart code); the package server is its `python d3_server.py`; Engine mode is `vllm-sr serve -e`, measured with one request per forward (*exact* in the chart, the former default) and with batching. In vLLM Semantic Router from main (its next development package and release), Engine mode batches concurrent requests by default and takes vLLM-style [engine arguments](https://github.com/vllm-project/semantic-router/blob/main/website/docs/model-runtime/engine-args.md): `--max-num-batched-tokens`, `--max-num-seqs`, `--batch-window-ms` and `--dtype`.
 
 Where Engine mode helps, and what it costs:
 
-- **Throughput:** 1.5× AutoModel's requests per second on one-question text, 1.3× on 8-question images and 1.8× on 10-second videos, with the better Engine profile of each row (8-question images: highest rates, since no side keeps p99 under 1 s there).
-- **Start-up:** the first answer comes 19.4 s after launch (20.4 s with `batching`) with empty kernel caches, against 38.4 s for AutoModel, whose first image then takes another 6.20 s, and 80.8 s for the package server.
-- **Many questions and long inputs:** one request at a time, Engine mode (`exact`) answers 16 text questions in 548 ms (AutoModel: 572 ms), 8 questions about one image in 1.87 s (1.95 s) and a 32k-token text in 4.21 s (4.23 s); `batching` runs the 8 image questions together in 1.56 s, 20 % less than AutoModel.
-- **GPU memory:** 65.4 GiB per replica at peak with `exact`, about the same as the package server's 65.2 GiB, and 62.1 GiB with `batching`; AutoModel, serving one request at a time, peaked at 61.8 GiB. Engine mode keeps up to 2 GiB of image features cached.
-- **Answers:** Engine mode (`exact`) returned answers byte-identical to AutoModel's on all 7 request types (text with 1, 4 or 16 questions, an image with 1 or 8, the video and a 32k-token text) when AutoModel used the same kernel settings: 135 Engine replies and 21 AutoModel calls compared. With the kernel autotuning AutoModel runs in each new process, 5 of the 7 were byte-identical and no decision changed (largest probability difference 0.005). `batching` changed 0.048 % of decisions on a 4,000-request sample of the public suite (14 of 29,339 questions, near-ties), with gold accuracy 92.109 → 92.126 %.
+- **Throughput:** 1.5× AutoModel's requests per second on one-question text, 1.3× on 8-question images and 1.8× on 10-second videos, with the better of the two Engine setups in each row (8-question images: highest rates, since no side keeps p99 under 1 s there).
+- **Start-up:** the first answer comes 19.4 s after launch (one request per forward; 20.4 s with batching) with empty kernel caches, against 38.4 s for AutoModel, whose first image then takes another 6.20 s, and 80.8 s for the package server.
+- **Many questions and long inputs:** one request at a time, Engine mode with one request per forward answered 16 text questions in 548 ms (AutoModel: 572 ms), 8 questions about one image in 1.87 s (1.95 s) and a 32k-token text in 4.21 s (4.23 s); batching runs the 8 image questions together in 1.56 s, 20 % less than AutoModel.
+- **GPU memory:** 65.4 GiB per replica at peak with one request per forward, about the same as the package server's 65.2 GiB, and 62.1 GiB with batching; AutoModel, serving one request at a time, peaked at 61.8 GiB. Engine mode keeps up to 2 GiB of image features cached.
+- **Answers:** Engine mode with one request per forward returned answers byte-identical to AutoModel's on all 7 request types (text with 1, 4 or 16 questions, an image with 1 or 8, the video and a 32k-token text) when AutoModel used the same kernel settings: 135 Engine replies and 21 AutoModel calls compared. With the kernel autotuning AutoModel runs in each new process, 5 of the 7 were byte-identical and no decision changed (largest probability difference 0.005). Batching changed 0.048 % of decisions on a 4,000-request sample of the public suite (14 of 29,339 questions, near-ties), with gold accuracy 92.109 → 92.126 %.
 
 A single short request is fastest in your own process: AutoModel answers one text question 0.9 ms sooner than Engine mode and one new image 22 ms sooner, the cost of going through the router to the model worker and its media path. Engine mode's gains come from serving many requests at once.
 
@@ -223,7 +223,7 @@ A single short request is fastest in your own process: AutoModel answers one tex
 
 One request at a time, warm (p50 / p90, ms):
 
-| Request | AutoModel | Package server | Engine, `exact` | Engine, `batching` |
+| Request | AutoModel | Package server | Engine, one request per forward | Engine, batching (now the default) |
 | --- | ---: | ---: | ---: | ---: |
 | Text, 1 question | 54.4 / 54.6 | 55.0 / 55.3 | 55.3 / 55.6 | 57.2 / 57.9 |
 | Text, 16 questions | 572 / 574 | 571 / 572 | 548 / 549 | 548 / 549 |
@@ -234,7 +234,7 @@ One request at a time, warm (p50 / p90, ms):
 
 Requests per second on one GPU, closed loop with up to 64 clients (p99 under 1 s, except *):
 
-| | AutoModel † | Package server | Engine, `exact` | Engine, `batching` |
+| | AutoModel † | Package server | Engine, one request per forward | Engine, batching (now the default) |
 | --- | ---: | ---: | ---: | ---: |
 | Text, 1 question | 18.4 | 17.7 | 18.1 | 27 |
 | Image (1.6 MP), 8 questions | 0.51 * | 0.53 * | 0.53 * | 0.67 * |