Xunzhuo commited on
Commit
7f257ad
·
verified ·
1 Parent(s): cbff3a3

Card: run with vLLM Semantic Router (Engine mode)

Browse files

Adds the "Run with vLLM Semantic Router" section: install the vllm-sr CLI from the development channel, `vllm-sr serve vllm-sr/d3 --engine --platform rocm`, and text, image and video requests to the System One API. MODEL_MANIFEST.json records the new README.md (SHA-256 and size) in the same commit. Weights, runtime files and the v3.1.0 tag are unchanged.

Files changed (2) hide show
  1. MODEL_MANIFEST.json +2 -2
  2. README.md +55 -0
MODEL_MANIFEST.json CHANGED
@@ -592,7 +592,7 @@
592
  "files_sha256": {
593
  "LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
594
  "NOTICE": "61f4ea488e1fed0c3fce97171506a23152c2591116442f78f15827fb9e6af469",
595
- "README.md": "62c45b5b40c4236aa52b65fb7c83da623c35edab4df7469f104c1597c4235d58",
596
  "assets/banner.png": "eb7ecc949232e1de544247899dd55ee629a36cfe276042d5795f65178b9c64c5",
597
  "assets/example-receipt.png": "b9dbb8b103bacd45f1de824d844a3c07b81d57cb1a7466712ef060ad7dcb3926",
598
  "assets/example-video.mp4": "f4968ef89d07e97f20df4fd8047510a7a8e9df5c2d07c35e363a97927f164ab9",
@@ -633,7 +633,7 @@
633
  "files_bytes": {
634
  "LICENSE": 11357,
635
  "NOTICE": 100,
636
- "README.md": 7892,
637
  "assets/banner.png": 2233137,
638
  "assets/example-receipt.png": 65918,
639
  "assets/example-video.mp4": 26300,
 
592
  "files_sha256": {
593
  "LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
594
  "NOTICE": "61f4ea488e1fed0c3fce97171506a23152c2591116442f78f15827fb9e6af469",
595
+ "README.md": "fec0348ded4425d20f15b87cbe782eb6c76646b871806e0d444ccad3ed77de4f",
596
  "assets/banner.png": "eb7ecc949232e1de544247899dd55ee629a36cfe276042d5795f65178b9c64c5",
597
  "assets/example-receipt.png": "b9dbb8b103bacd45f1de824d844a3c07b81d57cb1a7466712ef060ad7dcb3926",
598
  "assets/example-video.mp4": "f4968ef89d07e97f20df4fd8047510a7a8e9df5c2d07c35e363a97927f164ab9",
 
633
  "files_bytes": {
634
  "LICENSE": 11357,
635
  "NOTICE": 100,
636
+ "README.md": 10307,
637
  "assets/banner.png": 2233137,
638
  "assets/example-receipt.png": 65918,
639
  "assets/example-video.mp4": 26300,
README.md CHANGED
@@ -142,6 +142,61 @@ print(json.dumps(result["answers"], indent=2))
142
 
143
  Images go before the text of the request, each read at up to 1.6 megapixels; videos follow the images, read at 2 frames per second (at most 32 frames spread over the whole video, each at up to 0.2 megapixels). Every question of the request sees all of them.
144
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
145
  ## Evaluation
146
 
147
  ### Text: Jev Decision Index 0.3.1
 
142
 
143
  Images go before the text of the request, each read at up to 1.6 megapixels; videos follow the images, read at 2 frames per second (at most 32 frames spread over the whole video, each at up to 0.2 megapixels). Every question of the request sees all of them.
144
 
145
+ ## Run with vLLM Semantic Router
146
+
147
+ [vLLM Semantic Router](https://github.com/vllm-project/semantic-router) serves Decision 3.0 natively in Engine mode, which runs one model behind the System One API on your own machine. Its answers match this package's own `d3_server.py` exactly, for text, images and videos. On an AMD Instinct MI325X:
148
+
149
+ ```bash
150
+ curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel dev --no-launch
151
+ export PATH="$HOME/.local/bin:$PATH"
152
+ vllm-sr serve vllm-sr/d3 --engine --platform rocm
153
+ ```
154
+
155
+ Engine mode and Decision 3.0 are in the development channel (`--channel dev`) until a stable vLLM Semantic Router release includes them. The first start pulls the `vllm-sr-rocm` image and downloads the model. Then ask questions about a request:
156
+
157
+ ```bash
158
+ curl -s localhost:8899/v1/systemone -H 'content-type: application/json' -d '{
159
+ "model": "vllm-sr/d3",
160
+ "state": "Write a Python function that merges two sorted lists.",
161
+ "questions": {
162
+ "kind": {"type": "choice", "instructions": "What kind of work is this?",
163
+ "criteria": {"code": "Writing or fixing code", "math": "Mathematics", "chat": "Anything else"}},
164
+ "reasoning": {"type": "noul", "instructions": "Does answering this need multi-step reasoning?"}
165
+ }
166
+ }'
167
+ ```
168
+
169
+ Send images as base64 data URLs (PNG, JPEG or WebP) in `images` and videos (MP4, WebM, QuickTime or Matroska) in `videos`; every question sees them:
170
+
171
+ ```bash
172
+ curl -s localhost:8899/v1/systemone -H 'content-type: application/json' --data-binary @- <<EOF
173
+ {
174
+ "model": "vllm-sr/d3",
175
+ "state": "What does the attached image show?",
176
+ "images": ["data:image/png;base64,$(base64 -w0 chart.png)"],
177
+ "questions": {
178
+ "kind": {"type": "choice", "instructions": "What kind of image is this?",
179
+ "criteria": {"chart": "A chart or plot", "photo": "A photograph", "document": "A document"}}
180
+ }
181
+ }
182
+ EOF
183
+ ```
184
+
185
+ ```bash
186
+ curl -s localhost:8899/v1/systemone -H 'content-type: application/json' --data-binary @- <<EOF
187
+ {
188
+ "model": "vllm-sr/d3",
189
+ "state": "What happens in the clip?",
190
+ "videos": ["data:video/mp4;base64,$(base64 -w0 clip.mp4)"],
191
+ "questions": {
192
+ "moving": {"type": "noul", "instructions": "Does something move across the scene?"}
193
+ }
194
+ }
195
+ EOF
196
+ ```
197
+
198
+ `vllm-sr stop` stops it. See [Router and Engine modes](https://github.com/vllm-project/semantic-router/blob/main/src/vllm-sr/INSTANCE_MODES.md#ask-about-images-and-videos) for limits and options.
199
+
200
  ## Evaluation
201
 
202
  ### Text: Jev Decision Index 0.3.1