Model card rewrite; launch.sh downloads respanai/span-02

README.md

@@ -18,247 +18,257 @@ tags:
 
 # span-02
 
-span-02 is a **decision model**: it answers typed questions about a state (JSON or text, plus up to 4 images)
-with a full probability distribution over the options, in the Decision Index `/v1/systemone` wire format.
-It is a fine-tune of Gemma 4 26B-A4B (mixture of experts, about 4B active parameters per token), quantized
-(NVFP4 experts, FP8 dense layers) and served by a custom Rust/CUDA engine for Blackwell GPUs (sm_120: RTX 5090,
-RTX PRO 6000) that is included in this repo (`engine/`).
-
-Before answering it may think. If the top probability of the no-think answer is below 0.8, the model writes a
-greedy thought. Its length limit comes from a deterministic thought budget: a fixed cost model (`think_cost.json`)
-turns a 750 ms target into a per-question token cap that depends only on the request (prompt length, images,
-number of questions and of thinking questions), so the same request always gets the same thought. After a thought
-the answer is read out again, and the final distribution is the average of the two:
-0.5 x the no-think distribution + 0.5 x the post-thought distribution, for every question type. A choice answer is
-the option with the highest final probability. Thoughts and number answers are decoded with speculative decoding
-(Gemma's public MTP draft). The verification is greedy, so both are token-identical to plain greedy decoding.
-
-Outputs are typed: `choice`, `noul` (yes/no probability), `score` (ordinal levels) and `number` (a free-form number).
-The model never returns free text as an answer.
+**span-02 is a decision model.** Give it a state (text or JSON, plus up to 4 images) and a set of typed questions,
+and it returns a calibrated probability distribution for every answer. No free text, no parsing.
 
-## Quantization
-**This checkpoint is engine-native.** It cannot be loaded by `transformers` or vLLM.
+- **Typed answers:** pick one option (`choice`), yes/no (`noul`), a rating on a scale (`score`), or a number
+  (`number`).
+- **Fast:** a typical request answers in tens of milliseconds. When the model is unsure, it thinks briefly first,
+  and the request still finishes well under a second.
+- **Small and efficient:** a fine-tune of Gemma 4 26B-A4B (mixture of experts, ~4B active parameters), quantized to
+  NVFP4/FP8 (16.4 GB) and served by its own Rust/CUDA engine, which is in this repo.
+- **Runs on one Blackwell GPU:** RTX PRO 6000 or RTX 5090.
 
-| part | format |
+| Decision Index 0.3 (public) | score |
 |---|---|
-| MoE experts | NVFP4 W4A4 (e2m1 weights, FP8-e4m3 scale per 16 elements, FP32 global scale; activations NVFP4 with static per-layer global scales) |
-| attention and dense MLP linears | FP8 e4m3 W8A8 (per-output-row weight scale, dynamic per-token activation scale) |
-| tied embedding / LM head | FP8 e4m3 per row |
-| vision tower, router, norms, KV cache | bf16 |
+| Text index | **63.62** (all 140,178 requests answered) |
+| Vision | **70.69** (all 12,210 requests answered) |
 
-`model.safetensors` is 16.4 GB. GPU memory in use at serving time is about 26 GB.
-
-## Hardware
-* An NVIDIA Blackwell GPU with compute capability 12.0 (`sm_120a`). Tested on an **RTX PRO 6000 Blackwell** (96 GB),
-  where the latency below was measured, and on an **RTX 5090** (32 GB).
-* NVIDIA driver with CUDA 12.9 support (R575 or newer).
-* To build: CUDA toolkit 12.8+ (`nvcc`) and Rust. No GPU is needed at build time.
-
-Other GPU architectures are not supported, because the kernels use the sm_120a block-scaled FP4/FP8 tensor-core
-instructions.
-
-## Files
-| file | what |
-|---|---|
-| `model.safetensors`, `config.json` | quantized weights and the Gemma-4 config |
-| `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `generation_config.json` | tokenizer and chat template |
-| `processor_config.json` | Gemma-4 image processor at the served budget: `max_soft_tokens` 560 (about 1.3 MP) |
-| `draft_vocab_16384.json` | pruned 16384-token vocabulary for the MTP draft head |
-| `think_cost.json` | fixed cost model of the deterministic thought budget (`--think-cost`; calibrated on an RTX PRO 6000) |
-| `engine/` | source of `dllm-server` (Rust + CUDA; see `engine/README.md`) |
-| `launch.sh` | finds or downloads the weights and the draft, then starts the server with the settings below |
-| `Dockerfile`, `.dockerignore` | two-stage image: CUDA devel build, then CUDA base runtime |
-| `NOTICE` | licenses and attributions |
-
-The MTP draft model is **not** in this repo. `launch.sh` downloads the public
-[`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant).
+> The weights only run with the included engine (`engine/`). They cannot be loaded with `transformers` or vLLM.
 
 ## Quickstart
-Docker:
-```
-hf download lihe07/span-02 --local-dir span-02 && cd span-02
+
+With Docker (needs an NVIDIA driver with CUDA 12.9 support and the NVIDIA container toolkit):
+```bash
+hf download respanai/span-02 --local-dir span-02 && cd span-02
 docker build -t span-02 .
 docker run --gpus all -p 8000:8000 -v "$PWD":/model:ro -e DLLM_MODEL_DIR=/model -v span02-data:/data span-02
 ```
-Without `-v "$PWD":/model ... DLLM_MODEL_DIR`, the container downloads this repo itself. Pass `-e HF_TOKEN` for
-that while the repo is private.
 
-Without Docker (CUDA 12.8+ toolkit and Rust installed):
-```
-hf download lihe07/span-02 --local-dir span-02 && cd span-02
+Without Docker (CUDA toolkit 12.8+ and Rust; no GPU needed to build):
+```bash
+hf download respanai/span-02 --local-dir span-02 && cd span-02
 (cd engine && NVCC=/usr/local/cuda/bin/nvcc cargo build --release -p dllm-server)
-bash launch.sh          # serves on 0.0.0.0:8000; PORT / HOST / AR_API_KEY are optional
-```
-Startup takes about 10 s plus about 7 s of warmup. `GET /health` returns `{"ok":true,"model":"span-02"}` when the server is ready.
-
-## API
-`POST /v1/systemone`; the format is TypeSafe's
-System One API (docs.typesafe.ai/api) plus the extensions marked below. `GET /health` returns `{"ok": true, "model": "span-02"}`.
-
-Request fields:
-* `state` (required): any JSON value (string, object, array, ...).
-* `questions` (required): {question id: question}; answers come back in this order.
-* `model`: accepted and ignored (e.g. `"jev-latest"`); the response `model` is always the served id `span-02`.
-* `assets`: TypeSafe image assets, `{placeholder: {"type": "image", "format": "png" | "jpg" | "jpeg", "data_base64": ...}}`,
-  or `{"part": name}` with a `multipart/form-data` body (part `request` = the JSON request, one file part per image).
-  Referencing an asset is optional: a placeholder that occurs in `state` or in a question's `instructions` /
-  `criteria` (any string value, also mid-string; object keys are never replaced) is replaced by the image's label.
-  See "Images" below.
-* `images` (extension, sent by the Decision Index kit's vision runner): a list of images as data URIs or bare base64. At most 4 images per
-  request in total (`images` + `assets`).
-* `reasoning` (extension): `{"mode": "auto" | "on" | "off", "budget_ms": <ms>, "trace": <bool>}`, all optional.
-  `mode`: `auto` (default) uses the tau 0.8 gate, `on` thinks for every question, `off` never thinks.
-  `budget_ms` (0 to 600000) is the wall-clock thinking budget of the request; `0` means none (a thought ends at the
-  end-of-thought token, EOS or the cap). Without it, `mode: "on"` thinks to completion (up to the cap) and other
-  requests use the server's `--budget-ms`. The deterministic `--think-ms` cap of the submission settings applies
-  regardless; `budget_ms` only adds or removes a wall-clock deadline (the submission server has none), so requests
-  without it stay deterministic. `trace: true` returns each thought's text.
-* Other top-level fields are ignored.
-* Bodies may be compressed with `Content-Encoding: zstd` (the TypeSafe Python SDK default) or `gzip`.
-
-Question types:
-* `choice`: `criteria` maps each option key to its description (1 to 679 options). The answer is the argmax,
-  its `confidence` = (p_max - 1/n) / (1 - 1/n), and the probabilities in criteria order.
-* `noul`: the probability that the statement is true. `criteria` is optional: `{"true": ..., "false": ...}`.
-* `score`: `criteria` is an ordered array of 2 to 10 level descriptions, from low to high. The answer is
-  `score` = sum_i i x p_i, with the per-level probabilities, a `legend` (the levels as sent) and a `confidence`.
-* `number` (extension): a free-form number, no `criteria`. The model writes the number greedily after its
-  (optional) thought, as it was trained (the plain decimal, e.g. `391` or `2.5`, at most 32 tokens). The answer is
-  `{"type": "number", "number": 391.0, "text": "391", "confidence": 0.98}`: `number` is the parsed value (`null` if
-  the text is not a number), `confidence` = exp(mean log-probability of the generated answer tokens). The think gate
-  is trained for option questions only, so number questions think only with `"reasoning": {"mode": "on"}`.
-
-Request:
+bash launch.sh        # serves on 0.0.0.0:8000
 ```
+`launch.sh` also downloads the speculative-decoding draft model
+([`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant)).
+The server is ready in about 20 s; `GET /health` then returns `{"ok": true, "model": "span-02"}`.
+
+## Example
+
+```bash
 curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
-  "model": "span-02",
   "state": {"customer": "My package arrived damaged and nobody answers my emails. I want a refund now.",
             "order": {"id": 123, "days_late": 4}},
   "questions": {
     "dept":   {"type": "choice", "instructions": "Which team should handle this?",
-               "criteria": {"returns": "Exchanges, wrong or damaged items",
+               "criteria": {"returns":  "Exchanges, wrong or damaged items",
                             "shipping": "Delivery status, delays, lost packages",
-                            "billing": "Charges, invoices, payment problems"}},
+                            "billing":  "Charges, invoices, payment problems"}},
     "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
     "anger":  {"type": "score", "instructions": "How frustrated is the customer?",
-               "criteria": ["Not frustrated: calm or positive", "Somewhat frustrated: mild annoyance",
-                            "Very frustrated: angry, hostile or threatening to leave"]}},
-  "reasoning": {"mode": "auto", "trace": true}}'
-```
-Response:
+               "criteria": ["Calm or positive", "Mildly annoyed", "Angry or threatening to leave"]},
+    "late":   {"type": "number", "instructions": "How many days late is the order?"}
+  }
+}'
 ```
+```json
 {"model": "span-02",
  "answers": {
    "dept":   {"type": "choice", "choice": "returns", "confidence": 0.945,
               "probabilities": {"returns": 0.963, "shipping": 0.024, "billing": 0.013}},
    "refund": {"type": "noul", "noul": 0.99999},
-   "anger":  {"type": "score", "score": 1.9999,
-              "legend": {"0": "Not frustrated: calm or positive", "1": "Somewhat frustrated: mild annoyance",
-                         "2": "Very frustrated: angry, hostile or threatening to leave"},
-              "probabilities": {"0": 0.000008, "1": 0.00012, "2": 0.99987}, "confidence": 0.9998}},
- "usage": {"input_tokens": 412, "reasoning_tokens": 0, "output_tokens": 0, "latency_ms": 41.7}}
+   "anger":  {"type": "score", "score": 1.9999, "confidence": 0.9998,
+              "legend": {"0": "Calm or positive", "1": "Mildly annoyed", "2": "Angry or threatening to leave"},
+              "probabilities": {"0": 0.000008, "1": 0.00012, "2": 0.99987}},
+   "late":   {"type": "number", "number": 4.0, "text": "4", "confidence": 0.99}},
+ "usage": {"input_tokens": 540, "reasoning_tokens": 0, "output_tokens": 2, "latency_ms": 45.2}}
 ```
-(The token counts and latency in this response are illustrative.)
+(Values are illustrative.)
+
+The API follows TypeSafe's [System One API](https://docs.typesafe.ai/api), so its clients work unchanged; `number`,
+`reasoning` and `images` are span-02 extensions that those clients can ignore.
+
+## How it works
+
+1. **Read the answer directly.** Each question is one prompt; the answer is read from the model's probabilities
+   for the option labels in a single forward pass. All questions of a request share the state, and the engine
+   computes it once.
+2. **Think if unsure.** If the top answer has probability below 0.8, the model first writes a short thought, then
+   the answer is read again. The final distribution is the average of the two:
+   0.5 × before thinking + 0.5 × after thinking.
+3. **Stay on time.** Thoughts are limited by a time budget (700 ms by default) and decoded with speculative decoding
+   (Gemma's public MTP draft, verified greedily), at about 480 thought tokens/s on an RTX PRO 6000.
+
+Number answers are generated token by token after the (optional) thought, with the same speculative decoding.
+
+## API reference
+
+`POST /v1/systemone` takes a request and returns the answers. `GET /health` reports readiness.
+
+### Request
+
+| field | | |
+|---|---|---|
+| `state` | required | Any JSON value. The model treats it as data, not instructions. |
+| `questions` | required | `{id: question}`. Answers come back in the same order. |
+| `assets` | optional | Images keyed by id: `{"type": "image", "format": "png" \| "jpg" \| "jpeg", "data_base64": "..."}`. Mentioning an id in the state or a question is optional; see [Images](#images). |
+| `images` | optional | A list of images as data URIs or base64 (used by the Decision Index kit). At most 4 images in total. |
+| `reasoning` | optional | `{"mode": "auto" \| "on" \| "off", "budget_ms": 0-600000, "trace": true \| false}` |
+| `model` | optional | Accepted and ignored; responses always say `span-02`. |
+
+- **`mode`:** `auto` (default) thinks only when unsure; `on` always thinks; `off` never thinks.
+- **`budget_ms`:** the thinking time limit for this request; `0` means no limit. If omitted, `on` thinks until the
+  thought ends (at most 4,096 tokens) and `auto` uses the server default (`--budget-ms`, 700).
+- **`trace`:** return each thought's text.
+- Bodies can be JSON, `multipart/form-data` (a `request` part plus one file part per image, referenced as
+  `{"part": name}`), or compressed with `Content-Encoding: zstd` or `gzip`. Unknown top-level fields are ignored.
+
+### Question types
+
+| type | `criteria` | answer |
+|---|---|---|
+| `choice` | `{key: description}`, 1 to 679 options | `choice`, `confidence`, `probabilities` |
+| `noul` | optional `{"true": ..., "false": ...}` | `noul` = probability of true |
+| `score` | ordered list of 2 to 10 levels, low to high | `score` = Σ i·pᵢ, `legend`, `probabilities`, `confidence` |
+| `number` | none | `number` (`null` if the text is not a number), `text`, `confidence` |
+
+- `choice` confidence = (p_max − 1/n) / (1 − 1/n): 0 for a uniform guess, 1 for certainty.
+- `score` confidence = 1 − (expected distance from the most likely level) / (the same for a uniform guess).
+- `number` confidence = exp(mean log-probability of the answer tokens). Number questions think only with
+  `mode: "on"`, because the think gate is trained on option questions.
 
-* An answer whose question thought has `"reasoning": {"tokens": <thought tokens>, "trace": "<thought text>"}`
-  (`trace` only when the request asked for it); answers that did not think have no `reasoning` field.
-* `usage` holds request totals: `input_tokens` = the prompt tokens of every question (the shared state is counted
-  once per question; each image counts its soft tokens + 2), `reasoning_tokens` = thought tokens generated,
-  `output_tokens` = answer tokens generated (the tokens of `number` answers; choice, noul and score answers are read
-  from one forward pass and generate none), `latency_ms` = server time from arrival to response.
-* With images, the response also has a top-level `image_tokens` field (the total image soft tokens).
+### Response
+
+- `answers[id]`: the typed answer. If the question thought, it also has `reasoning: {"tokens": N}` (plus `trace`
+  when requested).
+- `usage`: `input_tokens` (the prompt tokens of every question; the shared state counts once per question and each
+  image its image tokens + 2), `reasoning_tokens` (thought tokens), `output_tokens` (number-answer tokens; other
+  answers are read in one pass and generate none), `latency_ms` (server time).
+- `image_tokens`: total image tokens, present only when the request has images.
 
 ### Images
-The model was trained with all images (up to 4) in front of the text, in order, and with the text calling them
-"Image 1", "Image 2", ... So the server puts the `images` first and then the assets in the order they are sent, and
-replaces every occurrence of an asset placeholder in the state, instructions and criteria by that image's label.
-For `{"state": {"photo": "a3f1c2e8-found"}, "assets": {"a3f1c2e8-found": ..., "77b0d4a1-poster": ...}}` the model
-reads `{"photo": "Image 1"}`, and an instruction "... as in this picture: 77b0d4a1-poster?" becomes "... as in this
-picture: Image 2?". Each asset is shown once, however often it is mentioned; an asset that is never mentioned is
-shown all the same (one unreferenced asset renders exactly like one `images` entry). A score `legend` echoes the placeholder,
-not the label. Requests without assets are rendered exactly as before. Assets are PNG or JPEG (the `format` must
-match the bytes), at most 16 MiB each; they are resized to the image budget, not rejected for their dimensions.
+
+The model was trained with all images (up to 4) before the text, referred to as "Image 1", "Image 2", and so on.
+The server shows `images` first, then `assets` in the order sent, and replaces every mention of an asset id in the
+state, instructions and criteria with that image's label. For example
+`{"state": {"photo": "a3f1c2e8-found"}, "assets": {"a3f1c2e8-found": ..., "77b0d4a1-poster": ...}}` reaches the model
+as `{"photo": "Image 1"}`, and "the dog in this picture: 77b0d4a1-poster" becomes "the dog in this picture: Image 2".
+Assets that are never mentioned are still shown. Images are PNG or JPEG, at most 16 MiB each, and are resized to the
+image budget (280 tokens per image by default, 560 in the Decision Index settings).
 
 ### Errors
-`{"error": {"type": ..., "message": ...}}`, one status per kind:
+
+Errors look like `{"error": {"type": "...", "message": "..."}}`.
 
 | status | type | when |
 |---|---|---|
-| 400 | `invalid_request` | invalid JSON, a malformed compressed or multipart body |
-| 400 | `invalid_asset` | a rejected asset: unknown `type` or `format`, bad base64, not a PNG/JPEG, missing or unused multipart part, more than 4 images |
-| 400 | `max_tokens_exceeded` | the prompt (+ images + thought reservation) exceeds the context, or the request cannot fit in the whole KV cache; the message contains "maximum context length" |
-| 401 | `authentication_error` | missing or wrong bearer token (only with `AR_API_KEY`) |
-| 404 | `not_found` | any path other than `POST /v1/systemone` and `GET /health` |
-| 413 | `request_too_large` | body over 64 MB (also after decompression), an image over 16 MiB |
-| 415 | `unsupported_encoding` | a Content-Encoding other than zstd / gzip, or more than one layer |
-| 422 | `validation_error` | schema errors: missing fields, bad question types or criteria, too many options, bad `reasoning`, more than 4 or undecodable `images` |
-| 529 | `overloaded` | concurrent requests hold the KV cache; retry after a short delay |
+| 400 | `invalid_request` | invalid JSON, or a broken compressed or multipart body |
+| 400 | `invalid_asset` | unknown asset type or format, bad base64, not a PNG/JPEG, more than 4 images |
+| 400 | `max_tokens_exceeded` | the request does not fit the context or the KV cache ("maximum context length") |
+| 401 | `authentication_error` | missing or wrong key (only when the server sets `AR_API_KEY`) |
+| 404 | `not_found` | any other path |
+| 413 | `request_too_large` | body over 64 MB, or an image over 16 MiB |
+| 415 | `unsupported_encoding` | a `Content-Encoding` other than zstd or gzip |
+| 422 | `validation_error` | missing fields, bad question types or criteria, too many options, bad `reasoning` |
+| 529 | `overloaded` | the server is busy; retry shortly |
 | 500 | `server_error` | an internal failure |
 
-The capacity errors carry the Decision Index capacity markers, so the DI client records `unsupported`.
+## Performance
+
+### Latency (Decision Index settings, one RTX PRO 6000, one request at a time)
+
+Client wall time per request, including HTTP, in ms. The Decision Index requires median, mean and p80 of at most
+1,000 ms each.
+
+| request mix | n | median | mean | p80 | accuracy (choice questions) |
+|---|---|---|---|---|---|
+| DI public text 0.3, stratified sample (42 benchmarks) | 400 | 71 | 308 | 712 | 0.923 |
+| DI public vision: CV-Bench, BLINK, RealWorldQA | 150 | 186 | 291 | 642 | 0.747 |
+| image rows of 6 DI vision benchmarks | 210 | 292 | 389 | 699 | 0.652 |
+| think-heavy set (78% of questions think) | 150 | 616 | 514 | 770 | 0.562 |
+
+On the full public text run: median 36 ms, mean 241 ms, p95 850 ms.
 
-## Serving settings (Decision Index submission)
+### Throughput (64 requests in flight)
+
+Short single-question requests, 700 ms thinking budget. RTX PRO 6000 with
+`--max-rows 16384 --think-max-rows 8192 --kv-gb 48 --pages 24000`; RTX 5090 with the defaults.
+
+| GPU | requests/s, no thinking | requests/s, `auto` thinking | p99, `auto` |
+|---|---|---|---|
+| RTX PRO 6000 | 167 | 150 | 0.85 s |
+| RTX 5090 | 127 | 102 | 0.70 s |
+
+On the same RTX 5090, the engine is 1.3 to 7× faster than vLLM without thinking and keeps a lower tail latency
+with it.
+
+## Hardware
+
+- An NVIDIA Blackwell GPU with compute capability 12.0 (`sm_120a`): **RTX PRO 6000 Blackwell** (96 GB) or
+  **RTX 5090** (32 GB). Other architectures are not supported: the kernels use sm_120a FP4/FP8 tensor-core
+  instructions.
+- NVIDIA driver with CUDA 12.9 support (R575 or newer).
+- GPU memory in use: about 28 GB with the default settings; the Decision Index settings below use about 41 GB.
+
+## Quantization
+
+| part | format |
+|---|---|
+| MoE experts | NVFP4 W4A4: e2m1 weights with an FP8 scale per 16 elements and an FP32 global scale; activations NVFP4 with static per-layer global scales |
+| attention and dense MLP | FP8 e4m3 W8A8: per-row weight scales, dynamic per-token activation scales |
+| tied embedding / LM head | FP8 e4m3, per row |
+| vision tower, router, norms, KV cache | bf16 |
+
+## Decision Index settings
+
+The Decision Index submission runs `launch.sh` as shipped:
 ```
 --max-len 131072 --budget-ms 0 --tau 0.8 --cap 1024 --think-ms 750 --think-cost think_cost.json
 --image-soft-tokens 560 --serial --no-prefix-cache --spec-max-branches 16 --pages 12000 --kv-gb 24
 ```
-* `--budget-ms 0`: there is no wall-clock limit; the clock is never read for thinking decisions.
-* `--think-ms 750 --think-cost think_cost.json`: deterministic thought budget. Each request's per-question thought
-  cap = floor((750 - fixed) / per_token(n)), where fixed is the modelled prompt, image and readout time of that
-  request and per_token(n) the modelled time per thought token with n of its questions thinking together. It is a
-  pure function of the request, so answers do not depend on the machine or its load. A thought ends at the
-  end-of-thought token, at EOS, or at that cap (at most `--cap 1024`). A short single question gets about 330
-  thought tokens, a request with many thinking questions or a long prompt fewer.
-* `--tau 0.8`: think when the no-think top probability is below 0.8. The final distribution is then
-  0.5 x no-think + 0.5 x post-thought (choice, score and noul alike).
-* `--spec-draft` (passed by `launch.sh`): MTP speculative decoding of thoughts and number answers with an adaptive draft
-  depth and a pruned 16384-token draft vocabulary. It is lossless (token-identical to greedy decoding).
-* `--image-soft-tokens 560`: the Gemma-4 image budget (about 1.3 MP; aspect ratio kept).
-* `--serial --no-prefix-cache --spec-max-branches 16`: each answer is a deterministic function of its request.
-  It does not depend on load or on earlier requests (for up to 16 questions of one request thinking at once).
-  `--serial` runs one request at a time; concurrent clients queue.
-* `--max-len 131072`: the context limit per question (prompt + images + thought).
-* `--pages 12000 --kv-gb 24`: KV cache pools (full-attention layers 12000 x 16 = 192K tokens, 3.9 GB;
-  sliding-window layers get the rest of `--kv-gb 24`, ~20 GB), sized so that every Decision Index request fits on its own under `--serial` (the
-  largest needs about 3.7K full pages: many questions with long tails). The pool size does not change any answer.
-  The server uses about 41 GB of GPU memory.
-
-## Latency
-Measured with the submission settings on one RTX PRO 6000 Blackwell, one request at a time, as client wall time
-per request including HTTP (ms). The Decision Index rule requires the median, mean and p80 each to be at most
-1000 ms.
+- **Deterministic thinking budget:** `--budget-ms 0 --think-ms 750 --think-cost think_cost.json`. The wall clock is
+  never read. Instead, a fixed cost model turns a 750 ms target into a thought-token cap that depends only on the
+  request (prompt length, images, number of thinking questions), so the result does not depend on the machine or
+  its load. A short single question gets about 330 thought tokens; requests with many thinking questions or long
+  prompts get fewer. `--cap 1024` is the hard limit.
+- **One request at a time, no prefix cache:** `--serial --no-prefix-cache`, so an answer does not depend on other
+  requests. Requests where several questions think at once can still vary very slightly in their post-thought
+  probabilities between runs.
+- **Image budget:** `--image-soft-tokens 560`, about 1.3 MP with the aspect ratio kept.
+- **KV cache:** `--pages 12000 --kv-gb 24`, sized so every Decision Index request fits on its own.
 
-| request mix | n | median | mean | p80 | accuracy (choice questions) |
-|---|---|---|---|---|---|
-| DI public text 0.3, stratified sample (42 benchmarks), DI kit | 400 | 71 | 308 | 712 | 0.923 (2725 questions) |
-| DI public vision: CV-Bench, BLINK, RealWorldQA, DI kit | 150 | 186 | 291 | 642 | 0.747 |
-| image rows of 6 DI vision benchmarks (incl. MMMU-Pro vision), DI kit | 210 | 292 | 389 | 699 | 0.652 |
-| think-heavy question set (78 % of questions think) | 150 | 616 | 514 | 770 | 0.562 |
-
-The rule is met on every mix measured, with margin. Against a plain 256-token cap on the same machine and requests,
-accuracy is the same within noise (pooled paired difference -0.03 points, 95 % CI [-0.39, +0.34]) and the tail is much
-shorter (DI text requests over 1 s: 34 -> 4 of 400; slowest 2.8 s -> 1.07 s). Against a blend that puts the post-thought
-half on the post-thought top option only (one-hot), the averaged distributions have the same accuracy within noise
-(+0.13 points, 95 % CI [-0.06, +0.35], 3,707 choice questions, same thoughts) and are better calibrated (log-loss
-0.409 -> 0.397, Brier score 0.200 -> 0.193; on the questions that thought 1.368 -> 1.290 and 0.734 -> 0.689). Speculative decoding runs at about 480
-thought tokens/s at batch 1 on real thoughts on this GPU (2.1 ms per token).
-
-## Capacity limits
-Requests over these limits are rejected with the DI capacity markers. Nothing is truncated.
-* Choice questions: 1 to 679 options.
-* Images: at most 4 per request (`images`: PNG, JPEG, GIF, BMP, TIFF or WebP; `assets`: PNG or JPEG). Each image uses at most 560 soft tokens.
-* Context: the prompt tokens + 562 per image + the reserved thought tokens + 3 must fit in 131072 tokens
-  (`--max-len 131072`; the sliding-window layers keep only their 1024-token window, so long prompts stay cheap).
-  The reservation is the deterministic cap of a lone thinking question of that size; at long prompts it is small
-  or zero (the cost model leaves no thought time).
-* KV cache: a request whose prompt pages (shared prefix + every question's own tail) exceed the whole KV pool is
-  rejected with 400 `max_tokens_exceeded` ("maximum context length"); none of the DI 0.3 public requests does with
-  the pools above.
-* Request body: at most 64 MB.
+## Limits
+
+- Choice questions: 1 to 679 options. Score questions: 2 to 10 levels.
+- Images: at most 4 per request.
+- Context: 131,072 tokens per question (prompt + images + thought). Long prompts stay cheap: the sliding-window
+  layers keep only their 1,024-token window.
+- KV cache: a request too large for the whole KV cache is rejected with 400 `max_tokens_exceeded`. With the default
+  `--kv-gb 9` (sized for a 32 GB RTX 5090) this can happen for requests with several long questions, for example
+  4 questions of 8K tokens each. On a larger GPU, raise `--kv-gb` and `--pages` (e.g. `--kv-gb 48 --pages 24000` on
+  an RTX PRO 6000).
+- Number answers: at most 32 tokens. Thoughts: at most 4,096 tokens (1,024 in the Decision Index settings).
+- Request body: at most 64 MB. Nothing is truncated; requests over a limit are rejected.
+
+## Files
+
+| file | contents |
+|---|---|
+| `model.safetensors`, `config.json` | quantized weights (16.4 GB) and the Gemma 4 config |
+| `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `generation_config.json` | tokenizer and chat template |
+| `processor_config.json` | Gemma 4 image processor at 560 image tokens |
+| `draft_vocab_16384.json` | pruned vocabulary for the speculative-decoding draft |
+| `think_cost.json` | cost model for the deterministic thinking budget (calibrated on an RTX PRO 6000) |
+| `engine/` | the `dllm-server` source (Rust + CUDA); see `engine/README.md` for build and flags |
+| `launch.sh` | downloads what is missing and starts the server with the Decision Index settings |
+| `Dockerfile`, `.dockerignore` | two-stage image: CUDA build, then a small runtime |
+| `NOTICE` | licenses and attributions |
 
 ## License
-The weights are a derivative of Google's Gemma 4 (`google/gemma-4-26B-A4B-it`) and are distributed under the
-Gemma 4 license terms, which are Apache 2.0:
-[ai.google.dev/gemma/docs/gemma_4_license](https://ai.google.dev/gemma/docs/gemma_4_license).
-The serving engine in `engine/` is Apache-2.0. It ports algorithms from vLLM, SGLang, FlashInfer and other
-projects, which are credited in `NOTICE` and in the source file headers.
+
+The weights are a derivative of Google's Gemma 4 (`google/gemma-4-26B-A4B-it`), distributed under the
+[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license) (Apache 2.0). The engine in `engine/` is
+Apache-2.0; it ports algorithms from vLLM, SGLang, FlashInfer and other projects, credited in `NOTICE` and in the
+source file headers.

launch.sh

@@ -8,7 +8,7 @@
 # Env:
 #   DLLM_MODEL_DIR   directory with model.safetensors, config.json, tokenizer.json, draft_vocab_16384.json, think_cost.json
 #                    (default: this script's directory if it holds model.safetensors, else $DLLM_DATA/model, downloaded from $DLLM_HF_REPO)
-#   DLLM_HF_REPO     model repo to download (lihe07/span-02)
+#   DLLM_HF_REPO     model repo to download (respanai/span-02)
 #   DLLM_DRAFT_REPO  MTP draft for speculative decoding (google/gemma-4-26B-A4B-it-assistant; public)
 #   DLLM_DATA        download directory (/data/dllm)
 #   DLLM_SERVER_BIN  server binary (default: dllm-server on PATH, else engine/target/release/dllm-server next to this script)
@@ -18,7 +18,7 @@
 # Extra arguments are passed to dllm-server (`dllm-server --help` lists them).
 set -euo pipefail
 HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
-REPO=${DLLM_HF_REPO:-lihe07/span-02}
+REPO=${DLLM_HF_REPO:-respanai/span-02}
 DRAFT=${DLLM_DRAFT_REPO:-google/gemma-4-26B-A4B-it-assistant}
 D=${DLLM_DATA:-/data/dllm}
 if [ -n "${DLLM_MODEL_DIR:-}" ]; then M=$DLLM_MODEL_DIR