| # DS4 API surface matrix |
|
|
| This guide describes the Rust host's HTTP behavior. Configuration and resolved |
| plans belong to the [serving contract](serving-contract.md); qualification and |
| workload limits belong to the [release ledgers](README.md#release-ledgers). |
| The original route oracle came from the v0.5.6/v0.6.0 API-promotion arc. |
|
|
| Wire contracts are shared across the [supported families](../README.md#supported-model-families). |
| Native state and serving-lane support remain family-specific. |
| Tokenizer, prompt/tool syntax, and stop-token handling are dispatched by the |
| loaded model family without changing the endpoint schemas. |
|
|
| DS4 serves four wire surfaces. "Surface" means a distinct wire contract |
| (object types, stream events, ID scheme, finish mapping, error envelope), |
| not an API vendor. OpenAI Chat and legacy Completions are different |
| surfaces: they share an endpoint family but stream different objects. |
|
|
| | Surface | Endpoint | Buffered object | Stream objects | |
| |---|---|---|---| |
| | OpenAI Chat | `POST /v1/chat/completions` | `chat.completion` | `chat.completion.chunk` deltas, `[DONE]` | |
| | OpenAI Completion | `POST /v1/completions` | `text_completion` | `text_completion` chunks with `choices[].text`, `[DONE]` | |
| | Anthropic Messages | `POST /v1/messages` | `type:"message"` | `message_start` .. `content_block_*` .. `message_delta`, `message_stop` | |
| | OpenAI Responses | `POST /v1/responses` | `object:"response"` | `response.created` .. item/delta events with `sequence_number`, `response.completed` | |
|
|
| ## Compatibility boundary |
|
|
| The project attempts to preserve these public contracts through `v0.x`: |
|
|
| - core `ds4-*` CLI flags and `DS4_*` environment variables; |
| - HTTP request/response behavior and observable sampling semantics; |
| - KVC files, EXT_TOOL_MAP, KTM trailers, recurrent payloads, and image replay |
| keys across C-save/Rust-load and Rust-save/C-load; |
| - explicit distributed byte codecs. |
|
|
| The following are internal and may change during `v0.x`: crate APIs, Rust |
| types, file layout, scheduler internals, and the native bridge ABI. Native and |
| Rust objects must be built from the same commit. Distributed transport has no |
| wire version, encryption, or authentication; use matching binaries on a |
| trusted network. |
|
|
| ## Serving lanes and current routing |
|
|
| Three lanes serve generation requests: |
|
|
| - **serial** (`generate_job`): one request at a time on the session graph. |
| Full feature set, including live tool continuation and corrective tool |
| recovery. |
| - **continuous** (`generate_continuous_jobs`): the batched engine with |
| per-row sampling, streaming, stops, and tools. The Rust scheduler operates |
| over the configured/native-fitted N-bank width, including width one, and |
| refills idle rows from the live queue during an epoch. |
| - **static** (`generate_batch_jobs`): coalesced buffered greedy batches; |
| the computed routing-needs mask must be zero. |
|
|
| The shared `route_decide` function selects a lane from each request's computed |
| needs. The routing table is: |
|
|
| | Surface | serial | continuous | static | |
| |---|---|---|---| |
| | OpenAI Chat | fallback | yes (prompt fits one bank) | buffered + greedy + non-thinking + no stops/tools | |
| | OpenAI Completion | fallback | yes (no tools/echo) | same conditions as Chat | |
| | Anthropic Messages | buffered tools, prefill-only zero, serial-owned live frontiers, fallback | streaming tools and bank-owned output-only continuations | needs-free buffered | |
| | OpenAI Responses | buffered tools, live reasoning state, serial-owned live frontiers, fallback | same as Anthropic | needs-free buffered | |
|
|
| Buffered tool generation stays serial because its model-visible corrective |
| retry has no row-local batched equivalent. Streaming tool turns publish |
| bank-owned continuation records. Output-only follow-ups reclaim the bank only |
| when its generation and frontier still match; eviction respects the record's |
| grace and pin windows. |
|
|
| `DS4_SERVER_CONT_ANTHROPIC` and `DS4_SERVER_CONT_RESPONSES` control stateless |
| continuous routing. `DS4_SERVER_CONT_TOOLS_ANTHROPIC` and |
| `DS4_SERVER_CONT_TOOLS_RESPONSES` control tool routing, and take effect only |
| while the corresponding stateless switch is on. |
|
|
| Within OpenAI, still serial: non-streaming `return_token_ids` chat, |
| completion-kind requests with `return_token_ids`, and completion-kind |
| requests carrying tools. |
|
|
| GLM 5.3 text and image requests use the serial lane. Inline PNG/JPEG inputs |
| have message shapes in Chat Completions, Responses and Anthropic Messages; |
| legacy Completions has no image form. GLM live evidence covers text/image |
| Chat with the exact Q2 main GGUF and vision sidecar. Responses and Anthropic |
| image shapes have model-free parser coverage, not equivalent GLM live gates. |
| See the [artifact and API scope](ds4-dfm-model-families.md#glm-53-flash-release-scope). |
|
|
| Inkling Small uses serial CUDA serving with text output. Chat, Responses and |
| Messages accept PNG/JPEG images; only Chat accepts 16 kHz PCM/float WAV audio. |
| Legacy Completions accepts text only. The optional eight-layer MTP sidecar |
| preserves the wire contracts and uses speculative decoding for unforced greedy |
| tokens. See [MQ85GB validation and limits](inkling-small.md). |
|
|
| ## Inline images |
|
|
| Qwen and GLM image content is accepted in the existing API-native shapes: |
|
|
| ```text |
| Chat: {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}} |
| Responses: {"type":"input_image","image_url":"data:image/jpeg;base64,..."} |
| Anthropic: {"type":"image","source":{"type":"base64","media_type":"image/jpeg","data":"..."}} |
| ``` |
|
|
| ## Output-budget (`max_tokens`) semantics today |
| |
| All four parsers reject negative decode budgets with an endpoint-native |
| error naming the supplied field. Numeric values above `i32::MAX` are capped; |
| fractional values are truncated, so this is not strict integer validation. |
| The shared `request_decode_budget` helper distinguishes: |
| |
| - **omitted** — the server default (`--tokens`, default 393216); |
| - **explicit zero** — zero requested decode tokens; |
| - **positive** — the requested budget. |
| |
| The serial lane supports zero-decode prefill. Anthropic explicitly adds the |
| prefill-only routing need for a zero budget, preserving its cache-prewarm |
| contract (`stop_reason: "max_tokens"`, empty content). Other surfaces can |
| route zero-budget requests to a batched lane, whose native engine floors |
| `max_new` at one to retire the admission. Those requests may decode one token. |
| Do not rely on route-independent zero-decode behavior outside Anthropic. |
|
|
| The Anthropic parser requires `messages` but does not require |
| `max_tokens` (upstream requires it); an omitted value gets the server |
| default like every other surface. |
|
|
| ## Trust domain |
|
|
| <a id="trust-domain-inc-5"></a> |
|
|
| The server is one trust domain. The continuation registry and exact-DSML tool |
| memory are global, with no authentication or tenant namespace. Registry records |
| are keyed by `(protocol, call_id)` for each Anthropic/Responses tool-call turn. |
|
|
| A tool-call ID grants access to its conversation: an output-only continuation |
| resumes the owning serial session or batch bank. Queued continuations can evict |
| other clients' serial work or prevent bank eviction until the bounded grace/pin |
| deadline. IDs are unguessable but appear in responses. Serve mutually untrusted |
| clients through an authenticating proxy or separate servers; see the |
| [compatibility boundary](#compatibility-boundary). |
|
|
| ## Explicitly unsupported |
|
|
| - **Schema-constrained output** (v0.6.3): `response_format` with type |
| `json_object`/`json_schema` (OpenAI Chat and Completion), |
| `text.format` (Responses), and `output_format` / `output_config.format` |
| (Anthropic) are refused at parse time with HTTP 400 and a typed |
| message naming the mode, in the endpoint's native envelope |
| (ds4-on-spark#10). `{"type":"text"}`, `null`, omitted, and a typeless |
| object are accepted unchanged; the string spelling of a schema mode |
| is refused too. Decode is never schema-constrained. |
| - **Responses durable references**: non-null `previous_response_id` or |
| `conversation` values are rejected at parse time with |
| "not supported; replay full input instead". DS4 serves a stateless |
| Responses subset; clients replay full history. Literal `null` is |
| accepted and ignored. |
| - **`Idempotency-Key`**: the HTTP reader parses only `Content-Length`, |
| `Transfer-Encoding` (chunked request bodies, v0.6.3; |
| `DS4_SERVER_CHUNKED=0` restores the Content-Length-only reader), and |
| `Accept`; the header is accepted and discarded, so a retry is a new |
| generation with new IDs. |
| - **`/v1/batch`** is not implemented by the Rust host (404); the retained C |
| bulk path does not make it a production Rust endpoint. |
| - **`return_token_ids`** is a DS4 extension on OpenAI Chat and Completion, |
| with the serial-routing restrictions described above; it is not a shared |
| four-surface feature. |
|
|
| ## Historical defects and retained fixtures |
|
|
| 1. **FIXED (Inc 0b): continuous legacy-Completion streaming emitted chat |
| deltas.** `cont_on_token` projected every streaming row through the |
| chat delta machine, so a `text_completion` client received |
| `chat.completion.chunk` objects. Completion rows now stream the |
| serial oracle's plain `text_completion` chunks |
| (`cont_stream_emit_plain`); the Inc 0a negative fixture is inverted |
| (`test_cont_completion_stream_matches_serial_oracle`) and |
| `speed-bench/completion_stream_gate.sh` holds the live schema + |
| cont-engagement line. |
| 2. **FIXED (Rust host RC.3): engine failures after SSE starts now terminate |
| with the surface-native error event.** Responses preserve their live |
| `sequence_number`, Anthropic emits `event: error` with its error envelope, |
| and OpenAI streams emit an error object instead of a misleading `length` |
| finish. |
| 3. **FIXED (Inc 2b): error envelopes are endpoint-native.** Anthropic |
| buffered errors carry the documented `{"type":"error","error":{...}}` |
| envelope with a status-mapped type (400/409 `invalid_request_error`, |
| 404 `not_found_error`, 429 `rate_limit_error`, 500 `api_error`, |
| 503 `overloaded_error` — retryable in the native SDKs); Responses |
| stream errors are protocol `data:` events (`{"type":"error",...}`) |
| whose spliced `sequence_number` continues a live machine's counter. |
| OpenAI chat/completions and Responses buffered errors keep the OpenAI |
| envelope — it is the native family shape there. The flip happened |
| inside `wire_http_error`/`wire_stream_error` (Inc 1c made the surface |
| explicit at every call site, including the two 409 continuation-state |
| refusals). Negative decode budgets now reject at parse with the |
| client's own field name (`max_tokens` / `max_completion_tokens` / |
| `max_output_tokens`); explicit ZERO stays supported on every surface |
| with the lane-dependent behavior described under output budgets above. |
| 4. **FIXED (Inc 0b): admission accounting dropped decode-growth |
| commitments** once a bank's prefill landed (the old `outstanding` |
| charge covered pending prompt targets only). Every continuous |
| admission now holds a lifetime credit — its full normalized target |
| `min(prompt + decode budget, seq_cap)` — from install until the row |
| ends, and both admission verdicts (comp-cache budget and live |
| memory floor) charge the page UNION of all live credits plus the |
| candidate. The union matters: per-layer bank strides are narrower |
| than VMM pages are wide, so neighbor banks share edge pages and the |
| true union of k full banks is far below `k x virtual/bank` — summing |
| per-bank rounded projections would silently shrink live width. The |
| verdict total (resident + projected credits) is timing-independent: |
| the promise holds no matter how much of a row's growth has faulted |
| in. Gate: `speed-bench/admission_credit_gate.sh` (achieved width at |
| the default budget, pinned-budget hard-promise reject with serial |
| fallback, credit release on row death and on mid-prefill abort). |
| 5. **FIXED (Inc 0b): the v0.5.5 budget-cut honesty fix (#13) was |
| serial-only.** On the continuous lane — where chat+tools actually |
| routes — an unrepairable `max_tokens` cut inside a tool call |
| reported `finish="error"` (and a repairable one still silently |
| completes to `tool_calls`, matching serial's repair tier). The cont |
| lane now mirrors serial: the finish stays the honest `length`, the |
| partial call returns as assistant content, and the same |
| "tool call cut by token budget" marker is logged — so |
| `finish_reason_gate.sh`'s engagement oracle finally fires on the |
| lane it actually exercises. Found by the Inc 0a baseline battery |
| (deterministic fail, reproduced byte-identical on the tip-parent |
| binary — the gate had never actually proven the cont lane). The |
| full failure-cause taxonomy (stranding, aborts) remains Inc 2 |
| typed-outcome work. |
|
|
| ## Recorded quirks (current behavior, not upstream-shaped) |
|
|
| - **Anthropic response IDs**: the host mints a shared `chatcmpl-N` / |
| `cmpl-N` job ID for every surface, and `anthropic_final_response` / |
| the Anthropic stream use it directly — so Anthropic clients see |
| `"id":"chatcmpl-N"` instead of an upstream-shaped `msg_*` ID. |
| Responses is unaffected (`responses_final_response` and |
| `responses_stream_init` mint their own `resp_*`/`rs_*`/`msg_*` IDs). |
| This is the current compatibility behavior. |
|
|
| ## Route observation metrics |
|
|
| `GET /metrics` exposes `ds4_route_requests_total{surface=...,lane=...}` |
| (fixed cardinality: 4 surfaces x 3 lanes, all cells always emitted); |
| `GET /v1/stats` mirrors it as the `routes` section. One increment per job |
| at the moment a lane takes it — a failed batched attempt that falls back |
| to serial increments both lanes. These counters are observation only; |
| they exist so route promotion can prove engagement (an eligible request |
| actually moved lanes) instead of inferring it. |
|
|
| ## Retained C oracle fixtures (`ds4_test --server`) |
| |
| These fixtures replay deterministic token/text tapes through the C projectors, |
| validated by protocol event validators (event order, one open/close per |
| block/item, contiguous Responses `sequence_number`, UTF-8 hold-back): |
|
|
| - `test_tape_openai_chat_stream_projection` (thinking + UTF-8-split tapes) |
| - `test_tape_openai_completion_stream_projection` (the oracle for defect 1) |
| - `test_tape_anthropic_stream_projection` |
| - `test_tape_responses_stream_projection` |
| - `test_tape_buffered_final_responses` (all four buffered objects + |
| finish mapping, including Anthropic `length -> max_tokens`) |
| - `test_cont_completion_stream_matches_serial_oracle` (the inverted |
| Inc 0a negative fixture for defect 1 — cont and serial now share the |
| legacy-Completion stream shape) |
| - `test_route_decisions_record_current_dispatch` |
| - `test_idempotency_key_header_is_ignored` |
| - `test_responses_durable_references_rejected_at_parse` |
| - `test_error_envelopes_native_shapes` (defect 3, now asserting the |
| native shapes + the negative-budget parse rejections) |
|
|
| Existing per-feature streaming tests (`test_openai_tool_stream_*`, |
| `test_anthropic_tool_stream_*`, `test_responses_*`) cover tool-call |
| projection per surface and remain part of the oracle. Current Rust checks |
| include `make test-server-parity` and the `ds4-server` crate tests; see |
| [CONTRIBUTING.md](../CONTRIBUTING.md). |
|
|
| Live-sampled output is never a byte oracle: continuous temp-0 emissions |
| jitter run-to-run, so live end-to-end gates assert schema, event automata, |
| route engagement, and semantic equivalence — not byte identity. |
|
|
