# Serving contract

Use one set of options across families. Implementation, memory and verified
limits remain specific to the [artifact](ds4-dfm-model-families.md).
See the [v0.1.3 ledger](releases/v0.1.3.md) for recorded gates.

[Flags](#flags) · [Inspect](#inspect) · [Request trace](#request-trace) ·
[Owner/worker](#weight-owner-and-worker) · [Model replacement](#replacing-a-model)

The host resolves one [`ServingRequest`](../crates/ds4-core/src/serving.rs)
into a [`ResolvedPlan`](../crates/ds4-core/src/serving.rs): requested,
effective, and qualified. `--print-plan` prints it. `--check-config`
exits 2 if any requested feature is unsupported. `GET /v1/stats` includes
the same object plus `last_request`.

## Flags

| Flag | Meaning | Aliases |
|---|---|---|
| `--ctx` / `-c` | Per-sequence context limit, capped per family | |
| `--max-seqs N\|auto` | Concurrent banks/sequences | `--cont-width`, `DS4_SERVER_COALESCE_MAX` |
| `--prefix-reuse off\|exact\|partial\|auto` | Conversation reuse policy | `DS4_SERVER_FORK`, `DS4_SERVER_FORK_PARTIAL` |
| `--mtp-mode off\|auto\|on` | Speculation policy | `DS4_MTP_SPEC_DISABLE` for off-with-weights |
| `--mtp PATH`, `--mtp-draft N` | Sidecar and draft length | |
| `--ignore-eos-in-reasoning` | Suppress EOS/EOT while the reasoning block is open | |
| `--ignore-eos` | Suppress EOS/EOT for the whole generation | |
| `--kv-disk-dir`, `--kv-disk-space-mb` | Persistent checkpoint store | `--kv-disk-space 32G` |
| `--prefill-chunk`, `--prefill-chunk-live` | Scheduler yield sizes, capped at 8,192 | `DS4_CONT_PREFILL_CHUNK`, `DS4_CONT_PREFILL_CHUNK_LIVE`, `DS4_CONT_PREFILL_NOFENCE=1` lifts the cap |
| `--mem-floor-gb` | Single host floor | `DS4_MEM_FLOOR_GB` (published for native) |
| `--print-plan` | Print resolved JSON and continue | |
| `--check-config` | Print resolved JSON and exit | |

### Prefix reuse

`auto` reuse is the best *qualified* path. Forced `partial` on a family
that only has exact-frontier reuse is an error, and so is forcing it when
this process cannot run it: checkpoint replay lives in the bank driver and
needs the opened runtime's checkpoint store, so serial-only serving or a
runtime without that store rejects `partial` and downgrades `auto` to
`exact` with a warning rather than reporting reuse it will not perform.

### Stop policy

EOS follows the model by default. These server-wide flags are explicit
workarounds for models that emit a terminator too early. Both mask the
model's EOS and any EOT registered as a generation stop before sampling;
other generation stops, request stop strings, and output/context limits
still apply. The reasoning flag stops masking
after `</think>`; `--ignore-eos` takes precedence if both are set. Global
suppression can make generation run until another stop or a length limit.
The policy is enforced in serial and continuous decoding, including MTP
target sampling. `--check-config` validates that an opted-in model has EOS.

### Context and concurrency

`--max-seqs` is not context length. Keeping N banks is not the same as
batching N requests in one kernel. Step banks each own KV and prefill
scratch; more banks are not a linear tok/s gain. `auto` may fit fewer
banks than it asked for; an explicit `--max-seqs N` the native fit
cannot honour is an error, not a narrower start.

### Memory fit

The quote's `floor` includes native fit headroom where required: normally
the host floor plus a 2 GiB burst reserve. `DS4_BATCH_FIT_HEADROOM_MB`
overrides that reserve; `DS4_BATCH_FIT_HEADROOM_DERIVED=0` selects 6 GiB,
otherwise `DS4_BATCH_FIT_BURST_MB` changes the burst. The quote always
preserves at least `--mem-floor-gb`.
CUDA serial graph fitting also reserves 1 GiB by default, configurable
with `DS4_SESSION_GRAPH_HEADROOM_MB` (disabled by `DS4_SESSION_GRAPH_FIT=0`).
The quote keeps the larger of the applicable serial and batch reserves.
DeepSeek shared graph costs include its initial caches and native workspace;
packed cache mirrors are conservatively included even when native VMM
support may disable them. The same native chunk sizes serial and batch graphs.
DeepSeek bank costs use full-depth compressed-cache capacity plus raw rings
and rollback states. Loaded DSpark runtime costs count even with `DS4_CONT_DSPARK=0`.
A manifest with drafter ranges defers the pre-open quote: import can soft-fail.
After open, a successful import excludes shared drafter weights; fallback
retains the local weight cost. Runtime allocations remain charged in both cases.

### Serving lanes

`--cont-width 0` and `DS4_SERVER_COALESCE_MAX=0` keep the legacy serial
meaning: no bank lane. `DS4_SERVER_CONTINUOUS=0` is narrower — it forces
the static/serial route, so the banks stay for the static lane to coalesce
over while no request enters the bank driver. Qwen and DeepSeek speculate only inside that
lane, so the combination rejects `--mtp-mode on` instead of reporting
MTP enabled. Inkling and Step also speculate on the serial engine and
are unaffected.

### Disk checkpoints

Disk KV is not active-bank offload. Resident bank state, partial
checkpoint memory, and disk budget are separate. A directory does not
persist conversations below the bank persist threshold (default 8,192
tokens on the continuous lane, `DS4_SERVER_PERSIST_MIN_TOKENS`). That is a
different number from the disk store's record minimum
(`--kv-cache-min-tokens`, default 512); the plan reports both.
MiMo's [256K mixed gate](mimo2-serving-2026-09-25.md) uses disk KV for text
bank continuation; media runs on the serial lane.

Resident-bank protection and SSD checkpoint eligibility are independent:
`DS4_SERVER_PIN_MIN_TOKENS` defaults to 65,536, while
`DS4_SERVER_PERSIST_MIN_TOKENS` defaults to 8,192. Lowering the persistence
threshold does not pin shallow sessions in memory.

HTTP disk records require a matching `local-file-stat-v1` identity: all
GGUF/sidecar file metadata, template contents, runtime files and effective
inference settings. This is local file identity, not full weight-content
attestation. Inputs must remain unchanged during model open. Legacy records
without this identity miss safely; all identities share the directory budget.
Native restore holds the validated file open through the payload read.

### Draft execution

MTP weights loaded is not "this request speculated". Sampled Step
requests keep predictor state and use ordinary decode.

## Inspect

```sh
./ds4-server --check-config --cuda -m "$MODEL" \
  --prefix-reuse auto --max-seqs 2 --mem-floor-gb 12 \
  --kv-disk-dir /tmp/ds4-kv --kv-disk-space 32G --mtp-mode auto
```

The JSON has `requested`, `effective`, `qualified`, `issues`, and when
the host supplied a memory ceiling, `quote`. `effective.native_chunk` is
the allocated workspace/graph max, not the scheduler yield.
`qualified.prompt` is the verified request length when it is smaller
than configured `--ctx`. A family's session cap is separate from its
qualified context: GLM sessions refuse anything above 2,048, so the
shared default is an error there, not a warning. `--native-chunk` sets
the allocated native capacity so `--check-config` can refuse a yield
the process cannot run.

## Request trace

`GET /v1/stats` field `last_request`:

| Field | Meaning |
|---|---|
| `effective_lane` | `serial`, `continuous`, or `static` |
| `reuse_kind` | `cold`, `exact`, `partial`, or `fork` |
| `speculation_active` | This request used speculative decode |
| `reuse_miss` | Why a candidate was refused, when one was |
| `fallback_reason` | Why a requested path was not used |

The same fields may appear next to HTTP `timings`.

Each field is recorded where the decision is made, not inferred from
counters. `exact` reuses a state that ends at this prompt's common
prefix and prefills only the appended turn, so cached and computed
tokens are both positive; `partial` restores a checkpoint below that
prefix and replays the gap, including a partial copy into another bank;
`fork` copies a complete retained frontier into another bank and preserves
the source. `speculation_active` follows the executed path: the serial
engine's speculative eval, or a native sequence that ran draft rows.

## Miss reasons

Restore miss is not "disk broken". `reuse_miss` carries the reason from the
decision that produced it, and is absent when nothing was refused:

- `rendered prefix changed` (template dropped an empty thinking block).
  Step and Motif official follow-up renders omit a generation-only empty
  `<think>` pair. Their history checkpoints capture matching native KV,
  tokens and logits before that suffix; the completed bank retains its
  original token sequence. A Motif append can therefore require `partial`
  rollback even when the visible messages only grow. Without a compatible
  checkpoint, the request stays cold: shortening a text key cannot change KV.
- `below minimum token threshold`
- `payload family/layout mismatch`. Also the answer when the chosen
  payload cannot be read back — a truncated or corrupt record is refused
  by its payload, whatever wrote it.
- `no checkpoint at or below LCP`
- `session state requires prefix replay`. Cached tokens count only the
  prefix preserved by the native sync plan. Dots3 MTP replays an unaligned
  append from zero (`cold`); plain Dots3 replays its final partial chunk
  (`partial`). An identical prompt or aligned append retains its full hit.

## Capability table

The [generated capability table](serving-capabilities.md) reads
`ds4_core::serving_caps` and the same resolved controls consumed by
`ds4-perf serving-controls`. Its model-free check runs with the core tests;
the page includes the regeneration command. Dated reports stay historical.

## Weight owner and worker

On a 128 GB unified-memory machine, keep one weight owner alive and restart
only inference workers while developing or profiling. The owner maps split
GGUFs as one logical model, uploads VMM ranges, builds byte-neutral aligned
IQ2/Q2K expert artifacts, and brokers POSIX file descriptors to workers.

Keep the manifest path short: its Unix socket is `<manifest>.sock`.
Start with a dry run:

```sh
MODEL=/path/to/model.gguf
RUN=/path/to/run-directory

./ds4_weight_server \
  --base "$MODEL" \
  --manifest "$RUN/weights.manifest" \
  --backend vmm \
  --scope base \
  --reserve-gb 24 \
  --no-repack-q8-aligned \
  --dry-run
```

If the memory preflight passes, run the same command without `--dry-run` in a
durable tmux session. Do not start a worker until the owner reports both
`broker listening` and `ready manifest=...`.

A VMM-backed worker uses this launch shape. Select its context, memory reserve,
repack policy and sidecars from the [family recipe](ds4-dfm-model-families.md).

```sh
DS4_CUDA_WEIGHT_IPC_MANIFEST="$RUN/weights.manifest" \
DS4_CUDA_WEIGHT_IPC_SCOPE=base \
./ds4-server -m "$MODEL" --cuda -c 2048 \
  --host 127.0.0.1 --port 8001 --no-update-check
```

For a split GGUF, `MODEL` is its first shard. Keep the owner resident across
worker restarts; enable drafting only through the artifact's explicit contract.
Use the [memory guard](host-memory-guard.md) for large-model validation.

## Replacing a model

Before replacing your large model on DGX Spark:

1. Inspect compute processes in `nvtop` and process memory in `btop` or `htop`.
2. Stop your inference worker; verify its PID and listening port are gone.
3. Stop its weight owner; verify that owner's PID and GPU allocations are gone.
4. Run `/usr/local/bin/clear_cache` after those processes have exited.
5. Recheck process memory, `free -h` and swap before starting the next owner.

Preserve unrelated servers and resident owners. Cache reclamation cannot free
allocations held by a live CUDA process. Do not load independent huge model
copies concurrently on the reference machine.