code/dspark/CONTRACT.md · ProprietaryLegal/dsv41-async-hostflag-handoff at main

dsv41-async-hostflag-handoff / code /dspark /CONTRACT.md
2026-10-23 update: measured on GB10 (out-of-process proxy: -0.9 to -3.9 %/round one card, -3.9 to -6.5 % two cards, TTFT parity, 0 watchdog fires); in-process cost finding; hardening (late-proxy corruption, start fallback, stall detection, respawn, shutdown) with independent-review evidence; full proxy code
da83667 verified 11 days ago
preview code
Raw History Blame Contribute Delete
41.3 kB
# Remote DSpark drafter — wire and state contract (DSV41-OFFLOAD)

Contract version **1.8** (2026-10-22). Owner: lane O3 (Spark side). Lane O2 implements the x86host worker against it.
Every change is listed in the change log (§13) with a version bump; the wire `version` field changes only on an
incompatible change. Propose changes in `O2.progress.md` (the coordinator relays them).

This contract comes from the proposer that runs in the live serve. File references point to the read-only copy of the
image at `/data/dsv41-offload/src-image/` (vLLM `0.31.1.dev0+g6e517b15.dsv41gb10`, image `sha256:435a5236`). It
does not come from the upstream remote-dspark mod (vLLM 0.29, GLM/Kimi), which uses a different state model.

## 1. Plain summary

- Each engine step, Spark rank 0 sends the worker the **target tap rows** for the positions the target has just
  accepted. Rows are only sent for those positions, and never more than the last 128 per request. Rank 0 also sends
  one **anchor token** per request.
- The worker keeps a **128-position window** of those rows per request slot. It drafts a block of **5 tokens**, with
  a **confidence** for each, by greedy argmax.
- Rank 0 broadcasts the drafts and confidences to rank 1 over the existing TP group. The verifier, rejection sampler,
  depth policy and k schedule stay unchanged on the Sparks.
- Nothing a draft produces is ever committed to the window. Only context rows that the target has verified are stored.
- Prefix-cache hits, preemption, resume and reconnection need no special worker logic. The engine already recomputes
  the last 128 tokens of a prefix hit (SWA bounded replay), and rank 0 keeps a copy of every live window, so it can
  re-send one at any time.

## 2. Facts from the running proposer (verified in source)

| fact | value | source |
|---|---|---|
| speculator | `DSparkSpeculator(DFlashSpeculator)`; `method="dspark"` → `init_speculator` | `v1/worker/gpu/spec_decode/__init__.py:35-40` |
| block / k | `num_speculative_tokens = 5` = `dspark_block_size`; anchor-as-first: N = 5 query tokens per request (`[anchor, noise×4]`), every query position predicts the next token | `dspark/speculator.py:47-56`, `dflash/speculator.py:624-671` (`SAMPLE_FROM_ANCHOR`) |
| noise token | `dspark_noise_token_id = 128799` (`parallel_drafting_token_id`) | config.json; `spec_decode/utils.py:85` |
| conditioning | `main_x = main_norm(main_proj(cat(aux_hidden_states, -1)))`, `main_proj` 15,360 → 5,120 (`ReplicatedLinear`, EXL3 K5 in this checkpoint), `main_norm` RMSNorm eps 1e-20 | `nvidia/dspark.py:104-112, 163-168`; `dflash/speculator.py:376-382`; config `quantization_config.exl3_moe.dense.drafter.main_proj = 5` |
| taps | `aux_hidden_states` = 3 tensors `[T, 5120]` bf16, in the order of `dspark_target_layer_ids = [37, 38, 39]`. Tap L = mean over the 4 hc copies of the residual stream **entering** decoder layer L (0-based), i.e. the output of layer L−1 | `eagle3_utils.py:113-120` ("v4.1 reads the attention inputs"), `nvidia/model.py:540-541, 1119-1138` |
| tap validity | in eager prefill steps, layers ≥ 20 (`max(kv_source_layer_ids)`) run only on each request's **last min(q, 128) rows** of the step (decoder SWA bounded replay, patch 0034). The trimmed rows' taps are **zeros**. CUDA-graph (decode) steps compute every row | `nvidia/model.py:793-848`, `decoder_replay_layers.py:65-85`, `nvidia/model_state.py:407-433` (`kept_lens = min(query_lens, window)`) |
| tap placement over TP | replicated on both ranks. The live config has no sequence parallel (no `--enable-expert-parallel`); with SP the engine all-gathers aux before returning. **Rank 0 alone holds complete taps** | `nvidia/model.py:245-255, 1134-1138` |
| context KV | each draft layer: `kv = kv_norm(wkv part of fused_wqa_wkv(main_x))`, RoPE + quant + insert at the context positions | `nvidia/dspark.py:170-201, 256-279` |
| context rows per step | the target's rows minus the rejected suffix: `num_valid_ctx = num_scheduled − num_rejected`, positions = the target's positions of those rows | `dflash/speculator.py:570-622` |
| anchor | `last_sampled[req]` if `num_sampled > 0`, else `next_prefill_tokens[0, req]` (partial prefill). Anchor position = last valid context position + 1 | `dflash/speculator.py:578-585, 625` |
| draft window | the draft layers are pure sliding-window attention (`compress_ratios[40:43] = 0`), window W = 128, but the DSpark draft block does **not** use the causal per-token window (`[max(P−127, 0), P]`, `sparse_swa.py:1060-1066`, the target's SWA layers). Its non-causal kernel anchors one window to the whole block: with the context ending at L = anchor_pos − 1, **every** query token of the block attends to the 128 context positions `[max(anchor_pos − 128, 0), anchor_pos − 1]` plus all N query tokens of the block (the earlier and the later ones) | config `sliding_window = 128`; `v1/attention/backends/mla/sparse_swa.py:1253-1255` (`ComputeDSparkNoncausalSWAIndicesKernel`: `start_pos = max(prefix_len − window_size, 0)`, `end_pos = seq_len`), `:550-562` + `compressor_utils.py:21-27` (index width W + N, padded); `nvidia/dspark.py:9-10` |
| query clamp | query positions `min(pos, max_model_len − 1)` with max_model_len = 327,680 | `dflash/speculator.py:656-657, 679-681` |
| sampling (live) | greedy drafts, `draft_logits is None`. `VLLM_DSPARK_LOCAL_ARGMAX=1` gives the argmax of (base logits + Markov bias) and resolves ties to the lower vocabulary index (= `torch.argmax`). The sequential Markov step uses the previous **sampled** token (step 0: the anchor). Markov bias rows come from the cache `VLLM_DSPARK_MARKOV_CACHE=/root/.cache/markov_top_tokens.json` | `dspark/speculator.py:173-230, 120-130`; `nvidia/dspark.py:369-450`; live container env |
| confidence | `sigmoid(confidence_head(cat(head_hidden, markov_embed)))` per draft position. Consumed by the depth policy `VLLM_DSPARK_DEPTH_POLICY=conf` (live) | `dspark/speculator.py:223-230`; `nvidia/dspark.py:466-471`; `depth_policy.py` |
| k actually verified | decided **on the Spark only**: 0040 schedule `[[1,2,5],[3,4,4]]` (k = 5 at 1–2 requests, 4 at 3–4), 0047 adaptive window in the scheduler (`VLLM_DSV41_ADAPTIVE_K=1`, from accepted-draft EMAs), then per-request depth from the confidences (rows past the depth are dead and never accepted) | `v1/core/sched/scheduler.py:1491-1500, 2027-2036`; `model_runner.py:1528-1540, 1693-1698`; `depth_policy.py` |
| prefix-cache hit | `CacheConfig.swa_bounded_replay = True` (default; V2 runner, no PCP, not ROCm): sliding-window groups (the draft layers included) are **not** prefix-cacheable. On a hit of h tokens the scheduler rewinds to `replay_start = h − 128` and the target **recomputes those 128 tokens**, so their taps reach the drafter like any prefill rows. Hits ≤ 128 tokens are discarded and fully recomputed | `config/cache.py:240`; `models/deepseek_v41/attention.py:529-570`; `v1/kv_cache_interface.py:950-955`; `v1/core/sched/scheduler.py:1002-1012, 1055-1065, 3126-3135` |
| preemption | the V2 runner removes preempted requests (`finish_requests` unions `preempted_req_ids`) and re-adds them as new requests (prefix hit + replay as above) | `model_runner.py:1187-1199, 1216-1258` |
| request slot | `idx_mapping[i]` = request-state index 0..`max_num_seqs`−1 (4). Slots are reused | `model_runner.py:1341-1342`; `states.py` |

## 3. Conditioning layout: send the raw taps (decided with numbers)

Two candidates: `main_x` (5,120 bf16 = **10,240 B/row**, computed on rank 0) or the raw taps (3 × 5,120 bf16 =
**30,720 B/row**, `main_proj` + `main_norm` on the worker).

| item | main_x on Spark | taps to worker |
|---|---|---|
| Spark work per round | `main_proj` GEMM on rank 0. EXL3 K5 weights are 46.9 MiB; at the 273 GB/s LPDDR5x floor that is ≥ 0.18 ms per round, ~0.2–0.3 ms in practice | none (row gather only) |
| Spark memory (capacity mode) | `main_proj` + `main_norm` must stay loaded on rank 0 (≈ 47 MiB) | **no drafter tensor on either Spark** |
| wire per decode round (at most 1+k rows/request) | 1 req ≤ 61 KiB; 4 req ≤ 200 KiB | 1 req ≤ 180 KiB; 4 req ≤ 600 KiB |
| extra wire time for taps at 4.6 GB/s effective (40 Gb/s) | — | +26 µs (1 req, 6 rows) … +89 µs (4 req, 20 rows) |
| worker `main_proj` on the 3090 | — | ≈ 0.05–0.08 ms (47 MiB at ~900 GB/s) |
| prefill (≤ 128 rows/request/step, §4.3) | 1.25 MiB per request-chunk | 3.75 MiB per request-chunk (+0.5 ms against a ~1 s chunk) |

**Decision: payload kind 1 = TAPS_BF16.** The taps cost 25–90 µs more on the wire but save ≥ 0.18 ms on rank 0,
and they keep both Sparks free of drafter weights. The result: about 0.05–0.15 ms better per round at 1–4 requests,
plus a clean capacity mode. Kind 2 = MAINX_BF16 is reserved in HELLO. Revisit it only if the measured link (lane S1)
or the worker's `main_proj` time (lane O2) reverses the sign.
Row layout: `row = cat(tap37, tap38, tap39)` as little-endian bf16. That is exactly row r of
`torch.cat(aux_hidden_states, dim=-1)`, the tensor `combine_hidden_states` consumes, so the worker computes
`main_x = main_norm(main_proj(row))` with the same weights.
## 4. Per-round data flow
### 4.1 Who talks to whom
- **Rank 0** (spark-a) is the only client. It uses one connection to the worker over its direct link
  (10.202.28.2 → 10.202.28.1) through lane O1's transport (TCP or RDMA RC). It sends one request and waits for the
  reply.
- **Rank 1** (spark-b) never talks to x86host. It receives drafts, confidences and a status word in one NCCL broadcast
  from rank 0 on the TP group. Both ranks must feed identical draft tokens to the target, because TP ranks diverge
  otherwise. The broadcast also makes every fallback decision identical on both ranks.
- Why not have both ranks receive from x86host? It would save one ~20 µs NCCL hop. But the two links can fail
  independently, so the ranks would still need an agreement collective, and then nothing is saved. Rejected for v1.
  Revisit only if the coordinator measures that the TP broadcast dominates.

### 4.2 Sequence (sync mode, all policies)
1. The target forward and the rejection sampler finish on both ranks, as today.
2. Rank 0, on its GPU stream: gathers the candidate rows of the taps, drops rejected rows (device-side compaction
   using `num_rejected`, no host sync), and updates its device ring (§7.3). It then copies rows and metadata into a
   pinned staging buffer and waits for that copy.
3. Rank 0 sends `STEP` and waits for the reply, with a deadline (§10).
4. Rank 0 uploads `[status, drafts, confidences]` and broadcasts them on the TP group. Rank 1 receives the same
   tensor. Both ranks write `draft_tokens` and `draft_token_confidence_probs`.
5. The engine continues unchanged: the depth policy records confidences, the scheduler takes the drafts, the next
   verify runs.

An optional async variant exists for capacity mode (`VLLM_DSPARK_REMOTE_ASYNC=1`). The engine stream waits on a
device flag (`cuStreamWaitValue32`) that the proxy thread's side stream writes after the reply is in device memory,
so rank 0's CPU never blocks. It is transparent to the worker: the same messages, the same order.

### 4.3 Which rows are sent (the trailing-window rule)
For request i in the step, with q_i scheduled rows and `num_rejected_i` rejected:
- the candidate rows are the **last `n_i = min(q_i, W)` rows** of the request's span, with W = 128;
- the valid rows are the first `n_i − num_rejected_i` of those. `num_rejected_i > 0` only for decode rows, where
  `q_i ≤ 1 + k ≤ 6 < W`, so every candidate row is a decode row;
- valid rows are sent in position order: `first_pos` … `first_pos + n_rows − 1`, and `anchor_pos = first_pos + n_rows`.

Why this is complete:
- Within a step, only the last min(q, 128) rows have real taps (decoder bounded replay).
- The draft block reads exactly the W = 128 context positions before the anchor, `[anchor_pos − 128, anchor_pos − 1]`
  (fewer near position 0), for every query token of the block (§2, block-anchored window). It never reads further
  back, and it needs all 128: the window is 128 rows, not the 127 of a causal per-token window.
- By induction over steps, the worker always holds every position in `[max(lo, anchor_pos − W), anchor_pos − 1]`,
  where `lo` is where the request's rows begin. That is position 0 for a fresh prompt, or `replay_start` after a
  prefix hit. Bounded replay recomputes the 128 tokens before the hit (`replay_start = h − 128`), so once the first
  chunk is done every one of the 128 rows the anchor reads exists.

## 5. Transport framing

The contract is transport-independent. One request message gets one reply message. Requests on a connection are
processed in order, with one outstanding request per connection in v1.
- **TCP stub:** each message is a little-endian `u32 length` followed by `length` bytes (AppHeader + body).
  `TCP_NODELAY`, explicit local bind to the direct-link address.
- **O1 transport `ofl`** (lane O1, `offload/transport/API.md`, commit 41cf2f7 or later): one contract message is one
  ofl request.
  - ofl `op` = AppHeader.op (1–5; ofl's op 0 is its built-in echo); ofl `generation` = AppHeader.seq (ofl echoes
    it).
  - Payload = the whole contract message (AppHeader + body). The reply is the whole contract reply frame,
    RDMA-written into one registered 64 KiB range.
  - service `dspark-v1`, `max_inflight` 1, `max_req_bytes` 16 MiB (worst case STEP: 32 + 192 + 512 × 30,720 B
    = 15.7 MB), `max_resp_bytes` 64 KiB.
  - Ports: **18620** = drafter on the Spark A link (server 10.202.28.1, client binds 10.202.28.2); 18621 is
    reserved for the Spark B link. O1 uses 18610 (Engram) and 18600–18601 (self-tests).
  - An ofl timeout breaks the connection before returning (the QP goes to error), so a late reply never lands. The
    proxy then reconnects (same session) and resyncs (§7.5).
  Endpoints, as the Spark proxy writes them: `o1rdma://10.202.28.1:18620?bind=10.202.28.2` (RDMA RC) and
  `o1tcp://…` (ofl's TCP backend). The contract's own TCP stub is `tcp://…` (§5 first bullet).

## 6. Messages (little-endian; all offsets in bytes)

### 6.1 AppHeader (32 B, every message)
| off | type | field | value |
|---|---|---|---|
| 0 | u32 | magic | `0x31505344` (bytes `D S P 1`) |
| 4 | u16 | version | 1 |
| 6 | u16 | op | 1 HELLO, 2 STEP, 3 RESET, 4 STATS, 5 PING |
| 8 | u64 | seq | client sequence number, strictly increasing within a session (gaps allowed); the reply echoes it |
| 16 | u32 | session | nonzero random id chosen when the Spark proxy starts; the reply echoes it |
| 20 | u32 | body_len | bytes after the header |
| 24 | i32 | status | 0 in requests; frame status in replies (§9) |
| 28 | u32 | flags | 0 (reserved) |
### 6.2 HELLO (op 1): request body 192 B
| off | type | field | live value |
|---|---|---|---|
| 0 | u32 | proto_min | 1 |
| 4 | u32 | proto_max | 1 |
| 8 | u32 | num_spec (N) | 5 |
| 12 | u32 | window (W) | 128 |
| 16 | u32 | hidden | 5120 |
| 20 | u32 | n_taps | 3 |
| 24 | u32[4] | tap_layer_ids | 37, 38, 39, 0 |
| 40 | u32 | payload_kind | 1 = TAPS_BF16 (2 = MAINX_BF16 reserved) |
| 44 | u32 | row_bytes | 30720 |
| 48 | u32 | noise_token_id | 128799 |
| 52 | u32 | vocab_size | 129280 |
| 56 | u32 | max_model_len | 327680 |
| 60 | u32 | max_slots | 4 (`max_num_seqs`) |
| 64 | u32 | max_rows_per_step | 512 (= max_slots × W) |
| 68 | u32 | sample_mode | 1 = GREEDY (argmax of base + Markov bias, lowest index on ties, confidence head on) |
| 72 | u32 | tp_size | 2 (informative) |
| 76 | u32 | reserved | 0 |
| 80 | u8[32] | drafter_identity | sha256 (§6.7) |
| 112 | u8[32] | markov_identity | sha256 of the bytes of `markov_top_tokens.json` (zeros if no Markov cache) |
| 144 | u8[48] | client_name | ASCII, NUL-padded |

HELLO reply body (128 B), status 0 or an error from §9:
| off | type | field |
|---|---|---|
| 0 | u32 | proto (chosen) |
| 4 | u32 | max_slots |
| 8 | u32 | max_rows_per_step |
| 12 | u32 | capabilities (bit0 TAPS_BF16, bit1 MAINX_BF16, bit2 HAS_MARGIN) |
| 16 | u32 | window |
| 20 | u32 | num_spec |
| 24 | u8[32] | drafter_identity (the worker's) |
| 56 | u8[32] | markov_identity (the worker's) |
| 88 | u8[40] | worker_name (ASCII) |
The worker rejects the HELLO in these cases:
- `IDENTITY_MISMATCH` if either identity differs;
- `CONFIG_MISMATCH` if N, W, hidden, n_taps, tap ids, row_bytes, noise id, vocab or sample_mode differ, or the
  payload kind is unsupported.
A HELLO with a new session id makes the worker drop all state of any previous session. The worker serves one client.
A HELLO with the current session id (a reconnect) keeps the windows, but the client resyncs every live slot anyway
(§7.4).
### 6.3 STEP (op 2): the per-round message
Request body:
```
StepHeader (32 B)
  0  u64 step_id      rank-0 engine step counter (monotonic; echoed; informative)
  8  u16 n_req        1..max_slots
 10  u16 step_flags   bit0 WANT_MARGIN (return top1-top2 margins, §6.3 reply), others 0
 12  u32 n_rows       total rows in the row block
 16  u32 row_bytes    = HELLO row_bytes
 20  u32 rows_offset  offset of the row block from the start of the body, multiple of 64
 24  u32 reserved[2]
ReqEntry[n_req] (32 B each), batch order
  0  u8  slot         request-state index 0..max_slots-1 (unique within a STEP)
  1  u8  req_flags    bit0 NEW_GEN  discard the slot's window before applying rows
                      bit1 NEED_DRAFT  return drafts/confidences (0 = partial prefill chunk: append only)
                      bit2 RESYNC  the rows are a re-send of the window from the Spark's ring (informative)
  2  u8  k_window     drafts the target verified for this request in the step that produced these rows
                      (0 for prefill; informative, for worker acceptance stats)
  3  u8  n_out        draft positions the Spark will consume, 0 = all N (v1.3; sent as 0 before). The worker
                      still runs the full N-token block, so positions < n_out are bit-identical to a full-block
                      draft; it may skip head / Markov / confidence for positions >= n_out, which it returns
                      as draft 0, confidence 0.0, margin 0.0
  4  u32 generation   per-slot generation (§7.1)
  8  i64 first_pos    position of this request's first row (ignored when n_rows == 0)
 16  u32 n_rows       rows of this request in the row block: valid context rows only, 0 <= n_rows <= W
 20  i32 anchor_token token at anchor_pos
 24  i64 anchor_pos   last valid context position + 1 (= first_pos + n_rows when n_rows > 0)
zero padding up to rows_offset
Row block: n_rows x row_bytes; entries in ReqEntry order, positions ascending within an entry
```
Reply body:
```
ReplyHeader (32 B)
  0  u64 step_id        echo
  8  u16 n_req          echo
 10  u16 num_spec       N
 12  u32 reply_flags    bit0 HAS_MARGIN
 16  u32 worker_us      apply + draft compute time, µs
 20  u32 recv_to_send_us  µs from request fully received to reply handed to the transport
 24  u32 reserved[2]
ReqStatus[n_req] (8 B): u16 status (§9.2), u16 window_rows, u32 generation (echo)
draft_ids  : i32 [n_req][N]   target vocabulary ids (full-vocab drafter; no remap)
confidence : f32 [n_req][N]   sigmoid outputs in [0, 1]
margin     : f32 [n_req][N]   only if HAS_MARGIN: top1 − top2 of (base + Markov) logits at each step
```
For entries without NEED_DRAFT, and for entries with a nonzero per-request error, the worker writes draft ids 0 and
confidences 0.0. The same applies to positions ≥ n_out (§6.3 ReqEntry).
**Reply values the Spark refuses (v1.4).** The drafts become the target's input ids on both ranks and the confidences
feed the depth policy, so rank 0 checks every reply before anything is broadcast. Besides the shape and the echoes
(`step_id`, `n_req`, `num_spec`, per-entry `generation`), a reply is refused when, for any entry:
- `status` is not a §9.2 code;
- `window_rows > min(W, anchor_pos)`; or, for a NEED_DRAFT entry, status `OK` with `window_rows != min(W, anchor_pos)`
  or `PARTIAL_WINDOW` with `window_rows ≥ min(W, anchor_pos)` (§7.2 step 5);
- a consumed position (NEED_DRAFT, status OK or PARTIAL_WINDOW, position < n_out) has a draft id outside
  `[0, vocab_size)`, a confidence that is not a finite value in [0, 1], or (HAS_MARGIN) a margin that is not finite
  and ≥ 0;
- any other position has a nonzero draft id, confidence or margin.
A refused reply is a failed round (counter `fallback_bad_reply`, `last_bad_reply` in the stats): no draft of that
reply is used, rank 0 drops the connection, and after the reconnect every live slot resyncs (§7.4).
**n_out (v1.3).** The Spark sends `n_out = cap[n]` when a dynamic speculation schedule is set and the depth policy runs
a mode that masks past the cap (`conf`, `confb` or `confr`, with no `VLLM_DSPARK_DEPTH_FILE`, which could switch the
mode at runtime). All of these are live, with schedule `[[1,2,5],[3,4,4]]`, so n_out is 4 at 3–4 requests. Otherwise
it sends 0 (all N): `fixed:<d>` and `off` verify by the scheduler window alone, which can exceed `cap[n]` when the
next step has fewer requests. The shadow policy always sends 0, so parity covers every position.
`VLLM_DSPARK_REMOTE_N_OUT=0` turns n_out off. In the cases where n_out is sent, nothing reads the positions ≥ cap:
- `DSparkDepthPolicy.record` zeroes the confidences at positions ≥ `cap[num_reqs]` of the current step before
  choosing depths (`depth_policy.py:432-434`).
- Rows past a request's depth are dead and never accepted (`model_runner.py:1528-1540, 1693-1698`), even when the
  next step has fewer requests.
- Calibration only uses positions below the verified depth.

Measured by lane O2: at 4 requests × n_out 4 the head runs 16 instead of 20 rows (a 16-row instead of a 32-row EXL3
tile), saving ≈ 0.55 ms per round.
### 6.4 RESET (op 3)
Body `u32 slot_mask` (bit i = slot i; `0xFFFFFFFF` = all). The worker discards those windows. Reply: empty body.

### 6.5 STATS (op 4)
Empty body. Reply body: UTF-8 JSON of worker counters. Suggested keys: `steps`, `rows`, `drafted_requests`,
`partial_windows`, `stale_generation`, `bad_entries`, `worker_us_p50/p99`, `accepted_hist` (from n_rows−1 against
k_window), `gpu_mem_bytes`, `session`, `uptime_s`.

### 6.6 PING (op 5)
Empty body. Reply: empty body, status 0. Used for health checks and RTT probes. A PING never changes state.

### 6.7 drafter_identity
`sha256` over the lines `"<basename> <sha256-hex>\n"`, sorted by basename, for every source file the worker reads
tensors or configuration from. For the live build these are:
- `config.json`, `exl3_moe_plan.json`;
- `model-00002`, `model-00043` … `model-00046-of-00046.safetensors`;
- `model-exl3-head.safetensors`, `model-exl3-mtp.safetensors`, `model-exl3-mtp-experts.safetensors`.
Each file's sha256 is the **Spark-side source hash** that lane S2 recorded in `DSPARK_COPY_DONE` (`files[].sha256`;
x86host `/data/dsv41-offload/dspark/DSPARK_COPY_DONE`). For the eight tensor files this equals the deployment's
pinned `manifest.json`. config.json and exl3_moe_plan.json are legal-v2a's own and are pinned by the receipt.
`offload/dspark/spark/tools/identity.py` computes the digest from the receipt; the coordinator passes it to the Spark
as `VLLM_DSPARK_REMOTE_IDENTITY`.

The worker computes the same digest from the same receipt. It must also check that the extracted files it actually
loads match the receipt's `extracted_files[].sha256`, which is how its tensors are tied to these sources. A worker
built from the baseline checkpoint, or any other one, is refused at HELLO.

**Live values (legal-v2a, 2026-10-22):**
- `drafter_identity = be2c1436fa9f9e66b398e1b1da2e140e6fddc82a11939e2ac4240a0ca79543bd`
- `markov_identity = 0b6e71e638abd5abce2db433965bb5ff523cbbe92ed7ae54324a50ccaeffa19a`, the sha256 of the live
  `markov_top_tokens.json`. It is byte-identical on both Sparks and on x86host.

## 7. State and lifecycle

### 7.1 Slot, generation and the Spark's bookkeeping (rank 0)
- The worker keeps per slot: `generation`, the contiguous position range `[lo, hi]` of the rows it holds (or empty),
  and the row data keyed by `pos mod W`.
- Rank 0 keeps per slot: `owner` (request id), `generation`, the same `[lo, hi]`, the device ring rows (§7.3) and a
  `dirty` bit.
- Each round, for each request in the batch:
  - If its request id differs from the slot's `owner`, the slot was reused (finish, abort, preemption or a new
    request). Rank 0 sets `owner`, increments `generation` (u32, wraps; compared by serial-number arithmetic), clears
    the ring range and sends NEW_GEN.
  - Nothing else changes a slot's generation on rank 0 (failures, resyncs and reconnects keep it), so within a
    session rank 0 never sends a slot a generation older than one it sent before, with or without NEW_GEN.
  - If the same request comes back at a lower position (re-add after preemption, streaming-input update, prefix
    replay), `first_pos ≤ hi`. Both sides roll back: rows at positions ≥ `first_pos` are dropped (§7.2). Rows below
    stay valid because a causal hidden state depends only on earlier tokens. The generation does not change.
- No CLOSE message exists. A finished request's window stays on the worker until its slot is reused, which takes at
  most `max_slots` × (W × row_bytes + KV) of worker memory. Cancellation and abort are therefore implicit. An
  explicit RESET is available.
### 7.2 Worker apply rule (per ReqEntry, in order)
1. If `generation` is older than the slot's: status `STALE_GENERATION`; ignore the entry (the window is kept).
   This check comes first and holds **whatever the flags**: NEW_GEN never makes an older generation acceptable.
   If it is newer, or NEW_GEN is set (with the slot's own generation: a resync): empty the window and adopt the
   generation.
2. Validate: `0 ≤ n_rows ≤ W`; `first_pos ≥ 0`; if `n_rows > 0` then `anchor_pos == first_pos + n_rows`; the rows
   fit in the block. On failure: `BAD_ENTRY`.
3. Apply rows `[p0, p0+n)`:
   - if the window is empty, or `p0 > hi + 1` (a gap): `lo = p0`;
   - else (`p0 ≤ hi + 1`): drop rows ≥ `p0`, i.e. roll back, and set `lo = min(lo, p0)`;
   - then `hi = p0 + n − 1` and `lo = max(lo, hi − W + 1)`.

   If `n_rows == 0`: require `hi + 1 ≥ anchor_pos`, then roll back to `hi = anchor_pos − 1`.
4. After applying, require `hi == anchor_pos − 1`, else `BAD_ENTRY`. Then
   `window_rows = hi − max(lo, anchor_pos − W) + 1` and `expected = min(W, anchor_pos)`.
5. If NEED_DRAFT:
   - draft with queries `[anchor_token, noise × (N−1)]` at positions
     `min(anchor_pos + j, max_model_len − 1)`, j = 0..N−1, over the window rows;
   - status `PARTIAL_WINDOW` if `window_rows < expected`. This is non-fatal: the drafts are produced but degraded,
     and the verifier keeps the output exact. Otherwise status `OK`.
6. Draft (query) rows and draft tokens are **never** inserted into the window. The next round's rows replace them.
### 7.3 Rank 0's device ring (the window's backup copy)
- Rank 0 writes every valid row it sends into `ring[slot, pos mod W]` with the same `[lo, hi]` rule, on the GPU.
  Size: 4 slots × 128 × 30,720 B = **15 MiB**, rank 0 only.
- This ring is rank 0's own copy of every live window. The worker's copy is a cache that rank 0 can rebuild at any
  time. In capacity mode the 15 MiB counts as retained memory.

### 7.4 Resync (failure recovery without a local drafter)
Rank 0 marks a slot `dirty` when:
- a STEP that carried rows for it failed, timed out or got no reply (rank 0 cannot know whether the worker applied
  it);
- after a reconnect, `NO_SESSION` or `STALE_SEQ`;
- the slot got a per-request status other than `OK`.

The next time a dirty slot appears in a batch, rank 0 sends `NEW_GEN | RESYNC` with the slot's whole ring window.
That means rows `[max(lo, anchor_pos − W), hi]`, already including this round's new rows. The worker's window then
holds exactly the rows it would have held with no failure. Its derived KV is recomputed in a different batch shape
(≤128 rows at once instead of 1–6 per step), and EXL3 kernel rounding depends on the batch size, so drafts after a
resync can differ at near-ties (lane O2 observed 2 of 5 positions in one test). The verifier keeps the output
exact; only proposals can change. No content-addressed prefix store is needed while SWA bounded replay is on (§2). If a future config turns bounded replay off, a hit at h leaves
`[h − 128, h − 1]` without rows: the worker reports `PARTIAL_WINDOW` and rank 0 counts it. A `PREFIX_RESTORE` op is
reserved (op 6) for that case and is not part of v1.

### 7.5 Sessions, sequence numbers, connections
- `session`: a random nonzero u32 per Spark proxy start. It also changes after an engine restart.
- `seq`: starts at 1 and strictly increases. The worker records the last applied seq and rejects `seq ≤ last` with
  `STALE_SEQ`, without applying the frame. The client never retries a STEP; after a failure it sends a new STEP (with
  RESYNC) under a new seq.
- **Timeout:** the client closes the connection and later reconnects with the same session, sending HELLO again.
  Frames from the dead connection that the worker processes later are either already older than the RESYNC (the
  worker processes frames in the order received) or rejected by seq.
- The worker accepts a new connection on a listener by closing the previous one on that listener. Newest wins.

## 8. Budgets

### 8.1 Bytes per round (TAPS_BF16, 30,720 B/row; headers excluded except where shown)
| case | rows sent (max) | request | reply |
|---|---|---|---|
| decode, 1 request, k=5 | 1 + accepted ≤ 6 | ≤ 184,320 B + 96 B headers | 32 + 8 + 40 = 80 B |
| decode, 2 requests, k=5 | ≤ 12 | ≤ 368,640 B | 32 + 16 + 80 = 128 B |
| decode, 3 requests, k=4 | ≤ 15 | ≤ 460,800 B | 32 + 24 + 120 = 176 B |
| decode, 4 requests, k=4 | ≤ 20 | ≤ 614,400 B | 32 + 32 + 160 = 224 B |
| one 4,096-token prefill chunk (1 request) | 128 | 3,932,160 B | 80 B |
| a step holding r prefill requests (r ≤ 4) | 128 each | r × 3.93 MB | |
| without the trailing-window rule (for contrast) | 4,096 | 125.8 MB | |
At the nominal 40 Gb/s (≈ 4.6 GB/s effective) these come to:
- 40 µs for one 6-row decode request, 134 µs for 4 requests;
- 0.85 ms per prefill request-chunk.
Lane S1's measured latency and throughput replace these figures. With main_x the numbers would be one third (§3).

### 8.2 Memory
| where | item | size |
|---|---|---|
| Spark rank 0 | device ring (§7.3) | 15 MiB |
| Spark rank 0 | device staging (gather + ring-resync stage) | 2 × 512 rows × 30 KiB = 30 MiB, bounded by max_rows_per_step |
| Spark rank 0 | pinned message buffer (rows sent zero-copy over TCP; copied once into the ofl send slot) | 15 MiB |
| Spark rank 0/1 | broadcast tensor | < 1 KiB |
| worker | context rows | 4 × 128 × 30 KiB = 15 MiB (or the derived KV, worker's choice) |
| worker | per-step input | ≤ 512 rows × 30 KiB ≈ 15 MiB |
## 9. Status codes
### 9.1 Frame status (AppHeader.status in replies)
| code | name | client action |
|---|---|---|
| 0 | OK | |
| 1 | BAD_VERSION | fatal configuration error: log, disable remote (policy fallback) |
| 2 | IDENTITY_MISMATCH | fatal: log both digests, disable remote |
| 3 | CONFIG_MISMATCH | fatal: log, disable remote |
| 4 | NO_SESSION | HELLO, then resync all live slots |
| 5 | STALE_SEQ | the frame was not applied: HELLO, resync |
| 6 | MALFORMED | round failed: resync the slots in that STEP; count it |
| 7 | CAPACITY | round failed (too many rows or slots): resync; count it |
| 8 | INTERNAL | round failed: resync; count it |
| 9 | UNSUPPORTED_OP | fatal: disable remote |
### 9.2 Per-request status (ReqStatus.status)
| code | name | meaning | client action |
|---|---|---|---|
| 0 | OK | drafts valid | use them |
| 1 | PARTIAL_WINDOW | drafts valid, window had fewer than expected rows | use them; count it; mark the slot dirty (resync next round) |
| 2 | STALE_GENERATION | entry ignored | zero drafts for this request this round; dirty. Rank 0 never sends an older generation (§7.1), so the worker's state does not match rank 0's: rank 0 starts a **new session** (a HELLO with a new session id drops every window, §6.2) and resyncs every live slot (v1.5) |
| 3 | BAD_ENTRY | entry rejected (positions or sizes) | zero drafts this round; dirty; log the entry |
| 4 | INTERNAL | the worker failed on this entry | zero drafts this round; dirty |

## 10. Failure policies (Spark side; `VLLM_DSPARK_REMOTE_POLICY`)
The per-round deadline is `VLLM_DSPARK_REMOTE_TIMEOUT_MS`, default 50 ms. A normal round is a few ms; one verify
step is 33–48 ms.
| policy | local drafter | on a failed round | memory |
|---|---|---|---|
| `capacity` | **not allocated**: no draft weights, graphs or draft KV layers | drafts = 0, confidences = 0.0. Depth policy `conf` then picks depth 0 (target-only cost). Counted. The circuit breaker below applies. `VLLM_DSPARK_REMOTE_ON_FAIL=raise` raises instead (engine-fatal; for benchmarks) | freed: drafter weights, Markov cache, draft graphs and workspace, draft SWA KV layers. Retained: 15 MiB ring + staging |
| `resilience` | kept, its context KV updated every round (eager precompute of the new rows) | that round runs the local drafter (the decision is broadcast, so both ranks run it); the circuit breaker below applies | nothing freed; the local drafter is retained and reported as not freed |
| `shadow` | kept, full local propose every round | n/a | remote and local proposals compared and logged (per-position agreement, first-mismatch histogram, margins). Local drafts are used unless `VLLM_DSPARK_REMOTE_SHADOW_USE=remote` |
**Circuit breaker (rank 0, every policy; v1.6).** A worker that answers but keeps failing would otherwise cost every
engine round its round trip (frame errors, per-request errors) or, after each reconnect, one full deadline
(timeouts). Rank 0 therefore counts consecutive **failed attempts**: rounds that sent a STEP and got a timeout, a
transport error, a refused reply (§6.3), a frame error, or per-request errors for every request that needed drafts.
- After `VLLM_DSPARK_REMOTE_TRIP` failed attempts in a row (default 3), the breaker opens. The next
  `VLLM_DSPARK_REMOTE_COOLDOWN` engine rounds (default 256) send nothing. They fail fast like a round without a
  connection: capacity zero drafts, resilience the local drafter, shadow no comparison. They are counted as
  `fallback_cooldown`.
- After the cooldown, the next round that sends a STEP *with a request that needs drafts* is a probe. Success closes
  the breaker; failure opens it again for another cooldown.
- The cooldown counts engine rounds, connected or not. These rounds are neither failures nor successes (v1.8):
  - rounds that send nothing (no connection, nothing to send, or a connection newer than the one the round's entries
    were built for);
  - rounds whose STEP succeeds but has no request that needs drafts (prefill appends only). They never exercise
    drafting, so they neither reset the count nor close a probe. If such a STEP fails, it is a failure.
- A success resets the count. A round where only some requests failed is a success.
- Rank 0 keeps its ring exact while the breaker is open. Every slot of a skipped round is marked dirty, so the probe
  resyncs those windows (§7.4).
- Transport failures also keep the client's wall-clock reconnect backoff: 1 s, doubling up to 30 s, reset by a
  successful HELLO.
- `VLLM_DSPARK_REMOTE_TRIP=0` turns the breaker off: every round tries again, as before v1.6.
- Steady-state cost with a worker that times out on every STEP: one 50 ms deadline per 257 rounds (≈ 9 s at
  35 ms/round, ≈ 0.5 %). Without the breaker it is one deadline per reconnect cycle (≈ 1 s, ≈ 5 %).
A failure in the middle of a request is safe in every policy:
- `capacity` only loses proposals for that round. The target's KV and the verifier are untouched, and the worker's
  window is rebuilt from rank 0's ring (§7.4).
- `resilience` has the local context KV in sync every round, so the local drafter can take over at any round
  boundary.

## 11. What the worker must match numerically (lane O2)
- Same tensors: the identity in §6.7. Same `main_proj` (EXL3 K5) and `main_norm`.
- Context KV computed exactly like `precompute_and_store_context_kv`: wkv from the fused `wq_a|wkv`, `kv_norm`,
  RoPE at the context positions. RoPE parameters are those of the drafter's attention layers (target-config RoPE;
  `compress_ratio = 0` layers).
- Query block: `[anchor, noise×4]`, non-causal inside the block, window 128.
- The draft MoE: 128 experts, top-3, plus the shared expert.
- mHC with hc_mult 4, ending in `hc_collapse` with the last layer's FFN pre-mix.
- Base logits = the target's head (`model-exl3-head`, EXL3 K6) over `norm(head_hidden)`.
- Sequential Markov bias from the previous sampled token, using the cached rows of `markov_top_tokens.json`
  (`markov_cache.py`) where they exist and the GEMM path elsewhere, like `markov_bias_for_ids`.
- Greedy argmax with lowest-index ties.
- Confidence from `cat(head_hidden, markov_embed)` per position.
- Representation changes are allowed only on the drafter side, and must be recorded (e.g. bf16 window KV instead of
  the Spark's fp8 record).

## 12. Optional extensions (reserved, not v1)
- `PREFIX_RESTORE` (op 6): content-addressed window restore, for configurations without SWA bounded replay.
- A split round, `STEP_CTX` + `STEP_ANCHOR`: send the taps before sampling finishes, so the worker can precompute
  context KV while the Spark samples. Worth ≈ the tap transfer time per round.
- Payload kind 2 (MAINX_BF16), if measurements reverse §3.
- Probabilistic drafting (draft probabilities over the 129,280-entry vocabulary are ~2.5 MB per request per round)
  is out of scope. The remote speculator refuses to start unless `draft_sample_method == "greedy"`, with no
  `dspark_draft_topk` and no watermarking.
## 13. Change log
| version | date | change |
|---|---|---|
| 1.0 | 2026-10-22 | first version, derived from the live proposer (lane O3) |
| 1.1 | 2026-10-22 | wire unchanged (`version` stays 1). §5: O1 `ofl` mapping, limits and ports (18620). §6.7: identity from Spark-side source hashes; the worker must tie its extracted files to them; live digests recorded. |
| 1.2 | 2026-10-22 | wire unchanged. §7.4: after a resync the window rows are exact, but drafts may differ at near-ties because KV is recomputed in another batch shape (lane O2). §8.2: rank-0 buffers corrected (45 MiB device, 15 MiB pinned). §4.2: async variant described. |
| 1.3 | 2026-10-22 | ReqEntry byte 3 = `n_out` (proposed by lane O2). Backward compatible both ways (old senders send 0 = all N; old workers ignore it), so the wire version stays 1. Positions ≥ n_out return zeros; positions < n_out equal the full-block draft. |
| 1.4 | 2026-10-22 | Wire format unchanged (`version` stays 1). §6.3: the Spark now refuses replies with values the target must not see (draft ids outside the vocabulary, confidences outside [0, 1] or not finite, bad margins, unknown status, window_rows out of bounds or inconsistent with the status, a nonzero n_out tail or nonzero drafts for unconsumed entries) as a failed round + resync (S5 #3). A worker that follows §6.3 and §7.2 is unaffected (lane O2's DraftCore passes O3's suites unchanged). |
| 1.5 | 2026-10-22 | Wire format unchanged. §7.2 step 1 made explicit: an older generation is STALE_GENERATION even with NEW_GEN (S5 #2; the reference worker exempted NEW_GEN, now fixed; lane O2 fixes its worker). §7.1: rank 0's slot generation changes only on a new owner, so it never sends an older one. §9.2: on STALE_GENERATION rank 0 starts a new session and resyncs, since a same-session resync would stay stale. |
| 1.6 | 2026-10-22 | Wire format unchanged. §10: the circuit breaker 1.0–1.5 promised (`VLLM_DSPARK_REMOTE_TRIP` 3, `VLLM_DSPARK_REMOTE_COOLDOWN` 256 rounds) is now implemented as specified (S5 #5: it did not exist; frame errors were retried every round). The text now states exactly which rounds are failed attempts, which are neutral, and how the probe and the resync work; "reconnect + HELLO" after the cooldown is replaced by "one probe round" (transport failures reconnect on their own backoff). |
| 1.7 | 2026-10-22 | Wire format unchanged; explanation corrected (S5 optional). §2 / §4.3 described the draft window with the causal per-token formula (127 context rows). The DSpark block kernel (`sparse_swa.py:1253-1255`) anchors one 128-row context window `[anchor_pos − 128, anchor_pos − 1]` to the whole block, plus the full query block. The rules themselves (W = 128 rows kept and sent, `expected = min(W, anchor_pos)`) were already 128-row; lane O2's worker matches the non-causal kernel. |
| 1.8 | 2026-10-22 | Wire format unchanged. §10 refined after an independent review of the 1.6 breaker: a successful STEP that carries no request needing drafts (prefill appends only) is neutral, so it cannot reset the failure count or close a probe without drafting having worked. A STEP whose connection is newer than the epoch its round was built for is refused on rank 0 (nothing sent, neutral) and the next round resyncs. |