offload/dspark/INTEGRATION.md

Status 2026-10-23: Original integration checklist retained; capacity/async production confirmed W5. Use promotion README and roadmap for current TP2/native gates. See live roadmap and measured record; campaign review decisions are in LANES.md / lane reports outside the repo.

Remote DSpark drafter — Spark-side integration checklist (lane O3 → coordinator)

Everything below is for the coordinator (session coordinator). Lanes never restart the serve. Every step runs inside an open window (offload/launch/window.sh open …), through offload/launch/start_exp.sh <config> on :8001. To roll back, run ./start.sh, or start any config without the overlays.

Wire/state contract: offload/dspark/CONTRACT.md (v1.8). Code: offload/dspark/spark/ (lane O3, branch offload/o3-proxy). Worker: lane O2, offload/dspark/worker/ (server.py, CONTRACT §6–§9).

0. What the overlay changes, and what it does not

• One image file is replaced: vllm/v1/worker/gpu/spec_decode/__init__.py. The replacement is the image file plus one guarded branch: when VLLM_DSPARK_REMOTE is non-empty, method == "dspark" builds the remote speculator.
  • Unset, the image path runs unchanged. tests/test_speculator.py::test_overlay_is_image_plus_guarded_branch pins the diff.
  • The base file's sha256 61b3acba…aa1b7e8 equals the live container's file (checked 2026-10-22 on sparkhead).
• One package is added: vllm/v1/worker/gpu/spec_decode/remote_dspark/, the contents of offload/dspark/spark/remote_dspark/.
• Unchanged:
  • The target model, verifier, rejection sampler, depth policy, 0040 schedule, 0047 adaptive window, the scheduler, and the API/aliases.
  • SPEC JSON, --kv-cache-memory-bytes and all production env vars stay as they are, unless a step below says otherwise.
• Both ranks need the same overlays and the same env: TP ranks must make identical decisions. Only rank 0 connects to x86host; rank 1 receives drafts through one NCCL broadcast on the TP group.
• Compatible with the coordinator's capture overlay (offload/dspark/capture/speculator.py, mounted over dspark/speculator.py). The two can run together; for timing, use neither.

1. Preparation

1.1 Files (from the merged offload-x86host checkout on sparkhead)

start_exp.sh copies OVERLAYS to ~/dsv41-offload-run/overlay/ on both nodes and bind-mounts them read-only. The ready configs in offload/dspark/spark/launch-configs/ already list both overlays:

offload/dspark/spark/remote_dspark=/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu/spec_decode/remote_dspark
offload/dspark/spark/overlay/vllm/v1/worker/gpu/spec_decode/__init__.py=/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu/spec_decode/__init__.py

Copy the config you want into offload/launch/configs/, or pass its path directly.

1.2 Identities

python3 offload/dspark/spark/tools/identity.py <S2's DSPARK_COPY_DONE> prints the two values the configs carry (live legal-v2a, 2026-10-22):

VLLM_DSPARK_REMOTE_IDENTITY=be2c1436fa9f9e66b398e1b1da2e140e6fddc82a11939e2ac4240a0ca79543bd
VLLM_DSPARK_REMOTE_MARKOV_IDENTITY=0b6e71e638abd5abce2db433965bb5ff523cbbe92ed7ae54324a50ccaeffa19a

The worker must report the same values at HELLO; otherwise rank 0 disables the remote and logs both digests.

1.3 Transport

The endpoint scheme selects the protocol. The worker has to speak the same one.

• o1rdma://10.202.28.1:18620?bind=10.202.28.2: lane O1's ofl, RDMA RC. This is what all four configs use, and what O2's server is expected to serve (an ofl server, service dspark-v1).
  • Measured Spark A ↔ x86host p50: echo 5.6 µs, 75 KB 33 µs (O1 API.md).
  • o1tcp://… selects ofl's TCP backend on the same server and port, for debugging.
  • Both need libofl.so + ofl.py in VLLM_DSPARK_REMOTE_OFL_DIR; the configs mount ~/dsv41-offload-run/ofl at /opt/dsv41-ofl. Build on sparkhead's host, aarch64 (O1 API.md "Build"), and put the same directory on sparkworker so the bind mount source exists there too; rank 1 never loads it:

    cd ~/dsv41-deploy/offload/transport && make OUT=$HOME/dsv41-offload-run/ofl $HOME/dsv41-offload-run/ofl/libofl.so \
      && cp ofl.py $HOME/dsv41-offload-run/ofl/
    ssh sparkworker mkdir -p dsv41-offload-run/ofl && scp -q $HOME/dsv41-offload-run/ofl/{libofl.so,ofl.py} sparkworker:dsv41-offload-run/ofl/
    

  • O1 commit f256568 or later gives a stable payload_buf. The proxy does not use it yet: each STEP is copied into the ofl send slot once, ≈ 20–40 µs per decode round and ≈ 0.5–1 ms per 4-request prefill step.
• tcp://10.202.28.1:18620?bind=10.202.28.2: the contract's own TCP stub (length-prefixed frames). It is a different wire protocol from ofl's TCP backend. Only the reference mock_worker (TCP-stub mode) or a worker that implements the stub speaks it. It needs no library.
• remote_dspark/mock_worker.py can stand in for the worker on x86host, for plumbing checks without the GPU drafter. It serves either protocol: --ofl-dir <dir with libofl.so + ofl.py>, or the stub. Its drafts are fake, so use shadow policy only.

1.4 Offline checks (Controller, CPU)

cd offload/dspark/spark && python3 -m pytest -q tests        # 160 pass (2026-10-22, O3-FIX)

• An end-to-end engine simulator checks exact drafts through: chunked prefill with trimmed taps, decode with random k and rejections, slot reuse, re-add/replay, timeouts, worker restart, frame and per-request errors, and dropped connections.
• Also covered: the speculators with vLLM stubs, the overlay diff and dispatch, and the proxy over ofl's TCP backend (.build/ofl).

1.5 Pre-flight against the real worker (before the experiment start)

Run inside an open window or with the serve idle. The probe refuses unless /metrics shows 0 running and 0 waiting. During a window nothing listens on :8000, so pass --no-metrics-check only when the serve is down.

Run the probe on sparkhead's host with the system python3. It needs no numpy and no torch (checked with numpy hidden): it uses only remote_dspark/{wire,transport,o1_adapter}.py from the merged checkout.

VLLM_DSPARK_REMOTE_OFL_DIR=$HOME/dsv41-offload-run/ofl python3 ~/dsv41-deploy/offload/dspark/spark/tools/probe.py \
  --endpoint 'o1rdma://10.202.28.1:18620?bind=10.202.28.2' \
  --identity be2c1436fa9f9e66b398e1b1da2e140e6fddc82a11939e2ac4240a0ca79543bd \
  --markov-identity 0b6e71e638abd5abce2db433965bb5ff523cbbe92ed7ae54324a50ccaeffa19a --hello --step-rows 128

Expected:

• hello_status: OK, with the worker's identity and markov digests equal to ours;
• ping_us (TCP ~30–60 µs, RDMA ~6 µs);
• step_status: OK and step_req_status: OK for a 128-row window. step_worker_us is O2's draft time for B=1.

--hello opens a new session, which drops an engine session's windows on the worker. Use it only while no engine is connected; the engine would recover by itself (NO_SESSION → HELLO → resync).

2. Order of in-engine checks

step	config	purpose	pass criterion
1	C-shadow.env	wiring + parity. Production drafts unchanged; the remote runs alongside	the worker connects (log remote DSpark worker '…' ready); rounds_ok ≈ rounds; shadow.full_block_agree_rate high; mismatches are near-ties (examples[].remote_margin small)
2	C-shadow-remote.env	acceptance with remote proposals (local still compared)	vllm:spec_decode_num_accepted_tokens_total / num_drafts_total within noise of A on the same prompts
3	timing inputs	read latency.local_propose_us and latency.rank0_round_us from the shadow stats	replace the PLACEHOLDER REMOTE_ROUND_MS / REMOTE_DRAFT_MS in C-capacity / C-resilience (§4)
4	C-capacity.env	the capacity mode: no local drafter	starts; reclaimed memory measured (§5); outputs sane; speed vs A (bench2)
4b	C-capacity-async-cal1.env (lane A2's hand-off; CUDA_MODULE_LOADING=EAGER stays: the proxy needs the GIL, and pybind11 extension launches hold it — S9 P1)	same, with async rounds (the CPU never blocks; the engine stream waits on a host word the proxy thread writes with no CUDA call)	log remote DSpark async mode on (host-flag hand-off self-test passed; …); stats async: true, async_handoff; speed vs step 4. Deploy libdsv41_hostflag.so next to libofl.so (§1.3) or let rank 0 compile hostflag.c at startup
4c	A-local.env vs C-capacity.env (then C-capacity-nodrop.env if needed)	prefix-cache hits without the drafter's KV layers (§5, "KV group annotation")	the second request of a repeated ≥ 32K prompt has the same cached_tokens as under A
5	C-capacity(-async).env with KV_BYTES_OVERRIDE	re-sized KV after step 4's measurement	larger GPU KV cache size / Maximum concurrency in the log; the 256K/300K runs of plan §4.5
6	C-resilience.env	per-round fallback	stop the worker mid-run → local_fallback_rounds grows, outputs continue; restart it → remote resumes (resync_rounds)
7	failure drill in capacity	stop/restart the worker mid-run	the target-only rounds are counted (fallback_not_ready, fallback_timeout); after restart hellos +1, resync_rounds +1, rounds_ok grows again

Step 1 shows rank 0's gather in gather_us, and the network plus worker time in rtt_us / worker_us. In shadow mode the step is slower than A because both drafters run; do not benchmark shadow.

3. What to read

• Rank 0 stats file: docker exec dsv41-serve cat /tmp/dsv41-remote-dspark.json on sparkhead. It is written every VLLM_DSPARK_REMOTE_LOG_EVERY rounds; a log line is emitted at the same time.
  • counters: rounds_ok, rounds_failed, rounds_some_failed, fallback_<reason> (not_ready, disabled, reconnected (the connection changed after the round's entries were built; nothing sent), cooldown, timeout, transport_error, bad_reply, frame_<status>, internal_error), req_status_<status>, resync_rounds, resync_rows, rows_sent, bytes_sent, bytes_received, hellos, disconnects, connect_failures, local_fallback_rounds (resilience), rounds_n_out_cut (rounds sent with n_out < N), breaker_trips, breaker_recoveries, fallback_cooldown (rounds skipped by an open breaker).
  • breaker: state (closed / open / half_open), consecutive_failures, cooldown_left, trips, and the configured trip / cooldown (CONTRACT §10). Log lines: remote DSpark circuit breaker open (…) and … closed: the probe round succeeded.
  • last_bad_reply: why the last reply was refused. fallback_bad_reply counts replies refused for their framing or their values (CONTRACT §6.3 v1.4: draft ids outside the vocabulary, confidences not in [0, 1], bad window or status, nonzero n_out tail). Any nonzero count is a worker bug to report to lane O2; the round ran target-only (capacity) or local (resilience) and the slots resynced.
  • stale_generation_new_sessions (and req_status_stale_generation): the worker reported a slot generation newer than rank 0's. Rank 0 never sends an older one, so this means the worker's state diverged; rank 0 started a new session (the worker dropped every window) and resynced (CONTRACT §9.2, v1.5). Expected 0; report any other value.
  • latency (µs, p50/p90/p99/max):
    • rank0_round_us: rank 0's whole remote round. This is the remote cost the depth policy should get.
    • gather_us: device gather + D2H + sync.
    • rtt_us: network + worker.
    • worker_us: the worker's own report.
    • local_propose_us: shadow only; the local drafter, CUDA events.
  • shadow: agree_rate_by_position, first_mismatch_hist (index 5 = whole block agrees), full_block_agree_rate, max_conf_abs_diff, examples (local vs remote ids, worker margins).
  • worker, policy, endpoint, rank0_buffers (45 MiB device, 15 MiB pinned), async / async_fallback_reason / async_handoff (mechanism, library path, spin/nap), and fused_gather (on, check).
  • Async counters: fallback_resync_pending (a resync waited one round for its staged windows), resync_windows_staged (slot windows the engine stream copied for the proxy); rank0_submit_us (the engine thread's enqueue cost per round).
    • The one-launch Triton row gather (remote_dspark/fused_gather.py) replaces ~40 eager torch ops per round on rank 0's stream.
    • It is self-checked at startup against the torch path on the GPU (bit-identical rows, ring and metadata) and turned off with a warning on any mismatch or error. It was verified offline under TRITON_INTERPRET=1.
    • VLLM_DSPARK_REMOTE_FUSED_GATHER=0 forces the torch path.
  • n_out_by_reqs, n_out_reason (CONTRACT 1.3, §6.3): the draft positions requested from the worker by request count, and why.
    • In capacity and resilience, with the live conf policy and schedule, this is [0, 5, 5, 4, 4]: n_out 4 at 3–4 requests, which saves the worker ≈ 0.55 ms per round at 4 requests (O2).
    • Shadow always shows null (all N, so parity covers every position).
    • VLLM_DSPARK_REMOTE_N_OUT=0 turns it off.
    • A VLLM_DSPARK_DEPTH_FILE, or a fixed:/off depth policy, also turns it off.
• Serve logs:
  • remote DSpark (<policy> policy) rank 0 -> <endpoint>
  • remote DSpark worker … ready … (session …, epoch N): the epoch rises on every reconnect.
  • remote DSpark disabled for this engine run: …: identity, config or version refused. Fix it and restart.
  • In remote modes, the DSpark depth policy conf (k=5), round cost ms by depth […] line must show the remote table.
• vLLM /metrics: as bench2 reads them: vllm:spec_decode_num_drafts_total, …num_draft_tokens_total, …num_accepted_tokens_total, …num_accepted_tokens_per_pos_total.
• Worker side: O2's STATS op (tools/probe.py --hello prints it before the run) and its own logs.

4. Depth-policy costs in remote modes (recalibration)

The live VLLM_DSPARK_ROUND_MS=33.26,… is "verify forward + drafter" by depth, with VLLM_DSPARK_DRAFT_MS=0. Its cost already includes the local drafter. In capacity and resilience the remote round must be an explicit input:

• VLLM_DSPARK_REMOTE_ROUND_MS replaces VLLM_DSPARK_ROUND_MS with a verify-only table. First estimate: each live entry minus shadow local_propose_us p50 (in ms). Better: measure with fixed depths, the way the live table was made.
• VLLM_DSPARK_REMOTE_DRAFT_MS replaces VLLM_DSPARK_DRAFT_MS with the remote round: shadow rank0_round_us p50 in ms (it includes gather, network and worker; add ~0.05 ms for the TP broadcast).
• The overlay writes both into the process env before the model runner builds the depth policy. Both ranks get the same env, so the tables stay identical across TP ranks.
• With the depth policy on (live: conf), capacity and resilience refuse to start unless both REMOTE_DRAFT_MS and REMOTE_ROUND_MS are set. The policy prices depth d as ROUND_MS[d] + DRAFT_MS, so either one alone mixes the live table (local drafter included) with a remote cost. The error names what is missing, and nothing is overridden. Checked at startup:
  • REMOTE_DRAFT_MS: one finite value ≥ 0;
  • REMOTE_ROUND_MS: exactly k + 1 = 6 finite values > 0 (depth 0..5), non-decreasing. The launch configs' placeholders pass.
• VLLM_DSPARK_DEPTH_POLICY=confr:<rows ms> is refused in capacity and resilience. Its rows table is part of the policy string, measured with the local drafter, and the remote inputs cannot replace it. Use conf (live) or confb.
• A VLLM_DSPARK_DEPTH_FILE can replace the cost table at runtime (conf:<ms…>, confb:<ms…>, confr:<ms…>). In remote modes, any table in that file must be a remote-mode measurement. The overlay cannot check it.
• fixed:<d> never prices a depth, so it needs neither value unless a VLLM_DSPARK_DEPTH_FILE is set (the file can switch the mode to conf at runtime). This lets you measure the remote verify table with fixed depths without dummy costs.
• With the depth policy off, nothing reads either value. They are only parsed: a value given alone is written, and a malformed one is refused.
• Shadow ignores both and keeps the local measurements.

5. Capacity mode: what disappears, and how to measure it

Not allocated on either Spark:

• the drafter weights. mtp.* is 3.77 GiB on disk, TP-sharded, so roughly 1.9 GiB per rank plus the replicated main_proj (47 MiB) and Markov head (126 MiB);
• the Markov bias cache (4,096 rows);
• the draft CUDA graphs (FULL_DECODE_ONLY captures of the query block);
• the draft workspace lane;
• the three draft attention layers. They are never registered, so get_kv_cache_spec has no draft SWA layers and the KV-cache groups change.

Measure, A vs C-capacity, both ranks:

1. Log lines: Model loading took X GiB, Graph capturing finished … took X GiB, GPU KV cache size: N tokens, Maximum concurrency …, and the KV-cache group printout (count of groups/layers).
2. free -g / MemAvailable on each Spark after warmup, plus nvidia-smi per-process usage.
3. With --kv-cache-memory-bytes unchanged (4026531840), the token capacity should rise a little: the same bytes now serve fewer layers. The freed non-KV memory is what KV_BYTES_OVERRIDE can then take. Keep the margin the production fit was sized with; Sparks run at 5–7 GiB MemAvailable.
4. Retained for the remote path: rank 0 holds 45 MiB of device buffers (15 MiB ring + two 15 MiB stages), a 15 MiB pinned message buffer and the ofl send slot (16 MiB). Rank 1 holds a < 1 KiB broadcast tensor. The stats file reports rank0_buffers.

Also note:

• KV group annotation and prefix-cache hits (check in the capacity window).
  • What happens:
    • SPEC still says dspark, so use_eagle_block_drop() stays true (config/speculative.py:1947-1949).
    • For deepseek_v41, _uses_trailing_mtp_layers turns on _annotate_eagle_groups' positional fallback (v1/core/kv_cache_utils.py:2155-2171, :2234-2242). That fallback flags the group holding the last registered layer as the "eagle" group.
    • Under A, that group is the drafter's SWA group (the three mtp.* layers register last). In capacity mode the draft layers do not exist, so it is the group holding the last target layer: a target group.
    • If the partition check fails, no group is flagged. The coordinator then flags every group (v1/core/kv_cache_coordinator.py:104-110).
    • A flagged group loses its trailing scheduler block from prefix-cache hits, the "EAGLE last-block drop" (comment at kv_cache_coordinator.py:112-122). Hits land on scheduler-block boundaries, so a hit can come out up to one scheduler block shorter than under A, and the request prefills that much more.
    • Output is never affected, only cached tokens and prefill time.
  • How to check, in the window. Run the same steps under A (A-local.env) and under C-capacity.env:
    1. Startup log of both ranks: kv cache group sizes [...] and kv lcm block sizes N. Record how capacity differs from A; the drafter's layers are gone, so a group may disappear or merge. N is the scheduler block, the most a hit can lose. No log line names the flagged group, so step 2 is the real test.
    2. Send one long legal prompt (≥ 32K tokens, from bench2) twice in a row on an idle serve. Read the second response's usage.prompt_tokens_details.cached_tokens; production returns it.
    3. Read the deltas of vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total on /metrics across those two requests.
    4. Compare A with capacity. Equal cached tokens: nothing to do. Capacity lower by about N tokens per hit: the fallback flagged a target group.
  • Remedy, capacity mode only: launch-configs/C-capacity-nodrop.env, which is C-capacity with "disable_eagle_block_drop": true added to SPEC_OVERRIDE.
    • The drop exists so that a prefix hit never reuses a draft-KV slot written with the chunk's lookahead token (kv_cache_coordinator.py:112-122). Capacity mode has no local draft KV, so dropping a target block protects nothing.
    • Rerun step 2 under it: cached tokens should match A.
    • Never in resilience or shadow: they keep the local draft KV and need the drop.
• Wasted 160 MiB per rank in all modes, not touched here. The target allocates _mtp_hidden_buffer (4096 × 20480 bf16) because use_eagle() includes dspark (nvidia/model.py:874-886), and copies the hidden states into it every step. DSpark never reads it: the aux taps exist (dflash/speculator.py:376-381). Freeing it needs a model-side change: skip the buffer for dspark, and return a zero-width view from get_mtp_target_hidden_states, which model_runner.py:2336-2338 slices. That would reclaim 160 MiB per rank in A as well.

6. Failure policies (CONTRACT §10)

• Circuit breaker, all policies (CONTRACT §10 v1.6).
  • After VLLM_DSPARK_REMOTE_TRIP failed attempts in a row (default 3), the next VLLM_DSPARK_REMOTE_COOLDOWN rounds (default 256) skip the network. A failed attempt is a STEP that timed out, was refused or got a frame error, or a STEP where every request that needed drafts failed. Then one probe round.
  • Rounds without a connection do not count, and neither do successful rounds with nothing to draft (prefill appends only, v1.8). A round where only some requests failed is a success.
  • Why it exists: without it, a worker that answers but fails every round costs each round its RTT, and a worker that misses every deadline costs one 50 ms stall per reconnect (≈ 5 %). With it, the cost is one attempt per ≈ 9 s.
  • VLLM_DSPARK_REMOTE_TRIP=0 turns it off. That is useful in a failure drill when every attempt should hit the worker.
  • Drill expectation (step 7): with the worker stopped, rounds are fallback_not_ready while the reconnect backoff runs and never trip the breaker, since no STEP is sent. A worker that is up but broken trips it after 3 rounds.
• capacity.
  • A failed round sends zero drafts with confidence 0, so the depth policy verifies nothing that round.
  • Rounds never wait on a connect: a daemon thread reconnects with backoff (1 s doubling to 30 s), and while it is down rounds fail in ~µs.
  • On recovery: HELLO, then the whole window of every live slot is resent from rank 0's ring. Drafts are exact again from the next round.
  • VLLM_DSPARK_REMOTE_ON_FAIL=raise, benchmarks only: any failed round raises on both ranks (engine-fatal), and connect + HELLO must succeed at startup.
  • A clean failure of a single request is not possible from inside the speculator.
• resilience. After a failed round both ranks run the local drafter that round, as decided by the broadcast status. Requests that failed on the worker get local drafts; the others keep remote drafts. The local context KV is kept current every round with an eager precompute of the new rows. Nothing is freed.
• shadow. Both run. Local drafts are used unless VLLM_DSPARK_REMOTE_SHADOW_USE=remote is set; with it, failed remote requests fall back to local.
• Never a hang: every network wait has a deadline (VLLM_DSPARK_REMOTE_TIMEOUT_MS, default 50 ms), and an exception on rank 0 still broadcasts a failed status so rank 1 never waits forever.

7. Rollback

• Immediate: ./start.sh (production config; no overlay, no remote env).
• In an experiment: start the same config without OVERLAYS/EXP_DOCKER_ENV, e.g. A-local.env. Without VLLM_DSPARK_REMOTE the overlay is inert, but the plain image is the reference.
• Worker: stop O2's unit. A connected engine then runs per its policy: capacity → target-only, resilience → local drafter, shadow → unchanged.

8. Known limits

• Greedy drafts only (live: local argmax). Probabilistic drafting, dspark_draft_topk, watermarking, adaptive verification (enable_adaptive_verification, off live), PP, DP and context parallelism are refused at startup with a message.
• Sync rounds are a host round trip on rank 0. The rank-0 CPU waits for the target step, then for the network and the worker, so the next step's CPU preparation is no longer overlapped with GPU work, as it is with the local drafter. That is a fixed per-step cost of ≈ 0.5–2 ms on top of the drafter time.
  • In shadow it is visible as rank0_round_us minus rtt_us.
  • Async rounds (VLLM_DSPARK_REMOTE_ASYNC=1, capacity only) remove it. Lane A2 design, v2 after review S9 (remote_dspark/streamsync.py, hostflag.c): one C-owned page of registered, device-mapped host memory holds three 32-bit words (gathered, released, claimed) and the reply. The engine stream writes gathered = t after the gather's copies (cuStreamWriteValue32, fenced), waits for released >= t (cuStreamWaitValue32), then copies the reply from the page (cuMemcpyHtoDAsync) and broadcasts it. The proxy thread acquire-spins on gathered, runs the round on the CPU and the network, builds the reply in a private buffer and publishes it in one GIL-free C call: claim t, copy the reply into the page, release t (monotonic). It makes no CUDA or torch call: the first async design (side-stream H2D + device flag written by the proxy thread) deadlocked on GB10 against a Triton module load on the engine thread while the stream was parked (offload/results/C/lazy-repro/RESULT.md); a host word is released by a CPU store, which needs no CUDA lock. The engine's Engram spin reader uses the same handshake on GB10.
  • A native watchdog thread (C: no Python, no CUDA) fails and releases a round that was gathered but not released within the deadline; the claim word makes it and the proxy's publish mutually exclusive per round. Releases never move backwards (cyclic max), so a late proxy cannot undo a later release (S9 F3). If the proxy thread ends for any reason other than close(), every pending and later round is failed and released at once (far-ahead claim) and later rounds skip the hand-off (zero drafts) (S9 F2). close() fails everything still pending the same way and logs an error if the thread did not stop within 5 s.
  • Resync rows in async mode come from ring windows the engine stream copies to pinned host memory on request: the first round that needs a resync asks for them and fails (zero drafts), the next one is exact. fallback_resync_pending counts those rounds (one per failure/reconnect event; two if the engine had already submitted the next round).
  • The C helper libdsv41_hostflag.so (v2: page allocation + mlock, acquire wait with GIL released, monotonic release, claim, publish, fail-release, watchdog). Production staging (spark/stage_overlay.sh, S9 F8) copies a pinned aarch64 build next to libofl.so and refuses when async is configured and the build is missing or not the pinned one (the pins: source HOSTFLAG_C_SHA, build HOSTFLAG_SO_SHA, set after building on the head node: cc -O2 -Wall -shared -fPIC -o ~/dsv41-offload-run/ofl/libdsv41_hostflag.so offload/dspark/spark/remote_dspark/hostflag.c -lpthread). Experiments may leave it out: rank 0 then compiles the overlay's hostflag.c with cc into a private temp directory at startup (async_handoff.lib says which). No C compiler and no library: async is refused with a warning (synchronous rounds). VLLM_DSPARK_REMOTE_HOSTFLAG_LIB names an explicit path. A v1 library is refused (version check).
  • The proxy thread spins on one rank-0 core while a round is pending (as cudaEventSynchronize did); VLLM_DSPARK_REMOTE_ASYNC_NAP_US > 0 sleeps between checks after VLLM_DSPARK_REMOTE_ASYNC_SPIN_US.
  • Startup self-test (HostFlagSync.self_test): the CPU sees the copied data with gathered; the stream does not pass the wait before the publish, and its reply copy after the wait reads the bytes the publish wrote (the production path). Its rescue is a CPU store; on failure the mode turns off with the reason. If the stream still does not drain after the rescue (a device that never observes host stores) it raises StreamStuckError, which stops startup instead of falling back to a stream that is parked for good (S9 F4).
  • Tested (lane A2, 2026-10-23): CPU suites (a profiler test fails on any torch call from the proxy thread), and on an RTX 3090 the GPU harness tests_gpu/gpu_async_harness.py with the reference worker on localhost: the first design deadlocked on a fresh cuModuleLoadData during a parked round (killed at 90 s); this one completed 30/30 such rounds (each load waited ~41 ms = the injected worker delay, then returned) and a 10,000-round soak pipelined 3 deep with 270 module loads during parked rounds and 49 injected faults: 19,082 drafts bit-exact, 9,319 failed rows zero, 73 resyncs from staged windows (124 resync_pending rounds), proxy round p50 664 µs vs 861 µs for the first design without module loads. Evidence: $CAMPAIGN_LOGS_ROOT/DSV41-OFFLOAD/A2-evidence/. GB10 is first run by the coordinator.
  • CUDA_MODULE_LOADING=EAGER stays in the async configs (S9 F1). The proxy needs the GIL, and pybind11 extension launches (every vllm_exl3_moe extension: plain m.def, no gil_scoped_release) hold it; under lazy loading such a launch waits for the parked stream while holding the GIL (S9 P1). EAGER moves those loads to startup.
  • The native watchdog (hostflag v2, S9 F2/F5) is the backstop for what EAGER cannot cover (an extension first imported while serving, a GC finalizer on the proxy thread making a CUDA call, proxy death, interpreter exit): a round the stream gathered but nobody released within VLLM_DSPARK_REMOTE_ASYNC_WATCHDOG_MS (auto max(4 x TIMEOUT_MS, 200)) is failed (zero drafts) and released from C, with no Python and no CUDA. Counters: async_watchdog.fires, watchdog_preempted (the proxy found its round already failed), fallback_proxy_dead / async_dead (the proxy thread ended: every later round is target-only, no hand-off).
  • The proxy thread shares the GIL with the engine thread, so async mode lowers the interpreter switch interval to 500 µs (VLLM_DSPARK_REMOTE_SWITCH_INTERVAL_US) to bound the waits.
• No local timing claims. The overlay ran only offline (CPU, Controller). Every number on GB10 comes from the coordinator's runs.