Blame for offload/launch/OPERATIONS.md

# Startup warm-up (OPS1, reviewed patch; not deployed)

The old helper requested 32 tokens but did not require them. MEASURED:
`/opt/dsv41/campaigns/DSV41-OFFLOAD/K1-production/start.log` records
5.629 s, 5,041 prompt tokens and **1 completion token**. Its raw completion
request has effective explicit max=32, default min=0, ignore_eos=false, no
explicit stop. The installed completion protocol merges generation-config
stop IDs. EOS or a default stop is a plausible explanation, **not a confirmed
cause**: the old helper discarded finish_reason, stop_reason, text and metrics.

Read-only `docker logs` on sparkhead (container started 20:46:08Z on October 23)
confirms deepseek_v41 tokenizer, reasoning and tool parsers, TP2 target,
max_model_len 327680, max_num_seqs 4, max_num_batched_tokens 4096, speculative
k=5 for B1/B2 and k=4 for B3/B4, process proxy and remote pair connection.
Raw `/v1/completions` does not apply chat templating or invoke chat reasoning/
tool output parsers. The old response's raw token IDs, EOS/stop reason,
reasoning count and attributable worker-round count are **unavailable**.
Completion usage is not visible-answer usage or a worker-round count. Do not
infer any of them from the nominal 32-token budget or unrelated request logs.

The local read-only engine snapshot supports min_tokens and ignore_eos in
`vllm/entrypoints/openai/completion/protocol.py:to_sampling_params`, and exposes
finish_reason/stop_reason from the actual RequestOutput in completion/serving.py.
Per-request speculative metrics are emitted only when enabled in the engine;
missing metrics and reasoning counts are reported as null, never zero.

## Patch and limits

`warmup_serve.sh` keeps the current start.sh interface; its new stdlib-only
`warmup.py` issues internal raw completions with **min=max=32**, ignore_eos=true,
no explicit stops, greedy sampling, priority 20. The current main start.sh
already invokes the wrapper after async startup. No start.sh/env edit or server
sampling-default change is needed. Its one-token response is now an error,
with sanitized counts and finish/stop reasons. Generated text/IDs are never
printed or saved. There is no separate hidden reasoning budget: at most 32
completion tokens per request, including any reasoning-like continuation.

Four bounded waves request B1/B2/B3/B4 (ten requests, 320 completion tokens
maximum). B1 has a unique synthetic prefix and roughly the previous 5K size
(ESTIMATED); returned prompt count must exceed4096 and reported cached count
must be zero. B2/B3/B4 use short synthetic prompts, so long serial prefills do
not consume the whole32-token decode interval before other streams can join.
Barrier-released clients target both live depth bands. Client concurrency is
**not proof of engine batch shape**: request completion skew, chunked prefill
and adaptive verification can produce smaller shapes or tail rows. Long-context
attention, every verification depth and future B6/B8 are not covered by this
startup sequence. Time bounds: each socket operation waits at most the time left
of a 90 s helper deadline (and never more than 60 s); a wave never starts after the
deadline and a run that finishes late fails; `warmup_serve.sh` ends the whole process
at 95 s (`timeout --kill-after=5s`, overridable by `WARMUP_HARD_S`) because a trickling
read or a waiting worker thread can outlive the helper's own deadline. Any of these is a
failure, which `start.sh` reports as its existing WARN; with the last K1 start at 432 s
the warm-up cannot reach K1's 600 s start guard.

Before every wave, both request gauges must exist and be zero, otherwise the
helper refuses. This is an additional guard, not a producer lock or a
preemption guarantee. Coordinator must keep D2/local-law and other producers
paused behind HOLD throughout startup and warm-up. An already submitted wave
has a bounded token budget; no later wave starts if other work remains.
The main caller preserves its existing WARN-on-failure behavior, so inspect
warm-up completion records before releasing HOLD; service readiness alone does
not establish warm-up success.

## Qualification / application (coordinator only)

1. Review/cherry-pick this item from offload/ops1 into the intended production
   branch. This lane did not deploy, request, restart or benchmark anything.
   Preserve K1's saved dsv41.env/start.sh hashes; this patch changes neither.
2. CPU: `CUDA_VISIBLE_DEVICES='' python3 -m unittest discover -s offload/launch/tests -p test_warmup.py -v`;
   `bash -n offload/launch/warmup_serve.sh`.
3. In the next exclusive startup window, save fresh x86host worker status and
   proxy counters **before** warm-up and immediately **after** it, with matching
   pid/session, timestamps and no foreign traffic. Worker `steps` delta measures
   actual worker RPC steps; `drafted_requests` delta counts per-request proposals.
   Engine per-request `num_spec_steps` is a verification boundary, not necessarily
   a worker RPC boundary. Do not mix them or rely on the proxy's 1024-round
   periodic stats to resolve a 32-token request. Missing/stale data leaves that
   check incomplete. Existing status is written on a timer; wait for a fresh
   timestamp before reading each snapshot.
4. Require all ten responses to return completion_tokens=32, finish_reason=length,
   stop_reason=null; record actual prompt counts and cached tokens separately.
   Correlate worker graph `(B,k,n_out)` and target verify-row dispatch/capture
   records for B1/B2 k5 and B3/B4 k4, including smaller acceptance tails. If shapes
   are not observed, add a bounded coordinator-only shape test; do not lengthen
   hidden reasoning or claim the wave size proves capture coverage.
5. Compare the first subsequent short/32K request and B3/B4 burst with later
   warm bursts (TTFT, visible-answer timing, worker rounds, failure/watchdog
   deltas). Existing semantics/greedy regressions still apply. Reject startup
   timeout regressions. Keep long-prefill first-use qualification separate.
6. Roll back only `offload/launch/warmup_serve.sh` and the added helper to the
   parent revision if needed; K1 manage.py restore and service changes are
   coordinator-only and not part of this patch.