code/offload/dspark/probabilistic/pp4/SERVER-PROBE.md

Round 11 X86host-only server diagnostic — written, not run

Ready for independent source/CPU review. The lane has made no host contact, GPU call, production mutation, staging operation or serving request this round. This diagnostic cannot qualify Spark admission, performance or promotion. The existing S78/S76 greedy-generation salt blocker is still open.

server_probe runs ON X86host, from a clean independently reviewed candidate checkout. It contains no SSH, Spark API, HOLD or engine-launch operation. Production's target keeps serving while prod_window.sh pair COORD temporarily releases its drafter pair; the diagnostic listener uses an allocated port 18640–18699 on X86host's own 10.202.28.1. It never uses production's 18620. The local client uses real OFL TCP callbacks; RDMA is outside this diagnostic.

Inputs and identity

Stage two separate clean local checkouts under /opt/dsv41/x86host-data/pp4:

• Baseline: full 1b8122992fb440edc668ee9e628ee9624ae2faf2.
• Candidate: full final independently reviewed Round 11 commit, containing the driver. It must contain exactly the graph override deletion and graph-mode readiness log in server.py; every other non-test worker Python/CUDA/C++ header and implementation byte must equal the baseline. The driver enforces this.

No staging is performed by the driver. Transfer a reviewed bundle to X86host only and make these checkouts there; do not use pp4_admission_stage.sh, which also stages a Spark. No source tree or shared runtime is changed during a run. Delete staging clones after the coordinator has retained the reviewed evidence.

Supply the preserved raw physical-pair driver receipt.json from the baseline 1b81229 qualification, not an admission summary or solo receipt. Its exact matrix, physical UUIDs, both-rank q/ID replay, isolated timeout/sticky teardown, whole-K2/N4 checks, commit and original source hashes are validated. These unchanged numerical sources remain the reference qualification; that receipt DID NOT qualify the service builder. It is not relabelled or rehashed. This avoids requiring another Spark HOLD merely to diagnose the builder mismatch.

The actual new graph reference is produced and preserved by a separate owned candidate process before baseline and candidate run. All three processes use identical reviewed runtime/checkpoint bytes. Candidate output signatures must match the separately preserved reference exactly. Both variants' server.py hashes, all worker Python source hashes, driver/bootstrap/frame/trace hashes, runtime receipt SHA256 and physical-receipt SHA256 are recorded in identity.json. Before imports the child rechecks its own run, staged directory, worker bytes, qualified runtime manifest and inherited GPU lock inodes. Loaded extension and OFL module paths/hashes must match the reviewed inputs.

The required miniconda interpreter is /home/operator/miniconda3/bin/python3; Torch must be 2.12.1+cu130. The existing checkpoint is read and checked by the real production builder at /opt/dsv41/x86host-data/dspark. No model download, conversion or private capture is needed.

Prepare this local runtime manifest in PP4 scratch, with actual absolute paths and SHA256 values, after independently reviewing the files. files must have EXACTLY these six labels:

{
  "schema": "pp4-server-probe-runtime-v1",
  "reviewed": true,
  "python": "/home/operator/miniconda3/bin/python3",
  "torch": "2.12.1+cu130",
  "staging_receipt": "ABSOLUTE_REVIEWED_STAGING_RECEIPT",
  "staging_sha256": "ACTUAL_RECEIPT_SHA256",
  "files": {
    "stock": {"path": "ABSOLUTE_STOCK_CPYTHON313_SO", "sha256": "ACTUAL_SHA256"},
    "facc": {"path": "ABSOLUTE_FACC_CPYTHON313_SO", "sha256": "ACTUAL_SHA256"},
    "p2p": {"path": "ABSOLUTE_P2P_ROOT/SOURCE_KEY/dspark_p2p.so", "sha256": "ACTUAL_SHA256"},
    "ofl": {"path": "ABSOLUTE_LIBOFL_SO", "sha256": "ACTUAL_SHA256"},
    "ofl_python": {"path": "ABSOLUTE_OFL_PY", "sha256": "ACTUAL_SHA256"},
    "tune": {"path": "ABSOLUTE_REVIEWED_TUNE_BIN", "sha256": "ACTUAL_SHA256"}
  }
}

The staging receipt must be REVIEWED for the candidate commit, with the existing three-entry CPython3.13 runtime_artifact_sha256, runtime_python and runtime_torch fields. Stock/facc/P2P hashes must equal it. The P2P source-key folder and sibling dspark_p2p.so.sha256 must already exist; the actual loader refuses missing or wrong bytes rather than compiling P2P. The OFL Python module and shared library are separately pinned here. These are local consistency and review gates; the driver does not authenticate invented review receipts.

Each process gets fresh private empty Triton, CUDA, Torch-extension, TorchInductor, XDG, HF and temporary cache directories. The tune file starts from a private byte-identical COPY; shared tune bytes are never written. Relevant resolved CUDA/Triton/allocator/runtime settings and library versions are retained. Inherited DSPARK/TRITON/CUDA/EXLLAMAV3 feature settings are removed before the fixed experiment settings are installed; CUDA module loading uses the runtime default, recorded as unset if not explicitly supplied. Cold here means empty process-private compiler/driver caches, not evicted OS page cache. Warm means repetition within that same process. No page-cache drop is attempted.

Actual execution and evidence

The bootstrap arms a repeating ten-second native Python traceback timer and SIGUSR1 stack dump before model imports. It executes the ACTUAL staged worker.server as __main__, with --tp 2, sampled switch ON and the caller's default graphs. A one-shot Python profile hook installs diagnostic host wrappers at that module's real main entry, before its actual build_core_tp2 call. No model or kernel implementation is copied into the probe.

The order inside each owned process is:

1. Real builder/loading/default capture. Resolved backend.graphs and the real readiness log are preserved. Baseline should show False; candidate/reference must show True. Startup is bounded at 600 seconds, the existing whole-worker qualification bound; this is not a measured new graph-startup duration.
2. From one ordinary Python thread, real encoded HELLO/STEP/RESET/PING frames enter actual DraftCore.handle. After HELLO, a labelled window-only setup STEP inserts 128 rows for each of four generation1 slots, with no draft flag and zero masks. It has the same 30-second frame bound. The FIRST positive draft is B1/two rows, IMMEDIATELY followed by B4/28 rows (seven per request), in the same session with no RESET, resync, other width or draft between them. Slot0 advances128->130->137; slots1..3 advance128->135. All windows remain full and generations unchanged. Only AFTER that pair, the existing reset/repetition matrix runs: B1/B4/B2/B3, ten changed-anchor/ round/key iterations per width and mask, all positive/all T0/mixed. These warm resyncs still send full128-row windows; subsequent increments are two/seven/five rows per request at B1/B4/B2–3. A new session repeats setup, cold transition and matrix. Service depth is HELLO-negotiated fixed5. Synthetic finite BF16 taps are deterministic by slot/position and match regardless of insertion chunks. These are observed public shapes with synthetic values, not recorded private tensors and not a claim to reproduce an unknown full public request.
3. Actual model APIs first run consecutive positive B1/B4 calls at output depth4, then consecutive positive B1/B4 at depth5, without an intervening model draft or reset within either pair. The API consumes preloaded KV; row-delta shapes are exercised by the protocol steps above. Afterwards the existing B1/B4/B2/B3 matrix runs at output depths4/5, all three masks and ten changed keys/anchors using the deterministic synthetic N4 fixtures. T0 IDs and confidence must exactly equal the original greedy model path.
4. The real server main starts real OflServer. A separate X86host-local CPU client sends the same frames through its native connection-thread callbacks, closes and reconnects a new connection/session. Direct/OFL ID/confidence/q signatures must match exactly. OFL runs after direct/model setup, so this is explicitly warm model traffic with fresh native connections, not a cold OFL-start claim.

Each STEP checks frame identity/status, complete windows, fixed depth, CRC/support/mask, exact dyadic24 mass and sampled-token support using the production codec. Every frame must finish inside the existing 30-second status freshness window; there is no cold-request exemption or timeout extension. Server and client have independent flushed per-phase/request records and Python/native thread IDs. Builder/capture, reset/stage/flush, whole forward, selector warmup/load, each rank submission, result checks and Python/native reply serialization are timed with perf_counter_ns. Logs contain hashes, counts and timing only, without text or tensor payloads.

Rank submissions buffer their records in memory until BOTH return. There is no log-file operation, print, lock, event or synchronization inserted between paired submissions. Native traceback timers still name a blocked call; buffered rank records from a call that never returns may not reach disk. Stream/device handles and host submission durations are recorded on successful submissions. After result drain ONLY, inspect both ranks' sticky/error/done/epochs, exact support/q AND proposed IDs; require healthy state and advancing matching epochs. No CUDA scheduling/overlap claim is made without a CUDA activity trace. Exact healthy peer exchange is evidence of completed handshake, not a timeline.

Candidate/reference must use graph dispatch, have graph submission and healthy peer evidence, make no positive eager-selector call or Triton warmup/handle-load in real direct/OFL requests, complete all frames and match exact signatures. Setup in the separately labelled model-level phase is not silently treated as service request work. Every step duration is retained; a two-sample percentile cannot conceal a slow wider batch.

Ownership, deadlines and close

Only the coordinator may execute. Local preflight refuses off X86host, a dirty or wrong tree, wrong source delta, missing/changed qualification/runtime, another listener, insufficient48GiB RAM or any used swap. It opens no service/window. The real run registers a X86host-local shared scoped_remote phase BEFORE pair-open, writes intent before submission, and opens prod_window.sh pair COORD INSIDE that owned scope. No lane-private registry is introduced. It takes BOTH existing GPU locks nonblocking, verifies physical UUID inventory, and passes the inherited locks to each new-session worker process. UUID order and actual two-device properties are rechecked; emulation refuses.

Only this owned scope receives MemoryMax48G/MemorySwapMax0. Workers run taskset -c 12-15,28-31 nice -n 19, eight compute threads/one interop thread. No power/clock setting, installed unit, current link, shared tune or image is changed. Production unit stop/start is exclusively the coordinator's existing pair-window script, as explicitly requested for this diagnostic.

MEASURED driver/startup duration: UNAVAILABLE. ESTIMATED attendance allowance: up to72minutes from immutable60-minute probe scope plus12-minute restoration reserve. Three startups each have a600s bound; real graph startup is unknown. The client also has a600s overall OFL-phase bound; every request independently has30s. These bounds do not promise the matrix fits: the shared absolute probe lease wins and truncates an overrun. Do not extend it without reviewed cold-build measurements and a separately labelled setup design.

On success, error, lost pair-open reply, request/startup bound or ordinary signal, stop/wait ONLY the processes this driver created, prove their groups and allocated listener absent, then fence/reap/prove the probe scope absent. Only then submit close in a SEPARATE shared owned scope. Close is bounded by the ORIGINAL probe+12-minute ceiling, with90s reserved for final close-scope absence proof. Second signals are masked during cleanup. Exact PRODUCTION HOLD OK from prod_window.sh close is required. Failure of absence, close or its final proof yields STOP/RESTORE FAILED and coordinator inspection; it cannot claim a restored pair. The unsafe alternative of closing while an old pair-open can still run is refused. No other process is killed by name.

Shared ownership fence/control receipts use the existing module's persistent control path; all probe data, caches and stage trees stay in authorized PP4 scratch. Persistent ABORT records must not be deleted. This is an attended coordinator diagnostic, not a replacement for W8, six-index greedy restoration, full admission or the post-close watch of a later Spark HOLD.

Coordinator command after independent review — not executed

On X86host, with both source trees staged, the reviewed runtime manifest and raw baseline physical receipt, and a FRESH output directory:

source ~/.config/ndm/paths.sh
: "${PP4_PROBE_CANDIDATE:?absolute candidate checkout under X86host PP4 scratch}"
: "${PP4_PROBE_BASELINE:?absolute separate baseline checkout under X86host PP4 scratch}"
: "${PP4_REVIEWED_COMMIT:?full independently reviewed final Round10 commit}"
: "${PP4_PROBE_RUNTIME:?absolute runtime manifest}"
: "${PP4_BASELINE_PAIR_RECEIPT:?absolute original raw 1b81229 pair receipt}"
: "${PP4_PROBE_OUT:?fresh output beneath /opt/dsv41/x86host-data/pp4}"
: "${PP4_PROBE_RUN:?fresh PP4-server- tag, <=51 characters}"
cd "$PP4_PROBE_CANDIDATE"
export PYTHONDONTWRITEBYTECODE=1 PYTHONPATH="$PWD:$PWD/offload/dspark"
WINDOW_COORDINATOR=Deepseek/coordinator \
 /home/operator/miniconda3/bin/python3 -m offload.dspark.probabilistic.pp4.server_probe \
 --execute --baseline "$PP4_PROBE_BASELINE" --candidate "$PP4_PROBE_CANDIDATE" \
 --commit "$PP4_REVIEWED_COMMIT" --runtime "$PP4_PROBE_RUNTIME" \
 --pair-receipt "$PP4_BASELINE_PAIR_RECEIPT" --out "$PP4_PROBE_OUT" \
 --run "$PP4_PROBE_RUN" --port 18644 --weights /opt/dsv41/x86host-data/dspark

Default preview requires no arguments and makes no host or model call: python -m offload.dspark.probabilistic.pp4.server_probe with the required interpreter and worktree PYTHONPATH.

Interpretation is conditional. Baseline reproduction at a named phase plus candidate clearance under the same cold/warm controls supports the graph correction as the remedy. A cold setup that completes then repeats quickly supports a readiness/setup diagnosis. Failure only in OFL prioritizes callback or transport handling; failure in a common GPU phase means the one-line change is insufficient. A candidate pass without baseline reproduction closes the dispatch mismatch but leaves the original cause unresolved. Any probe failure must stop before another Spark admission. No automatic promotion occurs.