# Whole-worker qualification runtime

PP4 rebuilt the public EXL3 dependencies for the required x86host interpreter:
`/home/operator/miniconda3/bin/python3`, CPython 3.13.12, Torch 2.12.1+cu130
and Triton 3.7.1. Stock EXL3, FP32-accumulation EXL3 and the worker's P2P
extension all import together successfully. This is an ABI qualification;
the P2P binary has not executed on either production-pair GPU.

[runtime-build.json](runtime-build.json) records the module, compiler wheel,
source-tree and object-reuse certificate hashes. Every binary remains in PP4
scratch and must be deleted when the lane closes. The coordinator must rebuild
the prerequisites before a future window. No installed package, production
file, service or power setting was changed.

## Reproduce the prerequisite build

[build_runtime.sh](build_runtime.sh) is the CPU-only build entry point. Run it
from an owned named tmux session on x86host, with `TMUX_TMPDIR` set to its private
PP4 directory **before starting the tmux server**. It applies the reserved
CPU affinity 12–15,28–31, nice 19, eight-job/thread limits and a 48 GiB virtual
limit for the CPU build. No GPU is visible. GPU execution uses measured RSS
and the 48 GiB resident-memory guard instead; a virtual limit would reject
CUDA's large address mappings even when resident use is small.

The two public source trees are copied from
`~/dsv41-offload/o2/src/{exllamav3,exllamav3-facc}`. Both are upstream commit
`151539c77abc7ab7425d30da7a4e8e3c5c154e7b`; the facc tree retains O2's existing
FP32-accumulation change and distinct pybind internals. PP4 changes no numerical
C++/CUDA source. Its scratch setup.py loads only the build reuse hook.

The installed CUDA 12.6 compiler does not match Torch's CUDA 13.0 build. Another
existing environment's CUDA 13.3 compiler also fails the compiler/header
compatibility check with its mixed headers. The build instead uses the pinned
public compiler packages in [requirements-runtime.lock](requirements-runtime.lock):

| Package | Version |
|---|---|
| nvidia-cuda-nvcc | 13.0.88 |
| nvidia-nvvm | 13.0.88 |
| nvidia-cuda-crt | 13.0.88 |
| nvidia-cuda-cccl | 13.0.85 |

The existing interpreter's nvidia-cuda-runtime 13.0.96 headers and libraries
remain read-only dependencies. A scratch `libcudart.so` symlink supplies the
linker name for the installed `libcudart.so.13`. All downloads, toolchain files,
build outputs and caches stay under `/opt/dsv41/x86host-data/pp4/runtime`.
The build targets SM86 only. [build_p2p.py](build_p2p.py) builds/imports P2P
without creating a CUDA context and writes its normal source-key/hash pin.

[runtime_reuse.py](runtime_reuse.py) certified 82 facc objects for reuse;
108 were rebuilt. It compares the actual Ninja compile commands and all
transitive source/header bytes. The module and pybind-internals definitions
may differ only when neither identifier appears in any input. A changed
numerical header, a pybind dependency, a missing input or any other flag change
requires compilation. At use it rechecks every input and object hash, writes
the complete dependency file and lets Ninja record the real wrapper command.
Reused objects retain their original stock paths in diagnostic metadata.
The complete certificate is preserved outside scratch as campaign artifact
`PP4.runtime-reuse-manifest.json.gz`; its uncompressed hash is in the receipt.

A reproduction attempt exposed an idempotence defect: extracting unchanged
wheel files updated header timestamps and triggered an unnecessary rebuild.
That owned job was stopped before linking; both qualified module hashes stayed
unchanged. The script now preserves unchanged file contents and timestamps.
The final CPU P2P helper executed successfully; the revised full build entry
point has shell validation but has not been rerun from an empty directory.

## Whole-worker measurement

[full_worker_run_locked.sh](full_worker_run_locked.sh) takes only the assigned
solo UUID's lock with `flock -w 3600`, checks its existing 225 W cap and
occupancy, and applies the CPU limits. Set `PP4_SOURCE_ROOT` to the staged
PP4 source and `PP4_EXL3_PATH` to the stock/facc module directories, separated
by a colon. Private CUDA, Triton and Torch caches are mandatory.

[full_worker.py](full_worker.py) executes both complete logical K2 TP2 ranks
on the solo card with production weights, query block 5 and output depths
4/5. The matched timing fixture contains four real N4 request states. It
interleaves greedy and sampled graphs over five repetitions, reversing cell
order each repetition. CUDA events bracket each complete worker call and
include host gaps inside it. Physical-pair collectives, transport and target
verification are excluded.

On all 24 NB1/N4 states, both output depths and every batch size 1–4, the
switch-enabled all-T0 request must match the original graph's proposals and
confidence exactly. Batches 2–4 also use both alternating T0/T1 masks, checking
every T0 row's proposals/confidence and zero meaningful q payload. Unused q
padding beyond the output depth is outside that check.

The 24 states comprise twelve frozen synthetic states and twelve real N4
captures; captures stay on x86host. A separate B1 comparison checks both depths
against the archived NB1 K2 digests in
[n4-greedy-reference.json](n4-greedy-reference.json). That file contains only
hashes, with the archived source receipt pinned. The comparison detects runtime
arithmetic drift; matching ON/OFF outputs in the rebuilt environment alone
cannot establish identity with production's older binary/runtime.

## Incidental writes and cleanup

The first failed build briefly created its own tmux socket at
`/tmp/tmux-1000/pp4-runtime`. The exact owned process group was stopped and its
socket removed; no bulk data was written there. Replacement servers use the
private socket directory from startup.

Two initial pip index queries created four HTTP metadata cache files outside
scratch, totaling 10,427 bytes. Their exact package names were verified and
only those four owned files were moved into scratch for cleanup. Subsequent
pip commands use the private cache. Early local mypy/ruff commands also used
the worktree caches; later checks use the prescribed private cache/no-cache
options. The root lane owns final worktree-cache and scratch removal.