offload/operations/README.md

Production profiles and operating ledger (OPS1, 2026-10-23)

The recorded production profile is K1-327680 with TP2 + async. This updates the older roadmap/comparison snapshot that still describes one-card async. It does not supersede the coordinator's campaign register or authorize a change.

profiles.v1.json is the versioned identity/configuration registry. It names A, D, async, TP2+async, K1-327680, K1-524288 HOLD, native-source-faithful and adapted separately, including config hashes, source refs/trees, drafter/Markov identities, overlay manifests, evidence and rollback sections below. An unbuilt adapter identity is null, never borrowed from the unadapted model. Historical config files are references, not scripts to source over current production. Native means the unadapted official-source backbone with the declared shared EXL3 K6 head; it does not claim every tensor is official BF16.

Read-only observation

From Controller (no GPU/import of torch, no Spark HTTP, no remote writes):

source ~/.config/ndm/paths.sh
CUDA_VISIBLE_DEVICES='' python3 offload/operations/live_profile.py --live
# Save aggregate output only under the campaign directory if a receipt is needed.
python3 offload/operations/live_profile.py --snapshot /path/to/saved-output.json
CUDA_VISIBLE_DEVICES='' python3 -m unittest discover -s offload/operations -p test_live_profile.py -v

Only fixed git log/status/diff, docker inspect/logs on sparkhead and fixed systemctl --user status, readlink, status-file cat on x86host are issued. An inactive systemd unit's status code 3 is handled explicitly; other failed reads fail. No credentials, arbitrary Docker environment entries, prompt/text or tensor data are printed. Snapshot mode makes no remote call; its result is as-of the saved observation, not a fresh health check. Exit 0 means one exact CONFIGURATION_MATCH; exit 2 means DRIFT_OR_UNKNOWN (including live HOLD), exit 1 means an observation/schema/read error.

Match includes image ID, startup cap/KV/sequence/chunk settings, expected env and drafter/Markov identities, registered HEAD/dirty state/diff digest, worker release, unit enablement/PID and fresh worker status, the complete K1 profile mount set and K1 mount coverage/read-only/version. The checker does not prove mounted file bytes, target manifest bytes, current clock settings, GPU lock holders, adapter loading or the second Spark's state. It never equates a healthy unit, a symbolic link or a container image label with those checks. Observe both ranks with the existing coordinator runtime verifier when a production qualification is needed. Separate SSH reads are not atomic; a concurrent switch can yield a deliberate refusal. Rerun after the coordinator's window closes. Failed/stale reads are never replaced by the previous saved result.

Current registration intentionally expects the still-uncommitted K1 diff. After the prepared commit/tag below, the checker must report drift until the coordinator updates that profile's allowed HEAD, dirty state and diff digest, reviewing the new observation. Future arbitrary commits do not inherit trust. Native/adapted candidates have no automatic match rule until qualification.

Closed operating ledger / evidence boundary

All numerical observations in this table are MEASURED in the cited records; no new model requests or performance benchmarks were run by OPS1.

Item	Disposition	Evidence / remaining gate
Async process isolation / recovery	Promoted; preserve	W5 in COMPARISON; R3/S12 reports in campaign directory. No replay of solved fixes
TP2 production worker	Promoted 20:21Z; separately attributable to K1	Campaign LANES.md 20:21 entry; W6-prod records; live unit active/current fb68c52
K1 at 327680	Promoted 21:19Z; source record not yet committed on sparkhead	Campaign K1-production start/post logs and S18/S22; live cap 327680, pool 2502600, bytes/rank 6442450944, seqs 4, batch 4096
K1 at 524288	HOLD	S18 final: cold prefill +35%; valid correctness does not waive speed gate. PF1-R2/coordinator own explanation and new qualification
Sparkhead source record/tag	Prepared, not run	HEAD ad2a4a0; only dsv41.env/start.sh dirty. Procedure below creates prod-k1-327-20261023 without restart
Native source-faithful arm	Not production qualified by N5; outcome qualification separately authorized	N4/N5; expert continuation rows 3/7. kx2 and packed identities recorded separately; stale DRAFT_MS must be calibrated
Adapted drafter	Not built/qualified	D2 pilot and serving receipts required; null identity is intentional
Old startup warm-up	Confirmed deficient; patch CPU-qualified, not applied	Warm-up report, branch offload/ops1 item f4bc392; actual old EOS/reasoning/worker rounds unavailable
Concurrency 6/8 / clocks	Plans only	Controlled plans; no setting changed
D2 fairness	Separate owner D2-FIX1	Expert continuation row 5; legal priority unchanged. No scheduling change in OPS1

Live observation from OPS1 at 22:39Z: sparkhead container 1ff67bf41da156b143abcbbc8071389d7d92652c5d593b74b792a72e143fbbdb, started 20:46:08Z, running, image ID recorded in the registry. X86host TP2 worker pid 1339801, current -> fb68c52, one-card unit inactive, status time 22:39:08Z, 42,524 worker steps / 126,987 drafted requests / zero TP communication errors. These are cumulative counters, not warm-up-only results or new performance measurements. No lock inspection was authorized/performed by OPS1.

The container logs' long target-only interval was the known native pair window, not a newly diagnosed failure: campaign LANES.md records opening at 21:19:25Z, closing with PRODUCTION HOLD OK at 21:43Z. At 21:42:55Z the proxy reconnected (epoch 2). Thus 29,624 cumulative failed rounds in the proxy logs must not be reported as a new steady-state failure rate. Attributable worker rounds for the old one-token warm-up still cannot be reconstructed.

Prepared K1 source record (do not run from this lane)

Coordinator only, on sparkhead in ~/dsv41-deploy. This records the already applied files; no start/stop or benchmark is needed. Do not mix the warm-up patch or any later changes into this commit. Preserve logs/k1-promotion/: its saved original files, receipt/report and promoted hashes remain the guard and rollback authority even after committing. Git commit/tag do not change those file bytes. If any precheck differs, inspect; do not overwrite/restage.

source ~/.config/ndm/paths.sh
cd ~/dsv41-deploy
set -euo pipefail
test "$(git branch --show-current)" = main
test "$(git rev-parse HEAD)" = ad2a4a07f5da5d81d46a838882f1c02bc932e14b
git diff --cached --exit-code
python3 offload/promote/k1/manage.py verify --profile 327680
python3 - <<'PY'
import subprocess
expected = [' M dsv41.env', ' M start.sh']
actual = subprocess.check_output(['git','status','--porcelain=v1','--untracked-files=normal'], text=True).splitlines()
if actual != expected:
    raise SystemExit('REFUSED: dirty paths differ: ' + repr(actual))
PY
git apply --reverse --check offload/promote/k1/dsv41.env.327680.patch offload/promote/k1/start.sh.patch
if git show-ref --verify --quiet refs/tags/prod-k1-327-20261023; then
  echo 'REFUSED: tag already exists' >&2
  exit 1
fi
git diff --check
git diff -- dsv41.env start.sh
# Coordinator reviews that exact diff against the saved promotion receipt.
git add -- dsv41.env start.sh
git commit -m 'production: record reviewed K1 packing at 327680 with TP2 async'
python3 offload/promote/k1/manage.py verify --profile 327680
git tag -a prod-k1-327-20261023 -m 'K1 327680; KV 2502600; async process proxy; x86host TP2 current fb68c52; S18/S22 qualified; 524288 HOLD'
git log -1 --format=fuller
git status --short

Record the resulting commit/tag object SHA in this registry and the campaign ledger; refresh live observation and require only the expected clean-source change. The tag pins Spark deployment sources, not x86host's service/link; those identities must remain in the same receipt. Never push from this lane.

Rollback commands (coordinator window only)

These are prepared, not executed or newly rehearsed by OPS1. Before any rollback, pause producers/D2, drain both running/waiting gauges, use offload/launch/window.sh open 'reviewed profile rollback' and preserve all current diffs. Do not use a tag checkout to discard later edits. Check target path/alias remain legal-v2a and retain exact local Engram fallback. Both nodes must pass fresh runtime, request, semantics and hold checks before offload/launch/window.sh close. An old tag is not proof of current health.

K1

On sparkhead, after drain, rollback K1 to the actual saved TP2+async state:

timeout --kill-after=15s 45s bash ./stop.sh </dev/null
python3 offload/promote/k1/manage.py restore
timeout --kill-after=15s 600s bash ./start.sh </dev/null

The restore refuses later edits and verifies byte identity; keep HOLD on error. Check old cap327680/pool1604926, process proxy, both-rank hashes, TP2 worker, accepted drafts and greedy regression. Do not replace manage.py restore with PROD_K1=0 or a tag checkout. See K1 package.

TP2+async

K1 rollback above selects this profile, retaining x86host's TP2 unit/current fb68c52. To roll the Spark configuration back from an unpromoted candidate, use that candidate's saved environment/overlay transaction; no blanket checkout of the experimental branch. Async/TP2 package owns the separately reviewed worker switch and failure drills.

async

After K1 restore, a separately reviewed worker rollback selects the one-card worker. In a coordinator pair window on x86host (never OPS1):

systemctl --user disable --now dsv41-dspark-worker-tp2-prod.service
test -d "$HOME/dsv41-offload/prod/dspark-worker/d275f6b/offload/dspark/worker"
ln -sfn d275f6b "$HOME/dsv41-offload/prod/dspark-worker/current"
systemctl --user enable --now dsv41-dspark-worker.service

Before those commands, verify the staged d275f6b source/identity and preserve the current link. worker/run/prod_tp2.sh documents this layout but deliberately does not flip the link itself. The historical one-card unit has no ExecStart GPU-flock wrapper: the coordinator must retain its one-card ownership/hold and prevent lane admission through the whole handoff; do not assume this unit provides the TP2 unit's two-lock guarantee. A later approved one-card runtime gets a new registry entry. Check no pair consumer or waiting lane can race the handoff. Existing production window protocol remains authoritative; do not disable the unit outside its owner window.

D

To remove async after restoring K1 and preserving the current diff, coordinator on sparkhead selects the recorded synchronous D configuration files:

git restore --source=prod-d-20261023 -- dsv41.env start.sh serve.sh
./stop.sh
./start.sh

Use the reviewed one-card worker rollback above to match historical D's worker identity. Current safety-hardened runtime code remains; this command restores configuration files only. Confirm sync mode, KV1604926/max327680 and local-law contract. Inspect the diff before restart; preserve any unrelated configuration change rather than overwriting it blindly.

A

After the same K1 restoration and diff preservation, the historical local checkpoint drafter/local Engram configuration is:

git restore --source=baseline-20261022 -- dsv41.env start.sh serve.sh
./stop.sh
./start.sh

Require MODEL_DIR still /srv/dsv41/legal-v2a, cap327680, KV989460, bytes/rank4026531840; verify target/source identities against the campaign baseline before this older configuration is used. The unused x86host worker can remain owned by its production unit; no lane releases its locks. Do not load stock target weights to claim rollback.

native/adapted

No native/adapted production rollback receipt exists yet. Each future candidate must snapshot the qualified K1-327680 target env/mounts, worker unit/current, arm, drafter/Markov/source/adapter identities and existing GPU lock ownership before switching. Its rollback uses the coordinator's prod_window.sh close protocol and saved arm transaction, then verifies the registered K2 identities and healthy process proxy. Until an actual candidate transaction exists, there is deliberately no speculative command that overwrites worker links.

The registered worker max-model-len1048576 is a historical protocol upper bound, not permission for >524288 target requests or new 1M experiments. The active target cap is 327680 and K1-524288 remains HOLD.