# HyperFlow release and evidence runbook This runbook covers the authenticated v2 application, separate durable worker, explicitly labelled replay workspaces, and account-scoped operations. A release is not complete until an independent person has completed the reviewer checklist. A local build or simulation run does not establish live provider access. ## Local setup Use Python 3.11 for the pinned numerical dependencies. The locked Vite release requires Node `^20.19.0 || >=22.12.0`. Start from the repository root: ```bash python3.11 -m venv .venv source .venv/bin/activate python -m pip install -r requirements.txt npm --prefix frontend ci ``` For an explicit local SQLite demonstration: ```bash export HYPERFLOW_DEMO=1 export DATABASE_URL=sqlite:///$PWD/.local/hyperflow.db mkdir -p .local python -m backend.flagship.manage migrate python -m backend.flagship.manage provision admin --role admin python -m backend.flagship.manage provision operator --role operator python -m backend.flagship.manage provision courier --role courier ``` Provisioning prompts for a password; minimum length is 12 characters. Passwords are not placed in shell arguments. User roles are assigned at provisioning, never selected by a login request. Demo mode permits SQLite. Connecting real provider credentials requires an explicitly configured persistent `VAULT_SECRET_KEY`, including in demo mode. Simulation-only operation needs no provider key. Run these processes in separate terminals with the same database environment: ```bash uvicorn backend.api.main:app --host 127.0.0.1 --port 8000 python -m backend.flagship.worker npm --prefix frontend run dev -- --host 127.0.0.1 ``` Open the frontend URL printed by Vite. `/health` checks the API process. `/ready` checks database schema and a recent worker heartbeat; it returns 503 when the worker is missing or stale. `/docs` describes the API. Reviewer access does not require provider credentials and uses isolated replay state. For a static production frontend served by the API, build before starting it: ```bash npm --prefix frontend run build ``` The frontend and API must agree on their origin. Use `VITE_BACKEND_URL` during frontend build only when the API is on a different origin. Configure allowed origins with `HYPERFLOW_ORIGINS`; the default allows local Vite development and preview ports. Session credentials currently persist in browser storage and are sent in Authorization headers, including authenticated event streams. This makes XSS prevention a session-security boundary. Verify reload and offline behavior against the mounted workspace; do not claim offline sign-out without testing. ## PostgreSQL and Docker deployment Outside demonstration mode, configure a persistent PostgreSQL `DATABASE_URL`. In every mode, configure an encryption key before connecting any live provider account. Generate a key locally without committing it: ```bash python -c 'import base64,secrets; print(base64.b64encode(secrets.token_bytes(32)).decode())' ``` Store the result as `VAULT_SECRET_KEY` in the deployment's secret configuration. Keep that key available for restores; encrypted provider credentials depend on it. Redis notifications are optional acceleration. PostgreSQL rows remain the source of truth for jobs and event replay. Kafka is not required by the v2 deployment. The Compose deployment defines PostgreSQL, Redis, API, and worker services. Inspect its required environment values before starting. Avoid placing a real provider token in shared deployment configuration; users connect their own credentials through the authenticated application. ```bash docker compose config --quiet docker compose build docker compose up -d postgres redis docker compose run --rm --no-deps api python -m backend.flagship.manage migrate docker compose run --rm --no-deps api python -m backend.flagship.manage provision admin --role admin docker compose up -d api worker ``` A build is a packaging check, not a live integration test. Record Docker/runtime versions and the exact source revision with release results. Do not report a container test unless Docker actually ran successfully on the target machine. ## Backup, restore, and rollback Back up the database, model registry, uploaded dataset references, and vault secret as one release recovery set. Store backups outside the application volume. Do not copy a running SQLite file as a consistent backup; stop API and worker first or use SQLite's backup command. A PostgreSQL custom-format database backup can be produced without embedding its password in the shell command: ```bash docker compose exec -T postgres sh -c 'pg_dump -Fc -U "$POSTGRES_USER" "$POSTGRES_DB"' > hyperflow-backup.dump ``` Restore into a separate empty test database first. Verify account counts, schema version, action histories, artifact hashes, and readiness before changing traffic. Use `pg_restore --no-owner` against the explicitly selected restoration database. Never restore over the active database as a first validation step. Model rollback is an authenticated admin operation at `POST /api/v2/models/rollback`. It restores the prior verified artifact pointer. For application rollback, stop API and worker, restore the previously tested image, and verify schema compatibility before restarting. The current migration is additive and records legacy provenance as `legacy_unknown`; there is no automatic destructive downgrade command. ## Verification and failure rehearsal ```bash python -m pytest tests/flagship tests/flagship_ops tests/flagship_ml -q npm --prefix frontend run build python scripts/eval_dispatch.py --seed 42 --cases 20 --output /tmp/hyperflow-dispatch.json python scripts/study_censoring.py --output /tmp/hyperflow-censoring ``` Run the retail study separately. A local compatible input remains user-supplied until its provenance is established; `--download-uci` explicitly downloads the public UCI dataset and preserves its source identity. Do not silently convert this UK retail study into Indian quick-commerce evidence. ```bash python scripts/study_retail.py --input /path/to/retail.csv --output /tmp/hyperflow-retail python scripts/study_retail.py --download-uci --output /tmp/hyperflow-retail-uci ``` Rehearse `stockout`, `price_change`, `external_edit`, `lost_response`, and `uncertain` in replay mode. Check that renewed approval is required after a changed proposal, unknown outcomes remain visibly uncertain, and retries do not blindly create a second placement. Restart the worker after a queued job, then inspect its terminal state and attempt count. Disconnect Redis and verify database-backed event replay. Run live discovery only after explicit credential setup. Paid model requests and real purchases/bookings are outside unattended CI. Preserve redacted receipts and capability evidence separately from simulation outputs. ## Independent reviewer checklist A person unfamiliar with the code should record their environment and source revision, then complete these checks without verbal assistance: - Start from a fresh checkout and run migrations, API, frontend, and worker. - Open an isolated reviewer workspace and identify every replay/simulation label. - Complete a basket workflow and explain infeasibility or renewed approval. - Trigger a lost response and identify the saved action outcome without guessing. - Enroll a courier, assign a HyperFlow-owned delivery, and complete its lifecycle. - Confirm another account cannot inspect or mutate that job. - Reproduce the dispatch and ML studies, including their source limitations. - Identify the currently served model and its artifact manifest. - Use keyboard navigation and a mobile viewport; record inaccessible controls. - Confirm empty and unavailable states contain no invented operational metrics. Keep feedback and unresolved issues in the release evidence. Do not mark independent setup/usability validation complete based only on automated tests or an author-led demonstration. ## ₹0 campaign and provider connection Use existing hardware; do not create paid cloud resources or enable a billable model fallback. Keep `HYPERFLOW_ENABLE_LLM` unset for ordinary verification. The frozen free-quota budgets and loop limits are in [methodology](../benchmark_methodology.md). Purchases and bookings remain disabled for this campaign. Configure `VAULT_SECRET_KEY` through local secret configuration, then sign in and use the credential form. Never paste a token into documentation, a committed `.env`, shell arguments or evidence. Saving a token establishes storage only. Read-only provider verification must establish the current credential's status; unknown expiry is not an invented five-day window. A registered OAuth client and an exact configured redirect are required for the browser authorization flow. See [integration](../../INTEGRATION.md) for required behavior. ## Incident drill record For each drill save source/environment identities, incident start, observed symptom, request/action IDs, authoritative record/event snapshots, redacted receipt evidence, diagnosis, recovery command, recovered state, duration and remaining uncertainty. Use isolated data. Redact account details and credentials. Rehearse API loss, worker restart, Redis notification loss, provider timeout after simulated commit and an artifact-write failure. Confirm that UNKNOWN blocks another placement, retained events support reconnect, failed training preserves the active artifact and readiness reflects actual dependency failure. `doctor` and `trace` output support diagnosis; their mere presence is not proof of deployment or FDE delivery. Never use teamwork metadata as worker liveness. Backup/restore rehearsal must restore a real backup into a separate database and verify schema, account/action/event counts and artifact hashes. Application rollback must use an identified prior build with compatible schema. Record actual commands and exit statuses; a written runbook is not a completed drill. ## Volunteer pilot form ```text Date / participant identifier (with consent): Source revision and dirty-file hashes / environment: User problem, constraints and agreed success criteria: Fresh setup start/end / assistance count and details: Task 1 — create a basket: completion, time, errors, assistance: Task 2 — handle changed proposal: completion, time, understanding: Task 3 — inspect uncertain outcome: completion, time, recovery choice: Observed defect and reproduction: Repair / regression / retest result: What became easier; what remains confusing: Limitations: one tester; prior exposure; assistance and learning effects: ``` Do not claim the pilot until a person other than the author actually performs it. The three-task pilot does not replace the fuller release checklist above. ## Release evidence bundle Retain setup/build/test logs, current source hashes, benchmark manifests and raw results, recovery/restore drill records, pilot feedback, open findings, architecture diagram and the three-minute demonstration. Update [status](status.md) and [reviews](reviews.md) with exact results. A public URL, live transaction evidence and Docker execution remain separate gates; none is implied by a local build. For 21 October applications, use only completed evidence while later phases remain open. ## Current model and evaluation controls Model experiments additionally require `HYPERFLOW_LLM_FREE_QUOTA_CONFIRMED=1` after the operator verifies a free-quota-only provider configuration. This flag does not inspect billing. Keep both model flags disabled for ordinary verification. Interrupted turns require inspection through `/api/v2/agent/turns` before starting a new attempt. Unified offline evaluations: `python -m backend.flagship.manage evaluate --suite intents --output .local/evidence/intents` (also solver, commerce, parsing, censoring, retail and dispatch). Retail requires an existing local input workbook. See each command's `--help`; no dataset or model downloads are implied. Current completed and incomplete drills are recorded in [status](status.md). The initial interrupted socket campaign remains historical. The separate complete campaign satisfies the three-run local performance protocol; see the status register for measurements and exclusions.