File size: 12,299 Bytes
2b86c60 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 | # HyperFlow release and evidence runbook
This runbook covers the authenticated v2 application, separate durable worker,
explicitly labelled replay workspaces, and account-scoped operations. A release is
not complete until an independent person has completed the reviewer checklist.
A local build or simulation run does not establish live provider access.
## Local setup
Use Python 3.11 for the pinned numerical dependencies. The locked Vite release
requires Node `^20.19.0 || >=22.12.0`. Start from the repository root:
```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
npm --prefix frontend ci
```
For an explicit local SQLite demonstration:
```bash
export HYPERFLOW_DEMO=1
export DATABASE_URL=sqlite:///$PWD/.local/hyperflow.db
mkdir -p .local
python -m backend.flagship.manage migrate
python -m backend.flagship.manage provision admin --role admin
python -m backend.flagship.manage provision operator --role operator
python -m backend.flagship.manage provision courier --role courier
```
Provisioning prompts for a password; minimum length is 12 characters. Passwords
are not placed in shell arguments. User roles are assigned at provisioning, never
selected by a login request. Demo mode permits SQLite. Connecting real provider credentials requires an explicitly configured persistent `VAULT_SECRET_KEY`, including in demo mode. Simulation-only operation needs no provider key.
Run these processes in separate terminals with the same database environment:
```bash
uvicorn backend.api.main:app --host 127.0.0.1 --port 8000
python -m backend.flagship.worker
npm --prefix frontend run dev -- --host 127.0.0.1
```
Open the frontend URL printed by Vite. `/health` checks the API process. `/ready`
checks database schema and a recent worker heartbeat; it returns 503 when the
worker is missing or stale. `/docs` describes the API. Reviewer access does not
require provider credentials and uses isolated replay state.
For a static production frontend served by the API, build before starting it:
```bash
npm --prefix frontend run build
```
The frontend and API must agree on their origin. Use `VITE_BACKEND_URL` during
frontend build only when the API is on a different origin. Configure allowed
origins with `HYPERFLOW_ORIGINS`; the default allows local Vite development and
preview ports. Session credentials currently persist in browser storage and are sent in Authorization headers, including authenticated event streams. This makes XSS prevention a session-security boundary. Verify reload and offline behavior against the mounted workspace; do not claim offline sign-out without testing.
## PostgreSQL and Docker deployment
Outside demonstration mode, configure a persistent PostgreSQL `DATABASE_URL`. In every mode, configure an encryption key before connecting any live provider account. Generate a key
locally without committing it:
```bash
python -c 'import base64,secrets; print(base64.b64encode(secrets.token_bytes(32)).decode())'
```
Store the result as `VAULT_SECRET_KEY` in the deployment's secret configuration.
Keep that key available for restores; encrypted provider credentials depend on it.
Redis notifications are optional acceleration. PostgreSQL rows remain the source
of truth for jobs and event replay. Kafka is not required by the v2 deployment.
The Compose deployment defines PostgreSQL, Redis, API, and worker services. Inspect
its required environment values before starting. Avoid placing a real provider
token in shared deployment configuration; users connect their own credentials
through the authenticated application.
```bash
docker compose config --quiet
docker compose build
docker compose up -d postgres redis
docker compose run --rm --no-deps api python -m backend.flagship.manage migrate
docker compose run --rm --no-deps api python -m backend.flagship.manage provision admin --role admin
docker compose up -d api worker
```
A build is a packaging check, not a live integration test. Record Docker/runtime
versions and the exact source revision with release results. Do not report a
container test unless Docker actually ran successfully on the target machine.
## Backup, restore, and rollback
Back up the database, model registry, uploaded dataset references, and vault secret
as one release recovery set. Store backups outside the application volume.
Do not copy a running SQLite file as a consistent backup; stop API and worker first
or use SQLite's backup command.
A PostgreSQL custom-format database backup can be produced without embedding its
password in the shell command:
```bash
docker compose exec -T postgres sh -c 'pg_dump -Fc -U "$POSTGRES_USER" "$POSTGRES_DB"' > hyperflow-backup.dump
```
Restore into a separate empty test database first. Verify account counts, schema
version, action histories, artifact hashes, and readiness before changing traffic.
Use `pg_restore --no-owner` against the explicitly selected restoration database.
Never restore over the active database as a first validation step.
Model rollback is an authenticated admin operation at `POST /api/v2/models/rollback`.
It restores the prior verified artifact pointer. For application rollback, stop
API and worker, restore the previously tested image, and verify schema compatibility
before restarting. The current migration is additive and records legacy provenance
as `legacy_unknown`; there is no automatic destructive downgrade command.
## Verification and failure rehearsal
```bash
python -m pytest tests/flagship tests/flagship_ops tests/flagship_ml -q
npm --prefix frontend run build
python scripts/eval_dispatch.py --seed 42 --cases 20 --output /tmp/hyperflow-dispatch.json
python scripts/study_censoring.py --output /tmp/hyperflow-censoring
```
Run the retail study separately. A local compatible input remains user-supplied
until its provenance is established; `--download-uci` explicitly downloads the
public UCI dataset and preserves its source identity. Do not silently convert this
UK retail study into Indian quick-commerce evidence.
```bash
python scripts/study_retail.py --input /path/to/retail.csv --output /tmp/hyperflow-retail
python scripts/study_retail.py --download-uci --output /tmp/hyperflow-retail-uci
```
Rehearse `stockout`, `price_change`, `external_edit`, `lost_response`, and `uncertain`
in replay mode. Check that renewed approval is required after a changed proposal,
unknown outcomes remain visibly uncertain, and retries do not blindly create a
second placement. Restart the worker after a queued job, then inspect its terminal
state and attempt count. Disconnect Redis and verify database-backed event replay.
Run live discovery only after explicit credential setup. Paid model requests and
real purchases/bookings are outside unattended CI. Preserve redacted receipts and
capability evidence separately from simulation outputs.
## Independent reviewer checklist
A person unfamiliar with the code should record their environment and source
revision, then complete these checks without verbal assistance:
- Start from a fresh checkout and run migrations, API, frontend, and worker.
- Open an isolated reviewer workspace and identify every replay/simulation label.
- Complete a basket workflow and explain infeasibility or renewed approval.
- Trigger a lost response and identify the saved action outcome without guessing.
- Enroll a courier, assign a HyperFlow-owned delivery, and complete its lifecycle.
- Confirm another account cannot inspect or mutate that job.
- Reproduce the dispatch and ML studies, including their source limitations.
- Identify the currently served model and its artifact manifest.
- Use keyboard navigation and a mobile viewport; record inaccessible controls.
- Confirm empty and unavailable states contain no invented operational metrics.
Keep feedback and unresolved issues in the release evidence. Do not mark
independent setup/usability validation complete based only on automated tests or
an author-led demonstration.
## ₹0 campaign and provider connection
Use existing hardware; do not create paid cloud resources or enable a billable model fallback. Keep `HYPERFLOW_ENABLE_LLM` unset for ordinary verification. The frozen free-quota budgets and loop limits are in [methodology](../benchmark_methodology.md). Purchases and bookings remain disabled for this campaign.
Configure `VAULT_SECRET_KEY` through local secret configuration, then sign in and use the credential form. Never paste a token into documentation, a committed `.env`, shell arguments or evidence. Saving a token establishes storage only. Read-only provider verification must establish the current credential's status; unknown expiry is not an invented five-day window. A registered OAuth client and an exact configured redirect are required for the browser authorization flow. See [integration](../../INTEGRATION.md) for required behavior.
## Incident drill record
For each drill save source/environment identities, incident start, observed symptom, request/action IDs, authoritative record/event snapshots, redacted receipt evidence, diagnosis, recovery command, recovered state, duration and remaining uncertainty. Use isolated data. Redact account details and credentials.
Rehearse API loss, worker restart, Redis notification loss, provider timeout after simulated commit and an artifact-write failure. Confirm that UNKNOWN blocks another placement, retained events support reconnect, failed training preserves the active artifact and readiness reflects actual dependency failure. `doctor` and `trace` output support diagnosis; their mere presence is not proof of deployment or FDE delivery. Never use teamwork metadata as worker liveness.
Backup/restore rehearsal must restore a real backup into a separate database and verify schema, account/action/event counts and artifact hashes. Application rollback must use an identified prior build with compatible schema. Record actual commands and exit statuses; a written runbook is not a completed drill.
## Volunteer pilot form
```text
Date / participant identifier (with consent):
Source revision and dirty-file hashes / environment:
User problem, constraints and agreed success criteria:
Fresh setup start/end / assistance count and details:
Task 1 — create a basket: completion, time, errors, assistance:
Task 2 — handle changed proposal: completion, time, understanding:
Task 3 — inspect uncertain outcome: completion, time, recovery choice:
Observed defect and reproduction:
Repair / regression / retest result:
What became easier; what remains confusing:
Limitations: one tester; prior exposure; assistance and learning effects:
```
Do not claim the pilot until a person other than the author actually performs it. The three-task pilot does not replace the fuller release checklist above.
## Release evidence bundle
Retain setup/build/test logs, current source hashes, benchmark manifests and raw results, recovery/restore drill records, pilot feedback, open findings, architecture diagram and the three-minute demonstration. Update [status](status.md) and [reviews](reviews.md) with exact results. A public URL, live transaction evidence and Docker execution remain separate gates; none is implied by a local build. For 21 October applications, use only completed evidence while later phases remain open.
## Current model and evaluation controls
Model experiments additionally require `HYPERFLOW_LLM_FREE_QUOTA_CONFIRMED=1` after the operator verifies a free-quota-only provider configuration. This flag does not inspect billing. Keep both model flags disabled for ordinary verification. Interrupted turns require inspection through `/api/v2/agent/turns` before starting a new attempt.
Unified offline evaluations: `python -m backend.flagship.manage evaluate --suite intents --output .local/evidence/intents` (also solver, commerce, parsing, censoring, retail and dispatch). Retail requires an existing local input workbook. See each command's `--help`; no dataset or model downloads are implied.
Current completed and incomplete drills are recorded in [status](status.md). The initial interrupted socket campaign remains historical. The separate complete campaign satisfies the three-run local performance protocol; see the status register for measurements and exclusions.
|