forgesight / docs /runbook.md
Ankit
Deploy ForgeSight to Hugging Face Spaces
1d0f191
|
Raw History Blame Contribute Delete
13.2 kB

Runbook

Everything needed to run ForgeSight locally, with containers, and to read its output honestly. No step here requires a paid service, a GPU, or a credential.

1. Local, no container

This is the default and it is not a toy path: the full stack runs this way, and that is how the numbers in the README were produced.

make install      # uv sync with all extras
make fetch        # pinned weights, every sha256 verified
make export       # ONNX export with recorded provenance
make datasets     # synthetic evaluation sets

make fetch and make export are not optional. They populate the model registry, and with an empty registry POST /v1/sessions returns 503 and names the directories it expected. That refusal is deliberate: an earlier version minted the session anyway and handed back a workspace with no candidates and no active release, where every upload is refused and the token explains nothing.

The same applies to the containers: the models and artifacts volumes are bind-mounted from the host for exactly this reason. A fresh clone with empty volumes boots a healthy API that cannot process a page.

Why the containers use CPU-only torch

On Linux, uv.lock resolves torch to 2.14.1+cpu rather than the PyPI wheel, which is a CUDA build. uv lock does this automatically, so make install and podman compose build agree without extra flags.

The reason is measurement integrity rather than disk space. Initialising a CUDA context reserves host memory that has nothing to do with the model, and the headline output of this project is a per-runtime peak RSS figure. A CUDA torch in the torch worker would inflate exactly the number the system exists to measure, while the ONNX worker's figure stayed honest. The comparison would have been structurally unfair, in a direction that flattered whichever runtime got the smaller image.

Measured on the build host:

image size torch CUDA
INSTALL_EXTRAS=torch 2.1 GB 2.14.1+cpu not compiled in
INSTALL_EXTRAS=ort 1.3 GB absent n/a

Both models run on the CPU-only torch build: heron in 4550 ms and egret-medium in 1627 ms for a single 640×640 page on 4 threads, at 827 MB peak RSS.

macOS is unaffected. The marker is sys_platform == 'linux', so make install on the M5 still gets the native PyPI wheel, which is also CPU-only. There is no CUDA build for macOS.

Then four processes, in four terminals:

make api          # 127.0.0.1:8000, serves the built UI too
make worker-torch # the PyTorch pool
make worker-ort   # the ONNX Runtime pool
make reaper       # lease expiry, batch rollup, orphan sweep

Open http://127.0.0.1:8000. The first load creates a session and keeps its token in localStorage, so a reload works. A token that no longer validates is discarded and a fresh session is minted, once.

Database

SQLite by default, at data/live.db. For PostgreSQL:

export FORGESIGHT_DATABASE_URL='postgresql://forgesight:pw@127.0.0.1:5432/forgesight'
make migrate && make api

PostgreSQL is the deployment engine and the one the ledger is designed for. SQLite is the development engine; see ADR 0002 for exactly what is and is not equivalent.

2. Containers

cp .env.example .env      # then edit it: set the token pepper
podman compose up -d --build
podman compose logs -f api
podman compose down

.env needs FORGESIGHT_TOKEN_PEPPER set to a long random value. It is passed to the API, both workers and the reaper, and it must be identical in all of them: a per-process pepper means a token minted by one is rejected by the next.

The compose file builds the ORT worker from a stage that installs no torch, on purpose. If torch were present, the ORT pool's peak RSS could include torch's allocator and the per-runtime memory figures would stop meaning anything.

Read this before quoting any container number. A Podman machine is a Linux VM with its own CPU count and its own memory. A benchmark taken inside these containers describes that VM, not the host. It is a separate, labelled experiment, never comparable with a native run.

3. Benchmarks

make bench                  # throughput and latency, Tier 1 matrix
make report                 # the above plus the quality and gate passes

The runner refuses by default to measure on battery power, with Low Power Mode on, or when free memory is under 25%. The environment manifest records the CPU and its core layout, the OS, the power state, memory pressure before and after, and every library version.

uv run python scripts/run_bench.py --tier 1 --allow-unfavourable

takes a labelled measurement anyway. The report then carries the refusal reason on its face and says the numbers should not be compared with a nominal run. This is deliberate: on a machine that cannot be put on AC power, a labelled measurement is more useful than nothing, and an unlabelled one is worse than either.

Useful flags: --pages, --trials, --cooldown, --models, --runtimes, --quality, --out.

A configuration whose trial throughput varies by more than 5%, or drifts more than 10% from the first trial to the last, is marked unstable and excluded from every comparison. The report prints the reasons.

Output lands in data/reports/: a Markdown report, its JSON twin, and data/reports/raw/ with one JSON file of raw samples per trial. Every number in the report is read from one of those files.

4. Reading a report

Start with Limits. It states that the pages are synthetic, that the absolute mAP is not comparable with published numbers, and that the measurements are valid only for the host and date in the manifest.

Then check, in this order:

  1. Host stamp. If the CPU or library versions are not the ones you care about, stop.
  2. Stability. Anything marked unstable is excluded from comparison, and the report says which configurations were dropped and why.
  3. Comparisons. A speedup sentence is only emitted between stable configurations from the same run with the same env manifest, and it always carries its interval. If a comparison is refused, the sentence says so.
  4. Memory model. Read whether the fit is marked usable. On this host it is not; see ADR 0004.

5. Operations

The remote PostgreSQL

The dual-dialect suite needs a real PostgreSQL. If the host you have is ephemeral, such as a Lightning AI Studio or a CI sandbox, anything that discards state when it stops: read this first.

scripts/remote_pg.sh is the supported way to provision it, and it is idempotent: run it again after a restart instead of rebuilding by hand.

export SSH_REMOTE='<user>@<host>'
scripts/remote_pg.sh up       # create or start, then wait for readiness
scripts/remote_pg.sh status   # running? how many tables?
scripts/remote_pg.sh down     # remove the container

Then open the tunnel and point the suite at it:

ssh -f -N -L 55432:127.0.0.1:5432 "$SSH_REMOTE"
export FORGESIGHT_TEST_PG='postgresql://forgesight:forgesight@127.0.0.1:55432/forgesight'
make test-all

Running the suite on the remote host needs no tunnel. Use postgresql://forgesight:forgesight@127.0.0.1:5432/forgesight there.

Do not put PGDATA on a FUSE mount

PG_DIR is unset by default, and that is deliberate. PGDATA is container-local and recreated on every start.

The obvious improvement, bind-mounting PGDATA onto the only durable path, does not work on a Lightning AI Studio, and it fails in a way that looks like it worked. The persistent location is the Studio folder:

$ mount | grep ' type lightning '
lightning on /home/zeus type lightning (rw,relatime)
lightning on /teamspace/studios/this_studio type lightning (rw,relatime)

/teamspace/studios/this_studio is the one to use; this_studio is Lightning's own stable alias, so it does not change with the account. It is a network FUSE mount, and PostgreSQL needs more from a filesystem than durable file contents. With PGDATA there, the container came back after a host restart looking healthy. The tables were still on disk. Then it refused to start:

FATAL:  could not open directory "pg_notify": No such file or directory

An earlier check of this setup proved a table survived docker rm -f and concluded persistence worked. That tested file durability; the property that matters is whether PostgreSQL can start from the directory. The two are different, and only the second one caught this.

Set PG_DIR only on a real POSIX filesystem, such as a local ext4 or xfs volume, never on FUSE or a network mount.

Recreating the database costs nothing here: the test suite drops and recreates the schema in a fixture, so there is no state worth keeping. The script also passes --restart unless-stopped for an in-place restart, and removes a stopped container left by a previous host before recreating it.

/teamspace/uploads is mounted read-only, so it is not usable for anything.

Migrations

Plain numbered SQL, applied in order inside a transaction, recorded in schema_migration. Re-running is a no-op.

make migrate

There is no down-migration. Immutable data is never migrated destructively: a schema change adds a column or a table, and old rows are left readable.

Backups

Back up two things, and note that one of them is content-addressed:

pg_dump "$FORGESIGHT_DATABASE_URL" > backup.sql
tar czf objects.tgz data/objects

data/objects is keyed by content hash, so the same bytes are stored once however many workspaces uploaded them. Restoring is a matter of putting both back; the database references objects by key, never by absolute path.

Rotating the demo token

psql "$FORGESIGHT_DATABASE_URL" -c \
  "UPDATE api_token SET revoked_at = now() WHERE revoked_at IS NULL;"

Existing tokens stop working immediately. New sessions are unaffected.

Purging sandbox data

In public mode, sandbox workspaces carry a 24 h TTL. To purge expired ones:

psql "$FORGESIGHT_DATABASE_URL" -c "DELETE FROM workspace WHERE expires_at < now();"

Objects become unreferenced and the reaper deletes them once they pass the grace period (orphan_object_grace_h, 24 h by default). That delay is why the sweep is safe to run: it only touches keys that are both unreferenced and old.

When a worker wedges

Nothing is lost. The lease expires, the reaper requeues the item, and another worker picks it up. The wedged worker is fenced out when it comes back.

tail -f logs/worker-torch.log     # look for "lease lost"
podman compose restart worker-torch

A worker also recycles itself if its RSS exceeds the hard cap. The cap is rss_hard_cap_fraction (default 0.85) of total host memory, checked between micro-batches. On crossing it the worker finishes the batch in hand, persists it, stops claiming, and exits for the supervisor to restart, so the replacement picks the work up immediately rather than waiting out a lease.

The cap is deliberately not derived from worker_mem_budget. That number is admission's soft target and it is a prediction about work not yet run; on this host the calibration fit produced a negative intercept and was marked invalid (ADR 0004). The hard cap is a measurement of what the process already did, anchored to the machine, and it is only ever a fallback. If RSS cannot be read at all the worker does not recycle. Exiting a healthy process over a failed measurement is the wrong trade.

Recycles are counted in the worker's own stats.recycles, and are not yet surfaced through the API. GET /v1/system/status reports queue depth, the memory budget, the safety margin and admission_mispredictions (read from stored evaluation records), but not live worker counters. A recycle therefore shows up in logs/worker-<pool>.log as recycling: RSS is over the hard cap. Wiring stats through is the obvious next step if this ever runs unattended.

6. Troubleshooting

"running on battery power; connect AC to measure". The benchmark guard working as designed. Plug in, or use --allow-unfavourable and accept the label.

"database is locked". SQLite with more writers than it can serialise. WAL and the busy timeout are set on every connection; if this still appears, the filesystem does not support WAL locking (some network mounts), so use PostgreSQL.

401 on every request. The token pepper differs between processes, or the database was reset under a live token. FORGESIGHT_TOKEN_PEPPER must be set identically everywhere; the browser recovers on its own by minting a new session.

"artifact sha mismatch". The weights on disk differ from models.lock.json. The load is refused rather than trusted. Re-fetch with make fetch --force.

A batch stays queued. No worker for that pool. The design routes work to a pool by the candidate's runtime, so a torch candidate needs make worker-torch running. Check GET /v1/system/status for queue depth per pool.