docs/DEPLOYMENT.md

# Deployment

**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `OPEN` · `DEFERRED` · `BLOCKED` · `BY DESIGN`.

The live SatQuery AI system runs across **three private repositories** plus one **public umbrella
repository**. The monorepo working copy is **not** the deployed source. This document records the live
topology, every environment variable, the deploy mechanics, the cold-start behaviour, the known
issues, and the platform traps that cost real debugging time.

> **The single most important trap in this document.** `deploy/` inside the monorepo is **stale and
> untracked**. It is **not** the deployed source. Edits must go to the three real repositories.

---

## 1. Live revisions (VERIFIED)

Read from the GitHub API during the release reconnaissance.

| Component | Repository | Visibility | Branch | Revision | Host |
|---|---|---|---|---|---|
| Frontend | `Anish-lab-blip/SatQuery-Frontend` | **private** | `main` | **`2d7ae53b482d`** | Cloudflare Pages → `satquery.pages.dev` |
| Backend / orchestrator | `Anish-lab-blip/SatQuery-Backend` | **private** | `main` | **`89d80eaddec5`** | Render → `satquery-backend-m4yv.onrender.com` |
| Inference | `Anish-lab-blip/SatQuery-Inference` | **private** | `main` | **`5a0936ace491`** | Codespace `potential-space-trout-r4ppw969w45j2pvvw`, port 8000 |
| Public umbrella | `Anish-lab-blip/SatQuery-AI` | **public** | `main` | `3dcabd32da41` ("Initial commit") | this release home |
| Monorepo (working copy) | `C:/Users/anish/satquery-ai` | local only | `master` | `9d57aed` | **no git remote**; 334 dirty entries |

The public umbrella repository previously contained **only a 13-byte README** (`# SatQuery-AI`). It is
the intended home for this release.

## 2. The four tiers

```
Browser
  │  HTTPS
  ▼
Cloudflare Pages  —  satquery.pages.dev                (static frontend, 11 pages)
  │  HTTPS / JSON  →  /api/*
  ▼
Render            —  satquery-backend-m4yv.onrender.com (orchestrator / gateway)
  │  outbound long-poll  POST /tunnel/agent
  ▼
GitHub Codespace  —  FastAPI inference, CPU, port 8000
  │  build_space_app()
  ▼
specialists: SmolVLM · RemoteCLIP · MiniLM · CROMA · STANet
  │
  ▼
ResultEnvelope  →  tunnel  →  Render  →  browser
```

```mermaid
flowchart LR
  U[Browser] -->|HTTPS| CF["Cloudflare Pages<br/>static frontend"]
  CF -->|"HTTPS JSON<br/>/api/health · /api/capabilities · /api/infer · /api/assets"| R["Render<br/>orchestrator / gateway"]
  R -->|"outbound long-poll<br/>POST /tunnel/agent"| C["GitHub Codespace<br/>FastAPI inference :8000"]
  C --> S[(SmolVLM · RemoteCLIP<br/>MiniLM · CROMA · STANet)]
  C -->|ResultEnvelope| R
  R -->|"envelope + error translation"| CF
```

## 3. Why each tier exists

### 3.1 Cloudflare Pages — the static tier

Serves the frontend. **No backend, no secrets, and no API calls of its own** on the static pages. The
one exception is the Analyze console, which calls the gateway. Staged by `scripts/stage_pages.mjs` and
deployed with `npx wrangler pages deploy`.

### 3.2 Render — the gateway

A deliberately **thin, stateless** orchestrator. Its responsibilities, from
`docs/DEPLOYMENT_ARCHITECTURE.md` §2:

| Responsibility | Detail |
|---|---|
| Schema validation | reject malformed requests before they cost inference |
| Size limits | whole-request body cap, shared with the Codespace |
| Rate limiting | per-IP count + window — **fairness, NOT a security control** |
| CORS allowlist | the Pages origin; **never `*`** |
| Request ids | correlate a request across tiers |
| Timeouts | sit inside the task budget |
| Secret custody | holds credentials that must never reach the browser |
| Error translation | upstream failures → the documented error envelope |

It is **not** a model host. It has **no database, no auth, and no queue**.

> **Hard rule.** The gateway must **not** retry `POST /api/infer` on its own — a retry would consume
> inference a second time. The client decides on retry.

### 3.3 GitHub Codespace — the inference tier

Runs the real inference service: `build_space_app()` from `app/space_app.py`, served by
`deploy/codespace/serve.py` on `$PORT`, in **CPU mode**. It honours the four-endpoint contract
(`/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets`), imports cheaply without torch, reuses
`app/serving.py` as the composition root, and **degrades rather than crashes** on absent artifacts.

### 3.4 Hugging Face — the model tier

Holds the six trained artifacts and the model card. It is **not** the runtime inference host; the
Codespace resolves the pinned backbones from the Hub at run time.

## 4. Why the transport is an outbound tunnel

The inference host is a Codespace in a **private** repository. A forwarded port for a private repo
returns **`302`**, so an inbound-forwarding design cannot work. Instead:

- the Codespace runs `deploy/codespace/tunnel_agent.py`;
- the agent **dials out** to `POST /tunnel/agent` and long-polls;
- work is executed against `http://127.0.0.1:8000` **locally**.

This inverts the usual direction: the inference host needs **no inbound firewall hole**. It also means
the transport is only alive while the agent is polling.

**Measured:** `GET /api/health` reported `tunnel.agent_connected: true` with a non-zero `completed`
counter, and `POST /api/infer {}` returned `422 invalid_request` with the response header
`x-satquery-transport: tunnel`.

## 5. The full live health payload (VERIFIED, probed)

```json
{
  "status": "ok",
  "service": "satquery-orchestrator",
  "tunnel": {
    "agent_connected": true,
    "agent_id": "codespaces-fd1038",
    "pending": 0,
    "completed": 97
  },
  "config": {
    "codespace_name": "potential-space-trout-r4ppw969w45j2pvvw\n",
    "codespace_port": 8000,
    "transport_mode": "auto",
    "tunnel_timeout_s": 150.0,
    "wake_timeout_s": 120.0,
    "upstream_timeout_s": 90.0,
    "device": "cpu",
    "has_github_token": true
  }
}
```

Note `codespace_name` still carries a **trailing `\n`** — this is **B-02**, cosmetic and `OPEN`; the
wake path strips it (`_codespace_name()`), so only the health payload reports the raw value.

## 6. Environment variables

### 6.1 Render (gateway) — measured live values

| Variable | Value (live) | Purpose |
|---|---|---|
| `CODESPACE_NAME` | `potential-space-trout-r4ppw969w45j2pvvw` | which Codespace to wake |
| `CODESPACE_PORT` | `8000` | the inference port |
| `SATQUERY_ALLOWED_ORIGINS` | `https://satquery.pages.dev` | CORS allowlist (never `*`) |
| `SATQUERY_DEVICE` | `cpu` | device preference |
| `SATQUERY_TRANSPORT` | `auto` | tunnel first, then forward |
| `SATQUERY_TUNNEL_TIMEOUT_S` | `150` | how long to wait on the tunnel |
| `SATQUERY_WAKE_TIMEOUT_S` | `120` | how long to wait for a cold start |
| `SATQUERY_UPSTREAM_TIMEOUT_S` | `90` | gateway → upstream budget |
| `GITHUB_TOKEN` | present | Codespace control |
| `PORT` | platform-supplied | Render's own listen port |

> **Measured absence.** There is **no** `SATQUERY_UPSTREAM_URL` and **no** `HF_TOKEN` in the live
> config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older
> `docs/DEPLOYMENT_TOPOLOGY.md` §3.2 table, which predates the tunnel design.

### 6.2 Codespace (inference)

| Variable | Purpose |
|---|---|
| `PORT` | platform-assigned; **must be read** (a historical blocker was that no code read it) |
| `SATQUERY_DEVICE` | `cpu` \| `cuda` \| `mps` \| `null`; read **without importing torch** |
| `SATQUERY_MAX_FILE_BYTES` | per-file cap, shared with Render |
| `SATQUERY_ASSET_ENABLED` / `SATQUERY_ASSET_DIR` | both required for `/v1/assets`; **fails closed (503)** otherwise |
| `SATQUERY_ASSET_MAX_FILES` / `SATQUERY_ASSET_TTL_S` | optional handle capacity / lifetime |

### 6.3 Config-loader environment overrides

Two registry values can be overridden from the environment without editing the YAML
(`core/config.py`):

| Variable | Effect |
|---|---|
| `SATQUERY_PRECISION` | overrides `training.precision` |
| `SATQUERY_TORCH_COMPILE` | overrides `deployment.torch_compile` (`"true"` → `True`) |

Both are still validated by the loader — setting `SATQUERY_TORCH_COMPILE=true` **fails startup**
because finding C-8 forbids `torch.compile`.

## 7. Deploy mechanics

| Tier | Mechanism |
|---|---|
| Frontend → Cloudflare Pages | `scripts/stage_pages.mjs` builds a Pages bundle; `npx wrangler pages deploy` |
| Backend → Render | `render.yaml` blueprint; `main.py` exposes the ASGI object `app` |
| Inference → Codespace | `deploy/codespace/serve.py` binds `build_space_app()` to `$PORT`; `.devcontainer/` forwards 8000 and starts the tunnel agent via `postStartCommand` |
| Repository writes | the **GitHub Git Data API** — blob → tree → commit → `PATCH` ref |

### 7.1 Why the Git Data API instead of `git push`

Every deployed file is uploaded as a **blob** whose sha256 is **computed locally and verified against
the uploaded blob**, then assembled into a tree, committed, and the branch ref patched. This means:

- each file is **content-verified** rather than trusted;
- deletions are expressed explicitly as `sha: null` tree entries;
- the deploy is **idempotent** — re-running it with identical content produces no change.

**Measured:** 9 deployed files were re-read from the API and found **sha256 byte-identical** to the
local copies, with the deployed HEAD re-read independently (`verify_deployed_head.py`).

## 8. Cold start (documented, not hidden)

Render's free tier sleeps when idle, and the Codespace may be stopped. Before a request can be served,
Render must start the Codespace (if stopped) and wait for the tunnel agent to reconnect. The frontend
shows *"Waking inference engine…"* during this.

| Property | Value |
|---|---|
| Cold start | **tens of seconds** |
| Tunnel wait before falling through | `SATQUERY_TUNNEL_TIMEOUT_S` = 150 s |
| Wake wait | `SATQUERY_WAKE_TIMEOUT_S` = 120 s |
| Upstream budget | `SATQUERY_UPSTREAM_TIMEOUT_S` = 90 s |

Cold start is **documented rather than papered over**: an honest "this will take a while the first
time" is better than a silent hang.

### 8.1 The `transport_mode: auto` fallthrough — B-07 (`OPEN`)

`SATQUERY_TRANSPORT=auto` means: **try the tunnel; on timeout, fall through to the forward path.**

The forward path to a **private** repo returns `302` quickly — but the wake step still consumes
`SATQUERY_WAKE_TIMEOUT_S` (120 s) **first**. So a worst-case failed request takes roughly

```
150 s (tunnel timeout)  +  120 s (wake timeout on a 302)  ≈  249 s
```

This is the **root shape** of the observed transient tunnel gap, and it is why a request can appear to
hang and then fail. A patch (`fix-b07-forward-unavailable.patch`) was authored and verified
(`py_compile` clean, applies cleanly to the deployed `main.py`), adding distinct
`forward_unavailable` (503, terminal 302/401/403 on the forward path) and `upstream_timeout`
(504, tunnel healthy but slow) codes, plus the `codespace_name` `.strip()` fix.

> **Status: B-07 is `OPEN`.** The patch is **prepared but NOT deployed.** The deployed health payload
> still shows the trailing `\n`.

## 9. The five historical backend blockers

Before any backend could boot, five verified blockers had to be closed. The current design closes them:

| # | Blocker (verified) | How it is closed |
|---|---|---|
| 1 | `requirements.txt` declared no `fastapi` / `uvicorn` / `httpx` / `starlette` | the runtime installs the ASGI stack so `build_space_app()` and the gateway can import |
| 2 | no code read `$PORT` — a platform port would be ignored | `deploy/codespace/serve.py` binds to `$PORT`; Render reads its own |
| 3 | hand-rolled CORS raised `405` on `OPTIONS`, so browser preflight failed | the gateway registers `OPTIONS` explicitly / uses Starlette's CORS middleware |
| 4 | module-level `app = create_app()` swallowed config errors into `app = None` | construction errors now **propagate** (fail-fast) instead of leaving a dead `app` |
| 5 | adapter integrity unverified on load (`_adapter_sha256` computed but never compared) | the load path compares the digest against an expected value, or fails startup |

## 10. Platform traps (recorded so they are not rediscovered)

| Trap | Detail |
|---|---|
| **Cloudflare `_headers` CONCATENATE** | Two matching rules are **merged, not overridden**. A specific rule nested under a broad `/assets/img/*` rule yields `max-age=604800, …, max-age=0, must-revalidate` — and Chromium takes the **FIRST** `max-age`. The file's own "later rules override" comment is **false**. |
| **Cloudflare 308 redirect** | `X.html` → `/X`. Reference the extensionless path. |
| **Forwarded port returns 302** | for a private repo — this is *why* the tunnel exists. |
| **Tunnel agent must start on boot** | via the devcontainer `postStartCommand`, or a restarted Codespace comes up with `agent_connected: false`. |
| **Never retry `/api/infer` at the gateway** | a retry consumes inference twice. |
| **`deploy/` is stale and untracked** | not the deployed source. |

## 11. The superseded design, and what did NOT change

The earlier design ran inference on an **HF Space with ZeroGPU** (5 GPU-min/day, `@spaces.GPU`
decoration) behind a **Railway** gateway. The active design changes three things:

1. **CPU-first instead of ZeroGPU.** No code change was required — `device_preference` honours
   `SATQUERY_DEVICE` and defaults to CPU, every specialist defaults to `device="cpu"`, and all
   placement is `.to(device)` (never `.cuda()`).
2. **A real, always-buildable inference environment.** A Codespace gives a reproducible container
   without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy
   loading as the cold-start story.
3. **No GPU quota to protect at the gateway.** Rate/size limits remain, but as **fairness** controls
   rather than quota protection.

**What did NOT change:**

| Unchanged | Detail |
|---|---|
| the four-endpoint contract | `/v1/health`, `/v1/capabilities`, `/v1/analyze`, `/v1/assets` |
| the gateway responsibility table | §3.2 above |
| the env-var vocabulary | only host names moved (`SATQUERY_SPACE_URL` → `SATQUERY_UPSTREAM_URL`) |
| the config freeze | `78f1e3700da15aa1` |

`configs/deploy.yaml` still describes the old HF-Space/ZeroGPU target (`platform: huggingface-spaces`,
`sdk: gradio`, `zerogpu: true`, the `gpu_duration_*` values). It is **frozen paperwork**: no Gradio
runtime exists in code, and editing it would move `Config.hash`. It is left undisturbed.

## 12. Deployment preconditions (open items)

1. Cloudflare Pages project name / domain (needed for the deploy command and `robots.txt` sitemap).
2. Artifacts present, **or** capabilities honestly ship `available: false` — the service degrades
   rather than crashes on absent artifacts.
3. `HF_TOKEN` on Render **if** the HF proxy path is used (not used in the live config).
4. Codespace `.devcontainer/` forwarding `:8000` **and** starting the tunnel agent.
5. The five blockers in §9 closed and verified on the first live run.

## 13. What is `NOT RUN` / `OPEN` / `BLOCKED` for this topic

| Item | State |
|---|---|
| B-07 tunnel fallthrough | **OPEN** — patch prepared, NOT deployed |
| B-02 `codespace_name` trailing `\n` | **OPEN** (cosmetic) |
| A deployed system-level load test | **NOT RUN** |
| Multi-region / HA deployment | **NOT RUN** |
| A production database / persistence | **BY DESIGN** (absent) |
| The ZeroGPU/Gradio deployment target | **REJECTED** (superseded; frozen paperwork only) |

## 14. Where the evidence lives

| Evidence | Location |
|---|---|
| Live health + capabilities probe | `release/CURRENT_RELEASE_STATE.md` §1 |
| Topology contract | `docs/DEPLOYMENT_TOPOLOGY.md`, `docs/DEPLOYMENT_ARCHITECTURE.md` |
| The undeployed B-07 patch | session scratch: `fix-b07-forward-unavailable.patch` |
| Deployed-artifact integrity check | `verify_deployed_head.py` (session scratch) |
| Live validation (3 passes) | `.workbuddy-ai/scratch/live_validation/` |