docs/DEPLOYMENT.md

Deployment

Status tags: IMPLEMENTED · VERIFIED · MEASURED · OPEN · DEFERRED · BLOCKED · BY DESIGN.

The live SatQuery AI system runs across three private repositories plus one public umbrella repository. The monorepo working copy is not the deployed source. This document records the live topology, every environment variable, the deploy mechanics, the cold-start behaviour, the known issues, and the platform traps that cost real debugging time.

The single most important trap in this document. deploy/ inside the monorepo is stale and untracked. It is not the deployed source. Edits must go to the three real repositories.

---

1. Live revisions (VERIFIED)

Read from the GitHub API during the release reconnaissance.

Component	Repository	Visibility	Branch	Revision	Host
Frontend	Anish-lab-blip/SatQuery-Frontend	private	main	2d7ae53b482d	Cloudflare Pages → satquery.pages.dev
Backend / orchestrator	Anish-lab-blip/SatQuery-Backend	private	main	89d80eaddec5	Render → satquery-backend-m4yv.onrender.com
Inference	Anish-lab-blip/SatQuery-Inference	private	main	5a0936ace491	Codespace potential-space-trout-r4ppw969w45j2pvvw, port 8000
Public umbrella	Anish-lab-blip/SatQuery-AI	public	main	3dcabd32da41 ("Initial commit")	this release home
Monorepo (working copy)	C:/Users/anish/satquery-ai	local only	master	9d57aed	no git remote; 334 dirty entries

The public umbrella repository previously contained only a 13-byte README (# SatQuery-AI). It is the intended home for this release.

2. The four tiers

Browser
  │  HTTPS
  ▼
Cloudflare Pages  —  satquery.pages.dev                (static frontend, 11 pages)
  │  HTTPS / JSON  →  /api/*
  ▼
Render            —  satquery-backend-m4yv.onrender.com (orchestrator / gateway)
  │  outbound long-poll  POST /tunnel/agent
  ▼
GitHub Codespace  —  FastAPI inference, CPU, port 8000
  │  build_space_app()
  ▼
specialists: SmolVLM · RemoteCLIP · MiniLM · CROMA · STANet
  │
  ▼
ResultEnvelope  →  tunnel  →  Render  →  browser

flowchart LR
  U[Browser] -->|HTTPS| CF["Cloudflare Pages<br/>static frontend"]
  CF -->|"HTTPS JSON<br/>/api/health · /api/capabilities · /api/infer · /api/assets"| R["Render<br/>orchestrator / gateway"]
  R -->|"outbound long-poll<br/>POST /tunnel/agent"| C["GitHub Codespace<br/>FastAPI inference :8000"]
  C --> S[(SmolVLM · RemoteCLIP<br/>MiniLM · CROMA · STANet)]
  C -->|ResultEnvelope| R
  R -->|"envelope + error translation"| CF

3. Why each tier exists

3.1 Cloudflare Pages — the static tier

Serves the frontend. No backend, no secrets, and no API calls of its own on the static pages. The one exception is the Analyze console, which calls the gateway. Staged by scripts/stage_pages.mjs and deployed with npx wrangler pages deploy.

3.2 Render — the gateway

A deliberately thin, stateless orchestrator. Its responsibilities, from docs/DEPLOYMENT_ARCHITECTURE.md §2:

Responsibility	Detail
Schema validation	reject malformed requests before they cost inference
Size limits	whole-request body cap, shared with the Codespace
Rate limiting	per-IP count + window — fairness, NOT a security control
CORS allowlist	the Pages origin; never *
Request ids	correlate a request across tiers
Timeouts	sit inside the task budget
Secret custody	holds credentials that must never reach the browser
Error translation	upstream failures → the documented error envelope

It is not a model host. It has no database, no auth, and no queue.

Hard rule. The gateway must not retry POST /api/infer on its own — a retry would consume inference a second time. The client decides on retry.

3.3 GitHub Codespace — the inference tier

Runs the real inference service: build_space_app() from app/space_app.py, served by deploy/codespace/serve.py on $PORT, in CPU mode. It honours the four-endpoint contract (/v1/health, /v1/capabilities, /v1/analyze, /v1/assets), imports cheaply without torch, reuses app/serving.py as the composition root, and degrades rather than crashes on absent artifacts.

3.4 Hugging Face — the model tier

Holds the six trained artifacts and the model card. It is not the runtime inference host; the Codespace resolves the pinned backbones from the Hub at run time.

4. Why the transport is an outbound tunnel

The inference host is a Codespace in a private repository. A forwarded port for a private repo returns 302, so an inbound-forwarding design cannot work. Instead:

• the Codespace runs deploy/codespace/tunnel_agent.py;
• the agent dials out to POST /tunnel/agent and long-polls;
• work is executed against http://127.0.0.1:8000 locally.

This inverts the usual direction: the inference host needs no inbound firewall hole. It also means the transport is only alive while the agent is polling.

Measured: GET /api/health reported tunnel.agent_connected: true with a non-zero completed counter, and POST /api/infer {} returned 422 invalid_request with the response header x-satquery-transport: tunnel.

5. The full live health payload (VERIFIED, probed)

{
  "status": "ok",
  "service": "satquery-orchestrator",
  "tunnel": {
    "agent_connected": true,
    "agent_id": "codespaces-fd1038",
    "pending": 0,
    "completed": 97
  },
  "config": {
    "codespace_name": "potential-space-trout-r4ppw969w45j2pvvw\n",
    "codespace_port": 8000,
    "transport_mode": "auto",
    "tunnel_timeout_s": 150.0,
    "wake_timeout_s": 120.0,
    "upstream_timeout_s": 90.0,
    "device": "cpu",
    "has_github_token": true
  }
}

Note codespace_name still carries a trailing \n — this is B-02, cosmetic and OPEN; the wake path strips it (_codespace_name()), so only the health payload reports the raw value.

6. Environment variables

6.1 Render (gateway) — measured live values

Variable	Value (live)	Purpose
CODESPACE_NAME	potential-space-trout-r4ppw969w45j2pvvw	which Codespace to wake
CODESPACE_PORT	8000	the inference port
SATQUERY_ALLOWED_ORIGINS	https://satquery.pages.dev	CORS allowlist (never *)
SATQUERY_DEVICE	cpu	device preference
SATQUERY_TRANSPORT	auto	tunnel first, then forward
SATQUERY_TUNNEL_TIMEOUT_S	150	how long to wait on the tunnel
SATQUERY_WAKE_TIMEOUT_S	120	how long to wait for a cold start
SATQUERY_UPSTREAM_TIMEOUT_S	90	gateway → upstream budget
GITHUB_TOKEN	present	Codespace control
PORT	platform-supplied	Render's own listen port

Measured absence. There is no SATQUERY_UPSTREAM_URL and no HF_TOKEN in the live config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older docs/DEPLOYMENT_TOPOLOGY.md §3.2 table, which predates the tunnel design.

6.2 Codespace (inference)

Variable	Purpose
PORT	platform-assigned; must be read (a historical blocker was that no code read it)
SATQUERY_DEVICE	cpu | cuda | mps | null; read without importing torch
SATQUERY_MAX_FILE_BYTES	per-file cap, shared with Render
SATQUERY_ASSET_ENABLED / SATQUERY_ASSET_DIR	both required for /v1/assets; fails closed (503) otherwise
SATQUERY_ASSET_MAX_FILES / SATQUERY_ASSET_TTL_S	optional handle capacity / lifetime

6.3 Config-loader environment overrides

Two registry values can be overridden from the environment without editing the YAML (core/config.py):

Variable	Effect
SATQUERY_PRECISION	overrides training.precision
SATQUERY_TORCH_COMPILE	overrides deployment.torch_compile ("true" → True)

Both are still validated by the loader — setting SATQUERY_TORCH_COMPILE=true fails startup because finding C-8 forbids torch.compile.

7. Deploy mechanics

Tier	Mechanism
Frontend → Cloudflare Pages	scripts/stage_pages.mjs builds a Pages bundle; npx wrangler pages deploy
Backend → Render	render.yaml blueprint; main.py exposes the ASGI object app
Inference → Codespace	deploy/codespace/serve.py binds build_space_app() to $PORT; .devcontainer/ forwards 8000 and starts the tunnel agent via postStartCommand
Repository writes	the GitHub Git Data API — blob → tree → commit → PATCH ref

7.1 Why the Git Data API instead of git push

Every deployed file is uploaded as a blob whose sha256 is computed locally and verified against the uploaded blob, then assembled into a tree, committed, and the branch ref patched. This means:

• each file is content-verified rather than trusted;
• deletions are expressed explicitly as sha: null tree entries;
• the deploy is idempotent — re-running it with identical content produces no change.

Measured: 9 deployed files were re-read from the API and found sha256 byte-identical to the local copies, with the deployed HEAD re-read independently (verify_deployed_head.py).

8. Cold start (documented, not hidden)

Render's free tier sleeps when idle, and the Codespace may be stopped. Before a request can be served, Render must start the Codespace (if stopped) and wait for the tunnel agent to reconnect. The frontend shows "Waking inference engine…" during this.

Property	Value
Cold start	tens of seconds
Tunnel wait before falling through	SATQUERY_TUNNEL_TIMEOUT_S = 150 s
Wake wait	SATQUERY_WAKE_TIMEOUT_S = 120 s
Upstream budget	SATQUERY_UPSTREAM_TIMEOUT_S = 90 s

Cold start is documented rather than papered over: an honest "this will take a while the first time" is better than a silent hang.

8.1 The transport_mode: auto fallthrough — B-07 (OPEN)

SATQUERY_TRANSPORT=auto means: try the tunnel; on timeout, fall through to the forward path.

The forward path to a private repo returns 302 quickly — but the wake step still consumes SATQUERY_WAKE_TIMEOUT_S (120 s) first. So a worst-case failed request takes roughly

150 s (tunnel timeout)  +  120 s (wake timeout on a 302)  ≈  249 s

This is the root shape of the observed transient tunnel gap, and it is why a request can appear to hang and then fail. A patch (fix-b07-forward-unavailable.patch) was authored and verified (py_compile clean, applies cleanly to the deployed main.py), adding distinct forward_unavailable (503, terminal 302/401/403 on the forward path) and upstream_timeout (504, tunnel healthy but slow) codes, plus the codespace_name .strip() fix.

Status: B-07 is OPEN. The patch is prepared but NOT deployed. The deployed health payload still shows the trailing \n.

9. The five historical backend blockers

Before any backend could boot, five verified blockers had to be closed. The current design closes them:

#	Blocker (verified)	How it is closed
1	requirements.txt declared no fastapi / uvicorn / httpx / starlette	the runtime installs the ASGI stack so build_space_app() and the gateway can import
2	no code read $PORT — a platform port would be ignored	deploy/codespace/serve.py binds to $PORT; Render reads its own
3	hand-rolled CORS raised 405 on OPTIONS, so browser preflight failed	the gateway registers OPTIONS explicitly / uses Starlette's CORS middleware
4	module-level app = create_app() swallowed config errors into app = None	construction errors now propagate (fail-fast) instead of leaving a dead app
5	adapter integrity unverified on load (_adapter_sha256 computed but never compared)	the load path compares the digest against an expected value, or fails startup

10. Platform traps (recorded so they are not rediscovered)

Trap	Detail
Cloudflare _headers CONCATENATE	Two matching rules are merged, not overridden. A specific rule nested under a broad /assets/img/* rule yields max-age=604800, …, max-age=0, must-revalidate — and Chromium takes the FIRST max-age. The file's own "later rules override" comment is false.
Cloudflare 308 redirect	X.html → /X. Reference the extensionless path.
Forwarded port returns 302	for a private repo — this is why the tunnel exists.
Tunnel agent must start on boot	via the devcontainer postStartCommand, or a restarted Codespace comes up with agent_connected: false.
Never retry /api/infer at the gateway	a retry consumes inference twice.
deploy/ is stale and untracked	not the deployed source.

11. The superseded design, and what did NOT change

The earlier design ran inference on an HF Space with ZeroGPU (5 GPU-min/day, @spaces.GPU decoration) behind a Railway gateway. The active design changes three things:

1. CPU-first instead of ZeroGPU. No code change was required — device_preference honours SATQUERY_DEVICE and defaults to CPU, every specialist defaults to device="cpu", and all placement is .to(device) (never .cuda()).
2. A real, always-buildable inference environment. A Codespace gives a reproducible container without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy loading as the cold-start story.
3. No GPU quota to protect at the gateway. Rate/size limits remain, but as fairness controls rather than quota protection.

What did NOT change:

Unchanged	Detail
the four-endpoint contract	/v1/health, /v1/capabilities, /v1/analyze, /v1/assets
the gateway responsibility table	§3.2 above
the env-var vocabulary	only host names moved (SATQUERY_SPACE_URL → SATQUERY_UPSTREAM_URL)
the config freeze	78f1e3700da15aa1

configs/deploy.yaml still describes the old HF-Space/ZeroGPU target (platform: huggingface-spaces, sdk: gradio, zerogpu: true, the gpu_duration_* values). It is frozen paperwork: no Gradio runtime exists in code, and editing it would move Config.hash. It is left undisturbed.

12. Deployment preconditions (open items)

1. Cloudflare Pages project name / domain (needed for the deploy command and robots.txt sitemap).
2. Artifacts present, or capabilities honestly ship available: false — the service degrades rather than crashes on absent artifacts.
3. HF_TOKEN on Render if the HF proxy path is used (not used in the live config).
4. Codespace .devcontainer/ forwarding :8000 and starting the tunnel agent.
5. The five blockers in §9 closed and verified on the first live run.

13. What is NOT RUN / OPEN / BLOCKED for this topic

Item	State
B-07 tunnel fallthrough	OPEN — patch prepared, NOT deployed
B-02 codespace_name trailing \n	OPEN (cosmetic)
A deployed system-level load test	NOT RUN
Multi-region / HA deployment	NOT RUN
A production database / persistence	BY DESIGN (absent)
The ZeroGPU/Gradio deployment target	REJECTED (superseded; frozen paperwork only)

14. Where the evidence lives

Evidence	Location
Live health + capabilities probe	release/CURRENT_RELEASE_STATE.md §1
Topology contract	docs/DEPLOYMENT_TOPOLOGY.md, docs/DEPLOYMENT_ARCHITECTURE.md
The undeployed B-07 patch	session scratch: fix-b07-forward-unavailable.patch
Deployed-artifact integrity check	verify_deployed_head.py (session scratch)
Live validation (3 passes)	.workbuddy-ai/scratch/live_validation/