THOX Open WebUI: branded image + Dataset-backed state + connection seeding

Dockerfile

@@ -0,0 +1,42 @@
+# THOX Open WebUI — Hugging Face Space (CPU, private)
+#
+# Branding is BAKED, not mounted. open-webui's config.py copies /app/build/static/*
+# into the served static dir on EVERY boot: a read-only mount crashes that copy with
+# Errno 30, and a read-write mount gets overwritten by the stock source. Writing THOX
+# assets into BOTH the copy source and the destination makes the boot-copy THOX->THOX
+# and therefore idempotent.
+FROM ghcr.io/open-webui/open-webui:main
+
+USER root
+
+# zstd/tar drive the assets archive; huggingface_hub moves state to/from the Dataset.
+RUN apt-get update && apt-get install -y --no-install-recommends zstd tar \
+ && rm -rf /var/lib/apt/lists/* \
+ && pip install --no-cache-dir "huggingface_hub>=0.34" hf_transfer
+
+# --- branding: copy SOURCE (read at boot) and SERVED target ---
+COPY brand/static/favicon.png /app/build/favicon.png
+COPY brand/static/            /app/build/static/
+COPY brand/static/            /app/backend/open_webui/static/
+
+# --- THOX runtime ---
+COPY brand/model-avatar.png   /app/thox/thox-model-avatar.png
+COPY scripts/                 /app/thox/
+RUN chmod +x /app/thox/thox-space-boot.sh
+
+ENV WEBUI_NAME="THOX" \
+    ENV=prod \
+    PORT=8080 \
+    HF_HUB_ENABLE_HF_TRANSFER=1 \
+    DATA_DIR=/app/backend/data \
+    STATE_REPO=tommytracx/thox-openwebui-state \
+    BACKUP_INTERVAL=300 \
+    SPACE_OWNER=tommytracx \
+    SPACE_PREFIX=thox-zero- \
+    EXTRA_OPENAI_URLS=https://route.thox.ai/v1 \
+    SCARF_NO_ANALYTICS=true \
+    DO_NOT_TRACK=true \
+    ANONYMIZED_TELEMETRY=false
+
+EXPOSE 8080
+ENTRYPOINT ["bash", "/app/thox/thox-space-boot.sh"]

README.md

@@ -1,10 +1,73 @@
 ---
-title: Thox Openwebui
-emoji: 📈
-colorFrom: red
-colorTo: blue
+title: THOX Open WebUI
+emoji: 💚
+colorFrom: green
+colorTo: gray
 sdk: docker
+app_port: 8080
 pinned: false
+short_description: THOX-branded Open WebUI with Dataset-backed state
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# THOX Open WebUI
+
+Your AI. Your Data. Your Rules.™
+
+THOX-branded Open WebUI, restored from the live THOX instance and wired to the
+THOX ZeroGPU model Spaces.
+
+## Persistence — read this before trusting it
+
+Spaces have **ephemeral** storage. The paid persistent-storage tier is unavailable
+on this account (`whoami-v2` reports `canPay: false`), so state lives in the private
+Dataset **`tommytracx/thox-openwebui-state`** and is restored on every cold start.
+
+| Artifact | Contents | Cadence |
+|---|---|---|
+| `webui.db` | users, chats, models, prompts, folders, settings | every 5 min, only when changed |
+| `assets.tar.zst` | `uploads/` + `vector_db/` | every 12th cycle (~1h), only when changed |
+
+**What this does not guarantee.** Writes made since the last successful backup are
+**lost** on an unexpected restart. A Space stop does not reliably deliver `SIGTERM`,
+so the periodic loop — not the shutdown hook — is the real durability mechanism. If
+two instances ever run at once, the database is last-writer-wins. This is genuinely
+weaker than a Docker volume; it is the ceiling of the free tier.
+
+Two implementation details that matter:
+
+- `webui.db` is live SQLite in **WAL** mode, so backups use `VACUUM INTO` rather
+  than a file copy, which would otherwise capture a torn database.
+- `vector_db/` is ~4250 Chroma directories. It travels as a single tar because
+  syncing it as loose files costs ~4250 requests per pass.
+
+## Model connections
+
+Seeded on boot by `scripts/seed_connections.py`, which **writes the `config` table
+directly**. `OPENAI_API_BASE_URLS` cannot be used: it is a `PersistentConfig` value
+that the environment seeds only on first boot, and this Space restores an existing
+database every time, so the env var would be ignored.
+
+Connections are discovered by listing Spaces under `SPACE_OWNER` matching
+`SPACE_PREFIX` (`thox-zero-`), so model Spaces are picked up as they come online,
+plus `EXTRA_OPENAI_URLS` (ThoxRoute at `https://route.thox.ai/v1`).
+
+The model Spaces are private; Hugging Face gates them at the edge and the HF token
+is what satisfies that check, so it doubles as the connection API key.
+
+## Required Space secrets
+
+| Name | Purpose |
+|---|---|
+| `HF_TOKEN` | restore/backup against the state Dataset, and auth to the model Spaces |
+| `WEBUI_SECRET_KEY` | session signing |
+
+## Branding
+
+Assets are **baked into the image**, not mounted. `config.py` re-copies
+`/app/build/static/*` into the served directory on every boot; a `:ro` mount makes
+that copy fail with `Errno 30` and a `:rw` mount is overwritten by the stock source.
+Writing THOX assets to both locations makes the copy idempotent.
+
+Palette is brand v2 (`ttracx/thox-design-system`): `#05A451` primary, `#45e887`
+accent, dark-first. Primary controls use **black** ink — white on `#05A451` is
+3.26:1 and fails WCAG AA, while black is 6.43:1.

brand/static/custom.css

@@ -0,0 +1,59 @@
+/* =====================================================================
+   THOX Open WebUI — brand v2 overlay
+   Source of truth: ttracx/thox-design-system (standards/brand-v2.md, ADR-0001)
+
+   CONTRAST NOTE — this is the whole reason the ink is black, not white:
+     #FFFFFF on #05A451 = 3.26:1  -> FAILS WCAG AA
+     #000000 on #05A451 = 6.43:1  -> PASSES  (primary-button pairing)
+     #05A451 on #09090B = 6.09:1  -> PASSES  (green as text on dark canvas)
+     #37C77A on #09090B = 9.1:1   -> focus rings
+   The reflex "white text on the brand green" is wrong here. Do not "fix" it.
+
+   #10B981 / #34D399 are DEPRECATED v1 emeralds. Purple #A855F7 is MagStack-only
+   and neon #00FF88 is device-illumination-only — neither belongs in UI chrome.
+   ===================================================================== */
+:root {
+  --thox-green:#05A451;        /* canonical primary */
+  --thox-green-ink:#000000;    /* the only AA-safe ink on --thox-green */
+  --thox-accent:#45e887;       /* accent / highlight on dark */
+  --thox-focus:#37C77A;        /* 9.1:1 on canvas */
+  --thox-green-deep:#03763B;   /* for light surfaces: + white ink = 5.74:1 */
+  --thox-bg:#09090B;
+  --thox-surface:#18181B;
+  --thox-border:#27272A;
+  --thox-text:#FAFAFA;
+  --thox-font-sans:'Inter',-apple-system,BlinkMacSystemFont,'Segoe UI',system-ui,sans-serif;
+  --thox-font-mono:'JetBrains Mono',ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;
+}
+
+body, .font-primary { font-family: var(--thox-font-sans); -webkit-font-smoothing: antialiased; }
+code, pre, kbd, .font-mono, textarea.font-mono { font-family: var(--thox-font-mono); }
+
+:is(button, a, input, textarea, select):focus-visible {
+  outline: 2px solid var(--thox-focus);
+  outline-offset: 2px;
+}
+
+/* Emphasis controls -> THOX green with black ink. open-webui uses bg-black in
+   light mode and bg-white in dark for its primary affordance. */
+button.bg-black:not(.text-black),
+.dark button.bg-white:not(.text-white) {
+  background-color: var(--thox-green) !important;
+  color: var(--thox-green-ink) !important;
+}
+button.bg-black:hover,
+.dark button.bg-white:hover { background-color: var(--thox-accent) !important; }
+
+/* Green as *text* on the dark canvas is AA-clean, so links use it directly. */
+a:not(.no-thox) { color: var(--thox-accent); }
+a:not(.no-thox):hover { color: var(--thox-green); }
+
+/* On light surfaces the plain green is not approved for body text. */
+@media (prefers-color-scheme: light) {
+  a:not(.no-thox) { color: var(--thox-green-deep); }
+}
+
+::selection { background: var(--thox-green); color: var(--thox-green-ink); }
+* { scrollbar-color: var(--thox-border) transparent; }
+::-webkit-scrollbar-thumb { background: var(--thox-border); border-radius: 6px; }
+::-webkit-scrollbar-thumb:hover { background: var(--thox-green); }

scripts/seed_connections.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Point Open WebUI's OpenAI connections at the THOX ZeroGPU model Spaces.
+
+WHY THIS WRITES TO THE DATABASE INSTEAD OF USING ENV VARS
+    OPENAI_API_BASE_URLS / OPENAI_API_KEYS are Open WebUI `PersistentConfig`
+    values: the environment seeds them on FIRST boot only, and from then on the
+    value stored in the database wins. This Space restores an existing database
+    every boot, so env vars would be silently ignored. Seeding therefore has to
+    write the `config` table directly.
+
+DISCOVERY
+    The ZeroGPU model Spaces are built by a separate lane from a shared template
+    and are still being rolled out, so rather than pin a list that goes stale we
+    enumerate Spaces matching SPACE_PREFIX under the account and add whichever
+    exist. Re-running is safe: entries are matched by URL and never duplicated.
+
+AUTH
+    The model Spaces are private, so Hugging Face gates them at the edge. The HF
+    token is what satisfies that check and is used as the connection's API key.
+"""
+import json, os, sqlite3, sys
+
+DB     = os.environ.get("DATA_DIR", "/app/backend/data") + "/webui.db"
+TOKEN  = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+OWNER  = os.environ.get("SPACE_OWNER", "tommytracx")
+PREFIX = os.environ.get("SPACE_PREFIX", "thox-zero-")
+EXTRA  = [u for u in os.environ.get("EXTRA_OPENAI_URLS", "https://route.thox.ai/v1").split(",") if u.strip()]
+
+
+def log(m): print(f"[thox-connections] {m}", flush=True)
+
+
+def discover():
+    """(url, key) pairs for each ZeroGPU model Space that currently exists."""
+    urls = []
+    if TOKEN:
+        try:
+            from huggingface_hub import HfApi
+            for s in HfApi(token=TOKEN).list_spaces(author=OWNER):
+                name = s.id.split("/")[-1]
+                if name.startswith(PREFIX):
+                    host = f"{OWNER}-{name}".lower().replace("_", "-")
+                    urls.append((f"https://{host}.hf.space/v1", TOKEN))
+        except Exception as exc:
+            log(f"space discovery failed ({type(exc).__name__}) - using extras only")
+    for u in EXTRA:
+        urls.append((u.strip(), ""))
+    return urls
+
+
+def get(cur, key, default):
+    row = cur.execute("select value from config where key=?", (key,)).fetchone()
+    if not row:
+        return default
+    try:
+        return json.loads(row[0])
+    except (TypeError, json.JSONDecodeError):
+        return default
+
+
+def put(cur, key, value):
+    payload = json.dumps(value)
+    if cur.execute("select 1 from config where key=?", (key,)).fetchone():
+        cur.execute("update config set value=? where key=?", (payload, key))
+    else:
+        cur.execute("insert into config (key,value) values (?,?)", (key, payload))
+
+
+def main():
+    if not os.path.exists(DB):
+        log("no database yet - skipping"); return
+    targets = discover()
+    if not targets:
+        log("nothing to seed"); return
+
+    con = sqlite3.connect(DB); cur = con.cursor()
+    base = get(cur, "openai.api_base_urls", [])
+    keys = get(cur, "openai.api_keys", [])
+    cfgs = get(cur, "openai.api_configs", {})
+
+    # api_base_urls and api_keys are positional partners; api_configs is keyed by
+    # the same index. Appending keeps every existing index stable.
+    while len(keys) < len(base):
+        keys.append("")
+
+    added = 0
+    for url, key in targets:
+        if url in base:
+            idx = base.index(url)
+            if key:
+                keys[idx] = key                      # refresh a rotated token
+            continue
+        base.append(url); keys.append(key)
+        cfgs[str(len(base) - 1)] = {"enable": True, "tags": [], "prefix_id": "",
+                                    "model_ids": [], "connection_type": "external"}
+        added += 1
+        log(f"added {url}")
+
+    put(cur, "openai.api_base_urls", base)
+    put(cur, "openai.api_keys", keys)
+    put(cur, "openai.api_configs", cfgs)
+    put(cur, "openai.enable", True)
+    con.commit(); con.close()
+    log(f"seeded {added} new connection(s); {len(base)} total")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/set-model-avatars.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+"""Set every OpenWebUI model's avatar (meta.profile_image_url) to the THOX logo.
+
+The per-model avatar lives in the SQLite DB (/app/backend/data/webui.db, in the
+persistent volume), stored as a base64 data URI — the same shape OpenWebUI uses for
+uploaded avatars. Storing a self-contained data URI (not a /static path) means the
+avatar survives image updates AND does not depend on any static asset path.
+
+Idempotent. Re-run after re-provisioning / re-importing models so a WebUI update
+cannot revert avatars to the default chip.
+
+Usage (inside the container):
+    python set-model-avatars.py [--only-empty] [AVATAR_PNG] [DB_PATH]
+
+    --only-empty  Only set the avatar on models that have no image or a stock
+                  default (empty / None / a /static/favicon* path). Used by the
+                  on-boot hook so it re-brands re-provisioned/new models WITHOUT
+                  clobbering any avatar a user later sets by hand. Omit it to force
+                  the THOX logo onto EVERY model.
+Defaults: AVATAR_PNG=/tmp/thox-model-avatar.png  DB_PATH=/app/backend/data/webui.db
+"""
+import sqlite3, json, base64, sys, os
+
+args = [a for a in sys.argv[1:] if a != "--only-empty"]
+ONLY_EMPTY = "--only-empty" in sys.argv[1:]
+AVATAR = args[0] if len(args) > 0 else "/tmp/thox-model-avatar.png"
+DB     = args[1] if len(args) > 1 else "/app/backend/data/webui.db"
+
+if not os.path.exists(AVATAR):
+    sys.exit(f"avatar not found: {AVATAR}")
+if not os.path.exists(DB):
+    sys.exit(f"db not found: {DB}")
+
+def is_default(url):
+    """True if this avatar is empty or a stock/default (i.e. safe to (re)brand)."""
+    if not url:
+        return True
+    return url.startswith("/static/favicon") or url in ("/favicon.png", "/static/favicon.png")
+
+with open(AVATAR, "rb") as f:
+    data_uri = "data:image/png;base64," + base64.b64encode(f.read()).decode()
+
+con = sqlite3.connect(DB)
+cur = con.cursor()
+# Tolerate a not-yet-migrated DB (fresh volume, model table not created yet).
+tabs = [r[0] for r in cur.execute("SELECT name FROM sqlite_master WHERE type='table'")]
+if "model" not in tabs:
+    print("model table not present yet; nothing to do")
+    sys.exit(0)
+rows = cur.execute("SELECT id, meta FROM model").fetchall()
+changed = 0
+for mid, meta in rows:
+    m = json.loads(meta) if meta else {}
+    cur_url = m.get("profile_image_url")
+    if cur_url == data_uri:
+        continue
+    if ONLY_EMPTY and not is_default(cur_url):
+        continue
+    m["profile_image_url"] = data_uri
+    cur.execute("UPDATE model SET meta = ? WHERE id = ?", (json.dumps(m), mid))
+    changed += 1
+con.commit()
+con.close()
+print(f"models_total={len(rows)} updated={changed} only_empty={ONLY_EMPTY} avatar_bytes={len(data_uri)}")

scripts/thox-space-boot.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# THOX Open WebUI (Hugging Face Space) entrypoint.
+#
+#   restore state  ->  seed connections  ->  background backup loop  ->  exec app
+#
+# Storage on a Space is ephemeral and the paid persistent tier is unavailable on
+# this account, so /app/backend/data is rebuilt from a private HF Dataset on every
+# cold start. See scripts/thox_state.py for the hot/cold split and its limits.
+set -euo pipefail
+
+echo "[thox-boot] THOX Open WebUI starting"
+[ -n "${HF_TOKEN:-}" ] && echo "[thox-boot] HF_TOKEN present" \
+                       || echo "[thox-boot] WARNING: HF_TOKEN unset - no restore, no backup"
+
+python /app/thox/thox_state.py restore      || echo "[thox-boot] restore failed - continuing with empty data"
+python /app/thox/seed_connections.py        || echo "[thox-boot] connection seeding failed - configure manually in Settings"
+
+# Re-apply the THOX avatar to models that have none. Runs after the app has had a
+# chance to migrate the schema, and never overwrites a user-set image.
+if [ -f /app/thox/set-model-avatars.py ]; then
+  (
+    for _ in $(seq 1 40); do
+      if [ -f /app/backend/data/webui.db ]; then
+        python /app/thox/set-model-avatars.py --only-empty \
+               /app/thox/thox-model-avatar.png /app/backend/data/webui.db 2>&1 && break
+      fi
+      sleep 5
+    done
+  ) >/tmp/thox-avatars.log 2>&1 &
+fi
+
+python /app/thox/thox_state.py loop &
+LOOP_PID=$!
+
+# Best-effort final flush. SIGTERM is not guaranteed on a Space stop, which is
+# precisely why the periodic loop above is the real durability mechanism.
+trap 'echo "[thox-boot] SIGTERM - final backup"; kill $LOOP_PID 2>/dev/null || true; \
+      python /app/thox/thox_state.py backup || true' TERM INT
+
+cd /app/backend
+exec bash start.sh

scripts/thox_state.py

@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""THOX Open WebUI state sync: HF Dataset <-> /app/backend/data.
+
+WHY THIS EXISTS
+    HF Spaces have ephemeral storage. The paid persistent-storage tier is not
+    available on this account (whoami-v2 reports canPay:false), so a private HF
+    Dataset is the backing store instead.
+
+HOT / COLD SPLIT
+    webui.db          ~30MB, changes constantly -> snapshot+upload every cycle
+    assets.tar.zst    ~1.2GB, changes rarely    -> upload only when the hash moves
+
+    vector_db/ is ~4250 Chroma collection directories. Synced as loose files that
+    is 4250 requests per pass, so it travels as a single tar.
+
+CONSISTENCY
+    webui.db is live SQLite in WAL mode. A plain copy while Open WebUI is writing
+    yields a torn file, so backups go through `VACUUM INTO`, which takes an atomic
+    snapshot against concurrent writers.
+
+WHAT THIS DOES NOT GUARANTEE
+    Writes made since the last successful backup are lost on an unexpected
+    restart. This is weaker than a real volume; it is the ceiling on the free tier.
+"""
+import hashlib, os, pathlib, shutil, sqlite3, subprocess, sys, time
+
+DATA   = pathlib.Path(os.environ.get("DATA_DIR", "/app/backend/data"))
+REPO   = os.environ.get("STATE_REPO", "tommytracx/thox-openwebui-state")
+TOKEN  = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+INTERVAL = int(os.environ.get("BACKUP_INTERVAL", "300"))
+WORK   = pathlib.Path("/tmp/thox-state"); WORK.mkdir(parents=True, exist_ok=True)
+ASSETS = ["uploads", "vector_db"]
+
+
+def log(msg):
+    print(f"[thox-state] {msg}", flush=True)
+
+
+def _api():
+    from huggingface_hub import HfApi
+    return HfApi(token=TOKEN)
+
+
+def _sha(path, chunk=1 << 20):
+    h = hashlib.sha256()
+    with open(path, "rb") as fh:
+        for b in iter(lambda: fh.read(chunk), b""):
+            h.update(b)
+    return h.hexdigest()
+
+
+def _marker(name):
+    return WORK / f"{name}.sha256"
+
+
+# ---------------------------------------------------------------- restore
+def restore():
+    if not TOKEN:
+        log("no HF_TOKEN -> starting with an empty database (nothing restored)")
+        return False
+    from huggingface_hub import hf_hub_download
+    DATA.mkdir(parents=True, exist_ok=True)
+    ok = False
+
+    try:
+        p = hf_hub_download(REPO, "webui.db", repo_type="dataset", token=TOKEN,
+                            local_dir=str(WORK / "dl"))
+        shutil.copy2(p, DATA / "webui.db")
+        # A restored snapshot must not inherit a stale WAL from a previous boot.
+        for sidecar in ("webui.db-wal", "webui.db-shm"):
+            (DATA / sidecar).unlink(missing_ok=True)
+        n = sqlite3.connect(DATA / "webui.db").execute(
+            "select count(*) from chat").fetchone()[0]
+        _marker("webui.db").write_text(_sha(DATA / "webui.db"))
+        log(f"restored webui.db ({(DATA / 'webui.db').stat().st_size/1e6:.1f} MB, {n} chats)")
+        ok = True
+    except Exception as exc:                     # first boot, or repo still empty
+        log(f"no webui.db restored ({type(exc).__name__}) -> fresh database")
+
+    try:
+        p = hf_hub_download(REPO, "assets.tar.zst", repo_type="dataset", token=TOKEN,
+                            local_dir=str(WORK / "dl"))
+        subprocess.run(["tar", "--use-compress-program=unzstd", "-xf", p, "-C", str(DATA)],
+                       check=True)
+        _marker("assets").write_text(_sha(p))
+        log(f"restored assets.tar.zst -> {', '.join(ASSETS)}")
+        ok = True
+    except Exception as exc:
+        log(f"no assets restored ({type(exc).__name__}) -> knowledge bases will be empty")
+
+    return ok
+
+
+# ---------------------------------------------------------------- backup
+def _snapshot_db():
+    """Atomic, self-consistent copy of a live WAL database."""
+    out = WORK / "webui-snap.db"
+    out.unlink(missing_ok=True)
+    con = sqlite3.connect(f"file:{DATA/'webui.db'}?mode=ro", uri=True)
+    try:
+        con.execute("VACUUM INTO ?", (str(out),))
+    finally:
+        con.close()
+    return out
+
+
+def backup_db():
+    if not TOKEN or not (DATA / "webui.db").exists():
+        return False
+    snap = _snapshot_db()
+    digest = _sha(snap)
+    if _marker("webui.db").exists() and _marker("webui.db").read_text().strip() == digest:
+        return False                              # unchanged since last push
+    _api().upload_file(path_or_fileobj=str(snap), path_in_repo="webui.db",
+                       repo_id=REPO, repo_type="dataset",
+                       commit_message=f"state: webui.db {digest[:12]}")
+    _marker("webui.db").write_text(digest)
+    log(f"backed up webui.db ({snap.stat().st_size/1e6:.1f} MB, {digest[:12]})")
+    return True
+
+
+def backup_assets():
+    if not TOKEN:
+        return False
+    present = [a for a in ASSETS if (DATA / a).exists()]
+    if not present:
+        return False
+    tar = WORK / "assets.tar.zst"
+    tar.unlink(missing_ok=True)
+    subprocess.run(f"tar -C {DATA} -cf - {' '.join(present)} | zstd -3 -T0 -q -o {tar}",
+                   shell=True, check=True)
+    digest = _sha(tar)
+    if _marker("assets").exists() and _marker("assets").read_text().strip() == digest:
+        tar.unlink(missing_ok=True)
+        return False
+    _api().upload_file(path_or_fileobj=str(tar), path_in_repo="assets.tar.zst",
+                       repo_id=REPO, repo_type="dataset",
+                       commit_message=f"state: assets {digest[:12]}")
+    _marker("assets").write_text(digest)
+    log(f"backed up assets.tar.zst ({tar.stat().st_size/1e6:.0f} MB, {digest[:12]})")
+    tar.unlink(missing_ok=True)
+    return True
+
+
+def loop():
+    log(f"backup loop every {INTERVAL}s (db each cycle, assets every 12th)")
+    i = 0
+    while True:
+        time.sleep(INTERVAL)
+        i += 1
+        try:
+            backup_db()
+            if i % 12 == 0:
+                backup_assets()
+        except Exception as exc:                  # never let a sync fault kill the app
+            log(f"backup failed, will retry next cycle: {type(exc).__name__}: {exc}")
+
+
+if __name__ == "__main__":
+    cmd = sys.argv[1] if len(sys.argv) > 1 else "restore"
+    {"restore": restore, "backup": lambda: (backup_db(), backup_assets()),
+     "backup-db": backup_db, "backup-assets": backup_assets, "loop": loop}[cmd]()