Spaces:
Running
Running
| # GitHub content checkpoints (format 1) | |
| The HF entrypoint enables content backup by default. Configure **GITHUB_PAT as a | |
| Space secret**, with Contents read/write permission limited to the **private** | |
| `JsonLord/nodepad-brain-backup` repository. Never put it in the source repository, | |
| remote URLs, chat, console settings, or the archive. `SYSTEM_PAT` remains a | |
| separate Space secret. Keep deployment's `HF_TOKEN` outside the Space runtime. | |
| Missing `GITHUB_PAT` with backup enabled prevents startup; it never falls back to | |
| another credential. No deployment is performed by this implementation. | |
| Variables: | |
| | Variable | Default | Purpose | | |
| |---|---|---| | |
| | `BRAIN_BACKUP_ENABLED` | `true` in HF entrypoint; `false` otherwise | Enable restore gate, commit tracking, checkpoint daemon | | |
| | `BRAIN_BACKUP_REPOSITORY` | `JsonLord/nodepad-brain-backup` | GitHub owner/repository | | |
| | `BRAIN_BACKUP_BRANCH` | `main` | Checkpoint branch; never force pushed | | |
| | `BRAIN_BACKUP_INTERVAL_SECONDS` | `300` | Debounce/coalescing window and reconciliation interval | | |
| | `BRAIN_BACKUP_ROOT` | `$PERSISTENT_ROOT/content-backup` | Private local checkout, checkpoint receipt, operation lock | | |
| | `BRAIN_BACKUP_STATE_DIR` | `/tmp/brainapi-backup-state` | Shared dirty generation and public status (no secrets) | | |
| | `GITHUB_PAT` (secret) | none | GitHub HTTPS authentication only | | |
| `BRAIN_BACKUP_STARTUP_ID` is generated by the entrypoint for each container start; | |
| do not configure it manually. API and MCP refuse to start without a successful | |
| restore gate for that container. The runtime image includes Git and PostgreSQL 15 | |
| logical backup tools. Normal API/worker profiles remain PostgreSQL + NetworkX, | |
| EmbeddingGemma 2 native 768 → normalized 512, all five vector stores at 512. | |
| ## Format | |
| ``` | |
| manifest.json | |
| system/registry.json | |
| brains/<canonical-brain-id>.dump.part0000 | |
| brains/<canonical-brain-id>.dump.part0001 # only when needed | |
| ``` | |
| The registry JSON contains only explicit Brain identity fields and typed workspace | |
| metadata, preserving IDs and timestamps; legacy PAT fields are removed. It is | |
| restored with parameterized inserts, not arbitrary registry SQL. Brain payloads | |
| are PostgreSQL custom-format **logical** pg_dump archives split into 32 MiB Git | |
| files. No PGDATA, Redis, console/session/CSRF state, passwords, internal secrets, | |
| or access-token registry/hashes are exported. Known deployment secrets and Brain | |
| PATs accidentally pasted in content cause checkpoint rejection rather than upload. | |
| Access tokens must be reissued separately after an ephemeral restore. Existing | |
| auth state on a durable local database is left alone. | |
| Set `BUILD_SHA` to the deployed Git commit (Docker build argument or pinned Space | |
| variable) for Git provenance; unpinned local builds record `unknown`. | |
| The manifest records format/schema version, PostgreSQL major version, application | |
| version, source SHA, creation time, canonical Brain names, full embedding profile, | |
| native/stored dimensions, payload order, per-file SHA-256 checksums, and semantic | |
| content fingerprints. Archive timestamps never cause a redundant commit. Every | |
| content table is allowlisted; unexpected per-Brain tables stop backup for operator | |
| review rather than silently losing unmodelled content or copying credentials. | |
| Unregistered databases and orphan workspaces are not discovered or exported. | |
| Physical-name collisions among canonical names are rejected. | |
| ## Startup and recovery | |
| 1. Start temporary PostgreSQL/Redis and initialize the system database/schema. | |
| 2. Fetch the configured Git branch using environment-only credentials through a | |
| token-free askpass program. Git errors are fixed codes, with captured output | |
| omitted. No token is written to a Git config, URL, archive, or exception. | |
| 3. Validate the complete manifest, payload checksums/layout, PostgreSQL major, and | |
| configured embedding identity. Validate each logical archive's TOC. | |
| 4. If registry is empty, reject destination database collisions, create only the | |
| registered Brain databases, restore archives transactionally per database and | |
| validate all vector profile rows/dimensions. Publish registry/workspaces **last** | |
| in one transaction. On failure drop only databases created by this restore. | |
| 5. Record the checkpoint lineage locally, then bootstrap workspaces, stop temporary | |
| services, and start Supervisor/API/MCP/workers. | |
| An empty remote, or a README-only initialized repository with no snapshot in its | |
| history, permits first boot and later seeding from local content. A deleted | |
| manifest, backup payloads without a manifest, corrupt/incompatible snapshot, or | |
| missing credential fails closed. Startup still fetches and validates when local data is | |
| nonempty. A matching local receipt retains newer local changes without restoring | |
| an older snapshot. Unknown/nonmatching lineage fails with | |
| `BACKUP_LOCAL_REMOTE_CONFLICT`; deleted remote history also fails. It never | |
| merges unrelated snapshots or overwrites populated local databases. | |
| For recovery, first stop API/MCP/workers and preserve the local databases and | |
| checkpoint receipt. Inspect local and remote commits. Restore into a **fresh, | |
| isolated cluster** using the normal entrypoint and verified configuration; reissue | |
| scoped tokens using SYSTEM_PAT. Do not delete legitimate databases, edit lineage | |
| receipts to bypass conflicts, force push, or use the old offline Redis/volume | |
| backup as a content checkpoint. If two writers diverge, choose/merge content in an | |
| isolated recovery environment, qualify it, and publish a reviewed successor. | |
| ### Planned Hugging Face restarts | |
| Before restarting/rebuilding an ephemeral Space, quiesce content writers and wait | |
| for a successful checkpoint containing their completed writes. Confirm the remote | |
| manifest/commit matches `/health` and validate the snapshot before replacement. | |
| HF can start a replacement before the old container finishes its final shutdown | |
| checkpoint; the replacement can therefore restore the preceding commit. A final | |
| shutdown attempt alone is not a maintenance checkpoint barrier. | |
| After startup, compare the restored commit with the intended remote checkpoint | |
| and verify identities and retrieval before resuming writers. If a later shutdown | |
| commit appeared, preserve both histories and local state, stop writers, and follow | |
| the isolated recovery procedure above. Do not bypass lineage protection or force | |
| push. See [live qualification evidence](content-backup-live-qualification.md). | |
| ## Runtime and limits | |
| A PostgreSQL connection hook marks successful content mutations dirty across API, | |
| Celery, graph and vector clients. Auth-only writes and rolled-back transactions do | |
| not count. Every 300 seconds the daemon reconciles logical content as well, covering | |
| administrative SQL and missed dirty markers. This coalesces writes; no new commit | |
| is created when semantic contents/schema/profiles are unchanged. A file lock | |
| serializes attempts across processes. Each Brain's dump and fingerprint share a | |
| repeatable-read snapshot; registry/workspace changes during export reject the | |
| attempt. This is per-database consistency, not a distributed transaction across | |
| all Brains. Keep the operation root on the same durable mount as local PostgreSQL | |
| when such a mount exists. | |
| Push compares the fetched head with the local receipt and uses a normal atomic, | |
| non-force Git push. Remote changes or a racing push cause a visible degraded | |
| state and retain local data; there is no unsafe rebase/retry. Retry occurs at the | |
| next interval only when the expected remote lineage remains unchanged. Health | |
| exposes safe checkpoint state/commit/time/error separately from local persistence. | |
| On graceful shutdown Supervisor stops request/worker writers first, then gives the | |
| checkpoint daemon **120 seconds** for its final attempt, then stops PostgreSQL. | |
| Forced termination can lose changes since the last successful checkpoint. This is | |
| an asynchronous backup with roughly interval + upload time RPO, not a substitute | |
| for a durable database or a zero-loss production guarantee. Existing explicit | |
| `ALLOW_EPHEMERAL_STORAGE` and degraded ephemeral health rules are preserved. | |
| Large snapshots/history can eventually exceed GitHub repository/storage limits; | |
| monitor growth and failed/stale checkpoint status. No automatic history rewrite. | |
| ## Qualification | |
| Run `pytest tests/test_brain_backup.py` for Git/manifest/startup safety gates. | |
| For a real restore round-trip, use **only a disposable PostgreSQL 15 + pgvector | |
| cluster**, configure the HF CPU profile and PostgreSQL credentials/tools, then: | |
| ``` | |
| BACKUP_QUALIFICATION_DISPOSABLE=true \ | |
| BACKUP_QUALIFICATION_RETRIEVAL=true \ | |
| python scripts/qualify_content_backup.py | |
| ``` | |
| It creates random test brains and unique facts, verifies dirty tracking/no-op | |
| commits, auth and orphan exclusions, all five profiles and mismatch rejection, | |
| drops only those test databases, restores their IDs/workspaces/facts, calls actual | |
| search/context controllers, and checks a new mutation produces a new commit. | |
| Retrieval qualification requires the preloaded real Ollama model and normal NLP | |
| runtime. It never pulls a model at application startup or deploys a Space. | |
| ## Verified maintenance barrier and RPO readiness | |
| `SYSTEM_PAT` administrators have these operations; scoped tokens cannot use them: | |
| | Operation | Effect | | |
| | --- | --- | | |
| | `POST /system/backup/quiesce` | Close new HTTP/MCP content admission; accepted requests/tasks may finish. | | |
| | `POST /system/backup/checkpoint?timeout_seconds=120` | Quiesce, drain admitted requests and all tracked Celery deliveries/children/retries, seal, snapshot, push, fetch HEAD and validate manifest/checksums. Return exact durable SHA. | | |
| | `GET /system/backup/status` | Read hold state, pending task count, durable SHA and safe error code. | | |
| | `POST /system/backup/resume` | Verify/publish the hold release before reopening admission. Also cancels maintenance after accepted work settles. | | |
| A successful barrier returns `state=durable`, `checkpoint=<sha>`, `dirty=false`, | |
| `remote_verified=true`, `quiesced=true`. Writes stay held until explicit resume. | |
| A drain timeout, ambiguous task publication, Git failure or verification conflict | |
| keeps admission closed and reports degraded state. Accepted work settles at Celery | |
| SUCCESS/FAILURE terminal delivery, including all already-published descendants; | |
| failed ingestion does not become successful merely because a checkpoint exists. | |
| An active delivery is counted separately from its publication so broker | |
| redelivery cannot make the barrier finish while a duplicate execution is active. | |
| External administrative SQL or independently running writers must also be stopped; | |
| this barrier controls the application's HTTP/MCP and managed Celery writers. | |
| The local control/accepted-work ledger contains only opaque IDs/states and is not | |
| backed up. A version-1 manifest optionally carries `maintenance` as an empty object | |
| or `{ "id": "<opaque-id>", "quiesced": true }`. This non-secret handoff metadata | |
| makes a freshly restored replacement remain quiesced. Publishing/releasing that | |
| hold is a real control-state change and may create a commit with unchanged content. | |
| Normal unchanged reconciliation still creates no commit. | |
| Planned restart sequence: | |
| 1. Quiesce writers and wait accepted work to settle using the checkpoint operation. | |
| 2. Record its remotely verified checkpoint SHA **A**; do not resume the old container. | |
| 3. Trigger the HF rebuild/restart. | |
| 4. Verify replacement `/health`: `restored_checkpoint` is A or a verified descendant, | |
| `remote_checkpoint` matches, `checkpoint_lag=0`, and no backup error is present. | |
| 5. Verify canonical identities and retrieval while the hold remains closed. | |
| 6. Explicitly resume through `/system/backup/resume`, then verify write admission. | |
| Startup re-fetches HEAD after restoration, and again after workspace bootstrap | |
| before Supervisor/API startup. Any advance fails closed with | |
| `BACKUP_STARTUP_REMOTE_ADVANCED`; it never serves stale writable content or merges | |
| active writers. The normal periodic/shutdown daemon and maintenance operations | |
| share a file lock. A shutdown checkpoint is still best-effort and is not the | |
| maintenance barrier. | |
| Once the hold is sealed, the periodic/shutdown daemon only verifies the pinned | |
| checkpoint; it does not publish a successor until explicit checkpoint/resume. | |
| Read paths can initialize derived indexes/profiles locally, but must not move the | |
| operator's handoff SHA through a late shutdown snapshot. Qualification accepts a | |
| descendant only with an operator-fetched Git checkout proving ancestry. | |
| `BRAIN_BACKUP_MAX_RPO_SECONDS` defaults to `BRAIN_BACKUP_INTERVAL_SECONDS` (300). | |
| Health exposes restored/remote SHA, lag, dynamic dirty state, last successful | |
| checkpoint/reconciliation time and age. An unchanged semantic reconciliation | |
| refreshes the successful verification time without creating a commit. Git reachability, | |
| auth/database/schema/profile and baked-model availability, plus an actual Uvicorn | |
| formatter self-test, are checked; expensive probes are cached for at most 10 seconds. | |
| Production readiness is withdrawn on observed Git failure, lineage conflict, | |
| checkpoint age beyond the RPO, or an unhealthy dependency. | |
| `checkpoint_lag` is an equality indicator: 0 means matching HEAD, 1 means a | |
| different HEAD, and null means unknown; it is not a count of missing commits. | |
| `ALLOW_CHECKPOINTED_PRODUCTION=false` is the default: ephemeral local storage | |
| remains production NO-GO. An operator may explicitly set it to `true` to accept | |
| bounded-loss GitHub checkpoint recovery. Readiness then also requires a qualified | |
| restore, current verified remote lineage, a successful checkpoint within the maximum | |
| age, and all dependency checks. Local storage is still reported `ephemeral`, content | |
| recovery `github-checkpoint`, `zero_loss=false`; this setting does not improve the | |
| underlying durability. With interval and maximum age both 300, long snapshots or | |
| uploads can temporarily exceed the policy and withdraw readiness; monitor this. | |
| Unexpected container death can lose writes since the last successful checkpoint. | |
| Zero-loss operation would require a synchronous external durable write path. This | |
| implementation does not commit to Git synchronously for every Brain write. | |