docs/content-backup.md

GitHub content checkpoints (format 1)

The HF entrypoint enables content backup by default. Configure GITHUB_PAT as a Space secret, with Contents read/write permission limited to the private JsonLord/nodepad-brain-backup repository. Never put it in the source repository, remote URLs, chat, console settings, or the archive. SYSTEM_PAT remains a separate Space secret. Keep deployment's HF_TOKEN outside the Space runtime. Missing GITHUB_PAT with backup enabled prevents startup; it never falls back to another credential. No deployment is performed by this implementation.

Variables:

Variable	Default	Purpose
BRAIN_BACKUP_ENABLED	true in HF entrypoint; false otherwise	Enable restore gate, commit tracking, checkpoint daemon
BRAIN_BACKUP_REPOSITORY	JsonLord/nodepad-brain-backup	GitHub owner/repository
BRAIN_BACKUP_BRANCH	main	Checkpoint branch; never force pushed
BRAIN_BACKUP_INTERVAL_SECONDS	300	Debounce/coalescing window and reconciliation interval
BRAIN_BACKUP_ROOT	$PERSISTENT_ROOT/content-backup	Private local checkout, checkpoint receipt, operation lock
BRAIN_BACKUP_STATE_DIR	/tmp/brainapi-backup-state	Shared dirty generation and public status (no secrets)
GITHUB_PAT (secret)	none	GitHub HTTPS authentication only

BRAIN_BACKUP_STARTUP_ID is generated by the entrypoint for each container start; do not configure it manually. API and MCP refuse to start without a successful restore gate for that container. The runtime image includes Git and PostgreSQL 15 logical backup tools. Normal API/worker profiles remain PostgreSQL + NetworkX, EmbeddingGemma 2 native 768 → normalized 512, all five vector stores at 512.

Format

manifest.json
system/registry.json
brains/<canonical-brain-id>.dump.part0000
brains/<canonical-brain-id>.dump.part0001  # only when needed

The registry JSON contains only explicit Brain identity fields and typed workspace metadata, preserving IDs and timestamps; legacy PAT fields are removed. It is restored with parameterized inserts, not arbitrary registry SQL. Brain payloads are PostgreSQL custom-format logical pg_dump archives split into 32 MiB Git files. No PGDATA, Redis, console/session/CSRF state, passwords, internal secrets, or access-token registry/hashes are exported. Known deployment secrets and Brain PATs accidentally pasted in content cause checkpoint rejection rather than upload. Access tokens must be reissued separately after an ephemeral restore. Existing auth state on a durable local database is left alone.

Set BUILD_SHA to the deployed Git commit (Docker build argument or pinned Space variable) for Git provenance; unpinned local builds record unknown.

The manifest records format/schema version, PostgreSQL major version, application version, source SHA, creation time, canonical Brain names, full embedding profile, native/stored dimensions, payload order, per-file SHA-256 checksums, and semantic content fingerprints. Archive timestamps never cause a redundant commit. Every content table is allowlisted; unexpected per-Brain tables stop backup for operator review rather than silently losing unmodelled content or copying credentials. Unregistered databases and orphan workspaces are not discovered or exported. Physical-name collisions among canonical names are rejected.

Startup and recovery

1. Start temporary PostgreSQL/Redis and initialize the system database/schema.
2. Fetch the configured Git branch using environment-only credentials through a token-free askpass program. Git errors are fixed codes, with captured output omitted. No token is written to a Git config, URL, archive, or exception.
3. Validate the complete manifest, payload checksums/layout, PostgreSQL major, and configured embedding identity. Validate each logical archive's TOC.
4. If registry is empty, reject destination database collisions, create only the registered Brain databases, restore archives transactionally per database and validate all vector profile rows/dimensions. Publish registry/workspaces last in one transaction. On failure drop only databases created by this restore.
5. Record the checkpoint lineage locally, then bootstrap workspaces, stop temporary services, and start Supervisor/API/MCP/workers.

An empty remote, or a README-only initialized repository with no snapshot in its history, permits first boot and later seeding from local content. A deleted manifest, backup payloads without a manifest, corrupt/incompatible snapshot, or missing credential fails closed. Startup still fetches and validates when local data is nonempty. A matching local receipt retains newer local changes without restoring an older snapshot. Unknown/nonmatching lineage fails with BACKUP_LOCAL_REMOTE_CONFLICT; deleted remote history also fails. It never merges unrelated snapshots or overwrites populated local databases.

For recovery, first stop API/MCP/workers and preserve the local databases and checkpoint receipt. Inspect local and remote commits. Restore into a fresh, isolated cluster using the normal entrypoint and verified configuration; reissue scoped tokens using SYSTEM_PAT. Do not delete legitimate databases, edit lineage receipts to bypass conflicts, force push, or use the old offline Redis/volume backup as a content checkpoint. If two writers diverge, choose/merge content in an isolated recovery environment, qualify it, and publish a reviewed successor.

Planned Hugging Face restarts

Before restarting/rebuilding an ephemeral Space, quiesce content writers and wait for a successful checkpoint containing their completed writes. Confirm the remote manifest/commit matches /health and validate the snapshot before replacement. HF can start a replacement before the old container finishes its final shutdown checkpoint; the replacement can therefore restore the preceding commit. A final shutdown attempt alone is not a maintenance checkpoint barrier.

After startup, compare the restored commit with the intended remote checkpoint and verify identities and retrieval before resuming writers. If a later shutdown commit appeared, preserve both histories and local state, stop writers, and follow the isolated recovery procedure above. Do not bypass lineage protection or force push. See live qualification evidence.

Runtime and limits

A PostgreSQL connection hook marks successful content mutations dirty across API, Celery, graph and vector clients. Auth-only writes and rolled-back transactions do not count. Every 300 seconds the daemon reconciles logical content as well, covering administrative SQL and missed dirty markers. This coalesces writes; no new commit is created when semantic contents/schema/profiles are unchanged. A file lock serializes attempts across processes. Each Brain's dump and fingerprint share a repeatable-read snapshot; registry/workspace changes during export reject the attempt. This is per-database consistency, not a distributed transaction across all Brains. Keep the operation root on the same durable mount as local PostgreSQL when such a mount exists.

Push compares the fetched head with the local receipt and uses a normal atomic, non-force Git push. Remote changes or a racing push cause a visible degraded state and retain local data; there is no unsafe rebase/retry. Retry occurs at the next interval only when the expected remote lineage remains unchanged. Health exposes safe checkpoint state/commit/time/error separately from local persistence.

On graceful shutdown Supervisor stops request/worker writers first, then gives the checkpoint daemon 120 seconds for its final attempt, then stops PostgreSQL. Forced termination can lose changes since the last successful checkpoint. This is an asynchronous backup with roughly interval + upload time RPO, not a substitute for a durable database or a zero-loss production guarantee. Existing explicit ALLOW_EPHEMERAL_STORAGE and degraded ephemeral health rules are preserved. Large snapshots/history can eventually exceed GitHub repository/storage limits; monitor growth and failed/stale checkpoint status. No automatic history rewrite.

Qualification

Run pytest tests/test_brain_backup.py for Git/manifest/startup safety gates. For a real restore round-trip, use only a disposable PostgreSQL 15 + pgvector cluster, configure the HF CPU profile and PostgreSQL credentials/tools, then:

BACKUP_QUALIFICATION_DISPOSABLE=true \
BACKUP_QUALIFICATION_RETRIEVAL=true \
python scripts/qualify_content_backup.py

It creates random test brains and unique facts, verifies dirty tracking/no-op commits, auth and orphan exclusions, all five profiles and mismatch rejection, drops only those test databases, restores their IDs/workspaces/facts, calls actual search/context controllers, and checks a new mutation produces a new commit. Retrieval qualification requires the preloaded real Ollama model and normal NLP runtime. It never pulls a model at application startup or deploys a Space.

Verified maintenance barrier and RPO readiness

SYSTEM_PAT administrators have these operations; scoped tokens cannot use them:

Operation	Effect
POST /system/backup/quiesce	Close new HTTP/MCP content admission; accepted requests/tasks may finish.
POST /system/backup/checkpoint?timeout_seconds=120	Quiesce, drain admitted requests and all tracked Celery deliveries/children/retries, seal, snapshot, push, fetch HEAD and validate manifest/checksums. Return exact durable SHA.
GET /system/backup/status	Read hold state, pending task count, durable SHA and safe error code.
POST /system/backup/resume	Verify/publish the hold release before reopening admission. Also cancels maintenance after accepted work settles.

A successful barrier returns state=durable, checkpoint=<sha>, dirty=false, remote_verified=true, quiesced=true. Writes stay held until explicit resume. A drain timeout, ambiguous task publication, Git failure or verification conflict keeps admission closed and reports degraded state. Accepted work settles at Celery SUCCESS/FAILURE terminal delivery, including all already-published descendants; failed ingestion does not become successful merely because a checkpoint exists. An active delivery is counted separately from its publication so broker redelivery cannot make the barrier finish while a duplicate execution is active. External administrative SQL or independently running writers must also be stopped; this barrier controls the application's HTTP/MCP and managed Celery writers.

The local control/accepted-work ledger contains only opaque IDs/states and is not backed up. A version-1 manifest optionally carries maintenance as an empty object or { "id": "<opaque-id>", "quiesced": true }. This non-secret handoff metadata makes a freshly restored replacement remain quiesced. Publishing/releasing that hold is a real control-state change and may create a commit with unchanged content. Normal unchanged reconciliation still creates no commit.

Planned restart sequence:

1. Quiesce writers and wait accepted work to settle using the checkpoint operation.
2. Record its remotely verified checkpoint SHA A; do not resume the old container.
3. Trigger the HF rebuild/restart.
4. Verify replacement /health: restored_checkpoint is A or a verified descendant, remote_checkpoint matches, checkpoint_lag=0, and no backup error is present.
5. Verify canonical identities and retrieval while the hold remains closed.
6. Explicitly resume through /system/backup/resume, then verify write admission.

Startup re-fetches HEAD after restoration, and again after workspace bootstrap before Supervisor/API startup. Any advance fails closed with BACKUP_STARTUP_REMOTE_ADVANCED; it never serves stale writable content or merges active writers. The normal periodic/shutdown daemon and maintenance operations share a file lock. A shutdown checkpoint is still best-effort and is not the maintenance barrier.

Once the hold is sealed, the periodic/shutdown daemon only verifies the pinned checkpoint; it does not publish a successor until explicit checkpoint/resume. Read paths can initialize derived indexes/profiles locally, but must not move the operator's handoff SHA through a late shutdown snapshot. Qualification accepts a descendant only with an operator-fetched Git checkout proving ancestry.

BRAIN_BACKUP_MAX_RPO_SECONDS defaults to BRAIN_BACKUP_INTERVAL_SECONDS (300). Health exposes restored/remote SHA, lag, dynamic dirty state, last successful checkpoint/reconciliation time and age. An unchanged semantic reconciliation refreshes the successful verification time without creating a commit. Git reachability, auth/database/schema/profile and baked-model availability, plus an actual Uvicorn formatter self-test, are checked; expensive probes are cached for at most 10 seconds. Production readiness is withdrawn on observed Git failure, lineage conflict, checkpoint age beyond the RPO, or an unhealthy dependency. checkpoint_lag is an equality indicator: 0 means matching HEAD, 1 means a different HEAD, and null means unknown; it is not a count of missing commits.

ALLOW_CHECKPOINTED_PRODUCTION=false is the default: ephemeral local storage remains production NO-GO. An operator may explicitly set it to true to accept bounded-loss GitHub checkpoint recovery. Readiness then also requires a qualified restore, current verified remote lineage, a successful checkpoint within the maximum age, and all dependency checks. Local storage is still reported ephemeral, content recovery github-checkpoint, zero_loss=false; this setting does not improve the underlying durability. With interval and maximum age both 300, long snapshots or uploads can temporarily exceed the policy and withdraw readiness; monitor this.

Unexpected container death can lose writes since the last successful checkpoint. Zero-loss operation would require a synchronous external durable write path. This implementation does not commit to Git synchronously for every Brain write.