|
Download scripts/ops/README.md from cryptorugmunch/rugmuncher-backend: direct link, hf CLI and curl.
- Browser
- Download file 2.69 kB
-
https://huggingface.co/cryptorugmunch/rugmuncher-backend/resolve/main/scripts/ops/README.md
- Command line
-
hf download hf://cryptorugmunch/rugmuncher-backend/scripts/ops/README.md
-
curl -L -o README.md https://huggingface.co/cryptorugmunch/rugmuncher-backend/resolve/main/scripts/ops/README.md
2.69 kB
| # ops/ β Server-side operations scripts | |
| Scripts that run on **netcup** (not in the FastAPI container). Deploy via: | |
| ```bash | |
| scp scripts/ops/backup.sh root@netcup:/root/scripts/backup.sh | |
| scp scripts/ops/restore_test.sh root@netcup:/root/scripts/restore_test.sh | |
| ssh netcup "chmod +x /root/scripts/*.sh" | |
| ``` | |
| ## backup.sh | |
| Daily backup of all RMI data + system configs to `/backups/<timestamp>/`. | |
| **What it backs up:** | |
| 1. Postgres (`postgres.sql.gz`) | |
| 2. Neo4j (`neo4j.dump`) | |
| 3. Qdrant (`qdrant/<collection>/<snapshot>`) | |
| 4. MinIO (`minio/`) | |
| 5. GlitchTip Postgres (`glitchtip.sql.gz`) | |
| 6. **System configs** (`etc_root.tar.gz`) β DR-critical, ~19 MB, 2440 files: | |
| - `/etc/prometheus/` β alert rules, prometheus.yml, alertmanager configs | |
| - `/root/scripts/` β this backup script, restore_test, cron health check | |
| - `/root/.hermes/` β Hermes gateway config (NOT sessions β too large) | |
| - `/root/.ssh/` β SSH keys for git/auth | |
| - `/root/.bashrc`, `/root/.profile` β shell config | |
| - `/root/backend/.env` β backend secrets (gopass-backed in source) | |
| 7. Old backups rotated (>7 days deleted) | |
| 8. ntfy notification on success | |
| **Cadence:** Runs via cron (daily, see `crontab -l` on netcup). | |
| **Restore test:** `/root/scripts/restore_test.sh` spins up test Postgres on alt port 15432, restores, verifies table count β₯ 10, samples row counts, now also validates `etc_root.tar.gz` contains `etc/prometheus/prometheus.yml`. | |
| ## restore_test.sh | |
| Monthly restore-test cron (1st of month) that proves a backup can actually be restored, not just written. | |
| **What it checks:** | |
| 1. Postgres table count β₯ 10 (fail if too few) | |
| 2. Row counts on `tokens`, `wallets`, `news_items` tables | |
| 3. `etc_root.tar.gz` contains `etc/prometheus/prometheus.yml` | |
| 4. `etc_root.tar.gz` file count > 1000 (catches truncated backups) | |
| 5. Notifies via ntfy topic `rmi-critical` on failure, `rmi-info` on success | |
| ## Cron health check (cron_health_check.py) | |
| Python watchdog that monitors cron job health and auto-restarts failed crons. | |
| ## T10 (RMIV5) β Backup /etc + /root | |
| Implemented in `backup.sh` step 6 (etc_root tarball). Verified 2026-06-22: | |
| - Tarball: 19 MB, 2440 files | |
| - Validates `etc/prometheus/prometheus.yml` exists | |
| - Excludes: `.cache`, `.ollama`, `.hermes/sessions`, `.cargo`, `.rustup` | |
| - Restore test validates tarball contents before declaring success | |
| **Bug found and fixed 2026-06-22:** | |
| Original script tried to back up `/etc/caddy/Caddyfile` which does not exist on | |
| netcup (we use nginx, not caddy). The validation silently passed because grep | |
| returned nothing. Fixed to validate `/etc/prometheus/prometheus.yml` instead, | |
| which actually exists and is the file we need for DR. |