Spaces:
Running
Running
| # Formbricks v4 Operational Runbook | |
| ## Overview | |
| Formbricks v4 uses a **5-container microservice architecture** for self-hosted deployments. This runbook covers day-to-day operations, troubleshooting, and disaster recovery for the MicProtocol deployment. | |
| ## Architecture | |
| ``` | |
| ┌─────────────────────────────────────────────────────────────────────┐ | |
| │ Formbricks v4 Stack │ | |
| ├─────────────────────────────────────────────────────────────────────┤ | |
| │ │ | |
| │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ | |
| │ │ formbricks │◄───►│formbricks-hub│◄───►│formbricks-cube│ │ | |
| │ │ (web) │ │ (API) │ │ (Analytics) │ │ | |
| │ │ :3000 │ │ :8080 │ │ :4000 │ │ | |
| │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ | |
| │ │ │ │ │ | |
| │ └───────────────────┼───────────────────┘ │ | |
| │ ▼ │ | |
| │ ┌─────────────────┐ │ | |
| │ │ PostgreSQL │ │ | |
| │ │ (formbricks DB)│ │ | |
| │ └─────────────────┘ │ | |
| │ │ │ | |
| │ ▼ │ | |
| │ ┌─────────────────┐ │ | |
| │ │ Redis │ │ | |
| │ └─────────────────┘ │ | |
| │ │ | |
| └─────────────────────────────────────────────────────────────────────┘ | |
| ``` | |
| ### Container Details | |
| | Container | Image | Port | Purpose | Health Check | | |
| |-----------|-------|------|---------|--------------| | |
| | `micprotocol-formbricks-migrate` | `ghcr.io/formbricks/formbricks:latest` | - | DB migrations (one-shot) | N/A (completion) | | |
| | `micprotocol-formbricks` | `ghcr.io/formbricks/formbricks:latest` | 3001 | Main web application | `/api/health` | | |
| | `micprotocol-formbricks-hub-migrate` | `ghcr.io/formbricks/hub:latest` | - | Hub DB migrations (one-shot) | N/A (completion) | | |
| | `micprotocol-formbricks-hub` | `ghcr.io/formbricks/hub:latest` | 8080 | Hub API (survey delivery) | `/health` | | |
| | `micprotocol-formbricks-cube` | `cubejs/cube:v1.6.6` | 4000 | Analytics engine (Cube.js) | `/readyz` | | |
| --- | |
| ## Required Environment Variables | |
| ### Critical Secrets (Generate Once, Never Rotate) | |
| ```bash | |
| # Core Auth - NEVER ROTATE AFTER INITIAL SETUP | |
| FORMBRICKS_NEXTAUTH_SECRET=$(openssl rand -base64 32) | |
| FORMBRICKS_ENCRYPTION_KEY=$(openssl rand -hex 32) # 64 chars | |
| FORMBRICKS_HUB_API_KEY=$(openssl rand -hex 32) | |
| FORMBRICKS_CUBE_API_SECRET=$(openssl rand -hex 32) | |
| # Rotatable Secrets | |
| FORMBRICKS_CRON_SECRET=$(openssl rand -base64 16) | |
| FORMBRICKS_WEBHOOK_SECRET=$(openssl rand -hex 32) | |
| ``` | |
| ### Service Configuration | |
| ```bash | |
| # URLs | |
| FORMBRICKS_URL=https://formbricks.yourdomain.com | |
| WEBAPP_URL=${FORMBRICKS_URL} | |
| NEXTAUTH_URL=${FORMBRICKS_URL} | |
| # Database | |
| DATABASE_URL=postgresql://user:pass@postgres:5432/formbricks?schema=public | |
| # Redis | |
| REDIS_URL=redis://:password@redis:6379 | |
| # Inter-service Communication | |
| HUB_API_URL=http://formbricks-hub:8080 | |
| HUB_API_KEY=${FORMBRICKS_HUB_API_KEY} | |
| CUBEJS_API_URL=http://formbricks-cube:4000 | |
| CUBEJS_API_SECRET=${FORMBRICKS_CUBE_API_SECRET} | |
| CUBEJS_JWT_ISSUER=formbricks-web | |
| CUBEJS_JWT_AUDIENCE=formbricks-cube | |
| MAIL_FROM=noreply@yourdomain.com | |
| SMTP_HOST=smtp.yourprovider.com | |
| SMTP_PORT=587 | |
| SMTP_USER=your_user | |
| SMTP_PASSWORD=your_password | |
| # Security (Production) | |
| RATE_LIMITING_DISABLED=0 | |
| TELEMETRY_DISABLED=1 | |
| EMAIL_VERIFICATION_DISABLED=1 | |
| PASSWORD_RESET_DISABLED=1 | |
| ``` | |
| --- | |
| ## Deployment Procedures | |
| ### Initial Deployment | |
| ```bash | |
| # 1. Generate all secrets | |
| ./scripts/generate-secrets.sh > .env.production.formbricks | |
| # 2. Start PostgreSQL and Redis first | |
| docker compose up -d postgres redis | |
| # 3. Run database migrations (one-shot) | |
| docker compose up formbricks-migrate | |
| docker compose up formbricks-hub-migrate | |
| # 4. Start Cube.js (analytics) | |
| docker compose up -d formbricks-cube | |
| # 5. Wait for Cube.js to be healthy | |
| docker compose logs -f formbricks-cube | grep -q "Cube.js server is listening" | |
| # 6. Start Hub API | |
| docker compose up -d formbricks-hub | |
| # 7. Start main web application | |
| docker compose up -d formbricks | |
| # 8. Verify all services | |
| docker compose ps | |
| ``` | |
| ### Rolling Updates | |
| ```bash | |
| # Update main web app (zero-downtime if replicas > 1) | |
| docker compose pull formbricks | |
| docker compose up -d --no-deps formbricks | |
| # Update Hub API | |
| docker compose pull formbricks-hub | |
| docker compose up -d --no-deps formbricks-hub | |
| # Update Cube.js (requires brief downtime) | |
| docker compose pull formbricks-cube | |
| docker compose up -d --no-deps formbricks-cube | |
| ``` | |
| --- | |
| ## Health Checks & Monitoring | |
| ### Service Health Endpoints | |
| | Service | Endpoint | Expected Response | | |
| |---------|----------|-------------------| | |
| | Web App | `GET http://localhost:3001/api/health` | `{ "status": "ok" }` | | |
| | Hub API | `GET http://localhost:8080/health` | `{ "status": "healthy" }` | | |
| | Cube.js | `GET http://localhost:4000/readyz` | `200 OK` | | |
| ### Docker Health Checks (Automatic) | |
| ```yaml | |
| # In docker-compose.yml | |
| healthcheck: | |
| test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"] | |
| interval: 30s | |
| timeout: 10s | |
| retries: 5 | |
| start_period: 40s | |
| ``` | |
| ### Prometheus Metrics | |
| Key metrics to alert on: | |
| ```promql | |
| # Service up/down | |
| up{job="formbricks-web"} == 0 | |
| up{job="formbricks-hub"} == 0 | |
| up{job="formbricks-cube"} == 0 | |
| # High error rate | |
| rate(formbricks_webhook_errors_total[5m]) > 0.1 | |
| # Slow responses | |
| histogram_quantile(0.95, rate(formbricks_http_request_duration_seconds_bucket[5m])) > 5 | |
| # Database connections | |
| formbricks_db_connections_active / formbricks_db_connections_max > 0.8 | |
| ``` | |
| ### Log Aggregation | |
| ```bash | |
| # View all Formbricks logs | |
| docker compose logs -f formbricks formbricks-hub formbricks-cube | |
| # Filter for errors | |
| docker compose logs formbricks 2>&1 | grep -i error | |
| # Follow specific service | |
| docker compose logs -f --tail=100 formbricks | |
| ``` | |
| --- | |
| ## Common Operations | |
| ### Create New Environment (Per Workspace) | |
| ```bash | |
| # 1. Access Formbricks Admin UI | |
| # 2. Go to Settings → Environments → Create Environment | |
| # 3. Copy Environment ID | |
| # 4. Set in MicProtocol: NEXT_PUBLIC_FORMBRICKS_ENVIRONMENT_ID=<env-id> | |
| ``` | |
| ### Configure Survey Targeting | |
| ```bash | |
| # Via API (for automation) | |
| curl -X POST https://formbricks.yourdomain.com/api/v1/surveys \ | |
| -H "Authorization: Bearer ${FORMBRICKS_API_KEY}" \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "name": "Post-Recording NPS", | |
| "type": "web", | |
| "questions": [...], | |
| "targeting": { | |
| "attributes": { | |
| "workspace_id": "eq:ws_123", | |
| "plan": "in:pro,enterprise" | |
| } | |
| } | |
| }' | |
| ``` | |
| ### Update Webhook URLs (After Domain Change) | |
| ```bash | |
| # In Formbricks Admin: Settings → Webhooks | |
| # Update all webhook URLs to new domain | |
| # Or via API: | |
| curl -X PUT https://formbricks.yourdomain.com/api/v1/webhooks/${WEBHOOK_ID} \ | |
| -H "Authorization: Bearer ${FORMBRICKS_API_KEY}" \ | |
| -d '{"url": "https://api.yourdomain.com/integrations/webhooks/formbricks"}' | |
| ``` | |
| ### Backup Database | |
| ```bash | |
| # Full backup | |
| docker exec micprotocol-postgres pg_dump -U micprotocol formbricks > formbricks_backup_$(date +%Y%m%d).sql | |
| # Compressed backup | |
| docker exec micprotocol-postgres pg_dump -U micprotocol formbricks | gzip > formbricks_backup_$(date +%Y%m%d).sql.gz | |
| # Restore | |
| gunzip -c formbricks_backup_20240115.sql.gz | docker exec -i micprotocol-postgres psql -U micprotocol formbricks | |
| ``` | |
| ### Migrate to New PostgreSQL Instance | |
| ```bash | |
| # 1. Dump current database | |
| docker exec micprotocol-postgres pg_dump -U micprotocol formbricks > dump.sql | |
| # 2. Create new database on new instance | |
| createdb -h new-postgres -U micprotocol formbricks | |
| # 3. Run migrations on new instance | |
| docker run --rm \ | |
| -e DATABASE_URL="postgresql://user:pass@new-postgres:5432/formbricks" \ | |
| ghcr.io/formbricks/formbricks:latest \ | |
| node packages/database/dist/scripts/apply-migrations.js | |
| # 4. Restore data | |
| psql -h new-postgres -U micprotocol formbricks < dump.sql | |
| # 5. Update docker-compose.yml DATABASE_URL and restart | |
| docker compose up -d formbricks-migrate formbricks-hub-migrate | |
| docker compose up -d formbricks formbricks-hub formbricks-cube | |
| ``` | |
| --- | |
| ## Troubleshooting Guide | |
| ### Issue: Web App Returns 500 / Won't Start | |
| **Symptoms:** Container restarts, logs show database connection errors | |
| **Diagnosis:** | |
| ```bash | |
| # Check database connectivity | |
| docker exec micprotocol-formbricks pg_isready -h postgres -U micprotocol | |
| # Check migration status | |
| docker compose logs formbricks-migrate | |
| # Check environment variables | |
| docker exec micprotocol-formbricks env | grep -E "DATABASE|NEXTAUTH|ENCRYPTION" | |
| ``` | |
| **Resolution:** | |
| 1. Ensure `formbricks-migrate` completed successfully | |
| 2. Verify `DATABASE_URL` includes `?schema=public` | |
| 3. Check `NEXTAUTH_SECRET` and `ENCRYPTION_KEY` are set | |
| 4. Restart: `docker compose restart formbricks` | |
| ### Issue: Hub API Unhealthy | |
| **Symptoms:** Surveys don't load, widget shows "Failed to load survey" | |
| **Diagnosis:** | |
| ```bash | |
| # Check Hub logs | |
| docker compose logs formbricks-hub | |
| # Test Hub health | |
| curl http://localhost:8080/health | |
| # Check Hub database | |
| docker exec micprotocol-formbricks-hub pg_isready -h postgres -U micprotocol | |
| ``` | |
| **Resolution:** | |
| 1. Ensure `formbricks-hub-migrate` completed | |
| 2. Verify `HUB_API_KEY` matches between web and hub | |
| 3. Check Redis connectivity | |
| 4. Restart: `docker compose restart formbricks-hub` | |
| ### Issue: Cube.js Analytics Not Working | |
| **Symptoms:** Dashboard shows no data, "Failed to load" errors | |
| **Diagnosis:** | |
| ```bash | |
| # Check Cube.js logs | |
| docker compose logs formbricks-cube | |
| # Test Cube.js API | |
| curl -H "Authorization: ${CUBEJS_API_SECRET}" http://localhost:4000/readyz | |
| # Check Cube.js can reach PostgreSQL | |
| docker exec micprotocol-formbricks-cube pg_isready -h postgres -U micprotocol | |
| ``` | |
| **Resolution:** | |
| 1. Verify `CUBEJS_API_SECRET` matches between web and cube | |
| 2. Check `CUBEJS_DB_*` environment variables | |
| 3. Ensure `CUBEJS_JWT_ISSUER` and `CUBEJS_JWT_AUDIENCE` match | |
| 4. Restart: `docker compose restart formbricks-cube` | |
| ### Issue: Webhooks Failing (401 Unauthorized) | |
| **Symptoms:** MicProtocol API returns 401 for Formbricks webhooks | |
| **Diagnosis:** | |
| ```bash | |
| # Check webhook secret matches | |
| grep FORMBRICKS_WEBHOOK_SECRET .env | |
| # Compare with Formbricks Admin → Settings → Webhooks → Secret | |
| ``` | |
| **Resolution:** | |
| 1. Ensure `FORMBRICKS_WEBHOOK_SECRET` in `.env` matches Formbricks UI | |
| 2. Regenerate if needed: `openssl rand -hex 32` | |
| 3. Update both `.env` and Formbricks Admin | |
| 4. Restart API: `docker compose restart api` | |
| ### Issue: Rate Limiting Blocking Legitimate Requests | |
| **Symptoms:** 429 responses, surveys not loading for some users | |
| **Diagnosis:** | |
| ```bash | |
| # Check rate limit config | |
| docker exec micprotocol-formbricks env | grep RATE_LIMITING | |
| # Check Redis for rate limit keys | |
| docker exec micprotocol-redis redis-cli KEYS "ratelimit:*" | |
| ``` | |
| **Resolution:** | |
| 1. Ensure `RATE_LIMITING_DISABLED=0` in production | |
| 2. Tune limits in Formbricks Admin → Settings → Rate Limiting | |
| 3. For high-traffic: increase Redis capacity or use Redis Cluster | |
| ### Issue: Email Notifications Not Sending | |
| **Symptoms:** No survey invitation emails, password reset emails fail | |
| **Diagnosis:** | |
| ```bash | |
| # Check SMTP config | |
| docker exec micprotocol-formbricks env | grep -i SMTP | |
| # Check mailhog (dev) or SMTP logs | |
| docker compose logs mailhog | |
| ``` | |
| **Resolution:** | |
| 1. Verify `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD` | |
| 2. Test with: `docker exec micprotocol-formbricks nc -zv smtp.example.com 587` | |
| 3. Check `MAIL_FROM` is verified in your SMTP provider | |
| --- | |
| ## Disaster Recovery | |
| ### Scenario: Complete Data Loss (PostgreSQL Corruption) | |
| **RTO: 2 hours | RPO: 24 hours (daily backups)** | |
| 1. Provision new PostgreSQL instance | |
| 2. Restore latest backup: | |
| ```bash | |
| gunzip -c formbricks_backup_latest.sql.gz | psql -h new-pg -U micprotocol formbricks | |
| ``` | |
| 3. Run migrations to ensure schema is current: | |
| ```bash | |
| docker compose up formbricks-migrate formbricks-hub-migrate | |
| ``` | |
| 4. Restart all services: | |
| ```bash | |
| docker compose up -d formbricks-cube formbricks-hub formbricks | |
| ``` | |
| 5. Verify data integrity via Admin UI | |
| ### Scenario: Formbricks Hub Database Corruption | |
| **RTO: 30 minutes | RPO: 1 hour** | |
| 1. Hub uses same PostgreSQL database (`formbricks` schema) | |
| 2. Restore from backup (same as above) | |
| 3. Re-run hub migrations: | |
| ```bash | |
| docker compose up formbricks-hub-migrate | |
| docker compose restart formbricks-hub | |
| ``` | |
| ### Scenario: Lost Encryption Keys (NEXTAUTH_SECRET, ENCRYPTION_KEY) | |
| **IMPACT: All existing sessions invalidated, encrypted data unrecoverable** | |
| **Mitigation:** | |
| 1. These keys are **never rotated** after initial setup | |
| 2. Store in secure password manager (1Password, Bitwarden) | |
| 3. Document in runbook with access restricted to platform team | |
| **Recovery:** | |
| 1. Generate new keys | |
| 2. Update `.env` and restart all services | |
| 3. All users must re-authenticate | |
| 4. Historical encrypted data (webhook payloads, etc.) will be unreadable | |
| --- | |
| ## Security Checklist | |
| ### Monthly | |
| - [ ] Rotate `FORMBRICKS_CRON_SECRET` and `FORMBRICKS_WEBHOOK_SECRET` | |
| - [ ] Review webhook delivery logs for anomalies | |
| - [ ] Check for failed login attempts in logs | |
| - [ ] Verify SSL certificates are valid (> 30 days) | |
| ### Quarterly | |
| - [ ] Update all container images to latest patches | |
| - [ ] Review and prune old survey responses (GDPR) | |
| - [ ] Audit API keys in Formbricks Admin | |
| - [ ] Test backup restoration procedure | |
| ### Annually | |
| - [ ] Rotate SMTP credentials | |
| - [ ] Review Redis AUTH password | |
| - [ ] Penetration test webhook endpoints | |
| - [ ] Update Cube.js and Hub to latest major versions | |
| --- | |
| ## Performance Tuning | |
| ### Database | |
| ```sql | |
| -- Add indexes for common query patterns | |
| CREATE INDEX idx_surveys_environment_status ON surveys(environment_id, status); | |
| CREATE INDEX idx_responses_survey_created ON responses(survey_id, created_at); | |
| CREATE INDEX idx_attributes_user_key ON user_attributes(user_id, key); | |
| ``` | |
| ### Redis | |
| ```bash | |
| # Increase maxmemory for rate limiting | |
| # In redis.conf or docker-compose: | |
| command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru | |
| ``` | |
| ### Cube.js | |
| ```yaml | |
| # In docker-compose.yml for formbricks-cube | |
| environment: | |
| CUBEJS_CACHE_AND_QUEUE_DRIVER: redis | |
| CUBEJS_REDIS_URL: redis://:password@redis:6379 | |
| CUBEJS_DB_MAX_POOL: 20 | |
| ``` | |
| --- | |
| ## Contact & Escalation | |
| | Issue Severity | Response Time | Contact | | |
| |----------------|---------------|---------| | |
| | P0 (All surveys down) | 15 min | Platform On-Call | | |
| | P1 (Single workspace affected) | 1 hour | Platform Team | | |
| | P2 (Analytics delayed) | 4 hours | Platform Team | | |
| | P3 (Feature request) | Next sprint | Product Team | | |
| **Runbook Owner:** Platform Engineering | |
| **Last Updated:** 2024-01-15 | |
| **Next Review:** 2024-04-15 |