Spaces:

Benaah
/

MicProtocol

Running

App Files Files Community

MicProtocol / docs /formbricks-runbook.md

Benaah

Upload folder using huggingface_hub

b17baae verified about 20 hours ago

preview code

Raw History Blame Contribute Delete

16.1 kB

Formbricks v4 Operational Runbook

Overview

Formbricks v4 uses a 5-container microservice architecture for self-hosted deployments. This runbook covers day-to-day operations, troubleshooting, and disaster recovery for the MicProtocol deployment.

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        Formbricks v4 Stack                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐           │
│  │  formbricks │◄───►│formbricks-hub│◄───►│formbricks-cube│          │
│  │   (web)     │     │   (API)      │     │ (Analytics)  │          │
│  │   :3000     │     │   :8080      │     │   :4000      │          │
│  └──────┬──────┘     └──────┬──────┘     └──────┬──────┘          │
│         │                   │                   │                  │
│         └───────────────────┼───────────────────┘                  │
│                             ▼                                       │
│                    ┌─────────────────┐                             │
│                    │  PostgreSQL     │                             │
│                    │  (formbricks DB)│                             │
│                    └─────────────────┘                             │
│                             │                                       │
│                             ▼                                       │
│                    ┌─────────────────┐                             │
│                    │     Redis       │                             │
│                    └─────────────────┘                             │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Container Details

Container	Image	Port	Purpose	Health Check
`micprotocol-formbricks-migrate`	`ghcr.io/formbricks/formbricks:latest`	-	DB migrations (one-shot)	N/A (completion)
`micprotocol-formbricks`	`ghcr.io/formbricks/formbricks:latest`	3001	Main web application	`/api/health`
`micprotocol-formbricks-hub-migrate`	`ghcr.io/formbricks/hub:latest`	-	Hub DB migrations (one-shot)	N/A (completion)
`micprotocol-formbricks-hub`	`ghcr.io/formbricks/hub:latest`	8080	Hub API (survey delivery)	`/health`
`micprotocol-formbricks-cube`	`cubejs/cube:v1.6.6`	4000	Analytics engine (Cube.js)	`/readyz`

Required Environment Variables

Critical Secrets (Generate Once, Never Rotate)

# Core Auth - NEVER ROTATE AFTER INITIAL SETUP
FORMBRICKS_NEXTAUTH_SECRET=$(openssl rand -base64 32)
FORMBRICKS_ENCRYPTION_KEY=$(openssl rand -hex 32)        # 64 chars
FORMBRICKS_HUB_API_KEY=$(openssl rand -hex 32)
FORMBRICKS_CUBE_API_SECRET=$(openssl rand -hex 32)

# Rotatable Secrets
FORMBRICKS_CRON_SECRET=$(openssl rand -base64 16)
FORMBRICKS_WEBHOOK_SECRET=$(openssl rand -hex 32)

Service Configuration

# URLs
FORMBRICKS_URL=https://formbricks.yourdomain.com
WEBAPP_URL=${FORMBRICKS_URL}
NEXTAUTH_URL=${FORMBRICKS_URL}

# Database
DATABASE_URL=postgresql://user:pass@postgres:5432/formbricks?schema=public

# Redis
REDIS_URL=redis://:password@redis:6379

# Inter-service Communication
HUB_API_URL=http://formbricks-hub:8080
HUB_API_KEY=${FORMBRICKS_HUB_API_KEY}
CUBEJS_API_URL=http://formbricks-cube:4000
CUBEJS_API_SECRET=${FORMBRICKS_CUBE_API_SECRET}
CUBEJS_JWT_ISSUER=formbricks-web
CUBEJS_JWT_AUDIENCE=formbricks-cube

# Email
MAIL_FROM=noreply@yourdomain.com
SMTP_HOST=smtp.yourprovider.com
SMTP_PORT=587
SMTP_USER=your_user
SMTP_PASSWORD=your_password

# Security (Production)
RATE_LIMITING_DISABLED=0
TELEMETRY_DISABLED=1
EMAIL_VERIFICATION_DISABLED=1
PASSWORD_RESET_DISABLED=1

Deployment Procedures

Initial Deployment

# 1. Generate all secrets
./scripts/generate-secrets.sh > .env.production.formbricks

# 2. Start PostgreSQL and Redis first
docker compose up -d postgres redis

# 3. Run database migrations (one-shot)
docker compose up formbricks-migrate
docker compose up formbricks-hub-migrate

# 4. Start Cube.js (analytics)
docker compose up -d formbricks-cube

# 5. Wait for Cube.js to be healthy
docker compose logs -f formbricks-cube | grep -q "Cube.js server is listening"

# 6. Start Hub API
docker compose up -d formbricks-hub

# 7. Start main web application
docker compose up -d formbricks

# 8. Verify all services
docker compose ps

Rolling Updates

# Update main web app (zero-downtime if replicas > 1)
docker compose pull formbricks
docker compose up -d --no-deps formbricks

# Update Hub API
docker compose pull formbricks-hub
docker compose up -d --no-deps formbricks-hub

# Update Cube.js (requires brief downtime)
docker compose pull formbricks-cube
docker compose up -d --no-deps formbricks-cube

Health Checks & Monitoring

Service Health Endpoints

Service	Endpoint	Expected Response
Web App	`GET http://localhost:3001/api/health`	`{ "status": "ok" }`
Hub API	`GET http://localhost:8080/health`	`{ "status": "healthy" }`
Cube.js	`GET http://localhost:4000/readyz`	`200 OK`

Docker Health Checks (Automatic)

# In docker-compose.yml
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
  interval: 30s
  timeout: 10s
  retries: 5
  start_period: 40s

Prometheus Metrics

Key metrics to alert on:

# Service up/down
up{job="formbricks-web"} == 0
up{job="formbricks-hub"} == 0
up{job="formbricks-cube"} == 0

# High error rate
rate(formbricks_webhook_errors_total[5m]) > 0.1

# Slow responses
histogram_quantile(0.95, rate(formbricks_http_request_duration_seconds_bucket[5m])) > 5

# Database connections
formbricks_db_connections_active / formbricks_db_connections_max > 0.8

Log Aggregation

# View all Formbricks logs
docker compose logs -f formbricks formbricks-hub formbricks-cube

# Filter for errors
docker compose logs formbricks 2>&1 | grep -i error

# Follow specific service
docker compose logs -f --tail=100 formbricks

Common Operations

Create New Environment (Per Workspace)

# 1. Access Formbricks Admin UI
# 2. Go to Settings → Environments → Create Environment
# 3. Copy Environment ID
# 4. Set in MicProtocol: NEXT_PUBLIC_FORMBRICKS_ENVIRONMENT_ID=<env-id>

Configure Survey Targeting

# Via API (for automation)
curl -X POST https://formbricks.yourdomain.com/api/v1/surveys \
  -H "Authorization: Bearer ${FORMBRICKS_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Post-Recording NPS",
    "type": "web",
    "questions": [...],
    "targeting": {
      "attributes": {
        "workspace_id": "eq:ws_123",
        "plan": "in:pro,enterprise"
      }
    }
  }'

Update Webhook URLs (After Domain Change)

# In Formbricks Admin: Settings → Webhooks
# Update all webhook URLs to new domain
# Or via API:
curl -X PUT https://formbricks.yourdomain.com/api/v1/webhooks/${WEBHOOK_ID} \
  -H "Authorization: Bearer ${FORMBRICKS_API_KEY}" \
  -d '{"url": "https://api.yourdomain.com/integrations/webhooks/formbricks"}'

Backup Database

# Full backup
docker exec micprotocol-postgres pg_dump -U micprotocol formbricks > formbricks_backup_$(date +%Y%m%d).sql

# Compressed backup
docker exec micprotocol-postgres pg_dump -U micprotocol formbricks | gzip > formbricks_backup_$(date +%Y%m%d).sql.gz

# Restore
gunzip -c formbricks_backup_20240115.sql.gz | docker exec -i micprotocol-postgres psql -U micprotocol formbricks

Migrate to New PostgreSQL Instance

# 1. Dump current database
docker exec micprotocol-postgres pg_dump -U micprotocol formbricks > dump.sql

# 2. Create new database on new instance
createdb -h new-postgres -U micprotocol formbricks

# 3. Run migrations on new instance
docker run --rm \
  -e DATABASE_URL="postgresql://user:pass@new-postgres:5432/formbricks" \
  ghcr.io/formbricks/formbricks:latest \
  node packages/database/dist/scripts/apply-migrations.js

# 4. Restore data
psql -h new-postgres -U micprotocol formbricks < dump.sql

# 5. Update docker-compose.yml DATABASE_URL and restart
docker compose up -d formbricks-migrate formbricks-hub-migrate
docker compose up -d formbricks formbricks-hub formbricks-cube

Troubleshooting Guide

Issue: Web App Returns 500 / Won't Start

Symptoms: Container restarts, logs show database connection errors

Diagnosis:

# Check database connectivity
docker exec micprotocol-formbricks pg_isready -h postgres -U micprotocol

# Check migration status
docker compose logs formbricks-migrate

# Check environment variables
docker exec micprotocol-formbricks env | grep -E "DATABASE|NEXTAUTH|ENCRYPTION"

Resolution:

Ensure formbricks-migrate completed successfully
Verify DATABASE_URL includes ?schema=public
Check NEXTAUTH_SECRET and ENCRYPTION_KEY are set
Restart: docker compose restart formbricks

Issue: Hub API Unhealthy

Symptoms: Surveys don't load, widget shows "Failed to load survey"

Diagnosis:

# Check Hub logs
docker compose logs formbricks-hub

# Test Hub health
curl http://localhost:8080/health

# Check Hub database
docker exec micprotocol-formbricks-hub pg_isready -h postgres -U micprotocol

Resolution:

Ensure formbricks-hub-migrate completed
Verify HUB_API_KEY matches between web and hub
Check Redis connectivity
Restart: docker compose restart formbricks-hub

Issue: Cube.js Analytics Not Working

Symptoms: Dashboard shows no data, "Failed to load" errors

Diagnosis:

# Check Cube.js logs
docker compose logs formbricks-cube

# Test Cube.js API
curl -H "Authorization: ${CUBEJS_API_SECRET}" http://localhost:4000/readyz

# Check Cube.js can reach PostgreSQL
docker exec micprotocol-formbricks-cube pg_isready -h postgres -U micprotocol

Resolution:

Verify CUBEJS_API_SECRET matches between web and cube
Check CUBEJS_DB_* environment variables
Ensure CUBEJS_JWT_ISSUER and CUBEJS_JWT_AUDIENCE match
Restart: docker compose restart formbricks-cube

Issue: Webhooks Failing (401 Unauthorized)

Symptoms: MicProtocol API returns 401 for Formbricks webhooks

Diagnosis:

# Check webhook secret matches
grep FORMBRICKS_WEBHOOK_SECRET .env
# Compare with Formbricks Admin → Settings → Webhooks → Secret

Resolution:

Ensure FORMBRICKS_WEBHOOK_SECRET in .env matches Formbricks UI
Regenerate if needed: openssl rand -hex 32
Update both .env and Formbricks Admin
Restart API: docker compose restart api

Issue: Rate Limiting Blocking Legitimate Requests

Symptoms: 429 responses, surveys not loading for some users

Diagnosis:

# Check rate limit config
docker exec micprotocol-formbricks env | grep RATE_LIMITING

# Check Redis for rate limit keys
docker exec micprotocol-redis redis-cli KEYS "ratelimit:*"

Resolution:

Ensure RATE_LIMITING_DISABLED=0 in production
Tune limits in Formbricks Admin → Settings → Rate Limiting
For high-traffic: increase Redis capacity or use Redis Cluster

Issue: Email Notifications Not Sending

Symptoms: No survey invitation emails, password reset emails fail

Diagnosis:

# Check SMTP config
docker exec micprotocol-formbricks env | grep -i SMTP

# Check mailhog (dev) or SMTP logs
docker compose logs mailhog

Resolution:

Verify SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD
Test with: docker exec micprotocol-formbricks nc -zv smtp.example.com 587
Check MAIL_FROM is verified in your SMTP provider

Disaster Recovery

Scenario: Complete Data Loss (PostgreSQL Corruption)

RTO: 2 hours | RPO: 24 hours (daily backups)

Provision new PostgreSQL instance

Restore latest backup:

gunzip -c formbricks_backup_latest.sql.gz | psql -h new-pg -U micprotocol formbricks

Run migrations to ensure schema is current:

docker compose up formbricks-migrate formbricks-hub-migrate

Restart all services:

docker compose up -d formbricks-cube formbricks-hub formbricks

Verify data integrity via Admin UI

Scenario: Formbricks Hub Database Corruption

RTO: 30 minutes | RPO: 1 hour

Hub uses same PostgreSQL database (formbricks schema)
Restore from backup (same as above)

Re-run hub migrations:

docker compose up formbricks-hub-migrate
docker compose restart formbricks-hub

Scenario: Lost Encryption Keys (NEXTAUTH_SECRET, ENCRYPTION_KEY)

IMPACT: All existing sessions invalidated, encrypted data unrecoverable

Mitigation:

These keys are never rotated after initial setup
Store in secure password manager (1Password, Bitwarden)
Document in runbook with access restricted to platform team

Recovery:

Generate new keys
Update .env and restart all services
All users must re-authenticate
Historical encrypted data (webhook payloads, etc.) will be unreadable

Security Checklist

Monthly

Rotate FORMBRICKS_CRON_SECRET and FORMBRICKS_WEBHOOK_SECRET
Review webhook delivery logs for anomalies
Check for failed login attempts in logs
Verify SSL certificates are valid (> 30 days)

Quarterly

Update all container images to latest patches
Review and prune old survey responses (GDPR)
Audit API keys in Formbricks Admin
Test backup restoration procedure

Annually

Rotate SMTP credentials
Review Redis AUTH password
Penetration test webhook endpoints
Update Cube.js and Hub to latest major versions

Performance Tuning

Database

-- Add indexes for common query patterns
CREATE INDEX idx_surveys_environment_status ON surveys(environment_id, status);
CREATE INDEX idx_responses_survey_created ON responses(survey_id, created_at);
CREATE INDEX idx_attributes_user_key ON user_attributes(user_id, key);

Redis

# Increase maxmemory for rate limiting
# In redis.conf or docker-compose:
command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru

Cube.js

# In docker-compose.yml for formbricks-cube
environment:
  CUBEJS_CACHE_AND_QUEUE_DRIVER: redis
  CUBEJS_REDIS_URL: redis://:password@redis:6379
  CUBEJS_DB_MAX_POOL: 20

Contact & Escalation

Issue Severity	Response Time	Contact
P0 (All surveys down)	15 min	Platform On-Call
P1 (Single workspace affected)	1 hour	Platform Team
P2 (Analytics delayed)	4 hours	Platform Team
P3 (Feature request)	Next sprint	Product Team

Runbook Owner: Platform Engineering
Last Updated: 2024-01-15
Next Review: 2024-04-15