Blame for docs/formbricks-runbook.md

# Formbricks v4 Operational Runbook

## Overview

Formbricks v4 uses a **5-container microservice architecture** for self-hosted deployments. This runbook covers day-to-day operations, troubleshooting, and disaster recovery for the MicProtocol deployment.

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Formbricks v4 Stack                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐           │
│  │  formbricks │◄───►│formbricks-hub│◄───►│formbricks-cube│          │
│  │   (web)     │     │   (API)      │     │ (Analytics)  │          │
│  │   :3000     │     │   :8080      │     │   :4000      │          │
│  └──────┬──────┘     └──────┬──────┘     └──────┬──────┘          │
│         │                   │                   │                  │
│         └───────────────────┼───────────────────┘                  │
│                             ▼                                       │
│                    ┌─────────────────┐                             │
│                    │  PostgreSQL     │                             │
│                    │  (formbricks DB)│                             │
│                    └─────────────────┘                             │
│                             │                                       │
│                             ▼                                       │
│                    ┌─────────────────┐                             │
│                    │     Redis       │                             │
│                    └─────────────────┘                             │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### Container Details

| Container | Image | Port | Purpose | Health Check |
|-----------|-------|------|---------|--------------|
| `micprotocol-formbricks-migrate` | `ghcr.io/formbricks/formbricks:latest` | - | DB migrations (one-shot) | N/A (completion) |
| `micprotocol-formbricks` | `ghcr.io/formbricks/formbricks:latest` | 3001 | Main web application | `/api/health` |
| `micprotocol-formbricks-hub-migrate` | `ghcr.io/formbricks/hub:latest` | - | Hub DB migrations (one-shot) | N/A (completion) |
| `micprotocol-formbricks-hub` | `ghcr.io/formbricks/hub:latest` | 8080 | Hub API (survey delivery) | `/health` |
| `micprotocol-formbricks-cube` | `cubejs/cube:v1.6.6` | 4000 | Analytics engine (Cube.js) | `/readyz` |

---

## Required Environment Variables

### Critical Secrets (Generate Once, Never Rotate)

```bash
# Core Auth - NEVER ROTATE AFTER INITIAL SETUP
FORMBRICKS_NEXTAUTH_SECRET=$(openssl rand -base64 32)
FORMBRICKS_ENCRYPTION_KEY=$(openssl rand -hex 32)        # 64 chars
FORMBRICKS_HUB_API_KEY=$(openssl rand -hex 32)
FORMBRICKS_CUBE_API_SECRET=$(openssl rand -hex 32)

# Rotatable Secrets
FORMBRICKS_CRON_SECRET=$(openssl rand -base64 16)
FORMBRICKS_WEBHOOK_SECRET=$(openssl rand -hex 32)
```

### Service Configuration

```bash
# URLs
FORMBRICKS_URL=https://formbricks.yourdomain.com
WEBAPP_URL=${FORMBRICKS_URL}
NEXTAUTH_URL=${FORMBRICKS_URL}

# Database
DATABASE_URL=postgresql://user:pass@postgres:5432/formbricks?schema=public

# Redis
REDIS_URL=redis://:password@redis:6379

# Inter-service Communication
HUB_API_URL=http://formbricks-hub:8080
HUB_API_KEY=${FORMBRICKS_HUB_API_KEY}
CUBEJS_API_URL=http://formbricks-cube:4000
CUBEJS_API_SECRET=${FORMBRICKS_CUBE_API_SECRET}
CUBEJS_JWT_ISSUER=formbricks-web
CUBEJS_JWT_AUDIENCE=formbricks-cube

# Email
MAIL_FROM=noreply@yourdomain.com
SMTP_HOST=smtp.yourprovider.com
SMTP_PORT=587
SMTP_USER=your_user
SMTP_PASSWORD=your_password

# Security (Production)
RATE_LIMITING_DISABLED=0
TELEMETRY_DISABLED=1
EMAIL_VERIFICATION_DISABLED=1
PASSWORD_RESET_DISABLED=1
```

---

## Deployment Procedures

### Initial Deployment

```bash
# 1. Generate all secrets
./scripts/generate-secrets.sh > .env.production.formbricks

# 2. Start PostgreSQL and Redis first
docker compose up -d postgres redis

# 3. Run database migrations (one-shot)
docker compose up formbricks-migrate
docker compose up formbricks-hub-migrate

# 4. Start Cube.js (analytics)
docker compose up -d formbricks-cube

# 5. Wait for Cube.js to be healthy
docker compose logs -f formbricks-cube | grep -q "Cube.js server is listening"

# 6. Start Hub API
docker compose up -d formbricks-hub

# 7. Start main web application
docker compose up -d formbricks

# 8. Verify all services
docker compose ps
```

### Rolling Updates

```bash
# Update main web app (zero-downtime if replicas > 1)
docker compose pull formbricks
docker compose up -d --no-deps formbricks

# Update Hub API
docker compose pull formbricks-hub
docker compose up -d --no-deps formbricks-hub

# Update Cube.js (requires brief downtime)
docker compose pull formbricks-cube
docker compose up -d --no-deps formbricks-cube
```

---

## Health Checks & Monitoring

### Service Health Endpoints

| Service | Endpoint | Expected Response |
|---------|----------|-------------------|
| Web App | `GET http://localhost:3001/api/health` | `{ "status": "ok" }` |
| Hub API | `GET http://localhost:8080/health` | `{ "status": "healthy" }` |
| Cube.js | `GET http://localhost:4000/readyz` | `200 OK` |

### Docker Health Checks (Automatic)

```yaml
# In docker-compose.yml
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
  interval: 30s
  timeout: 10s
  retries: 5
  start_period: 40s
```

### Prometheus Metrics

Key metrics to alert on:

```promql
# Service up/down
up{job="formbricks-web"} == 0
up{job="formbricks-hub"} == 0
up{job="formbricks-cube"} == 0

# High error rate
rate(formbricks_webhook_errors_total[5m]) > 0.1

# Slow responses
histogram_quantile(0.95, rate(formbricks_http_request_duration_seconds_bucket[5m])) > 5

# Database connections
formbricks_db_connections_active / formbricks_db_connections_max > 0.8
```

### Log Aggregation

```bash
# View all Formbricks logs
docker compose logs -f formbricks formbricks-hub formbricks-cube

# Filter for errors
docker compose logs formbricks 2>&1 | grep -i error

# Follow specific service
docker compose logs -f --tail=100 formbricks
```

---

## Common Operations

### Create New Environment (Per Workspace)

```bash
# 1. Access Formbricks Admin UI
# 2. Go to Settings → Environments → Create Environment
# 3. Copy Environment ID
# 4. Set in MicProtocol: NEXT_PUBLIC_FORMBRICKS_ENVIRONMENT_ID=<env-id>
```

### Configure Survey Targeting

```bash
# Via API (for automation)
curl -X POST https://formbricks.yourdomain.com/api/v1/surveys \
  -H "Authorization: Bearer ${FORMBRICKS_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Post-Recording NPS",
    "type": "web",
    "questions": [...],
    "targeting": {
      "attributes": {
        "workspace_id": "eq:ws_123",
        "plan": "in:pro,enterprise"
      }
    }
  }'
```

### Update Webhook URLs (After Domain Change)

```bash
# In Formbricks Admin: Settings → Webhooks
# Update all webhook URLs to new domain
# Or via API:
curl -X PUT https://formbricks.yourdomain.com/api/v1/webhooks/${WEBHOOK_ID} \
  -H "Authorization: Bearer ${FORMBRICKS_API_KEY}" \
  -d '{"url": "https://api.yourdomain.com/integrations/webhooks/formbricks"}'
```

### Backup Database

```bash
# Full backup
docker exec micprotocol-postgres pg_dump -U micprotocol formbricks > formbricks_backup_$(date +%Y%m%d).sql

# Compressed backup
docker exec micprotocol-postgres pg_dump -U micprotocol formbricks | gzip > formbricks_backup_$(date +%Y%m%d).sql.gz

# Restore
gunzip -c formbricks_backup_20240115.sql.gz | docker exec -i micprotocol-postgres psql -U micprotocol formbricks
```

### Migrate to New PostgreSQL Instance

```bash
# 1. Dump current database
docker exec micprotocol-postgres pg_dump -U micprotocol formbricks > dump.sql

# 2. Create new database on new instance
createdb -h new-postgres -U micprotocol formbricks

# 3. Run migrations on new instance
docker run --rm \
  -e DATABASE_URL="postgresql://user:pass@new-postgres:5432/formbricks" \
  ghcr.io/formbricks/formbricks:latest \
  node packages/database/dist/scripts/apply-migrations.js

# 4. Restore data
psql -h new-postgres -U micprotocol formbricks < dump.sql

# 5. Update docker-compose.yml DATABASE_URL and restart
docker compose up -d formbricks-migrate formbricks-hub-migrate
docker compose up -d formbricks formbricks-hub formbricks-cube
```

---

## Troubleshooting Guide

### Issue: Web App Returns 500 / Won't Start

**Symptoms:** Container restarts, logs show database connection errors

**Diagnosis:**
```bash
# Check database connectivity
docker exec micprotocol-formbricks pg_isready -h postgres -U micprotocol

# Check migration status
docker compose logs formbricks-migrate

# Check environment variables
docker exec micprotocol-formbricks env | grep -E "DATABASE|NEXTAUTH|ENCRYPTION"
```

**Resolution:**
1. Ensure `formbricks-migrate` completed successfully
2. Verify `DATABASE_URL` includes `?schema=public`
3. Check `NEXTAUTH_SECRET` and `ENCRYPTION_KEY` are set
4. Restart: `docker compose restart formbricks`

### Issue: Hub API Unhealthy

**Symptoms:** Surveys don't load, widget shows "Failed to load survey"

**Diagnosis:**
```bash
# Check Hub logs
docker compose logs formbricks-hub

# Test Hub health
curl http://localhost:8080/health

# Check Hub database
docker exec micprotocol-formbricks-hub pg_isready -h postgres -U micprotocol
```

**Resolution:**
1. Ensure `formbricks-hub-migrate` completed
2. Verify `HUB_API_KEY` matches between web and hub
3. Check Redis connectivity
4. Restart: `docker compose restart formbricks-hub`

### Issue: Cube.js Analytics Not Working

**Symptoms:** Dashboard shows no data, "Failed to load" errors

**Diagnosis:**
```bash
# Check Cube.js logs
docker compose logs formbricks-cube

# Test Cube.js API
curl -H "Authorization: ${CUBEJS_API_SECRET}" http://localhost:4000/readyz

# Check Cube.js can reach PostgreSQL
docker exec micprotocol-formbricks-cube pg_isready -h postgres -U micprotocol
```

**Resolution:**
1. Verify `CUBEJS_API_SECRET` matches between web and cube
2. Check `CUBEJS_DB_*` environment variables
3. Ensure `CUBEJS_JWT_ISSUER` and `CUBEJS_JWT_AUDIENCE` match
4. Restart: `docker compose restart formbricks-cube`

### Issue: Webhooks Failing (401 Unauthorized)

**Symptoms:** MicProtocol API returns 401 for Formbricks webhooks

**Diagnosis:**
```bash
# Check webhook secret matches
grep FORMBRICKS_WEBHOOK_SECRET .env
# Compare with Formbricks Admin → Settings → Webhooks → Secret
```

**Resolution:**
1. Ensure `FORMBRICKS_WEBHOOK_SECRET` in `.env` matches Formbricks UI
2. Regenerate if needed: `openssl rand -hex 32`
3. Update both `.env` and Formbricks Admin
4. Restart API: `docker compose restart api`

### Issue: Rate Limiting Blocking Legitimate Requests

**Symptoms:** 429 responses, surveys not loading for some users

**Diagnosis:**
```bash
# Check rate limit config
docker exec micprotocol-formbricks env | grep RATE_LIMITING

# Check Redis for rate limit keys
docker exec micprotocol-redis redis-cli KEYS "ratelimit:*"
```

**Resolution:**
1. Ensure `RATE_LIMITING_DISABLED=0` in production
2. Tune limits in Formbricks Admin → Settings → Rate Limiting
3. For high-traffic: increase Redis capacity or use Redis Cluster

### Issue: Email Notifications Not Sending

**Symptoms:** No survey invitation emails, password reset emails fail

**Diagnosis:**
```bash
# Check SMTP config
docker exec micprotocol-formbricks env | grep -i SMTP

# Check mailhog (dev) or SMTP logs
docker compose logs mailhog
```

**Resolution:**
1. Verify `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`
2. Test with: `docker exec micprotocol-formbricks nc -zv smtp.example.com 587`
3. Check `MAIL_FROM` is verified in your SMTP provider

---

## Disaster Recovery

### Scenario: Complete Data Loss (PostgreSQL Corruption)

**RTO: 2 hours | RPO: 24 hours (daily backups)**

1. Provision new PostgreSQL instance
2. Restore latest backup:
   ```bash
   gunzip -c formbricks_backup_latest.sql.gz | psql -h new-pg -U micprotocol formbricks
   ```
3. Run migrations to ensure schema is current:
   ```bash
   docker compose up formbricks-migrate formbricks-hub-migrate
   ```
4. Restart all services:
   ```bash
   docker compose up -d formbricks-cube formbricks-hub formbricks
   ```
5. Verify data integrity via Admin UI

### Scenario: Formbricks Hub Database Corruption

**RTO: 30 minutes | RPO: 1 hour**

1. Hub uses same PostgreSQL database (`formbricks` schema)
2. Restore from backup (same as above)
3. Re-run hub migrations:
   ```bash
   docker compose up formbricks-hub-migrate
   docker compose restart formbricks-hub
   ```

### Scenario: Lost Encryption Keys (NEXTAUTH_SECRET, ENCRYPTION_KEY)

**IMPACT: All existing sessions invalidated, encrypted data unrecoverable**

**Mitigation:**
1. These keys are **never rotated** after initial setup
2. Store in secure password manager (1Password, Bitwarden)
3. Document in runbook with access restricted to platform team

**Recovery:**
1. Generate new keys
2. Update `.env` and restart all services
3. All users must re-authenticate
4. Historical encrypted data (webhook payloads, etc.) will be unreadable

---

## Security Checklist

### Monthly
- [ ] Rotate `FORMBRICKS_CRON_SECRET` and `FORMBRICKS_WEBHOOK_SECRET`
- [ ] Review webhook delivery logs for anomalies
- [ ] Check for failed login attempts in logs
- [ ] Verify SSL certificates are valid (> 30 days)

### Quarterly
- [ ] Update all container images to latest patches
- [ ] Review and prune old survey responses (GDPR)
- [ ] Audit API keys in Formbricks Admin
- [ ] Test backup restoration procedure

### Annually
- [ ] Rotate SMTP credentials
- [ ] Review Redis AUTH password
- [ ] Penetration test webhook endpoints
- [ ] Update Cube.js and Hub to latest major versions

---

## Performance Tuning

### Database

```sql
-- Add indexes for common query patterns
CREATE INDEX idx_surveys_environment_status ON surveys(environment_id, status);
CREATE INDEX idx_responses_survey_created ON responses(survey_id, created_at);
CREATE INDEX idx_attributes_user_key ON user_attributes(user_id, key);
```

### Redis

```bash
# Increase maxmemory for rate limiting
# In redis.conf or docker-compose:
command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru
```

### Cube.js

```yaml
# In docker-compose.yml for formbricks-cube
environment:
  CUBEJS_CACHE_AND_QUEUE_DRIVER: redis
  CUBEJS_REDIS_URL: redis://:password@redis:6379
  CUBEJS_DB_MAX_POOL: 20
```

---

## Contact & Escalation

| Issue Severity | Response Time | Contact |
|----------------|---------------|---------|
| P0 (All surveys down) | 15 min | Platform On-Call |
| P1 (Single workspace affected) | 1 hour | Platform Team |
| P2 (Analytics delayed) | 4 hours | Platform Team |
| P3 (Feature request) | Next sprint | Product Team |

**Runbook Owner:** Platform Engineering  
**Last Updated:** 2024-01-15  
**Next Review:** 2024-04-15