docs/operations-runbook.md

Operations Runbook

Day-to-day and incident-runbook guidance for MicProtocol in production. This is the central ops doc; per-service deep dives live in Formbricks Runbook and the relevant sections of Self-Hosted Stack.

1. Service Inventory

For every container and the resource you need to keep healthy, see the catalog in Self-Hosted Stack → Service Catalog. Collectively you operate:

• Core infra: Postgres (pgvector), Redis, Meilisearch, MinIO, Mailhog (dev) / SMTP relay (prod)
• MicProtocol apps: API, worker, CLIP scorer, mic-realtime, web, admin
• Self-hosted integrations: Cal.com, Formbricks (5 containers), Documenso, Chatwoot (2 containers), Listmonk, Umami, LiveKit

2. Standard Operating Procedures

2.1 Daily checks

# Cluster health
docker compose ps
curl -s http://localhost:4000/health/detailed | jq
curl -s http://localhost:4000/integrations/health | jq

# Job queue depth
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:outreach:wait"
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:content:wait"
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:clip-scoring:wait"

# API error rate (last hour)
curl -s http://localhost:4000/tracing/metrics | jq '.errorRate'

Thresholds:

• Queue depth: alert at > 500 per queue for 15 min
• API p95 latency: alert at > 500 ms for 5 min (read), > 2000 ms for 5 min (write/AI)
• API error rate: alert at > 1% sustained

2.2 Weekly maintenance window

# Pull latest images (out-of-band from CI deploys)
docker compose pull

# Roll each integration in isolation (see Self-Hosted Stack → Rolling update a single integration)
docker compose up -d --no-deps calcom
# Formbricks requires re-running migrations (see runbook)

# Restart worker to release orphaned Redis locks
docker compose restart worker

# Vacuum Postgres
docker exec micprotocol-postgres psql -U micprotocol -d micprotocol -c 'VACUUM ANALYZE;'

# Check disk usage
docker system df

2.3 Backup

Daily backups are the minimum RPO target. Example cron against the live container:

# Snapshot all integration databases
for db in micprotocol chatwoot calcom documenso formbricks listmonk umami; do
  docker exec micprotocol-postgres pg_dump -U micprotocol -Fc "$db" | \
    gzip > "/backups/${db}_$(date +%Y%m%d).sql.gz"
done

# Retention: prune anything older than 30 days
find /backups -name '*.sql.gz' -mtime +30 -delete

Test restore quarterly into a sandbox cluster — an untested backup is no backup.

3. Health & Monitoring

3.1 Health endpoints

Every service exposes its own health check; see Self-Hosted Stack → Health Endpoints for the full list. The single consolidated health check is:

curl -s http://localhost:4000/health/detailed

This pings DB, Redis, Meilisearch, and reports the API's own subsystem status. For self-hosted integration reachability:

curl -s http://localhost:4000/integrations/health

3.2 Recommended exporters

Metric	Source	Alert on
Container up/down	docker compose ps / orchestration	up == 0 for > 1 min
API p95 latency	/tracing/metrics or Datadog APM	> 500 ms for 5 min
API error rate	/tracing/metrics	> 1% sustained
Queue depth	Redis LLEN bull:*:wait	> 500 pending per queue
Postgres connections	pg_stat_activity	Active / max > 0.8
Redis memory	INFO memory	used_memory_peak / maxmemory > 0.85
Webhook 401s	API logs (UnauthorizedException)	Spike rate
CLIP scorer errors	Worker logs	Any timeout / subprocess non-zero exit

3.3 Sentry integration

Set SENTRY_DSN to enable error grouping and release tracking. Optional sampling:

SENTRY_TRACE_SAMPLE_RATE=0.1     # 10% of traces
SENTRY_PROFILE_SAMPLE_RATE=0.05  # 5% of profiles

4. Incident Response

4.1 Severity definitions

Severity	Definition	Response time	Owner
P0	All API unavailable, no bookings, no recordings	15 min	Platform on-call
P1	Single workspace affected, or one integration down	1 hour	Platform team
P2	Analytics delayed, non-critical background jobs backed up	4 hours	Platform team
P3	Feature request / cosmetic	Next sprint	Product team

4.2 P0 playbook — API down

1. curl -s http://localhost:4000/health/detailed — confirms scope (DB, Redis, search).
2. docker compose ps — for any container not in Up state.
3. Tail API logs: docker compose logs -f --tail=200 api.
4. Restart in dependency order: postgres → redis → meilisearch → api → worker.
5. If DB is the issue, fail back to the latest snapshot (see §2.3).
6. Confirm recovery via /health/detailed and a synthetic agent sweep.
7. File an incident report within 24 hours; review in the next retro.

4.3 P0 playbook — recording sessions failing (LiveKit)

1. curl -s http://localhost:4000/integrations/health — confirms LiveKit reachable.
2. docker logs micprotocol-livekit — check for ICE/UDP port binding errors.
3. Confirm UDP 7882 is open end-to-end (firewall, security group, NAT).
4. Verify LIVEKIT_API_KEY and LIVEKIT_API_SECRET match the configured keys inside the LiveKit container.
5. For U-Fiber/nat'd networks, configure rtc.use_external_ip: true or run behind a TURN server.

4.4 P1 playbook — webhook storm

Symptom: API CPU pegged, UnauthorizedException or IntegrationsWebhookController log spam.

1. docker compose logs -f --tail=200 api | grep IntegrationsWebhookController — identifies the offender.
2. Confirm secret rotation hasn't broken verification (*_WEBHOOK_SECRET on both sides).
3. If a provider is redelivering failed events in a tight loop, temporarily pause redelivery upstream (Formbricks / Cal.com / Listmonk admin UI).
4. Refresh the WebhookIdempotencyService cache if the duplicate-detection rate has dropped to zero.
5. Restore redelivery once root cause is fixed.

4.5 P1 playbook — clip-scoring stuck

Symptom: clips stuck in "Scoring…" badge indefinitely.

1. Scoring runs inside the worker container via a Python subprocess — docker ps | grep micprotocol-worker confirms the process is up.
2. docker logs micprotocol-worker — look for model download failures or FFmpeg errors.
3. Orphaned Redis distributed locks self-expire with the 5-minute TTL; restarting the worker clears them immediately.
4. Re-queue affected clips via the Content Studio UI.

See CLIP Scoring Engine → Troubleshooting for full detail.

4.6 P0 playbook — Postgres corruption

Recovery point objective (RPO): 24 hours (daily backups). Recovery time objective (RTO): 2 hours.

1. Provision a new Postgres instance.
2. Restore the latest snapshot: gunzip -c micprotocol_latest.sql.gz | psql -h new-pg -U micprotocol micprotocol.
3. Re-run schema migrations to ensure fresh state.
4. Update DATABASE_URL and restart the API.
5. Verify data integrity via Prisma Studio + synthetic agent sweep.

Per-integration recovery: Formbricks, Chatwoot, Cal.com etc. each have their own DB. Restore only what's affected — restoring one integration's DB from a 12-hour-old snapshot does not compromise the others.

5. Performance Tuning

5.1 Postgres

-- Indexes for the heaviest MicProtocol queries (already in schema, audit periodically)
-- guests(podcast_id, status), guests(podcast_id, priority_score)
-- outreach_sequences(guest_id, status)
-- bookings(start_time, end_time)
-- asset_requests(episode_id, type)

-- Slow query log
ALTER SYSTEM SET log_min_duration_statement = 200;
SELECT pg_reload_conf();

5.2 Redis

Production Redis with BullMQ should not share its memory budget with caching:

# docker-compose override
command: redis-server --maxmemory 1gb --maxmemory-policy noeviction --requirepass "${REDIS_PASSWORD}"

noeviction ensures BullMQ job metadata is never evicted; for caching keys, prefer a separate Redis instance.

5.3 CLIP scorer

CLIP is CPU-bound; GPU optional but cuts per-clip time by 5–10×. See CLIP Scoring → Scaling Considerations and Deployment → Scaling Considerations for CLIP Scorer.

5.4 Meilisearch

Re-indexing is cheap; if search latency regresses after a large guest import, trigger a re-index via the API or curl -X POST http://localhost:7700/indexes/guests/update.

6. Scaling Triggers

See Deployment & Scaling → Scaling Triggers for the full phase-by-phase plan. Operators should plan capacity against the documented phase triggers rather than waiting for alerts.

Phase	Trigger	Architectural change
1 (≤1K podcasts)	Launch	Single ECS task per service; RDS t4g.micro; ElastiCache Serverless
2 (1–5K podcasts)	P95 > 500ms OR CPU > 70% sustained	Split workers from API; dedicated Redis for BullMQ vs. cache; RDS read replica
3 (5–20K podcasts)	Email > 100K/day OR queue depth > 1K	Extract Outreach Domain to microservice; migrate orchestration to Temporal.io
4 (20K+ podcasts)	Cross-domain query > 2s P95	Full microservices split; NATS JetStream event bus; per-service RDS

7. Common Operations

7.1 Rotate a webhook secret

NEW=$(openssl rand -hex 32)
# Update .env: CALCOM_WEBHOOK_SECRET=${NEW}
# Update Cal.com Admin UI: Settings → Webhooks → secret
docker compose restart api
# Tail logs to confirm webhooks still pass verification
docker compose logs -f api | grep -i webhook

7.2 Add a new workspace for a customer

Synthetic agent seed: apps/api/prisma/seed-agents.ts is the reference for deterministic workspace creation. Use the same pattern via tRPC workspace.create with the customer's Clerk user ID.

7.3 Run a synthetic agent sweep manually

pnpm db:seed:agents   # reset seed (optional)
pnpm agents:orchestrator  # requires API running on :4000
cat scripts/synthetic-agents/test-report.txt
jq '.[] | select(.severity == "high")' scripts/synthetic-agents/weaknesses.json

7.4 Onboard a new self-hosted integration

1. Add the service block to docker-compose.yml (mirror an existing block).
2. Add its database to scripts/init-databases.sql if it needs a Postgres schema.
3. Create the API client methods in apps/api/src/integrations/clients/self-hosted-clients.service.ts (mirroring the calCom*, chatwoot*, formbricks*, listmonk*, documenso* families).
4. Add a webhook handler in apps/api/src/integrations/integrations-webhook.controller.ts.
5. Document the new endpoint in Webhooks and Third-Party & Self-Hosted Integrations.
6. Add a *_WEBHOOK_SECRET to .env.example and scripts/generate-secrets.sh.
7. Add a health check method and wire it into the /integrations/health aggregator.

8. Change Management

• Production deploys happen only through the CI/CD pipeline (see Deployment & Scaling → CI/CD Pipeline).
• Database migrations: prefer additive changes (new columns nullable, new tables empty, new indexes CONCURRENTLY). Prisma migrations ship with the API image.
• Configuration changes (env vars): update .env in the secrets store first; orchestrator reads new value on next deploy.
• Backwards-incompatible API changes: gate behind feature flags (ENABLE_AI_FEATURES, ENABLE_CONTENT_STUDIO, ENABLE_ANALYTICS) for at least one release.

9. Contacts & Escalation

Issue Severity	Response Time	Contact
P0 (all API down)	15 min	Platform on-call
P1 (single workspace)	1 hour	Platform team
P2 (analytics delayed)	4 hours	Platform team
P3 (feature request)	Next sprint	Product team

Runbook owner: Platform Engineering. Review quarterly.