# Operations Runbook

Day-to-day and incident-runbook guidance for MicProtocol in production. This is the central ops doc; per-service deep dives live in [Formbricks Runbook](./formbricks-runbook.md) and the relevant sections of [Self-Hosted Stack](./self-hosted-stack.md).

## 1. Service Inventory

For every container and the resource you need to keep healthy, see the catalog in [Self-Hosted Stack → Service Catalog](./self-hosted-stack.md#service-catalog). Collectively you operate:

- Core infra: Postgres (`pgvector`), Redis, Meilisearch, MinIO, Mailhog (dev) / SMTP relay (prod)
- MicProtocol apps: API, worker, CLIP scorer, mic-realtime, web, admin
- Self-hosted integrations: Cal.com, Formbricks (5 containers), Documenso, Chatwoot (2 containers), Listmonk, Umami, LiveKit

## 2. Standard Operating Procedures

### 2.1 Daily checks

```bash
# Cluster health
docker compose ps
curl -s http://localhost:4000/health/detailed | jq
curl -s http://localhost:4000/integrations/health | jq

# Job queue depth
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:outreach:wait"
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:content:wait"
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:clip-scoring:wait"

# API error rate (last hour)
curl -s http://localhost:4000/tracing/metrics | jq '.errorRate'
```

Thresholds:

- Queue depth: alert at > 500 per queue for 15 min
- API p95 latency: alert at > 500 ms for 5 min (read), > 2000 ms for 5 min (write/AI)
- API error rate: alert at > 1% sustained

### 2.2 Weekly maintenance window

```bash
# Pull latest images (out-of-band from CI deploys)
docker compose pull

# Roll each integration in isolation (see Self-Hosted Stack → Rolling update a single integration)
docker compose up -d --no-deps calcom
# Formbricks requires re-running migrations (see runbook)

# Restart worker to release orphaned Redis locks
docker compose restart worker

# Vacuum Postgres
docker exec micprotocol-postgres psql -U micprotocol -d micprotocol -c 'VACUUM ANALYZE;'

# Check disk usage
docker system df
```

### 2.3 Backup

Daily backups are the minimum RPO target. Example cron against the live container:

```bash
# Snapshot all integration databases
for db in micprotocol chatwoot calcom documenso formbricks listmonk umami; do
  docker exec micprotocol-postgres pg_dump -U micprotocol -Fc "$db" | \
    gzip > "/backups/${db}_$(date +%Y%m%d).sql.gz"
done

# Retention: prune anything older than 30 days
find /backups -name '*.sql.gz' -mtime +30 -delete
```

Test restore quarterly into a sandbox cluster — an untested backup is no backup.

## 3. Health & Monitoring

### 3.1 Health endpoints

Every service exposes its own health check; see [Self-Hosted Stack → Health Endpoints](./self-hosted-stack.md#health-endpoints) for the full list. The single consolidated health check is:

```bash
curl -s http://localhost:4000/health/detailed
```

This pings DB, Redis, Meilisearch, and reports the API's own subsystem status. For self-hosted integration reachability:

```bash
curl -s http://localhost:4000/integrations/health
```

### 3.2 Recommended exporters

| Metric | Source | Alert on |
|---|---|---|
| Container up/down | `docker compose ps` / orchestration | `up == 0` for > 1 min |
| API p95 latency | `/tracing/metrics` or Datadog APM | > 500 ms for 5 min |
| API error rate | `/tracing/metrics` | > 1% sustained |
| Queue depth | Redis `LLEN bull:*:wait` | > 500 pending per queue |
| Postgres connections | `pg_stat_activity` | Active / max > 0.8 |
| Redis memory | `INFO memory` | `used_memory_peak / maxmemory > 0.85` |
| Webhook 401s | API logs (`UnauthorizedException`) | Spike rate |
| CLIP scorer errors | Worker logs | Any timeout / subprocess non-zero exit |

### 3.3 Sentry integration

Set `SENTRY_DSN` to enable error grouping and release tracking. Optional sampling:

```
SENTRY_TRACE_SAMPLE_RATE=0.1     # 10% of traces
SENTRY_PROFILE_SAMPLE_RATE=0.05  # 5% of profiles
```

## 4. Incident Response

### 4.1 Severity definitions

| Severity | Definition | Response time | Owner |
|---|---|---|---|
| P0 | All API unavailable, no bookings, no recordings | 15 min | Platform on-call |
| P1 | Single workspace affected, or one integration down | 1 hour | Platform team |
| P2 | Analytics delayed, non-critical background jobs backed up | 4 hours | Platform team |
| P3 | Feature request / cosmetic | Next sprint | Product team |

### 4.2 P0 playbook — API down

1. `curl -s http://localhost:4000/health/detailed` — confirms scope (DB, Redis, search).
2. `docker compose ps` — for any container not in `Up` state.
3. Tail API logs: `docker compose logs -f --tail=200 api`.
4. Restart in dependency order: `postgres` → `redis` → `meilisearch` → `api` → `worker`.
5. If DB is the issue, fail back to the latest snapshot (see §2.3).
6. Confirm recovery via `/health/detailed` and a synthetic agent sweep.
7. File an incident report within 24 hours; review in the next retro.

### 4.3 P0 playbook — recording sessions failing (LiveKit)

1. `curl -s http://localhost:4000/integrations/health` — confirms LiveKit reachable.
2. `docker logs micprotocol-livekit` — check for ICE/UDP port binding errors.
3. Confirm UDP `7882` is open end-to-end (firewall, security group, NAT).
4. Verify `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET` match the configured keys inside the LiveKit container.
5. For U-Fiber/nat'd networks, configure `rtc.use_external_ip: true` or run behind a TURN server.

### 4.4 P1 playbook — webhook storm

Symptom: API CPU pegged, `UnauthorizedException` or `IntegrationsWebhookController` log spam.

1. `docker compose logs -f --tail=200 api | grep IntegrationsWebhookController` — identifies the offender.
2. Confirm secret rotation hasn't broken verification (`*_WEBHOOK_SECRET` on both sides).
3. If a provider is redelivering failed events in a tight loop, temporarily pause redelivery upstream (Formbricks / Cal.com / Listmonk admin UI).
4. Refresh the `WebhookIdempotencyService` cache if the duplicate-detection rate has dropped to zero.
5. Restore redelivery once root cause is fixed.

### 4.5 P1 playbook — clip-scoring stuck

Symptom: clips stuck in "Scoring…" badge indefinitely.

1. Scoring runs inside the worker container via a Python subprocess — `docker ps | grep micprotocol-worker` confirms the process is up.
2. `docker logs micprotocol-worker` — look for model download failures or FFmpeg errors.
3. Orphaned Redis distributed locks self-expire with the 5-minute TTL; restarting the worker clears them immediately.
4. Re-queue affected clips via the Content Studio UI.

See [CLIP Scoring Engine → Troubleshooting](./clip-scoring.md#troubleshooting) for full detail.

### 4.6 P0 playbook — Postgres corruption

Recovery point objective (RPO): 24 hours (daily backups). Recovery time objective (RTO): 2 hours.

1. Provision a new Postgres instance.
2. Restore the latest snapshot: `gunzip -c micprotocol_latest.sql.gz | psql -h new-pg -U micprotocol micprotocol`.
3. Re-run schema migrations to ensure fresh state.
4. Update `DATABASE_URL` and restart the API.
5. Verify data integrity via Prisma Studio + synthetic agent sweep.

Per-integration recovery: Formbricks, Chatwoot, Cal.com etc. each have their own DB. Restore only what's affected — restoring one integration's DB from a 12-hour-old snapshot does not compromise the others.

## 5. Performance Tuning

### 5.1 Postgres

```sql
-- Indexes for the heaviest MicProtocol queries (already in schema, audit periodically)
-- guests(podcast_id, status), guests(podcast_id, priority_score)
-- outreach_sequences(guest_id, status)
-- bookings(start_time, end_time)
-- asset_requests(episode_id, type)

-- Slow query log
ALTER SYSTEM SET log_min_duration_statement = 200;
SELECT pg_reload_conf();
```

### 5.2 Redis

Production Redis with BullMQ should not share its memory budget with caching:

```yaml
# docker-compose override
command: redis-server --maxmemory 1gb --maxmemory-policy noeviction --requirepass "${REDIS_PASSWORD}"
```

`noeviction` ensures BullMQ job metadata is never evicted; for caching keys, prefer a separate Redis instance.

### 5.3 CLIP scorer

CLIP is CPU-bound; GPU optional but cuts per-clip time by 5–10×. See [CLIP Scoring → Scaling Considerations](./clip-scoring.md) and [Deployment → Scaling Considerations for CLIP Scorer](./deployment.md#scaling-considerations-for-clip-scorer).

### 5.4 Meilisearch

Re-indexing is cheap; if search latency regresses after a large guest import, trigger a re-index via the API or `curl -X POST http://localhost:7700/indexes/guests/update`.

## 6. Scaling Triggers

See [Deployment & Scaling → Scaling Triggers](./deployment.md#scaling-triggers) for the full phase-by-phase plan. Operators should plan capacity against the documented phase triggers rather than waiting for alerts.

| Phase | Trigger | Architectural change |
|---|---|---|
| 1 (≤1K podcasts) | Launch | Single ECS task per service; RDS `t4g.micro`; ElastiCache Serverless |
| 2 (1–5K podcasts) | P95 > 500ms OR CPU > 70% sustained | Split workers from API; dedicated Redis for BullMQ vs. cache; RDS read replica |
| 3 (5–20K podcasts) | Email > 100K/day OR queue depth > 1K | Extract Outreach Domain to microservice; migrate orchestration to Temporal.io |
| 4 (20K+ podcasts) | Cross-domain query > 2s P95 | Full microservices split; NATS JetStream event bus; per-service RDS |

## 7. Common Operations

### 7.1 Rotate a webhook secret

```bash
NEW=$(openssl rand -hex 32)
# Update .env: CALCOM_WEBHOOK_SECRET=${NEW}
# Update Cal.com Admin UI: Settings → Webhooks → secret
docker compose restart api
# Tail logs to confirm webhooks still pass verification
docker compose logs -f api | grep -i webhook
```

### 7.2 Add a new workspace for a customer

Synthetic agent seed: [`apps/api/prisma/seed-agents.ts`](../apps/api/prisma/seed-agents.ts) is the reference for deterministic workspace creation. Use the same pattern via tRPC `workspace.create` with the customer's Clerk user ID.

### 7.3 Run a synthetic agent sweep manually

```bash
pnpm db:seed:agents   # reset seed (optional)
pnpm agents:orchestrator  # requires API running on :4000
cat scripts/synthetic-agents/test-report.txt
jq '.[] | select(.severity == "high")' scripts/synthetic-agents/weaknesses.json
```

### 7.4 Onboard a new self-hosted integration

1. Add the service block to `docker-compose.yml` (mirror an existing block).
2. Add its database to `scripts/init-databases.sql` if it needs a Postgres schema.
3. Create the API client methods in `apps/api/src/integrations/clients/self-hosted-clients.service.ts` (mirroring the `calCom*`, `chatwoot*`, `formbricks*`, `listmonk*`, `documenso*` families).
4. Add a webhook handler in `apps/api/src/integrations/integrations-webhook.controller.ts`.
5. Document the new endpoint in [Webhooks](./webhooks.md) and [Third-Party & Self-Hosted Integrations](./third-party-integrations.md).
6. Add a `*_WEBHOOK_SECRET` to `.env.example` and `scripts/generate-secrets.sh`.
7. Add a health check method and wire it into the `/integrations/health` aggregator.

## 8. Change Management

- Production deploys happen only through the CI/CD pipeline (see [Deployment & Scaling → CI/CD Pipeline](./deployment.md#cicd-pipeline)).
- Database migrations: prefer additive changes (new columns nullable, new tables empty, new indexes `CONCURRENTLY`). Prisma migrations ship with the API image.
- Configuration changes (env vars): update `.env` in the secrets store first; orchestrator reads new value on next deploy.
- Backwards-incompatible API changes: gate behind feature flags (`ENABLE_AI_FEATURES`, `ENABLE_CONTENT_STUDIO`, `ENABLE_ANALYTICS`) for at least one release.

## 9. Contacts & Escalation

| Issue Severity | Response Time | Contact |
|---|---|---|
| P0 (all API down) | 15 min | Platform on-call |
| P1 (single workspace) | 1 hour | Platform team |
| P2 (analytics delayed) | 4 hours | Platform team |
| P3 (feature request) | Next sprint | Product team |

Runbook owner: Platform Engineering. Review quarterly.