Spaces:

Benaah
/

MicProtocol

Running

App Files Files Community

MicProtocol / docs /operations-runbook.md

Benaah

Upload folder using huggingface_hub

b17baae verified about 21 hours ago

preview code

Raw History Blame Contribute Delete

12.1 kB

# Operations Runbook

Day-to-day and incident-runbook guidance for MicProtocol in production. This is the central ops doc; per-service deep dives live in [Formbricks Runbook](./formbricks-runbook.md) and the relevant sections of [Self-Hosted Stack](./self-hosted-stack.md).

## 1. Service Inventory

For every container and the resource you need to keep healthy, see the catalog in [Self-Hosted Stack → Service Catalog](./self-hosted-stack.md#service-catalog). Collectively you operate:

- Core infra: Postgres (`pgvector`), Redis, Meilisearch, MinIO, Mailhog (dev) / SMTP relay (prod)
- MicProtocol apps: API, worker, CLIP scorer, mic-realtime, web, admin
- Self-hosted integrations: Cal.com, Formbricks (5 containers), Documenso, Chatwoot (2 containers), Listmonk, Umami, LiveKit

## 2. Standard Operating Procedures

### 2.1 Daily checks

```bash
# Cluster health
docker compose ps
curl -s http://localhost:4000/health/detailed | jq
curl -s http://localhost:4000/integrations/health | jq
# Job queue depth
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:outreach:wait"
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:content:wait"
docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:clip-scoring:wait"
# API error rate (last hour)
curl -s http://localhost:4000/tracing/metrics | jq '.errorRate'
```

Thresholds:

- Queue depth: alert at > 500 per queue for 15 min
- API p95 latency: alert at > 500 ms for 5 min (read), > 2000 ms for 5 min (write/AI)
- API error rate: alert at > 1% sustained

### 2.2 Weekly maintenance window

```bash
# Pull latest images (out-of-band from CI deploys)
docker compose pull
# Roll each integration in isolation (see Self-Hosted Stack → Rolling update a single integration)
docker compose up -d --no-deps calcom
# Formbricks requires re-running migrations (see runbook)
# Restart worker to release orphaned Redis locks
docker compose restart worker
# Vacuum Postgres
docker exec micprotocol-postgres psql -U micprotocol -d micprotocol -c 'VACUUM ANALYZE;'
# Check disk usage
docker system df
```

### 2.3 Backup

Daily backups are the minimum RPO target. Example cron against the live container:

```bash
# Snapshot all integration databases
for db in micprotocol chatwoot calcom documenso formbricks listmonk umami; do
  docker exec micprotocol-postgres pg_dump -U micprotocol -Fc "$db" | \
    gzip > "/backups/${db}_$(date +%Y%m%d).sql.gz"
done
# Retention: prune anything older than 30 days
find /backups -name '*.sql.gz' -mtime +30 -delete
```

Test restore quarterly into a sandbox cluster — an untested backup is no backup.

## 3. Health & Monitoring

### 3.1 Health endpoints

Every service exposes its own health check; see [Self-Hosted Stack → Health Endpoints](./self-hosted-stack.md#health-endpoints) for the full list. The single consolidated health check is:

```bash
curl -s http://localhost:4000/health/detailed
```

This pings DB, Redis, Meilisearch, and reports the API's own subsystem status. For self-hosted integration reachability:

```bash
curl -s http://localhost:4000/integrations/health
```

### 3.2 Recommended exporters

| Metric | Source | Alert on |
|---|---|---|
| Container up/down | `docker compose ps` / orchestration | `up == 0` for > 1 min |
| API p95 latency | `/tracing/metrics` or Datadog APM | > 500 ms for 5 min |
| API error rate | `/tracing/metrics` | > 1% sustained |
| Queue depth | Redis `LLEN bull:*:wait` | > 500 pending per queue |
| Postgres connections | `pg_stat_activity` | Active / max > 0.8 |
| Redis memory | `INFO memory` | `used_memory_peak / maxmemory > 0.85` |
| Webhook 401s | API logs (`UnauthorizedException`) | Spike rate |
| CLIP scorer errors | Worker logs | Any timeout / subprocess non-zero exit |

### 3.3 Sentry integration

Set `SENTRY_DSN` to enable error grouping and release tracking. Optional sampling:

```
SENTRY_TRACE_SAMPLE_RATE=0.1     # 10% of traces
SENTRY_PROFILE_SAMPLE_RATE=0.05  # 5% of profiles
```

## 4. Incident Response

### 4.1 Severity definitions

| Severity | Definition | Response time | Owner |
|---|---|---|---|
| P0 | All API unavailable, no bookings, no recordings | 15 min | Platform on-call |
| P1 | Single workspace affected, or one integration down | 1 hour | Platform team |
| P2 | Analytics delayed, non-critical background jobs backed up | 4 hours | Platform team |
| P3 | Feature request / cosmetic | Next sprint | Product team |

### 4.2 P0 playbook — API down

1. `curl -s http://localhost:4000/health/detailed` — confirms scope (DB, Redis, search).
2. `docker compose ps` — for any container not in `Up` state.
3. Tail API logs: `docker compose logs -f --tail=200 api`.
4. Restart in dependency order: `postgres` → `redis` → `meilisearch` → `api` → `worker`.
5. If DB is the issue, fail back to the latest snapshot (see §2.3).
6. Confirm recovery via `/health/detailed` and a synthetic agent sweep.
7. File an incident report within 24 hours; review in the next retro.

### 4.3 P0 playbook — recording sessions failing (LiveKit)

1. `curl -s http://localhost:4000/integrations/health` — confirms LiveKit reachable.
2. `docker logs micprotocol-livekit` — check for ICE/UDP port binding errors.
3. Confirm UDP `7882` is open end-to-end (firewall, security group, NAT).
4. Verify `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET` match the configured keys inside the LiveKit container.
5. For U-Fiber/nat'd networks, configure `rtc.use_external_ip: true` or run behind a TURN server.

### 4.4 P1 playbook — webhook storm

Symptom: API CPU pegged, `UnauthorizedException` or `IntegrationsWebhookController` log spam.

1. `docker compose logs -f --tail=200 api | grep IntegrationsWebhookController` — identifies the offender.
2. Confirm secret rotation hasn't broken verification (`*_WEBHOOK_SECRET` on both sides).
3. If a provider is redelivering failed events in a tight loop, temporarily pause redelivery upstream (Formbricks / Cal.com / Listmonk admin UI).
4. Refresh the `WebhookIdempotencyService` cache if the duplicate-detection rate has dropped to zero.
5. Restore redelivery once root cause is fixed.

### 4.5 P1 playbook — clip-scoring stuck

Symptom: clips stuck in "Scoring…" badge indefinitely.

1. Scoring runs inside the worker container via a Python subprocess — `docker ps | grep micprotocol-worker` confirms the process is up.
2. `docker logs micprotocol-worker` — look for model download failures or FFmpeg errors.
3. Orphaned Redis distributed locks self-expire with the 5-minute TTL; restarting the worker clears them immediately.
4. Re-queue affected clips via the Content Studio UI.

See [CLIP Scoring Engine → Troubleshooting](./clip-scoring.md#troubleshooting) for full detail.

### 4.6 P0 playbook — Postgres corruption

Recovery point objective (RPO): 24 hours (daily backups). Recovery time objective (RTO): 2 hours.

1. Provision a new Postgres instance.
2. Restore the latest snapshot: `gunzip -c micprotocol_latest.sql.gz | psql -h new-pg -U micprotocol micprotocol`.
3. Re-run schema migrations to ensure fresh state.
4. Update `DATABASE_URL` and restart the API.
5. Verify data integrity via Prisma Studio + synthetic agent sweep.

Per-integration recovery: Formbricks, Chatwoot, Cal.com etc. each have their own DB. Restore only what's affected — restoring one integration's DB from a 12-hour-old snapshot does not compromise the others.

## 5. Performance Tuning

### 5.1 Postgres

```sql
-- Indexes for the heaviest MicProtocol queries (already in schema, audit periodically)
-- guests(podcast_id, status), guests(podcast_id, priority_score)
-- outreach_sequences(guest_id, status)
-- bookings(start_time, end_time)
-- asset_requests(episode_id, type)
-- Slow query log
ALTER SYSTEM SET log_min_duration_statement = 200;
SELECT pg_reload_conf();
```

### 5.2 Redis

Production Redis with BullMQ should not share its memory budget with caching:

```yaml
# docker-compose override
command: redis-server --maxmemory 1gb --maxmemory-policy noeviction --requirepass "${REDIS_PASSWORD}"
```

`noeviction` ensures BullMQ job metadata is never evicted; for caching keys, prefer a separate Redis instance.

### 5.3 CLIP scorer

CLIP is CPU-bound; GPU optional but cuts per-clip time by 5–10×. See [CLIP Scoring → Scaling Considerations](./clip-scoring.md) and [Deployment → Scaling Considerations for CLIP Scorer](./deployment.md#scaling-considerations-for-clip-scorer).

### 5.4 Meilisearch

Re-indexing is cheap; if search latency regresses after a large guest import, trigger a re-index via the API or `curl -X POST http://localhost:7700/indexes/guests/update`.

## 6. Scaling Triggers

See [Deployment & Scaling → Scaling Triggers](./deployment.md#scaling-triggers) for the full phase-by-phase plan. Operators should plan capacity against the documented phase triggers rather than waiting for alerts.

| Phase | Trigger | Architectural change |
|---|---|---|
| 1 (≤1K podcasts) | Launch | Single ECS task per service; RDS `t4g.micro`; ElastiCache Serverless |
| 2 (1–5K podcasts) | P95 > 500ms OR CPU > 70% sustained | Split workers from API; dedicated Redis for BullMQ vs. cache; RDS read replica |
| 3 (5–20K podcasts) | Email > 100K/day OR queue depth > 1K | Extract Outreach Domain to microservice; migrate orchestration to Temporal.io |
| 4 (20K+ podcasts) | Cross-domain query > 2s P95 | Full microservices split; NATS JetStream event bus; per-service RDS |

## 7. Common Operations

### 7.1 Rotate a webhook secret

```bash
NEW=$(openssl rand -hex 32)
# Update .env: CALCOM_WEBHOOK_SECRET=${NEW}
# Update Cal.com Admin UI: Settings → Webhooks → secret
docker compose restart api
# Tail logs to confirm webhooks still pass verification
docker compose logs -f api | grep -i webhook
```

### 7.2 Add a new workspace for a customer

Synthetic agent seed: [`apps/api/prisma/seed-agents.ts`](../apps/api/prisma/seed-agents.ts) is the reference for deterministic workspace creation. Use the same pattern via tRPC `workspace.create` with the customer's Clerk user ID.

### 7.3 Run a synthetic agent sweep manually

```bash
pnpm db:seed:agents   # reset seed (optional)
pnpm agents:orchestrator  # requires API running on :4000
cat scripts/synthetic-agents/test-report.txt
jq '.[] | select(.severity == "high")' scripts/synthetic-agents/weaknesses.json
```

### 7.4 Onboard a new self-hosted integration

1. Add the service block to `docker-compose.yml` (mirror an existing block).
2. Add its database to `scripts/init-databases.sql` if it needs a Postgres schema.
3. Create the API client methods in `apps/api/src/integrations/clients/self-hosted-clients.service.ts` (mirroring the `calCom*`, `chatwoot*`, `formbricks*`, `listmonk*`, `documenso*` families).
4. Add a webhook handler in `apps/api/src/integrations/integrations-webhook.controller.ts`.
5. Document the new endpoint in [Webhooks](./webhooks.md) and [Third-Party & Self-Hosted Integrations](./third-party-integrations.md).
6. Add a `*_WEBHOOK_SECRET` to `.env.example` and `scripts/generate-secrets.sh`.
7. Add a health check method and wire it into the `/integrations/health` aggregator.

## 8. Change Management

- Production deploys happen only through the CI/CD pipeline (see [Deployment & Scaling → CI/CD Pipeline](./deployment.md#cicd-pipeline)).
- Database migrations: prefer additive changes (new columns nullable, new tables empty, new indexes `CONCURRENTLY`). Prisma migrations ship with the API image.
- Configuration changes (env vars): update `.env` in the secrets store first; orchestrator reads new value on next deploy.
- Backwards-incompatible API changes: gate behind feature flags (`ENABLE_AI_FEATURES`, `ENABLE_CONTENT_STUDIO`, `ENABLE_ANALYTICS`) for at least one release.

## 9. Contacts & Escalation

| Issue Severity | Response Time | Contact |
|---|---|---|
| P0 (all API down) | 15 min | Platform on-call |
| P1 (single workspace) | 1 hour | Platform team |
| P2 (analytics delayed) | 4 hours | Platform team |
| P3 (feature request) | Next sprint | Product team |

Runbook owner: Platform Engineering. Review quarterly.

	# Operations Runbook

	Day-to-day and incident-runbook guidance for MicProtocol in production. This is the central ops doc; per-service deep dives live in [Formbricks Runbook](./formbricks-runbook.md) and the relevant sections of [Self-Hosted Stack](./self-hosted-stack.md).

	## 1. Service Inventory

	For every container and the resource you need to keep healthy, see the catalog in [Self-Hosted Stack → Service Catalog](./self-hosted-stack.md#service-catalog). Collectively you operate:

	- Core infra: Postgres (`pgvector`), Redis, Meilisearch, MinIO, Mailhog (dev) / SMTP relay (prod)
	- MicProtocol apps: API, worker, CLIP scorer, mic-realtime, web, admin
	- Self-hosted integrations: Cal.com, Formbricks (5 containers), Documenso, Chatwoot (2 containers), Listmonk, Umami, LiveKit

	## 2. Standard Operating Procedures

	### 2.1 Daily checks

	```bash
	# Cluster health
	docker compose ps
	curl -s http://localhost:4000/health/detailed \| jq
	curl -s http://localhost:4000/integrations/health \| jq

	# Job queue depth
	docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:outreach:wait"
	docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:content:wait"
	docker exec micprotocol-redis redis-cli -a "$REDIS_PASSWORD" llen "bull:clip-scoring:wait"

	# API error rate (last hour)
	curl -s http://localhost:4000/tracing/metrics \| jq '.errorRate'
	```

	Thresholds:

	- Queue depth: alert at > 500 per queue for 15 min
	- API p95 latency: alert at > 500 ms for 5 min (read), > 2000 ms for 5 min (write/AI)
	- API error rate: alert at > 1% sustained

	### 2.2 Weekly maintenance window

	```bash
	# Pull latest images (out-of-band from CI deploys)
	docker compose pull

	# Roll each integration in isolation (see Self-Hosted Stack → Rolling update a single integration)
	docker compose up -d --no-deps calcom
	# Formbricks requires re-running migrations (see runbook)

	# Restart worker to release orphaned Redis locks
	docker compose restart worker

	# Vacuum Postgres
	docker exec micprotocol-postgres psql -U micprotocol -d micprotocol -c 'VACUUM ANALYZE;'

	# Check disk usage
	docker system df
	```

	### 2.3 Backup

	Daily backups are the minimum RPO target. Example cron against the live container:

	```bash
	# Snapshot all integration databases
	for db in micprotocol chatwoot calcom documenso formbricks listmonk umami; do
	docker exec micprotocol-postgres pg_dump -U micprotocol -Fc "$db" \| \
	gzip > "/backups/${db}_$(date +%Y%m%d).sql.gz"
	done

	# Retention: prune anything older than 30 days
	find /backups -name '*.sql.gz' -mtime +30 -delete
	```

	Test restore quarterly into a sandbox cluster — an untested backup is no backup.

	## 3. Health & Monitoring

	### 3.1 Health endpoints

	Every service exposes its own health check; see [Self-Hosted Stack → Health Endpoints](./self-hosted-stack.md#health-endpoints) for the full list. The single consolidated health check is:

	```bash
	curl -s http://localhost:4000/health/detailed
	```

	This pings DB, Redis, Meilisearch, and reports the API's own subsystem status. For self-hosted integration reachability:

	```bash
	curl -s http://localhost:4000/integrations/health
	```

	### 3.2 Recommended exporters

	\| Metric \| Source \| Alert on \|
	\|---\|---\|---\|
	\| Container up/down \| `docker compose ps` / orchestration \| `up == 0` for > 1 min \|
	\| API p95 latency \| `/tracing/metrics` or Datadog APM \| > 500 ms for 5 min \|
	\| API error rate \| `/tracing/metrics` \| > 1% sustained \|
	\| Queue depth \| Redis `LLEN bull:*:wait` \| > 500 pending per queue \|
	\| Postgres connections \| `pg_stat_activity` \| Active / max > 0.8 \|
	\| Redis memory \| `INFO memory` \| `used_memory_peak / maxmemory > 0.85` \|
	\| Webhook 401s \| API logs (`UnauthorizedException`) \| Spike rate \|
	\| CLIP scorer errors \| Worker logs \| Any timeout / subprocess non-zero exit \|

	### 3.3 Sentry integration

	Set `SENTRY_DSN` to enable error grouping and release tracking. Optional sampling:

	```
	SENTRY_TRACE_SAMPLE_RATE=0.1 # 10% of traces
	SENTRY_PROFILE_SAMPLE_RATE=0.05 # 5% of profiles
	```

	## 4. Incident Response

	### 4.1 Severity definitions

	\| Severity \| Definition \| Response time \| Owner \|
	\|---\|---\|---\|---\|
	\| P0 \| All API unavailable, no bookings, no recordings \| 15 min \| Platform on-call \|
	\| P1 \| Single workspace affected, or one integration down \| 1 hour \| Platform team \|
	\| P2 \| Analytics delayed, non-critical background jobs backed up \| 4 hours \| Platform team \|
	\| P3 \| Feature request / cosmetic \| Next sprint \| Product team \|

	### 4.2 P0 playbook — API down

	1. `curl -s http://localhost:4000/health/detailed` — confirms scope (DB, Redis, search).
	2. `docker compose ps` — for any container not in `Up` state.
	3. Tail API logs: `docker compose logs -f --tail=200 api`.
	4. Restart in dependency order: `postgres` → `redis` → `meilisearch` → `api` → `worker`.
	5. If DB is the issue, fail back to the latest snapshot (see §2.3).
	6. Confirm recovery via `/health/detailed` and a synthetic agent sweep.
	7. File an incident report within 24 hours; review in the next retro.

	### 4.3 P0 playbook — recording sessions failing (LiveKit)

	1. `curl -s http://localhost:4000/integrations/health` — confirms LiveKit reachable.
	2. `docker logs micprotocol-livekit` — check for ICE/UDP port binding errors.
	3. Confirm UDP `7882` is open end-to-end (firewall, security group, NAT).
	4. Verify `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET` match the configured keys inside the LiveKit container.
	5. For U-Fiber/nat'd networks, configure `rtc.use_external_ip: true` or run behind a TURN server.

	### 4.4 P1 playbook — webhook storm

	Symptom: API CPU pegged, `UnauthorizedException` or `IntegrationsWebhookController` log spam.

	1. `docker compose logs -f --tail=200 api \| grep IntegrationsWebhookController` — identifies the offender.
	2. Confirm secret rotation hasn't broken verification (`*_WEBHOOK_SECRET` on both sides).
	3. If a provider is redelivering failed events in a tight loop, temporarily pause redelivery upstream (Formbricks / Cal.com / Listmonk admin UI).
	4. Refresh the `WebhookIdempotencyService` cache if the duplicate-detection rate has dropped to zero.
	5. Restore redelivery once root cause is fixed.

	### 4.5 P1 playbook — clip-scoring stuck

	Symptom: clips stuck in "Scoring…" badge indefinitely.

	1. Scoring runs inside the worker container via a Python subprocess — `docker ps \| grep micprotocol-worker` confirms the process is up.
	2. `docker logs micprotocol-worker` — look for model download failures or FFmpeg errors.
	3. Orphaned Redis distributed locks self-expire with the 5-minute TTL; restarting the worker clears them immediately.
	4. Re-queue affected clips via the Content Studio UI.

	See [CLIP Scoring Engine → Troubleshooting](./clip-scoring.md#troubleshooting) for full detail.

	### 4.6 P0 playbook — Postgres corruption

	Recovery point objective (RPO): 24 hours (daily backups). Recovery time objective (RTO): 2 hours.

	1. Provision a new Postgres instance.
	2. Restore the latest snapshot: `gunzip -c micprotocol_latest.sql.gz \| psql -h new-pg -U micprotocol micprotocol`.
	3. Re-run schema migrations to ensure fresh state.
	4. Update `DATABASE_URL` and restart the API.
	5. Verify data integrity via Prisma Studio + synthetic agent sweep.

	Per-integration recovery: Formbricks, Chatwoot, Cal.com etc. each have their own DB. Restore only what's affected — restoring one integration's DB from a 12-hour-old snapshot does not compromise the others.

	## 5. Performance Tuning

	### 5.1 Postgres

	```sql
	-- Indexes for the heaviest MicProtocol queries (already in schema, audit periodically)
	-- guests(podcast_id, status), guests(podcast_id, priority_score)
	-- outreach_sequences(guest_id, status)
	-- bookings(start_time, end_time)
	-- asset_requests(episode_id, type)

	-- Slow query log
	ALTER SYSTEM SET log_min_duration_statement = 200;
	SELECT pg_reload_conf();
	```

	### 5.2 Redis

	Production Redis with BullMQ should not share its memory budget with caching:

	```yaml
	# docker-compose override
	command: redis-server --maxmemory 1gb --maxmemory-policy noeviction --requirepass "${REDIS_PASSWORD}"
	```

	`noeviction` ensures BullMQ job metadata is never evicted; for caching keys, prefer a separate Redis instance.

	### 5.3 CLIP scorer

	CLIP is CPU-bound; GPU optional but cuts per-clip time by 5–10×. See [CLIP Scoring → Scaling Considerations](./clip-scoring.md) and [Deployment → Scaling Considerations for CLIP Scorer](./deployment.md#scaling-considerations-for-clip-scorer).

	### 5.4 Meilisearch

	Re-indexing is cheap; if search latency regresses after a large guest import, trigger a re-index via the API or `curl -X POST http://localhost:7700/indexes/guests/update`.

	## 6. Scaling Triggers

	See [Deployment & Scaling → Scaling Triggers](./deployment.md#scaling-triggers) for the full phase-by-phase plan. Operators should plan capacity against the documented phase triggers rather than waiting for alerts.

	\| Phase \| Trigger \| Architectural change \|
	\|---\|---\|---\|
	\| 1 (≤1K podcasts) \| Launch \| Single ECS task per service; RDS `t4g.micro`; ElastiCache Serverless \|
	\| 2 (1–5K podcasts) \| P95 > 500ms OR CPU > 70% sustained \| Split workers from API; dedicated Redis for BullMQ vs. cache; RDS read replica \|
	\| 3 (5–20K podcasts) \| Email > 100K/day OR queue depth > 1K \| Extract Outreach Domain to microservice; migrate orchestration to Temporal.io \|
	\| 4 (20K+ podcasts) \| Cross-domain query > 2s P95 \| Full microservices split; NATS JetStream event bus; per-service RDS \|

	## 7. Common Operations

	### 7.1 Rotate a webhook secret

	```bash
	NEW=$(openssl rand -hex 32)
	# Update .env: CALCOM_WEBHOOK_SECRET=${NEW}
	# Update Cal.com Admin UI: Settings → Webhooks → secret
	docker compose restart api
	# Tail logs to confirm webhooks still pass verification
	docker compose logs -f api \| grep -i webhook
	```

	### 7.2 Add a new workspace for a customer

	Synthetic agent seed: [`apps/api/prisma/seed-agents.ts`](../apps/api/prisma/seed-agents.ts) is the reference for deterministic workspace creation. Use the same pattern via tRPC `workspace.create` with the customer's Clerk user ID.

	### 7.3 Run a synthetic agent sweep manually

	```bash
	pnpm db:seed:agents # reset seed (optional)
	pnpm agents:orchestrator # requires API running on :4000
	cat scripts/synthetic-agents/test-report.txt
	jq '.[] \| select(.severity == "high")' scripts/synthetic-agents/weaknesses.json
	```

	### 7.4 Onboard a new self-hosted integration

	1. Add the service block to `docker-compose.yml` (mirror an existing block).
	2. Add its database to `scripts/init-databases.sql` if it needs a Postgres schema.
	3. Create the API client methods in `apps/api/src/integrations/clients/self-hosted-clients.service.ts` (mirroring the `calCom`, `chatwoot`, `formbricks`, `listmonk`, `documenso*` families).
	4. Add a webhook handler in `apps/api/src/integrations/integrations-webhook.controller.ts`.
	5. Document the new endpoint in [Webhooks](./webhooks.md) and [Third-Party & Self-Hosted Integrations](./third-party-integrations.md).
	6. Add a `*_WEBHOOK_SECRET` to `.env.example` and `scripts/generate-secrets.sh`.
	7. Add a health check method and wire it into the `/integrations/health` aggregator.

	## 8. Change Management

	- Production deploys happen only through the CI/CD pipeline (see [Deployment & Scaling → CI/CD Pipeline](./deployment.md#cicd-pipeline)).
	- Database migrations: prefer additive changes (new columns nullable, new tables empty, new indexes `CONCURRENTLY`). Prisma migrations ship with the API image.
	- Configuration changes (env vars): update `.env` in the secrets store first; orchestrator reads new value on next deploy.
	- Backwards-incompatible API changes: gate behind feature flags (`ENABLE_AI_FEATURES`, `ENABLE_CONTENT_STUDIO`, `ENABLE_ANALYTICS`) for at least one release.

	## 9. Contacts & Escalation

	\| Issue Severity \| Response Time \| Contact \|
	\|---\|---\|---\|
	\| P0 (all API down) \| 15 min \| Platform on-call \|
	\| P1 (single workspace) \| 1 hour \| Platform team \|
	\| P2 (analytics delayed) \| 4 hours \| Platform team \|
	\| P3 (feature request) \| Next sprint \| Product team \|

	Runbook owner: Platform Engineering. Review quarterly.