Blame for docs/self-hosted-stack.md

# Self-Hosted Stack

MicProtocol ships a complete self-hostable integration fleet in `docker-compose.yml`. Every service replaces a paid SaaS equivalent, giving you maximum data ownership, no per-seat licensing, and zero vendor lock-in.

This doc is the unified operations reference. For per-integration API behavior and webhook side effects, see [Third-Party & Self-Hosted Integrations](./third-party-integrations.md) and [Webhooks](./webhooks.md).

## Architecture

```mermaid
graph TD
    API[MicProtocol NestJS API :4000] --> POSTGRES[(PostgreSQL pgvector :5432)]
    API --> REDIS[(Redis :6379)]
    API --> MEILI[(Meilisearch :7700)]
    API --> MINIO[(MinIO :9001)]
    API --> MICRT[mic-realtime :4001]

    API -. webhook .-> CALCOM[Cal.com :3002]
    API -. webhook .-> FORMBRICKS[Formbricks stack :3001]
    API -. webhook .-> CHATWOOT[Chatwoot :3005]
    API -. webhook .-> DOCUMENSO[Documenso :3004]
    API -. webhook .-> LISTMONK[Listmonk :9000]
    API -. webhook .-> LIVEKIT[LiveKit :7880]

    CALCOM --> POSTGRES
    FORMBRICKS --> POSTGRES
    CHATWOOT --> POSTGRES
    DOCUMENSO --> POSTGRES
    LISTMONK --> POSTGRES
    UMAMI --> POSTGRES

    CALCOM --> REDIS
    CHATWOOT --> REDIS
    FORMBRICKS --> REDIS

    CALCOM --> MINIO
    DOCUMENSO --> MINIO
    FORMBRICKS --> MINIO

    MAILHOG[Mailhog :8025] -. SMTP .-> ALL[All services]

    style API fill:#e1f5e1
    style MICRT fill:#fff3e0
```

## Service Catalog

### Core Infrastructure

| Service | Container | Image | Port(s) | Database |
|---|---|---|---|---|
| PostgreSQL | `micprotocol-postgres` | `pgvector/pgvector:pg16` | 5432 | All databases (via `init-databases.sql`) |
| Redis | `micprotocol-redis` | `redis:7-alpine` | 6379 | n/a (in-memory) |
| Meilisearch | `micprotocol-meilisearch` | `getmeili/meilisearch:v1.6` | 7700 | n/a (own volume) |
| MinIO (S3 storage) | `micprotocol-minio` | `minio/minio:latest` | 9001 (API), 9090 (Console) | n/a (own volume) |
| MinIO bucket creator | `micprotocol-minio-mc` | `minio/mc:latest` | - (one-shot) | n/a |
| Mailhog (dev SMTP) | `micprotocol-mailhog` | `mailhog/mailhog:latest` | 1025 (SMTP), 8025 (UI) | n/a |

### MicProtocol Apps

| Service | Container | Port | Notes |
|---|---|---|---|
| API | `micprotocol-api` | 4000 | tRPC + REST + Swagger at `/api/docs` |
| Background worker | `micprotocol-worker` | - | Same app image as the API (no published ports); runs BullMQ WorkerHosts incl. the Python clip-scoring subprocess |
| mic-realtime (Erlang/OTP) | `micprotocol-mic-realtime` | 4001 | Chat fanout + presence |
| Web | `micprotocol-web` | 3000 | Next.js 14 |
| Admin | `micprotocol-admin` | 3006 | Next.js platform admin |

### Self-Hosted Integrations

| Service | Container(s) | Port | Replacement | DB schema |
|---|---|---|---|---|
| Cal.com | `micprotocol-calcom` | 3002 | Calendly | `calcom` |
| Formbricks v4 | `micprotocol-formbricks` + `-hub` + `-cube` + 2 migrate jobs | 3001 (web), 8080 (hub), 4000 (cube) | Typeform / SurveyMonkey | `formbricks` (pgvector) |
| Documenso | `micprotocol-documenso` | 3004 | DocuSign / PandaDoc | `documenso` |
| Chatwoot | `micprotocol-chatwoot-web` + `-worker` | 3005 | Intercom / Zendesk | `chatwoot` (pgvector) |
| Listmonk | `micprotocol-listmonk` | 9000 | Mailchimp / ConvertKit | `listmonk` |
| Umami | `micprotocol-umami` | 3003 | Google Analytics / Plausible cloud | `umami` |
| LiveKit | `micprotocol-livekit` | 7880 (HTTP), 7881 (TCP), 7882 (UDP) | Twilio Video / Zoom SDK | n/a |

### Formbricks v4 substack

```mermaid
graph LR
    MIGRATE1[migrate job] --> DB[(Postgres formbricks DB)]
    MIGRATE2[hub migrate job] --> DB
    HUB[formbricks-hub :8080] --> DB
    HUB --> CUBE[cube.js :4000]
    CUBE --> DB
    WEB[formbricks :3001] --> HUB
    WEB --> CUBE
    WEB --> DB
    WEB --> REDIS[(Redis)]
```

Containers in the Formbricks substack:

1. `micprotocol-formbricks-migrate` (one-shot, runs Prisma migrations)
2. `micprotocol-formbricks` (web app, `:3001`)
3. `micprotocol-formbricks-hub-migrate` (one-shot, runs goose + river migrations)
4. `micprotocol-formbricks-hub` (Hub API, `:8080`)
5. `micprotocol-formbricks-cube` (Cube.js analytics, `:4000`)

Formbricks-specific runbook: [`formbricks-runbook.md`](./formbricks-runbook.md).

## Port Map

Fixed ports in `docker-compose.yml` — change in `docker-compose.yml` and matching env vars together:

| Port | Service |
|---|---|
| 3000 | Web (also guest portal) |
| 3001 | Formbricks |
| 3002 | Cal.com |
| 3003 | Umami |
| 3004 | Documenso |
| 3005 | Chatwoot |
| 3006 | Admin |
| 4000 | API |
| 4001 | mic-realtime |
| 5432 | Postgres |
| 6379 | Redis |
| 7700 | Meilisearch |
| 7880 | LiveKit HTTP |
| 7881 | LiveKit TCP |
| 7882 | LiveKit UDP |
| 8025 | Mailhog UI |
| 9000 | Listmonk |
| 9001 | MinIO API |
| 9090 | MinIO Console |
| 1025 | Mailhog SMTP |

## Database Layout

`init-databases.sql` runs on first Postgres boot and creates one database per integration:

| Database | Owner | pgvector | pgcrypto |
|---|---|---|---|
| `micprotocol` (default) | MicProtocol API | - | yes |
| `chatwoot` | Chatwoot | yes | yes |
| `formbricks` | Formbricks | yes | yes |
| `calcom` | Cal.com | - | yes |
| `documenso` | Documenso | - | yes |
| `listmonk` | Listmonk | - | yes |
| `umami` | Umami | - | yes |

If a volume already exists, `init-databases.sql` will not run. To bootstrap missing databases on an existing volume:

```bash
docker exec -i micprotocol-postgres psql -U micprotocol -d micprotocol < scripts/init-databases.sql
```

## Shared Object Storage (MinIO)

MinIO is the default storage backend for every self-hosted integration that uploads files. `minio-createbuckets` is a one-shot service that creates:

| Bucket | Owner |
|---|---|
| `micprotocol-storage` | MicProtocol API (media, headshots, clips) |
| `calcom-uploads` | Cal.com |
| `documenso-uploads` | Documenso |
| `formbricks-uploads` | Formbricks |

`micprotocol-storage` is set to public-read (used for guest-facing media). The others are private.

In production, point MicProtocol at Cloudflare R2 instead of MinIO by changing `R2_ENDPOINT`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_BUCKET`, and `STORAGE_PUBLIC_URL` — no code changes required, both speak the S3 API.

## Shared SMTP (Mailhog in dev)

Every self-hosted integration sends outbound mail through the same SMTP settings:

```
SMTP_HOST=mailhog          # in compose network
SMTP_PORT=1025
SMTP_USER=                 # empty for Mailhog
SMTP_PASSWORD=             # empty for Mailhog
EMAIL_FROM=noreply@<yourdomain>
```

In production, swap Mailhog for your transactional relay (Resend, SES, Postmark, or self-hosted Postal/Haraka) by setting `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`, and `EMAIL_FROM` to the provider's values. The integrations read these env vars directly — no code changes.

## Shared Redis (namespaced)

Chatwoot, Formbricks, and Cal.com all share the `micprotocol-redis` instance. Chatwoot and Cal.com namespace keys internally; Formbricks uses Redis for rate-limiting and Cube.js queue. There is currently no isolation beyond provider namespacing — for high-throughput production, split into one Redis per integration.

## Co-located Secrets

Every self-hosted service reads its secrets from environment variables that the API also needs. The complete list lives in [`.env.example`](../.env.example). Critical guidance:

| Category | Examples | Rotation policy |
|---|---|---|
| Postgres / Redis passwords | `POSTGRES_PASSWORD`, `REDIS_PASSWORD` | Rotatable; requires service restart |
| Encryption keys (immutable) | `FORMBRICKS_ENCRYPTION_KEY`, `DOCUMENSO_ENCRYPTION_KEY`, `CALCOM_ENCRYPTION_KEY`, `NEXTAUTH_SECRET` variants | Never rotate after first start; old encrypted data becomes unrecoverable |
| Webhook signatures | `*_WEBHOOK_SECRET` | Rotatable; update on both sides simultaneously |
| Admin / API tokens | `CALCOM_API_KEY`, `LISTMONK_API_TOKEN`, `CHATWOOT_API_KEY` | Rotatable via the admin UI of each integration |
| App secrets | `UMAMI_APP_SECRET`, `FORMBRICKS_CRON_SECRET` | Rotatable |

Use `bash scripts/generate-secrets.sh` to produce strong random values for every variable in one shot. See [Getting Started → Environment Configuration](./getting-started.md#3-environment-configuration) for details.

## Service-Specific Configuration Notes

### Cal.com

- Uses its own `calcom` database (Prisma migrations need a clean schema).
- `NEXT_PUBLIC_SINGLE_ORG_SLUG=micprotocol` and `ORGANIZATIONS_ENABLED=0` disable paid multi-org features for self-hosted single-tenant use.
- Generate an API key in Cal.com Settings → API Keys; set `CALCOM_API_KEY` in `.env`.
- Encrypts DB columns with `CALCOM_ENCRYPTION_KEY` (24-char hex, **never rotate** after init).

### Formbricks v4

- Five containers, must start in order: `formbricks-migrate` → `formbricks-hub-migrate` → `formbricks-cube` → `formbricks-hub` → `formbricks`. Compose `depends_on` chains handle this automatically.
- `SKIP_STARTUP_MIGRATION=true` prevents the web container from re-migrating (the dedicated migrate job already did it).
- `SKIP_STARTUP_MIGRATION` must remain `true`; running migrations inside the web container will race with the migrate job.
- Analytics require Cube.js (`CUBEJS_API_SECRET` must match between web and cube containers).
- Sign-up is disabled without an invite (`EMAIL_VERIFICATION_DISABLED=1`).

### Documenso

- Requires a signing certificate. Generate with the commands in [`scripts/documenso-cert/README.md`](../scripts/documenso-cert/README.md).
- Mount the cert directory read-only at `/opt/documenso/cert`; the app looks for `cert.p12` inside.
- `DOCUMENSO_ENCRYPTION_KEY` and `DOCUMENSO_ENCRYPTION_SECONDARY_KEY` are both required and **never rotated**.

### Chatwoot

- Rails + Sidekiq worker pair (`chatwoot-web` + `chatwoot-worker`).
- Uses the `chatwoot` database with `pgvector` (required by Chatwoot itself).
- Storage is local (`RAILS_STORAGE_LOCATION=/app/storage`, mounted as `chatwoot_storage` volume). For larger deployments, switch `ACTIVE_STORAGE_SERVICE` to S3/MinIO.
- `FRONTEND_URL` must match the public URL or the live-chat widget will fail to load.
- Generate `SECRET_KEY_BASE` with `openssl rand -hex 64`.

### Listmonk

- Container runs `./listmonk --install --idempotent --yes && ./listmonk --upgrade --yes && ./listmonk --config ''` on first boot — idempotent, safe on restarts.
- `LISTMONK_db__database=listmonk` (must match the name in `init-databases.sql`).
- API token is generated in Listmonk Admin → Settings → API; set `LISTMONK_API_TOKEN` in `.env`.
- `LISTMONK_ADMIN_USER`/`LISTMONK_ADMIN_PASSWORD` bootstrap the super-admin on first run.

### Umami

- Single container, Postgres-backed.
- `UMAMI_APP_SECRET` is required; generate with `openssl rand -base64 32`.
- Add a website in the Umami UI and copy the website ID to `NEXT_PUBLIC_UMAMI_WEBSITE_ID`.

### LiveKit

- SFU for the in-app recording studio. MicProtocol owns the browser client (`apps/web`); LiveKit owns the WebRTC media routing.
- Dev defaults (`devkey` / `devsecret`) work in Compose but **must be changed for production**.
- Webhooks from LiveKit back to the API call `POST /integrations/webhooks/livekit` and trigger `RecordingSession` state transitions.
- For internet-facing deployments, expose UDP `7882` and configure `rtc.use_external_ip: true` (or run behind a TURN server).

### mic-realtime

Erlang/OTP service that owns:

- Chat fanout (per-workspace conversation messages)
- Presence (who is online in a workspace)
- Live domain events (published by the NestJS API via `/internal/dispatch`)

Key env vars:

| Var | Default | Notes |
|---|---|---|
| `MIC_REALTIME_PORT` | 4001 | HTTP listener |
| `MIC_REALTIME_HOST` | 0.0.0.0 | Bind address |
| `MICPROTOCOL_API_URL` | http://api:4000 | Used by mic-realtime to call back into the API |
| `MIC_REALTIME_INTERNAL_TOKEN` | (change me) | Bearer used by API to publish events |
| `MIC_REALTIME_SHARED_SECRET` | (change me) | JWT signing secret for WS auth tokens |
| `MIC_REALTIME_PUBLIC_URL` | ws://localhost:4001 | Public WS URL handed to web clients |
| `MIC_REALTIME_MAX_MSG` | 65536 | Max message size in bytes |
| `MIC_REALTIME_RATE_LIMIT` | 120 | Messages per minute per connection |
| `MIC_REALTIME_PRESENCE_TTL` | 90 | Presence TTL (seconds) |

Both `MIC_REALTIME_INTERNAL_TOKEN` and `MIC_REALTIME_SHARED_SECRET` must be strong random strings in production.

#### Production deployment

`mic-realtime` ships in every deployment topology:

| Topology | Where | Public route |
|---|---|---|
| Local dev | `docker-compose.yml` (service `mic-realtime`) | `ws://localhost:4001` |
| Coolify | `docker-compose.coolify.yml` (service `mic-realtime`, internal `:4001`) | `wss://realtime.<root>` via Caddy |
| Render | `render.yaml` (`micprotocol-realtime`, `rootDir: services/mic-realtime`) | `wss://micprotocol-realtime.onrender.com` (or custom domain) |

The [Caddyfile](../Caddyfile) routes `realtime.<root>` to `mic-realtime:4001`
(the specific host block wins over the `*.` wildcard). Caddy handles the
WebSocket upgrade transparently on the same route, so one URL serves both the
`GET /healthz` probe and `GET /ws?token=...`.

Secrets are **fail-closed**: `RealtimeClient`
(`apps/api/src/chat/realtime.client.ts`) and `mic_config.erl` both refuse to
start in production when `MIC_REALTIME_INTERNAL_TOKEN` /
`MIC_REALTIME_SHARED_SECRET` are missing or equal to the known defaults.
Generate strong values (`openssl rand -hex 32`) and set
`MIC_REALTIME_PUBLIC_URL` to the public `wss://` URL; the web/admin frontends
receive it as `NEXT_PUBLIC_MIC_REALTIME_URL`. The Coolify stack defaults the
API→realtime base URL to the internal `http://mic-realtime:4001`
(`MIC_REALTIME_URL`), so API→realtime callbacks never leave the Docker
network.

> ⚠️ Render **free** instances sleep after ~15 min of inactivity and cap at
> 750 hours/month — a sleeping WebSocket engine drops every live chat/presence
> socket. Use `plan: starter` or higher for realtime in production.

## Start / Stop / Update

### Bring up the full stack

```bash
docker compose up -d
```

### Bring up only infra (for local app dev)

```bash
docker compose up -d postgres redis meilisearch minio minio-createbuckets
```

### Rolling update a single integration

```bash
docker compose pull formbricks
docker compose up -d --no-deps formbricks
```

Formbricks v4 migrations need the migrate jobs to re-run after a version bump:

```bash
docker compose up formbricks-migrate
docker compose up formbricks-hub-migrate
docker compose up -d --no-deps formbricks formbricks-hub formbricks-cube
```

### Tear down

```bash
docker compose down            # keep data
docker compose down -v         # wipe all volumes (destructive)
```

## Health Endpoints

| Service | Endpoint |
|---|---|
| API | `GET http://localhost:4000/health` |
| Cal.com | `GET http://localhost:3002/api/health` |
| Formbricks web | `GET http://localhost:3001/api/health` |
| Formbricks Hub | `GET http://localhost:8080/health` |
| Formbricks Cube | `GET http://localhost:4000/formbricks/readyz` (via cube container) |
| Chatwoot | `GET http://localhost:3005/health` |
| Listmonk | `GET http://localhost:9000/health` |
| Documenso | `GET http://localhost:3004/api/health` |
| Umami | `GET http://localhost:3003/api/health` |
| MinIO | `GET http://localhost:9001/minio/health/live` |
| Meilisearch | `GET http://localhost:7700/health` |
| mic-realtime | `GET http://localhost:4001/healthz` |
| LiveKit | `GET http://localhost:7880/health` (HTTP only; UDP port 7882 for WebRTC) |
| Mailhog | UI at `:8025` always returns 200 |

The API exposes a consolidated integration health check at:

```bash
curl -s http://localhost:4000/integrations/health | jq
```

This pings each self-hosted service's health endpoint in parallel and reports reachability.

### Integration Management Dashboard

The Admin panel (`apps/admin`) provides a full management interface for all self-hosted integrations:

| Feature | Description |
|---|---|
| **Health Monitoring** | Real-time health status, latency, and version for Chatwoot, Cal.com, Formbricks, Listmonk, Documenso, Umami, and LiveKit |
| **Webhook Secret Management** | View and rotate shared webhook secrets directly from the admin panel |
| **Admin Panel Links** | Direct "Open Admin" buttons that open each integration's native admin UI in a new tab |
| **Credential Management** | Connect, update, or disconnect API keys for self-hosted integrations |
| **Webhook Delivery Log** | Recent webhook deliveries with status, latency, and error details |
| **Provider Metrics** | Per-provider request counts, average latency, and error rates |

Access these features from the **Monitoring** and **Integrations** tabs in the admin navigation sidebar.

## Resource Limits

Default `deploy.resources.limits` in `docker-compose.yml`:

| Service | CPUs | Memory |
|---|---|---|
| API | 2 | 2G |
| Worker | 2 | 2G |
| mic-realtime | 1.5 | 1G |

Other services have no explicit limits — tune for your hardware. CLIP scoring (Python ML subprocess) runs inside the worker and is by far the most CPU- and memory-hungry: see [CLIP Scoring Engine](./clip-scoring.md) for scale guidance.

## Alternatives if You Skip Self-Hosting

If you don't want to run a particular self-hosted service, simply remove its block from `docker-compose.yml` and omit its env vars from `.env`. The MicProtocol API degrades gracefully — each integration's API client is called only when its corresponding URL env var is set. Missing integrations return `false` from the health check and the API logs a warning rather than 500-ing.

For the externally-hosted (non-self-hosted) integrations (Clerk, Resend, Cloudflare R2, NVIDIA NIM, DeepSeek, Moonshot, Stripe, Paystack, M-Pesa, Africa's Talking, Twilio, Google/Outlook calendars), see [Third-Party & Self-Hosted Integrations](./third-party-integrations.md).