Blame for docs/getting-started.md

# Getting Started

This guide takes you from a clean machine to a running MicProtocol stack with all self-hosted integrations. Read end-to-end on first install; use the checklists for re-installs.

## 1. Prerequisites

| Tool | Version | Purpose |
|---|---|---|
| Node.js | >= 20.0.0 | API, Web, Admin, Mobile tooling |
| pnpm | >= 9.15.0 (locked to `pnpm@10.30.3`) | Workspace package manager |
| Docker + Docker Compose | latest stable | Infra (Postgres, Redis, Meilisearch, integrations) |
| Python | >= 3.11 | CLIP scoring engine |
| uv | latest | Python package manager for `services/clip-scorer` |
| FFmpeg | latest | Audio/video frame extraction (also installed in API image) |
| PostgreSQL client | 15+ | Optional local DB without Docker |
| Redis CLI | 7+ | Optional local Redis without Docker |
| podman + podman-compose | latest | Required only for `pnpm agents:run` |

Install uv:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

## 2. Repository

```bash
git clone <repo-url> micprotocol
cd micprotocol
```

The repo is a Turborepo monorepo. Top-level layout:

```
micprotocol/
├── apps/
│   ├── api/        NestJS backend (tRPC + REST + Socket.io)
│   ├── web/        Next.js 14 host dashboard + guest portal
│   ├── admin/      Next.js platform-admin app
│   └── mobile/     Expo React Native app
├── packages/
│   ├── shared-types/   Zod schemas + domain enums
│   ├── ui/             Cross-platform UI primitives
│   ├── tsconfig/       Shared TS configs
│   └── eslint-config/  Shared ESLint config
├── services/
│   ├── clip-scorer/  Python ML engine (OpenCLIP + pyloudnorm + bart-large-mnli)
│   └── mic-realtime/ Erlang/OTP chat fanout + presence
├── scripts/
│   ├── init-databases.sql          Per-integration Postgres databases
│   ├── generate-secrets.sh         Generates all self-hosted secrets
│   ├── documenso-cert/README.md    How to generate the Documenso signing cert
│   └── synthetic-agents/           Synthetic agent test framework
├── docs/             This documentation set
├── docker-compose.yml           Production Docker Compose (full stack)
├── docker-compose.coolify.yml   Coolify override
├── turbo.json        Turborepo pipeline
└── pnpm-workspace.yaml
```

## 3. Environment Configuration

```bash
cp .env.example .env
cp .env.example.coolify .env.coolify   # only if deploying via Coolify
```

Generate strong secrets for the self-hosted integrations:

```bash
bash scripts/generate-secrets.sh > .env.production.formbricks
# Manually merge generated values into your .env
```

`scripts/generate-secrets.sh` produces:

- Core infrastructure (Postgres, Redis, Meilisearch, MinIO passwords)
- Encryption keys (never rotate after initial setup)
- Chatwoot, Cal.com, Formbricks, Documenso, Listmonk secrets
- LiveKit API key/secret/webhook secret
- Umami app secret

For each `*_ENCRYPTION_KEY` and `*_NEXTAUTH_SECRET`, treat them as immutable after first start — rotating will invalidate sessions and unrecoverable encrypted data.

### Generate the Documenso signing certificate (one-time)

```bash
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
  -days 365 -nodes -subj "/CN=MicProtocol Documenso"
openssl pkcs12 -export -out scripts/documenso-cert/cert.p12 \
  -inkey key.pem -in cert.pem \
  -passout pass:"$(openssl rand -hex 32)"
rm key.pem cert.pem
```

Set `DOCUMENSO_SIGNING_PASSPHRASE` in `.env` to the passphrase used above. Production should use a CA-issued cert, never the self-signed one.

## 4. First Boot — Full Stack (Docker)

```bash
# 4.1. Start core infra (Postgres, Redis, Meilisearch)
docker compose up -d postgres redis meilisearch

# 4.2. Generate Prisma client and push schema
pnpm db:generate
pnpm db:push

# 4.3. Start the entire stack (API + worker + CLIP scorer + web + admin
#       + all self-hosted integrations + realtime)
docker compose up -d

# 4.4. Verify
docker compose ps
curl -s http://localhost:4000/health
```

Default ports:

| Service | Port | URL |
|---|---|---|
| Web (host dashboard + guest portal) | 3000 | http://localhost:3000 |
| Admin | 3006 | http://localhost:3006 |
| API | 4000 | http://localhost:4000 |
| Swagger (API) | 4000 | http://localhost:4000/api/docs |
| Meilisearch | 7700 | http://localhost:7700 |
| mic-realtime (WS) | 4001 | ws://localhost:4001 |
| Cal.com | 3002 | http://localhost:3002 |
| Formbricks | 3001 | http://localhost:3001 |
| Umami | 3003 | http://localhost:3003 |
| Documenso | 3004 | http://localhost:3004 |
| Chatwoot | 3005 | http://localhost:3005 |
| Listmonk | 9000 | http://localhost:9000 |
| MinIO API | 9000 (internal) | http://localhost:9001 |
| MinIO Console | 9090 | http://localhost:9090 |
| LiveKit | 7880 (HTTP), 7881 (TCP), 7882 (UDP) | http://localhost:7880 |
| Mailhog UI | 8025 | http://localhost:8025 |
| Mailhog SMTP | 1025 | n/a |

## 5. First Boot — Local Dev (No Docker for App Code)

Useful when iterating on the API or web app and you have the infra running in Docker.

```bash
# 5.1. Install all JS deps
pnpm install

# 5.2. Install Python deps for CLIP scorer
cd services/clip-scorer
uv sync
cd ../..

# 5.3. Start only infra in Docker
docker compose up -d postgres redis meilisearch

# 5.4. Generate Prisma client and push schema
pnpm db:generate
pnpm db:push

# 5.5. Start all apps in dev mode
pnpm dev
```

The Turborepo pipeline runs `api`, `web`, `admin` (if configured), and any app with a `dev` script. Hot reload is enabled for both the API (NestJS) and Web (Next.js).

## 6. Verifying the Stack

### API

```bash
# Liveness
curl -s http://localhost:4000/health

# Detailed
curl -s http://localhost:4000/health/detailed

# Readiness (K8s style)
curl -s http://localhost:4000/health/ready
```

### Database

```bash
pnpm db:studio   # opens Prisma Studio on http://localhost:5555
```

### Self-hosted integrations

After first boot, complete these one-time setup steps inside each UI:

| Service | Action |
|---|---|
| Cal.com | Create org `micprotocol`, generate an API key in Settings → API Keys; set `CALCOM_API_KEY` in `.env` and restart API |
| Formbricks | Create Environment, copy ID to `NEXT_PUBLIC_FORMBRICKS_ENVIRONMENT_ID`; generate API key in Settings → API Keys |
| Listmonk | Log in with `LISTMONK_ADMIN_USER`/`LISTMONK_ADMIN_PASSWORD`, generate API token in Settings → API Tokens |
| Documenso | Upload signing cert (already generated in step 3) and configure SMTP |
| Chatwoot | Create user, create inbox (Website), copy inbox ID and website token to `.env` |
| Umami | Add website, copy website ID to `NEXT_PUBLIC_UMAMI_WEBSITE_ID` |
| MinIO | Log in with `MINIO_ROOT_USER`/`MINIO_ROOT_PASSWORD`; buckets are auto-created by `minio-createbuckets` |

### Webhook smoke tests

Each self-hosted integration has a webhook the API consumes. Use the Integration Health controller to confirm reachability:

```bash
curl -s http://localhost:4000/integrations/health | jq
```

See [Webhooks](./webhooks.md) for the full endpoint catalog and signature headers.

## 7. Running Tests

### API unit tests (Jest)

```bash
pnpm --filter @micprotocol/api test
```

### API E2E tests

```bash
pnpm --filter @micprotocol/api test:e2e
```

### Synthetic agents (full-stack scenario sweep)

```bash
pnpm agents:run   # full pipeline (pod up + seed + API + scenarios)
pnpm agents:orchestrator   # orchestrator only (API must be running)
pnpm db:seed:agents        # seed only
```

Requires `podman` + `podman-compose`. Outputs:

- `scripts/synthetic-agents/test-report.txt`
- `scripts/synthetic-agents/weaknesses.json`

## 8. Common Day-to-Day Commands

| Task | Command |
|---|---|
| Start everything | `pnpm dev` (JS apps) or `docker compose up -d` (full stack) |
| Rebuild after schema change | `pnpm db:generate && pnpm db:push` |
| Create a real migration | `pnpm --filter @micprotocol/api prisma:migrate` |
| Open Prisma Studio | `pnpm db:studio` |
| Run lint across the monorepo | `pnpm lint` |
| Run type check | `pnpm type-check` |
| Build for production | `pnpm build` |
| Tail logs | `docker compose logs -f api worker clip-scorer web` |
| Restart a single service | `docker compose restart <name>` |
| Stop everything | `docker compose down` |
| Wipe volumes (destructive) | `docker compose down -v` |

## 9. Troubleshooting First Run

### Postgres won't start

`docker compose logs postgres` — most often it's a port conflict (`5432` already in use). Either stop the local Postgres or change the host port mapping in `docker-compose.yml`.

### `pnpm db:push` complains about missing tables for `chatwoot`, `calcom`, etc.

You must run the init script that creates per-integration databases. The `init-databases.sql` is mounted into the Postgres container on first boot only — if the volume already exists, run manually:

```bash
docker exec -i micprotocol-postgres psql -U micprotocol -d micprotocol < scripts/init-databases.sql
```

### API returns 500 on first hit

Check that `CLERK_SECRET_KEY` and `CLERK_PUBLISHABLE_KEY` are set; without them the auth guard rejects all requests. For local-only development without Clerk, see [Security & Compliance](./security.md) for the dev-mode `x-user-id` header bypass.

### CLIP scorer stuck in "Scoring…"

See [CLIP Scoring Engine → Troubleshooting](./clip-scoring.md#troubleshooting).

### Webhook 401s

Every webhook signature is verified with `crypto.timingSafeEqual()`. See [Security & Compliance](./security.md) and [Webhooks](./webhooks.md) for the exact header names per provider.

### Port already in use

The compose file uses fixed ports for every integration. Stop the conflicting process or remap the host port in `docker-compose.yml`.

## 10. Next Steps

- Read [Architecture](./architecture.md) for the bounded-context breakdown.
- Read [Self-Hosted Stack](./self-hosted-stack.md) before changing any container config.
- Read [Webhooks](./webhooks.md) before wiring a new integration.
- Read [Operations Runbook](./operations-runbook.md) before going to production.