Spaces:

Benaah
/

MicProtocol

Running

App Files Files Community

MicProtocol / docs /getting-started.md

Benaah

Upload folder using huggingface_hub

b17baae verified about 20 hours ago

preview code

Raw History Blame Contribute Delete

10.1 kB

# Getting Started

This guide takes you from a clean machine to a running MicProtocol stack with all self-hosted integrations. Read end-to-end on first install; use the checklists for re-installs.

## 1. Prerequisites

| Tool | Version | Purpose |
|---|---|---|
| Node.js | >= 20.0.0 | API, Web, Admin, Mobile tooling |
| pnpm | >= 9.15.0 (locked to `pnpm@10.30.3`) | Workspace package manager |
| Docker + Docker Compose | latest stable | Infra (Postgres, Redis, Meilisearch, integrations) |
| Python | >= 3.11 | CLIP scoring engine |
| uv | latest | Python package manager for `services/clip-scorer` |
| FFmpeg | latest | Audio/video frame extraction (also installed in API image) |
| PostgreSQL client | 15+ | Optional local DB without Docker |
| Redis CLI | 7+ | Optional local Redis without Docker |
| podman + podman-compose | latest | Required only for `pnpm agents:run` |

Install uv:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

## 2. Repository

```bash
git clone <repo-url> micprotocol
cd micprotocol
```

The repo is a Turborepo monorepo. Top-level layout:

```
micprotocol/
├── apps/
│   ├── api/        NestJS backend (tRPC + REST + Socket.io)
│   ├── web/        Next.js 14 host dashboard + guest portal
│   ├── admin/      Next.js platform-admin app
│   └── mobile/     Expo React Native app
├── packages/
│   ├── shared-types/   Zod schemas + domain enums
│   ├── ui/             Cross-platform UI primitives
│   ├── tsconfig/       Shared TS configs
│   └── eslint-config/  Shared ESLint config
├── services/
│   ├── clip-scorer/  Python ML engine (OpenCLIP + pyloudnorm + bart-large-mnli)
│   └── mic-realtime/ Erlang/OTP chat fanout + presence
├── scripts/
│   ├── init-databases.sql          Per-integration Postgres databases
│   ├── generate-secrets.sh         Generates all self-hosted secrets
│   ├── documenso-cert/README.md    How to generate the Documenso signing cert
│   └── synthetic-agents/           Synthetic agent test framework
├── docs/             This documentation set
├── docker-compose.yml           Production Docker Compose (full stack)
├── docker-compose.coolify.yml   Coolify override
├── turbo.json        Turborepo pipeline
└── pnpm-workspace.yaml
```

## 3. Environment Configuration

```bash
cp .env.example .env
cp .env.example.coolify .env.coolify   # only if deploying via Coolify
```

Generate strong secrets for the self-hosted integrations:

```bash
bash scripts/generate-secrets.sh > .env.production.formbricks
# Manually merge generated values into your .env
```

`scripts/generate-secrets.sh` produces:

- Core infrastructure (Postgres, Redis, Meilisearch, MinIO passwords)
- Encryption keys (never rotate after initial setup)
- Chatwoot, Cal.com, Formbricks, Documenso, Listmonk secrets
- LiveKit API key/secret/webhook secret
- Umami app secret

For each `*_ENCRYPTION_KEY` and `*_NEXTAUTH_SECRET`, treat them as immutable after first start — rotating will invalidate sessions and unrecoverable encrypted data.

### Generate the Documenso signing certificate (one-time)

```bash
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
  -days 365 -nodes -subj "/CN=MicProtocol Documenso"
openssl pkcs12 -export -out scripts/documenso-cert/cert.p12 \
  -inkey key.pem -in cert.pem \
  -passout pass:"$(openssl rand -hex 32)"
rm key.pem cert.pem
```

Set `DOCUMENSO_SIGNING_PASSPHRASE` in `.env` to the passphrase used above. Production should use a CA-issued cert, never the self-signed one.

## 4. First Boot — Full Stack (Docker)

```bash
# 4.1. Start core infra (Postgres, Redis, Meilisearch)
docker compose up -d postgres redis meilisearch
# 4.2. Generate Prisma client and push schema
pnpm db:generate
pnpm db:push
# 4.3. Start the entire stack (API + worker + CLIP scorer + web + admin
#       + all self-hosted integrations + realtime)
docker compose up -d
# 4.4. Verify
docker compose ps
curl -s http://localhost:4000/health
```

Default ports:

| Service | Port | URL |
|---|---|---|
| Web (host dashboard + guest portal) | 3000 | http://localhost:3000 |
| Admin | 3006 | http://localhost:3006 |
| API | 4000 | http://localhost:4000 |
| Swagger (API) | 4000 | http://localhost:4000/api/docs |
| Meilisearch | 7700 | http://localhost:7700 |
| mic-realtime (WS) | 4001 | ws://localhost:4001 |
| Cal.com | 3002 | http://localhost:3002 |
| Formbricks | 3001 | http://localhost:3001 |
| Umami | 3003 | http://localhost:3003 |
| Documenso | 3004 | http://localhost:3004 |
| Chatwoot | 3005 | http://localhost:3005 |
| Listmonk | 9000 | http://localhost:9000 |
| MinIO API | 9000 (internal) | http://localhost:9001 |
| MinIO Console | 9090 | http://localhost:9090 |
| LiveKit | 7880 (HTTP), 7881 (TCP), 7882 (UDP) | http://localhost:7880 |
| Mailhog UI | 8025 | http://localhost:8025 |
| Mailhog SMTP | 1025 | n/a |

## 5. First Boot — Local Dev (No Docker for App Code)

Useful when iterating on the API or web app and you have the infra running in Docker.

```bash
# 5.1. Install all JS deps
pnpm install
# 5.2. Install Python deps for CLIP scorer
cd services/clip-scorer
uv sync
cd ../..
# 5.3. Start only infra in Docker
docker compose up -d postgres redis meilisearch
# 5.4. Generate Prisma client and push schema
pnpm db:generate
pnpm db:push
# 5.5. Start all apps in dev mode
pnpm dev
```

The Turborepo pipeline runs `api`, `web`, `admin` (if configured), and any app with a `dev` script. Hot reload is enabled for both the API (NestJS) and Web (Next.js).

## 6. Verifying the Stack

### API

```bash
# Liveness
curl -s http://localhost:4000/health
# Detailed
curl -s http://localhost:4000/health/detailed
# Readiness (K8s style)
curl -s http://localhost:4000/health/ready
```

### Database

```bash
pnpm db:studio   # opens Prisma Studio on http://localhost:5555
```

### Self-hosted integrations

After first boot, complete these one-time setup steps inside each UI:

| Service | Action |
|---|---|
| Cal.com | Create org `micprotocol`, generate an API key in Settings → API Keys; set `CALCOM_API_KEY` in `.env` and restart API |
| Formbricks | Create Environment, copy ID to `NEXT_PUBLIC_FORMBRICKS_ENVIRONMENT_ID`; generate API key in Settings → API Keys |
| Listmonk | Log in with `LISTMONK_ADMIN_USER`/`LISTMONK_ADMIN_PASSWORD`, generate API token in Settings → API Tokens |
| Documenso | Upload signing cert (already generated in step 3) and configure SMTP |
| Chatwoot | Create user, create inbox (Website), copy inbox ID and website token to `.env` |
| Umami | Add website, copy website ID to `NEXT_PUBLIC_UMAMI_WEBSITE_ID` |
| MinIO | Log in with `MINIO_ROOT_USER`/`MINIO_ROOT_PASSWORD`; buckets are auto-created by `minio-createbuckets` |

### Webhook smoke tests

Each self-hosted integration has a webhook the API consumes. Use the Integration Health controller to confirm reachability:

```bash
curl -s http://localhost:4000/integrations/health | jq
```

See [Webhooks](./webhooks.md) for the full endpoint catalog and signature headers.

## 7. Running Tests

### API unit tests (Jest)

```bash
pnpm --filter @micprotocol/api test
```

### API E2E tests

```bash
pnpm --filter @micprotocol/api test:e2e
```

### Synthetic agents (full-stack scenario sweep)

```bash
pnpm agents:run   # full pipeline (pod up + seed + API + scenarios)
pnpm agents:orchestrator   # orchestrator only (API must be running)
pnpm db:seed:agents        # seed only
```

Requires `podman` + `podman-compose`. Outputs:

- `scripts/synthetic-agents/test-report.txt`
- `scripts/synthetic-agents/weaknesses.json`

## 8. Common Day-to-Day Commands

| Task | Command |
|---|---|
| Start everything | `pnpm dev` (JS apps) or `docker compose up -d` (full stack) |
| Rebuild after schema change | `pnpm db:generate && pnpm db:push` |
| Create a real migration | `pnpm --filter @micprotocol/api prisma:migrate` |
| Open Prisma Studio | `pnpm db:studio` |
| Run lint across the monorepo | `pnpm lint` |
| Run type check | `pnpm type-check` |
| Build for production | `pnpm build` |
| Tail logs | `docker compose logs -f api worker clip-scorer web` |
| Restart a single service | `docker compose restart <name>` |
| Stop everything | `docker compose down` |
| Wipe volumes (destructive) | `docker compose down -v` |

## 9. Troubleshooting First Run

### Postgres won't start

`docker compose logs postgres` — most often it's a port conflict (`5432` already in use). Either stop the local Postgres or change the host port mapping in `docker-compose.yml`.

### `pnpm db:push` complains about missing tables for `chatwoot`, `calcom`, etc.

You must run the init script that creates per-integration databases. The `init-databases.sql` is mounted into the Postgres container on first boot only — if the volume already exists, run manually:

```bash
docker exec -i micprotocol-postgres psql -U micprotocol -d micprotocol < scripts/init-databases.sql
```

### API returns 500 on first hit

Check that `CLERK_SECRET_KEY` and `CLERK_PUBLISHABLE_KEY` are set; without them the auth guard rejects all requests. For local-only development without Clerk, see [Security & Compliance](./security.md) for the dev-mode `x-user-id` header bypass.

### CLIP scorer stuck in "Scoring…"

See [CLIP Scoring Engine → Troubleshooting](./clip-scoring.md#troubleshooting).

### Webhook 401s

Every webhook signature is verified with `crypto.timingSafeEqual()`. See [Security & Compliance](./security.md) and [Webhooks](./webhooks.md) for the exact header names per provider.

### Port already in use

The compose file uses fixed ports for every integration. Stop the conflicting process or remap the host port in `docker-compose.yml`.

## 10. Next Steps

- Read [Architecture](./architecture.md) for the bounded-context breakdown.
- Read [Self-Hosted Stack](./self-hosted-stack.md) before changing any container config.
- Read [Webhooks](./webhooks.md) before wiring a new integration.
- Read [Operations Runbook](./operations-runbook.md) before going to production.

	# Getting Started

	This guide takes you from a clean machine to a running MicProtocol stack with all self-hosted integrations. Read end-to-end on first install; use the checklists for re-installs.

	## 1. Prerequisites

	\| Tool \| Version \| Purpose \|
	\|---\|---\|---\|
	\| Node.js \| >= 20.0.0 \| API, Web, Admin, Mobile tooling \|
	\| pnpm \| >= 9.15.0 (locked to `pnpm@10.30.3`) \| Workspace package manager \|
	\| Docker + Docker Compose \| latest stable \| Infra (Postgres, Redis, Meilisearch, integrations) \|
	\| Python \| >= 3.11 \| CLIP scoring engine \|
	\| uv \| latest \| Python package manager for `services/clip-scorer` \|
	\| FFmpeg \| latest \| Audio/video frame extraction (also installed in API image) \|
	\| PostgreSQL client \| 15+ \| Optional local DB without Docker \|
	\| Redis CLI \| 7+ \| Optional local Redis without Docker \|
	\| podman + podman-compose \| latest \| Required only for `pnpm agents:run` \|

	Install uv:

	```bash
	curl -LsSf https://astral.sh/uv/install.sh \| sh
	```

	## 2. Repository

	```bash
	git clone <repo-url> micprotocol
	cd micprotocol
	```

	The repo is a Turborepo monorepo. Top-level layout:

	```
	micprotocol/
	├── apps/
	│ ├── api/ NestJS backend (tRPC + REST + Socket.io)
	│ ├── web/ Next.js 14 host dashboard + guest portal
	│ ├── admin/ Next.js platform-admin app
	│ └── mobile/ Expo React Native app
	├── packages/
	│ ├── shared-types/ Zod schemas + domain enums
	│ ├── ui/ Cross-platform UI primitives
	│ ├── tsconfig/ Shared TS configs
	│ └── eslint-config/ Shared ESLint config
	├── services/
	│ ├── clip-scorer/ Python ML engine (OpenCLIP + pyloudnorm + bart-large-mnli)
	│ └── mic-realtime/ Erlang/OTP chat fanout + presence
	├── scripts/
	│ ├── init-databases.sql Per-integration Postgres databases
	│ ├── generate-secrets.sh Generates all self-hosted secrets
	│ ├── documenso-cert/README.md How to generate the Documenso signing cert
	│ └── synthetic-agents/ Synthetic agent test framework
	├── docs/ This documentation set
	├── docker-compose.yml Production Docker Compose (full stack)
	├── docker-compose.coolify.yml Coolify override
	├── turbo.json Turborepo pipeline
	└── pnpm-workspace.yaml
	```

	## 3. Environment Configuration

	```bash
	cp .env.example .env
	cp .env.example.coolify .env.coolify # only if deploying via Coolify
	```

	Generate strong secrets for the self-hosted integrations:

	```bash
	bash scripts/generate-secrets.sh > .env.production.formbricks
	# Manually merge generated values into your .env
	```

	`scripts/generate-secrets.sh` produces:

	- Core infrastructure (Postgres, Redis, Meilisearch, MinIO passwords)
	- Encryption keys (never rotate after initial setup)
	- Chatwoot, Cal.com, Formbricks, Documenso, Listmonk secrets
	- LiveKit API key/secret/webhook secret
	- Umami app secret

	For each `_ENCRYPTION_KEY` and `_NEXTAUTH_SECRET`, treat them as immutable after first start — rotating will invalidate sessions and unrecoverable encrypted data.

	### Generate the Documenso signing certificate (one-time)

	```bash
	openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
	-days 365 -nodes -subj "/CN=MicProtocol Documenso"
	openssl pkcs12 -export -out scripts/documenso-cert/cert.p12 \
	-inkey key.pem -in cert.pem \
	-passout pass:"$(openssl rand -hex 32)"
	rm key.pem cert.pem
	```

	Set `DOCUMENSO_SIGNING_PASSPHRASE` in `.env` to the passphrase used above. Production should use a CA-issued cert, never the self-signed one.

	## 4. First Boot — Full Stack (Docker)

	```bash
	# 4.1. Start core infra (Postgres, Redis, Meilisearch)
	docker compose up -d postgres redis meilisearch

	# 4.2. Generate Prisma client and push schema
	pnpm db:generate
	pnpm db:push

	# 4.3. Start the entire stack (API + worker + CLIP scorer + web + admin
	# + all self-hosted integrations + realtime)
	docker compose up -d

	# 4.4. Verify
	docker compose ps
	curl -s http://localhost:4000/health
	```

	Default ports:

	\| Service \| Port \| URL \|
	\|---\|---\|---\|
	\| Web (host dashboard + guest portal) \| 3000 \| http://localhost:3000 \|
	\| Admin \| 3006 \| http://localhost:3006 \|
	\| API \| 4000 \| http://localhost:4000 \|
	\| Swagger (API) \| 4000 \| http://localhost:4000/api/docs \|
	\| Meilisearch \| 7700 \| http://localhost:7700 \|
	\| mic-realtime (WS) \| 4001 \| ws://localhost:4001 \|
	\| Cal.com \| 3002 \| http://localhost:3002 \|
	\| Formbricks \| 3001 \| http://localhost:3001 \|
	\| Umami \| 3003 \| http://localhost:3003 \|
	\| Documenso \| 3004 \| http://localhost:3004 \|
	\| Chatwoot \| 3005 \| http://localhost:3005 \|
	\| Listmonk \| 9000 \| http://localhost:9000 \|
	\| MinIO API \| 9000 (internal) \| http://localhost:9001 \|
	\| MinIO Console \| 9090 \| http://localhost:9090 \|
	\| LiveKit \| 7880 (HTTP), 7881 (TCP), 7882 (UDP) \| http://localhost:7880 \|
	\| Mailhog UI \| 8025 \| http://localhost:8025 \|
	\| Mailhog SMTP \| 1025 \| n/a \|

	## 5. First Boot — Local Dev (No Docker for App Code)

	Useful when iterating on the API or web app and you have the infra running in Docker.

	```bash
	# 5.1. Install all JS deps
	pnpm install

	# 5.2. Install Python deps for CLIP scorer
	cd services/clip-scorer
	uv sync
	cd ../..

	# 5.3. Start only infra in Docker
	docker compose up -d postgres redis meilisearch

	# 5.4. Generate Prisma client and push schema
	pnpm db:generate
	pnpm db:push

	# 5.5. Start all apps in dev mode
	pnpm dev
	```

	The Turborepo pipeline runs `api`, `web`, `admin` (if configured), and any app with a `dev` script. Hot reload is enabled for both the API (NestJS) and Web (Next.js).

	## 6. Verifying the Stack

	### API

	```bash
	# Liveness
	curl -s http://localhost:4000/health

	# Detailed
	curl -s http://localhost:4000/health/detailed

	# Readiness (K8s style)
	curl -s http://localhost:4000/health/ready
	```

	### Database

	```bash
	pnpm db:studio # opens Prisma Studio on http://localhost:5555
	```

	### Self-hosted integrations

	After first boot, complete these one-time setup steps inside each UI:

	\| Service \| Action \|
	\|---\|---\|
	\| Cal.com \| Create org `micprotocol`, generate an API key in Settings → API Keys; set `CALCOM_API_KEY` in `.env` and restart API \|
	\| Formbricks \| Create Environment, copy ID to `NEXT_PUBLIC_FORMBRICKS_ENVIRONMENT_ID`; generate API key in Settings → API Keys \|
	\| Listmonk \| Log in with `LISTMONK_ADMIN_USER`/`LISTMONK_ADMIN_PASSWORD`, generate API token in Settings → API Tokens \|
	\| Documenso \| Upload signing cert (already generated in step 3) and configure SMTP \|
	\| Chatwoot \| Create user, create inbox (Website), copy inbox ID and website token to `.env` \|
	\| Umami \| Add website, copy website ID to `NEXT_PUBLIC_UMAMI_WEBSITE_ID` \|
	\| MinIO \| Log in with `MINIO_ROOT_USER`/`MINIO_ROOT_PASSWORD`; buckets are auto-created by `minio-createbuckets` \|

	### Webhook smoke tests

	Each self-hosted integration has a webhook the API consumes. Use the Integration Health controller to confirm reachability:

	```bash
	curl -s http://localhost:4000/integrations/health \| jq
	```

	See [Webhooks](./webhooks.md) for the full endpoint catalog and signature headers.

	## 7. Running Tests

	### API unit tests (Jest)

	```bash
	pnpm --filter @micprotocol/api test
	```

	### API E2E tests

	```bash
	pnpm --filter @micprotocol/api test:e2e
	```

	### Synthetic agents (full-stack scenario sweep)

	```bash
	pnpm agents:run # full pipeline (pod up + seed + API + scenarios)
	pnpm agents:orchestrator # orchestrator only (API must be running)
	pnpm db:seed:agents # seed only
	```

	Requires `podman` + `podman-compose`. Outputs:

	- `scripts/synthetic-agents/test-report.txt`
	- `scripts/synthetic-agents/weaknesses.json`

	## 8. Common Day-to-Day Commands

	\| Task \| Command \|
	\|---\|---\|
	\| Start everything \| `pnpm dev` (JS apps) or `docker compose up -d` (full stack) \|
	\| Rebuild after schema change \| `pnpm db:generate && pnpm db:push` \|
	\| Create a real migration \| `pnpm --filter @micprotocol/api prisma:migrate` \|
	\| Open Prisma Studio \| `pnpm db:studio` \|
	\| Run lint across the monorepo \| `pnpm lint` \|
	\| Run type check \| `pnpm type-check` \|
	\| Build for production \| `pnpm build` \|
	\| Tail logs \| `docker compose logs -f api worker clip-scorer web` \|
	\| Restart a single service \| `docker compose restart <name>` \|
	\| Stop everything \| `docker compose down` \|
	\| Wipe volumes (destructive) \| `docker compose down -v` \|

	## 9. Troubleshooting First Run

	### Postgres won't start

	`docker compose logs postgres` — most often it's a port conflict (`5432` already in use). Either stop the local Postgres or change the host port mapping in `docker-compose.yml`.

	### `pnpm db:push` complains about missing tables for `chatwoot`, `calcom`, etc.

	You must run the init script that creates per-integration databases. The `init-databases.sql` is mounted into the Postgres container on first boot only — if the volume already exists, run manually:

	```bash
	docker exec -i micprotocol-postgres psql -U micprotocol -d micprotocol < scripts/init-databases.sql
	```

	### API returns 500 on first hit

	Check that `CLERK_SECRET_KEY` and `CLERK_PUBLISHABLE_KEY` are set; without them the auth guard rejects all requests. For local-only development without Clerk, see [Security & Compliance](./security.md) for the dev-mode `x-user-id` header bypass.

	### CLIP scorer stuck in "Scoring…"

	See [CLIP Scoring Engine → Troubleshooting](./clip-scoring.md#troubleshooting).

	### Webhook 401s

	Every webhook signature is verified with `crypto.timingSafeEqual()`. See [Security & Compliance](./security.md) and [Webhooks](./webhooks.md) for the exact header names per provider.

	### Port already in use

	The compose file uses fixed ports for every integration. Stop the conflicting process or remap the host port in `docker-compose.yml`.

	## 10. Next Steps

	- Read [Architecture](./architecture.md) for the bounded-context breakdown.
	- Read [Self-Hosted Stack](./self-hosted-stack.md) before changing any container config.
	- Read [Webhooks](./webhooks.md) before wiring a new integration.
	- Read [Operations Runbook](./operations-runbook.md) before going to production.