MicProtocol / docs /architecture.md
Benaah's picture
Upload folder using huggingface_hub
b17baae verified
|
Raw History Blame Contribute Delete
8.03 kB

MicProtocol – System Architecture

Philosophy

MicProtocol is a modular monolith with clear bounded contexts, designed to split into microservices once specific domains hit scale. The architecture prioritizes workflow automation over simple CRUD, recognizing that significant time savings come from eliminating friction in asynchronous coordination between hosts, guests, and production teams.

Tech Stack

Layer Technology Justification
Monorepo Turborepo + pnpm workspaces Shared types, UI components, and validation schemas across web/mobile
Backend NestJS (Node.js) Enterprise-grade DI, built-in CQRS/Event Sourcing support, gradual microservice migration
API Protocol tRPC + WebSockets End-to-end type safety between frontend and backend; real-time updates for scheduling
Database PostgreSQL 15+ (primary) + Redis ACID for financial/booking data; Redis for session, cache, and BullMQ job queues
ORM Prisma Type-safe migrations, excellent relations handling for complex guest-episode-asset graphs
Search Meilisearch Instant guest lookup, fuzzy matching on names/companies
Storage Cloudflare R2 (S3-compatible) Zero egress fees for media sharing; perfect for audio files and headshots
Queue BullMQ (Redis-backed) Reliable scheduling of follow-up emails, reminder workflows
AI Layer NVIDIA NIM + DeepSeek + Moonshot Outreach personalization, audiogram generation, transcript processing
CLIP Scoring Python + OpenCLIP + bart-large-mnli Multimodal clip evaluation (visual/audio/transcript) via subprocess
Web Frontend Next.js 14 (App Router) Server Components for heavy dashboards; Edge runtime for guest portal
Mobile React Native (Expo) 70% code reuse with web, native calendar integration
Auth Clerk Handles OAuth (Google/Apple), MFA, and session management
Calendar Nylas API Unified Google/Outlook/Exchange integration; bi-directional sync
Email Resend + React Email Programmable transactional emails with React components

System Architecture

graph TB
    subgraph "Client Layer"
        WEB[Next.js 14 Web App<br/>App Router]
        MOBILE[React Native<br/>Expo SDK]
        GUEST_PORTAL[Guest Portal<br/>Next.js Edge Runtime]
    end

    subgraph "API Gateway Layer"
        GW[NestJS API Gateway<br/>tRPC Router]
        WS[WebSocket Gateway<br/>Socket.io]
    end

    subgraph "Core Domains - Modular Monolith"
        AUTH[Auth Module]
        GUEST[Guest Management<br/>Domain]
        OUTREACH[Outreach Automation<br/>Domain]
        SCHEDULE[Scheduling Engine<br/>Domain]
        ASSETS[Asset Pipeline<br/>Domain]
        RECORDING[Recording Infrastructure<br/>Domain]
        CONTENT_STUDIO[Content Studio<br/>CLIP Scoring]
        ANALYTICS[Analytics Engine]
    end

    subgraph "Infrastructure Layer"
        DB[(PostgreSQL<br/>Primary Data)]
        CACHE[(Redis<br/>BullMQ + Cache)]
        SEARCH[(Meilisearch)]
        OBJ_STORE[(R2 Object Storage)]
    end

    subgraph "ML Pipeline"
        CLIP[CLIP Scorer<br/>Python + OpenCLIP]
        FF[FFmpeg<br/>Frame/Audio Extract]
    end

    subgraph "External Integrations"
        CAL[Calendar APIs<br/>Nylas/Google/Outlook]
        EMAIL[Email Providers<br/>Resend/SendGrid]
        AI[NVIDIA NIM + DeepSeek + Moonshot]
        PODCAST_HOST[Podcast Hosting<br/>Buzzsprout/Transistor]
    end

    WEB -->|tRPC| GW
    MOBILE -->|tRPC| GW
    GUEST_PORTAL -->|Public REST| GW
    
    GW --> AUTH
    GW --> GUEST
    GW --> OUTREACH
    GW --> SCHEDULE
    GW --> ASSETS
    GW --> RECORDING
    GW --> CONTENT_STUDIO
    GW --> ANALYTICS
    
    GUEST --> DB
    GUEST --> SEARCH
    OUTREACH --> DB
    OUTREACH --> CACHE
    SCHEDULE --> DB
    SCHEDULE --> CAL
    ASSETS --> DB
    ASSETS --> OBJ_STORE
    RECORDING --> DB
    RECORDING --> OBJ_STORE
    
    OUTREACH --> EMAIL
    ASSETS --> AI
    ANALYTICS --> DB
    CONTENT_STUDIO --> CLIP
    CONTENT_STUDIO --> OBJ_STORE
    CLIP --> FF
    
    CACHE -->|BullMQ Workers| EMAIL
    CACHE -->|BullMQ Workers| OBJ_STORE
    CACHE -->|BullMQ Workers| CAL
    
    style GUEST_PORTAL fill:#e1f5e1
    style WS fill:#fff3e0

Bounded Contexts

Guest Management Domain

Owns the guest lifecycle from prospect to published. Manages guest profiles, tags, priority scoring, and status transitions.

Outreach Automation Domain

Manages email sequences, message tracking (sent/opened/clicked), and AI-powered personalization. Primary candidate for future microservice extraction due to high write volume on email events.

Scheduling Engine Domain

Handles episode creation, booking management, calendar integration (Nylas), and recording session coordination.

Asset Pipeline Domain

Manages the pre-production asset collection workflow: requesting materials from guests (headshots, bios, audio intros), tracking submissions, and processing uploads through AI validation.

Recording Infrastructure Domain

Provides a highly resilient, local-first double-ender audio recording system tailored for low-bandwidth environments (e.g., Africa). It handles environmental diagnostics (Green Room API), asynchronous chunked 48kHz audio uploading, OPUS fallback codecs, and BullMQ worker-based edge post-processing.

Analytics Engine

Aggregates metrics across all domains: guest pipeline funnel, outreach performance (open/click rates), scheduling efficiency, and asset completion tracking.

Content Studio Domain

Provides viral clip detection, scoring, and multi-platform export. Integrates with the CLIP scoring engine (Python subprocess) to evaluate clips on visual composition, audio quality, and transcript engagement. Handles clip approval workflow and scheduled exports.

Project Structure

MicProtocol/
β”œβ”€β”€ apps/
β”‚   β”œβ”€β”€ api/                    # NestJS backend
β”‚   β”‚   β”œβ”€β”€ prisma/             # Database schema & migrations
β”‚   β”‚   └── src/
β”‚   β”‚       β”œβ”€β”€ analytics/      # Analytics domain module
β”‚   β”‚       β”œβ”€β”€ assets/         # Asset pipeline domain module
β”‚   β”‚       β”œβ”€β”€ auth/           # Clerk authentication
β”‚   β”‚       β”œβ”€β”€ guest/          # Guest management domain module
β”‚   β”‚       β”œβ”€β”€ outreach/       # Outreach automation domain module
β”‚   β”‚       β”œβ”€β”€ prisma/         # Database service
β”‚   β”‚       β”œβ”€β”€ queue/          # BullMQ workers (content, clip-scoring)
β”‚   β”‚       β”œβ”€β”€ recording/      # Recording infrastructure domain & edge processing
β”‚   β”‚       β”œβ”€β”€ content-studio/ # Viral clip detection + scoring + export
β”‚   β”‚       β”œβ”€β”€ scheduling/     # Scheduling engine domain module
β”‚   β”‚       β”œβ”€β”€ trpc/           # tRPC router setup
β”‚   β”‚       └── websocket/      # Real-time events
β”‚   └── web/                    # Next.js 14 frontend
β”‚       └── src/
β”‚           β”œβ”€β”€ app/
β”‚           β”‚   β”œβ”€β”€ (dashboard)/ # Protected host workspace
β”‚           β”‚   └── (guest-portal)/ # Public guest-facing pages
β”‚           β”œβ”€β”€ lib/            # tRPC client, providers
β”‚           └── styles/         # Design system CSS
β”œβ”€β”€ services/
β”‚   └── clip-scorer/            # Python ML engine (CLIP + audio + NLP)
β”‚       β”œβ”€β”€ scoring/            # Visual, audio, transcript modules
β”‚       β”œβ”€β”€ pipeline.py         # CLI entrypoint
β”‚       β”œβ”€β”€ pyproject.toml      # uv-managed dependencies
β”‚       └── Dockerfile
β”œβ”€β”€ packages/
β”‚   β”œβ”€β”€ shared-types/           # Domain enums, Zod schemas
β”‚   β”œβ”€β”€ ui/                     # Design tokens
β”‚   β”œβ”€β”€ eslint-config/          # Shared linting
β”‚   └── tsconfig/               # Shared TS configs
└── docs/                       # Architecture documentation