|
Download ARCHITECTURE_INTEGRATION_GUIDE.md from ineso22/Mozdef: direct link, hf CLI and curl.
- Browser
- Download file 21.4 kB
-
https://huggingface.co/ineso22/Mozdef/resolve/main/ARCHITECTURE_INTEGRATION_GUIDE.md
- Command line
-
hf download hf://ineso22/Mozdef/ARCHITECTURE_INTEGRATION_GUIDE.md
-
curl -L -o ARCHITECTURE_INTEGRATION_GUIDE.md https://huggingface.co/ineso22/Mozdef/resolve/main/ARCHITECTURE_INTEGRATION_GUIDE.md
21.4 kB
| # MozDef Architecture: Frontend, Backend & Services Integration Guide | |
| ## Table of Contents | |
| 1. [Frontend vs Backend Overview](#frontend-vs-backend-overview) | |
| 2. [Services Breakdown](#services-breakdown) | |
| 3. [How Services Integrate](#how-services-integrate) | |
| 4. [Complete Data Flow](#complete-data-flow) | |
| 5. [Service Communication Patterns](#service-communication-patterns) | |
| 6. [How They Run Together](#how-they-run-together) | |
| --- | |
| ## Frontend vs Backend Overview | |
| ### Frontend Components (User-Facing) | |
| **Frontend** = What users interact with directly | |
| | Component | Technology | Purpose | Port | | |
| |-----------|-----------|---------|------| | |
| | **Meteor Web Interface** | Node.js/Meteor | Real-time web UI for incident handling | 80 (via Nginx) | | |
| | **Kibana** | JavaScript/React | Log visualization and analysis | 9090 (via Nginx) | | |
| **Key Characteristics:** | |
| - **Meteor**: Runs on Node.js, provides real-time collaboration | |
| - **Kibana**: Standalone visualization tool, connects directly to Elasticsearch | |
| - Both are served through **Nginx** as a reverse proxy | |
| ### Backend Components (Processing & API) | |
| **Backend** = Services that process data and provide APIs | |
| | Component | Technology | Purpose | Port | | |
| |-----------|-----------|---------|------| | |
| | **REST API** | Python/Bottle.py | RESTful API for web interface | 8081 | | |
| | **Loginput** | Python/Bottle.py | HTTP endpoint for event ingestion | 8080 | | |
| | **MQ Workers** | Python | Process events from RabbitMQ queue | N/A (internal) | | |
| | **Alerts Engine** | Python/Celery | Alert correlation and generation | N/A (internal) | | |
| | **Alert Actions** | Python | Post-alert processing | N/A (internal) | | |
| | **Bot** | Python | Slack/IRC notifications | N/A (internal) | | |
| **Key Characteristics:** | |
| - All backend services are **Python-based** | |
| - Use **uWSGI** for process management | |
| - Communicate via **RabbitMQ** message queues | |
| - Query/write to **Elasticsearch** and **MongoDB** | |
| --- | |
| ## Services Breakdown | |
| ### Infrastructure Services (Supporting Services) | |
| These are the foundational services that other components depend on: | |
| | Service | Technology | Purpose | Port | | |
| |---------|-----------|---------|------| | |
| | **Elasticsearch** | Java | Event storage, indexing, search | 9200 (internal) | | |
| | **RabbitMQ** | Erlang | Message queue for event processing | 5672 (internal), 15672 (admin) | | |
| | **MongoDB** | C++ | Database for Meteor (incidents, users) | 3002 (internal) | | |
| | **Nginx** | C | Reverse proxy and web server | 80, 8080, 8081, 9090 | | |
| ### Application Services (MozDef Components) | |
| These are the MozDef-specific services: | |
| #### **Ingest Services** (Data Ingestion) | |
| - **Loginput**: Receives JSON events via HTTP | |
| - **MQ Workers**: Process events from queue | |
| - **Syslog**: Receives syslog messages | |
| #### **Alert Services** (Security Alerting) | |
| - **Alerts**: Celery-based alert engine | |
| - **Alert Actions**: Post-alert processing | |
| - **Bot**: Chat notifications | |
| #### **Web Services** (User Interface) | |
| - **REST API**: Backend API for web interface | |
| - **Meteor**: Frontend web application | |
| - **Kibana**: Log visualization (separate service) | |
| --- | |
| ## How Services Integrate | |
| ### Integration Architecture Diagram | |
| ``` | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β USER BROWSER β | |
| βββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β HTTP/WebSocket | |
| βΌ | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β NGINX (Reverse Proxy) β | |
| β ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ β | |
| β β Port 80 β β Port 8080β β Port 8081β β Port 9090β β | |
| β β Meteor β β Loginput β β REST API β β Kibana β β | |
| β ββββββ¬ββββββ ββββββ¬ββββββ ββββββ¬ββββββ ββββββ¬ββββββ β | |
| βββββββββΌββββββββββββββΌβββββββββββββββΌββββββββββββββΌβββββββββββββββ | |
| β β β β | |
| βΌ βΌ βΌ βΌ | |
| βββββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ | |
| β METEOR β β LOGINPUT β β REST API β β KIBANA β | |
| β (Node.js) β β (Python) β β (Python) β β β | |
| ββββββββ¬βββββββ ββββββ¬ββββββ ββββββ¬ββββββ ββββββ¬ββββββ | |
| β β β β | |
| β β β β | |
| β βΌ β β | |
| β ββββββββββββββββ β β | |
| β β RABBITMQ β β β | |
| β β (Message β β β | |
| β β Queue) β β β | |
| β ββββββββ¬ββββββββ β β | |
| β β β β | |
| β βΌ β β | |
| β ββββββββββββββββ β β | |
| β β MQ WORKERS β β β | |
| β β (Python) β β β | |
| β ββββββββ¬ββββββββ β β | |
| β β β β | |
| β β β β | |
| βββββββββββββββΌββββββββββββββββΌβββββββββββββββ | |
| β β | |
| βΌ βΌ | |
| ββββββββββββββββ ββββββββββββββββ | |
| β ELASTICSEARCHβ β MONGODB β | |
| β (Storage) β β (Database) β | |
| ββββββββ¬ββββββββ ββββββββ¬ββββββββ | |
| β β | |
| β β | |
| βΌ β | |
| ββββββββββββββββ β | |
| β ALERTS β β | |
| β (Celery) β β | |
| ββββββββ¬ββββββββ β | |
| β β | |
| βΌ β | |
| ββββββββββββββββ β | |
| β ALERT ACTIONSβ β | |
| β & BOT β β | |
| ββββββββββββββββ β | |
| ``` | |
| --- | |
| ## Complete Data Flow | |
| ### 1. Event Ingestion Flow | |
| ``` | |
| External Log Shipper | |
| β | |
| β JSON over HTTP POST | |
| βΌ | |
| βββββββββββββββββββ | |
| β NGINX β (Port 8080) | |
| β Reverse Proxy β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β uWSGI socket | |
| βΌ | |
| βββββββββββββββββββ | |
| β LOGINPUT β (Python/Bottle.py) | |
| β - Receives JSONβ | |
| β - Validates β | |
| β - Adds metadataβ | |
| ββββββββββ¬βββββββββ | |
| β | |
| β Publishes to RabbitMQ | |
| βΌ | |
| βββββββββββββββββββ | |
| β RABBITMQ β (Message Queue) | |
| β Exchange: β | |
| β eventtask β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β Consumes from queue | |
| βΌ | |
| βββββββββββββββββββ | |
| β MQ WORKERS β (Python) | |
| β - Normalize β | |
| β - Enrich β | |
| β - GeoIP β | |
| β - Watchlist β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β Index document | |
| βΌ | |
| βββββββββββββββββββ | |
| β ELASTICSEARCH β | |
| β - Stores event β | |
| β - Indexes β | |
| β - Searchable β | |
| βββββββββββββββββββ | |
| ``` | |
| ### 2. Frontend-Backend Communication Flow | |
| ``` | |
| User Browser | |
| β | |
| β HTTP Request | |
| βΌ | |
| βββββββββββββββββββ | |
| β NGINX β (Port 80) | |
| β Reverse Proxy β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β Proxy to Meteor | |
| βΌ | |
| βββββββββββββββββββ | |
| β METEOR β (Node.js) | |
| β Web Interface β | |
| β β | |
| β Client Code: β | |
| β - UI Componentsβ | |
| β - Reactivity β | |
| β β | |
| β Server Code: β | |
| β - Methods β | |
| β - HTTP calls β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β HTTP POST/GET | |
| β (via mozdef.rootAPI) | |
| βΌ | |
| βββββββββββββββββββ | |
| β REST API β (Python/Bottle.py) | |
| β Port 8081 β | |
| β β | |
| β Endpoints: β | |
| β - /events β | |
| β - /alerts β | |
| β - /incidents β | |
| β - /watchitem β | |
| β - /blockip β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β Query/Write | |
| ββββββββββββββββ | |
| βΌ βΌ | |
| ββββββββββββββββ ββββββββββββββββ | |
| β ELASTICSEARCHβ β MONGODB β | |
| β (Events) β β (Incidents) β | |
| ββββββββββββββββ ββββββββββββββββ | |
| ``` | |
| **Example: User blocks an IP address** | |
| 1. User clicks "Block IP" in Meteor UI | |
| 2. Meteor client calls Meteor method: `Meteor.call('blockip', {ip: '1.2.3.4'})` | |
| 3. Meteor server method executes: `blockIP(formobj)` | |
| 4. Meteor server makes HTTP POST: `HTTP.post(mozdef.rootAPI + '/blockip', {data: formobj})` | |
| 5. REST API receives POST at `/blockip` | |
| 6. REST API processes request (may call plugins, update watchlist, etc.) | |
| 7. REST API returns response | |
| 8. Meteor receives response and updates UI reactively | |
| ### 3. Alert Processing Flow | |
| ``` | |
| βββββββββββββββββββ | |
| β ELASTICSEARCH β | |
| β (Event Store) β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β Periodic Query | |
| β (Every X minutes) | |
| βΌ | |
| βββββββββββββββββββ | |
| β ALERTS β (Python/Celery) | |
| β Alert Engine β | |
| β β | |
| β - Queries ES β | |
| β - Correlates β | |
| β - Aggregates β | |
| β - Generates β | |
| ββββββββββ¬βββββββββ | |
| β | |
| β Creates alert document | |
| ββββββββββββββββββββββββ | |
| βΌ βΌ | |
| ββββββββββββββββ ββββββββββββββββ | |
| β ELASTICSEARCHβ β RABBITMQ β | |
| β (alerts-*) β β (alerts) β | |
| ββββββββββββββββ ββββββββ¬ββββββββ | |
| β | |
| β Consume alert | |
| βΌ | |
| ββββββββββββββββ | |
| β ALERT ACTIONS β | |
| β (Python) β | |
| β β | |
| β - Notify β | |
| β - Enrich β | |
| β - Respond β | |
| ββββββββ¬ββββββββ | |
| β | |
| βΌ | |
| ββββββββββββββββ | |
| β BOT β | |
| β (Slack/IRC) β | |
| β β | |
| β Sends messageβ | |
| ββββββββββββββββ | |
| ``` | |
| **Example: SSH Brute Force Alert** | |
| 1. **Alerts Engine** (Celery) runs every 5 minutes | |
| 2. Queries Elasticsearch: "Find SSH failed login events in last 5 minutes" | |
| 3. Aggregates by source IP: "Count failures per IP" | |
| 4. If IP has > 10 failures: Create alert | |
| 5. Alert saved to Elasticsearch `alerts-*` index | |
| 6. Alert published to RabbitMQ `alerts` exchange | |
| 7. **Alert Actions** worker consumes alert | |
| 8. **Alert Actions** enriches alert (GeoIP, threat intel) | |
| 9. **Bot** consumes alert and sends to Slack channel | |
| ### 4. Real-Time Collaboration Flow (Meteor) | |
| ``` | |
| User 1 Browser User 2 Browser | |
| β β | |
| β β | |
| βΌ βΌ | |
| βββββββββββ βββββββββββ | |
| β METEOR β β METEOR β | |
| β Client β β Client β | |
| ββββββ¬βββββ ββββββ¬βββββ | |
| β β | |
| β WebSocket/DDP β | |
| β (Real-time) β | |
| β β | |
| ββββββββββββ¬βββββββββββ | |
| β | |
| βΌ | |
| ββββββββββββββββ | |
| β METEOR β | |
| β Server β | |
| β (Node.js) β | |
| ββββββββ¬ββββββββ | |
| β | |
| β Read/Write | |
| βΌ | |
| ββββββββββββββββ | |
| β MONGODB β | |
| β (Incidents) β | |
| ββββββββββββββββ | |
| ``` | |
| **Example: Two users working on same incident** | |
| 1. User 1 opens incident #123 in Meteor UI | |
| 2. User 2 opens same incident #123 | |
| 3. Both users subscribe to MongoDB collection: `incidents` | |
| 4. User 1 adds a comment | |
| 5. Meteor server updates MongoDB | |
| 6. MongoDB change triggers Meteor reactivity | |
| 7. Both User 1 and User 2 see the comment appear **instantly** (no page refresh) | |
| --- | |
| ## Service Communication Patterns | |
| ### 1. HTTP/REST Communication | |
| **Used for:** | |
| - Meteor β REST API | |
| - External systems β Loginput | |
| - Kibana β Elasticsearch | |
| **Protocol:** HTTP/HTTPS | |
| **Format:** JSON | |
| **Example:** | |
| ```javascript | |
| // Meteor calling REST API | |
| HTTP.post('http://rest:8081/blockip', { | |
| data: {ip: '1.2.3.4', reason: 'malicious'} | |
| }); | |
| ``` | |
| ### 2. Message Queue Communication (RabbitMQ) | |
| **Used for:** | |
| - Loginput β MQ Workers | |
| - Alerts β Alert Actions | |
| - Alerts β Bot | |
| **Protocol:** AMQP (Advanced Message Queuing Protocol) | |
| **Format:** JSON messages | |
| **Exchanges:** | |
| - `eventtask`: Events from loginput | |
| - `mozdef.event`: Processed events (topic exchange) | |
| - `alerts`: Generated alerts | |
| **Example:** | |
| ```python | |
| # Loginput publishes to RabbitMQ | |
| mqConn.ensure(mqproducer, mqproducer.publish, max_retries=10) | |
| mqproducer.publish( | |
| eventDict, | |
| exchange=eventTaskExchange, | |
| routing_key=options.taskexchange | |
| ) | |
| # MQ Worker consumes from RabbitMQ | |
| taskConsumer(mqConn, eventTaskQueue, eventTopicExchange, es).run() | |
| ``` | |
| ### 3. Direct Database Access | |
| **Used for:** | |
| - REST API β Elasticsearch (queries) | |
| - REST API β MongoDB (incidents) | |
| - MQ Workers β Elasticsearch (indexing) | |
| - Alerts β Elasticsearch (queries) | |
| - Meteor β MongoDB (real-time subscriptions) | |
| **Protocol:** Native client libraries | |
| - Elasticsearch: `elasticsearch` Python client | |
| - MongoDB: `pymongo` (Python) or native (Meteor) | |
| **Example:** | |
| ```python | |
| # REST API querying Elasticsearch | |
| es = ElasticsearchClient(options.esservers) | |
| search_query = SearchQuery(minutes=20) | |
| search_query.add_must([TermMatch('category', 'authentication')]) | |
| results = es.query(search_query) | |
| ``` | |
| ### 4. WebSocket/DDP (Meteor) | |
| **Used for:** | |
| - Browser β Meteor server (real-time updates) | |
| **Protocol:** DDP (Distributed Data Protocol) over WebSocket | |
| **Format:** JSON | |
| **Example:** | |
| ```javascript | |
| // Meteor automatically handles WebSocket | |
| // When MongoDB changes, clients are notified | |
| incidents.find({_id: incidentId}).observe({ | |
| added: function(doc) { | |
| // Real-time update received | |
| } | |
| }); | |
| ``` | |
| --- | |
| ## How They Run Together | |
| ### Docker Compose Orchestration | |
| All services are orchestrated via `docker-compose.yml`: | |
| ```yaml | |
| services: | |
| nginx: # Reverse proxy | |
| elasticsearch: # Event storage | |
| rabbitmq: # Message queue | |
| mongodb: # Database | |
| kibana: # Visualization | |
| # MozDef Services | |
| loginput: # Event ingestion | |
| mq_worker: # Event processing | |
| rest: # REST API | |
| meteor: # Web interface | |
| alerts: # Alert engine | |
| alertactions: # Alert processing | |
| bot: # Notifications (optional) | |
| cron: # Scheduled tasks | |
| ``` | |
| ### Startup Sequence | |
| 1. **Infrastructure Services Start First:** | |
| ``` | |
| elasticsearch β rabbitmq β mongodb β kibana | |
| ``` | |
| 2. **Bootstrap Service:** | |
| - Waits for Elasticsearch to be ready | |
| - Creates initial indices | |
| - Sets up Kibana dashboards | |
| 3. **Backend Services Start:** | |
| ``` | |
| loginput β mq_worker β rest β alerts β alertactions | |
| ``` | |
| 4. **Frontend Services Start:** | |
| ``` | |
| meteor (depends on mongodb and rest) | |
| ``` | |
| 5. **Nginx Starts Last:** | |
| - Depends on all services being ready | |
| - Routes traffic to appropriate services | |
| ### Service Dependencies | |
| ``` | |
| nginx | |
| βββ meteor (port 80) | |
| βββ loginput (port 8080) | |
| βββ rest (port 8081) | |
| βββ kibana (port 9090) | |
| meteor | |
| βββ mongodb (user data, incidents) | |
| βββ rest (API calls) | |
| rest | |
| βββ elasticsearch (event queries) | |
| βββ mongodb (incident storage) | |
| loginput | |
| βββ rabbitmq (publish events) | |
| mq_worker | |
| βββ rabbitmq (consume events) | |
| βββ elasticsearch (index events) | |
| alerts | |
| βββ elasticsearch (query events) | |
| βββ rabbitmq (publish alerts) | |
| βββ rest (sync schedules) | |
| alertactions | |
| βββ rabbitmq (consume alerts) | |
| kibana | |
| βββ elasticsearch (direct connection) | |
| ``` | |
| ### Health Checks | |
| Each service has health checks to ensure proper startup: | |
| ```yaml | |
| healthcheck: | |
| test: ["CMD", "curl", "-f", "127.0.0.1:8080/status"] | |
| interval: 30s | |
| timeout: 3s | |
| retries: 10 | |
| ``` | |
| Services wait for dependencies to be healthy before starting. | |
| ### Configuration Integration | |
| **Environment Variables:** | |
| - Services read from environment variables | |
| - Docker Compose sets these via `environment:` section | |
| - Settings cascade: env vars β config files β defaults | |
| **Key Configuration Points:** | |
| - `OPTIONS_METEOR_ROOTAPI`: REST API URL (default: `http://rest:8081`) | |
| - `OPTIONS_METEOR_KIBANAURL`: Kibana URL (default: `http://localhost:9090/app/kibana`) | |
| - `OPTIONS_MQPROTOCOL`: Message queue protocol (`amqp` or `sqs`) | |
| ### Network Communication | |
| All services run in the same Docker network (`default`): | |
| - Services can communicate via service names (e.g., `http://rest:8081`) | |
| - Ports are exposed only where needed (80, 8080, 8081, 9090) | |
| - Internal ports (Elasticsearch 9200, RabbitMQ 5672) are not exposed | |
| --- | |
| ## Summary | |
| ### Frontend | |
| - **Meteor**: Real-time web UI (Node.js) | |
| - **Kibana**: Log visualization (JavaScript) | |
| ### Backend | |
| - **REST API**: Backend API (Python) | |
| - **Loginput**: Event ingestion (Python) | |
| - **MQ Workers**: Event processing (Python) | |
| - **Alerts**: Alert engine (Python/Celery) | |
| - **Alert Actions**: Alert processing (Python) | |
| - **Bot**: Notifications (Python) | |
| ### Services | |
| - **Elasticsearch**: Event storage | |
| - **RabbitMQ**: Message queue | |
| - **MongoDB**: Database | |
| - **Nginx**: Reverse proxy | |
| ### Integration Points | |
| 1. **HTTP**: Meteor β REST API, External β Loginput | |
| 2. **RabbitMQ**: Loginput β MQ Workers, Alerts β Alert Actions | |
| 3. **Elasticsearch**: All services query/write events | |
| 4. **MongoDB**: Meteor stores incidents, REST API reads/writes | |
| 5. **WebSocket**: Browser β Meteor (real-time) | |
| ### How They Run | |
| - Docker Compose orchestrates all services | |
| - Services start in dependency order | |
| - Health checks ensure proper startup | |
| - All services communicate via Docker network | |
| This architecture allows MozDef to: | |
| - **Scale horizontally** (multiple MQ workers, loginput instances) | |
| - **Process events asynchronously** (via RabbitMQ) | |
| - **Provide real-time collaboration** (via Meteor) | |
| - **Handle high event volumes** (300M+ events/day) | |