| # Enterprise Monitoring, Telemetry & Observability Mesh |
|
|
| This directory contains the production-grade observability stack for the **Privacy-Preserving Cross-Bank Fraud Detection Platform (CFI)**, integrating **Prometheus** (metrics aggregation), **Grafana** (real-time dashboards), **Alertmanager** (incident routing), **Loki** & **Promtail** (structured log shipping), and **Jaeger** (distributed tracing). |
|
|
| --- |
|
|
| ## 1. Directory Structure |
|
|
| ```text |
| monitoring/ |
| ├── README.md # Telemetry architecture, metric dictionaries & SLA alerts |
| ├── prometheus.yml # Prometheus server configuration & scrape targets |
| ├── jaeger-ui-config.json # Jaeger distributed tracing UI configuration |
| ├── alertmanager/ |
| │ └── config.yml # Incident routing, alert grouping & webhook notifications |
| ├── prometheus/ |
| │ └── alert_rules.yml # Prometheus SLA alert definitions (Latency, Quorum, DP, Drift) |
| ├── grafana/ |
| │ ├── provisioning/ |
| │ │ ├── datasources/ |
| │ │ │ └── datasources.yml # Auto-provisioned Prometheus, Jaeger & Loki datasources |
| │ │ └── dashboards/ |
| │ │ └── dashboards.yml # Dashboard provider registration |
| │ └── dashboards/ |
| │ ├── cfi-overview.json # High-level HTTP throughput, latency p95 & error rates |
| │ ├── cfi_operational_dashboard.json # Active bank nodes, scoring throughput & latency p50/p95/p99 |
| │ └── cfi_privacy_budget_dashboard.json # DP epsilon consumption, budget projection & exhaustion |
| ├── loki/ |
| │ └── loki-config.yml # Grafana Loki log aggregation engine configuration |
| └── promtail/ |
| └── promtail-config.yml # Promtail container log shipping agent |
| ``` |
|
|
| --- |
|
|
| ## 2. Core Observability Components |
|
|
| ### 2.1 Prometheus Metrics Scraper (`prometheus.yml`) |
| - **Scrape Interval**: 15s (`evaluation_interval: 15s`). |
| - **Scrape Targets**: |
| - `cfi-gateway` (`gateway:8000/metrics`): Reverse proxy and public REST endpoints. |
| - `cfi-fl-coordinator` (`fl-coordinator:8001/metrics`): Federated Learning round progression and consensus. |
| - `cfi-identity-graph` (`identity-graph:8002/metrics`): Inductive GraphSAGE embeddings & UBO topology. |
| - `cfi-fraud-alert` (`fraud-alert:8003/metrics`): Real-time inference scoring, AML rules, and sanction screening. |
| - `prometheus` (`localhost:9090`): Self-monitoring. |
|
|
| ### 2.2 Alert Rules & SLA Invariants (`prometheus/alert_rules.yml`) |
| |
| The platform enforces four automated SLA alerts: |
| |
| | Alert Name | Severity | Condition / PromQL | Description | |
| |:---|:---|:---|:---| |
| | **`CFI_HighInferenceLatency`** | `critical` | `histogram_quantile(0.95, sum(rate(cfi_inference_latency_ms_bucket[5m])) by (le)) > 100` | Real-time transaction scoring p95 latency exceeds 100ms for > 2m. | |
| | **`CFI_QuorumRisk`** | `warning` | `cfi_active_bank_nodes < 2` | Active participating bank nodes drop below minimum consortium quorum (2 nodes) for > 5m. | |
| | **`CFI_DPBudgetDepleting`** | `warning` | `cfi_dp_epsilon_consumed_total > 6.4` | Differential privacy epsilon consumption exceeds 80% (6.4 / 8.0) of maximum budget. | |
| | **`CFI_ModelDrift`** | `critical` | `cfi_champion_model_auc < 0.75` | Champion global model holdout evaluation AUC drops below critical threshold 0.75. | |
| |
| ### 2.3 Alertmanager Incident Routing (`alertmanager/config.yml`) |
| - Groups alerts by `alertname` and `severity`. |
| - Emits webhooks to `http://gateway:8000/api/v1/monitoring/alerts/webhook` with `repeat_interval: 1h`. |
| |
| ### 2.4 Grafana Dashboards (`grafana/dashboards/`) |
| Auto-loaded upon container boot via Grafana provisioning: |
| 1. **`cfi-overview.json`**: |
| - HTTP Request Rate (`reqps`) |
| - Latency p95 (`ms`) |
| - Active In-Flight Requests |
| 2. **`cfi_operational_dashboard.json`**: |
| - Active Consortium Bank Nodes (Gauge: Red `<2`, Yellow `2`, Green `3+`) |
| - Real-Time Inference Latency (p50, p95, p99 timeseries) |
| - Transaction Scoring Volume Rate (Barchart) |
| 3. **`cfi_privacy_budget_dashboard.json`**: |
| - DP Epsilon Consumed per Bank Node (Stacked barchart) |
| - Time to Privacy Budget Exhaustion Projection (Hours to exhaustion) |
| - Remaining Privacy Budget Percentage (Gauge) |
|
|
| ### 2.5 Centralized Logging & Tracing |
| - **Loki (`loki/loki-config.yml`)**: Single-tenant TSDB engine storing compressed log chunks with 24h retention indices. |
| - **Promtail (`promtail/promtail-config.yml`)**: Scrapes container logs from `/var/log/containers/*log` and streams to Loki on port 3100. |
| - **Jaeger (`jaeger-ui-config.json`)**: Distributed trace viewer inspecting OpenTelemetry spans (`cfi_grpc_request_duration_seconds`, etc.). |
|
|
| --- |
|
|
| ## 3. Automated Verification |
|
|
| The monitoring configurations and dashboard schemas are continuously validated in CI: |
|
|
| ```bash |
| # Run telemetry & dashboard schema verification tests |
| python -m pytest backend/tests/unit/test_telemetry_metrics.py -v |
| ``` |
|
|
