File size: 5,027 Bytes
eb0499d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | # Enterprise Monitoring, Telemetry & Observability Mesh
This directory contains the production-grade observability stack for the **Privacy-Preserving Cross-Bank Fraud Detection Platform (CFI)**, integrating **Prometheus** (metrics aggregation), **Grafana** (real-time dashboards), **Alertmanager** (incident routing), **Loki** & **Promtail** (structured log shipping), and **Jaeger** (distributed tracing).
---
## 1. Directory Structure
```text
monitoring/
├── README.md # Telemetry architecture, metric dictionaries & SLA alerts
├── prometheus.yml # Prometheus server configuration & scrape targets
├── jaeger-ui-config.json # Jaeger distributed tracing UI configuration
├── alertmanager/
│ └── config.yml # Incident routing, alert grouping & webhook notifications
├── prometheus/
│ └── alert_rules.yml # Prometheus SLA alert definitions (Latency, Quorum, DP, Drift)
├── grafana/
│ ├── provisioning/
│ │ ├── datasources/
│ │ │ └── datasources.yml # Auto-provisioned Prometheus, Jaeger & Loki datasources
│ │ └── dashboards/
│ │ └── dashboards.yml # Dashboard provider registration
│ └── dashboards/
│ ├── cfi-overview.json # High-level HTTP throughput, latency p95 & error rates
│ ├── cfi_operational_dashboard.json # Active bank nodes, scoring throughput & latency p50/p95/p99
│ └── cfi_privacy_budget_dashboard.json # DP epsilon consumption, budget projection & exhaustion
├── loki/
│ └── loki-config.yml # Grafana Loki log aggregation engine configuration
└── promtail/
└── promtail-config.yml # Promtail container log shipping agent
```
---
## 2. Core Observability Components
### 2.1 Prometheus Metrics Scraper (`prometheus.yml`)
- **Scrape Interval**: 15s (`evaluation_interval: 15s`).
- **Scrape Targets**:
- `cfi-gateway` (`gateway:8000/metrics`): Reverse proxy and public REST endpoints.
- `cfi-fl-coordinator` (`fl-coordinator:8001/metrics`): Federated Learning round progression and consensus.
- `cfi-identity-graph` (`identity-graph:8002/metrics`): Inductive GraphSAGE embeddings & UBO topology.
- `cfi-fraud-alert` (`fraud-alert:8003/metrics`): Real-time inference scoring, AML rules, and sanction screening.
- `prometheus` (`localhost:9090`): Self-monitoring.
### 2.2 Alert Rules & SLA Invariants (`prometheus/alert_rules.yml`)
The platform enforces four automated SLA alerts:
| Alert Name | Severity | Condition / PromQL | Description |
|:---|:---|:---|:---|
| **`CFI_HighInferenceLatency`** | `critical` | `histogram_quantile(0.95, sum(rate(cfi_inference_latency_ms_bucket[5m])) by (le)) > 100` | Real-time transaction scoring p95 latency exceeds 100ms for > 2m. |
| **`CFI_QuorumRisk`** | `warning` | `cfi_active_bank_nodes < 2` | Active participating bank nodes drop below minimum consortium quorum (2 nodes) for > 5m. |
| **`CFI_DPBudgetDepleting`** | `warning` | `cfi_dp_epsilon_consumed_total > 6.4` | Differential privacy epsilon consumption exceeds 80% (6.4 / 8.0) of maximum budget. |
| **`CFI_ModelDrift`** | `critical` | `cfi_champion_model_auc < 0.75` | Champion global model holdout evaluation AUC drops below critical threshold 0.75. |
### 2.3 Alertmanager Incident Routing (`alertmanager/config.yml`)
- Groups alerts by `alertname` and `severity`.
- Emits webhooks to `http://gateway:8000/api/v1/monitoring/alerts/webhook` with `repeat_interval: 1h`.
### 2.4 Grafana Dashboards (`grafana/dashboards/`)
Auto-loaded upon container boot via Grafana provisioning:
1. **`cfi-overview.json`**:
- HTTP Request Rate (`reqps`)
- Latency p95 (`ms`)
- Active In-Flight Requests
2. **`cfi_operational_dashboard.json`**:
- Active Consortium Bank Nodes (Gauge: Red `<2`, Yellow `2`, Green `3+`)
- Real-Time Inference Latency (p50, p95, p99 timeseries)
- Transaction Scoring Volume Rate (Barchart)
3. **`cfi_privacy_budget_dashboard.json`**:
- DP Epsilon Consumed per Bank Node (Stacked barchart)
- Time to Privacy Budget Exhaustion Projection (Hours to exhaustion)
- Remaining Privacy Budget Percentage (Gauge)
### 2.5 Centralized Logging & Tracing
- **Loki (`loki/loki-config.yml`)**: Single-tenant TSDB engine storing compressed log chunks with 24h retention indices.
- **Promtail (`promtail/promtail-config.yml`)**: Scrapes container logs from `/var/log/containers/*log` and streams to Loki on port 3100.
- **Jaeger (`jaeger-ui-config.json`)**: Distributed trace viewer inspecting OpenTelemetry spans (`cfi_grpc_request_duration_seconds`, etc.).
---
## 3. Automated Verification
The monitoring configurations and dashboard schemas are continuously validated in CI:
```bash
# Run telemetry & dashboard schema verification tests
python -m pytest backend/tests/unit/test_telemetry_metrics.py -v
```
|