monitoring/README.md

Enterprise Monitoring, Telemetry & Observability Mesh

This directory contains the production-grade observability stack for the Privacy-Preserving Cross-Bank Fraud Detection Platform (CFI), integrating Prometheus (metrics aggregation), Grafana (real-time dashboards), Alertmanager (incident routing), Loki & Promtail (structured log shipping), and Jaeger (distributed tracing).

---

1. Directory Structure

monitoring/
├── README.md                          # Telemetry architecture, metric dictionaries & SLA alerts
├── prometheus.yml                     # Prometheus server configuration & scrape targets
├── jaeger-ui-config.json              # Jaeger distributed tracing UI configuration
├── alertmanager/
│   └── config.yml                     # Incident routing, alert grouping & webhook notifications
├── prometheus/
│   └── alert_rules.yml                # Prometheus SLA alert definitions (Latency, Quorum, DP, Drift)
├── grafana/
│   ├── provisioning/
│   │   ├── datasources/
│   │   │   └── datasources.yml        # Auto-provisioned Prometheus, Jaeger & Loki datasources
│   │   └── dashboards/
│   │       └── dashboards.yml         # Dashboard provider registration
│   └── dashboards/
│       ├── cfi-overview.json          # High-level HTTP throughput, latency p95 & error rates
│       ├── cfi_operational_dashboard.json # Active bank nodes, scoring throughput & latency p50/p95/p99
│       └── cfi_privacy_budget_dashboard.json # DP epsilon consumption, budget projection & exhaustion
├── loki/
│   └── loki-config.yml                # Grafana Loki log aggregation engine configuration
└── promtail/
    └── promtail-config.yml            # Promtail container log shipping agent

---

2. Core Observability Components

2.1 Prometheus Metrics Scraper (prometheus.yml)

• Scrape Interval: 15s (evaluation_interval: 15s).
• Scrape Targets:
  • cfi-gateway (gateway:8000/metrics): Reverse proxy and public REST endpoints.
  • cfi-fl-coordinator (fl-coordinator:8001/metrics): Federated Learning round progression and consensus.
  • cfi-identity-graph (identity-graph:8002/metrics): Inductive GraphSAGE embeddings & UBO topology.
  • cfi-fraud-alert (fraud-alert:8003/metrics): Real-time inference scoring, AML rules, and sanction screening.
  • prometheus (localhost:9090): Self-monitoring.

2.2 Alert Rules & SLA Invariants (prometheus/alert_rules.yml)

The platform enforces four automated SLA alerts:

Alert Name	Severity	Condition / PromQL	Description
CFI_HighInferenceLatency	critical	histogram_quantile(0.95, sum(rate(cfi_inference_latency_ms_bucket[5m])) by (le)) > 100	Real-time transaction scoring p95 latency exceeds 100ms for > 2m.
CFI_QuorumRisk	warning	cfi_active_bank_nodes < 2	Active participating bank nodes drop below minimum consortium quorum (2 nodes) for > 5m.
CFI_DPBudgetDepleting	warning	cfi_dp_epsilon_consumed_total > 6.4	Differential privacy epsilon consumption exceeds 80% (6.4 / 8.0) of maximum budget.
CFI_ModelDrift	critical	cfi_champion_model_auc < 0.75	Champion global model holdout evaluation AUC drops below critical threshold 0.75.

2.3 Alertmanager Incident Routing (alertmanager/config.yml)

• Groups alerts by alertname and severity.
• Emits webhooks to http://gateway:8000/api/v1/monitoring/alerts/webhook with repeat_interval: 1h.

2.4 Grafana Dashboards (grafana/dashboards/)

Auto-loaded upon container boot via Grafana provisioning:

1. cfi-overview.json:
   • HTTP Request Rate (reqps)
   • Latency p95 (ms)
   • Active In-Flight Requests
2. cfi_operational_dashboard.json:
   • Active Consortium Bank Nodes (Gauge: Red <2, Yellow 2, Green 3+)
   • Real-Time Inference Latency (p50, p95, p99 timeseries)
   • Transaction Scoring Volume Rate (Barchart)
3. cfi_privacy_budget_dashboard.json:
   • DP Epsilon Consumed per Bank Node (Stacked barchart)
   • Time to Privacy Budget Exhaustion Projection (Hours to exhaustion)
   • Remaining Privacy Budget Percentage (Gauge)

2.5 Centralized Logging & Tracing

• Loki (loki/loki-config.yml): Single-tenant TSDB engine storing compressed log chunks with 24h retention indices.
• Promtail (promtail/promtail-config.yml): Scrapes container logs from /var/log/containers/*log and streams to Loki on port 3100.
• Jaeger (jaeger-ui-config.json): Distributed trace viewer inspecting OpenTelemetry spans (cfi_grpc_request_duration_seconds, etc.).

---

3. Automated Verification

The monitoring configurations and dashboard schemas are continuously validated in CI:

# Run telemetry & dashboard schema verification tests
python -m pytest backend/tests/unit/test_telemetry_metrics.py -v