File size: 15,234 Bytes
eff6797 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | # Unified Experiment Tracking, Serialization & Publication Suite
> **CF-Intelligence Experiment Infrastructure Specification**
> **Package:** `experiments/harness/`
> **Schema Version:** `1.0.0` (Pydantic v2 & TypeScript Synchronization)
> **Status:** Production-Ready & Tested
---
## 1. Architectural Overview & Design Objectives
The **Unified Experiment Infrastructure** provides a rigorous, reproducible framework for designing, executing, tracking, and publishing collaborative financial crime machine learning experiments.
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ EXPERIMENT HARNESS ARCHITECTURAL TOPOLOGY │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────┐ ┌────────────────────────┐ │
│ │ ExperimentConfig │ │ DatasetMetadata │ │
│ │ (Hyperparams/Seeds) │ │ (SHA-256 / Partitions) │ │
│ └──────────┬──────────┘ └───────────┬────────────┘ │
│ │ │ │
│ v v │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ ExperimentTracker │ │
│ │ - Context Manager (__enter__ / __exit__) │ │
│ │ - Automatic Hardware Environment Probing (CPU, RAM, Torch) │ │
│ │ - Automated Git Provenance Capture (SHA-1 / Branch) │ │
│ │ - Step-by-Step Training & Validation Trajectory Logging │ │
│ │ - Evaluation Curve Synthesis (ROC, PR, Brier, Confusion) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ v │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ ExperimentExporter │ │
│ │ - Atomic JSON Serialization (results.json) │ │
│ │ - Tabular Trajectory Logging (metrics.csv) │ │
│ │ - High-Performance Columnar Traces (traces.parquet) │ │
│ └──────────────┬──────────────────────────────┬───────────────┘ │
│ │ │ │
│ v v │
│ ┌──────────────────────────────┐ ┌────────────────────────────┐ │
│ │ Publication Plot Suite │ │ ReportCompiler │ │
│ │ - ROC Curve (300 DPI) │ │ - Executive Summary Matrix │ │
│ │ - PR Curve (Prevalence line) │ │ - Confusion Matrix Table │ │
│ │ - Reliability Diagram │ │ - Hardware & Hyperparams │ │
│ │ - Confusion Matrix Heatmap │ │ - REPORT.md Audit Dossier │ │
│ └──────────────────────────────┘ └────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### Core Design Principles:
1. **Zero Fabrication & Deterministic Reproducibility**:
- Every experiment run captures the exact `git_commit` SHA-1 hash, execution branch, dataset `sha256_hash`, evaluated random seeds, and host hardware environment.
2. **Atomic Disk Operations**:
- Serialization to `results.json`, `metrics.csv`, and `traces.parquet` uses staged atomic file replacement (`.tmp` → target) to prevent corrupted states during unexpected halts.
3. **Multi-Seed Statistical Aggregation**:
- Rather than single-seed evaluations, the runner aggregates across multiple random seeds (e.g. [42, 123, 456]), reporting mean ($\mu$), sample standard deviation ($\sigma$), min, max, and exact 95% confidence intervals ($\mu \pm 1.96 \cdot \frac{\sigma}{\sqrt{N}}$).
4. **Publication-Grade Visual Assets**:
- Plots are generated directly from raw execution arrays using headless Matplotlib (`Agg` backend) at 300 DPI with tight bounding boxes, suitable for academic papers and regulatory dossiers.
---
## 2. Directory Layout & Artifact Specification
### 2.1 Standardized Per-Dataset Experiment Directory Topology
Every canonical financial crime benchmark in `experiments/<dataset>/` satisfies a strict **5-Artifact Standard**:
```
experiments/<dataset>/
├── config.json # Machine-readable hyperparameter configuration
├── results.json # Authoritative execution contract & final metrics
├── metrics.csv # Step-by-step tabular trajectory & round history
├── report.md # Publication-grade Markdown audit dossier
└── plots/ # High-resolution 300 DPI publication figures (*.png)
```
#### Canonical Dataset Experiment Inventory
| Dataset Directory | Benchmark Scope | Core Model / Strategy | Primary Metrics Captured | Status |
| :--- | :--- | :--- | :--- | :---: |
| [`experiments/paysim/`](../experiments/paysim/) | PaySim Mobile Money Remittance | `PaySimNeuralClassifier` (FedAvg) | PR-AUC: 0.1184, ROC-AUC: 0.8700 | `STANDARDIZED` |
| [`experiments/ieee_cis/`](../experiments/ieee_cis/) | IEEE-CIS Card-Not-Present Fraud | `DeepFraudMLP` (FedAvg/FedProx) | PR-AUC: 0.0691, ROC-AUC: 0.6632 | `STANDARDIZED` |
| [`experiments/credit_card/`](../experiments/credit_card/) | European Credit Card PCA Fraud | `DeepFraudMLP` (FedAvg) | PR-AUC: 0.7750, ROC-AUC: 0.9837 | `STANDARDIZED` |
| [`experiments/elliptic/`](../experiments/elliptic/) | Elliptic Bitcoin Blockchain Graph | `Inductive GraphSAGE (Multi-Seed)` | PR-AUC: 0.3761, ROC-AUC: 0.8325 | `STANDARDIZED` |
| [`experiments/amlsim/`](../experiments/amlsim/) | IBM AMLSim Graph Network | GraphSAGE vs Tabular MLP | PR-AUC: 0.6527, Cycle Rec: 67.4% | `STANDARDIZED` |
| [`experiments/synthaml/`](../experiments/synthaml/) | Spar Nord SynthAML Lookback | `DeepFraudMLP` (FedAvg) | PR-AUC: 0.9985, ROC-AUC: 0.9995 | `STANDARDIZED` |
| [`experiments/amlnet/`](../experiments/amlnet/) | AUSTRAC Rare-Event AMLNet | `DeepFraudMLP` (FedAvg) | PR-AUC: 1.0000, ROC-AUC: 1.0000 | `STANDARDIZED` |
| [`experiments/cross_bank/`](../experiments/cross_bank/) | CFI-CrossBank-01 Consortium | 3-Bank Consortium (FedAvg) | +55.59% Uplift, 836,000 USD Averted | `STANDARDIZED` |
### 2.2 Execution Run Directory Topology (`experiments/results/`)
Dynamic experiment runs initiated via the runner harness create isolated execution snapshots:
```
experiments/results/<experiment_id>/
├── results.json # Authoritative machine-readable execution contract
├── metrics.csv # Step-by-step tabular trajectory
├── traces.parquet # High-performance binary columnar traces (PyArrow)
├── report.md # Senior-engineering Markdown audit dossier
├── summary.json # Multi-seed aggregate summary (for multi-seed runs)
└── plots/ # 300 DPI publication figures
├── roc_curve.png
├── pr_curve.png
├── calibration_curve.png
└── confusion_matrix.png
```
---
## 3. Machine-Readable Schema Specification
### 3.1 Hardware Metadata (`HardwareMetadata`)
Captured dynamically at execution initialization:
- `os_platform`: Operating system family (`Windows`, `Linux`, `Darwin`).
- `os_release` & `os_version`: OS kernel and build version.
- `cpu_model` & `cpu_architecture`: Processor name and architecture string.
- `cpu_physical_cores` & `cpu_logical_cores`: Core count from OS / `psutil`.
- `total_ram_gb` & `available_ram_gb`: Physical host memory in Gigabytes.
- `python_version` & `torch_version`: Runtime and neural library versions.
- `cuda_available` & `device_name`: GPU acceleration state and device descriptor.
### 3.2 Dataset Metadata (`DatasetMetadata`)
- `dataset_name`: Canonical identifier (e.g. `SyntheticFinancialConsortium`, `PaySim`, `IEEE-CIS`).
- `sha256_hash`: 64-character SHA-256 checksum computed over raw record bytes.
- `total_samples` & `num_features`: Matrix dimensionality.
- `fraud_samples` & `fraud_rate`: Positive class distribution and prevalence.
- `split_ratios`: Proportions assigned to `train`, `val`, and `test` splits.
### 3.3 Experiment Result Contract (`ExperimentResult`)
Top-level payload stored in `results.json`:
- `schema_version`: Semantic schema version (currently `1.0.0`).
- `experiment_id`: Unique run identifier.
- `config`: Hyperparameters, model architecture, strategy, and training parameters.
- `hardware`: Environment snapshot.
- `dataset`: Dataset integrity and feature distribution.
- `git_commit`: Active git commit SHA-1.
- `git_branch`: Active git branch.
- `status`: Execution state (`PENDING`, `RUNNING`, `COMPLETED`, `FAILED`).
- `start_time_utc` & `end_time_utc`: ISO 8601 timestamps.
- `total_duration_seconds`: Monotonic runtime duration.
- `final_metrics`: Key evaluation indicators (`pr_auc`, `roc_auc`, `f1_score`, `precision`, `recall`, `brier_score`).
- `history`: List of `StepMetric` snapshots.
- `curves`: Decimated $(x, y)$ coordinate pairs for ROC and PR curves.
- `confusion_matrix`: Discrete binary confusion metrics ($TN, FP, FN, TP$).
- `calibration`: Binned empirical fraud prevalence vs predicted probabilities.
- `artifact_paths`: Manifest of relative links to generated files and figures.
---
## 4. Multi-Seed Aggregate Summary (`AggregateSummary`)
When evaluating algorithms across varying seeds $S = \{s_1, s_2, \dots, s_n\}$:
$$\mu = \frac{1}{n} \sum_{i=1}^n x_i, \quad s = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \mu)^2}$$
$$\mathrm{CI}_{0.95} = \left[ \max\left(0, \mu - 1.96 \cdot \frac{s}{\sqrt{n}}\right), \quad \min\left(1, \mu + 1.96 \cdot \frac{s}{\sqrt{n}}\right) \right]$$
Stored in `summary.json`:
```json
{
"experiment_name": "ProductionFraudMLP",
"num_runs": 3,
"seeds": [42, 123, 456],
"metrics": {
"pr_auc": {
"mean": 0.812450,
"std": 0.015230,
"min": 0.795120,
"max": 0.824900,
"ci_95_lower": 0.795202,
"ci_95_upper": 0.829698
},
"roc_auc": {
"mean": 0.965400,
"std": 0.004120,
"min": 0.961000,
"max": 0.969100,
"ci_95_lower": 0.960737,
"ci_95_upper": 0.970063
}
}
}
```
---
## 5. Usage & CLI Quickstart
### 5.1 Programmatic Tracking (Python API)
```python
from experiments.harness import ExperimentConfig, DatasetMetadata, ExperimentTracker
config = ExperimentConfig(
experiment_id="fed_fraud_v1",
experiment_name="CrossBank_Fraud_Detection",
model_type="FederatedMLP",
strategy="FedAvg",
num_rounds=10,
seeds=[42],
learning_rate=0.001,
)
dataset_meta = DatasetMetadata.from_bytes(
data=raw_csv_bytes,
dataset_name="PaySim_Partition1",
total_samples=10000,
num_features=16,
fraud_samples=500,
)
with ExperimentTracker(config, dataset_meta) as tracker:
for round_idx in range(1, 11):
# ... client training and server aggregation ...
tracker.log_step(
step=round_idx,
train_loss=loss,
val_loss=val_loss,
pr_auc=pr_auc,
roc_auc=roc_auc,
)
# Set final validation predictions to synthesize curves & confusion matrix
tracker.set_evaluation_predictions(y_test, y_pred_prob)
```
### 5.2 Command-Line Interface
```bash
# 1. Standardize and compile report dossiers across all 8 canonical datasets
python experiments/harness/compile_reports.py --all
# 2. Verify artifact hierarchy compliance across all 8 datasets
python experiments/harness/compile_reports.py --verify
# 3. Standardize a specific dataset directory
python experiments/harness/compile_reports.py --dataset paysim
# 4. Run multi-seed experiment and compile execution snapshot
python -m experiments.harness.runner --name FraudMLP_Production --rounds 5 --seeds 42 123 456
# 5. Generate all publication charts from raw benchmarks and experiments
python scripts/generate_charts.py --include-experiments
```
### 5.3 Make Targets
```bash
make experiment-all # Runs multi-seed experiment harness and chart compilation
make experiment-clean # Removes temporary artifacts
```
---
## 6. Frontend Schema Synchronization
TypeScript interfaces mirroring all Pydantic v2 schemas are maintained in:
- `frontend/src/types/benchmark.ts`
- Exported via `frontend/src/types.ts`
Any frontend component or dashboard consuming `results.json` receives fully typed, validated models ensuring zero schema drift across the full stack.
---
## 7. Verification Test Suites
Artifact standardization, file schemas, report completeness, and verification CLI operations are validated by dedicated automated unit tests:
- [`backend/tests/unit/test_experiment_artifact_hierarchy.py`](../backend/tests/unit/test_experiment_artifact_hierarchy.py): 9 comprehensive unit tests asserting existence and integrity of all 5 artifacts across all 8 datasets (`paysim`, `ieee_cis`, `credit_card`, `elliptic`, `amlsim`, `synthaml`, `amlnet`, `cross_bank`), `config.json` schema, `metrics.csv` headers/rows, `report.md` required sections, `plots/` image validity, and `--verify` CLI exit code (**9 tests, 100% passing**).
|