# Unified Experiment Tracking, Serialization & Publication Suite > **CF-Intelligence Experiment Infrastructure Specification** > **Package:** `experiments/harness/` > **Schema Version:** `1.0.0` (Pydantic v2 & TypeScript Synchronization) > **Status:** Production-Ready & Tested --- ## 1. Architectural Overview & Design Objectives The **Unified Experiment Infrastructure** provides a rigorous, reproducible framework for designing, executing, tracking, and publishing collaborative financial crime machine learning experiments. ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ EXPERIMENT HARNESS ARCHITECTURAL TOPOLOGY │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────┐ ┌────────────────────────┐ │ │ │ ExperimentConfig │ │ DatasetMetadata │ │ │ │ (Hyperparams/Seeds) │ │ (SHA-256 / Partitions) │ │ │ └──────────┬──────────┘ └───────────┬────────────┘ │ │ │ │ │ │ v v │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ ExperimentTracker │ │ │ │ - Context Manager (__enter__ / __exit__) │ │ │ │ - Automatic Hardware Environment Probing (CPU, RAM, Torch) │ │ │ │ - Automated Git Provenance Capture (SHA-1 / Branch) │ │ │ │ - Step-by-Step Training & Validation Trajectory Logging │ │ │ │ - Evaluation Curve Synthesis (ROC, PR, Brier, Confusion) │ │ │ └──────────────────────────────┬──────────────────────────────┘ │ │ │ │ │ v │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ ExperimentExporter │ │ │ │ - Atomic JSON Serialization (results.json) │ │ │ │ - Tabular Trajectory Logging (metrics.csv) │ │ │ │ - High-Performance Columnar Traces (traces.parquet) │ │ │ └──────────────┬──────────────────────────────┬───────────────┘ │ │ │ │ │ │ v v │ │ ┌──────────────────────────────┐ ┌────────────────────────────┐ │ │ │ Publication Plot Suite │ │ ReportCompiler │ │ │ │ - ROC Curve (300 DPI) │ │ - Executive Summary Matrix │ │ │ │ - PR Curve (Prevalence line) │ │ - Confusion Matrix Table │ │ │ │ - Reliability Diagram │ │ - Hardware & Hyperparams │ │ │ │ - Confusion Matrix Heatmap │ │ - REPORT.md Audit Dossier │ │ │ └──────────────────────────────┘ └────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ### Core Design Principles: 1. **Zero Fabrication & Deterministic Reproducibility**: - Every experiment run captures the exact `git_commit` SHA-1 hash, execution branch, dataset `sha256_hash`, evaluated random seeds, and host hardware environment. 2. **Atomic Disk Operations**: - Serialization to `results.json`, `metrics.csv`, and `traces.parquet` uses staged atomic file replacement (`.tmp` → target) to prevent corrupted states during unexpected halts. 3. **Multi-Seed Statistical Aggregation**: - Rather than single-seed evaluations, the runner aggregates across multiple random seeds (e.g. [42, 123, 456]), reporting mean ($\mu$), sample standard deviation ($\sigma$), min, max, and exact 95% confidence intervals ($\mu \pm 1.96 \cdot \frac{\sigma}{\sqrt{N}}$). 4. **Publication-Grade Visual Assets**: - Plots are generated directly from raw execution arrays using headless Matplotlib (`Agg` backend) at 300 DPI with tight bounding boxes, suitable for academic papers and regulatory dossiers. --- ## 2. Directory Layout & Artifact Specification ### 2.1 Standardized Per-Dataset Experiment Directory Topology Every canonical financial crime benchmark in `experiments//` satisfies a strict **5-Artifact Standard**: ``` experiments// ├── config.json # Machine-readable hyperparameter configuration ├── results.json # Authoritative execution contract & final metrics ├── metrics.csv # Step-by-step tabular trajectory & round history ├── report.md # Publication-grade Markdown audit dossier └── plots/ # High-resolution 300 DPI publication figures (*.png) ``` #### Canonical Dataset Experiment Inventory | Dataset Directory | Benchmark Scope | Core Model / Strategy | Primary Metrics Captured | Status | | :--- | :--- | :--- | :--- | :---: | | [`experiments/paysim/`](../experiments/paysim/) | PaySim Mobile Money Remittance | `PaySimNeuralClassifier` (FedAvg) | PR-AUC: 0.1184, ROC-AUC: 0.8700 | `STANDARDIZED` | | [`experiments/ieee_cis/`](../experiments/ieee_cis/) | IEEE-CIS Card-Not-Present Fraud | `DeepFraudMLP` (FedAvg/FedProx) | PR-AUC: 0.0691, ROC-AUC: 0.6632 | `STANDARDIZED` | | [`experiments/credit_card/`](../experiments/credit_card/) | European Credit Card PCA Fraud | `DeepFraudMLP` (FedAvg) | PR-AUC: 0.7750, ROC-AUC: 0.9837 | `STANDARDIZED` | | [`experiments/elliptic/`](../experiments/elliptic/) | Elliptic Bitcoin Blockchain Graph | `Inductive GraphSAGE (Multi-Seed)` | PR-AUC: 0.3761, ROC-AUC: 0.8325 | `STANDARDIZED` | | [`experiments/amlsim/`](../experiments/amlsim/) | IBM AMLSim Graph Network | GraphSAGE vs Tabular MLP | PR-AUC: 0.6527, Cycle Rec: 67.4% | `STANDARDIZED` | | [`experiments/synthaml/`](../experiments/synthaml/) | Spar Nord SynthAML Lookback | `DeepFraudMLP` (FedAvg) | PR-AUC: 0.9985, ROC-AUC: 0.9995 | `STANDARDIZED` | | [`experiments/amlnet/`](../experiments/amlnet/) | AUSTRAC Rare-Event AMLNet | `DeepFraudMLP` (FedAvg) | PR-AUC: 1.0000, ROC-AUC: 1.0000 | `STANDARDIZED` | | [`experiments/cross_bank/`](../experiments/cross_bank/) | CFI-CrossBank-01 Consortium | 3-Bank Consortium (FedAvg) | +55.59% Uplift, 836,000 USD Averted | `STANDARDIZED` | ### 2.2 Execution Run Directory Topology (`experiments/results/`) Dynamic experiment runs initiated via the runner harness create isolated execution snapshots: ``` experiments/results// ├── results.json # Authoritative machine-readable execution contract ├── metrics.csv # Step-by-step tabular trajectory ├── traces.parquet # High-performance binary columnar traces (PyArrow) ├── report.md # Senior-engineering Markdown audit dossier ├── summary.json # Multi-seed aggregate summary (for multi-seed runs) └── plots/ # 300 DPI publication figures ├── roc_curve.png ├── pr_curve.png ├── calibration_curve.png └── confusion_matrix.png ``` --- ## 3. Machine-Readable Schema Specification ### 3.1 Hardware Metadata (`HardwareMetadata`) Captured dynamically at execution initialization: - `os_platform`: Operating system family (`Windows`, `Linux`, `Darwin`). - `os_release` & `os_version`: OS kernel and build version. - `cpu_model` & `cpu_architecture`: Processor name and architecture string. - `cpu_physical_cores` & `cpu_logical_cores`: Core count from OS / `psutil`. - `total_ram_gb` & `available_ram_gb`: Physical host memory in Gigabytes. - `python_version` & `torch_version`: Runtime and neural library versions. - `cuda_available` & `device_name`: GPU acceleration state and device descriptor. ### 3.2 Dataset Metadata (`DatasetMetadata`) - `dataset_name`: Canonical identifier (e.g. `SyntheticFinancialConsortium`, `PaySim`, `IEEE-CIS`). - `sha256_hash`: 64-character SHA-256 checksum computed over raw record bytes. - `total_samples` & `num_features`: Matrix dimensionality. - `fraud_samples` & `fraud_rate`: Positive class distribution and prevalence. - `split_ratios`: Proportions assigned to `train`, `val`, and `test` splits. ### 3.3 Experiment Result Contract (`ExperimentResult`) Top-level payload stored in `results.json`: - `schema_version`: Semantic schema version (currently `1.0.0`). - `experiment_id`: Unique run identifier. - `config`: Hyperparameters, model architecture, strategy, and training parameters. - `hardware`: Environment snapshot. - `dataset`: Dataset integrity and feature distribution. - `git_commit`: Active git commit SHA-1. - `git_branch`: Active git branch. - `status`: Execution state (`PENDING`, `RUNNING`, `COMPLETED`, `FAILED`). - `start_time_utc` & `end_time_utc`: ISO 8601 timestamps. - `total_duration_seconds`: Monotonic runtime duration. - `final_metrics`: Key evaluation indicators (`pr_auc`, `roc_auc`, `f1_score`, `precision`, `recall`, `brier_score`). - `history`: List of `StepMetric` snapshots. - `curves`: Decimated $(x, y)$ coordinate pairs for ROC and PR curves. - `confusion_matrix`: Discrete binary confusion metrics ($TN, FP, FN, TP$). - `calibration`: Binned empirical fraud prevalence vs predicted probabilities. - `artifact_paths`: Manifest of relative links to generated files and figures. --- ## 4. Multi-Seed Aggregate Summary (`AggregateSummary`) When evaluating algorithms across varying seeds $S = \{s_1, s_2, \dots, s_n\}$: $$\mu = \frac{1}{n} \sum_{i=1}^n x_i, \quad s = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \mu)^2}$$ $$\mathrm{CI}_{0.95} = \left[ \max\left(0, \mu - 1.96 \cdot \frac{s}{\sqrt{n}}\right), \quad \min\left(1, \mu + 1.96 \cdot \frac{s}{\sqrt{n}}\right) \right]$$ Stored in `summary.json`: ```json { "experiment_name": "ProductionFraudMLP", "num_runs": 3, "seeds": [42, 123, 456], "metrics": { "pr_auc": { "mean": 0.812450, "std": 0.015230, "min": 0.795120, "max": 0.824900, "ci_95_lower": 0.795202, "ci_95_upper": 0.829698 }, "roc_auc": { "mean": 0.965400, "std": 0.004120, "min": 0.961000, "max": 0.969100, "ci_95_lower": 0.960737, "ci_95_upper": 0.970063 } } } ``` --- ## 5. Usage & CLI Quickstart ### 5.1 Programmatic Tracking (Python API) ```python from experiments.harness import ExperimentConfig, DatasetMetadata, ExperimentTracker config = ExperimentConfig( experiment_id="fed_fraud_v1", experiment_name="CrossBank_Fraud_Detection", model_type="FederatedMLP", strategy="FedAvg", num_rounds=10, seeds=[42], learning_rate=0.001, ) dataset_meta = DatasetMetadata.from_bytes( data=raw_csv_bytes, dataset_name="PaySim_Partition1", total_samples=10000, num_features=16, fraud_samples=500, ) with ExperimentTracker(config, dataset_meta) as tracker: for round_idx in range(1, 11): # ... client training and server aggregation ... tracker.log_step( step=round_idx, train_loss=loss, val_loss=val_loss, pr_auc=pr_auc, roc_auc=roc_auc, ) # Set final validation predictions to synthesize curves & confusion matrix tracker.set_evaluation_predictions(y_test, y_pred_prob) ``` ### 5.2 Command-Line Interface ```bash # 1. Standardize and compile report dossiers across all 8 canonical datasets python experiments/harness/compile_reports.py --all # 2. Verify artifact hierarchy compliance across all 8 datasets python experiments/harness/compile_reports.py --verify # 3. Standardize a specific dataset directory python experiments/harness/compile_reports.py --dataset paysim # 4. Run multi-seed experiment and compile execution snapshot python -m experiments.harness.runner --name FraudMLP_Production --rounds 5 --seeds 42 123 456 # 5. Generate all publication charts from raw benchmarks and experiments python scripts/generate_charts.py --include-experiments ``` ### 5.3 Make Targets ```bash make experiment-all # Runs multi-seed experiment harness and chart compilation make experiment-clean # Removes temporary artifacts ``` --- ## 6. Frontend Schema Synchronization TypeScript interfaces mirroring all Pydantic v2 schemas are maintained in: - `frontend/src/types/benchmark.ts` - Exported via `frontend/src/types.ts` Any frontend component or dashboard consuming `results.json` receives fully typed, validated models ensuring zero schema drift across the full stack. --- ## 7. Verification Test Suites Artifact standardization, file schemas, report completeness, and verification CLI operations are validated by dedicated automated unit tests: - [`backend/tests/unit/test_experiment_artifact_hierarchy.py`](../backend/tests/unit/test_experiment_artifact_hierarchy.py): 9 comprehensive unit tests asserting existence and integrity of all 5 artifacts across all 8 datasets (`paysim`, `ieee_cis`, `credit_card`, `elliptic`, `amlsim`, `synthaml`, `amlnet`, `cross_bank`), `config.json` schema, `metrics.csv` headers/rows, `report.md` required sections, `plots/` image validity, and `--verify` CLI exit code (**9 tests, 100% passing**).