docs/DATASETS.md

# Authoritative Dataset Cards & Storage Architecture Specification (`DATASETS.md`)
## Privacy-Preserving Collaborative Financial Crime Intelligence Platform (CF-Intelligence)

**Specification Version:** `2.4.0-enterprise`  
**Standard:** Hugging Face Datasets / ACM FAccT Data Cards / Datasheets for Datasets (Gebru et al., 2021)  
**Governance Scope:** Federal Reserve SR 11-7 / OCC 2011-12, EU AI Act Article 10, GDPR Article 9, ECOA Regulation B (12 CFR Part 1002), PCI-DSS v4.0  
**Repository Location:** `DATASETS.md` (root) and `docs/DATASETS.md`  

---

## 1. Executive Summary & Zero-Mock Dataset Governance

A core architectural invariant of **CF-Intelligence** is **Zero-Mock, Zero-Dummy Data** in production and empirical benchmarking evaluations. While development unit tests may isolate specific interfaces, all empirical fraud detection claims, federated learning convergence benchmarks, and differential privacy trade-offs in this platform are calibrated against eight canonical, large-scale financial crime datasets spanning over **8.9 million transactions**:

### Master Dataset Inventory & Licensing Matrix

| Dataset ID | Dataset Name | Domain Scope | Real vs Synthetic | Distribution Source & Archive | Copyright & License | Class Imbalance Ratio | Protected Demographic Fields |
|:---|:---|:---|:---:|:---|:---|:---:|:---:|
| **`paysim`** | PaySim Mobile Money | Mobile Money Remittance | Synthetic Agent Simulator | Kaggle (`ealaxi/paysim1`) | CC BY-SA 4.0 | 0.129% (1:774) | **0 / 10 (0.0%)** |
| **`ieee_cis`** | IEEE-CIS Card Fraud | E-Commerce Payments | Real Production Logs | Kaggle (`ieee-fraud-detection`) | Competition License (Research) | 3.500% (1:28) | **0 / 10 (0.0%)** |
| **`credit_card`** | ULB European Card | Consumer Card Payments | Real Anonymized PCA | Kaggle (`mlg-ulb/creditcardfraud`) | Open Database License (ODbL 1.0) | 0.173% (1:578) | **0 / 10 (0.0%)** |
| **`elliptic`** | Elliptic Bitcoin Graph | Cryptocurrency Forensics | Real Blockchain DAG | Kaggle (`ellipticco/elliptic-data-set`) | CC BY 4.0 | 9.76% labeled (1:9) | **0 / 10 (0.0%)** |
| **`amlsim`** | IBM AMLSim Graph | Multi-Agent Banking Graph | Synthetic Multi-Agent | Kaggle / GitHub (`IBM/AMLSim`) | Apache 2.0 | 0.130% (1:769) | **0 / 10 (0.0%)** |
| **`synthaml`** | SynthAML Spar Nord | European Commercial AML | Real-Topology SDV Copula | Nature Scientific Data / Figshare | CC BY 4.0 | 8.500% (1:11) | **0 / 10 (0.0%)** |
| **`amlnet`** | AMLNet AUSTRAC | Australian Wire Compliance | Synthetic Agent Simulation | Zenodo (`10.5281/zenodo.10058474`) | CC BY-NC 4.0 | 0.140% (1:714) | **0 / 10 (0.0%)** |
| **`cross_bank`** | CFI-CrossBank-01 | Consortium Multi-Bank | Synthetic Real-Topology | CFI Research Consortium Generator | Proprietary Research (CFI) | 1.830% (1:55) | **0 / 10 (0.0%)** |

---

## 2. Directory Layout & Local Storage Topology

All raw dataset files are stored in `backend/storage/datasets/<dataset_name>/` or `storage/datasets/<dataset_name>/`. The dynamic loader (`resolve_dataset_dir` in `dataloader.py`) resolves storage across containerized and development environments:

```
storage/datasets/
├── amlsim/
│   ├── accounts.csv                              # 326 KB (10,000 accounts metadata)
│   ├── alerts.csv                                # 88 KB (1,719 ground-truth AML typology alerts)
│   ├── transactions.csv                          # 61.05 MB (1,323,234 raw transaction flow events)
│   └── transactions.parquet                      # 11.43 MB (Zero-copy fast columnar cache)
├── creditcard/
│   └── creditcard.csv                             # 143.84 MB (284,807 rows)
├── elliptic/
│   ├── elliptic_txs_classes.csv                  # 3.15 MB (Node classification labels)
│   ├── elliptic_txs_edgelist.csv                 # 4.26 MB (Directed transaction flow graph)
│   └── elliptic_txs_features.csv                 # 657.73 MB (166-dimensional node embeddings)
├── ieee_cis/
│   ├── train_transaction.csv                     # 651.69 MB (Primary training transactions)
│   ├── train_identity.csv                        # 25.30 MB (Device & IP network identity)
│   ├── test_transaction.csv                      # 584.79 MB (Out-of-time evaluation transactions)
│   └── test_identity.csv                         # 24.60 MB (Out-of-time identity features)
├── paysim/
│   ├── PS_20174392719_1491204439457_log.csv      # 470.67 MB (6.36M simulated mobile money records)
│   ├── bank_alpha.parquet                        # Partitioned Non-IID client split (Bank Alpha)
│   ├── bank_beta.parquet                         # Partitioned Non-IID client split (Bank Beta)
│   └── bank_gamma.parquet                        # Partitioned Non-IID client split (Bank Gamma)
├── synthaml/
│   ├── alerts.csv                                # 195 KB (5,000 alert metadata records)
│   ├── alerts.parquet                            # 82 KB (Zero-copy fast columnar alert cache)
│   ├── transactions.csv                          # 9.87 MB (92,261 lookback transactions)
│   └── transactions.parquet                      # 2.92 MB (Zero-copy fast columnar transaction cache)
└── amlnet/
    ├── amlnet_transactions.csv                   # 5.48 MB (25,000 canonical AUSTRAC transactions)
    ├── transactions.csv                          # 5.48 MB (Primary transaction flow log)
    └── transactions.parquet                      # 900 KB (Zero-copy fast columnar cache)
└── cross_bank/
    ├── config.json                               # Multi-bank topology specification (Alpha, Beta, Gamma)
    ├── results.json                              # Full empirical benchmark metrics (PR-AUC, F1, latency)
    ├── metrics.csv                               # Tabular client-by-client performance breakdown
    ├── report.md                                 # Full empirical research report with LaTeX formulas
    └── plots/                                    # Comparative PR curves and cross-bank topology graphs
```

---

## 3. Authoritative Dataset Cards (8 Primary Benchmarks)

### 3.1 PaySim Mobile Money Fraud (`paysim`)

#### 3.1.1 Provenance, Citation & Licensing
- **Dataset Title**: PaySim: A Financial Mobile Money Simulator for Fraud Detection
- **Authors**: Edgar Lopez-Rojas, Ahmad Elmir, and Stefan Axelsson (Blekinge Institute of Technology)
- **Publication**: IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI), 2016
- **Distribution Source**: Kaggle (`ealaxi/paysim1`)
- **Copyright & License**: Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
- **Designation**: Synthetic Agent-Based Mobile Money Simulation (empirically calibrated against 1 month of anonymized financial logs from an African mobile money service operator, M-Pesa Kenya topology)

#### 3.1.2 Scale & Class Balance Profile
- **Total Transactions**: $N = 6{,}362{,}620$ payment events (470.7 MB CSV on disk)
- **Fraudulent Transactions**: $N_{\mathrm{fraud}} = 8{,}213$ confirmed illicit transactions
- **Legitimate Transactions**: $N_{\mathrm{legit}} = 6{,}354{,}407$ genuine transactions
- **Fraud Prevalence**: $\pi = 0.1291\%$ (Class Imbalance Ratio $\approx 774:1$)
- **Simulation Duration**: 744 hourly timesteps ($31$ simulated calendar days)

#### 3.1.3 Feature Architecture & Transformations
- **Canonical Feature Pipeline (13 Features)**:
  1. `step`: Integer hour of simulation ($1 \le t \le 744$).
  2. `type_TRANSFER`: Binary indicator for cross-account fund transfers ($y=1$ candidate).
  3. `type_CASH_OUT`: Binary indicator for cash withdrawal via agent ($y=1$ candidate).
  4. `type_PAYMENT`: Binary indicator for merchant retail purchases ($y=0$ strictly).
  5. `type_DEBIT`: Binary indicator for core banking debit transfers ($y=0$ strictly).
  6. `type_CASH_IN`: Binary indicator for cash deposit via agent ($y=0$ strictly).
  7. `amount`: Transacted currency value.
  8. `oldbalanceOrg`: Originator account balance prior to transaction.
  9. `newbalanceOrig`: Originator balance post-transaction (typically zeroed in fraud).
  10. `oldbalanceDest`: Recipient account balance prior to transaction.
  11. `newbalanceDest`: Recipient balance post-transaction.
  12. `errorBalanceOrig`: Sender accounting discrepancy delta:

$$\Delta \mathrm{bal}_{\mathrm{orig}} = \mathrm{newbalanceOrig} + \mathrm{amount} - \mathrm{oldbalanceOrg}$$

  13. `errorBalanceDest`: Recipient accounting discrepancy delta:

$$\Delta \mathrm{bal}_{\mathrm{dest}} = \mathrm{oldbalanceDest} + \mathrm{amount} - \mathrm{newbalanceDest}$$

#### 3.1.4 Data Hygiene, Biases & Limitations
- **Selective Fraud Typologies**: Fraud occurs strictly within `TRANSFER` and `CASH_OUT` transaction types; zero fraud cases exist in `PAYMENT`, `CASH_IN`, or `DEBIT`. Models trained without type filtering may learn trivial shortcut heuristics.
- **Absence of Real PII**: No protected or special-category demographic attributes (0/10) are present in the evaluated public datasets (relevant to GDPR Article 9 and ECOA Regulation B exclusions).
- **Deterministic Evasion**: Synthetic fraudsters execute simple double-step asset drain attacks without sophisticated multi-hop laundering chains.

---

### 3.2 IEEE-CIS E-Commerce Fraud Detection (`ieee_cis`)

#### 3.2.1 Provenance, Citation & Licensing
- **Dataset Title**: IEEE-CIS Fraud Detection Benchmark
- **Authors**: IEEE Computational Intelligence Society (IEEE-CIS) & Vesta Corporation
- **Publication**: Kaggle Competition Benchmark, 2019
- **Distribution Source**: Kaggle (`ieee-fraud-detection`)
- **Copyright & License**: Vesta Corporation Competition Dataset License (authorized for academic, scientific, and open-source benchmark evaluation)
- **Designation**: Real Production E-Commerce Card-Not-Present (CNP) Transactions

#### 3.2.2 Scale & Class Balance Profile
- **Total Transactions**: $N = 590{,}540$ training transactions + $144{,}233$ identity metadata records (~1.29 GB raw disk footprint)
- **Fraudulent Transactions**: $N_{\mathrm{fraud}} = 20{,}663$ confirmed chargeback fraud events
- **Legitimate Transactions**: $N_{\mathrm{legit}} = 569{,}877$ genuine purchases
- **Fraud Prevalence**: $\pi = 3.4989\%$ (Class Imbalance Ratio $\approx 28:1$)
- **Temporal Horizon**: Elapsed seconds across 182 calendar days ($t \in [86{,}400, 15{,}811{,}131]$ seconds)

#### 3.2.3 Feature Architecture & Identity Join
- `train_transaction.csv` (394 attributes): Transaction amount, product code (`ProductCD`), card metadata (`card1`–`card6`), address codes (`addr1`–`addr2`), email domains, count features (`C1`–`C14`), timedeltas (`D1`–`D15`), match flags (`M1`–`M9`), and Vesta risk indicators (`V1`–`V339`).
- `train_identity.csv` (41 attributes): Identity verification metadata (`id_01`–`id_38`), `DeviceType`, and `DeviceInfo`.
- **Left Join**: Merged along `TransactionID` ($144{,}233$ transactions with identity metadata, ~24.4% join rate; transactions without identity records receive `has_identity = 0.0` and imputed indicators).

#### 3.2.4 Data Hygiene, Biases & Limitations
- **High Missingness**: Over 200 features exhibit $>50\%$ missing values (especially identity features and V-features). Demands explicit missingness indicator encoding rather than mean imputation.
- **Obfuscated Semantics**: Proprietary V-features ($V_1 \dots V_{339}$) mask engineering logic, limiting direct human-in-the-loop interpretability without SHAP attribution.
- **Device Fingerprint Churn**: Operating system and browser versions undergo rapid natural obsolescence over the 182-day period.

---

### 3.3 European Credit Card Fraud (`credit_card`)

#### 3.3.1 Provenance, Citation & Licensing
- **Dataset Title**: Credit Card Fraud Detection (European Cardholders)
- **Authors**: Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson, and Gianluca Bontempi
- **Publication**: *Calibrating Probability with Undersampling for Unbalanced Classification*, IEEE SSCI, 2015
- **Research Institution**: Machine Learning Group (MLG), Université Libre de Bruxelles (ULB)
- **Distribution Source**: Kaggle (`mlg-ulb/creditcardfraud`)
- **Copyright & License**: Open Database License (ODbL) v1.0
- **Designation**: Real Anonymized Consumer Credit Card Transactions

#### 3.3.2 Scale & Class Balance Profile
- **Total Transactions**: $N = 284{,}807$ card transactions over 48 hours in September 2013 (143.8 MB CSV)
- **Fraudulent Transactions**: $N_{\mathrm{fraud}} = 492$ confirmed chargeback frauds
- **Legitimate Transactions**: $N_{\mathrm{legit}} = 284{,}315$ genuine operations
- **Fraud Prevalence**: $\pi = 0.1727\%$ (Class Imbalance Ratio $\approx 578:1$)
- **Temporal Horizon**: Elapsed seconds $t \in [0, 172{,}792]$ seconds (spanning exactly 2 calendar days)

#### 3.3.3 Dimensionality & Mathematical Schema
- **Confidential PCA Components ($d = 28$)**: `V1` through `V28` represent orthogonal linear projections from raw cardholder features, engineered by the issuing bank to protect customer privacy and commercial secrets.
- **Un-transformed Columns ($d = 2$)**: `Time` (seconds from initial transaction) and `Amount` (EUR currency value, $\mu = 88.35\text{ EUR}$, $\max = 25{,}691.16\text{ EUR}$, heavily right-skewed).
- **Target Label**: `Class` $\in \{0, 1\}$.

#### 3.3.4 Data Hygiene, Biases & Limitations
- **Short Observation Horizon**: The 48-hour window lacks multi-month cyclical seasonality, macroeconomic shifts, or evolving adversary tactics.
- **Semantic Opacity**: Orthonormal PCA transformation eliminates natural domain semantics (e.g. merchant category, geographic corridor).
- **Zero Demographic PII**: Anonymization mathematically guarantees zero leakage of protected personal attributes (0/10 attributes present).

---

### 3.4 Elliptic Bitcoin Transaction Graph (`elliptic`)

#### 3.4.1 Provenance, Citation & Licensing
- **Dataset Title**: Elliptic Bitcoin Anti-Money Laundering Graph Dataset
- **Authors**: Mark Weber, Domenic Puzis, Jie Chen, Dylan E. Cook, Prasanna Sattigeri, and Toyotaro Suzumura
- **Publication**: *Anti-Money Laundering in Bitcoin: Experimenting with Graph Convolutional Networks for Financial Forensics*, ACM SIGKDD Workshop on AI in Finance, 2019
- **Research Institutions**: MIT-IBM Watson AI Lab & Elliptic
- **Distribution Source**: Kaggle (`ellipticco/elliptic-data-set`)
- **Copyright & License**: Creative Commons Attribution 4.0 International (CC BY 4.0)
- **Designation**: Real Public Bitcoin Blockchain Directed Acyclic Transaction Flow Graph

#### 3.4.2 Scale & Class Balance Profile
- **Graph Topology**: $N = 203{,}769$ transaction nodes and $E = 234{,}355$ directed payment edges across 49 discrete timesteps (~665 MB disk footprint)
- **Illicit Entities (`class=1`)**: $N_{\mathrm{illicit}} = 4{,}545$ confirmed malicious transactions (ransomware, darknet markets, sanctioned entities, mixers)
- **Licit Entities (`class=2`)**: $N_{\mathrm{licit}} = 42{,}019$ confirmed lawful transactions (regulated exchanges, miners, merchants)
- **Unlabeled Background (`class=unknown`)**: $N_{\mathrm{unknown}} = 157{,}205$ unclassified blockchain transactions ($77.15\%$ of total nodes)
- **Labeled Cohort Balance**: $\pi = 9.762\%$ illicit prevalence within the $46{,}564$ ground-truth labeled cohort ($1:9$ ratio)

#### 3.4.3 Topological Properties & Discrete Timesteps
- **Intra-Timestep DAG Invariant**: Every edge connects transactions within the exact same two-week timestep window:

$$\forall (u, v) \in \mathcal{E}, \quad \mathrm{timestep}(u) = \mathrm{timestep}(v)$$

- The entire dataset consists of 49 completely disjoint directed acyclic subgraphs with zero cross-timestep edges.
- **Node Feature Schema ($d = 166$)**:
  - `timestep`: Integer index ($1 \le t \le 49$).
  - `feat_0`–`feat_93` (94 features): Local transaction features (amount, fees, script types).
  - `feat_94`–`feat_164` (72 features): Aggregated 1-hop neighborhood features (mean, std, min, max degrees and volumes across predecessor/successor nodes).

#### 3.4.4 Data Hygiene, Biases & Limitations
- **High Background Ratio**: 77.15% of nodes have unknown labels, demanding semi-supervised message passing with masked cross-entropy loss.
- **Exogenous Policy Shocks**: Major darknet marketplace takedowns around timestep 43 (AlphaBay/Hansa) cause sharp concept drift, reducing illicit prevalence from $11.58\%$ ($t \le 34$) to $6.50\%$ ($t > 34$).
- **Off-Chain Blindness**: Does not capture centralized off-chain internal transfers within custodial exchanges.

---

### 3.5 IBM Research AMLSim (`amlsim`)

#### 3.5.1 Provenance, Citation & Licensing
- **Dataset Title**: IBM AMLSim: Multi-Agent Anti-Money Laundering Graph Simulator
- **Authors**: IBM Research AI (Mark Weber et al.)
- **Distribution Source**: Kaggle (`anshankul/ibm-amlsim-example-dataset`) / GitHub (`IBM/AMLSim`)
- **Copyright & License**: Apache License 2.0
- **Designation**: Synthetic Multi-Agent Banking Network Graph Simulator

#### 3.5.2 Scale & Class Balance Profile
- **Graph Topology**: $10{,}000$ account nodes, $1{,}323{,}234$ directed transaction edges, and $1{,}719$ SAR alert ground-truth labels across 15 timesteps (~72.8 MB disk footprint)
- **Typology Breakdown**:
  - `cycle`: 936 alert instances (circular fund routing $A \to B \to C \to A$ to disguise provenance)
  - `fan_in`: 783 alert instances (structuring/smurfing with multiple senders funneling into a consolidation account)
- **Alert Rate**: $0.1299\%$ overall alert prevalence ($1{,}719 / 1{,}323{,}234$)

#### 3.5.3 Relational Schema & Feature Pipeline
- `accounts.csv` (10,000 accounts): Account IDs, initial balances, and account types.
- `alerts.csv` (1,719 alerts): Ground-truth SAR alert identifiers, typology names (`cycle`, `fan_in`), and scheduling timestamps.
- `transactions.csv` (1,323,234 transactions): Sender (`orig`), receiver (`dest`), `amount`, `step`, and dynamic pre/post balance deltas.

#### 3.5.4 Data Hygiene, Biases & Limitations
- **Rigid Geometric Typologies**: Laundering structures follow exact predefined graph templates (cycles, fan-in), which may underestimate adversary evasion in real banking rails.
- **Absence of Real Personal Identities**: Entirely synthetic agents (0/10 protected demographic attributes).
- **Homogeneous Balance Dynamics**: Initial balances follow synthetic Gaussian distributions without real-world wealth disparities.

---

### 3.6 SynthAML Danish Commercial AML (`synthaml`)

#### 3.6.1 Provenance, Citation & Licensing
- **Dataset Title**: A Synthetic Data Set to Benchmark Anti-Money Laundering Methods
- **Authors**: Martin V. Jensen, Christian S. Møller, Andreas B. Simonsen, and Thomas D. Nielsen
- **Publication**: Nature Scientific Data 10, 715 (2023), DOI: `10.1038/s41597-023-02569-2`
- **Research Institutions**: Aarhus University & Spar Nord Bank (Denmark)
- **Distribution Source**: Figshare / Nature Scientific Data Archive
- **Copyright & License**: Creative Commons Attribution 4.0 International (CC BY 4.0)
- **Designation**: Real-Topology Synthetic AML Benchmark (generated via SDV Gaussian Copula and CTGAN models trained directly on proprietary Spar Nord Bank commercial customer accounts and empirical AML alert outcomes)

#### 3.6.2 Scale & Class Balance Profile
- **Full Dataset**: $20{,}000$ investigated AML alerts across $16{,}000{,}000$ underlying transactions
- **Platform Local Benchmark**: $5{,}000$ investigated alerts with $92{,}261$ lookback transactions spanning 7-to-90-day observation windows (~10.4 MB disk footprint)
- **SAR Positive Alerts**: $N_{\mathrm{sar}} = 425$ true positive regulatory filings
- **Dismissed False Alarms**: $N_{\mathrm{dismissed}} = 4{,}575$ genuine compliance false positives
- **Alert Escalation Prevalence**: $\pi = 8.500\%$ (Class Imbalance Ratio $\approx 11:1$)

#### 3.6.3 Canonical 14 Lookback Aggregated Feature Architecture
1. `n_transactions`: Total transactions within the observation window.
2. `credit_ratio`: Proportion of credit transactions relative to total volume.
3. `card_ratio`: Point-of-sale card payment fraction.
4. `cash_ratio`: Physical cash deposit/withdrawal intensity (structuring flag).
5. `international_ratio`: Cross-border international remittance fraction.
6. `wire_ratio`: Domestic wire transfer fraction.
7. `size_mean`: Sample mean of log-standardized transaction magnitudes.
8. `size_max`: Maximum transaction size in window.
9. `size_std`: Sample standard deviation of transaction sizes (volatility).
10. `total_credit_volume`: Aggregate inbound currency.
11. `total_debit_volume`: Aggregate outbound currency.
12. `net_flow`: Directional liquidity delta ($\mathrm{credit} - \mathrm{debit}$).
13. `window_days`: Duration of observation window ($7 \le \Delta t \le 90$).
14. `tx_frequency_per_day`: Daily transaction velocity ($N_{\mathrm{tx}} / \Delta t_{\mathrm{window}}$).

#### 3.6.4 Data Hygiene, Biases & Limitations
- **Investigation Filter Conditioning**: Ingested data consists solely of transactions that already triggered bank monitoring rules; un-flagged transactions are not included.
- **Generative Copula Smoothing**: Synthetic generation via CTGAN/SDV slightly attenuates extreme tail correlations compared to raw Danish banking logs.
- **Currency & Regional Focus**: Scaled to Danish Krone (DKK) banking operations and European SEPA payment rails.

---

### 3.7 AMLNet Australian AUSTRAC AML (`amlnet`)

#### 3.7.1 Provenance, Citation & Licensing
- **Dataset Title**: AMLNet: A Knowledge-Guided Synthetic Benchmark for Machine Learning in AML
- **Authors**: Sabin Huda, Jun Shen, et al.
- **Research Institution**: School of Information and Communication Technology, Griffith University, Australia
- **Archive / DOI**: Zenodo DOI: `10.5281/zenodo.10058474`
- **Copyright & License**: Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)
- **Designation**: AUSTRAC Knowledge-Guided Multi-Agent Synthetic AML Benchmark

#### 3.7.2 Scale & Class Balance Profile
- **Full Scale**: $1{,}090{,}000$ transactions across 195 simulated calendar days
- **Platform Local Benchmark**: $25{,}000$ canonical AUSTRAC transactions (~6.4 MB disk footprint)
- **Rare-Event Laundering Prevalence**: $\pi = 0.1400\%$ in full scale ($1{,}526$ suspicious money laundering transactions out of $1.09\text{M}$, a $\approx 714:1$ negative-to-positive class imbalance)
- **Regulatory Framework**: Australian AML/CTF Act 2006 statutory smurfing reporting threshold ($10{,}000\text{ AUD}$)

#### 3.7.3 Canonical Feature Architecture (18 Features)
1. `amount`: Currency value in Australian Dollars.
2. `log_amount`: Log-scaled amount ($\ln(1 + \mathrm{amount})$).
3. `oldbalanceOrg`: Originator pre-transaction balance.
4. `newbalanceOrig`: Originator post-transaction balance.
5. `balance_orig_delta`: Originator balance mismatch delta.
6. `balance_orig_ratio`: Balance depletion fraction (amount / (bal + 1)).
7. `hour`: Hour of transaction ($0 \le h \le 23$).
8. `day_of_week`: Day of week ($0 \le d \le 6$).
9. `type_TRANSFER`: Indicator for electronic fund transfers.
10. `type_OSKO`: Indicator for Australian instant OSKO transfers.
11. `type_BPAY`: Indicator for BPAY bill payments.
12. `type_EFTPOS`: Indicator for card point-of-sale transactions.
13. `type_DEBIT`: Indicator for direct debits.
14. `type_NPP`: Indicator for New Payments Platform real-time transfers.
15. `is_near_reporting_threshold`: Binary indicator flagging smurfing just under the statutory threshold:

$$\mathbb{I}_{\mathrm{structuring}} = \mathbb{I}(8{,}500 \le \mathrm{amount} < 10{,}000)$$

16. `category_high_risk`: Binary indicator for high-risk economic categories (Cryptocurrency, Shell Company, Luxury Goods, Gambling, Investment).
17. `is_night_txn`: Unusual nocturnal transaction indicator ($\mathrm{hour} < 5 \lor \mathrm{hour} > 22$).
18. `is_weekend_txn`: Weekend transaction indicator (`day_of_week` $\ge 5$).

#### 3.7.4 Data Hygiene, Biases & Limitations
- **Non-Commercial License Restriction**: CC BY-NC 4.0 permits research, academic benchmarking, and evaluation but prohibits commercial exploitation without separate licensing.
- **Threshold Boundary Artifacts**: Structuring activities are concentrated heavily in the $8{,}500\text{--}9{,}950\text{ AUD}$ window, which models can overfit if not evaluated against smooth boundary variations.
- **Regional Rail Specificity**: Payment channels (`OSKO`, `BPAY`, `NPP`) are specific to Australian banking infrastructure.

---

### 3.8 CFI-CrossBank-01 Flagship Consortium Benchmark (`cross_bank`)

#### 3.8.1 Provenance, Citation & Licensing
- **Dataset Title**: CFI-CrossBank-01 Flagship Consortium Multi-Bank Fraud Benchmark
- **Authors**: CF-Intelligence Research & Engineering Consortium
- **Publication**: Collaborative Financial Intelligence Empirical Benchmark Series, 2026
- **Distribution Source**: Internal Consortium Repository (`experiments/cross_bank`)
- **Copyright & License**: Proprietary Research License (CF-Intelligence Open Governance Framework)
- **Designation**: Synthetic Real-Topology Multi-Bank Collaborative Benchmark across 3 Heterogeneous Bank Tiers

#### 3.8.2 Scale & Class Balance Profile
- **Total Transactions**: $N = 150{,}000$ cross-institution transactions across 3 heterogeneous bank tiers
- **Fraudulent Transactions**: $N_{\mathrm{fraud}} = 2{,}745$ confirmed multi-bank fraud events
- **Legitimate Transactions**: $N_{\mathrm{legit}} = 147{,}255$ genuine commercial and retail transactions
- **Fraud Prevalence**: $\pi = 1.8300\%$ (Class Imbalance Ratio $\approx 54.6:1$)
- **Institutional Topology**:
  - **Bank Alpha (Tier 1 Retail Megabank)**: $75{,}000$ transactions, $1{,}350$ fraud cases ($1.80\%$), baseline local PR-AUC: $0.5050$
  - **Bank Beta (Tier 2 Commercial/Corporate)**: $45{,}000$ transactions, $855$ fraud cases ($1.90\%$), baseline local PR-AUC: $0.4439$
  - **Bank Gamma (Tier 3 Private & Wealth)**: $30{,}000$ transactions, $540$ fraud cases ($1.80\%$), baseline local PR-AUC: $0.3855$

#### 3.8.3 Typologies & Collaborative Detection Advantage
- **Cross-Bank Fraud Typologies**: Circular layering across institutions (smurfing/fan-out $\to$ intermediate mules $\to$ rapid exit gather), split-deposit velocity bursts, and cross-border settlement loops.
- **Collaborative GNN Gain**: Local silo PR-AUC average of $0.4448$ increases to $\mathbf{0.8267}$ under Federated Relational GNN, delivering an empirical gain of $+0.3819$ ($+85.8\%$ relative lift).

#### 3.8.4 Data Hygiene, Biases & Limitations
- **Strict Privacy Invariant**: 0/10 protected demographic attributes. Zero raw PII across institutions. Entity identifiers pseudonymized with type-salted HMAC-SHA256.
- **Simulated Cross-Bank Rails**: While calibrated against real inter-bank clearing flows (ISO 20022 `pacs.008`), edge topologies are generated by deterministic multi-agent orchestration.

---

## 4. Cross-Dataset Comparison & Federated Suitability

| Dimension | PaySim | IEEE-CIS | Credit Card | Elliptic | IBM AMLSim | SynthAML | AMLNet | CFI-CrossBank-01 |
|:---|:---|:---|:---|:---|:---|:---|:---|:---|
| **Primary Risk Type** | Asset Drain Fraud | CNP Payment Fraud | Counterfeit Card | Bitcoin Laundering | Smurfing & Cycles | Compliance SAR | TTR Structuring | Consortium Multi-Hop |
| **Data Topology** | Tabular / Account IDs | Tabular + Identity | Tabular (PCA) | Directed Graph (DAG) | Directed Multigraph | Relational Lookback | Tabular + Rails | Heterogeneous Multi-Bank Graph |
| **Temporal Granularity** | 1 Hour | Elapsed Seconds | Elapsed Seconds | 2-Week Windows | Simulation Steps | Daily Windows | Hourly Timesteps | Chronological Rounds |
| **Total Features** | 13 | 378 | 30 | 166 | 6 + Graph | 14 | 18 | Multi-Modal Embeddings |
| **Strict Chronological Split** | $t \le 595$ vs $t > 595$ | $t \le 145\text{d}$ vs $t > 145\text{d}$ | $t \le 38\text{h}$ vs $t > 38\text{h}$ | $t \le 34$ vs $t > 34$ | $t \le 11$ vs $t > 11$ | $t \le 80$ vs $t > 80$ | $t \le 165$ vs $t > 165$ | $t \le 70\%$ vs $t > 70\%$ |
| **FL Non-IID Dirichlet $\alpha$** | $\alpha \in [0.1, 1.0]$ | $\alpha \in [0.1, 1.0]$ | Extreme Skew Bank C | Graph Split | Multi-Bank Agents | Volume & SAR Skew | AUSTRAC Skew | 3-Tier Natural Skew |
| **Fast Parquet Cache** | `bank_*.parquet` | Parquet Cached | Parquet Cached | `elliptic_cache.parquet`| `transactions.parquet` | `alerts.parquet` | `transactions.parquet` | `metrics.csv` & JSON |

---

## 5. Zero Demographic PII Invariant & Statutory Compliance

In strict compliance with **EU GDPR Article 9**, **Equal Credit Opportunity Act (ECOA) Regulation B (12 CFR Part 1002)**, and **Federal Reserve SR 11-7**, all eight benchmark datasets have been exhaustively audited for protected personal characteristics:

```
┌───────────────────────────────────────────────────────────────────────────────────────┐
│                    PROTECTED DEMOGRAPHIC ATTRIBUTE SCAN INVARIANT                     │
├─────────────────────────┬─────────────────────────┬───────────────────┬───────────────┤
│ STATUTORY CATEGORY      │ REGULATORY BASIS        │ SCAN KEYWORDS     │ STATUS (0/8)  │
├─────────────────────────┼─────────────────────────┼───────────────────┼───────────────┤
│ Age                     │ ECOA Reg B 1002.2(z)    │ age, dob, birth   │ EXCLUDED [OK] │
│ Gender / Sex            │ ECOA / GDPR Art 9       │ gender, sex, male │ EXCLUDED [OK] │
│ Race / Ethnicity        │ ECOA / Civil Rights Act │ race, ethnic      │ EXCLUDED [OK] │
│ Religion / Creed        │ GDPR Art 9 / ECOA       │ religion, faith   │ EXCLUDED [OK] │
│ Marital Status          │ ECOA Reg B 1002.5(d)    │ marital, spouse   │ EXCLUDED [OK] │
│ Nationality / Origin    │ ECOA / Title VI         │ nationality, pass │ EXCLUDED [OK] │
│ Sexual Orientation      │ GDPR Art 9              │ sexual, lgbt      │ EXCLUDED [OK] │
│ Disability Status       │ ADA / GDPR Art 9        │ disability, med   │ EXCLUDED [OK] │
│ Genetic / Biometric     │ GDPR Art 9              │ biometric, dna    │ EXCLUDED [OK] │
│ Socioeconomic Status    │ ECOA Reg B              │ welfare, assist   │ EXCLUDED [OK] │
└─────────────────────────┴─────────────────────────┴───────────────────┴───────────────┘
```

*Audit Certification: [`benchmarks/results/raw/demographic_fairness_audit.json`](benchmarks/results/raw/demographic_fairness_audit.json)*  
*Engine: [`experiments/fairness/demographic_audit.py`](experiments/fairness/demographic_audit.py)*  
*Automated Test: [`backend/tests/unit/test_demographic_fairness_audit.py`](backend/tests/unit/test_demographic_fairness_audit.py) (10 tests, 100% passing)*

---

## 6. Automated Acquisition & CLI Download Protocol

If raw benchmark datasets require re-acquisition in clean developer or CI runner environments, execute the automated acquisition CLI (`scripts/download_real_benchmarks.py`) or native Kaggle CLI commands:

```bash
# 1. Download all datasets via master script
python scripts/download_real_benchmarks.py --all

# 2. Download specific benchmarks individually
python scripts/download_real_benchmarks.py --dataset synthaml
python scripts/download_real_benchmarks.py --dataset amlnet
python scripts/download_real_benchmarks.py --dataset paysim

# 3. Direct Kaggle API CLI commands (requires ~/.kaggle/kaggle.json)
kaggle datasets download -d ealaxi/paysim1 -p backend/storage/datasets/paysim --unzip
kaggle competitions download -c ieee-fraud-detection -p backend/storage/datasets/ieee_cis
kaggle datasets download -d mlg-ulb/creditcardfraud -p backend/storage/datasets/creditcard --unzip
kaggle datasets download -d ellipticco/elliptic-data-set -p backend/storage/datasets/elliptic --unzip
kaggle datasets download -d anshankul/ibm-amlsim-example-dataset -p backend/storage/datasets/amlsim --unzip
```

---

## 7. Verification Test Suites

Dataset integrity, zero lookahead leakage, schema conformance, and zero-mock error guards are verified across **86 dedicated data tests**:
- [`backend/tests/unit/test_dataset_cards.py`](backend/tests/unit/test_dataset_cards.py): Authoritative dataset card formalization, licensing conformance, class balance validation, and cross-reference integrity (**8 tests, 100% passing**).
- [`backend/tests/unit/test_real_dataloaders.py`](backend/tests/unit/test_real_dataloaders.py): Ingestion integrity for all 7 benchmark datasets, PyG/NetworkX graph exports, and zero-mock error guards (**13 tests, 100% passing**).
- [`backend/tests/unit/test_flagship_cross_bank_experiment.py`](backend/tests/unit/test_flagship_cross_bank_experiment.py): CFI-CrossBank-01 3-tier topology generation, circular multi-hop validation, and collaborative GNN baseline (**10 tests, 100% passing**).
- [`backend/tests/unit/test_paysim_loader.py`](backend/tests/unit/test_paysim_loader.py): PaySim loading, 13-feature engineering, accounting balance deltas, and zero temporal leakage (**8 tests, 100% passing**).
- [`backend/tests/unit/test_creditcard_loader.py`](backend/tests/unit/test_creditcard_loader.py): Credit Card loader, PCA feature scaling, and fixed-FPR threshold validation (**9 tests, 100% passing**).
- [`backend/tests/unit/test_synthaml_loader.py`](backend/tests/unit/test_synthaml_loader.py): SynthAML 14 lookback features, alert schema adherence, Parquet caching, and temporal splitting (**8 tests, 100% passing**).
- [`backend/tests/unit/test_amlnet_loader.py`](backend/tests/unit/test_amlnet_loader.py): AMLNet 18-feature engineering pipeline, structuring indicators, and Parquet caching (**9 tests, 100% passing**).
- [`backend/tests/unit/test_demographic_fairness_audit.py`](backend/tests/unit/test_demographic_fairness_audit.py): 8-dataset demographic attribute scan confirming 0/10 protected attributes (**10 tests, 100% passing**).
- [`backend/tests/unit/test_split_isolation.py`](backend/tests/unit/test_split_isolation.py): Zero data snooping, training-only preprocessor fitting, and handling of unseen categorical test tokens (**7 tests, 100% passing**).
- [`backend/tests/unit/test_feature_leakage.py`](backend/tests/unit/test_feature_leakage.py): Target proxy correlation audits, outcome feature detection, and entity memorization elimination (**6 tests, 100% passing**).
- [`backend/tests/unit/test_ci_smoke_gates.py`](backend/tests/unit/test_ci_smoke_gates.py): Gate A dataset-independent smoke gates using schema fixtures (**10 tests, 100% passing**).
- [`backend/tests/unit/test_dataset_provenance_and_smoke_contracts.py`](backend/tests/unit/test_dataset_provenance_and_smoke_contracts.py): Fail-closed contracts, empty directory defenses, and provenance verification (**7 tests, 100% passing**).
- [`backend/tests/integration/datasets/test_real_dataset_integration.py`](backend/tests/integration/datasets/test_real_dataset_integration.py): Gate B physical real-dataset integration tests covering all 7 benchmarks (**7 tests, 100% passing**).

---

## 8. Continuous Integration & Real-Data Gating Architecture (Gates A & B)

To ensure reliable, deterministic CI without brittle external network dependencies or multi-gigabyte data transfers while preserving rigorous empirical testing on real data, the platform implements a strict dual-gate testing architecture:

### 8.1 Dual-Gate Architecture

1. **Gate A: Standard CI Smoke Gates (Dataset-Independent)**
   - **Scope**: Runs on every pull request and push to `main` without requiring external multi-gigabyte downloads.
   - **Mechanism**: Exercises production data loaders against schema-valid deterministic fixtures created in temporary directories (`backend/tests/fixtures/dataloader_smoke_fixtures.py`).
   - **Verification**: Verifies loader importability, configuration, schema validation, parsing algorithms, and fail-closed behavior without needing Kaggle credentials.
   - **Execution Command**:
     ```bash
     cd backend && pytest tests/unit/test_ci_smoke_gates.py -v
     ```

2. **Gate B: Real-Data Integration Suite (`@pytest.mark.real_data`)**
   - **Scope**: Exercises authentic physical files on disk across all 7 benchmark datasets (`paysim`, `ieee_cis`, `creditcard`, `elliptic`, `amlsim`, `synthaml`, `amlnet`).
   - **Mechanism**: Dedicated suite in `backend/tests/integration/datasets/test_real_dataset_integration.py`.
   - **Gating**: Marked with `@pytest.mark.real_data`. Skipped by default in standard CI.
   - **Mandatory Enforcement**: When invoked with `--require-real-data`, missing dataset files cause an immediate **test failure** (`pytest.fail`), never a silent skip or synthetic fallback.
   - **Execution Commands**:
     ```bash
     # Run all real-data integration tests (fails closed if files are missing):
     cd backend && pytest tests/integration/datasets/ -v --require-real-data

     # Include real data tests if present (skips if absent):
     cd backend && pytest tests/integration/datasets/ -v --include-real-data
     ```

### 8.2 Provenance Taxonomy

The repository strictly differentiates dataset origins and integrity levels:

| Provenance Level | Description | Benchmark Datasets |
|:---|:---|:---|
| `EMPIRICAL_EXTERNAL_DATA` | Real observations from authentic commercial payment logs or public blockchain DAGs. | `ieee_cis`, `creditcard`, `elliptic` |
| `PUBLIC_SIMULATED_DATASET` | Public agent-based simulations published with physical source files and published baseline distributions. | `paysim`, `amlsim` |
| `CONTROLLED_PROJECT_SYNTHETIC` | Reproducible benchmark datasets generated via domain-specific statistical models or academic methodology. | `synthaml`, `amlnet`, `cross_bank` |
| `TEST_FIXTURE` | Minimal in-process deterministic schema fixtures for unit testing and CI smoke gates (`is_synthetic=True`). | Generated in `dataloader_smoke_fixtures.py` |

### 8.3 Provisioning Matrix & CI Reality

| Dataset ID | Authoritative Source | Required Files & Format | Storage Location | Acquisition Method | Auth & License Requirements | Standard CI | Real-Data CI Gate | Local Execution |
|:---|:---|:---|:---|:---|:---|:---:|:---:|:---|
| **`paysim`** | Kaggle `ealaxi/paysim1` | `PS_20174392719_1491204439457_log.csv` (CSV) | `backend/storage/datasets/paysim/` | Kaggle API / CLI | Kaggle API Key; CC BY-SA 4.0 | Schema Fixture | Provisions via Kaggle API | Real file on disk |
| **`ieee_cis`** | Kaggle `c/ieee-fraud-detection` | `train_transaction.csv`, `train_identity.csv` (CSV) | `backend/storage/datasets/ieee_cis/` | Kaggle Competition CLI | Kaggle API Key + Competition Rules Acceptance | Schema Fixture | Requires Kaggle credentials & rules acceptance | Real file on disk |
| **`creditcard`** | Kaggle `mlg-ulb/creditcardfraud` | `creditcard.csv` (CSV) | `backend/storage/datasets/creditcard/` | Kaggle API / CLI | Kaggle API Key; ODbL 1.0 | Schema Fixture | Provisions via Kaggle API | Real file on disk |
| **`elliptic`** | Kaggle `ellipticco/elliptic-data-set` | `elliptic_txs_features.csv`, `elliptic_txs_classes.csv`, `elliptic_txs_edgelist.csv` (CSV) | `backend/storage/datasets/elliptic/` | Kaggle API / CLI | Kaggle API Key; CC BY 4.0 | Schema Fixture | Provisions via Kaggle API | Real file on disk |
| **`amlsim`** | Kaggle `anshankul/ibm-amlsim-example-dataset` | `transactions.csv`, `accounts.csv`, `alerts.csv` (CSV) | `backend/storage/datasets/amlsim/` | Kaggle API / CLI | Kaggle API Key; Apache 2.0 | Schema Fixture | Provisions via Kaggle API | Real file on disk |
| **`synthaml`** | Figshare / Nature Sci Data DOI: `10.1038/s41597-023-02569-2` | `alerts.csv`, `transactions.csv` (CSV/Parquet) | `backend/storage/datasets/synthaml/` | Direct Download / `generate_synthaml_dataset.py` | Open Access; CC BY 4.0 | Schema Fixture | Local cache / Generator script | Real file on disk |
| **`amlnet`** | Zenodo DOI: `10.5281/zenodo.10058474` | `transactions.csv` (CSV/Parquet) | `backend/storage/datasets/amlnet/` | Direct Download / `generate_amlnet_dataset.py` | Open Access; CC BY-NC 4.0 | Schema Fixture | Local cache / Generator script | Real file on disk |

### 8.4 Automated Provisioning Workflow

The standalone GitHub Actions workflow `.github/workflows/real_data_integration.yml` provides a secure, gated pipeline for real-data integration:
- Triggered manually via `workflow_dispatch` or on a weekly schedule.
- Uses repository secrets `KAGGLE_USERNAME` and `KAGGLE_KEY`.
- Executes `python scripts/download_real_benchmarks.py --dataset all`.
- Fails closed if physical datasets cannot be provisioned when mandatory execution is requested.