getSTEAV/system-one-fdb-sparknov

System One fraud detector for FDB sparknov: Simulated Credit Card Transactions (Sparkov)

This model detects fraudulent credit card transactions. It is a two-stage stack: a gradient-boosted tree model scores each transaction from engineered features, and System One, a typed-decision model (Laya ModernBERT-large encoder with a LoRA adapter and a decision head), reads that score, a few readable engineered fields and the original fields as JSON evidence, and returns a calibrated probability that the transaction is fraudulent.

On FDB's official test split (20,000 transactions, 93 fraudulent) it scores 0.9994 AUROC and catches 97.9% of fraud at a 1% false-positive rate, against the best published 0.998 AUROC (AFD OFI) and 100.0% recall (AFD OFI). It trails the published recall, but the published value lies inside this model's 95% confidence interval.

It was trained for the Fraud Dataset Benchmark (FDB) and is a benchmark model, not a production fraud system; read Limitations first.

Results

Official FDB test split, scored once per model. AUROC and recall at 1% FPR follow FDB's own evaluation (recall is np.interp(0.01, fpr, tpr) on the test ROC). 95% confidence intervals: class-stratified bootstrap, 1,000 resamples.

Model	AUROC [95% CI]	Recall at 1% FPR [95% CI]	Average precision	Recall at 0.1% FPR
This model: System One	0.9994 [0.9989, 0.9999]	97.9% [94.6, 100.0]	0.9494	90.3%
Tree alone (System One's input)	0.9994 [0.9988, 0.9999]	97.9% [94.6, 100.0]	0.9498	91.4%
Blend fitted on System One's holdout split (not released)	0.9994 [0.9988, 0.9999]	97.9% [94.6, 100.0]	0.9496	91.4%
Best published AUROC: AFD OFI	0.998
Best published recall at 1% FPR: AFD OFI		100.0%

AUROC leaders are from the FDB paper (arXiv 2208.14417 v3). The paper reports no recall, so recall leaders are from the results table in FDB's README at commit 54cdefa211; later versions of the README keep that table inside an HTML comment, so it no longer renders. With 93 fraudulent transactions in the test set, each one is worth 1.1 points of recall.

How the released configuration was chosen. All models were trained and calibrated without test data, and each test set was scored once per candidate. After the test results were known, one model per benchmark was released: System One on every set (for a uniform, calibrated decision interface), and on this set it ties the tree and the blend on both headline metrics. Choosing among candidates after seeing test results can flatter the headline slightly, so the table shows every candidate.

Model details

• Developed by: STEAV (steav.io), trained with CID Model Studio.
• Model type: stacked binary classifier: gradient-boosted trees, then System One with offset stacking (System One adds a learned correction to the tree's log-odds).
• System One base: convaiinnovations/laya at revision 7b928d828b7b (Apache-2.0), a ModernBERT-large encoder (about 395M parameters) with Laya's typed decision head. The base weights are downloaded from the Hub at that pinned revision and checked against their sha256; they are not redistributed here.
• Trained parameters: a LoRA adapter (rank 16, alpha 32, on the attention Wqkv/Wo and MLP Wo projections of all 28 layers; 4.39M parameters) and the decision head (26.2M parameters, initialised from Laya's head; its 0.26M-parameter action sub-head stays frozen).
• Question: "Is this card transaction fraudulent, meaning it was not made by the legitimate cardholder?" (a yes/no decision; the output is P(yes)).
• License: Apache-2.0 for the weights and code; some data files carry their own terms (see License and attribution).
• Version: 1.0.0.
• Contact: questions and issues go to this repository's Community tab.

Uses

Intended: benchmark research and comparison on FDB; a reference implementation of stacking a typed-decision model on a tree model's score; a starting point for fine-tuning on your own data.

Out of scope: scoring real card transactions; any automated decision about a person without human review; use on data from a different distribution without retraining and validation.

How to use

Python 3.12 with the pinned versions in requirements.txt. On the reference GPU (NVIDIA GB10 in a DGX Spark, torch 2.11.0, bfloat16) the code reproduced the evaluated scores bit for bit; other GPUs and CPUs give slightly different scores (see Reproducibility). On CPU (Apple M4 Max, macOS arm64, float32) System One takes about 0.14 s per transaction, about 0.8 h for the 20,000-row test set.

hf download getSTEAV/system-one-fdb-sparknov --local-dir system-one-fdb-sparknov
cd system-one-fdb-sparknov && pip install -r requirements.txt

# run from the repository folder
from system_one_fraud import FraudPipeline
from system_one_fraud.fdb import load_split

pipe = FraudPipeline.from_pretrained(".")      # the tree, the adapter and the pinned Laya base
train, test, labels = load_split("sparknov")       # FDB's loader (needs Kaggle API credentials)
p_fraud = pipe.predict_proba(test)              # one probability per row

predict_proba returns, per row, the probability that the transaction is fraudulent, shifted from System One's rebalanced training sample to the natural fraud rate of FDB's training data (0.579%); the shift does not change the ranking. output="tree" returns the tree model's probability alone. states, tree_out = pipe.states(rows) returns the exact JSON evidence System One reads and the tree model's outputs. examples/quickstart.py runs the same steps on a few test rows.

The adapter does not load with PeftModel.from_pretrained alone: Laya's root config.json is a stub that transformers cannot instantiate, and the decision head and evidence format live in the bundled system_one_fraud package. On macOS the tree model runs in a helper Python process, because PyTorch and the gradient-boosting libraries load two OpenMP runtimes that crash in one process; results are identical. System One's bfloat16 GPU scores depend slightly on how rows are batched (padding); the reference scores used batch size 128 in file order.

Input contract. The tree reads cc_num (the card), EVENT_TIMESTAMP (UTC), amt, category (one of the 14 Sparkov categories), gender, city_pop and dob; System One also reads job and state. Every other column is ignored, and a missing field raises an error. Card-profile features use the same card's observed transactions older than 7 days: by default the bundled training-period history (all 1,296,675 FDB training transactions) plus the earlier rows of the same call, which is how FDB's test was scored, so pass the whole test set in one call to reproduce the reported scores. For new data, pass history= (columns cc_num, EVENT_TIMESTAMP, amt, category) to predict_proba. FDB's loader moves Sparkov's timestamps forward by 20 calendar months (dateutil.relativedelta(months=20)), which does not preserve weekdays; apply the same shift to raw Kaggle-format rows.

Training data

• Source: Synthetic card transactions for 1,000 customers and 800 merchants generated with the Sparkov data generator and published on Kaggle by Kartik Shenoy (kartik2112/fraud-detection, version 1). FDB uses the whole source training file (1,296,675 transactions) for training and a random 20,000-row sample of the later source test file for testing, and moves every timestamp forward by 20 calendar months.
• Benchmark split: out of time: the test window starts five minutes after the last training transaction and spans 191.5 days; the 20,000 test rows are a 3.6% random sample of that window. Training: 1,296,675 transactions (7,506 fraudulent, 0.58%). Test: 20,000 transactions (93 fraudulent).
• Licence: CC0 1.0 (public domain), set by the person who generated the data; the generator (Sparkov Data Generation, Brandon Harris) is MIT-licensed. Fully synthetic: no real cardholders.

Training procedure

Stage 1, tree model. An equal-weight blend of three LightGBM and three CatBoost models (seeds 0-2), each trained on every training fraud plus a 20% sample of legitimate transactions and corrected back to the natural fraud rate. The 12 features are the amount, merchant category, hour and weekday, customer age, gender, city size, and card profiles computed only from the card's transactions older than 7 days (amount z-scores against the card's history overall and within the category, category, hour and night-time shares), shrunk towards population averages when the history is thin. Selection used five contiguous time blocks of the training data (blocked out-of-fold scores) and a test-like simulation: the last 191.5 days of training scored with only the history a sparse FDB test row has. Recent-activity (velocity) features were deliberately left out: FDB's test is a 3.6% sample, so a test row's recent transactions are mostly missing, and a velocity model falls from 0.99985 to 0.9958 AUROC on test-like data.

Training rows carry out-of-fold tree scores: the models that scored a training row never saw it, so System One never learns from a score fit on its own row.

Stage 2, System One. Each training row is rendered as a JSON state: the tree's out-of-fold score (Tree_fraud_score), its percentile among all out-of-fold training scores (Tree_percentile), readable engineered fields (Context_*) and the original fields. System One trained on a 125,000-row sample: every fraud from time blocks 2-5 (5,533) plus random legitimate transactions, a 4.43% fraud rate; outputs are shifted back to the natural 0.579% rate. The rows were split 100,000 / 12,500 / 12,500 (train / validation / holdout). On the official test set every state fits in System One's 256-token window.

• LoRA r = 16, alpha = 32, dropout 0.05; learning rate 1e-4 (head 5e-5), weight decay 0.01, 5% warm-up, batch 32; bfloat16; maximum sequence length 256 tokens; seed 42.
• one supervised epoch with soft cross-entropy, then one RLCD epoch (reinforcement learning on the decision distribution with proper-scoring-rule rewards, following Laya's method).
• 6,250 optimiser steps, 34.0M training tokens, 2.2 h on one NVIDIA GB10 (DGX Spark).
• Calibration: one temperature fitted on the validation split (L-BFGS).
• Validation: accuracy 0.9954, expected calibration error 0.0017.

Evaluation

• Protocol. FDB's official split; the test set was never used for training, feature selection or calibration, and labels were read only to compute the metrics.
• Calibration on test (10 equal-width bins, released output): expected calibration error 0.0020, Brier score 0.00164.
• Cost of high detection (released output): catching 95% of fraud requires a false-positive rate of 0.15%, and catching 99% requires 2.05%.
• Reproduce: python eval/reproduce_fdb.py rebuilds the split with FDB's own loader, checks it against the frames used here, scores it with the released pipeline and compares the metrics with eval/test_metrics.json.

Limitations

• Synthetic data. Sparkov's fraud is a simple generative pattern (bursts of unusual-category, unusual-amount, mostly night-time transactions). Results do not transfer to real card fraud.
• A known simulator artefact is not used. Every card has at most one fraud burst in this generator; a feature built from that ("this card already had its burst") would raise recall but cannot transfer to real data, so it was left out on purpose.
• Card history required. The profile features need the card's history older than 7 days. The package bundles the training-period history so the FDB test reproduces; for new transactions you must supply each card's history (see How to use).
• Small positive count. The test has 93 frauds: one fraud is about 1.1 points of recall, so the gap to the published 100% recall (2 frauds) is within noise.

Bias, risks and ethical considerations

Fraud models make errors in both directions: false positives block or delay legitimate customers, and their burden can fall unevenly across groups. No fairness evaluation was done for this model; the benchmark data carries no protected attributes we could audit. Use the score as one input to a reviewed decision, monitor error rates on your own population, and retrain when the data drifts.

Reproducibility

These checks passed before release:

• Tree model. Refit from scratch, the tree model in tree/ reproduces the evaluated tree scores bit for bit on all 20,000 test rows (Apple M4 Max, macOS arm64, the pinned versions). On other platforms (other BLAS or OpenMP builds) the last bits can differ, which can occasionally change the Tree_fraud_score text System One reads.
• Evidence. The state builder reproduces the evaluated System One inputs exactly on all 20,000 test rows.
• System One. The inference code in system_one_fraud/ reproduces the evaluated scores bit for bit on all 20,000 test rows on the reference GPU (NVIDIA GB10 in a DGX Spark, torch 2.11.0, bfloat16, batch size 128 in file order, base downloaded from the Hub), run in the training environment with the same pinned versions rather than a fresh install.
• Fresh install. A new virtual environment built from requirements.txt (Apple M4 Max, macOS arm64, pandas 3.0.6, torch 2.11.0) reproduces the tree scores bit for bit and the System One inputs exactly.
• Fresh download. FDB's loader, run fresh with the pinned pandas, rebuilds the evaluated frames exactly (content hashes in eval/fdb_split_hashes.json) apart from the identifier and timestamp columns FDB randomises on every load, which the models never read.
• End to end on CPU. eval/reproduce_fdb.py (fresh download, released pipeline, Apple M4 Max, macOS arm64, float32) gives 0.9994 AUROC and 97.9% recall at 1% FPR, against the reported 0.9994 and 97.9% (identical to 4 decimals).

Measured, not gated: CPU drift. On CPU in float32, System One's scores differ slightly from the bfloat16 GPU reference. On 512 holdout transactions (256 fraudulent) the mean absolute difference in probability is 2.3e-05 and the largest 3.5e-03; decisions at 0.5 agree on 100.0% of rows, and AUROC on that sample is 0.9979 on the GPU and 0.9979 on CPU. The largest difference in any of the 5 CPU comparisons was 1.4e-02, on 512 holdout transactions, in an earlier check.

Environmental impact

System One training used one NVIDIA GB10 (DGX Spark, on premises) for 2.2 h, roughly 0.5 kWh at the system's 240 W rating (an upper-bound estimate). The tree models trained in minutes on a laptop CPU.

Files

Path	Contents
adapter_model.safetensors, adapter_config.json	LoRA adapter
system_one/head.safetensors	decision head
system_one/config.json, system_one/calibration.json, release.json	head configuration, temperature, release manifest (question, pinned base, priors)
system_one/percentile_reference.npy	sorted out-of-fold training tree scores, for the Tree_percentile evidence field
tree/	three LightGBM models (model-text format) and three CatBoost models (.cbm), the bundled training-period card history (parquet), population priors and configuration (JSON), the sorted out-of-fold score reference (.npy) and manifest.json with every file's sha256 (checked on load)
system_one_fraud/	inference code: pipeline.py (end to end), state.py (JSON evidence), system_one.py (System One), tree_sparknov.py (tree model), tree_runner.py (macOS helper process), fdb.py (FDB loader)
eval/	reproduce_fdb.py, the split hashes and the test metrics
examples/quickstart.py	load, score and print a few test rows
LICENSE, NOTICE	licence texts and attributions
requirements.txt	pinned dependencies
checksums.sha256	sha256 of every file except this README

Embedded data. tree/card_history.parquet holds raw rows of the synthetic training data: all 1,296,675 training transactions as card number, time, amount, category and hour, without labels or personal fields; the profile features need them. The data is fully synthetic and CC0, so this is allowed. The other files hold model parameters, label-free population priors and sorted out-of-fold scores.

Citation

@misc{steav2026systemonefdb,
  title        = {System One fraud detectors for the Fraud Dataset Benchmark},
  author       = {{STEAV}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/getSTEAV/system-one-fdb-sparknov}}
}

@misc{grover2023fraud,
  title         = {Fraud Dataset Benchmark and Applications},
  author        = {Prince Grover and Julia Xu and Justin Tittelfitz and Anqi Cheng and Zheng Li and Jakub Zablocki and Jianbo Liu and Hao Zhou},
  year          = {2023},
  eprint        = {2208.14417},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}

@misc{convai2026laya,
  title        = {Laya: a non-autoregressive System 1 decision model},
  author       = {{Convai Innovations}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/convaiinnovations/laya}},
  note         = {Revision 7b928d828b7b0e022f929d9bd2e44165aa270148}
}

@misc{warner2024modernbert,
  title         = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author        = {Benjamin Warner and Antoine Chaffin and Benjamin Clavi{\'e} and Orion Weller and Oskar Hallstr{\"o}m and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
  year          = {2024},
  eprint        = {2412.13663},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

License and attribution

Files	Terms
tree/card_history.parquet	synthetic training transactions from the Sparkov dataset by Kartik Shenoy, CC0 1.0
everything else (weights, code, configuration, model files)	Apache-2.0 (LICENSE)

Attributions are collected in NOTICE.

• Data: Trained on the Credit Card Transactions Fraud Detection dataset by Kartik Shenoy (kaggle.com/datasets/kartik2112/fraud-detection, CC0 1.0), generated with Sparkov Data Generation (Brandon Harris, MIT). tree/card_history.parquet holds synthetic training transactions from that dataset (CC0 1.0).
• Base model: Laya by Convai Innovations (Apache-2.0), built on ModernBERT-large by Answer.AI and LightOn (Apache-2.0). The sequence layout and the decision-head architecture follow Laya's reference code.
• Benchmark: Fraud Dataset Benchmark, Grover et al. 2023 (MIT; Copyright (c) 2021-2022 Prince Grover and Zheng Li, and (c) 2022 Jianbo Liu, Jakub Zablocki, Hao Zhou, Julia Xu and Anqi Cheng).