Deliverius refund fraud model
A calibrated probability that a refund, coupon or cancellation claim on the fictional Deliverius delivery app is fraudulent, plus the rule that sends a claim to a human reviewer when the expected loss reaches the cost of a review. It is a logistic regression over 27 point-in-time features, calibrated with Platt scaling.
Synthetic data only. Every customer, order and claim was generated. The model learned the generator, not real fraud. Do not use it to decide anything about real people.
Intended use
- Demonstrating a reproducible tabular ML pipeline: time split, baselines, calibration, cost-based decision, pinned artifacts and a CI quality gate.
- Scoring claims inside the Deliverius demo, where a person reviews every flagged case.
Not intended: decisions about real customers, credit or identity checks, or any use where a score alone denies someone a refund.
Data
- Dataset:
G0m7s/deliverius-aiversion 1.0.0, commitcc5be0958e3e4971c284031ceb6406c43e96ae84. - Population: claims the policy engine did not deny (denied claims are never scored).
- Split by time: training May to August (hyperparameters by expanding monthly folds), validation September (model choice, calibration, threshold), test October (once).
- Test set: 462 claims, 51 frauds.
Features
Only facts strictly before the claim instant; no name, address, document, message text, customer profile or reviewer outcome. Missing values get the training median.
account_age_days, orders_90d, claims_30d, claims_90d, refund_requested_30d_cents, claims_per_order_90d, reviews_90d, order_total_cents, requested_cents, requested_fraction, minutes_since_delivery, marked_delivered, local_hour, action_refund, action_coupon, action_cancellation, reason_item_missing, reason_wrong_item, reason_quality_issue, reason_late_delivery, reason_not_delivered, reason_duplicate_charge, reason_customer_cancellation, payment_pix, payment_credit_card, payment_debit_card, payment_meal_voucher
Feature schema version 1.0.0, hash 2616a1e1f99103eef888ab3857584ad465f54863e4904435882812615c4709a2.
How the model was chosen
The rule was written before the first run and uses validation only: LightGBM is chosen when the 95% paired-bootstrap interval of PR-AUC(LightGBM) - PR-AUC(logistic) is entirely above zero; otherwise the simpler model stays. On September the interval was [-0.182, -0.007], so the logistic regression stayed.
Test metrics (October)
| Scorer | PR-AUC [95% CI] | ROC-AUC | Brier | Recall at top 5% |
|---|---|---|---|---|
| Logistic (calibrated) | 0.452 [0.348, 0.596] | 0.839 | 0.0770 | 0.333 |
| LightGBM (calibrated) | 0.372 [0.272, 0.507] | 0.802 | 0.0841 | 0.235 |
| Business rule | 0.123 [0.110, 0.167] | 0.526 | - | 0.078 |
Expected cost per claim, with a review costing R$ 4.00:
| Rule | Reviewed | Frauds caught | Cost per claim |
|---|---|---|---|
| Never review | 0 | 0 | R$ 3.22 |
| Review everything | 462 | 51 | R$ 4.00 |
| Business rule (3+ claims in 30 days) | 15 | 4 | R$ 3.14 |
| Fixed threshold (TunedThresholdClassifierCV) | 57 | 27 | R$ 1.65 |
| Cost rule: p x amount >= review cost | 93 | 31 | R$ 1.25 |
Files
Plain JSON with numbers only, no pickle: loading the model runs no code.
logistic.json: medians, means, scales, coefficients and intercept, in feature order.calibrator.json:p = 1 / (1 + exp(a * raw + b))over the raw score.decision.json: the cost of one review, in cents.manifest.json: versions, lineage, metrics and the SHA-256 of every file.
Limitations
- Synthetic data: the signal is the generator's, and real fraud is more varied.
- Few positives (51 frauds in the test set): the intervals are wide.
- Orders start on May 1st, so the 90-day windows are incomplete until the end of July; validation and test have complete windows.
- The calibration is sigmoid: about 60 positives per month are too few for isotonic.
Lineage
- Code commit
3a20f85f486574fdaa7ede97af759d5f8daa99ca, MLflow run122a02d18bf241208d0db8c4f17bc00f, seed 20261109.