Deliverius refund fraud model

A calibrated probability that a refund, coupon or cancellation claim on the fictional Deliverius delivery app is fraudulent, plus the rule that sends a claim to a human reviewer when the expected loss reaches the cost of a review. It is a logistic regression over 27 point-in-time features, calibrated with Platt scaling.

Synthetic data only. Every customer, order and claim was generated. The model learned the generator, not real fraud. Do not use it to decide anything about real people.

Intended use

  • Demonstrating a reproducible tabular ML pipeline: time split, baselines, calibration, cost-based decision, pinned artifacts and a CI quality gate.
  • Scoring claims inside the Deliverius demo, where a person reviews every flagged case.

Not intended: decisions about real customers, credit or identity checks, or any use where a score alone denies someone a refund.

Data

  • Dataset: G0m7s/deliverius-ai version 1.0.0, commit cc5be0958e3e4971c284031ceb6406c43e96ae84.
  • Population: claims the policy engine did not deny (denied claims are never scored).
  • Split by time: training May to August (hyperparameters by expanding monthly folds), validation September (model choice, calibration, threshold), test October (once).
  • Test set: 462 claims, 51 frauds.

Features

Only facts strictly before the claim instant; no name, address, document, message text, customer profile or reviewer outcome. Missing values get the training median.

account_age_days, orders_90d, claims_30d, claims_90d, refund_requested_30d_cents, claims_per_order_90d, reviews_90d, order_total_cents, requested_cents, requested_fraction, minutes_since_delivery, marked_delivered, local_hour, action_refund, action_coupon, action_cancellation, reason_item_missing, reason_wrong_item, reason_quality_issue, reason_late_delivery, reason_not_delivered, reason_duplicate_charge, reason_customer_cancellation, payment_pix, payment_credit_card, payment_debit_card, payment_meal_voucher

Feature schema version 1.0.0, hash 2616a1e1f99103eef888ab3857584ad465f54863e4904435882812615c4709a2.

How the model was chosen

The rule was written before the first run and uses validation only: LightGBM is chosen when the 95% paired-bootstrap interval of PR-AUC(LightGBM) - PR-AUC(logistic) is entirely above zero; otherwise the simpler model stays. On September the interval was [-0.182, -0.007], so the logistic regression stayed.

Test metrics (October)

Scorer PR-AUC [95% CI] ROC-AUC Brier Recall at top 5%
Logistic (calibrated) 0.452 [0.348, 0.596] 0.839 0.0770 0.333
LightGBM (calibrated) 0.372 [0.272, 0.507] 0.802 0.0841 0.235
Business rule 0.123 [0.110, 0.167] 0.526 - 0.078

Expected cost per claim, with a review costing R$ 4.00:

Rule Reviewed Frauds caught Cost per claim
Never review 0 0 R$ 3.22
Review everything 462 51 R$ 4.00
Business rule (3+ claims in 30 days) 15 4 R$ 3.14
Fixed threshold (TunedThresholdClassifierCV) 57 27 R$ 1.65
Cost rule: p x amount >= review cost 93 31 R$ 1.25

Files

Plain JSON with numbers only, no pickle: loading the model runs no code.

  • logistic.json: medians, means, scales, coefficients and intercept, in feature order.
  • calibrator.json: p = 1 / (1 + exp(a * raw + b)) over the raw score.
  • decision.json: the cost of one review, in cents.
  • manifest.json: versions, lineage, metrics and the SHA-256 of every file.

Limitations

  • Synthetic data: the signal is the generator's, and real fraud is more varied.
  • Few positives (51 frauds in the test set): the intervals are wide.
  • Orders start on May 1st, so the 90-day windows are incomplete until the end of July; validation and test have complete windows.
  • The calibration is sigmoid: about 60 positives per month are too few for isotonic.

Lineage

  • Code commit 3a20f85f486574fdaa7ede97af759d5f8daa99ca, MLflow run 122a02d18bf241208d0db8c4f17bc00f, seed 20261109.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Dataset used to train G0m7s/deliverius-fraud