megathon_travellers - green-product propensity models (CatBoost)

Eight CatBoost binary classifiers (catboost_GP01.cbm ... catboost_GP08.cbm), one per green product, predicting the probability that an eligible customer takes the product up. Built for Megathon 26 on a synthetic customer dataset (not real people). Scores are Platt-calibrated (calibration.json).

Files

• catboost_GP0x.cbm - model per product
• manifest.json - features, categorical features and calibration per product
• calibration.json - Platt parameters: p = sigmoid(a * logit(raw) + b)
• train_catboost.py, predict_catboost.py - training and scoring code (need the project's data/ feature files, not included)
• cat_boostreport.md - full metrics report

Use

import json, numpy as np
from catboost import CatBoostClassifier
man = json.load(open("manifest.json"))["GP03"]
model = CatBoostClassifier().load_model("catboost_GP03.cbm")
# X: DataFrame with man["features"]; categorical columns (man["cat_features"]) as strings, NaN -> "NA"
raw = model.predict_proba(X)[:, 1]
c = man["calibration"]; r = np.clip(raw, 1e-6, 1-1e-6)
propensity = 1 / (1 + np.exp(-(c["a"] * np.log(r / (1 - r)) + c["b"])))

Results (held-out test customers)

Per-product CatBoost classifiers (GP01-GP08), reloaded from models/*.cbm by predict_catboost.py (no retraining) and scored on the test split: customers never used in training or hyper-parameter tuning. Training: train_catboost.py (Optuna, 20 trials x 3-fold CV maximising PR-AUC; Platt calibration on out-of-fold scores).

Ranking quality (test split)

product_id	n	positives	base_rate	roc_auc	oracle_auc	pr_auc	lift_top10	capture_top10
GP01	2057	63	0.031	0.724 [0.670, 0.780]	0.822	0.086 [0.054, 0.147]	2.853	0.286
GP02	1731	38	0.022	0.685 [0.589, 0.772]	0.788	0.116 [0.036, 0.219]	2.633	0.263
GP03	1398	34	0.024	0.812 [0.740, 0.880]	0.900	0.213 [0.099, 0.377]	5.874	0.588
GP04	1020	39	0.038	0.672 [0.571, 0.771]	0.705	0.100 [0.058, 0.190]	3.077	0.308
GP05	1758	32	0.018	0.765 [0.680, 0.845]	0.826	0.095 [0.045, 0.216]	3.746	0.375
GP06	1578	24	0.015	0.667 [0.535, 0.797]	0.782	0.076 [0.018, 0.186]	2.497	0.250
GP07	2895	41	0.014	0.684 [0.609, 0.755]	0.833	0.030 [0.018, 0.077]	2.678	0.268
GP08	1736	19	0.011	0.756 [0.655, 0.847]	0.816	0.039 [0.017, 0.086]	3.676	0.368

ROC-AUC / PR-AUC in brackets: 95% bootstrap CI. oracle_auc is the ceiling from the hidden true probabilities. lift_top10 / capture_top10: lift and recall within the top 10% of scores.

F1, precision, recall (test split)

product_id	rule	thr	flagged	precision	recall	f1
GP01	OOF-tuned best-F1 threshold	0.097	106	0.094	0.159	0.118
GP01	top 10% flagged	0.067	206	0.087	0.286	0.134
GP02	OOF-tuned best-F1 threshold	0.118	39	0.128	0.132	0.130
GP02	top 10% flagged	0.044	173	0.058	0.263	0.095
GP03	OOF-tuned best-F1 threshold	0.158	40	0.250	0.294	0.270
GP03	top 10% flagged	0.042	140	0.143	0.588	0.230
GP04	OOF-tuned best-F1 threshold	0.069	77	0.130	0.256	0.172
GP04	top 10% flagged	0.064	102	0.118	0.308	0.170
GP05	OOF-tuned best-F1 threshold	0.049	105	0.076	0.250	0.117
GP05	top 10% flagged	0.038	176	0.068	0.375	0.115
GP06	OOF-tuned best-F1 threshold	0.039	47	0.064	0.125	0.085
GP06	top 10% flagged	0.025	158	0.038	0.250	0.066
GP07	OOF-tuned best-F1 threshold	0.077	37	0.027	0.024	0.026
GP07	top 10% flagged	0.029	290	0.038	0.268	0.066
GP08	OOF-tuned best-F1 threshold	0.051	15	0.000	0.000	0.000
GP08	top 10% flagged	0.026	174	0.040	0.368	0.073

OOF-tuned best-F1 threshold: the calibrated-propensity cut-off that maximises F1 on dev out-of-fold scores, applied unchanged to the test split. top 10% flagged: flag the 10% highest-scoring customers. Take-up is rare, so F1 is inherently low; treat it alongside the ranking metrics.

Take-up is rare (1-4%) and some test sets have under 40 positives, so confidence intervals are wide and F1 is low; the ranking metrics (ROC-AUC, lift) are the better summary. No license has been set yet.