Model card: five-fold results, out-of-time run, released folds
Browse files
README.md
CHANGED
|
@@ -39,14 +39,19 @@ amount-bin edges the model was trained on; histories must be tokenized with it).
|
|
| 39 |
|---|---|---|---|
|
| 40 |
| `aligned-s42`, `aligned-s43` | transactions and geolocation | aligned, query-key normalization | 0.786, 0.788 |
|
| 41 |
| `aligned-profile-s42`, `aligned-profile-s43` | as above, with early fusion of the profile fields | aligned, query-key normalization | 0.803, 0.807 with late fusion of the profile fields |
|
| 42 |
-
| `trx-s42`, `trx-s43` | transactions only | random offsets | 0.
|
| 43 |
| `trx-oot-s42`, `trx-oot-s43` | transactions only, pretrained on events before February 2022 | random offsets | 0.790, 0.788 |
|
| 44 |
|
| 45 |
-
The official mean is the mean ROC-AUC over MBD's four product targets on
|
| 46 |
head on the frozen embedding. On the same rows, CatBoost on engineered features reaches 0.816 and on
|
| 47 |
token counts over the full history 0.803. A rank-averaged ensemble of EventGPT (with late fusion of the
|
| 48 |
profile fields and token counts) and the engineered features reaches 0.830.
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
Without labels, the next-event distribution scores five "first transaction of a kind next month" targets
|
| 51 |
at ROC-AUC 0.78 to 0.91 (aligned variants, mean of two seeds).
|
| 52 |
|
|
@@ -66,7 +71,9 @@ Install the code from the GitHub repository. Tokenizing MBD-mini (`egpt prepare`
|
|
| 66 |
The vocabulary is MBD's event schema. Another institution's events need their own tokenization and their
|
| 67 |
own pretraining; these weights serve to reproduce and extend the paper's results on MBD. The models were
|
| 68 |
pretrained on the pretraining clients' events from 2021 and 2022, which include the months in which the
|
| 69 |
-
test clients' labels fall (never the test clients themselves)
|
|
|
|
|
|
|
| 70 |
|
| 71 |
## License and citation
|
| 72 |
|
|
|
|
| 39 |
|---|---|---|---|
|
| 40 |
| `aligned-s42`, `aligned-s43` | transactions and geolocation | aligned, query-key normalization | 0.786, 0.788 |
|
| 41 |
| `aligned-profile-s42`, `aligned-profile-s43` | as above, with early fusion of the profile fields | aligned, query-key normalization | 0.803, 0.807 with late fusion of the profile fields |
|
| 42 |
+
| `trx-s42`, `trx-s43` | transactions only | random offsets | 0.794, 0.791 |
|
| 43 |
| `trx-oot-s42`, `trx-oot-s43` | transactions only, pretrained on events before February 2022 | random offsets | 0.790, 0.788 |
|
| 44 |
|
| 45 |
+
The official mean is the mean ROC-AUC over MBD's four product targets on test fold 0, with a logistic
|
| 46 |
head on the frozen embedding. On the same rows, CatBoost on engineered features reaches 0.816 and on
|
| 47 |
token counts over the full history 0.803. A rank-averaged ensemble of EventGPT (with late fusion of the
|
| 48 |
profile fields and token counts) and the engineered features reaches 0.830.
|
| 49 |
|
| 50 |
+
The paper repeats this comparison with each of MBD's five folds as the test fold, pretraining new models
|
| 51 |
+
on the other four. Averaged over the five folds, the embedding scores 0.787 against 0.808 for the
|
| 52 |
+
engineered features, 0.804 with the profile fields and token counts, and the ensemble 0.824. The ensemble
|
| 53 |
+
is above the engineered features on every fold and seed.
|
| 54 |
+
|
| 55 |
Without labels, the next-event distribution scores five "first transaction of a kind next month" targets
|
| 56 |
at ROC-AUC 0.78 to 0.91 (aligned variants, mean of two seeds).
|
| 57 |
|
|
|
|
| 71 |
The vocabulary is MBD's event schema. Another institution's events need their own tokenization and their
|
| 72 |
own pretraining; these weights serve to reproduce and extend the paper's results on MBD. The models were
|
| 73 |
pretrained on the pretraining clients' events from 2021 and 2022, which include the months in which the
|
| 74 |
+
test clients' labels fall (never the test clients themselves); pretraining only on transactions before
|
| 75 |
+
February 2022 (`trx-oot`) lowers the official mean by 0.004. The weights here are pretrained with fold 0
|
| 76 |
+
held out; the paper's models for the other four test folds are not released.
|
| 77 |
|
| 78 |
## License and citation
|
| 79 |
|