deburky commited on
Commit
6e91060
·
verified ·
1 Parent(s): eff5f15

Model card: five-fold results, out-of-time run, released folds

Browse files
Files changed (1) hide show
  1. README.md +10 -3
README.md CHANGED
@@ -39,14 +39,19 @@ amount-bin edges the model was trained on; histories must be tokenized with it).
39
  |---|---|---|---|
40
  | `aligned-s42`, `aligned-s43` | transactions and geolocation | aligned, query-key normalization | 0.786, 0.788 |
41
  | `aligned-profile-s42`, `aligned-profile-s43` | as above, with early fusion of the profile fields | aligned, query-key normalization | 0.803, 0.807 with late fusion of the profile fields |
42
- | `trx-s42`, `trx-s43` | transactions only | random offsets | 0.795, 0.791 |
43
  | `trx-oot-s42`, `trx-oot-s43` | transactions only, pretrained on events before February 2022 | random offsets | 0.790, 0.788 |
44
 
45
- The official mean is the mean ROC-AUC over MBD's four product targets on the test fold, with a logistic
46
  head on the frozen embedding. On the same rows, CatBoost on engineered features reaches 0.816 and on
47
  token counts over the full history 0.803. A rank-averaged ensemble of EventGPT (with late fusion of the
48
  profile fields and token counts) and the engineered features reaches 0.830.
49
 
 
 
 
 
 
50
  Without labels, the next-event distribution scores five "first transaction of a kind next month" targets
51
  at ROC-AUC 0.78 to 0.91 (aligned variants, mean of two seeds).
52
 
@@ -66,7 +71,9 @@ Install the code from the GitHub repository. Tokenizing MBD-mini (`egpt prepare`
66
  The vocabulary is MBD's event schema. Another institution's events need their own tokenization and their
67
  own pretraining; these weights serve to reproduce and extend the paper's results on MBD. The models were
68
  pretrained on the pretraining clients' events from 2021 and 2022, which include the months in which the
69
- test clients' labels fall (never the test clients themselves). All results are on one test fold.
 
 
70
 
71
  ## License and citation
72
 
 
39
  |---|---|---|---|
40
  | `aligned-s42`, `aligned-s43` | transactions and geolocation | aligned, query-key normalization | 0.786, 0.788 |
41
  | `aligned-profile-s42`, `aligned-profile-s43` | as above, with early fusion of the profile fields | aligned, query-key normalization | 0.803, 0.807 with late fusion of the profile fields |
42
+ | `trx-s42`, `trx-s43` | transactions only | random offsets | 0.794, 0.791 |
43
  | `trx-oot-s42`, `trx-oot-s43` | transactions only, pretrained on events before February 2022 | random offsets | 0.790, 0.788 |
44
 
45
+ The official mean is the mean ROC-AUC over MBD's four product targets on test fold 0, with a logistic
46
  head on the frozen embedding. On the same rows, CatBoost on engineered features reaches 0.816 and on
47
  token counts over the full history 0.803. A rank-averaged ensemble of EventGPT (with late fusion of the
48
  profile fields and token counts) and the engineered features reaches 0.830.
49
 
50
+ The paper repeats this comparison with each of MBD's five folds as the test fold, pretraining new models
51
+ on the other four. Averaged over the five folds, the embedding scores 0.787 against 0.808 for the
52
+ engineered features, 0.804 with the profile fields and token counts, and the ensemble 0.824. The ensemble
53
+ is above the engineered features on every fold and seed.
54
+
55
  Without labels, the next-event distribution scores five "first transaction of a kind next month" targets
56
  at ROC-AUC 0.78 to 0.91 (aligned variants, mean of two seeds).
57
 
 
71
  The vocabulary is MBD's event schema. Another institution's events need their own tokenization and their
72
  own pretraining; these weights serve to reproduce and extend the paper's results on MBD. The models were
73
  pretrained on the pretraining clients' events from 2021 and 2022, which include the months in which the
74
+ test clients' labels fall (never the test clients themselves); pretraining only on transactions before
75
+ February 2022 (`trx-oot`) lowers the official mean by 0.004. The weights here are pretrained with fold 0
76
+ held out; the paper's models for the other four test folds are not released.
77
 
78
  ## License and citation
79