Blame for README.md

---
license: cc-by-4.0
datasets:
- ai-lab/MBD-mini
tags:
- event-sequences
- banking
- transactions
- time-series
- foundation-model
- pytorch
library_name: pytorch
---

# EventGPT

EventGPT is an open generative pretrained transformer for banking event sequences. It reads a customer's
history of transactions and other events, predicts the next event token by token, and gives one
embedding per customer and point in time. It was pretrained on the public Multimodal Banking Dataset
(MBD-mini: 100,224 clients, 38.8 million transactions and 66.3 million geolocation pings).

- Paper: *EventGPT: Next-Event Pretraining on Banking Customer Histories* (Burakov, 2026), https://arxiv.org/abs/2611.03302
- Code: https://github.com/deburky/EventGPT

## Model

EventGPT is hierarchical. An event encoder turns the tokens of one event (name, type, amount bin, time gap
and other fields) into a vector; a causal history decoder reads these vectors with rotary positions taken
from elapsed time since the client's first event; a token decoder predicts the next event's tokens from
the history state. The models here have 23 million parameters (width 384) and were trained on 655
million tokens.

## Variants

Each variant is a subfolder with `model.safetensors`, `config.json` and `vocab.json` (the token table and
amount-bin edges the model was trained on; histories must be tokenized with it).

| variant | events | training windows | official mean ROC-AUC |
|---|---|---|---|
| `aligned-s42`, `aligned-s43` | transactions and geolocation | aligned, query-key normalization | 0.786, 0.788 |
| `aligned-profile-s42`, `aligned-profile-s43` | as above, with early fusion of the profile fields | aligned, query-key normalization | 0.803, 0.807 with late fusion of the profile fields |
| `trx-s42`, `trx-s43` | transactions only | random offsets | 0.794, 0.791 |
| `trx-oot-s42`, `trx-oot-s43` | transactions only, pretrained on events before February 2022 | random offsets | 0.790, 0.788 |

The official mean is the mean ROC-AUC over MBD's four product targets on test fold 0, with a logistic
head on the frozen embedding. On the same rows, CatBoost on engineered features reaches 0.816 and on
token counts over the full history 0.803. A rank-averaged ensemble of EventGPT (with late fusion of the
profile fields and token counts) and the engineered features reaches 0.830.

The paper repeats this comparison with each of MBD's five folds as the test fold, pretraining new models
on the other four. Averaged over the five folds, the embedding scores 0.787 against 0.808 for the
engineered features, 0.804 with the profile fields and token counts, and the ensemble 0.824. The ensemble
is above the engineered features on every fold and seed.

Without labels, the next-event distribution scores five "first transaction of a kind next month" targets
at ROC-AUC 0.78 to 0.91 (aligned variants, mean of two seeds).

## Use

```python
from eventgpt.hub import from_pretrained

model, vocab = from_pretrained("deburky/EventGPT", "aligned-profile-s42")
```

Install the code from the GitHub repository. Tokenizing MBD-mini (`egpt prepare`), embedding histories
(`egpt probe`) and zero-shot scoring (`egpt zero-shot`) are commands of the `egpt` CLI.

## Limitations

The vocabulary is MBD's event schema. Another institution's events need their own tokenization and their
own pretraining; these weights serve to reproduce and extend the paper's results on MBD. The models were
pretrained on the pretraining clients' events from 2021 and 2022, which include the months in which the
test clients' labels fall (never the test clients themselves); pretraining only on transactions before
February 2022 (`trx-oot`) lowers the official mean by 0.004 to 0.006. The weights here are pretrained with fold 0
held out; the paper's models for the other four test folds are not released.

## License and citation

The weights are released under CC BY 4.0, the license of MBD-mini, on which they were trained. Please cite
the paper and MBD (Mollaev et al., 2024).

```bibtex
@misc{burakov2026eventgpt,
  title  = {EventGPT: Next-Event Pretraining on Banking Customer Histories},
  author = {Burakov, Denis},
  year   = {2026},
  eprint = {2611.03302},
  archivePrefix = {arXiv},
  primaryClass = {stat.ML},
  url    = {https://arxiv.org/abs/2611.03302}
}
```