File size: 12,200 Bytes

---
license: mit
library_name: pytorch
pipeline_tag: tabular-classification
tags:
- process-mining
- predictive-process-monitoring
- next-activity-prediction
- foundation-model
- transformer
- scaling-laws
- event-log
datasets:
- Polifemus/event-log-mixer
---

# Process-FM — event-log foundation models

Causal transformer foundation models for **predictive process monitoring**: given the prefix of a case (activities, resources, timestamps), predict the next activity and the next resource. All models share the same global vocabulary and the same `inference.py`; they differ in size and training budget.

This repository contains the **fm-v6** foundation checkpoint, the **scaling suite** (sc5 → sc27 → sc80 → sc300) and two **real-only ablations** on an expanded 333-log vocabulary.

## Models

| folder | params | d_model / layers | training corpus | positions seen | val next-activity acc | BPIC 2015-1 zero-shot | BPIC 2015-1 FT test acc | FT top3 | FT macro-F1 |
|---|---:|---|---:|---:|---:|---:|---:|---:|---:|
| `fm-v6/` | 27.4M | 512 / 8 | v3: 18 real + 40 synthetic (58 logs) | ~0.43B | **0.7527** | 0.3480 | **0.4495** | **0.6436** | **0.205** |
| `scaling/sc5/` | 6.32M | 320 / 4 | v4: 18 real + 300 synthetic (308 logs) | 0.04B | 0.6026 | 0.2977 | 0.3931 | 0.5642 | 0.163 |
| `scaling/sc27/` | 27.36M | 512 / 8 | v4 | 0.13B | 0.6451 | 0.3068 | 0.4296 | 0.5865 | 0.1877 |
| `scaling/sc80/` | 88.19M | 768 / 12 | v4 | 0.38B | **0.6563** | **0.3600** | 0.4336 | 0.6280 | 0.1861 |
| `scaling/sc300/` | 306.4M | 1024 / 24 | v5 (1 epoch, reduced budget) | ~1.28B | 0.6080* | 0.3090 | 0.4267 | 0.6064 | 0.1932 |
| — (local, not released) | 306.4M | 1024 / 24 | v4: 18 real + 300 synthetic (308 logs) | ~1.28B | 0.6538† | **0.3929** | 0.4450 | 0.6206 | 0.1873 |
| `scaling/ablation/sc80-v5-rmix14/` | 88.19M | 768 / 12 | v5 rmix14 (18 real ×14 + 2,500 synth) | ~0.41B | 0.6158 | 0.3969 | **0.4695** | **0.6456** | **0.2096** |
| `scaling/ablation/real-only-cap/real18cap/` | 89.1M | 768 / 12 | real-only: 18 logs, cap 4000 (vocab v7) | ~0.38B | 0.8133‡ | 0.3703 | 0.4418 | 0.5960 | 0.1909 |
| `scaling/ablation/real-only-cap/real31cap/` | 89.1M | 768 / 12 | real-only: 31 logs (18 + 13 new), cap 4000 (vocab v7) | ~0.38B | 0.8291‡ | 0.3010 | 0.4457 | 0.6036 | 0.1892 |

Notes:
- `fm-v6` and the `sc*` models use **different corpora and vocabularies** (58 vs 308 seen logs) and are not directly comparable. The scaling curve is `sc5 → sc27 → sc80 → sc300`.
- † sc300-v4 is a controlled local run (best epoch 8); its checkpoint is not released. Validation is on corpus v4 and comparable with the other v4 points.
- The `sc*` points are **under-budget** by design (0.22-0.42× the 20-token/parameter target; sc300 uses 4.2 tokens/parameter), so the curve measures the under-trained regime, not Chinchilla-optimal performance. See `SCALING.md`.
- **sc300 on corpus v4 (local ablation, checkpoint not released)**: a controlled follow-up on the same corpus as sc5/sc27/sc80, 61 epochs (~1.28B positions), best epoch 8. It isolates scale at fixed mix: validation is flat vs sc80 (0.6538 vs 0.6563) while transfer improves (**+3.3pp zero-shot, +1.1pp FT**). The mix remains the stronger lever: sc80 on rmix14 reaches 0.4695 FT, still the best single model. See the data-mix table below.
- `sc300` is a **negative result** on the default v5 mix: trained on corpus v5 (2,500 synthetic shards + 18 real logs), it did not improve transfer over sc80. Its validation accuracy (0.6080) is measured on a different, synthetic-heavy validation set and is **not comparable** with sc5/sc27/sc80 (v4). Zero-shot on BPIC 2015-1 is 0.3090 (0.2869 without unseen-to-UNK) and fine-tuned test accuracy 0.4267, both below sc80. The **data-mix ablation** below shows the bottleneck is the real fraction: sc300 on rmix14 reaches 0.3969 zero-shot / 0.4448 fine-tuned, but still does not beat the 80M model on the same mix (0.4695). See `SCALING.md`.
- `‡` The two `real-only-cap` rows use an expanded vocabulary (333 log ids, 934/3,773 activity/resource rows) and a real-only corpus; their validation is on the respective held-out real-only cases and is **not comparable** with the `fm-v6`/`sc*` rows. See `scaling/ablation/real-only-cap/README.md`.

## Quickstart

```bash
pip install torch numpy
```

```python
import sys
sys.path.insert(0, ".")
from inference import load_model, predict

model, vocab = load_model("fm-v6")          # or "scaling/sc80"
events = [
    {"activity": "A_SUBMITTED",       "resource": "112", "timestamp": "2011-09-30 22:38:44"},
    {"activity": "A_PARTLYSUBMITTED", "resource": "112", "timestamp": "2011-09-30 22:38:44"},
    {"activity": "A_PREACCEPTED",     "resource": "112", "timestamp": "2011-09-30 22:39:37"},
]
print(predict(model, vocab, events, top_k=3, log="bpi2012"))
```

The models use a **global vocabulary** shipped in each folder: ids depend on the checkpoint (fm-v6/sc*: activities 4..776, resources 2..2739; real-only ablation: activities 4..933, resources 2..3772), with `UNK=1`; `vocab["log_ids"]` maps the seen logs. `inference.py` handles the global-id decoding. The log id is a conditioning input, not a task-specific head; for a new log you can use the UNK log id or fine-tune with the project code.

## Training

- Objective: causal next-event modeling (next activity + next resource) over 256-event windows.
- Architecture: pre-norm transformer encoder, learned positional embeddings, activity/resource/log embeddings + 6 numeric features; `id_dropout=0.15`, `log_dropout=0.15`.
- Data: event logs from BPIC/process-mining corpora plus synthetic logs generated by our DFG mixer (released as [`Polifemus/event-log-mixer`](https://huggingface.co/datasets/Polifemus/event-log-mixer)). Raw real logs are **not** redistributed. The real-only ablation uses 13 additional logs; sources and licenses are in [`scaling/ablation/real-only-cap/SOURCES.md`](scaling/ablation/real-only-cap/SOURCES.md).
- Fine-tuning protocol for the transfer numbers: full FT, causal attention for BPIC 2015-1, full bidirectional attention for traffic fines, 30 epochs, early stopping on validation, lr 3e-4.

## Scaling

`SCALING.md` contains the protocol, token budgets, caveats and the full table. Headline: on corpus v4, validation next-activity accuracy rises 0.6026 → 0.6451 → 0.6563 across 6.3M → 27.4M → 88.2M and then flattens at 306M (0.6538, best epoch 8), while zero-shot transfer keeps improving slightly (0.2977 → 0.3068 → 0.3600 → 0.3929). The scale effect at fixed mix is small; the **real-data fraction is the dominant lever**: rmix14 at 80M reaches **0.4695** fine-tuned and 0.3969 zero-shot, the best single model. The 306M point on corpus v5 (0.3090 zero-shot / 0.4267 FT) was confounded by the synthetic-heavy mix; see `SCALING.md`.

## Data-mix ablation (corpus v5)

The v5 corpus is 2,500 synthetic shards plus only 18 real logs. We ablated the real fraction by repeating the real shards ×14 (rmix14, ~9.5% of windows) and ×24 (rmix24, ~15.7%), at the same 80M architecture and an equal token budget (cap 600 windows/shard, 1 epoch):

| model | mix | real windows | zs BPIC 2015-1 | FT test acc | top3 | F1 |
|---|---|---:|---:|---:|---:|---:|
| sc80 (v4 ref) | v4 | 5.6% | 0.3810 | 0.4336 | 0.6280 | 0.1861 |
| sc80 v5 | default | 0.7% | 0.3363 | 0.4149 | 0.5918 | 0.1706 |
| **sc80 v5 rmix14** | ×14 | 9.5% | **0.3969** | **0.4695** | **0.6456** | **0.2096** |
| sc80 v5 rmix24 | ×24 | 15.7% | 0.3654 | 0.4597 | 0.6423 | 0.1967 |
| sc300 v5 | default | 0.7% | 0.3090 | 0.4267 | 0.6064 | 0.1932 |
| sc300 v5 rmix14 | ×14 | 9.5% | 0.3969 | 0.4448 | 0.6075 | 0.1945 |
| sc300 v4 (local) | v4 | 5.6% | **0.3929** | 0.4450 | 0.6206 | 0.1873 |

**Reading**: the real fraction is the dominant lever. rmix14 recovers +3.6pp fine-tuned and +1.6pp zero-shot over sc80-v4; more real is not monotone (rmix24 is lower). At 300M the mix recovers +8.8pp zero-shot / +1.8pp fine-tuned over the default v5, but on rmix14 the 300M model still does not beat the 80M (FT 0.4448 vs 0.4695). At fixed v4 mix the 300M is +3.3pp zero-shot / +1.1pp FT over 80M (validation flat): the scale effect is small, the mix effect is larger. With the corpus fixed at v4, the 300M model beats the 80M by +3.3pp zero-shot / +1.1pp fine-tuned (validation flat), so the earlier v5 negative result was partly confounded by the mix. Still, at this budget the mix lever (+3.6pp FT at 80M) is larger than the scale lever (80M→300M at fixed v4). The best single model is `sc80-v5-rmix14` (published here as an ablation checkpoint); the best overall result is its late fusion with `process-graph-fm/gfm-sc80` (test 0.4832, see process-graph-fm).

## Real-log expansion — 13 new real logs (real-only ablation)

To test whether **more real data** is the next lever, we added 13 real event logs
(CoSeLoG WABO 1–5, CoSeLoG Receipt, BPIC 2017 Offer, Airline, ICP, KIM, MCRM,
finale, Hospital Billing; ~1.13M events, ~0.58M training events) and rebuilt the
global vocabulary (`vocab v7`: 333 log ids, 930 activities, 3,771 resources).
Two 89.1M models were trained on **real logs only** at equal budget (~0.38B
positions, cap 4,000 windows per log):

| folder | real logs | epochs × steps | zero-shot | FT test acc | FT top3 | FT macro-F1 |
|---|---|---:|---:|---:|---:|---:|
| `scaling/ablation/real-only-cap/real18cap/` | 18 | 80 × 145 | 0.3703 | 0.4418 | 0.5960 | 0.1909 |
| `scaling/ablation/real-only-cap/real31cap/` | 31 (18 + 13 new) | 55 × 210 | 0.3010 | **0.4457** | **0.6036** | 0.1892 |

References: sc80 on corpus v4 (0.3600 / 0.4336), real-only 18 logs without cap
(0.3683 / 0.4377), sc80 v5 rmix14 (0.3969 / **0.4695**).

- **Adding 13 logs helps fine-tuning only marginally**: 0.4418 → 0.4457
  (+0.4pp), above sc80-v4 and the 18-log no-cap control, but far from rmix14.
- **Zero-shot drops** 0.3703 → 0.3010: the larger vocabulary and log mixture
  dilute the lexical overlap that drives transfer to BPIC 2015-1. Real-only
  scaling is not the lever; the next step is a mix/vocabulary-controlled run
  with the new logs.
- Raw logs are **not redistributed**; provenance, counts and licenses are in
  [`scaling/ablation/real-only-cap/SOURCES.md`](scaling/ablation/real-only-cap/SOURCES.md).
  The ICP log contains untimed `ArtificialStartTask` events: the prep imputes
  the case-minimum timestamp, and training now fails fast on non-finite
  features. Single seed per run.

## MoE ablation (archived)

A 1.1B-parameter sparse MoE (8 experts, top-2, ~340M active) was trained for 2 epochs as an ablation. It did **not** beat the dense models: on BPIC 2015-1 fine-tuning it reached **0.4098** test accuracy vs **0.4495** for `fm-v6`, and on traffic fines it tied (0.8832 vs 0.8835). The pretraining validation plateaued after epoch 1 and the run was closed. The checkpoint is archived locally; it is intentionally not released as a flagship model.

## Data & licensing

Model weights and vocabularies are released under **MIT**. The raw event logs remain under their original licenses (mostly 4TU General Terms of Use; BPIC 2018 is CC0; the Improving Process Trees logs are CC BY 4.0) and are not redistributed. The 13-log expansion used in the real-only ablation is documented with per-log sources, counts and licenses in [`scaling/ablation/real-only-cap/SOURCES.md`](scaling/ablation/real-only-cap/SOURCES.md). The synthetic half of the pretraining corpus is available at [`Polifemus/event-log-mixer`](https://huggingface.co/datasets/Polifemus/event-log-mixer) under MIT.

## Related repositories

- [`Polifemus/prefix-next-activity`](https://huggingface.co/Polifemus/prefix-next-activity) — GNN vs fine-tuned transformer study (the original 7-log benchmark).
- [`Polifemus/process-graph-fm`](https://huggingface.co/Polifemus/process-graph-fm) — graph (GPS) foundation model.
- [`Polifemus/event-log-mixer`](https://huggingface.co/datasets/Polifemus/event-log-mixer) — synthetic event-log corpus.

## Citation

```bibtex
@misc{process-fm,
  title  = {Process-FM: Event-Log Foundation Models and Scaling Suite},
  author = {Polidori, Filippo},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Polifemus/process-fm}}
}
```