Process-FM — event-log foundation models

Causal transformer foundation models for predictive process monitoring: given the prefix of a case (activities, resources, timestamps), predict the next activity and the next resource. All models share the same global vocabulary and the same inference.py; they differ in size and training budget.

This repository contains the fm-v6 foundation checkpoint, the scaling suite (sc5 → sc27 → sc80 → sc300) and two real-only ablations on an expanded 333-log vocabulary.

Models

folder	params	d_model / layers	training corpus	positions seen	val next-activity acc	BPIC 2015-1 zero-shot	BPIC 2015-1 FT test acc	FT top3	FT macro-F1
fm-v6/	27.4M	512 / 8	v3: 18 real + 40 synthetic (58 logs)	~0.43B	0.7527	0.3480	0.4495	0.6436	0.205
scaling/sc5/	6.32M	320 / 4	v4: 18 real + 300 synthetic (308 logs)	0.04B	0.6026	0.2977	0.3931	0.5642	0.163
scaling/sc27/	27.36M	512 / 8	v4	0.13B	0.6451	0.3068	0.4296	0.5865	0.1877
scaling/sc80/	88.19M	768 / 12	v4	0.38B	0.6563	0.3600	0.4336	0.6280	0.1861
scaling/sc300/	306.4M	1024 / 24	v5 (1 epoch, reduced budget)	~1.28B	0.6080*	0.3090	0.4267	0.6064	0.1932
— (local, not released)	306.4M	1024 / 24	v4: 18 real + 300 synthetic (308 logs)	~1.28B	0.6538†	0.3929	0.4450	0.6206	0.1873
scaling/ablation/sc80-v5-rmix14/	88.19M	768 / 12	v5 rmix14 (18 real ×14 + 2,500 synth)	~0.41B	0.6158	0.3969	0.4695	0.6456	0.2096
scaling/ablation/real-only-cap/real18cap/	89.1M	768 / 12	real-only: 18 logs, cap 4000 (vocab v7)	~0.38B	0.8133‡	0.3703	0.4418	0.5960	0.1909
scaling/ablation/real-only-cap/real31cap/	89.1M	768 / 12	real-only: 31 logs (18 + 13 new), cap 4000 (vocab v7)	~0.38B	0.8291‡	0.3010	0.4457	0.6036	0.1892

Notes:

• fm-v6 and the sc* models use different corpora and vocabularies (58 vs 308 seen logs) and are not directly comparable. The scaling curve is sc5 → sc27 → sc80 → sc300.
• † sc300-v4 is a controlled local run (best epoch 8); its checkpoint is not released. Validation is on corpus v4 and comparable with the other v4 points.
• The sc* points are under-budget by design (0.22-0.42× the 20-token/parameter target; sc300 uses 4.2 tokens/parameter), so the curve measures the under-trained regime, not Chinchilla-optimal performance. See SCALING.md.
• sc300 on corpus v4 (local ablation, checkpoint not released): a controlled follow-up on the same corpus as sc5/sc27/sc80, 61 epochs (~1.28B positions), best epoch 8. It isolates scale at fixed mix: validation is flat vs sc80 (0.6538 vs 0.6563) while transfer improves (+3.3pp zero-shot, +1.1pp FT). The mix remains the stronger lever: sc80 on rmix14 reaches 0.4695 FT, still the best single model. See the data-mix table below.
• sc300 is a negative result on the default v5 mix: trained on corpus v5 (2,500 synthetic shards + 18 real logs), it did not improve transfer over sc80. Its validation accuracy (0.6080) is measured on a different, synthetic-heavy validation set and is not comparable with sc5/sc27/sc80 (v4). Zero-shot on BPIC 2015-1 is 0.3090 (0.2869 without unseen-to-UNK) and fine-tuned test accuracy 0.4267, both below sc80. The data-mix ablation below shows the bottleneck is the real fraction: sc300 on rmix14 reaches 0.3969 zero-shot / 0.4448 fine-tuned, but still does not beat the 80M model on the same mix (0.4695). See SCALING.md.
• ‡ The two real-only-cap rows use an expanded vocabulary (333 log ids, 934/3,773 activity/resource rows) and a real-only corpus; their validation is on the respective held-out real-only cases and is not comparable with the fm-v6/sc* rows. See scaling/ablation/real-only-cap/README.md.

Quickstart

pip install torch numpy

import sys
sys.path.insert(0, ".")
from inference import load_model, predict

model, vocab = load_model("fm-v6")          # or "scaling/sc80"
events = [
    {"activity": "A_SUBMITTED",       "resource": "112", "timestamp": "2011-09-30 22:38:44"},
    {"activity": "A_PARTLYSUBMITTED", "resource": "112", "timestamp": "2011-09-30 22:38:44"},
    {"activity": "A_PREACCEPTED",     "resource": "112", "timestamp": "2011-09-30 22:39:37"},
]
print(predict(model, vocab, events, top_k=3, log="bpi2012"))

The models use a global vocabulary shipped in each folder: ids depend on the checkpoint (fm-v6/sc*: activities 4..776, resources 2..2739; real-only ablation: activities 4..933, resources 2..3772), with UNK=1; vocab["log_ids"] maps the seen logs. inference.py handles the global-id decoding. The log id is a conditioning input, not a task-specific head; for a new log you can use the UNK log id or fine-tune with the project code.

Training

• Objective: causal next-event modeling (next activity + next resource) over 256-event windows.
• Architecture: pre-norm transformer encoder, learned positional embeddings, activity/resource/log embeddings + 6 numeric features; id_dropout=0.15, log_dropout=0.15.
• Data: event logs from BPIC/process-mining corpora plus synthetic logs generated by our DFG mixer (released as Polifemus/event-log-mixer). Raw real logs are not redistributed. The real-only ablation uses 13 additional logs; sources and licenses are in scaling/ablation/real-only-cap/SOURCES.md.
• Fine-tuning protocol for the transfer numbers: full FT, causal attention for BPIC 2015-1, full bidirectional attention for traffic fines, 30 epochs, early stopping on validation, lr 3e-4.

Scaling

SCALING.md contains the protocol, token budgets, caveats and the full table. Headline: on corpus v4, validation next-activity accuracy rises 0.6026 → 0.6451 → 0.6563 across 6.3M → 27.4M → 88.2M and then flattens at 306M (0.6538, best epoch 8), while zero-shot transfer keeps improving slightly (0.2977 → 0.3068 → 0.3600 → 0.3929). The scale effect at fixed mix is small; the real-data fraction is the dominant lever: rmix14 at 80M reaches 0.4695 fine-tuned and 0.3969 zero-shot, the best single model. The 306M point on corpus v5 (0.3090 zero-shot / 0.4267 FT) was confounded by the synthetic-heavy mix; see SCALING.md.

Data-mix ablation (corpus v5)

The v5 corpus is 2,500 synthetic shards plus only 18 real logs. We ablated the real fraction by repeating the real shards ×14 (rmix14, ~9.5% of windows) and ×24 (rmix24, ~15.7%), at the same 80M architecture and an equal token budget (cap 600 windows/shard, 1 epoch):

model	mix	real windows	zs BPIC 2015-1	FT test acc	top3	F1
sc80 (v4 ref)	v4	5.6%	0.3810	0.4336	0.6280	0.1861
sc80 v5	default	0.7%	0.3363	0.4149	0.5918	0.1706
sc80 v5 rmix14	×14	9.5%	0.3969	0.4695	0.6456	0.2096
sc80 v5 rmix24	×24	15.7%	0.3654	0.4597	0.6423	0.1967
sc300 v5	default	0.7%	0.3090	0.4267	0.6064	0.1932
sc300 v5 rmix14	×14	9.5%	0.3969	0.4448	0.6075	0.1945
sc300 v4 (local)	v4	5.6%	0.3929	0.4450	0.6206	0.1873

Reading: the real fraction is the dominant lever. rmix14 recovers +3.6pp fine-tuned and +1.6pp zero-shot over sc80-v4; more real is not monotone (rmix24 is lower). At 300M the mix recovers +8.8pp zero-shot / +1.8pp fine-tuned over the default v5, but on rmix14 the 300M model still does not beat the 80M (FT 0.4448 vs 0.4695). At fixed v4 mix the 300M is +3.3pp zero-shot / +1.1pp FT over 80M (validation flat): the scale effect is small, the mix effect is larger. With the corpus fixed at v4, the 300M model beats the 80M by +3.3pp zero-shot / +1.1pp fine-tuned (validation flat), so the earlier v5 negative result was partly confounded by the mix. Still, at this budget the mix lever (+3.6pp FT at 80M) is larger than the scale lever (80M→300M at fixed v4). The best single model is sc80-v5-rmix14 (published here as an ablation checkpoint); the best overall result is its late fusion with process-graph-fm/gfm-sc80 (test 0.4832, see process-graph-fm).

Real-log expansion — 13 new real logs (real-only ablation)

To test whether more real data is the next lever, we added 13 real event logs (CoSeLoG WABO 1–5, CoSeLoG Receipt, BPIC 2017 Offer, Airline, ICP, KIM, MCRM, finale, Hospital Billing; 1.13M events, ~0.58M training events) and rebuilt the global vocabulary (vocab v7: 333 log ids, 930 activities, 3,771 resources). Two 89.1M models were trained on real logs only at equal budget (0.38B positions, cap 4,000 windows per log):

folder	real logs	epochs × steps	zero-shot	FT test acc	FT top3	FT macro-F1
scaling/ablation/real-only-cap/real18cap/	18	80 × 145	0.3703	0.4418	0.5960	0.1909
scaling/ablation/real-only-cap/real31cap/	31 (18 + 13 new)	55 × 210	0.3010	0.4457	0.6036	0.1892

References: sc80 on corpus v4 (0.3600 / 0.4336), real-only 18 logs without cap (0.3683 / 0.4377), sc80 v5 rmix14 (0.3969 / 0.4695).

• Adding 13 logs helps fine-tuning only marginally: 0.4418 → 0.4457 (+0.4pp), above sc80-v4 and the 18-log no-cap control, but far from rmix14.
• Zero-shot drops 0.3703 → 0.3010: the larger vocabulary and log mixture dilute the lexical overlap that drives transfer to BPIC 2015-1. Real-only scaling is not the lever; the next step is a mix/vocabulary-controlled run with the new logs.
• Raw logs are not redistributed; provenance, counts and licenses are in scaling/ablation/real-only-cap/SOURCES.md. The ICP log contains untimed ArtificialStartTask events: the prep imputes the case-minimum timestamp, and training now fails fast on non-finite features. Single seed per run.

MoE ablation (archived)

A 1.1B-parameter sparse MoE (8 experts, top-2, ~340M active) was trained for 2 epochs as an ablation. It did not beat the dense models: on BPIC 2015-1 fine-tuning it reached 0.4098 test accuracy vs 0.4495 for fm-v6, and on traffic fines it tied (0.8832 vs 0.8835). The pretraining validation plateaued after epoch 1 and the run was closed. The checkpoint is archived locally; it is intentionally not released as a flagship model.

Data & licensing

Model weights and vocabularies are released under MIT. The raw event logs remain under their original licenses (mostly 4TU General Terms of Use; BPIC 2018 is CC0; the Improving Process Trees logs are CC BY 4.0) and are not redistributed. The 13-log expansion used in the real-only ablation is documented with per-log sources, counts and licenses in scaling/ablation/real-only-cap/SOURCES.md. The synthetic half of the pretraining corpus is available at Polifemus/event-log-mixer under MIT.

Related repositories

• Polifemus/prefix-next-activity — GNN vs fine-tuned transformer study (the original 7-log benchmark).
• Polifemus/process-graph-fm — graph (GPS) foundation model.
• Polifemus/event-log-mixer — synthetic event-log corpus.

Citation

@misc{process-fm,
  title  = {Process-FM: Event-Log Foundation Models and Scaling Suite},
  author = {Polidori, Filippo},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Polifemus/process-fm}}
}