Polifemus commited on
Commit
3daadd8
·
verified ·
1 Parent(s): 648d744

Add real-only cap4000 ablation (18 vs 31 real logs, vocab v7)

Browse files
scaling/ablation/real-only-cap/README.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Real-only cap4000 ablation — 18 vs 31 real logs
2
+
3
+ Two 89.1M-parameter causal transformers trained on **real logs only**, at equal
4
+ token budget (~0.38B positions), to test whether adding 13 new real event logs
5
+ improves transfer. Both use the global vocabulary v7 (333 log ids, 934/3,773
6
+ activity/resource embedding rows) and a cap of 4,000 windows per log.
7
+
8
+ | folder | real logs | epochs × steps | params | val next-activity acc* | BPIC 2015-1 zero-shot | FT test acc | FT top3 | FT macro-F1 |
9
+ |---|---|---:|---:|---:|---:|---:|---:|---:|
10
+ | `real18cap/` | 18 | 80 × 145 | 89.1M | 0.8133 | 0.3703 | 0.4418 | 0.5960 | 0.1909 |
11
+ | `real31cap/` | 31 (18 + 13 new) | 55 × 210 | 89.1M | 0.8291 | 0.3010 | 0.4457 | 0.6036 | 0.1892 |
12
+
13
+ \* Validation is measured on the real-only held-out cases of the respective
14
+ corpus and is **not comparable** with the `sc*` rows in the main README, which
15
+ use a different corpus and vocabulary.
16
+
17
+ ## References
18
+
19
+ | model | zero-shot | FT test acc |
20
+ |---|---:|---:|
21
+ | sc80 on corpus v4 | 0.3600 | 0.4336 |
22
+ | real-only 18 logs, no cap (41 epochs) | 0.3683 | 0.4377 |
23
+ | sc80 on v5 rmix14 | 0.3969 | 0.4695 |
24
+
25
+ ## Reading
26
+
27
+ - Adding the 13 logs gives a small fine-tuning gain: 0.4418 → **0.4457**
28
+ (+0.4pp), and both runs beat the 18-log no-cap control (0.4377) and sc80-v4
29
+ (0.4336) on FT.
30
+ - Zero-shot drops sharply: 0.3703 → **0.3010** (−6.9pp). The larger vocabulary
31
+ and log mixture dilute the lexical overlap that drives the canonical
32
+ zero-shot transfer to BPIC 2015-1.
33
+ - The best single model remains `scaling/ablation/sc80-v5-rmix14`
34
+ (0.3969 / 0.4695): more real logs alone do not replace a mix anchored on the
35
+ real corpus. The next lever is a mix/vocabulary-controlled run with the new
36
+ logs, not real-only scaling.
37
+
38
+ ## Files
39
+
40
+ - `real18cap/`, `real31cap/` — `model.pt` (state dict), `config.json`,
41
+ `metrics.json`, `vocab.json` (global vocab v7, self-contained for
42
+ `inference.py`).
43
+ - `SOURCES.md` — provenance, per-log counts and licensing of the 13 new logs
44
+ (raw logs are not redistributed).
45
+
46
+ ## Reproduce
47
+
48
+ Training and evaluation used the project code (`prep.py`, `fm_extend.py`,
49
+ `fm_pretrain.py`, `fm_finetune.py`) with seed 42, window 256, stride 128,
50
+ batch 128, `id_dropout=0.15`, `log_dropout=0.15`, AdamW. The two chains are
51
+ `chain_realonly_cap18.sh` and `chain_realonly_cap31.sh`. Raw data must be
52
+ downloaded from the original sources listed in `SOURCES.md`.
53
+
54
+ ## Caveats
55
+
56
+ - Single seed per run.
57
+ - Raw event logs are not redistributed; only derived artifacts are released.
58
+ - The validation sets differ from the `sc*` suite; compare those columns only
59
+ within this ablation.
scaling/ablation/real-only-cap/SOURCES.md ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Real-log expansion — provenance and licensing
2
+
3
+ This ablation adds 13 real event logs to the 18 logs used by the scaling suite.
4
+ The raw logs are **not redistributed** here; this folder contains only model
5
+ checkpoints, metrics and the derived global vocabulary. The original licenses
6
+ and terms of use apply to the raw data.
7
+
8
+ Vocabulary `vocab.json` (v7): 333 log ids (332 + UNK), 930 activities
9
+ (`n_act` = 934 with offsets) and 3,771 resources (`n_res` = 3,773).
10
+
11
+ ## New logs
12
+
13
+ Counts are computed after the project's `prep.py` (COMPLETE lifecycle, prefix
14
+ generation); `activities` and `resources` are the per-log vocabulary sizes.
15
+
16
+ | log | source | identifier | events | cases | activities | resources |
17
+ |---|---|---|---:|---:|---:|---:|
18
+ | cose_wabo_1 | CoSeLoG WABO, municipality 1 (4TU.ResearchData) | item `a928c371-d8e8-4c14-95db-2c28078085b5` | 38,943 | 937 | 380 | 18 |
19
+ | cose_wabo_2 | CoSeLoG WABO, municipality 2 (4TU.ResearchData) | item `a1a87437-ce83-4b34-9c0e-d8fc53401e4d` | 33,373 | 645 | 376 | 8 |
20
+ | cose_wabo_3 | CoSeLoG WABO, municipality 3 (4TU.ResearchData) | item `3b3687b9-2828-4304-8548-e066f8dbf1df` | 44,801 | 1,087 | 369 | 14 |
21
+ | cose_wabo_4 | CoSeLoG WABO, municipality 4 (4TU.ResearchData) | item `944cbf48-361f-4bbe-b227-a9ad3d2a5ac0` | 34,803 | 787 | 330 | 8 |
22
+ | cose_wabo_5 | CoSeLoG WABO, municipality 5 (4TU.ResearchData) | item `5a29e45a-f115-465c-9aeb-da07b88e69ea` | 43,872 | 891 | 349 | 13 |
23
+ | cose_receipt | CoSeLoG Receipt phase (4TU.ResearchData) | item `2db2e3c1-9499-4699-9098-1a28c15a5913` | 8,577 | 1,434 | 27 | 45 |
24
+ | bpi2017_offer | BPI Challenge 2017 Offer log (4TU.ResearchData) | item `cc497753-1175-41f6-a107-425787c54266` | 193,849 | 42,995 | 8 | 125 |
25
+ | airline | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 59,529 | 2,680 | 25 | 1 |
26
+ | icp | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 65,653 | 12,391 | 70 | 1 |
27
+ | kim | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 124,217 | 24,770 | 18 | 1 |
28
+ | mcrm | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 11,218 | 956 | 22 | 1 |
29
+ | finale | Help desk log of an Italian company (4TU.ResearchData) | article 12675977, DOI `10.4121/uuid:0c60edf1-6f83-4e75-9367-4c63b3e9d5bb` | 21,348 | 4,580 | 14 | 19 |
30
+ | hospital_billing | Hospital Billing — Event Log (4TU.ResearchData) | DOI `10.4121/uuid:76c46b83-c930-4798-a1c9-4be94dfeb741` | 451,359 | 100,000 | 18 | 989 |
31
+
32
+ Total: 1,131,542 events and 194,153 cases (all splits); about 0.58M training
33
+ events after the standard 70/15/15 case-level split.
34
+
35
+ - figshare article: <https://figshare.com/articles/dataset/Improving_Process_Trees_-_Event_Log/25962715>
36
+ - 4TU.ResearchData: <https://data.4tu.nl/> (search by item UUID or DOI)
37
+ - finale: <https://data.4tu.nl/articles/dataset/Dataset_belonging_to_the_help_desk_log_of_an_Italian_Company/12675977>
38
+ - hospital_billing: <https://doi.org/10.4121/uuid:76c46b83-c930-4798-a1c9-4be94dfeb741>
39
+
40
+ ## Notes
41
+
42
+ - `hospital_billing` carries 100,000 traces and ~451k events; it dominates the
43
+ added volume.
44
+ - `icp` contains 12,391 `ArtificialStartTask` events without timestamps. The
45
+ prep imputes each missing timestamp with the minimum timestamp of its case;
46
+ no NaT remains in the feature matrix. The training script now fails fast if
47
+ any npz contains non-finite numeric features.
48
+ - The raw XES/CSV files stay under their original licenses (4TU General Terms
49
+ of Use / figshare CC BY 4.0) and are not part of this repository.
scaling/ablation/real-only-cap/real18cap/config.json ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "EventTransformerPretrained",
3
+ "task": "causal next-activity + next-resource pretraining",
4
+ "corpus": "real-only: 18 real logs, cap 4000 windows/log (vocab v7)",
5
+ "vocab": "vocab.json (global vocab v7)",
6
+ "n_act": 934,
7
+ "n_res": 3773,
8
+ "n_logs": 334,
9
+ "n_classes": 934,
10
+ "d_model": 768,
11
+ "n_layers": 12,
12
+ "n_heads": 8,
13
+ "dropout": 0.1,
14
+ "max_len": 256,
15
+ "act_dim": 64,
16
+ "res_dim": 32,
17
+ "log_dim": 16,
18
+ "attention": "causal",
19
+ "id_dropout": 0.15,
20
+ "log_dropout": 0.15,
21
+ "log_unk": true,
22
+ "seen_logs": [
23
+ "bpi2012",
24
+ "bpi2017",
25
+ "bpi2013_incidents",
26
+ "bpi2013_closed",
27
+ "bpi2013_open",
28
+ "bpi2020_intl",
29
+ "bpi2020_dom",
30
+ "bpi2020_prepaid",
31
+ "bpi2020_rfp",
32
+ "sepsis",
33
+ "helpdesk",
34
+ "bpi2019",
35
+ "bpi2018",
36
+ "bpi2014",
37
+ "bpi2015_2",
38
+ "bpi2015_3",
39
+ "bpi2015_4",
40
+ "bpi2015_5"
41
+ ],
42
+ "training": {
43
+ "epochs": 80,
44
+ "steps_per_epoch": 145,
45
+ "batch_size": 128,
46
+ "window": 256,
47
+ "stride": 128,
48
+ "cap_windows_per_log": 4000,
49
+ "seed": 42,
50
+ "positions_approx": 380000000.0
51
+ },
52
+ "best": {
53
+ "val_loss": 0.9187226933950141,
54
+ "val_act_acc": 0.8133117652098105,
55
+ "epoch": 11
56
+ },
57
+ "metrics": {
58
+ "bpi2015_1_zero_shot_acc": 0.37025948103792417,
59
+ "bpi2015_1_ft_test_acc": 0.441830624465355,
60
+ "bpi2015_1_ft_top3": 0.5959509552323924,
61
+ "bpi2015_1_ft_macro_f1": 0.19086491526933283
62
+ },
63
+ "notes": [
64
+ "Real-only control at ~0.38B positions; 18 established logs.",
65
+ "Validation accuracy is measured on the real-only held-out splits of this corpus and is not comparable with the sc* rows.",
66
+ "Single seed; raw event logs are not redistributed."
67
+ ]
68
+ }
scaling/ablation/real-only-cap/real18cap/metrics.json ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "pretrain": {
3
+ "val_loss": 0.9187226933950141,
4
+ "val_act_acc": 0.8133117652098105,
5
+ "epoch": 11
6
+ },
7
+ "zero_shot_bpi2015_1": {
8
+ "accuracy": 0.37025948103792417,
9
+ "top3": 0.44682064442543484,
10
+ "macro_f1": 0.19028307298266744
11
+ },
12
+ "finetune_bpi2015_1": {
13
+ "accuracy": 0.441830624465355,
14
+ "top3": 0.5959509552323924,
15
+ "macro_f1": 0.19086491526933283,
16
+ "best_epoch": 20,
17
+ "val_accuracy": 0.48997621474685693,
18
+ "zero_shot": {
19
+ "accuracy": 0.3109495295124038,
20
+ "top3": 0.5236669518106644,
21
+ "macro_f1": 0.15299290147576208
22
+ }
23
+ },
24
+ "budget": {
25
+ "epochs": 80,
26
+ "steps_per_epoch": 145,
27
+ "batch_size": 128,
28
+ "window": 256,
29
+ "positions_approx": 380000000.0
30
+ },
31
+ "refs": {
32
+ "sc80_v4": {
33
+ "zero_shot": 0.36,
34
+ "ft_test": 0.4336,
35
+ "top3": 0.628,
36
+ "f1": 0.1861
37
+ },
38
+ "real_only18_no_cap": {
39
+ "zero_shot": 0.3683,
40
+ "ft_test": 0.4377
41
+ },
42
+ "sc80_v5_rmix14": {
43
+ "zero_shot": 0.3969,
44
+ "ft_test": 0.4695,
45
+ "top3": 0.6456,
46
+ "f1": 0.2096
47
+ }
48
+ },
49
+ "caveats": [
50
+ "Single seed.",
51
+ "Validation is not comparable across different corpora/vocabularies.",
52
+ "Raw logs are not redistributed (original licenses apply)."
53
+ ]
54
+ }
scaling/ablation/real-only-cap/real18cap/model.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07c62f40c6398c1466217962812323820c5d18c5fb06183176131aa600b299d3
3
+ size 356642563
scaling/ablation/real-only-cap/real18cap/vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
scaling/ablation/real-only-cap/real31cap/config.json ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "EventTransformerPretrained",
3
+ "task": "causal next-activity + next-resource pretraining",
4
+ "corpus": "real-only: 31 real logs (18 + 13 new), cap 4000 windows/log (vocab v7)",
5
+ "vocab": "vocab.json (global vocab v7)",
6
+ "n_act": 934,
7
+ "n_res": 3773,
8
+ "n_logs": 334,
9
+ "n_classes": 934,
10
+ "d_model": 768,
11
+ "n_layers": 12,
12
+ "n_heads": 8,
13
+ "dropout": 0.1,
14
+ "max_len": 256,
15
+ "act_dim": 64,
16
+ "res_dim": 32,
17
+ "log_dim": 16,
18
+ "attention": "causal",
19
+ "id_dropout": 0.15,
20
+ "log_dropout": 0.15,
21
+ "log_unk": true,
22
+ "seen_logs": [
23
+ "bpi2012",
24
+ "bpi2017",
25
+ "bpi2013_incidents",
26
+ "bpi2013_closed",
27
+ "bpi2013_open",
28
+ "bpi2020_intl",
29
+ "bpi2020_dom",
30
+ "bpi2020_prepaid",
31
+ "bpi2020_rfp",
32
+ "sepsis",
33
+ "helpdesk",
34
+ "bpi2019",
35
+ "bpi2018",
36
+ "bpi2014",
37
+ "bpi2015_2",
38
+ "bpi2015_3",
39
+ "bpi2015_4",
40
+ "bpi2015_5",
41
+ "cose_wabo_1",
42
+ "cose_wabo_2",
43
+ "cose_wabo_3",
44
+ "cose_wabo_4",
45
+ "cose_wabo_5",
46
+ "cose_receipt",
47
+ "bpi2017_offer",
48
+ "airline",
49
+ "icp",
50
+ "kim",
51
+ "mcrm",
52
+ "finale",
53
+ "hospital_billing"
54
+ ],
55
+ "training": {
56
+ "epochs": 55,
57
+ "steps_per_epoch": 210,
58
+ "batch_size": 128,
59
+ "window": 256,
60
+ "stride": 128,
61
+ "cap_windows_per_log": 4000,
62
+ "seed": 42,
63
+ "positions_approx": 380000000.0
64
+ },
65
+ "best": {
66
+ "val_loss": 0.86675356285237,
67
+ "val_act_acc": 0.8290903582146996,
68
+ "epoch": 13
69
+ },
70
+ "metrics": {
71
+ "bpi2015_1_zero_shot_acc": 0.30096948959224407,
72
+ "bpi2015_1_ft_test_acc": 0.44568006843455943,
73
+ "bpi2015_1_ft_top3": 0.6036498431708013,
74
+ "bpi2015_1_ft_macro_f1": 0.1891734804765475
75
+ },
76
+ "notes": [
77
+ "Real-only A/B with 13 new real logs (vocab v7, 333 log ids).",
78
+ "Validation accuracy is measured on the real-only held-out splits of this corpus and is not comparable with the sc* rows.",
79
+ "Single seed; raw event logs are not redistributed."
80
+ ]
81
+ }
scaling/ablation/real-only-cap/real31cap/metrics.json ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "pretrain": {
3
+ "val_loss": 0.86675356285237,
4
+ "val_act_acc": 0.8290903582146996,
5
+ "epoch": 13
6
+ },
7
+ "zero_shot_bpi2015_1": {
8
+ "accuracy": 0.30096948959224407,
9
+ "top3": 0.35186769318505845,
10
+ "macro_f1": 0.17075795416010806
11
+ },
12
+ "finetune_bpi2015_1": {
13
+ "accuracy": 0.44568006843455943,
14
+ "top3": 0.6036498431708013,
15
+ "macro_f1": 0.1891734804765475,
16
+ "best_epoch": 20,
17
+ "val_accuracy": 0.4931475818325971,
18
+ "zero_shot": {
19
+ "accuracy": 0.2978329056173368,
20
+ "top3": 0.5028514399771885,
21
+ "macro_f1": 0.17080535911148195
22
+ }
23
+ },
24
+ "budget": {
25
+ "epochs": 55,
26
+ "steps_per_epoch": 210,
27
+ "batch_size": 128,
28
+ "window": 256,
29
+ "positions_approx": 380000000.0
30
+ },
31
+ "refs": {
32
+ "sc80_v4": {
33
+ "zero_shot": 0.36,
34
+ "ft_test": 0.4336,
35
+ "top3": 0.628,
36
+ "f1": 0.1861
37
+ },
38
+ "real_only18_no_cap": {
39
+ "zero_shot": 0.3683,
40
+ "ft_test": 0.4377
41
+ },
42
+ "sc80_v5_rmix14": {
43
+ "zero_shot": 0.3969,
44
+ "ft_test": 0.4695,
45
+ "top3": 0.6456,
46
+ "f1": 0.2096
47
+ }
48
+ },
49
+ "caveats": [
50
+ "Single seed.",
51
+ "Validation is not comparable across different corpora/vocabularies.",
52
+ "Raw logs are not redistributed (original licenses apply)."
53
+ ]
54
+ }
scaling/ablation/real-only-cap/real31cap/model.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7d5df65ba694474dbe3413ecfc5b170c7a09832462bc306eabaa3c8e4da4e8d1
3
+ size 356642563
scaling/ablation/real-only-cap/real31cap/vocab.json ADDED
The diff for this file is too large to render. See raw diff