Add real-only cap4000 ablation (18 vs 31 real logs, vocab v7)
Browse files- scaling/ablation/real-only-cap/README.md +59 -0
- scaling/ablation/real-only-cap/SOURCES.md +49 -0
- scaling/ablation/real-only-cap/real18cap/config.json +68 -0
- scaling/ablation/real-only-cap/real18cap/metrics.json +54 -0
- scaling/ablation/real-only-cap/real18cap/model.pt +3 -0
- scaling/ablation/real-only-cap/real18cap/vocab.json +0 -0
- scaling/ablation/real-only-cap/real31cap/config.json +81 -0
- scaling/ablation/real-only-cap/real31cap/metrics.json +54 -0
- scaling/ablation/real-only-cap/real31cap/model.pt +3 -0
- scaling/ablation/real-only-cap/real31cap/vocab.json +0 -0
scaling/ablation/real-only-cap/README.md
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Real-only cap4000 ablation — 18 vs 31 real logs
|
| 2 |
+
|
| 3 |
+
Two 89.1M-parameter causal transformers trained on **real logs only**, at equal
|
| 4 |
+
token budget (~0.38B positions), to test whether adding 13 new real event logs
|
| 5 |
+
improves transfer. Both use the global vocabulary v7 (333 log ids, 934/3,773
|
| 6 |
+
activity/resource embedding rows) and a cap of 4,000 windows per log.
|
| 7 |
+
|
| 8 |
+
| folder | real logs | epochs × steps | params | val next-activity acc* | BPIC 2015-1 zero-shot | FT test acc | FT top3 | FT macro-F1 |
|
| 9 |
+
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
| 10 |
+
| `real18cap/` | 18 | 80 × 145 | 89.1M | 0.8133 | 0.3703 | 0.4418 | 0.5960 | 0.1909 |
|
| 11 |
+
| `real31cap/` | 31 (18 + 13 new) | 55 × 210 | 89.1M | 0.8291 | 0.3010 | 0.4457 | 0.6036 | 0.1892 |
|
| 12 |
+
|
| 13 |
+
\* Validation is measured on the real-only held-out cases of the respective
|
| 14 |
+
corpus and is **not comparable** with the `sc*` rows in the main README, which
|
| 15 |
+
use a different corpus and vocabulary.
|
| 16 |
+
|
| 17 |
+
## References
|
| 18 |
+
|
| 19 |
+
| model | zero-shot | FT test acc |
|
| 20 |
+
|---|---:|---:|
|
| 21 |
+
| sc80 on corpus v4 | 0.3600 | 0.4336 |
|
| 22 |
+
| real-only 18 logs, no cap (41 epochs) | 0.3683 | 0.4377 |
|
| 23 |
+
| sc80 on v5 rmix14 | 0.3969 | 0.4695 |
|
| 24 |
+
|
| 25 |
+
## Reading
|
| 26 |
+
|
| 27 |
+
- Adding the 13 logs gives a small fine-tuning gain: 0.4418 → **0.4457**
|
| 28 |
+
(+0.4pp), and both runs beat the 18-log no-cap control (0.4377) and sc80-v4
|
| 29 |
+
(0.4336) on FT.
|
| 30 |
+
- Zero-shot drops sharply: 0.3703 → **0.3010** (−6.9pp). The larger vocabulary
|
| 31 |
+
and log mixture dilute the lexical overlap that drives the canonical
|
| 32 |
+
zero-shot transfer to BPIC 2015-1.
|
| 33 |
+
- The best single model remains `scaling/ablation/sc80-v5-rmix14`
|
| 34 |
+
(0.3969 / 0.4695): more real logs alone do not replace a mix anchored on the
|
| 35 |
+
real corpus. The next lever is a mix/vocabulary-controlled run with the new
|
| 36 |
+
logs, not real-only scaling.
|
| 37 |
+
|
| 38 |
+
## Files
|
| 39 |
+
|
| 40 |
+
- `real18cap/`, `real31cap/` — `model.pt` (state dict), `config.json`,
|
| 41 |
+
`metrics.json`, `vocab.json` (global vocab v7, self-contained for
|
| 42 |
+
`inference.py`).
|
| 43 |
+
- `SOURCES.md` — provenance, per-log counts and licensing of the 13 new logs
|
| 44 |
+
(raw logs are not redistributed).
|
| 45 |
+
|
| 46 |
+
## Reproduce
|
| 47 |
+
|
| 48 |
+
Training and evaluation used the project code (`prep.py`, `fm_extend.py`,
|
| 49 |
+
`fm_pretrain.py`, `fm_finetune.py`) with seed 42, window 256, stride 128,
|
| 50 |
+
batch 128, `id_dropout=0.15`, `log_dropout=0.15`, AdamW. The two chains are
|
| 51 |
+
`chain_realonly_cap18.sh` and `chain_realonly_cap31.sh`. Raw data must be
|
| 52 |
+
downloaded from the original sources listed in `SOURCES.md`.
|
| 53 |
+
|
| 54 |
+
## Caveats
|
| 55 |
+
|
| 56 |
+
- Single seed per run.
|
| 57 |
+
- Raw event logs are not redistributed; only derived artifacts are released.
|
| 58 |
+
- The validation sets differ from the `sc*` suite; compare those columns only
|
| 59 |
+
within this ablation.
|
scaling/ablation/real-only-cap/SOURCES.md
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Real-log expansion — provenance and licensing
|
| 2 |
+
|
| 3 |
+
This ablation adds 13 real event logs to the 18 logs used by the scaling suite.
|
| 4 |
+
The raw logs are **not redistributed** here; this folder contains only model
|
| 5 |
+
checkpoints, metrics and the derived global vocabulary. The original licenses
|
| 6 |
+
and terms of use apply to the raw data.
|
| 7 |
+
|
| 8 |
+
Vocabulary `vocab.json` (v7): 333 log ids (332 + UNK), 930 activities
|
| 9 |
+
(`n_act` = 934 with offsets) and 3,771 resources (`n_res` = 3,773).
|
| 10 |
+
|
| 11 |
+
## New logs
|
| 12 |
+
|
| 13 |
+
Counts are computed after the project's `prep.py` (COMPLETE lifecycle, prefix
|
| 14 |
+
generation); `activities` and `resources` are the per-log vocabulary sizes.
|
| 15 |
+
|
| 16 |
+
| log | source | identifier | events | cases | activities | resources |
|
| 17 |
+
|---|---|---|---:|---:|---:|---:|
|
| 18 |
+
| cose_wabo_1 | CoSeLoG WABO, municipality 1 (4TU.ResearchData) | item `a928c371-d8e8-4c14-95db-2c28078085b5` | 38,943 | 937 | 380 | 18 |
|
| 19 |
+
| cose_wabo_2 | CoSeLoG WABO, municipality 2 (4TU.ResearchData) | item `a1a87437-ce83-4b34-9c0e-d8fc53401e4d` | 33,373 | 645 | 376 | 8 |
|
| 20 |
+
| cose_wabo_3 | CoSeLoG WABO, municipality 3 (4TU.ResearchData) | item `3b3687b9-2828-4304-8548-e066f8dbf1df` | 44,801 | 1,087 | 369 | 14 |
|
| 21 |
+
| cose_wabo_4 | CoSeLoG WABO, municipality 4 (4TU.ResearchData) | item `944cbf48-361f-4bbe-b227-a9ad3d2a5ac0` | 34,803 | 787 | 330 | 8 |
|
| 22 |
+
| cose_wabo_5 | CoSeLoG WABO, municipality 5 (4TU.ResearchData) | item `5a29e45a-f115-465c-9aeb-da07b88e69ea` | 43,872 | 891 | 349 | 13 |
|
| 23 |
+
| cose_receipt | CoSeLoG Receipt phase (4TU.ResearchData) | item `2db2e3c1-9499-4699-9098-1a28c15a5913` | 8,577 | 1,434 | 27 | 45 |
|
| 24 |
+
| bpi2017_offer | BPI Challenge 2017 Offer log (4TU.ResearchData) | item `cc497753-1175-41f6-a107-425787c54266` | 193,849 | 42,995 | 8 | 125 |
|
| 25 |
+
| airline | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 59,529 | 2,680 | 25 | 1 |
|
| 26 |
+
| icp | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 65,653 | 12,391 | 70 | 1 |
|
| 27 |
+
| kim | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 124,217 | 24,770 | 18 | 1 |
|
| 28 |
+
| mcrm | Improving Process Trees — Event Log (figshare) | article 25962715, CC BY 4.0 | 11,218 | 956 | 22 | 1 |
|
| 29 |
+
| finale | Help desk log of an Italian company (4TU.ResearchData) | article 12675977, DOI `10.4121/uuid:0c60edf1-6f83-4e75-9367-4c63b3e9d5bb` | 21,348 | 4,580 | 14 | 19 |
|
| 30 |
+
| hospital_billing | Hospital Billing — Event Log (4TU.ResearchData) | DOI `10.4121/uuid:76c46b83-c930-4798-a1c9-4be94dfeb741` | 451,359 | 100,000 | 18 | 989 |
|
| 31 |
+
|
| 32 |
+
Total: 1,131,542 events and 194,153 cases (all splits); about 0.58M training
|
| 33 |
+
events after the standard 70/15/15 case-level split.
|
| 34 |
+
|
| 35 |
+
- figshare article: <https://figshare.com/articles/dataset/Improving_Process_Trees_-_Event_Log/25962715>
|
| 36 |
+
- 4TU.ResearchData: <https://data.4tu.nl/> (search by item UUID or DOI)
|
| 37 |
+
- finale: <https://data.4tu.nl/articles/dataset/Dataset_belonging_to_the_help_desk_log_of_an_Italian_Company/12675977>
|
| 38 |
+
- hospital_billing: <https://doi.org/10.4121/uuid:76c46b83-c930-4798-a1c9-4be94dfeb741>
|
| 39 |
+
|
| 40 |
+
## Notes
|
| 41 |
+
|
| 42 |
+
- `hospital_billing` carries 100,000 traces and ~451k events; it dominates the
|
| 43 |
+
added volume.
|
| 44 |
+
- `icp` contains 12,391 `ArtificialStartTask` events without timestamps. The
|
| 45 |
+
prep imputes each missing timestamp with the minimum timestamp of its case;
|
| 46 |
+
no NaT remains in the feature matrix. The training script now fails fast if
|
| 47 |
+
any npz contains non-finite numeric features.
|
| 48 |
+
- The raw XES/CSV files stay under their original licenses (4TU General Terms
|
| 49 |
+
of Use / figshare CC BY 4.0) and are not part of this repository.
|
scaling/ablation/real-only-cap/real18cap/config.json
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "EventTransformerPretrained",
|
| 3 |
+
"task": "causal next-activity + next-resource pretraining",
|
| 4 |
+
"corpus": "real-only: 18 real logs, cap 4000 windows/log (vocab v7)",
|
| 5 |
+
"vocab": "vocab.json (global vocab v7)",
|
| 6 |
+
"n_act": 934,
|
| 7 |
+
"n_res": 3773,
|
| 8 |
+
"n_logs": 334,
|
| 9 |
+
"n_classes": 934,
|
| 10 |
+
"d_model": 768,
|
| 11 |
+
"n_layers": 12,
|
| 12 |
+
"n_heads": 8,
|
| 13 |
+
"dropout": 0.1,
|
| 14 |
+
"max_len": 256,
|
| 15 |
+
"act_dim": 64,
|
| 16 |
+
"res_dim": 32,
|
| 17 |
+
"log_dim": 16,
|
| 18 |
+
"attention": "causal",
|
| 19 |
+
"id_dropout": 0.15,
|
| 20 |
+
"log_dropout": 0.15,
|
| 21 |
+
"log_unk": true,
|
| 22 |
+
"seen_logs": [
|
| 23 |
+
"bpi2012",
|
| 24 |
+
"bpi2017",
|
| 25 |
+
"bpi2013_incidents",
|
| 26 |
+
"bpi2013_closed",
|
| 27 |
+
"bpi2013_open",
|
| 28 |
+
"bpi2020_intl",
|
| 29 |
+
"bpi2020_dom",
|
| 30 |
+
"bpi2020_prepaid",
|
| 31 |
+
"bpi2020_rfp",
|
| 32 |
+
"sepsis",
|
| 33 |
+
"helpdesk",
|
| 34 |
+
"bpi2019",
|
| 35 |
+
"bpi2018",
|
| 36 |
+
"bpi2014",
|
| 37 |
+
"bpi2015_2",
|
| 38 |
+
"bpi2015_3",
|
| 39 |
+
"bpi2015_4",
|
| 40 |
+
"bpi2015_5"
|
| 41 |
+
],
|
| 42 |
+
"training": {
|
| 43 |
+
"epochs": 80,
|
| 44 |
+
"steps_per_epoch": 145,
|
| 45 |
+
"batch_size": 128,
|
| 46 |
+
"window": 256,
|
| 47 |
+
"stride": 128,
|
| 48 |
+
"cap_windows_per_log": 4000,
|
| 49 |
+
"seed": 42,
|
| 50 |
+
"positions_approx": 380000000.0
|
| 51 |
+
},
|
| 52 |
+
"best": {
|
| 53 |
+
"val_loss": 0.9187226933950141,
|
| 54 |
+
"val_act_acc": 0.8133117652098105,
|
| 55 |
+
"epoch": 11
|
| 56 |
+
},
|
| 57 |
+
"metrics": {
|
| 58 |
+
"bpi2015_1_zero_shot_acc": 0.37025948103792417,
|
| 59 |
+
"bpi2015_1_ft_test_acc": 0.441830624465355,
|
| 60 |
+
"bpi2015_1_ft_top3": 0.5959509552323924,
|
| 61 |
+
"bpi2015_1_ft_macro_f1": 0.19086491526933283
|
| 62 |
+
},
|
| 63 |
+
"notes": [
|
| 64 |
+
"Real-only control at ~0.38B positions; 18 established logs.",
|
| 65 |
+
"Validation accuracy is measured on the real-only held-out splits of this corpus and is not comparable with the sc* rows.",
|
| 66 |
+
"Single seed; raw event logs are not redistributed."
|
| 67 |
+
]
|
| 68 |
+
}
|
scaling/ablation/real-only-cap/real18cap/metrics.json
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"pretrain": {
|
| 3 |
+
"val_loss": 0.9187226933950141,
|
| 4 |
+
"val_act_acc": 0.8133117652098105,
|
| 5 |
+
"epoch": 11
|
| 6 |
+
},
|
| 7 |
+
"zero_shot_bpi2015_1": {
|
| 8 |
+
"accuracy": 0.37025948103792417,
|
| 9 |
+
"top3": 0.44682064442543484,
|
| 10 |
+
"macro_f1": 0.19028307298266744
|
| 11 |
+
},
|
| 12 |
+
"finetune_bpi2015_1": {
|
| 13 |
+
"accuracy": 0.441830624465355,
|
| 14 |
+
"top3": 0.5959509552323924,
|
| 15 |
+
"macro_f1": 0.19086491526933283,
|
| 16 |
+
"best_epoch": 20,
|
| 17 |
+
"val_accuracy": 0.48997621474685693,
|
| 18 |
+
"zero_shot": {
|
| 19 |
+
"accuracy": 0.3109495295124038,
|
| 20 |
+
"top3": 0.5236669518106644,
|
| 21 |
+
"macro_f1": 0.15299290147576208
|
| 22 |
+
}
|
| 23 |
+
},
|
| 24 |
+
"budget": {
|
| 25 |
+
"epochs": 80,
|
| 26 |
+
"steps_per_epoch": 145,
|
| 27 |
+
"batch_size": 128,
|
| 28 |
+
"window": 256,
|
| 29 |
+
"positions_approx": 380000000.0
|
| 30 |
+
},
|
| 31 |
+
"refs": {
|
| 32 |
+
"sc80_v4": {
|
| 33 |
+
"zero_shot": 0.36,
|
| 34 |
+
"ft_test": 0.4336,
|
| 35 |
+
"top3": 0.628,
|
| 36 |
+
"f1": 0.1861
|
| 37 |
+
},
|
| 38 |
+
"real_only18_no_cap": {
|
| 39 |
+
"zero_shot": 0.3683,
|
| 40 |
+
"ft_test": 0.4377
|
| 41 |
+
},
|
| 42 |
+
"sc80_v5_rmix14": {
|
| 43 |
+
"zero_shot": 0.3969,
|
| 44 |
+
"ft_test": 0.4695,
|
| 45 |
+
"top3": 0.6456,
|
| 46 |
+
"f1": 0.2096
|
| 47 |
+
}
|
| 48 |
+
},
|
| 49 |
+
"caveats": [
|
| 50 |
+
"Single seed.",
|
| 51 |
+
"Validation is not comparable across different corpora/vocabularies.",
|
| 52 |
+
"Raw logs are not redistributed (original licenses apply)."
|
| 53 |
+
]
|
| 54 |
+
}
|
scaling/ablation/real-only-cap/real18cap/model.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:07c62f40c6398c1466217962812323820c5d18c5fb06183176131aa600b299d3
|
| 3 |
+
size 356642563
|
scaling/ablation/real-only-cap/real18cap/vocab.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
scaling/ablation/real-only-cap/real31cap/config.json
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "EventTransformerPretrained",
|
| 3 |
+
"task": "causal next-activity + next-resource pretraining",
|
| 4 |
+
"corpus": "real-only: 31 real logs (18 + 13 new), cap 4000 windows/log (vocab v7)",
|
| 5 |
+
"vocab": "vocab.json (global vocab v7)",
|
| 6 |
+
"n_act": 934,
|
| 7 |
+
"n_res": 3773,
|
| 8 |
+
"n_logs": 334,
|
| 9 |
+
"n_classes": 934,
|
| 10 |
+
"d_model": 768,
|
| 11 |
+
"n_layers": 12,
|
| 12 |
+
"n_heads": 8,
|
| 13 |
+
"dropout": 0.1,
|
| 14 |
+
"max_len": 256,
|
| 15 |
+
"act_dim": 64,
|
| 16 |
+
"res_dim": 32,
|
| 17 |
+
"log_dim": 16,
|
| 18 |
+
"attention": "causal",
|
| 19 |
+
"id_dropout": 0.15,
|
| 20 |
+
"log_dropout": 0.15,
|
| 21 |
+
"log_unk": true,
|
| 22 |
+
"seen_logs": [
|
| 23 |
+
"bpi2012",
|
| 24 |
+
"bpi2017",
|
| 25 |
+
"bpi2013_incidents",
|
| 26 |
+
"bpi2013_closed",
|
| 27 |
+
"bpi2013_open",
|
| 28 |
+
"bpi2020_intl",
|
| 29 |
+
"bpi2020_dom",
|
| 30 |
+
"bpi2020_prepaid",
|
| 31 |
+
"bpi2020_rfp",
|
| 32 |
+
"sepsis",
|
| 33 |
+
"helpdesk",
|
| 34 |
+
"bpi2019",
|
| 35 |
+
"bpi2018",
|
| 36 |
+
"bpi2014",
|
| 37 |
+
"bpi2015_2",
|
| 38 |
+
"bpi2015_3",
|
| 39 |
+
"bpi2015_4",
|
| 40 |
+
"bpi2015_5",
|
| 41 |
+
"cose_wabo_1",
|
| 42 |
+
"cose_wabo_2",
|
| 43 |
+
"cose_wabo_3",
|
| 44 |
+
"cose_wabo_4",
|
| 45 |
+
"cose_wabo_5",
|
| 46 |
+
"cose_receipt",
|
| 47 |
+
"bpi2017_offer",
|
| 48 |
+
"airline",
|
| 49 |
+
"icp",
|
| 50 |
+
"kim",
|
| 51 |
+
"mcrm",
|
| 52 |
+
"finale",
|
| 53 |
+
"hospital_billing"
|
| 54 |
+
],
|
| 55 |
+
"training": {
|
| 56 |
+
"epochs": 55,
|
| 57 |
+
"steps_per_epoch": 210,
|
| 58 |
+
"batch_size": 128,
|
| 59 |
+
"window": 256,
|
| 60 |
+
"stride": 128,
|
| 61 |
+
"cap_windows_per_log": 4000,
|
| 62 |
+
"seed": 42,
|
| 63 |
+
"positions_approx": 380000000.0
|
| 64 |
+
},
|
| 65 |
+
"best": {
|
| 66 |
+
"val_loss": 0.86675356285237,
|
| 67 |
+
"val_act_acc": 0.8290903582146996,
|
| 68 |
+
"epoch": 13
|
| 69 |
+
},
|
| 70 |
+
"metrics": {
|
| 71 |
+
"bpi2015_1_zero_shot_acc": 0.30096948959224407,
|
| 72 |
+
"bpi2015_1_ft_test_acc": 0.44568006843455943,
|
| 73 |
+
"bpi2015_1_ft_top3": 0.6036498431708013,
|
| 74 |
+
"bpi2015_1_ft_macro_f1": 0.1891734804765475
|
| 75 |
+
},
|
| 76 |
+
"notes": [
|
| 77 |
+
"Real-only A/B with 13 new real logs (vocab v7, 333 log ids).",
|
| 78 |
+
"Validation accuracy is measured on the real-only held-out splits of this corpus and is not comparable with the sc* rows.",
|
| 79 |
+
"Single seed; raw event logs are not redistributed."
|
| 80 |
+
]
|
| 81 |
+
}
|
scaling/ablation/real-only-cap/real31cap/metrics.json
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"pretrain": {
|
| 3 |
+
"val_loss": 0.86675356285237,
|
| 4 |
+
"val_act_acc": 0.8290903582146996,
|
| 5 |
+
"epoch": 13
|
| 6 |
+
},
|
| 7 |
+
"zero_shot_bpi2015_1": {
|
| 8 |
+
"accuracy": 0.30096948959224407,
|
| 9 |
+
"top3": 0.35186769318505845,
|
| 10 |
+
"macro_f1": 0.17075795416010806
|
| 11 |
+
},
|
| 12 |
+
"finetune_bpi2015_1": {
|
| 13 |
+
"accuracy": 0.44568006843455943,
|
| 14 |
+
"top3": 0.6036498431708013,
|
| 15 |
+
"macro_f1": 0.1891734804765475,
|
| 16 |
+
"best_epoch": 20,
|
| 17 |
+
"val_accuracy": 0.4931475818325971,
|
| 18 |
+
"zero_shot": {
|
| 19 |
+
"accuracy": 0.2978329056173368,
|
| 20 |
+
"top3": 0.5028514399771885,
|
| 21 |
+
"macro_f1": 0.17080535911148195
|
| 22 |
+
}
|
| 23 |
+
},
|
| 24 |
+
"budget": {
|
| 25 |
+
"epochs": 55,
|
| 26 |
+
"steps_per_epoch": 210,
|
| 27 |
+
"batch_size": 128,
|
| 28 |
+
"window": 256,
|
| 29 |
+
"positions_approx": 380000000.0
|
| 30 |
+
},
|
| 31 |
+
"refs": {
|
| 32 |
+
"sc80_v4": {
|
| 33 |
+
"zero_shot": 0.36,
|
| 34 |
+
"ft_test": 0.4336,
|
| 35 |
+
"top3": 0.628,
|
| 36 |
+
"f1": 0.1861
|
| 37 |
+
},
|
| 38 |
+
"real_only18_no_cap": {
|
| 39 |
+
"zero_shot": 0.3683,
|
| 40 |
+
"ft_test": 0.4377
|
| 41 |
+
},
|
| 42 |
+
"sc80_v5_rmix14": {
|
| 43 |
+
"zero_shot": 0.3969,
|
| 44 |
+
"ft_test": 0.4695,
|
| 45 |
+
"top3": 0.6456,
|
| 46 |
+
"f1": 0.2096
|
| 47 |
+
}
|
| 48 |
+
},
|
| 49 |
+
"caveats": [
|
| 50 |
+
"Single seed.",
|
| 51 |
+
"Validation is not comparable across different corpora/vocabularies.",
|
| 52 |
+
"Raw logs are not redistributed (original licenses apply)."
|
| 53 |
+
]
|
| 54 |
+
}
|
scaling/ablation/real-only-cap/real31cap/model.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7d5df65ba694474dbe3413ecfc5b170c7a09832462bc306eabaa3c8e4da4e8d1
|
| 3 |
+
size 356642563
|
scaling/ablation/real-only-cap/real31cap/vocab.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|