File size: 7,895 Bytes
6332e03 04022e4 648d744 04022e4 6332e03 648d744 6332e03 648d744 6332e03 9cc1000 648d744 9cc1000 648d744 9cc1000 01a6fe4 6332e03 648d744 6332e03 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | # Scaling suite — protocol and results
## Protocol
- **Data**: corpus v4 for sc5/sc27/sc80 (18 real logs + 300 synthetic logs; 308 seen logs, global vocabulary 777 activities / 2740 resources). sc300 uses corpus v5 (1 epoch, reduced number of windows).
- **Objective**: causal next-event modeling (next activity + next resource), 256-event windows, stride 128.
- **Regularization**: activity/resource identity dropout 0.15, log dropout 0.15.
- **Splits**: case-level temporal split; validation used for model selection; test touched once at the end.
- **Evaluation**: next-activity accuracy on validation; transfer to held-out BPIC 2015-1 (zero-shot and fine-tuning) and traffic fines (fine-tuning).
- **Budget**: all points are **under-budget** relative to 20 tokens/parameter (0.22-0.42×; sc300 uses 4.2 tokens/parameter). The curve therefore describes the under-trained regime and should not be read as a Chinchilla-optimal scaling law.
## Results
| model | params | d_model / layers | positions seen | tokens/param | val next-activity acc | BPIC 2015-1 zero-shot | BPIC 2015-1 FT test acc | FT top3 | FT macro-F1 |
|---|---:|---|---:|---:|---:|---:|---:|---:|---:|
| sc5 | 6.32M | 320 / 4 | 0.04B | 6.3 | 0.6026 | 0.2977 | 0.3931 | 0.5642 | 0.1630 |
| sc27 | 27.36M | 512 / 8 | 0.13B | 4.8 | 0.6451 | 0.3068 | 0.4296 | 0.5865 | 0.1877 |
| sc80 | 88.19M | 768 / 12 | 0.38B | 4.3 | **0.6563** | **0.3600** | 0.4336 | 0.6280 | 0.1861 |
| sc300 | 306.4M | 1024 / 24 | ~1.28B | 4.2 | 0.6080* | 0.3090 | 0.4267 | 0.6064 | 0.1932 |
| sc300 v4 (local ablation) | 306.4M | 1024 / 24 | ~1.28B | 4.2 | 0.6538† | 0.3929 | 0.4450 | 0.6206 | 0.1873 |
\* sc300 was trained on **corpus v5** (2,500 synthetic shards + 18 real logs), so its validation set is different and synthetic-heavy: **0.6080 is not comparable** with the v4 points. The transfer columns are the honest comparison.
† sc300-v4 is a controlled local run on **corpus v4** (same corpus as sc5/sc27/sc80), 61 epochs (~1.28B positions), best epoch 8; its checkpoint is not released. Its validation is on the v4 split and is comparable with the other v4 points.
Reference: `fm-v6` (27.4M, corpus v3, ~0.43B positions) achieves val 0.7527, zero-shot 0.3480, FT 0.4495 / top3 0.6436 / F1 0.205. It is not part of the v4 scaling curve because it uses a different corpus/vocabulary.
## Observations
- Validation accuracy is monotone in this under-budget regime: **+4.3pp** from 6.3M to 27.4M, **+1.1pp** from 27.4M to 88.2M.
- Zero-shot transfer to the held-out log also rises (0.2977 → 0.3068 → 0.3600); fine-tuned transfer is flatter (0.3931 → 0.4296 → 0.4336).
- **On corpus v4 the curve flattens after 80M**: validation 0.6026 → 0.6451 → 0.6563 → 0.6538 (sc300-v4, best epoch), while transfer improves modestly: zero-shot 0.2977 → 0.3068 → 0.3600 → **0.3929** and fine-tuned 0.3931 → 0.4296 → 0.4336 → **0.4450**.
- **The v5 sc300 point was confounded by the mix**: on v5 (2,500 synthetic shards + 18 real logs) zero-shot drops to 0.3090 and fine-tuned accuracy to 0.4267 (top3 0.6064, F1 0.1932), below sc80; on traffic fines it ties the established models (0.8837 vs 0.8835 for fm-v6). With the corpus held at v4, the 306M model is better than sc80 on transfer (+3.3pp zero-shot, +1.1pp FT).
- **Interpretation**: at ~4.2 tokens/parameter, scale adds only a small transfer gain once the mix is held fixed; the dominant lever is the real fraction. At 80M, rmix14 reaches 0.3969 zero-shot / **0.4695** FT, still the best single model. The sc300 folders are published as evidence, not as flagship models.
## Data-mix ablation (corpus v5)
The v5 corpus has only 18 real logs among 2,500 synthetic shards. We varied the real fraction by repeating the real shards (`rmix14` ×14 ≈ 9.5% of windows, `rmix24` ×24 ≈ 15.7%), at equal architecture and token budget (cap 600/shard, 1 epoch):
| model | mix | real windows | zs BPIC 2015-1 | FT test acc | top3 | F1 |
|---|---|---:|---:|---:|---:|---:|
| sc80 (v4 ref) | v4 | 5.6% | 0.3810 | 0.4336 | 0.6280 | 0.1861 |
| sc80 v5 | default | 0.7% | 0.3363 | 0.4149 | 0.5918 | 0.1706 |
| **sc80 v5 rmix14** | ×14 | 9.5% | **0.3969** | **0.4695** | **0.6456** | **0.2096** |
| sc80 v5 rmix24 | ×24 | 15.7% | 0.3654 | 0.4597 | 0.6423 | 0.1967 |
| sc300 v5 | default | 0.7% | 0.3090 | 0.4267 | 0.6064 | 0.1932 |
| sc300 v5 rmix14 | ×14 | 9.5% | 0.3969 | 0.4448 | 0.6075 | 0.1945 |
| sc300 v4 (local) | v4 | 5.6% | **0.3929** | 0.4450 | 0.6206 | 0.1873 |
- The real fraction is the **dominant lever**: rmix14 recovers +3.6pp FT and +1.6pp zero-shot over sc80-v4; more real is not monotone (rmix24 is lower).
- At 300M the mix recovers +8.8pp zero-shot / +1.8pp FT over the default v5, but on rmix14 the 300M still does not beat the 80M (FT 0.4448 vs 0.4695). At fixed v4 mix, 300M beats 80M by +3.3pp zero-shot / +1.1pp FT: the scale effect is real but smaller than the mix effect.
- At fixed v4 mix, 80M→300M adds +3.3pp zero-shot / +1.1pp FT (validation flat); the mix effect at fixed scale is larger, so the mix is the primary lever at this budget.
- The best single model is `scaling/ablation/sc80-v5-rmix14` (0.4695 FT); its late fusion with `process-graph-fm/gfm-sc80` reaches test 0.4832.
## Real-only cap4000 ablation (vocab v7)
- **Data**: real logs only, 18 vs 31 logs (31 = 18 + 13 new logs), global
vocabulary v7 (333 log ids, 934/3,773 activity/resource rows).
- **Budget**: 89.1M params (d768/12L), cap 4,000 windows/log, ~0.38B positions:
18 logs × 80 epochs × 145 steps; 31 logs × 55 epochs × 210 steps.
- **Evaluation**: same BPIC 2015-1 zero-shot and fine-tuning protocol as the
scaling suite; single seed per run.
| run | real logs | epochs × steps | zero-shot | FT test acc | FT top3 | FT macro-F1 |
|---|---|---:|---:|---:|---:|---:|
| real-only 18 cap4000 | 18 | 80 × 145 | 0.3703 | 0.4418 | 0.5960 | 0.1909 |
| real-only 31 cap4000 | 31 | 55 × 210 | 0.3010 | **0.4457** | **0.6036** | 0.1892 |
| sc80 v4 (ref) | 18 | 18 × 639 | **0.3810** | 0.4336 | 0.6280 | 0.1861 |
| real-only 18 no-cap (ref) | 18 | 41 ep | 0.3683 | 0.4377 | 0.6032 | 0.1764 |
| sc80 v5 rmix14 (ref) | 18 ×14 + synth | 1 ep | **0.3969** | **0.4695** | **0.6456** | **0.2096** |
- The 13 new logs give **+0.4pp fine-tuned** (0.4418 → 0.4457) but **−6.9pp
zero-shot** (0.3703 → 0.3010). The larger vocabulary/log mixture dilutes the
lexical overlap that drives zero-shot transfer to BPIC 2015-1.
- Real-only scaling is not the dominant lever at this budget; the best single
model remains sc80-v5-rmix14 (0.3969 / 0.4695). The next experiment should
control the vocabulary and mix the new logs with the real anchor corpus.
- Validation for these two rows is on the respective real-only held-out cases
and is **not comparable** with the v4/v5 points above.
- Raw logs are not redistributed; per-log provenance and licenses are in
`scaling/ablation/real-only-cap/SOURCES.md`.
## Caveats
- Single seed per point; no error bars across seeds at these budgets.
- The real-vs-synthetic mix is ablated at 80M (def/rmix14/rmix24) and at 300M (def/rmix14). The mix is the dominant lever; at equal rmix14 mix, 300M does not beat 80M (FT 0.4448 vs 0.4695), while at fixed v4 mix 300M gains +1.1pp FT over 80M.
- sc300 (v5) is a reduced local run (1.28B positions, not the 6B full-budget point), chosen to stay within ~3 days on one GB10; it changes two variables at once (scale and mix). The sc300-v4 control fixes the mix and isolates scale: it shows a small transfer gain over sc80-v4, while the mix ablation shows a larger gain.
- sc300-v4 is a single-seed local ablation whose checkpoint is not released; its numbers come from the same pipeline and metrics as the other points.
- Numbers are test-set accuracy after validation-only model selection; the first-order argmax baseline and majority baseline are reported in the study repo.
|