SCALING.md

Scaling suite — protocol and results

Protocol

• Data: corpus v4 for sc5/sc27/sc80 (18 real logs + 300 synthetic logs; 308 seen logs, global vocabulary 777 activities / 2740 resources). sc300 uses corpus v5 (1 epoch, reduced number of windows).
• Objective: causal next-event modeling (next activity + next resource), 256-event windows, stride 128.
• Regularization: activity/resource identity dropout 0.15, log dropout 0.15.
• Splits: case-level temporal split; validation used for model selection; test touched once at the end.
• Evaluation: next-activity accuracy on validation; transfer to held-out BPIC 2015-1 (zero-shot and fine-tuning) and traffic fines (fine-tuning).
• Budget: all points are under-budget relative to 20 tokens/parameter (0.22-0.42×; sc300 uses 4.2 tokens/parameter). The curve therefore describes the under-trained regime and should not be read as a Chinchilla-optimal scaling law.

Results

model	params	d_model / layers	positions seen	tokens/param	val next-activity acc	BPIC 2015-1 zero-shot	BPIC 2015-1 FT test acc	FT top3	FT macro-F1
sc5	6.32M	320 / 4	0.04B	6.3	0.6026	0.2977	0.3931	0.5642	0.1630
sc27	27.36M	512 / 8	0.13B	4.8	0.6451	0.3068	0.4296	0.5865	0.1877
sc80	88.19M	768 / 12	0.38B	4.3	0.6563	0.3600	0.4336	0.6280	0.1861
sc300	306.4M	1024 / 24	~1.28B	4.2	0.6080*	0.3090	0.4267	0.6064	0.1932
sc300 v4 (local ablation)	306.4M	1024 / 24	~1.28B	4.2	0.6538†	0.3929	0.4450	0.6206	0.1873

* sc300 was trained on corpus v5 (2,500 synthetic shards + 18 real logs), so its validation set is different and synthetic-heavy: 0.6080 is not comparable with the v4 points. The transfer columns are the honest comparison.

† sc300-v4 is a controlled local run on corpus v4 (same corpus as sc5/sc27/sc80), 61 epochs (~1.28B positions), best epoch 8; its checkpoint is not released. Its validation is on the v4 split and is comparable with the other v4 points.

Reference: fm-v6 (27.4M, corpus v3, ~0.43B positions) achieves val 0.7527, zero-shot 0.3480, FT 0.4495 / top3 0.6436 / F1 0.205. It is not part of the v4 scaling curve because it uses a different corpus/vocabulary.

Observations

• Validation accuracy is monotone in this under-budget regime: +4.3pp from 6.3M to 27.4M, +1.1pp from 27.4M to 88.2M.
• Zero-shot transfer to the held-out log also rises (0.2977 → 0.3068 → 0.3600); fine-tuned transfer is flatter (0.3931 → 0.4296 → 0.4336).
• On corpus v4 the curve flattens after 80M: validation 0.6026 → 0.6451 → 0.6563 → 0.6538 (sc300-v4, best epoch), while transfer improves modestly: zero-shot 0.2977 → 0.3068 → 0.3600 → 0.3929 and fine-tuned 0.3931 → 0.4296 → 0.4336 → 0.4450.
• The v5 sc300 point was confounded by the mix: on v5 (2,500 synthetic shards + 18 real logs) zero-shot drops to 0.3090 and fine-tuned accuracy to 0.4267 (top3 0.6064, F1 0.1932), below sc80; on traffic fines it ties the established models (0.8837 vs 0.8835 for fm-v6). With the corpus held at v4, the 306M model is better than sc80 on transfer (+3.3pp zero-shot, +1.1pp FT).
• Interpretation: at ~4.2 tokens/parameter, scale adds only a small transfer gain once the mix is held fixed; the dominant lever is the real fraction. At 80M, rmix14 reaches 0.3969 zero-shot / 0.4695 FT, still the best single model. The sc300 folders are published as evidence, not as flagship models.

Data-mix ablation (corpus v5)

The v5 corpus has only 18 real logs among 2,500 synthetic shards. We varied the real fraction by repeating the real shards (rmix14 ×14 ≈ 9.5% of windows, rmix24 ×24 ≈ 15.7%), at equal architecture and token budget (cap 600/shard, 1 epoch):

model	mix	real windows	zs BPIC 2015-1	FT test acc	top3	F1
sc80 (v4 ref)	v4	5.6%	0.3810	0.4336	0.6280	0.1861
sc80 v5	default	0.7%	0.3363	0.4149	0.5918	0.1706
sc80 v5 rmix14	×14	9.5%	0.3969	0.4695	0.6456	0.2096
sc80 v5 rmix24	×24	15.7%	0.3654	0.4597	0.6423	0.1967
sc300 v5	default	0.7%	0.3090	0.4267	0.6064	0.1932
sc300 v5 rmix14	×14	9.5%	0.3969	0.4448	0.6075	0.1945
sc300 v4 (local)	v4	5.6%	0.3929	0.4450	0.6206	0.1873

• The real fraction is the dominant lever: rmix14 recovers +3.6pp FT and +1.6pp zero-shot over sc80-v4; more real is not monotone (rmix24 is lower).
• At 300M the mix recovers +8.8pp zero-shot / +1.8pp FT over the default v5, but on rmix14 the 300M still does not beat the 80M (FT 0.4448 vs 0.4695). At fixed v4 mix, 300M beats 80M by +3.3pp zero-shot / +1.1pp FT: the scale effect is real but smaller than the mix effect.
• At fixed v4 mix, 80M→300M adds +3.3pp zero-shot / +1.1pp FT (validation flat); the mix effect at fixed scale is larger, so the mix is the primary lever at this budget.
• The best single model is scaling/ablation/sc80-v5-rmix14 (0.4695 FT); its late fusion with process-graph-fm/gfm-sc80 reaches test 0.4832.

Real-only cap4000 ablation (vocab v7)

• Data: real logs only, 18 vs 31 logs (31 = 18 + 13 new logs), global vocabulary v7 (333 log ids, 934/3,773 activity/resource rows).
• Budget: 89.1M params (d768/12L), cap 4,000 windows/log, ~0.38B positions: 18 logs × 80 epochs × 145 steps; 31 logs × 55 epochs × 210 steps.
• Evaluation: same BPIC 2015-1 zero-shot and fine-tuning protocol as the scaling suite; single seed per run.

run	real logs	epochs × steps	zero-shot	FT test acc	FT top3	FT macro-F1
real-only 18 cap4000	18	80 × 145	0.3703	0.4418	0.5960	0.1909
real-only 31 cap4000	31	55 × 210	0.3010	0.4457	0.6036	0.1892
sc80 v4 (ref)	18	18 × 639	0.3810	0.4336	0.6280	0.1861
real-only 18 no-cap (ref)	18	41 ep	0.3683	0.4377	0.6032	0.1764
sc80 v5 rmix14 (ref)	18 ×14 + synth	1 ep	0.3969	0.4695	0.6456	0.2096

• The 13 new logs give +0.4pp fine-tuned (0.4418 → 0.4457) but −6.9pp zero-shot (0.3703 → 0.3010). The larger vocabulary/log mixture dilutes the lexical overlap that drives zero-shot transfer to BPIC 2015-1.
• Real-only scaling is not the dominant lever at this budget; the best single model remains sc80-v5-rmix14 (0.3969 / 0.4695). The next experiment should control the vocabulary and mix the new logs with the real anchor corpus.
• Validation for these two rows is on the respective real-only held-out cases and is not comparable with the v4/v5 points above.
• Raw logs are not redistributed; per-log provenance and licenses are in scaling/ablation/real-only-cap/SOURCES.md.

Caveats

• Single seed per point; no error bars across seeds at these budgets.
• The real-vs-synthetic mix is ablated at 80M (def/rmix14/rmix24) and at 300M (def/rmix14). The mix is the dominant lever; at equal rmix14 mix, 300M does not beat 80M (FT 0.4448 vs 0.4695), while at fixed v4 mix 300M gains +1.1pp FT over 80M.
• sc300 (v5) is a reduced local run (1.28B positions, not the 6B full-budget point), chosen to stay within ~3 days on one GB10; it changes two variables at once (scale and mix). The sc300-v4 control fixes the mix and isolates scale: it shows a small transfer gain over sc80-v4, while the mix ablation shows a larger gain.
• sc300-v4 is a single-seed local ablation whose checkpoint is not released; its numbers come from the same pipeline and metrics as the other points.
• Numbers are test-set accuracy after validation-only model selection; the first-order argmax baseline and majority baseline are reported in the study repo.