Add real-only cap4000 ablation to SCALING
Browse files- SCALING.md +28 -0
SCALING.md
CHANGED
|
@@ -52,6 +52,34 @@ The v5 corpus has only 18 real logs among 2,500 synthetic shards. We varied the
|
|
| 52 |
- At fixed v4 mix, 80M→300M adds +3.3pp zero-shot / +1.1pp FT (validation flat); the mix effect at fixed scale is larger, so the mix is the primary lever at this budget.
|
| 53 |
- The best single model is `scaling/ablation/sc80-v5-rmix14` (0.4695 FT); its late fusion with `process-graph-fm/gfm-sc80` reaches test 0.4832.
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
## Caveats
|
| 56 |
|
| 57 |
- Single seed per point; no error bars across seeds at these budgets.
|
|
|
|
| 52 |
- At fixed v4 mix, 80M→300M adds +3.3pp zero-shot / +1.1pp FT (validation flat); the mix effect at fixed scale is larger, so the mix is the primary lever at this budget.
|
| 53 |
- The best single model is `scaling/ablation/sc80-v5-rmix14` (0.4695 FT); its late fusion with `process-graph-fm/gfm-sc80` reaches test 0.4832.
|
| 54 |
|
| 55 |
+
## Real-only cap4000 ablation (vocab v7)
|
| 56 |
+
|
| 57 |
+
- **Data**: real logs only, 18 vs 31 logs (31 = 18 + 13 new logs), global
|
| 58 |
+
vocabulary v7 (333 log ids, 934/3,773 activity/resource rows).
|
| 59 |
+
- **Budget**: 89.1M params (d768/12L), cap 4,000 windows/log, ~0.38B positions:
|
| 60 |
+
18 logs × 80 epochs × 145 steps; 31 logs × 55 epochs × 210 steps.
|
| 61 |
+
- **Evaluation**: same BPIC 2015-1 zero-shot and fine-tuning protocol as the
|
| 62 |
+
scaling suite; single seed per run.
|
| 63 |
+
|
| 64 |
+
| run | real logs | epochs × steps | zero-shot | FT test acc | FT top3 | FT macro-F1 |
|
| 65 |
+
|---|---|---:|---:|---:|---:|---:|
|
| 66 |
+
| real-only 18 cap4000 | 18 | 80 × 145 | 0.3703 | 0.4418 | 0.5960 | 0.1909 |
|
| 67 |
+
| real-only 31 cap4000 | 31 | 55 × 210 | 0.3010 | **0.4457** | **0.6036** | 0.1892 |
|
| 68 |
+
| sc80 v4 (ref) | 18 | 18 × 639 | **0.3810** | 0.4336 | 0.6280 | 0.1861 |
|
| 69 |
+
| real-only 18 no-cap (ref) | 18 | 41 ep | 0.3683 | 0.4377 | 0.6032 | 0.1764 |
|
| 70 |
+
| sc80 v5 rmix14 (ref) | 18 ×14 + synth | 1 ep | **0.3969** | **0.4695** | **0.6456** | **0.2096** |
|
| 71 |
+
|
| 72 |
+
- The 13 new logs give **+0.4pp fine-tuned** (0.4418 → 0.4457) but **−6.9pp
|
| 73 |
+
zero-shot** (0.3703 → 0.3010). The larger vocabulary/log mixture dilutes the
|
| 74 |
+
lexical overlap that drives zero-shot transfer to BPIC 2015-1.
|
| 75 |
+
- Real-only scaling is not the dominant lever at this budget; the best single
|
| 76 |
+
model remains sc80-v5-rmix14 (0.3969 / 0.4695). The next experiment should
|
| 77 |
+
control the vocabulary and mix the new logs with the real anchor corpus.
|
| 78 |
+
- Validation for these two rows is on the respective real-only held-out cases
|
| 79 |
+
and is **not comparable** with the v4/v5 points above.
|
| 80 |
+
- Raw logs are not redistributed; per-log provenance and licenses are in
|
| 81 |
+
`scaling/ablation/real-only-cap/SOURCES.md`.
|
| 82 |
+
|
| 83 |
## Caveats
|
| 84 |
|
| 85 |
- Single seed per point; no error bars across seeds at these budgets.
|