sc300: negative result (data-mix bottleneck)
Browse files
README.md
CHANGED
|
@@ -28,12 +28,12 @@ This repository contains the **fm-v6** foundation checkpoint and the **scaling s
|
|
| 28 |
| `scaling/sc5/` | 6.32M | 320 / 4 | v4: 18 real + 300 synthetic (308 logs) | 0.04B | 0.6026 | 0.2977 | 0.3931 | 0.5642 | 0.163 |
|
| 29 |
| `scaling/sc27/` | 27.36M | 512 / 8 | v4 | 0.13B | 0.6451 | 0.3068 | 0.4296 | 0.5865 | 0.1877 |
|
| 30 |
| `scaling/sc80/` | 88.19M | 768 / 12 | v4 | 0.38B | **0.6563** | **0.3600** | 0.4336 | 0.6280 | 0.1861 |
|
| 31 |
-
| `scaling/sc300/` | 306.4M | 1024 / 24 | v5 (1 epoch, reduced budget) | ~1.28B |
|
| 32 |
|
| 33 |
Notes:
|
| 34 |
- `fm-v6` and the `sc*` models use **different corpora and vocabularies** (58 vs 308 seen logs) and are not directly comparable. The scaling curve is `sc5 β sc27 β sc80 β sc300`.
|
| 35 |
- The `sc*` points are **under-budget** by design (0.22-0.42Γ the 20-token/parameter target; sc300 uses 4.2 tokens/parameter), so the curve measures the under-trained regime, not Chinchilla-optimal performance. See `SCALING.md`.
|
| 36 |
-
- `sc300` is
|
| 37 |
|
| 38 |
## Quickstart
|
| 39 |
|
|
@@ -66,7 +66,7 @@ The models use a **global vocabulary**: activities are ids 4..776, resources 2..
|
|
| 66 |
|
| 67 |
## Scaling
|
| 68 |
|
| 69 |
-
`SCALING.md` contains the protocol, token budgets, caveats and the full table. Headline: validation next-activity accuracy rises 0.6026 β 0.6451 β 0.6563 across 6.3M β 27.4M β 88.2M under the same under-budget recipe, while transfer improves mainly in the larger models (zero-shot 0.2977 β 0.3068 β 0.3600).
|
| 70 |
|
| 71 |
## MoE ablation (archived)
|
| 72 |
|
|
|
|
| 28 |
| `scaling/sc5/` | 6.32M | 320 / 4 | v4: 18 real + 300 synthetic (308 logs) | 0.04B | 0.6026 | 0.2977 | 0.3931 | 0.5642 | 0.163 |
|
| 29 |
| `scaling/sc27/` | 27.36M | 512 / 8 | v4 | 0.13B | 0.6451 | 0.3068 | 0.4296 | 0.5865 | 0.1877 |
|
| 30 |
| `scaling/sc80/` | 88.19M | 768 / 12 | v4 | 0.38B | **0.6563** | **0.3600** | 0.4336 | 0.6280 | 0.1861 |
|
| 31 |
+
| `scaling/sc300/` | 306.4M | 1024 / 24 | v5 (1 epoch, reduced budget) | ~1.28B | 0.6080* | 0.3090 | 0.4267 | 0.6064 | 0.1932 |
|
| 32 |
|
| 33 |
Notes:
|
| 34 |
- `fm-v6` and the `sc*` models use **different corpora and vocabularies** (58 vs 308 seen logs) and are not directly comparable. The scaling curve is `sc5 β sc27 β sc80 β sc300`.
|
| 35 |
- The `sc*` points are **under-budget** by design (0.22-0.42Γ the 20-token/parameter target; sc300 uses 4.2 tokens/parameter), so the curve measures the under-trained regime, not Chinchilla-optimal performance. See `SCALING.md`.
|
| 36 |
+
- `sc300` is a **negative result**: trained on corpus v5 (2,500 synthetic shards + 18 real logs), it did not improve transfer over sc80. Its validation accuracy (0.6080) is measured on a different, synthetic-heavy validation set and is **not comparable** with sc5/sc27/sc80 (v4). Zero-shot on BPIC 2015-1 is 0.3090 (0.2869 without unseen-to-UNK) and fine-tuned test accuracy 0.4267, both below sc80. This points to the data mix, not model scale, as the bottleneck. See `SCALING.md`.
|
| 37 |
|
| 38 |
## Quickstart
|
| 39 |
|
|
|
|
| 66 |
|
| 67 |
## Scaling
|
| 68 |
|
| 69 |
+
`SCALING.md` contains the protocol, token budgets, caveats and the full table. Headline: validation next-activity accuracy rises 0.6026 β 0.6451 β 0.6563 across 6.3M β 27.4M β 88.2M under the same under-budget v4 recipe, while transfer improves mainly in the larger models (zero-shot 0.2977 β 0.3068 β 0.3600). The 306M point (trained on corpus v5) does **not** continue the trend: it is a negative result for scale, suggesting the data mix is the bottleneck.
|
| 70 |
|
| 71 |
## MoE ablation (archived)
|
| 72 |
|