Polifemus commited on
Commit
e428791
Β·
verified Β·
1 Parent(s): 918219f

sc300: negative result (data-mix bottleneck)

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -28,12 +28,12 @@ This repository contains the **fm-v6** foundation checkpoint and the **scaling s
28
  | `scaling/sc5/` | 6.32M | 320 / 4 | v4: 18 real + 300 synthetic (308 logs) | 0.04B | 0.6026 | 0.2977 | 0.3931 | 0.5642 | 0.163 |
29
  | `scaling/sc27/` | 27.36M | 512 / 8 | v4 | 0.13B | 0.6451 | 0.3068 | 0.4296 | 0.5865 | 0.1877 |
30
  | `scaling/sc80/` | 88.19M | 768 / 12 | v4 | 0.38B | **0.6563** | **0.3600** | 0.4336 | 0.6280 | 0.1861 |
31
- | `scaling/sc300/` | 306.4M | 1024 / 24 | v5 (1 epoch, reduced budget) | ~1.28B | _pending_ | _pending_ | _pending_ | _pending_ | _pending_ |
32
 
33
  Notes:
34
  - `fm-v6` and the `sc*` models use **different corpora and vocabularies** (58 vs 308 seen logs) and are not directly comparable. The scaling curve is `sc5 β†’ sc27 β†’ sc80 β†’ sc300`.
35
  - The `sc*` points are **under-budget** by design (0.22-0.42Γ— the 20-token/parameter target; sc300 uses 4.2 tokens/parameter), so the curve measures the under-trained regime, not Chinchilla-optimal performance. See `SCALING.md`.
36
- - `sc300` is being trained locally; the folder/report will be added when the run and its evaluation finish.
37
 
38
  ## Quickstart
39
 
@@ -66,7 +66,7 @@ The models use a **global vocabulary**: activities are ids 4..776, resources 2..
66
 
67
  ## Scaling
68
 
69
- `SCALING.md` contains the protocol, token budgets, caveats and the full table. Headline: validation next-activity accuracy rises 0.6026 β†’ 0.6451 β†’ 0.6563 across 6.3M β†’ 27.4M β†’ 88.2M under the same under-budget recipe, while transfer improves mainly in the larger models (zero-shot 0.2977 β†’ 0.3068 β†’ 0.3600).
70
 
71
  ## MoE ablation (archived)
72
 
 
28
  | `scaling/sc5/` | 6.32M | 320 / 4 | v4: 18 real + 300 synthetic (308 logs) | 0.04B | 0.6026 | 0.2977 | 0.3931 | 0.5642 | 0.163 |
29
  | `scaling/sc27/` | 27.36M | 512 / 8 | v4 | 0.13B | 0.6451 | 0.3068 | 0.4296 | 0.5865 | 0.1877 |
30
  | `scaling/sc80/` | 88.19M | 768 / 12 | v4 | 0.38B | **0.6563** | **0.3600** | 0.4336 | 0.6280 | 0.1861 |
31
+ | `scaling/sc300/` | 306.4M | 1024 / 24 | v5 (1 epoch, reduced budget) | ~1.28B | 0.6080* | 0.3090 | 0.4267 | 0.6064 | 0.1932 |
32
 
33
  Notes:
34
  - `fm-v6` and the `sc*` models use **different corpora and vocabularies** (58 vs 308 seen logs) and are not directly comparable. The scaling curve is `sc5 β†’ sc27 β†’ sc80 β†’ sc300`.
35
  - The `sc*` points are **under-budget** by design (0.22-0.42Γ— the 20-token/parameter target; sc300 uses 4.2 tokens/parameter), so the curve measures the under-trained regime, not Chinchilla-optimal performance. See `SCALING.md`.
36
+ - `sc300` is a **negative result**: trained on corpus v5 (2,500 synthetic shards + 18 real logs), it did not improve transfer over sc80. Its validation accuracy (0.6080) is measured on a different, synthetic-heavy validation set and is **not comparable** with sc5/sc27/sc80 (v4). Zero-shot on BPIC 2015-1 is 0.3090 (0.2869 without unseen-to-UNK) and fine-tuned test accuracy 0.4267, both below sc80. This points to the data mix, not model scale, as the bottleneck. See `SCALING.md`.
37
 
38
  ## Quickstart
39
 
 
66
 
67
  ## Scaling
68
 
69
+ `SCALING.md` contains the protocol, token budgets, caveats and the full table. Headline: validation next-activity accuracy rises 0.6026 β†’ 0.6451 β†’ 0.6563 across 6.3M β†’ 27.4M β†’ 88.2M under the same under-budget v4 recipe, while transfer improves mainly in the larger models (zero-shot 0.2977 β†’ 0.3068 β†’ 0.3600). The 306M point (trained on corpus v5) does **not** continue the trend: it is a negative result for scale, suggesting the data mix is the bottleneck.
70
 
71
  ## MoE ablation (archived)
72