update benchmark table: add decision-model-large (DI 58.52)
Browse files
README.md
CHANGED
|
@@ -15,8 +15,10 @@ library_name: custom
|
|
| 15 |
|
| 16 |
| model | Decision Index | balanced_raw | head-to-head (43 benchmarks) |
|
| 17 |
|---|---|---|---|
|
| 18 |
-
|
|
| 19 |
-
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
Wins vs the 321B reference on identical cases: **When2Call** MCQ 0.785 vs 0.559 (+22.6pp), **SimpleBench** 0.500 vs 0.200 (+30pp), **CLINC150+OOS** macro-F1 0.878 vs 0.802 (+7.6pp), **GSM8K** 0.844 vs 0.836, **GPQA Diamond** 0.561 vs 0.520, **POP909-CL** 0.347 vs 0.196 (+15.1pp), **VAST** macro-F1 0.642 vs 0.529 (+11.3pp), **ForecastBench** Brier 0.162 vs 0.306 (lower is better). Full 155,390-row Decision Index (edition 0.2.1), 5 errors (0.003%).
|
| 22 |
|
|
|
|
| 15 |
|
| 16 |
| model | Decision Index | balanced_raw | head-to-head (43 benchmarks) |
|
| 17 |
|---|---|---|---|
|
| 18 |
+
| [decision-model-tiny](https://huggingface.co/someguystrainingmodels/decision-model-tiny) (0.8B) | 22.60 | 41.76 | 3 W / 1 T / 39 L |
|
| 19 |
+
| [decision-model-small](https://huggingface.co/someguystrainingmodels/decision-model-small) (4B) | 40.66 | 54.85 | 5 W / 2 T / 36 L |
|
| 20 |
+
| **decision-model-large** (176B-A6B, this repo) | 58.52 | 68.32 | 25 W / 1 T / 17 L |
|
| 21 |
+
| [decision-model-extra-large](https://huggingface.co/someguystrainingmodels/decision-model-extra-large) (321B, LoRA) | 57.79 | 67.83 | 2 W / 0 T / 36 L |
|
| 22 |
|
| 23 |
Wins vs the 321B reference on identical cases: **When2Call** MCQ 0.785 vs 0.559 (+22.6pp), **SimpleBench** 0.500 vs 0.200 (+30pp), **CLINC150+OOS** macro-F1 0.878 vs 0.802 (+7.6pp), **GSM8K** 0.844 vs 0.836, **GPQA Diamond** 0.561 vs 0.520, **POP909-CL** 0.347 vs 0.196 (+15.1pp), **VAST** macro-F1 0.642 vs 0.529 (+11.3pp), **ForecastBench** Brier 0.162 vs 0.306 (lower is better). Full 155,390-row Decision Index (edition 0.2.1), 5 errors (0.003%).
|
| 24 |
|