ultemica commited on
Commit
ae04637
Β·
verified Β·
1 Parent(s): 61c4d7a

Add Flash benchmark comparison

Browse files
Files changed (1) hide show
  1. README.md +47 -47
README.md CHANGED
@@ -4,55 +4,55 @@ license: apache-2.0
4
 
5
  ## Decision Index benchmark (sampled, adapted)
6
 
7
- The following is a local, **sample-only proxy**, not an official Decision Index score. Clef-oQ4e was tested against Decision Index **0.3**, with up to 100 selected requests per catalog, through oMLX `/v1/decisions`. Structured state, instructions and choice descriptions were rendered as compact JSON text when needed; true/false criteria were appended to instructions and could not be represented exactly. The endpoint may silently truncate state.
8
 
9
  The reference columns are Cloudflare's published [Clef-Flash and Clef results](https://huggingface.co/Cloudflare/clef-flash#results) for the full Decision Index **0.2.1** suite, run internally. **The versions, cases, request representation and execution environments differ; these numbers cannot isolate quantization effects or establish a head-to-head ranking.**
10
 
11
- The local run covered 43 catalogs and 4,198 requests: 4,163 `ok` and 35 `unsupported` (34 BRIGHT, 1 ToolRet). Unsupported cases are excluded from per-benchmark scores; BRIGHT scores cover 66 of 100 requests and ToolRet 99 of 100. Home appliance simulator has 88 requests and SimpleBench 10; all other catalogs have 100. Values are percentages rounded to one decimal. iSarcasmEval is a supplementary local case-exact accuracy, not its official multi-subtask aggregate. A dash means no corresponding row in the published reference table. ForecastBench appears in the reference but was not included locally.
12
 
13
- | Benchmark (metric) | Clef-oQ4e, 0.3 sample | Clef-Flash, 0.2.1 full | Clef, 0.2.1 full |
14
- |---|---:|---:|---:|
15
- | BFCL (case exact accuracy) | 100.0 | 98.8 | 98.5 |
16
- | ToolRet (nDCG@10) | 72.1 | 66.4 | 69.2 |
17
- | API-Bank (accuracy) | 92.0 | 93.1 | 91.9 |
18
- | BANKING77 (macro-F1) | 94.3 | 90.9 | 94.2 |
19
- | CLINC150+OOS (macro-F1) | 97.2 | 66.8 | 97.4 |
20
- | RouterBench (selected quality) | 79.5 | 79.9 | 79.7 |
21
- | Home appliance simulator (case exact accuracy) | 81.8 | 97.7 | 83.0 |
22
- | SGD/SGD-X (macro-F1) | 37.9 | 34.2 | 43.8 |
23
- | ContractNLI (macro-F1) | 80.0 | 84.3 | 81.4 |
24
- | ANLI (macro-F1) | 59.2 | 59.1 | 69.8 |
25
- | BPoMP (accuracy) | 97.0 | 95.4 | 96.9 |
26
- | Humicroedit (accuracy) | 71.0 | 75.1 | 66.7 |
27
- | POP909-CL (accuracy) | 16.0 | 1.6 | 15.8 |
28
- | cfcolor (accuracy) | 60.0 | 65.8 | 66.0 |
29
- | MMLU (accuracy) | 92.0 | 91.8 | 90.3 |
30
- | GPQA Diamond (accuracy) | 44.0 | 51.0 | 48.0 |
31
- | ARC-Easy (accuracy) | 99.0 | 99.5 | 99.0 |
32
- | ARC-Challenge (accuracy) | 98.0 | 98.3 | 97.7 |
33
- | WinoGrande (accuracy) | 92.0 | 97.5 | 93.5 |
34
- | HellaSwag (accuracy) | 96.0 | 98.6 | 98.2 |
35
- | GSM8K (accuracy) | 77.0 | 67.3 | 80.8 |
36
- | ChessBench (accuracy) | 22.0 | 23.0 | 24.7 |
37
- | MuSR (accuracy) | 83.0 | 86.0 | 83.5 |
38
- | SATA-Bench (case exact accuracy) | 31.0 | 36.7 | 33.8 |
39
- | SimpleBench (accuracy; 10 local requests) | 30.0 | β€” | β€” |
40
- | BRIGHT (nDCG@10; 66 scored of 100) | 45.8 | 39.3 | 45.9 |
41
- | Amazon ESCI (macro-F1) | 50.8 | 57.4 | 57.5 |
42
- | ACOS (per-review F1) | 36.5 | 25.9 | 33.3 |
43
- | FinEntity (macro-F1) | 95.7 | 97.1 | 96.2 |
44
- | iSarcasmEval (case exact accuracy, supplementary) | 89.0 | β€” | β€” |
45
- | VAST (macro-F1) | 65.9 | 49.6 | 59.5 |
46
- | NLI4CT (macro-F1) | 80.8 | 78.6 | 82.9 |
47
- | CRUXEval (accuracy) | 86.0 | 86.1 | 86.7 |
48
- | CLadder (accuracy) | 94.0 | 97.7 | 94.0 |
49
- | HLE (accuracy) | 13.0 | β€” | β€” |
50
- | Habermas Machine (accuracy) | 62.0 | 71.8 | 68.7 |
51
- | PhishNChips (accuracy) | 79.0 | 75.0 | 79.6 |
52
- | MMLU-Pro (accuracy) | 66.0 | 65.3 | 65.9 |
53
- | BBH (accuracy) | 71.0 | 68.9 | 73.7 |
54
- | RAGTruth (hallucination F1) | 72.2 | 35.6 | 79.4 |
55
- | HoVer (accuracy) | 65.0 | 61.2 | 65.2 |
56
- | When2Call MCQ (accuracy) | 69.0 | 65.6 | 72.4 |
57
- | New Yorker (accuracy) | 75.0 | 66.1 | 69.5 |
58
 
 
4
 
5
  ## Decision Index benchmark (sampled, adapted)
6
 
7
+ The following is a local, **sample-only proxy**, not an official Decision Index score. Clef-oQ4e and Clef-Flash-oQ4e were tested on the same selected Decision Index **0.3** sample, with up to 100 requests per catalog, through oMLX `/v1/decisions`. The Flash results are also published on the [Clef-Flash-oQ4e model card](https://huggingface.co/qleap/clef-flash-oQ4e). Structured state, instructions and choice descriptions were rendered as compact JSON text when needed; true/false criteria were appended to instructions and could not be represented exactly. The endpoint may silently truncate state.
8
 
9
  The reference columns are Cloudflare's published [Clef-Flash and Clef results](https://huggingface.co/Cloudflare/clef-flash#results) for the full Decision Index **0.2.1** suite, run internally. **The versions, cases, request representation and execution environments differ; these numbers cannot isolate quantization effects or establish a head-to-head ranking.**
10
 
11
+ Each local run covered 43 catalogs and 4,198 requests: 4,163 `ok` and 35 `unsupported` (34 BRIGHT, 1 ToolRet). Unsupported cases are excluded from per-benchmark scores; BRIGHT scores cover 66 of 100 requests and ToolRet 99 of 100. Home appliance simulator has 88 requests and SimpleBench 10; all other catalogs have 100. Values are percentages rounded to one decimal. iSarcasmEval is a supplementary local case-exact accuracy, not its official multi-subtask aggregate. A dash means no corresponding row in the published reference table. ForecastBench appears in the reference but was not included locally.
12
 
13
+ | Benchmark (metric) | Clef-oQ4e, 0.3 sample | Clef-Flash-oQ4e, 0.3 sample | Clef-Flash, 0.2.1 full | Clef, 0.2.1 full |
14
+ |---|---:|---:|---:|---:|
15
+ | BFCL (case exact accuracy) | 100.0 | 99.0 | 98.8 | 98.5 |
16
+ | ToolRet (nDCG@10) | 72.1 | 69.2 | 66.4 | 69.2 |
17
+ | API-Bank (accuracy) | 92.0 | 93.0 | 93.1 | 91.9 |
18
+ | BANKING77 (macro-F1) | 94.3 | 87.7 | 90.9 | 94.2 |
19
+ | CLINC150+OOS (macro-F1) | 97.2 | 62.3 | 66.8 | 97.4 |
20
+ | RouterBench (selected quality) | 79.5 | 80.0 | 79.9 | 79.7 |
21
+ | Home appliance simulator (case exact accuracy) | 81.8 | 95.5 | 97.7 | 83.0 |
22
+ | SGD/SGD-X (macro-F1) | 37.9 | 28.5 | 34.2 | 43.8 |
23
+ | ContractNLI (macro-F1) | 80.0 | 83.9 | 84.3 | 81.4 |
24
+ | ANLI (macro-F1) | 59.2 | 55.3 | 59.1 | 69.8 |
25
+ | BPoMP (accuracy) | 97.0 | 96.0 | 95.4 | 96.9 |
26
+ | Humicroedit (accuracy) | 71.0 | 75.0 | 75.1 | 66.7 |
27
+ | POP909-CL (accuracy) | 16.0 | 2.0 | 1.6 | 15.8 |
28
+ | cfcolor (accuracy) | 60.0 | 53.0 | 65.8 | 66.0 |
29
+ | MMLU (accuracy) | 92.0 | 92.0 | 91.8 | 90.3 |
30
+ | GPQA Diamond (accuracy) | 44.0 | 44.0 | 51.0 | 48.0 |
31
+ | ARC-Easy (accuracy) | 99.0 | 100.0 | 99.5 | 99.0 |
32
+ | ARC-Challenge (accuracy) | 98.0 | 99.0 | 98.3 | 97.7 |
33
+ | WinoGrande (accuracy) | 92.0 | 97.0 | 97.5 | 93.5 |
34
+ | HellaSwag (accuracy) | 96.0 | 96.0 | 98.6 | 98.2 |
35
+ | GSM8K (accuracy) | 77.0 | 57.0 | 67.3 | 80.8 |
36
+ | ChessBench (accuracy) | 22.0 | 18.0 | 23.0 | 24.7 |
37
+ | MuSR (accuracy) | 83.0 | 87.0 | 86.0 | 83.5 |
38
+ | SATA-Bench (case exact accuracy) | 31.0 | 36.0 | 36.7 | 33.8 |
39
+ | SimpleBench (accuracy; 10 local requests) | 30.0 | 50.0 | β€” | β€” |
40
+ | BRIGHT (nDCG@10; 66 scored of 100) | 45.8 | 41.1 | 39.3 | 45.9 |
41
+ | Amazon ESCI (macro-F1) | 50.8 | 49.8 | 57.4 | 57.5 |
42
+ | ACOS (per-review F1) | 36.5 | 25.5 | 25.9 | 33.3 |
43
+ | FinEntity (macro-F1) | 95.7 | 97.6 | 97.1 | 96.2 |
44
+ | iSarcasmEval (case exact accuracy, supplementary) | 89.0 | 89.0 | β€” | β€” |
45
+ | VAST (macro-F1) | 65.9 | 51.4 | 49.6 | 59.5 |
46
+ | NLI4CT (macro-F1) | 80.8 | 73.0 | 78.6 | 82.9 |
47
+ | CRUXEval (accuracy) | 86.0 | 87.0 | 86.1 | 86.7 |
48
+ | CLadder (accuracy) | 94.0 | 97.0 | 97.7 | 94.0 |
49
+ | HLE (accuracy) | 13.0 | 10.0 | β€” | β€” |
50
+ | Habermas Machine (accuracy) | 62.0 | 74.0 | 71.8 | 68.7 |
51
+ | PhishNChips (accuracy) | 79.0 | 76.0 | 75.0 | 79.6 |
52
+ | MMLU-Pro (accuracy) | 66.0 | 68.0 | 65.3 | 65.9 |
53
+ | BBH (accuracy) | 71.0 | 70.0 | 68.9 | 73.7 |
54
+ | RAGTruth (hallucination F1) | 72.2 | 25.6 | 35.6 | 79.4 |
55
+ | HoVer (accuracy) | 65.0 | 66.0 | 61.2 | 65.2 |
56
+ | When2Call MCQ (accuracy) | 69.0 | 65.0 | 65.6 | 72.4 |
57
+ | New Yorker (accuracy) | 75.0 | 64.0 | 66.1 | 69.5 |
58