clef-oQ4e / README.md
ultemica's picture
Add Flash benchmark comparison
ae04637 verified
|
Raw History Blame Contribute Delete
4.17 kB
metadata
license: apache-2.0

Decision Index benchmark (sampled, adapted)

The following is a local, sample-only proxy, not an official Decision Index score. Clef-oQ4e and Clef-Flash-oQ4e were tested on the same selected Decision Index 0.3 sample, with up to 100 requests per catalog, through oMLX /v1/decisions. The Flash results are also published on the Clef-Flash-oQ4e model card. Structured state, instructions and choice descriptions were rendered as compact JSON text when needed; true/false criteria were appended to instructions and could not be represented exactly. The endpoint may silently truncate state.

The reference columns are Cloudflare's published Clef-Flash and Clef results for the full Decision Index 0.2.1 suite, run internally. The versions, cases, request representation and execution environments differ; these numbers cannot isolate quantization effects or establish a head-to-head ranking.

Each local run covered 43 catalogs and 4,198 requests: 4,163 ok and 35 unsupported (34 BRIGHT, 1 ToolRet). Unsupported cases are excluded from per-benchmark scores; BRIGHT scores cover 66 of 100 requests and ToolRet 99 of 100. Home appliance simulator has 88 requests and SimpleBench 10; all other catalogs have 100. Values are percentages rounded to one decimal. iSarcasmEval is a supplementary local case-exact accuracy, not its official multi-subtask aggregate. A dash means no corresponding row in the published reference table. ForecastBench appears in the reference but was not included locally.

Benchmark (metric) Clef-oQ4e, 0.3 sample Clef-Flash-oQ4e, 0.3 sample Clef-Flash, 0.2.1 full Clef, 0.2.1 full
BFCL (case exact accuracy) 100.0 99.0 98.8 98.5
ToolRet (nDCG@10) 72.1 69.2 66.4 69.2
API-Bank (accuracy) 92.0 93.0 93.1 91.9
BANKING77 (macro-F1) 94.3 87.7 90.9 94.2
CLINC150+OOS (macro-F1) 97.2 62.3 66.8 97.4
RouterBench (selected quality) 79.5 80.0 79.9 79.7
Home appliance simulator (case exact accuracy) 81.8 95.5 97.7 83.0
SGD/SGD-X (macro-F1) 37.9 28.5 34.2 43.8
ContractNLI (macro-F1) 80.0 83.9 84.3 81.4
ANLI (macro-F1) 59.2 55.3 59.1 69.8
BPoMP (accuracy) 97.0 96.0 95.4 96.9
Humicroedit (accuracy) 71.0 75.0 75.1 66.7
POP909-CL (accuracy) 16.0 2.0 1.6 15.8
cfcolor (accuracy) 60.0 53.0 65.8 66.0
MMLU (accuracy) 92.0 92.0 91.8 90.3
GPQA Diamond (accuracy) 44.0 44.0 51.0 48.0
ARC-Easy (accuracy) 99.0 100.0 99.5 99.0
ARC-Challenge (accuracy) 98.0 99.0 98.3 97.7
WinoGrande (accuracy) 92.0 97.0 97.5 93.5
HellaSwag (accuracy) 96.0 96.0 98.6 98.2
GSM8K (accuracy) 77.0 57.0 67.3 80.8
ChessBench (accuracy) 22.0 18.0 23.0 24.7
MuSR (accuracy) 83.0 87.0 86.0 83.5
SATA-Bench (case exact accuracy) 31.0 36.0 36.7 33.8
SimpleBench (accuracy; 10 local requests) 30.0 50.0 — —
BRIGHT (nDCG@10; 66 scored of 100) 45.8 41.1 39.3 45.9
Amazon ESCI (macro-F1) 50.8 49.8 57.4 57.5
ACOS (per-review F1) 36.5 25.5 25.9 33.3
FinEntity (macro-F1) 95.7 97.6 97.1 96.2
iSarcasmEval (case exact accuracy, supplementary) 89.0 89.0 — —
VAST (macro-F1) 65.9 51.4 49.6 59.5
NLI4CT (macro-F1) 80.8 73.0 78.6 82.9
CRUXEval (accuracy) 86.0 87.0 86.1 86.7
CLadder (accuracy) 94.0 97.0 97.7 94.0
HLE (accuracy) 13.0 10.0 — —
Habermas Machine (accuracy) 62.0 74.0 71.8 68.7
PhishNChips (accuracy) 79.0 76.0 75.0 79.6
MMLU-Pro (accuracy) 66.0 68.0 65.3 65.9
BBH (accuracy) 71.0 70.0 68.9 73.7
RAGTruth (hallucination F1) 72.2 25.6 35.6 79.4
HoVer (accuracy) 65.0 66.0 61.2 65.2
When2Call MCQ (accuracy) 69.0 65.0 65.6 72.4
New Yorker (accuracy) 75.0 64.0 66.1 69.5