clef-flash-oQ4e / README.md
ultemica's picture
Remove benchmark latency results
9dbb412 verified
|
Raw History Blame Contribute Delete
3.69 kB
metadata
license: apache-2.0

Decision Index benchmark (sampled, adapted)

The following is a local, sample-only proxy, not an official Decision Index score. Clef-Flash-oQ4e was tested against Decision Index 0.3, with up to 100 selected requests per catalog, through oMLX /v1/decisions. Structured state, instructions and choice descriptions were rendered as compact JSON text when needed; true/false criteria were appended to instructions and could not be represented exactly. The endpoint may silently truncate state.

The reference columns are Cloudflare's published Clef-Flash and Clef results for the full Decision Index 0.2.1 suite, run internally. The versions, cases, request representation and execution environments differ; these numbers cannot isolate quantization effects or establish a head-to-head ranking.

The local run covered 43 catalogs and 4,198 requests: 4,163 ok and 35 unsupported (34 BRIGHT, 1 ToolRet). Unsupported cases are excluded from per-benchmark scores; BRIGHT scores cover 66 of 100 requests and ToolRet 99 of 100. Home appliance simulator has 88 requests and SimpleBench 10; all other catalogs have 100. Values are percentages rounded to one decimal. iSarcasmEval is a supplementary local case-exact accuracy, not its official multi-subtask aggregate. A dash means no corresponding row in the published reference table. ForecastBench appears in the reference but was not included locally.

Benchmark (metric) Clef-Flash-oQ4e, 0.3 sample Clef-Flash, 0.2.1 full Clef, 0.2.1 full
BFCL (case exact accuracy) 99.0 98.8 98.5
ToolRet (nDCG@10) 69.2 66.4 69.2
API-Bank (accuracy) 93.0 93.1 91.9
BANKING77 (macro-F1) 87.7 90.9 94.2
CLINC150+OOS (macro-F1) 62.3 66.8 97.4
RouterBench (selected quality) 80.0 79.9 79.7
Home appliance simulator (case exact accuracy) 95.5 97.7 83.0
SGD/SGD-X (macro-F1) 28.5 34.2 43.8
ContractNLI (macro-F1) 83.9 84.3 81.4
ANLI (macro-F1) 55.3 59.1 69.8
BPoMP (accuracy) 96.0 95.4 96.9
Humicroedit (accuracy) 75.0 75.1 66.7
POP909-CL (accuracy) 2.0 1.6 15.8
cfcolor (accuracy) 53.0 65.8 66.0
MMLU (accuracy) 92.0 91.8 90.3
GPQA Diamond (accuracy) 44.0 51.0 48.0
ARC-Easy (accuracy) 100.0 99.5 99.0
ARC-Challenge (accuracy) 99.0 98.3 97.7
WinoGrande (accuracy) 97.0 97.5 93.5
HellaSwag (accuracy) 96.0 98.6 98.2
GSM8K (accuracy) 57.0 67.3 80.8
ChessBench (accuracy) 18.0 23.0 24.7
MuSR (accuracy) 87.0 86.0 83.5
SATA-Bench (case exact accuracy) 36.0 36.7 33.8
SimpleBench (accuracy; 10 local requests) 50.0 — —
BRIGHT (nDCG@10; 66 scored of 100) 41.1 39.3 45.9
Amazon ESCI (macro-F1) 49.8 57.4 57.5
ACOS (per-review F1) 25.5 25.9 33.3
FinEntity (macro-F1) 97.6 97.1 96.2
iSarcasmEval (case exact accuracy, supplementary) 89.0 — —
VAST (macro-F1) 51.4 49.6 59.5
NLI4CT (macro-F1) 73.0 78.6 82.9
CRUXEval (accuracy) 87.0 86.1 86.7
CLadder (accuracy) 97.0 97.7 94.0
HLE (accuracy) 10.0 — —
Habermas Machine (accuracy) 74.0 71.8 68.7
PhishNChips (accuracy) 76.0 75.0 79.6
MMLU-Pro (accuracy) 68.0 65.3 65.9
BBH (accuracy) 70.0 68.9 73.7
RAGTruth (hallucination F1) 25.6 35.6 79.4
HoVer (accuracy) 66.0 61.2 65.2
When2Call MCQ (accuracy) 65.0 65.6 72.4
New Yorker (accuracy) 64.0 66.1 69.5