--- license: apache-2.0 --- ## Decision Index benchmark (sampled, adapted) The following is a local, **sample-only proxy**, not an official Decision Index score. Clef-oQ4e and Clef-Flash-oQ4e were tested on the same selected Decision Index **0.3** sample, with up to 100 requests per catalog, through oMLX `/v1/decisions`. The Flash results are also published on the [Clef-Flash-oQ4e model card](https://huggingface.co/qleap/clef-flash-oQ4e). Structured state, instructions and choice descriptions were rendered as compact JSON text when needed; true/false criteria were appended to instructions and could not be represented exactly. The endpoint may silently truncate state. The reference columns are Cloudflare's published [Clef-Flash and Clef results](https://huggingface.co/Cloudflare/clef-flash#results) for the full Decision Index **0.2.1** suite, run internally. **The versions, cases, request representation and execution environments differ; these numbers cannot isolate quantization effects or establish a head-to-head ranking.** Each local run covered 43 catalogs and 4,198 requests: 4,163 `ok` and 35 `unsupported` (34 BRIGHT, 1 ToolRet). Unsupported cases are excluded from per-benchmark scores; BRIGHT scores cover 66 of 100 requests and ToolRet 99 of 100. Home appliance simulator has 88 requests and SimpleBench 10; all other catalogs have 100. Values are percentages rounded to one decimal. iSarcasmEval is a supplementary local case-exact accuracy, not its official multi-subtask aggregate. A dash means no corresponding row in the published reference table. ForecastBench appears in the reference but was not included locally. | Benchmark (metric) | Clef-oQ4e, 0.3 sample | Clef-Flash-oQ4e, 0.3 sample | Clef-Flash, 0.2.1 full | Clef, 0.2.1 full | |---|---:|---:|---:|---:| | BFCL (case exact accuracy) | 100.0 | 99.0 | 98.8 | 98.5 | | ToolRet (nDCG@10) | 72.1 | 69.2 | 66.4 | 69.2 | | API-Bank (accuracy) | 92.0 | 93.0 | 93.1 | 91.9 | | BANKING77 (macro-F1) | 94.3 | 87.7 | 90.9 | 94.2 | | CLINC150+OOS (macro-F1) | 97.2 | 62.3 | 66.8 | 97.4 | | RouterBench (selected quality) | 79.5 | 80.0 | 79.9 | 79.7 | | Home appliance simulator (case exact accuracy) | 81.8 | 95.5 | 97.7 | 83.0 | | SGD/SGD-X (macro-F1) | 37.9 | 28.5 | 34.2 | 43.8 | | ContractNLI (macro-F1) | 80.0 | 83.9 | 84.3 | 81.4 | | ANLI (macro-F1) | 59.2 | 55.3 | 59.1 | 69.8 | | BPoMP (accuracy) | 97.0 | 96.0 | 95.4 | 96.9 | | Humicroedit (accuracy) | 71.0 | 75.0 | 75.1 | 66.7 | | POP909-CL (accuracy) | 16.0 | 2.0 | 1.6 | 15.8 | | cfcolor (accuracy) | 60.0 | 53.0 | 65.8 | 66.0 | | MMLU (accuracy) | 92.0 | 92.0 | 91.8 | 90.3 | | GPQA Diamond (accuracy) | 44.0 | 44.0 | 51.0 | 48.0 | | ARC-Easy (accuracy) | 99.0 | 100.0 | 99.5 | 99.0 | | ARC-Challenge (accuracy) | 98.0 | 99.0 | 98.3 | 97.7 | | WinoGrande (accuracy) | 92.0 | 97.0 | 97.5 | 93.5 | | HellaSwag (accuracy) | 96.0 | 96.0 | 98.6 | 98.2 | | GSM8K (accuracy) | 77.0 | 57.0 | 67.3 | 80.8 | | ChessBench (accuracy) | 22.0 | 18.0 | 23.0 | 24.7 | | MuSR (accuracy) | 83.0 | 87.0 | 86.0 | 83.5 | | SATA-Bench (case exact accuracy) | 31.0 | 36.0 | 36.7 | 33.8 | | SimpleBench (accuracy; 10 local requests) | 30.0 | 50.0 | — | — | | BRIGHT (nDCG@10; 66 scored of 100) | 45.8 | 41.1 | 39.3 | 45.9 | | Amazon ESCI (macro-F1) | 50.8 | 49.8 | 57.4 | 57.5 | | ACOS (per-review F1) | 36.5 | 25.5 | 25.9 | 33.3 | | FinEntity (macro-F1) | 95.7 | 97.6 | 97.1 | 96.2 | | iSarcasmEval (case exact accuracy, supplementary) | 89.0 | 89.0 | — | — | | VAST (macro-F1) | 65.9 | 51.4 | 49.6 | 59.5 | | NLI4CT (macro-F1) | 80.8 | 73.0 | 78.6 | 82.9 | | CRUXEval (accuracy) | 86.0 | 87.0 | 86.1 | 86.7 | | CLadder (accuracy) | 94.0 | 97.0 | 97.7 | 94.0 | | HLE (accuracy) | 13.0 | 10.0 | — | — | | Habermas Machine (accuracy) | 62.0 | 74.0 | 71.8 | 68.7 | | PhishNChips (accuracy) | 79.0 | 76.0 | 75.0 | 79.6 | | MMLU-Pro (accuracy) | 66.0 | 68.0 | 65.3 | 65.9 | | BBH (accuracy) | 71.0 | 70.0 | 68.9 | 73.7 | | RAGTruth (hallucination F1) | 72.2 | 25.6 | 35.6 | 79.4 | | HoVer (accuracy) | 65.0 | 66.0 | 61.2 | 65.2 | | When2Call MCQ (accuracy) | 69.0 | 65.0 | 65.6 | 72.4 | | New Yorker (accuracy) | 75.0 | 64.0 | 66.1 | 69.5 |