Download README.md from qleap/clef-flash-oQ4e: direct link, hf CLI and curl.
- Browser
- Download file 3.69 kB
-
https://huggingface.co/qleap/clef-flash-oQ4e/resolve/main/README.md
- Command line
-
hf download hf://qleap/clef-flash-oQ4e/README.md
-
curl -L -o README.md https://huggingface.co/qleap/clef-flash-oQ4e/resolve/main/README.md
license: apache-2.0
Decision Index benchmark (sampled, adapted)
The following is a local, sample-only proxy, not an official Decision Index score. Clef-Flash-oQ4e was tested against Decision Index 0.3, with up to 100 selected requests per catalog, through oMLX /v1/decisions. Structured state, instructions and choice descriptions were rendered as compact JSON text when needed; true/false criteria were appended to instructions and could not be represented exactly. The endpoint may silently truncate state.
The reference columns are Cloudflare's published Clef-Flash and Clef results for the full Decision Index 0.2.1 suite, run internally. The versions, cases, request representation and execution environments differ; these numbers cannot isolate quantization effects or establish a head-to-head ranking.
The local run covered 43 catalogs and 4,198 requests: 4,163 ok and 35 unsupported (34 BRIGHT, 1 ToolRet). Unsupported cases are excluded from per-benchmark scores; BRIGHT scores cover 66 of 100 requests and ToolRet 99 of 100. Home appliance simulator has 88 requests and SimpleBench 10; all other catalogs have 100. Values are percentages rounded to one decimal. iSarcasmEval is a supplementary local case-exact accuracy, not its official multi-subtask aggregate. A dash means no corresponding row in the published reference table. ForecastBench appears in the reference but was not included locally.
| Benchmark (metric) | Clef-Flash-oQ4e, 0.3 sample | Clef-Flash, 0.2.1 full | Clef, 0.2.1 full |
|---|---|---|---|
| BFCL (case exact accuracy) | 99.0 | 98.8 | 98.5 |
| ToolRet (nDCG@10) | 69.2 | 66.4 | 69.2 |
| API-Bank (accuracy) | 93.0 | 93.1 | 91.9 |
| BANKING77 (macro-F1) | 87.7 | 90.9 | 94.2 |
| CLINC150+OOS (macro-F1) | 62.3 | 66.8 | 97.4 |
| RouterBench (selected quality) | 80.0 | 79.9 | 79.7 |
| Home appliance simulator (case exact accuracy) | 95.5 | 97.7 | 83.0 |
| SGD/SGD-X (macro-F1) | 28.5 | 34.2 | 43.8 |
| ContractNLI (macro-F1) | 83.9 | 84.3 | 81.4 |
| ANLI (macro-F1) | 55.3 | 59.1 | 69.8 |
| BPoMP (accuracy) | 96.0 | 95.4 | 96.9 |
| Humicroedit (accuracy) | 75.0 | 75.1 | 66.7 |
| POP909-CL (accuracy) | 2.0 | 1.6 | 15.8 |
| cfcolor (accuracy) | 53.0 | 65.8 | 66.0 |
| MMLU (accuracy) | 92.0 | 91.8 | 90.3 |
| GPQA Diamond (accuracy) | 44.0 | 51.0 | 48.0 |
| ARC-Easy (accuracy) | 100.0 | 99.5 | 99.0 |
| ARC-Challenge (accuracy) | 99.0 | 98.3 | 97.7 |
| WinoGrande (accuracy) | 97.0 | 97.5 | 93.5 |
| HellaSwag (accuracy) | 96.0 | 98.6 | 98.2 |
| GSM8K (accuracy) | 57.0 | 67.3 | 80.8 |
| ChessBench (accuracy) | 18.0 | 23.0 | 24.7 |
| MuSR (accuracy) | 87.0 | 86.0 | 83.5 |
| SATA-Bench (case exact accuracy) | 36.0 | 36.7 | 33.8 |
| SimpleBench (accuracy; 10 local requests) | 50.0 | — | — |
| BRIGHT (nDCG@10; 66 scored of 100) | 41.1 | 39.3 | 45.9 |
| Amazon ESCI (macro-F1) | 49.8 | 57.4 | 57.5 |
| ACOS (per-review F1) | 25.5 | 25.9 | 33.3 |
| FinEntity (macro-F1) | 97.6 | 97.1 | 96.2 |
| iSarcasmEval (case exact accuracy, supplementary) | 89.0 | — | — |
| VAST (macro-F1) | 51.4 | 49.6 | 59.5 |
| NLI4CT (macro-F1) | 73.0 | 78.6 | 82.9 |
| CRUXEval (accuracy) | 87.0 | 86.1 | 86.7 |
| CLadder (accuracy) | 97.0 | 97.7 | 94.0 |
| HLE (accuracy) | 10.0 | — | — |
| Habermas Machine (accuracy) | 74.0 | 71.8 | 68.7 |
| PhishNChips (accuracy) | 76.0 | 75.0 | 79.6 |
| MMLU-Pro (accuracy) | 68.0 | 65.3 | 65.9 |
| BBH (accuracy) | 70.0 | 68.9 | 73.7 |
| RAGTruth (hallucination F1) | 25.6 | 35.6 | 79.4 |
| HoVer (accuracy) | 66.0 | 61.2 | 65.2 |
| When2Call MCQ (accuracy) | 65.0 | 65.6 | 72.4 |
| New Yorker (accuracy) | 64.0 | 66.1 | 69.5 |