Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> | |
| <title>ToolUse.1 — Tool-Calling Benchmark with Abstention</title> | |
| <meta name="description" content="ToolUse.1 measures whether a model calls tools correctly — and only when it should. 114 English cases · JSON / XML / plain-text protocols · deterministic grading, no LLM judge." /> | |
| <link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='7' fill='%230a0b0d'/><path d='M8 22V10h3.4v9.2H21V22z' fill='%23e5484d'/></svg>" /> | |
| <link rel="stylesheet" href="styles.css" /> | |
| </head> | |
| <body> | |
| <nav class="nav"> | |
| <div class="wrap"> | |
| <a class="brand" href="index.html"><span class="dot"></span>ToolUse<span class="mark">.1</span></a> | |
| <div class="nav-links"> | |
| <a href="index.html" aria-current="page">Overview</a> | |
| <a href="leaderboard.html">Leaderboard</a> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1" target="_blank" rel="noopener">Dataset</a> | |
| <a class="nav-cta" href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions" target="_blank" rel="noopener">Submit a model</a> | |
| </div> | |
| </div> | |
| </nav> | |
| <header class="hero"> | |
| <div class="wrap"> | |
| <p class="eyebrow anim" style="animation-delay:.05s">Hardcore AI · benchmark</p> | |
| <h1 class="anim" style="animation-delay:.12s">ToolUse<span class="mark">.1</span></h1> | |
| <p class="lede anim" style="animation-delay:.2s"> | |
| Can a model call a tool when it should — and stay quiet when it shouldn't? | |
| ToolUse.1 scores correct invocation, correct abstention, and correct clarification | |
| separately, on 114 authored cases in English. | |
| </p> | |
| <ul class="stats anim" style="animation-delay:.3s"> | |
| <li><b data-count="114">114</b><span>cases</span></li> | |
| <li><b data-count="78">78</b><span>require a call</span></li> | |
| <li><b data-count="36">36</b><span>abstain / clarify</span></li> | |
| <li><b data-count="9">9</b><span>tool schemas</span></li> | |
| <li><b data-count="3">3</b><span>protocols</span></li> | |
| <li><b data-count="0">0</b><span>LLM judges</span></li> | |
| </ul> | |
| </div> | |
| </header> | |
| <main class="wrap"> | |
| <section class="metrics reveal"> | |
| <div class="metric"> | |
| <div class="label">Overall score</div> | |
| <div class="value" data-metric="overall">77.2%</div> | |
| <div class="note">Passed cases over all cases, three protocols combined. Each entry reports a Wilson 95% interval.</div> | |
| </div> | |
| <div class="metric"> | |
| <div class="label">Correct call</div> | |
| <div class="value" data-metric="call">71.8%</div> | |
| <div class="note">Right tool with semantically correct arguments when a tool applies. Expressions are evaluated — <code>45+12</code> equals <code>45 + 12</code>.</div> | |
| </div> | |
| <div class="metric"> | |
| <div class="label">Abstention</div> | |
| <div class="value" data-metric="abstain">88.9%</div> | |
| <div class="note">No call emitted when no tool applies: knowledge questions, absent capabilities, explicit no-tool instructions.</div> | |
| </div> | |
| <p class="metric-leader">Current leader: <b data-leader>HardcoreAI/Nova-1.2-0.8B</b> — the figures above are its run.</p> | |
| </section> | |
| <section class="grid reveal"> | |
| <div class="card"> | |
| <div class="k">What it measures</div> | |
| <h3>Right call, right time</h3> | |
| <p>Nine tools across arithmetic, weather, search, translation, conversion, timers, notes and time. Every case is a realistic user turn — no trivia, no synthetic templates.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="k">Three protocols</div> | |
| <h3>JSON · XML · plain-text</h3> | |
| <p>The output protocol is a run-time axis, not a data axis. Each case is rendered in all three; grading uses canonical call semantics, so a model is judged on intent, not formatting.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="k">Deterministic grading</div> | |
| <h3>No LLM judge</h3> | |
| <p>Arguments are compared as values with alias tables for cities, currencies, units and timezones. Exactly one call is required for call cases. The oracle self-check scores 100%.</p> | |
| </div> | |
| </section> | |
| <section class="reveal"> | |
| <h2 class="section">Leaderboard — top of the board</h2> | |
| <div class="board" data-board> | |
| <div class="board-head"> | |
| <h2>ToolUse.1 — Leaderboard</h2> | |
| <p class="stamp" data-board-stamp>4 entries · v1.0.0 · updated 2026-10-18</p> | |
| </div> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th class="col-rank">#</th> | |
| <th class="col-model">Model</th> | |
| <th class="num" title="Passed cases / all cases. Wilson 95% interval shown underneath.">Score</th> | |
| <th class="num" title="Correct call when a tool applies">Correct call</th> | |
| <th class="num" title="Correct abstention when no tool applies">Abstention</th> | |
| <th class="num" title="No-call cases where any call was emitted. Lower is better.">Spurious ↓</th> | |
| <th class="num" title="Cases scored">n</th> | |
| </tr> | |
| </thead> | |
| <tbody data-board-body> | |
| <tr class="in" data-score="1.0000" data-correct_call="1.0000" data-abstention="1.0000" data-spurious_call_rate="0.0000" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge rank-1">1</span></td> | |
| <td class="col-model"><span class="model-name">oracle</span><span class="model-sub">all protocols · reads the expectations out of the corpus</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:100.0%"></i></div><span class="score-value">100.0%</span></div><span class="ci">98.9% – 100.0%</span></td> | |
| <td class="num">100.0%</td> | |
| <td class="num">100.0%</td> | |
| <td class="num">0.0%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="tag">grader self-check</span></td> | |
| </tr> | |
| <tr class="in" data-score="0.7719" data-correct_call="0.7179" data-abstention="0.8889" data-spurious_call_rate="0.0463" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge rank-2">2</span></td> | |
| <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/HardcoreAI/Nova-1.2-0.8B" target="_blank" rel="noopener">HardcoreAI/Nova-1.2-0.8B</a><span class="model-sub">all protocols · LoRA on Nova-1.1 · greedy · thinking off</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:77.2%"></i></div><span class="score-value">77.2%</span></div><span class="ci">72.5% – 81.3%</span></td> | |
| <td class="num">71.8%</td> | |
| <td class="num">88.9%</td> | |
| <td class="num">4.6%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:82.5%"></i></div><span class="pv">82.5%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:73.7%"></i></div><span class="pv">73.7%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:75.4%"></i></div><span class="pv">75.4%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="ci">LoRA · bf16 · batch 8</span></td> | |
| </tr> | |
| <tr class="in" data-score="0.6491" data-correct_call="0.5983" data-abstention="0.7593" data-spurious_call_rate="0.1667" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge rank-3">3</span></td> | |
| <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/HardcoreAI/Nova-1.1-0.8B" target="_blank" rel="noopener">HardcoreAI/Nova-1.1-0.8B</a><span class="model-sub">all protocols · greedy · thinking off</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:64.9%"></i></div><span class="score-value">64.9%</span></div><span class="ci">59.7% – 69.8%</span></td> | |
| <td class="num">59.8%</td> | |
| <td class="num">75.9%</td> | |
| <td class="num spurious-mid">16.7%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:67.5%"></i></div><span class="pv">67.5%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:65.8%"></i></div><span class="pv">65.8%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:61.4%"></i></div><span class="pv">61.4%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="ci">base model · bf16 · batch 8</span></td> | |
| </tr> | |
| <tr class="in" data-score="0.5965" data-correct_call="0.4829" data-abstention="0.8426" data-spurious_call_rate="0.0648" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge">4</span></td> | |
| <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/Qwen/Qwen3.5-0.8B" target="_blank" rel="noopener">Qwen/Qwen3.5-0.8B</a><span class="model-sub">all protocols · greedy · thinking off</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:59.7%"></i></div><span class="score-value">59.7%</span></div><span class="ci">54.4% – 64.7%</span></td> | |
| <td class="num">48.3%</td> | |
| <td class="num">84.3%</td> | |
| <td class="num">6.5%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:67.5%"></i></div><span class="pv">67.5%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:68.4%"></i></div><span class="pv">68.4%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:43.0%"></i></div><span class="pv">43.0%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="ci">base model · bf16 · batch 8</span></td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p class="empty" data-board-empty hidden>No model results yet. See <strong>Submit a model</strong> below.</p> | |
| </div> | |
| <p style="margin:1rem 0 0"><a href="leaderboard.html">Open the full leaderboard →</a></p> | |
| </section> | |
| <section class="submit reveal"> | |
| <h2 class="section">Submit a model</h2> | |
| <p>Run the benchmark yourself, keep the raw results JSON the harness writes, and open a discussion on the dataset repository with the model id and configuration.</p> | |
| <pre><code>git clone https://huggingface.co/datasets/HardcoreAI/ToolUse.1 | |
| pip install -r requirements.txt # transformers 5.x for Nova / Qwen3.5 | |
| python -m tooluse1 selftest # oracle must score exactly 100% | |
| python -m tooluse1 run --model <id> \ | |
| --backend transformers --protocol all --batch-size 8 --out results/</code></pre> | |
| <ol class="steps"> | |
| <li><b>Verify the harness</b> — run the self-check first; it validates the grader against the corpus.</li> | |
| <li><b>Run all three protocols</b> and keep the results JSON the harness emits.</li> | |
| <li><b>Open a discussion</b> on <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions" target="_blank" rel="noopener">HardcoreAI/ToolUse.1</a> with the model id, protocols, case count, decoding config, and the raw JSON attachment.</li> | |
| </ol> | |
| <p style="margin-top:1rem;color:var(--dim);font-size:0.86rem;"> | |
| Recommended: greedy decoding (<code>do_sample=False</code>), <code>max_new_tokens=256</code>, unmodified system prompt, | |
| no few-shot examples, reasoning mode disabled where supported. One dtype per row — bfloat16 on Ampere+, float16 on T4. | |
| </p> | |
| </section> | |
| </main> | |
| <footer> | |
| <div class="wrap"> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1">Dataset</a> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/blob/main/SPEC.md">Specification</a> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions">Discussions</a> | |
| <a href="https://huggingface.co/HardcoreAI">Hardcore AI</a> | |
| <span class="license">Apache-2.0 · v1.0.0 · <span id="year">2026</span></span> | |
| </div> | |
| </footer> | |
| <script src="app.js"></script> | |
| </body> | |
| </html> | |