index.html

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>ToolUse.1 — Tool-Calling Benchmark with Abstention</title>
<meta name="description" content="ToolUse.1 measures whether a model calls tools correctly — and only when it should. 114 English cases · JSON / XML / plain-text protocols · deterministic grading, no LLM judge." />
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='7' fill='%230a0b0d'/><path d='M8 22V10h3.4v9.2H21V22z' fill='%23e5484d'/></svg>" />
<link rel="stylesheet" href="styles.css" />
</head>
<body>

<nav class="nav">
  <div class="wrap">
    <a class="brand" href="index.html"><span class="dot"></span>ToolUse<span class="mark">.1</span></a>
    <div class="nav-links">
      <a href="index.html" aria-current="page">Overview</a>
      <a href="leaderboard.html">Leaderboard</a>
      <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1" target="_blank" rel="noopener">Dataset</a>
      <a class="nav-cta" href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions" target="_blank" rel="noopener">Submit a model</a>
    </div>
  </div>
</nav>

<header class="hero">
  <div class="wrap">
    <p class="eyebrow anim" style="animation-delay:.05s">Hardcore AI · benchmark</p>
    <h1 class="anim" style="animation-delay:.12s">ToolUse<span class="mark">.1</span></h1>
    <p class="lede anim" style="animation-delay:.2s">
      Can a model call a tool when it should — and stay quiet when it shouldn't?
      ToolUse.1 scores correct invocation, correct abstention, and correct clarification
      separately, on 114 authored cases in English.
    </p>
    <ul class="stats anim" style="animation-delay:.3s">
      <li><b data-count="114">114</b><span>cases</span></li>
      <li><b data-count="78">78</b><span>require a call</span></li>
      <li><b data-count="36">36</b><span>abstain / clarify</span></li>
      <li><b data-count="9">9</b><span>tool schemas</span></li>
      <li><b data-count="3">3</b><span>protocols</span></li>
      <li><b data-count="0">0</b><span>LLM judges</span></li>
    </ul>
  </div>
</header>

<main class="wrap">

  <section class="metrics reveal">
    <div class="metric">
      <div class="label">Overall score</div>
      <div class="value" data-metric="overall">77.2%</div>
      <div class="note">Passed cases over all cases, three protocols combined. Each entry reports a Wilson 95% interval.</div>
    </div>
    <div class="metric">
      <div class="label">Correct call</div>
      <div class="value" data-metric="call">71.8%</div>
      <div class="note">Right tool with semantically correct arguments when a tool applies. Expressions are evaluated — <code>45+12</code> equals <code>45 + 12</code>.</div>
    </div>
    <div class="metric">
      <div class="label">Abstention</div>
      <div class="value" data-metric="abstain">88.9%</div>
      <div class="note">No call emitted when no tool applies: knowledge questions, absent capabilities, explicit no-tool instructions.</div>
    </div>
    <p class="metric-leader">Current leader: <b data-leader>HardcoreAI/Nova-1.2-0.8B</b> — the figures above are its run.</p>
  </section>

  <section class="grid reveal">
    <div class="card">
      <div class="k">What it measures</div>
      <h3>Right call, right time</h3>
      <p>Nine tools across arithmetic, weather, search, translation, conversion, timers, notes and time. Every case is a realistic user turn — no trivia, no synthetic templates.</p>
    </div>
    <div class="card">
      <div class="k">Three protocols</div>
      <h3>JSON · XML · plain-text</h3>
      <p>The output protocol is a run-time axis, not a data axis. Each case is rendered in all three; grading uses canonical call semantics, so a model is judged on intent, not formatting.</p>
    </div>
    <div class="card">
      <div class="k">Deterministic grading</div>
      <h3>No LLM judge</h3>
      <p>Arguments are compared as values with alias tables for cities, currencies, units and timezones. Exactly one call is required for call cases. The oracle self-check scores 100%.</p>
    </div>
  </section>

  <section class="reveal">
    <h2 class="section">Leaderboard — top of the board</h2>
    <div class="board" data-board>
      <div class="board-head">
        <h2>ToolUse.1 — Leaderboard</h2>
        <p class="stamp" data-board-stamp>4 entries · v1.0.0 · updated 2026-10-18</p>
      </div>
      <table>
        <thead>
          <tr>
            <th class="col-rank">#</th>
            <th class="col-model">Model</th>
            <th class="num" title="Passed cases / all cases. Wilson 95% interval shown underneath.">Score</th>
            <th class="num" title="Correct call when a tool applies">Correct call</th>
            <th class="num" title="Correct abstention when no tool applies">Abstention</th>
            <th class="num" title="No-call cases where any call was emitted. Lower is better.">Spurious ↓</th>
            <th class="num" title="Cases scored">n</th>
          </tr>
        </thead>
        <tbody data-board-body>
          <tr class="in" data-score="1.0000" data-correct_call="1.0000" data-abstention="1.0000" data-spurious_call_rate="0.0000" data-n_cases="342">
            <td class="col-rank"><span class="rank-badge rank-1">1</span></td>
            <td class="col-model"><span class="model-name">oracle</span><span class="model-sub">all protocols · reads the expectations out of the corpus</span></td>
            <td class="num"><div class="score-cell"><div class="bar"><i style="width:100.0%"></i></div><span class="score-value">100.0%</span></div><span class="ci">98.9% – 100.0%</span></td>
            <td class="num">100.0%</td>
            <td class="num">100.0%</td>
            <td class="num">0.0%</td>
            <td class="num">342</td>
            <td><div class="proto-mini">
              <div><span class="pl">json</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div>
              <div><span class="pl">xml</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div>
              <div><span class="pl">plain</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div>
            </div></td>
            <td class="col-notes"><span class="tag">grader self-check</span></td>
          </tr>
          <tr class="in" data-score="0.7719" data-correct_call="0.7179" data-abstention="0.8889" data-spurious_call_rate="0.0463" data-n_cases="342">
            <td class="col-rank"><span class="rank-badge rank-2">2</span></td>
            <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/HardcoreAI/Nova-1.2-0.8B" target="_blank" rel="noopener">HardcoreAI/Nova-1.2-0.8B</a><span class="model-sub">all protocols · LoRA on Nova-1.1 · greedy · thinking off</span></td>
            <td class="num"><div class="score-cell"><div class="bar"><i style="width:77.2%"></i></div><span class="score-value">77.2%</span></div><span class="ci">72.5% – 81.3%</span></td>
            <td class="num">71.8%</td>
            <td class="num">88.9%</td>
            <td class="num">4.6%</td>
            <td class="num">342</td>
            <td><div class="proto-mini">
              <div><span class="pl">json</span><div class="pb"><i style="width:82.5%"></i></div><span class="pv">82.5%</span></div>
              <div><span class="pl">xml</span><div class="pb"><i style="width:73.7%"></i></div><span class="pv">73.7%</span></div>
              <div><span class="pl">plain</span><div class="pb"><i style="width:75.4%"></i></div><span class="pv">75.4%</span></div>
            </div></td>
            <td class="col-notes"><span class="ci">LoRA · bf16 · batch 8</span></td>
          </tr>
          <tr class="in" data-score="0.6491" data-correct_call="0.5983" data-abstention="0.7593" data-spurious_call_rate="0.1667" data-n_cases="342">
            <td class="col-rank"><span class="rank-badge rank-3">3</span></td>
            <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/HardcoreAI/Nova-1.1-0.8B" target="_blank" rel="noopener">HardcoreAI/Nova-1.1-0.8B</a><span class="model-sub">all protocols · greedy · thinking off</span></td>
            <td class="num"><div class="score-cell"><div class="bar"><i style="width:64.9%"></i></div><span class="score-value">64.9%</span></div><span class="ci">59.7% – 69.8%</span></td>
            <td class="num">59.8%</td>
            <td class="num">75.9%</td>
            <td class="num spurious-mid">16.7%</td>
            <td class="num">342</td>
            <td><div class="proto-mini">
              <div><span class="pl">json</span><div class="pb"><i style="width:67.5%"></i></div><span class="pv">67.5%</span></div>
              <div><span class="pl">xml</span><div class="pb"><i style="width:65.8%"></i></div><span class="pv">65.8%</span></div>
              <div><span class="pl">plain</span><div class="pb"><i style="width:61.4%"></i></div><span class="pv">61.4%</span></div>
            </div></td>
            <td class="col-notes"><span class="ci">base model · bf16 · batch 8</span></td>
          </tr>
          <tr class="in" data-score="0.5965" data-correct_call="0.4829" data-abstention="0.8426" data-spurious_call_rate="0.0648" data-n_cases="342">
            <td class="col-rank"><span class="rank-badge">4</span></td>
            <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/Qwen/Qwen3.5-0.8B" target="_blank" rel="noopener">Qwen/Qwen3.5-0.8B</a><span class="model-sub">all protocols · greedy · thinking off</span></td>
            <td class="num"><div class="score-cell"><div class="bar"><i style="width:59.7%"></i></div><span class="score-value">59.7%</span></div><span class="ci">54.4% – 64.7%</span></td>
            <td class="num">48.3%</td>
            <td class="num">84.3%</td>
            <td class="num">6.5%</td>
            <td class="num">342</td>
            <td><div class="proto-mini">
              <div><span class="pl">json</span><div class="pb"><i style="width:67.5%"></i></div><span class="pv">67.5%</span></div>
              <div><span class="pl">xml</span><div class="pb"><i style="width:68.4%"></i></div><span class="pv">68.4%</span></div>
              <div><span class="pl">plain</span><div class="pb"><i style="width:43.0%"></i></div><span class="pv">43.0%</span></div>
            </div></td>
            <td class="col-notes"><span class="ci">base model · bf16 · batch 8</span></td>
          </tr>
        </tbody>
      </table>
      <p class="empty" data-board-empty hidden>No model results yet. See <strong>Submit a model</strong> below.</p>
    </div>
    <p style="margin:1rem 0 0"><a href="leaderboard.html">Open the full leaderboard →</a></p>
  </section>

  <section class="submit reveal">
    <h2 class="section">Submit a model</h2>
    <p>Run the benchmark yourself, keep the raw results JSON the harness writes, and open a discussion on the dataset repository with the model id and configuration.</p>
    <pre><code>git clone https://huggingface.co/datasets/HardcoreAI/ToolUse.1
pip install -r requirements.txt         # transformers 5.x for Nova / Qwen3.5

python -m tooluse1 selftest             # oracle must score exactly 100%
python -m tooluse1 run --model &lt;id&gt; \
    --backend transformers --protocol all --batch-size 8 --out results/</code></pre>
    <ol class="steps">
      <li><b>Verify the harness</b> — run the self-check first; it validates the grader against the corpus.</li>
      <li><b>Run all three protocols</b> and keep the results JSON the harness emits.</li>
      <li><b>Open a discussion</b> on <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions" target="_blank" rel="noopener">HardcoreAI/ToolUse.1</a> with the model id, protocols, case count, decoding config, and the raw JSON attachment.</li>
    </ol>
    <p style="margin-top:1rem;color:var(--dim);font-size:0.86rem;">
      Recommended: greedy decoding (<code>do_sample=False</code>), <code>max_new_tokens=256</code>, unmodified system prompt,
      no few-shot examples, reasoning mode disabled where supported. One dtype per row — bfloat16 on Ampere+, float16 on T4.
    </p>
  </section>

</main>

<footer>
  <div class="wrap">
    <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1">Dataset</a>
    <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/blob/main/SPEC.md">Specification</a>
    <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions">Discussions</a>
    <a href="https://huggingface.co/HardcoreAI">Hardcore AI</a>
    <span class="license">Apache-2.0 · v1.0.0 · <span id="year">2026</span></span>
  </div>
</footer>

<script src="app.js"></script>
</body>
</html>