Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> | |
| <title>Leaderboard — ToolUse.1</title> | |
| <meta name="description" content="Model leaderboard for ToolUse.1: overall score, correct invocation, correct abstention and spurious-call rate across JSON, XML and plain-text protocols." /> | |
| <link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='7' fill='%230a0b0d'/><path d='M8 22V10h3.4v9.2H21V22z' fill='%23e5484d'/></svg>" /> | |
| <link rel="stylesheet" href="styles.css" /> | |
| </head> | |
| <body> | |
| <nav class="nav"> | |
| <div class="wrap"> | |
| <a class="brand" href="index.html"><span class="dot"></span>ToolUse<span class="mark">.1</span></a> | |
| <div class="nav-links"> | |
| <a href="index.html">Overview</a> | |
| <a href="leaderboard.html" aria-current="page">Leaderboard</a> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1" target="_blank" rel="noopener">Dataset</a> | |
| <a class="nav-cta" href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions" target="_blank" rel="noopener">Submit a model</a> | |
| </div> | |
| </div> | |
| </nav> | |
| <header class="hero" style="padding:3.4rem 0 1.2rem"> | |
| <div class="wrap"> | |
| <p class="eyebrow anim" style="animation-delay:.05s">ToolUse.1 · results</p> | |
| <h1 class="anim" style="animation-delay:.12s">Leader<span class="mark">board</span></h1> | |
| <p class="lede anim" style="animation-delay:.2s"> | |
| One row per submitted run. Sorted by overall score; click any numeric header to re-rank. | |
| Every entry reports invocation, abstention and calibration — a high score with a high | |
| spurious-call rate is not the same model as a high score with a clean one. | |
| </p> | |
| </div> | |
| </header> | |
| <main class="wrap" style="padding-top:1.6rem"> | |
| <section class="reveal"> | |
| <div class="board" data-board> | |
| <div class="board-head"> | |
| <h2>All submissions</h2> | |
| <div class="board-tools"> | |
| <p class="stamp" data-board-stamp>4 entries · v1.0.0 · updated 2026-10-18</p> | |
| </div> | |
| </div> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th class="col-rank">#</th> | |
| <th class="col-model">Model</th> | |
| <th class="num" data-sort="score" title="Passed cases / all cases. Wilson 95% interval underneath.">Score <span class="arrow">↑↓</span></th> | |
| <th class="num" data-sort="correct_call" title="Correct call when a tool applies">Correct call <span class="arrow">↑↓</span></th> | |
| <th class="num" data-sort="abstention" title="Correct abstention when no tool applies">Abstention <span class="arrow">↑↓</span></th> | |
| <th class="num" data-sort="spurious_call_rate" title="No-call cases where any call was emitted. Lower is better.">Spurious ↓ <span class="arrow">↑↓</span></th> | |
| <th class="num" data-sort="n_cases" title="Cases scored">n <span class="arrow">↑↓</span></th> | |
| <th data-col-protocol>By protocol</th> | |
| <th class="col-notes" data-col-notes>Notes</th> | |
| </tr> | |
| </thead> | |
| <tbody data-board-body> | |
| <tr class="in" data-score="1.0000" data-correct_call="1.0000" data-abstention="1.0000" data-spurious_call_rate="0.0000" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge rank-1">1</span></td> | |
| <td class="col-model"><span class="model-name">oracle</span><span class="model-sub">all protocols · reads the expectations out of the corpus</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:100.0%"></i></div><span class="score-value">100.0%</span></div><span class="ci">98.9% – 100.0%</span></td> | |
| <td class="num">100.0%</td> | |
| <td class="num">100.0%</td> | |
| <td class="num">0.0%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:100.0%"></i></div><span class="pv">100.0%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="tag">grader self-check</span></td> | |
| </tr> | |
| <tr class="in" data-score="0.7719" data-correct_call="0.7179" data-abstention="0.8889" data-spurious_call_rate="0.0463" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge rank-2">2</span></td> | |
| <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/HardcoreAI/Nova-1.2-0.8B" target="_blank" rel="noopener">HardcoreAI/Nova-1.2-0.8B</a><span class="model-sub">all protocols · LoRA on Nova-1.1 · greedy · thinking off</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:77.2%"></i></div><span class="score-value">77.2%</span></div><span class="ci">72.5% – 81.3%</span></td> | |
| <td class="num">71.8%</td> | |
| <td class="num">88.9%</td> | |
| <td class="num">4.6%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:82.5%"></i></div><span class="pv">82.5%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:73.7%"></i></div><span class="pv">73.7%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:75.4%"></i></div><span class="pv">75.4%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="ci">LoRA · bf16 · batch 8</span></td> | |
| </tr> | |
| <tr class="in" data-score="0.6491" data-correct_call="0.5983" data-abstention="0.7593" data-spurious_call_rate="0.1667" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge rank-3">3</span></td> | |
| <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/HardcoreAI/Nova-1.1-0.8B" target="_blank" rel="noopener">HardcoreAI/Nova-1.1-0.8B</a><span class="model-sub">all protocols · greedy · thinking off</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:64.9%"></i></div><span class="score-value">64.9%</span></div><span class="ci">59.7% – 69.8%</span></td> | |
| <td class="num">59.8%</td> | |
| <td class="num">75.9%</td> | |
| <td class="num spurious-mid">16.7%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:67.5%"></i></div><span class="pv">67.5%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:65.8%"></i></div><span class="pv">65.8%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:61.4%"></i></div><span class="pv">61.4%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="ci">base model · bf16 · batch 8</span></td> | |
| </tr> | |
| <tr class="in" data-score="0.5965" data-correct_call="0.4829" data-abstention="0.8426" data-spurious_call_rate="0.0648" data-n_cases="342"> | |
| <td class="col-rank"><span class="rank-badge">4</span></td> | |
| <td class="col-model"><a class="model-link model-name" href="https://huggingface.co/Qwen/Qwen3.5-0.8B" target="_blank" rel="noopener">Qwen/Qwen3.5-0.8B</a><span class="model-sub">all protocols · greedy · thinking off</span></td> | |
| <td class="num"><div class="score-cell"><div class="bar"><i style="width:59.7%"></i></div><span class="score-value">59.7%</span></div><span class="ci">54.4% – 64.7%</span></td> | |
| <td class="num">48.3%</td> | |
| <td class="num">84.3%</td> | |
| <td class="num">6.5%</td> | |
| <td class="num">342</td> | |
| <td><div class="proto-mini"> | |
| <div><span class="pl">json</span><div class="pb"><i style="width:67.5%"></i></div><span class="pv">67.5%</span></div> | |
| <div><span class="pl">xml</span><div class="pb"><i style="width:68.4%"></i></div><span class="pv">68.4%</span></div> | |
| <div><span class="pl">plain</span><div class="pb"><i style="width:43.0%"></i></div><span class="pv">43.0%</span></div> | |
| </div></td> | |
| <td class="col-notes"><span class="ci">base model · bf16 · batch 8</span></td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p class="empty" data-board-empty hidden>No model results yet. Open a discussion to submit the first one.</p> | |
| </div> | |
| </section> | |
| <section class="grid reveal"> | |
| <div class="card"> | |
| <div class="k">Score</div> | |
| <h3>What counts</h3> | |
| <p>A case passes when the emitted call set matches the expectation: the right tool, semantically correct arguments, and — on no-call cases — nothing emitted at all. Wilson 95% interval in parentheses.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="k">Correct call ↑</div> | |
| <h3>Recall on tool cases</h3> | |
| <p>Share of call-eligible cases where the model produced the expected invocation with correct arguments. High abstention at low correct-call means the model is too shy.</p> | |
| </div> | |
| <div class="card"> | |
| <div class="k">Spurious ↓</div> | |
| <h3>Calibration</h3> | |
| <p>Share of no-call cases where the model emitted *some* call anyway. High abstention at high spurious means it is noisy, not cautious — the two are not interchangeable.</p> | |
| </div> | |
| </section> | |
| <section class="submit reveal"> | |
| <h2 class="section">Submit a model</h2> | |
| <p>Run the harness, keep the raw results JSON, and open a discussion on the dataset repository. Rows are added by hand — include the model id, protocols, case count, decoding configuration and the JSON attachment.</p> | |
| <pre><code>python -m tooluse1 selftest # oracle must score exactly 100% | |
| python -m tooluse1 run --model <id> --backend transformers \ | |
| --protocol all --batch-size 8 --out results/</code></pre> | |
| <p style="margin:1rem 0 0"> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions" target="_blank" rel="noopener">Open a submission discussion →</a> | |
| </p> | |
| </section> | |
| </main> | |
| <footer> | |
| <div class="wrap"> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1">Dataset</a> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/blob/main/SPEC.md">Specification</a> | |
| <a href="https://huggingface.co/datasets/HardcoreAI/ToolUse.1/discussions">Discussions</a> | |
| <a href="https://huggingface.co/HardcoreAI">Hardcore AI</a> | |
| <span class="license">Apache-2.0 · v1.0.0 · <span id="year">2026</span></span> | |
| </div> | |
| </footer> | |
| <script src="app.js"></script> | |
| </body> | |
| </html> | |