| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta name="description" content="InferenceNet research leaderboard: single-pass generation, DeepAgents and an independent DeepSeek Harness result on 1,000 econometric replication tasks."> |
| <meta name="color-scheme" content="light dark"> |
| <title>Leaderboard — InferenceNet Challenge</title> |
| <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='6' fill='%230c1219'/%3E%3Ctext x='5' y='23' font-family='sans-serif' font-size='21' fill='%2378dbc0'%3EiN%3C/text%3E%3C/svg%3E"> |
| <link rel="stylesheet" href="./leaderboard.css"> |
| <link rel="stylesheet" href="./mountain.css"> |
| <link rel="stylesheet" href="./portal.css"> |
| <script type="module" src="./leaderboard.js"></script> |
| </head> |
| <body> |
| <a class="skip" href="#rankings">Skip to rankings</a> |
| <main class="page"> |
| <nav class="topbar" aria-label="Main navigation"><a class="brand" href="./index.html">InferenceNet <small>CHALLENGE</small></a><div class="nav-links"><a href="./index.html"><span data-en="Home" data-zh="主页" >Home</span></a><a href="./data.html"><span data-en="Data" data-zh="数据" >Data</span></a><a href="./leaderboard.html" aria-current="page"><span data-en="Leaderboard" data-zh="榜单" >Leaderboard</span></a><a href="./agent.html"><span data-en="Agent / Harness" data-zh="Agent / Harness" >Agent / Harness</span></a><button id="language" type="button" aria-label="Switch to Chinese">中文</button><button id="theme" type="button" aria-pressed="false"><span class="theme-icon" aria-hidden="true"></span><span id="theme-label">Light</span></button></div></nav> |
|
|
| <header class="hero"> |
| <p class="eyebrow"><span data-i18n="edition">LEADERBOARD · SEPTEMBER 2026</span></p> |
| <h1 data-i18n="title">InferenceNet Challenge Leaderboard</h1> |
| <p class="intro" data-i18n="intro">Models and harnesses on 1,000 econometric replication tasks.</p> |
| <div class="facts" aria-label="Dataset coverage"> |
| <span><strong id="model-count">—</strong><span data-i18n="models">model aliases</span></span> |
| <span><strong id="group-count">—</strong><span data-i18n="groups">experiment groups</span></span> |
| <span><strong id="task-count">—</strong><span data-i18n="tasks">tasks per group</span></span> |
| <span><strong id="record-count">—</strong><span data-i18n="records">task records</span></span> |
| <span class="update" id="published-date"></span> |
| </div> |
| </header> |
|
|
|
|
| <section id="rankings" aria-labelledby="rankings-title"> |
| <div class="section-head"><div><p class="eyebrow" data-i18n="resultsLabel">01 / RESULTS</p><h2 id="rankings-title" class="sr-only" data-i18n="rankings">Agent & Harness leaderboard</h2></div><span class="badge" data-i18n="research">Research results · provisional</span></div> |
|
|
|
|
|
|
| <div class="toolbar"> |
| <div class="view-switch" role="group" aria-label="Leaderboard view"> |
| <button type="button" data-view="all" aria-pressed="true" data-i18n="all">All results</button> |
| <button type="button" data-view="compare" aria-pressed="false" data-i18n="compare">Baseline / DeepAgents pairs</button> |
| <button type="button" data-view="baseline" aria-pressed="false" data-i18n="single">Single-pass Agent</button> |
| <button type="button" data-view="harness" aria-pressed="false" data-i18n="harness">All harness runs</button> |
| </div> |
| <a class="download" href="./leaderboard.csv" download data-i18n="download">Download CSV ↓</a> |
| </div> |
| <div class="filters"> |
| <label class="metric-control"><span data-i18n="metric">Rank by</span><select id="metric"> |
| <optgroup label="Local paper profile" data-group="local"> |
| <option value="perfect" data-i18n="perfect">Full replication · local</option> |
| </optgroup> |
| <optgroup label="HF-style profile · parity unverified" data-group="hf"> |
| <option value="partial_replication" data-i18n="partial_replication">Partial replication · HF-style</option> |
| <option value="compilation_success" data-i18n="compilation_success">Execution success · HF-style</option> |
| <option value="coefficient_direction" data-i18n="coefficient_direction">Coefficient direction · HF-style</option> |
| <option value="significance_level" data-i18n="significance_level">Significance level · HF-style</option> |
| </optgroup> |
| </select></label> |
| <label id="sort-control"><span data-i18n="sort">Sort by</span><select id="sort"><option value="deepagents" data-i18n="harnessScore">DeepAgents score</option><option value="baseline" data-i18n="singleScore">Single-pass score</option><option value="gain" data-i18n="lift">Lift</option></select></label> |
| <label class="search-control"><span data-i18n="search">Find a model</span><input id="search" type="search" placeholder="Search models…" autocomplete="off"></label> |
| </div> |
| <div class="display-tools"> |
| <p id="metric-description" class="metric-description"></p> |
| <div class="display-switch" role="group" aria-label="Display format"> |
| <button type="button" data-display="table" aria-pressed="true" data-i18n="tableView">Table</button> |
| <button type="button" data-display="mountain" aria-pressed="false" data-i18n="mountainView">Mountain view</button> |
| </div> |
| </div> |
| <p id="independent-summary" class="independent-summary" hidden><span class="independent-dot" aria-hidden="true"></span><span data-i18n="dshShort">DSH is an independent run with technical recovery, not a paired lift.</span> <a href="#protocol-details" id="show-protocol" data-i18n="detailsLink">Details ↗</a></p> |
| <figure id="ranking-mountain" class="mountain" hidden></figure> |
| <p id="load-state" role="status" class="load-state">Loading archived results…</p> |
| <div id="error" class="error" role="alert" hidden><p data-i18n="error">Results could not be loaded. Please retry or open the results archive.</p><button type="button" id="retry" data-i18n="retry">Retry</button></div> |
| <div id="table-wrap" class="table-wrap" tabindex="0" aria-label="Scrollable leaderboard" hidden> |
| <table><caption id="caption" class="sr-only"></caption><thead id="table-head"></thead><tbody id="table-body"></tbody></table> |
| </div> |
| <p id="empty" class="load-state" hidden data-i18n="empty">No matching models.</p> |
| <div class="table-foot"><span id="result-count" aria-live="polite"></span><span data-i18n="denominator">Every score uses all 1,000 tasks. Unknown and invalid outcomes earn no successes.</span></div> |
| <p class="publication-note" data-i18n="publicationNote">Research snapshot · local scoring; official scorer parity unverified. Run budgets differ.</p> |
| </section> |
|
|
| <section class="method" aria-label="Evaluation details"> |
| <details id="protocol-details" class="evidence"> |
| <summary data-i18n="evaluationDetails">Evaluation protocol, definitions & evidence</summary> |
| <p class="scope-note" data-i18n="scope">Locally scored, archived results. Official scorer parity is not yet verified. Different budgets and run protocols mean the lift is descriptive, not an equal-cost comparison.</p> |
| <section class="protocols" aria-label="Experimental settings"> |
| <div><p class="protocol-tag baseline"><span aria-hidden="true">A</span><strong data-i18n="single">Single-pass Agent</strong></p><p data-i18n="singleDesc">One code-generation call, followed by execution. No iterative agent loop.</p></div> |
| <div><p class="protocol-tag harness"><span aria-hidden="true">B</span><strong data-i18n="deepagents">DeepAgents</strong></p><p data-i18n="harnessDesc">DeepAgents plans, uses tools and revises code. Up to 6 model calls, plus final execution.</p></div> |
| <div><p class="protocol-tag independent"><span aria-hidden="true">C</span><strong data-i18n="dsh">DeepSeek Harness · separate run</strong></p><p data-i18n="dshDesc">Official DSH SDK, requested v4-pro API alias. A fixed technical recovery selection; underlying model weights are unverified.</p></div> |
| </section> |
| <aside id="independent-notice" class="independent-notice" hidden><strong data-i18n="dshNoticeTitle">DSH is an independent result.</strong><p data-i18n="dshNotice">The request used deepseek-v4-pro; the underlying model weights were not verified. The fixed 1,000-task selection includes 87 technical replacement attempts (27 with changed input-schema prompts), with 30 task outcomes still unknown. It is excluded from the paired DeepAgents lift.</p><a id="dsh-protocol" data-i18n="dshProtocol" target="_blank" rel="noopener noreferrer">Read run protocol & evidence ↗</a></aside> |
| <div class="method-grid"> |
| <article><h3 data-i18n="fixedTasks">Fixed task set</h3><p data-i18n="fixedDesc">The same Selected_1000 task IDs and dataset revision are used for every group. Each score is successes ÷ 1,000; failed or unknown tasks stay in the denominator.</p></article> |
| <article><h3 data-i18n="separateMetrics">Two scoring profiles</h3><p data-i18n="profilesDesc">Full replication is the local paper metric. The four HF-style metrics reproduce the published descriptions locally; they are not yet verified against the official scorer.</p></article> |
| <article><h3 data-i18n="budgetTitle">Read lift with the budgets</h3><p data-i18n="budgetDesc">Baseline: 1 model call. DeepAgents and DSH: up to 6 model calls and 4 trial tool calls per attempt. DSH adds 87 recovery attempts; full environment and reasoning parity is unverified. DSH has no paired lift; these are not equal-cost causal estimates.</p></article> |
| </div> |
| <h3 class="evidence-heading" data-i18n="evidence">Evidence status & metric definitions</h3> |
| <p data-i18n="unknownDesc">“Metric unknown” means the selected metric cannot be assessed, including failed execution or unavailable outputs. It is different from a task whose final status is unknown. Neither is dropped or counted as a success.</p> |
| <ul id="definitions"></ul> |
| <div class="evidence-table-wrap"><table class="evidence-table"><caption data-i18n="evidenceCaption">Status from each accepted archive</caption><thead><tr><th data-i18n="model">Model / API alias</th><th data-i18n="protocol">Protocol</th><th data-i18n="evidenceCol">Evidence</th></tr></thead><tbody id="evidence-body"></tbody></table></div> |
| <p data-i18n="verification">HF-style counts are checked against archived task flags. Full-replication counts for Sol and Opus come from accepted summaries; all other displayed runs, including DSH, have per-task local flags. This refresh does not rerun or rescore any experiment.</p> |
| <p data-i18n="archiveNote">Per-run archive files remain frozen, including their original publication metadata. This page is a new display derived from those archives. GPT-5.5 and unarchived experiments are not included in this edition.</p> |
|
|
| </details> |
| <a class="historical" href="./legacy.html" data-i18n="historical">Historical leaderboard ↗</a> |
| </section> |
| <footer><span>InferenceNet <span class="muted">/</span> <span data-i18n="footer">Econometric replication, measured.</span></span><div><a id="revision-link" href="https://huggingface.co/spaces/CamoAiLab/InferenceNet-Leaderboard/tree/main/results" target="_blank" rel="noopener noreferrer" data-i18n="source">Source snapshot ↗</a><a href="./leaderboard-data.json" data-i18n="json">Data JSON ↗</a><a href="https://easonai-5589.github.io/inferencenet-challenge/" target="_blank" rel="noopener noreferrer" data-i18n="project">Project ↗</a></div></footer> |
| </main> |
| <noscript><p>This leaderboard needs JavaScript. <a href="./leaderboard.csv">Download the results CSV</a> or <a href="https://huggingface.co/spaces/CamoAiLab/InferenceNet-Leaderboard/tree/main/results">browse the archives</a>.</p></noscript> |
| </body> |
| </html> |
|
|
