Add the InferenceNet project portal with Home, Data and Agent pages
Browse filesMake the Space a project homepage with shared navigation, bilingual text and theme preferences. Keep the existing leaderboard on leaderboard.html and read its published snapshot on Home. Add verified Selected_1000 distributions with a pinned CSV hash and a recorded agent walkthrough. Preserve all published scores and result archives.
- README.md +32 -3
- agent.html +186 -0
- build_dataset_summary.py +35 -0
- data.html +229 -0
- dataset-summary.json +113 -0
- index.html +194 -112
- leaderboard.html +111 -0
- leaderboard.js +2 -20
- portal.css +6 -0
- portal.js +64 -0
- site-shell.js +33 -0
README.md
CHANGED
|
@@ -1,15 +1,27 @@
|
|
| 1 |
---
|
| 2 |
-
title: InferenceNet
|
| 3 |
emoji: 📊
|
| 4 |
colorFrom: green
|
| 5 |
colorTo: blue
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
| 8 |
license: apache-2.0
|
| 9 |
-
short_description:
|
| 10 |
---
|
| 11 |
|
| 12 |
-
# InferenceNet
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
The September 2026 research leaderboard displays **six requested model aliases,
|
| 15 |
thirteen groups, and 13,000 archived task records**. Switch between the original
|
|
@@ -104,3 +116,20 @@ display refresh, not a fresh inference run, native artifact audit, or rescore.
|
|
| 104 |
[Project website](https://easonai-5589.github.io/inferencenet-challenge/) ·
|
| 105 |
[Project code](https://github.com/EasonAI-5589/NTU-GIFTS-InferenceNet) ·
|
| 106 |
[Evaluation framework](https://github.com/YTZSR/chatpilot_evaluator/tree/dev-gyc-eval)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: InferenceNet Challenge
|
| 3 |
emoji: 📊
|
| 4 |
colorFrom: green
|
| 5 |
colorTo: blue
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
| 8 |
license: apache-2.0
|
| 9 |
+
short_description: Explore the benchmark data, agent harnesses and leaderboard
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# InferenceNet Challenge
|
| 13 |
+
|
| 14 |
+
The project portal brings the benchmark, data documentation, execution walkthrough and published results into one Hugging Face Space.
|
| 15 |
+
|
| 16 |
+
- [Home](https://camoailab-inferencenet-leaderboard.static.hf.space/index.html)
|
| 17 |
+
- [Data](https://camoailab-inferencenet-leaderboard.static.hf.space/data.html)
|
| 18 |
+
- [Leaderboard](https://camoailab-inferencenet-leaderboard.static.hf.space/leaderboard.html)
|
| 19 |
+
- [Agent / Harness](https://camoailab-inferencenet-leaderboard.static.hf.space/agent.html)
|
| 20 |
+
- [Dataset repository](https://huggingface.co/datasets/CamoAiLab/InferenceNet)
|
| 21 |
+
|
| 22 |
+
The original Space URL remains valid. Data files continue to live in the dataset repository; the portal includes only a small, reproducible distribution summary. Home and Leaderboard both read `leaderboard-data.json`. `dataset-summary.json` is derived from the same pinned task-list revision.
|
| 23 |
+
|
| 24 |
+
## Agent & Harness Leaderboard
|
| 25 |
|
| 26 |
The September 2026 research leaderboard displays **six requested model aliases,
|
| 27 |
thirteen groups, and 13,000 archived task records**. Switch between the original
|
|
|
|
| 116 |
[Project website](https://easonai-5589.github.io/inferencenet-challenge/) ·
|
| 117 |
[Project code](https://github.com/EasonAI-5589/NTU-GIFTS-InferenceNet) ·
|
| 118 |
[Evaluation framework](https://github.com/YTZSR/chatpilot_evaluator/tree/dev-gyc-eval)
|
| 119 |
+
|
| 120 |
+
## Editing the portal
|
| 121 |
+
|
| 122 |
+
`index.html`, `data.html`, `leaderboard.html` and `agent.html` are the four static pages.
|
| 123 |
+
`site-shell.js` shares language and theme preferences; `portal.css` extends the
|
| 124 |
+
existing leaderboard styles. `portal.js` renders the publication counts and data
|
| 125 |
+
distributions. Published metric values still come only from `leaderboard-data.json`.
|
| 126 |
+
|
| 127 |
+
To regenerate the data distribution summary from the exact pinned CSV:
|
| 128 |
+
|
| 129 |
+
```sh
|
| 130 |
+
python build_dataset_summary.py /path/to/1000_new.csv
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
The script verifies the CSV SHA-256 before generating `dataset-summary.json`.
|
| 134 |
+
Serve the repository with a local HTTP server to preview module scripts and JSON
|
| 135 |
+
loading; no frontend build step is required.
|
agent.html
ADDED
|
@@ -0,0 +1,186 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8">
|
| 5 |
+
<meta name="viewport" content="width=device-width,initial-scale=1">
|
| 6 |
+
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='6' fill='%230c1219'/%3E%3Ctext x='5' y='23' font-family='sans-serif' font-size='21' fill='%2378dbc0'%3EiN%3C/text%3E%3C/svg%3E">
|
| 7 |
+
<meta name="color-scheme" content="light dark">
|
| 8 |
+
<meta name="description" content="How single-pass execution, DeepAgents and DeepSeek Harness approach an InferenceNet task.">
|
| 9 |
+
<title>Agent / Harness — InferenceNet Challenge</title>
|
| 10 |
+
<link rel="stylesheet" href="./leaderboard.css">
|
| 11 |
+
<link rel="stylesheet" href="./portal.css">
|
| 12 |
+
<script type="module" src="./portal.js">
|
| 13 |
+
</script>
|
| 14 |
+
</head>
|
| 15 |
+
<body>
|
| 16 |
+
<a class="skip" href="#main">
|
| 17 |
+
<span data-en="Skip to content" data-zh="跳转到正文" >Skip to content</span>
|
| 18 |
+
</a>
|
| 19 |
+
<div class="page">
|
| 20 |
+
<nav class="topbar" aria-label="Main navigation">
|
| 21 |
+
<a class="brand" href="./index.html">InferenceNet <small>CHALLENGE</small>
|
| 22 |
+
</a>
|
| 23 |
+
<div class="nav-links">
|
| 24 |
+
<a href="./index.html">
|
| 25 |
+
<span data-en="Home" data-zh="主页" >Home</span>
|
| 26 |
+
</a>
|
| 27 |
+
<a href="./data.html">
|
| 28 |
+
<span data-en="Data" data-zh="数据" >Data</span>
|
| 29 |
+
</a>
|
| 30 |
+
<a href="./leaderboard.html">
|
| 31 |
+
<span data-en="Leaderboard" data-zh="榜单" >Leaderboard</span>
|
| 32 |
+
</a>
|
| 33 |
+
<a href="./agent.html" aria-current="page">
|
| 34 |
+
<span data-en="Agent / Harness" data-zh="Agent / Harness" >Agent / Harness</span>
|
| 35 |
+
</a>
|
| 36 |
+
<button id="language" type="button" aria-label="Switch to Chinese">中文</button>
|
| 37 |
+
<button id="theme" type="button" aria-pressed="false">
|
| 38 |
+
<span class="theme-icon" aria-hidden="true">
|
| 39 |
+
</span>
|
| 40 |
+
<span id="theme-label">Light</span>
|
| 41 |
+
</button>
|
| 42 |
+
</div>
|
| 43 |
+
</nav>
|
| 44 |
+
<main id="main">
|
| 45 |
+
<header class="portal-hero">
|
| 46 |
+
<p class="eyebrow">AGENT / HARNESS</p>
|
| 47 |
+
<h1 data-en="From one attempt to an execution loop." data-zh="从单次生成,到执行反馈循环。" >From one attempt to an execution loop.</h1>
|
| 48 |
+
<p data-en="The model proposes the analysis. The harness manages tools, feedback, budgets and final artifacts. InferenceNet makes the resulting process and output inspectable." data-zh="模型提出分析方案;Harness 管理工具、反馈、预算与最终产物。InferenceNet 让执行过程和结果都可以被检查。" class="portal-lead">The model proposes the analysis. The harness manages tools, feedback, budgets and final artifacts. InferenceNet makes the resulting process and output inspectable.</p>
|
| 49 |
+
</header>
|
| 50 |
+
<section class="portal-section">
|
| 51 |
+
<div class="portal-heading">
|
| 52 |
+
<p data-en="01 / EXECUTION MODES" data-zh=" 01 / EXECUTION MODES" class="eyebrow">01 / EXECUTION MODES</p>
|
| 53 |
+
<h2 data-en="The harness is part of the experiment." data-zh="Harness 是实验的一部分。" >The harness is part of the experiment.</h2>
|
| 54 |
+
</div>
|
| 55 |
+
<div class="profile-grid">
|
| 56 |
+
<article class="profile">
|
| 57 |
+
<p class="eyebrow">A / BASELINE</p>
|
| 58 |
+
<h3>Single-pass Agent</h3>
|
| 59 |
+
<div class="profile-budget">1 <span data-en="model call" data-zh="次模型调用" >model call</span>
|
| 60 |
+
</div>
|
| 61 |
+
<p data-en="Generate one program, then execute it. There is no iterative tool-feedback loop." data-zh="生成一个程序后执行,不进行多轮工具反馈循环。" >Generate one program, then execute it. There is no iterative tool-feedback loop.</p>
|
| 62 |
+
<a href="./leaderboard.html" class="text-link">
|
| 63 |
+
<span data-en="See published results →" data-zh="查看已发布结果 →" >See published results →</span>
|
| 64 |
+
</a>
|
| 65 |
+
</article>
|
| 66 |
+
<article class="profile">
|
| 67 |
+
<p class="eyebrow">B / AGENT LOOP</p>
|
| 68 |
+
<h3>DeepAgents</h3>
|
| 69 |
+
<div class="profile-budget">≤ 6 <span data-en="model calls" data-zh="次模型调用" >model calls</span>
|
| 70 |
+
</div>
|
| 71 |
+
<p data-en="Plan, inspect data, execute trial programs and revise. Up to four trial tool calls, followed by final execution." data-zh="规划、检查数据、试运行与修订;最多四次试运行工具调用,然后执行最终程序。" >Plan, inspect data, execute trial programs and revise. Up to four trial tool calls, followed by final execution.</p>
|
| 72 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/agent.html" class="text-link" target="_blank" rel="noopener noreferrer">
|
| 73 |
+
<span data-en="Read the full walkthrough ↗" data-zh="阅读完整执行记录 ↗" >Read the full walkthrough ↗</span>
|
| 74 |
+
</a>
|
| 75 |
+
</article>
|
| 76 |
+
<article class="profile">
|
| 77 |
+
<p class="eyebrow">C / INDEPENDENT</p>
|
| 78 |
+
<h3>DeepSeek Harness · DSH</h3>
|
| 79 |
+
<div class="profile-budget">≤ 6 <span data-en="calls / attempt" data-zh="次调用 / 尝试" >calls / attempt</span>
|
| 80 |
+
</div>
|
| 81 |
+
<p data-en="Official DSH SDK. The published run uses technical recovery and stays separate from the baseline / DeepAgents paired comparison." data-zh="使用官方 DSH SDK;已发布实验包含技术补测,独立于单次生成 / DeepAgents 配对比较展示。" >Official DSH SDK. The published run uses technical recovery and stays separate from the baseline / DeepAgents paired comparison.</p>
|
| 82 |
+
<a href="https://huggingface.co/spaces/CamoAiLab/InferenceNet-Leaderboard/blob/6ea7c5684664bfe5c6e3d282a5e28f68244acd9e/results/deepseek-v4-pro/dsh/README.md" class="text-link" target="_blank" rel="noopener noreferrer">
|
| 83 |
+
<span data-en="Read the run protocol ↗" data-zh="查看运行协议 ↗" >Read the run protocol ↗</span>
|
| 84 |
+
</a>
|
| 85 |
+
</article>
|
| 86 |
+
</div>
|
| 87 |
+
<p data-en="These are the published run settings, not universal framework limits. Unequal total budgets and differing protocols do not support an equal-cost causal claim." data-zh="以上为已发布实验的设定,并非框架本身的限制。各组总预算与协议存在差异,不能据此作等成本的因果判断。" class="source-note">These are the published run settings, not universal framework limits. Unequal total budgets and differing protocols do not support an equal-cost causal claim.</p>
|
| 88 |
+
</section>
|
| 89 |
+
<section class="portal-section">
|
| 90 |
+
<div class="portal-heading">
|
| 91 |
+
<p data-en="02 / FEEDBACK" data-zh=" 02 / FEEDBACK" class="eyebrow">02 / FEEDBACK</p>
|
| 92 |
+
<h2 data-en="A loop with an explicit stopping point." data-zh="有明确停止条件的反馈循环。" >A loop with an explicit stopping point.</h2>
|
| 93 |
+
</div>
|
| 94 |
+
<div class="agent-loop">
|
| 95 |
+
<ol class="workflow">
|
| 96 |
+
<li>
|
| 97 |
+
<span class="step-number">01</span>
|
| 98 |
+
<h3 data-en="Read the task" data-zh="读取任务" >Read the task</h3>
|
| 99 |
+
<p data-en="A research specification and its source data." data-zh="研究要求与对应的数据文件。" >A research specification and its source data.</p>
|
| 100 |
+
</li>
|
| 101 |
+
<li>
|
| 102 |
+
<span class="step-number">02</span>
|
| 103 |
+
<h3 data-en="Write & run code" data-zh="编写并执行代码" >Write & run code</h3>
|
| 104 |
+
<p data-en="Fit the requested econometric model." data-zh="按照要求估计计量模型。" >Fit the requested econometric model.</p>
|
| 105 |
+
</li>
|
| 106 |
+
<li>
|
| 107 |
+
<span class="step-number">03</span>
|
| 108 |
+
<h3 data-en="Inspect & revise" data-zh="检查并修订" >Inspect & revise</h3>
|
| 109 |
+
<p data-en="A harness can use execution feedback." data-zh="Harness 可根据执行反馈修订程序。" >A harness can use execution feedback.</p>
|
| 110 |
+
</li>
|
| 111 |
+
<li>
|
| 112 |
+
<span class="step-number">04</span>
|
| 113 |
+
<h3 data-en="Recover the finding" data-zh="复现研究结果" >Recover the finding</h3>
|
| 114 |
+
<p data-en="Return the coefficient, standard error and p-value." data-zh="输出系数、标准误和 p 值。" >Return the coefficient, standard error and p-value.</p>
|
| 115 |
+
</li>
|
| 116 |
+
</ol>
|
| 117 |
+
<div class="loop-feedback">↶ <span data-en="Execution feedback returns to the model until it finalizes or reaches its budget." data-zh="执行反馈返回模型,直到模型提交最终程序或达到预算上限。" >Execution feedback returns to the model until it finalizes or reaches its budget.</span>
|
| 118 |
+
</div>
|
| 119 |
+
</div>
|
| 120 |
+
</section>
|
| 121 |
+
<section class="portal-section" id="case">
|
| 122 |
+
<div class="portal-heading">
|
| 123 |
+
<p data-en="03 / RECORDED CASE" data-zh=" 03 / RECORDED CASE" class="eyebrow">03 / RECORDED CASE</p>
|
| 124 |
+
<h2 data-en="Same task. Two execution paths." data-zh="同一道任务,两种执行路径。" >Same task. Two execution paths.</h2>
|
| 125 |
+
</div>
|
| 126 |
+
<p class="portal-lead">
|
| 127 |
+
<span data-en="Task 0011 asks for a monthly-clustered regression on Spain, 1905–1945. This recorded GPT-5.5 example shows why inspecting the data can change an outcome." data-zh="任务 0011 要求对 1905–1945 年的西班牙样本进行按月聚类的回归。这个已记录的 GPT-5.5 案例展示了检查数据如何影响执行结果。" >Task 0011 asks for a monthly-clustered regression on Spain, 1905–1945. This recorded GPT-5.5 example shows why inspecting the data can change an outcome.</span>
|
| 128 |
+
</p>
|
| 129 |
+
<div class="case-pair">
|
| 130 |
+
<article>
|
| 131 |
+
<p class="eyebrow">SINGLE-PASS</p>
|
| 132 |
+
<h3 data-en="A guessed schema stops execution." data-zh="对数据结构的猜测导致执行失败。" >A guessed schema stops execution.</h3>
|
| 133 |
+
<p data-en="The generated program searches for common month/date column names. It misses the dataset’s tid column and exits before producing a result." data-zh="生成的程序只搜索常见的月份 / 日期列名,遗漏数据中的 tid 列,因此未能输出结果。" >The generated program searches for common month/date column names. It misses the dataset’s tid column and exits before producing a result.</p>
|
| 134 |
+
<div class="case-outcome">1 <span data-en="model call · no valid result JSON" data-zh="次模型调用 · 未生成有效结果 JSON" >model call · no valid result JSON</span>
|
| 135 |
+
</div>
|
| 136 |
+
</article>
|
| 137 |
+
<article>
|
| 138 |
+
<p class="eyebrow">DEEPAGENTS</p>
|
| 139 |
+
<h3 data-en="Inspect, estimate, then finalize." data-zh="先检查,再估计,最后提交。" >Inspect, estimate, then finalize.</h3>
|
| 140 |
+
<p data-en="The agent inspects 38 columns, checks the time structure, then fits OLS with tid-clustered errors on 977 observations. Its final program recovers all three statistics." data-zh="Agent 检查了 38 列数据和时间结构,然后对 977 条观测进行按 tid 聚类的 OLS 估计。最终程序复现了三项统计量。" >The agent inspects 38 columns, checks the time structure, then fits OLS with tid-clustered errors on 977 observations. Its final program recovers all three statistics.</p>
|
| 141 |
+
<div class="case-outcome">4 <span data-en="model calls · 3 tool calls" data-zh="次模型调用 · 3 次工具调用" >model calls · 3 tool calls</span>
|
| 142 |
+
</div>
|
| 143 |
+
</article>
|
| 144 |
+
</div>
|
| 145 |
+
<div class="artifact">
|
| 146 |
+
<div>
|
| 147 |
+
<h3 data-en="The final artifact" data-zh="最终产物" >The final artifact</h3>
|
| 148 |
+
<p data-en="A program writes these three numeric fields. Execution logs and the evaluation record are stored separately." data-zh="程序写出这三个数值字段;执行日志与评分记录单独保存。" >A program writes these three numeric fields. Execution logs and the evaluation record are stored separately.</p>
|
| 149 |
+
</div>
|
| 150 |
+
<pre>
|
| 151 |
+
<code>{
|
| 152 |
+
"coefficient": 0.27273630753072026,
|
| 153 |
+
"standard_error": 0.054224701707830926,
|
| 154 |
+
"p_value": 6.899021793750105e-7
|
| 155 |
+
}</code>
|
| 156 |
+
</pre>
|
| 157 |
+
</div>
|
| 158 |
+
<p data-en="Illustrative internal run from 13 September 2026, not a result in the current leaderboard. One example does not establish an average harness benefit." data-zh="2026 年 9 月 13 日的内部执行示例,不计入当前榜单。单个案例不代表 Harness 的平均提升。" class="source-note">Illustrative internal run from 13 September 2026, not a result in the current leaderboard. One example does not establish an average harness benefit.</p>
|
| 159 |
+
<div class="actions">
|
| 160 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/agent.html#walkthrough" class="text-link" target="_blank" rel="noopener noreferrer">
|
| 161 |
+
<span data-en="Inspect the complete trace ↗" data-zh="查看完整轨迹 ↗" >Inspect the complete trace ↗</span>
|
| 162 |
+
</a>
|
| 163 |
+
<a href="./leaderboard.html#protocol-details" class="text-link">
|
| 164 |
+
<span data-en="Scoring definitions & evidence →" data-zh="评分定义与证据 →" >Scoring definitions & evidence →</span>
|
| 165 |
+
</a>
|
| 166 |
+
</div>
|
| 167 |
+
</section>
|
| 168 |
+
</main>
|
| 169 |
+
<footer>
|
| 170 |
+
<span>InferenceNet <span class="muted">/</span> <span data-en="AI for empirical research." data-zh="面向实证研究的 AI。" >AI for empirical research.</span>
|
| 171 |
+
</span>
|
| 172 |
+
<div>
|
| 173 |
+
<a href="https://huggingface.co/datasets/CamoAiLab/InferenceNet" class="" target="_blank" rel="noopener noreferrer">
|
| 174 |
+
<span data-en="Dataset ↗" data-zh="数据集 ↗" >Dataset ↗</span>
|
| 175 |
+
</a>
|
| 176 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/" class="" target="_blank" rel="noopener noreferrer">
|
| 177 |
+
<span data-en="Project website ↗" data-zh="项目官网 ↗" >Project website ↗</span>
|
| 178 |
+
</a>
|
| 179 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/teams.html" class="" target="_blank" rel="noopener noreferrer">
|
| 180 |
+
<span data-en="Team & contact ↗" data-zh="团队与联系 ↗" >Team & contact ↗</span>
|
| 181 |
+
</a>
|
| 182 |
+
</div>
|
| 183 |
+
</footer>
|
| 184 |
+
</div>
|
| 185 |
+
</body>
|
| 186 |
+
</html>
|
build_dataset_summary.py
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Build lightweight portal statistics from the published pinned task CSV.
|
| 2 |
+
|
| 3 |
+
Usage: python build_dataset_summary.py /path/to/1000_new.csv
|
| 4 |
+
Download that CSV from the dataset revision recorded below, not the moving main.
|
| 5 |
+
"""
|
| 6 |
+
import argparse
|
| 7 |
+
from collections import Counter
|
| 8 |
+
import csv
|
| 9 |
+
import hashlib
|
| 10 |
+
import io
|
| 11 |
+
import json
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
REVISION = '59f9512a38e594528807744214a60ee00367434e'
|
| 15 |
+
MAPPING = {'OLS': 'OLS', 'panel OLS': 'OLS', 'DID': 'DID', 'IV': 'IV', 'IV-2SLS': 'IV', 'RDD': 'RDD', 'PROBIT': 'Other', 'Probit': 'Other', 'LOGIT': 'Other', 'Logit': 'Other', 'Negative binomial': 'Other', 'LMM': 'Other', 'PSM': 'Other', 'Cox proportional hazards': 'Other'}
|
| 16 |
+
p = argparse.ArgumentParser(description=__doc__)
|
| 17 |
+
p.add_argument('task_list', type=Path)
|
| 18 |
+
p.add_argument('--output', type=Path, default=Path('dataset-summary.json'))
|
| 19 |
+
a = p.parse_args()
|
| 20 |
+
raw = a.task_list.read_bytes()
|
| 21 |
+
assert hashlib.sha256(raw).hexdigest() == '5338c8af548b35dab0e92a63b0acfebb0bc29c7901da4088851025f5709d6bb3', 'Input does not match the pinned task list'
|
| 22 |
+
rows = list(csv.DictReader(io.StringIO(raw.decode('utf-8-sig'))))
|
| 23 |
+
assert len(rows) == 1000 and len({r['sequence_id'] for r in rows}) == 1000
|
| 24 |
+
journals = Counter(r['journal'] for r in rows)
|
| 25 |
+
methods = Counter(MAPPING[r['method']] for r in rows)
|
| 26 |
+
result = {
|
| 27 |
+
'dataset': 'CamoAiLab/InferenceNet', 'revision': REVISION,
|
| 28 |
+
'task_list': 'Selected_1000/1000_new.csv', 'task_list_sha256': hashlib.sha256(raw).hexdigest(),
|
| 29 |
+
'tasks': len(rows), 'journal_count': len(journals), 'distinct_article_titles': len({r['article'] for r in rows}),
|
| 30 |
+
'raw_method_labels': len({r['method'] for r in rows}), 'languages': dict(Counter(r['language'] for r in rows)),
|
| 31 |
+
'methods': [{'label': k, 'count': v} for k,v in methods.most_common()],
|
| 32 |
+
'journals': [{'label': k, 'count': v} for k,v in journals.most_common()], 'method_mapping': MAPPING,
|
| 33 |
+
}
|
| 34 |
+
a.output.write_text(json.dumps(result, indent=2, ensure_ascii=False) + '\n')
|
| 35 |
+
print(f"Wrote {a.output}: {len(rows)} tasks, {len(journals)} journals")
|
data.html
ADDED
|
@@ -0,0 +1,229 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8">
|
| 5 |
+
<meta name="viewport" content="width=device-width,initial-scale=1">
|
| 6 |
+
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='6' fill='%230c1219'/%3E%3Ctext x='5' y='23' font-family='sans-serif' font-size='21' fill='%2378dbc0'%3EiN%3C/text%3E%3C/svg%3E">
|
| 7 |
+
<meta name="color-scheme" content="light dark">
|
| 8 |
+
<meta name="description" content="Explore the pinned Selected_1000 benchmark: methods, journals, task structure and dataset access.">
|
| 9 |
+
<title>Data — InferenceNet Challenge</title>
|
| 10 |
+
<link rel="stylesheet" href="./leaderboard.css">
|
| 11 |
+
<link rel="stylesheet" href="./portal.css">
|
| 12 |
+
<script type="module" src="./portal.js">
|
| 13 |
+
</script>
|
| 14 |
+
</head>
|
| 15 |
+
<body>
|
| 16 |
+
<a class="skip" href="#main">
|
| 17 |
+
<span data-en="Skip to content" data-zh="跳转到正文" >Skip to content</span>
|
| 18 |
+
</a>
|
| 19 |
+
<div class="page">
|
| 20 |
+
<nav class="topbar" aria-label="Main navigation">
|
| 21 |
+
<a class="brand" href="./index.html">InferenceNet <small>CHALLENGE</small>
|
| 22 |
+
</a>
|
| 23 |
+
<div class="nav-links">
|
| 24 |
+
<a href="./index.html">
|
| 25 |
+
<span data-en="Home" data-zh="主页" >Home</span>
|
| 26 |
+
</a>
|
| 27 |
+
<a href="./data.html" aria-current="page">
|
| 28 |
+
<span data-en="Data" data-zh="数据" >Data</span>
|
| 29 |
+
</a>
|
| 30 |
+
<a href="./leaderboard.html">
|
| 31 |
+
<span data-en="Leaderboard" data-zh="榜单" >Leaderboard</span>
|
| 32 |
+
</a>
|
| 33 |
+
<a href="./agent.html">
|
| 34 |
+
<span data-en="Agent / Harness" data-zh="Agent / Harness" >Agent / Harness</span>
|
| 35 |
+
</a>
|
| 36 |
+
<button id="language" type="button" aria-label="Switch to Chinese">中文</button>
|
| 37 |
+
<button id="theme" type="button" aria-pressed="false">
|
| 38 |
+
<span class="theme-icon" aria-hidden="true">
|
| 39 |
+
</span>
|
| 40 |
+
<span id="theme-label">Light</span>
|
| 41 |
+
</button>
|
| 42 |
+
</div>
|
| 43 |
+
</nav>
|
| 44 |
+
<main id="main">
|
| 45 |
+
<header class="portal-hero">
|
| 46 |
+
<p class="eyebrow">DATA · SELECTED_1000</p>
|
| 47 |
+
<h1 data-en="The benchmark, task by task." data-zh="逐题了解评测数据。" >The benchmark, task by task.</h1>
|
| 48 |
+
<p data-en="Each task names an outcome, a treatment, controls, an estimation method and a data file. The goal is to recover one reported effect: its coefficient, standard error and p-value." data-zh="每道任务指定结果变量、处理变量、控制变量、估计方法与数据文件,要求复现一个研究效应的系数、标准误和 p 值。" class="portal-lead">Each task names an outcome, a treatment, controls, an estimation method and a data file. The goal is to recover one reported effect: its coefficient, standard error and p-value.</p>
|
| 49 |
+
<div class="actions">
|
| 50 |
+
<a href="https://huggingface.co/datasets/CamoAiLab/InferenceNet" class="action primary" target="_blank" rel="noopener noreferrer">
|
| 51 |
+
<span data-en="Open the dataset ↗" data-zh="打开数据集 ↗" >Open the dataset ↗</span>
|
| 52 |
+
</a>
|
| 53 |
+
<a href="https://huggingface.co/datasets/CamoAiLab/InferenceNet/resolve/59f9512a38e594528807744214a60ee00367434e/Selected_1000/1000_new.csv?download=true" class="action" target="_blank" rel="noopener noreferrer">
|
| 54 |
+
<span data-en="Download task list ↓" data-zh="下载任务清单 ↓" >Download task list ↓</span>
|
| 55 |
+
</a>
|
| 56 |
+
</div>
|
| 57 |
+
</header>
|
| 58 |
+
<div class="scale-strip">
|
| 59 |
+
<div>
|
| 60 |
+
<strong>1,000</strong>
|
| 61 |
+
<span data-en="fixed evaluation tasks" data-zh="固定评测任务" >fixed evaluation tasks</span>
|
| 62 |
+
</div>
|
| 63 |
+
<div>
|
| 64 |
+
<strong>15</strong>
|
| 65 |
+
<span data-en="journals in this subset" data-zh="子集涵盖期刊" >journals in this subset</span>
|
| 66 |
+
</div>
|
| 67 |
+
<div>
|
| 68 |
+
<strong>286</strong>
|
| 69 |
+
<span data-en="distinct article titles" data-zh="不同论文标题" >distinct article titles</span>
|
| 70 |
+
</div>
|
| 71 |
+
<div>
|
| 72 |
+
<strong>5</strong>
|
| 73 |
+
<span data-en="estimation classes" data-zh="估计方法大类" >estimation classes</span>
|
| 74 |
+
</div>
|
| 75 |
+
</div>
|
| 76 |
+
<p data-en="Selected_1000 · Counts from the pinned task list, revision 59f9512." data-zh="Selected_1000 · 按固定任务清单统计,版本 59f9512。" class="source-note">Selected_1000 · Counts from the pinned task list, revision 59f9512.</p>
|
| 77 |
+
<section class="portal-section">
|
| 78 |
+
<div class="portal-heading">
|
| 79 |
+
<p data-en="01 / DISTRIBUTION" data-zh=" 01 / DISTRIBUTION" class="eyebrow">01 / DISTRIBUTION</p>
|
| 80 |
+
<h2 data-en="Different research questions. Different demands." data-zh="不同的研究问题,不同的复现要求。" >Different research questions. Different demands.</h2>
|
| 81 |
+
</div>
|
| 82 |
+
<div class="distribution-grid">
|
| 83 |
+
<figure class="distribution">
|
| 84 |
+
<figcaption>
|
| 85 |
+
<h3 data-en="Estimation methods" data-zh="估计方法" >Estimation methods</h3>
|
| 86 |
+
<p data-en="Five classes · 1,000 tasks" data-zh="五大类 · 1,000 题" >Five classes · 1,000 tasks</p>
|
| 87 |
+
</figcaption>
|
| 88 |
+
<div id="method-chart" class="bar-chart" role="status">
|
| 89 |
+
<span data-en="Loading distributions…" data-zh="正在加载分布…" >Loading distributions…</span>
|
| 90 |
+
</div>
|
| 91 |
+
<p data-en="OLS includes panel OLS; IV includes IV-2SLS. Other combines Probit, Logit, negative binomial, LMM, PSM and Cox models." data-zh="OLS 包含面板 OLS,IV 包含 IV-2SLS;其他类合并 Probit、Logit、负二项、LMM、PSM 与 Cox 模型。" class="source-note">OLS includes panel OLS; IV includes IV-2SLS. Other combines Probit, Logit, negative binomial, LMM, PSM and Cox models.</p>
|
| 92 |
+
</figure>
|
| 93 |
+
<figure class="distribution">
|
| 94 |
+
<figcaption>
|
| 95 |
+
<h3 data-en="Journal coverage" data-zh="期刊分布" >Journal coverage</h3>
|
| 96 |
+
<p data-en="The five largest sources; all 15 below" data-zh="五个最大的任务来源;展开可查看全部 15 本" >The five largest sources; all 15 below</p>
|
| 97 |
+
</figcaption>
|
| 98 |
+
<div id="journal-chart" class="bar-chart" role="status">
|
| 99 |
+
<span data-en="Loading distributions…" data-zh="正在加载分布…" >Loading distributions…</span>
|
| 100 |
+
</div>
|
| 101 |
+
<details class="journal-rest">
|
| 102 |
+
<summary>
|
| 103 |
+
<span data-en="All 15 journals" data-zh="全部 15 本期刊" >All 15 journals</span>
|
| 104 |
+
</summary>
|
| 105 |
+
<div id="journal-table">
|
| 106 |
+
</div>
|
| 107 |
+
</details>
|
| 108 |
+
</figure>
|
| 109 |
+
</div>
|
| 110 |
+
<p data-en="Percentages use all 1,000 tasks. These distributions are recomputed from the pinned CSV, not from the moving dataset main branch." data-zh="百分比均以 1,000 题为分母;分布重新统计自固定版本的 CSV,不随数据集主分支变化。" class="source-note">Percentages use all 1,000 tasks. These distributions are recomputed from the pinned CSV, not from the moving dataset main branch.</p>
|
| 111 |
+
</section>
|
| 112 |
+
<section class="portal-section">
|
| 113 |
+
<div class="portal-heading">
|
| 114 |
+
<p data-en="02 / TASK FORMAT" data-zh=" 02 / TASK FORMAT" class="eyebrow">02 / TASK FORMAT</p>
|
| 115 |
+
<h2 data-en="A clear input and output contract." data-zh="明确任务输入与输出。" >A clear input and output contract.</h2>
|
| 116 |
+
</div>
|
| 117 |
+
<div class="anatomy-grid">
|
| 118 |
+
<article>
|
| 119 |
+
<h3 data-en="What the agent receives" data-zh="Agent 的输入" >What the agent receives</h3>
|
| 120 |
+
<p data-en="Target and explanatory variables, controls, estimator, analysis requirements, and mounted source data." data-zh="目标变量与解释变量、控制变量、估计方法、分析要求,以及挂载的数据文件。" >Target and explanatory variables, controls, estimator, analysis requirements, and mounted source data.</p>
|
| 121 |
+
</article>
|
| 122 |
+
<article>
|
| 123 |
+
<h3 data-en="What the agent produces" data-zh="Agent 的输出" >What the agent produces</h3>
|
| 124 |
+
<p data-en="An executable program and a result JSON containing coefficient, standard_error and p_value." data-zh="可执行程序,以及包含 coefficient、standard_error、p_value 的结果 JSON。" >An executable program and a result JSON containing coefficient, standard_error and p_value.</p>
|
| 125 |
+
</article>
|
| 126 |
+
<article>
|
| 127 |
+
<h3 data-en="What the evaluator keeps separate" data-zh="评测侧独立保存的内容" >What the evaluator keeps separate</h3>
|
| 128 |
+
<p data-en="Reference answers and replication programs are kept outside the model input during evaluation. Public dataset downloads can contain references." data-zh="评测时参考答案与复现程序不进入模型输入;公开数据集下载文件可能包含这些参考资料。" >Reference answers and replication programs are kept outside the model input during evaluation. Public dataset downloads can contain references.</p>
|
| 129 |
+
</article>
|
| 130 |
+
</div>
|
| 131 |
+
</section>
|
| 132 |
+
<section class="portal-section">
|
| 133 |
+
<div class="portal-heading">
|
| 134 |
+
<p data-en="03 / EXAMPLE" data-zh=" 03 / EXAMPLE" class="eyebrow">03 / EXAMPLE</p>
|
| 135 |
+
<h2 data-en="One task, in context." data-zh="一个具体任务。" >One task, in context.</h2>
|
| 136 |
+
</div>
|
| 137 |
+
<div class="case-summary">
|
| 138 |
+
<div>
|
| 139 |
+
<p class="eyebrow">TASK 0011 · OLS</p>
|
| 140 |
+
<h3 data-en="Economic uncertainty and the “two Spains”" data-zh="经济不确定性与“两个西班牙”" >Economic uncertainty and the “two Spains”</h3>
|
| 141 |
+
<p data-en="Recover the relationship between socio-economic conflict and policy uncertainty, with three controls and month-clustered standard errors." data-zh="在加入三个控制变量、按月聚类标准误的设定下,复现社会经济冲突与政策不确定性的关系。" >Recover the relationship between socio-economic conflict and policy uncertainty, with three controls and month-clustered standard errors.</p>
|
| 142 |
+
<a href="./agent.html#case" class="text-link">
|
| 143 |
+
<span data-en="See the agent walkthrough →" data-zh="查看 Agent 执行过程 →" >See the agent walkthrough →</span>
|
| 144 |
+
</a>
|
| 145 |
+
</div>
|
| 146 |
+
<dl class="compact-spec">
|
| 147 |
+
<dt>y</dt>
|
| 148 |
+
<dd>
|
| 149 |
+
<code>EPU0month_simsn_w</code>
|
| 150 |
+
</dd>
|
| 151 |
+
<dt>x</dt>
|
| 152 |
+
<dd>
|
| 153 |
+
<code>Wscmonth_simsn_w</code>
|
| 154 |
+
</dd>
|
| 155 |
+
<dt>
|
| 156 |
+
<span data-en="Data" data-zh="数据" >Data</span>
|
| 157 |
+
</dt>
|
| 158 |
+
<dd>
|
| 159 |
+
<code>data_np.dta</code>
|
| 160 |
+
</dd>
|
| 161 |
+
<dt>
|
| 162 |
+
<span data-en="Sample" data-zh="样本" >Sample</span>
|
| 163 |
+
</dt>
|
| 164 |
+
<dd>1905–1945 · 977 <span data-en="estimation observations" data-zh="估计样本观测" >estimation observations</span>
|
| 165 |
+
</dd>
|
| 166 |
+
</dl>
|
| 167 |
+
</div>
|
| 168 |
+
</section>
|
| 169 |
+
<section class="portal-section">
|
| 170 |
+
<div class="portal-heading">
|
| 171 |
+
<p data-en="04 / SOURCE" data-zh=" 04 / SOURCE" class="eyebrow">04 / SOURCE</p>
|
| 172 |
+
<h2 data-en="A fixed, inspectable snapshot." data-zh="固定且可核查的数据版本。" >A fixed, inspectable snapshot.</h2>
|
| 173 |
+
</div>
|
| 174 |
+
<dl class="provenance">
|
| 175 |
+
<dt>
|
| 176 |
+
<span data-en="Dataset" data-zh="数据集" >Dataset</span>
|
| 177 |
+
</dt>
|
| 178 |
+
<dd>
|
| 179 |
+
<a href="https://huggingface.co/datasets/CamoAiLab/InferenceNet" class="" target="_blank" rel="noopener noreferrer">
|
| 180 |
+
<span data-en="CamoAiLab/InferenceNet ↗" data-zh="CamoAiLab/InferenceNet ↗" >CamoAiLab/InferenceNet ↗</span>
|
| 181 |
+
</a>
|
| 182 |
+
</dd>
|
| 183 |
+
<dt>
|
| 184 |
+
<span data-en="Pinned revision" data-zh="固定版本" >Pinned revision</span>
|
| 185 |
+
</dt>
|
| 186 |
+
<dd>
|
| 187 |
+
<code>59f9512a38e594528807744214a60ee00367434e</code>
|
| 188 |
+
</dd>
|
| 189 |
+
<dt>
|
| 190 |
+
<span data-en="Task list" data-zh="任务清单" >Task list</span>
|
| 191 |
+
</dt>
|
| 192 |
+
<dd>
|
| 193 |
+
<code>Selected_1000/1000_new.csv</code>
|
| 194 |
+
</dd>
|
| 195 |
+
<dt>
|
| 196 |
+
<span data-en="Subset labels" data-zh="子集标签" >Subset labels</span>
|
| 197 |
+
</dt>
|
| 198 |
+
<dd>
|
| 199 |
+
<span data-en="1,000 Stata-labelled tasks; 14 raw method labels folded into five classes. Article count is the number of distinct title strings." data-zh="1,000 题均标注为 Stata;14 种原始方法标签归并为五大类。论文数量按不同标题字符串统计。" >1,000 Stata-labelled tasks; 14 raw method labels folded into five classes. Article count is the number of distinct title strings.</span>
|
| 200 |
+
</dd>
|
| 201 |
+
</dl>
|
| 202 |
+
<div class="actions">
|
| 203 |
+
<a href="./dataset-summary.json" class="text-link">
|
| 204 |
+
<span data-en="Download distribution JSON ↓" data-zh="下载分布 JSON ↓" >Download distribution JSON ↓</span>
|
| 205 |
+
</a>
|
| 206 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/data.html" class="text-link" target="_blank" rel="noopener noreferrer">
|
| 207 |
+
<span data-en="Extended data documentation ↗" data-zh="完整数据说明 ↗" >Extended data documentation ↗</span>
|
| 208 |
+
</a>
|
| 209 |
+
</div>
|
| 210 |
+
</section>
|
| 211 |
+
</main>
|
| 212 |
+
<footer>
|
| 213 |
+
<span>InferenceNet <span class="muted">/</span> <span data-en="AI for empirical research." data-zh="面向实证研究的 AI。" >AI for empirical research.</span>
|
| 214 |
+
</span>
|
| 215 |
+
<div>
|
| 216 |
+
<a href="https://huggingface.co/datasets/CamoAiLab/InferenceNet" class="" target="_blank" rel="noopener noreferrer">
|
| 217 |
+
<span data-en="Dataset ↗" data-zh="数据集 ↗" >Dataset ↗</span>
|
| 218 |
+
</a>
|
| 219 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/" class="" target="_blank" rel="noopener noreferrer">
|
| 220 |
+
<span data-en="Project website ↗" data-zh="项目官网 ↗" >Project website ↗</span>
|
| 221 |
+
</a>
|
| 222 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/teams.html" class="" target="_blank" rel="noopener noreferrer">
|
| 223 |
+
<span data-en="Team & contact ↗" data-zh="团队与联系 ↗" >Team & contact ↗</span>
|
| 224 |
+
</a>
|
| 225 |
+
</div>
|
| 226 |
+
</footer>
|
| 227 |
+
</div>
|
| 228 |
+
</body>
|
| 229 |
+
</html>
|
dataset-summary.json
ADDED
|
@@ -0,0 +1,113 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"dataset": "CamoAiLab/InferenceNet",
|
| 3 |
+
"revision": "59f9512a38e594528807744214a60ee00367434e",
|
| 4 |
+
"task_list": "Selected_1000/1000_new.csv",
|
| 5 |
+
"task_list_sha256": "5338c8af548b35dab0e92a63b0acfebb0bc29c7901da4088851025f5709d6bb3",
|
| 6 |
+
"tasks": 1000,
|
| 7 |
+
"journal_count": 15,
|
| 8 |
+
"distinct_article_titles": 286,
|
| 9 |
+
"raw_method_labels": 14,
|
| 10 |
+
"languages": {
|
| 11 |
+
"stata": 1000
|
| 12 |
+
},
|
| 13 |
+
"methods": [
|
| 14 |
+
{
|
| 15 |
+
"label": "OLS",
|
| 16 |
+
"count": 773
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
"label": "DID",
|
| 20 |
+
"count": 135
|
| 21 |
+
},
|
| 22 |
+
{
|
| 23 |
+
"label": "IV",
|
| 24 |
+
"count": 55
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"label": "Other",
|
| 28 |
+
"count": 21
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"label": "RDD",
|
| 32 |
+
"count": 16
|
| 33 |
+
}
|
| 34 |
+
],
|
| 35 |
+
"journals": [
|
| 36 |
+
{
|
| 37 |
+
"label": "Management Science",
|
| 38 |
+
"count": 300
|
| 39 |
+
},
|
| 40 |
+
{
|
| 41 |
+
"label": "Journal of Economic History",
|
| 42 |
+
"count": 156
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"label": "Econometrica",
|
| 46 |
+
"count": 152
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"label": "The Journal of Law and Economics",
|
| 50 |
+
"count": 113
|
| 51 |
+
},
|
| 52 |
+
{
|
| 53 |
+
"label": "Journal of Financial Economics",
|
| 54 |
+
"count": 92
|
| 55 |
+
},
|
| 56 |
+
{
|
| 57 |
+
"label": "American Economic Review",
|
| 58 |
+
"count": 56
|
| 59 |
+
},
|
| 60 |
+
{
|
| 61 |
+
"label": "Journal of Political Economy",
|
| 62 |
+
"count": 40
|
| 63 |
+
},
|
| 64 |
+
{
|
| 65 |
+
"label": "Marketing Science",
|
| 66 |
+
"count": 27
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"label": "Review of Economic Studies",
|
| 70 |
+
"count": 21
|
| 71 |
+
},
|
| 72 |
+
{
|
| 73 |
+
"label": "Journal of Finance",
|
| 74 |
+
"count": 13
|
| 75 |
+
},
|
| 76 |
+
{
|
| 77 |
+
"label": "Journal of Financial and Quantitative Analysis",
|
| 78 |
+
"count": 13
|
| 79 |
+
},
|
| 80 |
+
{
|
| 81 |
+
"label": "Quarterly Journal of Economics",
|
| 82 |
+
"count": 10
|
| 83 |
+
},
|
| 84 |
+
{
|
| 85 |
+
"label": "The Review of Financial Studies",
|
| 86 |
+
"count": 3
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"label": "Journal of Health Economics",
|
| 90 |
+
"count": 3
|
| 91 |
+
},
|
| 92 |
+
{
|
| 93 |
+
"label": "Review of Finance",
|
| 94 |
+
"count": 1
|
| 95 |
+
}
|
| 96 |
+
],
|
| 97 |
+
"method_mapping": {
|
| 98 |
+
"OLS": "OLS",
|
| 99 |
+
"panel OLS": "OLS",
|
| 100 |
+
"DID": "DID",
|
| 101 |
+
"IV": "IV",
|
| 102 |
+
"IV-2SLS": "IV",
|
| 103 |
+
"RDD": "RDD",
|
| 104 |
+
"PROBIT": "Other",
|
| 105 |
+
"Probit": "Other",
|
| 106 |
+
"LOGIT": "Other",
|
| 107 |
+
"Logit": "Other",
|
| 108 |
+
"Negative binomial": "Other",
|
| 109 |
+
"LMM": "Other",
|
| 110 |
+
"PSM": "Other",
|
| 111 |
+
"Cox proportional hazards": "Other"
|
| 112 |
+
}
|
| 113 |
+
}
|
index.html
CHANGED
|
@@ -1,119 +1,201 @@
|
|
| 1 |
<!doctype html>
|
| 2 |
<html lang="en">
|
| 3 |
<head>
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
|
|
|
| 13 |
</head>
|
| 14 |
<body>
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
<
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
</body>
|
| 119 |
</html>
|
|
|
|
| 1 |
<!doctype html>
|
| 2 |
<html lang="en">
|
| 3 |
<head>
|
| 4 |
+
<meta charset="utf-8">
|
| 5 |
+
<meta name="viewport" content="width=device-width,initial-scale=1">
|
| 6 |
+
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='6' fill='%230c1219'/%3E%3Ctext x='5' y='23' font-family='sans-serif' font-size='21' fill='%2378dbc0'%3EiN%3C/text%3E%3C/svg%3E">
|
| 7 |
+
<meta name="color-scheme" content="light dark">
|
| 8 |
+
<meta name="description" content="InferenceNet: a benchmark for AI agents that reproduce empirical research. Explore the data, agent harnesses and published leaderboard.">
|
| 9 |
+
<title>Home — InferenceNet Challenge</title>
|
| 10 |
+
<link rel="stylesheet" href="./leaderboard.css">
|
| 11 |
+
<link rel="stylesheet" href="./portal.css">
|
| 12 |
+
<script type="module" src="./portal.js">
|
| 13 |
+
</script>
|
| 14 |
</head>
|
| 15 |
<body>
|
| 16 |
+
<a class="skip" href="#main">
|
| 17 |
+
<span data-en="Skip to content" data-zh="跳转到正文" >Skip to content</span>
|
| 18 |
+
</a>
|
| 19 |
+
<div class="page">
|
| 20 |
+
<nav class="topbar" aria-label="Main navigation">
|
| 21 |
+
<a class="brand" href="./index.html">InferenceNet <small>CHALLENGE</small>
|
| 22 |
+
</a>
|
| 23 |
+
<div class="nav-links">
|
| 24 |
+
<a href="./index.html" aria-current="page">
|
| 25 |
+
<span data-en="Home" data-zh="主页" >Home</span>
|
| 26 |
+
</a>
|
| 27 |
+
<a href="./data.html">
|
| 28 |
+
<span data-en="Data" data-zh="数据" >Data</span>
|
| 29 |
+
</a>
|
| 30 |
+
<a href="./leaderboard.html">
|
| 31 |
+
<span data-en="Leaderboard" data-zh="榜单" >Leaderboard</span>
|
| 32 |
+
</a>
|
| 33 |
+
<a href="./agent.html">
|
| 34 |
+
<span data-en="Agent / Harness" data-zh="Agent / Harness" >Agent / Harness</span>
|
| 35 |
+
</a>
|
| 36 |
+
<button id="language" type="button" aria-label="Switch to Chinese">中文</button>
|
| 37 |
+
<button id="theme" type="button" aria-pressed="false">
|
| 38 |
+
<span class="theme-icon" aria-hidden="true">
|
| 39 |
+
</span>
|
| 40 |
+
<span id="theme-label">Light</span>
|
| 41 |
+
</button>
|
| 42 |
+
</div>
|
| 43 |
+
</nav>
|
| 44 |
+
<main id="main">
|
| 45 |
+
<header class="portal-hero home-hero">
|
| 46 |
+
<div>
|
| 47 |
+
<p class="eyebrow">INFERENCENET CHALLENGE</p>
|
| 48 |
+
<h1 data-en="Can AI reproduce an empirical finding?" data-zh="AI 能复现真实的实证研究吗?" >Can AI reproduce an empirical finding?</h1>
|
| 49 |
+
<p data-en="Real research data. Executable analysis. Results you can inspect." data-zh="真实研究数据,可执行的分析过程,可核查的结果。" class="hero-line">Real research data. Executable analysis. Results you can inspect.</p>
|
| 50 |
+
<p data-en="InferenceNet evaluates how models and agent harnesses turn an econometric specification into a reproducible result." data-zh="InferenceNet 评测模型与 Agent Harness 能否将计量经济学任务转化为可复现的研究结果。" class="portal-lead">InferenceNet evaluates how models and agent harnesses turn an econometric specification into a reproducible result.</p>
|
| 51 |
+
<div class="actions">
|
| 52 |
+
<a href="./leaderboard.html" class="action primary">
|
| 53 |
+
<span data-en="Explore the leaderboard →" data-zh="查看榜单 →" >Explore the leaderboard →</span>
|
| 54 |
+
</a>
|
| 55 |
+
<a href="./data.html" class="action">
|
| 56 |
+
<span data-en="Meet the data →" data-zh="了解数据 →" >Meet the data →</span>
|
| 57 |
+
</a>
|
| 58 |
+
</div>
|
| 59 |
+
</div>
|
| 60 |
+
<aside class="task-figure" aria-label="A replication task">
|
| 61 |
+
<div class="figure-label">
|
| 62 |
+
<span data-en="A REPLICATION TASK" data-zh="一道复现任务" >A REPLICATION TASK</span>
|
| 63 |
+
<span>0011 / OLS</span>
|
| 64 |
+
</div>
|
| 65 |
+
<div class="figure-question">
|
| 66 |
+
<h2 data-en="Economic uncertainty & divisive politics" data-zh="经济不确定性与政治分歧" >Economic uncertainty & divisive politics</h2>
|
| 67 |
+
<p data-en="Spain · 1905–1945 · month-clustered errors" data-zh="西班牙 · 1905–1945 · 按月聚类标准误" >Spain · 1905–1945 · month-clustered errors</p>
|
| 68 |
+
</div>
|
| 69 |
+
<div class="figure-stage">
|
| 70 |
+
<span class="file-chip">data_np.dta</span>
|
| 71 |
+
<span aria-hidden="true">+</span>
|
| 72 |
+
<span data-en="Research specification" data-zh="研究要求" >Research specification</span>
|
| 73 |
+
</div>
|
| 74 |
+
<div class="figure-arrow" aria-hidden="true">↓</div>
|
| 75 |
+
<div class="figure-model">Model + Harness</div>
|
| 76 |
+
<div class="figure-arrow" aria-hidden="true">↓</div>
|
| 77 |
+
<div class="output-fields">
|
| 78 |
+
<span>coefficient</span>
|
| 79 |
+
<span>standard_error</span>
|
| 80 |
+
<span>p_value</span>
|
| 81 |
+
</div>
|
| 82 |
+
<a href="./agent.html#case" class="figure-link">
|
| 83 |
+
<span data-en="Follow the recorded case ↗" data-zh="查看真实执行案例 ↗" >Follow the recorded case ↗</span>
|
| 84 |
+
</a>
|
| 85 |
+
</aside>
|
| 86 |
+
</header>
|
| 87 |
+
<div class="scale-strip">
|
| 88 |
+
<div>
|
| 89 |
+
<strong>1,000</strong>
|
| 90 |
+
<span data-en="fixed evaluation tasks" data-zh="固定评测任务" >fixed evaluation tasks</span>
|
| 91 |
+
</div>
|
| 92 |
+
<div>
|
| 93 |
+
<strong>15</strong>
|
| 94 |
+
<span data-en="journals in this subset" data-zh="子集涵盖期刊" >journals in this subset</span>
|
| 95 |
+
</div>
|
| 96 |
+
<div>
|
| 97 |
+
<strong>286</strong>
|
| 98 |
+
<span data-en="distinct article titles" data-zh="不同论文标题" >distinct article titles</span>
|
| 99 |
+
</div>
|
| 100 |
+
<div>
|
| 101 |
+
<strong>5</strong>
|
| 102 |
+
<span data-en="estimation classes" data-zh="估计方法大类" >estimation classes</span>
|
| 103 |
+
</div>
|
| 104 |
+
</div>
|
| 105 |
+
<p data-en="Selected_1000 · Counts from the pinned task list, revision 59f9512." data-zh="Selected_1000 · 按固定任务清单统计,版本 59f9512。" class="source-note">Selected_1000 · Counts from the pinned task list, revision 59f9512.</p>
|
| 106 |
+
<section class="portal-section">
|
| 107 |
+
<div class="portal-heading">
|
| 108 |
+
<p data-en="01 / THE CHALLENGE" data-zh=" 01 / THE CHALLENGE" class="eyebrow">01 / THE CHALLENGE</p>
|
| 109 |
+
<h2 data-en="From a research question to a reproducible result." data-zh="从研究问题,到可复现的结果。" >From a research question to a reproducible result.</h2>
|
| 110 |
+
</div>
|
| 111 |
+
<ol class="workflow">
|
| 112 |
+
<li>
|
| 113 |
+
<span class="step-number">01</span>
|
| 114 |
+
<h3 data-en="Read the task" data-zh="读取任务" >Read the task</h3>
|
| 115 |
+
<p data-en="A research specification and its source data." data-zh="研究要求与对应的数据文件。" >A research specification and its source data.</p>
|
| 116 |
+
</li>
|
| 117 |
+
<li>
|
| 118 |
+
<span class="step-number">02</span>
|
| 119 |
+
<h3 data-en="Write & run code" data-zh="编写并执行代码" >Write & run code</h3>
|
| 120 |
+
<p data-en="Fit the requested econometric model." data-zh="按照要求估计计量模型。" >Fit the requested econometric model.</p>
|
| 121 |
+
</li>
|
| 122 |
+
<li>
|
| 123 |
+
<span class="step-number">03</span>
|
| 124 |
+
<h3 data-en="Inspect & revise" data-zh="检查并修订" >Inspect & revise</h3>
|
| 125 |
+
<p data-en="A harness can use execution feedback." data-zh="Harness 可根据执行反馈修订程序。" >A harness can use execution feedback.</p>
|
| 126 |
+
</li>
|
| 127 |
+
<li>
|
| 128 |
+
<span class="step-number">04</span>
|
| 129 |
+
<h3 data-en="Recover the finding" data-zh="复现研究结果" >Recover the finding</h3>
|
| 130 |
+
<p data-en="Return the coefficient, standard error and p-value." data-zh="输出系数、标准误和 p 值。" >Return the coefficient, standard error and p-value.</p>
|
| 131 |
+
</li>
|
| 132 |
+
</ol>
|
| 133 |
+
</section>
|
| 134 |
+
<section class="portal-section">
|
| 135 |
+
<div class="portal-heading">
|
| 136 |
+
<p data-en="02 / EXPLORE" data-zh=" 02 / EXPLORE" class="eyebrow">02 / EXPLORE</p>
|
| 137 |
+
<h2 data-en="One project. Three ways in." data-zh="从数据、结果与执行过程了解项目。" >One project. Three ways in.</h2>
|
| 138 |
+
</div>
|
| 139 |
+
<div class="entry-grid">
|
| 140 |
+
<a class="entry" href="./data.html">
|
| 141 |
+
<span class="eyebrow">01</span>
|
| 142 |
+
<h3 data-en="Explore the data" data-zh="了解数据" >Explore the data</h3>
|
| 143 |
+
<p data-en="Where the tasks come from, what they ask, and how to access them." data-zh="了解任务来源、研究要求与数据获取方式。" >Where the tasks come from, what they ask, and how to access them.</p>
|
| 144 |
+
<span class="entry-arrow" aria-hidden="true">↗</span>
|
| 145 |
+
</a>
|
| 146 |
+
<a class="entry" href="./leaderboard.html">
|
| 147 |
+
<span class="eyebrow">02</span>
|
| 148 |
+
<h3 data-en="Compare the results" data-zh="比较结果" >Compare the results</h3>
|
| 149 |
+
<p data-en="Single-pass models, DeepAgents and an independent DSH run." data-zh="单次生成、DeepAgents,以及独立的 DSH 实验。" >Single-pass models, DeepAgents and an independent DSH run.</p>
|
| 150 |
+
<span class="entry-arrow" aria-hidden="true">↗</span>
|
| 151 |
+
</a>
|
| 152 |
+
<a class="entry" href="./agent.html">
|
| 153 |
+
<span class="eyebrow">03</span>
|
| 154 |
+
<h3 data-en="Inside the agent loop" data-zh="走进 Agent 循环" >Inside the agent loop</h3>
|
| 155 |
+
<p data-en="Follow the code, tools and final artifact of a real replication task." data-zh="通���真实案例了解代码、工具与最终产物。" >Follow the code, tools and final artifact of a real replication task.</p>
|
| 156 |
+
<span class="entry-arrow" aria-hidden="true">↗</span>
|
| 157 |
+
</a>
|
| 158 |
+
</div>
|
| 159 |
+
</section>
|
| 160 |
+
<section class="portal-section">
|
| 161 |
+
<div class="portal-heading">
|
| 162 |
+
<p data-en="03 / RESULTS" data-zh=" 03 / RESULTS" class="eyebrow">03 / RESULTS</p>
|
| 163 |
+
<h2 data-en="A shared view of the evidence." data-zh="在同一份证据上比较。" >A shared view of the evidence.</h2>
|
| 164 |
+
</div>
|
| 165 |
+
<div class="release-strip">
|
| 166 |
+
<div>
|
| 167 |
+
<p class="eyebrow">
|
| 168 |
+
<span data-en="PUBLISHED RESEARCH SNAPSHOT" data-zh="已发布的研究快照" >PUBLISHED RESEARCH SNAPSHOT</span>
|
| 169 |
+
</p>
|
| 170 |
+
<div class="release-count">
|
| 171 |
+
<strong data-snapshot="groups">—</strong> <span data-en="experiment groups" data-zh="组实验" >experiment groups</span> <span class="muted">/</span> <strong data-snapshot="models">—</strong> <span data-en="model aliases" data-zh="个模型别名" >model aliases</span>
|
| 172 |
+
</div>
|
| 173 |
+
<p data-en="The homepage and leaderboard read the same published result file." data-zh="主页与榜单使用同一份已发布结果文件。" >The homepage and leaderboard read the same published result file.</p>
|
| 174 |
+
<p class="snapshot-date" data-snapshot="date" aria-live="polite">
|
| 175 |
+
</p>
|
| 176 |
+
</div>
|
| 177 |
+
<a href="./leaderboard.html" class="action">
|
| 178 |
+
<span data-en="View results →" data-zh="打开榜单 →" >View results →</span>
|
| 179 |
+
</a>
|
| 180 |
+
</div>
|
| 181 |
+
<p data-en="Research results use local scoring. Read the run protocols, scoring definitions and evidence status with every comparison." data-zh="研究结果采用本地评分;比较时请同时查看运行协议、指标定义与证据状态。" class="source-note">Research results use local scoring. Read the run protocols, scoring definitions and evidence status with every comparison.</p>
|
| 182 |
+
</section>
|
| 183 |
+
</main>
|
| 184 |
+
<footer>
|
| 185 |
+
<span>InferenceNet <span class="muted">/</span> <span data-en="AI for empirical research." data-zh="面向实证研究的 AI。" >AI for empirical research.</span>
|
| 186 |
+
</span>
|
| 187 |
+
<div>
|
| 188 |
+
<a href="https://huggingface.co/datasets/CamoAiLab/InferenceNet" class="" target="_blank" rel="noopener noreferrer">
|
| 189 |
+
<span data-en="Dataset ↗" data-zh="数据集 ↗" >Dataset ↗</span>
|
| 190 |
+
</a>
|
| 191 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/" class="" target="_blank" rel="noopener noreferrer">
|
| 192 |
+
<span data-en="Project website ↗" data-zh="项目官网 ↗" >Project website ↗</span>
|
| 193 |
+
</a>
|
| 194 |
+
<a href="https://easonai-5589.github.io/inferencenet-challenge/teams.html" class="" target="_blank" rel="noopener noreferrer">
|
| 195 |
+
<span data-en="Team & contact ↗" data-zh="团队与联系 ↗" >Team & contact ↗</span>
|
| 196 |
+
</a>
|
| 197 |
+
</div>
|
| 198 |
+
</footer>
|
| 199 |
+
</div>
|
| 200 |
</body>
|
| 201 |
</html>
|
leaderboard.html
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8">
|
| 5 |
+
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 6 |
+
<meta name="description" content="InferenceNet research leaderboard: single-pass generation, DeepAgents and an independent DeepSeek Harness result on 1,000 econometric replication tasks.">
|
| 7 |
+
<meta name="color-scheme" content="light dark">
|
| 8 |
+
<title>Leaderboard — InferenceNet Challenge</title>
|
| 9 |
+
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='6' fill='%230c1219'/%3E%3Ctext x='5' y='23' font-family='sans-serif' font-size='21' fill='%2378dbc0'%3EiN%3C/text%3E%3C/svg%3E">
|
| 10 |
+
<link rel="stylesheet" href="./leaderboard.css">
|
| 11 |
+
<link rel="stylesheet" href="./mountain.css">
|
| 12 |
+
<link rel="stylesheet" href="./portal.css">
|
| 13 |
+
<script type="module" src="./leaderboard.js"></script>
|
| 14 |
+
</head>
|
| 15 |
+
<body>
|
| 16 |
+
<a class="skip" href="#rankings">Skip to rankings</a>
|
| 17 |
+
<main class="page">
|
| 18 |
+
<nav class="topbar" aria-label="Main navigation"><a class="brand" href="./index.html">InferenceNet <small>CHALLENGE</small></a><div class="nav-links"><a href="./index.html"><span data-en="Home" data-zh="主页" >Home</span></a><a href="./data.html"><span data-en="Data" data-zh="数据" >Data</span></a><a href="./leaderboard.html" aria-current="page"><span data-en="Leaderboard" data-zh="榜单" >Leaderboard</span></a><a href="./agent.html"><span data-en="Agent / Harness" data-zh="Agent / Harness" >Agent / Harness</span></a><button id="language" type="button" aria-label="Switch to Chinese">中文</button><button id="theme" type="button" aria-pressed="false"><span class="theme-icon" aria-hidden="true"></span><span id="theme-label">Light</span></button></div></nav>
|
| 19 |
+
|
| 20 |
+
<header class="hero">
|
| 21 |
+
<p class="eyebrow"><span data-i18n="edition">LEADERBOARD · SEPTEMBER 2026</span></p>
|
| 22 |
+
<h1 data-i18n="title">InferenceNet Challenge Leaderboard</h1>
|
| 23 |
+
<p class="intro" data-i18n="intro">Models and harnesses on 1,000 econometric replication tasks.</p>
|
| 24 |
+
<div class="facts" aria-label="Dataset coverage">
|
| 25 |
+
<span><strong id="model-count">—</strong><span data-i18n="models">model aliases</span></span>
|
| 26 |
+
<span><strong id="group-count">—</strong><span data-i18n="groups">experiment groups</span></span>
|
| 27 |
+
<span><strong id="task-count">—</strong><span data-i18n="tasks">tasks per group</span></span>
|
| 28 |
+
<span><strong id="record-count">—</strong><span data-i18n="records">task records</span></span>
|
| 29 |
+
<span class="update" id="published-date"></span>
|
| 30 |
+
</div>
|
| 31 |
+
</header>
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
<section id="rankings" aria-labelledby="rankings-title">
|
| 35 |
+
<div class="section-head"><div><p class="eyebrow" data-i18n="resultsLabel">01 / RESULTS</p><h2 id="rankings-title" class="sr-only" data-i18n="rankings">Agent & Harness leaderboard</h2></div><span class="badge" data-i18n="research">Research results · provisional</span></div>
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
<div class="toolbar">
|
| 40 |
+
<div class="view-switch" role="group" aria-label="Leaderboard view">
|
| 41 |
+
<button type="button" data-view="all" aria-pressed="true" data-i18n="all">All results</button>
|
| 42 |
+
<button type="button" data-view="compare" aria-pressed="false" data-i18n="compare">Baseline / DeepAgents pairs</button>
|
| 43 |
+
<button type="button" data-view="baseline" aria-pressed="false" data-i18n="single">Single-pass Agent</button>
|
| 44 |
+
<button type="button" data-view="harness" aria-pressed="false" data-i18n="harness">All harness runs</button>
|
| 45 |
+
</div>
|
| 46 |
+
<a class="download" href="./leaderboard.csv" download data-i18n="download">Download CSV ↓</a>
|
| 47 |
+
</div>
|
| 48 |
+
<div class="filters">
|
| 49 |
+
<label class="metric-control"><span data-i18n="metric">Rank by</span><select id="metric">
|
| 50 |
+
<optgroup label="Local paper profile" data-group="local">
|
| 51 |
+
<option value="perfect" data-i18n="perfect">Full replication · local</option>
|
| 52 |
+
</optgroup>
|
| 53 |
+
<optgroup label="HF-style profile · parity unverified" data-group="hf">
|
| 54 |
+
<option value="partial_replication" data-i18n="partial_replication">Partial replication · HF-style</option>
|
| 55 |
+
<option value="compilation_success" data-i18n="compilation_success">Execution success · HF-style</option>
|
| 56 |
+
<option value="coefficient_direction" data-i18n="coefficient_direction">Coefficient direction · HF-style</option>
|
| 57 |
+
<option value="significance_level" data-i18n="significance_level">Significance level · HF-style</option>
|
| 58 |
+
</optgroup>
|
| 59 |
+
</select></label>
|
| 60 |
+
<label id="sort-control"><span data-i18n="sort">Sort by</span><select id="sort"><option value="deepagents" data-i18n="harnessScore">DeepAgents score</option><option value="baseline" data-i18n="singleScore">Single-pass score</option><option value="gain" data-i18n="lift">Lift</option></select></label>
|
| 61 |
+
<label class="search-control"><span data-i18n="search">Find a model</span><input id="search" type="search" placeholder="Search models…" autocomplete="off"></label>
|
| 62 |
+
</div>
|
| 63 |
+
<div class="display-tools">
|
| 64 |
+
<p id="metric-description" class="metric-description"></p>
|
| 65 |
+
<div class="display-switch" role="group" aria-label="Display format">
|
| 66 |
+
<button type="button" data-display="table" aria-pressed="true" data-i18n="tableView">Table</button>
|
| 67 |
+
<button type="button" data-display="mountain" aria-pressed="false" data-i18n="mountainView">Mountain view</button>
|
| 68 |
+
</div>
|
| 69 |
+
</div>
|
| 70 |
+
<p id="independent-summary" class="independent-summary" hidden><span class="independent-dot" aria-hidden="true"></span><span data-i18n="dshShort">DSH is an independent run with technical recovery, not a paired lift.</span> <a href="#protocol-details" id="show-protocol" data-i18n="detailsLink">Details ↗</a></p>
|
| 71 |
+
<figure id="ranking-mountain" class="mountain" hidden></figure>
|
| 72 |
+
<p id="load-state" role="status" class="load-state">Loading archived results…</p>
|
| 73 |
+
<div id="error" class="error" role="alert" hidden><p data-i18n="error">Results could not be loaded. Please retry or open the results archive.</p><button type="button" id="retry" data-i18n="retry">Retry</button></div>
|
| 74 |
+
<div id="table-wrap" class="table-wrap" tabindex="0" aria-label="Scrollable leaderboard" hidden>
|
| 75 |
+
<table><caption id="caption" class="sr-only"></caption><thead id="table-head"></thead><tbody id="table-body"></tbody></table>
|
| 76 |
+
</div>
|
| 77 |
+
<p id="empty" class="load-state" hidden data-i18n="empty">No matching models.</p>
|
| 78 |
+
<div class="table-foot"><span id="result-count" aria-live="polite"></span><span data-i18n="denominator">Every score uses all 1,000 tasks. Unknown and invalid outcomes earn no successes.</span></div>
|
| 79 |
+
<p class="publication-note" data-i18n="publicationNote">Research snapshot · local scoring; official scorer parity unverified. Run budgets differ.</p>
|
| 80 |
+
</section>
|
| 81 |
+
|
| 82 |
+
<section class="method" aria-label="Evaluation details">
|
| 83 |
+
<details id="protocol-details" class="evidence">
|
| 84 |
+
<summary data-i18n="evaluationDetails">Evaluation protocol, definitions & evidence</summary>
|
| 85 |
+
<p class="scope-note" data-i18n="scope">Locally scored, archived results. Official scorer parity is not yet verified. Different budgets and run protocols mean the lift is descriptive, not an equal-cost comparison.</p>
|
| 86 |
+
<section class="protocols" aria-label="Experimental settings">
|
| 87 |
+
<div><p class="protocol-tag baseline"><span aria-hidden="true">A</span><strong data-i18n="single">Single-pass Agent</strong></p><p data-i18n="singleDesc">One code-generation call, followed by execution. No iterative agent loop.</p></div>
|
| 88 |
+
<div><p class="protocol-tag harness"><span aria-hidden="true">B</span><strong data-i18n="deepagents">DeepAgents</strong></p><p data-i18n="harnessDesc">DeepAgents plans, uses tools and revises code. Up to 6 model calls, plus final execution.</p></div>
|
| 89 |
+
<div><p class="protocol-tag independent"><span aria-hidden="true">C</span><strong data-i18n="dsh">DeepSeek Harness · separate run</strong></p><p data-i18n="dshDesc">Official DSH SDK, requested v4-pro API alias. A fixed technical recovery selection; underlying model weights are unverified.</p></div>
|
| 90 |
+
</section>
|
| 91 |
+
<aside id="independent-notice" class="independent-notice" hidden><strong data-i18n="dshNoticeTitle">DSH is an independent result.</strong><p data-i18n="dshNotice">The request used deepseek-v4-pro; the underlying model weights were not verified. The fixed 1,000-task selection includes 87 technical replacement attempts (27 with changed input-schema prompts), with 30 task outcomes still unknown. It is excluded from the paired DeepAgents lift.</p><a id="dsh-protocol" data-i18n="dshProtocol" target="_blank" rel="noopener noreferrer">Read run protocol & evidence ↗</a></aside>
|
| 92 |
+
<div class="method-grid">
|
| 93 |
+
<article><h3 data-i18n="fixedTasks">Fixed task set</h3><p data-i18n="fixedDesc">The same Selected_1000 task IDs and dataset revision are used for every group. Each score is successes ÷ 1,000; failed or unknown tasks stay in the denominator.</p></article>
|
| 94 |
+
<article><h3 data-i18n="separateMetrics">Two scoring profiles</h3><p data-i18n="profilesDesc">Full replication is the local paper metric. The four HF-style metrics reproduce the published descriptions locally; they are not yet verified against the official scorer.</p></article>
|
| 95 |
+
<article><h3 data-i18n="budgetTitle">Read lift with the budgets</h3><p data-i18n="budgetDesc">Baseline: 1 model call. DeepAgents and DSH: up to 6 model calls and 4 trial tool calls per attempt. DSH adds 87 recovery attempts; full environment and reasoning parity is unverified. DSH has no paired lift; these are not equal-cost causal estimates.</p></article>
|
| 96 |
+
</div>
|
| 97 |
+
<h3 class="evidence-heading" data-i18n="evidence">Evidence status & metric definitions</h3>
|
| 98 |
+
<p data-i18n="unknownDesc">“Metric unknown” means the selected metric cannot be assessed, including failed execution or unavailable outputs. It is different from a task whose final status is unknown. Neither is dropped or counted as a success.</p>
|
| 99 |
+
<ul id="definitions"></ul>
|
| 100 |
+
<div class="evidence-table-wrap"><table class="evidence-table"><caption data-i18n="evidenceCaption">Status from each accepted archive</caption><thead><tr><th data-i18n="model">Model / API alias</th><th data-i18n="protocol">Protocol</th><th data-i18n="evidenceCol">Evidence</th></tr></thead><tbody id="evidence-body"></tbody></table></div>
|
| 101 |
+
<p data-i18n="verification">HF-style counts are checked against archived task flags. Full-replication counts for Sol and Opus come from accepted summaries; all other displayed runs, including DSH, have per-task local flags. This refresh does not rerun or rescore any experiment.</p>
|
| 102 |
+
<p data-i18n="archiveNote">Per-run archive files remain frozen, including their original publication metadata. This page is a new display derived from those archives. GPT-5.5 and unarchived experiments are not included in this edition.</p>
|
| 103 |
+
|
| 104 |
+
</details>
|
| 105 |
+
<a class="historical" href="./legacy.html" data-i18n="historical">Historical leaderboard ↗</a>
|
| 106 |
+
</section>
|
| 107 |
+
<footer><span>InferenceNet <span class="muted">/</span> <span data-i18n="footer">Econometric replication, measured.</span></span><div><a id="revision-link" href="https://huggingface.co/spaces/CamoAiLab/InferenceNet-Leaderboard/tree/main/results" target="_blank" rel="noopener noreferrer" data-i18n="source">Source snapshot ↗</a><a href="./leaderboard-data.json" data-i18n="json">Data JSON ↗</a><a href="https://easonai-5589.github.io/inferencenet-challenge/" target="_blank" rel="noopener noreferrer" data-i18n="project">Project ↗</a></div></footer>
|
| 108 |
+
</main>
|
| 109 |
+
<noscript><p>This leaderboard needs JavaScript. <a href="./leaderboard.csv">Download the results CSV</a> or <a href="https://huggingface.co/spaces/CamoAiLab/InferenceNet-Leaderboard/tree/main/results">browse the archives</a>.</p></noscript>
|
| 110 |
+
</body>
|
| 111 |
+
</html>
|
leaderboard.js
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
|
|
| 1 |
import { mountain } from './mountain.js';
|
| 2 |
'use strict';
|
| 3 |
const $ = id => document.getElementById(id);
|
|
@@ -69,8 +70,6 @@ try { language = localStorage.getItem('inferencenet-language') === 'zh' ? 'zh' :
|
|
| 69 |
let data = null;
|
| 70 |
let view = 'all';
|
| 71 |
let display = 'table';
|
| 72 |
-
let theme = 'light';
|
| 73 |
-
try { theme = localStorage.getItem('inferencenet-theme') === 'dark' ? 'dark' : 'light'; } catch (_) {}
|
| 74 |
let sortBy = 'deepagents';
|
| 75 |
const t = key => (language === 'zh' ? zh[key] : en[key]) ?? en[key] ?? key;
|
| 76 |
const esc = value => String(value).replace(/[&<>"']/g, char => ({'&':'&', '<':'<', '>':'>', '"':'"', "'":'''}[char]));
|
|
@@ -104,13 +103,6 @@ Object.assign(zh, {
|
|
| 104 |
metric: '排名指标', single: '单次生成', harness: '模型 + Harness', compare: '配对提升',
|
| 105 |
});
|
| 106 |
const shortKeys = ['fullShort', 'partialShort', 'executionShort', 'directionShort', 'significanceShort'];
|
| 107 |
-
function applyTheme() {
|
| 108 |
-
document.documentElement.dataset.theme = theme;
|
| 109 |
-
$('theme').setAttribute('aria-pressed', String(theme === 'dark'));
|
| 110 |
-
$('theme-label').textContent = t(theme);
|
| 111 |
-
$('theme').setAttribute('aria-label', language === 'zh' ? `切换到${theme === 'light' ? '深色' : '浅色'}主题` : `Switch to ${theme === 'light' ? 'dark' : 'light'} theme`);
|
| 112 |
-
}
|
| 113 |
-
|
| 114 |
function applyLanguage() {
|
| 115 |
document.documentElement.lang = language === 'zh' ? 'zh-CN' : 'en';
|
| 116 |
document.querySelectorAll('[data-i18n]').forEach(el => { el.innerHTML = t(el.dataset.i18n); });
|
|
@@ -121,7 +113,6 @@ function applyLanguage() {
|
|
| 121 |
document.querySelector('[data-group=local]').label = t('groupLocal');
|
| 122 |
document.querySelector('[data-group=hf]').label = t('groupHF');
|
| 123 |
$('definitions').innerHTML = metricKeys.map(key => `<li><strong>${esc(t(key))}:</strong> ${esc(t(key + 'Desc'))}</li>`).join('');
|
| 124 |
-
applyTheme();
|
| 125 |
render();
|
| 126 |
}
|
| 127 |
|
|
@@ -278,11 +269,6 @@ $('metric').addEventListener('change', render);
|
|
| 278 |
$('sort').addEventListener('change', event => { sortBy = event.target.value; render(); });
|
| 279 |
$('search').addEventListener('input', render);
|
| 280 |
$('retry').addEventListener('click', loadData);
|
| 281 |
-
$('language').addEventListener('click', () => {
|
| 282 |
-
language = language === 'en' ? 'zh' : 'en';
|
| 283 |
-
try { localStorage.setItem('inferencenet-language', language); } catch (_) {}
|
| 284 |
-
applyLanguage();
|
| 285 |
-
});
|
| 286 |
function setDisplay(next) {
|
| 287 |
display = next;
|
| 288 |
document.querySelectorAll('[data-display]').forEach(el => el.setAttribute('aria-pressed', String(el.dataset.display === next)));
|
|
@@ -296,10 +282,6 @@ $('table-head').addEventListener('click', event => {
|
|
| 296 |
document.addEventListener('click', event => {
|
| 297 |
if (event.target.closest('a[href="#protocol-details"]')) $('protocol-details').open = true;
|
| 298 |
});
|
| 299 |
-
|
| 300 |
-
theme = theme === 'light' ? 'dark' : 'light';
|
| 301 |
-
try { localStorage.setItem('inferencenet-theme', theme); } catch (_) {}
|
| 302 |
-
applyTheme();
|
| 303 |
-
});
|
| 304 |
applyLanguage();
|
| 305 |
loadData();
|
|
|
|
| 1 |
+
import { initSite } from './site-shell.js';
|
| 2 |
import { mountain } from './mountain.js';
|
| 3 |
'use strict';
|
| 4 |
const $ = id => document.getElementById(id);
|
|
|
|
| 70 |
let data = null;
|
| 71 |
let view = 'all';
|
| 72 |
let display = 'table';
|
|
|
|
|
|
|
| 73 |
let sortBy = 'deepagents';
|
| 74 |
const t = key => (language === 'zh' ? zh[key] : en[key]) ?? en[key] ?? key;
|
| 75 |
const esc = value => String(value).replace(/[&<>"']/g, char => ({'&':'&', '<':'<', '>':'>', '"':'"', "'":'''}[char]));
|
|
|
|
| 103 |
metric: '排名指标', single: '单次生成', harness: '模型 + Harness', compare: '配对提升',
|
| 104 |
});
|
| 105 |
const shortKeys = ['fullShort', 'partialShort', 'executionShort', 'directionShort', 'significanceShort'];
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
function applyLanguage() {
|
| 107 |
document.documentElement.lang = language === 'zh' ? 'zh-CN' : 'en';
|
| 108 |
document.querySelectorAll('[data-i18n]').forEach(el => { el.innerHTML = t(el.dataset.i18n); });
|
|
|
|
| 113 |
document.querySelector('[data-group=local]').label = t('groupLocal');
|
| 114 |
document.querySelector('[data-group=hf]').label = t('groupHF');
|
| 115 |
$('definitions').innerHTML = metricKeys.map(key => `<li><strong>${esc(t(key))}:</strong> ${esc(t(key + 'Desc'))}</li>`).join('');
|
|
|
|
| 116 |
render();
|
| 117 |
}
|
| 118 |
|
|
|
|
| 269 |
$('sort').addEventListener('change', event => { sortBy = event.target.value; render(); });
|
| 270 |
$('search').addEventListener('input', render);
|
| 271 |
$('retry').addEventListener('click', loadData);
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 272 |
function setDisplay(next) {
|
| 273 |
display = next;
|
| 274 |
document.querySelectorAll('[data-display]').forEach(el => el.setAttribute('aria-pressed', String(el.dataset.display === next)));
|
|
|
|
| 282 |
document.addEventListener('click', event => {
|
| 283 |
if (event.target.closest('a[href="#protocol-details"]')) $('protocol-details').open = true;
|
| 284 |
});
|
| 285 |
+
initSite({ onLanguageChange: next => { language = next; applyLanguage(); } });
|
|
|
|
|
|
|
|
|
|
|
|
|
| 286 |
applyLanguage();
|
| 287 |
loadData();
|
portal.css
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/* Shared project portal. Typography and colors inherit the leaderboard tokens. */
|
| 2 |
+
.nav-links>a[aria-current=page]{color:var(--ink);text-decoration:underline;text-underline-offset:8px;text-decoration-thickness:1px}.nav-links>a:nth-child(3){display:block}.nav-links{gap:22px}.portal-hero{padding:64px 0 48px;border-bottom:1px solid var(--line)}.portal-hero h1{max-width:870px;margin-top:18px;font-size:clamp(36px,4.4vw,62px);font-weight:600;line-height:1.09;letter-spacing:-.045em;overflow-wrap:break-word}.portal-lead{font-size:17px;color:var(--soft);line-height:1.8;max-width:850px;margin-top:20px}.hero-line{font-size:19px;color:var(--ink);margin-top:26px;max-width:510px}.home-hero{display:grid;grid-template-columns:1.1fr 1fr;gap:70px;align-items:center;padding-block:65px}.home-hero .portal-lead{font-size:15px;margin-top:13px;max-width:520px}.actions{display:flex;gap:12px 22px;align-items:center;flex-wrap:wrap;margin-top:30px}.action{display:inline-flex;padding:10px 17px;border:1px solid var(--strong-line);border-radius:3px;font-size:13px;align-items:center;min-height:42px}.action:hover{text-decoration:none;background:var(--tint)}.action.primary{background:var(--ink);color:var(--paper);border-color:var(--ink)}.action.primary:hover{opacity:.85}.text-link{text-decoration:underline;text-underline-offset:5px;font-size:13px}.task-figure{border:1px solid var(--line);background:var(--tint);padding:26px 28px;border-radius:4px}.figure-label{display:flex;justify-content:space-between;gap:15px;font-size:9px;letter-spacing:.08em;color:var(--muted)}.figure-label>span:last-child{font-family:var(--mono)}.figure-question{padding:22px 0}.figure-question h2{font-size:21px;font-weight:550;letter-spacing:-.025em;line-height:1.4}.figure-question p{font-size:10px;color:var(--muted);margin-top:5px}.figure-stage{display:flex;justify-content:center;align-items:center;gap:14px;font-size:11px;padding:14px 5px;border-top:1px solid var(--line)}.file-chip{font-family:var(--mono);padding:4px 7px;background:var(--surface);border:1px solid var(--line);border-radius:3px}.figure-arrow{text-align:center;color:var(--muted);font-size:16px;margin:2px 0}.figure-model{padding:10px;border:1px solid var(--strong-line);text-align:center;background:var(--surface);font-size:12px;letter-spacing:.02em}.output-fields{display:flex;gap:8px;justify-content:center;margin:10px 0 22px;flex-wrap:wrap}.output-fields>span{font:10px var(--mono);padding:5px 8px;border:1px solid var(--line);border-radius:3px;background:var(--surface)}.figure-link{font-size:11px;color:var(--soft);text-decoration:underline;text-underline-offset:4px}.scale-strip{display:grid;grid-template-columns:repeat(4,1fr);gap:24px;margin-top:35px}.scale-strip>div{border-left:1px solid var(--line);padding-left:24px;display:flex;flex-direction:column;gap:3px}.scale-strip>div:first-child{border:0;padding:0}.scale-strip strong{font-size:32px;font-weight:550;letter-spacing:-.035em;line-height:1.3;font-variant-numeric:tabular-nums}.scale-strip span{font-size:12px;color:var(--muted)}.source-note{font-size:11px;color:var(--muted);line-height:1.8;margin-top:20px;max-width:970px}.portal-section{padding:43px 0 40px;border-top:1px solid var(--line);margin-top:40px}.portal-section+.portal-section{margin-top:0}.portal-heading{margin-bottom:28px}.portal-heading h2{font-size:clamp(24px,2.6vw,32px);line-height:1.3;font-weight:550;letter-spacing:-.035em;margin-top:10px}.workflow{padding:0;margin:0;list-style:none;display:grid;grid-template-columns:repeat(4,1fr);gap:32px}.workflow li{position:relative;min-width:0}.step-number{display:flex;align-items:center;justify-content:center;width:26px;height:26px;border:1px solid var(--strong-line);border-radius:50%;font:10px var(--mono);margin-bottom:17px}.workflow li:not(:last-child)::after{content:'→';position:absolute;top:0;left:50%;color:var(--muted)}.workflow h3{font-size:15px;font-weight:550;margin-bottom:7px}.workflow p{font-size:12px;color:var(--soft);max-width:220px}.entry-grid{display:grid;grid-template-columns:repeat(3,1fr);gap:24px}.entry{position:relative;display:block;border-top:1px solid var(--strong-line);padding:24px 28px 24px 0}.entry:hover{text-decoration:none}.entry:hover h3{text-decoration:underline;text-underline-offset:5px}.entry h3{font-size:21px;letter-spacing:-.03em;font-weight:550;margin:12px 0 8px}.entry p{font-size:13px;color:var(--soft);max-width:290px}.entry-arrow{position:absolute;top:28px;right:0;color:var(--muted)}.release-strip{display:flex;justify-content:space-between;gap:24px;align-items:center;padding:26px 30px;background:var(--tint);border:1px solid var(--line);border-radius:3px}.release-count{font-size:21px;margin-top:8px;letter-spacing:-.02em}.release-count strong{font-weight:600}.release-strip p:not(.eyebrow){font-size:12px;color:var(--soft);margin-top:8px}.release-strip .snapshot-date{font-size:10px!important;color:var(--muted)!important}.distribution-grid{display:grid;grid-template-columns:1fr 1fr;gap:65px}.distribution h3{font-size:19px;font-weight:550}.distribution figcaption p{font-size:12px;color:var(--muted);margin-top:4px}.bar-chart{margin-top:27px}.bar-row{display:grid;grid-template-columns:1fr auto;gap:9px;margin-bottom:18px;align-items:baseline}.bar-label{font-size:12px;line-height:1.4}.bar-count{font-size:11px;color:var(--muted);font-variant-numeric:tabular-nums;white-space:nowrap}.bar-track{grid-column:1/-1;height:6px;background:var(--tint);display:block;border-radius:1px}.bar-track>span{display:block;height:100%;background:var(--ink);border-radius:1px}.journal-rest{font-size:12px;margin-top:25px}.journal-rest summary{cursor:pointer;text-decoration:underline;text-underline-offset:4px}.journal-rest table{font-size:11px;margin-top:15px}.journal-rest caption{text-align:left;padding:10px 0;color:var(--muted)}.journal-rest th,.journal-rest td{padding:8px 4px}.journal-rest tbody th{font-weight:400}.journal-rest td{white-space:nowrap;text-align:right}.anatomy-grid{display:grid;grid-template-columns:repeat(3,1fr);gap:32px}.anatomy-grid h3{font-size:16px;font-weight:550}.anatomy-grid p{font-size:13px;color:var(--soft);margin-top:12px;line-height:1.8}.case-summary{display:grid;grid-template-columns:1fr 1fr;gap:65px}.case-summary h3{font-size:22px;font-weight:550;margin:12px 0}.case-summary p:not(.eyebrow){font-size:14px;color:var(--soft);margin-bottom:22px}.compact-spec{display:grid;grid-template-columns:auto 1fr;gap:12px 24px;font-size:12px;margin:0;align-content:start}.compact-spec dt{color:var(--muted)}dd{margin:0;min-width:0;overflow-wrap:anywhere}code{font-family:var(--mono);font-size:.9em}.provenance{display:grid;grid-template-columns:130px 1fr;gap:18px 24px;font-size:13px;margin:0}.provenance dt{color:var(--muted)}.profile-grid{display:grid;grid-template-columns:repeat(3,1fr);gap:25px}.profile{border:1px solid var(--line);padding:24px;display:flex;flex-direction:column}.profile h3{font-size:20px;font-weight:550;letter-spacing:-.025em;margin-top:17px}.profile-budget{font-size:14px;margin-top:9px}.profile>p:not(.eyebrow){font-size:13px;color:var(--soft);margin:16px 0 24px;line-height:1.8}.profile .text-link{margin-top:auto;font-size:11px}.agent-loop{border:1px solid var(--line);padding:28px;background:var(--tint)}.loop-feedback{border-top:1px dashed var(--strong-line);margin-top:27px;padding-top:18px;font-size:12px;text-align:center;color:var(--soft)}.case-pair{display:grid;grid-template-columns:1fr 1fr;gap:32px;margin-top:28px}.case-pair article{border-top:1px solid var(--strong-line);padding-top:22px}.case-pair h3{font-size:20px;font-weight:550;margin:12px 0}.case-pair p:not(.eyebrow){font-size:14px;color:var(--soft);line-height:1.8}.case-outcome{font-size:12px;margin-top:19px;font-weight:550}.artifact{display:grid;grid-template-columns:1fr 1fr;gap:40px;background:var(--tint);border:1px solid var(--line);padding:25px;margin-top:34px;align-items:center}.artifact h3{font-size:17px;font-weight:550}.artifact p{font-size:12px;color:var(--soft);margin-top:10px}.artifact pre{max-width:100%;overflow:auto;margin:0;font-size:12px;line-height:1.9;tab-size:2}.portal-section:last-child{padding-bottom:15px}#journal-table{overflow:auto}footer{flex-wrap:wrap}footer>div{flex-wrap:wrap}#main{min-height:60vh}
|
| 3 |
+
@media(max-width:1100px){.home-hero{gap:40px}.nav-links{gap:16px}.portal-hero h1{font-size:45px}.topbar{flex-wrap:wrap;gap:10px;padding:20px 0}.nav-links{flex-wrap:wrap}.distribution-grid,.case-summary{gap:35px}.profile{padding:20px}}
|
| 4 |
+
@media(max-width:720px){.home-hero{grid-template-columns:1fr;gap:35px}.portal-hero{padding:37px 0}.portal-hero h1{font-size:39px}.portal-lead{font-size:15px}.hero-line{font-size:17px}.task-figure{max-width:100%;padding:23px}.figure-question h2{font-size:23px}.scale-strip{grid-template-columns:1fr 1fr;gap:25px}.scale-strip>div:nth-child(3){padding-left:0;border-left:0}.scale-strip strong{font-size:30px}.workflow{grid-template-columns:1fr 1fr;gap:25px}.workflow li:nth-child(2)::after{content:none}.entry-grid,.distribution-grid,.anatomy-grid,.case-summary,.profile-grid,.case-pair,.artifact{grid-template-columns:1fr}.entry-grid{gap:6px}.entry{padding-bottom:26px}.entry p{max-width:100%}.entry-arrow{right:10px}.release-strip{align-items:start;flex-direction:column;padding:23px}.distribution-grid{gap:35px}.portal-section{padding-top:32px;margin-top:30px}.provenance{grid-template-columns:1fr;gap:6px}.provenance dd{margin-bottom:12px}.profile-grid{gap:15px}.profile{padding:23px}.artifact{gap:18px;padding:20px}.agent-loop{padding:20px}.figure-stage{flex-wrap:wrap}.nav-links{gap:13px;font-size:11px;width:100%}#language{margin-left:auto}.brand small{display:inline}.release-count{font-size:19px}}
|
| 5 |
+
@media(max-width:460px){.nav-links{gap:10px}.nav-links button{padding-inline:7px}.nav-links>a{font-size:10px}.nav-links #theme-label{display:none}.portal-hero h1{font-size:35px}.output-fields{gap:5px}.output-fields>span{font-size:9px;padding:5px}.compact-spec{gap:10px}.workflow{gap:24px 18px}.workflow p{font-size:11px}}
|
| 6 |
+
@media(prefers-reduced-motion:reduce){html{scroll-behavior:auto}}
|
portal.js
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import { initSite } from './site-shell.js';
|
| 2 |
+
const site = initSite({ onLanguageChange: render });
|
| 3 |
+
let snapshot = null, summary = null, failure = false;
|
| 4 |
+
const t = (en, zh) => site.language === 'zh' ? zh : en;
|
| 5 |
+
const number = value => value.toLocaleString('en-US');
|
| 6 |
+
function barChart(id, items, total) {
|
| 7 |
+
const container = document.getElementById(id);
|
| 8 |
+
container.replaceChildren();
|
| 9 |
+
for (const item of items) {
|
| 10 |
+
const row = document.createElement('div'); row.className = 'bar-row';
|
| 11 |
+
const label = document.createElement('span'); label.className = 'bar-label';
|
| 12 |
+
label.textContent = item.label === 'Other' ? t('Other', '其他') : item.label;
|
| 13 |
+
const count = document.createElement('span'); count.className = 'bar-count';
|
| 14 |
+
count.textContent = `${number(item.count)} · ${(item.count / total * 100).toFixed(1)}%`;
|
| 15 |
+
const track = document.createElement('span'); track.className = 'bar-track'; track.setAttribute('aria-hidden', 'true');
|
| 16 |
+
const fill = document.createElement('span'); fill.style.width = `${item.count / total * 100}%`;
|
| 17 |
+
track.append(fill); row.append(label, count, track); container.append(row);
|
| 18 |
+
}
|
| 19 |
+
}
|
| 20 |
+
function render() {
|
| 21 |
+
document.querySelectorAll('[data-snapshot]').forEach(el => {
|
| 22 |
+
if (!snapshot) {
|
| 23 |
+
if (el.dataset.snapshot === 'date' && failure) el.textContent = t('Snapshot unavailable. Open the leaderboard for details.', '快照暂不可用,请打开榜单查看。');
|
| 24 |
+
return;
|
| 25 |
+
}
|
| 26 |
+
const values = { groups: number(snapshot.groups), models: number(snapshot.models.length), date: t('Published ', '发布于 ') + snapshot.published_date };
|
| 27 |
+
el.textContent = values[el.dataset.snapshot];
|
| 28 |
+
});
|
| 29 |
+
if (!summary) {
|
| 30 |
+
if (failure) document.querySelectorAll('.bar-chart').forEach(el => { el.textContent = t('Distribution unavailable. Open the dataset or download the summary JSON below.', '分布暂不可用,请打开数据集或下载下方分布 JSON。'); });
|
| 31 |
+
return;
|
| 32 |
+
}
|
| 33 |
+
barChart('method-chart', summary.methods, summary.tasks);
|
| 34 |
+
barChart('journal-chart', summary.journals.slice(0, 5), summary.tasks);
|
| 35 |
+
const table = document.createElement('table');
|
| 36 |
+
const caption = document.createElement('caption'); caption.textContent = t('Tasks by journal · n = 1,000', '按期刊统计任务 · n = 1,000'); table.append(caption);
|
| 37 |
+
const thead = document.createElement('thead'), header = document.createElement('tr');
|
| 38 |
+
for (const label of [t('Journal','期刊'), t('Tasks','题数'), t('Share','比例')]) { const th = document.createElement('th'); th.scope = 'col'; th.textContent = label; header.append(th); }
|
| 39 |
+
thead.append(header); table.append(thead);
|
| 40 |
+
const tbody = document.createElement('tbody');
|
| 41 |
+
for (const item of summary.journals) {
|
| 42 |
+
const row = document.createElement('tr'), name = document.createElement('th'), count = document.createElement('td'), percent = document.createElement('td');
|
| 43 |
+
name.scope = 'row'; name.textContent = item.label; count.textContent = number(item.count); percent.textContent = (item.count / summary.tasks * 100).toFixed(1) + '%';
|
| 44 |
+
row.append(name,count,percent); tbody.append(row);
|
| 45 |
+
}
|
| 46 |
+
table.append(tbody); document.getElementById('journal-table').replaceChildren(table);
|
| 47 |
+
}
|
| 48 |
+
async function readJSON(path) {
|
| 49 |
+
const response = await fetch(path, { cache: 'no-cache' });
|
| 50 |
+
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
| 51 |
+
return response.json();
|
| 52 |
+
}
|
| 53 |
+
if (document.querySelector('[data-snapshot]')) {
|
| 54 |
+
readJSON('./leaderboard-data.json').then(data => {
|
| 55 |
+
if (!Array.isArray(data.models) || !Number.isInteger(data.groups) || data.records !== data.groups * data.tasks_per_group) throw new Error('Invalid snapshot');
|
| 56 |
+
snapshot = data; render();
|
| 57 |
+
}).catch(error => { console.error('Snapshot unavailable', error); failure = true; render(); });
|
| 58 |
+
}
|
| 59 |
+
if (document.getElementById('method-chart')) {
|
| 60 |
+
readJSON('./dataset-summary.json').then(data => {
|
| 61 |
+
if (data.tasks !== 1000 || data.methods.reduce((n, x) => n + x.count, 0) !== data.tasks || data.journals.reduce((n, x) => n + x.count, 0) !== data.tasks) throw new Error('Invalid distribution summary');
|
| 62 |
+
summary = data; render();
|
| 63 |
+
}).catch(error => { console.error('Distribution unavailable', error); failure = true; render(); });
|
| 64 |
+
}
|
site-shell.js
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Shared navigation preferences across the project portal and leaderboard.
|
| 2 |
+
export function initSite({ onLanguageChange = () => {} } = {}) {
|
| 3 |
+
let language = 'en', theme = 'light';
|
| 4 |
+
try {
|
| 5 |
+
language = localStorage.getItem('inferencenet-language') === 'zh' ? 'zh' : 'en';
|
| 6 |
+
theme = localStorage.getItem('inferencenet-theme') === 'dark' ? 'dark' : 'light';
|
| 7 |
+
} catch (_) { /* Storage is optional in embedded browsers. */ }
|
| 8 |
+
const languageButton = document.getElementById('language');
|
| 9 |
+
const themeButton = document.getElementById('theme');
|
| 10 |
+
function render() {
|
| 11 |
+
document.documentElement.lang = language === 'zh' ? 'zh-CN' : 'en';
|
| 12 |
+
document.documentElement.dataset.theme = theme;
|
| 13 |
+
document.querySelectorAll('[data-en][data-zh]').forEach(el => { el.textContent = el.dataset[language]; });
|
| 14 |
+
languageButton.textContent = language === 'en' ? '中文' : 'EN';
|
| 15 |
+
languageButton.setAttribute('aria-label', language === 'en' ? 'Switch to Chinese' : 'Switch to English');
|
| 16 |
+
themeButton.setAttribute('aria-pressed', String(theme === 'dark'));
|
| 17 |
+
document.getElementById('theme-label').textContent = language === 'zh' ? (theme === 'light' ? '浅色' : '深色') : (theme === 'light' ? 'Light' : 'Dark');
|
| 18 |
+
themeButton.setAttribute('aria-label', language === 'zh' ? `切换到${theme === 'light' ? '深色' : '浅色'}主题` : `Switch to ${theme === 'light' ? 'dark' : 'light'} theme`);
|
| 19 |
+
}
|
| 20 |
+
languageButton.addEventListener('click', () => {
|
| 21 |
+
language = language === 'en' ? 'zh' : 'en';
|
| 22 |
+
try { localStorage.setItem('inferencenet-language', language); } catch (_) {}
|
| 23 |
+
render();
|
| 24 |
+
onLanguageChange(language);
|
| 25 |
+
});
|
| 26 |
+
themeButton.addEventListener('click', () => {
|
| 27 |
+
theme = theme === 'light' ? 'dark' : 'light';
|
| 28 |
+
try { localStorage.setItem('inferencenet-theme', theme); } catch (_) {}
|
| 29 |
+
render();
|
| 30 |
+
});
|
| 31 |
+
render();
|
| 32 |
+
return { get language() { return language; } };
|
| 33 |
+
}
|