Spaces:
Running
Running
Add SchemeArena logo (nav, hero, favicon) and paper Figure 5 to Results
Browse files- .gitattributes +1 -0
- assets/figure5_scheming_rates.png +3 -0
- assets/logo.svg +24 -0
- css/main.css +5 -0
- index.html +12 -8
.gitattributes
CHANGED
|
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
assets/framework.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
assets/framework.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
assets/figure5_scheming_rates.png filter=lfs diff=lfs merge=lfs -text
|
assets/figure5_scheming_rates.png
ADDED
|
Git LFS Details
|
assets/logo.svg
ADDED
|
|
css/main.css
CHANGED
|
@@ -454,3 +454,8 @@ table.data.fixed col.lab { width: 30%; }
|
|
| 454 |
|
| 455 |
.verdict { align-items: flex-start; font-weight: 500; line-height: 1.5; }
|
| 456 |
.verdict svg { flex: none; margin-top: 2px; }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 454 |
|
| 455 |
.verdict { align-items: flex-start; font-weight: 500; line-height: 1.5; }
|
| 456 |
.verdict svg { flex: none; margin-top: 2px; }
|
| 457 |
+
|
| 458 |
+
.brand-mark { width: 28px; height: 28px; object-fit: contain; }
|
| 459 |
+
.hero-logo { display: block; width: 84px; height: auto; margin: 0 auto 18px; filter: drop-shadow(0 6px 14px rgba(75, 86, 98, .18)); }
|
| 460 |
+
.result-fig { display: block; width: 100%; height: auto; border-radius: 8px; margin-top: 6px; }
|
| 461 |
+
@media (max-width: 760px) { .hero-logo { width: 64px; margin-bottom: 14px; } }
|
index.html
CHANGED
|
@@ -8,19 +8,14 @@
|
|
| 8 |
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
| 9 |
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
| 10 |
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;600;700&family=Source+Serif+4:opsz,wght@8..60,500;8..60,600&display=swap" rel="stylesheet" />
|
|
|
|
| 11 |
<link rel="stylesheet" href="css/main.css" />
|
| 12 |
</head>
|
| 13 |
<body>
|
| 14 |
|
| 15 |
<nav id="topNav">
|
| 16 |
<a class="nav-brand" href="#overview">
|
| 17 |
-
<
|
| 18 |
-
<rect x="1" y="3" width="30" height="27" rx="8" fill="#b9470f"/>
|
| 19 |
-
<rect x="5" y="11" width="22" height="8" rx="4" fill="#15171b"/>
|
| 20 |
-
<circle cx="11" cy="15" r="2" fill="#f5b38c"/><circle cx="21" cy="15" r="2" fill="#f5b38c"/>
|
| 21 |
-
<rect x="11" y="23" width="10" height="2.4" rx="1.2" fill="#fbeee5"/>
|
| 22 |
-
<rect x="15" y="0" width="2" height="4" rx="1" fill="#b9470f"/>
|
| 23 |
-
</svg>
|
| 24 |
SchemeArena
|
| 25 |
</a>
|
| 26 |
<div class="nav-tabs" role="tablist">
|
|
@@ -46,6 +41,7 @@
|
|
| 46 |
<section id="tab-overview" class="tab-panel active">
|
| 47 |
|
| 48 |
<header class="hero">
|
|
|
|
| 49 |
<span class="pill"><span class="dot"></span>Benchmark · AI Safety · LLM Agents</span>
|
| 50 |
<h1><span class="name">SchemeArena</span>: Factorized Stress Testing of Scheming in LLM Agents</h1>
|
| 51 |
<p class="subtitle">A 400-scenario benchmark that varies instrumental goals, pressure, strategic hints, and oversight to study when LLM agents covertly pursue misaligned goals.</p>
|
|
@@ -325,11 +321,19 @@
|
|
| 325 |
</div>
|
| 326 |
</section>
|
| 327 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 328 |
<section class="block" style="margin-top:20px">
|
| 329 |
<div class="card chart-card">
|
| 330 |
<div class="chart-head"><h3>Scheming capability (Setups 5–7)</h3></div>
|
| 331 |
<div class="table-wrap"><table class="data fixed" id="capTable"></table></div>
|
| 332 |
-
<p class="footnote">Values reported in the text of the paper (§5.3); changes are relative to Setup 5. "—":
|
| 333 |
</div>
|
| 334 |
</section>
|
| 335 |
|
|
|
|
| 8 |
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
| 9 |
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
| 10 |
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;600;700&family=Source+Serif+4:opsz,wght@8..60,500;8..60,600&display=swap" rel="stylesheet" />
|
| 11 |
+
<link rel="icon" type="image/svg+xml" href="assets/logo.svg" />
|
| 12 |
<link rel="stylesheet" href="css/main.css" />
|
| 13 |
</head>
|
| 14 |
<body>
|
| 15 |
|
| 16 |
<nav id="topNav">
|
| 17 |
<a class="nav-brand" href="#overview">
|
| 18 |
+
<img class="brand-mark" src="assets/logo.svg" alt="" />
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
SchemeArena
|
| 20 |
</a>
|
| 21 |
<div class="nav-tabs" role="tablist">
|
|
|
|
| 41 |
<section id="tab-overview" class="tab-panel active">
|
| 42 |
|
| 43 |
<header class="hero">
|
| 44 |
+
<img class="hero-logo" src="assets/logo.svg" alt="SchemeArena logo" />
|
| 45 |
<span class="pill"><span class="dot"></span>Benchmark · AI Safety · LLM Agents</span>
|
| 46 |
<h1><span class="name">SchemeArena</span>: Factorized Stress Testing of Scheming in LLM Agents</h1>
|
| 47 |
<p class="subtitle">A 400-scenario benchmark that varies instrumental goals, pressure, strategic hints, and oversight to study when LLM agents covertly pursue misaligned goals.</p>
|
|
|
|
| 321 |
</div>
|
| 322 |
</section>
|
| 323 |
|
| 324 |
+
<section class="block" style="margin-top:20px">
|
| 325 |
+
<div class="card chart-card">
|
| 326 |
+
<div class="chart-head"><h3>Scheming rates across all seven setups</h3></div>
|
| 327 |
+
<img class="result-fig" src="assets/figure5_scheming_rates.png" alt="Line chart of scheming rates with error bars for o4-mini, o1, Claude-3.7-Sonnet, Qwen3-8B and Qwen3-32B across Setups 1 to 7, with Setup 6 split into action-only and full oversight" loading="lazy" />
|
| 328 |
+
<p class="footnote">Scheming rates for all setups that evaluate propensity (blue background) and capability (yellow background). Strategic hints (Setups 5 and 7) strongly enable scheming, especially for Qwen models. Action-only monitoring (Setup 6 w/ Action) slightly increases scheming among the closed models, while full-trajectory oversight yields only limited reductions. Error bars: 95% bootstrap confidence intervals. (Paper, Figure 5.)</p>
|
| 329 |
+
</div>
|
| 330 |
+
</section>
|
| 331 |
+
|
| 332 |
<section class="block" style="margin-top:20px">
|
| 333 |
<div class="card chart-card">
|
| 334 |
<div class="chart-head"><h3>Scheming capability (Setups 5–7)</h3></div>
|
| 335 |
<div class="table-wrap"><table class="data fixed" id="capTable"></table></div>
|
| 336 |
+
<p class="footnote">Values reported in the text of the paper (§5.3); changes are relative to Setup 5. "—": not stated numerically in the text; see the figure above, where the Qwen models stay near saturation across all oversight conditions. Setup 6 here is action-only oversight.</p>
|
| 337 |
</div>
|
| 338 |
</section>
|
| 339 |
|