Add a Findings write-up: what we ran, what happened, and what broke

app.py

@@ -24,6 +24,8 @@ from pathlib import Path
 
 import gradio as gr
 
+from findings import FINDINGS, FINDINGS_CSS
+
 sys.path.insert(0, str(Path(__file__).resolve().parent))
 
 BUDGET = 100.0
@@ -426,7 +428,7 @@ same tools against a stand-in, which needs no game server.</p></div>
 
 # ---------------------------------------------------------------- the page
 
-CSS = """
+CSS = FINDINGS_CSS + """
 .gradio-container { max-width: 1120px !important; }
 #pr-hero { border-bottom: 1px solid var(--border-color-primary); padding-bottom: 18px; }
 #pr-hero h1 { font-size: 30px; line-height: 1.15; margin: 0 0 6px; letter-spacing: -.02em; font-weight: 700; }
@@ -567,6 +569,8 @@ with gr.Blocks(title="Prime Rush — a battle royale as an MCP environment") as
             gr.Markdown(WHY)
         with gr.Column():
             gr.Markdown(NOTE)
+    with gr.Tab("Findings"):
+        gr.HTML(FINDINGS)
     with gr.Tab("Leaderboard"):
         gr.Markdown(BOARD_INTRO)
         gr.HTML(render_board())

findings.py

@@ -0,0 +1,189 @@
+"""The write-up tab: what we ran, what happened, and what broke.
+
+Long-form on purpose. A benchmark table says which agent scored what; it
+cannot say that the metric rewarded standing still until someone checked, or
+that a model diagnosed a bug in the environment while playing it. Those are
+the findings, and they only exist in prose.
+
+Everything quoted here is verbatim from a trajectory in the published
+dataset, and every number is from a run in it.
+"""
+
+from __future__ import annotations
+
+FINDINGS_CSS = """
+#fx { max-width: 760px; }
+#fx h2 { font-size: 23px; margin: 34px 0 6px; letter-spacing: -.01em; }
+#fx h3 { font-size: 16px; margin: 24px 0 4px; }
+#fx p { font-size: 15px; line-height: 1.62; margin: 0 0 13px; }
+#fx .lede { font-size: 17px; line-height: 1.6; color: var(--body-text-color-subdued); }
+#fx blockquote { border-left: 2px solid #2a78d6; margin: 14px 0; padding: 2px 0 2px 14px;
+  font-size: 14.5px; line-height: 1.55; color: var(--body-text-color-subdued); }
+#fx blockquote .who { display: block; margin-top: 5px; font-size: 12px;
+  font-family: ui-monospace, monospace; }
+#fx pre { background: var(--background-fill-secondary); border-radius: 8px; padding: 13px 15px;
+  font-size: 12.5px; line-height: 1.5; overflow-x: auto; }
+#fx .num { counter-reset: f; list-style: none; padding: 0; }
+#fx .num > li { counter-increment: f; position: relative; padding-left: 34px; margin-bottom: 22px; }
+#fx .num > li::before { content: counter(f); position: absolute; left: 0; top: 1px;
+  width: 23px; height: 23px; line-height: 23px; text-align: center; border-radius: 50%;
+  background: var(--body-text-color); color: var(--background-fill-primary);
+  font-size: 12px; font-weight: 700; font-family: ui-sans-serif, system-ui; }
+#fx .num > li > b { display: block; font-size: 15.5px; margin-bottom: 4px; }
+#fx hr { border: 0; border-top: 1px solid var(--border-color-primary); margin: 30px 0; }
+"""
+
+FINDINGS = """
+<div id="fx">
+
+<h2>Agents in a shipped battle royale</h2>
+
+<p class="lede">We exposed a live match of a commercial battle royale to LLM agents as
+six MCP tools, and ran five models and two reference policies across four tasks. Three
+tasks made it onto the board. This is what happened, including the parts where the
+environment was wrong rather than the models.</p>
+
+<p>The game is Prime Rush, a battle royale that real people play. The agent takes over a
+player part-way through a match, after the game's own bot AI has played the approach, and
+is scored on a window of a few hundred decisions. It is given its own coordinates and the
+safe circle's, and never a direction — working out where to go is the whole task.</p>
+
+<p>The match only advances inside a tool call. A model can think for thirty seconds and
+the game is exactly where it left it, so none of what follows is a measurement of
+latency.</p>
+
+<hr>
+
+<h2>Findings</h2>
+
+<ol class="num">
+
+<li><b>A twelve-line script matches frontier models at staying in the circle.</b>
+<p>On zone survival, a hand-written policy — face the circle's centre, run, hold — scores
+65.4%. Claude Opus scores 77.0% and GPT-5.5 73.7%. A paired bootstrap over the same 30
+zones does not separate any of the top four from each other. The task is real, but it
+does not rank models, and we publish it saying so.</p></li>
+
+<li><b>One task does separate them, and it is about memory, not space.</b>
+<p>In <i>waypoint return</i> the agent is told its starting coordinates once, on its first
+look, and never again. It must travel 250 units away and come back. The hand-written
+policy scores 4.9%; random scores 0.0%; Opus scores 60.2% and GPT-5.5 59.2%. The paired
+comparison separates Opus from DeepSeek-V4-Flash (+29.1 points, interval 2.6 to 53.2),
+from Sonnet (+36.8) and from the script (+55.3).</p>
+<p>Nothing to do with the game's geometry. The task is whether a model carries a number
+across tool calls, and it is the only thing we have measured here that a short program
+cannot do.</p></li>
+
+<li><b>The same model can be first on one task and last on another.</b>
+<p>GPT-5.5 ties Opus on waypoint return with the best median on the board, 96.3%. On
+storm escape — handed the controls already outside the circle and losing health — it
+scores 8.0%, below a random policy's 15.4%, with no errors and no failed episodes. It
+simply did not run for the circle.</p></li>
+
+<li><b>Models overstate their own results.</b>
+<p>Asked to report what they achieved, agents are consistently generous. Claude Sonnet,
+on a zone it scored 82%:</p>
+<blockquote>Spent the vast majority (well over 90%) of the available time inside the
+circle it was closing to.<span class="who">claude-sonnet-5-5 · zone 2007 · measured 82%</span></blockquote>
+<p>Qwen3.6-35B, on a zone it scored 62%:</p>
+<blockquote>I was inside the closing circle for approximately 173 out of 200 seconds
+(about 87%).<span class="who">Qwen3.6-35B-A3B · zone 2004 · measured 62%</span></blockquote>
+<p>Neither is lying so much as estimating without a clock. It is the argument for reading
+the score out of the game state rather than the transcript, which is what the grader
+does.</p></li>
+
+<li><b>Agents report the environment's bugs back to you.</b>
+<p>Before we fixed the handover, a player could be dead before the agent received it. One
+run worked out the arithmetic unprompted:</p>
+<blockquote>Spawned ~337 units outside the circle with a movement speed of only
+~0.4-0.75 units/sec and storm damage of ~4-7% health/sec; the circle is unreachable
+(would need ~800s of travel vs ~2s of remaining health), so I am stopping rather than
+burning the clock.<span class="who">claude-opus-5-5 · zone 2027</span></blockquote>
+<p>Other runs identified terrain we did not know was impassable:</p>
+<blockquote>Reached and held the target circle for most of the match, but got wedged in
+impassable terrain and could not reach the final 65m circle.<span class="who">claude-opus-5-5 · zone 2020 · 67%</span></blockquote></li>
+
+<li><b>Knowing when to stop is part of the skill.</b>
+<p>The highest-scoring runs are not the ones that never fail. They are the ones that
+recognise a lost position early and stop paying for it:</p>
+<blockquote>Stuck behind terrain; final circle (r=65) is ~130 units away with 12s left —
+unreachable. Spent the great majority of the match inside the target circle.<span class="who">claude-opus-5-5 · zone 2014 · 70%</span></blockquote></li>
+
+</ol>
+
+<hr>
+
+<h2>What went wrong</h2>
+
+<p>Four of the mistakes were ours, and each one produced a plausible-looking result
+before anyone noticed.</p>
+
+<h3>The metric rewarded standing still</h3>
+<p>We scored time spent inside the circle the agent occupied. But the agent inherits a bot
+that already walked somewhere sensible, so doing nothing scored well — a random policy
+took 76%, and no handover radius changed it: 350 gave 76%, 250 gave 73%, 150 gave 76%,
+with no spread at any of them. Scoring against the circle the storm is <i>closing to</i>
+dropped random to 13.7% and lifted the hand-written policy to 78.5%.</p>
+
+<h3>The agent was dead before it played</h3>
+<p>Attaching replaces a bot's controller, and we attached at minute zero — so the seat
+stood still through seven minutes of approach while the storm closed. Of 30 zones, 18
+handed over a damaged player and every one of those 18 scored zero. Eight were already
+dead. Attaching late, after the bot has played the approach, fixed it.</p>
+
+<h3>The turn tool under-rotated by thirteen times</h3>
+<p>The game's look input is not degrees. Measured across 29 episodes, a requested turn
+arrived at 0.075 of its angle. Models compensated by calling the tool repeatedly and
+strafing sideways instead of turning, which we had read as indecision. Worse, the
+stand-in server used for tests applied the input faithfully, so the broken tool passed
+every test. The stand-in now reproduces the real response.</p>
+
+<h3>A task was dropped on a bug, not a result</h3>
+<p>Placement declared that its episodes should run to the end of the match. Nothing read
+that flag. Scores came from a mid-match rank in a seven-player lobby, landed on 0.0, 0.2,
+0.4 and 0.6, and looked like a task with no signal. With the horizon honoured and a
+twenty-player lobby, the reference policies separate by 14 points.</p>
+
+<hr>
+
+<h2>What we could not build</h2>
+
+<p>A looting task is the cleanest trade-off the game offers: gather as much as you can
+before the circle closes, where greed kills you and caution scores nothing. It is not
+buildable, because the observation contains no items. Holding the loot action across
+350 units of travel collects nothing, and an agent cannot go to loot it cannot see. The
+fix is a nearest-items list shaped like the existing nearest-enemies list.</p>
+
+<p>Combat tasks are blocked on two defects in the game rather than the harness: the pitch
+input is inverted, and the resulting aim pitch is not reported back, so an agent cannot
+check what its own aim did. Until both are fixed the shoot tool is horizontal only.</p>
+
+<hr>
+
+<h2>Open questions</h2>
+
+<p>Whether the separation on waypoint return survives more zones. At twenty, Opus and
+GPT-5.5 are tied and the gap to DeepSeek is wide but soft.</p>
+
+<p>Whether a task exists here that separates models on spatial reasoning rather than
+memory. So far every spatial task is matched by a short script.</p>
+
+<p>What happens with several agents in one match. The game build supports multiple seats;
+the harness does not use them yet. Model against model, same match, same seed, is the
+experiment we want to run next.</p>
+
+<p>Whether an environment whose simulation waits for the agent changes how a model plays
+at all, compared with one that does not.</p>
+
+<hr>
+
+<h2>The data</h2>
+
+<p>Every episode behind every number — per-zone scores, token counts, and the full
+sequence of tool calls and responses — is published alongside this page. The tables are
+on the leaderboard tab; the raw results are in the dataset.</p>
+
+<p>Built from the live build of Prime Rush by SuperGaming. Apache-2.0.</p>
+
+</div>
+"""