Spaces:
Running
Running
| <html lang="en"><head><meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width,initial-scale=1"> | |
| <title>Same length, different guidance · EvalArc</title> | |
| <style> | |
| :root{color-scheme:dark}body{font:17px/1.65 system-ui,sans-serif;background:#101725; | |
| color:#e6ecf5;max-width:1040px;margin:auto;padding:32px 20px}a{color:#8ee1dc} | |
| a:focus-visible{outline:3px solid #ffc87d}h1{font-size:clamp(30px,5vw,48px);line-height:1.2} | |
| h2{font-size:23px}p{max-width:80ch}.grid{display:grid; | |
| grid-template-columns:repeat(auto-fit,minmax(min(100%,300px),1fr));gap:20px} | |
| article,.note{padding:18px 22px;background:#1c293c;border:1px solid #3c4961;border-radius:8px} | |
| article h2{margin:0}dl{display:grid;grid-template-columns:1fr 1fr;gap:6px} | |
| dd{margin:0;text-align:right}dt{color:#b1c4dd}strong{color:#ffc87d} | |
| code{overflow-wrap:anywhere}details{margin:24px 0}summary{cursor:pointer} | |
| </style></head><body> | |
| <nav><a href="https://noteflowai.github.io/evalarc/context-controls/index.html"> | |
| Both cohorts online</a> · | |
| <a href="context-controls.zip?download=true" download>Offline cohort evidence</a></nav> | |
| <h1>Same length. Different guidance.</h1> | |
| <p><strong>Initial cohort: an EOF example is the supplied execution check.</strong></p> | |
| <p>Six actual Qwen3-8B trials compare relevant robot-review guidance with unrelated | |
| descriptive prose, delivered through the same Skills Anywhere MCP route. | |
| Both skill-load results contain <strong>476 tokens</strong>, including hashes.</p> | |
| <p class="note">The model, initial prompt, task, catalog description, tools and budgets | |
| are fixed. The loaded content changes. EvalArc independently executes the generated | |
| or unchanged starter programs in Docker; a successful skill load alone does not pass the task.</p> | |
| <section class="note"><h2>0 / 6 tasks resolved</h2><p>48 / 48 recorded cases had a response timeout. An EOF example can appear to work while a persistent JSONL service waits on buffered output. The original failures remain here; the subsequent cohort supplies a protocol diagnostic to both conditions.</p><p>A harness stop of <code>finished</code> means the agent called finish. The independent grader decides whether the task was resolved.</p></section> | |
| <div class="grid"><article><h2>relevant · seed 17</h2><dl><dt>Program score</dt><dd>0.0%</dd><dt>Task resolved</dt><dd>No</dd><dt>Harness stop</dt><dd>finished</dd><dt>Skill-open calls</dt><dd>1</dd><dt>Generated tokens</dt><dd>1,671</dd><dt>Summed input tokens</dt><dd>48,112</dd><dt>Harness wall time</dt><dd>79.3 s</dd></dl><details><summary>Inspect 8 case outcomes</summary><ul><li>seed 41 / world-meters: response timeout</li><li>seed 41 / millimeters-sensor-frame: response timeout</li><li>seed 41 / centimeters-offset-clock: response timeout</li><li>seed 41 / incomplete-recording: response timeout</li><li>seed 97 / world-meters: response timeout</li><li>seed 97 / millimeters-sensor-frame: response timeout</li><li>seed 97 / centimeters-offset-clock: response timeout</li><li>seed 97 / incomplete-recording: response timeout</li></ul></details><p><a href="relevant/01-mcp-17/trial.json">Original trial</a> · <a href="relevant/01-mcp-17/candidate/main.py">Program</a> · <a href="relevant/01-mcp-17/evaluation.json">Independent grade</a></p></article><article><h2>neutral · seed 17</h2><dl><dt>Program score</dt><dd>0.0%</dd><dt>Task resolved</dt><dd>No</dd><dt>Harness stop</dt><dd>finished</dd><dt>Skill-open calls</dt><dd>1</dd><dt>Generated tokens</dt><dd>2,829</dd><dt>Summed input tokens</dt><dd>59,980</dd><dt>Harness wall time</dt><dd>127.6 s</dd></dl><details><summary>Inspect 8 case outcomes</summary><ul><li>seed 41 / world-meters: response timeout</li><li>seed 41 / millimeters-sensor-frame: response timeout</li><li>seed 41 / centimeters-offset-clock: response timeout</li><li>seed 41 / incomplete-recording: response timeout</li><li>seed 97 / world-meters: response timeout</li><li>seed 97 / millimeters-sensor-frame: response timeout</li><li>seed 97 / centimeters-offset-clock: response timeout</li><li>seed 97 / incomplete-recording: response timeout</li></ul></details><p><a href="neutral/02-mcp-17/trial.json">Original trial</a> · <a href="neutral/02-mcp-17/candidate/main.py">Program</a> · <a href="neutral/02-mcp-17/evaluation.json">Independent grade</a></p></article><article><h2>neutral · seed 41</h2><dl><dt>Program score</dt><dd>0.0%</dd><dt>Task resolved</dt><dd>No</dd><dt>Harness stop</dt><dd>finished</dd><dt>Skill-open calls</dt><dd>1</dd><dt>Generated tokens</dt><dd>3,336</dd><dt>Summed input tokens</dt><dd>67,223</dd><dt>Harness wall time</dt><dd>149.7 s</dd></dl><details><summary>Inspect 8 case outcomes</summary><ul><li>seed 41 / world-meters: response timeout</li><li>seed 41 / millimeters-sensor-frame: response timeout</li><li>seed 41 / centimeters-offset-clock: response timeout</li><li>seed 41 / incomplete-recording: response timeout</li><li>seed 97 / world-meters: response timeout</li><li>seed 97 / millimeters-sensor-frame: response timeout</li><li>seed 97 / centimeters-offset-clock: response timeout</li><li>seed 97 / incomplete-recording: response timeout</li></ul></details><p><a href="neutral/03-mcp-41/trial.json">Original trial</a> · <a href="neutral/03-mcp-41/candidate/main.py">Program</a> · <a href="neutral/03-mcp-41/evaluation.json">Independent grade</a></p></article><article><h2>relevant · seed 41</h2><dl><dt>Program score</dt><dd>0.0%</dd><dt>Task resolved</dt><dd>No</dd><dt>Harness stop</dt><dd>finished</dd><dt>Skill-open calls</dt><dd>1</dd><dt>Generated tokens</dt><dd>1,549</dd><dt>Summed input tokens</dt><dd>47,626</dd><dt>Harness wall time</dt><dd>72.2 s</dd></dl><details><summary>Inspect 8 case outcomes</summary><ul><li>seed 41 / world-meters: response timeout</li><li>seed 41 / millimeters-sensor-frame: response timeout</li><li>seed 41 / centimeters-offset-clock: response timeout</li><li>seed 41 / incomplete-recording: response timeout</li><li>seed 97 / world-meters: response timeout</li><li>seed 97 / millimeters-sensor-frame: response timeout</li><li>seed 97 / centimeters-offset-clock: response timeout</li><li>seed 97 / incomplete-recording: response timeout</li></ul></details><p><a href="relevant/04-mcp-41/trial.json">Original trial</a> · <a href="relevant/04-mcp-41/candidate/main.py">Program</a> · <a href="relevant/04-mcp-41/evaluation.json">Independent grade</a></p></article><article><h2>relevant · seed 97</h2><dl><dt>Program score</dt><dd>0.0%</dd><dt>Task resolved</dt><dd>No</dd><dt>Harness stop</dt><dd>finished</dd><dt>Skill-open calls</dt><dd>1</dd><dt>Generated tokens</dt><dd>2,735</dd><dt>Summed input tokens</dt><dd>59,944</dd><dt>Harness wall time</dt><dd>123.0 s</dd></dl><details><summary>Inspect 8 case outcomes</summary><ul><li>seed 41 / world-meters: response timeout</li><li>seed 41 / millimeters-sensor-frame: response timeout</li><li>seed 41 / centimeters-offset-clock: response timeout</li><li>seed 41 / incomplete-recording: response timeout</li><li>seed 97 / world-meters: response timeout</li><li>seed 97 / millimeters-sensor-frame: response timeout</li><li>seed 97 / centimeters-offset-clock: response timeout</li><li>seed 97 / incomplete-recording: response timeout</li></ul></details><p><a href="relevant/05-mcp-97/trial.json">Original trial</a> · <a href="relevant/05-mcp-97/candidate/main.py">Program</a> · <a href="relevant/05-mcp-97/evaluation.json">Independent grade</a></p></article><article><h2>neutral · seed 97</h2><dl><dt>Program score</dt><dd>0.0%</dd><dt>Task resolved</dt><dd>No</dd><dt>Harness stop</dt><dd>finished</dd><dt>Skill-open calls</dt><dd>1</dd><dt>Generated tokens</dt><dd>5,646</dd><dt>Summed input tokens</dt><dd>87,948</dd><dt>Harness wall time</dt><dd>260.5 s</dd></dl><details><summary>Inspect 8 case outcomes</summary><ul><li>seed 41 / world-meters: response timeout</li><li>seed 41 / millimeters-sensor-frame: response timeout</li><li>seed 41 / centimeters-offset-clock: response timeout</li><li>seed 41 / incomplete-recording: response timeout</li><li>seed 97 / world-meters: response timeout</li><li>seed 97 / millimeters-sensor-frame: response timeout</li><li>seed 97 / centimeters-offset-clock: response timeout</li><li>seed 97 / incomplete-recording: response timeout</li></ul></details><p><a href="neutral/06-mcp-97/trial.json">Original trial</a> · <a href="neutral/06-mcp-97/candidate/main.py">Program</a> · <a href="neutral/06-mcp-97/evaluation.json">Independent grade</a></p></article></div> | |
| <details><summary>What is matched, and what can still differ?</summary> | |
| <p>The match covers the JSON tool-result payload using the pinned Qwen3 tokenizer. | |
| It includes the skill name, description, instruction body and file/bundle hashes. | |
| Real MCP preflight results must equal the prepared payloads before inference.</p> | |
| <p>Later generated programs, messages, tool calls and total token usage can differ. | |
| Seeds 17, 41 and 97 each receive both conditions in alternating order. Each trial | |
| has 12 model turns, up to 4096 generated tokens per turn and a 600-second wall budget.</p> | |
| <p>This is one public development task with three model seeds, not six independent | |
| tasks or a held-out benchmark. The earlier 27-trial engineering profiles use their | |
| own recorded context and library versions; their results are not pooled here.</p></details> | |
| <h2>Review every attempt</h2> | |
| <p><a href="experiment.json">Preselected plan</a> · | |
| <a href="summary.json">All six results</a> · | |
| <a href="preparation/match.json">Both exact skill payloads</a> · | |
| <a href="tokenization-check.json">Native tokenizer check</a> · | |
| <a href="README.md">Methods and commands</a> · | |
| <a href="manifest.json">File identities</a></p> | |
| <p>The page replays saved evidence. It does not call a model or execute a program. | |
| Negative outcomes remain in the download. These small controls do not establish | |
| a general skill benefit, context-length effect size, or client ranking.</p> | |
| </body></html> | |