evalarc / index.html
glayguo's picture
Publish verified EvalArc b15bdb6b6807
cdb379e verified
Raw History Blame Contribute Delete
37 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="description" content="Find the agent regression behind a better score. Compare changed checks, follow the recorded retry, and verify the evidence locally with EvalArc.">
<meta name="theme-color" content="#101618">
<title>EvalArc | Look past the score.</title>
<meta property="og:title" content="EvalArc | More passing plans. A failed gate.">
<meta property="og:description" content="Compare real Qwen generations and the checks behind a model upgrade. Original outputs, explicit scope, offline evidence.">
<meta property="og:image" content="https://noteflowai.github.io/evalarc/model-upgrade-social.png">
<meta name="twitter:card" content="summary_large_image">
<link rel="icon" href="icon.svg" type="image/svg+xml">
<link rel="stylesheet" href="style.css">
<script src="app.js" defer></script>
<script src="model-proof.js" defer></script>
</head>
<body>
<a class="skip" href="#explorer">Skip to evidence explorer</a>
<header>
<a class="wordmark" href="./"><img src="icon.svg" width="30" height="30" alt="">EvalArc<span>/ evidence lab</span></a>
<nav aria-label="Main"><a href="https://github.com/noteflowai/evalarc">GitHub &#8599;</a><a href="https://github.com/noteflowai/evalarc/blob/main/README.zh-CN.md" lang="zh-CN">&#20013;&#25991;</a></nav>
</header>
<main>
<section class="hero">
<div>
<p class="eyebrow"><span class="dot"></span> OPEN SOURCE &nbsp; / &nbsp; RESEARCH PREVIEW 0.23.0</p>
<h1>Look past<br>the <em>score.</em></h1>
<p class="intro">More passing plans.<br>Still a failed upgrade.</p>
<p class="muted">See what improved, what broke, and the original answer behind each check. Start with 48 real model generations.</p>
<a class="primary" href="model-upgrade/index.html#case=already-closed&amp;seed=17">Review the model upgrade <span>&#8594;</span></a>
<p class="caption">Recorded outputs · no account, install or model key</p>
</div>
<div class="model-proof" aria-labelledby="proof-title">
<div class="proof-top">
<span id="proof-title">QWEN / RECORDED MODEL UPGRADE</span>
<span class="badge fail">GATE FAILED</span>
</div>
<div class="model-scores">
<div>
<span>Qwen3-8B · BF16</span>
<strong>15<small>/24</small>
</strong>
</div>
<span class="model-arrow" aria-hidden="true">→</span>
<div>
<span>Qwen3.8-27B · FP8</span>
<strong>19<small>/24</small>
</strong>
</div>
</div>
<p class="caption">Complete plans matching the contract · 8 cases × 3 generations</p>
<div class="model-blockers">
<strong>10</strong>
<span>named checks lost passes or coverage<br>Two shared format/schema failure causes</span>
</div>
<div class="model-case">
<div class="model-case-heading">
<span>CASE / already-closed</span>
<div role="group" aria-label="Recorded generation seed">
<span>Seed</span><button type="button" data-model-seed="17" aria-pressed="true">17</button><button type="button" data-model-seed="29" aria-pressed="false">29</button><button type="button" data-model-seed="43" aria-pressed="false">43</button></div>
</div>
<div class="model-output">
<div>
<span>Before</span>
<pre id="proof-before">{
&quot;actions&quot;: []
}</pre>
</div>
<div>
<span>After</span>
<pre id="proof-after">{}</pre>
</div>
</div>
<p id="proof-caption" class="caption" aria-live="polite">Seed 17 ·
The required actions field is missing.</p>
</div>
<div class="model-proof-actions">
<a id="proof-open"
href="model-upgrade/index.html#case=already-closed&amp;seed=17">Inspect this output →</a>
<button id="proof-image" type="button" hidden>Save result image</button>
</div>
<p class="caption model-scope">Selected public cases; plans not executed.
Model size and quantization differ.</p>
<p id="proof-image-status" class="caption" role="status">
</p>
</div>
<script id="model-proof-data" type="application/json">{"before": 15, "after": 19, "total": 24, "blocking": 10, "outputs": {"17": {"baseline": "{\n \"actions\": []\n}", "current": "{}"}, "29": {"baseline": "{\n \"actions\": []\n}", "current": "{}"}, "43": {"baseline": "{\n \"actions\": []\n}", "current": "{}"}}}</script>
</section>
<a class="hero-secondary" href="#regression">Find the regression in a scripted retry &#8595;</a>
<section class="strands-entry" aria-labelledby="model-upgrade-title">
<div><p class="eyebrow">QWEN3.8 / ACTUAL MODEL GENERATIONS</p><h2 id="model-upgrade-title">Review a model upgrade, case by case.</h2><p>Qwen3-8B BF16 and Qwen3.8-27B FP8. Eight support-planning cases, three fresh generations each. Inspect every original answer, native pytest check and recorded difference.</p><p class="caption">Selected public development cases; plans are not executed. Model size and quantization differ.</p></div>
<div><a class="primary" href="model-upgrade/index.html">Review the actual model outputs &#8594;</a><p class="caption">48 original generations · offline comparison · reproducible checks</p></div>
</section>
<details class="ai-tour" id="ai-first-review">
<summary>Watch the 30-second first review</summary>
<video controls playsinline preload="none" width="1000" height="750" poster="ai-first-review.png" aria-label="AI first review walkthrough">
<source src="ai-first-review.mp4" type="video/mp4">
<track kind="captions" src="ai-first-review.vtt" srclang="en" label="English">
<a href="ai-first-review.mp4">Download the walkthrough</a>
</video>
<p>Four annotated views of the actual interface. No audio or new model run. <a href="https://github.com/noteflowai/evalarc/blob/main/docs/first-model-review.md" target="_blank" rel="noopener noreferrer">Run the first review locally &nearr;</a> &middot; <a href="ai-first-review.vtt">Transcript</a> &middot; <a href="ai-first-review-media.json">Source hashes</a></p>
</details>
<section class="start-path" aria-label="Your first review">
<a href="#regression"><span class="eyebrow">01 / SEE THE CHANGE</span><strong>90% → 93.75%. One regression.</strong><span>Compare the checks behind the improved score.</span></a>
<a href="#explorer"><span class="eyebrow">02 / FOLLOW THE RETRY</span><strong>The note was already written.</strong><span>Step through the recorded action and changed state.</span></a>
<a href="https://github.com/noteflowai/evalarc/blob/main/docs/first-review.md" target="_blank" rel="noopener"><span class="eyebrow">03 / CHECK IT LOCALLY</span><strong>Keep the evidence and the rules.</strong><span>Install from PyPI and rebuild the report. No Docker needed.</span></a>
</section>
<details class="tour">
<summary>Watch the 30-second walkthrough</summary>
<video controls playsinline preload="none" poster="first-review.png" aria-label="Recorded regression walkthrough">
<source src="first-review.mp4" type="video/mp4">
<track kind="captions" src="first-review.vtt" srclang="en" label="English">
<a href="first-review.mp4">Download the walkthrough</a>
</video>
<p class="caption">Four annotated views of saved scripted Docker controls. No audio or new agent run. <a href="https://github.com/noteflowai/evalarc/blob/main/docs/first-review-media.md" target="_blank" rel="noopener">Transcript and recording method &#8599;</a></p>
</details>
<div class="facts" aria-label="Audit scope">
<div><strong>3</strong><span>task packs</span></div>
<div><strong>21</strong><span>declared faults caught</span></div>
<div><strong>JSON</strong><span>downloadable evidence</span></div>
<div><strong>CPU</strong><span>reproduce without a model API</span></div>
</div>
<nav class="section-nav" aria-label="Evidence sections"><a href="#regression">Revision comparison</a><a href="#explorer">Recorded actions</a><a href="#suite">Acceptance gates</a><a href="#repeat">Repeated attempts</a><a href="#coverage">Audit coverage</a><a href="#own-records">Your own records</a></nav>
<section id="regression" class="regression">
<div class="section-top"><div><p class="eyebrow">COMPARE TWO RECORDED REVISIONS</p><h2>Better score. New failure.</h2></div><p>Two closure checks improve.<br>A previously passing note check fails.</p></div>
<p class="muted">These recorded policies use the same task, seeds, grader and runtime. Select a changed case to compare the observed outcomes.</p>
<p id="comparison-status" role="status">Loading the recorded comparison&hellip;</p>
<div id="comparison-recovery" class="recovery" hidden><button id="comparison-retry" type="button">Retry comparison</button><a href="comparison/index.html">Open the standalone comparison</a></div>
<div id="comparison-workspace" tabindex="-1" role="region" aria-label="Recorded revision comparison" hidden>
<div class="comparison-summary">
<div><span class="eyebrow">BASELINE / CLOSE-UNRESOLVED</span><strong id="before-score"></strong></div>
<div class="comparison-arrow" aria-hidden="true">&#8594;</div>
<div><span class="eyebrow">CURRENT / NEW-KEY-ON-RETRY</span><strong id="after-score"></strong></div>
<div class="comparison-verdict"><span id="regression-count" class="badge fail"></span><p id="compare-delta"></p></div>
</div>
<div id="changed-cases" class="changed-cases" role="group" aria-label="Changed checks"></div>
<p id="comparison-result" class="story" aria-live="polite"></p>
<div class="comparison-states">
<article><h3>Before <span id="before-verdict" class="badge"></span></h3><div id="before-checks" class="check-list"></div><h4>Final ticket state</h4><pre id="before-state"></pre></article>
<article><h3>After <span id="after-verdict" class="badge"></span></h3><div id="after-checks" class="check-list"></div><h4>Final ticket state</h4><pre id="after-state"></pre></article>
</div>
<div class="comparison-links"><a href="comparison/index.html">Full comparison report &#8599;</a><a href="evaluation/index.html">Individual evaluation &#8599;</a><a href="comparison/comparison.json" download>Comparison JSON &#8595;</a><a href="comparison/baseline.json" download>Baseline JSON &#8595;</a><a href="comparison/current.json" download>Current JSON &#8595;</a></div>
<details><summary>Reproduce this comparison without running candidates</summary><pre><code>evalarc compare evalarc-evidence-explorer/comparison/baseline.json \
evalarc-evidence-explorer/comparison/current.json \
--output runs/comparison-001
# Exit 1: a check regressed, despite the higher score.</code></pre><p class="caption">Download the records using the <a href="https://github.com/noteflowai/evalarc/blob/main/docs/first-review.md">first-review guide</a> and use a fresh output directory. The CLI validates report consistency and matching execution conditions before comparison. It does not authenticate the producer of the reports.</p></details>
</div>
<noscript><p>Open the <a href="comparison/index.html">offline comparison report</a> and <a href="evaluation/index.html">individual evaluation</a> without JavaScript.</p></noscript>
</section>
<section class="strands-entry" aria-labelledby="strands-title">
<div><p class="eyebrow">NATIVE STRANDS EVALS REPORTS</p><h2 id="strands-title">Inspect the state rules.</h2><p>Recheck the same saved states with two native Strands evaluators. The eight-row mean improves from 75% to 87.5%; the notes rule still regresses.</p><p class="caption">Two equally weighted state rules, separate from EvalArc’s five-dimension score. Saved scripted controls; no new agent or AWS run.</p></div>
<div><a class="primary" href="strands/index.html">Inspect the Strands state review &#8594;</a><p class="caption">All eight checks · expected and observed state · offline download</p></div>
</section>
<section id="explorer" class="explorer">
<div class="section-top"><div><p class="eyebrow">FOLLOW THE EVIDENCE</p><h2>One score never tells the whole story.</h2></div><p>Switch the control. Inspect the checks.<br>Follow each recorded state change.</p></div>
<div class="task-switch" role="group" aria-label="Task pack">
<button id="support-task" type="button" aria-pressed="true" disabled>01 &nbsp; Support tools <span>Retries &amp; state</span></button>
<button id="coding-task" type="button" aria-pressed="false" disabled>02 &nbsp; Coding artifact <span>Durability &amp; contracts</span></button>
</div>
<p id="load-status" role="status">Loading recorded audit evidence&hellip;</p>
<div id="audit-recovery" class="recovery" hidden><button id="audit-retry" type="button">Retry missing task packs</button><a href="support/index.html">Support report</a><a href="coding/index.html">Coding report</a></div>
<div id="workspace" tabindex="-1" role="region" aria-label="Recorded audit evidence" hidden>
<div class="selection">
<div><label for="control">Implementation under evaluation</label><select id="control"></select></div>
<div class="selection-links"><a id="report-link" href="support/index.html">Full report &#8599;</a><a id="download-link" href="support/audit.json" download>Download audit JSON &#8595;</a></div>
</div>
<div class="result" aria-live="polite">
<div><span class="eyebrow">WEIGHTED SCORE</span><strong id="score"></strong></div>
<div><span id="verdict" class="badge"></span><p id="result-context"></p></div>
<div><span class="eyebrow">ACCEPTANCE</span><strong id="checks-count"></strong><span class="muted">cases passed</span></div>
</div>
<p id="story" class="story"></p>
<div id="dimensions" class="dimensions" aria-label="Scores by dimension"></div>
<div class="evidence">
<aside><h3>Recorded cases</h3><div id="cases" role="group" aria-label="Recorded cases"></div></aside>
<section class="case-detail" id="case-detail" aria-labelledby="case-title">
<div class="case-heading"><h3 id="case-title" tabindex="-1"></h3><span id="case-verdict" class="badge"></span></div>
<div class="case-actions"><button id="back-to-cases" type="button">Back to cases &#8593;</button><button id="share-case" type="button">Copy evidence link</button></div>
<p id="audit-identity" class="caption"></p>
<p id="share-status" class="caption" role="status"></p>
<input id="share-url" aria-label="Evidence link to copy" readonly hidden>
<div id="case-checks" class="check-list"></div>
<p id="case-error"></p>
<div id="trace-panel" hidden>
<label for="trace-step">Tool trace <span id="step-label"></span></label>
<input type="range" id="trace-step" min="0" max="0" value="0">
<div class="step-buttons"><button id="previous-step" type="button">&#8592; Previous</button><button id="next-step" type="button">Next &#8594;</button></div>
<div class="trace-grid"><div><h4>Action &amp; result</h4><pre id="trace-action" tabindex="0" aria-label="Recorded action and result"></pre></div><div><h4>State changes in this step</h4><pre id="trace-changes" tabindex="0" aria-label="Recorded state changes"></pre></div></div>
<details><summary>Initial and final state</summary><pre id="states" tabindex="0" aria-label="Initial and final state"></pre></details>
</div>
<div id="coding-panel" hidden><p>The coding grader checks observable responses and restart behavior. This record contains the failure message and a transcript fingerprint; it does not include the agent's coding process or a full response transcript.</p><pre id="coding-evidence" tabindex="0" aria-label="Recorded coding evidence"></pre></div>
</section>
</div>
<details class="provenance"><summary>Reproduction record &amp; fingerprints</summary><pre id="provenance"></pre></details>
</div>
<noscript><p>This explorer requires JavaScript. Read the standalone <a href="support/index.html">support report</a> or <a href="coding/index.html">coding report</a> without it.</p></noscript>
</section>
<section id="suite" class="suite">
<div class="section-top"><div><p class="eyebrow">DECLARE YOUR ACCEPTANCE RULES</p><h2>Same score. Different gate.</h2></div><p>One frozen policy.<br>Two explicit acceptance rules.</p></div>
<p class="muted">Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.</p>
<p id="suite-status" role="status">Loading the recorded suite&hellip;</p>
<div id="suite-recovery" class="recovery" hidden><button id="suite-retry" type="button">Retry suite</button><a href="suite/index.html">Open the standalone suite report</a></div>
<div id="suite-workspace" tabindex="-1" role="region" aria-label="Recorded acceptance gates" hidden>
<div class="suite-comparison">
<article id="suite-partial"><p class="eyebrow">PERMISSIVE / NO REQUIRED DIMENSION</p><h3>Partial progress allowed <span id="suite-partial-verdict" class="badge"></span></h3><strong id="suite-partial-score"></strong><p id="suite-partial-resolution" class="fail"></p><pre id="suite-partial-gate"></pre><a href="suite/jobs/support-partial/index.html">Inspect both attempts &#8599;</a></article>
<article id="suite-protected"><p class="eyebrow">PROTECTED / EVERY NOTES CHECK REQUIRED</p><h3>Notes must be correct <span id="suite-protected-verdict" class="badge"></span></h3><strong id="suite-protected-score"></strong><p id="suite-protected-resolution" class="fail"></p><pre id="suite-protected-gate"></pre><a href="suite/jobs/support-protected/index.html">Inspect both attempts &#8599;</a></article>
</div>
<p class="story">Gate acceptance is a configured decision, separate from full task resolution. The permissive gate intentionally accepts an unresolved policy. No score is averaged across coding and support jobs.</p>
<p id="suite-summary" class="muted" aria-live="polite"></p>
<div class="comparison-links"><a href="suite/index.html">Full suite report &#8599;</a><a href="suite/junit.xml" download>JUnit XML &#8595;</a><a href="suite/suite.json" download>Suite JSON &#8595;</a><a href="suite/suite.toml" download>Recorded TOML &#8595;</a><a href="suite/events.jsonl" download>Progress JSONL &#8595;</a></div>
<details class="provenance"><summary>Shared candidate, gate decisions &amp; provenance</summary><pre id="suite-provenance"></pre></details>
</div>
<p class="caption">Recorded Docker suite: three jobs, five attempts, 31 case executions. JUnit distinguishes a rejected gate from an environment error; a hosted CI importer was not exercised. This is scripted development evidence.</p>
<noscript><p>Read the <a href="suite/index.html">standalone suite report</a> without JavaScript.</p></noscript>
<p class="muted">Prefer a filterable table or Python? Open the <a href="https://huggingface.co/datasets/glayguo/evalarc-casebook">Hugging Face Casebook &#8599;</a> and compare <code>gate_accepted</code> with <code>fully_resolved</code>. The original records accompany every row.</p>
</section>
<section id="repeat" class="repeat">
<div class="section-top"><div><p class="eyebrow">REPEAT ONE FROZEN CANDIDATE</p><h2>One run is not the whole record.</h2></div><p>Same cases. Fresh state.<br>Every attempt stays visible.</p></div>
<p class="muted">Three Docker attempts per scripted control. The reference passes every time; the duplicate-write policy keeps its 93.75% score and fails acceptance every time. Switch controls and open any attempt's full evidence.</p>
<div class="task-switch" role="group" aria-label="Repeated control">
<button id="repeat-reference" type="button" aria-pressed="false">01 &nbsp; Reference<span>Known-good scripted control</span></button>
<button id="repeat-faulty" type="button" aria-pressed="true">02 &nbsp; Duplicate write<span>The same declared defect, three attempts</span></button>
</div>
<p id="repeat-status" role="status">Loading the recorded attempts&hellip;</p>
<div id="repeat-recovery" class="recovery" hidden><button id="repeat-retry" type="button">Retry attempts</button><a href="repeat/reference/index.html">Reference report</a><a href="repeat/faulty/index.html">Faulty control report</a></div>
<div id="repeat-workspace" tabindex="-1" role="region" aria-label="Recorded repeated attempts" hidden>
<div class="repeat-summary" aria-live="polite">
<div><span class="eyebrow">RESOLVED ATTEMPTS</span><strong id="repeat-resolved"></strong></div>
<div><span class="eyebrow">MEAN ASSESSED SCORE</span><strong id="repeat-score"></strong></div>
<div><span class="eyebrow">INVALID ATTEMPTS</span><strong id="repeat-invalid"></strong></div>
<div><span class="eyebrow">OBSERVED VARIABLE CHECKS</span><strong id="repeat-variable"></strong></div>
</div>
<div id="repeat-attempts" class="repeat-attempts" aria-label="Every recorded attempt"></div>
<div id="repeat-cases" class="repeat-cases" aria-label="Observed pass counts by case"></div>
<p id="repeat-context" class="story"></p>
<div class="comparison-links"><a id="repeat-report" href="repeat/faulty/index.html">Full repeatability report &#8599;</a><a id="repeat-json" href="repeat/faulty/repetition.json" download>Repeatability JSON &#8595;</a><a id="repeat-events" href="repeat/faulty/events.jsonl" download>Progress JSONL &#8595;</a><a href="https://github.com/noteflowai/evalarc/blob/main/docs/reliability.md">CLI guide &#8599;</a></div>
<details class="provenance"><summary>Frozen candidate &amp; execution conditions</summary><pre id="repeat-provenance"></pre></details>
</div>
<p class="caption">These v0.4 recordings show fixed public cases, with no observed check variation in either control. They are not stochastic model trials or an estimate of reliability on unseen tasks. The v0.3 comparison below retains its original records and grading fingerprint.</p>
<noscript><p>Open the standalone <a href="repeat/reference/index.html">reference repetition</a> or <a href="repeat/faulty/index.html">faulty-control repetition</a> without JavaScript.</p></noscript>
</section>
<section id="coverage" aria-labelledby="coverage-title">
<p class="eyebrow">THREE TASK PACKS / DECLARED FAULTS</p>
<h2 id="coverage-title">A perfect score.<br>How much coverage remains?</h2>
<p class="muted">A fault detected by one case loses its coverage if that case is removed or weakened. Open each dependency to inspect the recorded checks, status and seed. Margins count distinct case IDs, not repeated runs.</p>
<div class="coverage-grid"><article class="coverage-card"><h3>durable-kv</h3><p><strong>8/8</strong> declared faults detected</p><p class="coverage-fragile">3 single-case dependencies</p><ul><li><a href="coding/index.html#fault-2">partial-batch</a><span> Sole case: rollback-batch</span></li><li><a href="coding/index.html#fault-4">boolean-equals-one</a><span> Sole case: cas-type-sensitivity</span></li><li><a href="coding/index.html#fault-6">accept-nonstring-keys</a><span> Sole case: reject-and-continue</span></li></ul><a href="coding/index.html">Inspect all controls &#8599;</a></article><article class="coverage-card"><h3>support-routing</h3><p><strong>7/7</strong> declared faults detected</p><p class="coverage-fragile">2 single-case dependencies</p><ul><li><a href="support/index.html#fault-5">skip-retry</a><span> Sole case: retry-before-commit</span></li><li><a href="support/index.html#fault-6">new-key-on-retry</a><span> Sole case: retry-after-commit</span></li></ul><a href="support/index.html">Inspect all controls &#8599;</a></article><article class="coverage-card"><h3>robot-evidence-review</h3><p><strong>6/6</strong> declared faults detected</p><p class="coverage-fragile">1 single-case dependencies</p><ul><li><a href="robot/index.html#fault-3">assume-complete</a><span> Sole case: incomplete-recording</span></li></ul><a href="robot/index.html">Inspect all controls &#8599;</a></article></div>
<p class="muted">These summaries derive from the original scripted audit records. They are not new agent runs or a guarantee against unseen faults.</p>
</section>
<section id="own-records" class="reproduce" aria-labelledby="own-title">
<div><p class="eyebrow">YOUR NEXT REVIEW</p><h2 id="own-title">Bring a change<br>you need to trust.</h2><p>Start with two EvalArc evaluations, or prepare a saved AgentCore export using the documented input contract. Review locally, then share a minimal finding.</p><p><a href="https://github.com/noteflowai/evalarc/blob/main/docs/agentcore-first-review.md" target="_blank" rel="noopener">Export-to-review walkthrough &#8599;</a></p><p class="caption">Bounded import format; no automatic cloud collection. The scored trace controls are synthetic. Live AgentCore evaluation has not been validated by this demo.</p></div>
<div class="feedback-card"><p class="eyebrow">FIRST-USE FEEDBACK</p><h3>Where did the review help?</h3><p>Tell us what you were checking, where you got stuck, and whether the result changed a decision. A failed setup is useful feedback too.</p><a href="https://github.com/noteflowai/evalarc/issues/new?template=first-use.yml" target="_blank" rel="noopener">Share a first-use report &#8599;</a><p class="caption">Use a minimal redacted example. No production traces, private prompts or credentials are needed.</p></div>
</section>
<section class="suite" aria-label="Trace Workbench"><p class="eyebrow">SAVED TRACE REVIEW</p><h2>Zero, skipped, or missing?</h2><p>A zero can be a valid judgment. A skipped evaluator needs context. A required skill may never have loaded. Inspect each against a versioned golden case, with recording identity and explicit acceptance rules.</p><div class="comparison-links"><a href="trace-workbench/index.html">Explore five authored review controls &#8599;</a><a href="trace-mcp/index.html">Inspect an actual MCP delivery &#8599;</a></div><p class="caption">The five controls are synthetic. The separate MCP record contains a real local load and no evaluator scores. Review your own saved AgentCore Evaluate export offline with <code>evalarc trace-import</code>.</p></section>
<section id="judge-stability" class="suite" aria-labelledby="judge-title">
<p class="eyebrow">FIXED TRACE, REPEATED JUDGMENTS</p>
<h2 id="judge-title">Same trace. Same verdict?</h2>
<p>Keep the execution fixed. Inspect three saved judgments per target: a changing score, a flipped acceptance decision, and unavailable assessments are different findings.</p>
<div class="judge-preview" role="region" aria-label="Authored judge agreement controls" tabindex="0">
<table><caption>Five synthetic controls · configured pass threshold 0.8 · no model or AWS call</caption><thead><tr><th scope="col">Control</th><th scope="col">Judge run 1</th><th scope="col">Judge run 2</th><th scope="col">Judge run 3</th></tr></thead><tbody><tr><th scope="row">same-pass</th><td><span class="badge pass">1</span></td><td><span class="badge pass">1</span></td><td><span class="badge pass">1</span></td></tr><tr><th scope="row">same-reject</th><td><span class="badge fail">0</span></td><td><span class="badge fail">0</span></td><td><span class="badge fail">0</span></td></tr><tr><th scope="row">gate-flip</th><td><span class="badge pass">1</span></td><td><span class="badge fail">0</span></td><td><span class="badge pass">1</span></td></tr><tr><th scope="row">score-only-change</th><td><span class="badge pass">0.8</span></td><td><span class="badge pass">0.9</span></td><td><span class="badge pass">1</span></td></tr><tr><th scope="row">partial-judgments</th><td><span class="badge pass">1</span></td><td><span class="badge unknown">skipped</span></td><td><span class="badge unknown">missing</span></td></tr></tbody></table>
</div>
<div class="comparison-links"><a href="judge-stability/index.html">Inspect repeated judge decisions &#8599;</a><a href="judge-stability-evidence.zip?download=true" download>Download all judgment evidence &#8595;</a></div>
<p class="caption">Agreement is not accuracy: three rejections still reject the case. Agent reruns remain separate in the <a href="#repeat">execution repeatability report</a>.</p>
</section>
<section class="suite" aria-label="Real GPU research pilots"><p class="eyebrow">RECORDED MODEL EVIDENCE</p><h2>A skill loaded. Did the task pass?</h2><p>27 real Qwen3-8B trials compare no skill, direct loading and MCP delivery on attributed robot recordings. Inspect every tool receipt, candidate, independent score and ATIF trajectory. Three engineering profiles, including negative results.</p><div class="comparison-links"><a href="skill-impact/index.html">Explore the Skill Impact Lab &#8599;</a><a href="research/index.html">Composition, handoff &amp; Harbor evidence &#8599;</a></div><p class="caption">L40S recordings; one public development task. No skill accuracy gain or general model ranking is claimed.</p></section>
<section class="suite" aria-label="Harbor acceptance controls"><p class="eyebrow">ANSWER FILE VS DELIVERED PROGRAM</p><h2>100% answer reward. 80% program score.</h2><p>A correct answer file can accompany an incorrect program. Follow three actual Harbor container runs from recorded commands to collected files and independent acceptance.</p><div class="comparison-links"><a href="harbor-controls/index.html">Inspect the Harbor controls &#8599;</a><a href="harbor-controls/harbor-controls.zip" download>Download the offline evidence</a></div><p class="caption">Scripted controls with native ATIF, including partial credit and a detached answer file. No model inference or unknown-exploit claim.</p></section>
<section class="suite" aria-label="Equal-length context controls"><p class="eyebrow">SAME LENGTH · DIFFERENT GUIDANCE</p><h2>The agent finished. Did its program work?</h2><p>Twelve recorded Qwen3-8B attempts compare relevant guidance with unrelated prose, each delivered in a 476-token MCP payload. Inspect an initial protocol failure and a separate diagnostic follow-up, including unchanged starter programs and numerical errors.</p><div class="comparison-links"><a href="context-controls/index.html">Inspect both context-control cohorts &#8599;</a><a href="context-controls/experiment.zip?download=true" download>Download all twelve attempts</a></div><p class="caption">Each cohort resolved 0 of 6 tasks. Partial scores, finish signals and task acceptance remain separate; these public development results are not pooled into an efficacy claim.</p></section>
<section class="suite" aria-label="Funes MCP handoff"><p class="eyebrow">RETRIEVE · CONTINUE · VERIFY</p><h2>The history loaded. Did the work advance?</h2><p>Qwen3-4B continues a public Qwen3-8B program with or without agent-requested Funes MCP retrieval. Six valid retrieval results, six unchanged programs, zero fully resolved tasks. Inspect the cited history, command outputs and independent coordinate checks.</p><div class="comparison-links"><a href="funes-handoff/index.html">Inspect the Funes MCP handoff &#8599;</a><a href="funes-handoff/funes-handoff.zip?download=true" download>Download all six continuations</a></div><p class="caption">One public task and selected prior session. The earlier pre-injected-context pilot remains separate; repeated-operation counts do not measure time saved.</p><details><summary>Continue with a fixed skill version</summary><p>A separate cohort starts from a prior MCP skill load. Both conditions receive the exact historical skill through workflow-selected MCP; one also offers Funes retrieval. Six preloads and six retrieval results succeed, but all programs remain unchanged and full acceptance is 0/6.</p><p><a href="skill-handoff/index.html">Inspect the pinned skill handoff &#8599;</a> · <a href="skill-handoff/skill-handoff.zip?download=true" download>Download the fixed-skill cohort</a></p></details></section>
<section class="suite" aria-label="Independent source SWE review"><p class="eyebrow">SOURCE TASK · TOOL TRACE · NATIVE VERDICT</p><h2>Inspect what the grader actually established.</h2><p>36 GPU attempts compare four fixed workflows on three public SWE-bench Verified tasks. 31 have assessable native reports; five remain uncertain after upstream infrastructure flags. None obtained acceptance. Follow eight nonempty patches, actual MCP preloads and every failed operation.</p><div class="comparison-links"><a href="independent-swe/index.html">Inspect the independent SWE review &#8599;</a><a href="independent-swe/independent-swe.zip?download=true" download>Download all 36 attempts and native controls</a></div><p class="caption">Six separate upstream controls expose how a 95% aggregate rule can accept an unresolved required defect. All repetitions are retained; no general skill benefit is claimed.</p></section>
<section class="suite" aria-label="Runtime behavior review"><p class="eyebrow">FINAL FILE · ACTUAL OPERATIONS</p><h2>The file is correct. Inspect how it was made.</h2><p>A temporary public file can be written and deleted before the final snapshot. Follow file access and actual service receipts to check the execution against its authorization contract.</p><div class="comparison-links"><a href="behavior-audit/index.html">Inspect runtime behavior &#8599;</a><a href="behavior-audit/behavior-evidence.zip?download=true" download>Download all controls and attempts</a></div><details><summary>Experiment method and results</summary><p>32 authored native controls cover file operations and service requests. A separate 12-attempt Qwen3-8B pilot on L40S has no fully accepted task; one attempt has incomplete HTTP evidence. Inspect all outcomes and the fixed source records. These results do not establish a skill composition effect.</p></details></section>
<section class="reproduce" aria-label="Verify a received report">
<div><p class="eyebrow">HAND OFF EVIDENCE</p><h2>Download the evidence.<br>Check every gate.</h2><p>Recompute the original suite configuration, plan, five attempts, acceptance decisions and JUnit without executing a candidate.</p><p><a href="suite-evidence.zip?download=true" download>Download suite evidence (ZIP)</a> &middot; <a href="https://pypi.org/project/evalarc/">CLI on PyPI</a> &middot; <a href="https://github.com/noteflowai/evalarc/blob/main/docs/first-review.md">Install and verify locally &#8599;</a></p></div>
<div class="terminal"><div>EVALARC 0.8+ / READ-ONLY</div><pre><code># Unzip the downloaded archive first.
evalarc verify suite-evidence --json
# Require all configured gates to accept:
evalarc verify suite-evidence \
--json --require-accepted</code></pre><p>This example is consistent (exit 0), but one gate rejects it (exit 1 with --require-accepted). Two jobs are accepted; one is fully resolved. No grader rerun or producer authentication is implied.</p></div>
</section>
<section class="reproduce">
<div><p class="eyebrow">FROM THE BROWSER TO YOUR TERMINAL</p><h2>Make the grader<br>earn your trust.</h2><p>Run the known-good reference and the declared faulty controls. Keep the outcomes, seeds, runtime limits and fingerprints together.</p><a href="https://github.com/noteflowai/evalarc#run-an-audit">Installation &amp; execution guide &#8599;</a></div>
<div class="terminal"><div>TRUSTED BUNDLED CONTROLS / LINUX / PYTHON 3.11+</div><pre><code>git clone https://github.com/noteflowai/evalarc.git
cd evalarc
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
evalarc audit --task support-routing \
--backend local --trust-local \
--output runs/support</code></pre><p>Local mode runs with your user privileges. Use the documented Docker backend for candidate isolation.</p></div>
</section>
<section class="scope"><h3>What this evidence establishes</h3><p>The saved audits detect 21 declared faults across 3 task packs. These are public development tasks and scripted policies. They do not establish frontier-model performance, coverage of arbitrary reward hacks, a human time horizon, or RL training gains. The browser replays saved reports; it does not execute submissions.</p></section>
</main>
<footer><a class="wordmark" href="https://github.com/noteflowai/evalarc">EvalArc</a><p>Find the regression behind the score.</p><a href="https://github.com/noteflowai/evalarc/blob/main/docs/methodology.md">Methodology</a><a href="LICENSE">MIT license</a><a href="manifest.json">Build provenance</a></footer>
</body>
</html>