Add OFT All 140k to Easy, Hard and Plus with verified completion counts

README.md

@@ -52,3 +52,9 @@ Single-episode coverage is not the upstream multi-trial protocol; vision-noise i
 The [Plus panel](https://rle-bench-programmable-vla.static.hf.space/plus/index.html) now uses the Easy/Hard task browser, comparison table, episode tabs, video player, evidence panels and VLA input/output viewer. All **1,200** videos use the same head + left wrist + right wrist layout, 2880 × 720 output, H.264 CRF 20, 40 FPS / 4× action-index playback, posters and last-frame controls. Native 320 × 240 camera views are upscaled; wrist observations update at inference boundaries. No new observations or benchmark outcomes were generated.
 
 All original scores (94/400, 222/400, 294/400), native videos, evidence ZIPs, seeds and instructions are unchanged. VLA I/O includes every recorded call and byte-identical input/source PNGs. [Presentation protocol](https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/robotwin-plus/20261101-native/presentation-v2/presentation-protocol.json). Dataset revision: `b2d67a67dd2f9d8e12c1f8db13ba8de309739883`.
+
+## Non-VLAct OFT All 140k
+
+The Easy, Hard and Plus panels include `StarVLA/Qwen3-VL-OFT-RoboTwin2-All` (140k; 27,500 clean/randomized training demonstrations; 50-action chunks). Each panel preserves its established instruction and native evaluation protocol. Completion counts are reported separately from successes. All prior episodes remain unchanged.
+
+[Campaign evidence](https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/summary.json).

app.js

@@ -12,7 +12,7 @@ const compact = v => v == null ? '—' : Math.abs(v) >= 1e9 ? (v/1e9).toFixed(2)
 const pct = v => v == null ? '—' : (v*100).toFixed(1)+'%';
 const elapsed = v => v == null ? '—' : v >= 3600 ? (v/3600).toFixed(1)+' h' : v >= 60 ? (v/60).toFixed(1)+' min' : v.toFixed(1)+' s';
 const arrow = '<span aria-hidden="true">↗</span>';
-const colors = {'standalone-vlact-clean':'#a75f7f','standalone-vlact-all':'#bd733b','standalone-official':'#39819c','standalone':'#39819c','kinex_policy':'#756ee4','kinex':'#27847a','codex-astra-spectator':'#a76a34','codex-astra':'#39819c','kinex-astra':'#756ee4','cc-opus':'#b48566','kinex-opus':'#958bc8','cc-opus-vertex-high':'#c9875a','cc-opus-vertex-xhigh':'#8c5a3a','kinex-opus-vertex-high':'#5b8bd8','kinex-opus-vertex-xhigh':'#3a5ba6','cc-opus-vertex-high-run2':'#e0a37a','cc-opus-vertex-xhigh-run2':'#a8704a','kinex-opus-vertex-high-run2':'#86aae6','kinex-opus-vertex-xhigh-run2':'#5876c0'};
+const colors = {'standalone-oft-all140k':'#547a36','standalone-vlact-clean':'#a75f7f','standalone-vlact-all':'#bd733b','standalone-official':'#39819c','standalone':'#39819c','kinex_policy':'#756ee4','kinex':'#27847a','codex-astra-spectator':'#a76a34','codex-astra':'#39819c','kinex-astra':'#756ee4','cc-opus':'#b48566','kinex-opus':'#958bc8','cc-opus-vertex-high':'#c9875a','cc-opus-vertex-xhigh':'#8c5a3a','kinex-opus-vertex-high':'#5b8bd8','kinex-opus-vertex-xhigh':'#3a5ba6','cc-opus-vertex-high-run2':'#e0a37a','cc-opus-vertex-xhigh-run2':'#a8704a','kinex-opus-vertex-high-run2':'#86aae6','kinex-opus-vertex-xhigh-run2':'#5876c0'};
 let data, epoch = 0, current = {}, matrixMetric = 'success';
 let filters = {query:'',family:'',status:'',variant:'standalone-vlact-clean',sort:'catalog'};
 let sessionState = {filter:'all',query:'',count:40,view:'timeline'};
@@ -98,7 +98,7 @@ function comparisonCell(v,f) {
 
 function comparisonTable() {
   const scope='Per-task averages divide totals by the number of valid evaluated tasks in each environment and variant, including passes and failures. Pending and invalid tasks are excluded; missing values remain —.';
-  return `<section class="comparison-panel" aria-labelledby="comparison-title"><div class="section-heading bare"><div><h2 id="comparison-title">Compare environments</h2><p>All three standalone checkpoints use official RoboTwin evaluation across all 50 tasks. Kinex conditions cover three tasks. The current Kinex + VLA condition uses VLAct Clean 50k with an open conditioning interface.</p></div><div class="segmented" aria-label="Comparison metric">${[['success','Success rate'],['tokens','Tokens / task'],['cost','Cost / task']].map(([k,l])=>`<button data-matrix="${k}" aria-pressed="${matrixMetric===k}">${l}</button>`).join('')}</div></div><p class="comparison-definition">${matrixMetric==='tokens'?scope+' '+usageDefinition():matrixMetric==='cost'?scope+' Estimated API-equivalent USD per task.':'Native passes / evaluated tasks. Pending tasks are excluded; compare coverage alongside success rates.'}</p><div class="model-comparisons">${modelGroups().map((g,i)=>`<details class="model-comparison panel" data-model-group="${esc(g.key)}" ${expandedModels?expandedModels.has(g.key)?'open':'':i===0?'open':''}><summary><span><strong>${esc(g.model)}</strong><span class="model-group-meta">${g.key==='baselines'?'One episode per task':esc(g.effort)+' effort'} · ${g.variants.map(v=>esc(v.label||v.agent)).join(' vs ')}</span></span><span class="model-group-count">${g.variants.length} variants <span class="disclosure-arrow">⌄</span></span></summary><div class="table-scroll"><table class="comparison-table grouped-comparison ${matrixMetric}"><thead><tr><th>Environment</th>${g.variants.map(v=>`<th><a href="#runs/${v.id}" class="agent-inline">${icon(v)}${esc(v.label||v.agent)} ${arrow}</a></th>`).join('')}</tr></thead><tbody>${data.families.map(f=>`<tr><th scope="row"><span class="family-code">${f.id.slice(-2)}</span>${esc(f.name)}<small>${f.tasks} tasks</small></th>${g.variants.map(v=>comparisonCell(v,f)).join('')}</tr>`).join('')}</tbody></table></div></details>`).join('')}</div></section>`;
+  return `<section class="comparison-panel" aria-labelledby="comparison-title"><div class="section-heading bare"><div><h2 id="comparison-title">Compare environments</h2><p>Four standalone checkpoints are scheduled under official RoboTwin evaluation across all 50 tasks. Check completion counts alongside scores. Kinex conditions cover three tasks. The current Kinex + VLA condition uses VLAct Clean 50k with an open conditioning interface.</p></div><div class="segmented" aria-label="Comparison metric">${[['success','Success rate'],['tokens','Tokens / task'],['cost','Cost / task']].map(([k,l])=>`<button data-matrix="${k}" aria-pressed="${matrixMetric===k}">${l}</button>`).join('')}</div></div><p class="comparison-definition">${matrixMetric==='tokens'?scope+' '+usageDefinition():matrixMetric==='cost'?scope+' Estimated API-equivalent USD per task.':'Native passes / evaluated tasks. Pending tasks are excluded; compare coverage alongside success rates.'}</p><div class="model-comparisons">${modelGroups().map((g,i)=>`<details class="model-comparison panel" data-model-group="${esc(g.key)}" ${expandedModels?expandedModels.has(g.key)?'open':'':i===0?'open':''}><summary><span><strong>${esc(g.model)}</strong><span class="model-group-meta">${g.key==='baselines'?'One episode per task':esc(g.effort)+' effort'} · ${g.variants.map(v=>esc(v.label||v.agent)).join(' vs ')}</span></span><span class="model-group-count">${g.variants.length} variants <span class="disclosure-arrow">⌄</span></span></summary><div class="table-scroll"><table class="comparison-table grouped-comparison ${matrixMetric}"><thead><tr><th>Environment</th>${g.variants.map(v=>`<th><a href="#runs/${v.id}" class="agent-inline">${icon(v)}${esc(v.label||v.agent)} ${arrow}</a></th>`).join('')}</tr></thead><tbody>${data.families.map(f=>`<tr><th scope="row"><span class="family-code">${f.id.slice(-2)}</span>${esc(f.name)}<small>${f.tasks} tasks</small></th>${g.variants.map(v=>comparisonCell(v,f)).join('')}</tr>`).join('')}</tbody></table></div></details>`).join('')}</div></section>`;
 }
 function recordingPilotProgress(){
  const p=data.recording_pilot;if(!p)return '';

data-source.json

@@ -1,38 +1,23 @@
 {
   "schema": "programvla/dataset-source/1",
   "dataset": "RLE-Bench/programmable-vla",
-  "revision": "c2e39c9ea8f3a9d3e7680e4818d294770f032c21",
-  "evidence_root": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/",
+  "revision": "f24090b8f6c0ba26574f9248f6bfe2a6e78528ad",
+  "evidence_root": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/",
   "files": {
-    "attempt-history.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/attempt-history.json"
-    },
-    "cost-data.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/cost-data.json"
-    },
-    "data-source.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/data-source.json"
-    },
     "data.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/data.json"
-    },
-    "instruction-audit.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/instruction-audit.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/easy/data.json"
     },
-    "manifest.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/manifest.json"
+    "cost-data.json": {
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/easy/cost-data.json"
     },
     "measured-instructions.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/measured-instructions.json"
-    },
-    "migration.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/migration.json"
-    },
-    "presentation-provenance.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/presentation-provenance.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/easy/measured-instructions.json"
     },
     "sources.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/sources.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/easy/sources.json"
+    },
+    "instruction-audit.json": {
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/easy/instruction-audit.json"
     }
   }
 }

hard/app.js

@@ -12,7 +12,7 @@ const compact = v => v == null ? '—' : Math.abs(v) >= 1e9 ? (v/1e9).toFixed(2)
 const pct = v => v == null ? '—' : (v*100).toFixed(1)+'%';
 const elapsed = v => v == null ? '—' : v >= 3600 ? (v/3600).toFixed(1)+' h' : v >= 60 ? (v/60).toFixed(1)+' min' : v.toFixed(1)+' s';
 const arrow = '<span aria-hidden="true">↗</span>';
-const colors = {'standalone-vlact-clean':'#a75f7f','standalone-vlact-all':'#bd733b','standalone-official':'#39819c','standalone':'#39819c','kinex_policy':'#756ee4','kinex':'#27847a','codex-astra-spectator':'#a76a34','codex-astra':'#39819c','kinex-astra':'#756ee4','cc-opus':'#b48566','kinex-opus':'#958bc8','cc-opus-vertex-high':'#c9875a','cc-opus-vertex-xhigh':'#8c5a3a','kinex-opus-vertex-high':'#5b8bd8','kinex-opus-vertex-xhigh':'#3a5ba6','cc-opus-vertex-high-run2':'#e0a37a','cc-opus-vertex-xhigh-run2':'#a8704a','kinex-opus-vertex-high-run2':'#86aae6','kinex-opus-vertex-xhigh-run2':'#5876c0'};
+const colors = {'standalone-oft-all140k':'#547a36','standalone-vlact-clean':'#a75f7f','standalone-vlact-all':'#bd733b','standalone-official':'#39819c','standalone':'#39819c','kinex_policy':'#756ee4','kinex':'#27847a','codex-astra-spectator':'#a76a34','codex-astra':'#39819c','kinex-astra':'#756ee4','cc-opus':'#b48566','kinex-opus':'#958bc8','cc-opus-vertex-high':'#c9875a','cc-opus-vertex-xhigh':'#8c5a3a','kinex-opus-vertex-high':'#5b8bd8','kinex-opus-vertex-xhigh':'#3a5ba6','cc-opus-vertex-high-run2':'#e0a37a','cc-opus-vertex-xhigh-run2':'#a8704a','kinex-opus-vertex-high-run2':'#86aae6','kinex-opus-vertex-xhigh-run2':'#5876c0'};
 let data, epoch = 0, current = {}, matrixMetric = 'success';
 let filters = {query:'',family:'',status:'',variant:'standalone-vlact-clean',sort:'catalog'};
 let sessionState = {filter:'all',query:'',count:40,view:'timeline'};
@@ -119,7 +119,7 @@ function opusPilotProgress(){
  return `<section class="panel" style="padding:18px 24px;margin-bottom:24px" aria-label="Claude Code campaign"><strong>Claude Code · Opus 5.5 / high</strong><p>${number(n.finished||0)} / ${number(c.planned_episodes)} episodes completed · ${number(n.running||0)} running · ${number(n.queued||0)} pending${n.needs_review?' · '+number(n.needs_review)+' under review':''}${n.interrupted?' · '+number(n.interrupted)+' interrupted':''}.</p><p class="source-note">${c.quota_waiting?'Waiting for model-service quota recovery · ':''}${number(c.published_episodes)} episode records published · <a href="${esc(c.status_url)}" target="_blank" rel="noopener">Results and protocol ↗</a></p></section>`;
 }
 function methodology() {
- return `<details class="panel methodology" id="methodology"><summary><span><span class="eyebrow">Reading the Hard benchmark</span><strong>Protocol, metrics & provenance</strong></span><span class="summary-plus">+</span></summary><div class="method-grid"><div><h3>50 tasks × 4 conditions</h3><p>RoboTwin 2.0 Hard, ALOHA AgileX, demo_randomized. Each condition completed one valid episode per task. Successes and valid failures are both retained; no valid outcome was retried. This is RoboTwin 2.0 Hard, not RoboTwin 2.0-Plus.</p></div><div><h3>Native unseen instructions</h3><p>Every episode uses the original upstream unseen instruction with its scene bindings. No RLE task03 instruction revision is applied. The exact instruction is available on each episode page and in the recorded model inputs.</p></div><div><h3>Checkpoint identity</h3><p>Base 40k: StarVLA/Qwen3-VL-OFT-Robotwin2. VLAct Clean 50k: StarVLA/VLAct_Qwen3OFT_Robotwin_Finetune. VLAct All 100k: StarVLA/VLAct_Qwen3OFT_Robotwin_all_Finetune. Full revisions and weight checksums are linked below. Kinex uses GPT-6 Astra / high without a VLA extension.</p></div><div><h3>Execution and seeds</h3><p>Standalone VLA uses official expert admission from scene seed 100000 and official take_action with full predicted chunks. Pure Kinex uses rosbridge at 25 Hz and the first initializable seed in ascending order from 0, selected before the first model query. Seeds, admission rules and physical durations are not matched across the interfaces; this is not a paired causal comparison. Per-episode seeds and budgets are retained.</p></div><div><h3>Initialization provenance</h3><p>Five infrastructure startup attempts produced no completed model response and are excluded from the 200 valid outcomes. For hanging_mug, a minimal runtime repair retains native language bindings if the expert path fails before finishing. Selected seed 0, task physics and the native verifier are preserved.</p></div><div><h3>Usage and costs</h3><p>${usageDefinition()} Estimated API-equivalent costs use recorded Pi session values. Actual account charges and GPU costs are not measured. Standalone VLA has zero language-agent API calls, not zero compute cost.</p></div></div><div class="method-links"><a href="${esc(resultURL('sources.json'))}" target="_blank">Data sources ↗</a><a href="${esc(evidenceURL('research/robotwin-hard-20261101/REPORT.md'))}">Report ↗</a><a href="${esc(evidenceURL('research/robotwin-hard-20261101/protocol.json'))}">Protocol & checkpoints ↗</a><a href="${esc(evidenceURL('research/robotwin-hard-20261101/execution-audit.json'))}">200-episode audit ↗</a><a href="../index.html#overview">Retained Easy results ↗</a><button class="text-button" data-export="all">Download comparison CSV ↓</button></div></details>`;
+ return `<details class="panel methodology" id="methodology"><summary><span><span class="eyebrow">Reading the Hard benchmark</span><strong>Protocol, metrics & provenance</strong></span><span class="summary-plus">+</span></summary><div class="method-grid"><div><h3>50 tasks × 5 conditions</h3><p>RoboTwin 2.0 Hard, ALOHA AgileX, demo_randomized. Each condition is scheduled for one valid episode per task; completion is shown separately. Successes and valid failures are both retained; no valid outcome was retried. This is RoboTwin 2.0 Hard, not RoboTwin 2.0-Plus.</p></div><div><h3>Native unseen instructions</h3><p>Every episode uses the original upstream unseen instruction with its scene bindings. No RLE task03 instruction revision is applied. The exact instruction is available on each episode page and in the recorded model inputs.</p></div><div><h3>Checkpoint identity</h3><p>Base 40k: StarVLA/Qwen3-VL-OFT-Robotwin2. VLAct Clean 50k: StarVLA/VLAct_Qwen3OFT_Robotwin_Finetune. VLAct All 100k: StarVLA/VLAct_Qwen3OFT_Robotwin_all_Finetune. Non-VLAct All 140k: StarVLA/Qwen3-VL-OFT-RoboTwin2-All, trained on 27,500 clean and randomized demonstrations. Full revisions and weight checksums are linked below. Kinex uses GPT-6 Astra / high without a VLA extension.</p></div><div><h3>Execution and seeds</h3><p>Standalone VLA uses official expert admission from scene seed 100000 and official take_action with full predicted chunks. Pure Kinex uses rosbridge at 25 Hz and the first initializable seed in ascending order from 0, selected before the first model query. Seeds, admission rules and physical durations are not matched across the interfaces; this is not a paired causal comparison. Per-episode seeds and budgets are retained.</p></div><div><h3>Initialization provenance</h3><p>Five infrastructure startup attempts produced no completed model response and are excluded from the 200 valid outcomes. For hanging_mug, a minimal runtime repair retains native language bindings if the expert path fails before finishing. Selected seed 0, task physics and the native verifier are preserved.</p></div><div><h3>Usage and costs</h3><p>${usageDefinition()} Estimated API-equivalent costs use recorded Pi session values. Actual account charges and GPU costs are not measured. Standalone VLA has zero language-agent API calls, not zero compute cost.</p></div></div><div class="method-links"><a href="${esc(resultURL('sources.json'))}" target="_blank">Data sources ↗</a><a href="${esc(evidenceURL('research/robotwin-hard-20261101/REPORT.md'))}">Report ↗</a><a href="${esc(evidenceURL('research/robotwin-hard-20261101/protocol.json'))}">Protocol & checkpoints ↗</a><a href="${esc(evidenceURL('research/robotwin-hard-20261101/execution-audit.json'))}">200-episode audit ↗</a><a href="../index.html#overview">Retained Easy results ↗</a><button class="text-button" data-export="all">Download comparison CSV ↓</button></div></details>`;
 }
 function taskBrowser(variant=null) {
   return `<section class="task-browser" id="tasks"><div class="section-heading bare"><div><div class="eyebrow">${variant?'Episode index':'Browse the benchmark'}</div><h2>Explore tasks <span class="heading-count">${plannedTasks(variant).length}</span></h2></div><button class="button secondary" data-export="${variant||'all'}">Export CSV ↓</button></div><div class="panel"><div class="filters"><label class="search-field"><span aria-hidden="true">⌕</span><input id="task-search" type="search" aria-label="Search tasks" placeholder="Search task name, instruction or ID" value="${esc(filters.query)}"></label><select id="task-family" aria-label="Environment"><option value="">All environments</option>${data.families.map(f=>`<option value="${f.id}" ${filters.family===f.id?'selected':''}>${f.name}</option>`).join('')}</select>${variant?'':`<select id="task-variant" aria-label="Agent for outcome filter">${data.variants.map(v=>`<option value="${v.id}" ${filters.variant===v.id?'selected':''}>${v.label||v.agent} · ${v.model}</option>`).join('')}</select>`}<select id="task-status" aria-label="Outcome"><option value="">All outcomes</option>${['pass','fail','pending'].map(s=>`<option value="${s}" ${filters.status===s?'selected':''}>${s[0].toUpperCase()+s.slice(1)}</option>`).join('')}</select><select id="task-sort" aria-label="Sort tasks"><option value="catalog" ${filters.sort==='catalog'?'selected':''}>Catalog order</option><option value="tokens" ${filters.sort==='tokens'?'selected':''}>Most fresh input tokens</option><option value="cost" ${filters.sort==='cost'?'selected':''}>Highest cost</option><option value="name" ${filters.sort==='name'?'selected':''}>Task name</option></select></div><div class="table-scroll"><table class="task-table ${variant?'single-variant':''}"><thead><tr><th>Task</th>${variant?'<th>Native result</th><th>Fresh input<small>Excludes read and write</small></th><th>Cache read</th><th>Cache write</th><th>Cache hit</th><th>Output</th><th>Total cost<small>Estimated USD</small></th><th>Action steps</th>':data.variants.map(v=>`<th class="${v.id===filters.variant?'focus-column':''}">${v.label||v.agent}<small>${v.model} · ${v.effort}</small><small>Native result · Estimated USD</small></th>`).join('')}</tr></thead><tbody id="task-rows"></tbody></table></div><div class="table-footer"><span id="task-count" aria-live="polite"></span><span id="filter-context"></span><button class="text-button" id="reset-filters">Reset filters</button></div></div></section>`;

hard/data-source.json

@@ -1,23 +1,23 @@
 {
   "schema": "programvla/dataset-source/1",
   "dataset": "RLE-Bench/programmable-vla",
-  "revision": "b07d88579523eeeefdf7ad2ca14ee65340af5d4e",
-  "evidence_root": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/",
+  "revision": "f24090b8f6c0ba26574f9248f6bfe2a6e78528ad",
+  "evidence_root": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/",
   "files": {
     "data.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/data.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/hard/data.json"
     },
     "cost-data.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/cost-data.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/hard/cost-data.json"
     },
     "measured-instructions.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/measured-instructions.json"
-    },
-    "instruction-audit.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/instruction-audit.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/hard/measured-instructions.json"
     },
     "sources.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/sources.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/hard/sources.json"
+    },
+    "instruction-audit.json": {
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/hard/instruction-audit.json"
     }
   }
 }

hard/index.html

@@ -9,16 +9,16 @@
 <link rel="icon" href="icon.svg" type="image/svg+xml">
 <link rel="stylesheet" href="style.css?v=bd1a59147e1d">
 <script src="usage-metrics.js?v=330e92f8eb96" defer></script>
-<script src="app.js?v=hard-20261101" defer></script>
+<script src="app.js?v=oft-all140k-20261103" defer></script>
 <link rel="stylesheet" href="policy.css">
 </head>
 <body>
 <a class="skip" href="#main">Skip to content</a>
 <header class="site-header"><div class="header-inner"><a class="brand" href="#overview" aria-label="Programmable VLA home"><img src="assets/kinex-logo.svg" width="34" height="34" alt="Kinex"><span>programmable<span class="brand-divider">/</span><span class="brand-benchmark">vla</span></span></a><nav aria-label="Main navigation"><a href="../plus/index.html">Plus</a><a href="../index.html#overview">Easy</a><a href="#overview" data-nav="overview">Hard</a><a href="#tasks" data-nav="tasks">Explore tasks</a><a href="#methodology" data-nav="methodology">Protocol</a></nav><a class="header-link" href="https://huggingface.co/spaces/RLE-Bench/programmable-vla" target="_blank" rel="noopener">Hugging Face <span aria-hidden="true">↗</span></a></div></header>
 <main id="main" tabindex="-1"><div class="loading"><span class="spinner"></span>Loading benchmark results…</div></main>
-<footer class="site-footer"><div><img src="assets/kinex-logo.svg" width="22" height="22" alt=""><strong>Programmable VLA</strong><span class="footer-org">by RLE-Bench</span></div><div><a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/data.json" target="_blank">Hard results JSON</a><a data-result-file="sources.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/sources.json" target="_blank">Data sources</a><a href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/manifest.json" target="_blank">Manifest</a><span id="updated"></span></div></footer>
+<footer class="site-footer"><div><img src="assets/kinex-logo.svg" width="22" height="22" alt=""><strong>Programmable VLA</strong><span class="footer-org">by RLE-Bench</span></div><div><a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/hard/data.json" target="_blank">Hard results JSON</a><a data-result-file="sources.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/sources.json" target="_blank">Data sources</a><a href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/manifest.json" target="_blank">Manifest</a><span id="updated"></span></div></footer>
 <dialog id="image-viewer" aria-labelledby="image-caption"><div class="viewer-toolbar"><span id="image-caption"></span><a id="image-original" target="_blank" rel="noopener">Open original ↗</a><button class="icon-button" id="close-viewer" aria-label="Close image">×</button></div><div class="viewer-image"><button id="previous-image" class="viewer-arrow" aria-label="Previous image">←</button><img id="large-image" alt="Recorded agent observation"><button id="next-image" class="viewer-arrow" aria-label="Next image">→</button></div><p>Recorded agent observation · Use ← → to browse · Esc to close</p></dialog>
 <div id="toast" role="status" aria-live="polite"></div>
-<noscript><p>This site needs JavaScript. <a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/data.json">Download the complete results index.</a></p></noscript>
+<noscript><p>This site needs JavaScript. <a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/hard/data.json">Download the complete results index.</a></p></noscript>
 </body>
 </html>

index.html

@@ -9,16 +9,16 @@
 <link rel="icon" href="icon.svg" type="image/svg+xml">
 <link rel="stylesheet" href="style.css?v=bd1a59147e1d">
 <script src="usage-metrics.js?v=330e92f8eb96" defer></script>
-<script src="app.js?v=hard-navigation-20261101" defer></script>
+<script src="app.js?v=oft-all140k-20261103" defer></script>
 <link rel="stylesheet" href="policy.css">
 </head>
 <body>
 <a class="skip" href="#main">Skip to content</a>
 <header class="site-header"><div class="header-inner"><a class="brand" href="#overview" aria-label="Programmable VLA home"><img src="assets/kinex-logo.svg" width="34" height="34" alt="Kinex"><span>programmable<span class="brand-divider">/</span><span class="brand-benchmark">vla</span></span></a><nav aria-label="Main navigation"><a href="plus/index.html">Plus</a><a href="#overview" data-nav="overview">Easy</a><a href="hard/index.html#overview">Hard</a><a href="#tasks" data-nav="tasks">Explore tasks</a><a href="#methodology" data-nav="methodology">Protocol</a></nav><a class="header-link" href="https://huggingface.co/spaces/RLE-Bench/programmable-vla" target="_blank" rel="noopener">Hugging Face <span aria-hidden="true">↗</span></a></div></header>
 <main id="main" tabindex="-1"><div class="loading"><span class="spinner"></span>Loading benchmark results…</div></main>
-<footer class="site-footer"><div><img src="assets/kinex-logo.svg" width="22" height="22" alt=""><strong>Programmable VLA</strong><span class="footer-org">by RLE-Bench</span></div><div><a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/data.json" target="_blank">Results JSON</a><a data-result-file="sources.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/sources.json" target="_blank">Data sources</a><a href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/manifest.json" target="_blank">Manifest</a><span id="updated"></span></div></footer>
+<footer class="site-footer"><div><img src="assets/kinex-logo.svg" width="22" height="22" alt=""><strong>Programmable VLA</strong><span class="footer-org">by RLE-Bench</span></div><div><a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/easy/data.json" target="_blank">Results JSON</a><a data-result-file="sources.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/sources.json" target="_blank">Data sources</a><a href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/manifest.json" target="_blank">Manifest</a><span id="updated"></span></div></footer>
 <dialog id="image-viewer" aria-labelledby="image-caption"><div class="viewer-toolbar"><span id="image-caption"></span><a id="image-original" target="_blank" rel="noopener">Open original ↗</a><button class="icon-button" id="close-viewer" aria-label="Close image">×</button></div><div class="viewer-image"><button id="previous-image" class="viewer-arrow" aria-label="Previous image">←</button><img id="large-image" alt="Recorded agent observation"><button id="next-image" class="viewer-arrow" aria-label="Next image">→</button></div><p>Recorded agent observation · Use ← → to browse · Esc to close</p></dialog>
 <div id="toast" role="status" aria-live="polite"></div>
-<noscript><p>This site needs JavaScript. <a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/c2e39c9ea8f3a9d3e7680e4818d294770f032c21/data.json">Download the complete results index.</a></p></noscript>
+<noscript><p>This site needs JavaScript. <a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/easy/data.json">Download the complete results index.</a></p></noscript>
 </body>
 </html>

manifest.json

@@ -7,8 +7,8 @@
   },
   "files": {
     "app.js": {
-      "bytes": 94913,
-      "sha256": "19672228deb8715bb6ccd5cf18dcc6fe70b450bf5d70dca35fb620155e6819fc"
+      "bytes": 95001,
+      "sha256": "bd4dddc2edd26159432c60c1acf4edc9b0257c6b1a425610b8592b6acd96972a"
     },
     "history-app.js": {
       "bytes": 87082,
@@ -19,8 +19,8 @@
       "sha256": "34a4d6e437537e78bc242418805a3af3d71b43980280482fd8fc6bfda91f79f8"
     },
     "README.md": {
-      "bytes": 6279,
-      "sha256": "510d4c6b23afb20d2fd087878cc9c3d44d8d7cee4bee24979c3e1de76fce9166"
+      "bytes": 6808,
+      "sha256": "d3233e6c7ffe60693e5223321eb0a39c81dfb561e9a9caf77506e537f3b9760d"
     },
     "icon.svg": {
       "bytes": 725,
@@ -35,8 +35,8 @@
       "sha256": "8b0691838063224bce2a991d7bb0672897ef8f51bbc39ce38afb4a67cb6e39d0"
     },
     "index.html": {
-      "bytes": 3333,
-      "sha256": "5c60da97019262ace5b0335f3d41c7c6ca9a8f7b6cff16abe5745df1213ba13f"
+      "bytes": 3413,
+      "sha256": "d1a509dab9c934a0a7028cd20a04312de2470a991cee1849d844239a41d409ac"
     },
     "policy.css": {
       "bytes": 1067,
@@ -47,8 +47,8 @@
       "sha256": "bd1a59147e1d5e6c4fc5abd0871fb7b2e5e282a9ee3d15ca4df8a219509af69c"
     },
     "data-source.json": {
-      "bytes": 2053,
-      "sha256": "03adf23d135a31081819c23f9244b922ea26a6ea20a89970b149738dcc7f6ee9"
+      "bytes": 1376,
+      "sha256": "2049b4660d8f2843c9c508a7acca345379c22bc1b93d00e24588bbdce746eb52"
     },
     "history/data-source.json": {
       "bytes": 2109,
@@ -75,8 +75,8 @@
       "sha256": "a53108594a23b740fbb08623f6fffdda1d73608b552988a9ffda9f0944ae66a1"
     },
     "hard/app.js": {
-      "bytes": 93978,
-      "sha256": "c8700f8239045156d52df89c3a9d7d02fde81bacc55d8933123fee7467eeb914"
+      "bytes": 94163,
+      "sha256": "41d700d19497d91c7e9af762207b384bf30eee3199fc5a77a4642290a8b1b3a8"
     },
     "hard/icon.svg": {
       "bytes": 725,
@@ -87,8 +87,8 @@
       "sha256": "330e92f8eb9670cb506f2909a2a47ee8f0bd569199898df3bfa044239ae64161"
     },
     "hard/index.html": {
-      "bytes": 3351,
-      "sha256": "5ab0faa63b0141aad0ea623006bd93f4ca688262a2ea84852e968cf27b2876c5"
+      "bytes": 3432,
+      "sha256": "18500aae0c13e5306eed4e8e6ec817df21adaff493d24a325e8fd9eb54756d96"
     },
     "hard/policy.css": {
       "bytes": 1067,
@@ -99,8 +99,8 @@
       "sha256": "bd1a59147e1d5e6c4fc5abd0871fb7b2e5e282a9ee3d15ca4df8a219509af69c"
     },
     "hard/data-source.json": {
-      "bytes": 1191,
-      "sha256": "ac332c73322a1efb2018f9e87641fd2b6becb4fad18141edd3013d66e9ee60df"
+      "bytes": 1376,
+      "sha256": "4c09a8086bd7b321211f77f0d3498c0fcafc40da58ea55ae25f443a591d1dba2"
     },
     "probes/bridge-20261028/index.html": {
       "bytes": 6573,
@@ -127,20 +127,20 @@
       "sha256": "a53108594a23b740fbb08623f6fffdda1d73608b552988a9ffda9f0944ae66a1"
     },
     "plus/app.js": {
-      "bytes": 95721,
-      "sha256": "b5bce2b4c28d54ba82dbc244598d91115a240c21fc29c3b316d3f886660ec95f"
+      "bytes": 95880,
+      "sha256": "6bb2137d077cd9ba888da6206d1f9a84e1e6f9c8cf52c493d939dca9b17f9e47"
     },
     "plus/index.html": {
-      "bytes": 3370,
-      "sha256": "84b8f8f968ecfc1d9032af7408c5dfa1fe3b089bec4339d70ba4deabfc830408"
+      "bytes": 3441,
+      "sha256": "530a860dea717e6d25284e3d259ed5d687ddf71ea15231574a2725ec4e6b78c0"
     },
     "plus/style.css": {
       "bytes": 60419,
       "sha256": "bd1a59147e1d5e6c4fc5abd0871fb7b2e5e282a9ee3d15ca4df8a219509af69c"
     },
     "plus/data-source.json": {
-      "bytes": 1396,
-      "sha256": "7a0ad5e9303ccdefb75d450cb17edbbc764466c5403af02c98095aacec4ec82e"
+      "bytes": 1376,
+      "sha256": "49d3b3f177a31d67dafea61b8640541fe256a860bf0774c83697b4d81f6e74b3"
     },
     "plus/icon.svg": {
       "bytes": 725,
@@ -186,5 +186,10 @@
     "revision": "b2d67a67dd2f9d8e12c1f8db13ba8de309739883",
     "prefix": "robotwin-plus/20261101-native/presentation-v2",
     "original_evidence_revision": "9f8a68a40148e1ad013b35a2d8774ff113cb0c5e"
+  },
+  "oft_all140k_dataset": {
+    "repo_id": "RLE-Bench/programmable-vla",
+    "revision": "f24090b8f6c0ba26574f9248f6bfe2a6e78528ad",
+    "prefix": "research/oft-all140k-20261103"
   }
 }

plus/app.js

@@ -12,7 +12,7 @@ const compact = v => v == null ? '—' : Math.abs(v) >= 1e9 ? (v/1e9).toFixed(2)
 const pct = v => v == null ? '—' : (v*100).toFixed(1)+'%';
 const elapsed = v => v == null ? '—' : v >= 3600 ? (v/3600).toFixed(1)+' h' : v >= 60 ? (v/60).toFixed(1)+' min' : v.toFixed(1)+' s';
 const arrow = '<span aria-hidden="true">↗</span>';
-const colors = {'standalone-vlact-clean':'#a75f7f','standalone-vlact-all':'#bd733b','standalone-official':'#39819c','standalone':'#39819c','kinex_policy':'#756ee4','kinex':'#27847a','codex-astra-spectator':'#a76a34','codex-astra':'#39819c','kinex-astra':'#756ee4','cc-opus':'#b48566','kinex-opus':'#958bc8','cc-opus-vertex-high':'#c9875a','cc-opus-vertex-xhigh':'#8c5a3a','kinex-opus-vertex-high':'#5b8bd8','kinex-opus-vertex-xhigh':'#3a5ba6','cc-opus-vertex-high-run2':'#e0a37a','cc-opus-vertex-xhigh-run2':'#a8704a','kinex-opus-vertex-high-run2':'#86aae6','kinex-opus-vertex-xhigh-run2':'#5876c0'};
+const colors = {'standalone-oft-all140k':'#547a36','standalone-vlact-clean':'#a75f7f','standalone-vlact-all':'#bd733b','standalone-official':'#39819c','standalone':'#39819c','kinex_policy':'#756ee4','kinex':'#27847a','codex-astra-spectator':'#a76a34','codex-astra':'#39819c','kinex-astra':'#756ee4','cc-opus':'#b48566','kinex-opus':'#958bc8','cc-opus-vertex-high':'#c9875a','cc-opus-vertex-xhigh':'#8c5a3a','kinex-opus-vertex-high':'#5b8bd8','kinex-opus-vertex-xhigh':'#3a5ba6','cc-opus-vertex-high-run2':'#e0a37a','cc-opus-vertex-xhigh-run2':'#a8704a','kinex-opus-vertex-high-run2':'#86aae6','kinex-opus-vertex-xhigh-run2':'#5876c0'};
 let data, epoch = 0, current = {}, matrixMetric = 'success';
 let filters = {query:'',family:'',status:'',variant:'standalone-vlact-clean',sort:'catalog'};
 let sessionState = {filter:'all',query:'',count:40,view:'timeline'};
@@ -119,7 +119,7 @@ function opusPilotProgress(){
  return `<section class="panel" style="padding:18px 24px;margin-bottom:24px" aria-label="Claude Code campaign"><strong>Claude Code · Opus 5.5 / high</strong><p>${number(n.finished||0)} / ${number(c.planned_episodes)} episodes completed · ${number(n.running||0)} running · ${number(n.queued||0)} pending${n.needs_review?' · '+number(n.needs_review)+' under review':''}${n.interrupted?' · '+number(n.interrupted)+' interrupted':''}.</p><p class="source-note">${c.quota_waiting?'Waiting for model-service quota recovery · ':''}${number(c.published_episodes)} episode records published · <a href="${esc(c.status_url)}" target="_blank" rel="noopener">Results and protocol ↗</a></p></section>`;
 }
 function methodology() {
- return `<details class="panel methodology" id="methodology"><summary><span><span class="eyebrow">Reading the Plus benchmark</span><strong>Protocol, metrics & provenance</strong></span><span class="summary-plus">+</span></summary><div class="method-grid"><div><h3>50 tasks × 8 configurations × 3 checkpoints</h3><p>All 1,200 episodes are complete and valid. Clean and seven perturbation configurations each contain 50 tasks per checkpoint. Overall scores include Clean. Native successes and failures are retained; no valid outcome was retried. Kinex was not scheduled for this panel.</p></div><div><h3>Native instructions</h3><p>Original Plus instructions are sent unchanged. Non-language configurations use the native seen pool; language-plus uses the native combined pool. No RLE instruction revision is applied. Exact instructions and model inputs are available on every episode page; sampled prompts may differ across checkpoints.</p></div><div><h3>Native execution and coverage</h3><p>Standalone policies use the upstream expert-success admission rule, starting at scene seed 100000 in seed group 0. Each task/configuration uses accepted episode index 0, with its actual scene seed recorded. Native take_action(qpos) executes complete predicted chunks: 16 actions for Base and 32 for VLAct. This single-episode evaluation is not the upstream multi-trial protocol. Vision-noise index 0 covers motion noise, not all noise subtypes.</p></div><div><h3>Recorded recovery</h3><p>All 100k stopped once before inference because a validator rejected angle-bracket text in a native instruction. The fix preserves whole native Plus instructions in all eight configurations while keeping custom RLE templates strict. The 116 completed All outcomes were preserved; the remaining 284 used the recorded recovery source. The interrupted attempt made zero model calls. Per-episode source phases and the recovery receipt are retained.</p></div><div><h3>Same video presentation as Easy/Hard</h3><p>Head, left wrist and right wrist appear side by side at 2880 × 720. Native 320 × 240 views are upscaled with the same H.264 CRF 20 encoding. Playback is 4× the official 10 FPS action-index recording, not physical simulation time. Wrist views hold the recorded inference observation until the next inference. Original frames and videos remain available; no simulator rerun or new camera frame was generated.</p></div><div><h3>Usage and evidence</h3><p>${usageDefinition()} These standalone runs have zero language-agent API calls; local VLA compute and GPU cost are not measured. The VLA input/output tab shows every recorded request, exact input PNG, raw response and native action chunk. Original evidence ZIPs, native videos, checkpoint revisions and source hashes are retained.</p></div></div><div class="method-links"><a href="${esc(resultURL('sources.json'))}" target="_blank">Data sources ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/REPORT.md'))}">Report ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/protocol.json'))}">Protocol & checkpoints ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/recovery/receipt.json'))}">Recovery receipt ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/presentation-v2/presentation-protocol.json'))}">Video presentation protocol ↗</a><button class="text-button" data-export="all">Download comparison CSV ↓</button></div></details>`;
+ return `<details class="panel methodology" id="methodology"><summary><span><span class="eyebrow">Reading the Plus benchmark</span><strong>Protocol, metrics & provenance</strong></span><span class="summary-plus">+</span></summary><div class="method-grid"><div><h3>50 tasks × 8 configurations × 4 checkpoints</h3><p>The original 1,200 valid episodes are retained. All 140k adds 400 scheduled episodes; completion is shown separately. Clean and seven perturbation configurations each contain 50 tasks per checkpoint. Overall scores include Clean. Native successes and failures are retained; no valid outcome was retried. Kinex was not scheduled for this panel.</p></div><div><h3>Native instructions</h3><p>Original Plus instructions are sent unchanged. Non-language configurations use the native seen pool; language-plus uses the native combined pool. No RLE instruction revision is applied. Exact instructions and model inputs are available on every episode page; sampled prompts may differ across checkpoints.</p></div><div><h3>Native execution and coverage</h3><p>Standalone policies use the upstream expert-success admission rule, starting at scene seed 100000 in seed group 0. Each task/configuration uses accepted episode index 0, with its actual scene seed recorded. Native take_action(qpos) executes complete predicted chunks: 16 actions for Base, 32 for VLAct, and 50 for non-VLAct All 140k. This single-episode evaluation is not the upstream multi-trial protocol. Vision-noise index 0 covers motion noise, not all noise subtypes.</p></div><div><h3>Recorded recovery</h3><p>VLAct All 100k stopped once before inference because a validator rejected angle-bracket text in a native instruction. The fix preserves whole native Plus instructions in all eight configurations while keeping custom RLE templates strict. The 116 completed All outcomes were preserved; the remaining 284 used the recorded recovery source. The interrupted attempt made zero model calls. Per-episode source phases and the recovery receipt are retained.</p></div><div><h3>Same video presentation as Easy/Hard</h3><p>Head, left wrist and right wrist appear side by side at 2880 × 720. Native 320 × 240 views are upscaled with the same H.264 CRF 20 encoding. Playback is 4× the official 10 FPS action-index recording, not physical simulation time. Wrist views hold the recorded inference observation until the next inference. Original frames and videos remain available; no simulator rerun or new camera frame was generated.</p></div><div><h3>Usage and evidence</h3><p>${usageDefinition()} These standalone runs have zero language-agent API calls; local VLA compute and GPU cost are not measured. The VLA input/output tab shows every recorded request, exact input PNG, raw response and native action chunk. Original evidence ZIPs, native videos, checkpoint revisions and source hashes are retained.</p></div></div><div class="method-links"><a href="${esc(resultURL('sources.json'))}" target="_blank">Data sources ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/REPORT.md'))}">Report ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/protocol.json'))}">Protocol & checkpoints ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/recovery/receipt.json'))}">Recovery receipt ↗</a><a href="${esc(evidenceURL('robotwin-plus/20261101-native/presentation-v2/presentation-protocol.json'))}">Video presentation protocol ↗</a><button class="text-button" data-export="all">Download comparison CSV ↓</button></div></details>`;
 }
 function taskBrowser(variant=null) {
   return `<section class="task-browser" id="tasks"><div class="section-heading bare"><div><div class="eyebrow">${variant?'Episode index':'Browse the benchmark'}</div><h2>Explore tasks <span class="heading-count">${plannedTasks(variant).length}</span></h2></div><button class="button secondary" data-export="${variant||'all'}">Export CSV ↓</button></div><div class="panel"><div class="filters"><label class="search-field"><span aria-hidden="true">⌕</span><input id="task-search" type="search" aria-label="Search tasks" placeholder="Search task name, instruction or ID" value="${esc(filters.query)}"></label><select id="task-family" aria-label="Configuration"><option value="">All configurations</option>${data.families.map(f=>`<option value="${f.id}" ${filters.family===f.id?'selected':''}>${f.name}</option>`).join('')}</select>${variant?'':`<select id="task-variant" aria-label="Agent for outcome filter">${data.variants.map(v=>`<option value="${v.id}" ${filters.variant===v.id?'selected':''}>${v.label||v.agent} · ${v.model}</option>`).join('')}</select>`}<select id="task-status" aria-label="Outcome"><option value="">All outcomes</option>${['pass','fail','pending'].map(s=>`<option value="${s}" ${filters.status===s?'selected':''}>${s[0].toUpperCase()+s.slice(1)}</option>`).join('')}</select><select id="task-sort" aria-label="Sort tasks"><option value="catalog" ${filters.sort==='catalog'?'selected':''}>Catalog order</option><option value="tokens" ${filters.sort==='tokens'?'selected':''}>Most fresh input tokens</option><option value="cost" ${filters.sort==='cost'?'selected':''}>Highest cost</option><option value="name" ${filters.sort==='name'?'selected':''}>Task name</option></select></div><div class="table-scroll"><table class="task-table ${variant?'single-variant':''}"><thead><tr><th>Task</th>${variant?'<th>Native result</th><th>Fresh input<small>Excludes read and write</small></th><th>Cache read</th><th>Cache write</th><th>Cache hit</th><th>Output</th><th>Total cost<small>Estimated USD</small></th><th>Action steps</th>':data.variants.map(v=>`<th class="${v.id===filters.variant?'focus-column':''}">${v.label||v.agent}<small>${v.model} · ${v.effort}</small><small>Native result · Estimated USD</small></th>`).join('')}</tr></thead><tbody id="task-rows"></tbody></table></div><div class="table-footer"><span id="task-count" aria-live="polite"></span><span id="filter-context"></span><button class="text-button" id="reset-filters">Reset filters</button></div></div></section>`;
@@ -142,7 +142,7 @@ function researchRerunNote() {
 
 function overview() {
   current={type:'overview'};
-  $('#main').innerHTML=`<section class="hero"><div><div class="eyebrow"><span class="tiny-logo">✦</span> RLE-Bench · Embodied agent evaluation</div><h1>RoboTwin 2.0-Plus.<br><span>VLA evaluation.</span></h1><p>Native instructions · 50 tasks × 8 configurations × one admitted episode.<br>Explore all 1,200 native verdicts, three-camera recordings and model evidence.</p></div><div class="hero-facts"><div><strong>${data.families.length}</strong><span>configurations</span></div><div><strong>${data.tasks.length}</strong><span>task/configuration pairs</span></div><div><strong>${data.variants.length}</strong><span>VLA checkpoints</span></div><span class="progress-label"><i></i>${data.benchmark_complete?'Evaluation complete':'Evaluation in progress'} · selected seeds</span></div></section>
+  $('#main').innerHTML=`<section class="hero"><div><div class="eyebrow"><span class="tiny-logo">✦</span> RLE-Bench · Embodied agent evaluation</div><h1>RoboTwin 2.0-Plus.<br><span>VLA evaluation.</span></h1><p>Native instructions · 50 tasks × 8 configurations × one admitted episode.<br>Explore native verdicts, three-camera recordings and model evidence for all four checkpoints.</p></div><div class="hero-facts"><div><strong>${data.families.length}</strong><span>configurations</span></div><div><strong>${data.tasks.length}</strong><span>task/configuration pairs</span></div><div><strong>${data.variants.length}</strong><span>VLA checkpoints</span></div><span class="progress-label"><i></i>${data.benchmark_complete?'Evaluation complete':'Evaluation in progress'} · selected seeds</span></div></section>
   ${researchRerunNote()}${recordingPilotProgress()}${campaignProgress()}${opusPilotProgress()}<div class="agent-cards">${data.variants.map(v=>{const s=stats(episodes(v.id));return `<a class="agent-card ${s.n?'':'planned'}" href="#runs/${v.id}" style="--variant:${colors[v.id]}"><div class="card-header"><span class="agent-name">${icon(v)}${esc(v.label||v.agent)}</span><span class="state-label">${s.n===plannedTasks(v.id).length?'Complete':s.n?'Partial':'Planned'}</span></div><p class="card-model">${modelLabel(v)}</p><div class="card-rate"><strong>${pct(s.rate)}</strong><span>native success</span></div><div class="card-stats"><span><b>${s.n?s.passes+' / '+s.n:'—'}</b> passed</span><span class="card-cost" title="${esc(costNote(s))}"><b>${costValue(s)}</b> total cost<small>Estimated USD</small></span></div><div class="completion-track"><span style="width:${s.n/plannedTasks(v.id).length*100}%"></span></div><div class="card-bottom"><span>${s.n} / ${plannedTasks(v.id).length} evaluated</span><span>${s.n?'Explore runs':'View planned'} ${arrow}</span></div></a>`;}).join('')}</div><div class="scope-note"><span><i class="status-dot"></i>Success rates and costs cover selected evaluated tasks only.</span><span>Native Plus instructions · One admitted episode per task/configuration</span></div><div id="comparison-mount">${comparisonTable()}</div>${taskBrowser()}${methodology()}`;
   renderRows();
 }

plus/data-source.json

@@ -1,23 +1,23 @@
 {
   "schema": "programvla/dataset-source/1",
   "dataset": "RLE-Bench/programmable-vla",
-  "revision": "b2d67a67dd2f9d8e12c1f8db13ba8de309739883",
-  "evidence_root": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/",
+  "revision": "f24090b8f6c0ba26574f9248f6bfe2a6e78528ad",
+  "evidence_root": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/",
   "files": {
     "data.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/robotwin-plus/20261101-native/presentation-v2/data.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/plus/data.json"
     },
     "cost-data.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/robotwin-plus/20261101-native/presentation-v2/cost-data.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/plus/cost-data.json"
     },
     "measured-instructions.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/robotwin-plus/20261101-native/presentation-v2/measured-instructions.json"
-    },
-    "instruction-audit.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/robotwin-plus/20261101-native/presentation-v2/instruction-audit.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/plus/measured-instructions.json"
     },
     "sources.json": {
-      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/robotwin-plus/20261101-native/presentation-v2/sources.json"
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/plus/sources.json"
+    },
+    "instruction-audit.json": {
+      "url": "https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/plus/instruction-audit.json"
     }
   }
 }

plus/index.html

@@ -9,16 +9,16 @@
 <link rel="icon" href="icon.svg" type="image/svg+xml">
 <link rel="stylesheet" href="style.css?v=bd1a59147e1d">
 <script src="usage-metrics.js?v=330e92f8eb96" defer></script>
-<script src="app.js?v=plus-reference-20261102" defer></script>
+<script src="app.js?v=oft-all140k-20261103" defer></script>
 <link rel="stylesheet" href="policy.css">
 </head>
 <body>
 <a class="skip" href="#main">Skip to content</a>
 <header class="site-header"><div class="header-inner"><a class="brand" href="#overview" aria-label="Programmable VLA home"><img src="assets/kinex-logo.svg" width="34" height="34" alt="Kinex"><span>programmable<span class="brand-divider">/</span><span class="brand-benchmark">vla</span></span></a><nav aria-label="Main navigation"><a href="../index.html#overview">Easy</a><a href="../hard/index.html#overview">Hard</a><a href="#overview" data-nav="overview">Plus</a><a href="#tasks" data-nav="tasks">Explore tasks</a><a href="#methodology" data-nav="methodology">Protocol</a></nav><a class="header-link" href="https://huggingface.co/spaces/RLE-Bench/programmable-vla" target="_blank" rel="noopener">Hugging Face <span aria-hidden="true">↗</span></a></div></header>
 <main id="main" tabindex="-1"><div class="loading"><span class="spinner"></span>Loading benchmark results…</div></main>
-<footer class="site-footer"><div><img src="assets/kinex-logo.svg" width="22" height="22" alt=""><strong>Programmable VLA</strong><span class="footer-org">by RLE-Bench</span></div><div><a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/data.json" target="_blank">Plus results JSON</a><a data-result-file="sources.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/sources.json" target="_blank">Data sources</a><a href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/manifest.json" target="_blank">Manifest</a><span id="updated"></span></div></footer>
+<footer class="site-footer"><div><img src="assets/kinex-logo.svg" width="22" height="22" alt=""><strong>Programmable VLA</strong><span class="footer-org">by RLE-Bench</span></div><div><a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/plus/data.json" target="_blank">Plus results JSON</a><a data-result-file="sources.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/sources.json" target="_blank">Data sources</a><a href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b2d67a67dd2f9d8e12c1f8db13ba8de309739883/manifest.json" target="_blank">Manifest</a><span id="updated"></span></div></footer>
 <dialog id="image-viewer" aria-labelledby="image-caption"><div class="viewer-toolbar"><span id="image-caption"></span><a id="image-original" target="_blank" rel="noopener">Open original ↗</a><button class="icon-button" id="close-viewer" aria-label="Close image">×</button></div><div class="viewer-image"><button id="previous-image" class="viewer-arrow" aria-label="Previous image">←</button><img id="large-image" alt="Recorded agent observation"><button id="next-image" class="viewer-arrow" aria-label="Next image">→</button></div><p>Recorded agent observation · Use ← → to browse · Esc to close</p></dialog>
 <div id="toast" role="status" aria-live="polite"></div>
-<noscript><p>This site needs JavaScript. <a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/b07d88579523eeeefdf7ad2ca14ee65340af5d4e/hard/data.json">Download the complete results index.</a></p></noscript>
+<noscript><p>This site needs JavaScript. <a data-result-file="data.json" href="https://huggingface.co/datasets/RLE-Bench/programmable-vla/resolve/f24090b8f6c0ba26574f9248f6bfe2a6e78528ad/research/oft-all140k-20261103/panels/plus/data.json">Download the complete results index.</a></p></noscript>
 </body>
 </html>