yzzhao commited on
Commit
050b961
·
verified ·
1 Parent(s): 2b0bec7

Select the three Kinex VLA reruns and preserve previous valid attempts

Browse files
Files changed (8) hide show
  1. README.md +3 -1
  2. app.js +10 -5
  3. attempt-history.json +2087 -0
  4. cost-data.json +174 -0
  5. data.json +0 -0
  6. index.html +1 -1
  7. manifest.json +0 -0
  8. measured-instructions.json +21 -0
README.md CHANGED
@@ -10,8 +10,10 @@ license: apache-2.0
10
 
11
  # Programmable VLA
12
 
13
- RoboTwin 2.0 results: official standalone VLA evaluation on 50 tasks (one episode each, 25/50 successful), alongside the retained Kinex and Kinex + VLA rosbridge pilot. Instructions, policy inputs and outputs, sessions, usage, videos, and artifacts.
14
 
15
  Standalone and agent results use different execution protocols and initializations. The previous standalone rosbridge pilot remains available in `history.html`.
16
 
17
  Official videos show head, left wrist and right wrist at 4× the original 10 FPS recording. Wrist observations update at inference boundaries. This is action-index playback, not physical simulation time. Native 320×240 views are upscaled to the shared presentation layout.
 
 
 
10
 
11
  # Programmable VLA
12
 
13
+ RoboTwin 2.0 results: official standalone VLA evaluation on 50 tasks (one episode each, 25/50 successful), alongside the retained Kinex pilot and the October 29 Kinex + VLA research rerun (1/3 successful, three valid seed-0 episodes). Instructions, policy inputs and outputs, sessions, usage, videos, and artifacts.
14
 
15
  Standalone and agent results use different execution protocols and initializations. The previous standalone rosbridge pilot remains available in `history.html`.
16
 
17
  Official videos show head, left wrist and right wrist at 4× the original 10 FPS recording. Wrist observations update at inference boundaries. This is action-index playback, not physical simulation time. Native 320×240 views are upscaled to the shared presentation layout.
18
+
19
+ The latest Kinex + VLA runs made 0, 7 and 23 VLA calls. Of 30 inferences, 28 used the full task and two used agent-selected stage instructions. The successful handover used geometry without VLA. These outcomes do not isolate agentic incompatibility as the cause of failure. Previous valid episodes remain in each task's Previous attempts tab and the archived pilot.
app.js CHANGED
@@ -119,7 +119,7 @@ function opusPilotProgress(){
119
  return `<section class="panel" style="padding:18px 24px;margin-bottom:24px" aria-label="Claude Code campaign"><strong>Claude Code · Opus 5.5 / high</strong><p>${number(n.finished||0)} / ${number(c.planned_episodes)} episodes completed · ${number(n.running||0)} running · ${number(n.queued||0)} pending${n.needs_review?' · '+number(n.needs_review)+' under review':''}${n.interrupted?' · '+number(n.interrupted)+' interrupted':''}.</p><p class="source-note">${c.quota_waiting?'Waiting for model-service quota recovery · ':''}${number(c.published_episodes)} episode records published · <a href="${esc(c.status_url)}" target="_blank" rel="noopener">Results and protocol ↗</a></p></section>`;
120
  }
121
  function methodology() {
122
- return `<details class="panel methodology" id="methodology"><summary><span><span class="eyebrow">Reading the benchmark</span><strong>Protocol, metrics & provenance</strong></span><span class="summary-plus">+</span></summary><div class="method-grid"><div><h3>One selected episode</h3><p>Standalone: 50 tasks, one episode each, official seed group 0 with native expert admission. Kinex and Kinex + VLA: the retained three-task seed-0 pilot.</p></div><div><h3>Native success</h3><p>Success rate is passes / evaluated tasks, pooled over the selected episodes. Pending tasks do not count as failures. Results follow the original native verifier.</p></div><div><h3>Native control budget</h3><p>RLE-Bench task03 · RoboTwin 2.0 · ALOHA AgileX. Standalone uses official take_action and task-specific limits. Kinex uses rosbridge control frames. Initializations and physical durations are not matched.</p></div><div><h3>Policy & agent</h3><p>StarVLA/Qwen3-VL-OFT-Robotwin2, step 40,000. Kinex uses GPT-6 Astra / high. Kinex writes its own control loop around native run_policy; each inference receives three RGB images and language. Encoder changes and progress monitors are not used.</p></div><div><h3>Token accounting</h3><p>${usageDefinition()} API-equivalent cost uses the recorded Pi session estimates; account charges and GPU costs are not reported.</p></div></div><div class="method-links"><a href="${esc(resultURL('sources.json'))}" target="_blank">Data sources ↗</a><a href="history.html#overview">Previous rosbridge pilot ↗</a><a href="official/results.json">Official results JSON ↗</a><button class="text-button" data-export="all">Download comparison CSV ↓</button></div></details>`;
123
  }
124
  function taskBrowser(variant=null) {
125
  return `<section class="task-browser" id="tasks"><div class="section-heading bare"><div><div class="eyebrow">${variant?'Episode index':'Browse the benchmark'}</div><h2>Explore tasks <span class="heading-count">${plannedTasks(variant).length}</span></h2></div><button class="button secondary" data-export="${variant||'all'}">Export CSV ↓</button></div><div class="panel"><div class="filters"><label class="search-field"><span aria-hidden="true">⌕</span><input id="task-search" type="search" aria-label="Search tasks" placeholder="Search task name, instruction or ID" value="${esc(filters.query)}"></label><select id="task-family" aria-label="Environment"><option value="">All environments</option>${data.families.map(f=>`<option value="${f.id}" ${filters.family===f.id?'selected':''}>${f.name}</option>`).join('')}</select>${variant?'':`<select id="task-variant" aria-label="Agent for outcome filter">${data.variants.map(v=>`<option value="${v.id}" ${filters.variant===v.id?'selected':''}>${v.label||v.agent} · ${v.model}</option>`).join('')}</select>`}<select id="task-status" aria-label="Outcome"><option value="">All outcomes</option>${['pass','fail','pending'].map(s=>`<option value="${s}" ${filters.status===s?'selected':''}>${s[0].toUpperCase()+s.slice(1)}</option>`).join('')}</select><select id="task-sort" aria-label="Sort tasks"><option value="catalog" ${filters.sort==='catalog'?'selected':''}>Catalog order</option><option value="tokens" ${filters.sort==='tokens'?'selected':''}>Most fresh input tokens</option><option value="cost" ${filters.sort==='cost'?'selected':''}>Highest cost</option><option value="name" ${filters.sort==='name'?'selected':''}>Task name</option></select></div><div class="table-scroll"><table class="task-table ${variant?'single-variant':''}"><thead><tr><th>Task</th>${variant?'<th>Native result</th><th>Fresh input<small>Excludes read and write</small></th><th>Cache read</th><th>Cache write</th><th>Cache hit</th><th>Output</th><th>Total cost<small>Estimated USD</small></th><th>Action steps</th>':data.variants.map(v=>`<th class="${v.id===filters.variant?'focus-column':''}">${v.label||v.agent}<small>${v.model} · ${v.effort}</small><small>Native result · Estimated USD</small></th>`).join('')}</tr></thead><tbody id="task-rows"></tbody></table></div><div class="table-footer"><span id="task-count" aria-live="polite"></span><span id="filter-context"></span><button class="text-button" id="reset-filters">Reset filters</button></div></div></section>`;
@@ -135,10 +135,15 @@ function renderRows() {
135
  $('#filter-context').textContent=variant?variantName(variant):'Outcome filter & usage/cost sort: '+variantName(filters.variant);
136
  $$('.task-table thead th').forEach((th,i)=>{if(!variant&&i)th.classList.toggle('focus-column',data.variants[i-1].id===filters.variant);});
137
  }
 
 
 
 
 
138
  function overview() {
139
  current={type:'overview'};
140
  $('#main').innerHTML=`<section class="hero"><div><div class="eyebrow"><span class="tiny-logo">✦</span> RLE-Bench · Embodied agent evaluation</div><h1>Coding agents.<br><span>Physical tasks.</span></h1><p>Compare native success, token usage and recorded behavior.<br>Every result opens into the full agent session and its evidence.</p></div><div class="hero-facts"><div><strong>${data.families.length}</strong><span>environment</span></div><div><strong>${data.tasks.length}</strong><span>distinct tasks</span></div><div><strong>${data.variants.length}</strong><span>agent–model variants</span></div><span class="progress-label"><i></i>${data.benchmark_complete?'Evaluation complete':'Evaluation in progress'} · selected seeds</span></div></section>
141
- ${recordingPilotProgress()}${campaignProgress()}${opusPilotProgress()}<div class="agent-cards">${data.variants.map(v=>{const s=stats(episodes(v.id));return `<a class="agent-card ${s.n?'':'planned'}" href="#runs/${v.id}" style="--variant:${colors[v.id]}"><div class="card-header"><span class="agent-name">${icon(v)}${esc(v.label||v.agent)}</span><span class="state-label">${s.n===plannedTasks(v.id).length?'Complete':s.n?'Partial':'Planned'}</span></div><p class="card-model">${modelLabel(v)}</p><div class="card-rate"><strong>${pct(s.rate)}</strong><span>native success</span></div><div class="card-stats"><span><b>${s.n?s.passes+' / '+s.n:'—'}</b> passed</span><span class="card-cost" title="${esc(costNote(s))}"><b>${costValue(s)}</b> total cost<small>Estimated USD</small></span></div><div class="completion-track"><span style="width:${s.n/plannedTasks(v.id).length*100}%"></span></div><div class="card-bottom"><span>${s.n} / ${plannedTasks(v.id).length} evaluated</span><span>${s.n?'Explore runs':'View planned'} ${arrow}</span></div></a>`;}).join('')}</div><div class="scope-note"><span><i class="status-dot"></i>Success rates and costs cover selected evaluated tasks only.</span><span>Recorded effort · Cache-aware costs · Native verdicts preserved</span></div><div id="comparison-mount">${comparisonTable()}</div>${taskBrowser()}${methodology()}`;
142
  renderRows();
143
  }
144
  function runsPage(variant,params,restoredFilters=null) {
@@ -324,7 +329,7 @@ function evidencePanel(task,ep) {
324
  return `<div class="evidence-intro"><h2>Evidence & provenance</h2><p>Original files retained with their published checksums. Native verdicts and measured source identities remain unchanged.</p></div><div class="evidence-grid">${evidenceGroups.map(([title,desc,items])=>evidenceCard(title,desc,items,ep)).join('')}${extras.length?evidenceCard('Additional records','Episode-specific checks and retained references.',extras.map(k=>[k,k.replaceAll('_',' ')]),ep):''}</div><section class="panel provenance-panel"><div class="section-heading"><div><h2>Recorded identity</h2><p>Source and protocol belonging to this episode.</p></div></div><dl class="provenance-list"><dt>Episode</dt><dd><code>${esc(ep.source_episode_id)}</code></dd><dt>Model</dt><dd>${esc(ep.model)} / ${ep.effort}</dd><dt>Space</dt><dd>${external('https://huggingface.co/spaces/'+source.space,source.space)}</dd><dt>Original evidence revision</dt><dd><code>${source.evidence_revision}</code></dd>${ep.kinex_revision?`<dt>Kinex revision</dt><dd><code>${esc(ep.kinex_revision)}</code></dd>`:''}${Object.entries(ep.provenance?.sources||{}).map(([k,v])=>`<dt>${esc(k)}</dt><dd><code>${esc(v.revision||'Unrecorded revision')}</code>${v.dirty?' <span class="tag">Local changes recorded</span>':''}</dd>`).join('')}<dt>Recorded action-step budget</dt><dd>${number(actionSteps(ep.measured_protocol?.max_steps,ep))} steps</dd><dt>Campaign planned action-step budget</dt><dd>${number(actionSteps(plannedProtocol(task,current.variant).max_control_steps,task))} steps</dd><dt>Current benchmark default</dt><dd>${number(currentBudget(task))} action steps</dd></dl><details class="protocol-json"><summary>Inspect the measured protocol JSON</summary><pre>${esc(JSON.stringify(ep.measured_protocol,null,2))}</pre></details></section><section class="panel evidence-preview-panel"><div class="section-heading"><div><h2>Preview an evidence file</h2><p>Inspect the recorded JSON without leaving this page.</p></div><select id="evidence-select" aria-label="Choose evidence preview"><option value="">Choose a file…</option>${[['verdict','Native verdict'],['timing','Episode timing'],['protocol','Measured protocol'],['provenance','Source identities'],['usage','Token usage'],['analysis','Episode analysis'],['media_check','Media validation'],['recording_manifest','Recording manifest']].filter(([k])=>ep.links[k]).map(([k,l])=>`<option value="${k}">${l}</option>`).join('')}</select></div><pre id="evidence-preview" hidden tabindex="0"></pre></section>`;
325
  }
326
  function historyPanel(history,task) {
327
- return `<section class="panel history-panel"><div class="section-heading"><div><h2>Previous attempts</h2><p>Archived attempts are preserved separately and excluded from the current success rate, usage and costs.</p></div><a href="${esc(resultURL('attempt-history.json'))}" target="_blank" class="button secondary">Full archive ↗</a></div>${history.length?history.map(h=>`<article class="history-row"><div class="history-outcome">${badge(h)}<small>${h.valid===false?'Interrupted / invalid':typeof h.success==='boolean'?'Archived result':'Verdict not recorded'}</small></div><div><strong>${esc((h.status||'Archived attempt').replaceAll('_',' '))}</strong><p><code>${esc(h.id||h.job||h.dataset_revision||'Historical record')}</code></p><span class="history-stats">${h.steps!=null?number(actionSteps(h.steps,task))+' action steps · ':''}${h.usage?'Fresh input '+compact(UsageMetrics.usage(h).fresh)+' · cache read '+compact(h.usage.cached_input_tokens)+' · cache write '+compact(UsageMetrics.usage(h).cacheWrite)+' · output '+compact(UsageMetrics.usage(h).output)+' · ':''}${h.dataset_revision?'Revision '+h.dataset_revision.slice(0,8):''}</span>${h.analysis?.execution?`<p>${esc(h.analysis.execution)}</p>`:''}${h.instruction?`<details><summary>Measured instruction</summary><p>${esc(h.instruction)}</p></details>`:''}<div class="inline-links">${Object.entries(h.links||{}).filter(([k])=>['video','verdict','provenance','transcript','terminal_frame','native_session'].includes(k)).map(([k,u])=>external(u,k.replaceAll('_',' '))).join('')}</div></div></article>`).join(''):'<div class="empty-state">No earlier attempts are listed for this task and agent combination.</div>'}</section>`;
328
  }
329
  function notFound() {$('#main').innerHTML='<div class="empty-state"><h1>Page not found</h1><p><a href="#overview">Return to the benchmark</a></p></div>';}
330
  // Keep each task list's view with its browser history entry.
@@ -464,7 +469,7 @@ function renderPolicy(index) {
464
  const textParts=p=>(p||[]).filter(x=>x.type!=='image');
465
  const toolImages=caller?caller.tool_result.parts.filter(p=>p.type==='image').map(p=>{const url=new URL(p.src,new URL(ep.links.transcript,location.href)).href;return `<a href="${esc(url)}" target="_blank"><img src="${esc(url)}" alt="Observation returned to the language agent" style="width:240px;max-width:100%"></a>`;}).join(''):'';
466
  const cols=record.contract.output_layout;
467
- $('#policy-body').innerHTML=`<label>Inference <select class="policy-select" id="policy-call" aria-label="VLA inference">${record.calls.map((x,i)=>`<option value="${i}" ${i===index?'selected':''}>${i+1} / ${record.calls.length} · ${x.call_id.slice(0,12)}</option>`).join('')}</select></label><div class="inline-links"><code>${esc(c.call_id)}</code><span>${c.service_response.inference_s.toFixed(3)} s · 16 × 14 actions</span></div><h3>Task instruction</h3><div class="instruction-text">${esc(c.request.task)}</div><div class="policy-images">${images}</div><p class="subtle">Model images use the recorded RGB inputs with the executed resize to 224 × 224. No joint state or agent conversation is passed to the VLA.</p>${jsonBlock('Model-service request · retained image paths',c.request)}${jsonBlock('Backbone message · reconstructed from executed code',c.backbone_messages)}<h3>Model-service output</h3><p class="subtle">The complete response consumed by Kinex run_policy. Joint values are absolute radians; grippers use −1 open, +1 closed.</p><div class="policy-table"><table><thead><tr><th>Action</th>${cols.map(x=>`<th>${esc(x.replace('left_','L ').replace('right_','R '))}</th>`).join('')}</tr></thead><tbody>${c.service_response.actions.map((row,i)=>`<tr><th>${i+1}</th>${row.map(x=>`<td title="${x}">${x.toFixed(5)}</td>`).join('')}</tr>`).join('')}</tbody></table></div>${jsonBlock('Complete service response · full precision',c.service_response)}${jsonBlock('Internal diagnostic · denormalized training-order actions',c.internal_diagnostics.denormalized_actions)}${caller?`<h3>Kinex tool call & result</h3><p class="subtle">This inference falls within the recorded exec call below. One exec can contain several inferences. The language agent receives the tool result, not the model's raw action array.</p>${jsonBlock('Agent tool call',caller.tool_call,true)}${jsonBlock('Tool result received by the language agent',{...caller.tool_result,parts:textParts(caller.tool_result.parts)},true)}<div class="inline-links">${toolImages}<a href="${taskRoute(ep.task_key,ep.variant,'session')}">Read the full session ↗</a><a href="${taskRoute(ep.task_key,ep.variant,'artifacts')}">Inspect agent artifacts ↗</a></div>${c.executed_tool_source?jsonBlock('Agent-authored code at this call',c.executed_tool_source):''}`:'<p class="subtle">Standalone execution; no language-agent tool call.</p>'}<details class="tool-block"><summary>Recording coverage & model contract</summary><p>Temporary image filenames, normalized action tensors and processor tensors were not retained. Recorded images, task text and both complete action arrays are available for every inference.</p><pre>${esc(JSON.stringify(record.contract,null,2))}</pre><div class="inline-links">${external(record.contract.model_metadata,'Model configuration')}${external(record.contract.executed_sources,'Executed source checksums')}${external('policy-contract/model_node.py','Model service')}${external('policy-contract/starvla.py','Policy adapter')}${external('policy-contract/kinex_policy.py','Kinex run_policy')}</div></details>`;
468
  $('#policy-call').onchange=e=>renderPolicy(Number(e.target.value));
469
  }
470
  function toolsShell(ep) {return `<section class="panel session-panel"><div class="section-heading"><div><h2>Tool calls</h2><p>Original tool inputs and outputs from the agent trace.</p></div>${external(ep.links.tool_calls,'Tool calls JSON')}</div><div class="policy-body" id="tools-body">Loading tool calls…</div></section>`;}
@@ -492,6 +497,6 @@ boot();
492
  function renderOfficialPolicy(index) {
493
  const r=policyRecord,c=r.calls[index],ep=current.ep;
494
  const ims=c.images.map(im=>`<figure><a href="${esc(im.model_image)}" target="_blank"><img src="${esc(im.model_image)}" alt="${esc(im.camera)} model input"></a><figcaption>${esc(im.camera)} · 224 × 224<br><a href="${esc(im.recorded)}" target="_blank">Recorded RGB · ${im.recorded_size.join(' × ')} ↗</a></figcaption></figure>`).join('');
495
- $('#policy-body').innerHTML=`<label>Inference <select class="policy-select" id="policy-call" aria-label="VLA inference">${r.calls.map((x,i)=>`<option value="${i}" ${i===index?'selected':''}>${i+1} / ${r.calls.length} · action ${x.native_step}</option>`).join('')}</select></label><div class="inline-links"><code>${esc(c.call_id)}</code><span>${c.service_response.inference_s.toFixed(3)} s · 16 × 14 actions</span></div><h3>Task instruction</h3><div class="instruction-text">${esc(c.request.task)}</div><div class="policy-images">${ims}</div><p class="subtle">Exact transmitted RGB pixels. No joint state or agent conversation is passed to the VLA.</p>${jsonBlock('Complete model request · image arrays linked as PNGs',c.request.wire_payload)}<h3>Native action chunk</h3><p class="subtle">Reordered model output consumed by the official evaluator. Absolute joint radians; gripper opening: 0 closed, 1 open.</p><div class="policy-table"><table><thead><tr><th>Action</th>${r.contract.output_layout.map(x=>`<th>${esc(x.replace('left_','L ').replace('right_','R '))}</th>`).join('')}</tr></thead><tbody>${c.service_response.actions.map((row,i)=>`<tr><th>${i+1}</th>${row.map(x=>`<td title="${x}">${x.toFixed(5)}</td>`).join('')}</tr>`).join('')}</tbody></table></div>${jsonBlock('Complete recorded StarVLA response · full precision',c.raw_response)}${jsonBlock('Reordered native actions · full precision',c.service_response.actions)}<p class="subtle">Standalone execution; no language-agent session or tool call. Token and API costs cover the language agent only; local GPU cost is not reported.</p>${jsonBlock('Recording coverage & model contract',r.contract)}<div class="inline-links">${external(r.contract.model_metadata,'Model configuration')}${external(r.contract.executed_sources,'Source identities')}${external('official/adapters.py','Policy adapter')}${external('official/integrations.py','Official evaluator integration')}</div>`;
496
  $('#policy-call').onchange=e=>renderPolicy(Number(e.target.value));
497
  }
 
119
  return `<section class="panel" style="padding:18px 24px;margin-bottom:24px" aria-label="Claude Code campaign"><strong>Claude Code · Opus 5.5 / high</strong><p>${number(n.finished||0)} / ${number(c.planned_episodes)} episodes completed · ${number(n.running||0)} running · ${number(n.queued||0)} pending${n.needs_review?' · '+number(n.needs_review)+' under review':''}${n.interrupted?' · '+number(n.interrupted)+' interrupted':''}.</p><p class="source-note">${c.quota_waiting?'Waiting for model-service quota recovery · ':''}${number(c.published_episodes)} episode records published · <a href="${esc(c.status_url)}" target="_blank" rel="noopener">Results and protocol ↗</a></p></section>`;
120
  }
121
  function methodology() {
122
+ return `<details class="panel methodology" id="methodology"><summary><span><span class="eyebrow">Reading the benchmark</span><strong>Protocol, metrics & provenance</strong></span><span class="summary-plus">+</span></summary><div class="method-grid"><div><h3>One selected episode</h3><p>Standalone: 50 tasks, one episode each, official seed group 0 with native expert admission. Kinex: the retained three-task seed-0 pilot. Kinex + VLA: the October 29 research rerun on those same three tasks and seed 0; previous valid attempts remain archived.</p></div><div><h3>Native success</h3><p>Success rate is passes / evaluated tasks, pooled over the selected episodes. Pending tasks do not count as failures. Results follow the original native verifier.</p></div><div><h3>Native control budget</h3><p>RLE-Bench task03 · RoboTwin 2.0 · ALOHA AgileX. Standalone uses official take_action and task-specific limits. Kinex uses rosbridge control frames. Initializations and physical durations are not matched.</p></div><div><h3>Policy & agent</h3><p>StarVLA/Qwen3-VL-OFT-Robotwin2, step 40,000. Kinex uses GPT-6 Astra / high. Kinex chooses whether to use VLA, writes its own control loop, and chooses full-task or stage language. Each inference receives three RGB images and task text. The rerun returns untimed joint waypoints, executed by Kinex trajectories or agent-authored code. Encoder changes and progress monitors are not used.</p></div><div><h3>Token accounting</h3><p>${usageDefinition()} API-equivalent cost uses the recorded Pi session estimates; account charges and GPU costs are not reported.</p></div></div><div class="method-links"><a href="${esc(resultURL('sources.json'))}" target="_blank">Data sources ↗</a><a href="history.html#overview">Previous rosbridge pilot ↗</a><a href="official/results.json">Official results JSON ↗</a><button class="text-button" data-export="all">Download comparison CSV ↓</button></div></details>`;
123
  }
124
  function taskBrowser(variant=null) {
125
  return `<section class="task-browser" id="tasks"><div class="section-heading bare"><div><div class="eyebrow">${variant?'Episode index':'Browse the benchmark'}</div><h2>Explore tasks <span class="heading-count">${plannedTasks(variant).length}</span></h2></div><button class="button secondary" data-export="${variant||'all'}">Export CSV ↓</button></div><div class="panel"><div class="filters"><label class="search-field"><span aria-hidden="true">⌕</span><input id="task-search" type="search" aria-label="Search tasks" placeholder="Search task name, instruction or ID" value="${esc(filters.query)}"></label><select id="task-family" aria-label="Environment"><option value="">All environments</option>${data.families.map(f=>`<option value="${f.id}" ${filters.family===f.id?'selected':''}>${f.name}</option>`).join('')}</select>${variant?'':`<select id="task-variant" aria-label="Agent for outcome filter">${data.variants.map(v=>`<option value="${v.id}" ${filters.variant===v.id?'selected':''}>${v.label||v.agent} · ${v.model}</option>`).join('')}</select>`}<select id="task-status" aria-label="Outcome"><option value="">All outcomes</option>${['pass','fail','pending'].map(s=>`<option value="${s}" ${filters.status===s?'selected':''}>${s[0].toUpperCase()+s.slice(1)}</option>`).join('')}</select><select id="task-sort" aria-label="Sort tasks"><option value="catalog" ${filters.sort==='catalog'?'selected':''}>Catalog order</option><option value="tokens" ${filters.sort==='tokens'?'selected':''}>Most fresh input tokens</option><option value="cost" ${filters.sort==='cost'?'selected':''}>Highest cost</option><option value="name" ${filters.sort==='name'?'selected':''}>Task name</option></select></div><div class="table-scroll"><table class="task-table ${variant?'single-variant':''}"><thead><tr><th>Task</th>${variant?'<th>Native result</th><th>Fresh input<small>Excludes read and write</small></th><th>Cache read</th><th>Cache write</th><th>Cache hit</th><th>Output</th><th>Total cost<small>Estimated USD</small></th><th>Action steps</th>':data.variants.map(v=>`<th class="${v.id===filters.variant?'focus-column':''}">${v.label||v.agent}<small>${v.model} · ${v.effort}</small><small>Native result · Estimated USD</small></th>`).join('')}</tr></thead><tbody id="task-rows"></tbody></table></div><div class="table-footer"><span id="task-count" aria-live="polite"></span><span id="filter-context"></span><button class="text-button" id="reset-filters">Reset filters</button></div></div></section>`;
 
135
  $('#filter-context').textContent=variant?variantName(variant):'Outcome filter & usage/cost sort: '+variantName(filters.variant);
136
  $$('.task-table thead th').forEach((th,i)=>{if(!variant&&i)th.classList.toggle('focus-column',data.variants[i-1].id===filters.variant);});
137
  }
138
+ function researchRerunNote() {
139
+ if(!data.research_update)return '';
140
+ return `<aside class="native-note" data-research-rerun><span class="note-symbol">i</span><div><strong>Kinex + VLA · October 29 rerun</strong><p>1 / 3 native successes · 30 VLA calls: 28 full-task and 2 stage instructions. The successful handover used geometric control with no VLA calls. Agent strategy remained autonomous; these three episodes do not isolate the cause of failure.</p><div class="inline-links"><a href="#runs/kinex_policy">Explore the new runs ↗</a><a href="research/kinex-vla-rerun-20261029/summary.json">Calling behavior & interpretation ↗</a><a href="history.html#overview">Previous pilot ↗</a></div></div></aside>`;
141
+ }
142
+
143
  function overview() {
144
  current={type:'overview'};
145
  $('#main').innerHTML=`<section class="hero"><div><div class="eyebrow"><span class="tiny-logo">✦</span> RLE-Bench · Embodied agent evaluation</div><h1>Coding agents.<br><span>Physical tasks.</span></h1><p>Compare native success, token usage and recorded behavior.<br>Every result opens into the full agent session and its evidence.</p></div><div class="hero-facts"><div><strong>${data.families.length}</strong><span>environment</span></div><div><strong>${data.tasks.length}</strong><span>distinct tasks</span></div><div><strong>${data.variants.length}</strong><span>agent–model variants</span></div><span class="progress-label"><i></i>${data.benchmark_complete?'Evaluation complete':'Evaluation in progress'} · selected seeds</span></div></section>
146
+ ${researchRerunNote()}${recordingPilotProgress()}${campaignProgress()}${opusPilotProgress()}<div class="agent-cards">${data.variants.map(v=>{const s=stats(episodes(v.id));return `<a class="agent-card ${s.n?'':'planned'}" href="#runs/${v.id}" style="--variant:${colors[v.id]}"><div class="card-header"><span class="agent-name">${icon(v)}${esc(v.label||v.agent)}</span><span class="state-label">${s.n===plannedTasks(v.id).length?'Complete':s.n?'Partial':'Planned'}</span></div><p class="card-model">${modelLabel(v)}</p><div class="card-rate"><strong>${pct(s.rate)}</strong><span>native success</span></div><div class="card-stats"><span><b>${s.n?s.passes+' / '+s.n:'—'}</b> passed</span><span class="card-cost" title="${esc(costNote(s))}"><b>${costValue(s)}</b> total cost<small>Estimated USD</small></span></div><div class="completion-track"><span style="width:${s.n/plannedTasks(v.id).length*100}%"></span></div><div class="card-bottom"><span>${s.n} / ${plannedTasks(v.id).length} evaluated</span><span>${s.n?'Explore runs':'View planned'} ${arrow}</span></div></a>`;}).join('')}</div><div class="scope-note"><span><i class="status-dot"></i>Success rates and costs cover selected evaluated tasks only.</span><span>Recorded effort · Cache-aware costs · Native verdicts preserved</span></div><div id="comparison-mount">${comparisonTable()}</div>${taskBrowser()}${methodology()}`;
147
  renderRows();
148
  }
149
  function runsPage(variant,params,restoredFilters=null) {
 
329
  return `<div class="evidence-intro"><h2>Evidence & provenance</h2><p>Original files retained with their published checksums. Native verdicts and measured source identities remain unchanged.</p></div><div class="evidence-grid">${evidenceGroups.map(([title,desc,items])=>evidenceCard(title,desc,items,ep)).join('')}${extras.length?evidenceCard('Additional records','Episode-specific checks and retained references.',extras.map(k=>[k,k.replaceAll('_',' ')]),ep):''}</div><section class="panel provenance-panel"><div class="section-heading"><div><h2>Recorded identity</h2><p>Source and protocol belonging to this episode.</p></div></div><dl class="provenance-list"><dt>Episode</dt><dd><code>${esc(ep.source_episode_id)}</code></dd><dt>Model</dt><dd>${esc(ep.model)} / ${ep.effort}</dd><dt>Space</dt><dd>${external('https://huggingface.co/spaces/'+source.space,source.space)}</dd><dt>Original evidence revision</dt><dd><code>${source.evidence_revision}</code></dd>${ep.kinex_revision?`<dt>Kinex revision</dt><dd><code>${esc(ep.kinex_revision)}</code></dd>`:''}${Object.entries(ep.provenance?.sources||{}).map(([k,v])=>`<dt>${esc(k)}</dt><dd><code>${esc(v.revision||'Unrecorded revision')}</code>${v.dirty?' <span class="tag">Local changes recorded</span>':''}</dd>`).join('')}<dt>Recorded action-step budget</dt><dd>${number(actionSteps(ep.measured_protocol?.max_steps,ep))} steps</dd><dt>Campaign planned action-step budget</dt><dd>${number(actionSteps(plannedProtocol(task,current.variant).max_control_steps,task))} steps</dd><dt>Current benchmark default</dt><dd>${number(currentBudget(task))} action steps</dd></dl><details class="protocol-json"><summary>Inspect the measured protocol JSON</summary><pre>${esc(JSON.stringify(ep.measured_protocol,null,2))}</pre></details></section><section class="panel evidence-preview-panel"><div class="section-heading"><div><h2>Preview an evidence file</h2><p>Inspect the recorded JSON without leaving this page.</p></div><select id="evidence-select" aria-label="Choose evidence preview"><option value="">Choose a file…</option>${[['verdict','Native verdict'],['timing','Episode timing'],['protocol','Measured protocol'],['provenance','Source identities'],['usage','Token usage'],['analysis','Episode analysis'],['media_check','Media validation'],['recording_manifest','Recording manifest']].filter(([k])=>ep.links[k]).map(([k,l])=>`<option value="${k}">${l}</option>`).join('')}</select></div><pre id="evidence-preview" hidden tabindex="0"></pre></section>`;
330
  }
331
  function historyPanel(history,task) {
332
+ return `<section class="panel history-panel"><div class="section-heading"><div><h2>Previous attempts</h2><p>Archived attempts are preserved separately and excluded from the current success rate, usage and costs.</p></div><a href="${esc(resultURL('attempt-history.json'))}" target="_blank" class="button secondary">Full archive ↗</a></div>${history.length?history.map(h=>`<article class="history-row"><div class="history-outcome">${badge(h)}<small>${h.valid===false?'Interrupted / invalid':typeof h.success==='boolean'?'Archived result':'Verdict not recorded'}</small></div><div><strong>${esc((h.status||'Archived attempt').replaceAll('_',' '))}</strong><p><code>${esc(h.id||h.job||h.dataset_revision||'Historical record')}</code></p><span class="history-stats">${h.steps!=null?number(actionSteps(h.steps,task))+' action steps · ':''}${h.usage?'Fresh input '+compact(UsageMetrics.usage(h).fresh)+' · cache read '+compact(h.usage.cached_input_tokens)+' · cache write '+compact(UsageMetrics.usage(h).cacheWrite)+' · output '+compact(UsageMetrics.usage(h).output)+' · ':''}${h.dataset_revision?'Revision '+h.dataset_revision.slice(0,8):''}</span>${h.analysis?.execution?`<p>${esc(h.analysis.execution)}</p>`:''}${h.instruction?`<details><summary>Measured instruction</summary><p>${esc(h.instruction)}</p></details>`:''}<div class="inline-links">${Object.entries(h.links||{}).filter(([k])=>['video','verdict','provenance','transcript','terminal_frame','native_session','result_page'].includes(k)).map(([k,u])=>external(u,k.replaceAll('_',' '))).join('')}</div></div></article>`).join(''):'<div class="empty-state">No earlier attempts are listed for this task and agent combination.</div>'}</section>`;
333
  }
334
  function notFound() {$('#main').innerHTML='<div class="empty-state"><h1>Page not found</h1><p><a href="#overview">Return to the benchmark</a></p></div>';}
335
  // Keep each task list's view with its browser history entry.
 
469
  const textParts=p=>(p||[]).filter(x=>x.type!=='image');
470
  const toolImages=caller?caller.tool_result.parts.filter(p=>p.type==='image').map(p=>{const url=new URL(p.src,new URL(ep.links.transcript,location.href)).href;return `<a href="${esc(url)}" target="_blank"><img src="${esc(url)}" alt="Observation returned to the language agent" style="width:240px;max-width:100%"></a>`;}).join(''):'';
471
  const cols=record.contract.output_layout;
472
+ $('#policy-body').innerHTML=`<label>Inference <select class="policy-select" id="policy-call" aria-label="VLA inference">${record.calls.map((x,i)=>`<option value="${i}" ${i===index?'selected':''}>${i+1} / ${record.calls.length} · ${x.call_id.slice(0,12)} · ${x.request.task===ep.instruction?'full task':'stage / changed task'}</option>`).join('')}</select></label><div class="inline-links"><code>${esc(c.call_id)}</code><span>${c.service_response.inference_s.toFixed(3)} s · 16 × 14 actions</span></div><h3>Task instruction</h3><div class="instruction-text">${esc(c.request.task)}</div><div class="policy-images">${images}</div><p class="subtle">Model images use the recorded RGB inputs with the executed resize to 224 × 224. No joint state or agent conversation is passed to the VLA.</p>${jsonBlock('Model-service request · retained image paths',c.request)}${jsonBlock('Backbone message · reconstructed from executed code',c.backbone_messages)}<h3>Model-service output</h3><p class="subtle">${c.service_response.waypoints?'Untimed joint waypoints consumed by the recorded caller. Kinex or agent-authored code determines their physical timing.':'The recorded action response consumed by the caller.'} Joint values are absolute radians; grippers use −1 open, +1 closed.</p><div class="policy-table"><table><thead><tr><th>Action</th>${cols.map(x=>`<th>${esc(x.replace('left_','L ').replace('right_','R '))}</th>`).join('')}</tr></thead><tbody>${(c.service_response.waypoints||c.service_response.actions).map((row,i)=>`<tr><th>${i+1}</th>${row.map(x=>`<td title="${x}">${x.toFixed(5)}</td>`).join('')}</tr>`).join('')}</tbody></table></div>${jsonBlock('Complete service response · full precision',c.service_response)}${c.execution?jsonBlock('Actual execution · target count and world time',{executor:c.planner,...c.execution},true):''}${jsonBlock('Internal diagnostic · denormalized training-order actions',c.internal_diagnostics.denormalized_actions)}${caller?`<h3>Kinex tool call & result</h3><p class="subtle">This inference is joined to the recorded tool call and result below. Agent-authored code may consume the model output directly; the language agent receives the displayed tool response.</p>${jsonBlock('Agent tool call',caller.tool_call,true)}${jsonBlock('Tool result received by the language agent',{...caller.tool_result,parts:textParts(caller.tool_result.parts)},true)}<div class="inline-links">${toolImages}<a href="${taskRoute(ep.task_key,ep.variant,'session')}">Read the full session ↗</a><a href="${taskRoute(ep.task_key,ep.variant,'artifacts')}">Inspect agent artifacts ↗</a></div>${c.executed_tool_source?jsonBlock('Agent-authored code at this call',c.executed_tool_source):''}`:'<p class="subtle">Standalone execution; no language-agent tool call.</p>'}<details class="tool-block"><summary>Recording coverage & model contract</summary><p>Temporary image filenames, normalized action tensors and processor tensors were not retained. Recorded images, task text and both complete action arrays are available for every inference.</p><pre>${esc(JSON.stringify(record.contract,null,2))}</pre><div class="inline-links">${external(record.contract.model_metadata,'Model configuration')}${external(record.contract.executed_sources,'Executed source checksums')}${external(record.contract.model_service_source||'policy-contract/model_node.py','Model service')}${external(record.contract.adapter_source||'policy-contract/starvla.py','Policy adapter')}${external(record.contract.helper_source||'policy-contract/kinex_policy.py','Kinex run_policy')}</div></details>`;
473
  $('#policy-call').onchange=e=>renderPolicy(Number(e.target.value));
474
  }
475
  function toolsShell(ep) {return `<section class="panel session-panel"><div class="section-heading"><div><h2>Tool calls</h2><p>Original tool inputs and outputs from the agent trace.</p></div>${external(ep.links.tool_calls,'Tool calls JSON')}</div><div class="policy-body" id="tools-body">Loading tool calls…</div></section>`;}
 
497
  function renderOfficialPolicy(index) {
498
  const r=policyRecord,c=r.calls[index],ep=current.ep;
499
  const ims=c.images.map(im=>`<figure><a href="${esc(im.model_image)}" target="_blank"><img src="${esc(im.model_image)}" alt="${esc(im.camera)} model input"></a><figcaption>${esc(im.camera)} · 224 × 224<br><a href="${esc(im.recorded)}" target="_blank">Recorded RGB · ${im.recorded_size.join(' × ')} ↗</a></figcaption></figure>`).join('');
500
+ $('#policy-body').innerHTML=`<label>Inference <select class="policy-select" id="policy-call" aria-label="VLA inference">${r.calls.map((x,i)=>`<option value="${i}" ${i===index?'selected':''}>${i+1} / ${r.calls.length} · action ${x.native_step}</option>`).join('')}</select></label><div class="inline-links"><code>${esc(c.call_id)}</code><span>${c.service_response.inference_s.toFixed(3)} s · 16 × 14 actions</span></div><h3>Task instruction</h3><div class="instruction-text">${esc(c.request.task)}</div><div class="policy-images">${ims}</div><p class="subtle">Exact transmitted RGB pixels. No joint state or agent conversation is passed to the VLA.</p>${jsonBlock('Complete model request · image arrays linked as PNGs',c.request.wire_payload)}<h3>Native action chunk</h3><p class="subtle">Reordered model output consumed by the official evaluator. Absolute joint radians; gripper opening: 0 closed, 1 open.</p><div class="policy-table"><table><thead><tr><th>Action</th>${r.contract.output_layout.map(x=>`<th>${esc(x.replace('left_','L ').replace('right_','R '))}</th>`).join('')}</tr></thead><tbody>${(c.service_response.waypoints||c.service_response.actions).map((row,i)=>`<tr><th>${i+1}</th>${row.map(x=>`<td title="${x}">${x.toFixed(5)}</td>`).join('')}</tr>`).join('')}</tbody></table></div>${jsonBlock('Complete recorded StarVLA response · full precision',c.raw_response)}${jsonBlock('Reordered native actions · full precision',c.service_response.actions)}<p class="subtle">Standalone execution; no language-agent session or tool call. Token and API costs cover the language agent only; local GPU cost is not reported.</p>${jsonBlock('Recording coverage & model contract',r.contract)}<div class="inline-links">${external(r.contract.model_metadata,'Model configuration')}${external(r.contract.executed_sources,'Source identities')}${external('official/adapters.py','Policy adapter')}${external('official/integrations.py','Official evaluator integration')}</div>`;
501
  $('#policy-call').onchange=e=>renderPolicy(Number(e.target.value));
502
  }
attempt-history.json ADDED
@@ -0,0 +1,2087 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "id": "kinex_policy-1",
4
+ "variant": "kinex_policy",
5
+ "task_key": "task03/01",
6
+ "family": "task03",
7
+ "source_family": "task03",
8
+ "provider": "rosbridge",
9
+ "slot": "01",
10
+ "seed": 0,
11
+ "source_episode_id": "kinex_policy-1",
12
+ "status": "archived_completed",
13
+ "success": true,
14
+ "valid": true,
15
+ "execution": {
16
+ "status": "finished",
17
+ "reason": null
18
+ },
19
+ "verdict": {
20
+ "evidence_valid": true,
21
+ "success": true,
22
+ "termination": "success",
23
+ "steps": 523
24
+ },
25
+ "steps": 523,
26
+ "simulation_time_s": 20.92,
27
+ "wall_time_s": 518.141632,
28
+ "model": "GPT-6 Astra",
29
+ "effort": "high",
30
+ "harness": "kinex",
31
+ "instruction": "use the left arm to grasp the red block on the table, hand it over to the right arm, and set it down in the center of the blue pad. Release it with the right gripper open",
32
+ "native_instruction": "use the left arm to grasp the red block on the table, handover it to the right arm and place it on the blue pad",
33
+ "instruction_policy": "modified",
34
+ "usage": {
35
+ "accounting": "reported-responses",
36
+ "audit_complete": true,
37
+ "audit_source": "Provider terminal response usage",
38
+ "cache_hit_rate": 0.9718491681630644,
39
+ "cache_read_input_tokens": 1536512,
40
+ "cache_reported_input_tokens": 1581019,
41
+ "cache_write_basis": "provider-reported",
42
+ "cache_write_input_tokens": 0,
43
+ "cache_write_reported_input_tokens": 1581019,
44
+ "cached_input_tokens": 1536512,
45
+ "cost_usd": null,
46
+ "fresh_input_tokens": 44507,
47
+ "harbor_mismatches": {},
48
+ "input_breakdown_complete": true,
49
+ "input_tokens": 1581019,
50
+ "known_cache_write_input_tokens": 0,
51
+ "known_cached_input_tokens": 1536512,
52
+ "known_fresh_input_tokens": 44507,
53
+ "known_input_tokens": 1581019,
54
+ "known_output_tokens": 6998,
55
+ "known_reasoning_output_tokens": 1710,
56
+ "output_tokens": 6998,
57
+ "parser": "harbor/0.22.0",
58
+ "reasoning_output_tokens": 1710,
59
+ "reasoning_reported_output_tokens": 6998,
60
+ "rejected_attempts": 0,
61
+ "reported_responses": {
62
+ "cache_reported_input_tokens": 48,
63
+ "cache_write_input_tokens": 48,
64
+ "cache_write_reported_input_tokens": 48,
65
+ "cached_input_tokens": 48,
66
+ "fresh_input_tokens": 48,
67
+ "input_tokens": 48,
68
+ "output_tokens": 48,
69
+ "reasoning_output_tokens": 48,
70
+ "reasoning_reported_output_tokens": 48
71
+ },
72
+ "request_attempts": 48,
73
+ "response_count": 48,
74
+ "response_ids_complete": true,
75
+ "schema": "rlebench/token-usage/1",
76
+ "source": "Harbor session metrics",
77
+ "total_tokens": 1588017,
78
+ "uncached_input_tokens": 44507,
79
+ "unresolved_attempts": [],
80
+ "usage_records": [
81
+ {
82
+ "cache_write_input_tokens": 0,
83
+ "cached_input_tokens": 0,
84
+ "input_tokens": 11317,
85
+ "output_tokens": 19,
86
+ "reasoning_output_tokens": 0
87
+ },
88
+ {
89
+ "cache_write_input_tokens": 0,
90
+ "cached_input_tokens": 11136,
91
+ "input_tokens": 11357,
92
+ "output_tokens": 29,
93
+ "reasoning_output_tokens": 0
94
+ },
95
+ {
96
+ "cache_write_input_tokens": 0,
97
+ "cached_input_tokens": 11264,
98
+ "input_tokens": 11490,
99
+ "output_tokens": 30,
100
+ "reasoning_output_tokens": 0
101
+ },
102
+ {
103
+ "cache_write_input_tokens": 0,
104
+ "cached_input_tokens": 11392,
105
+ "input_tokens": 15932,
106
+ "output_tokens": 52,
107
+ "reasoning_output_tokens": 0
108
+ },
109
+ {
110
+ "cache_write_input_tokens": 0,
111
+ "cached_input_tokens": 15872,
112
+ "input_tokens": 16139,
113
+ "output_tokens": 252,
114
+ "reasoning_output_tokens": 0
115
+ },
116
+ {
117
+ "cache_write_input_tokens": 0,
118
+ "cached_input_tokens": 16256,
119
+ "input_tokens": 21248,
120
+ "output_tokens": 114,
121
+ "reasoning_output_tokens": 0
122
+ },
123
+ {
124
+ "cache_write_input_tokens": 0,
125
+ "cached_input_tokens": 21248,
126
+ "input_tokens": 23413,
127
+ "output_tokens": 107,
128
+ "reasoning_output_tokens": 0
129
+ },
130
+ {
131
+ "cache_write_input_tokens": 0,
132
+ "cached_input_tokens": 23424,
133
+ "input_tokens": 23737,
134
+ "output_tokens": 65,
135
+ "reasoning_output_tokens": 0
136
+ },
137
+ {
138
+ "cache_write_input_tokens": 0,
139
+ "cached_input_tokens": 23680,
140
+ "input_tokens": 24765,
141
+ "output_tokens": 63,
142
+ "reasoning_output_tokens": 21
143
+ },
144
+ {
145
+ "cache_write_input_tokens": 0,
146
+ "cached_input_tokens": 24704,
147
+ "input_tokens": 27118,
148
+ "output_tokens": 23,
149
+ "reasoning_output_tokens": 0
150
+ },
151
+ {
152
+ "cache_write_input_tokens": 0,
153
+ "cached_input_tokens": 27008,
154
+ "input_tokens": 27309,
155
+ "output_tokens": 412,
156
+ "reasoning_output_tokens": 0
157
+ },
158
+ {
159
+ "cache_write_input_tokens": 0,
160
+ "cached_input_tokens": 27520,
161
+ "input_tokens": 27743,
162
+ "output_tokens": 37,
163
+ "reasoning_output_tokens": 0
164
+ },
165
+ {
166
+ "cache_write_input_tokens": 0,
167
+ "cached_input_tokens": 27648,
168
+ "input_tokens": 28194,
169
+ "output_tokens": 37,
170
+ "reasoning_output_tokens": 0
171
+ },
172
+ {
173
+ "cache_write_input_tokens": 0,
174
+ "cached_input_tokens": 28032,
175
+ "input_tokens": 28640,
176
+ "output_tokens": 36,
177
+ "reasoning_output_tokens": 0
178
+ },
179
+ {
180
+ "cache_write_input_tokens": 0,
181
+ "cached_input_tokens": 28544,
182
+ "input_tokens": 29084,
183
+ "output_tokens": 54,
184
+ "reasoning_output_tokens": 16
185
+ },
186
+ {
187
+ "cache_write_input_tokens": 0,
188
+ "cached_input_tokens": 28928,
189
+ "input_tokens": 29546,
190
+ "output_tokens": 59,
191
+ "reasoning_output_tokens": 20
192
+ },
193
+ {
194
+ "cache_write_input_tokens": 0,
195
+ "cached_input_tokens": 29568,
196
+ "input_tokens": 29928,
197
+ "output_tokens": 39,
198
+ "reasoning_output_tokens": 0
199
+ },
200
+ {
201
+ "cache_write_input_tokens": 0,
202
+ "cached_input_tokens": 29824,
203
+ "input_tokens": 30376,
204
+ "output_tokens": 178,
205
+ "reasoning_output_tokens": 51
206
+ },
207
+ {
208
+ "cache_write_input_tokens": 0,
209
+ "cached_input_tokens": 30464,
210
+ "input_tokens": 31921,
211
+ "output_tokens": 171,
212
+ "reasoning_output_tokens": 70
213
+ },
214
+ {
215
+ "cache_write_input_tokens": 0,
216
+ "cached_input_tokens": 32000,
217
+ "input_tokens": 32167,
218
+ "output_tokens": 39,
219
+ "reasoning_output_tokens": 0
220
+ },
221
+ {
222
+ "cache_write_input_tokens": 0,
223
+ "cached_input_tokens": 32000,
224
+ "input_tokens": 32438,
225
+ "output_tokens": 342,
226
+ "reasoning_output_tokens": 0
227
+ },
228
+ {
229
+ "cache_write_input_tokens": 0,
230
+ "cached_input_tokens": 32640,
231
+ "input_tokens": 33195,
232
+ "output_tokens": 954,
233
+ "reasoning_output_tokens": 252
234
+ },
235
+ {
236
+ "cache_write_input_tokens": 0,
237
+ "cached_input_tokens": 34048,
238
+ "input_tokens": 34170,
239
+ "output_tokens": 74,
240
+ "reasoning_output_tokens": 0
241
+ },
242
+ {
243
+ "cache_write_input_tokens": 0,
244
+ "cached_input_tokens": 34048,
245
+ "input_tokens": 34460,
246
+ "output_tokens": 75,
247
+ "reasoning_output_tokens": 0
248
+ },
249
+ {
250
+ "cache_write_input_tokens": 0,
251
+ "cached_input_tokens": 34432,
252
+ "input_tokens": 34892,
253
+ "output_tokens": 113,
254
+ "reasoning_output_tokens": 28
255
+ },
256
+ {
257
+ "cache_write_input_tokens": 0,
258
+ "cached_input_tokens": 34816,
259
+ "input_tokens": 35366,
260
+ "output_tokens": 52,
261
+ "reasoning_output_tokens": 0
262
+ },
263
+ {
264
+ "cache_write_input_tokens": 0,
265
+ "cached_input_tokens": 35200,
266
+ "input_tokens": 35776,
267
+ "output_tokens": 115,
268
+ "reasoning_output_tokens": 27
269
+ },
270
+ {
271
+ "cache_write_input_tokens": 0,
272
+ "cached_input_tokens": 35712,
273
+ "input_tokens": 36251,
274
+ "output_tokens": 228,
275
+ "reasoning_output_tokens": 143
276
+ },
277
+ {
278
+ "cache_write_input_tokens": 0,
279
+ "cached_input_tokens": 36352,
280
+ "input_tokens": 36839,
281
+ "output_tokens": 296,
282
+ "reasoning_output_tokens": 212
283
+ },
284
+ {
285
+ "cache_write_input_tokens": 0,
286
+ "cached_input_tokens": 36992,
287
+ "input_tokens": 37492,
288
+ "output_tokens": 51,
289
+ "reasoning_output_tokens": 12
290
+ },
291
+ {
292
+ "cache_write_input_tokens": 0,
293
+ "cached_input_tokens": 37376,
294
+ "input_tokens": 37866,
295
+ "output_tokens": 428,
296
+ "reasoning_output_tokens": 25
297
+ },
298
+ {
299
+ "cache_write_input_tokens": 0,
300
+ "cached_input_tokens": 38144,
301
+ "input_tokens": 38314,
302
+ "output_tokens": 31,
303
+ "reasoning_output_tokens": 0
304
+ },
305
+ {
306
+ "cache_write_input_tokens": 0,
307
+ "cached_input_tokens": 38144,
308
+ "input_tokens": 38699,
309
+ "output_tokens": 162,
310
+ "reasoning_output_tokens": 79
311
+ },
312
+ {
313
+ "cache_write_input_tokens": 0,
314
+ "cached_input_tokens": 38656,
315
+ "input_tokens": 39217,
316
+ "output_tokens": 159,
317
+ "reasoning_output_tokens": 77
318
+ },
319
+ {
320
+ "cache_write_input_tokens": 0,
321
+ "cached_input_tokens": 39168,
322
+ "input_tokens": 39737,
323
+ "output_tokens": 77,
324
+ "reasoning_output_tokens": 16
325
+ },
326
+ {
327
+ "cache_write_input_tokens": 0,
328
+ "cached_input_tokens": 39680,
329
+ "input_tokens": 40178,
330
+ "output_tokens": 87,
331
+ "reasoning_output_tokens": 0
332
+ },
333
+ {
334
+ "cache_write_input_tokens": 0,
335
+ "cached_input_tokens": 40064,
336
+ "input_tokens": 40625,
337
+ "output_tokens": 213,
338
+ "reasoning_output_tokens": 130
339
+ },
340
+ {
341
+ "cache_write_input_tokens": 0,
342
+ "cached_input_tokens": 40704,
343
+ "input_tokens": 41196,
344
+ "output_tokens": 30,
345
+ "reasoning_output_tokens": 0
346
+ },
347
+ {
348
+ "cache_write_input_tokens": 0,
349
+ "cached_input_tokens": 41088,
350
+ "input_tokens": 41576,
351
+ "output_tokens": 339,
352
+ "reasoning_output_tokens": 256
353
+ },
354
+ {
355
+ "cache_write_input_tokens": 0,
356
+ "cached_input_tokens": 41856,
357
+ "input_tokens": 42274,
358
+ "output_tokens": 83,
359
+ "reasoning_output_tokens": 43
360
+ },
361
+ {
362
+ "cache_write_input_tokens": 0,
363
+ "cached_input_tokens": 42240,
364
+ "input_tokens": 42605,
365
+ "output_tokens": 159,
366
+ "reasoning_output_tokens": 77
367
+ },
368
+ {
369
+ "cache_write_input_tokens": 0,
370
+ "cached_input_tokens": 42624,
371
+ "input_tokens": 43122,
372
+ "output_tokens": 73,
373
+ "reasoning_output_tokens": 10
374
+ },
375
+ {
376
+ "cache_write_input_tokens": 0,
377
+ "cached_input_tokens": 43008,
378
+ "input_tokens": 43328,
379
+ "output_tokens": 30,
380
+ "reasoning_output_tokens": 0
381
+ },
382
+ {
383
+ "cache_write_input_tokens": 0,
384
+ "cached_input_tokens": 43264,
385
+ "input_tokens": 43420,
386
+ "output_tokens": 51,
387
+ "reasoning_output_tokens": 0
388
+ },
389
+ {
390
+ "cache_write_input_tokens": 0,
391
+ "cached_input_tokens": 43392,
392
+ "input_tokens": 43756,
393
+ "output_tokens": 127,
394
+ "reasoning_output_tokens": 95
395
+ },
396
+ {
397
+ "cache_write_input_tokens": 0,
398
+ "cached_input_tokens": 43776,
399
+ "input_tokens": 43901,
400
+ "output_tokens": 229,
401
+ "reasoning_output_tokens": 50
402
+ },
403
+ {
404
+ "cache_write_input_tokens": 0,
405
+ "cached_input_tokens": 44032,
406
+ "input_tokens": 44156,
407
+ "output_tokens": 567,
408
+ "reasoning_output_tokens": 0
409
+ },
410
+ {
411
+ "cache_write_input_tokens": 0,
412
+ "cached_input_tokens": 44544,
413
+ "input_tokens": 44746,
414
+ "output_tokens": 37,
415
+ "reasoning_output_tokens": 0
416
+ }
417
+ ],
418
+ "usage_source": "provider-raw"
419
+ },
420
+ "call_activity": {
421
+ "schema": "rlebench/call-activity/1",
422
+ "source": "Kinex native trace",
423
+ "model_tool_calls": 58,
424
+ "model_tool_calls_by_name": {
425
+ "ls": 1,
426
+ "read": 17,
427
+ "find": 1,
428
+ "bash": 1,
429
+ "devices": 4,
430
+ "exec": 26,
431
+ "device": 2,
432
+ "write": 4,
433
+ "device_stop": 1,
434
+ "edit": 1
435
+ },
436
+ "python_instrumented_model_tool_calls": null,
437
+ "python_tool_invocations": null,
438
+ "nested_python_tool_invocations": null,
439
+ "python_tool_invocations_by_origin": null,
440
+ "python_device_rpc_attempts": null,
441
+ "python_device_rpc_attempts_by_action": null,
442
+ "python_device_rpc_errors": null,
443
+ "python_scope": "Recorded instrumented Python invocations only; other paths and automatic fatal stops are not counted."
444
+ },
445
+ "media": {
446
+ "schema": "rlebench/native-camera-media/3",
447
+ "passed": true,
448
+ "speed": 4,
449
+ "source_fps": 10,
450
+ "output_fps": 40,
451
+ "width": 2880,
452
+ "height": 720,
453
+ "native_frames": 210,
454
+ "published_frames": 210,
455
+ "source_sample_coverage": 1.0,
456
+ "source_samples_complete": true,
457
+ "source_dropped_samples": 0,
458
+ "terminal_frame_retained": true,
459
+ "terminal_source_index": 209,
460
+ "terminal_hold_s": 0,
461
+ "simulation_time_s": 20.92,
462
+ "duration_s": 5.25,
463
+ "sha256": "ac15e983e0afbe68b6c357a9563df0beae4c8eef2128af0ed33e76bd302a474d",
464
+ "source_sha256": {
465
+ "third_person.mp4": "363adc62fca2294fe530d8ca0684b0aa98ddaff80ba91aae290a9c825721f05c",
466
+ "left_wrist.mp4": "03b8216df236ca19b6cc9fd4b050fa6d465d536befc8bbcecb41bf566e84695c",
467
+ "right_wrist.mp4": "9eab1d7fcf5bdc04e0b6a4d6377427b2059a97fdcfacd6f85bddffcfd3238eb0"
468
+ },
469
+ "all_frames_compared": 210,
470
+ "minimum_frame_psnr_db": 43.772015440757116,
471
+ "maximum_frame_rgb_mae": 1.2994675636291504,
472
+ "decode_ok": true,
473
+ "pts_verified": true,
474
+ "playback_kind": "simulation_time",
475
+ "resolution_kind": "independent_720_high",
476
+ "agent_observation_video": false,
477
+ "camera_views": [
478
+ {
479
+ "name": "third_person",
480
+ "width": 960,
481
+ "height": 720,
482
+ "channel": "spectator"
483
+ },
484
+ {
485
+ "name": "left_wrist",
486
+ "width": 960,
487
+ "height": 720,
488
+ "channel": "spectator"
489
+ },
490
+ {
491
+ "name": "right_wrist",
492
+ "width": 960,
493
+ "height": 720,
494
+ "channel": "spectator"
495
+ }
496
+ ],
497
+ "coverage_basis": "Every scheduled 10 Hz control-boundary sample, plus the terminal state.",
498
+ "frame_mapping": "Each captured sample exactly once; no appended or held frames.",
499
+ "sampling_note": "Samples follow the native simulation clock; model thinking adds no footage.",
500
+ "scope": "Live owner-only spectator views; no replay, rerender, upscaling or invented frames.",
501
+ "recording": {
502
+ "schema": "rosbridge/recording/1",
503
+ "run_id": "83632f6147cc",
504
+ "task": "robotwin/handover-block-aloha-agilex",
505
+ "state": "complete",
506
+ "views": {
507
+ "third_person": {
508
+ "received": 211,
509
+ "dropped": 0,
510
+ "segments": [
511
+ {
512
+ "file": "third_person.mp4",
513
+ "frames": 210,
514
+ "width": 960,
515
+ "height": 720,
516
+ "fps": 10.0,
517
+ "first_stamp_ns": 0,
518
+ "last_stamp_ns": 20920000000
519
+ }
520
+ ],
521
+ "error": null
522
+ },
523
+ "left_wrist": {
524
+ "received": 211,
525
+ "dropped": 0,
526
+ "segments": [
527
+ {
528
+ "file": "left_wrist.mp4",
529
+ "frames": 210,
530
+ "width": 960,
531
+ "height": 720,
532
+ "fps": 10.0,
533
+ "first_stamp_ns": 0,
534
+ "last_stamp_ns": 20920000000
535
+ }
536
+ ],
537
+ "error": null
538
+ },
539
+ "right_wrist": {
540
+ "received": 211,
541
+ "dropped": 0,
542
+ "segments": [
543
+ {
544
+ "file": "right_wrist.mp4",
545
+ "frames": 210,
546
+ "width": 960,
547
+ "height": 720,
548
+ "fps": 10.0,
549
+ "first_stamp_ns": 0,
550
+ "last_stamp_ns": 20920000000
551
+ }
552
+ ],
553
+ "error": null
554
+ }
555
+ },
556
+ "type": "mp4",
557
+ "spectator_views": [
558
+ "third_person",
559
+ "left_wrist",
560
+ "right_wrist"
561
+ ],
562
+ "fps": 10.0,
563
+ "topics": [],
564
+ "error": null,
565
+ "bag_messages": 0
566
+ }
567
+ },
568
+ "analysis": {
569
+ "observed": "Kinex wrote its own one-chunk policy tool with fresh feedback. Five 16-action inferences used the complete original instruction, without subtask rewriting. It then switched to agent-written geometric control and succeeded at native frame 523. This episode shows autonomous switching; it does not establish a planning benefit from VLA.",
570
+ "execution": "Previous valid episode; superseded by the authorized 2026-10-29 rerun, not removed or retried to select a success."
571
+ },
572
+ "links": {
573
+ "video": "experiments/robotwin-native/kinex_policy-1/presentation/video.mp4",
574
+ "poster": "experiments/robotwin-native/kinex_policy-1/presentation/poster.jpg",
575
+ "terminal_frame": "experiments/robotwin-native/kinex_policy-1/presentation/terminal.jpg",
576
+ "transcript": "experiments/robotwin-native/kinex_policy-1/transcript.json",
577
+ "usage": "experiments/robotwin-native/kinex_policy-1/usage.json",
578
+ "analysis": "experiments/robotwin-native/kinex_policy-1/analysis.json",
579
+ "protocol": "experiments/robotwin-native/kinex_policy-1/protocol.json",
580
+ "native_goal": "experiments/robotwin-native/kinex_policy-1/instruction.json",
581
+ "verdict": "experiments/robotwin-native/kinex_policy-1/episode.json",
582
+ "provenance": "sources.json",
583
+ "media_check": "experiments/robotwin-native/kinex_policy-1/presentation/media-validation.json",
584
+ "recording_manifest": "experiments/robotwin-native/kinex_policy-1/presentation/recording.json",
585
+ "workspace": "experiments/robotwin-native/kinex_policy-1/workspace.tar.gz",
586
+ "workspace_changes": "experiments/robotwin-native/kinex_policy-1/workspace.json",
587
+ "evidence_archive": "experiments/robotwin-native/kinex_policy-1/evidence.tar.gz",
588
+ "tool_calls": "experiments/robotwin-native/kinex_policy-1/tool-calls.json",
589
+ "artifact_index": "experiments/robotwin-native/kinex_policy-1/artifact-index.json",
590
+ "native_session": "experiments/robotwin-native/kinex_policy-1/session.jsonl",
591
+ "trace": "experiments/robotwin-native/kinex_policy-1/trace.json",
592
+ "provider_usage": "experiments/robotwin-native/kinex_policy-1/provider-requests.jsonl",
593
+ "vla_io": "experiments/robotwin-native/kinex_policy-1/vla-io.json",
594
+ "policy_trace": "experiments/robotwin-native/kinex_policy-1/policy/calls.jsonl",
595
+ "result_page": "https://huggingface.co/spaces/RLE-Bench/programmable-vla/blob/91f9b775b2f5a5b8355aa11989ac24a5c7f72214/data.json"
596
+ },
597
+ "resources": [
598
+ {
599
+ "name": ".kinex/tools/cartesian_move.py",
600
+ "file": "experiments/robotwin-native/kinex_policy-1/workspace/.kinex/tools/cartesian_move.py",
601
+ "kind": "Created during this episode; no imported earlier task resources."
602
+ },
603
+ {
604
+ "name": ".kinex/tools/color_geometry.py",
605
+ "file": "experiments/robotwin-native/kinex_policy-1/workspace/.kinex/tools/color_geometry.py",
606
+ "kind": "Created during this episode; no imported earlier task resources."
607
+ },
608
+ {
609
+ "name": ".kinex/tools/handover_policy.py",
610
+ "file": "experiments/robotwin-native/kinex_policy-1/workspace/.kinex/tools/handover_policy.py",
611
+ "kind": "Created during this episode; no imported earlier task resources."
612
+ },
613
+ {
614
+ "name": ".kinex/memos/handover-block.md",
615
+ "file": "experiments/robotwin-native/kinex_policy-1/workspace/.kinex/memos/handover-block.md",
616
+ "kind": "Created during this episode; no imported earlier task resources."
617
+ }
618
+ ],
619
+ "session_counts": {
620
+ "visible_events": 108,
621
+ "observed_images": 24
622
+ },
623
+ "measured_protocol": {
624
+ "max_steps": 800,
625
+ "seed": 0,
626
+ "episodes": 1,
627
+ "control_frequency_hz": 25,
628
+ "provider": "rosbridge",
629
+ "realtime": false,
630
+ "stop_on_success": true
631
+ },
632
+ "vla_calls": 5,
633
+ "kinex_revision": "f4ab59e9e93aa2f569d88386799a2bed0670d59c",
634
+ "kinex_version": "1.1.0",
635
+ "dataset_revision": "91f9b775b2f5a5b8355aa11989ac24a5c7f72214"
636
+ },
637
+ {
638
+ "id": "kinex_policy-2",
639
+ "variant": "kinex_policy",
640
+ "task_key": "task03/02",
641
+ "family": "task03",
642
+ "source_family": "task03",
643
+ "provider": "rosbridge",
644
+ "slot": "02",
645
+ "seed": 0,
646
+ "source_episode_id": "kinex_policy-2",
647
+ "status": "archived_completed",
648
+ "success": false,
649
+ "valid": true,
650
+ "execution": {
651
+ "status": "finished",
652
+ "reason": null
653
+ },
654
+ "verdict": {
655
+ "evidence_valid": true,
656
+ "success": false,
657
+ "termination": "stopped",
658
+ "steps": 498
659
+ },
660
+ "steps": 498,
661
+ "simulation_time_s": 19.92,
662
+ "wall_time_s": 817.625303,
663
+ "model": "GPT-6 Astra",
664
+ "effort": "high",
665
+ "harness": "kinex",
666
+ "instruction": "use both arms to pick up the two shoes on the table and put them flat in the shoebox, with the shoe tips pointing to the left and the shoe initially on the left nearer the robot. Leave both grippers open",
667
+ "native_instruction": "use both arms to pick up the two shoes on the table and put them in the shoebox, with the shoe tip pointing to the left",
668
+ "instruction_policy": "modified",
669
+ "usage": {
670
+ "accounting": "reported-responses",
671
+ "audit_complete": true,
672
+ "audit_source": "Provider terminal response usage",
673
+ "cache_hit_rate": 0.9554559175384522,
674
+ "cache_read_input_tokens": 2249856,
675
+ "cache_reported_input_tokens": 2354746,
676
+ "cache_write_basis": "provider-reported",
677
+ "cache_write_input_tokens": 0,
678
+ "cache_write_reported_input_tokens": 2354746,
679
+ "cached_input_tokens": 2249856,
680
+ "cost_usd": null,
681
+ "fresh_input_tokens": 104890,
682
+ "harbor_mismatches": {},
683
+ "input_breakdown_complete": true,
684
+ "input_tokens": 2354746,
685
+ "known_cache_write_input_tokens": 0,
686
+ "known_cached_input_tokens": 2249856,
687
+ "known_fresh_input_tokens": 104890,
688
+ "known_input_tokens": 2354746,
689
+ "known_output_tokens": 9984,
690
+ "known_reasoning_output_tokens": 3108,
691
+ "output_tokens": 9984,
692
+ "parser": "harbor/0.22.0",
693
+ "reasoning_output_tokens": 3108,
694
+ "reasoning_reported_output_tokens": 9984,
695
+ "rejected_attempts": 0,
696
+ "reported_responses": {
697
+ "cache_reported_input_tokens": 63,
698
+ "cache_write_input_tokens": 63,
699
+ "cache_write_reported_input_tokens": 63,
700
+ "cached_input_tokens": 63,
701
+ "fresh_input_tokens": 63,
702
+ "input_tokens": 63,
703
+ "output_tokens": 63,
704
+ "reasoning_output_tokens": 63,
705
+ "reasoning_reported_output_tokens": 63
706
+ },
707
+ "request_attempts": 63,
708
+ "response_count": 63,
709
+ "response_ids_complete": true,
710
+ "schema": "rlebench/token-usage/1",
711
+ "source": "Harbor session metrics",
712
+ "total_tokens": 2364730,
713
+ "uncached_input_tokens": 104890,
714
+ "unresolved_attempts": [],
715
+ "usage_records": [
716
+ {
717
+ "cache_write_input_tokens": 0,
718
+ "cached_input_tokens": 0,
719
+ "input_tokens": 11328,
720
+ "output_tokens": 19,
721
+ "reasoning_output_tokens": 0
722
+ },
723
+ {
724
+ "cache_write_input_tokens": 0,
725
+ "cached_input_tokens": 11264,
726
+ "input_tokens": 11368,
727
+ "output_tokens": 29,
728
+ "reasoning_output_tokens": 0
729
+ },
730
+ {
731
+ "cache_write_input_tokens": 0,
732
+ "cached_input_tokens": 11264,
733
+ "input_tokens": 11501,
734
+ "output_tokens": 30,
735
+ "reasoning_output_tokens": 0
736
+ },
737
+ {
738
+ "cache_write_input_tokens": 0,
739
+ "cached_input_tokens": 11392,
740
+ "input_tokens": 14555,
741
+ "output_tokens": 47,
742
+ "reasoning_output_tokens": 0
743
+ },
744
+ {
745
+ "cache_write_input_tokens": 0,
746
+ "cached_input_tokens": 14464,
747
+ "input_tokens": 15031,
748
+ "output_tokens": 252,
749
+ "reasoning_output_tokens": 0
750
+ },
751
+ {
752
+ "cache_write_input_tokens": 0,
753
+ "cached_input_tokens": 15104,
754
+ "input_tokens": 20140,
755
+ "output_tokens": 167,
756
+ "reasoning_output_tokens": 0
757
+ },
758
+ {
759
+ "cache_write_input_tokens": 0,
760
+ "cached_input_tokens": 20224,
761
+ "input_tokens": 23684,
762
+ "output_tokens": 107,
763
+ "reasoning_output_tokens": 0
764
+ },
765
+ {
766
+ "cache_write_input_tokens": 0,
767
+ "cached_input_tokens": 23680,
768
+ "input_tokens": 24008,
769
+ "output_tokens": 40,
770
+ "reasoning_output_tokens": 0
771
+ },
772
+ {
773
+ "cache_write_input_tokens": 0,
774
+ "cached_input_tokens": 23936,
775
+ "input_tokens": 26338,
776
+ "output_tokens": 75,
777
+ "reasoning_output_tokens": 19
778
+ },
779
+ {
780
+ "cache_write_input_tokens": 0,
781
+ "cached_input_tokens": 26240,
782
+ "input_tokens": 26580,
783
+ "output_tokens": 465,
784
+ "reasoning_output_tokens": 0
785
+ },
786
+ {
787
+ "cache_write_input_tokens": 0,
788
+ "cached_input_tokens": 26880,
789
+ "input_tokens": 27068,
790
+ "output_tokens": 40,
791
+ "reasoning_output_tokens": 0
792
+ },
793
+ {
794
+ "cache_write_input_tokens": 0,
795
+ "cached_input_tokens": 27008,
796
+ "input_tokens": 27615,
797
+ "output_tokens": 37,
798
+ "reasoning_output_tokens": 0
799
+ },
800
+ {
801
+ "cache_write_input_tokens": 0,
802
+ "cached_input_tokens": 27520,
803
+ "input_tokens": 28159,
804
+ "output_tokens": 39,
805
+ "reasoning_output_tokens": 0
806
+ },
807
+ {
808
+ "cache_write_input_tokens": 0,
809
+ "cached_input_tokens": 28032,
810
+ "input_tokens": 28708,
811
+ "output_tokens": 39,
812
+ "reasoning_output_tokens": 0
813
+ },
814
+ {
815
+ "cache_write_input_tokens": 0,
816
+ "cached_input_tokens": 0,
817
+ "input_tokens": 29258,
818
+ "output_tokens": 39,
819
+ "reasoning_output_tokens": 0
820
+ },
821
+ {
822
+ "cache_write_input_tokens": 0,
823
+ "cached_input_tokens": 29184,
824
+ "input_tokens": 29804,
825
+ "output_tokens": 39,
826
+ "reasoning_output_tokens": 0
827
+ },
828
+ {
829
+ "cache_write_input_tokens": 0,
830
+ "cached_input_tokens": 29696,
831
+ "input_tokens": 30352,
832
+ "output_tokens": 39,
833
+ "reasoning_output_tokens": 0
834
+ },
835
+ {
836
+ "cache_write_input_tokens": 0,
837
+ "cached_input_tokens": 30208,
838
+ "input_tokens": 30899,
839
+ "output_tokens": 39,
840
+ "reasoning_output_tokens": 0
841
+ },
842
+ {
843
+ "cache_write_input_tokens": 0,
844
+ "cached_input_tokens": 30848,
845
+ "input_tokens": 31448,
846
+ "output_tokens": 42,
847
+ "reasoning_output_tokens": 0
848
+ },
849
+ {
850
+ "cache_write_input_tokens": 0,
851
+ "cached_input_tokens": 31360,
852
+ "input_tokens": 32035,
853
+ "output_tokens": 40,
854
+ "reasoning_output_tokens": 0
855
+ },
856
+ {
857
+ "cache_write_input_tokens": 0,
858
+ "cached_input_tokens": 31872,
859
+ "input_tokens": 32577,
860
+ "output_tokens": 58,
861
+ "reasoning_output_tokens": 14
862
+ },
863
+ {
864
+ "cache_write_input_tokens": 0,
865
+ "cached_input_tokens": 32384,
866
+ "input_tokens": 33144,
867
+ "output_tokens": 61,
868
+ "reasoning_output_tokens": 20
869
+ },
870
+ {
871
+ "cache_write_input_tokens": 0,
872
+ "cached_input_tokens": 33152,
873
+ "input_tokens": 33713,
874
+ "output_tokens": 91,
875
+ "reasoning_output_tokens": 49
876
+ },
877
+ {
878
+ "cache_write_input_tokens": 0,
879
+ "cached_input_tokens": 33664,
880
+ "input_tokens": 34314,
881
+ "output_tokens": 68,
882
+ "reasoning_output_tokens": 29
883
+ },
884
+ {
885
+ "cache_write_input_tokens": 0,
886
+ "cached_input_tokens": 34304,
887
+ "input_tokens": 34705,
888
+ "output_tokens": 102,
889
+ "reasoning_output_tokens": 68
890
+ },
891
+ {
892
+ "cache_write_input_tokens": 0,
893
+ "cached_input_tokens": 34688,
894
+ "input_tokens": 34869,
895
+ "output_tokens": 128,
896
+ "reasoning_output_tokens": 84
897
+ },
898
+ {
899
+ "cache_write_input_tokens": 0,
900
+ "cached_input_tokens": 34816,
901
+ "input_tokens": 35207,
902
+ "output_tokens": 353,
903
+ "reasoning_output_tokens": 21
904
+ },
905
+ {
906
+ "cache_write_input_tokens": 0,
907
+ "cached_input_tokens": 35456,
908
+ "input_tokens": 35789,
909
+ "output_tokens": 120,
910
+ "reasoning_output_tokens": 76
911
+ },
912
+ {
913
+ "cache_write_input_tokens": 0,
914
+ "cached_input_tokens": 35712,
915
+ "input_tokens": 36413,
916
+ "output_tokens": 159,
917
+ "reasoning_output_tokens": 119
918
+ },
919
+ {
920
+ "cache_write_input_tokens": 0,
921
+ "cached_input_tokens": 36352,
922
+ "input_tokens": 37077,
923
+ "output_tokens": 106,
924
+ "reasoning_output_tokens": 72
925
+ },
926
+ {
927
+ "cache_write_input_tokens": 0,
928
+ "cached_input_tokens": 36992,
929
+ "input_tokens": 37245,
930
+ "output_tokens": 37,
931
+ "reasoning_output_tokens": 0
932
+ },
933
+ {
934
+ "cache_write_input_tokens": 0,
935
+ "cached_input_tokens": 37120,
936
+ "input_tokens": 37606,
937
+ "output_tokens": 152,
938
+ "reasoning_output_tokens": 22
939
+ },
940
+ {
941
+ "cache_write_input_tokens": 0,
942
+ "cached_input_tokens": 37632,
943
+ "input_tokens": 38344,
944
+ "output_tokens": 39,
945
+ "reasoning_output_tokens": 0
946
+ },
947
+ {
948
+ "cache_write_input_tokens": 0,
949
+ "cached_input_tokens": 38272,
950
+ "input_tokens": 38615,
951
+ "output_tokens": 269,
952
+ "reasoning_output_tokens": 0
953
+ },
954
+ {
955
+ "cache_write_input_tokens": 0,
956
+ "cached_input_tokens": 38784,
957
+ "input_tokens": 39478,
958
+ "output_tokens": 238,
959
+ "reasoning_output_tokens": 199
960
+ },
961
+ {
962
+ "cache_write_input_tokens": 0,
963
+ "cached_input_tokens": 39680,
964
+ "input_tokens": 40182,
965
+ "output_tokens": 672,
966
+ "reasoning_output_tokens": 53
967
+ },
968
+ {
969
+ "cache_write_input_tokens": 0,
970
+ "cached_input_tokens": 40704,
971
+ "input_tokens": 40875,
972
+ "output_tokens": 117,
973
+ "reasoning_output_tokens": 0
974
+ },
975
+ {
976
+ "cache_write_input_tokens": 0,
977
+ "cached_input_tokens": 40832,
978
+ "input_tokens": 41422,
979
+ "output_tokens": 151,
980
+ "reasoning_output_tokens": 126
981
+ },
982
+ {
983
+ "cache_write_input_tokens": 0,
984
+ "cached_input_tokens": 41472,
985
+ "input_tokens": 41743,
986
+ "output_tokens": 314,
987
+ "reasoning_output_tokens": 81
988
+ },
989
+ {
990
+ "cache_write_input_tokens": 0,
991
+ "cached_input_tokens": 41856,
992
+ "input_tokens": 42462,
993
+ "output_tokens": 248,
994
+ "reasoning_output_tokens": 79
995
+ },
996
+ {
997
+ "cache_write_input_tokens": 0,
998
+ "cached_input_tokens": 42624,
999
+ "input_tokens": 42845,
1000
+ "output_tokens": 316,
1001
+ "reasoning_output_tokens": 177
1002
+ },
1003
+ {
1004
+ "cache_write_input_tokens": 0,
1005
+ "cached_input_tokens": 43008,
1006
+ "input_tokens": 43844,
1007
+ "output_tokens": 71,
1008
+ "reasoning_output_tokens": 20
1009
+ },
1010
+ {
1011
+ "cache_write_input_tokens": 0,
1012
+ "cached_input_tokens": 43776,
1013
+ "input_tokens": 44351,
1014
+ "output_tokens": 75,
1015
+ "reasoning_output_tokens": 0
1016
+ },
1017
+ {
1018
+ "cache_write_input_tokens": 0,
1019
+ "cached_input_tokens": 44288,
1020
+ "input_tokens": 44453,
1021
+ "output_tokens": 137,
1022
+ "reasoning_output_tokens": 0
1023
+ },
1024
+ {
1025
+ "cache_write_input_tokens": 0,
1026
+ "cached_input_tokens": 44416,
1027
+ "input_tokens": 45140,
1028
+ "output_tokens": 271,
1029
+ "reasoning_output_tokens": 144
1030
+ },
1031
+ {
1032
+ "cache_write_input_tokens": 0,
1033
+ "cached_input_tokens": 45184,
1034
+ "input_tokens": 45842,
1035
+ "output_tokens": 107,
1036
+ "reasoning_output_tokens": 49
1037
+ },
1038
+ {
1039
+ "cache_write_input_tokens": 0,
1040
+ "cached_input_tokens": 45824,
1041
+ "input_tokens": 46269,
1042
+ "output_tokens": 119,
1043
+ "reasoning_output_tokens": 43
1044
+ },
1045
+ {
1046
+ "cache_write_input_tokens": 0,
1047
+ "cached_input_tokens": 46336,
1048
+ "input_tokens": 46820,
1049
+ "output_tokens": 363,
1050
+ "reasoning_output_tokens": 267
1051
+ },
1052
+ {
1053
+ "cache_write_input_tokens": 0,
1054
+ "cached_input_tokens": 47104,
1055
+ "input_tokens": 47506,
1056
+ "output_tokens": 77,
1057
+ "reasoning_output_tokens": 27
1058
+ },
1059
+ {
1060
+ "cache_write_input_tokens": 0,
1061
+ "cached_input_tokens": 47488,
1062
+ "input_tokens": 47904,
1063
+ "output_tokens": 327,
1064
+ "reasoning_output_tokens": 81
1065
+ },
1066
+ {
1067
+ "cache_write_input_tokens": 0,
1068
+ "cached_input_tokens": 48000,
1069
+ "input_tokens": 48655,
1070
+ "output_tokens": 444,
1071
+ "reasoning_output_tokens": 158
1072
+ },
1073
+ {
1074
+ "cache_write_input_tokens": 0,
1075
+ "cached_input_tokens": 48896,
1076
+ "input_tokens": 49238,
1077
+ "output_tokens": 231,
1078
+ "reasoning_output_tokens": 115
1079
+ },
1080
+ {
1081
+ "cache_write_input_tokens": 0,
1082
+ "cached_input_tokens": 49280,
1083
+ "input_tokens": 49903,
1084
+ "output_tokens": 203,
1085
+ "reasoning_output_tokens": 19
1086
+ },
1087
+ {
1088
+ "cache_write_input_tokens": 0,
1089
+ "cached_input_tokens": 50048,
1090
+ "input_tokens": 50456,
1091
+ "output_tokens": 220,
1092
+ "reasoning_output_tokens": 126
1093
+ },
1094
+ {
1095
+ "cache_write_input_tokens": 0,
1096
+ "cached_input_tokens": 28544,
1097
+ "input_tokens": 50996,
1098
+ "output_tokens": 54,
1099
+ "reasoning_output_tokens": 0
1100
+ },
1101
+ {
1102
+ "cache_write_input_tokens": 0,
1103
+ "cached_input_tokens": 50944,
1104
+ "input_tokens": 51369,
1105
+ "output_tokens": 508,
1106
+ "reasoning_output_tokens": 431
1107
+ },
1108
+ {
1109
+ "cache_write_input_tokens": 0,
1110
+ "cached_input_tokens": 50560,
1111
+ "input_tokens": 52308,
1112
+ "output_tokens": 48,
1113
+ "reasoning_output_tokens": 15
1114
+ },
1115
+ {
1116
+ "cache_write_input_tokens": 0,
1117
+ "cached_input_tokens": 52224,
1118
+ "input_tokens": 52418,
1119
+ "output_tokens": 297,
1120
+ "reasoning_output_tokens": 200
1121
+ },
1122
+ {
1123
+ "cache_write_input_tokens": 0,
1124
+ "cached_input_tokens": 52608,
1125
+ "input_tokens": 53035,
1126
+ "output_tokens": 76,
1127
+ "reasoning_output_tokens": 14
1128
+ },
1129
+ {
1130
+ "cache_write_input_tokens": 0,
1131
+ "cached_input_tokens": 52992,
1132
+ "input_tokens": 53429,
1133
+ "output_tokens": 209,
1134
+ "reasoning_output_tokens": 56
1135
+ },
1136
+ {
1137
+ "cache_write_input_tokens": 0,
1138
+ "cached_input_tokens": 53504,
1139
+ "input_tokens": 53665,
1140
+ "output_tokens": 544,
1141
+ "reasoning_output_tokens": 0
1142
+ },
1143
+ {
1144
+ "cache_write_input_tokens": 0,
1145
+ "cached_input_tokens": 54016,
1146
+ "input_tokens": 54232,
1147
+ "output_tokens": 148,
1148
+ "reasoning_output_tokens": 35
1149
+ },
1150
+ {
1151
+ "cache_write_input_tokens": 0,
1152
+ "cached_input_tokens": 54144,
1153
+ "input_tokens": 54409,
1154
+ "output_tokens": 32,
1155
+ "reasoning_output_tokens": 0
1156
+ }
1157
+ ],
1158
+ "usage_source": "provider-raw"
1159
+ },
1160
+ "call_activity": {
1161
+ "schema": "rlebench/call-activity/1",
1162
+ "source": "Kinex native trace",
1163
+ "model_tool_calls": 72,
1164
+ "model_tool_calls_by_name": {
1165
+ "ls": 1,
1166
+ "read": 16,
1167
+ "find": 1,
1168
+ "bash": 1,
1169
+ "exec": 38,
1170
+ "devices": 3,
1171
+ "device": 2,
1172
+ "write": 3,
1173
+ "device_stop": 3,
1174
+ "grep": 1,
1175
+ "edit": 3
1176
+ },
1177
+ "python_instrumented_model_tool_calls": null,
1178
+ "python_tool_invocations": null,
1179
+ "nested_python_tool_invocations": null,
1180
+ "python_tool_invocations_by_origin": null,
1181
+ "python_device_rpc_attempts": null,
1182
+ "python_device_rpc_attempts_by_action": null,
1183
+ "python_device_rpc_errors": null,
1184
+ "python_scope": "Recorded instrumented Python invocations only; other paths and automatic fatal stops are not counted."
1185
+ },
1186
+ "media": {
1187
+ "schema": "rlebench/native-camera-media/3",
1188
+ "passed": true,
1189
+ "speed": 4,
1190
+ "source_fps": 10,
1191
+ "output_fps": 40,
1192
+ "width": 2880,
1193
+ "height": 720,
1194
+ "native_frames": 200,
1195
+ "published_frames": 200,
1196
+ "source_sample_coverage": 1.0,
1197
+ "source_samples_complete": true,
1198
+ "source_dropped_samples": 0,
1199
+ "terminal_frame_retained": true,
1200
+ "terminal_source_index": 199,
1201
+ "terminal_hold_s": 0,
1202
+ "simulation_time_s": 19.92,
1203
+ "duration_s": 5.0,
1204
+ "sha256": "da23802e54cf63701938fc000dd89ce5802fb5cd8cd7a221c288c235ce3f8ba5",
1205
+ "source_sha256": {
1206
+ "third_person.mp4": "b1fbd04d7a0097d489360513098d6d5081b2cca3c556c50147ab7dd4e4444574",
1207
+ "left_wrist.mp4": "8c6aef985461d23c5e36fb3a91d33894ceb34f6871d641459a969c8a601264ff",
1208
+ "right_wrist.mp4": "611a79fab90c57f18084f6b7b4616283cd3525af4024e1fffa06d80efc285318"
1209
+ },
1210
+ "all_frames_compared": 200,
1211
+ "minimum_frame_psnr_db": 41.467497047947965,
1212
+ "maximum_frame_rgb_mae": 1.5497524738311768,
1213
+ "decode_ok": true,
1214
+ "pts_verified": true,
1215
+ "playback_kind": "simulation_time",
1216
+ "resolution_kind": "independent_720_high",
1217
+ "agent_observation_video": false,
1218
+ "camera_views": [
1219
+ {
1220
+ "name": "third_person",
1221
+ "width": 960,
1222
+ "height": 720,
1223
+ "channel": "spectator"
1224
+ },
1225
+ {
1226
+ "name": "left_wrist",
1227
+ "width": 960,
1228
+ "height": 720,
1229
+ "channel": "spectator"
1230
+ },
1231
+ {
1232
+ "name": "right_wrist",
1233
+ "width": 960,
1234
+ "height": 720,
1235
+ "channel": "spectator"
1236
+ }
1237
+ ],
1238
+ "coverage_basis": "Every scheduled 10 Hz control-boundary sample, plus the terminal state.",
1239
+ "frame_mapping": "Each captured sample exactly once; no appended or held frames.",
1240
+ "sampling_note": "Samples follow the native simulation clock; model thinking adds no footage.",
1241
+ "scope": "Live owner-only spectator views; no replay, rerender, upscaling or invented frames.",
1242
+ "recording": {
1243
+ "schema": "rosbridge/recording/1",
1244
+ "run_id": "ebcedf7e2ae7",
1245
+ "task": "robotwin/place-dual-shoes-aloha-agilex",
1246
+ "state": "complete",
1247
+ "views": {
1248
+ "third_person": {
1249
+ "received": 201,
1250
+ "dropped": 0,
1251
+ "segments": [
1252
+ {
1253
+ "file": "third_person.mp4",
1254
+ "frames": 200,
1255
+ "width": 960,
1256
+ "height": 720,
1257
+ "fps": 10.0,
1258
+ "first_stamp_ns": 0,
1259
+ "last_stamp_ns": 19920000000
1260
+ }
1261
+ ],
1262
+ "error": null
1263
+ },
1264
+ "left_wrist": {
1265
+ "received": 201,
1266
+ "dropped": 0,
1267
+ "segments": [
1268
+ {
1269
+ "file": "left_wrist.mp4",
1270
+ "frames": 200,
1271
+ "width": 960,
1272
+ "height": 720,
1273
+ "fps": 10.0,
1274
+ "first_stamp_ns": 0,
1275
+ "last_stamp_ns": 19920000000
1276
+ }
1277
+ ],
1278
+ "error": null
1279
+ },
1280
+ "right_wrist": {
1281
+ "received": 201,
1282
+ "dropped": 0,
1283
+ "segments": [
1284
+ {
1285
+ "file": "right_wrist.mp4",
1286
+ "frames": 200,
1287
+ "width": 960,
1288
+ "height": 720,
1289
+ "fps": 10.0,
1290
+ "first_stamp_ns": 0,
1291
+ "last_stamp_ns": 19920000000
1292
+ }
1293
+ ],
1294
+ "error": null
1295
+ }
1296
+ },
1297
+ "type": "mp4",
1298
+ "spectator_views": [
1299
+ "third_person",
1300
+ "left_wrist",
1301
+ "right_wrist"
1302
+ ],
1303
+ "fps": 10.0,
1304
+ "topics": [],
1305
+ "error": null,
1306
+ "bag_messages": 0
1307
+ }
1308
+ },
1309
+ "analysis": {
1310
+ "observed": "Kinex ran 15 policy inferences (240 control frames), always with the full original instruction. The video shows pickup followed by lost placement. It then tried agent-written geometric recovery and stopped at frame 498 of 600, explicitly reporting failure and opening both grippers. This was an agent decision to stop, not budget exhaustion.",
1311
+ "execution": "Previous valid episode; superseded by the authorized 2026-10-29 rerun, not removed or retried to select a success."
1312
+ },
1313
+ "links": {
1314
+ "video": "experiments/robotwin-native/kinex_policy-2/presentation/video.mp4",
1315
+ "poster": "experiments/robotwin-native/kinex_policy-2/presentation/poster.jpg",
1316
+ "terminal_frame": "experiments/robotwin-native/kinex_policy-2/presentation/terminal.jpg",
1317
+ "transcript": "experiments/robotwin-native/kinex_policy-2/transcript.json",
1318
+ "usage": "experiments/robotwin-native/kinex_policy-2/usage.json",
1319
+ "analysis": "experiments/robotwin-native/kinex_policy-2/analysis.json",
1320
+ "protocol": "experiments/robotwin-native/kinex_policy-2/protocol.json",
1321
+ "native_goal": "experiments/robotwin-native/kinex_policy-2/instruction.json",
1322
+ "verdict": "experiments/robotwin-native/kinex_policy-2/episode.json",
1323
+ "provenance": "sources.json",
1324
+ "media_check": "experiments/robotwin-native/kinex_policy-2/presentation/media-validation.json",
1325
+ "recording_manifest": "experiments/robotwin-native/kinex_policy-2/presentation/recording.json",
1326
+ "workspace": "experiments/robotwin-native/kinex_policy-2/workspace.tar.gz",
1327
+ "workspace_changes": "experiments/robotwin-native/kinex_policy-2/workspace.json",
1328
+ "evidence_archive": "experiments/robotwin-native/kinex_policy-2/evidence.tar.gz",
1329
+ "tool_calls": "experiments/robotwin-native/kinex_policy-2/tool-calls.json",
1330
+ "artifact_index": "experiments/robotwin-native/kinex_policy-2/artifact-index.json",
1331
+ "native_session": "experiments/robotwin-native/kinex_policy-2/session.jsonl",
1332
+ "trace": "experiments/robotwin-native/kinex_policy-2/trace.json",
1333
+ "provider_usage": "experiments/robotwin-native/kinex_policy-2/provider-requests.jsonl",
1334
+ "vla_io": "experiments/robotwin-native/kinex_policy-2/vla-io.json",
1335
+ "policy_trace": "experiments/robotwin-native/kinex_policy-2/policy/calls.jsonl",
1336
+ "result_page": "https://huggingface.co/spaces/RLE-Bench/programmable-vla/blob/91f9b775b2f5a5b8355aa11989ac24a5c7f72214/data.json"
1337
+ },
1338
+ "resources": [
1339
+ {
1340
+ "name": ".kinex/tools/cartesian_segment.py",
1341
+ "file": "experiments/robotwin-native/kinex_policy-2/workspace/.kinex/tools/cartesian_segment.py",
1342
+ "kind": "Created during this episode; no imported earlier task resources."
1343
+ },
1344
+ {
1345
+ "name": ".kinex/tools/dual_shoes_policy.py",
1346
+ "file": "experiments/robotwin-native/kinex_policy-2/workspace/.kinex/tools/dual_shoes_policy.py",
1347
+ "kind": "Created during this episode; no imported earlier task resources."
1348
+ },
1349
+ {
1350
+ "name": ".kinex/memos/dual_shoes.md",
1351
+ "file": "experiments/robotwin-native/kinex_policy-2/workspace/.kinex/memos/dual_shoes.md",
1352
+ "kind": "Created during this episode; no imported earlier task resources."
1353
+ }
1354
+ ],
1355
+ "session_counts": {
1356
+ "visible_events": 137,
1357
+ "observed_images": 42
1358
+ },
1359
+ "measured_protocol": {
1360
+ "max_steps": 600,
1361
+ "seed": 0,
1362
+ "episodes": 1,
1363
+ "control_frequency_hz": 25,
1364
+ "provider": "rosbridge",
1365
+ "realtime": false,
1366
+ "stop_on_success": true
1367
+ },
1368
+ "vla_calls": 15,
1369
+ "kinex_revision": "f4ab59e9e93aa2f569d88386799a2bed0670d59c",
1370
+ "kinex_version": "1.1.0",
1371
+ "dataset_revision": "91f9b775b2f5a5b8355aa11989ac24a5c7f72214"
1372
+ },
1373
+ {
1374
+ "id": "kinex_policy-3",
1375
+ "variant": "kinex_policy",
1376
+ "task_key": "task03/03",
1377
+ "family": "task03",
1378
+ "source_family": "task03",
1379
+ "provider": "rosbridge",
1380
+ "slot": "03",
1381
+ "seed": 0,
1382
+ "source_episode_id": "kinex_policy-3",
1383
+ "status": "archived_completed",
1384
+ "success": true,
1385
+ "valid": true,
1386
+ "execution": {
1387
+ "status": "finished",
1388
+ "reason": null
1389
+ },
1390
+ "verdict": {
1391
+ "evidence_valid": true,
1392
+ "success": true,
1393
+ "termination": "success",
1394
+ "steps": 1277
1395
+ },
1396
+ "steps": 1277,
1397
+ "simulation_time_s": 51.08,
1398
+ "wall_time_s": 737.794664,
1399
+ "model": "GPT-6 Astra",
1400
+ "effort": "high",
1401
+ "harness": "kinex",
1402
+ "instruction": "use arms to grab the bottles and put them into the dustbin to the left of the table",
1403
+ "native_instruction": "use arms to grab the bottles and put them into the dustbin to the left of the table",
1404
+ "instruction_policy": "original_native",
1405
+ "usage": {
1406
+ "accounting": "reported-responses",
1407
+ "audit_complete": true,
1408
+ "audit_source": "Provider terminal response usage",
1409
+ "cache_hit_rate": 0.9726861083715472,
1410
+ "cache_read_input_tokens": 2026752,
1411
+ "cache_reported_input_tokens": 2083665,
1412
+ "cache_write_basis": "provider-reported",
1413
+ "cache_write_input_tokens": 0,
1414
+ "cache_write_reported_input_tokens": 2083665,
1415
+ "cached_input_tokens": 2026752,
1416
+ "cost_usd": null,
1417
+ "fresh_input_tokens": 56913,
1418
+ "harbor_mismatches": {},
1419
+ "input_breakdown_complete": true,
1420
+ "input_tokens": 2083665,
1421
+ "known_cache_write_input_tokens": 0,
1422
+ "known_cached_input_tokens": 2026752,
1423
+ "known_fresh_input_tokens": 56913,
1424
+ "known_input_tokens": 2083665,
1425
+ "known_output_tokens": 5246,
1426
+ "known_reasoning_output_tokens": 328,
1427
+ "output_tokens": 5246,
1428
+ "parser": "harbor/0.22.0",
1429
+ "reasoning_output_tokens": 328,
1430
+ "reasoning_reported_output_tokens": 5246,
1431
+ "rejected_attempts": 0,
1432
+ "reported_responses": {
1433
+ "cache_reported_input_tokens": 60,
1434
+ "cache_write_input_tokens": 60,
1435
+ "cache_write_reported_input_tokens": 60,
1436
+ "cached_input_tokens": 60,
1437
+ "fresh_input_tokens": 60,
1438
+ "input_tokens": 60,
1439
+ "output_tokens": 60,
1440
+ "reasoning_output_tokens": 60,
1441
+ "reasoning_reported_output_tokens": 60
1442
+ },
1443
+ "request_attempts": 60,
1444
+ "response_count": 60,
1445
+ "response_ids_complete": true,
1446
+ "schema": "rlebench/token-usage/1",
1447
+ "source": "Harbor session metrics",
1448
+ "total_tokens": 2088911,
1449
+ "uncached_input_tokens": 56913,
1450
+ "unresolved_attempts": [],
1451
+ "usage_records": [
1452
+ {
1453
+ "cache_write_input_tokens": 0,
1454
+ "cached_input_tokens": 0,
1455
+ "input_tokens": 11303,
1456
+ "output_tokens": 19,
1457
+ "reasoning_output_tokens": 0
1458
+ },
1459
+ {
1460
+ "cache_write_input_tokens": 0,
1461
+ "cached_input_tokens": 11136,
1462
+ "input_tokens": 11343,
1463
+ "output_tokens": 29,
1464
+ "reasoning_output_tokens": 0
1465
+ },
1466
+ {
1467
+ "cache_write_input_tokens": 0,
1468
+ "cached_input_tokens": 11264,
1469
+ "input_tokens": 11476,
1470
+ "output_tokens": 30,
1471
+ "reasoning_output_tokens": 0
1472
+ },
1473
+ {
1474
+ "cache_write_input_tokens": 0,
1475
+ "cached_input_tokens": 11392,
1476
+ "input_tokens": 15499,
1477
+ "output_tokens": 180,
1478
+ "reasoning_output_tokens": 0
1479
+ },
1480
+ {
1481
+ "cache_write_input_tokens": 0,
1482
+ "cached_input_tokens": 15360,
1483
+ "input_tokens": 17418,
1484
+ "output_tokens": 210,
1485
+ "reasoning_output_tokens": 0
1486
+ },
1487
+ {
1488
+ "cache_write_input_tokens": 0,
1489
+ "cached_input_tokens": 17280,
1490
+ "input_tokens": 22354,
1491
+ "output_tokens": 77,
1492
+ "reasoning_output_tokens": 10
1493
+ },
1494
+ {
1495
+ "cache_write_input_tokens": 0,
1496
+ "cached_input_tokens": 22272,
1497
+ "input_tokens": 22900,
1498
+ "output_tokens": 81,
1499
+ "reasoning_output_tokens": 0
1500
+ },
1501
+ {
1502
+ "cache_write_input_tokens": 0,
1503
+ "cached_input_tokens": 22784,
1504
+ "input_tokens": 23208,
1505
+ "output_tokens": 77,
1506
+ "reasoning_output_tokens": 0
1507
+ },
1508
+ {
1509
+ "cache_write_input_tokens": 0,
1510
+ "cached_input_tokens": 23040,
1511
+ "input_tokens": 24559,
1512
+ "output_tokens": 107,
1513
+ "reasoning_output_tokens": 0
1514
+ },
1515
+ {
1516
+ "cache_write_input_tokens": 0,
1517
+ "cached_input_tokens": 24448,
1518
+ "input_tokens": 24883,
1519
+ "output_tokens": 23,
1520
+ "reasoning_output_tokens": 0
1521
+ },
1522
+ {
1523
+ "cache_write_input_tokens": 0,
1524
+ "cached_input_tokens": 24704,
1525
+ "input_tokens": 25073,
1526
+ "output_tokens": 40,
1527
+ "reasoning_output_tokens": 0
1528
+ },
1529
+ {
1530
+ "cache_write_input_tokens": 0,
1531
+ "cached_input_tokens": 24960,
1532
+ "input_tokens": 27403,
1533
+ "output_tokens": 385,
1534
+ "reasoning_output_tokens": 0
1535
+ },
1536
+ {
1537
+ "cache_write_input_tokens": 0,
1538
+ "cached_input_tokens": 27264,
1539
+ "input_tokens": 27811,
1540
+ "output_tokens": 39,
1541
+ "reasoning_output_tokens": 0
1542
+ },
1543
+ {
1544
+ "cache_write_input_tokens": 0,
1545
+ "cached_input_tokens": 27648,
1546
+ "input_tokens": 28265,
1547
+ "output_tokens": 61,
1548
+ "reasoning_output_tokens": 22
1549
+ },
1550
+ {
1551
+ "cache_write_input_tokens": 0,
1552
+ "cached_input_tokens": 28032,
1553
+ "input_tokens": 28738,
1554
+ "output_tokens": 38,
1555
+ "reasoning_output_tokens": 0
1556
+ },
1557
+ {
1558
+ "cache_write_input_tokens": 0,
1559
+ "cached_input_tokens": 28544,
1560
+ "input_tokens": 29189,
1561
+ "output_tokens": 38,
1562
+ "reasoning_output_tokens": 0
1563
+ },
1564
+ {
1565
+ "cache_write_input_tokens": 0,
1566
+ "cached_input_tokens": 28928,
1567
+ "input_tokens": 29640,
1568
+ "output_tokens": 49,
1569
+ "reasoning_output_tokens": 9
1570
+ },
1571
+ {
1572
+ "cache_write_input_tokens": 0,
1573
+ "cached_input_tokens": 29568,
1574
+ "input_tokens": 30101,
1575
+ "output_tokens": 37,
1576
+ "reasoning_output_tokens": 0
1577
+ },
1578
+ {
1579
+ "cache_write_input_tokens": 0,
1580
+ "cached_input_tokens": 29952,
1581
+ "input_tokens": 30552,
1582
+ "output_tokens": 37,
1583
+ "reasoning_output_tokens": 0
1584
+ },
1585
+ {
1586
+ "cache_write_input_tokens": 0,
1587
+ "cached_input_tokens": 30464,
1588
+ "input_tokens": 31003,
1589
+ "output_tokens": 38,
1590
+ "reasoning_output_tokens": 0
1591
+ },
1592
+ {
1593
+ "cache_write_input_tokens": 0,
1594
+ "cached_input_tokens": 30848,
1595
+ "input_tokens": 31456,
1596
+ "output_tokens": 39,
1597
+ "reasoning_output_tokens": 0
1598
+ },
1599
+ {
1600
+ "cache_write_input_tokens": 0,
1601
+ "cached_input_tokens": 31360,
1602
+ "input_tokens": 31913,
1603
+ "output_tokens": 39,
1604
+ "reasoning_output_tokens": 0
1605
+ },
1606
+ {
1607
+ "cache_write_input_tokens": 0,
1608
+ "cached_input_tokens": 31744,
1609
+ "input_tokens": 32368,
1610
+ "output_tokens": 39,
1611
+ "reasoning_output_tokens": 0
1612
+ },
1613
+ {
1614
+ "cache_write_input_tokens": 0,
1615
+ "cached_input_tokens": 32128,
1616
+ "input_tokens": 32821,
1617
+ "output_tokens": 38,
1618
+ "reasoning_output_tokens": 0
1619
+ },
1620
+ {
1621
+ "cache_write_input_tokens": 0,
1622
+ "cached_input_tokens": 32640,
1623
+ "input_tokens": 33269,
1624
+ "output_tokens": 414,
1625
+ "reasoning_output_tokens": 35
1626
+ },
1627
+ {
1628
+ "cache_write_input_tokens": 0,
1629
+ "cached_input_tokens": 33152,
1630
+ "input_tokens": 33712,
1631
+ "output_tokens": 72,
1632
+ "reasoning_output_tokens": 0
1633
+ },
1634
+ {
1635
+ "cache_write_input_tokens": 0,
1636
+ "cached_input_tokens": 33664,
1637
+ "input_tokens": 34232,
1638
+ "output_tokens": 44,
1639
+ "reasoning_output_tokens": 0
1640
+ },
1641
+ {
1642
+ "cache_write_input_tokens": 0,
1643
+ "cached_input_tokens": 34176,
1644
+ "input_tokens": 34724,
1645
+ "output_tokens": 43,
1646
+ "reasoning_output_tokens": 0
1647
+ },
1648
+ {
1649
+ "cache_write_input_tokens": 0,
1650
+ "cached_input_tokens": 34560,
1651
+ "input_tokens": 35218,
1652
+ "output_tokens": 45,
1653
+ "reasoning_output_tokens": 0
1654
+ },
1655
+ {
1656
+ "cache_write_input_tokens": 0,
1657
+ "cached_input_tokens": 35072,
1658
+ "input_tokens": 35712,
1659
+ "output_tokens": 59,
1660
+ "reasoning_output_tokens": 12
1661
+ },
1662
+ {
1663
+ "cache_write_input_tokens": 0,
1664
+ "cached_input_tokens": 35584,
1665
+ "input_tokens": 36237,
1666
+ "output_tokens": 45,
1667
+ "reasoning_output_tokens": 0
1668
+ },
1669
+ {
1670
+ "cache_write_input_tokens": 0,
1671
+ "cached_input_tokens": 36096,
1672
+ "input_tokens": 36747,
1673
+ "output_tokens": 67,
1674
+ "reasoning_output_tokens": 20
1675
+ },
1676
+ {
1677
+ "cache_write_input_tokens": 0,
1678
+ "cached_input_tokens": 36608,
1679
+ "input_tokens": 37276,
1680
+ "output_tokens": 59,
1681
+ "reasoning_output_tokens": 20
1682
+ },
1683
+ {
1684
+ "cache_write_input_tokens": 0,
1685
+ "cached_input_tokens": 37120,
1686
+ "input_tokens": 37660,
1687
+ "output_tokens": 65,
1688
+ "reasoning_output_tokens": 21
1689
+ },
1690
+ {
1691
+ "cache_write_input_tokens": 0,
1692
+ "cached_input_tokens": 37504,
1693
+ "input_tokens": 37935,
1694
+ "output_tokens": 129,
1695
+ "reasoning_output_tokens": 15
1696
+ },
1697
+ {
1698
+ "cache_write_input_tokens": 0,
1699
+ "cached_input_tokens": 37888,
1700
+ "input_tokens": 38650,
1701
+ "output_tokens": 33,
1702
+ "reasoning_output_tokens": 0
1703
+ },
1704
+ {
1705
+ "cache_write_input_tokens": 0,
1706
+ "cached_input_tokens": 38528,
1707
+ "input_tokens": 38788,
1708
+ "output_tokens": 105,
1709
+ "reasoning_output_tokens": 29
1710
+ },
1711
+ {
1712
+ "cache_write_input_tokens": 0,
1713
+ "cached_input_tokens": 38656,
1714
+ "input_tokens": 39125,
1715
+ "output_tokens": 220,
1716
+ "reasoning_output_tokens": 0
1717
+ },
1718
+ {
1719
+ "cache_write_input_tokens": 0,
1720
+ "cached_input_tokens": 39040,
1721
+ "input_tokens": 39697,
1722
+ "output_tokens": 558,
1723
+ "reasoning_output_tokens": 60
1724
+ },
1725
+ {
1726
+ "cache_write_input_tokens": 0,
1727
+ "cached_input_tokens": 39552,
1728
+ "input_tokens": 40278,
1729
+ "output_tokens": 63,
1730
+ "reasoning_output_tokens": 0
1731
+ },
1732
+ {
1733
+ "cache_write_input_tokens": 0,
1734
+ "cached_input_tokens": 40192,
1735
+ "input_tokens": 40564,
1736
+ "output_tokens": 53,
1737
+ "reasoning_output_tokens": 0
1738
+ },
1739
+ {
1740
+ "cache_write_input_tokens": 0,
1741
+ "cached_input_tokens": 40448,
1742
+ "input_tokens": 40840,
1743
+ "output_tokens": 61,
1744
+ "reasoning_output_tokens": 0
1745
+ },
1746
+ {
1747
+ "cache_write_input_tokens": 0,
1748
+ "cached_input_tokens": 40704,
1749
+ "input_tokens": 41123,
1750
+ "output_tokens": 45,
1751
+ "reasoning_output_tokens": 0
1752
+ },
1753
+ {
1754
+ "cache_write_input_tokens": 0,
1755
+ "cached_input_tokens": 40960,
1756
+ "input_tokens": 41615,
1757
+ "output_tokens": 58,
1758
+ "reasoning_output_tokens": 12
1759
+ },
1760
+ {
1761
+ "cache_write_input_tokens": 0,
1762
+ "cached_input_tokens": 41472,
1763
+ "input_tokens": 42118,
1764
+ "output_tokens": 46,
1765
+ "reasoning_output_tokens": 0
1766
+ },
1767
+ {
1768
+ "cache_write_input_tokens": 0,
1769
+ "cached_input_tokens": 41984,
1770
+ "input_tokens": 42611,
1771
+ "output_tokens": 45,
1772
+ "reasoning_output_tokens": 0
1773
+ },
1774
+ {
1775
+ "cache_write_input_tokens": 0,
1776
+ "cached_input_tokens": 42368,
1777
+ "input_tokens": 43105,
1778
+ "output_tokens": 46,
1779
+ "reasoning_output_tokens": 0
1780
+ },
1781
+ {
1782
+ "cache_write_input_tokens": 0,
1783
+ "cached_input_tokens": 43008,
1784
+ "input_tokens": 43598,
1785
+ "output_tokens": 81,
1786
+ "reasoning_output_tokens": 10
1787
+ },
1788
+ {
1789
+ "cache_write_input_tokens": 0,
1790
+ "cached_input_tokens": 43520,
1791
+ "input_tokens": 44118,
1792
+ "output_tokens": 43,
1793
+ "reasoning_output_tokens": 0
1794
+ },
1795
+ {
1796
+ "cache_write_input_tokens": 0,
1797
+ "cached_input_tokens": 44032,
1798
+ "input_tokens": 44607,
1799
+ "output_tokens": 46,
1800
+ "reasoning_output_tokens": 0
1801
+ },
1802
+ {
1803
+ "cache_write_input_tokens": 0,
1804
+ "cached_input_tokens": 44416,
1805
+ "input_tokens": 45102,
1806
+ "output_tokens": 79,
1807
+ "reasoning_output_tokens": 14
1808
+ },
1809
+ {
1810
+ "cache_write_input_tokens": 0,
1811
+ "cached_input_tokens": 44928,
1812
+ "input_tokens": 45404,
1813
+ "output_tokens": 46,
1814
+ "reasoning_output_tokens": 0
1815
+ },
1816
+ {
1817
+ "cache_write_input_tokens": 0,
1818
+ "cached_input_tokens": 45184,
1819
+ "input_tokens": 45896,
1820
+ "output_tokens": 47,
1821
+ "reasoning_output_tokens": 0
1822
+ },
1823
+ {
1824
+ "cache_write_input_tokens": 0,
1825
+ "cached_input_tokens": 45696,
1826
+ "input_tokens": 46388,
1827
+ "output_tokens": 45,
1828
+ "reasoning_output_tokens": 0
1829
+ },
1830
+ {
1831
+ "cache_write_input_tokens": 0,
1832
+ "cached_input_tokens": 46208,
1833
+ "input_tokens": 46878,
1834
+ "output_tokens": 43,
1835
+ "reasoning_output_tokens": 0
1836
+ },
1837
+ {
1838
+ "cache_write_input_tokens": 0,
1839
+ "cached_input_tokens": 46720,
1840
+ "input_tokens": 47354,
1841
+ "output_tokens": 44,
1842
+ "reasoning_output_tokens": 0
1843
+ },
1844
+ {
1845
+ "cache_write_input_tokens": 0,
1846
+ "cached_input_tokens": 47232,
1847
+ "input_tokens": 47529,
1848
+ "output_tokens": 28,
1849
+ "reasoning_output_tokens": 0
1850
+ },
1851
+ {
1852
+ "cache_write_input_tokens": 0,
1853
+ "cached_input_tokens": 47360,
1854
+ "input_tokens": 47619,
1855
+ "output_tokens": 49,
1856
+ "reasoning_output_tokens": 0
1857
+ },
1858
+ {
1859
+ "cache_write_input_tokens": 0,
1860
+ "cached_input_tokens": 47488,
1861
+ "input_tokens": 48050,
1862
+ "output_tokens": 531,
1863
+ "reasoning_output_tokens": 39
1864
+ },
1865
+ {
1866
+ "cache_write_input_tokens": 0,
1867
+ "cached_input_tokens": 47872,
1868
+ "input_tokens": 48610,
1869
+ "output_tokens": 39,
1870
+ "reasoning_output_tokens": 0
1871
+ }
1872
+ ],
1873
+ "usage_source": "provider-raw"
1874
+ },
1875
+ "call_activity": {
1876
+ "schema": "rlebench/call-activity/1",
1877
+ "source": "Kinex native trace",
1878
+ "model_tool_calls": 69,
1879
+ "model_tool_calls_by_name": {
1880
+ "ls": 3,
1881
+ "read": 15,
1882
+ "find": 1,
1883
+ "exec": 40,
1884
+ "devices": 3,
1885
+ "device": 1,
1886
+ "write": 3,
1887
+ "edit": 1,
1888
+ "relative_pose": 1,
1889
+ "device_stop": 1
1890
+ },
1891
+ "python_instrumented_model_tool_calls": null,
1892
+ "python_tool_invocations": null,
1893
+ "nested_python_tool_invocations": null,
1894
+ "python_tool_invocations_by_origin": null,
1895
+ "python_device_rpc_attempts": null,
1896
+ "python_device_rpc_attempts_by_action": null,
1897
+ "python_device_rpc_errors": null,
1898
+ "python_scope": "Recorded instrumented Python invocations only; other paths and automatic fatal stops are not counted."
1899
+ },
1900
+ "media": {
1901
+ "schema": "rlebench/native-camera-media/3",
1902
+ "passed": true,
1903
+ "speed": 4,
1904
+ "source_fps": 10,
1905
+ "output_fps": 40,
1906
+ "width": 2880,
1907
+ "height": 720,
1908
+ "native_frames": 512,
1909
+ "published_frames": 512,
1910
+ "source_sample_coverage": 1.0,
1911
+ "source_samples_complete": true,
1912
+ "source_dropped_samples": 0,
1913
+ "terminal_frame_retained": true,
1914
+ "terminal_source_index": 511,
1915
+ "terminal_hold_s": 0,
1916
+ "simulation_time_s": 51.08,
1917
+ "duration_s": 12.8,
1918
+ "sha256": "e182700df6a1b1b83c33ea1bef6ee088dec66690418ac77fb59dab6b0adb4d67",
1919
+ "source_sha256": {
1920
+ "third_person.mp4": "cc7356c76a92f5fa5a5ac2991295bdc3069c94774f69fef9b09e131fa538112f",
1921
+ "left_wrist.mp4": "40132ed56a64bbbaac618077114b8f34d5f0befdbc1b18291531ad36e88730b9",
1922
+ "right_wrist.mp4": "947a864a65109bd4bcbb11329f06bf31cbed24ad1ea3312ce7ea5e5b9547c94a"
1923
+ },
1924
+ "all_frames_compared": 512,
1925
+ "minimum_frame_psnr_db": 41.518423372042804,
1926
+ "maximum_frame_rgb_mae": 1.4636712074279785,
1927
+ "decode_ok": true,
1928
+ "pts_verified": true,
1929
+ "playback_kind": "simulation_time",
1930
+ "resolution_kind": "independent_720_high",
1931
+ "agent_observation_video": false,
1932
+ "camera_views": [
1933
+ {
1934
+ "name": "third_person",
1935
+ "width": 960,
1936
+ "height": 720,
1937
+ "channel": "spectator"
1938
+ },
1939
+ {
1940
+ "name": "left_wrist",
1941
+ "width": 960,
1942
+ "height": 720,
1943
+ "channel": "spectator"
1944
+ },
1945
+ {
1946
+ "name": "right_wrist",
1947
+ "width": 960,
1948
+ "height": 720,
1949
+ "channel": "spectator"
1950
+ }
1951
+ ],
1952
+ "coverage_basis": "Every scheduled 10 Hz control-boundary sample, plus the terminal state.",
1953
+ "frame_mapping": "Each captured sample exactly once; no appended or held frames.",
1954
+ "sampling_note": "Samples follow the native simulation clock; model thinking adds no footage.",
1955
+ "scope": "Live owner-only spectator views; no replay, rerender, upscaling or invented frames.",
1956
+ "recording": {
1957
+ "schema": "rosbridge/recording/1",
1958
+ "run_id": "88c15864ae34",
1959
+ "task": "robotwin/put-bottles-dustbin-aloha-agilex",
1960
+ "state": "complete",
1961
+ "views": {
1962
+ "third_person": {
1963
+ "received": 512,
1964
+ "dropped": 0,
1965
+ "segments": [
1966
+ {
1967
+ "file": "third_person.mp4",
1968
+ "frames": 512,
1969
+ "width": 960,
1970
+ "height": 720,
1971
+ "fps": 10.0,
1972
+ "first_stamp_ns": 0,
1973
+ "last_stamp_ns": 51080000000
1974
+ }
1975
+ ],
1976
+ "error": null
1977
+ },
1978
+ "left_wrist": {
1979
+ "received": 512,
1980
+ "dropped": 0,
1981
+ "segments": [
1982
+ {
1983
+ "file": "left_wrist.mp4",
1984
+ "frames": 512,
1985
+ "width": 960,
1986
+ "height": 720,
1987
+ "fps": 10.0,
1988
+ "first_stamp_ns": 0,
1989
+ "last_stamp_ns": 51080000000
1990
+ }
1991
+ ],
1992
+ "error": null
1993
+ },
1994
+ "right_wrist": {
1995
+ "received": 512,
1996
+ "dropped": 0,
1997
+ "segments": [
1998
+ {
1999
+ "file": "right_wrist.mp4",
2000
+ "frames": 512,
2001
+ "width": 960,
2002
+ "height": 720,
2003
+ "fps": 10.0,
2004
+ "first_stamp_ns": 0,
2005
+ "last_stamp_ns": 51080000000
2006
+ }
2007
+ ],
2008
+ "error": null
2009
+ }
2010
+ },
2011
+ "type": "mp4",
2012
+ "spectator_views": [
2013
+ "third_person",
2014
+ "left_wrist",
2015
+ "right_wrist"
2016
+ ],
2017
+ "fps": 10.0,
2018
+ "topics": [],
2019
+ "error": null,
2020
+ "bag_messages": 0
2021
+ }
2022
+ },
2023
+ "analysis": {
2024
+ "observed": "Kinex made 72 policy inferences and wrote its own loop of up to four freshly observed chunks per tool call. When grasping stalled, it measured RGB-D geometry and wrote Cartesian corrections, then resumed the policy. Native success occurred at frame 1,277. Every VLA call retained the full original task instruction; this is observed control recovery, not language subtask rewriting.",
2025
+ "execution": "Previous valid episode; superseded by the authorized 2026-10-29 rerun, not removed or retried to select a success."
2026
+ },
2027
+ "links": {
2028
+ "video": "experiments/robotwin-native/kinex_policy-3/presentation/video.mp4",
2029
+ "poster": "experiments/robotwin-native/kinex_policy-3/presentation/poster.jpg",
2030
+ "terminal_frame": "experiments/robotwin-native/kinex_policy-3/presentation/terminal.jpg",
2031
+ "transcript": "experiments/robotwin-native/kinex_policy-3/transcript.json",
2032
+ "usage": "experiments/robotwin-native/kinex_policy-3/usage.json",
2033
+ "analysis": "experiments/robotwin-native/kinex_policy-3/analysis.json",
2034
+ "protocol": "experiments/robotwin-native/kinex_policy-3/protocol.json",
2035
+ "native_goal": "experiments/robotwin-native/kinex_policy-3/instruction.json",
2036
+ "verdict": "experiments/robotwin-native/kinex_policy-3/episode.json",
2037
+ "provenance": "sources.json",
2038
+ "media_check": "experiments/robotwin-native/kinex_policy-3/presentation/media-validation.json",
2039
+ "recording_manifest": "experiments/robotwin-native/kinex_policy-3/presentation/recording.json",
2040
+ "workspace": "experiments/robotwin-native/kinex_policy-3/workspace.tar.gz",
2041
+ "workspace_changes": "experiments/robotwin-native/kinex_policy-3/workspace.json",
2042
+ "evidence_archive": "experiments/robotwin-native/kinex_policy-3/evidence.tar.gz",
2043
+ "tool_calls": "experiments/robotwin-native/kinex_policy-3/tool-calls.json",
2044
+ "artifact_index": "experiments/robotwin-native/kinex_policy-3/artifact-index.json",
2045
+ "native_session": "experiments/robotwin-native/kinex_policy-3/session.jsonl",
2046
+ "trace": "experiments/robotwin-native/kinex_policy-3/trace.json",
2047
+ "provider_usage": "experiments/robotwin-native/kinex_policy-3/provider-requests.jsonl",
2048
+ "vla_io": "experiments/robotwin-native/kinex_policy-3/vla-io.json",
2049
+ "policy_trace": "experiments/robotwin-native/kinex_policy-3/policy/calls.jsonl",
2050
+ "result_page": "https://huggingface.co/spaces/RLE-Bench/programmable-vla/blob/91f9b775b2f5a5b8355aa11989ac24a5c7f72214/data.json"
2051
+ },
2052
+ "resources": [
2053
+ {
2054
+ "name": ".kinex/tools/bottle_policy.py",
2055
+ "file": "experiments/robotwin-native/kinex_policy-3/workspace/.kinex/tools/bottle_policy.py",
2056
+ "kind": "Created during this episode; no imported earlier task resources."
2057
+ },
2058
+ {
2059
+ "name": ".kinex/tools/cartesian_adjust.py",
2060
+ "file": "experiments/robotwin-native/kinex_policy-3/workspace/.kinex/tools/cartesian_adjust.py",
2061
+ "kind": "Created during this episode; no imported earlier task resources."
2062
+ },
2063
+ {
2064
+ "name": ".kinex/memos/put-bottles-dustbin.md",
2065
+ "file": "experiments/robotwin-native/kinex_policy-3/workspace/.kinex/memos/put-bottles-dustbin.md",
2066
+ "kind": "Created during this episode; no imported earlier task resources."
2067
+ }
2068
+ ],
2069
+ "session_counts": {
2070
+ "visible_events": 131,
2071
+ "observed_images": 40
2072
+ },
2073
+ "measured_protocol": {
2074
+ "max_steps": 1700,
2075
+ "seed": 0,
2076
+ "episodes": 1,
2077
+ "control_frequency_hz": 25,
2078
+ "provider": "rosbridge",
2079
+ "realtime": false,
2080
+ "stop_on_success": true
2081
+ },
2082
+ "vla_calls": 72,
2083
+ "kinex_revision": "f4ab59e9e93aa2f569d88386799a2bed0670d59c",
2084
+ "kinex_version": "1.1.0",
2085
+ "dataset_revision": "91f9b775b2f5a5b8355aa11989ac24a5c7f72214"
2086
+ }
2087
+ ]
cost-data.json CHANGED
@@ -1521,6 +1521,180 @@
1521
  "cache_upper": 0,
1522
  "long_context_responses": 0,
1523
  "cache_write_assumed": false
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1524
  }
1525
  },
1526
  "sources": [],
 
1521
  "cache_upper": 0,
1522
  "long_context_responses": 0,
1523
  "cache_write_assumed": false
1524
+ },
1525
+ "kinex_policy-rerun-20261029-1": {
1526
+ "signature": [
1527
+ "kinex_policy-rerun-20261029-1",
1528
+ "experiments/kinex-vla-rerun-20261029/kinex_policy-rerun-20261029-1/usage.json",
1529
+ 1786672,
1530
+ 9118,
1531
+ 1739264,
1532
+ 0,
1533
+ null
1534
+ ],
1535
+ "lower": 2.669244,
1536
+ "upper": 2.669244,
1537
+ "actual_billed_usd": null,
1538
+ "basis": "native_pi_session_api_equivalent",
1539
+ "buckets": [
1540
+ {
1541
+ "category": "fresh_input",
1542
+ "tier": "Recorded session",
1543
+ "tokens": 47408,
1544
+ "rate": 10.0,
1545
+ "rate_upper": 10.0,
1546
+ "lower": 0.47408000000000006,
1547
+ "upper": 0.47408000000000006
1548
+ },
1549
+ {
1550
+ "category": "cache_read",
1551
+ "tier": "Recorded session",
1552
+ "tokens": 1739264,
1553
+ "rate": 0.9999999999999998,
1554
+ "rate_upper": 0.9999999999999998,
1555
+ "lower": 1.7392639999999995,
1556
+ "upper": 1.7392639999999995
1557
+ },
1558
+ {
1559
+ "category": "cache_write",
1560
+ "tier": "Recorded session",
1561
+ "tokens": 0,
1562
+ "rate": null,
1563
+ "rate_upper": null,
1564
+ "lower": 0.0,
1565
+ "upper": 0.0
1566
+ },
1567
+ {
1568
+ "category": "output",
1569
+ "tier": "Recorded session",
1570
+ "tokens": 9118,
1571
+ "rate": 50.00000000000002,
1572
+ "rate_upper": 50.00000000000002,
1573
+ "lower": 0.4559000000000002,
1574
+ "upper": 0.4559000000000002
1575
+ }
1576
+ ],
1577
+ "cache_lower": 1739264,
1578
+ "cache_upper": 1739264,
1579
+ "long_context_responses": 0,
1580
+ "cache_write_assumed": false,
1581
+ "evidence_url": "experiments/kinex-vla-rerun-20261029/kinex_policy-rerun-20261029-1/session.jsonl"
1582
+ },
1583
+ "kinex_policy-rerun-20261029-2": {
1584
+ "signature": [
1585
+ "kinex_policy-rerun-20261029-2",
1586
+ "experiments/kinex-vla-rerun-20261029/kinex_policy-rerun-20261029-2/usage.json",
1587
+ 1339050,
1588
+ 7068,
1589
+ 1290496,
1590
+ 0,
1591
+ null
1592
+ ],
1593
+ "lower": 2.129436,
1594
+ "upper": 2.129436,
1595
+ "actual_billed_usd": null,
1596
+ "basis": "native_pi_session_api_equivalent",
1597
+ "buckets": [
1598
+ {
1599
+ "category": "fresh_input",
1600
+ "tier": "Recorded session",
1601
+ "tokens": 48554,
1602
+ "rate": 10.0,
1603
+ "rate_upper": 10.0,
1604
+ "lower": 0.48554,
1605
+ "upper": 0.48554
1606
+ },
1607
+ {
1608
+ "category": "cache_read",
1609
+ "tier": "Recorded session",
1610
+ "tokens": 1290496,
1611
+ "rate": 1.0000000000000002,
1612
+ "rate_upper": 1.0000000000000002,
1613
+ "lower": 1.290496,
1614
+ "upper": 1.290496
1615
+ },
1616
+ {
1617
+ "category": "cache_write",
1618
+ "tier": "Recorded session",
1619
+ "tokens": 0,
1620
+ "rate": null,
1621
+ "rate_upper": null,
1622
+ "lower": 0.0,
1623
+ "upper": 0.0
1624
+ },
1625
+ {
1626
+ "category": "output",
1627
+ "tier": "Recorded session",
1628
+ "tokens": 7068,
1629
+ "rate": 49.999999999999986,
1630
+ "rate_upper": 49.999999999999986,
1631
+ "lower": 0.35339999999999994,
1632
+ "upper": 0.35339999999999994
1633
+ }
1634
+ ],
1635
+ "cache_lower": 1290496,
1636
+ "cache_upper": 1290496,
1637
+ "long_context_responses": 0,
1638
+ "cache_write_assumed": false,
1639
+ "evidence_url": "experiments/kinex-vla-rerun-20261029/kinex_policy-rerun-20261029-2/session.jsonl"
1640
+ },
1641
+ "kinex_policy-rerun-20261029-3": {
1642
+ "signature": [
1643
+ "kinex_policy-rerun-20261029-3",
1644
+ "experiments/kinex-vla-rerun-20261029/kinex_policy-rerun-20261029-3/usage.json",
1645
+ 3334082,
1646
+ 11045,
1647
+ 3258496,
1648
+ 0,
1649
+ null
1650
+ ],
1651
+ "lower": 4.566605999999999,
1652
+ "upper": 4.566605999999999,
1653
+ "actual_billed_usd": null,
1654
+ "basis": "native_pi_session_api_equivalent",
1655
+ "buckets": [
1656
+ {
1657
+ "category": "fresh_input",
1658
+ "tier": "Recorded session",
1659
+ "tokens": 75586,
1660
+ "rate": 10.000000000000002,
1661
+ "rate_upper": 10.000000000000002,
1662
+ "lower": 0.7558600000000002,
1663
+ "upper": 0.7558600000000002
1664
+ },
1665
+ {
1666
+ "category": "cache_read",
1667
+ "tier": "Recorded session",
1668
+ "tokens": 3258496,
1669
+ "rate": 0.9999999999999998,
1670
+ "rate_upper": 0.9999999999999998,
1671
+ "lower": 3.258495999999999,
1672
+ "upper": 3.258495999999999
1673
+ },
1674
+ {
1675
+ "category": "cache_write",
1676
+ "tier": "Recorded session",
1677
+ "tokens": 0,
1678
+ "rate": null,
1679
+ "rate_upper": null,
1680
+ "lower": 0.0,
1681
+ "upper": 0.0
1682
+ },
1683
+ {
1684
+ "category": "output",
1685
+ "tier": "Recorded session",
1686
+ "tokens": 11045,
1687
+ "rate": 50.0,
1688
+ "rate_upper": 50.0,
1689
+ "lower": 0.55225,
1690
+ "upper": 0.55225
1691
+ }
1692
+ ],
1693
+ "cache_lower": 3258496,
1694
+ "cache_upper": 3258496,
1695
+ "long_context_responses": 0,
1696
+ "cache_write_assumed": false,
1697
+ "evidence_url": "experiments/kinex-vla-rerun-20261029/kinex_policy-rerun-20261029-3/session.jsonl"
1698
  }
1699
  },
1700
  "sources": [],
data.json CHANGED
The diff for this file is too large to render. See raw diff
 
index.html CHANGED
@@ -9,7 +9,7 @@
9
  <link rel="icon" href="icon.svg" type="image/svg+xml">
10
  <link rel="stylesheet" href="style.css?v=30978a82c360">
11
  <script src="usage-metrics.js?v=330e92f8eb96" defer></script>
12
- <script src="app.js?v=3370b7d0b865" defer></script>
13
  <link rel="stylesheet" href="policy.css">
14
  </head>
15
  <body>
 
9
  <link rel="icon" href="icon.svg" type="image/svg+xml">
10
  <link rel="stylesheet" href="style.css?v=30978a82c360">
11
  <script src="usage-metrics.js?v=330e92f8eb96" defer></script>
12
+ <script src="app.js?v=85d9cccf453c" defer></script>
13
  <link rel="stylesheet" href="policy.css">
14
  </head>
15
  <body>
manifest.json CHANGED
The diff for this file is too large to render. See raw diff
 
measured-instructions.json CHANGED
@@ -412,6 +412,27 @@
412
  "text": "press the switch fully down",
413
  "kind": "task_instruction",
414
  "observed_native": null
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
415
  }
416
  }
417
  }
 
412
  "text": "press the switch fully down",
413
  "kind": "task_instruction",
414
  "observed_native": null
415
+ },
416
+ "kinex_policy-rerun-20261029-1": {
417
+ "task_key": "task03/01",
418
+ "initial_goal": "use the left arm to grasp the red block on the table, hand it over to the right arm, and set it down in the center of the blue pad. Release it with the right gripper open",
419
+ "text": "use the left arm to grasp the red block on the table, hand it over to the right arm, and set it down in the center of the blue pad. Release it with the right gripper open",
420
+ "kind": "task_instruction",
421
+ "observed_native": "use the left arm to grasp the red block on the table, handover it to the right arm and place it on the blue pad"
422
+ },
423
+ "kinex_policy-rerun-20261029-2": {
424
+ "task_key": "task03/02",
425
+ "initial_goal": "use both arms to pick up the two shoes on the table and put them flat in the shoebox, with the shoe tips pointing to the left and the shoe initially on the left nearer the robot. Leave both grippers open",
426
+ "text": "use both arms to pick up the two shoes on the table and put them flat in the shoebox, with the shoe tips pointing to the left and the shoe initially on the left nearer the robot. Leave both grippers open",
427
+ "kind": "task_instruction",
428
+ "observed_native": "use both arms to pick up the two shoes on the table and put them in the shoebox, with the shoe tip pointing to the left"
429
+ },
430
+ "kinex_policy-rerun-20261029-3": {
431
+ "task_key": "task03/03",
432
+ "initial_goal": "use arms to grab the bottles and put them into the dustbin to the left of the table",
433
+ "text": "use arms to grab the bottles and put them into the dustbin to the left of the table",
434
+ "kind": "task_instruction",
435
+ "observed_native": "use arms to grab the bottles and put them into the dustbin to the left of the table"
436
  }
437
  }
438
  }