`;
}
function recordingPilotProgress(){
const p=data.recording_pilot;if(!p)return '';
return `Codex + RoboBridge · GPT-6 Astra / high · GPU recording pilot
${number(p.completed)} / ${number(p.planned)} independent attempts published: LIBERO, RoboTwin, and two RoboDojo recording conditions. This pilot does not cover the full benchmark.
LIBERO and RoboTwin use independent 720-high spectator cameras. RoboDojo compares native camera recording with online HD recording. Both keep native agent camera resolution, but the RTX control test found that extra HD cameras change subsequent native image pixels. The HD control is shown separately.
Videos play at 4× simulation speed. Session galleries contain images actually submitted to the agent. One model episode per condition cannot establish a success-rate effect.
${c.quota_waiting?'Waiting for model-service quota recovery · ':''}${number(c.published_episodes)} episode records published · Results and protocol ↗
`;
}
function methodology() {
return `Reading the Hard benchmarkProtocol, metrics & provenance+
Full-suite comparison and input-guided pilot
RoboTwin 2.0 Hard, ALOHA AgileX, demo_randomized. The five original conditions cover 50 tasks each. Earlier All 140k autonomous and input-required results cover 50 RLE tasks, retaining their first ten results. The VLAct Clean 50k comparison adds 100 fresh episodes: 50 tasks in each of its two protocols. Input-guided retains its first 10 RLE tasks. Completion and denominators are shown separately. Successes and valid failures are both retained; no completed, protocol-valid agent outcome was retried. This is RoboTwin 2.0 Hard, not RoboTwin 2.0-Plus.
Instruction protocols
Pure Kinex and all Kinex + VLA conditions use the revised RLE Hard templates and public scene bindings. VLA checkpoints and archived Kinex results retain original upstream unseen instructions. Each episode records the exact measured instruction and its authorship policy.
Checkpoint identity
Base 40k: StarVLA/Qwen3-VL-OFT-Robotwin2. VLAct Clean 50k: StarVLA/VLAct_Qwen3OFT_Robotwin_Finetune. VLAct All 100k: StarVLA/VLAct_Qwen3OFT_Robotwin_all_Finetune. Non-VLAct All 140k: StarVLA/Qwen3-VL-OFT-RoboTwin2-All, trained on 27,500 clean and randomized demonstrations. Full revisions and weight checksums are linked below. Pure Kinex uses GPT-6 Astra / high. The three earlier Kinex + VLA protocols use All 140k. The new autonomous and input-required conditions use VLAct Clean 50k, with 50 fresh Hard episodes per condition, 32-action chunks and the released wrap decoder. In the retained All 140k study, autonomous left input selection unguided, input-guided added optional context suggestions, and input-required used five JSON fields with nullable targets. The new VLAct Clean 50k autonomous condition leaves caller text unconstrained. Its input-required condition uses numeric-target-text-v1: instruction and one or two numerical TCP/joint targets, with optional feedback. Targets include an observation timestamp, source and uncertainty; desired joint targets may be derived by IK. Null or qualitative targets are rejected. Validation checks syntax, not grounding, reachability or learned compliance. The full JSON enters the frozen model as language text without a trained numerical encoder. Both modes leave VLA invocation and action execution to Kinex; zero VLA calls are valid. These are distinct protocols, and historical records retain their original descriptions.
Execution and seeds
Standalone VLA uses official expert admission from scene seed 100000 and official take_action with full predicted chunks. Current pure Kinex uses rosbridge at 25 Hz and the authored RLE manifest seeds and overrides, frozen before model use. No seed substitution or expert-success filtering is performed during this campaign. Seeds, admission rules and physical durations are not matched across the interfaces; this is not a paired causal comparison. Per-episode seeds and budgets are retained. In the superseded protocol study, two preliminary guided attempts were native successes but are excluded because a profile serialization defect corrupted their interface; their diagnostics are retained in the protocol report. Two further attempts were interrupted by provider quota exhaustion (HTTP 429) and restarted with unchanged seeds and budgets after quota recovery; both interruption records are preserved. The Kinex + VLA protocols share RLE instructions, authored seeds, budgets and the nominal 60 ms data clock resampled to 40 ms controls; prefixes and other control choices belong to the agent. Historical standalone results use native TOPP and are not an execution-matched baseline for this comparison.
Initialization provenance
Five infrastructure startup attempts produced no completed model response and are excluded from the original 200 valid outcomes. For hanging_mug, a minimal runtime repair retains native language bindings if the expert path fails before finishing. Selected seed 0, task physics and the native verifier are preserved.
Usage and costs
Estimated API-equivalent costs use recorded Pi session values. Actual account charges and GPU costs are not measured. Standalone VLA has zero language-agent API calls, not zero compute cost.
`;
}
const expandedTaskVariants = new Set();
const visibleTaskVariants = () => data.variants.filter(v => expandedTaskVariants.has(v.id));
function taskColumnPicker() {
return `
Choose methods to compare
All method columns are collapsed. Select one or more to compare.
${data.variants.map(v=>``).join('')}
`;
}
function taskColumnHeaders() {
return '
Task
' + visibleTaskVariants().map(v=>`
${esc(methodLabel(v))}
${esc(v.model)} · ${esc(v.effort)}Native result · Estimated USD
`).join('');
}
function syncTaskColumns() {
if(current.type==='runs')return;
const variants=visibleTaskVariants(), table=$('#task-comparison');
table.querySelector('thead tr').innerHTML=taskColumnHeaders();
table.classList.toggle('has-methods',variants.length>0);
table.style.setProperty('--method-count',variants.length);
$$('[data-task-column].task-column-toggle').forEach(button=>{
const expanded=expandedTaskVariants.has(button.dataset.taskColumn);
button.setAttribute('aria-expanded',expanded);
button.querySelector('.column-toggle-icon').textContent=expanded?'−':'+';
});
$('#task-column-status').textContent=variants.length?`${variants.length} of ${data.variants.length} method columns expanded. Select another method to compare.`:'All method columns are collapsed. Select one or more to compare.';
$('[data-task-columns="none"]').disabled=variants.length===0;
$('[data-task-columns="all"]').disabled=variants.length===data.variants.length;
}
function taskBrowser(variant=null) {
return `
`;
$('#task-count').textContent=`${tasks.length} of ${data.tasks.length} tasks`;
$('#filter-context').textContent=variant?variantName(variant):'Outcome filter & usage/cost sort: '+variantName(filters.variant);
}
function researchRerunNote() {
const r=data.research_update;if(!r)return '';
return ``;
}
function overview() {
current={type:'overview'};
$('#main').innerHTML=`
✦ RLE-Bench · Embodied agent evaluation
RoboTwin 2.0. Hard evaluation.
demo_randomized · one episode per selected task. Kinex and the paired Kinex + VLA study use RLE revised instructions; four standalone VLA checkpoints retain native instructions.
${data.families.length}environment
${data.tasks.length}distinct tasks
${data.variants.length}agent–model variants
${data.benchmark_complete?'Evaluation complete':'Evaluation in progress'} · selected seeds
`;
}
// Align words and punctuation while preserving the exact original whitespace.
function instructionDiff(original, revised) {
const split = text => text.match(/\s+|[\p{L}\p{N}_]+|[^\s]/gu) || [];
const a=split(original), b=split(revised);
const lcs=Array.from({length:a.length+1},()=>new Uint16Array(b.length+1));
for(let i=a.length-1;i>=0;i--)for(let j=b.length-1;j>=0;j--)
lcs[i][j]=a[i]===b[j]?lcs[i+1][j+1]+1:Math.max(lcs[i+1][j],lcs[i][j+1]);
const before=[],after=[];
const append=(list,text,changed)=>{const last=list.at(-1);if(last&&last.changed===changed)last.text+=text;else list.push({text,changed});};
let i=0,j=0;
while(i=lcs[i+1][j]))append(after,b[j++],true);
else append(before,a[i++],true);
}
const html=(parts,kind)=>parts.map(p=>p.changed&&p.text.trim()?`<${kind==='removed'?'del':'mark'} class="instruction-${kind}">${esc(p.text)}`:esc(p.text)).join('');
return {original:html(before,'removed'),revised:html(after,'added')};
}
function instructionPair(task, ep) {
const candidate = ep ? measuredInstructions.episodes[ep.id] : null;
const record = candidate && candidate.initial_goal === ep.instruction && candidate.task_key === task.key ? candidate : null;
const recordedNative = record?.observed_native || (ep ? ep.native_instruction : task.native_instruction);
const measured = record?.text || (ep ? (ep.instruction || (ep.instruction_policy === 'original_native' ? recordedNative : null)) : null);
const reference = catalogInstructionAudit.tasks.find(item => item.published_task_key === task.key && item.id === task.native_id);
const nativeReference = !recordedNative && reference?.native ? reference : null;
const native = recordedNative || nativeReference?.native;
return {record, native, measured, nativeReference};
}
function instructionPanel(task,ep) {
const {record,native,measured:recorded,nativeReference}=instructionPair(task,ep);
const measured=ep?recorded:(task.catalog_instruction_by_variant?.[current.variant]||task.catalog_instruction);
const initial=record&&record.initial_goal!==measured?record.initial_goal:null;
const changed=Boolean(native&&measured&&native!==measured);
const diff=changed?instructionDiff(native,measured):null;
const choices=[['measured',ep?'Measured':'Planned',measured],['native','Original native',native],['current','Current catalog',null]].filter(([key,,text])=>text&&(key!=='current'||(ep&&text!==measured&&text!==initial)));
const currentChanged=native&&task.catalog_instruction&&native!==task.catalog_instruction;
const description=!ep?'The current catalog instruction for the planned evaluation.':record?.kind==='scene_bound_goal'?"The measured goal with this episode's objects and targets filled in from the agent's session.":record?.kind==='native_received'?'The native instruction received by the agent during this episode.':'The task instruction supplied for this episode.';
return `
Task instruction
${description}
${choices.map(([k,label],i)=>``).join('')}
${nativeReference?`
Native comparison uses the pinned upstream full_description reference${ep?'; native text was not recorded in this episode':''}. ${external(nativeReference.source_url,'Native source')}
`:''}${choices.map(([k,,text],i)=>{
const comparison=k==='current'?(currentChanged?instructionDiff(native,text):null):diff;
const html=comparison?(k==='native'?comparison.original:comparison.revised):esc(text);
const note=comparison?(k==='native'?'Struck-through text was removed or replaced in the '+(ep?'measured':'planned')+' instruction.':'Highlighted text was added or replaced relative to the original native instruction.'):(native?'No changes from the original native instruction.':'Original native text is unavailable for comparison.');
return `
${comparison?``:''}${note}${k==='current'?' This catalog text is not the measured episode instruction.':''}
${html}
`;
}).join('')}${record&&initial?`
${record.kind==='scene_bound_goal'?'Object names are resolved for display; the measured completion requirements are preserved.':'The initial input contained a generic placeholder; this native instruction was received in the recorded session.'} Original instruction record ↗ · Session evidence ↗
Original harness verdict: invalid. Recording manifest: incomplete. This attempt is excluded from formal success metrics; its usage is retained separately.
${seedLimit?.exhausted?'No native-valid seed at or below '+number(seedLimit.max_seed):initializationBlocked?'Native seed 0 could not initialize':'No recorded episode yet'}
${seedLimit?.exhausted?'Native checks exhausted seeds 1–'+number(seedLimit.max_seed)+'. No Codex model attempt was launched. The original Kinex result is retained; this unavailable Codex task is excluded from success rates.':initializationBlocked?'No model was called. The authored seed could not initialize; a seed-selection decision is pending. This case is excluded from native success rates.':'This task is included in the planned catalog. Results and usage will appear after evaluation.'}
${m.playback_kind==='official_action_index'?'Head + left wrist + right wrist · 4× official recording, not simulation time. Wrist views update only at inference boundaries. Native 320 × 240 views are upscaled for display; action-limit recordings omit the post-final-action frame.':ep.recording_mode==='native'?'Native camera recording · cached agent-resolution frames · '+(m.speed||1)+'× simulation speed.':ep.provider==='robobridge'&&m.playback_kind==='live_spectator'?'Independent spectator cameras · 720 px high per view · overview + wrist views · '+(m.speed||1)+'× simulation speed. Agent images retain native resolution and orientation.':m.playback_kind==='native_observation_snapshots'?'Native observation snapshots · 10 FPS playback; this is not simulation time.':`${m.source_fps?m.source_fps+' FPS source capture · ':''}${m.speed||1}× simulation speed.`} Session timestamps use wall time and are not synchronized to video frames.
No standalone tool or memo files were saved in this episode. The original session and workspace evidence remain available.
'}
${evidenceCard('Execution evidence','Original verdict, protocol and runtime identity.',[['verdict','Native verdict'],['protocol','Protocol'],['provenance','Source identity']],ep)}${evidenceCard('Session & interactions','Original tool calls, observations and edited workspace.',[['native_session','Native session'],['trace','Trace'],['workspace_changes','Workspace changes']],ep)}
`;
}
async function fetchText(url) {
if(fileCache.has(url))return fileCache.get(url);
const r=await fetch(url);if(!r.ok)throw Error('HTTP '+r.status);
if(Number(r.headers.get('content-length'))>3e6)throw Error('File exceeds the 3 MB preview limit; use Open file');
const text=await r.text();if(text.length>3e6)throw Error('File exceeds the preview limit; use Open file');
fileCache.set(url,text);if(fileCache.size>20)fileCache.delete(fileCache.keys().next().value);return text;
}
async function loadResource(index,token) {
const r=current.ep?.resources?.[index];if(!r)return;
current.resource=index;$('#resource-name').textContent=r.name;$('#resource-open').href=r.file;$('#resource-kind').textContent=r.kind||'Final workspace snapshot';$('#resource-preview').textContent='Loading file…';
$$('[data-resource]').forEach(b=>{b.classList.toggle('active',+b.dataset.resource===index);b.setAttribute('aria-pressed',+b.dataset.resource===index);});
try{const text=await fetchText(r.file);if(token===epoch&¤t.resource===index){$('#resource-preview').textContent=text;current.resourceText=text;}}
catch(e){if(token===epoch&¤t.resource===index){$('#resource-preview').textContent='Preview unavailable: '+e.message;current.resourceText=null;}}
}
const evidenceGroups=[
['Native evaluation','Owner verdicts, measured task and control protocol.',[['verdict','Native verdict'],['timing','Episode timing'],['protocol','Measured protocol'],['native_goal','Task instructions'],['owner_journal','Owner action journal']]],
['Session & workspace','The original agent trajectory and final files.',[['native_session','Native session'],['transcript','Readable transcript'],['trace','Agent trace'],['workspace','Workspace archive'],['workspace_changes','Workspace manifest']]],
['Recording & review','Published media and checks against the recorded evidence.',[['native_video','Original head recording'],['video','Video'],['poster','Poster'],['terminal_frame','Terminal frame'],['recording_manifest','Recording manifest'],['media_check','Media validation'],['analysis','Episode analysis']]],
['Usage & provenance','Provider counters and the actual code used by the episode.',[['usage','Token usage'],['provider_usage','Provider usage ledger'],['provenance','Source identities'],['tool_errors','Tool errors']]]
];
function evidenceCard(title,description,items,ep) {
return `
${esc(record.contract.image_preprocessing||"Model images follow the recorded checkpoint preprocessing.")} Additional information reaches the model only when present in the request and rendered language below.
${jsonBlock('Complete model-service request',c.request)}${c.model_input_text!==undefined?jsonBlock('Language passed to predict_action · recorded',c.model_input_text,true):''}${c.backbone_messages?jsonBlock('Backbone message · reconstructed from executed code',c.backbone_messages):''}
Model-service output
${response?.waypoints?'Untimed joint waypoints consumed by the recorded caller. Kinex or agent-authored code determines their physical timing.':'The recorded action response consumed by the caller.'} Joint values are absolute radians; grippers use −1 open, +1 closed.
${jsonBlock('Recorded model output · full precision',c.service_response)}${c.execution?jsonBlock('Actual execution · target count and world time',{executor:c.planner,...c.execution},true):''}${jsonBlock('Internal diagnostic · denormalized training-order actions',c.internal_diagnostics.denormalized_actions)}${caller?`
Kinex tool call & result
This inference is joined to the recorded tool call and result below. Agent-authored code may consume the model output directly; the language agent receives the displayed tool response.
${jsonBlock('Agent tool call',caller.tool_call,true)}${jsonBlock('Tool result received by the language agent',{...caller.tool_result,parts:textParts(caller.tool_result.parts)},true)}
${c.executed_tool_source?jsonBlock('Agent-authored code at this call',c.executed_tool_source):''}`:'
An exact caller-to-inference join is unavailable. Inspect the recorded Kinex session and authored code.
'}Recording coverage & model contract
See the per-run contract below for recorded and reconstructed fields. Original requests and rendered language are retained in the current rich-input run; older runs have different recording coverage.
${jsonBlock('Complete recorded StarVLA response · full precision',c.raw_response)}${jsonBlock('Reordered native actions · full precision',c.service_response.actions)}
Standalone execution; no language-agent session or tool call. Token and API costs cover the language agent only; local GPU cost is not reported.
${jsonBlock('Recording coverage & model contract',r.contract)}
`;
function stats(es) {
return UsageMetrics.stats(es);
}
window.benchmarkStats = stats;
const tokens = e => e && Number.isFinite(e.usage?.input_tokens) && Number.isFinite(e.usage?.output_tokens) ? e.usage.input_tokens+e.usage.output_tokens : null;
function metric(value,label,detail='',attrs='') {return `