Papers
arxiv:2611.25179

Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks

Published on Nov 18
ยท Submitted by
Zirui Song
on Nov 20
Authors:
,
,
,
,
,

Abstract

An LLM-based pipeline audits simulated manipulation success checkers by recreating disagreement scenarios and visually verifying whether observed behavior satisfies task instructions, revealing phantom successes and silent failures.

Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.

Community

Paper submitter

Are your robot policies really succeeding, or is the success checker lying? ๐Ÿค–

Simulated manipulation benchmarks rely on hand-written success checkers, which are usually validated only against expert demos. Learned policies behave differently, so checkers can:

  • โœ… accept incomplete executions โ†’ phantom success
  • โŒ reject executions that actually follow the instruction โ†’ silent failure

We introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. It reads the instruction and checker code, proposes states where they may disagree, recreates them in simulation, and verifies the outcome with multi-view renderings.

๐Ÿ” Key finding: checker errors appear across popular benchmarks (e.g., RoboTwin 2.0 errs in both directions), so reported success rates may not mean what we think.

Feedback welcome, especially from people building or using manipulation benchmarks!

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.25179
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.25179 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.25179 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.25179 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.