A language instruction is compiled into a typed graph of atomic robot skills. Each skill runs only when the gate in front of it verifies. Nothing is a black box: when the robot fails, the graph says which atom failed and why.
Boxes are atomic skills. Diamonds are verification gates. The rule is the whole idea: a skill fires only after the preceding gate passes. Press play — then run the failing branch and watch execution leave the pipeline the moment a gate says no.
The two green nodes are the perception atoms — the most expensive part of the pipeline, and the one this project rebuilt (§03).
An end-to-end policy that fails gives you one bit of information: it failed. A gated graph gives you the node, the inputs it saw, and the branch it took. That turns debugging a robot into reading a trace.
The gates are what make the failure legible. A run that dies at found? stops after about 51 control
steps — the arm has not moved, because the graph refused to hand an unverified object pose to the motion planner. A
run that clears every gate executes roughly 750 steps. You can classify a trial by its step count alone.
The perception atom dominated cost. It proposed ~8 candidate boxes and ran a single-elimination tournament of vision-language comparisons — one model call per elimination, several times per task.
The fix is to rank candidates against the target name first, keep the top few, and skip the tournament entirely when the leader is unambiguous. Same interface, same gates, far less compute:
| Task family | calls / trial · before | calls / trial · after | reduction |
|---|---|---|---|
| Tabletop objects | 14.4 | 2.6 | 5.5× |
| Spatial referring | 16.3 | 1.9 | 8.6× |
| Goal-directed | 14.0 | 1.3 | 10.8× |
| Average | 14.9 | 1.9 | ≈8× |
| Task family (10 tasks × 3 trials each) | baseline | + pre-ranking |
|---|---|---|
| Tabletop pick-and-place | 28/30 = 93.3% | 29/30 = 96.7% |
| Spatial referring expressions | 3/30 = 10.0% | 1/30 = 3.3% |
| Goal-directed manipulation | 5/30 = 16.7% | 6/30 = 20.0% |
| All 90 trials | 36/90 = 40.0% | 36/90 = 40.0% |
Read the collapse outside pick-and-place carefully — the traces show three different causes, and only one is a real ceiling.
Opening a drawer, turning a knob, pushing a plate — these are not pick-and-place. Those tasks ran their full trials with zero perception aborts and the longest executions measured: 2680–3297 control steps against ~720 for a success. The graph was built, perception verified, and the arm genuinely tried. The manipulation atoms simply do not exist. That is a library boundary, and it is visible precisely because the gates all passed.
For "the bowl between the plate and the ramekin", graph generation emits the target as just
"black bowl" — and the scene contains two identical bowls. No downstream atom can recover the missing
words. Worse, confident pre-ranking then locks onto the salient bowl rather than the referred one, which
is why that column moves the wrong way. The fix under test scores candidates by relation geometry against
detected anchor objects, at zero extra model calls.
One graph is sampled per task. A single malformed sample zeroes all three trials — one observed failure was a graph whose error branch routed to a label its own edge map never declared, so it died before the arm moved at all. That is graph-generation variance, not robot capability, and it is reported separately rather than folded into the score.
The reference implementation reports LIBERO grocery success around ~0.95. Earlier reproduction
attempts fell short only because they ran an open perception backend. Point the perceive atom at
the framework's default perception model — Gemini-3.1-flash-lite, on Google's own endpoint — and the
unmodified baseline reproduces that level. We ran the entire thing on a free API key.
| Grocery task · 3 trials each | baseline | + pre-ranking | ours calls/trial |
|---|---|---|---|
| 0 · alphabet soup can | 2/3 | 2/3 | 3.3 |
| 1 · cream cheese box | 3/3 | 3/3 | 4.0 |
| 2 · dark salad-dressing bottle | 3/3 | 3/3 | 2.7 |
| 3 · red bbq sauce bottle | 3/3 | 3/3 | 2.0 |
| 4 · ketchup bottle | 3/3 | 3/3 | 2.0 |
| 5 · tomato sauce can | 2/3 | 3/3 | 1.3 |
| 6 · butter box | 3/3 | 3/3 | 2.0 |
| 7 · milk carton | 3/3 | 3/3 | 2.7 |
| 8 · chocolate pudding box | 3/3 | 3/3 | 3.3 |
| 9 · orange juice | 3/3 | 3/3 | 2.7 |
| Aggregate · all 10 tasks | 28/30 = 93.3% | 29/30 = 96.7% | 2.6 vs 14.4 base |
Same-scene A/B across the whole gate. On this strong backend the baseline already sits near the ceiling, so accuracy headroom is small — only task 5 moves. The win here is parity-or-better accuracy at ~5.5× fewer perception calls: it prunes ~8 detector candidates to 3 before the tournament, and skips the container comparison outright when the leader is unambiguous.
On a strong perception backend the gates rarely trip, so pre-ranking buys efficiency. Put the same graph on a deliberately constrained backend, where the perception atom genuinely cannot resolve the object, and the same change stops being an optimisation and starts being the difference between moving and not moving.
Why the step count is the proof. A baseline trial dies after ~51 steps: the graph
reached perceive, the found? gate refused it, and the arm never moved. With pre-ranking the
same task runs ~746 steps — the gate passes, so the arm actually grasps and places. Independent failure
attribution agrees: 100% of baseline failures (28/28) are perception, exactly the atom this targets.
Built on an open-source typed skill-graph framework; the perception pre-ranking, the measurements, and the failure attribution are our own. All numbers come from real benchmark executions on 2×RTX 4090 — none are estimated.