Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies
Paper • 2610.10479 • Published • 1
r2s baseline: Agentic RSR style (GR00T N1.5 + pi0.5 students)
Baseline students for a real-to-sim-to-real comparison. The teacher was made in the style of Agentic RSR (arXiv 2610.10479); the students are VLA policies (GR00T N1.5 and pi0.5) trained on that teacher's demonstrations. This is a re-implementation of the method's pipeline inside our framework, not the original code.
What was done (per task)
- Input (same as our method): one teleoperated demonstration + the camera / scene calibration.
- Simulator. Our Sim R0 as built (protocol 261107,
sim_initseed 0: the pseudo world plus a uniform perturbation); no sim tuning. - Teacher (three LLM passes). A Python program
solve(view) -> Plan, developed in three passes: privileged view on the nominal scene, visual (noisy estimate) view on the nominal scene, visual view on 8 randomized scenes (+-5 mm / +-3 deg object poses, joint offsets). Each pass gets per-scene feedback (checker outcome, demo-filter reason). - Demos. The final program with the visual view on fresh scenes, rendered with the task's cameras; only rollouts that pass the checker and the demo filter are kept, up to 320 (at most 1280 scenes are tried).
- Students. One round, the demos above, 30k steps at batch 64 (the same budget as our method: 15k + 3 x 5k).
GR00T N1.5 (
nvidia/GR00T-N1.5-3B, lr 1e-4) and pi0.5 (openpi, lr 1e-5).
Evaluation (sim2sim protocol): set D = 96 fresh scenes in the pseudo world (the stand-in for reality, never seen by the method); for pnp_straw (r2s2r_data2 0000-0002) and banana (pnp_banana 0006/0007), the validation episodes' measured starts in the pseudo world and in the method's world. The validation episodes were not an input.
Results
| task | world dist (Sim R0 -> method) | sim step | teacher | demos | LLM $ | student | train | set D (pseudo) | val starts (pseudo) | val starts (method world) |
|---|---|---|---|---|---|---|---|---|---|---|
| straw | 0.306 -> 0.306 | Sim R0 as built | 0/1 / 0/1 / 0/8 | 0 / 1280 scenes | 15.98 | pi05 | no demos | - | - | - |
| straw | 0.306 -> 0.306 | Sim R0 as built | 0/1 / 0/1 / 0/8 | 0 / 1280 scenes | 15.98 | groot | no demos | - | - | - |
| banana_basket | 0.417 -> 0.417 | Sim R0 as built | 0/1 / 0/1 / 8/8 | 288 / 1280 scenes | 6.37 | pi05 | - | - | - | - |
| banana_basket | 0.417 -> 0.417 | Sim R0 as built | 0/1 / 0/1 / 8/8 | 288 / 1280 scenes | 6.37 | groot | 30000 st, 1.0 h | 65/96 (67.7%) | 5/16 (31.2%) | 10/16 (62.5%) |
| chilgyo | 0.417 -> 0.417 | Sim R0 as built | - | 0.00 | pi05 | pending | - | - | - | |
| chilgyo | 0.417 -> 0.417 | Sim R0 as built | - | 0.00 | groot | pending | - | - | - | |
| towel | 0.404 -> 0.404 | Sim R0 as built | 1/1 / 1/1 / 7/8 | 320 / 491 scenes | 4.06 | pi05 | - | - | - | - |
| towel | 0.404 -> 0.404 | Sim R0 as built | 1/1 / 1/1 / 7/8 | 320 / 491 scenes | 4.06 | groot | 30000 st, 0.6 h | 21/96 (21.9%) | - | - |
| air_fryer | 0.398 -> 0.398 | Sim R0 as built | 0/1 / 0/1 / 1/8 | 101 / 1280 scenes | 7.46 | pi05 | 30000 st, 0.1 h | - | - | - |
| air_fryer | 0.398 -> 0.398 | Sim R0 as built | 0/1 / 0/1 / 1/8 | 101 / 1280 scenes | 7.46 | groot | 30000 st, 1.9 h | 44/96 (45.8%) | - | - |
Layout
<task>/<student>/
pi05/:params/,assets/groot/:model/- every student also has
policy.json,baseline.json(the method record: the three passes, demos, LLM cost, the final program's source),world_final.json,train_time.jsonandeval/(set D and validation results, when present).