rooty2020/r2s-baseline-robosimstudio

r2s baseline: RoboSimStudio style (GR00T N1.5 + pi0.5 students)

Baseline students for a real-to-sim-to-real comparison. The teacher and the simulator were made in the style of RoboSimStudio (arXiv 2610.18835); the students are VLA policies (GR00T N1.5 and pi0.5) trained on that teacher's demonstrations. This is a re-implementation of the method's pipeline inside our framework, not the original RoboSimStudio code: the GaP runtime (graph-as-policy) is not used. The teacher is a Python program solve(view) -> Plan, improved with a GaP-style loop, and the policy that is evaluated is the student trained on the program's demos.

What was done (per task)

1. Input (same as our method): one teleoperated demonstration + the camera / scene calibration.
2. Simulator. Start from our Sim R0 (protocol 261107, sim_init seed 0: the pseudo world plus a uniform perturbation). One LLM sim-tuning pass on 16 reference clips (12 attempts, loss = the clips' object-track error), then three physics variants: tuned / low / high.
3. Teacher (GaP-style program improvement). One LLM program, executed on 30 layouts x 3 physics variants (90 scenes) per iteration, with per-scene feedback (checker outcome, demo-filter reason), at most 14 iterations, pass at >= 95 %.
4. Demos. The final program on fresh scenes (scene i in variant i % 3), rendered with the task's cameras; only rollouts that pass the checker and the demo filter are kept, up to 320.
5. Students. One round, 320 demos, 30k steps at batch 64 (the same budget as our method: 15k + 3 x 5k). GR00T N1.5 (nvidia/GR00T-N1.5-3B, lr 1e-4) and pi0.5 (openpi, lr 1e-5).

Evaluation (sim2sim protocol): set D = 96 fresh scenes in the pseudo world (the stand-in for reality, never seen by the method); for pnp_straw (r2s2r_data2 0000-0002) and banana (pnp_banana 0006/0007), the validation episodes' measured starts in the pseudo world and in the method's world. The validation episodes were not an input.

Results

task	world dist (Sim R0 -> method)	sim step	teacher	demos	LLM $	student	train	set D (pseudo)	val starts (pseudo)	val starts (method world)
straw	0.306 -> 0.344	tune loss 3.914 -> 3.613	GaP 4/90 (14 it, not passed)	-	0.00	pi05	pending	-	-	-
straw	0.306 -> 0.344	tune loss 3.914 -> 3.613	GaP 4/90 (14 it, not passed)	-	0.00	groot	pending	-	-	-
banana_basket	0.417 -> 0.298	tune loss 2.377 -> 0.279	GaP 88/90 (4 it, passed)	320 / 334 scenes	1.76	pi05	-	-	-	-
banana_basket	0.417 -> 0.298	tune loss 2.377 -> 0.279	GaP 88/90 (4 it, passed)	320 / 334 scenes	1.76	groot	30000 st, 2.3 h	91/96 (94.8%)	1/16 (6.2%)	0/16 (0.0%)
chilgyo	0.417 -> 0.320	tune loss 0.169 -> 0.055	GaP 88/90 (12 it, passed)	320 / 473 scenes	8.21	pi05	-	-	-	-
chilgyo	0.417 -> 0.320	tune loss 0.169 -> 0.055	GaP 88/90 (12 it, passed)	320 / 473 scenes	8.21	groot	30000 st, 1.1 h	7/96 (7.3%)	-	-
towel	0.404 -> 0.430	tune loss 4.544 -> 4.010	running	-	0.00	pi05	pending	-	-	-
towel	0.404 -> 0.430	tune loss 4.544 -> 4.010	running	-	0.00	groot	pending	-	-	-
air_fryer	0.398 -> 0.230	tune loss 6.029 -> 1.891	GaP 85/90 (14 it, not passed)	320 / 376 scenes	7.18	pi05	-	-	-	-
air_fryer	0.398 -> 0.230	tune loss 6.029 -> 1.891	GaP 85/90 (14 it, not passed)	320 / 376 scenes	7.18	groot	30000 st, 1.5 h	71/96 (74.0%)	-	-

Layout

<task>/<student>/

• pi05/: params/, assets/
• groot/: model/
• every student also has policy.json, baseline.json (the method record: sim tuning, GaP iterations, demos, LLM cost, the final program's source), world_final.json, train_time.json and eval/ (set D and validation results, when present).