LIBERO-Plus split by perturbation type for the three backbones?
Hi Yonghao, thanks for shipping all three checkpoints with release.py verify; that makes them easy to check.
The LIBERO-Plus table caught my eye: Wan averages 64.54, and Qwen3.5-2B gets 81.14 on the same recipe. Do you have the split by perturbation type? I'm mostly curious whether the gap sits in the language perturbations or in camera and lighting.
I ask because I red-team robot policies in simulation (Provael). On SmolVLA with LIBERO Object, a roleplay frame around the instruction made it leave the safe envelope in 42 of 50 matched pairs, against 1 of 50 for the benign control. So I measured the language side.
If the split exists, could you post it here?
Hi Sattyam, thanks for taking a look! Here’s the breakdown of success rates (%):
| Perturbation | JAM (Wan) | JAM (Ovis-U1) | JAM (Qwen3.5-2B) |
|---|---|---|---|
| Camera viewpoints | 25.4 | 83.8 | 80.8 |
| Robot initial states | 68.0 | 49.2 | 61.4 |
| Language instructions | 92.6 | 84.8 | 71.7 |
| Lighting | 93.2 | 95.7 | 93.8 |
| Background textures | 46.2 | 80.1 | 81.7 |
| Sensor noise | 52.7 | 91.9 | 94.2 |
| Object layouts | 81.4 | 72.3 | 84.5 |
| Overall | 64.5 | 79.9 | 81.1 |
All three were evaluated on the same 1,506-task subset, with one rollout per task. Overall is weighted by task count.
Qwen’s advantage over Wan mainly comes from camera viewpoints, backgrounds, and sensor noise. Wan does better on language perturbations, while lighting scores are similar. These are standard LIBERO-Plus task-success results, so they don’t directly measure robustness to the roleplay prompts you described.