Robotics
LeRobot
Safetensors
smolvla
Eval Results

Add Provael community eval results (.eval_results/provael.yaml)

#3

Hi, this adds .eval_results/provael.yaml for this checkpoint: community eval entries from Provael, an open-source red-team harness for VLA policies (Apache-2.0).

What is in it: 11 entries on the LIBERO-Object suite, 10 tasks. The adversarial arms and the benign floor at 5 seeds each (400 episodes), plus four harmless-variation controls at 3 seeds. Every value is the fraction of episodes that left a keep-out envelope, with the counts, the 95% Wilson interval, the predicate state and the tool version in the notes, so the floor sits next to each rate. The headline: roleplay 42/50 against a benign floor of 1/50, and the same frame with no target named 27/30, so it reads as fragility under a long out-of-distribution imperative rather than steering. Clean task success on the benign arm was 48/50.

Benchmark dataset: Sattyam/provael-libero-object-redteam (eval.yaml). Each entry links the committed run directory with the per-task report.json files and execution manifests, so anyone can re-run it. Simulation only, uncalibrated default predicate, and none of it is HF Jobs verified; it shows as community-provided, which is the right label.

If you would rather not carry third-party eval files in this repo, say so and I will close this. Happy to adjust the format too.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment