README.md

metadata

title: RoboFAC-7B
emoji: 🤖
colorFrom: blue
colorTo: gray
sdk: gradio
sdk_version: 6.30.0
app_file: app.py
short_description: Analyze robot manipulation videos for failures
python_version: '3.10'
startup_duration_timeout: 1h

RoboFAC-7B — Robotic Failure Analysis & Correction

Demo of MINT-SJTU/RoboFAC-7B, a 7B vision-language model fine-tuned from Qwen2.5-VL-7B-Instruct for analyzing robot-manipulation videos: task identification and planning, failure detection / identification / locating / explanation, and both high-level and low-level correction guidance.

Upload a video of a robot arm performing a task and ask a question about it. Following the authors' reference pipeline (mint-sjtu/robofac, evaluation/eval_utils.py), the app samples one keyframe per second from the video, resizes each frame to 512×512, and feeds all frames to the model together with the question in a single user message.

Example videos

The bundled example videos are test clips from the MINT-SJTU/RoboFAC-dataset (MIT license):

• examples/sim_microwave_spoon.mp4 — simulation, MicrowaveTask (put the spoon in the mug, then both in the microwave; the grasp fails).
• examples/realworld_insert_cylinder.mp4 — real-world SO-100 arm, InsertCylinder (insert a cylinder into a shelf hole; the reach is misaligned).

The example questions are the authors' own test QA prompts from the dataset's test_qa_sim / test_qa_realworld annotations.

Paper

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction (arXiv:2505.12224).