Spaces:
Sleeping
Sleeping
| title: RoboFAC-7B | |
| emoji: 🤖 | |
| colorFrom: blue | |
| colorTo: gray | |
| sdk: gradio | |
| sdk_version: 6.30.0 | |
| app_file: app.py | |
| short_description: Analyze robot manipulation videos for failures | |
| python_version: "3.10" | |
| startup_duration_timeout: 1h | |
| # RoboFAC-7B — Robotic Failure Analysis & Correction | |
| Demo of [`MINT-SJTU/RoboFAC-7B`](https://huggingface.co/MINT-SJTU/RoboFAC-7B), a 7B vision-language model fine-tuned from Qwen2.5-VL-7B-Instruct for analyzing robot-manipulation videos: task identification and planning, failure detection / identification / locating / explanation, and both high-level and low-level correction guidance. | |
| Upload a video of a robot arm performing a task and ask a question about it. Following the authors' reference pipeline ([mint-sjtu/robofac](https://github.com/mint-sjtu/robofac), `evaluation/eval_utils.py`), the app samples one keyframe per second from the video, resizes each frame to 512×512, and feeds all frames to the model together with the question in a single user message. | |
| ## Example videos | |
| The bundled example videos are test clips from the [`MINT-SJTU/RoboFAC-dataset`](https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset) (MIT license): | |
| - `examples/sim_microwave_spoon.mp4` — simulation, *MicrowaveTask* (put the spoon in the mug, then both in the microwave; the grasp fails). | |
| - `examples/realworld_insert_cylinder.mp4` — real-world SO-100 arm, *InsertCylinder* (insert a cylinder into a shelf hole; the reach is misaligned). | |
| The example questions are the authors' own test QA prompts from the dataset's `test_qa_sim` / `test_qa_realworld` annotations. | |
| ## Paper | |
| [RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction](https://huggingface.co/papers/2505.12224) (arXiv:2505.12224). |