---
title: RoboFAC-7B
emoji: 🤖
colorFrom: blue
colorTo: gray
sdk: gradio
sdk_version: 6.30.0
app_file: app.py
short_description: Analyze robot manipulation videos for failures
python_version: "3.10"
startup_duration_timeout: 1h
---

# RoboFAC-7B — Robotic Failure Analysis & Correction

Demo of [`MINT-SJTU/RoboFAC-7B`](https://huggingface.co/MINT-SJTU/RoboFAC-7B), a 7B vision-language model fine-tuned from Qwen2.5-VL-7B-Instruct for analyzing robot-manipulation videos: task identification and planning, failure detection / identification / locating / explanation, and both high-level and low-level correction guidance.

Upload a video of a robot arm performing a task and ask a question about it. Following the authors' reference pipeline ([mint-sjtu/robofac](https://github.com/mint-sjtu/robofac), `evaluation/eval_utils.py`), the app samples one keyframe per second from the video, resizes each frame to 512×512, and feeds all frames to the model together with the question in a single user message.

## Example videos

The bundled example videos are test clips from the [`MINT-SJTU/RoboFAC-dataset`](https://huggingface.co/datasets/MINT-SJTU/RoboFAC-dataset) (MIT license):

- `examples/sim_microwave_spoon.mp4` — simulation, *MicrowaveTask* (put the spoon in the mug, then both in the microwave; the grasp fails).
- `examples/realworld_insert_cylinder.mp4` — real-world SO-100 arm, *InsertCylinder* (insert a cylinder into a shelf hole; the reach is misaligned).

The example questions are the authors' own test QA prompts from the dataset's `test_qa_sim` / `test_qa_realworld` annotations.

## Paper

[RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction](https://huggingface.co/papers/2505.12224) (arXiv:2505.12224).