README.md

metadata

title: ProcObject-Qwen3-VL-4B Video QA
emoji: 🍳
colorFrom: red
colorTo: indigo
sdk: gradio
sdk_version: 6.31.0
app_file: app.py
short_description: Object-centric video QA with temporal grounding
python_version: '3.12'
startup_duration_timeout: 30m

ProcObject-Qwen3-VL-4B — Object-Centric Video QA & Temporal Grounding

Demo of BrightGuo/ProcObject-Qwen3-VL-4B, the fine-tuned model of ProcObject-10K: Benchmarking and Learning Object-Centric Procedural Understanding in Instructional Videos (NeurIPS 2026, Evaluations & Datasets Track).

Upload an instructional video clip and ask a question about an object — how its state evolves, what happened to it before a step, what went wrong with it, whether it is ready for the next step. The model answers and localizes the supporting moments in the clip, returning [[start_second, end_second], ...] evidence intervals.

How it works

• Frames: 48 frames sampled uniformly over the clip (the model's training frame budget), resized to 512×512.
• Prompt: each frame is sent as an image preceded by [Timestamp: <t>s] (seconds from the clip start), followed by the question — the exact protocol of the released benchmark harness.
• Decoding: greedy (temperature 0).
• Output: a JSON object {"answer": "...", "evidence": [[start, end], ...]}, parsed into an answer plus a highlighted timeline of the grounded evidence spans.

The five object-centric reasoning types

Type	Asks
Precondition Grounding	what state an object had to reach before a step
Object State Evolution	how an object's state changes across steps
Counterfactual Reasoning	what would happen to an object if a step were skipped or done differently
Mistake Recognition	what went wrong with an object, and why it is a mistake
Readiness Assessment	whether an object is ready for the next step, and which visible cues show it

Example clips

The bundled example clips come from the CaptainCook4D dataset (Apache-2.0 code license; videos distributed by its authors via the public downloader), one of the four sources of ProcObject-10K. They were cut to the exact benchmark clip windows used by the test split.

Paper

Benchmarking and Learning Object-Centric Procedural Understanding in Instructional Videos — Wenliang Guo, Yu Kong (Michigan State University). Code: github.com/WenliangGuo/ProcObject-10K; dataset: BrightGuo/ProcObject-10K.