Beyond Answering: Learning to Speak First and Act in Streaming Video
Abstract
AURA 2.0 is a streaming video large language model that can initiate conversations, call tools, and perform interactive visual grounding, with long-horizon memory and a three-stage training scheme for improved streaming interaction.
Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) lets the model act on what it sees by invoking external tools using the current visual and conversational context and incorporating returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.
Community
Hi Hugging Face community! 👋 We’re excited to share AURA 2.0 — Beyond Answering: Learning to Speak First and Act in Streaming Video.
What if a video AI could notice something worth discussing and start the conversation, rather than always waiting for a question?
AURA 2.0 is an open-source streaming video understanding model that brings together three capabilities:
- Model-Initiated Conversation (MIC): Decide whether to speak, when to speak, and what to say based on the evolving video and dialogue context—including when to remain silent.
- Stream-Aware Tool Calling (SATC): Select tools and construct arguments from the current visual context, then incorporate tool results into the conversation.
- Interactive Visual Grounding (IVG): Connect dialogue to specific objects through user-provided visual cues and model-generated, frame-aligned bounding boxes.
The model is trained on approximately 5,400 hours of video and 40 million frames, with three-stage training and long-horizon interaction memory. It achieves the highest reported aggregate scores across IPIBench, StreamArena, ProactiveVideoQA, and PhoStream among the models compared in our report.
Explore the demos and find code, model downloads, and deployment instructions on our project page.
We welcome feedback, reproductions, and discussion—especially on how proactive video assistants should balance useful intervention with knowing when to wait!
Get this paper in your agent:
hf papers read 2611.17048 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 2
aurateam/AURA_2_BF8
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper