: Live Humanoid Video Stream Generation with Streaming Multimodal References and Proactive Agent

Overview of FlowAct-R2. FlowAct-R2 enables proactive, lifelike humanoid video generation by coupling a two-stage proactive agent with a streaming multi-reference DiT. The agent pre-plans long-horizon agendas and reusable multimodal skills offline, and dynamically orchestrates online responses, tool execution, and real-time interruptions. Conditioned on rolling action prompts and streaming audio, the generative backbone incorporates video-driven RoPE and an explicitly trained (R+I)2V multi-reference mechanism—supporting dynamic image, audio, and video references to suppress cumulative drift and synthesize high-fidelity 720p avatar videos in real time.

Streaming Multimoal Reference DiT

Existing streaming video generation methods are typically adapted from image-to-video (I2V) or text-to-video (T2V) models, limiting their ability to incorporate dynamically evolving references and often requiring substantial architectural modifications for long-horizon consistency. However, interactive digital humans demand continuous multimodal control across diverse scenarios, such as e-commerce livestreaming, talent performances, and scene exploration.

We present FlowAct-R2, a streaming adaptation of Seedance 2.0 Mini designed for interactive digital humans. Rather than augmenting an I2V model with attention sinks, FlowAct-R2 builds directly on a pre-trained reference-to-video (R2V) backbone, inheriting its strong reference preservation capability. We employ Diffusion Forcing to enable autoregressive streaming inference while preserving the backbone and its bidirectional spatiotemporal attention. We further extend Diffusion Forcing to streaming multimodal reference generation, allowing image, audio, and video references, together with text prompts, to be continuously updated or switched during inference. A modified RoPE aligns incoming reference chunks with the corresponding generation timeline, ensuring temporally synchronized control.

The strong reference conditioning of the R2V backbone already substantially reduces autoregressive drift by continuously anchoring generation to the reference content. To further suppress residual error accumulation, we condition the model on partially noised historical motion frames rather than clean ones, improving its robustness to imperfect generated histories. Specialized audio-driven training enhances synchronization among speech, lip movements, expressions, and head poses, while coarse-to-fine inference progressively restores visual details. Together, these designs enable stable, hour-long, real-time 720p digital human generation under continuous multimodal control.

Proactive Interaction Agent

Proactivity in a streaming digital human extends beyond generating timely responses to user inputs: the digital human must continuously determine what to do next, even in the absence of explicit instructions, while maintaining a coherent persona and pursuing the long-term objectives of the live session. This creates a fundamental tension between long-horizon preparation and online adaptation. Purely online planning is often too myopic and latency-sensitive to organize an entire session or synthesize complex behaviors on demand, whereas a fixed offline script cannot accommodate unpredictable audience interactions or evolving contexts.

FlowAct-R2 therefore adopts a two-stage agent architecture. Offline, it derives a persona from multimodal character assets, organizes the session into a long-horizon agenda, and compiles complex behaviors into reusable skills. Online, a scheduling agent integrates the agenda, interaction events, and historical context to determine when to initiate, overlay, defer, interrupt, or resume behaviors, while invoking the appropriate skills for execution. This design enables the digital human to autonomously sustain the live session while remaining responsive to interactions.