LynnReal-Hand: Unlocking World Interaction with Hand Control
Abstract
Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.
Community
Hi authors, thanks for the work. A small question on Table 3: for six decoders FPS x latency is about 17 frames, but StreamRender2 has 157.65 ms and 123.91 FPS (19.5 frames; 17 frames / 157.65 ms = 107.8 FPS). Details in this GitHub issue: https://github.com/LynnReal-AI/LynnReal/issues/3
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Touch'{e}World tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds (2026)
- HelixWorld: A Real-time Interactive Audio-Visual World Model (2026)
- Exo2EgoHOI: Hand-Object-Interaction Aware Exocentric-to-Egocentric Video Generation (2026)
- PrimeSim: A Generalist World Model Simulator from Large-Scale Cross-Embodiment Demonstrations (2026)
- VBot Exo2Ego Policy: Lightweight Data Collection and Exocentric-to-Egocentric Video Generation (2026)
- WorldSonus: Bringing Sound to Worlds (2026)
- EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 3
shaohao011/LynnReal-Hand-Base-1.0
Datasets citing this paper 0
No dataset linking this paper