ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation
Abstract
Few-step autoregressive (AR) video diffusion enables low-latency streaming generation, but existing post-training methods predominantly rely on Distribution Matching Distillation (DMD), requiring both a large pretrained teacher and an online critic to estimate distributional discrepancies through diffusion scores. In this work, we ask whether this resource-intensive teacher--critic stack can be eliminated by post-training only the generator against a precomputed target distribution. Drawing inspiration from representation distribution matching (RDM) for one-step image generation, we systematically study its transfer to few-step causal video generation and identify three key barriers: a memory-intractable gradient path, a distinct video optimization regime, and representation distributions that underconstrain temporal dynamics. We introduce ViRDM, a teacher- and critic-free video post-training recipe that addresses these barriers sequentially. By coupling RDM with stochastically truncated clean-exit supervision, a lightweight VAE decoder, and staged vector--Jacobian products, ViRDM makes representation distribution matching memory-feasible for multi-step causal video rollouts. We further establish effective generated-population and initialization regimes for video RDM, and introduce lightweight dynamics regularization to compensate for the underconstrained temporal dynamics. ViRDM turns three-network distillation into generator-only post-training, reducing GPU memory use and training time while improving video quality. With only 20 generator updates, the recipe reaches 84.87 on the official VBench evaluation, outperforming the previous best few-step causal baseline by 0.36, while requiring 16 A100 GPU-hours. We additionally report exploratory results demonstrating the potential of the same recipe for lower causal sampling budget and for one-, two-, and four-step bidirectional generation.
Community
Hi everyone! 👋 We’re sharing ViRDM: A recipe for RDM-based few-step causal video post-training, along with the pitfalls and lessons learned (single-GPU training supported 💥)
Inspired by the impressive results of FD Loss and RDM in image generation, we explored whether the same idea could work for few-step causal video generation: matching a precomputed representation distribution to update only the generator, without the online diffusion teacher and learned critic commonly used in DMD-based methods such as Self Forcing and Causal Forcing.
The idea was straightforward, but getting it to work for video generation came with quite a few challenges 🥹
🔹 Making training fit in memory. Compared with one-step image generation, multi-step causal video generation has a much longer backward path, and backpropagating through the video VAE is particularly memory-intensive. We had to tackle OOM before anything else.
🔹 Rethinking population size and initialization. Image RDM uses very large generated populations and can start directly from a pretrained bidirectional model. For few-step causal video generation, we couldn’t simply carry over these choices.
🔹 Closing the motion dynamic gap. Even with video representation features, matching the feature distributions alone did not fully resolve the degraded dynamics issue.
After working through ablated experiments, we put the working configurations, practical lessons, and pitfalls into a recipe. Hopefully, this saves some time for the community in exploring RDM for video generation 🔧
Our final four-step causal recipe achieves better results than the DMD-based baselines evaluated in our paper, with substantially less post-training time and lower peak memory:
✅ 84.87 VBench Total
✅ Only 20 generator updates
✅ Approximately 2 hours of post-training on 8×A100.
The recipe also supports training on a single 80GB A100, a setup that is challenging for common DMD pipelines 💥
We’re releasing:
✔ Training and inference code
✔ Model weights and precomputed reference artifacts
✔ Reproduction configurations and evaluation scripts
✔ A single-80GB-GPU training launcher
🌐 Website & demos: https://neu-vi.github.io/ViRDM/
💻 GitHub: https://github.com/neu-vi/ViRDM
📄 Paper: https://arxiv.org/abs/2609.28923
Many thanks to the authors of FD Loss, RDM, Self Forcing, Causal Forcing, and the other foundational works we build on 🙏
Stars 🌟, PRs, issues, and discussions are all welcome. We’d love to hear from others experimenting with ViRDM!🤗
Get this paper in your agent:
hf papers read 2609.28923 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper