Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting

Paper | Project Page | Code

Yongqi Mao, Zijia Dai, Zhishuo Liu, Wei Xu, Kaiwei Wang, Guotao Meng

Zhejiang University · Manifold Tech · ShanghaiTech University · University of Cambridge

Model description

Manifold4D performs video re-shooting. It re-renders a monocular video of a dynamic scene along a user-specified camera trajectory.

A dominant paradigm lifts the source video into a 4D point cloud using per-frame depth, and rasterizes it along the target trajectory. The point-cloud render then serves as a geometric condition for trajectory-controlled generation, alongside the source video. These two signals can provide conflicting visual cues throughout denoising. We refer to this conflict as a trust dilemma. It can compromise trajectory control, 3D consistency of dynamic content, and visual quality.

Manifold4D instead injects the point-cloud render directly into the initial noise of flow matching. Generation no longer departs from standard Gaussian noise but from a geometry-bearing noise distribution that we term the point cloud rendered manifold. The render is used exactly once, and the source video remains the only visual condition during denoising.

On our benchmarks, Manifold4D attains the best camera-control accuracy on every metric. It lowers rotation error by 25-27% and translation error by up to 32% over the strongest baseline, while matching it in video fidelity and attaining the best overall photometric quality and 3D consistency of dynamic content.

Checkpoint contents

This repository contains the fine-tuned weights of Manifold4D, built on Wan2.1-T2V-14B:

File	Size	Contents
self_attn_full.pt	8.39 GB	Fine-tuned self-attention projections (Q/K/V/O weights and biases, with query/key RMSNorms) of all 40 DiT blocks
camera_encoder.pt	2.57 GB	Plücker camera encoder: a shared plucker_share_encoder and per-block cam_encoders, zero-initialized at training start
conditioning_modules.pt	2.57 GB	Four patchify embeddings (output_rgb, output_anchor, source_rgb, source_mask) and the conditioning projectors

Only the trainable modules are included. All other parameters stay frozen and come from the base model Wan-AI/Wan2.1-T2V-14B: feed-forward blocks, cross-attention, normalization layers, time and text embeddings, the output head, the 16-channel VAE, and the text encoder.

Usage

Download the checkpoints:

hf download manifoldtech/Manifold4D

Loading and inference require the code at ManifoldTechLtd/Manifold4D. The checkpoints are PyTorch pickles. Load them with torch.load following the instructions in the GitHub repository.

Training details

• Base model: Wan2.1-T2V-14B with its native 16-channel VAE.
• Resolution: 49 frames at 384x672, mapped by the VAE to 13 latent frames.
• Data: about 36K clips from DL3DV, DynPose, OpenVid-HD, MultiCamVideo, and HuMMan. Metric depth and camera poses come from VGGT-Omega for every source except HuMMan, which ships calibrated multi-camera captures.
• Optimization: AdamW with a constant learning rate of 1e-5, global batch size 8, 30K steps.

Citation

@article{mao2026manifold4d,
  title   = {Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting},
  author  = {Mao, Yongqi and Dai, Zijia and Liu, Zhishuo and Xu, Wei and Wang, Kaiwei and Meng, Guotao},
  journal = {arXiv preprint arXiv:2608.28174},
  year    = {2026}
}

License

These weights are released under the Apache 2.0 license. The base model Wan2.1-T2V-14B is also released under Apache 2.0. Refer to its model card for details.