README.md

---
license: apache-2.0
base_model: Wan-AI/Wan2.1-T2V-14B
pipeline_tag: video-to-video
tags:
  - video-diffusion
  - camera-control
  - novel-view-synthesis
---

# Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting

**[Paper](https://arxiv.org/abs/2608.28174)** | **[Project Page](https://yongxuqixiang.github.io/Manifold4D-Project-Page/)** | **[Code](https://github.com/ManifoldTechLtd/Manifold4D)**

Yongqi Mao, Zijia Dai, Zhishuo Liu, Wei Xu, Kaiwei Wang, Guotao Meng

Zhejiang University · Manifold Tech · ShanghaiTech University · University of Cambridge

## Model description

Manifold4D performs video re-shooting. It re-renders a monocular video of a dynamic scene along a user-specified camera trajectory.

A dominant paradigm lifts the source video into a 4D point cloud using per-frame depth, and rasterizes it along the target trajectory. The point-cloud render then serves as a geometric condition for trajectory-controlled generation, alongside the source video. These two signals can provide conflicting visual cues throughout denoising. We refer to this conflict as a trust dilemma. It can compromise trajectory control, 3D consistency of dynamic content, and visual quality.

Manifold4D instead injects the point-cloud render directly into the initial noise of flow matching. Generation no longer departs from standard Gaussian noise but from a geometry-bearing noise distribution that we term the point cloud rendered manifold. The render is used exactly once, and the source video remains the only visual condition during denoising.

On our benchmarks, Manifold4D attains the best camera-control accuracy on every metric. It lowers rotation error by 25-27% and translation error by up to 32% over the strongest baseline, while matching it in video fidelity and attaining the best overall photometric quality and 3D consistency of dynamic content.

## Checkpoint contents

This repository contains the fine-tuned weights of Manifold4D, built on [Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B):

| File | Size | Contents |
| --- | --- | --- |
| `self_attn_full.pt` | 8.39 GB | Fine-tuned self-attention projections (Q/K/V/O weights and biases, with query/key RMSNorms) of all 40 DiT blocks |
| `camera_encoder.pt` | 2.57 GB | Plücker camera encoder: a shared `plucker_share_encoder` and per-block `cam_encoders`, zero-initialized at training start |
| `conditioning_modules.pt` | 2.57 GB | Four patchify embeddings (`output_rgb`, `output_anchor`, `source_rgb`, `source_mask`) and the conditioning `projectors` |

Only the trainable modules are included. All other parameters stay frozen and come from the base model [Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B): feed-forward blocks, cross-attention, normalization layers, time and text embeddings, the output head, the 16-channel VAE, and the text encoder.

## Usage

Download the checkpoints:

```bash
hf download manifoldtech/Manifold4D
```

Loading and inference require the code at [ManifoldTechLtd/Manifold4D](https://github.com/ManifoldTechLtd/Manifold4D). The checkpoints are PyTorch pickles. Load them with `torch.load` following the instructions in the GitHub repository.

## Training details

- Base model: Wan2.1-T2V-14B with its native 16-channel VAE.
- Resolution: 49 frames at 384x672, mapped by the VAE to 13 latent frames.
- Data: about 36K clips from DL3DV, DynPose, OpenVid-HD, MultiCamVideo, and HuMMan. Metric depth and camera poses come from VGGT-Omega for every source except HuMMan, which ships calibrated multi-camera captures.
- Optimization: AdamW with a constant learning rate of 1e-5, global batch size 8, 30K steps.

## Citation

```bibtex
@article{mao2026manifold4d,
  title   = {Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting},
  author  = {Mao, Yongqi and Dai, Zijia and Liu, Zhishuo and Xu, Wei and Wang, Kaiwei and Meng, Guotao},
  journal = {arXiv preprint arXiv:2608.28174},
  year    = {2026}
}
```

## License

These weights are released under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. The base model [Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) is also released under Apache 2.0. Refer to its model card for details.