Update README.md

README.md

@@ -5,4 +5,99 @@ datasets:
 base_model:
 - alibaba-pai/Wan2.1-Fun-14B-Control
 - alibaba-pai/Wan2.1-Fun-14B-InP
----
+tags:
+  - 4d-generation
+  - image-to-4d
+  - diffusion
+  - novel-view-synthesis
+  - point-trajectory
+language:
+  - en
+pipeline_tag: image-to-video
+metrics:
+  - VBench
+---
+
+
+# MoGe4D: Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation
+
+**Yanran Zhang, Ziyi Wang, Wenzhao Zheng†, Zheng Zhu, Jie Zhou, Jiwen Lu**
+
+Department of Automation, Tsinghua University · GigaAI
+
+[[Paper (arXiv)]](https://arxiv.org/abs/2512.05044) · [[Project Page]](https://ivg-yanranzhang.github.io/MoGe4D/) · [[GitHub]](https://github.com/Zhangyr2022/MoGe4D) · [[Dataset (TrajScene-60K)]](https://www.modelscope.cn/datasets/YanranZhang/TrajScene-60K)
+
+> Accepted to **ECCV 2026**.
+
+## 📄 Paper Summary
+
+Generating interactive and dynamic 4D scenes from a single static image is a core challenge. Existing methods decouple geometry from motion — either *generate-then-reconstruct* (geometric inconsistency) or *reconstruct-then-generate* (limited, externally-constrained motion) — causing spatiotemporal inconsistency and poor generalization.
+
+**MoGe4D** (Motion and Geometry-aware image-to-4D synthesis) is a geometry-conditioned framework that models a scene as **dense 4D point trajectories**. Starting from an initial geometric prior of the input image, it predicts future time-varying trajectories through a diffusion process, tightly coupling geometric modeling with motion generation. This yields 4D scenes with strong temporal coherence, geometry-aware consistency, and compelling novel-view synthesis.
+
+**Contributions:**
+- **TrajScene-60K** — 60K videos with dense 4D point trajectories (3M+ frames, ~12B 3D points).
+- **4D-STraG** — a diffusion trajectory generator with *depth-guided motion normalization* and a *Motion Perception Module (MPM)*.
+- **4D-ViSM** — a view-synthesis module rendering the 4D representation under arbitrary camera trajectories.
+
+## 🧱 Model Structure
+
+This repository releases the three trained components of MoGe4D:
+
+| Path | Size | Description |
+|---|---|---|
+| `4D-STraG/diffusion_pytorch_model.safetensors` | ~31.9 GiB | 4D Scene Trajectory Generator (diffusion model, built on Wan2.1-14B) |
+| `4D-ViSM/lora_diffusion_pytorch_model.safetensors` | ~1.36 GiB | 4D View Synthesis Module (LoRA adapter) |
+| `VAE/vae/pytorch_model.bin` | ~484 MiB | Motion-sensitive VAE for trajectory signals |
+| `VAE/{encoder,decoder}_prompt/pytorch_model.bin` | ~1–2 MiB | VAE prompt encoder/decoder |
+| `VAE/{optimizer.bin, scheduler.bin, random_states_0.pkl}` | — | Training states for the VAE (optional, for resuming training) |
+
+## 🛠️ Usage
+
+### 1. Set up the environment
+
+```bash
+git clone https://github.com/Zhangyr2022/MoGe4D.git
+cd MoGe4D
+conda create -n MoGe4D python=3.10 && conda activate MoGe4D
+conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
+pip install -r requirements.txt
+```
+
+Install third-party deps: [UniDepth](https://github.com/lpiccinelli-eth/UniDepth) and [diff-gaussian-rasterization](https://github.com/graphdeco-inria/diff-gaussian-rasterization).
+
+### 2. Download the checkpoints
+
+```bash
+huggingface-cli download Yanran21/MoGe4D --local-dir ./models --resume-download
+```
+
+(Also place the base backbones [Wan2.1-Fun-V1.1-14B-Control/InP](https://huggingface.co/alibaba-pai), [OmniMAE](https://dl.fbaipublicfiles.com/omnivore/omnimae_ckpts/vitb_pretrain.torch), and [UniDepth](https://huggingface.co/lpiccinelli/unidepth-v2-vitl14) under `./models`.)
+
+### 3. Inference
+
+```bash
+bash scripts/inference/infer.sh        # whole pipeline: image → 4D scene → multi-view videos
+```
+
+See the [GitHub README](https://github.com/Zhangyr2022/MoGe4D) for training scripts and details.
+
+## 📊 Results
+
+MoGe4D delivers superior geometric consistency, dynamic realism, and visual fidelity over decoupled approaches (e.g., generate-then-reconstruct with VGGT). Please refer to the paper for quantitative metrics and qualitative comparisons.
+
+## 📖 Citation
+
+```bibtex
+@inproceedings{zhang2026moge4d,
+  title={Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation},
+  author={Zhang, Yanran and Wang, Ziyi and Zheng, Wenzhao and Zhu, Zheng and Zhou, Jie and Lu, Jiwen},
+  booktitle={European Conference on Computer Vision (ECCV)},
+  year={2026}
+}
+```
+
+## 📧 Contact
+
+- Yanran Zhang — zhangyr21@mails.tsinghua.edu.cn
+