--- license: cc-by-nc-4.0 pipeline_tag: text-to-3d tags: - 4d-generation - gaussian-splatting - 4d-gaussian-splatting - world-model - video-diffusion - image-to-3d - wan2.1 - neurips-2026 ---
# AEON: Unifying Video and 3D World Models ### NeurIPS 2026 [Weiqi Zhang](https://weiqi-zhang.github.io/), [Wenyuan Zhang](https://wen-yuan-zhang.github.io/), [Yu-Shen Liu](https://yushen-liu.github.io/)†, [Junsheng Zhou](https://junshengzhou.github.io/)† School of Software, Tsinghua University  ·  † Corresponding authors [**Paper**](https://huggingface.co/papers/2610.28631) | [**Project Page**](https://weiqi-zhang.github.io/AEON/) | [**Code**](https://github.com/weiqi-zhang/AEON)
![Four generated 4DGS worlds rendered at different timesteps and viewpoints with depth and normals](https://weiqi-zhang.github.io/AEON/static/img/teaser.webp) AEON ([paper](https://huggingface.co/papers/2610.28631)) is a generative world model that unifies video-based and 3D-based world models. It works in the latent space of a pretrained video diffusion model and produces an explicit **4D Gaussian Splatting (4DGS)** world together with its camera trajectory. The 4DGS world can be rendered in real time at any timestep and from new viewpoints, and it can be edited or used for simulation. This repository hosts the weights of the **AEON decoder**. The inference code is on [GitHub](https://github.com/weiqi-zhang/AEON), and more generated worlds, with videos, are on the [project page](https://weiqi-zhang.github.io/AEON/#worlds). ## Model details | | | |---|---| | File | `aeon.safetensors`: 3.26 GB, float32, about 812M parameters | | Components | domain adapter, a Conv3D + Perceiver bridge (`stitching_layer.*`), and student 4D Gaussian Reconstruction Model, 4DRM (`stitched_4d_model.*`) | | 4DRM backbone | DINOv2-Large encoder and an alternating-attention multi-view transformer, adapted from MapAnything | | Input | a Wan2.1 video latent, and the frames the Wan VAE decodes from it | | Output | per-pixel 4D Gaussians and per-frame cameras | | Video models | [Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers) for text-to-4DGS, [Wan2.1-I2V-14B-480P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P-Diffusers) for image-to-4DGS | The Wan2.1 models stay frozen and are not included in this repository. Both share the same VAE, so the same decoder serves both pipelines. ## Method ![Overview of AEON: the 4D Gaussian Reconstruction Model on top, generative world modeling with a domain adapter and a student 4DRM below](https://weiqi-zhang.github.io/AEON/static/img/method.webp) A 4D Gaussian Reconstruction Model (4DRM) first learns to recover a 4DGS world from monocular video. AEON then connects a frozen video diffusion model to this reconstruction model through a domain adapter, which translates generative video latents into reconstructive 4D latents. The adapter and a student 4DRM are trained by distillation, so a generated video latent can be decoded directly into 4D Gaussians. At inference time: 1. Wan2.1 generates a video latent from the prompt (and the image, for I2V). The Wan VAE decodes it into frames. 2. The domain adapter and the student 4DRM map the latent to per-pixel 4D Gaussians and per-frame cameras. 3. The Gaussians are rasterized with gsplat along the predicted cameras and, optionally, along a spiral novel-view path. ## Usage Requirements: Python 3.10 or newer, a CUDA GPU, PyTorch built for your CUDA version, and the CUDA toolkit (`nvcc`) and `ninja` on `PATH`. gsplat compiles its CUDA kernels the first time it runs. ```bash git clone https://github.com/weiqi-zhang/AEON.git cd AEON pip install -r requirements.txt pip install -e . # AEON decoder -> weights/aeon.safetensors, where the default config looks for it huggingface-cli download wiki1231/AEON aeon.safetensors --local-dir weights # Text -> 4DGS, plus a spiral novel-view video and an export of the Gaussians aeon-generate --prompt "A red fox running through deep snow, slow motion" --spiral --save_4dgs # Image + text -> 4DGS with the 14B model (--cpu_offload keeps GPU memory around 50 GB) aeon-generate --pipeline i2v --image ref.png --prompt "The cat walks to the window" --cpu_offload ``` The Wan2.1 models are loaded by their Hugging Face ids; pass `--wan_model` to use a local copy. Each sample writes the generated video, RGB / depth / normal renders of the 4DGS world and, with `--save_4dgs` or `--save_3dgs`, the Gaussians themselves (a static, Inria-format `point_cloud.ply` for standard 3DGS viewers in the latter case). Run `aeon-generate -h` for all options. From Python: ```python import torch from huggingface_hub import hf_hub_download from aeon4d.pipeline import (DEFAULT_CONFIG, denoise_video_latents, load_aeon, load_config, load_wan_pipeline, reconstruct_4dgs, zero_dxyzt_) from aeon4d.rendering import render_4dgs config = load_config(DEFAULT_CONFIG) device = torch.device("cuda:0") torch.manual_seed(42) pipe = load_wan_pipeline("Wan-AI/Wan2.1-T2V-1.3B-Diffusers", "t2v", flow_shift=3.0, device=device) model = load_aeon(config, hf_hub_download("wiki1231/AEON", "aeon.safetensors"), device) latents = denoise_video_latents(pipe, "A red fox running through deep snow", height=512, width=512, num_frames=81, num_inference_steps=50, guidance_scale=5.0, negative_prompt="Blurred, artifacts, low quality, distorted, deformed") video, views, res, gs_params = reconstruct_4dgs(model, pipe.vae, latents, vae_resolution=512, ff_resolution=448, device=device) zero_dxyzt_(gs_params) render = render_4dgs(gs_params, res, device, render_h=448, render_w=448, fix_k_mode="none") ``` ## Resource usage - **T2V-1.3B:** about 60 GB of GPU memory at peak. - **I2V-14B with** `--cpu_offload`**:** about 50 GB. Without offload, it needs roughly 90 GB. - **Speed:** a sample with the 14B model takes about 5–8× longer than one with the 1.3B model. Tested with PyTorch 2.13 + CUDA 13.0 on Python 3.12. ## Citation ```bibtex @inproceedings{zhang2026aeon, title={{AEON}: Unifying Video and {3D} World Models}, author={Zhang, Weiqi and Zhang, Wenyuan and Liu, Yu-Shen and Zhou, Junsheng}, booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, year={2026} } ``` ## Acknowledgements AEON builds on [Wan2.1](https://github.com/Wan-Video/Wan2.1), [MapAnything](https://github.com/facebookresearch/map-anything), [UniCeption](https://github.com/castacks/UniCeption), [DINOv2](https://github.com/facebookresearch/dinov2), [EasyVolcap](https://github.com/zju3dv/EasyVolcap), [diffusers](https://github.com/huggingface/diffusers) and [gsplat](https://github.com/nerfstudio-project/gsplat). We thank their authors for releasing their code and models. ## License The weights in this repository are released under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license and may be used for non-commercial research only: the 4DRM is initialized from [MapAnything](https://github.com/facebookresearch/map-anything) weights, and the training data include datasets licensed for non-commercial research only, such as DynamicReplica and Panda-70M. The AEON inference code is released under the [MIT License](https://github.com/weiqi-zhang/AEON/blob/main/LICENSE). The third-party code it includes keeps its original licenses, listed in [THIRD_PARTY_NOTICES.md](https://github.com/weiqi-zhang/AEON/blob/main/THIRD_PARTY_NOTICES.md). The Wan2.1 models are distributed by their authors under the Apache-2.0 license.