|
Download README.md from Maxtir/OctWorld-1.5: direct link, hf CLI and curl.
- Browser
- Download file 9.25 kB
-
https://huggingface.co/Maxtir/OctWorld-1.5/resolve/main/README.md
- Command line
-
hf download hf://Maxtir/OctWorld-1.5/README.md
-
curl -L -o README.md https://huggingface.co/Maxtir/OctWorld-1.5/resolve/main/README.md
9.25 kB
| license: apache-2.0 | |
| pipeline_tag: image-to-video | |
| base_model: Wan-AI/Wan2.1-I2V-14B-480P | |
| tags: | |
| - octworld | |
| - video-generation | |
| - camera-control | |
| - spatial-memory | |
| - octree | |
| - pytorch | |
| - arxiv:2609.03919 | |
| # OctWorld-1.5 | |
| **[Code](https://github.com/microsoft/OctWorld) | [Project Page & Demos](https://microsoft.github.io/OctWorld/) | [Paper](https://arxiv.org/abs/2609.03919)** | |
| **OctWorld: Long-Range World-Consistent Video Generation with Octree-based 3D Mapping** | |
| OctWorld-1.5 is the **Wan 14B-based world-generation checkpoint** for | |
| [OctWorld](https://github.com/microsoft/OctWorld). Starting from a single image, | |
| the framework generates video along user-specified camera trajectories, using | |
| text prompts, camera poses, and anchor depth as inputs. | |
| OctWorld maintains persistent **3D spatial memory** through **OctMap**, a | |
| spatially adaptive octree map. OctMap accumulates geometry and appearance from | |
| generated RGB-D observations and renders spatial conditions for subsequent | |
| video chunks, helping preserve scene consistency over long camera paths and | |
| revisits. | |
| ## Release Contents | |
| This repository contains the OctWorld-1.5 checkpoint and its pipeline assets: | |
| ```text | |
| OctWorld-1.5/ | |
| model/ | |
| .metadata | |
| __0_0.distcp | |
| ... | |
| __15_0.distcp | |
| pipe_config.json | |
| neg_text_embed.pt | |
| ``` | |
| | Asset | Description | | |
| | --- | --- | | |
| | `model/` | PyTorch distributed checkpoint: 16 weight shards and the required `.metadata` file. | | |
| | `pipe_config.json` | Wan 14B backbone, camera adapter, and FramePack configuration. | | |
| | `neg_text_embed.pt` | Precomputed negative-text embedding used by the inference pipeline. | | |
| The model package occupies approximately **101.84 GB (94.85 GiB)** on disk, | |
| excluding auxiliary models and generated outputs. Keep all checkpoint shards | |
| and `model/.metadata` together. | |
| Use the **official OctWorld inference code** to load this checkpoint. This is | |
| not a standalone Transformers or Diffusers pipeline. | |
| The code release includes Wan-based inference, OctMap, depth-prediction | |
| integration, bundled examples, and custom-input preparation. The training-data | |
| pipeline and end-to-end training/fine-tuning code are **not released yet**. | |
| ## Requirements | |
| The tested inference environment uses: | |
| - Linux x86_64 and Python **3.10** (tested with **3.10.21**). | |
| - An NVIDIA **Ampere-or-newer GPU with at least 70 GB of GPU memory**; | |
| tested on an **A100 80 GB**. | |
| - **CUDA Toolkit 12.6**, a compatible NVIDIA driver, a C++ compiler, Git, | |
| and FFmpeg. | |
| - **PyTorch 2.8.0+cu126**, **torchvision 0.23.0+cu126**, | |
| **xformers 0.0.32.post2**, and **flash-attn 2.8.3**. | |
| The CUDA toolkit is needed to build native rendering extensions; the CUDA | |
| runtime bundled with PyTorch alone is not sufficient. Follow the | |
| [installation guide](https://github.com/microsoft/OctWorld/blob/main/README.md) | |
| and the repository's pinned dependency setup. | |
| ## Quick Start | |
| Run the following commands from the code repository root after cloning it. | |
| Use a new or empty output directory for each generation run. | |
| ### 1. Install the Code | |
| ```bash | |
| git clone --recurse-submodules https://github.com/microsoft/OctWorld.git | |
| cd OctWorld | |
| conda create -n octworld python=3.10.21 -y | |
| conda activate octworld | |
| export CUDA_HOME=/usr/local/cuda-12.6 | |
| bash setup.sh | |
| ``` | |
| For an existing checkout, initialize the pinned depth-model submodules with | |
| `git submodule update --init --recursive`. Skip environment creation if you | |
| already have a suitable Python 3.10 environment. `setup.sh` installs dependencies | |
| into the selected environment; it does not install system packages or download | |
| model weights. | |
| ### 2. Download OctWorld-1.5 | |
| ```bash | |
| hf download Maxtir/OctWorld-1.5 --local-dir model_weights/octworld-1.5 | |
| ``` | |
| The `hf` CLI is included in the repository's `huggingface-hub` dependency. | |
| This downloads the checkpoint, configuration, and negative-text embedding into | |
| the layout expected by the examples. | |
| ### 3. Download Auxiliary Models | |
| OctWorld-1.5 alone is **not sufficient** to run the full framework: | |
| ```bash | |
| python scripts/download_models.py | |
| ``` | |
| | Component | Purpose | | |
| | --- | --- | | |
| | [Wan2.1 VAE, T5, CLIP, and tokenizers](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) | Image, text, and video encoding/decoding. The original Wan DiT is not downloaded. | | |
| | [Depth Anything 3](https://huggingface.co/depth-anything/DA3NESTED-GIANT-LARGE-1.1) | Per-chunk depth aligned to user cameras. | | |
| | [RAFT](https://github.com/pytorch/vision/tree/main/torchvision/models/optical_flow) | Optical-flow-based keyframe selection. | | |
| | [Sky segmentation](https://huggingface.co/JianyuanWang/skyseg) | Sky masking before map integration. | | |
| | [MoGe v1](https://huggingface.co/Ruicheng/moge-vitl) | Anchor depth and intrinsics for custom inputs. | | |
| | [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) | Prompt planning for custom inputs. | | |
| The default downloader prepares all six auxiliary components, approximately | |
| **42.44 GB (39.53 GiB)** in total. For bundled-case inference only, use | |
| `python scripts/download_models.py --models wan da3 raft skyseg`. | |
| Custom-input preparation additionally needs MoGe and Qwen3-VL. | |
| Every downloaded file and revision is pinned in the | |
| [model manifest](https://github.com/microsoft/OctWorld/blob/main/scripts/model_manifest.json). | |
| The auxiliary downloader does not download OctWorld-1.5. | |
| ### 4. Generate a Video | |
| ```bash | |
| python example/worldgen/wan/world_gen.py \ | |
| --dit_ckpt model_weights/octworld-1.5/model \ | |
| --model_weights_dir model_weights \ | |
| --save_dir outputs/dl3dv_gt | |
| ``` | |
| The default example uses the bundled DL3DV scene and its original camera | |
| trajectory. It generates **4 chunks / 321 frames** at **480 x 832 (height x | |
| width), 15 fps**, or approximately **21.4 seconds**. | |
| Successful single-case runs produce: | |
| | Output | Description | | |
| | --- | --- | | |
| | `output.mp4` | Generated RGB video. | | |
| | `depth.mp4` | Colorized inverse-depth visualization, **not metric depth**. | | |
| | `warp.mp4` | Rendered spatial conditions used during generation. | | |
| | `sha256.json` | Seed information and sampling/conditioning tensor hashes. | | |
| The default uses online OctMap integration, DA3 depth aligned to user cameras, | |
| and seed 425. The final chunk of a multi-chunk run skips depth prediction and | |
| map integration because there is no subsequent chunk to condition; its | |
| depth-video segment is black. | |
| See the [inference guide](https://github.com/microsoft/OctWorld/blob/main/example/worldgen/wan/README.md) | |
| for input formats, generation options, output semantics, and reproducibility | |
| limits. | |
| ## Batch Generation and Custom Inputs | |
| Generate all three bundled cases with one model load: | |
| ```bash | |
| python example/worldgen/wan/batch_world_gen.py \ | |
| --instances assets/standard/instances.json \ | |
| --dit_ckpt model_weights/octworld-1.5/model \ | |
| --model_weights_dir model_weights \ | |
| --base_folder outputs/batch | |
| ``` | |
| For your own images and camera paths, follow the | |
| [custom-input guide](https://github.com/microsoft/OctWorld/blob/main/example/customize_world_gen/README.md). | |
| The tools prepare inference inputs from an image and camera-movement | |
| instructions, with optional smooth camera interpolation, MoGe anchor depth, | |
| and Qwen3-VL prompt planning. They are not a training-data pipeline. | |
| ## Intended Use and Limitations | |
| OctWorld supports image-conditioned scene exploration, camera-controlled video | |
| generation, and research on long-range world-consistent generation. | |
| - Spatial memory helps preserve consistency but does not guarantee physically | |
| accurate geometry or artifact-free videos. Generated content and inferred | |
| depth should not be treated as ground-truth observations. | |
| - Anchor depth defines the metric scale of the user trajectory. The exported | |
| `depth.mp4` is only a visualization and must not be interpreted as metric | |
| depth measurements. | |
| - Results depend on inputs, camera paths, prompts, seeds, weights, and runtime | |
| environment. A matching seed or initial-noise hash alone does not guarantee | |
| pixel-identical outputs. | |
| - This checkpoint requires the official pipeline and auxiliary models; this | |
| repository does not include the full software environment or training code. | |
| For the method and experimental evaluation, see the | |
| [paper](https://arxiv.org/abs/2609.03919). For visual examples, visit the | |
| [project page](https://microsoft.github.io/OctWorld/). | |
| ## License | |
| The **OctWorld-1.5 model weights** are released under the | |
| [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). | |
| The [OctWorld code](https://github.com/microsoft/OctWorld) is released under the | |
| [MIT License](https://github.com/microsoft/OctWorld/blob/main/LICENSE). | |
| Third-party code, auxiliary model weights, and datasets remain subject to their | |
| respective licenses and usage terms. | |
| ## Citation | |
| If you find OctWorld useful for your research, please cite: | |
| ```bibtex | |
| @article{lv2026octworld, | |
| title = {OctWorld: Long-Range World-Consistent Video Generation | |
| with Octree-based 3D Mapping}, | |
| author = {Lv, Zelong and Xu, Sicheng and Xiang, Jianfeng and | |
| Wang, Ruicheng and Dong, Yue and Deng, Yu and | |
| Sun, Guangzhong and Yang, Jiaolong}, | |
| journal = {arXiv preprint arXiv:2609.03919}, | |
| year = {2026} | |
| } | |
| ``` | |