OctWorld-1.5 / README.md
Maxtir's picture
Add OctWorld-1.5 model card with official links and usage guide
fd37f52 verified
|
Raw History Blame Contribute Delete
9.25 kB
---
license: apache-2.0
pipeline_tag: image-to-video
base_model: Wan-AI/Wan2.1-I2V-14B-480P
tags:
- octworld
- video-generation
- camera-control
- spatial-memory
- octree
- pytorch
- arxiv:2609.03919
---
# OctWorld-1.5
**[Code](https://github.com/microsoft/OctWorld) | [Project Page & Demos](https://microsoft.github.io/OctWorld/) | [Paper](https://arxiv.org/abs/2609.03919)**
**OctWorld: Long-Range World-Consistent Video Generation with Octree-based 3D Mapping**
OctWorld-1.5 is the **Wan 14B-based world-generation checkpoint** for
[OctWorld](https://github.com/microsoft/OctWorld). Starting from a single image,
the framework generates video along user-specified camera trajectories, using
text prompts, camera poses, and anchor depth as inputs.
OctWorld maintains persistent **3D spatial memory** through **OctMap**, a
spatially adaptive octree map. OctMap accumulates geometry and appearance from
generated RGB-D observations and renders spatial conditions for subsequent
video chunks, helping preserve scene consistency over long camera paths and
revisits.
## Release Contents
This repository contains the OctWorld-1.5 checkpoint and its pipeline assets:
```text
OctWorld-1.5/
model/
.metadata
__0_0.distcp
...
__15_0.distcp
pipe_config.json
neg_text_embed.pt
```
| Asset | Description |
| --- | --- |
| `model/` | PyTorch distributed checkpoint: 16 weight shards and the required `.metadata` file. |
| `pipe_config.json` | Wan 14B backbone, camera adapter, and FramePack configuration. |
| `neg_text_embed.pt` | Precomputed negative-text embedding used by the inference pipeline. |
The model package occupies approximately **101.84 GB (94.85 GiB)** on disk,
excluding auxiliary models and generated outputs. Keep all checkpoint shards
and `model/.metadata` together.
Use the **official OctWorld inference code** to load this checkpoint. This is
not a standalone Transformers or Diffusers pipeline.
The code release includes Wan-based inference, OctMap, depth-prediction
integration, bundled examples, and custom-input preparation. The training-data
pipeline and end-to-end training/fine-tuning code are **not released yet**.
## Requirements
The tested inference environment uses:
- Linux x86_64 and Python **3.10** (tested with **3.10.21**).
- An NVIDIA **Ampere-or-newer GPU with at least 70 GB of GPU memory**;
tested on an **A100 80 GB**.
- **CUDA Toolkit 12.6**, a compatible NVIDIA driver, a C++ compiler, Git,
and FFmpeg.
- **PyTorch 2.8.0+cu126**, **torchvision 0.23.0+cu126**,
**xformers 0.0.32.post2**, and **flash-attn 2.8.3**.
The CUDA toolkit is needed to build native rendering extensions; the CUDA
runtime bundled with PyTorch alone is not sufficient. Follow the
[installation guide](https://github.com/microsoft/OctWorld/blob/main/README.md)
and the repository's pinned dependency setup.
## Quick Start
Run the following commands from the code repository root after cloning it.
Use a new or empty output directory for each generation run.
### 1. Install the Code
```bash
git clone --recurse-submodules https://github.com/microsoft/OctWorld.git
cd OctWorld
conda create -n octworld python=3.10.21 -y
conda activate octworld
export CUDA_HOME=/usr/local/cuda-12.6
bash setup.sh
```
For an existing checkout, initialize the pinned depth-model submodules with
`git submodule update --init --recursive`. Skip environment creation if you
already have a suitable Python 3.10 environment. `setup.sh` installs dependencies
into the selected environment; it does not install system packages or download
model weights.
### 2. Download OctWorld-1.5
```bash
hf download Maxtir/OctWorld-1.5 --local-dir model_weights/octworld-1.5
```
The `hf` CLI is included in the repository's `huggingface-hub` dependency.
This downloads the checkpoint, configuration, and negative-text embedding into
the layout expected by the examples.
### 3. Download Auxiliary Models
OctWorld-1.5 alone is **not sufficient** to run the full framework:
```bash
python scripts/download_models.py
```
| Component | Purpose |
| --- | --- |
| [Wan2.1 VAE, T5, CLIP, and tokenizers](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) | Image, text, and video encoding/decoding. The original Wan DiT is not downloaded. |
| [Depth Anything 3](https://huggingface.co/depth-anything/DA3NESTED-GIANT-LARGE-1.1) | Per-chunk depth aligned to user cameras. |
| [RAFT](https://github.com/pytorch/vision/tree/main/torchvision/models/optical_flow) | Optical-flow-based keyframe selection. |
| [Sky segmentation](https://huggingface.co/JianyuanWang/skyseg) | Sky masking before map integration. |
| [MoGe v1](https://huggingface.co/Ruicheng/moge-vitl) | Anchor depth and intrinsics for custom inputs. |
| [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) | Prompt planning for custom inputs. |
The default downloader prepares all six auxiliary components, approximately
**42.44 GB (39.53 GiB)** in total. For bundled-case inference only, use
`python scripts/download_models.py --models wan da3 raft skyseg`.
Custom-input preparation additionally needs MoGe and Qwen3-VL.
Every downloaded file and revision is pinned in the
[model manifest](https://github.com/microsoft/OctWorld/blob/main/scripts/model_manifest.json).
The auxiliary downloader does not download OctWorld-1.5.
### 4. Generate a Video
```bash
python example/worldgen/wan/world_gen.py \
--dit_ckpt model_weights/octworld-1.5/model \
--model_weights_dir model_weights \
--save_dir outputs/dl3dv_gt
```
The default example uses the bundled DL3DV scene and its original camera
trajectory. It generates **4 chunks / 321 frames** at **480 x 832 (height x
width), 15 fps**, or approximately **21.4 seconds**.
Successful single-case runs produce:
| Output | Description |
| --- | --- |
| `output.mp4` | Generated RGB video. |
| `depth.mp4` | Colorized inverse-depth visualization, **not metric depth**. |
| `warp.mp4` | Rendered spatial conditions used during generation. |
| `sha256.json` | Seed information and sampling/conditioning tensor hashes. |
The default uses online OctMap integration, DA3 depth aligned to user cameras,
and seed 425. The final chunk of a multi-chunk run skips depth prediction and
map integration because there is no subsequent chunk to condition; its
depth-video segment is black.
See the [inference guide](https://github.com/microsoft/OctWorld/blob/main/example/worldgen/wan/README.md)
for input formats, generation options, output semantics, and reproducibility
limits.
## Batch Generation and Custom Inputs
Generate all three bundled cases with one model load:
```bash
python example/worldgen/wan/batch_world_gen.py \
--instances assets/standard/instances.json \
--dit_ckpt model_weights/octworld-1.5/model \
--model_weights_dir model_weights \
--base_folder outputs/batch
```
For your own images and camera paths, follow the
[custom-input guide](https://github.com/microsoft/OctWorld/blob/main/example/customize_world_gen/README.md).
The tools prepare inference inputs from an image and camera-movement
instructions, with optional smooth camera interpolation, MoGe anchor depth,
and Qwen3-VL prompt planning. They are not a training-data pipeline.
## Intended Use and Limitations
OctWorld supports image-conditioned scene exploration, camera-controlled video
generation, and research on long-range world-consistent generation.
- Spatial memory helps preserve consistency but does not guarantee physically
accurate geometry or artifact-free videos. Generated content and inferred
depth should not be treated as ground-truth observations.
- Anchor depth defines the metric scale of the user trajectory. The exported
`depth.mp4` is only a visualization and must not be interpreted as metric
depth measurements.
- Results depend on inputs, camera paths, prompts, seeds, weights, and runtime
environment. A matching seed or initial-noise hash alone does not guarantee
pixel-identical outputs.
- This checkpoint requires the official pipeline and auxiliary models; this
repository does not include the full software environment or training code.
For the method and experimental evaluation, see the
[paper](https://arxiv.org/abs/2609.03919). For visual examples, visit the
[project page](https://microsoft.github.io/OctWorld/).
## License
The **OctWorld-1.5 model weights** are released under the
[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
The [OctWorld code](https://github.com/microsoft/OctWorld) is released under the
[MIT License](https://github.com/microsoft/OctWorld/blob/main/LICENSE).
Third-party code, auxiliary model weights, and datasets remain subject to their
respective licenses and usage terms.
## Citation
If you find OctWorld useful for your research, please cite:
```bibtex
@article{lv2026octworld,
title = {OctWorld: Long-Range World-Consistent Video Generation
with Octree-based 3D Mapping},
author = {Lv, Zelong and Xu, Sicheng and Xiang, Jianfeng and
Wang, Ruicheng and Dong, Yue and Deng, Yu and
Sun, Guangzhong and Yang, Jiaolong},
journal = {arXiv preprint arXiv:2609.03919},
year = {2026}
}
```