---
license: apache-2.0
pipeline_tag: image-to-video
base_model: Wan-AI/Wan2.1-I2V-14B-480P
tags:
  - octworld
  - video-generation
  - camera-control
  - spatial-memory
  - octree
  - pytorch
  - arxiv:2609.03919
---

# OctWorld-1.5

**[Code](https://github.com/microsoft/OctWorld) | [Project Page & Demos](https://microsoft.github.io/OctWorld/) | [Paper](https://arxiv.org/abs/2609.03919)**

**OctWorld: Long-Range World-Consistent Video Generation with Octree-based 3D Mapping**

OctWorld-1.5 is the **Wan 14B-based world-generation checkpoint** for
[OctWorld](https://github.com/microsoft/OctWorld). Starting from a single image,
the framework generates video along user-specified camera trajectories, using
text prompts, camera poses, and anchor depth as inputs.

OctWorld maintains persistent **3D spatial memory** through **OctMap**, a
spatially adaptive octree map. OctMap accumulates geometry and appearance from
generated RGB-D observations and renders spatial conditions for subsequent
video chunks, helping preserve scene consistency over long camera paths and
revisits.

## Release Contents

This repository contains the OctWorld-1.5 checkpoint and its pipeline assets:

```text
OctWorld-1.5/
  model/
    .metadata
    __0_0.distcp
    ...
    __15_0.distcp
  pipe_config.json
  neg_text_embed.pt
```

| Asset | Description |
| --- | --- |
| `model/` | PyTorch distributed checkpoint: 16 weight shards and the required `.metadata` file. |
| `pipe_config.json` | Wan 14B backbone, camera adapter, and FramePack configuration. |
| `neg_text_embed.pt` | Precomputed negative-text embedding used by the inference pipeline. |

The model package occupies approximately **101.84 GB (94.85 GiB)** on disk,
excluding auxiliary models and generated outputs. Keep all checkpoint shards
and `model/.metadata` together.

Use the **official OctWorld inference code** to load this checkpoint. This is
not a standalone Transformers or Diffusers pipeline.

The code release includes Wan-based inference, OctMap, depth-prediction
integration, bundled examples, and custom-input preparation. The training-data
pipeline and end-to-end training/fine-tuning code are **not released yet**.

## Requirements

The tested inference environment uses:

- Linux x86_64 and Python **3.10** (tested with **3.10.21**).
- An NVIDIA **Ampere-or-newer GPU with at least 70 GB of GPU memory**;
  tested on an **A100 80 GB**.
- **CUDA Toolkit 12.6**, a compatible NVIDIA driver, a C++ compiler, Git,
  and FFmpeg.
- **PyTorch 2.8.0+cu126**, **torchvision 0.23.0+cu126**,
  **xformers 0.0.32.post2**, and **flash-attn 2.8.3**.

The CUDA toolkit is needed to build native rendering extensions; the CUDA
runtime bundled with PyTorch alone is not sufficient. Follow the
[installation guide](https://github.com/microsoft/OctWorld/blob/main/README.md)
and the repository's pinned dependency setup.

## Quick Start

Run the following commands from the code repository root after cloning it.
Use a new or empty output directory for each generation run.

### 1. Install the Code

```bash
git clone --recurse-submodules https://github.com/microsoft/OctWorld.git
cd OctWorld

conda create -n octworld python=3.10.21 -y
conda activate octworld

export CUDA_HOME=/usr/local/cuda-12.6
bash setup.sh
```

For an existing checkout, initialize the pinned depth-model submodules with
`git submodule update --init --recursive`. Skip environment creation if you
already have a suitable Python 3.10 environment. `setup.sh` installs dependencies
into the selected environment; it does not install system packages or download
model weights.

### 2. Download OctWorld-1.5

```bash
hf download Maxtir/OctWorld-1.5 --local-dir model_weights/octworld-1.5
```

The `hf` CLI is included in the repository's `huggingface-hub` dependency.
This downloads the checkpoint, configuration, and negative-text embedding into
the layout expected by the examples.

### 3. Download Auxiliary Models

OctWorld-1.5 alone is **not sufficient** to run the full framework:

```bash
python scripts/download_models.py
```

| Component | Purpose |
| --- | --- |
| [Wan2.1 VAE, T5, CLIP, and tokenizers](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) | Image, text, and video encoding/decoding. The original Wan DiT is not downloaded. |
| [Depth Anything 3](https://huggingface.co/depth-anything/DA3NESTED-GIANT-LARGE-1.1) | Per-chunk depth aligned to user cameras. |
| [RAFT](https://github.com/pytorch/vision/tree/main/torchvision/models/optical_flow) | Optical-flow-based keyframe selection. |
| [Sky segmentation](https://huggingface.co/JianyuanWang/skyseg) | Sky masking before map integration. |
| [MoGe v1](https://huggingface.co/Ruicheng/moge-vitl) | Anchor depth and intrinsics for custom inputs. |
| [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) | Prompt planning for custom inputs. |

The default downloader prepares all six auxiliary components, approximately
**42.44 GB (39.53 GiB)** in total. For bundled-case inference only, use
`python scripts/download_models.py --models wan da3 raft skyseg`.
Custom-input preparation additionally needs MoGe and Qwen3-VL.

Every downloaded file and revision is pinned in the
[model manifest](https://github.com/microsoft/OctWorld/blob/main/scripts/model_manifest.json).
The auxiliary downloader does not download OctWorld-1.5.

### 4. Generate a Video

```bash
python example/worldgen/wan/world_gen.py \
    --dit_ckpt model_weights/octworld-1.5/model \
    --model_weights_dir model_weights \
    --save_dir outputs/dl3dv_gt
```

The default example uses the bundled DL3DV scene and its original camera
trajectory. It generates **4 chunks / 321 frames** at **480 x 832 (height x
width), 15 fps**, or approximately **21.4 seconds**.

Successful single-case runs produce:

| Output | Description |
| --- | --- |
| `output.mp4` | Generated RGB video. |
| `depth.mp4` | Colorized inverse-depth visualization, **not metric depth**. |
| `warp.mp4` | Rendered spatial conditions used during generation. |
| `sha256.json` | Seed information and sampling/conditioning tensor hashes. |

The default uses online OctMap integration, DA3 depth aligned to user cameras,
and seed 425. The final chunk of a multi-chunk run skips depth prediction and
map integration because there is no subsequent chunk to condition; its
depth-video segment is black.

See the [inference guide](https://github.com/microsoft/OctWorld/blob/main/example/worldgen/wan/README.md)
for input formats, generation options, output semantics, and reproducibility
limits.

## Batch Generation and Custom Inputs

Generate all three bundled cases with one model load:

```bash
python example/worldgen/wan/batch_world_gen.py \
    --instances assets/standard/instances.json \
    --dit_ckpt model_weights/octworld-1.5/model \
    --model_weights_dir model_weights \
    --base_folder outputs/batch
```

For your own images and camera paths, follow the
[custom-input guide](https://github.com/microsoft/OctWorld/blob/main/example/customize_world_gen/README.md).
The tools prepare inference inputs from an image and camera-movement
instructions, with optional smooth camera interpolation, MoGe anchor depth,
and Qwen3-VL prompt planning. They are not a training-data pipeline.

## Intended Use and Limitations

OctWorld supports image-conditioned scene exploration, camera-controlled video
generation, and research on long-range world-consistent generation.

- Spatial memory helps preserve consistency but does not guarantee physically
  accurate geometry or artifact-free videos. Generated content and inferred
  depth should not be treated as ground-truth observations.
- Anchor depth defines the metric scale of the user trajectory. The exported
  `depth.mp4` is only a visualization and must not be interpreted as metric
  depth measurements.
- Results depend on inputs, camera paths, prompts, seeds, weights, and runtime
  environment. A matching seed or initial-noise hash alone does not guarantee
  pixel-identical outputs.
- This checkpoint requires the official pipeline and auxiliary models; this
  repository does not include the full software environment or training code.

For the method and experimental evaluation, see the
[paper](https://arxiv.org/abs/2609.03919). For visual examples, visit the
[project page](https://microsoft.github.io/OctWorld/).

## License

The **OctWorld-1.5 model weights** are released under the
[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).

The [OctWorld code](https://github.com/microsoft/OctWorld) is released under the
[MIT License](https://github.com/microsoft/OctWorld/blob/main/LICENSE).
Third-party code, auxiliary model weights, and datasets remain subject to their
respective licenses and usage terms.

## Citation

If you find OctWorld useful for your research, please cite:

```bibtex
@article{lv2026octworld,
    title   = {OctWorld: Long-Range World-Consistent Video Generation
                         with Octree-based 3D Mapping},
    author  = {Lv, Zelong and Xu, Sicheng and Xiang, Jianfeng and
                         Wang, Ruicheng and Dong, Yue and Deng, Yu and
                         Sun, Guangzhong and Yang, Jiaolong},
    journal = {arXiv preprint arXiv:2609.03919},
    year    = {2026}
}
```