OpenNWM / README.md

Publish original OpenNWM Nav15 checkpoints and configs

c4c3a02 verified 6 days ago

4.74 kB

---
license: cc-by-nc-4.0
language:
- en
- zh
tags:
- OpenNWM
- navigation
- world-model
- latent-action-model
- NavAnywhere
library_name: pytorch
---
# OpenNWM

The Nav15 Pixel+Action release of OpenNWM: a latent action model (LAM), a
Stage 1 navigation world model pretrained with latent actions, and a Stage 2
world model adapted to real navigation actions.

- Code and setup: [B0B8K1ng/OpenNWM](https://github.com/B0B8K1ng/OpenNWM)
- ModelScope mirror: [LittleBoss/OpenNWM](https://modelscope.cn/models/LittleBoss/OpenNWM)
- Training data: [NavAnywhere](https://huggingface.co/datasets/LittleB0sS/NavAnywhere)

## Checkpoints

| File | Use | Bytes |
| --- | --- | ---: |
| `opennwm-nav15-stage2-joint100k.pth.tar` | Stage 2 real-action navigation; 3,000 adapter warmup + 100,000 joint steps | 3,137,820,686 |
| `opennwm-nav15-stage1-pretrain60k.pth.tar` | Stage 1 latent-action pretraining; 60,000 steps | 3,137,820,971 |
| `lam-nav15-pixel-action-step60k.ckpt` | Nav15 PixelActionLAM; 60,000 steps; 32-dimensional latent actions | 8,518,856,077 |

Use **Stage 2** for real-action prediction and planning. Stage 1 uses latent
actions and has not been adapted for direct real-action navigation. The LAM
maps image pairs to 32-dimensional latent actions. These are the final Nav15
checkpoints, distinct from the older Nav1 weights in `LittleBoss/LAM-weights`.

Files preserve their original checkpoint format and bytes, including training
state and historical configuration metadata. No optimizer tensors were removed
and no weight conversion was performed. Use the separate portable YAML configs
below instead of relying on historical machine paths embedded in checkpoints.
All payload files are covered by `SHA256SUMS`; `manifest.json` records checkpoint
roles, byte sizes and hashes.

## Model and inference settings

The world models use CDiT-B/2 with four 224×224 context frames and
`stabilityai/sd-vae-ft-ema` (latent scaling 0.18215). The VAE is an external
dependency and is downloaded separately. Stage 2 uses the checkpoint's `ema`
state, three real action dimensions, and 250 DDPM sampling steps. The real-action
normalization bounds are [-2.5, -4, -π] to [5, 4, π]. LAM images use a 4:3 center
crop followed by resize to 240×320; use the repository preprocessing code.

## Download and use

```python
from huggingface_hub import snapshot_download
snapshot_download("LittleB0sS/OpenNWM", local_dir="./weights/OpenNWM")
```

Install OpenNWM and configure its data and runtime paths following the code
repository README. From the OpenNWM source repository, import Stage 2 into a
new inference directory with its portable config:

```bash
python run.py nwm import-checkpoint \
  --checkpoint /path/to/weights/OpenNWM/opennwm-nav15-stage2-joint100k.pth.tar \
  --config /path/to/weights/OpenNWM/configs/opennwm-nav15-stage2.yaml \
  --run-dir /path/to/runs/opennwm-nav15-stage2
python run.py nwm infer exp_dir=/path/to/runs/opennwm-nav15-stage2 ckp=latest eval_type=time
```

`NWM_VAE_PATH` can point to a local VAE or use the default public identifier.
Dataset evaluation also requires `NWM_DATA_ROOT` and `NWM_SPLIT_ROOT`; use the
original split lists for historical score reproduction. The portable configs
are for inference and do not claim exact training resumption. Refer to the
repository NWM README for prediction, evaluation and planning, and the LAM
README for using `lam-nav15-pixel-action-step60k.ckpt` with LAM evaluation or
latent extraction. Stage 1's config additionally uses
`NWM_NAVANYWHERE_ROOT`, `NWM_NAVANYWHERE_RECIPE` and `NWM_LATENT_PROXY_ROOT`.

## Provenance and licenses

NWM code derives from Meta NWM and CompACT; its upstream **CC BY-NC 4.0**
license is preserved in `licenses/NWM-CC-BY-NC-4.0.txt`. DreamDojo LAM code's
**Apache 2.0** license is preserved in `licenses/DreamDojo-Apache-2.0.txt`.
The repository code provenance pins DreamDojo `8f3eeb7`, CompACT `8367a5e`,
and Meta NWM `3f6cd8e`. This release retains both upstream notices and does
not relicense upstream material. External VAE and data sources retain their
respective terms.

The weights support research on visual navigation and future observation
prediction. They are not a complete robot controller; predictions can drift
outside the training distribution. No new benchmark scores or deployment
claims are introduced by this release.

## 中文说明

本仓库发布 Nav15 Pixel+Action 的最终 LAM、Stage 1 和 Stage 2 权重。
真实动作导航预测与规划使用 Stage 2 的 EMA 权重。所有 checkpoint 保留原始
字节和训练状态；请传入本仓库提供的可迁移 YAML 配置，避免使用 checkpoint
中历史机器路径。完整使用方法见代码仓库，文件完整性由 SHA256SUMS 校验。

	---
	license: cc-by-nc-4.0
	language:
	- en
	- zh
	tags:
	- OpenNWM
	- navigation
	- world-model
	- latent-action-model
	- NavAnywhere
	library_name: pytorch
	---

	# OpenNWM

	The Nav15 Pixel+Action release of OpenNWM: a latent action model (LAM), a
	Stage 1 navigation world model pretrained with latent actions, and a Stage 2
	world model adapted to real navigation actions.

	- Code and setup: [B0B8K1ng/OpenNWM](https://github.com/B0B8K1ng/OpenNWM)
	- ModelScope mirror: [LittleBoss/OpenNWM](https://modelscope.cn/models/LittleBoss/OpenNWM)
	- Training data: [NavAnywhere](https://huggingface.co/datasets/LittleB0sS/NavAnywhere)

	## Checkpoints

	\| File \| Use \| Bytes \|
	\| --- \| --- \| ---: \|
	\| `opennwm-nav15-stage2-joint100k.pth.tar` \| Stage 2 real-action navigation; 3,000 adapter warmup + 100,000 joint steps \| 3,137,820,686 \|
	\| `opennwm-nav15-stage1-pretrain60k.pth.tar` \| Stage 1 latent-action pretraining; 60,000 steps \| 3,137,820,971 \|
	\| `lam-nav15-pixel-action-step60k.ckpt` \| Nav15 PixelActionLAM; 60,000 steps; 32-dimensional latent actions \| 8,518,856,077 \|

	Use Stage 2 for real-action prediction and planning. Stage 1 uses latent
	actions and has not been adapted for direct real-action navigation. The LAM
	maps image pairs to 32-dimensional latent actions. These are the final Nav15
	checkpoints, distinct from the older Nav1 weights in `LittleBoss/LAM-weights`.

	Files preserve their original checkpoint format and bytes, including training
	state and historical configuration metadata. No optimizer tensors were removed
	and no weight conversion was performed. Use the separate portable YAML configs
	below instead of relying on historical machine paths embedded in checkpoints.
	All payload files are covered by `SHA256SUMS`; `manifest.json` records checkpoint
	roles, byte sizes and hashes.

	## Model and inference settings

	The world models use CDiT-B/2 with four 224×224 context frames and
	`stabilityai/sd-vae-ft-ema` (latent scaling 0.18215). The VAE is an external
	dependency and is downloaded separately. Stage 2 uses the checkpoint's `ema`
	state, three real action dimensions, and 250 DDPM sampling steps. The real-action
	normalization bounds are [-2.5, -4, -π] to [5, 4, π]. LAM images use a 4:3 center
	crop followed by resize to 240×320; use the repository preprocessing code.

	## Download and use

	```python
	from huggingface_hub import snapshot_download
	snapshot_download("LittleB0sS/OpenNWM", local_dir="./weights/OpenNWM")
	```

	Install OpenNWM and configure its data and runtime paths following the code
	repository README. From the OpenNWM source repository, import Stage 2 into a
	new inference directory with its portable config:

	```bash
	python run.py nwm import-checkpoint \
	--checkpoint /path/to/weights/OpenNWM/opennwm-nav15-stage2-joint100k.pth.tar \
	--config /path/to/weights/OpenNWM/configs/opennwm-nav15-stage2.yaml \
	--run-dir /path/to/runs/opennwm-nav15-stage2
	python run.py nwm infer exp_dir=/path/to/runs/opennwm-nav15-stage2 ckp=latest eval_type=time
	```

	`NWM_VAE_PATH` can point to a local VAE or use the default public identifier.
	Dataset evaluation also requires `NWM_DATA_ROOT` and `NWM_SPLIT_ROOT`; use the
	original split lists for historical score reproduction. The portable configs
	are for inference and do not claim exact training resumption. Refer to the
	repository NWM README for prediction, evaluation and planning, and the LAM
	README for using `lam-nav15-pixel-action-step60k.ckpt` with LAM evaluation or
	latent extraction. Stage 1's config additionally uses
	`NWM_NAVANYWHERE_ROOT`, `NWM_NAVANYWHERE_RECIPE` and `NWM_LATENT_PROXY_ROOT`.

	## Provenance and licenses

	NWM code derives from Meta NWM and CompACT; its upstream CC BY-NC 4.0
	license is preserved in `licenses/NWM-CC-BY-NC-4.0.txt`. DreamDojo LAM code's
	Apache 2.0 license is preserved in `licenses/DreamDojo-Apache-2.0.txt`.
	The repository code provenance pins DreamDojo `8f3eeb7`, CompACT `8367a5e`,
	and Meta NWM `3f6cd8e`. This release retains both upstream notices and does
	not relicense upstream material. External VAE and data sources retain their
	respective terms.

	The weights support research on visual navigation and future observation
	prediction. They are not a complete robot controller; predictions can drift
	outside the training distribution. No new benchmark scores or deployment
	claims are introduced by this release.

	## 中文说明

	本仓库发布 Nav15 Pixel+Action 的最终 LAM、Stage 1 和 Stage 2 权重。
	真实动作导航预测与规划使用 Stage 2 的 EMA 权重。所有 checkpoint 保留原始
	字节和训练状态；请传入本仓库提供的可迁移 YAML 配置，避免使用 checkpoint
	中历史机器路径。完整使用方法见代码仓库，文件完整性由 SHA256SUMS 校验。