Shady0057/GWM

Grounded World Model: Latent Planning with Language Goals

Paper · Code and setup · WISER dataset

GWM predicts future visual embeddings from the current observation and candidate actions. A frozen video-language readout scores the predicted future against a language goal for planning.

Checkpoints

File	Training data	Intended workflow
checkpoint.pt	WISER simulation data	Learned world-model planning in the WISER testbed
real_data/checkpoint.pt	MolmoAct2-DROID and MolmoBot	DROID-sim and Franka hardware through the GWM scoring service

The real-data checkpoint is from step 34,000. Its model card records architecture, training configuration and checksums.

Both workflows require the separate Qwen3-VL-Embedding-8B encoder. These checkpoints contain the GWM predictor, not the Qwen encoder or the complete robotics stack.

Download and load

Install the source repository and dependencies using the setup instructions. Download the checkpoint needed for your workflow:

# WISER
hf download Shady0057/GWM checkpoint.pt --local-dir gwm_ckpt

# Real-data / robotics
hf download Shady0057/GWM real_data/checkpoint.pt --local-dir gwm_ckpt

# Companion encoder
hf download Qwen/Qwen3-VL-Embedding-8B --local-dir qwen3_vl_embedding_8b

Load the WISER checkpoint dictionary with the repository's compatibility loader:

from gwm_wiser.utils.checkpoint import load_gwm_checkpoint

checkpoint = load_gwm_checkpoint("gwm_ckpt/checkpoint.pt", map_location="cpu")
config = checkpoint["config"]
state_dict = checkpoint["model_state_dict"]

This handles the original WISER checkpoint's legacy configuration class name. The WISER planner uses the same loader when constructing the model. For the real-data checkpoint, run from a source checkout:

from real_data_train.gwm_model import load_canonical_like_planner

model, checkpoint = load_canonical_like_planner("gwm_ckpt/real_data/checkpoint.pt")
model.eval()

These loaders deserialize PyTorch checkpoints; use them only with trusted files. See the checkpoint loading checks and repository entrypoints for the complete planning setup.

Citation

@misc{li2026groundedworldmodellatentplanning,
      title={Grounded World Model: Latent Planning with Language Goals},
      author={Quanyi Li and Lan Feng and Haonan Zhang and Wuyang Li and Letian Wang and Alexandre Alahi and Harold Soh},
      year={2026},
      eprint={2604.11751},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2604.11751},
}