GILWM

GILWM (Grounded Instance Lifting Towards Physically Consistent Video World Models) uses first-frame object and geometric structure as priors for robot-interaction video prediction. Given an initial frame and a language instruction, it aims to support object persistence, geometric coherence, and plausible spatial relationships during generation. Physical consistency is a research objective, not a guarantee of the generated videos.

GILWM is built on the 2B-parameter NVIDIA Cosmos Predict2.5 Video2World backbone. It augments the video diffusion transformer with structure features extracted from the initial frame by SAM 3 and SAM 3D Objects, including instance masks, geometry, object shape, and pose. These features are injected through learned, zero-gated cross-attention adapters.

Project links

• Training and inference code: Coming soon (the GitHub repository remains private)
• Model weights: kurisu0306/GILWM
• Technical report: Coming soon

Model details

Property	Value
Task	Structure-conditioned image-to-video prediction
Domain	Robot manipulation and embodied interaction
Backbone	Cosmos Predict2.5 2B Video2World
Training clip / AR chunk	93 frames
Output resolution	640 × 480
Output length	Configurable through autoregressive chunks
Checkpoint	Consolidated bf16 PyTorch EMA state dict
Model file	model_ema_bf16.pt

The repository contains inference weights only. Optimizer, scheduler, and trainer states are not included.

Inputs and outputs

GILWM uses the following inputs:

• an initial RGB frame;
• a natural-language instruction describing the intended interaction;
• structure conditions extracted from the initial frame, including instance masks, geometry, object shape features, and pose features.

The model outputs a video prediction of the future scene. Longer sequences can be generated autoregressively from 93-frame chunks.

Intended use

GILWM is intended for research on:

• video world models for robot manipulation;
• object- and geometry-aware future prediction;
• embodied interaction simulation;
• evaluation of semantic, spatial, and physical consistency in generated videos.

The model predicts visual futures. It does not generate robot actions and is not a closed-loop control policy.

Usage

The model weights are publicly downloadable. The GILWM training and inference code remains private and will be released separately. Running the checkpoint requires the custom GILWM architecture and structure preprocessing; downloading these weights alone does not provide an end-to-end inference pipeline.

Download the weights with the Hugging Face CLI:

hf download kurisu0306/GILWM model_ema_bf16.pt \
  --local-dir checkpoints/GILWM

GILWM also requires the compatible Cosmos Predict2.5 text encoder and VAE, together with the SAM 3, SAM 3D Objects, DINOv2, and MoGe assets used by the structure preprocessing pipeline. Setup and inference documentation will accompany the code release.

Evaluation

GILWM was evaluated on robot-interaction video prediction tasks using semantic alignment and interaction-focused measurements. Qualitative results compare the generated future against the corresponding ground-truth video, with identical initial observations and task instructions.

Reported examples and benchmark results should be interpreted as open-loop video predictions rather than evidence of closed-loop robot-control performance.

Limitations

• Performance is strongest in robot-manipulation scenes similar to the training and evaluation domains.
• Long-horizon autoregressive generation may accumulate visual or geometric errors.
• Fine object contact, occlusion, and fast motion can reduce physical consistency.
• Output videos represent plausible visual futures and should not be treated as guaranteed physical outcomes.
• The preprocessing stack and model runtime require substantial accelerator memory and several external model assets.

GILWM has not been validated for safety-critical deployment or direct control of physical systems.

License and dependencies

This checkpoint is derived from Cosmos Predict2.5 and depends on third-party model components. Users must comply with the applicable licenses and usage terms for Cosmos Predict2.5, SAM 3, SAM 3D Objects, DINOv2, MoGe, and any other downloaded assets.

Citation

Citation information will be added with the GILWM technical report.

Acknowledgements

GILWM builds on NVIDIA Cosmos Predict2.5 and uses SAM 3, SAM 3D Objects, DINOv2, and MoGe for structure extraction and conditioning.