license: cc-by-nc-sa-4.0
library_name: pytorch
pipeline_tag: image-to-video
tags:
- surgical-robotics
- world-model
- video-prediction
- medical
- da-vinci
datasets:
- nvidia/PhysicalAI-Robotics-Open-H-Embodiment
Surgical World Model (EG-SWM)
Embodiment-grounded world models of robot-assisted surgery: an action-conditioned video world model whose action space mirrors the robot — an explicit endoscope ego-motion action plus one latent action per instrument arm, learned from unlabeled in-vivo da Vinci video and bound to the arms through the robot's own system logs.
Code, data pipeline, documentation and diagrams: https://github.com/GIND123/Surgical-World-Model
Checkpoints
One folder per finished run: model.safetensors (EMA weights), config.json (model and training config, step) and the evaluation summaries (replay.json: fidelity; control.json: endoscope control, locality, embodiment consistency; action_use.json: action-use diagnostics). Metrics on the held-out SurgVU test cases, 2.8 s horizon, action replay (protocol).
| Folder | Model | Size | Steps | PSNR ↑ | LPIPS ↓ | FVD ↓ | Endoscope gain ↑ |
|---|---|---|---|---|---|---|---|
S_none/ |
None (unconditional) | S | 60000 | 18.96 | 0.249 | 422 | – |
S_state_nocam/ |
Robot state | S | 60000 | 19.29 | 0.237 | 358 | – |
S_state/ |
Robot state + camera | S | 60000 | 20.44 | 0.206 | 266 | 0.79 |
S_global_nocam/ |
Monolithic latent (Genie) | S | 60000 | 22.49 | 0.154 | 172 | – |
S_global/ |
Monolithic latent + camera | S | 60000 | 22.47 | 0.154 | 180 | 0.16 |
S_global_vq/ |
Monolithic VQ + camera (SurgWM-style) | S | 60000 | 21.30 | 0.183 | 206 | 0.81 |
S_slots/ |
Slots + camera (FLAM-style) | S | 60000 | 22.77 | 0.149 | 169 | 0.05 |
S_arms_nocam/ |
EG-SWM, no camera action | S | 60000 | 21.52 | 0.176 | 215 | – |
S_arms/ |
EG-SWM | S | 60000 | 22.00 | 0.164 | 186 | 0.70 |
S_arms_scene/ |
EG-SWM + scene action | S | 60000 | 22.38 | 0.156 | 174 | 0.22 |
S_arms_nofwd/ |
EG-SWM, diffusion loss only | S | 60000 | 21.65 | 0.172 | 204 | 0.69 |
S_arms_poolglobal/ |
EG-SWM, arm actions not grounded | S | 60000 | 22.07 | 0.163 | 188 | 0.65 |
S_arms_shuffled/ |
EG-SWM, wrong arm binding | S | 60000 | 21.98 | 0.165 | 186 | 0.70 |
S_arms_ui/ |
EG-SWM + console readout | S | 60000 | 21.98 | 0.164 | 184 | 0.70 |
S_arms_hold/ |
EG-SWM, boxes held across detector gaps | S | 60000 | 22.06 | 0.163 | 185 | 0.65 |
Loading (with the swm package from the code repository):
import json
from safetensors.torch import load_model
from swm.models.world_model import WMConfig, WorldModel
meta = json.load(open("S_arms/config.json"))
model = WorldModel(WMConfig.from_dict(meta["model"])).eval()
load_model(model, "S_arms/model.safetensors")
The SurgWM re-implementation (surgwm/) has one file per stage (tokenizer, lam, dynamics), loaded into the classes of swm.baselines.surgwm with SurgWMConfig(**meta["model"]).
Training data
- SurgVU (Zia et al., Surgical Visual Understanding (SurgVU) Dataset, arXiv:2501.09209): 155 da Vinci porcine training-lab cases, 837.8 h, with robot-log instrument and endoscope occupancy per arm.
- For kinematic grounding / benchmarks: Open-H-Embodiment subsets (CC-BY-4.0), JIGSAWS (Gao et al., M2CAI 2014; data-use agreement), SAR-RARP50 (CC BY-NC-SA 4.0).
Licence and intended use
Weights are released under CC BY-NC-SA 4.0 for non-commercial research, in line with the licences of the training data. The models simulate endoscopic video; they are not medical devices and must not be used for clinical decision-making or to control a robot on a patient.