---
license: cc-by-nc-sa-4.0
library_name: pytorch
pipeline_tag: image-to-video
tags:
  - surgical-robotics
  - world-model
  - video-prediction
  - medical
  - da-vinci
datasets:
  - nvidia/PhysicalAI-Robotics-Open-H-Embodiment
---

# Surgical World Model (EG-SWM)

Embodiment-grounded world models of robot-assisted surgery: an action-conditioned video world model whose action space mirrors the robot — an explicit **endoscope ego-motion** action plus one latent action per **instrument arm**, learned from unlabeled in-vivo da Vinci video and bound to the arms through the robot's own system logs.

Code, data pipeline, documentation and diagrams: <https://github.com/GIND123/Surgical-World-Model>

![architecture](architecture.png)

## Checkpoints

One folder per finished run: `model.safetensors` (EMA weights), `config.json` (model and training config, step) and the evaluation summaries (`replay.json`: fidelity; `control.json`: endoscope control, locality, embodiment consistency; `action_use.json`: action-use diagnostics). Metrics on the held-out SurgVU test cases, 2.8 s horizon, action replay ([protocol](https://github.com/GIND123/Surgical-World-Model#experiments-and-evaluation-protocol)).

| Folder | Model | Size | Steps | PSNR ↑ | LPIPS ↓ | FVD ↓ | Endoscope gain ↑ |
|---|---|---|---:|---:|---:|---:|---:|
| `S_none/` | None (unconditional) | S | 60000 | 18.96 | 0.249 | 422 | – |
| `S_state_nocam/` | Robot state | S | 60000 | 19.29 | 0.237 | 358 | – |
| `S_state/` | Robot state + camera | S | 60000 | 20.44 | 0.206 | 266 | 0.79 |
| `S_global_nocam/` | Monolithic latent (Genie) | S | 60000 | 22.49 | 0.154 | 172 | – |
| `S_global/` | Monolithic latent + camera | S | 60000 | 22.47 | 0.154 | 180 | 0.16 |
| `S_global_vq/` | Monolithic VQ + camera (SurgWM-style) | S | 60000 | 21.30 | 0.183 | 206 | 0.81 |
| `S_slots/` | Slots + camera (FLAM-style) | S | 60000 | 22.77 | 0.149 | 169 | 0.05 |
| `S_arms_nocam/` | EG-SWM, no camera action | S | 60000 | 21.52 | 0.176 | 215 | – |
| `S_arms/` | EG-SWM | S | 60000 | 22.00 | 0.164 | 186 | 0.70 |
| `S_arms_scene/` | EG-SWM + scene action | S | 60000 | 22.38 | 0.156 | 174 | 0.22 |
| `S_arms_nofwd/` | EG-SWM, diffusion loss only | S | 60000 | 21.65 | 0.172 | 204 | 0.69 |
| `S_arms_poolglobal/` | EG-SWM, arm actions not grounded | S | 60000 | 22.07 | 0.163 | 188 | 0.65 |
| `S_arms_shuffled/` | EG-SWM, wrong arm binding | S | 60000 | 21.98 | 0.165 | 186 | 0.70 |
| `S_arms_ui/` | EG-SWM + console readout | S | 60000 | 21.98 | 0.164 | 184 | 0.70 |
| `S_arms_hold/` | EG-SWM, boxes held across detector gaps | S | 60000 | 22.06 | 0.163 | 185 | 0.65 |

Loading (with the `swm` package from the [code repository](https://github.com/GIND123/Surgical-World-Model)):

```python
import json
from safetensors.torch import load_model
from swm.models.world_model import WMConfig, WorldModel

meta = json.load(open("S_arms/config.json"))
model = WorldModel(WMConfig.from_dict(meta["model"])).eval()
load_model(model, "S_arms/model.safetensors")
```

The SurgWM re-implementation (`surgwm/`) has one file per stage (`tokenizer`, `lam`, `dynamics`), loaded into the classes of `swm.baselines.surgwm` with `SurgWMConfig(**meta["model"])`.

## Training data

* **SurgVU** (Zia et al., *Surgical Visual Understanding (SurgVU) Dataset*, arXiv:2501.09209): 155 da Vinci porcine training-lab cases, 837.8 h, with robot-log instrument and endoscope occupancy per arm.
* For kinematic grounding / benchmarks: Open-H-Embodiment subsets (CC-BY-4.0), JIGSAWS (Gao et al., M2CAI 2014; data-use agreement), SAR-RARP50 (CC BY-NC-SA 4.0).

## Licence and intended use

Weights are released under **CC BY-NC-SA 4.0** for non-commercial research, in line with the licences of the training data. The models simulate endoscopic video; they are **not** medical devices and must not be used for clinical decision-making or to control a robot on a patient.