Model card: checkpoint table

2b3cdf6 verified about 5 hours ago

3.91 kB

---
license: cc-by-nc-sa-4.0
library_name: pytorch
pipeline_tag: image-to-video
tags:
  - surgical-robotics
  - world-model
  - video-prediction
  - medical
  - da-vinci
datasets:
  - nvidia/PhysicalAI-Robotics-Open-H-Embodiment
---
# Surgical World Model (EG-SWM)

Embodiment-grounded world models of robot-assisted surgery: an action-conditioned video world model whose action space mirrors the robot — an explicit **endoscope ego-motion** action plus one latent action per **instrument arm**, learned from unlabeled in-vivo da Vinci video and bound to the arms through the robot's own system logs.

Code, data pipeline, documentation and diagrams: <https://github.com/GIND123/Surgical-World-Model>

![architecture](architecture.png)

## Checkpoints

One folder per finished run: `model.safetensors` (EMA weights), `config.json` (model and training config, step) and the evaluation summaries (`replay.json`: fidelity; `control.json`: endoscope control, locality, embodiment consistency; `action_use.json`: action-use diagnostics). Metrics on the held-out SurgVU test cases, 2.8 s horizon, action replay ([protocol](https://github.com/GIND123/Surgical-World-Model#experiments-and-evaluation-protocol)).

| Folder | Model | Size | Steps | PSNR ↑ | LPIPS ↓ | FVD ↓ | Endoscope gain ↑ |
|---|---|---|---:|---:|---:|---:|---:|
| `S_none/` | None (unconditional) | S | 60000 | 18.96 | 0.249 | 422 | – |
| `S_state_nocam/` | Robot state | S | 60000 | 19.29 | 0.237 | 358 | – |
| `S_state/` | Robot state + camera | S | 60000 | 20.44 | 0.206 | 266 | 0.79 |
| `S_global_nocam/` | Monolithic latent (Genie) | S | 60000 | 22.49 | 0.154 | 172 | – |
| `S_global/` | Monolithic latent + camera | S | 60000 | 22.47 | 0.154 | 180 | 0.16 |
| `S_global_vq/` | Monolithic VQ + camera (SurgWM-style) | S | 60000 | 21.30 | 0.183 | 206 | 0.81 |
| `S_slots/` | Slots + camera (FLAM-style) | S | 60000 | 22.77 | 0.149 | 169 | 0.05 |
| `S_arms_nocam/` | EG-SWM, no camera action | S | 60000 | 21.52 | 0.176 | 215 | – |
| `S_arms/` | EG-SWM | S | 60000 | 22.00 | 0.164 | 186 | 0.70 |
| `S_arms_scene/` | EG-SWM + scene action | S | 60000 | 22.38 | 0.156 | 174 | 0.22 |
| `S_arms_nofwd/` | EG-SWM, diffusion loss only | S | 60000 | 21.65 | 0.172 | 204 | 0.69 |
| `S_arms_poolglobal/` | EG-SWM, arm actions not grounded | S | 60000 | 22.07 | 0.163 | 188 | 0.65 |
| `S_arms_shuffled/` | EG-SWM, wrong arm binding | S | 60000 | 21.98 | 0.165 | 186 | 0.70 |
| `S_arms_ui/` | EG-SWM + console readout | S | 60000 | 21.98 | 0.164 | 184 | 0.70 |
| `S_arms_hold/` | EG-SWM, boxes held across detector gaps | S | 60000 | 22.06 | 0.163 | 185 | 0.65 |

Loading (with the `swm` package from the [code repository](https://github.com/GIND123/Surgical-World-Model)):

```python
import json
from safetensors.torch import load_model
from swm.models.world_model import WMConfig, WorldModel
meta = json.load(open("S_arms/config.json"))
model = WorldModel(WMConfig.from_dict(meta["model"])).eval()
load_model(model, "S_arms/model.safetensors")
```

The SurgWM re-implementation (`surgwm/`) has one file per stage (`tokenizer`, `lam`, `dynamics`), loaded into the classes of `swm.baselines.surgwm` with `SurgWMConfig(**meta["model"])`.

## Training data

* **SurgVU** (Zia et al., *Surgical Visual Understanding (SurgVU) Dataset*, arXiv:2501.09209): 155 da Vinci porcine training-lab cases, 837.8 h, with robot-log instrument and endoscope occupancy per arm.
* For kinematic grounding / benchmarks: Open-H-Embodiment subsets (CC-BY-4.0), JIGSAWS (Gao et al., M2CAI 2014; data-use agreement), SAR-RARP50 (CC BY-NC-SA 4.0).

## Licence and intended use

Weights are released under **CC BY-NC-SA 4.0** for non-commercial research, in line with the licences of the training data. The models simulate endoscopic video; they are **not** medical devices and must not be used for clinical decision-making or to control a robot on a patient.

	---
	license: cc-by-nc-sa-4.0
	library_name: pytorch
	pipeline_tag: image-to-video
	tags:
	- surgical-robotics
	- world-model
	- video-prediction
	- medical
	- da-vinci
	datasets:
	- nvidia/PhysicalAI-Robotics-Open-H-Embodiment
	---

	# Surgical World Model (EG-SWM)

	Embodiment-grounded world models of robot-assisted surgery: an action-conditioned video world model whose action space mirrors the robot — an explicit endoscope ego-motion action plus one latent action per instrument arm, learned from unlabeled in-vivo da Vinci video and bound to the arms through the robot's own system logs.

	Code, data pipeline, documentation and diagrams: <https://github.com/GIND123/Surgical-World-Model>

	![architecture](architecture.png)

	## Checkpoints

	One folder per finished run: `model.safetensors` (EMA weights), `config.json` (model and training config, step) and the evaluation summaries (`replay.json`: fidelity; `control.json`: endoscope control, locality, embodiment consistency; `action_use.json`: action-use diagnostics). Metrics on the held-out SurgVU test cases, 2.8 s horizon, action replay ([protocol](https://github.com/GIND123/Surgical-World-Model#experiments-and-evaluation-protocol)).

	\| Folder \| Model \| Size \| Steps \| PSNR ↑ \| LPIPS ↓ \| FVD ↓ \| Endoscope gain ↑ \|
	\|---\|---\|---\|---:\|---:\|---:\|---:\|---:\|
	\| `S_none/` \| None (unconditional) \| S \| 60000 \| 18.96 \| 0.249 \| 422 \| – \|
	\| `S_state_nocam/` \| Robot state \| S \| 60000 \| 19.29 \| 0.237 \| 358 \| – \|
	\| `S_state/` \| Robot state + camera \| S \| 60000 \| 20.44 \| 0.206 \| 266 \| 0.79 \|
	\| `S_global_nocam/` \| Monolithic latent (Genie) \| S \| 60000 \| 22.49 \| 0.154 \| 172 \| – \|
	\| `S_global/` \| Monolithic latent + camera \| S \| 60000 \| 22.47 \| 0.154 \| 180 \| 0.16 \|
	\| `S_global_vq/` \| Monolithic VQ + camera (SurgWM-style) \| S \| 60000 \| 21.30 \| 0.183 \| 206 \| 0.81 \|
	\| `S_slots/` \| Slots + camera (FLAM-style) \| S \| 60000 \| 22.77 \| 0.149 \| 169 \| 0.05 \|
	\| `S_arms_nocam/` \| EG-SWM, no camera action \| S \| 60000 \| 21.52 \| 0.176 \| 215 \| – \|
	\| `S_arms/` \| EG-SWM \| S \| 60000 \| 22.00 \| 0.164 \| 186 \| 0.70 \|
	\| `S_arms_scene/` \| EG-SWM + scene action \| S \| 60000 \| 22.38 \| 0.156 \| 174 \| 0.22 \|
	\| `S_arms_nofwd/` \| EG-SWM, diffusion loss only \| S \| 60000 \| 21.65 \| 0.172 \| 204 \| 0.69 \|
	\| `S_arms_poolglobal/` \| EG-SWM, arm actions not grounded \| S \| 60000 \| 22.07 \| 0.163 \| 188 \| 0.65 \|
	\| `S_arms_shuffled/` \| EG-SWM, wrong arm binding \| S \| 60000 \| 21.98 \| 0.165 \| 186 \| 0.70 \|
	\| `S_arms_ui/` \| EG-SWM + console readout \| S \| 60000 \| 21.98 \| 0.164 \| 184 \| 0.70 \|
	\| `S_arms_hold/` \| EG-SWM, boxes held across detector gaps \| S \| 60000 \| 22.06 \| 0.163 \| 185 \| 0.65 \|

	Loading (with the `swm` package from the [code repository](https://github.com/GIND123/Surgical-World-Model)):

	```python
	import json
	from safetensors.torch import load_model
	from swm.models.world_model import WMConfig, WorldModel

	meta = json.load(open("S_arms/config.json"))
	model = WorldModel(WMConfig.from_dict(meta["model"])).eval()
	load_model(model, "S_arms/model.safetensors")
	```

	The SurgWM re-implementation (`surgwm/`) has one file per stage (`tokenizer`, `lam`, `dynamics`), loaded into the classes of `swm.baselines.surgwm` with `SurgWMConfig(**meta["model"])`.

	## Training data

	* SurgVU (Zia et al., Surgical Visual Understanding (SurgVU) Dataset, arXiv:2501.09209): 155 da Vinci porcine training-lab cases, 837.8 h, with robot-log instrument and endoscope occupancy per arm.
	* For kinematic grounding / benchmarks: Open-H-Embodiment subsets (CC-BY-4.0), JIGSAWS (Gao et al., M2CAI 2014; data-use agreement), SAR-RARP50 (CC BY-NC-SA 4.0).

	## Licence and intended use

	Weights are released under CC BY-NC-SA 4.0 for non-commercial research, in line with the licences of the training data. The models simulate endoscopic video; they are not medical devices and must not be used for clinical decision-making or to control a robot on a patient.