GOVINDFROM/surgical-world-model

Surgical World Model (EG-SWM)

Embodiment-grounded world models of robot-assisted surgery: an action-conditioned video world model whose action space mirrors the robot — an explicit endoscope ego-motion action plus one latent action per instrument arm, learned from unlabeled in-vivo da Vinci video and bound to the arms through the robot's own system logs.

Code, data pipeline, documentation and diagrams: https://github.com/GIND123/Surgical-World-Model

Checkpoints

One folder per finished run: model.safetensors (EMA weights), config.json (model and training config, step) and the evaluation summaries (replay.json: fidelity; control.json: endoscope control, locality, embodiment consistency; action_use.json: action-use diagnostics). Metrics on the held-out SurgVU test cases, 2.8 s horizon, action replay (protocol).

Folder	Model	Size	Steps	PSNR ↑	LPIPS ↓	FVD ↓	Endoscope gain ↑
S_none/	None (unconditional)	S	60000	18.96	0.249	422	–
S_state_nocam/	Robot state	S	60000	19.29	0.237	358	–
S_state/	Robot state + camera	S	60000	20.44	0.206	266	0.79
S_global_nocam/	Monolithic latent (Genie)	S	60000	22.49	0.154	172	–
S_global/	Monolithic latent + camera	S	60000	22.47	0.154	180	0.16
S_global_vq/	Monolithic VQ + camera (SurgWM-style)	S	60000	21.30	0.183	206	0.81
S_slots/	Slots + camera (FLAM-style)	S	60000	22.77	0.149	169	0.05
S_arms_nocam/	EG-SWM, no camera action	S	60000	21.52	0.176	215	–
S_arms/	EG-SWM	S	60000	22.00	0.164	186	0.70
S_arms_scene/	EG-SWM + scene action	S	60000	22.38	0.156	174	0.22
S_arms_nofwd/	EG-SWM, diffusion loss only	S	60000	21.65	0.172	204	0.69
S_arms_poolglobal/	EG-SWM, arm actions not grounded	S	60000	22.07	0.163	188	0.65
S_arms_shuffled/	EG-SWM, wrong arm binding	S	60000	21.98	0.165	186	0.70
S_arms_ui/	EG-SWM + console readout	S	60000	21.98	0.164	184	0.70

Loading (with the swm package from the code repository):

import json
from safetensors.torch import load_model
from swm.models.world_model import WMConfig, WorldModel

meta = json.load(open("S_arms/config.json"))
model = WorldModel(WMConfig.from_dict(meta["model"])).eval()
load_model(model, "S_arms/model.safetensors")

The SurgWM re-implementation (surgwm/) has one file per stage (tokenizer, lam, dynamics), loaded into the classes of swm.baselines.surgwm with SurgWMConfig(**meta["model"]).

Training data

• SurgVU (Zia et al., Surgical Visual Understanding (SurgVU) Dataset, arXiv:2501.09209): 155 da Vinci porcine training-lab cases, 837.8 h, with robot-log instrument and endoscope occupancy per arm.
• For kinematic grounding / benchmarks: Open-H-Embodiment subsets (CC-BY-4.0), JIGSAWS (Gao et al., M2CAI 2014; data-use agreement), SAR-RARP50 (CC BY-NC-SA 4.0).

Licence and intended use

Weights are released under CC BY-NC-SA 4.0 for non-commercial research, in line with the licences of the training data. The models simulate endoscopic video; they are not medical devices and must not be used for clinical decision-making or to control a robot on a patient.