introvoyz041/cosmos3-nanoh-surgical-dvrk

Cosmos 3 Nano-H surgical dVRK

Cosmos 3 Nano-0H Surgical dVRK is a domain-specific post-trained Cosmos 3 Nano checkpoint for short-horizon surgical kinematics prediction and dVRK motion simulation. It adapts the Nano-scale Cosmos world model to dVRK/da Vinci endoscopic scenes and uses a native dVRK action representation that is compatible with GR00T-H-style dual-arm end-effector control.

The model is intended as a research world model for surgical robotics: predicting and comparing near-future tool motion, recovering action-like kinematic structure from short video clips, validating action interfaces and studying low-latency motion simulation behaviour in visually constrained robotic manipulation settings.

Video is a secondary diagnostic channel in this limited NanoH run, not the main product. The video objective remains much harder than the action/kinematic objectives: generated video quality is substantially weaker than the kinematics outputs even when the rollout is conditioned. Use this checkpoint as a compact surgical motion model, not as a surgical video generator.

It is not intended for clinical decision-making, autonomous surgery, medical diagnosis or patient-facing deployment.

Model details

• Base model: nvidia/Cosmos3-Nano
• Adaptation: supervised post-training on dVRK surgical windows
• Primary task: surgical kinematics prediction and short-horizon dVRK motion simulation
• Secondary task: action-conditioned visual rollout for context and diagnostics
• Prediction horizon: short-horizon 16-frame chunks
• Visual input: single endoscopic camera stream
• Action input: dual-PSM dVRK tensor, relative EEF motion with rot6D
• Training objective: mixed forward dynamics, inverse dynamics and policy
• Precision: bfloat16 training
• Distributed strategy: single-node FSDP
• Checkpoint format: Cosmos DCP checkpoints plus optional HF export

Action space

The model registers dVRK as a first-class Cosmos embodiment rather than mapping it onto an existing robot id.

embodiment: dvrk
domain_id: 22
raw_action_dim: 20
chunk_length: 16

Native dVRK tensor:

actions_dvrk: (16, 20)

left_xyz_rel(3), left_rot6d_rel(6), left_jaw_abs(1),
right_xyz_rel(3), right_rot6d_rel(6), right_jaw_abs(1)

Surgical simulator bridge:

actions_unified44: (16, 44)
actions_unified44[..., :20] == actions_dvrk
actions_unified44[..., 20:44] == 0

Cosmos action head:

actions_cosmos3: (16, 64)
raw_action_dim: 20

The 20D native tensor is padded to the Cosmos 3 64D action head while preserving the raw dimension for action-channel masking. A 44D zero-padded bridge is included for surgical simulator interoperability.

Training data

Training uses chrisvoncsefalvay/openembodiment-dvrk-subset, a convenience subset of nvidia/PhysicalAI-Robotics-Open-H-Embodiment.

The subset mirrors dVRK/da Vinci records and associated metadata so training machines can stream from one stable HF/Xet-backed repository. The underlying source data and credit remain with NVIDIA and the Open-H Embodiment contributors.

Training recipe

The release recipe is designed for a single 8xH100 node:

• FSDP over 8 GPUs
• bfloat16 precision
• activation checkpointing
• 16-frame action chunks
• single-camera video stream
• all-objective sampling across forward dynamics, inverse dynamics and policy
• checkpoint save interval of 100 iterations
• W&B metrics and qualitative visual logging
• interrupt-tolerant DCP checkpoints
• final artefact upload to a private Hugging Face model repository

The recipe demonstrates that useful domain adaptation of Cosmos 3 Nano does not require enterprise-scale cluster choreography. Larger runs remain valuable, but the single-node path is already strong enough to reveal meaningful surgical domain improvement.

Evaluation

Evaluation emphasises short-horizon kinematic consistency and action-space validity. Video metrics are retained as diagnostics, but kinematic and toolpath checks are the primary success criteria for this run:

• tensor validation for dVRK, 44D simulator bridge and Cosmos 64D padded action
• held-out episode splits
• FDS-L1 over short rollouts
• toolpath and end-effector trajectory comparison against ground truth
• inverse-kinematics consistency on held-out motion snippets
• action-space normalization and scale audits
• qualitative W&B panels with context, prediction, actual, error strip and action norms
• GR00T-H interop samples in native, 44D and 64D action formats
• PSNR, lightweight global SSIM and temporal frame-difference L1 as secondary video diagnostics

Latency and smoothness are first-class evaluation concerns. A short rollout that keeps motion coherent and fast is more valuable for this use case than a long rollout that looks impressive only until the simulator has to respond.

Intended use

Appropriate use:

• research on surgical robotics world models
• short-horizon dVRK kinematics and toolpath prediction
• action-space validation and interop work
• simulation prototyping and qualitative model comparison
• non-clinical robotics research demos

Out-of-scope use:

• clinical decision support
• autonomous surgical control
• diagnosis, treatment planning or patient triage
• safety-critical deployment
• replacing a validated simulator or physical robot test

Limitations

• The model is specialised to dVRK-like endoscopic surgical scenes.
• It is optimised for short-horizon consistency, not indefinite rollout length.
• It may hallucinate tool geometry, tissue response, lighting or contact effects.
• Video rollouts are weaker than the kinematics outputs and should be treated as qualitative diagnostics rather than photorealistic surgical video.
• It should not be treated as a calibrated dynamics oracle.
• Single-camera training limits viewpoint disentanglement.
• Dataset coverage reflects the dVRK subset available in Open-H Embodiment.

Safety

This checkpoint is a research artefact. Predicted toolpaths, actions and generated video should be interpreted as learned simulation outputs, not as measured surgical reality. Any downstream robotics use requires independent validation, simulator checks, hardware safety systems and domain-expert review.

Citation

Please cite the upstream Open-H Embodiment dataset and NVIDIA Cosmos materials when using this model or the associated dataset subset. Cite this checkpoint repository for the dVRK action mapping, kinematics-focused training recipe and released weights.

@misc{voncsefalvay2026cosmos3nanohsurgicaldvrk,
  title = {Cosmos 3 NanoH surgical dVRK},
  author = {von Csefalvay, Chris},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face model repository},
  url = {https://huggingface.co/chrisvoncsefalvay/cosmos3-nanoh-surgical-dvrk}
}

Artefacts

Expected repository contents after release:

model/                 exported HF checkpoint
checkpoints/           resumable Cosmos DCP checkpoints
eval/                  quantitative reports and qualitative references
interop/               dVRK, 44D simulator and 64D Cosmos action samples
README.md              this model card

License

Released model materials are licensed under OpenMDW-1.1. Model code and adapter utilities in the source repository remain Apache-2.0. Dataset files retain the terms of the upstream Open-H Embodiment release. Base model usage is subject to the terms of the upstream Cosmos 3 Nano release.