README.md

---
title: SyncWorld Simulator
emoji: 🦾
colorFrom: pink
colorTo: red
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
short_description: Visually-calibrated zero-shot robot world simulator
python_version: "3.12"
startup_duration_timeout: 1h
models:
  - yyuncong/SyncWorld
datasets:
  - yyuncong/SyncWorld-Evaluation
---

# 🦾 SyncWorld — visual calibration turns a world model into a zero-shot simulator

Demo for **[SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators](https://huggingface.co/papers/2609.09155)**
(UMass Embodied AGI) — [code](https://github.com/UMass-Embodied-AGI/SyncWorld) ·
[weights](https://huggingface.co/yyuncong/SyncWorld).

A pretrained action-conditioned video world model cannot be dropped into a new scene, because it
does not know how *this* camera sees *this* robot's action space. SyncWorld's fix is **visual
calibration**: prepend a short clip of the arm sweeping each degree of freedom, paired with the
actions that produced it, as extra fully-conditioning context. The model infers the unseen
camera↔robot mapping in-context and becomes a usable forward-dynamics simulator — with no
per-scene finetuning.

## What this Space does

Pick one of the authors' calibrated evaluation episodes, then either

- **replay the robot's own actions** — re-simulate the expert trajectory and compare against ground
  truth, or
- **drive the arm yourself** — command a constant per-step end-effector twist
  (Δx/Δy/Δz in cm/step, Δroll/Δpitch/Δyaw in °/step, plus the gripper) and watch the model render
  a counterfactual rollout.

Each press generates one or more autoregressive 16-frame windows at 512 px, closed-loop (each
window conditions on the previously *generated* frames).

## Inference path

Ported 1:1 from the authors' reference script `examples/eval_gripperhead_fdm_rollout.py`, keeping
the trained recipe exactly:

| | |
|---|---|
| Conditioning items | `[6 per-DoF calibration segments × 5 frames] + [25 sparse history frames @ stride 3] + [current frame + 16 future frames]` |
| Actions | 7-D `[Δpos_cm(3), Δeuler_xyz_deg(3), gripper(1)]`, `backward_framewise` convention, translation scale 100.0, rotation scale 57.2958 |
| Resolution / fps | 512 px square, conditioning fps 15 |
| Sampler | rectified flow, UniPC, 20 steps (adjustable) |
| Caption | the FDM training constant `"A robot arm interacts with the scene."` |
| Backbone | 16B Cosmos-3 mixture-of-transformers (Qwen3-VL-8B understanding expert + diffusion expert), Wan2.2 4×16×16 video VAE |

Deviations forced by the target hardware: single-process inference (`parallel_dims=None`),
`torch.compile` disabled (ZeroGPU forks a fresh worker per call), the audio expert disabled
(unused by this checkpoint), and the VLM processor served from a local copy of the seven tokenizer
files in `nvidia/Cosmos3-Nano` instead of downloading that repo's 35 GB of weights.

## Bundled scenes

Ten episodes (5 ManiSkill + 5 LIBERO) copied verbatim from the authors' own evaluation set,
[`yyuncong/SyncWorld-Evaluation`](https://huggingface.co/datasets/yyuncong/SyncWorld-Evaluation),
each with its paired calibration sweep and pose track. Both the code and the data are released
under **OpenMDW-1.1**; see `ATTRIBUTIONS.md` / `NOTICE` in the upstream repo.

## Citation

```bibtex
@article{syncworld2026,
  title  = {SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators},
  author = {UMass Embodied AGI},
  year   = {2026},
  url    = {https://huggingface.co/papers/2609.09155}
}
```