Doppel-8: one video world model across eight embodiments

Model weights for Render the Robot, Generate the World: One Video World Model Across Embodiments (Yash Jangir, Karan Mirakhor, Renos Zabounidis, Muhammad Zubair Irshad, Sergey Zakharov, Yonatan Bisk, Homanga Bharadhwaj; JHU / CMU / TRI). Project page: https://doppelwm.github.io. Code: https://github.com/DoppelWM/Doppel.

Doppel is an action-conditioned video world model. Given the first camera frame, a rendering of the embodiment along a proposed trajectory (seen from the same camera), and the numerical trajectory itself, it predicts the video of what happens: how the arm, hand or body moves and how the objects it touches respond.

This repository holds Doppel-8, the model trained jointly on eight datasets / eight embodiments (robot arms, a bimanual robot, a humanoid with dexterous hands, human hands and human bodies) with one shared action encoder and no embodiment label. The renderer tells the model which body it is looking at.

Backbone Wan 2.1 image-to-video 14B (480P, Diffusers format), all 40 DiT blocks fine-tuned
Conditioning render-action adapter ("REDAR", 30.6M params): render encoder + action encoder + render-to-action cross-attention, injected per video token through the flow-timestep modulation
Checkpoint epoch 2489 of 2490 (189,240 optimizer steps), EMA weights (decay 0.995)
Final validation loss (EMA) 0.0632
Training data 803,602 training clips from 68,748 episodes, about 670 hours of video
Output 17 RGB frames at 240x432

What is in this repository

This repo is an overlay on the public base model: it holds only the tensors that differ from Wan-AI/Wan2.1-I2V-14B-480P-Diffusers (revision b184e23a8a16b20f108f727c902e769e873ffc73), plus the new adapter. Because every DiT block was fine-tuned, the overlay covers 16.34B of the transformer's 16.43B parameters (99.5%); the rest (patch embedding, time/text/image embedders, output scale-shift table) and the VAE, CLIP image encoder, UMT5 text encoder, tokenizer and scheduler config come from the base repo.

README.md                         this card
config.json                       model, input, inference and training settings (no internal paths)
action_padding_layout.json        per-embodiment action layout inside the shared 159-slot vector
action_stats/<embodiment>.json    per-embodiment z-score statistics used in training (8 files)
transformer_overlay/
  model.safetensors.index.json    weight map
  model-0000{1..7}-of-00007.safetensors
SHA256SUMS                        sha256 of every file above
Tensors Count Params Stored dtype
blocks.{0..39}.* attention, FFN, norm2 1,240 16,152.3M bfloat16
blocks.{0..39}.scale_shift_table 40 1.2M float32
condition_embedder.time_proj.* 2 157.3M float32
proj_out.* 2 0.3M bfloat16
action_adapter_v4.* (render-action adapter) 72 30.6M float32
Total 1,356 16,341.7M 33.06 GB

Precision. Training kept fp32 master weights and an fp32 EMA; the model runs in bf16 with a small set of modules kept in fp32 (time_proj, scale_shift_table, the adapter, plus base time_embedder and norm_out). Each tensor here is stored in the dtype it has at inference, so loading this overlay gives a model that is bit-identical to the one used for every number in the paper (see Verification). The fp32 EMA master weights (65.4 GB) are not included.

The adapter keys use the training-code name action_adapter_v4.*; the doppel code loads them into render_action_adapter (same sub-module names) automatically.

The auxiliary point-track head (tracks_head.*) of the architecture was disabled in this run (loss weight 0); it is not shipped and is never used for video generation.

Training data

One model, eight datasets, sampled in proportion to their training clips (about 60 passes over each).

Dataset Embodiment (what the renderer draws) Action Train episodes Train clips Test shift
DROID Franka FR3 + 2F-85 8-D: 7 joints + gripper 16,726 83,511 unseen lab
BridgeData V2 WidowX 250s 8-D: 6 joints + gripper 12,387 61,935 unseen kitchen
RoboMIND UR5e + Robotiq 2F-85 8-D: 6 joints + gripper 14,801 74,005 unseen objects
ABC-130K bimanual YAM 14-D: 2 x (6 joints + gripper) 9,115 274,523 unseen tasks
T-Rex Vega-1 humanoid + Sharpa hands 58-D: 2 x (7 arm + 22 hand joints) 4,784 58,120 unseen backgrounds
Ego-Exo4D human body (SMPL-H) 159-D body + hands axis-angle 2,041 213,292 unseen tasks
GigaHands two human hands (MANO) 108-D: 2 x (wrist 6-D rot + xyz + 15 joints) 8,491 31,218 unseen activities
OakInk2 human hands (SMPL-X) 159-D (42 constant slots zeroed) 403 6,998 unseen tasks
All eight 68,748 803,602

Renderings: robots are drawn with per-robot Dr. Robot Gaussian-splat models along the logged joint trajectory; humans and hands from their fitted SMPL-H / SMPL-X / MANO models, all from the calibrated camera of the clip.

Training data licenses

These weights are released under CC BY-NC 4.0 (non-commercial). They were trained on data and body models with non-commercial or share-alike terms (GigaHands, OakInk2, and the MPI SMPL-H / SMPL-X / MANO models used for the human renderings), so the weights carry a non-commercial licence. The base model Wan 2.1 is Apache-2.0. Using the human embodiments also requires your own SMPL-H / SMPL-X / MANO licence from MPI; no body-model files are included here.

Dataset License / terms
DROID CC BY 4.0
BridgeData V2 CC BY 4.0
RoboMIND Apache-2.0 (gated on Hugging Face)
ABC-130K Apache-2.0 (gated on Hugging Face)
T-Rex MIT
Ego-Exo4D Ego-Exo4D license agreement (video); Ego-Exo4D Human Meshes CC BY 4.0
GigaHands CC BY-NC 4.0
OakInk2 CC BY-SA 4.0

No training videos or frames are included in this repository.

Inputs and outputs

  • First frame: one RGB image, resized (not aspect-preserving) to 240x432. Used as the I2V conditioning frame and CLIP-encoded.
  • Rendering: 17 frames of the embodiment drawn along the trajectory from the same camera, black background, 240x432.
  • Actions: (17, 159) float32. For an embodiment with native width D: take its per-frame state, unwrap the root axis-angle over the clip (SMPL-X / SMPL-H only), z-score with action_stats/<embodiment>.json ((a - mean) / std), set the masked slots to 0, then zero-extend to 159 slots. The exact layout of each embodiment is in action_padding_layout.json. No embodiment id is passed.
  • Prompt: empty string. Text prompts were ignored in training.
  • Output: 17 RGB frames, 240x432. Frame 0 reproduces the input frame; frames 1..16 are the prediction. The 17 frames are spread evenly over the clip window, so the time step depends on the clip length and the dataset's native rate (5 to 30 Hz).

Inference settings used for every reported number: 50 flow-matching Euler steps (base scheduler config), guidance scale 1.0 (no CFG), max_sequence_length=512, bf16 autocast, VAE tiling and slicing on. One 80 GB A100 runs the full pipeline without offloading, about 30 s per clip.

Usage

With the code from github.com/DoppelWM/Doppel:

hf download Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --revision b184e23a8a16b20f108f727c902e769e873ffc73 \
    --local-dir checkpoints/wan2.1-i2v-14b-480p
hf download sourlreapwr/doppel-emb8 --local-dir checkpoints/doppel-emb8
from doppel.inference import DoppelWorldModel

wm = DoppelWorldModel.from_pretrained(
    checkpoint="checkpoints/doppel-emb8",
    config="checkpoints/doppel-emb8/config.json",
    model_path="checkpoints/wan2.1-i2v-14b-480p")
video = wm.predict(image="frame0.png", render="render.mp4", actions=actions, robot_type="panda", seed=0)
# uint8 (17, 240, 432, 3); frame 0 is the input frame

actions are the raw per-frame states, shape (17, D); see the repository README for the layout of each embodiment.

Results

Numbers are averages over frames 1-16 on 200 clips per split, scored at this checkpoint.

Robot datasets, validation (unseen trajectories) / test (held-out shift):

Dataset Val PSNR Val SSIM Val LPIPS Test PSNR Test SSIM Test LPIPS
DROID 24.82 0.870 0.062 21.46 0.806 0.108
BridgeData V2 26.54 0.873 0.049 22.24 0.784 0.099
RoboMIND 31.30 0.930 0.036 30.66 0.921 0.038

On DROID and RoboMIND this one model matches or beats the per-dataset render+action model and the three-robot model on every image metric in both splits; on Bridge it is within 0.08 dB of the best model on validation and second on the test split, behind the three-robot model (22.47 dB). Against an action-only model trained on the same dataset it is +1.9 / +0.3 / +0.8 dB on validation (DROID / Bridge / RoboMIND) and +1.4 / +1.7 / +0.6 dB on test.

The five other datasets (same weights):

Dataset Val PSNR Val SSIM Val LPIPS Test PSNR Test SSIM Test LPIPS
ABC-130K 24.65 0.850 0.096 25.22 0.861 0.089
T-Rex 28.33 0.871 0.050 28.35 0.867 0.051
Ego-Exo4D 27.94 0.895 0.045 27.61 0.892 0.044
GigaHands 25.62 0.885 0.053 25.78 0.897 0.049
OakInk2 29.01 0.918 0.036 29.09 0.921 0.033

Intended use

  • Research on action-conditioned video prediction, world models and policy evaluation for robot manipulation and human motion, in settings like the eight training datasets.
  • Predicting short (17-frame) futures from a real first frame and a trajectory you can render: a robot with a calibrated camera and a renderable model, or a human body / hand pose sequence.

Limitations

  • Short horizon, low resolution: 17 frames at 240x432. Longer rollouts need chaining, which this model was not trained for.
  • Needs a calibrated rendering: the rendering must be aligned with the camera of the first frame. Misaligned renders (wrong extrinsics, wrong robot model) degrade predictions; the model has no other way to know the body.
  • Embodiments outside training are untested. The action layout is fixed per embodiment (action_padding_layout.json); a new robot needs its own statistics and was never seen by the action encoder.
  • Distribution shift: test-split scores (new labs, kitchens, objects, tasks) are lower than validation. On the Bridge held-out kitchen, scores fell slightly late in training (22.57 dB at epoch 1229, 22.24 at 2489) while in-distribution Bridge improved.
  • Physics is learned, not simulated: object motion can be implausible, objects can morph, appear or vanish, and the camera can drift, especially for in-the-wild video unlike the training data.
  • Not for safety-critical decisions: predictions are samples from a generative model, not measurements.
  • People in the data: three datasets show real people. Do not use the model to generate misleading video of identifiable people.

License

Citation

@article{doppel2026,
  title  = {Render the Robot, Generate the World: One Video World Model Across Embodiments},
  author = {Jangir, Yash and Mirakhor, Karan and Zabounidis, Renos and Irshad, Muhammad Zubair and
            Zakharov, Sergey and Bisk, Yonatan and Bharadhwaj, Homanga},
  year   = {2026}
}

Verification

Checked before release (2026-11-05) against the original training checkpoint (EMA, fp32, 65.4 GB, sha256 42803a1aa2dc5c09eeb30bac1bf55e196ae4bc8762b08912e76cab40b330bc48):

  • Weights: the evaluator's model was built from the base transformer and the original checkpoint was loaded the way every paper number was produced. All 1,356 tensors in this overlay are bit-identical to the parameters that model held. The 114 fp32 tensors (adapter, time_proj, scale_shift_table) are also bit-identical to the fp32 checkpoint. The 1,242 bf16 tensors differ from the fp32 masters only by bf16 rounding: max relative error 3.9e-3 (2^-8), relative RMS error 1.5e-3 to 1.8e-3 per tensor group.
  • End to end: one evaluation clip (DROID, seed 374915573), 50 steps on one A100, scored with the paper's evaluator: PSNR 28.5735 dB with this overlay vs 28.5666 dB with the original checkpoint in the same session vs 28.5727 dB stored from the paper run. SSIM, LPIPS and SCS agree to 7e-5, 7e-5 and 4e-4. The +-0.007 dB spread is GPU run-to-run nondeterminism: two runs of the original checkpoint differ by the same amount.
Downloads last month
20
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for sourlreapwr/doppel-emb8

Finetuned
(8)
this model

Space using sourlreapwr/doppel-emb8 1

Paper for sourlreapwr/doppel-emb8