Image-to-Video
1zeryu commited on
Commit
60dc1b8
Β·
verified Β·
1 Parent(s): 40d497d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +186 -0
README.md CHANGED
@@ -1,3 +1,189 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+ # Uranus
5
+
6
+ <!---
7
+ Copyright 2026 - The D-Robotics Large Model Team. All rights reserved.
8
+
9
+ Licensed under the Apache License, Version 2.0 (the "License");
10
+ you may not use this file except in compliance with the License.
11
+ You may obtain a copy of the License at
12
+
13
+ http://www.apache.org/licenses/LICENSE-2.0
14
+
15
+ Unless required by applicable law or agreed to in writing, software
16
+ distributed under the License is distributed on an "AS IS" BASIS,
17
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18
+ See the License for the specific language governing permissions and
19
+ limitations under the License.
20
+ -->
21
+
22
+ <p align="center">
23
+ <img src="https://github.com/D-Robotics-AI-Lab/Uranus-OSS/blob/main/assets/figures/logo.png" width="400"/>
24
+ </p>
25
+ <p align="center">
26
+ <a href="https://huggingface.co/collections/D-Robotics/uranus"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-yellow"></a>
27
+ <a href="https://www.modelscope.cn/collections/D-Robotics/Uranus"><img alt="ModelScope" src="https://img.shields.io/badge/%F0%9F%A4%96%20ModelScope-8A2BE2"></a>
28
+ <a href="https://huggingface.co/datasets/D-Robotics/Uranus-Demo-Data"><img alt="Test Samples" src="https://img.shields.io/badge/%F0%9F%A7%AA%20Test%20Samples-00B4D8"></a>
29
+ <!-- <a href="TODO_TECHNICAL_REPORT_URL"><img alt="Technical Report" src="https://img.shields.io/badge/Technical_Report-B31B1B?logo=arxiv&logoColor=white"></a> -->
30
+ <a href="https://d-robotics-ai-lab.github.io/large-model-team/blog/uranus/"><img alt="Blog" src="https://img.shields.io/badge/Blog-FF7A00?logo=githubpages&logoColor=white"></a>
31
+ <a href="https://github.com/D-Robotics-AI-Lab/Uranus-OSS/raw/main/assets/figures/wechat.jpg"><img alt="WeChat" src="https://img.shields.io/badge/WeChat-07C160?logo=wechat&logoColor=white"></a>
32
+ <a href="https://github.com/D-Robotics-AI-Lab/Uranus-OSS"><img alt="GitHub" src="https://img.shields.io/badge/OSS_Code-0077FF.svg?logo=github&logoColor=white"></a>
33
+ <a href="https://github.com/D-Robotics-AI-Lab/Uranus-SDK"><img alt="GitHub" src="https://img.shields.io/badge/SDK_Code-0077FF.svg?logo=github&logoColor=white"></a>
34
+ </p>
35
+
36
+ -----
37
+
38
+ In this repository, we present **Uranus**, a data-driven robot simulator built around a joint-trajectoryconditioned autoregressive diffusion model. Uranus offers three key capabilities:
39
+
40
+ - **streaming, open-ended rollout**, which receives future joint-position trajectories online and autoregressively
41
+ generates one latent frame per step, corresponding to four RGB frames, without a fixed horizon
42
+ - **low-latency generation**, achieving 24 FPS after inference optimization
43
+ - **scalable, extensible robot control**, providing a unified interface for synchronized multi-view generation
44
+ across diverse robot embodiments and camera configurations.
45
+
46
+ More technical details can be found in our [technical report](TODO_TECHNICAL_REPORT_URL), and visualization demos are available on our [blog](https://d-robotics-ai-lab.github.io/large-model-team/blog/uranus/).
47
+
48
+ ## Quickstart
49
+
50
+ #### Installation
51
+
52
+ Install uv
53
+
54
+ ```bash
55
+ curl -LsSf https://astral.sh/uv/install.sh | sh
56
+ ```
57
+
58
+ Create the environment and install dependencies
59
+
60
+ ```bash
61
+ git clone https://github.com/D-Robotics-AI-Lab/Uranus-OSS.git
62
+ cd Uranus-OSS
63
+ uv sync
64
+ ```
65
+
66
+ All commands in this repo are run through `uv run python <cmd>`, which resolves `.venv/` automatically without manual activation.
67
+
68
+ #### Download model weights
69
+
70
+ | Models | Download Link | Notes |
71
+ |--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
72
+ | Uranus-1.3B | πŸ€— [Huggingface](https://huggingface.co/D-Robotics/Uranus-1.3B) πŸ€– [ModelScope](https://www.modelscope.cn/models/D-Robotics/Uranus-1.3B) | SFT model, 384Γ—640, 25 inference steps |
73
+ | Uranus-1.3B-Distillation | πŸ€— [Huggingface](https://huggingface.co/D-Robotics/Uranus-1.3B-Distillation) πŸ€– [ModelScope](https://www.modelscope.cn/models/D-Robotics/Uranus-1.3B-Distillation) | Distilled model, 384Γ—640, 4 inference steps |
74
+
75
+
76
+ You can download the weights with the Hugging Face CLI (`pip install -U huggingface_hub` if you don't have it):
77
+
78
+ ```bash
79
+ # SFT model (17.5 GB) into ./weights/uranus-1.3b
80
+ hf download D-Robotics/Uranus-1.3B --local-dir ./weights/uranus-1.3b
81
+
82
+ # Distilled model (17.5 GB) into ./weights/uranus-1.3b-distillation
83
+ hf download D-Robotics/Uranus-1.3B-Distillation --local-dir ./weights/uranus-1.3b-distillation
84
+ ```
85
+
86
+ For gated repos, set `HF_TOKEN` first (`hf auth login` or `export HF_TOKEN=...`). The same repos are mirrored on ModelScope β€” use `modelscope download D-Robotics/Uranus-1.3B --local-dir ./weights/uranus-1.3b`.
87
+
88
+
89
+ Uranus consumes converted weights from a single directory. Expected layout:
90
+
91
+ ```
92
+ weights_dir/
93
+ β”œβ”€β”€ dit.pt # SpatialTemporalWanModel state_dict
94
+ β”œβ”€β”€ vae.pt # WanVideoVAE state_dict (loaded strict=False)
95
+ β”œβ”€β”€ text_encoder.pt # WanTextEncoder (T5) state_dict
96
+ β”œβ”€β”€ plucker_adapter.pt # PlΓΌcker-conditioning adapter
97
+ β”œβ”€β”€ vace_patch_embedding.pt # skeleton controller patch embedding
98
+ β”œβ”€β”€ tokenizer/ # HuggingFace tokenizer directory
99
+ └── metadata.json # model + inference metadata
100
+ ```
101
+
102
+ #### Download test samples
103
+
104
+ We provide a set of ready-to-run test samples covering a variety of robot embodiments (ALOHA, ARX5, UR5, Franka, G1, DOS-W1, X5) and data sources (AgiBot World, DROID, RC-Table). Download and unpack them from [Huggingface](https://huggingface.co/datasets/D-Robotics/Uranus-Data) into `examples/data/`.
105
+
106
+ ```bash
107
+ hf download D-Robotics/Uranus-Demo-Data --repo-type dataset --local-dir ./examples/data
108
+ ```
109
+
110
+ Each episode lands in `./examples/data/<episode_id>/` and can be passed directly to `main.py --sample-dir`.
111
+
112
+ Each sample is a self-contained "XML-environment" directory:
113
+
114
+ ```
115
+ sample_dir/
116
+ β”œβ”€β”€ meta.json # prompt, cameras, mjcf_path, end_effectors, skeleton, fps
117
+ β”œβ”€β”€ temporal.json # step_qpos: [{state, robot2world_transform}, ...]
118
+ β”œβ”€β”€ mjcf/
119
+ β”‚ └── <robot>.xml # robot MJCF with baked-in camera calibration & mounts
120
+ β”œβ”€β”€ ref_images/
121
+ β”‚ └── <camera>.png # one reference image per camera
122
+ └── gt/
123
+ └── <camera>.mp4 # ground-truth video per camera
124
+ ```
125
+
126
+ Your own robot data can be converted into this same format and fed to inference as well, as long as it provides the per-frame `qpos`, the robot MJCF with calibrated cameras, and one reference image per camera.
127
+
128
+
129
+ #### Run inference
130
+
131
+ Run generation on a test sample with the pretrained model:
132
+
133
+ ```bash
134
+ uv run python main.py \
135
+ --weights-dir <weights_dir> \
136
+ --sample-dir examples/data/000000 \
137
+ --output-dir ./output
138
+ ```
139
+
140
+ **Flag reference**
141
+
142
+ | Flag | Required | Description |
143
+ |------|----------|-------------|
144
+ | `--weights-dir` | yes | Path to the converted weights directory (see [Download model weights](#download-model-weights)). Must contain `metadata.json`. |
145
+ | `--sample-dir` | yes | Path to a single XML-environment sample (see [Download test samples](#download-test-samples)). |
146
+ | `--output-dir` | no | Root directory for generated videos. Defaults to `./output`. |
147
+ | `--num-chunks` | no | Number of autoregressive chunks to roll out. Omit to automatically use every frame in `temporal.json` (trailing partial chunk is padded with the last frame, then trimmed back to the original length in the output). |
148
+
149
+ **Hyperparameters are auto-resolved from `metadata.json`**
150
+
151
+ `num_inference_steps`, `step_length`, `default_height`, `default_width`, and `teacher_forcing_window_size` are read from the checkpoint's `metadata.json` automatically β€” no extra flags are needed. Any flag passed explicitly on the CLI takes precedence over the metadata. For example, the SFT model defaults to 25 denoising steps, while the distilled model defaults to 4.
152
+
153
+ **Outputs**
154
+
155
+ Each run writes four aligned video streams per camera under `--output-dir`:
156
+
157
+ - `gen/` β€” model-generated video
158
+ - `gt/` β€” ground-truth video (copied from the sample for comparison)
159
+ - `skeleton/` β€” rendered MuJoCo skeleton visualization
160
+ - `plucker/` β€” PlΓΌcker-coordinate conditioning visualization (force RGB + ray-direction RGB)
161
+
162
+ Plus a `preview.mp4` that stacks all four streams side by side for quick qualitative comparison.
163
+
164
+
165
+ ## License
166
+
167
+ This project is released under the [Apache License 2.0](LICENSE). By using, distributing, or contributing to this repository, you agree to the terms and conditions of the license.
168
+
169
+
170
+ ## Citation
171
+
172
+ 🌟 If you find our work helpful, please leave us a star and cite our paper.
173
+
174
+ ```
175
+ @article{drobotics2026uranus,
176
+ author = {D-Robotics Large Model Team},
177
+ title = {Uranus: Building the Next-Generation Simulation Infrastructure for Embodied AI},
178
+ journal = {D-Robotics AI Blog},
179
+ year = {2026},
180
+ }
181
+ ```
182
+
183
+ ## Contact Us
184
+
185
+ If you would like to leave a message to our research or product teams, feel free to join our [WeChat](https://github.com/D-Robotics-AI-Lab/Uranus-OSS/raw/main/assets/figures/wechat.jpg) groups!
186
+
187
+ <p align="center">
188
+ <img src="https://github.com/D-Robotics-AI-Lab/Uranus-OSS/blob/main/assets/figures/company.svg" alt="D-Robotics" width="400"/>
189
+ </p>