Image-to-Video
Diffusers
Safetensors
Wan2.2
astronex-world
world-model
video-generation
text-to-video
diffusion
video-diffusion-transformer
interactive
real-time
camera-control
action-conditioned
embodied-ai
robotics
autonomous-driving
Instructions to use Astronex-Lab/Astronex-World with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Wan2.2
How to use Astronex-Lab/Astronex-World with Wan2.2:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Trim the model card to capabilities, benchmarks and usage
Browse files
README.md
CHANGED
|
@@ -70,11 +70,8 @@ action-sequence output head β see [Action-sequence output](#action-sequence-ou
|
|
| 70 |
| Output | 832x480 at 24 fps; a 1280x704 preset uses the same weights |
|
| 71 |
| Hardware | trained on 2x NVIDIA L20 48 GB, runs on 1x L20 48 GB |
|
| 72 |
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
block-causal conversion by teacher forcing; **III** online UniPC trajectory distillation, 25-step teacher to 12-step
|
| 76 |
-
student; **IV** mixed-domain causal SFT on Control2V, CrossFPS, DrivingDojo and PhysicalAI; **V** asymmetric DMD/DMD2
|
| 77 |
-
with a motion-preservation term, the Stage I model acting as the real score, producing the causal release.
|
| 78 |
|
| 79 |
## What is in this repository
|
| 80 |
|
|
@@ -160,10 +157,10 @@ which is what makes the representation usable as an inverse-dynamics readout for
|
|
| 160 |
| Head | `LayerNorm -> Linear(dim, hidden) -> SiLU -> Linear(hidden, 64)`, output layer zero-initialised so inserting it leaves the model unchanged |
|
| 161 |
| Input | spatial hidden tokens mean-pooled per latent frame: `(B, L, dim) -> (B, F, 64)` |
|
| 162 |
| Output | 64-D action vector per latent frame; trained with MSE on masked samples so the head cannot copy its input |
|
| 163 |
-
| Install / train | `model_kwargs.action_output: true` installs the head; the `
|
| 164 |
|
| 165 |
```bash
|
| 166 |
-
bash scripts/
|
| 167 |
```
|
| 168 |
|
| 169 |
One setting decides whether the run means anything: `action_output_input_dropout: 1.0` withholds the commanded action
|
|
@@ -171,11 +168,9 @@ from the head's input, so the head has to read the frames rather than its own co
|
|
| 171 |
then connect the generic 64-D interface to joints, end effectors, grippers or a mobile base β for driving, ego
|
| 172 |
trajectory, steering, speed and throttle/brake go into the same 64-D stream.
|
| 173 |
|
| 174 |
-
The head ships in this repository
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
as a probe: a head that recovers the action from generated frames is direct evidence that the action conditioning is
|
| 178 |
-
really steering the rollout.
|
| 179 |
|
| 180 |
## Benchmarks
|
| 181 |
|
|
@@ -187,8 +182,7 @@ Each case is generated as one continuous sequence rather than restarted as i2v a
|
|
| 187 |
| Navi 158 | **73.5** | 78.2 | 73.5 | 63.4 | 83.6 | 68.6 |
|
| 188 |
| Full 289 | **70.0** | 78.3 | 73.8 | 47.6 | 82.4 | 68.1 |
|
| 189 |
|
| 190 |
-
Navi 158 measures navigation only; Full 289
|
| 191 |
-
Interaction drops. Against the open models on the WBench leaderboard (snapshot of 13 September 2026; peer scores from
|
| 192 |
the leaderboard, Astronex-World evaluated by us with the official code), β post-trained from a Wan prior:
|
| 193 |
|
| 194 |
| Model | Params | Training GPUs | Avg. | Qual. | Set. | Inter. | Cons. | Phys. |
|
|
@@ -201,9 +195,10 @@ the leaderboard, Astronex-World evaluated by us with the official code), β pos
|
|
| 201 |
| Helios (distilled)β | 14B | 64-128x H100 | 69.7 | 73.3 | 75.3 | 41.6 | 82.2 | 76.1 |
|
| 202 |
| LTX-2.3 | 22B | n/r | 70.9 | 77.0 | 85.2 | 49.4 | 78.0 | 65.1 |
|
| 203 |
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
|
|
|
| 207 |
|
| 208 |
**VBench 1.0** β official evaluation code, official prompt order, no sampling or selection. A *partial* run: 240 of
|
| 209 |
6,220 text-to-video and 118 of 5,590 image-to-video videos had been generated and scored at the time of the report
|
|
@@ -221,26 +216,10 @@ parameters β Helios uses 64 to 128 H100s per stage. Quality 78.3 is the highes
|
|
| 221 |
| I2V background | β | 0.997 |
|
| 222 |
| Camera motion | β | 0.485 |
|
| 223 |
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
mixed-domain SFT, 21.2 px at DMD1515 and 16.6 px for this release. The motion-preservation term keeps rollouts from
|
| 227 |
-
freezing without restoring the teacher's amplitude. That is why both forms ship: causal for interaction and length,
|
| 228 |
bidirectional when the motion has to be exact.
|
| 229 |
|
| 230 |
-
## Limitations
|
| 231 |
-
|
| 232 |
-
From the report, unchanged here:
|
| 233 |
-
|
| 234 |
-
1. **Weak complex interaction.** Event editing, subject action and perspective switching go through a text-switch
|
| 235 |
-
interface that cannot express several independent timed events.
|
| 236 |
-
2. **Long-horizon drift remains.** Four sink frames and the local window delay forgetting, but strong turns, loops and
|
| 237 |
-
low-light scenes still show colour shift, darkening, structural repainting and detail loss.
|
| 238 |
-
3. **Distillation suppresses motion.** See the amplitudes above.
|
| 239 |
-
4. **Limited physical fidelity.** No explicit physical state, depth, collision constraints or 3D scene representation β
|
| 240 |
-
video coherence is not proof of accurate physics.
|
| 241 |
-
5. **Action post-training required.** The 64-D action stream and the `(F, 64)` action output are reserved interfaces;
|
| 242 |
-
driving and robot control spaces are connected through post-training on domain data.
|
| 243 |
-
|
| 244 |
## Speed
|
| 245 |
|
| 246 |
Real-time on a single NVIDIA L20 48 GB: the causal form streams block by block with cross-block KV caching and few-step
|
|
@@ -262,9 +241,3 @@ UniPC sampling, so frames leave the sampler as the rollout is produced instead o
|
|
| 262 |
## License
|
| 263 |
|
| 264 |
Released under the Apache License, Version 2.0 β see [LICENSE](LICENSE). Copyright 2026 Astronex Robotics.
|
| 265 |
-
|
| 266 |
-
## Provenance of the causal export
|
| 267 |
-
|
| 268 |
-
Base `online_unipc25_to12_step20_full.pt` plus the 1545 `generator_ema` adapter, merged as
|
| 269 |
-
`W = W_base + (B @ A) * 128/64` in fp32 and cast to bf16. The merge was checked against the LoRA forward pass on real
|
| 270 |
-
tensors before writing: worst relative error 3.7e-7, which is fp32 rounding.
|
|
|
|
| 70 |
| Output | 832x480 at 24 fps; a 1280x704 preset uses the same weights |
|
| 71 |
| Hardware | trained on 2x NVIDIA L20 48 GB, runs on 1x L20 48 GB |
|
| 72 |
|
| 73 |
+
Five training stages, all on two L20 48 GB GPUs (Section 6 of the report): bidirectional control adaptation,
|
| 74 |
+
block-causal conversion, online UniPC trajectory distillation, mixed-domain causal SFT, and asymmetric DMD/DMD2.
|
|
|
|
|
|
|
|
|
|
| 75 |
|
| 76 |
## What is in this repository
|
| 77 |
|
|
|
|
| 157 |
| Head | `LayerNorm -> Linear(dim, hidden) -> SiLU -> Linear(hidden, 64)`, output layer zero-initialised so inserting it leaves the model unchanged |
|
| 158 |
| Input | spatial hidden tokens mean-pooled per latent frame: `(B, L, dim) -> (B, F, 64)` |
|
| 159 |
| Output | 64-D action vector per latent frame; trained with MSE on masked samples so the head cannot copy its input |
|
| 160 |
+
| Install / train | `model_kwargs.action_output: true` installs the head; the `action` recipe in the code release trains it |
|
| 161 |
|
| 162 |
```bash
|
| 163 |
+
bash scripts/post_train_action.sh --data <lmdb dir> # action in and out, backbone frozen
|
| 164 |
```
|
| 165 |
|
| 166 |
One setting decides whether the run means anything: `action_output_input_dropout: 1.0` withholds the commanded action
|
|
|
|
| 168 |
then connect the generic 64-D interface to joints, end effectors, grippers or a mobile base β for driving, ego
|
| 169 |
trajectory, steering, speed and throttle/brake go into the same 64-D stream.
|
| 170 |
|
| 171 |
+
The head ships in this repository as six zero-initialised tensors, so the parameters exist and are trainable as they
|
| 172 |
+
stand. Untrained it returns zeros; once trained it doubles as a probe, since a head that recovers the action from
|
| 173 |
+
generated frames is direct evidence that the action conditioning is steering the rollout.
|
|
|
|
|
|
|
| 174 |
|
| 175 |
## Benchmarks
|
| 176 |
|
|
|
|
| 182 |
| Navi 158 | **73.5** | 78.2 | 73.5 | 63.4 | 83.6 | 68.6 |
|
| 183 |
| Full 289 | **70.0** | 78.3 | 73.8 | 47.6 | 82.4 | 68.1 |
|
| 184 |
|
| 185 |
+
Navi 158 measures navigation only; Full 289 also covers event editing, subject action and perspective switching. Against the open models on the WBench leaderboard (snapshot of 13 September 2026; peer scores from
|
|
|
|
| 186 |
the leaderboard, Astronex-World evaluated by us with the official code), β post-trained from a Wan prior:
|
| 187 |
|
| 188 |
| Model | Params | Training GPUs | Avg. | Qual. | Set. | Inter. | Cons. | Phys. |
|
|
|
|
| 195 |
| Helios (distilled)β | 14B | 64-128x H100 | 69.7 | 73.3 | 75.3 | 41.6 | 82.2 | 76.1 |
|
| 196 |
| LTX-2.3 | 22B | n/r | 70.9 | 77.0 | 85.2 | 49.4 | 78.0 | 65.1 |
|
| 197 |
|
| 198 |
+
Among 5B-class models this one is above YUME 1.5, post-trained from the same Wan2.2 prior on A100s, and above the 4B
|
| 199 |
+
Kairos 3.0 by 4.3 points. With a third or less of the parameters it is above LongCat-Video (13.6B) and Helios (14B) β
|
| 200 |
+
Helios uses 64 to 128 H100s per training stage β and within 0.9 points of LTX-2.3 (22B). Quality 78.3 is the highest in
|
| 201 |
+
the table, on two L20 48 GB GPUs.
|
| 202 |
|
| 203 |
**VBench 1.0** β official evaluation code, official prompt order, no sampling or selection. A *partial* run: 240 of
|
| 204 |
6,220 text-to-video and 118 of 5,590 image-to-video videos had been generated and scored at the time of the report
|
|
|
|
| 216 |
| I2V background | β | 0.997 |
|
| 217 |
| Camera motion | β | 0.485 |
|
| 218 |
|
| 219 |
+
Smoothness, flicker and background preservation are the strongest axes, which is what block-causal KV caching, the
|
| 220 |
+
local window and the persistent sink frames are built for. Both forms ship: causal for interaction and length,
|
|
|
|
|
|
|
| 221 |
bidirectional when the motion has to be exact.
|
| 222 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
## Speed
|
| 224 |
|
| 225 |
Real-time on a single NVIDIA L20 48 GB: the causal form streams block by block with cross-block KV caching and few-step
|
|
|
|
| 241 |
## License
|
| 242 |
|
| 243 |
Released under the Apache License, Version 2.0 β see [LICENSE](LICENSE). Copyright 2026 Astronex Robotics.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|