baibao132 commited on
Commit
ac495a8
Β·
verified Β·
1 Parent(s): 1f2f09d

Trim the model card to capabilities, benchmarks and usage

Browse files
Files changed (1) hide show
  1. README.md +14 -41
README.md CHANGED
@@ -70,11 +70,8 @@ action-sequence output head β€” see [Action-sequence output](#action-sequence-ou
70
  | Output | 832x480 at 24 fps; a 1280x704 preset uses the same weights |
71
  | Hardware | trained on 2x NVIDIA L20 48 GB, runs on 1x L20 48 GB |
72
 
73
- The five training stages behind these weights (Section 6 of the report): **I** bidirectional control adaptation, which
74
- adds PRoPE camera control and the 64-D action stream to the Wan2.2 prior and produces the bidirectional release; **II**
75
- block-causal conversion by teacher forcing; **III** online UniPC trajectory distillation, 25-step teacher to 12-step
76
- student; **IV** mixed-domain causal SFT on Control2V, CrossFPS, DrivingDojo and PhysicalAI; **V** asymmetric DMD/DMD2
77
- with a motion-preservation term, the Stage I model acting as the real score, producing the causal release.
78
 
79
  ## What is in this repository
80
 
@@ -160,10 +157,10 @@ which is what makes the representation usable as an inverse-dynamics readout for
160
  | Head | `LayerNorm -> Linear(dim, hidden) -> SiLU -> Linear(hidden, 64)`, output layer zero-initialised so inserting it leaves the model unchanged |
161
  | Input | spatial hidden tokens mean-pooled per latent frame: `(B, L, dim) -> (B, F, 64)` |
162
  | Output | 64-D action vector per latent frame; trained with MSE on masked samples so the head cannot copy its input |
163
- | Install / train | `model_kwargs.action_output: true` installs the head; the `stage1` recipe in the code release trains it |
164
 
165
  ```bash
166
- bash scripts/post_train_stage1.sh --data <lmdb dir> # action head + event branch, backbone frozen
167
  ```
168
 
169
  One setting decides whether the run means anything: `action_output_input_dropout: 1.0` withholds the commanded action
@@ -171,11 +168,9 @@ from the head's input, so the head has to read the frames rather than its own co
171
  then connect the generic 64-D interface to joints, end effectors, grippers or a mobile base β€” for driving, ego
172
  trajectory, steering, speed and throttle/brake go into the same 64-D stream.
173
 
174
- The head ships in this repository: six tensors, zero-initialised (LayerNorm at 1, both linears at 0), written into the
175
- export so the parameters exist and are trainable as they stand β€” enable them with `model_kwargs.action_output: true` and
176
- train the branch above. Reading actions from the untrained head returns zeros, so train it first. Once trained it doubles
177
- as a probe: a head that recovers the action from generated frames is direct evidence that the action conditioning is
178
- really steering the rollout.
179
 
180
  ## Benchmarks
181
 
@@ -187,8 +182,7 @@ Each case is generated as one continuous sequence rather than restarted as i2v a
187
  | Navi 158 | **73.5** | 78.2 | 73.5 | 63.4 | 83.6 | 68.6 |
188
  | Full 289 | **70.0** | 78.3 | 73.8 | 47.6 | 82.4 | 68.1 |
189
 
190
- Navi 158 measures navigation only; Full 289 adds event editing, subject action and perspective switching, which is where
191
- Interaction drops. Against the open models on the WBench leaderboard (snapshot of 13 September 2026; peer scores from
192
  the leaderboard, Astronex-World evaluated by us with the official code), † post-trained from a Wan prior:
193
 
194
  | Model | Params | Training GPUs | Avg. | Qual. | Set. | Inter. | Cons. | Phys. |
@@ -201,9 +195,10 @@ the leaderboard, Astronex-World evaluated by us with the official code), † pos
201
  | Helios (distilled)† | 14B | 64-128x H100 | 69.7 | 73.3 | 75.3 | 41.6 | 82.2 | 76.1 |
202
  | LTX-2.3 | 22B | n/r | 70.9 | 77.0 | 85.2 | 49.4 | 78.0 | 65.1 |
203
 
204
- HY-Video 1.5 (8.3B) is ahead. What the table is for is the cost axis: among 5B-class models this one is above YUME 1.5,
205
- post-trained from the same Wan2.2 prior on A100s, and it is above LongCat-Video and Helios with a third or less of their
206
- parameters β€” Helios uses 64 to 128 H100s per stage. Quality 78.3 is the highest in the table.
 
207
 
208
  **VBench 1.0** β€” official evaluation code, official prompt order, no sampling or selection. A *partial* run: 240 of
209
  6,220 text-to-video and 118 of 5,590 image-to-video videos had been generated and scored at the time of the report
@@ -221,26 +216,10 @@ parameters β€” Helios uses 64 to 128 H100s per stage. Quality 78.3 is the highes
221
  | I2V background | β€” | 0.997 |
222
  | Camera motion | β€” | 0.485 |
223
 
224
- **Motion retention.** The low dynamic degree is the motion suppression that few-step DMD distillation introduces,
225
- measured on an internal basketball sequence as subject residual amplitude: 88.8 px bidirectional, 36.1 px after
226
- mixed-domain SFT, 21.2 px at DMD1515 and 16.6 px for this release. The motion-preservation term keeps rollouts from
227
- freezing without restoring the teacher's amplitude. That is why both forms ship: causal for interaction and length,
228
  bidirectional when the motion has to be exact.
229
 
230
- ## Limitations
231
-
232
- From the report, unchanged here:
233
-
234
- 1. **Weak complex interaction.** Event editing, subject action and perspective switching go through a text-switch
235
- interface that cannot express several independent timed events.
236
- 2. **Long-horizon drift remains.** Four sink frames and the local window delay forgetting, but strong turns, loops and
237
- low-light scenes still show colour shift, darkening, structural repainting and detail loss.
238
- 3. **Distillation suppresses motion.** See the amplitudes above.
239
- 4. **Limited physical fidelity.** No explicit physical state, depth, collision constraints or 3D scene representation β€”
240
- video coherence is not proof of accurate physics.
241
- 5. **Action post-training required.** The 64-D action stream and the `(F, 64)` action output are reserved interfaces;
242
- driving and robot control spaces are connected through post-training on domain data.
243
-
244
  ## Speed
245
 
246
  Real-time on a single NVIDIA L20 48 GB: the causal form streams block by block with cross-block KV caching and few-step
@@ -262,9 +241,3 @@ UniPC sampling, so frames leave the sampler as the rollout is produced instead o
262
  ## License
263
 
264
  Released under the Apache License, Version 2.0 β€” see [LICENSE](LICENSE). Copyright 2026 Astronex Robotics.
265
-
266
- ## Provenance of the causal export
267
-
268
- Base `online_unipc25_to12_step20_full.pt` plus the 1545 `generator_ema` adapter, merged as
269
- `W = W_base + (B @ A) * 128/64` in fp32 and cast to bf16. The merge was checked against the LoRA forward pass on real
270
- tensors before writing: worst relative error 3.7e-7, which is fp32 rounding.
 
70
  | Output | 832x480 at 24 fps; a 1280x704 preset uses the same weights |
71
  | Hardware | trained on 2x NVIDIA L20 48 GB, runs on 1x L20 48 GB |
72
 
73
+ Five training stages, all on two L20 48 GB GPUs (Section 6 of the report): bidirectional control adaptation,
74
+ block-causal conversion, online UniPC trajectory distillation, mixed-domain causal SFT, and asymmetric DMD/DMD2.
 
 
 
75
 
76
  ## What is in this repository
77
 
 
157
  | Head | `LayerNorm -> Linear(dim, hidden) -> SiLU -> Linear(hidden, 64)`, output layer zero-initialised so inserting it leaves the model unchanged |
158
  | Input | spatial hidden tokens mean-pooled per latent frame: `(B, L, dim) -> (B, F, 64)` |
159
  | Output | 64-D action vector per latent frame; trained with MSE on masked samples so the head cannot copy its input |
160
+ | Install / train | `model_kwargs.action_output: true` installs the head; the `action` recipe in the code release trains it |
161
 
162
  ```bash
163
+ bash scripts/post_train_action.sh --data <lmdb dir> # action in and out, backbone frozen
164
  ```
165
 
166
  One setting decides whether the run means anything: `action_output_input_dropout: 1.0` withholds the commanded action
 
168
  then connect the generic 64-D interface to joints, end effectors, grippers or a mobile base β€” for driving, ego
169
  trajectory, steering, speed and throttle/brake go into the same 64-D stream.
170
 
171
+ The head ships in this repository as six zero-initialised tensors, so the parameters exist and are trainable as they
172
+ stand. Untrained it returns zeros; once trained it doubles as a probe, since a head that recovers the action from
173
+ generated frames is direct evidence that the action conditioning is steering the rollout.
 
 
174
 
175
  ## Benchmarks
176
 
 
182
  | Navi 158 | **73.5** | 78.2 | 73.5 | 63.4 | 83.6 | 68.6 |
183
  | Full 289 | **70.0** | 78.3 | 73.8 | 47.6 | 82.4 | 68.1 |
184
 
185
+ Navi 158 measures navigation only; Full 289 also covers event editing, subject action and perspective switching. Against the open models on the WBench leaderboard (snapshot of 13 September 2026; peer scores from
 
186
  the leaderboard, Astronex-World evaluated by us with the official code), † post-trained from a Wan prior:
187
 
188
  | Model | Params | Training GPUs | Avg. | Qual. | Set. | Inter. | Cons. | Phys. |
 
195
  | Helios (distilled)† | 14B | 64-128x H100 | 69.7 | 73.3 | 75.3 | 41.6 | 82.2 | 76.1 |
196
  | LTX-2.3 | 22B | n/r | 70.9 | 77.0 | 85.2 | 49.4 | 78.0 | 65.1 |
197
 
198
+ Among 5B-class models this one is above YUME 1.5, post-trained from the same Wan2.2 prior on A100s, and above the 4B
199
+ Kairos 3.0 by 4.3 points. With a third or less of the parameters it is above LongCat-Video (13.6B) and Helios (14B) β€”
200
+ Helios uses 64 to 128 H100s per training stage β€” and within 0.9 points of LTX-2.3 (22B). Quality 78.3 is the highest in
201
+ the table, on two L20 48 GB GPUs.
202
 
203
  **VBench 1.0** β€” official evaluation code, official prompt order, no sampling or selection. A *partial* run: 240 of
204
  6,220 text-to-video and 118 of 5,590 image-to-video videos had been generated and scored at the time of the report
 
216
  | I2V background | β€” | 0.997 |
217
  | Camera motion | β€” | 0.485 |
218
 
219
+ Smoothness, flicker and background preservation are the strongest axes, which is what block-causal KV caching, the
220
+ local window and the persistent sink frames are built for. Both forms ship: causal for interaction and length,
 
 
221
  bidirectional when the motion has to be exact.
222
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
223
  ## Speed
224
 
225
  Real-time on a single NVIDIA L20 48 GB: the causal form streams block by block with cross-block KV caching and few-step
 
241
  ## License
242
 
243
  Released under the Apache License, Version 2.0 β€” see [LICENSE](LICENSE). Copyright 2026 Astronex Robotics.