| --- |
| license: apache-2.0 |
| base_model: |
| - alibaba-pai/Wan2.2-Fun-5B-Control |
| pipeline_tag: video-to-video |
| tags: |
| - matrix-game-4.0 |
| - causal-video |
| - game-rendering |
| --- |
| |
| # Matrix-Game 4.0 |
|
|
| [Source code](https://github.com/SkyworkAI/Matrix-Game) |
| | [Project website](https://matrix-game-v4-0.github.io/page/) |
|
|
| Scene-specialized causal video renderers for **Medieval Castle**, **Ancient |
| Egypt**, and **City Racing**. An executable game determines gameplay, physics |
| and input, while the model re-renders clean low-poly RGB. HUD and sound remain |
| separate, frame-aligned presentation layers. |
|
|
| ## Contents |
|
|
| | Path | Contents | |
| | --- | --- | |
| | `config.json` | Release version, architecture, upstream revision and scene file index | |
| | `fps-castle/` | Castle checkpoint, sink image and inference configuration | |
| | `fps-egypt/` | Ancient Egypt checkpoint, sink image and inference configuration | |
| | `driving_game/` | Racing checkpoint, sink image and inference configuration | |
| | `base/` | Full streaming TAEHV codec | |
| | `game/assets/` | Shooter geometry, character data, catalogs and combat sounds | |
| | `racing/assets/` | City map, routes, captions, vehicle audio and audio credits | |
| | `releases/` | Exact resource paths, byte lengths and SHA256 manifests | |
|
|
| Each scene checkpoint has 5,001,016,512 BF16 parameters. The weights require |
| the Matrix-Game 4.0 inference source; they are not drop-in Diffusers pipelines. |
| Training code and training data are not included. Shared encoders/codecs are |
| installed once for all scenes. See [game asset details](GAME_ASSETS.md). |
|
|
| ## Download and Layout |
|
|
| Use the source release's manifest downloader with a pinned Hub commit revision |
| and `--repo-id Skywork/Matrix-Game-4.0 --repo-type model` for both model and |
| game-asset manifests. See `releases/README.md` in the source repository for exact |
| commands. A scene selection downloads that scene plus its shared dependencies. |
|
|
| The shared root `config.json` is installed as `resources/release-config.json` |
| and verified by the model manifest. It describes this release; it is not a |
| Diffusers/Transformers pipeline configuration or the optional DiT |
| `resources/base/config.json`. Game-asset-only downloads do not fetch it. |
|
|
| The T5 encoder, tokenizer and Wan2.2 VAE are fetched directly from the official |
| [base-model repository](https://huggingface.co/alibaba-pai/Wan2.2-Fun-5B-Control/tree/b8bc1a65ab71d054ba4636dc0dac104aa4df2686). |
| The manifest pins that revision and verifies every file's SHA256; no manual |
| base-model download is needed. These shared files are not duplicated here. |
|
|
| The downloader maps Hub scene folders to the runtime paths |
| `resources/models/castle/`, `resources/models/ancient-egypt/` and |
| `resources/models/racing/`. Do not infer the local directory layout from the |
| Hub folder names alone. The `repo_path` field in `releases/models.json` describes |
| this mapping. Both length and SHA256 are checked before a file is installed. |
|
|
| ## Inference Contract |
|
|
| - Output resolution: 480x864 RGB. One latent chunk corresponds to four RGB |
| frames after startup; the startup chunk contains one RGB frame. |
| - Castle/Egypt use an independent identity sink with fixed-relative RoPE. |
| Racing uses the supplied image as output frame zero and an absolute-position |
| sink. These policies are not interchangeable. |
| - Quality preset: Wan VAE encoding/decoding, two denoising steps (1000, 500), |
| native DiT. Realtime preset: TAEHV encoding/decoding, one step (1000), optimized |
| single-GPU DiT and CUDA graphs. |
| - CFG is 1 and the scheduler shift is 5. The attention cache contains 21 latent |
| frames including the sink. Racing skips denoising on its supplied first frame. |
| - Shooter captions are fixed. Racing supports fixed or per-chunk dynamic engine |
| captions. Clean RGB, not the HUD or keyboard overlay, is the model input. |
|
|
| Use `base/taew2_2.pth` for the full streaming fast codec; a standalone decoder |
| file is not a replacement for its encoder. Use the exact shared files in the |
| manifest, not an arbitrary upstream version or Wan2.1's 16-channel VAE. |
|
|
| ## Limitations |
|
|
| The models are specialized to the supplied low-poly scenes and camera profiles. |
| Unseen geometry, rapid camera motion, occlusion and long sessions can cause |
| appearance drift or geometry errors. Neural output does not change collision |
| or gameplay. One-step/TAEHV and two-step/Wan have different quality and cost. |
| There is no guaranteed frame rate, consumer-GPU memory fit, or exact scene |
| reconstruction. This model card does not claim a performance benchmark. |
|
|
| ## License and Attribution |
|
|
| Project scene weights, sink images and game assets: **Apache-2.0**, Skywork AI. |
| The shared Wan/umT5 resources retain their upstream Apache-2.0 terms: |
| [Wan2.2-Fun-5B-Control](https://huggingface.co/alibaba-pai/Wan2.2-Fun-5B-Control), |
| [umT5](https://huggingface.co/google/umt5-xxl). |
| TAEHV is by Ollin Boer Bohan under MIT; see `licenses/TAEHV-LICENSE` and its |
| [upstream repository](https://github.com/madebyollin/taehv). |
| Racing audio includes CC0 and CC BY 3.0 material; retain |
| `racing/assets/audio/CREDITS.md`. See `THIRD_PARTY_NOTICES.md` for code dependencies. |
|
|
