| --- |
| license: other |
| license_name: minimax-h3-community |
| license_link: LICENSE |
| tags: |
| - video-generation |
| - streaming |
| - interactive |
| - world-model |
| - minimax-h3 |
| --- |
| |
| # StreamRender-H3: Interactive Coding Worlds with Real-time Streaming Diffusion Render |
|
|
| **Project page:** https://nvlabs.github.io/Sana/Sol-Engine/StreamRender-H3/ |
|
|
| StreamRender-H3 splits an interactive world into a **Code Game Engine** and a **Streaming Diffusion Render**. The engine keeps the world state in code (maps, physics, rules) and streams a semantic map for every frame. A two-step streaming video model built on MiniMax-H3 Ref2VA renders those maps into photorealistic video as the player drives, with the look set by a reference image and a prompt. |
|
|
| - Two denoising steps per chunk in the distilled student. |
| - Diffusion render of 217.96 ms per streaming step on 8 GB200 GPUs, a 3.67× speedup over the PyTorch baseline. |
| - Fast Causal H3 VAE decodes 2 latent frames at 720p in 24 ms on one GB200, built by a self-improving agent loop. |
|
|
| ## Released weights |
|
|
| This repository contains **two trained model components**: |
|
|
| | Directory | Model | Checkpoint | |
| | --- | --- | --- | |
| | `backbone/` | Streaming H3 renderer, full 50-layer EMA backbone with LoRA merged | 1008b / step224 | |
| | `causal_decoder/` | Fast Causal H3 VAE, a pruned and distilled 24-layer causal decoder | step6000 / 20261005-v1 | |
|
|
| The renderer uses two denoising evaluations with the S=2/W=2/C=2 streaming |
| policy. The decoder processes two available latent frames per call, using |
| cached past context without future latent frames. These are the trained |
| renderer and decoder checkpoints, not a bundle of unchanged upstream models. |
|
|
| ## Download and use |
|
|
| Runtime code and deployment instructions: |
| [NVlabs/Sana — StreamRender-H3](https://github.com/NVlabs/Sana/tree/sol-engine/models/streamrender_h3). |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| snapshot_download("Efficient-Large-Model/StreamRender-H3", |
| local_dir="./StreamRender-H3") |
| ``` |
|
|
| The download is approximately 72.75 GB. The renderer is stored as nine |
| safetensors shards, preserving all original tensor values. The current |
| runtime expects one merged backbone file. Reconstruct it with: |
|
|
| ```bash |
| python StreamRender-H3/download_assets.py |
| ``` |
|
|
| Reconstruction needs another 66.28 GB of disk and verifies the original |
| SHA-256. Point the runtime manifest's `backbone` entry to |
| `backbone/1008b_step224_backbone_ema_merged.safetensors` and its |
| `causal_decoder` entry to `causal_decoder/causal_vit24_step6000.safetensors`. |
| Provide the remaining runtime dependencies separately as described in the |
| runtime README and `configs/assets.example.json`; this release contains |
| only the two trained model components. `SHA256SUMS` and `inventory.json` |
| provide integrity information for the released files. |
|
|
| ## Upstream runtime dependencies |
|
|
| The trained-weight release does not include unchanged upstream models. |
| Download these components from their original sources: |
|
|
| | Runtime asset | Original download source | |
| | --- | --- | |
| | Qwen3-VL conditioning weights/config (`qwen_model`) | [MiniMax-H3 / Ref2VA/text_encoder](https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/Ref2VA/text_encoder) — the H3-compatible conditioning checkpoint | |
| | Tokenizer (`tokenizer`) | [MiniMax-H3 / Ref2VA/tokenizer](https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/Ref2VA/tokenizer) | |
| | Image/text processor (`qwen_processor`) | [MiniMax-H3 / Ref2VA/processor](https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/Ref2VA/processor) | |
| | Native H3 video VAE (`native_vae`) | [MiniMax-H3 / Ref2VA/video_vae/source](https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/Ref2VA/video_vae/source) — `model.safetensors`, used for decoder construction | |
| | Native H3 audio VAE (`audio_vae`) | [MiniMax-H3 / Ref2VA/audio_vae](https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/Ref2VA/audio_vae) — `model.safetensors` | |
| | Official H3 TAE (`tae`) | [TAeHV / taeh3.pth](https://github.com/madebyollin/taehv/blob/62f7591f59dfbb4c3c02b7a621d180a9eeaba26c/taeh3.pth) — official GitHub release; no official HF model repository was verified | |
|
|
| The [Qwen3-VL upstream model](https://huggingface.co/Qwen/Qwen3-VL-32B-Instruct) |
| is the upstream architecture. For this runtime, use the H3-distributed |
| `Ref2VA/text_encoder` checkpoint above rather than assuming another Qwen |
| checkpoint is interchangeable. |
|
|
| `qwen_dcp` is a PyTorch distributed-checkpoint representation of those |
| conditioning weights, not a separate model. The current runtime loader |
| requires a compatible local DCP conversion; the public upstream HF download |
| provides safetensors, not the local DCP directory. Do not point |
| `qwen_dcp` at the safetensors directory. |
|
|
| Download the H3 dependencies without its original transformer weights: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| snapshot_download( |
| "MiniMaxAI/MiniMax-H3", local_dir="./MiniMax-H3", |
| allow_patterns=["Ref2VA/text_encoder/*", "Ref2VA/tokenizer/*", |
| "Ref2VA/processor/*", "Ref2VA/video_vae/*", |
| "Ref2VA/audio_vae/*"], |
| ) |
| ``` |
|
|
| Download the verified official TAE weights: |
|
|
| ```bash |
| curl -fL -o taeh3_official.pth \ |
| https://raw.githubusercontent.com/madebyollin/taehv/62f7591f59dfbb4c3c02b7a621d180a9eeaba26c/taeh3.pth |
| ``` |
|
|
| Fill the corresponding paths in the runtime asset manifest. These links |
| identify external dependencies; they do not add their weights to this release. |
|
|
| ## License |
|
|
| These model weights are derivatives governed by the MiniMax H3 Community |
| License, including its territory, use and redistribution terms. See |
| `LICENSE` and `NOTICE`. The runtime code's license does not replace the |
| model license. |
|
|
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{streamrenderh3_2026, |
| title = {StreamRender-H3: Interactive Coding Worlds with Real-time Streaming Diffusion Render}, |
| author = {Yanzuo Lu and Tian Ye and Yitong Li and Shuchen Xue and Haozhe Liu and Song Han and Enze Xie}, |
| year = {2026}, |
| howpublished = {\url{https://nvlabs.github.io/Sana/Sol-Engine/StreamRender-H3/}}, |
| note = {Project page} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| We thank the [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) team |
| for the base video generation model and native video codec. Our streaming |
| renderer and distilled causal decoder build on these components. Their |
| upstream license and attribution notices remain applicable. |
|
|
