| --- |
| license: apache-2.0 |
| base_model: Wan-AI/Wan2.2-TI2V-5B-Diffusers |
| pipeline_tag: image-to-video |
| tags: |
| - world-model |
| - camera-control |
| - video-generation |
| - long-video |
| --- |
| |
| <div align="center"> |
|
|
| <h2>Memorizon: Training World Models Beyond Their Context Window</h2> |
|
|
| <p> |
| <a href="https://tingtingliao.github.io/" style="color:#0969da"><b>Tingting Liao</b></a> · |
| <span style="color:#0969da"><b>Xuezhi Liang</b></span> · |
| <a href="https://www.hao-li.com/Hao_Li/Hao_Li_-_about_me.html" style="color:#0969da"><b>Hao Li</b></a> · |
| <a href="https://guangyiliu.com/" style="color:#0969da"><b>Guangyi Liu</b></a> |
| <br> |
| <sub>Institute of Foundation Models (IFM), MBZUAI</sub> |
| </p> |
|
|
| <div align="center"> |
| <a href='https://github.com/TingtingLiao/memorizon'><img src='https://img.shields.io/badge/GitHub-code-black?logo=github&logoColor=black&labelColor=white'></a>   |
| <a href='https://tingtingliao.github.io/memorizon/'><img src='https://img.shields.io/badge/project-page-green?logo=googlechrome&logoColor=green&labelColor=white'></a>   |
| <a href='https://arxiv.org/abs/2610.00544'><img src='https://img.shields.io/badge/arXiv-2610.00544-red?logo=arxiv&logoColor=darkred&labelColor=white'></a>   |
| </div> |
|
|
| <br> |
|
|
| <img src="teaser.webp" width="100%"> |
|
|
| </div> |
|
|
| --- |
|
|
| ## ✨ This checkpoint |
|
|
| | | | |
| |---|---| |
| | **Model** | Memorizon, 4-step distilled (Self-Forcing + DMD, CFG distilled in) | |
| | **Base** | Wan2.2-TI2V-5B | |
| | **Input** | one image + keyboard actions or a camera trajectory | |
| | **Output** | 864×480, 16 fps, generated one second at a time | |
| | **Sampling** | 4 steps, no CFG (read from the config) | |
|
|
| ## 🚀 Usage |
|
|
| ```bash |
| git clone https://github.com/TingtingLiao/memorizon.git && cd memorizon && pip install -e . |
| |
| python scripts/generate.py --checkpoint Luffuly/memorizon --image photo.jpg \ |
| --actions "w*16 l*24 w*16 l*24" --output out.mp4 |
| ``` |
|
|
| ```python |
| from memorizon import MemorizonPipeline, actions_to_c2w, save_video |
| |
| pipe = MemorizonPipeline.from_pretrained("Luffuly/memorizon") |
| video = pipe("photo.jpg", actions_to_c2w("w*16 l*24 w*16 l*24")) |
| save_video(video, "out.mp4") |
| ``` |
|
|
| ### 🎮 Keyboard actions — one per 0.25 s |
|
|
| | Key | Action | Key | Action | |
| |:---:|:---|:---:|:---| |
| | `w` `s` | forward / backward · 0.25 m | `j` `l` | turn left / right · 7.5° | |
| | `a` `d` | left / right · 0.25 m | `i` `k` | look up / down · 7.5° | |
| | `.` | stay | `*N` | repeat N times | |
|
|
| <details> |
| <summary><b>Prompts</b> — captioned automatically</summary> |
|
|
| <br> |
|
|
| Without a prompt, the image is captioned by Qwen3-VL-8B in the training format |
| (`memorizon/caption.py`). Custom prompts should follow it — a perspective prefix, |
| then one paragraph: |
|
|
| > First-person perspective — character not visible. A winding paved path curves |
| > beneath a canopy of vibrant pink cherry blossoms, … |
|
|
| </details> |
|
|
| <details> |
| <summary><b>Camera trajectories</b></summary> |
|
|
| <br> |
|
|
| Instead of actions, pass `--trajectory` a `[T, 4, 4]` array of camera-to-world poses, |
| `T = 1 + 4 × seconds`, in the first frame's coordinates (OpenCV axes), translations |
| in metres / 4. |
|
|
| </details> |
|
|
| ## 📖 Citation |
|
|
| ```bibtex |
| @article{memorizon2026, |
| title = {Memorizon: Training World Models Beyond Their Context Window}, |
| author = {Tingting Liao, Xuezhi Liang, Hao Li, Guangyi Liu}, |
| journal = {arXiv preprint arXiv:2610.00544}, |
| year = {2026} |
| } |
| ``` |
|
|
