Instructions to use da-eun07/rest2art-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Wan2.2
How to use da-eun07/rest2art-models with Wan2.2:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Inference
- Notebooks
- Google Colab
- Kaggle
Upload folder using huggingface_hub
Browse files- .gitattributes +1 -34
- README.md +110 -0
- wan2.2-i2v-loras/README.md +29 -0
- wan2.2-i2v-loras/config/wan_moe_i2v_distill_with_lora_vbvr.json +46 -0
- wan2.2-i2v-loras/loras/Wan22_I2V_VBVR_HIGH_rank_64_fp16.safetensors +3 -0
- wan2.2-i2v-loras/loras/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors +3 -0
- wan2.2-i2v-loras/loras/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors +3 -0
- wan2.2-i2v-loras/scripts/download_assets.sh +18 -0
- wan2.2-i2v-loras/scripts/minimal_infer.py +83 -0
- wan2.2-i2v-loras/scripts/render_config.py +45 -0
.gitattributes
CHANGED
|
@@ -1,35 +1,2 @@
|
|
| 1 |
-
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
-
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
-
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
-
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
-
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
-
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
-
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
-
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
-
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
-
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
-
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
-
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
-
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
-
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
-
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
-
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
-
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
-
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
-
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
-
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
-
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
-
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
-
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
-
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
-
|
| 27 |
-
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
-
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
-
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
-
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
-
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
-
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
-
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
-
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
-
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
pretty_name: "rest2art Model Checkpoints"
|
| 4 |
+
base_model:
|
| 5 |
+
- Wan-AI/Wan2.2-I2V-A14B
|
| 6 |
+
task_categories:
|
| 7 |
+
- image-to-video
|
| 8 |
+
language:
|
| 9 |
+
- en
|
| 10 |
+
tags:
|
| 11 |
+
- articulated-objects
|
| 12 |
+
- reconstruction
|
| 13 |
+
- image-to-video
|
| 14 |
+
- lora
|
| 15 |
+
- wan2.2
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# [ECCV'26] Articulated Object Reconstruction from Rest-State Observation
|
| 19 |
+
|
| 20 |
+
<!-- TODO: replace with real links once public -->
|
| 21 |
+
[](https://example.com)
|
| 22 |
+
[](https://arxiv.org)
|
| 23 |
+
[](https://github.com)
|
| 24 |
+
[](https://huggingface.co/datasets/da-eun07/rest2art)
|
| 25 |
+
|
| 26 |
+
## Model Checkpoints
|
| 27 |
+
|
| 28 |
+
Pretrained models used by the rest2art pipeline. We only re-host the Wan2.2 video-generation
|
| 29 |
+
LoRAs, whose inference is config-sensitive and hard to reproduce. Every other model is linked to
|
| 30 |
+
its original repo (cleaner attribution and licensing), so fetch those from the source.
|
| 31 |
+
|
| 32 |
+
| Model | Hosting | Source | License |
|
| 33 |
+
|---|---|---|---|
|
| 34 |
+
| GPT (OpenAI API) | API, no ckpt | OpenAI | n/a |
|
| 35 |
+
| SAM 3 | link | [facebook/sam3](https://huggingface.co/facebook/sam3) | see repo |
|
| 36 |
+
| Qwen3-VL-8B-Instruct | link | [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) | Apache-2.0 |
|
| 37 |
+
| Wan2.2-I2V-A14B + 3 LoRAs | hosted (`wan2.2-i2v-loras/`) | [Wan-AI](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B), [lightx2v](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras), [Kijai](https://huggingface.co/Kijai/WanVideo_comfy) | Apache-2.0 |
|
| 38 |
+
| CoTracker3 (scaled_offline) | link | [facebook/cotracker3](https://huggingface.co/facebook/cotracker3) | CC BY-NC 4.0 |
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
## Get started
|
| 42 |
+
|
| 43 |
+
Put every checkpoint under one `ckpt/` root, then point the pipeline flags at it.
|
| 44 |
+
|
| 45 |
+
### 1. Download checkpoints
|
| 46 |
+
|
| 47 |
+
```bash
|
| 48 |
+
mkdir -p ckpt && cd ckpt
|
| 49 |
+
|
| 50 |
+
# this collection: Wan2.2 LoRAs + config + loader
|
| 51 |
+
huggingface-cli download da-eun07/rest2art-models --local-dir ./rest2art-models
|
| 52 |
+
|
| 53 |
+
# Wan2.2 base model, UMT5-XXL T5, VAE (~119 GB) from Wan-AI
|
| 54 |
+
bash ./rest2art-models/wan2.2-i2v-loras/scripts/download_assets.sh ./Wan-AI
|
| 55 |
+
|
| 56 |
+
# CoTracker3 offline checkpoint
|
| 57 |
+
huggingface-cli download facebook/cotracker3 scaled_offline.pth --local-dir ./co-tracker
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
SAM 3 (`facebook/sam3`) and Qwen3-VL-8B (`Qwen/Qwen3-VL-8B-Instruct`) are pulled from their repos
|
| 61 |
+
on first run, so no manual download is needed. Resulting layout:
|
| 62 |
+
|
| 63 |
+
```
|
| 64 |
+
ckpt/
|
| 65 |
+
rest2art-models/wan2.2-i2v-loras/loras/*.safetensors
|
| 66 |
+
Wan-AI/Wan2.2-I2V-A14B/
|
| 67 |
+
co-tracker/scaled_offline.pth
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
### 2. Render the Wan2.2 config
|
| 71 |
+
|
| 72 |
+
The LightX2V config needs absolute LoRA paths, so render a local copy:
|
| 73 |
+
|
| 74 |
+
```bash
|
| 75 |
+
cd ckpt/rest2art-models/wan2.2-i2v-loras
|
| 76 |
+
python scripts/render_config.py --repo . --base ../../Wan-AI/Wan2.2-I2V-A14B --out config.local.json
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
### 3. Run the pipeline
|
| 80 |
+
|
| 81 |
+
From the rest2art project root, point the flags at your `ckpt/` root:
|
| 82 |
+
|
| 83 |
+
```bash
|
| 84 |
+
python -m rest2art.pipeline.main \
|
| 85 |
+
--scene <scene> \
|
| 86 |
+
--wan_path ckpt/Wan-AI/Wan2.2-I2V-A14B \
|
| 87 |
+
--config_json ckpt/rest2art-models/wan2.2-i2v-loras/config.local.json \
|
| 88 |
+
--cotracker_ckpt ckpt/co-tracker/scaled_offline.pth \
|
| 89 |
+
--vlm_path Qwen/Qwen3-VL-8B-Instruct
|
| 90 |
+
```
|
| 91 |
+
|
| 92 |
+
To run only the Wan2.2 video step, follow [`wan2.2-i2v-loras/`](./wan2.2-i2v-loras).
|
| 93 |
+
|
| 94 |
+
## License
|
| 95 |
+
|
| 96 |
+
Hosted LoRAs are Apache-2.0 (Wan-AI, lightx2v); the VBVR LoRA follows
|
| 97 |
+
[Kijai/WanVideo_comfy](https://huggingface.co/Kijai/WanVideo_comfy) terms. Linked models keep
|
| 98 |
+
their own licenses (see table). Config and scripts are Apache-2.0. To request removal from this
|
| 99 |
+
mirror, open a discussion.
|
| 100 |
+
|
| 101 |
+
## Citation
|
| 102 |
+
|
| 103 |
+
```bibtex
|
| 104 |
+
@inproceedings{lee2026rest2art,
|
| 105 |
+
title={Articulated Object Reconstruction from Rest-State Observation},
|
| 106 |
+
author={Lee, Daeun and Lee, Jaeah and Kim, Woosung and Jung, Haebeom and Park, Jaesik},
|
| 107 |
+
booktitle={European Conference on Computer Vision (ECCV)},
|
| 108 |
+
year={2026}
|
| 109 |
+
}
|
| 110 |
+
```
|
wan2.2-i2v-loras/README.md
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Wan2.2 I2V 4-step Distill LoRAs (rest2art video generation)
|
| 2 |
+
|
| 3 |
+
The only re-hosted weights in [da-eun07/rest2art-models](../README.md): three rank-64, 4-step
|
| 4 |
+
distillation LoRAs for rest-state to articulated video synthesis (dual-expert MoE, two on the
|
| 5 |
+
high-noise expert including VBVR, one on the low-noise expert). Sources, attribution, and
|
| 6 |
+
licenses live in the [collection README](../README.md); this page is only how to run them.
|
| 7 |
+
|
| 8 |
+
## Contents
|
| 9 |
+
|
| 10 |
+
| File | Expert | dtype |
|
| 11 |
+
|---|---|---|
|
| 12 |
+
| `loras/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors` | high-noise | bf16 |
|
| 13 |
+
| `loras/Wan22_I2V_VBVR_HIGH_rank_64_fp16.safetensors` | high-noise (stacked) | fp16 |
|
| 14 |
+
| `loras/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors` | low-noise | bf16 |
|
| 15 |
+
|
| 16 |
+
## Reproduce
|
| 17 |
+
|
| 18 |
+
The base model, T5, and VAE are not re-hosted; `download_assets.sh` fetches them from Wan-AI.
|
| 19 |
+
|
| 20 |
+
```bash
|
| 21 |
+
bash scripts/download_assets.sh /path/to/base
|
| 22 |
+
python scripts/render_config.py --repo . --base /path/to/base/Wan2.2-I2V-A14B --out config.local.json
|
| 23 |
+
python scripts/minimal_infer.py --base /path/to/base/Wan2.2-I2V-A14B \
|
| 24 |
+
--config config.local.json --image front.png --prompt "..." --out out.mp4
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
Settings (`config/wan_moe_i2v_distill_with_lora_vbvr.json`): 4 steps, 720x1280, 81 frames,
|
| 28 |
+
`guide_scale=[3.5,3.5]`, `shift=5.0`, `cfg=false`, flash-attn 3, bf16. Both high-noise LoRAs are
|
| 29 |
+
stacked on the high-noise expert (each `strength: 1.0`); `use_image_encoder=false` (no CLIP).
|
wan2.2-i2v-loras/config/wan_moe_i2v_distill_with_lora_vbvr.json
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"infer_steps": 4,
|
| 3 |
+
"target_video_length": 81,
|
| 4 |
+
"text_len": 512,
|
| 5 |
+
"target_height": 720,
|
| 6 |
+
"target_width": 1280,
|
| 7 |
+
"self_attn_1_type": "flash_attn3",
|
| 8 |
+
"cross_attn_1_type": "flash_attn3",
|
| 9 |
+
"cross_attn_2_type": "flash_attn3",
|
| 10 |
+
"rope_type": "torch",
|
| 11 |
+
"sample_guide_scale": [
|
| 12 |
+
3.5,
|
| 13 |
+
3.5
|
| 14 |
+
],
|
| 15 |
+
"sample_shift": 5.0,
|
| 16 |
+
"enable_cfg": false,
|
| 17 |
+
"cpu_offload": false,
|
| 18 |
+
"offload_granularity": "block",
|
| 19 |
+
"t5_cpu_offload": false,
|
| 20 |
+
"vae_cpu_offload": false,
|
| 21 |
+
"use_image_encoder": false,
|
| 22 |
+
"boundary_step_index": 2,
|
| 23 |
+
"denoising_step_list": [
|
| 24 |
+
1000,
|
| 25 |
+
750,
|
| 26 |
+
500,
|
| 27 |
+
250
|
| 28 |
+
],
|
| 29 |
+
"lora_configs": [
|
| 30 |
+
{
|
| 31 |
+
"name": "high_noise_model",
|
| 32 |
+
"path": "loras/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors",
|
| 33 |
+
"strength": 1.0
|
| 34 |
+
},
|
| 35 |
+
{
|
| 36 |
+
"name": "high_noise_model",
|
| 37 |
+
"path": "loras/Wan22_I2V_VBVR_HIGH_rank_64_fp16.safetensors",
|
| 38 |
+
"strength": 1.0
|
| 39 |
+
},
|
| 40 |
+
{
|
| 41 |
+
"name": "low_noise_model",
|
| 42 |
+
"path": "loras/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors",
|
| 43 |
+
"strength": 1.0
|
| 44 |
+
}
|
| 45 |
+
]
|
| 46 |
+
}
|
wan2.2-i2v-loras/loras/Wan22_I2V_VBVR_HIGH_rank_64_fp16.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a4d2256f5c166c4a8d5b256b2134720aee2c56415c5efc37a6d060164eb06244
|
| 3 |
+
size 630695648
|
wan2.2-i2v-loras/loras/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:887c3bdeb74e83859c920438e16ca31f39ab18ce189abc5f0e36f8348c5bbb19
|
| 3 |
+
size 634645944
|
wan2.2-i2v-loras/loras/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8833bd4fd7c8eabebf0bc8ee5cfaf47f4f310ce116928a02c1adf8941dd4b0f1
|
| 3 |
+
size 739472104
|
wan2.2-i2v-loras/scripts/download_assets.sh
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Download the base assets that this repo intentionally does NOT re-host.
|
| 3 |
+
# All base weights belong to Wan-AI (Apache-2.0); we only mirror the LoRAs.
|
| 4 |
+
#
|
| 5 |
+
# Usage: bash scripts/download_assets.sh /path/to/dest
|
| 6 |
+
set -euo pipefail
|
| 7 |
+
|
| 8 |
+
DEST="${1:-./base}"
|
| 9 |
+
mkdir -p "$DEST"
|
| 10 |
+
|
| 11 |
+
echo ">> Downloading Wan-AI/Wan2.2-I2V-A14B (base DiT experts + UMT5-XXL T5 + VAE + tokenizer)"
|
| 12 |
+
echo ">> This is ~119 GB. Destination: $DEST/Wan2.2-I2V-A14B"
|
| 13 |
+
huggingface-cli download Wan-AI/Wan2.2-I2V-A14B \
|
| 14 |
+
--local-dir "$DEST/Wan2.2-I2V-A14B"
|
| 15 |
+
|
| 16 |
+
echo
|
| 17 |
+
echo ">> Done. Base model at: $DEST/Wan2.2-I2V-A14B"
|
| 18 |
+
echo ">> The 3 LoRAs are already in this repo under loras/."
|
wan2.2-i2v-loras/scripts/minimal_infer.py
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Minimal reproduction of our Wan2.2 I2V + VBVR 4-step video generation.
|
| 3 |
+
|
| 4 |
+
Distilled from the project's pipeline/video_gen.py. Loads the LightX2V pipeline
|
| 5 |
+
with the dual-expert distill LoRAs (incl. the stacked VBVR high-noise LoRA) and
|
| 6 |
+
runs one image-to-video generation.
|
| 7 |
+
|
| 8 |
+
Requires: lightx2v (https://github.com/ModelTC/LightX2V) + its deps (flash-attn 3).
|
| 9 |
+
|
| 10 |
+
python scripts/minimal_infer.py \
|
| 11 |
+
--base /path/to/Wan2.2-I2V-A14B \
|
| 12 |
+
--config config.local.json \
|
| 13 |
+
--image input.png --prompt "..." --out out.mp4
|
| 14 |
+
"""
|
| 15 |
+
import argparse
|
| 16 |
+
import torch
|
| 17 |
+
from lightx2v import LightX2VPipeline
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def patch_lightx2v_dtype(infer_dtype=torch.bfloat16):
|
| 21 |
+
"""Force all inference ops to bf16 (mixed fp16/bf16 LoRAs are cast on apply)."""
|
| 22 |
+
from lightx2v.common.ops.mm.mm_weight import MMWeight
|
| 23 |
+
from lightx2v.common.ops.conv.conv3d import Conv3dWeight
|
| 24 |
+
from lightx2v.common.ops.attn.flash_attn import FlashAttn2Weight, FlashAttn3Weight
|
| 25 |
+
|
| 26 |
+
_mm = MMWeight.apply
|
| 27 |
+
|
| 28 |
+
def mm(self, x, *a, **k):
|
| 29 |
+
if self.weight is not None and self.weight.dtype != infer_dtype:
|
| 30 |
+
self.weight = self.weight.to(infer_dtype)
|
| 31 |
+
return _mm(self, x.to(infer_dtype), *a, **k)
|
| 32 |
+
MMWeight.apply = mm
|
| 33 |
+
|
| 34 |
+
_cv = Conv3dWeight.apply
|
| 35 |
+
|
| 36 |
+
def cv(self, x, *a, **k):
|
| 37 |
+
if self.weight is not None and self.weight.dtype != infer_dtype:
|
| 38 |
+
self.weight = self.weight.to(infer_dtype)
|
| 39 |
+
return _cv(self, x.to(infer_dtype), *a, **k)
|
| 40 |
+
Conv3dWeight.apply = cv
|
| 41 |
+
|
| 42 |
+
def wrap(orig):
|
| 43 |
+
def f(self, q, k_, v, *a, **kw):
|
| 44 |
+
return orig(self, q.to(infer_dtype), k_.to(infer_dtype), v.to(infer_dtype), *a, **kw)
|
| 45 |
+
return f
|
| 46 |
+
FlashAttn2Weight.apply = wrap(FlashAttn2Weight.apply)
|
| 47 |
+
FlashAttn3Weight.apply = wrap(FlashAttn3Weight.apply)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
NEGATIVE_PROMPT = (
|
| 51 |
+
"色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,"
|
| 52 |
+
"整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,"
|
| 53 |
+
"画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,"
|
| 54 |
+
"静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走"
|
| 55 |
+
)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def main():
|
| 59 |
+
ap = argparse.ArgumentParser()
|
| 60 |
+
ap.add_argument("--base", required=True, help="Wan2.2-I2V-A14B dir")
|
| 61 |
+
ap.add_argument("--config", required=True, help="Rendered config (see render_config.py)")
|
| 62 |
+
ap.add_argument("--image", required=True)
|
| 63 |
+
ap.add_argument("--prompt", required=True)
|
| 64 |
+
ap.add_argument("--out", required=True)
|
| 65 |
+
ap.add_argument("--seed", type=int, default=42)
|
| 66 |
+
ap.add_argument("--negative", default=NEGATIVE_PROMPT)
|
| 67 |
+
args = ap.parse_args()
|
| 68 |
+
|
| 69 |
+
patch_lightx2v_dtype()
|
| 70 |
+
pipe = LightX2VPipeline(model_path=args.base, model_cls="wan2.2_moe_distill", task="i2v")
|
| 71 |
+
pipe.create_generator(config_json=args.config)
|
| 72 |
+
pipe.generate(
|
| 73 |
+
seed=args.seed,
|
| 74 |
+
image_path=args.image,
|
| 75 |
+
prompt=args.prompt,
|
| 76 |
+
negative_prompt=args.negative,
|
| 77 |
+
save_result_path=args.out,
|
| 78 |
+
)
|
| 79 |
+
print(f"Saved: {args.out}")
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
if __name__ == "__main__":
|
| 83 |
+
main()
|
wan2.2-i2v-loras/scripts/render_config.py
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Expand the portable config's relative LoRA paths into absolute paths.
|
| 3 |
+
|
| 4 |
+
The shipped config uses repo-relative LoRA paths (loras/*.safetensors) so the
|
| 5 |
+
bundle is portable. LightX2V expects absolute paths, so render a local copy.
|
| 6 |
+
|
| 7 |
+
Usage:
|
| 8 |
+
python scripts/render_config.py --repo . --base /path/to/Wan2.2-I2V-A14B \
|
| 9 |
+
--out config.local.json
|
| 10 |
+
"""
|
| 11 |
+
import argparse
|
| 12 |
+
import json
|
| 13 |
+
import os
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
def main():
|
| 17 |
+
ap = argparse.ArgumentParser()
|
| 18 |
+
ap.add_argument("--repo", required=True, help="Path to this bundle (dir with loras/, config/)")
|
| 19 |
+
ap.add_argument("--base", required=True, help="Path to Wan2.2-I2V-A14B base model dir")
|
| 20 |
+
ap.add_argument("--config", default=None,
|
| 21 |
+
help="Config template (default: <repo>/config/wan_moe_i2v_distill_with_lora_vbvr.json)")
|
| 22 |
+
ap.add_argument("--out", required=True, help="Where to write the rendered config")
|
| 23 |
+
args = ap.parse_args()
|
| 24 |
+
|
| 25 |
+
repo = os.path.abspath(args.repo)
|
| 26 |
+
cfg_path = args.config or os.path.join(repo, "config", "wan_moe_i2v_distill_with_lora_vbvr.json")
|
| 27 |
+
with open(cfg_path) as f:
|
| 28 |
+
cfg = json.load(f)
|
| 29 |
+
|
| 30 |
+
for lc in cfg.get("lora_configs", []):
|
| 31 |
+
p = lc["path"]
|
| 32 |
+
if not os.path.isabs(p):
|
| 33 |
+
p = os.path.join(repo, p)
|
| 34 |
+
if not os.path.exists(p):
|
| 35 |
+
raise FileNotFoundError(f"LoRA not found: {p}")
|
| 36 |
+
lc["path"] = os.path.abspath(p)
|
| 37 |
+
|
| 38 |
+
with open(args.out, "w") as f:
|
| 39 |
+
json.dump(cfg, f, indent=4)
|
| 40 |
+
print(f"Wrote {args.out}")
|
| 41 |
+
print(f"Base model dir (pass as model_path): {os.path.abspath(args.base)}")
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
if __name__ == "__main__":
|
| 45 |
+
main()
|