da-eun07 commited on
Commit
032d750
·
verified ·
1 Parent(s): fede4a2

Upload folder using huggingface_hub

Browse files
.gitattributes CHANGED
@@ -1,35 +1,2 @@
1
- *.7z filter=lfs diff=lfs merge=lfs -text
2
- *.arrow filter=lfs diff=lfs merge=lfs -text
3
- *.bin filter=lfs diff=lfs merge=lfs -text
4
- *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
- *.ftz filter=lfs diff=lfs merge=lfs -text
7
- *.gz filter=lfs diff=lfs merge=lfs -text
8
- *.h5 filter=lfs diff=lfs merge=lfs -text
9
- *.joblib filter=lfs diff=lfs merge=lfs -text
10
- *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
- *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
- *.model filter=lfs diff=lfs merge=lfs -text
13
- *.msgpack filter=lfs diff=lfs merge=lfs -text
14
- *.npy filter=lfs diff=lfs merge=lfs -text
15
- *.npz filter=lfs diff=lfs merge=lfs -text
16
- *.onnx filter=lfs diff=lfs merge=lfs -text
17
- *.ot filter=lfs diff=lfs merge=lfs -text
18
- *.parquet filter=lfs diff=lfs merge=lfs -text
19
- *.pb filter=lfs diff=lfs merge=lfs -text
20
- *.pickle filter=lfs diff=lfs merge=lfs -text
21
- *.pkl filter=lfs diff=lfs merge=lfs -text
22
- *.pt filter=lfs diff=lfs merge=lfs -text
23
- *.pth filter=lfs diff=lfs merge=lfs -text
24
- *.rar filter=lfs diff=lfs merge=lfs -text
25
  *.safetensors filter=lfs diff=lfs merge=lfs -text
26
- saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
- *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
- *.tflite filter=lfs diff=lfs merge=lfs -text
30
- *.tgz filter=lfs diff=lfs merge=lfs -text
31
- *.wasm filter=lfs diff=lfs merge=lfs -text
32
- *.xz filter=lfs diff=lfs merge=lfs -text
33
- *.zip filter=lfs diff=lfs merge=lfs -text
34
- *.zst filter=lfs diff=lfs merge=lfs -text
35
- *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  *.safetensors filter=lfs diff=lfs merge=lfs -text
2
+ *.pth filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
README.md ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ pretty_name: "rest2art Model Checkpoints"
4
+ base_model:
5
+ - Wan-AI/Wan2.2-I2V-A14B
6
+ task_categories:
7
+ - image-to-video
8
+ language:
9
+ - en
10
+ tags:
11
+ - articulated-objects
12
+ - reconstruction
13
+ - image-to-video
14
+ - lora
15
+ - wan2.2
16
+ ---
17
+
18
+ # [ECCV'26] Articulated Object Reconstruction from Rest-State Observation
19
+
20
+ <!-- TODO: replace with real links once public -->
21
+ [![Project Page](https://img.shields.io/badge/Project_Page-Rest2Art-blue)](https://example.com)
22
+ [![arXiv](https://img.shields.io/badge/arXiv-TODO-B31B1B?logo=arxiv)](https://arxiv.org)
23
+ [![GitHub](https://img.shields.io/badge/GitHub-Repository-black?logo=github)](https://github.com)
24
+ [![Dataset](https://img.shields.io/badge/Dataset-rest2art-yellow)](https://huggingface.co/datasets/da-eun07/rest2art)
25
+
26
+ ## Model Checkpoints
27
+
28
+ Pretrained models used by the rest2art pipeline. We only re-host the Wan2.2 video-generation
29
+ LoRAs, whose inference is config-sensitive and hard to reproduce. Every other model is linked to
30
+ its original repo (cleaner attribution and licensing), so fetch those from the source.
31
+
32
+ | Model | Hosting | Source | License |
33
+ |---|---|---|---|
34
+ | GPT (OpenAI API) | API, no ckpt | OpenAI | n/a |
35
+ | SAM 3 | link | [facebook/sam3](https://huggingface.co/facebook/sam3) | see repo |
36
+ | Qwen3-VL-8B-Instruct | link | [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) | Apache-2.0 |
37
+ | Wan2.2-I2V-A14B + 3 LoRAs | hosted (`wan2.2-i2v-loras/`) | [Wan-AI](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B), [lightx2v](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras), [Kijai](https://huggingface.co/Kijai/WanVideo_comfy) | Apache-2.0 |
38
+ | CoTracker3 (scaled_offline) | link | [facebook/cotracker3](https://huggingface.co/facebook/cotracker3) | CC BY-NC 4.0 |
39
+
40
+
41
+ ## Get started
42
+
43
+ Put every checkpoint under one `ckpt/` root, then point the pipeline flags at it.
44
+
45
+ ### 1. Download checkpoints
46
+
47
+ ```bash
48
+ mkdir -p ckpt && cd ckpt
49
+
50
+ # this collection: Wan2.2 LoRAs + config + loader
51
+ huggingface-cli download da-eun07/rest2art-models --local-dir ./rest2art-models
52
+
53
+ # Wan2.2 base model, UMT5-XXL T5, VAE (~119 GB) from Wan-AI
54
+ bash ./rest2art-models/wan2.2-i2v-loras/scripts/download_assets.sh ./Wan-AI
55
+
56
+ # CoTracker3 offline checkpoint
57
+ huggingface-cli download facebook/cotracker3 scaled_offline.pth --local-dir ./co-tracker
58
+ ```
59
+
60
+ SAM 3 (`facebook/sam3`) and Qwen3-VL-8B (`Qwen/Qwen3-VL-8B-Instruct`) are pulled from their repos
61
+ on first run, so no manual download is needed. Resulting layout:
62
+
63
+ ```
64
+ ckpt/
65
+ rest2art-models/wan2.2-i2v-loras/loras/*.safetensors
66
+ Wan-AI/Wan2.2-I2V-A14B/
67
+ co-tracker/scaled_offline.pth
68
+ ```
69
+
70
+ ### 2. Render the Wan2.2 config
71
+
72
+ The LightX2V config needs absolute LoRA paths, so render a local copy:
73
+
74
+ ```bash
75
+ cd ckpt/rest2art-models/wan2.2-i2v-loras
76
+ python scripts/render_config.py --repo . --base ../../Wan-AI/Wan2.2-I2V-A14B --out config.local.json
77
+ ```
78
+
79
+ ### 3. Run the pipeline
80
+
81
+ From the rest2art project root, point the flags at your `ckpt/` root:
82
+
83
+ ```bash
84
+ python -m rest2art.pipeline.main \
85
+ --scene <scene> \
86
+ --wan_path ckpt/Wan-AI/Wan2.2-I2V-A14B \
87
+ --config_json ckpt/rest2art-models/wan2.2-i2v-loras/config.local.json \
88
+ --cotracker_ckpt ckpt/co-tracker/scaled_offline.pth \
89
+ --vlm_path Qwen/Qwen3-VL-8B-Instruct
90
+ ```
91
+
92
+ To run only the Wan2.2 video step, follow [`wan2.2-i2v-loras/`](./wan2.2-i2v-loras).
93
+
94
+ ## License
95
+
96
+ Hosted LoRAs are Apache-2.0 (Wan-AI, lightx2v); the VBVR LoRA follows
97
+ [Kijai/WanVideo_comfy](https://huggingface.co/Kijai/WanVideo_comfy) terms. Linked models keep
98
+ their own licenses (see table). Config and scripts are Apache-2.0. To request removal from this
99
+ mirror, open a discussion.
100
+
101
+ ## Citation
102
+
103
+ ```bibtex
104
+ @inproceedings{lee2026rest2art,
105
+ title={Articulated Object Reconstruction from Rest-State Observation},
106
+ author={Lee, Daeun and Lee, Jaeah and Kim, Woosung and Jung, Haebeom and Park, Jaesik},
107
+ booktitle={European Conference on Computer Vision (ECCV)},
108
+ year={2026}
109
+ }
110
+ ```
wan2.2-i2v-loras/README.md ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Wan2.2 I2V 4-step Distill LoRAs (rest2art video generation)
2
+
3
+ The only re-hosted weights in [da-eun07/rest2art-models](../README.md): three rank-64, 4-step
4
+ distillation LoRAs for rest-state to articulated video synthesis (dual-expert MoE, two on the
5
+ high-noise expert including VBVR, one on the low-noise expert). Sources, attribution, and
6
+ licenses live in the [collection README](../README.md); this page is only how to run them.
7
+
8
+ ## Contents
9
+
10
+ | File | Expert | dtype |
11
+ |---|---|---|
12
+ | `loras/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors` | high-noise | bf16 |
13
+ | `loras/Wan22_I2V_VBVR_HIGH_rank_64_fp16.safetensors` | high-noise (stacked) | fp16 |
14
+ | `loras/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors` | low-noise | bf16 |
15
+
16
+ ## Reproduce
17
+
18
+ The base model, T5, and VAE are not re-hosted; `download_assets.sh` fetches them from Wan-AI.
19
+
20
+ ```bash
21
+ bash scripts/download_assets.sh /path/to/base
22
+ python scripts/render_config.py --repo . --base /path/to/base/Wan2.2-I2V-A14B --out config.local.json
23
+ python scripts/minimal_infer.py --base /path/to/base/Wan2.2-I2V-A14B \
24
+ --config config.local.json --image front.png --prompt "..." --out out.mp4
25
+ ```
26
+
27
+ Settings (`config/wan_moe_i2v_distill_with_lora_vbvr.json`): 4 steps, 720x1280, 81 frames,
28
+ `guide_scale=[3.5,3.5]`, `shift=5.0`, `cfg=false`, flash-attn 3, bf16. Both high-noise LoRAs are
29
+ stacked on the high-noise expert (each `strength: 1.0`); `use_image_encoder=false` (no CLIP).
wan2.2-i2v-loras/config/wan_moe_i2v_distill_with_lora_vbvr.json ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "infer_steps": 4,
3
+ "target_video_length": 81,
4
+ "text_len": 512,
5
+ "target_height": 720,
6
+ "target_width": 1280,
7
+ "self_attn_1_type": "flash_attn3",
8
+ "cross_attn_1_type": "flash_attn3",
9
+ "cross_attn_2_type": "flash_attn3",
10
+ "rope_type": "torch",
11
+ "sample_guide_scale": [
12
+ 3.5,
13
+ 3.5
14
+ ],
15
+ "sample_shift": 5.0,
16
+ "enable_cfg": false,
17
+ "cpu_offload": false,
18
+ "offload_granularity": "block",
19
+ "t5_cpu_offload": false,
20
+ "vae_cpu_offload": false,
21
+ "use_image_encoder": false,
22
+ "boundary_step_index": 2,
23
+ "denoising_step_list": [
24
+ 1000,
25
+ 750,
26
+ 500,
27
+ 250
28
+ ],
29
+ "lora_configs": [
30
+ {
31
+ "name": "high_noise_model",
32
+ "path": "loras/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors",
33
+ "strength": 1.0
34
+ },
35
+ {
36
+ "name": "high_noise_model",
37
+ "path": "loras/Wan22_I2V_VBVR_HIGH_rank_64_fp16.safetensors",
38
+ "strength": 1.0
39
+ },
40
+ {
41
+ "name": "low_noise_model",
42
+ "path": "loras/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors",
43
+ "strength": 1.0
44
+ }
45
+ ]
46
+ }
wan2.2-i2v-loras/loras/Wan22_I2V_VBVR_HIGH_rank_64_fp16.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a4d2256f5c166c4a8d5b256b2134720aee2c56415c5efc37a6d060164eb06244
3
+ size 630695648
wan2.2-i2v-loras/loras/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:887c3bdeb74e83859c920438e16ca31f39ab18ce189abc5f0e36f8348c5bbb19
3
+ size 634645944
wan2.2-i2v-loras/loras/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8833bd4fd7c8eabebf0bc8ee5cfaf47f4f310ce116928a02c1adf8941dd4b0f1
3
+ size 739472104
wan2.2-i2v-loras/scripts/download_assets.sh ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Download the base assets that this repo intentionally does NOT re-host.
3
+ # All base weights belong to Wan-AI (Apache-2.0); we only mirror the LoRAs.
4
+ #
5
+ # Usage: bash scripts/download_assets.sh /path/to/dest
6
+ set -euo pipefail
7
+
8
+ DEST="${1:-./base}"
9
+ mkdir -p "$DEST"
10
+
11
+ echo ">> Downloading Wan-AI/Wan2.2-I2V-A14B (base DiT experts + UMT5-XXL T5 + VAE + tokenizer)"
12
+ echo ">> This is ~119 GB. Destination: $DEST/Wan2.2-I2V-A14B"
13
+ huggingface-cli download Wan-AI/Wan2.2-I2V-A14B \
14
+ --local-dir "$DEST/Wan2.2-I2V-A14B"
15
+
16
+ echo
17
+ echo ">> Done. Base model at: $DEST/Wan2.2-I2V-A14B"
18
+ echo ">> The 3 LoRAs are already in this repo under loras/."
wan2.2-i2v-loras/scripts/minimal_infer.py ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Minimal reproduction of our Wan2.2 I2V + VBVR 4-step video generation.
3
+
4
+ Distilled from the project's pipeline/video_gen.py. Loads the LightX2V pipeline
5
+ with the dual-expert distill LoRAs (incl. the stacked VBVR high-noise LoRA) and
6
+ runs one image-to-video generation.
7
+
8
+ Requires: lightx2v (https://github.com/ModelTC/LightX2V) + its deps (flash-attn 3).
9
+
10
+ python scripts/minimal_infer.py \
11
+ --base /path/to/Wan2.2-I2V-A14B \
12
+ --config config.local.json \
13
+ --image input.png --prompt "..." --out out.mp4
14
+ """
15
+ import argparse
16
+ import torch
17
+ from lightx2v import LightX2VPipeline
18
+
19
+
20
+ def patch_lightx2v_dtype(infer_dtype=torch.bfloat16):
21
+ """Force all inference ops to bf16 (mixed fp16/bf16 LoRAs are cast on apply)."""
22
+ from lightx2v.common.ops.mm.mm_weight import MMWeight
23
+ from lightx2v.common.ops.conv.conv3d import Conv3dWeight
24
+ from lightx2v.common.ops.attn.flash_attn import FlashAttn2Weight, FlashAttn3Weight
25
+
26
+ _mm = MMWeight.apply
27
+
28
+ def mm(self, x, *a, **k):
29
+ if self.weight is not None and self.weight.dtype != infer_dtype:
30
+ self.weight = self.weight.to(infer_dtype)
31
+ return _mm(self, x.to(infer_dtype), *a, **k)
32
+ MMWeight.apply = mm
33
+
34
+ _cv = Conv3dWeight.apply
35
+
36
+ def cv(self, x, *a, **k):
37
+ if self.weight is not None and self.weight.dtype != infer_dtype:
38
+ self.weight = self.weight.to(infer_dtype)
39
+ return _cv(self, x.to(infer_dtype), *a, **k)
40
+ Conv3dWeight.apply = cv
41
+
42
+ def wrap(orig):
43
+ def f(self, q, k_, v, *a, **kw):
44
+ return orig(self, q.to(infer_dtype), k_.to(infer_dtype), v.to(infer_dtype), *a, **kw)
45
+ return f
46
+ FlashAttn2Weight.apply = wrap(FlashAttn2Weight.apply)
47
+ FlashAttn3Weight.apply = wrap(FlashAttn3Weight.apply)
48
+
49
+
50
+ NEGATIVE_PROMPT = (
51
+ "色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,"
52
+ "整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,"
53
+ "画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,"
54
+ "静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走"
55
+ )
56
+
57
+
58
+ def main():
59
+ ap = argparse.ArgumentParser()
60
+ ap.add_argument("--base", required=True, help="Wan2.2-I2V-A14B dir")
61
+ ap.add_argument("--config", required=True, help="Rendered config (see render_config.py)")
62
+ ap.add_argument("--image", required=True)
63
+ ap.add_argument("--prompt", required=True)
64
+ ap.add_argument("--out", required=True)
65
+ ap.add_argument("--seed", type=int, default=42)
66
+ ap.add_argument("--negative", default=NEGATIVE_PROMPT)
67
+ args = ap.parse_args()
68
+
69
+ patch_lightx2v_dtype()
70
+ pipe = LightX2VPipeline(model_path=args.base, model_cls="wan2.2_moe_distill", task="i2v")
71
+ pipe.create_generator(config_json=args.config)
72
+ pipe.generate(
73
+ seed=args.seed,
74
+ image_path=args.image,
75
+ prompt=args.prompt,
76
+ negative_prompt=args.negative,
77
+ save_result_path=args.out,
78
+ )
79
+ print(f"Saved: {args.out}")
80
+
81
+
82
+ if __name__ == "__main__":
83
+ main()
wan2.2-i2v-loras/scripts/render_config.py ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Expand the portable config's relative LoRA paths into absolute paths.
3
+
4
+ The shipped config uses repo-relative LoRA paths (loras/*.safetensors) so the
5
+ bundle is portable. LightX2V expects absolute paths, so render a local copy.
6
+
7
+ Usage:
8
+ python scripts/render_config.py --repo . --base /path/to/Wan2.2-I2V-A14B \
9
+ --out config.local.json
10
+ """
11
+ import argparse
12
+ import json
13
+ import os
14
+
15
+
16
+ def main():
17
+ ap = argparse.ArgumentParser()
18
+ ap.add_argument("--repo", required=True, help="Path to this bundle (dir with loras/, config/)")
19
+ ap.add_argument("--base", required=True, help="Path to Wan2.2-I2V-A14B base model dir")
20
+ ap.add_argument("--config", default=None,
21
+ help="Config template (default: <repo>/config/wan_moe_i2v_distill_with_lora_vbvr.json)")
22
+ ap.add_argument("--out", required=True, help="Where to write the rendered config")
23
+ args = ap.parse_args()
24
+
25
+ repo = os.path.abspath(args.repo)
26
+ cfg_path = args.config or os.path.join(repo, "config", "wan_moe_i2v_distill_with_lora_vbvr.json")
27
+ with open(cfg_path) as f:
28
+ cfg = json.load(f)
29
+
30
+ for lc in cfg.get("lora_configs", []):
31
+ p = lc["path"]
32
+ if not os.path.isabs(p):
33
+ p = os.path.join(repo, p)
34
+ if not os.path.exists(p):
35
+ raise FileNotFoundError(f"LoRA not found: {p}")
36
+ lc["path"] = os.path.abspath(p)
37
+
38
+ with open(args.out, "w") as f:
39
+ json.dump(cfg, f, indent=4)
40
+ print(f"Wrote {args.out}")
41
+ print(f"Base model dir (pass as model_path): {os.path.abspath(args.base)}")
42
+
43
+
44
+ if __name__ == "__main__":
45
+ main()