nickchiu
/

Refocusing-Any-Video

@@ -6,144 +6,64 @@ tags:
 - video-refocusing
 - video-diffusion
 - lora
-model-index: []
 license: other
 license_name: stable-video-diffusion-community
 license_link: https://huggingface.co/nickchiu/Refocusing-Any-Video/blob/main/LICENSE-SVD.txt
 ---
-# Refocus Any Video: inference checkpoints
-Powered by Stability AI.
-Model bundle for Refocusing Any Video (RAV). Video Defocus Deblurring (VDD) recovers an all-in-focus video from an out-of-focus input; Video Bokeh Rendering (VBR) renders a target focal plane using the recovered video and a depth sequence. Optional Identity Adapter and Track4Gen Refiner variants condition VDD on reference faces.
-Use these weights with the accompanying Refocus Any Video source implementation and its README. This bundle contains adapters and a learned input convolution; it is not a standalone Diffusers pipeline and does not include the Stable Video Diffusion base model.
-## Included files
-The repository root maps directly to the source repository's `checkpoints/` directory:
-```text
-vdd_weights/checkpoint-90000/
-  unet_lora/adapter_config.json
-  unet_lora/adapter_model.safetensors
-vbr_weights/checkpoint-100000/
-  unet_lora/adapter_config.json
-  unet_lora/adapter_model.safetensors
-  unet_conv_in_refocus.pt
-identity_adapter/ip_adapter.bin
-identity_adapter_track4gen/ip_adapter.bin
-facenet/
-  pnet.pt
-  rnet.pt
-  onet.pt
-  LICENSE.md
-README.md
-LICENSE-SVD.txt
-NOTICE.txt
-bundle_manifest.json
-```
-| Component | Purpose and checkpoint provenance |
-| --- | --- |
-| VDD LoRA | SVD UNet adapters from the supplied `checkpoint-90000` directory. |
-| VBR LoRA and input convolution | Adapters and learned 12-channel input convolution from the supplied `checkpoint-100000` directory. Both files are required for VBR. |
-| Identity Adapter | Copied from the supplied identity experiment's `checkpoint-5000/ip_adapter.bin`. Use with `infer_vdd_identity.sh`. |
-| Identity Adapter + Track4Gen Refiner | Copied from the supplied Track4Gen experiment's `checkpoint-25000/ip_adapter.bin`. Use with `infer_vdd_identity_track4gen.sh`. |
-| MTCNN | Third-party `pnet`, `rnet`, and `onet` detector weights from the implementation's vendored facenet-pytorch assets. The upstream MIT license is retained in `facenet/LICENSE.md`. |
-The VDD and VBR PEFT configurations both specify LoRA rank 32 and alpha 32. VDD also adapts `conv_in`; VBR saves its expanded input convolution separately from its LoRA. The identity variants load on top of the VDD checkpoint, rather than replacing it.
-Checkpoint step numbers above are the source directory labels. They do not independently establish dataset size, number of epochs, or training duration. Optimizer, scheduler, and random-state snapshots and large full-UNet experiment snapshots are excluded; this is an inference release rather than an exact training-resume bundle. `bundle_manifest.json` records repository-relative file paths, sizes, and SHA-256 checksums for the distributed assets.
-## Download
-From the source repository root, install and activate the environment described in its README, then run:
-```bash
-python tools/checkpoints/transfer_checkpoints.py download \
-  --repo-id nickchiu/Refocusing-Any-Video --local-dir ./checkpoints
-```
-Alternatively, download with the Hugging Face Python client:
-```python
-from huggingface_hub import snapshot_download
-snapshot_download(repo_id="nickchiu/Refocusing-Any-Video", local_dir="./checkpoints")
-```
-The download writes `checkpoints/vdd_weights/`, `checkpoints/vbr_weights/`, and the other folders shown above.
-## Base and auxiliary models
-Obtain these models separately from their providers. Their weights and licenses are separate from the authors' RAV assets.
-| Model | Source | Local path from source repository root |
-| --- | --- | --- |
-| Stable Video Diffusion, 14-frame img2vid | [stabilityai/stable-video-diffusion-img2vid](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) | `checkpoints/svdh/` |
-| Video Depth Anything, relative-depth Large | [Video Depth Anything](https://github.com/DepthAnything/Video-Depth-Anything) | `checkpoints/video_depth_anything_vitl.pth` |
-| FaRL ViT-B/16, ep16 | [FaRL pretrained backbones](https://github.com/FacePerceiver/FaRL#pre-trained-backbones) | `checkpoints/farl/FaRL-Base-Patch16-LAIONFace20M-ep16.pth` |
-| ArcFace IR-SE50 | [pSp pretrained-model instructions](https://github.com/eladrich/pixel2style2pixel#pretrained-models) | `checkpoints/arcface/model_ir_se50.pth` |
-| OpenAI CLIP ViT-B/16 | [OpenAI CLIP](https://github.com/openai/CLIP) | Downloaded into the CLIP cache by `clip.load("ViT-B/16")` for FaRL initialization. |
-The provided `python tools/checkpoints/download_svd_weights.py` helper downloads SVD to `checkpoints/svdh/`. Retain the SVD VAE and UNet `fp16` weight variants: the inference loaders request these files even when inference precision is changed. Follow each provider's authentication/access instructions when required.
-The shared VDD wrapper constructs ArcFace, FaRL, and MTCNN even when identity conditioning is disabled, so baseline VDD also requires these auxiliary models and the CLIP cache. Optional keypoint preprocessing, facial landmark extraction, and training/data generation have additional external model requirements documented in the code repository; these are not needed for the basic VDD → VBR example below.
-## Inference
-Run from the source repository root after completing the environment and all checkpoint setup. Put the input video at `inputs/input.mp4`.
-```bash
-mkdir -p inputs results
-# Estimate depth from the input video.
-python VideoDepthAnything/infer_video_depth.py \
-  --input ./inputs/input.mp4 --output ./inputs/depth.mp4 \
-  --checkpoint ./checkpoints/video_depth_anything_vitl.pth \
-  --encoder vitl --device cuda
-# Recover an all-in-focus video.
-INPUT_VIDEO=./inputs/input.mp4 \
-CHECKPOINT=./checkpoints/vdd_weights/checkpoint-90000 \
-OUTPUT_DIR=./results/vdd OUTPUT_FPS=24 FP16=1 \
-  bash infer_vdd.sh
-# Render a focal plane using the all-in-focus video and depth.
-INPUT_VIDEO=./results/vdd/output.mp4 DEPTH_INPUT=./inputs/depth.mp4 \
-CHECKPOINT=./checkpoints/vbr_weights/checkpoint-100000 \
-FOCAL_PLANE=0.5 COC=27 OUTPUT_DIR=./results/vbr OUTPUT_FPS=24 FP16=1 \
-  bash infer_vbr_defocus_map.sh
-```
-The final video is `results/vbr/output.mp4`. Keep RGB and depth frame counts, order, and framing aligned. `FOCAL_PLANE` uses normalized disparity in `[0, 1]`, with larger values focusing nearer. `COC` controls the defocus scale. The launchers save at the requested integer playback FPS without resampling and do not preserve audio. See the source README for environment setup, dry runs, time-varying focal planes, resolution controls, and troubleshooting.
-For optional identity-guided VDD, provide a clear reference face image and select the matching adapter:
 ```bash
-MODE=single INPUT_VIDEO=./inputs/input.mp4 \
-REFERENCE_IMAGE=./inputs/reference_face.jpg KEYPOINT_VIDEO="" \
-IP_ADAPTER_PATH=./checkpoints/identity_adapter/ip_adapter.bin \
-HEIGHT="" WIDTH="" OUTPUT_DIR=./results/vdd_identity \
-OUTPUT_FPS=24 FP16=1 bash infer_vdd_identity.sh
 ```
-For Track4Gen, use `infer_vdd_identity_track4gen.sh` and `IP_ADAPTER_PATH=./checkpoints/identity_adapter_track4gen/ip_adapter.bin`. Both use `checkpoints/vdd_weights/checkpoint-90000` by default. Missing adapter files can cause the current implementation to continue with initialized weights; supply the matching trained file for meaningful results.
-## Training and evaluation information
-The source distribution includes training runners, historical experiment configuration templates, losses, renderers, and dataset split identifiers. This release does not provide an independently verified, checkpoint-specific training-data description, complete training logs, or benchmark metrics. No quantitative performance claims are made here.
-The release preparation checked file inventory and source hashes, adapter configuration and stored tensor keys/headers, Python/shell syntax, and transfer previews. A full GPU end-to-end inference or training run was not performed during release preparation. These checks do not establish reconstruction quality, temporal consistency, or identity fidelity.
-## Intended use and limitations
-Use the implementation to explore video defocus deblurring, depth-guided refocusing, and reference-conditioned face restoration. Recovered details are model predictions and may differ from the original scene or identity. Depth errors, severe blur, motion, occlusions, or mismatched references can affect results. Hardware requirements and usable resolution depend on the input and available GPU memory; no measured minimum-memory guarantee is provided.
-## Licensing and third-party attribution
-No additional license for the authors' project-specific model weights was supplied in the source material; this card does not assign one. Stable Video Diffusion and any applicable derivatives remain subject to the base model's terms. The base SVD weights are downloaded separately from the provider. Refer to the retained `LICENSE-SVD.txt`, `NOTICE.txt`, and the provider's model page.
-MTCNN assets are redistributed with their upstream MIT notice, copyright © 2019 Timothy Esler, in `facenet/LICENSE.md`; that notice applies to those assets rather than to the complete RAV bundle. Other auxiliary models are obtained separately under their respective providers' terms.

 - video-refocusing
 - video-diffusion
 - lora
 license: other
 license_name: stable-video-diffusion-community
 license_link: https://huggingface.co/nickchiu/Refocusing-Any-Video/blob/main/LICENSE-SVD.txt
 ---
+# Refocusing Any Video: Temporally Consistent Video Refocusing for Dynamic Focal Planes
+**SIGGRAPH Asia 2026**
+Chee Kim Gan, [Pin-Yen Chiu](https://itsnickchiu.github.io/), I-Sheng Fang, Jun-Cheng Chen
+Research Center for Information Technology Innovation, Academia Sinica
+Inference checkpoints for Refocusing Any Video (RAV). **Video Defocus Deblurring (VDD)** recovers an all-in-focus video; **Video Bokeh Rendering (VBR)** refocuses it using depth and a chosen focal plane. Optional identity adapters guide face restoration.
+[Source code and inference instructions](https://github.com/aiiu-lab/Refocusing-Any-Video)
+## Release status
+- [x] Inference code and pretrained checkpoints
+- [ ] Training code — being prepared for a future release
+## Checkpoints
+| Folder | Contents |
+| --- | --- |
+| `vdd_weights/checkpoint-90000/` | VDD LoRA weights |
+| `vbr_weights/checkpoint-100000/` | VBR LoRA weights and required input convolution |
+| `identity_adapter/` | Identity Adapter |
+| `identity_adapter_track4gen/` | Identity Adapter with Track4Gen Refiner |
+| `facenet/` | MTCNN face detector weights |
+## Download and inference
+After setting up the [code environment](https://github.com/aiiu-lab/Refocusing-Any-Video#environment), run from the cloned code repository root:
 ```bash
+python tools/checkpoints/transfer_checkpoints.py download \
+  --repo-id nickchiu/Refocusing-Any-Video --local-dir ./checkpoints
 ```
+Download SVD, Video Depth Anything, and the auxiliary face models separately using the [checkpoint guide](https://github.com/aiiu-lab/Refocusing-Any-Video/blob/main/checkpoints/README.md), then follow the [inference example](https://github.com/aiiu-lab/Refocusing-Any-Video#inference). These adapters require the code and SVD base model. Generated details may differ from the original scene.
+## Models and licenses
+RAV uses Stable Video Diffusion (SVD), Video Depth Anything, FaRL, ArcFace, OpenAI CLIP, and facenet-pytorch.
+SVD and applicable derivatives are subject to the [SVD license](LICENSE-SVD.txt) and [NOTICE](NOTICE.txt). **Powered by Stability AI** (attribution required by the SVD license). MTCNN retains its [MIT license](facenet/LICENSE.md); other models follow their providers' terms. An additional project-specific license has not yet been specified.
+## Citation
+If you use this work, please cite:
+```bibtex
+@misc{gan2026refocusinganyvideo,
+  title  = {Refocusing Any Video: Temporally Consistent Video Refocusing for Dynamic Focal Planes},
+  author = {Gan, Chee Kim and Chiu, Pin-Yen and Fang, I-Sheng and Chen, Jun-Cheng},
+  year   = {2026},
+  note   = {SIGGRAPH Asia 2026},
+  url    = {https://github.com/aiiu-lab/Refocusing-Any-Video}
+}
+```