| ---
|
| license: apache-2.0
|
| tags:
|
| - video-frame-interpolation
|
| - frame-interpolation
|
| - video
|
| - comfyui
|
| - ema-vfi
|
| ---
|
|
|
| # EMA-VFI, released checkpoints as safetensors
|
|
|
| The four checkpoints released with **EMA-VFI**, converted from the original pickles to
|
| safetensors and nothing else. The weights are unchanged, tensor for tensor.
|
|
|
| Upstream: <https://github.com/MCG-NJU/EMA-VFI>. This is a mirror, not a new model. All credit
|
| for the network and the training belongs to its authors.
|
|
|
| ## Why this exists
|
|
|
| The originals are distributed as bare `state_dict` **pickles** inside a Google Drive folder.
|
| That is two problems for anyone building on them. A pickle executes arbitrary code when it is
|
| loaded, and a Drive folder gives out per-file links that are not stable enough to script
|
| against, so tools end up telling people to fetch a file by hand.
|
|
|
| Safetensors cannot execute anything on load, and a Hub URL stays put.
|
|
|
| ## The files
|
|
|
| | File | Size | Parameters | Quality | Multiplier above 2 |
|
| |---|--:|--:|---|---|
|
| | `ours_t.safetensors` | 250.5 MiB | 65.66M | best | **yes** |
|
| | `ours.safetensors` | 250.5 MiB | 65.66M | best | no |
|
| | `ours_small_t.safetensors` | 55.3 MiB | 14.49M | lower | **yes** |
|
| | `ours_small.safetensors` | 55.3 MiB | 14.49M | lower | no |
|
|
|
| Take **`ours_t`** unless you have a reason not to. The `_t` pair was trained to land anywhere
|
| between two frames, so it can do 3x, 4x and beyond. The other two only ever land halfway, which
|
| limits them to a multiplier of 2. The `small` pair is a narrower network: quicker, and visibly
|
| weaker on fast motion.
|
|
|
| ## Provenance
|
|
|
| Every file was downloaded from the authors' own Drive folder, `16jUa3HkQ85Z5lb5gce1yoaWkP-rdCd0o`,
|
| linked from the upstream README. No third-party mirror was used.
|
|
|
| | Original | SHA-256 of the source pickle |
|
| |---|---|
|
| | `ours_t.pkl` | `c09e82d193e303d7b91f1e36bf535886b820cb56105574355211d60a6b557a99` |
|
| | `ours.pkl` | `f9085bb32d8b2eaa70eb1df08cc25bc7e973c50a74d74fb593ebe0d5e100a11f` |
|
| | `ours_small_t.pkl` | `ad845671f7250d408937718aca19eaf87aa1ca79215070f7641eea332b703233` |
|
| | `ours_small.pkl` | `ef0e455a7fe75c24d415bbcd83c29a9c0134ba853942bde0cd4dc4ed01e3f98f` |
|
|
|
| Each hash is repeated in its safetensors file's own metadata, under `source_sha256`, alongside
|
| the architecture the weights belong to. `conversion_report.json` in this repo records the whole
|
| run.
|
|
|
| ## What was checked before publishing
|
|
|
| 1. Loaded with `torch.load(weights_only=True)`, which refuses arbitrary pickle opcodes, so a
|
| payload would have raised rather than run.
|
| 2. Built the network at the matching size and required `load_state_dict(strict=True)`, which
|
| proves the tensors really are this architecture and that none is missing or spare.
|
| 3. Reloaded each converted file and required the same strict fit, then compared every tensor
|
| against the original with `torch.equal`. All exact.
|
|
|
| ## What the conversion changed
|
|
|
| Two things, both mechanical, and both what a loader had to do anyway:
|
|
|
| - **The `module.` prefix is stripped.** The originals were saved from a `DataParallel` wrapper,
|
| so every key carried it.
|
| - **`attn_mask` and `HW` are dropped.** These are shifted-window buffers the backbone registers
|
| during `forward` from whatever padded size it is given, so a saved pair belongs to the
|
| resolution it was trained at and is wrong at any other. Upstream's own loader drops them too.
|
| 8 such keys in the full models, 4 in the small ones.
|
|
|
| Everything else is byte-identical, float32 throughout.
|
|
|
| ## Use it in ComfyUI
|
|
|
| With [WAS Node Suite](https://github.com/WASasquatch/was-node-suite-comfyui) and its
|
| **EMA-VFI Frame Interpolation** node, either way round.
|
|
|
| **Let it fetch.** Set `features.network: true` in the pack's `config.yaml`, pick a checkpoint in
|
| the node's menu and queue the graph. The file is downloaded once into `ComfyUI/models/EMA-VFI`
|
| and kept. Nothing is fetched at startup, only on the run that needs it.
|
|
|
| **Or place it yourself.** Download the file and put it here:
|
|
|
| ```
|
| ComfyUI/models/EMA-VFI/ours_t.safetensors
|
| ```
|
|
|
| Create the `EMA-VFI` directory if it is not there, and restart ComfyUI. The checkpoint list is
|
| read when the node builds its schema, so a file added while ComfyUI is running does not appear
|
| until it restarts.
|
|
|
| Keep the filename as it is. The node picks the architecture from the name, so a renamed file
|
| will not be recognised.
|
|
|
| ## Use it in plain torch
|
|
|
| The network itself is not in this repo. Take it from upstream, or from
|
| `modules/vendor/ema_vfi` in WAS Node Suite, then:
|
|
|
| ```python
|
| from safetensors.torch import load_file
|
|
|
| state = load_file("ours_t.safetensors")
|
| net.load_state_dict(state, strict=True) # no prefix to strip, no keys to drop
|
| ```
|
|
|
| ## Licence
|
|
|
| Apache-2.0, as released by the authors: "This project is released under the Apache 2.0 license."
|
| The full text is in `LICENSE` beside these files.
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @inproceedings{zhang2023extracting,
|
| title={Extracting motion and appearance via inter-frame attention for efficient video frame interpolation},
|
| author={Zhang, Guozhen and Zhu, Yuhan and Wang, Haonan and Chen, Youxin and Wu, Gangshan and Wang, Limin},
|
| booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
|
| pages={5682--5692},
|
| year={2023}
|
| }
|
| ```
|
|
|
