| --- |
| license: other |
| license_name: minimax-h3-community-license |
| license_link: LICENSE |
| base_model: MiniMaxAI/MiniMax-H3 |
| pipeline_tag: video-to-video |
| library_name: minimax-h3 |
| tags: |
| - video-editing |
| - character-replacement |
| - video-to-video |
| - distillation |
| - dmd |
| --- |
| |
| # Viggle-Animate |
|
|
| ### Character Replacement in Video from a Single Repainted Frame |
|
|
| **[Try the demo](https://huggingface.co/spaces/Viggle/viggle-animate)** · |
| **[Research write-up](https://viggle.ai/research/viggle-animate-character-replacement-from-a-repainted-frame?utm_source=huggingface&utm_medium=social&utm_campaign=viggle-animate&utm_content=viggle/viggle-animate)** · |
| **[ComfyUI nodes](https://github.com/Saganaki22/ComfyUI-Viggle-Animate-H3)** · |
| **[viggle.ai/h3](https://viggle.ai/h3)** · |
| Built on **[MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3)** |
|
|
| <a href="https://x.com/cocktailpeanut/status/2097332291844399514" style="display:block;max-width:460px;margin:0 0 .5em"> |
| <img src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/community-tweet.png" width="100%" style="display:block;border-radius:14px" alt="@cocktailpeanut on X: two AI CEOs dropped into a film scene with Viggle-Animate - 1.5M views, 14K likes"> |
| </a> |
|
|
| <sub>Made with Viggle-Animate by <a href="https://x.com/cocktailpeanut/status/2097332291844399514">@cocktailpeanut</a>, not by us. Click through to watch it on X.</sub> |
|
|
| <video autoplay controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/white-house-funk.mp4"></video> |
|
|
| <sub><b>White House Funk: a preview of Viggle-Animate v2, coming soon.</b> Not made with the weights in |
| this repository. The cast of a two-minute music video replaced through its fast cuts, from at most |
| one repainted frame per shot. AI-generated parody: the people shown did not take part in, approve or |
| endorse it.</sub> |
|
|
| **Viggle-Animate replaces the character in a video with whatever you paint into one of its own |
| frames** — motion, camera and timing untouched. You prepare that one frame in an image editor; |
| from there the video stage runs no pose estimator, segmenter, face tracker or text encoder. Two |
| inputs, three forward passes, 26 seconds a shot on one GPU. |
|
|
| **It is strongest where replacement is hardest: fast motion, and pose transfer accurate enough to |
| follow it.** Whipping heads, full kicks, jumps — tracked frame for frame, not smeared through. |
|
|
| ## Abstract |
|
|
| Controlled character replacement is usually built on intermediate representations — pose skeletons, |
| segmentation masks, background plates, face crops. Each needs its own extractor, and each extractor |
| is another model to run and another place to lose information. Recent work drops the skeleton but |
| keeps a mask channel. **Viggle-Animate uses neither.** Its two inputs are a driving video and one of |
| that video's own frames with the character repainted, and its only task is to propagate that edit |
| across the shot. |
|
|
| Because the reference is a frame of the clip, its pose, camera, framing and lighting already agree |
| with the footage, and nothing downstream has to align them again. The model is never told what the |
| new character is: no class, no identity encoder, and no user-provided text prompt. |
| **Viggle-Animate is a 33.1 B full finetune of MiniMax-H3's `ref2va` transformer, jointly distilled |
| with DMD to three forward passes.** In a matched comparison on the same machine and B200 GPU, using |
| the same source videos, output resolution and frame count, it renders 124 frames in 26 s, 6.1× |
| faster per clip than Wan2.2-Animate-14B. |
|
|
| ## Method |
|
|
| Character replacement asks two questions at once: *what does the new character look like*, and *how |
| does it move through this shot*. Systems that condition on a standalone character photograph must |
| answer both, and reconciling a photograph with footage it was never part of is what the scaffolding |
| exists for. |
|
|
| State-of-the-art image models have finished that job. Give `gpt-image` a frame and an instruction |
| and it replaces the character while following the prompt exactly — transferring the pose, matching |
| the lighting, preserving the background. The hard reconciliation is already solved, once, on one |
| image. This model is the second half of that pipeline, not the whole of it. |
|
|
| <img src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/pipeline.svg" |
| alt="Two inputs — a driving video and one of its own frames, repainted in any image editor — enter Viggle-Animate. No pose skeleton, segmentation mask, face crop, background plate, depth map or user-provided text prompt enters the video model." width="100%"> |
|
|
| Appearance enters only through the repainted frame; geometry enters only through the driving video. |
| **The text encoder is never loaded.** Conditioning is one frozen embedding shipped with the weights |
| ([`assets/fixed_prompt.txt`](assets/fixed_prompt.txt)), identical for every render. |
|
|
| Left panel is the driving video, right panel is this model: |
|
|
| <video autoplay controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/hero-duo.mp4"></video> |
|
|
| <video autoplay controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/hero-corgi.mp4"></video> |
|
|
| **It is fast twice over.** Once the repainted frame exists there is nothing else to run — no pose |
| estimator, no segmenter, no face tracker, no text encoder. And the sampler is distilled, so a |
| finished clip is three forward passes rather than thirty. The two compound: one model, one GPU, |
| and no orchestration to get wrong. |
|
|
| The distillation is **joint, across two teachers split by noise level.** Our finetune supervises the |
| high-noise end of the schedule, where the replacement itself is decided — it is the model that gets |
| the swap right. The original MiniMax-H3 supervises the low-noise end, where detail and texture are |
| decided — it is the model with the better image quality. Distilling each end against the teacher that |
| owns it keeps both properties in one student, instead of inheriting the finetune's visual |
| regressions along with its replacement ability. |
|
|
| **It generalizes past humans**, because nothing in the loop assumes one. A pose skeleton has a neck |
| and two arms; a mask has a person-shaped hole. We have neither, so the model holds no representation |
| that a character must be a person. What it can animate is bounded by what you can paint. |
|
|
| ## Efficiency |
|
|
| <div align="center"> |
| <div style="display:flex;gap:10px;flex-wrap:wrap;justify-content:center;margin:20px 0 6px;text-align:left"> |
|
|
| <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> |
| <div style="font-size:1.7em;font-weight:700;line-height:1.15">26 s</div> |
| <b>per render</b><br> |
| <small style="opacity:.7">124 frames at 24 fps, 480×832, a single B200</small> |
| </div> |
|
|
| <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> |
| <div style="font-size:1.7em;font-weight:700;line-height:1.15">3</div> |
| <b>forward passes</b><br> |
| <small style="opacity:.7"><code>--steps 4</code> names four sigma boundaries, so three passes</small> |
| </div> |
|
|
| <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> |
| <div style="font-size:1.7em;font-weight:700;line-height:1.15">2</div> |
| <b>inputs</b><br> |
| <small style="opacity:.7">a clip, and one of its own frames repainted</small> |
| </div> |
|
|
| <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> |
| <div style="font-size:1.7em;font-weight:700;line-height:1.15">0</div> |
| <b>other models</b><br> |
| <small style="opacity:.7">in the video stage — no pose estimator, segmenter, face tracker or text encoder</small> |
| </div> |
|
|
| </div> |
| </div> |
|
|
| One B200, 480×832, 124 frames at 24 fps, bf16, no compile, no offload. Wan ran its documented |
| replacement recipe — 20 steps, `sample_shift 5.0`, `--refert_num 1 --replace_flag |
| --use_relighting_lora`, `--w_len 1 --h_len 1` — after its own preprocessing pass. |
|
|
| | | Viggle-Animate | Wan2.2-Animate-14B | |
| |---|---|---| |
| | Inputs | driving video + one repainted frame | driving video + character image, then a **preprocessing pass** producing pose, face, mask and background tracks | |
| | Render, after weights load | **26 s** | 160 s | |
| | — of which sampling | **13.6 s** | 140 s | |
| | Forward passes | **3** | 40 (20 steps × 2 chunks) | |
| | Parameters | 33.1 B | 17.3 B | |
|
|
| **6.1× faster per render, 10.3× on sampling alone.** Wan's preprocessing pass is not counted in |
| its 160 s. |
|
|
| ### Qualitative comparison |
|
|
| Four panels each: **painted reference · driving video · this model · Wan2.2-Animate-14B**, the last |
| at its documented replacement settings. The gap is widest under fast motion: where the comparison |
| smears, this model stays sharp and lands the pose on the right frame. |
|
|
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-fastmotion.mp4"></video> |
|
|
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-highkick.mp4"></video> |
|
|
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/compare-cosplay.mp4"></video> |
|
|
| ## Generalization |
|
|
| The model is never told what it is animating, so how far the character can get from a person is an |
| empirical question rather than a list of supported categories. Three panels each: **painted |
| reference · driving video · this model.** Every clip below is one paint and one render at the |
| shipped defaults, `--seed 42` — no best-of-N. |
|
|
| **Animals.** Ears, eye patches and flippers move on limbs the driving clip does not have — the paint |
| places them, the render animates them as if they had always been arms and a head. |
|
|
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-corgi.mp4"></video> |
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-panda.mp4"></video> |
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-penguin.mp4"></video> |
|
|
| **Not humanoid.** The airliner is the hardest case we have: the paint binds wings to arms and landing |
| gear to legs, and the model's job is to keep that binding for 124 frames. The robot has to relight |
| specular metal as it turns. |
|
|
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-airliner.mp4"></video> |
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-robot.mp4"></video> |
|
|
| **Stylized.** The clay figure holds its style boundary for the whole clip. |
|
|
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-claymation.mp4"></video> |
|
|
| **More than one character.** The work moves into the paint prompt, which has to bind each one to a |
| position — "the one on the left". The last clip is a wide arena shot, each figure a few dozen pixels |
| tall. |
|
|
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-anime-duo.mp4"></video> |
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-photoreal-duo.mp4"></video> |
| <video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-wushu-animals.mp4"></video> |
|
|
| ## Limitations |
|
|
| **It inherits the image edit.** What the paint does not show, the model will not add, and where |
| paint and video disagree the video wins. Appearance comes from the paint but shape comes from the |
| driving pose: a LEGO minifigure kept its palette and yellow claw hands, yet reverted to human |
| anatomy — the airliner held because the paint tied its wings to real arms. |
|
|
| ~~**Lip-sync is weak.** Mouth shapes do not track speech closely in close-ups. Identity and expression |
| hold; it is the sync that lags, and we believe that is a training-data limit rather than anything |
| structural.~~ |
|
|
| **Retracted — and since improved.** Broader user testing already put lip-sync and facial |
| expression ahead of our own first read of them. The rest turned out not to be a training-data limit |
| either. The model animates the mouth to *its own* audio track, and the fixed prompt asks that track |
| to be silence — so in the shipped configuration there is nothing for the mouth to sync to. Give it a |
| real soundtrack instead: encode the driving clip's audio with the audio VAE and hold it in the |
| target audio rows as a **clean** latent for the whole denoise (H3's flow convention is reversed, so |
| clean is `t = 1`). The audio rows stop being something the model predicts and become something it |
| conditions on, and the mouth tracks that speech. This is an inference-time change only — no |
| retraining, no new weights — and the pairing of noisy video rows with clean audio rows is not a |
| state the model was trained on, but it holds up across the clips we have run. |
| Pass `--audio` to do it here; the [hosted demo](https://huggingface.co/spaces/Viggle/viggle-animate) |
| does it automatically whenever the clip you upload carries sound. |
|
|
| **Turn the sound on for these two.** The driving clip, then the same shot with a different actor in |
| it, speaking her lines: |
|
|
| <video controls playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/audio-pin-driving.mp4"></video> |
|
|
| <video controls playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/audio-pin-out.mp4"></video> |
|
|
| All three files ship with the weights — the driving clip, the repainted still it is animated from |
| ([`audio-pin-ref.png`](examples/media/audio-pin-ref.png)), and the take above — so the second clip |
| is a command rather than a claim. The same file is passed twice, once for the motion and once for |
| the soundtrack: |
|
|
| ```bash |
| python inference/sample.py --model-dir ./MiniMax-H3 \ |
| --cond examples/media/audio-pin-driving.mp4 \ |
| --ref examples/media/audio-pin-ref.png \ |
| --audio examples/media/audio-pin-driving.mp4 \ |
| --out audio-pin-out.mp4 |
| ``` |
|
|
| Everything else is the shipped default: 124 frames at 832x480, four steps, `--seed 42`; 51 s of |
| render on a B200 after the weights load, and it comes back at 46 dB against the file above. The |
| track you get back is the driving clip's own audio put through the audio VAE and decoded again — |
| same level, peaks rounded off (0.95 to 0.76 here); that round trip is the evidence the rows really |
| were pinned. Drop `--audio` and the same command writes a file whose audio track is digital silence, |
| sample for sample, and a mouth that moves without saying anything. |
|
|
| **Complex scenes are harder than single subjects.** Several characters at once, close interaction |
| between them, and shots that cut are all cases where quality drops off — enough that we would not |
| call them solved. |
|
|
| **We are training a substantially better model right now**, aimed squarely at what is left. This |
| release is the version we can ship today, not the ceiling. |
|
|
| ## What this repository contains |
|
|
| Two parts, both derived from [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)'s |
| `ref2va` transformer: |
|
|
| | | | |
| |---|---| |
| | `transformer/` | 33.1 B, bf16, 14 shards. A **full finetune** of the base `transformer_ref` on a character-replacement objective | |
| | `lora/` | rank 128 over 302 linear layers, 2.5 GB. A **DMD2-distilled** delta on that finetune — this is what collapses the sampler to three forward passes | |
|
|
| The LoRA is a delta on the *finetuned* transformer — loading it onto stock `transformer_ref` produces |
| garbage. |
|
|
| ## Quickstart |
|
|
| This repository ships only the transformer and the LoRA — the VAE, audio VAE and schedulers load |
| from your own copy of the base model. Inference touches 11 GB of its 269 GB: |
|
|
| ```bash |
| hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \ |
| --include "modular_model_index.json" "vae/*" "audio_vae/*" \ |
| "scheduler/*" "audio_scheduler/*" "assets/ref2va.mp4" |
| |
| hf download Viggle/Viggle-Animate --local-dir ./Viggle-Animate |
| |
| pip install torch "git+https://github.com/huggingface/diffusers@d6726f3" av |
| |
| python Viggle-Animate/inference/sample.py \ |
| --model-dir ./MiniMax-H3 \ |
| --cond driving.mp4 --ref ref.png --out swapped.mp4 |
| ``` |
|
|
| `d6726f3` is the tested `diffusers` commit; the upstream `minimax_h3` modular pipeline is enough, |
| no fork or patch. The last `--include` is the clip [`examples/demo.sh`](examples/demo.sh) needs. |
|
|
| **One 80 GB card is not enough at bf16** — the transformer is 62 GiB resident and a 480×832 / |
| 124-frame render peaks at 80.1 GiB allocated. Use a card with ≥ 96 GB, or pass `--offload` to |
| stream blocks from CPU (~12 GB resident, much slower). |
|
|
| **It also runs quantized on consumer hardware.** We deploy it on a single **RTX 5090 (32 GB)**: |
| NVFP4 weights — 4.5 bits/param, dispatching to the real sm_120 cutlass block-scaled kernel, 2.70× |
| bf16 per compiled linear — plus a low-rank `adaln_proj` and `torch.compile`. Quantization alone is |
| not enough for 32 GB: 13.0 B of the 33.1 B parameters sit in `adaln_proj`, which the linear-layer |
| quantizer does not touch. That deployment path is not shipped in this repository. |
|
|
| Defaults are the evaluated configuration: `--steps 4 --flow-shift 3 --num-frames 124` (≈ 5.2 s at |
| 24 fps) `--seed 42`. Output geometry follows the driving clip and must be a multiple of 32 on both |
| axes. Weights load in ~21 s, once per process. **Four steps is the operating point, not a shortcut** |
| — the distilled model already renders sharper than its teacher, and raising the step count tips that |
| into over-sharpening. `--steps 4` is four sigma boundaries and therefore three forward passes. |
| `sample.py` drops the driving clip's audio at input and the model emits its own track, which the |
| fixed prompt asks to be silence. `--audio <file>` overrides that and pins a real soundtrack instead, |
| which is what makes the mouth track speech — see [Limitations](#limitations). |
|
|
| Pull a frame with `ffmpeg -ss 1.5 -i driving.mp4 -frames:v 1 ref.png` and edit it at the same |
| resolution. **It does not have to be the first frame.** The still is passed to the model with no |
| frame index, so nothing downstream knows where in the clip it came from — pick whichever frame shows |
| the character most clearly, front-on and unoccluded. Name the change, and pin down what must *not* |
| change: pose, hands, props, framing, background, light. An editor that quietly reframes the shot will |
| fight the driving motion. Prefer clips that keep one side to camera, and bind any new limb to a real |
| one. |
|
|
| [`examples/demo.sh`](examples/demo.sh) runs a swap end to end on a clip that ships with the base |
| model, so it needs no media from you — and it is the fastest way to check that the LoRA loaded. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{viggle2026animate, |
| title = {Viggle-Animate: Character Replacement in Video from a Single Repainted Frame}, |
| author = {Viggle Research}, |
| year = {2026}, |
| url = {https://huggingface.co/Viggle/Viggle-Animate} |
| } |
| ``` |
|
|
| ## License |
|
|
| The weights are a Model Derivative of MiniMax H3, so the |
| [MiniMax H3 Community License](LICENSE) applies to them — read it before you redistribute them or |
| ship a product on them. Our changes are listed in [`MODIFICATIONS.md`](MODIFICATIONS.md). |
|
|
| The code in [`inference/`](inference) and [`examples/`](examples) is Apache 2.0 |
| ([`LICENSE-CODE`](LICENSE-CODE)). |
|
|
| Music in the teaser at the top of this page: "Electrodoodle" by Kevin MacLeod |
| ([incompetech.com](https://incompetech.com)), licensed under |
| [Creative Commons: By Attribution 4.0](http://creativecommons.org/licenses/by/4.0/). |
|
|
| ## Intended use |
|
|
| This model exists to put a consenting performer into footage they did not shoot, and it will just |
| as readily put someone into footage they never agreed to appear in. Note where that decision is |
| made: **the identity comes from the frame you paint**, so an image editor's safeguards are upstream |
| of this model and none of them are in it. It cannot verify identity or consent. Do not run it on |
| people who have not agreed to it, label what you generate as AI-generated, and see Section V.5 of |
| the Agreement if you offer this as a service. |
|
|
| Powered by MiniMax H3. |
|
|
