|
Download README.md from paralaif/anima-motion-module: direct link, hf CLI and curl.
- Browser
- Download file 11.8 kB
-
https://huggingface.co/paralaif/anima-motion-module/resolve/main/README.md
- Command line
-
hf download hf://paralaif/anima-motion-module/README.md
-
curl -L -o README.md https://huggingface.co/paralaif/anima-motion-module/resolve/main/README.md
11.8 kB
| license: other | |
| license_name: circlestone-labs-non-commercial-license | |
| license_link: https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md | |
| base_model: circlestone-labs/Anima | |
| tags: | |
| - anima | |
| - comfyui | |
| - motion-module | |
| - animatediff | |
| - video | |
| - anime | |
| # Anima Motion Module (experimental, v0.1) | |
| An AnimateDiff-style **motion module for [Anima](https://huggingface.co/circlestone-labs/Anima)** (Cosmos-Predict2 2B Text2Image), plus a ComfyUI node and a "freeze or animate anything" workflow. | |
| This is the **first experiment**, trained on only **20 clips**. It is useful today for one effect: freeze part of an image and animate the rest, or the other way round. It is not a general video model yet. See [Next plans](#next-plans). | |
| <video src="https://huggingface.co/paralaif/anima-motion-module/resolve/main/examples/frozen_character_night_beach.mp4" controls autoplay loop muted></video> | |
| *Frozen character, flowing night beach: Anima + charref keyframe, background redrawn by the motion module.* | |
| ## Files | |
| | File | What it is | | |
| |---|---| | |
| | `anima_motion_r2v_v0.1.safetensors` | **Recommended.** Trained with the [Anima Ref Adapter v2](https://huggingface.co/ElectronGirl/anima-ref-adapter) active (reference-to-video): every frame reads the character sheet. 2000 steps. | | |
| | `anima_motion_i2v_v0.1.safetensors` | First run, without a reference (I2V/T2V). 1500 steps. Kept for comparison. | | |
| | `ComfyUI-Anima-Motion/` | ComfyUI node **Anima Motion Module (Experimental)**. Copy into `custom_nodes/`. | | |
| | `workflows/anima_freeze_animate.json` | Freeze-or-animate workflow (keyframe β WeVisOne mask β motion module β video). | | |
| | `scripts/` | Data generation (MiniMax H3), latent encoding, training and evaluation scripts, as used (local paths; edit before running). | | |
| | `examples/` | Frame strips and clips from the tests below. | | |
| ## How it works | |
| - Anima's DiT is a video architecture (Cosmos), but the released weights were trained on images only. Running it on several frames gives haze and no motion. | |
| - The module **folds the frames into the batch**, so each frame is a plain Anima image at RoPE `t = 0` (full image quality, no haze). It adds a **zero-initialized temporal attention** after each of the 28 DiT blocks: attention across frames at every spatial position, with 1D RoPE on the frame index. | |
| - 117.5M trainable parameters (inner dim 512, 8 heads); Anima stays frozen. | |
| - Zero init: an untrained module reproduces Anima exactly. | |
| - **Per-frame timesteps.** A frame kept by the noise mask (a keyframe) gets timestep 0, as Cosmos-Predict2 I2V does. | |
| - **R2V.** After **Anima Ref Adapter (v2)** in the graph, the node takes over its reference: every frame attends to the cached character-sheet keys/values, with the adapter LoRA on. | |
| - **Decode each latent as an image** (`LatentCutToBatch` with `t`, slice 1). Decoding as video with the causal Wan VAE smears and blotches. | |
| ## Training | |
| | | | | |
| |---|---| | |
| | Data | 20 clips from **MiniMax H3** image-to-video, each from an Anima keyframe of a different character, with a fixed-camera prompt and one gesture (wave, stretch, turn, laughβ¦) | | |
| | Frames | 16 frames at 8 fps (every 3rd frame of 24 fps), 384Γ576, each encoded as a separate image | | |
| | Text | Anima tags + gesture: `masterpiece, best quality, score_7, safe, {character tags}, {motion}, anime screencap` | | |
| | Objective | Flow matching as ComfyUI CONST (shift 3, logit-normal Ο), MSE on Ξ΅ β xβ | | |
| | Optimizer | AdamW, lr 1e-4, grad clip 1.0, batch 1, bf16 autocast, gradient checkpointing | | |
| | I2V v0.1 | 1500 steps, 50% of steps with a clean keyframe (frame 0 at Ο = 0); RTX 5060 Ti, 5.8 s/step | | |
| | R2V v0.1 | 2000 steps from zero init, no keyframe, Ref Adapter v2 with each character's sheet; RunPod RTX 5090, 2.3 s/step (~77 min, about USD 1.50) | | |
| Final loss: about 0.088 (I2V) and 0.089 (R2V), plateaued from step ~1200. With 20 clips that is the memorization limit. | |
| ## Results | |
| Metrics are on 4 test cases (2 characters seen in training, 2 unseen), 16 frames each. **Frame-to-frame** is the mean absolute pixel difference between consecutive frames (lower = steadier). **Contrast** is the pixel std; a keyframe is about 55β75 and haze lowers it. | |
| | Setup | Frame-to-frame | First β last frame | Contrast | Notes | | |
| |---|---|---|---|---| | |
| | Anima alone, video latent | 3β6 | 5β6 | 28β59 | Hazy still image, no motion | | |
| | I2V, 500 steps | 41β47 | 34β54 | 49β64 | Sharp, but every frame is a different image | | |
| | I2V, 1500 steps | 16β36 | 28β45 | 41β64 | Temporal coherence appears; ignores the keyframe | | |
| | R2V, 1000 steps | 23β27 | 31β48 | 53β71 | Identity locked to the character sheet | | |
| | **R2V, 2000 steps** | **16β23** | 27β43 | 43β66 | Steadier, varied framing, small motion | | |
|  | |
| *Pairs of rows: I2V v0.1 (first frame is the keyframe), then R2V v0.1, for three characters. I2V drifts in outfit and identity (bottom pair: the pink-haired officer becomes someone else); R2V keeps the character sheet.* | |
| ### Masked: freeze the rest, animate a selection (training-free on top of the module) | |
| Module: R2V v0.1. The keyframe is repeated over 16 latent frames and only a **WeVisOne** matte (threshold 0.3, grow 16) is denoised (`SetLatentNoiseMask`); everything else stays pixel-identical. | |
| | Mask, denoise | Frame-to-frame | Result | | |
| |---|---|---| | |
| | character (grow 32), 1.0 | 11β20 | New pose; leftover silhouette fills with blotches | | |
| | character, 0.9 | 11β18 | Larger gestures, breaks the keyframe pose | | |
| | **character, 0.8** | 9β15 | Faithful pose; expression and small hand motion | | |
| | character, 0.6 | 9β14 | Facial expression only | | |
| | character, 0.4 | 8β13 | Blinks and breathing (cinemagraph) | | |
| | **face, 0.8** | 0.6β1.8 | Expression only (laughs, blinks, winks); nothing else moves | | |
|  | |
| *Character matte, denoise 0.8. The first column is the keyframe.* | |
|  | |
| *Face matte, denoise 0.8.* | |
| ### Inverted: freeze the character, let the world flow | |
| The same matte, inverted: the character is frozen and the background is redrawn with an AnimateDiff-like flow (waves, moon, sky). Two things matter: | |
| - **Run the background pass without the charref and without character tags** (`no humans` + scene). With the reference, the redrawn area grows a giant copy of the character. | |
| - **Use a tight, soft mask** (grow 4, blur 8). With grow 16 a ring of the old background stays stuck around the character. | |
|  | |
|  | |
| *Left: grow 16 (halo). Right: grow 4 + blur 8.* | |
| ## Usage (ComfyUI) | |
| **Custom nodes** (in `ComfyUI/custom_nodes/`): | |
| - **Anima Motion Module**: copy the `ComfyUI-Anima-Motion/` folder from this repo. | |
| - **Anima Ref Adapter**: `git clone https://github.com/ArcAnima/ComfyUI-Anima-Ref-Adapter` | |
| - **WeVisOne-VL**, my ComfyUI port of Tencent WeVisOne, which the workflow uses for the selection mask: `GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL` (skips the weights stored in that repo; download one file below). | |
| **Models:** | |
| | File | Folder | From | | |
| |---|---|---| | |
| | `anima_motion_r2v_v0.1.safetensors` | `models/anima_motion/` | this repo | | |
| | `wevisone_vl_4b_int8_convrot.safetensors` (5.7 GB) or `wevisone_vl_4b_bf16.safetensors` (9.7 GB) | `models/text_encoders/` | [paralaif/ComfyUI-WeVisOne-VL](https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL) | | |
| | `anima_ref_adapter_v2_r64.safetensors` | `models/loras/` | [ElectronGirl/anima-ref-adapter](https://huggingface.co/ElectronGirl/anima-ref-adapter) (the node also downloads it on first use) | | |
| | `split_files/diffusion_models/anima-turbo-v1.0.safetensors` | `models/diffusion_models/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) | | |
| | `split_files/text_encoders/qwen_3_06b_base.safetensors` | `models/text_encoders/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) | | |
| | `split_files/vae/qwen_image_vae.safetensors` | `models/vae/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) | | |
| Then load `workflows/anima_freeze_animate.json`. The same download list is in the workflow's note. | |
| **Workflow controls** | |
| - **Keyframe (two switches):** | |
| - **generate with the character sheet**: Anima + charref; the sheet is the video's reference; | |
| - **text only (T2V)**: plain Anima; the generated keyframe becomes the video's charref reference; | |
| - **your own image**: it keeps its aspect ratio (scaled to the Megapixels value, sides in multiples of 16) and is the video's charref reference. | |
| - **query:** what to select (`character`, `face`, `sky`, `sea`, `hair`β¦). | |
| - **Freeze/animate switch:** | |
| - **true**: freeze the selection and animate the rest, with no charref and the *THE REST* prompt. | |
| - **false**: animate only the selection, with charref and the *THE SELECTION* prompt. | |
| - **denoise** (video KSampler): 0.4 breathes, **0.8 recommended**, 1.0 mutates. | |
| - With your own image, describe its background in the *THE REST* prompt, or the background turns into whatever that prompt says. | |
|  | |
| *Top: switch true (frozen character). Bottom: switch false (frozen background).* | |
| Minimal graph without masks: `Anima Ref Adapter (v2, ref_guidance 1)` β `Anima Motion Module` β `KSampler` on a 16-frame latent β `LatentCutToBatch (t, 1)` β `VAEDecode` β `CreateVideo (8 fps)`. | |
| ## Limitations | |
| - **20 training clips.** Motion is small (gestures, expressions). The module memorizes frequent objects: a book from the data shows up in early checkpoints and in some unmasked generations. | |
| - Trained at 384Γ576 portrait. Other sizes work (the 16:9 beach is 832Γ480) but are untested for quality. | |
| - It never saw landscapes moving, so in the inverted mode the background **morphs** (AnimateDiff-like) instead of moving physically. | |
| - The R2V module was trained without a keyframe, so without a mask it does not start from your image. | |
| - Each frame is decoded as an independent image: some fine-line shimmer remains. | |
| - 16 frames (2 s at 8 fps). Longer clips are untested. | |
| ## Next plans | |
| 1. **More data: about 300 clips.** About 200 characters with varied gestures and **about 100 fixed-camera landscapes** with ambient motion (waves, clouds, lanterns, falling petals), all from Anima keyframes animated with MiniMax H3. | |
| 2. **Keyframe conditioning in R2V training.** A clean frame 0 in about 70β90% of the steps, so the module learns to continue an existing image instead of redrawing it. | |
| 3. **Train the way it is used.** Start some samples from a noised keyframe (partial denoise) and with spatial masks, to match the masked workflow. | |
| 4. **Bigger run on a cloud GPU** (L40S, batch 4, 3000β5000 steps). Mixed batches: character clips with the reference, landscape clips without it. | |
| 5. **Longer and wider clips**: 32 frames (4 s) and 16:9 / 9:16 buckets. | |
| 6. Re-check frame-to-frame stability, keyframe adherence and object memorization against this v0.1. | |
| ## License and credits | |
| - **License:** the [CircleStone Labs Non-Commercial License](https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md), as for Anima, since the module is trained on and runs with Anima. Training clips were generated with MiniMax H3; check its terms for your use. | |
| - [Anima](https://huggingface.co/circlestone-labs/Anima) by CircleStone Labs. | |
| - [Anima Ref Adapter](https://huggingface.co/ElectronGirl/anima-ref-adapter) by ElectronGirl. | |
| - WeVisOne-VL by Tencent (ComfyUI port: [paralaif/ComfyUI-WeVisOne-VL](https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL)). | |
| - Method inspired by [AnimateDiff](https://github.com/guoyww/AnimateDiff). | |