anima-motion-module / README.md
paralaif's picture
README: document the three keyframe modes
58231c6 verified
|
Raw History Blame Contribute Delete
11.8 kB
---
license: other
license_name: circlestone-labs-non-commercial-license
license_link: https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md
base_model: circlestone-labs/Anima
tags:
- anima
- comfyui
- motion-module
- animatediff
- video
- anime
---
# Anima Motion Module (experimental, v0.1)
An AnimateDiff-style **motion module for [Anima](https://huggingface.co/circlestone-labs/Anima)** (Cosmos-Predict2 2B Text2Image), plus a ComfyUI node and a "freeze or animate anything" workflow.
This is the **first experiment**, trained on only **20 clips**. It is useful today for one effect: freeze part of an image and animate the rest, or the other way round. It is not a general video model yet. See [Next plans](#next-plans).
<video src="https://huggingface.co/paralaif/anima-motion-module/resolve/main/examples/frozen_character_night_beach.mp4" controls autoplay loop muted></video>
*Frozen character, flowing night beach: Anima + charref keyframe, background redrawn by the motion module.*
## Files
| File | What it is |
|---|---|
| `anima_motion_r2v_v0.1.safetensors` | **Recommended.** Trained with the [Anima Ref Adapter v2](https://huggingface.co/ElectronGirl/anima-ref-adapter) active (reference-to-video): every frame reads the character sheet. 2000 steps. |
| `anima_motion_i2v_v0.1.safetensors` | First run, without a reference (I2V/T2V). 1500 steps. Kept for comparison. |
| `ComfyUI-Anima-Motion/` | ComfyUI node **Anima Motion Module (Experimental)**. Copy into `custom_nodes/`. |
| `workflows/anima_freeze_animate.json` | Freeze-or-animate workflow (keyframe β†’ WeVisOne mask β†’ motion module β†’ video). |
| `scripts/` | Data generation (MiniMax H3), latent encoding, training and evaluation scripts, as used (local paths; edit before running). |
| `examples/` | Frame strips and clips from the tests below. |
## How it works
- Anima's DiT is a video architecture (Cosmos), but the released weights were trained on images only. Running it on several frames gives haze and no motion.
- The module **folds the frames into the batch**, so each frame is a plain Anima image at RoPE `t = 0` (full image quality, no haze). It adds a **zero-initialized temporal attention** after each of the 28 DiT blocks: attention across frames at every spatial position, with 1D RoPE on the frame index.
- 117.5M trainable parameters (inner dim 512, 8 heads); Anima stays frozen.
- Zero init: an untrained module reproduces Anima exactly.
- **Per-frame timesteps.** A frame kept by the noise mask (a keyframe) gets timestep 0, as Cosmos-Predict2 I2V does.
- **R2V.** After **Anima Ref Adapter (v2)** in the graph, the node takes over its reference: every frame attends to the cached character-sheet keys/values, with the adapter LoRA on.
- **Decode each latent as an image** (`LatentCutToBatch` with `t`, slice 1). Decoding as video with the causal Wan VAE smears and blotches.
## Training
| | |
|---|---|
| Data | 20 clips from **MiniMax H3** image-to-video, each from an Anima keyframe of a different character, with a fixed-camera prompt and one gesture (wave, stretch, turn, laugh…) |
| Frames | 16 frames at 8 fps (every 3rd frame of 24 fps), 384Γ—576, each encoded as a separate image |
| Text | Anima tags + gesture: `masterpiece, best quality, score_7, safe, {character tags}, {motion}, anime screencap` |
| Objective | Flow matching as ComfyUI CONST (shift 3, logit-normal Οƒ), MSE on Ξ΅ βˆ’ xβ‚€ |
| Optimizer | AdamW, lr 1e-4, grad clip 1.0, batch 1, bf16 autocast, gradient checkpointing |
| I2V v0.1 | 1500 steps, 50% of steps with a clean keyframe (frame 0 at Οƒ = 0); RTX 5060 Ti, 5.8 s/step |
| R2V v0.1 | 2000 steps from zero init, no keyframe, Ref Adapter v2 with each character's sheet; RunPod RTX 5090, 2.3 s/step (~77 min, about USD 1.50) |
Final loss: about 0.088 (I2V) and 0.089 (R2V), plateaued from step ~1200. With 20 clips that is the memorization limit.
## Results
Metrics are on 4 test cases (2 characters seen in training, 2 unseen), 16 frames each. **Frame-to-frame** is the mean absolute pixel difference between consecutive frames (lower = steadier). **Contrast** is the pixel std; a keyframe is about 55–75 and haze lowers it.
| Setup | Frame-to-frame | First β†’ last frame | Contrast | Notes |
|---|---|---|---|---|
| Anima alone, video latent | 3–6 | 5–6 | 28–59 | Hazy still image, no motion |
| I2V, 500 steps | 41–47 | 34–54 | 49–64 | Sharp, but every frame is a different image |
| I2V, 1500 steps | 16–36 | 28–45 | 41–64 | Temporal coherence appears; ignores the keyframe |
| R2V, 1000 steps | 23–27 | 31–48 | 53–71 | Identity locked to the character sheet |
| **R2V, 2000 steps** | **16–23** | 27–43 | 43–66 | Steadier, varied framing, small motion |
![I2V vs R2V](examples/i2v_vs_r2v.jpg)
*Pairs of rows: I2V v0.1 (first frame is the keyframe), then R2V v0.1, for three characters. I2V drifts in outfit and identity (bottom pair: the pink-haired officer becomes someone else); R2V keeps the character sheet.*
### Masked: freeze the rest, animate a selection (training-free on top of the module)
Module: R2V v0.1. The keyframe is repeated over 16 latent frames and only a **WeVisOne** matte (threshold 0.3, grow 16) is denoised (`SetLatentNoiseMask`); everything else stays pixel-identical.
| Mask, denoise | Frame-to-frame | Result |
|---|---|---|
| character (grow 32), 1.0 | 11–20 | New pose; leftover silhouette fills with blotches |
| character, 0.9 | 11–18 | Larger gestures, breaks the keyframe pose |
| **character, 0.8** | 9–15 | Faithful pose; expression and small hand motion |
| character, 0.6 | 9–14 | Facial expression only |
| character, 0.4 | 8–13 | Blinks and breathing (cinemagraph) |
| **face, 0.8** | 0.6–1.8 | Expression only (laughs, blinks, winks); nothing else moves |
![Character animated, background frozen](examples/animate_character_frozen_background.jpg)
*Character matte, denoise 0.8. The first column is the keyframe.*
![Face only](examples/animate_face_only.jpg)
*Face matte, denoise 0.8.*
### Inverted: freeze the character, let the world flow
The same matte, inverted: the character is frozen and the background is redrawn with an AnimateDiff-like flow (waves, moon, sky). Two things matter:
- **Run the background pass without the charref and without character tags** (`no humans` + scene). With the reference, the redrawn area grows a giant copy of the character.
- **Use a tight, soft mask** (grow 4, blur 8). With grow 16 a ring of the old background stays stuck around the character.
![Night beach](examples/frozen_character_night_beach.jpg)
![Soft edge vs halo](examples/soft_edge_vs_halo.jpg)
*Left: grow 16 (halo). Right: grow 4 + blur 8.*
## Usage (ComfyUI)
**Custom nodes** (in `ComfyUI/custom_nodes/`):
- **Anima Motion Module**: copy the `ComfyUI-Anima-Motion/` folder from this repo.
- **Anima Ref Adapter**: `git clone https://github.com/ArcAnima/ComfyUI-Anima-Ref-Adapter`
- **WeVisOne-VL**, my ComfyUI port of Tencent WeVisOne, which the workflow uses for the selection mask: `GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL` (skips the weights stored in that repo; download one file below).
**Models:**
| File | Folder | From |
|---|---|---|
| `anima_motion_r2v_v0.1.safetensors` | `models/anima_motion/` | this repo |
| `wevisone_vl_4b_int8_convrot.safetensors` (5.7 GB) or `wevisone_vl_4b_bf16.safetensors` (9.7 GB) | `models/text_encoders/` | [paralaif/ComfyUI-WeVisOne-VL](https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL) |
| `anima_ref_adapter_v2_r64.safetensors` | `models/loras/` | [ElectronGirl/anima-ref-adapter](https://huggingface.co/ElectronGirl/anima-ref-adapter) (the node also downloads it on first use) |
| `split_files/diffusion_models/anima-turbo-v1.0.safetensors` | `models/diffusion_models/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) |
| `split_files/text_encoders/qwen_3_06b_base.safetensors` | `models/text_encoders/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) |
| `split_files/vae/qwen_image_vae.safetensors` | `models/vae/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) |
Then load `workflows/anima_freeze_animate.json`. The same download list is in the workflow's note.
**Workflow controls**
- **Keyframe (two switches):**
- **generate with the character sheet**: Anima + charref; the sheet is the video's reference;
- **text only (T2V)**: plain Anima; the generated keyframe becomes the video's charref reference;
- **your own image**: it keeps its aspect ratio (scaled to the Megapixels value, sides in multiples of 16) and is the video's charref reference.
- **query:** what to select (`character`, `face`, `sky`, `sea`, `hair`…).
- **Freeze/animate switch:**
- **true**: freeze the selection and animate the rest, with no charref and the *THE REST* prompt.
- **false**: animate only the selection, with charref and the *THE SELECTION* prompt.
- **denoise** (video KSampler): 0.4 breathes, **0.8 recommended**, 1.0 mutates.
- With your own image, describe its background in the *THE REST* prompt, or the background turns into whatever that prompt says.
![Workflow, both modes](examples/workflow_freeze_vs_animate.jpg)
*Top: switch true (frozen character). Bottom: switch false (frozen background).*
Minimal graph without masks: `Anima Ref Adapter (v2, ref_guidance 1)` β†’ `Anima Motion Module` β†’ `KSampler` on a 16-frame latent β†’ `LatentCutToBatch (t, 1)` β†’ `VAEDecode` β†’ `CreateVideo (8 fps)`.
## Limitations
- **20 training clips.** Motion is small (gestures, expressions). The module memorizes frequent objects: a book from the data shows up in early checkpoints and in some unmasked generations.
- Trained at 384Γ—576 portrait. Other sizes work (the 16:9 beach is 832Γ—480) but are untested for quality.
- It never saw landscapes moving, so in the inverted mode the background **morphs** (AnimateDiff-like) instead of moving physically.
- The R2V module was trained without a keyframe, so without a mask it does not start from your image.
- Each frame is decoded as an independent image: some fine-line shimmer remains.
- 16 frames (2 s at 8 fps). Longer clips are untested.
## Next plans
1. **More data: about 300 clips.** About 200 characters with varied gestures and **about 100 fixed-camera landscapes** with ambient motion (waves, clouds, lanterns, falling petals), all from Anima keyframes animated with MiniMax H3.
2. **Keyframe conditioning in R2V training.** A clean frame 0 in about 70–90% of the steps, so the module learns to continue an existing image instead of redrawing it.
3. **Train the way it is used.** Start some samples from a noised keyframe (partial denoise) and with spatial masks, to match the masked workflow.
4. **Bigger run on a cloud GPU** (L40S, batch 4, 3000–5000 steps). Mixed batches: character clips with the reference, landscape clips without it.
5. **Longer and wider clips**: 32 frames (4 s) and 16:9 / 9:16 buckets.
6. Re-check frame-to-frame stability, keyframe adherence and object memorization against this v0.1.
## License and credits
- **License:** the [CircleStone Labs Non-Commercial License](https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md), as for Anima, since the module is trained on and runs with Anima. Training clips were generated with MiniMax H3; check its terms for your use.
- [Anima](https://huggingface.co/circlestone-labs/Anima) by CircleStone Labs.
- [Anima Ref Adapter](https://huggingface.co/ElectronGirl/anima-ref-adapter) by ElectronGirl.
- WeVisOne-VL by Tencent (ComfyUI port: [paralaif/ComfyUI-WeVisOne-VL](https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL)).
- Method inspired by [AnimateDiff](https://github.com/guoyww/AnimateDiff).