--- license: other license_name: circlestone-labs-non-commercial-license license_link: https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md base_model: circlestone-labs/Anima tags: - anima - comfyui - motion-module - animatediff - video - anime --- # Anima Motion Module (experimental, v0.1) An AnimateDiff-style **motion module for [Anima](https://huggingface.co/circlestone-labs/Anima)** (Cosmos-Predict2 2B Text2Image), plus a ComfyUI node and a "freeze or animate anything" workflow. This is the **first experiment**, trained on only **20 clips**. It is useful today for one effect: freeze part of an image and animate the rest, or the other way round. It is not a general video model yet. See [Next plans](#next-plans). *Frozen character, flowing night beach: Anima + charref keyframe, background redrawn by the motion module.* ## Files | File | What it is | |---|---| | `anima_motion_r2v_v0.1.safetensors` | **Recommended.** Trained with the [Anima Ref Adapter v2](https://huggingface.co/ElectronGirl/anima-ref-adapter) active (reference-to-video): every frame reads the character sheet. 2000 steps. | | `anima_motion_i2v_v0.1.safetensors` | First run, without a reference (I2V/T2V). 1500 steps. Kept for comparison. | | `ComfyUI-Anima-Motion/` | ComfyUI node **Anima Motion Module (Experimental)**. Copy into `custom_nodes/`. | | `workflows/anima_freeze_animate.json` | Freeze-or-animate workflow (keyframe → WeVisOne mask → motion module → video). | | `scripts/` | Data generation (MiniMax H3), latent encoding, training and evaluation scripts, as used (local paths; edit before running). | | `examples/` | Frame strips and clips from the tests below. | ## How it works - Anima's DiT is a video architecture (Cosmos), but the released weights were trained on images only. Running it on several frames gives haze and no motion. - The module **folds the frames into the batch**, so each frame is a plain Anima image at RoPE `t = 0` (full image quality, no haze). It adds a **zero-initialized temporal attention** after each of the 28 DiT blocks: attention across frames at every spatial position, with 1D RoPE on the frame index. - 117.5M trainable parameters (inner dim 512, 8 heads); Anima stays frozen. - Zero init: an untrained module reproduces Anima exactly. - **Per-frame timesteps.** A frame kept by the noise mask (a keyframe) gets timestep 0, as Cosmos-Predict2 I2V does. - **R2V.** After **Anima Ref Adapter (v2)** in the graph, the node takes over its reference: every frame attends to the cached character-sheet keys/values, with the adapter LoRA on. - **Decode each latent as an image** (`LatentCutToBatch` with `t`, slice 1). Decoding as video with the causal Wan VAE smears and blotches. ## Training | | | |---|---| | Data | 20 clips from **MiniMax H3** image-to-video, each from an Anima keyframe of a different character, with a fixed-camera prompt and one gesture (wave, stretch, turn, laugh…) | | Frames | 16 frames at 8 fps (every 3rd frame of 24 fps), 384×576, each encoded as a separate image | | Text | Anima tags + gesture: `masterpiece, best quality, score_7, safe, {character tags}, {motion}, anime screencap` | | Objective | Flow matching as ComfyUI CONST (shift 3, logit-normal σ), MSE on ε − x₀ | | Optimizer | AdamW, lr 1e-4, grad clip 1.0, batch 1, bf16 autocast, gradient checkpointing | | I2V v0.1 | 1500 steps, 50% of steps with a clean keyframe (frame 0 at σ = 0); RTX 5060 Ti, 5.8 s/step | | R2V v0.1 | 2000 steps from zero init, no keyframe, Ref Adapter v2 with each character's sheet; RunPod RTX 5090, 2.3 s/step (~77 min, about USD 1.50) | Final loss: about 0.088 (I2V) and 0.089 (R2V), plateaued from step ~1200. With 20 clips that is the memorization limit. ## Results Metrics are on 4 test cases (2 characters seen in training, 2 unseen), 16 frames each. **Frame-to-frame** is the mean absolute pixel difference between consecutive frames (lower = steadier). **Contrast** is the pixel std; a keyframe is about 55–75 and haze lowers it. | Setup | Frame-to-frame | First → last frame | Contrast | Notes | |---|---|---|---|---| | Anima alone, video latent | 3–6 | 5–6 | 28–59 | Hazy still image, no motion | | I2V, 500 steps | 41–47 | 34–54 | 49–64 | Sharp, but every frame is a different image | | I2V, 1500 steps | 16–36 | 28–45 | 41–64 | Temporal coherence appears; ignores the keyframe | | R2V, 1000 steps | 23–27 | 31–48 | 53–71 | Identity locked to the character sheet | | **R2V, 2000 steps** | **16–23** | 27–43 | 43–66 | Steadier, varied framing, small motion | ![I2V vs R2V](examples/i2v_vs_r2v.jpg) *Pairs of rows: I2V v0.1 (first frame is the keyframe), then R2V v0.1, for three characters. I2V drifts in outfit and identity (bottom pair: the pink-haired officer becomes someone else); R2V keeps the character sheet.* ### Masked: freeze the rest, animate a selection (training-free on top of the module) Module: R2V v0.1. The keyframe is repeated over 16 latent frames and only a **WeVisOne** matte (threshold 0.3, grow 16) is denoised (`SetLatentNoiseMask`); everything else stays pixel-identical. | Mask, denoise | Frame-to-frame | Result | |---|---|---| | character (grow 32), 1.0 | 11–20 | New pose; leftover silhouette fills with blotches | | character, 0.9 | 11–18 | Larger gestures, breaks the keyframe pose | | **character, 0.8** | 9–15 | Faithful pose; expression and small hand motion | | character, 0.6 | 9–14 | Facial expression only | | character, 0.4 | 8–13 | Blinks and breathing (cinemagraph) | | **face, 0.8** | 0.6–1.8 | Expression only (laughs, blinks, winks); nothing else moves | ![Character animated, background frozen](examples/animate_character_frozen_background.jpg) *Character matte, denoise 0.8. The first column is the keyframe.* ![Face only](examples/animate_face_only.jpg) *Face matte, denoise 0.8.* ### Inverted: freeze the character, let the world flow The same matte, inverted: the character is frozen and the background is redrawn with an AnimateDiff-like flow (waves, moon, sky). Two things matter: - **Run the background pass without the charref and without character tags** (`no humans` + scene). With the reference, the redrawn area grows a giant copy of the character. - **Use a tight, soft mask** (grow 4, blur 8). With grow 16 a ring of the old background stays stuck around the character. ![Night beach](examples/frozen_character_night_beach.jpg) ![Soft edge vs halo](examples/soft_edge_vs_halo.jpg) *Left: grow 16 (halo). Right: grow 4 + blur 8.* ## Usage (ComfyUI) **Custom nodes** (in `ComfyUI/custom_nodes/`): - **Anima Motion Module**: copy the `ComfyUI-Anima-Motion/` folder from this repo. - **Anima Ref Adapter**: `git clone https://github.com/ArcAnima/ComfyUI-Anima-Ref-Adapter` - **WeVisOne-VL**, my ComfyUI port of Tencent WeVisOne, which the workflow uses for the selection mask: `GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL` (skips the weights stored in that repo; download one file below). **Models:** | File | Folder | From | |---|---|---| | `anima_motion_r2v_v0.1.safetensors` | `models/anima_motion/` | this repo | | `wevisone_vl_4b_int8_convrot.safetensors` (5.7 GB) or `wevisone_vl_4b_bf16.safetensors` (9.7 GB) | `models/text_encoders/` | [paralaif/ComfyUI-WeVisOne-VL](https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL) | | `anima_ref_adapter_v2_r64.safetensors` | `models/loras/` | [ElectronGirl/anima-ref-adapter](https://huggingface.co/ElectronGirl/anima-ref-adapter) (the node also downloads it on first use) | | `split_files/diffusion_models/anima-turbo-v1.0.safetensors` | `models/diffusion_models/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) | | `split_files/text_encoders/qwen_3_06b_base.safetensors` | `models/text_encoders/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) | | `split_files/vae/qwen_image_vae.safetensors` | `models/vae/` | [circlestone-labs/Anima](https://huggingface.co/circlestone-labs/Anima) | Then load `workflows/anima_freeze_animate.json`. The same download list is in the workflow's note. **Workflow controls** - **Keyframe (two switches):** - **generate with the character sheet**: Anima + charref; the sheet is the video's reference; - **text only (T2V)**: plain Anima; the generated keyframe becomes the video's charref reference; - **your own image**: it keeps its aspect ratio (scaled to the Megapixels value, sides in multiples of 16) and is the video's charref reference. - **query:** what to select (`character`, `face`, `sky`, `sea`, `hair`…). - **Freeze/animate switch:** - **true**: freeze the selection and animate the rest, with no charref and the *THE REST* prompt. - **false**: animate only the selection, with charref and the *THE SELECTION* prompt. - **denoise** (video KSampler): 0.4 breathes, **0.8 recommended**, 1.0 mutates. - With your own image, describe its background in the *THE REST* prompt, or the background turns into whatever that prompt says. ![Workflow, both modes](examples/workflow_freeze_vs_animate.jpg) *Top: switch true (frozen character). Bottom: switch false (frozen background).* Minimal graph without masks: `Anima Ref Adapter (v2, ref_guidance 1)` → `Anima Motion Module` → `KSampler` on a 16-frame latent → `LatentCutToBatch (t, 1)` → `VAEDecode` → `CreateVideo (8 fps)`. ## Limitations - **20 training clips.** Motion is small (gestures, expressions). The module memorizes frequent objects: a book from the data shows up in early checkpoints and in some unmasked generations. - Trained at 384×576 portrait. Other sizes work (the 16:9 beach is 832×480) but are untested for quality. - It never saw landscapes moving, so in the inverted mode the background **morphs** (AnimateDiff-like) instead of moving physically. - The R2V module was trained without a keyframe, so without a mask it does not start from your image. - Each frame is decoded as an independent image: some fine-line shimmer remains. - 16 frames (2 s at 8 fps). Longer clips are untested. ## Next plans 1. **More data: about 300 clips.** About 200 characters with varied gestures and **about 100 fixed-camera landscapes** with ambient motion (waves, clouds, lanterns, falling petals), all from Anima keyframes animated with MiniMax H3. 2. **Keyframe conditioning in R2V training.** A clean frame 0 in about 70–90% of the steps, so the module learns to continue an existing image instead of redrawing it. 3. **Train the way it is used.** Start some samples from a noised keyframe (partial denoise) and with spatial masks, to match the masked workflow. 4. **Bigger run on a cloud GPU** (L40S, batch 4, 3000–5000 steps). Mixed batches: character clips with the reference, landscape clips without it. 5. **Longer and wider clips**: 32 frames (4 s) and 16:9 / 9:16 buckets. 6. Re-check frame-to-frame stability, keyframe adherence and object memorization against this v0.1. ## License and credits - **License:** the [CircleStone Labs Non-Commercial License](https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md), as for Anima, since the module is trained on and runs with Anima. Training clips were generated with MiniMax H3; check its terms for your use. - [Anima](https://huggingface.co/circlestone-labs/Anima) by CircleStone Labs. - [Anima Ref Adapter](https://huggingface.co/ElectronGirl/anima-ref-adapter) by ElectronGirl. - WeVisOne-VL by Tencent (ComfyUI port: [paralaif/ComfyUI-WeVisOne-VL](https://huggingface.co/paralaif/ComfyUI-WeVisOne-VL)). - Method inspired by [AnimateDiff](https://github.com/guoyww/AnimateDiff).