40fuku: a small distilled image-to-video model

Give it a picture and a sentence, and it makes a 480×832 video: 2 seconds in one go (4 denoising steps per frame), or up to 30 seconds by continuing in 1-second windows. Optionally it adds a matching soundtrack with MMAudio.

40fuku is a 722M-parameter denoiser, half the depth of its teacher. It was distilled from Causal Forcing's frame-wise image-to-video model (1.42B), which is itself built on Wan2.1-T2V-1.3B. Everything here is Apache-2.0.

It keeps your image and starts the right motion: camera push-ins and orbits, people and animals moving. Expect it to get smeary and melty as the clip goes on, and long clips tend to drift sideways. That's the price of half the layers, and honestly part of the fun.

Denoiser	722,386,240 params (bf16, 1.44 GB): 15 of the teacher's 30 Wan blocks (0,2,4,…,12,15,17,…,29)
Also needs	Wan2.1 VAE (0.5 GB) + umT5-XXL text encoder (11.4 GB), downloaded automatically from Wan-AI/Wan2.1-T2V-1.3B
Output	480×832 @ 16 fps: 33 frames (2 s, trained length) up to 481 frames (30 s) in re-anchored 1 s windows
Sound (optional)	MMAudio large_44k_v2, 44.1 kHz; weights CC-BY-NC 4.0 (non-commercial)
Speed (2 s clip)	~2 s on an H100, ~10 s on an RTX 4070, ~25 s on a T4; each extra second ~5 s on a 4070 (plus text encoding for a new prompt)
GPU memory	~6.5 GB peak with the text encoder on CPU; runs on 8–12 GB cards. Sound: ~6 GB more (fits on a 12 GB card or a T4)

Quick start

# pip install diffusers==0.31.0 transformers omegaconf einops ftfy easydict imageio imageio-ffmpeg sentencepiece safetensors huggingface_hub
from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("fuku40", hf_hub_download("Kompaktion/40fuku", "fuku40.py"))
fuku40 = importlib.util.module_from_spec(spec); sys.modules["fuku40"] = fuku40; spec.loader.exec_module(fuku40)

pipe = fuku40.load()      # first run downloads ~13.5 GB and clones the Causal-Forcing code (~10 MB)
pipe.generate("my_photo.jpg", "a dog running toward the camera, cinematic", out="dog.mp4", seed=0)
pipe.generate("my_photo.jpg", "a dog running toward the camera, barking", out="dog5s.mp4", frames=81, sound=True)  # 5 s + sound

Command line: python fuku40.py --image my_photo.jpg --prompt "..." --frames 81 --sound --out out.mp4

• Length: frames=33 (2 s) is one window, exactly as trained. More frames continue in windows: each new window re-encodes a recent frame as its starting image, carries over the last 4 latent frames, and adds 1 s, so the model never sees more than its training length of 9 latent frames. It keeps going, but detail degrades the longer it runs.

• Sound (optional): sound=True / --sound, or pipe.add_sound(video, prompt) on any clip. Install first: pip install --no-deps "git+https://github.com/hkchengrex/MMAudio.git" && pip install av open_clip_torch timm torchdiffeq scipy librosa colorlog. First use downloads ~9 GB. MMAudio's weights are CC-BY-NC 4.0, so clips with sound are for non-commercial use.

• Text encoder modes (text_mode= / --text-mode, picked automatically from free VRAM): gpu (40+ GB cards), reload (16 GB cards: Colab/Kaggle T4; loads per new prompt), cpu (≤12 GB cards, ~1–2 min per new prompt on the CPU; generation itself stays on the GPU).

• Images are center-cropped to landscape 832×480, the only size it was trained on.

• Set HF_HOME / FUKU40_HOME to put the downloads on a drive with space.

Notebook (Colab / Kaggle): 40fuku_colab.ipynb in this repo. Demo Space: Kompaktion/40fuku-demo.

How it was made (≈ $30 of Modal GPU time)

1. Teacher data. 560 original prompts (no scraped prompt lists). The teacher draws a first frame from each prompt, then makes a 33-frame image-to-video clip. Every one of its 32 denoising calls is recorded. 48 clips are held out, using subjects that never appear in training.
2. Student. The teacher's own blocks 0,2,4,…,29 (15 of 30) with the same width, VAE and text conditioning. Untrained, it turns every video into colored static.
3. Distillation. Teacher-forced output regression: the student sees the teacher's noisy state and the clean earlier frames, and predicts the teacher's denoised output. Round 1 was 2,842 steps (mostly 5-frame clips, lr 5e-5). Round 2 added 1,003 steps on mostly full 9-latent-frame clips at lr 2e-5.
4. Learning from its own mistakes (round 3). Rounds 1–2 only ever saw the teacher's clean frames as context, so at inference its small errors piled up. Round 3 (DAgger-style) let 40fuku generate 256 clips of its own, many of them several windows long, and had the teacher label what each of its frames should have been given 40fuku's own drifted context. 741 steps (60% on these self-rollouts, 40% on the original teacher data, lr 2e-5). The released weights are round 3's EMA.

Held-out results (48 unseen images; every arm uses the same image and noise; errors are measured against the teacher's video):

	latent MSE ↓	PSNR vs teacher ↑	motion
40fuku (round 3, released)	0.534	12.6 dB	0.043
40fuku round 2 (same run)	0.568	12.3 dB	0.043
40fuku round 1	0.569	12.4 dB	0.041
untrained 15-block copy	1.82	7.5 dB	noise
frozen input image	0.76	11.0 dB	0
teacher (1.42B)	n/a	n/a	0.057

40fuku beats a frozen image, so its motion really follows the teacher. It moves a bit less than the teacher, and it loses detail late in the clip. (Rounds differ by ±0.02 MSE between runs, because the sampler's re-noising is random.) On 6-second clips of 8 held-out images, round 3 visibly keeps more of the scene (horizons, trees, textures) where round 2 dissolves into blur, though both still drift sideways.

Limits

• Not teacher quality: later frames blur, warp and melt. Faces and hands suffer the most.
• Landscape 832×480 only. Long clips keep degrading and tend to pan sideways; past ~10 s it is mostly abstract art.
• Prompt-following is limited: the 1.42B teacher itself often ignores the action you ask for.
• English prompts worked best (umT5 is multilingual, but the training prompts were English).
• It inherits the teacher's biases. Don't use it to make deceptive content about real people.

Credits & license

Apache-2.0. Built on Wan2.1 (Alibaba, Apache-2.0) and Causal Forcing (THU-ML, Apache-2.0). The teacher checkpoint is zhuhz22/Causal-Forcing framewise/causal_forcing.pt. Optional sound uses MMAudio (code MIT, weights CC-BY-NC 4.0), which is not bundled here. 40fuku was made by Claude (Anthropic) and Codex (OpenAI) for Kompaktion.