| --- |
| license: mit |
| pipeline_tag: image-to-video |
| library_name: diffusers |
| tags: |
| - speech-to-video |
| - avatar |
| - video-generation |
| - kandinsky |
| --- |
| |
| # Kandinsky S2V 1.0 Pro — Diffusers format |
|
|
| <p> |
| <a href="https://arxiv.org/pdf/2610.19290"><img src="https://img.shields.io/badge/arXiv-2610.19290-b31b1b" alt="arXiv"></a> |
| <a href="https://kandinskyavatar.github.io"><img src="https://img.shields.io/badge/Project-Page-green" alt="Project Page"></a> |
| <a href="https://github.com/kandinskylab/kandinsky-s2v"><img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"></a> |
| <a href="https://huggingface.co/collections/kandinskylab/kandinsky-s2v-10-6ab65ebe8f765b6dd9d706d7"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue" alt="Hugging Face"></a> |
| <a href="https://huggingface.co/spaces/kandinskylab/Kandinsky-S2V-1.0-Demo"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo-yellow" alt="Demo"></a> |
| </p> |
|
|
| Speech-to-video generation: an avatar video is produced from a **reference |
| frame** and a **speech track**. The whole figure is animated, not only the |
| head. The audio is a *condition* — the model does not generate sound, it |
| animates the person in the reference frame to match the speech it is given. |
|
|
| This repository holds the pro checkpoint laid out for |
| [Diffusers](https://github.com/huggingface/diffusers). The same weights in the |
| native layout live in |
| [`kandinskylab/Kandinsky-S2V-1.0-Pro`](https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Pro). |
|
|
| ## Contents |
|
|
| | Folder | Class | Size | |
| |---|---|---| |
| | `transformer/` | `KandinskyS2VTransformer3DModel` | 39.8 GiB | |
| | `vae/` | `AutoencoderKLHunyuanVideo` | 0.9 GiB | |
| | `text_encoder/` + `tokenizer/` | `Qwen2_5OmniThinkerForConditionalGeneration` | 16.6 GiB | |
| | `text_encoder_2/` + `tokenizer_2/` | `CLIPTextModel` | 0.5 GiB | |
| | `scheduler/` | `FlowMatchEulerDiscreteScheduler` | — | |
|
|
| The bundle is self-contained: the Thinker is exported without the Talker of the |
| full Omni checkpoint, and CLIP without its vision tower, since neither is used. |
|
|
| ## Usage |
|
|
| ```python |
| import librosa |
| import torch |
| from diffusers import ModularPipeline |
| from diffusers.utils import encode_video, load_image |
| |
| pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers") |
| pipe.load_components(dtype=torch.bfloat16) |
| pipe.to("cuda") |
| |
| image = load_image("reference.png") |
| audio, _ = librosa.load("speech.wav", sr=16000) # a mono waveform at 16 kHz, for the model |
| speech, sample_rate = librosa.load("speech.wav", sr=None) # the original track, added to the output video |
| |
| video = pipe(image=image, audio=audio, output="videos")[0] |
| encode_video( |
| video, |
| fps=25, |
| output_path="avatar.mp4", |
| audio=torch.from_numpy(speech)[None], |
| audio_sample_rate=sample_rate, |
| ) |
| ``` |
|
|
| Pro does not fit an 80 GB card with every component resident: beside the 7B speech encoder the VAE |
| decode of a chunk has nowhere to go. Offload through a `ComponentsManager` instead of `pipe.to("cuda")`; |
| the margin keeps room for the VAE decode of a chunk, which brings the peak to about 50 GiB: |
|
|
| ```python |
| from diffusers import ComponentsManager |
| |
| manager = ComponentsManager() |
| pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers", components_manager=manager) |
| pipe.load_components(dtype=torch.bfloat16) |
| manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="40GiB") |
| ``` |
|
|
| The pipeline and its blocks come from `modular_model_index.json`. Classifier-free guidance is the `guider` component, not a call argument. |
|
|
| **Kandinsky S2V 1.0 is not part of a released Diffusers version yet**; it lands as a modular pipeline. |
|
|
| ## Notes |
|
|
| Euler flow matching, 50 steps, classifier-free guidance 3.0. |
|
|
| Output is 125 frames per chunk at 25 fps (5 s) in one of the 512 buckets, chosen |
| from the reference frame's aspect ratio. Longer speech is generated chunk by |
| chunk, each conditioned on the previous one through FramePack memory. |
|
|
| ## Responsible use |
|
|
| The model animates a real person from a single photograph, driven by a |
| voice that need not be theirs. Use a reference frame you have the subject's |
| consent to animate, and do not present the result as a genuine recording of |
| anyone. |
|
|
