---
license: mit
pipeline_tag: image-to-video
library_name: diffusers
tags:
- speech-to-video
- avatar
- video-generation
- kandinsky
---

# Kandinsky S2V 1.0 Pro — Diffusers format

<p>
  <a href="https://arxiv.org/pdf/2610.19290"><img src="https://img.shields.io/badge/arXiv-2610.19290-b31b1b" alt="arXiv"></a>
  <a href="https://kandinskyavatar.github.io"><img src="https://img.shields.io/badge/Project-Page-green" alt="Project Page"></a>
  <a href="https://github.com/kandinskylab/kandinsky-s2v"><img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"></a>
  <a href="https://huggingface.co/collections/kandinskylab/kandinsky-s2v-10-6ab65ebe8f765b6dd9d706d7"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue" alt="Hugging Face"></a>
  <a href="https://huggingface.co/spaces/kandinskylab/Kandinsky-S2V-1.0-Demo"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo-yellow" alt="Demo"></a>
</p>

Speech-to-video generation: an avatar video is produced from a **reference
frame** and a **speech track**. The whole figure is animated, not only the
head. The audio is a *condition* — the model does not generate sound, it
animates the person in the reference frame to match the speech it is given.

This repository holds the pro checkpoint laid out for
[Diffusers](https://github.com/huggingface/diffusers). The same weights in the
native layout live in
[`kandinskylab/Kandinsky-S2V-1.0-Pro`](https://huggingface.co/kandinskylab/Kandinsky-S2V-1.0-Pro).

## Contents

| Folder | Class | Size |
|---|---|---|
| `transformer/` | `KandinskyS2VTransformer3DModel` | 39.8 GiB |
| `vae/` | `AutoencoderKLHunyuanVideo` | 0.9 GiB |
| `text_encoder/` + `tokenizer/` | `Qwen2_5OmniThinkerForConditionalGeneration` | 16.6 GiB |
| `text_encoder_2/` + `tokenizer_2/` | `CLIPTextModel` | 0.5 GiB |
| `scheduler/` | `FlowMatchEulerDiscreteScheduler` | — |

The bundle is self-contained: the Thinker is exported without the Talker of the
full Omni checkpoint, and CLIP without its vision tower, since neither is used.

## Usage

```python
import librosa
import torch
from diffusers import ModularPipeline
from diffusers.utils import encode_video, load_image

pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("reference.png")
audio, _ = librosa.load("speech.wav", sr=16000)  # a mono waveform at 16 kHz, for the model
speech, sample_rate = librosa.load("speech.wav", sr=None)  # the original track, added to the output video

video = pipe(image=image, audio=audio, output="videos")[0]
encode_video(
    video,
    fps=25,
    output_path="avatar.mp4",
    audio=torch.from_numpy(speech)[None],
    audio_sample_rate=sample_rate,
)
```

Pro does not fit an 80 GB card with every component resident: beside the 7B speech encoder the VAE
decode of a chunk has nowhere to go. Offload through a `ComponentsManager` instead of `pipe.to("cuda")`;
the margin keeps room for the VAE decode of a chunk, which brings the peak to about 50 GiB:

```python
from diffusers import ComponentsManager

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers", components_manager=manager)
pipe.load_components(dtype=torch.bfloat16)
manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="40GiB")
```

The pipeline and its blocks come from `modular_model_index.json`. Classifier-free guidance is the `guider` component, not a call argument.

**Kandinsky S2V 1.0 is not part of a released Diffusers version yet**; it lands as a modular pipeline.

## Notes

Euler flow matching, 50 steps, classifier-free guidance 3.0.

Output is 125 frames per chunk at 25 fps (5 s) in one of the 512 buckets, chosen
from the reference frame's aspect ratio. Longer speech is generated chunk by
chunk, each conditioned on the previous one through FramePack memory.

## Responsible use

The model animates a real person from a single photograph, driven by a
voice that need not be theirs. Use a reference frame you have the subject's
consent to animate, and do not present the result as a genuine recording of
anyone.