--- license: mit library_name: diffusers tags: - video - video-generation - text-to-video - image-to-video - audio-generation - kandinsky - diffusers --- > Mirror of [kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers). All credit to the original authors; license unchanged.

KandinskyLab Report PyPI HF Demo Diffusers vLLM SGLang FastVideo ComfyUI

Kandinsky 6.0: A family of diffusion models for Video + Audio generation

We present Kandinsky 6.0 Video, a family of foundation diffusion models for synchronized text-to-audio-video generation, comprising Kandinsky 6.0 Video Lite (3B parameters) and Kandinsky 6.0 Video Pro (29B parameters). Both models generate 5-second video clips with synchronized 44 kHz audio, including lip-sync, in text-to-audio-video (T2AV) and image-to-audio-video (TI2AV) modes; a plug-in super-resolution model raises the output resolution to Full-HD (1920×1080). ## Usage guide This repository contains **Kandinsky 6.0 Pro**: the main Pro checkpoint, the strongest quality setting of the Pro line. ### Text-to-audio-video (T2AV) ```python import torch from diffusers import Kandinsky6TI2VAPipeline from diffusers.utils import encode_video DEFAULT_PROMPT = ( "cinematic shot: a giant stone samurai on a stormy cliff above a neon city opens glowing golden eyes and " "raises a katana. Blue lightning strikes the blade, creating a massive shockwave through the clouds. The " "camera rapidly pulls back from a low angle. Photorealistic, epic scale, dark blue and gold lighting, rain, " "sparks, volumetric lightning, blockbuster quality. Audio: heavy rain, deep thunder, metallic sword hum, " "rising brass and choir, electrical crackle, perfectly synchronized lightning impact, sub-bass shockwave. " "No dialogue, text, or logos." ) pipe = Kandinsky6TI2VAPipeline.from_pretrained( "kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers", torch_dtype=torch.bfloat16 ) pipe.enable_model_cpu_offload() output = pipe( prompt=DEFAULT_PROMPT, height=480, width=864, num_frames=121, num_inference_steps=50, guidance_scale=5.0, ) encode_video( output.frames[0], fps=24, output_path="output.mp4", audio=output.audio[0][None], audio_sample_rate=pipe.audio_sample_rate, ) ``` ### Image-to-audio-video (TI2AV) Pass a reference image with `image=`. It conditions the first frame, and the pipeline resizes and center-crops it to `height` × `width`. The prompt should describe what happens next in that image, so the example uses its own prompt rather than `DEFAULT_PROMPT`. It reuses `pipe` from the T2AV example. ```python from diffusers.utils import load_image IMAGE_PROMPT = ( "cinematic shot: a red dragon wakes on a mound of gold coins in a sunlit cave, slowly unfolds its wings " "and lowers its head toward the glowing stream below. Coins slide and clink down the pile as the dragon " "shifts its weight. The camera slowly pushes in toward the dragon's eyes. Photorealistic, fantasy, warm " "golden light, drifting dust motes, lush cave flowers. Audio: deep rumbling breath, leathery wing flaps, " "clinking coins, trickling water, distant echoing roar. No dialogue, text, or logos." ) image = load_image( "https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers/resolve/main/assets/i2va_input.jpg" ) output = pipe( prompt=IMAGE_PROMPT, image=image, height=480, width=864, num_frames=121, num_inference_steps=50, guidance_scale=5.0, ) encode_video( output.frames[0], fps=24, output_path="output_ti2av.mp4", audio=output.audio[0][None], audio_sample_rate=pipe.audio_sample_rate, ) ``` ### Options Pass `sample_audio=False` to generate video only, and `expand_prompts=True` to let the Qwen2.5-VL text encoder rewrite a short prompt into a detailed one before encoding. `expand_prompts` adds latency but no extra model weights. ## Super Res guide Kandinsky 6.0 video super-resolution is a separate pipeline, `Kandinsky6SRPipeline`. It takes the frames produced above and upscales them by ×2, ×2.25 or ×4 with tiled diffusion in K-VAE latent space, blending overlapping tiles back with Hann windows. The example uses the flow-matching checkpoint, which runs in 4 steps per tile. ```python import torch from diffusers import Kandinsky6SRPipeline from diffusers.utils import encode_video torch._inductor.config.max_autotune = True # required: lets inductor pick flex-attention tiles that fit the SR block mask sr_pipe = Kandinsky6SRPipeline.from_pretrained( "kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers", torch_dtype=torch.bfloat16 ) sr_pipe.transformer.set_attention_backend("flex") sr_pipe.enable_model_cpu_offload() sr_pipe.transformer.compile_repeated_blocks(fullgraph=True) upscaled = sr_pipe( video=output.frames[0], resolution_scale=2.25, # 2, 2.25 or 4 num_inference_steps=4, # 2 with Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers ).frames[0] encode_video( upscaled, fps=24, output_path="output_sr.mp4", audio=output.audio[0][None], audio_sample_rate=pipe.audio_sample_rate, ) ``` The example continues from the Usage guide: `pipe` and `output` come from it. The SR pipeline works on frames only, so the audio is passed to `encode_video` from the generation step. - `resolution_scale=2.25` (the default, and the route used after Kandinsky 6 generation) is a ×1.125 bilinear pre-upscale followed by the ×2 path. `2` and `4` are the other options. - `min_overlap` (default `0.2`) sets the minimum tile overlap. `tiles_batch_size` (default `1`) sets how many tiles go through the model per call, trading VRAM for speed. ## Components | Folder | Class | Role | |---|---|---| | `transformer/` | `Kandinsky6Transformer3DModel` | Multimodal video+audio diffusion transformer (Pro) | | `vae/` | `AutoencoderKLHunyuanVideo` | Video VAE: encodes the reference image, decodes the generated video | | `text_encoder/` + `tokenizer/` | `Qwen2_5_VLForConditionalGeneration` + `Qwen2_5_VLProcessor` | Token-level text embeddings; also used for `expand_prompts=True` | | `text_encoder_2/` + `tokenizer_2/` | `CLIPTextModel` + `CLIPTokenizer` | Pooled text embedding | | `audio_vae/` | `MMAudioVAE` | Audio latents ↔ mel spectrogram | | `vocoder/` | `MMAudioVocoder` (BigVGAN-style) | Mel spectrogram → waveform | | `scheduler/` | `FlowMatchEulerDiscreteScheduler` | Flow matching, `shift = 5.0` | `model_index.json` names the pipeline class `Kandinsky6TI2VAPipeline`. ### Kandinsky 6.0 family | Checkpoint | Repository | Use | |---|---|---| | Pro (this repository) | [kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers) | generation | | Pro, distilled, 10 steps | [kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers) | generation | | Pro, pretrained | [kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers) | generation | | Lite | [kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers) | generation | | Lite, distilled, 10 steps | [kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers) | generation | | Lite, pretrained | [kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers) | generation | | VSR, flow matching, 4 steps/tile | [kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers) | upscale | | VSR, distilled, 2 steps/tile | [kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers) | upscale |