Model card

README.md

@@ -0,0 +1,192 @@
+---
+license: mit
+library_name: diffusers
+tags:
+- video
+- video-generation
+- text-to-video
+- image-to-video
+- audio-generation
+- kandinsky
+- diffusers
+---
+
+> Mirror of [kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers). All credit to the original authors; license unchanged.
+
+
+<div align="center">
+  <img src="https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers/resolve/main/assets/promo.webp" width="100%">
+</div>
+
+<br>
+
+<div align="center">
+<img src="https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers/resolve/main/assets/Kandinsky_LOGO_6_Horizontal_black.png" width="80%">
+</div>
+
+
+<div align="center">
+
+<a href="https://kandinskylab.ai/models/video/"><img alt="KandinskyLab" src="https://img.shields.io/badge/KandinskyLab-76E0B7?style=for-the-badge"></a>
+<a href="https://arxiv.org/pdf/2610.05608"><img alt="Report" src="https://img.shields.io/badge/Report-9C2731?style=for-the-badge"></a>
+<a href="#"><img alt="PyPI" src="https://img.shields.io/badge/PyPI-3171B2?style=for-the-badge"></a>
+<a href="https://huggingface.co/spaces/kandinskylab/Kandinsky-6.0-Pro-distill-5s"><img alt="HF Demo" src="https://img.shields.io/badge/HF%20Demo-D9622B?style=for-the-badge"></a>
+<a href="https://huggingface.co/docs/diffusers/main/en/api/pipelines/kandinsky6"><img alt="Diffusers" src="https://img.shields.io/badge/Diffusers-F8D44E?style=for-the-badge"></a>
+<a href="https://docs.vllm.ai/projects/vllm-omni/en/latest/api/vllm_omni/diffusion/models/kandinsky6/"><img alt="vLLM" src="https://img.shields.io/badge/vLLM-54A0F8?style=for-the-badge"></a>
+<a href="https://docs.sglang.io/cookbook/diffusion/Kandinsky/Kandinsky6"><img alt="SGLang" src="https://img.shields.io/badge/SGLang-C6602D?style=for-the-badge"></a>
+<a href="https://haoailab.com/FastVideo/inference/kandinsky6/"><img alt="FastVideo" src="https://img.shields.io/badge/FastVideo-436BF6?style=for-the-badge"></a>
+<a href="https://registry.comfy.org/nodes/kandinsky6"><img alt="ComfyUI" src="https://img.shields.io/badge/ComfyUI-EBF764?style=for-the-badge"></a>
+
+</div>
+
+<h1>Kandinsky 6.0: A family of diffusion models for Video + Audio generation</h1>
+
+We present Kandinsky 6.0 Video, a family of foundation diffusion models for synchronized text-to-audio-video generation, comprising Kandinsky 6.0 Video Lite (3B parameters) and Kandinsky 6.0 Video Pro (29B parameters). Both models generate 5-second video clips with synchronized 44 kHz audio, including lip-sync, in text-to-audio-video (T2AV) and image-to-audio-video (TI2AV) modes; a plug-in super-resolution model raises the output resolution to Full-HD (1920×1080).
+
+## Usage guide
+
+This repository contains **Kandinsky 6.0 Pro**: the main Pro checkpoint, the strongest quality setting of the Pro line.
+
+### Text-to-audio-video (T2AV)
+
+```python
+import torch
+from diffusers import Kandinsky6TI2VAPipeline
+from diffusers.utils import encode_video
+
+DEFAULT_PROMPT = (
+    "cinematic shot: a giant stone samurai on a stormy cliff above a neon city opens glowing golden eyes and "
+    "raises a katana. Blue lightning strikes the blade, creating a massive shockwave through the clouds. The "
+    "camera rapidly pulls back from a low angle. Photorealistic, epic scale, dark blue and gold lighting, rain, "
+    "sparks, volumetric lightning, blockbuster quality. Audio: heavy rain, deep thunder, metallic sword hum, "
+    "rising brass and choir, electrical crackle, perfectly synchronized lightning impact, sub-bass shockwave. "
+    "No dialogue, text, or logos."
+)
+
+pipe = Kandinsky6TI2VAPipeline.from_pretrained(
+    "kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers", torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+
+output = pipe(
+    prompt=DEFAULT_PROMPT,
+    height=480,
+    width=864,
+    num_frames=121,
+    num_inference_steps=50,
+    guidance_scale=5.0,
+)
+encode_video(
+    output.frames[0],
+    fps=24,
+    output_path="output.mp4",
+    audio=output.audio[0][None],
+    audio_sample_rate=pipe.audio_sample_rate,
+)
+```
+
+### Image-to-audio-video (TI2AV)
+
+Pass a reference image with `image=`. It conditions the first frame, and the pipeline resizes and center-crops it to `height` × `width`. The prompt should describe what happens next in that image, so the example uses its own prompt rather than `DEFAULT_PROMPT`. It reuses `pipe` from the T2AV example.
+
+```python
+from diffusers.utils import load_image
+
+IMAGE_PROMPT = (
+    "cinematic shot: a red dragon wakes on a mound of gold coins in a sunlit cave, slowly unfolds its wings "
+    "and lowers its head toward the glowing stream below. Coins slide and clink down the pile as the dragon "
+    "shifts its weight. The camera slowly pushes in toward the dragon's eyes. Photorealistic, fantasy, warm "
+    "golden light, drifting dust motes, lush cave flowers. Audio: deep rumbling breath, leathery wing flaps, "
+    "clinking coins, trickling water, distant echoing roar. No dialogue, text, or logos."
+)
+
+image = load_image(
+    "https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers/resolve/main/assets/i2va_input.jpg"
+)
+
+output = pipe(
+    prompt=IMAGE_PROMPT,
+    image=image,
+    height=480,
+    width=864,
+    num_frames=121,
+    num_inference_steps=50,
+    guidance_scale=5.0,
+)
+encode_video(
+    output.frames[0],
+    fps=24,
+    output_path="output_ti2av.mp4",
+    audio=output.audio[0][None],
+    audio_sample_rate=pipe.audio_sample_rate,
+)
+```
+
+### Options
+
+Pass `sample_audio=False` to generate video only, and `expand_prompts=True` to let the Qwen2.5-VL text encoder rewrite a short prompt into a detailed one before encoding. `expand_prompts` adds latency but no extra model weights.
+
+
+## Super Res guide
+
+Kandinsky 6.0 video super-resolution is a separate pipeline, `Kandinsky6SRPipeline`. It takes the frames produced above and upscales them by ×2, ×2.25 or ×4 with tiled diffusion in K-VAE latent space, blending overlapping tiles back with Hann windows. The example uses the flow-matching checkpoint, which runs in 4 steps per tile.
+
+```python
+import torch
+from diffusers import Kandinsky6SRPipeline
+from diffusers.utils import encode_video
+
+torch._inductor.config.max_autotune = True  # required: lets inductor pick flex-attention tiles that fit the SR block mask
+
+sr_pipe = Kandinsky6SRPipeline.from_pretrained(
+    "kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers", torch_dtype=torch.bfloat16
+)
+sr_pipe.transformer.set_attention_backend("flex")
+sr_pipe.enable_model_cpu_offload()
+sr_pipe.transformer.compile_repeated_blocks(fullgraph=True)
+
+upscaled = sr_pipe(
+    video=output.frames[0],
+    resolution_scale=2.25,  # 2, 2.25 or 4
+    num_inference_steps=4,  # 2 with Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers
+).frames[0]
+encode_video(
+    upscaled,
+    fps=24,
+    output_path="output_sr.mp4",
+    audio=output.audio[0][None],
+    audio_sample_rate=pipe.audio_sample_rate,
+)
+```
+
+The example continues from the Usage guide: `pipe` and `output` come from it. The SR pipeline works on frames only, so the audio is passed to `encode_video` from the generation step.
+
+- `resolution_scale=2.25` (the default, and the route used after Kandinsky 6 generation) is a ×1.125 bilinear pre-upscale followed by the ×2 path. `2` and `4` are the other options.
+- `min_overlap` (default `0.2`) sets the minimum tile overlap. `tiles_batch_size` (default `1`) sets how many tiles go through the model per call, trading VRAM for speed.
+
+## Components
+
+| Folder | Class | Role |
+|---|---|---|
+| `transformer/` | `Kandinsky6Transformer3DModel` | Multimodal video+audio diffusion transformer (Pro) |
+| `vae/` | `AutoencoderKLHunyuanVideo` | Video VAE: encodes the reference image, decodes the generated video |
+| `text_encoder/` + `tokenizer/` | `Qwen2_5_VLForConditionalGeneration` + `Qwen2_5_VLProcessor` | Token-level text embeddings; also used for `expand_prompts=True` |
+| `text_encoder_2/` + `tokenizer_2/` | `CLIPTextModel` + `CLIPTokenizer` | Pooled text embedding |
+| `audio_vae/` | `MMAudioVAE` | Audio latents ↔ mel spectrogram |
+| `vocoder/` | `MMAudioVocoder` (BigVGAN-style) | Mel spectrogram → waveform |
+| `scheduler/` | `FlowMatchEulerDiscreteScheduler` | Flow matching, `shift = 5.0` |
+
+`model_index.json` names the pipeline class `Kandinsky6TI2VAPipeline`.
+
+### Kandinsky 6.0 family
+
+| Checkpoint | Repository | Use |
+|---|---|---|
+| Pro (this repository) | [kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers) | generation |
+| Pro, distilled, 10 steps | [kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers) | generation |
+| Pro, pretrained | [kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-pretrain-5s-Diffusers) | generation |
+| Lite | [kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-5s-Diffusers) | generation |
+| Lite, distilled, 10 steps | [kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers) | generation |
+| Lite, pretrained | [kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-pretrain-5s-Diffusers) | generation |
+| VSR, flow matching, 4 steps/tile | [kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers) | upscale |
+| VSR, distilled, 2 steps/tile | [kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers) | upscale |